A Fine Slice Of SQL Server
Tomaz Kastrun starts winding down a series on Apache Spark. Part 22 covers Spark in Azure Databricks:
Azure Databricks is a platform build on top of Spark based analytical engine, that unifies data, data manipulation, analytics and machine learning.
Databricks uses notebooks to tackle all the tasks and is therefore made easy to collaborate. Let’s dig in and start using a Python API on top of Spark API.
Read on for that primer.
Comments closedWolfgang Strasser digs up some documents:
Today, I stumbled upon a very interesting link – the Azure Synapse Analytics – Success by Design site (follow this link).
If you need guidance, best practices links, POC playbooks, links to blogs & videos, tools, .. THIS is the site you need to bookmark.
Click through for a bit more information, as well as links to other relevant Azure Synapse Analytics resources.
Comments closedCatalin Cimpanu reports on a security problem:
Microsoft has notified earlier this month a select group of Azure customers impacted by a recently discovered bug that exposed the source code of their Azure web apps since at least September 2017.
The vulnerability was discovered by cloud security firm Wiz and reported to Microsoft in September. The issue was fixed in November, and Microsoft has spent the last few weeks investigating how many customers were impacted.
Click through for the full report, with the upshot that if you run Azure App Services on Linux, you were probably affected.
Comments closedGarland MacNeill finds one way to solve a problem:
Anyway, it’s been a while since I worked on this AG and I need to get the migration/upgrade done. As I was working on configuring jobs, I ran into a problem where the AG node (as a target node) wasn’t downloading jobs from the Master node, in fact, the last poll was in July.
When I tried to force a poll, I was met with an error message that the server wasn’t registered, never mind it was clearly listed as a target server. Google didn’t find anything useful, other than some questions from 2013. I did come across the syntax to forcefully eject the server as a target with SQL.
Read on to see how and what to do in the aftermath.
Comments closedChris Webb points out something of interest:
If you’re using Power BI in DirectQuery mode against a SQL Server-related data source (ie SQL Server on-prem, Azure SQL DB or Synapse) you may have noticed a new feature that was released a few weeks ago: query labels that allow you to link a SQL query to the Power BI visual that generated it.
There’s nothing you need to do to enable it – it happens automatically. Here’s an example of a TSQL query generated by a DirectQuery mode dataset with it in:
Though I’m not sure if on-prem SQL Server or even Azure SQL Database has a nice way of viewing data by label. I know that dedicated SQL pools do and they’re a rather helpful method for figuring out who (or what) is doing work.
Comments closedBill Fellows works from an…interesting…SSIS package:
Our job was to rewrite this into something more manageable and it appears Azure Data Factory will be the winner. Before we can do that, we need to document what the existing package is doing (the vendor has supplied the incremental load logic) so we can replicate it but in a more economical form. It appears to have the pattern (squint really hard at the picture) Execute SQL Task -> Execute SQL Task -> Sequence container => many data flows -> Data Flow -> Execute SQL Task. The Data Flow Task is named after the table being loaded. An ODBC source with a expression based query, named “ODBC Source 1” wired to an OLE DB Destination, named “OLE DB Destination”. How would you do it, especially given that there are 236 Data Flow Tasks embedded in a single container?
Click through for the answer.
Comments closedJoe Obbish goes beyond the obvious reason:
Upon reading the title, you may be thinking that of course parallel scans will be slow in the cloud. Cloud storage storage simply isn’t very fast. I would argue that there’s a bit more to it.
Click through for a deep dive with some advice on what you might do to fix a specific (but common) scenario.
Comments closedTomaz Kastrun continues a series on Spark with a look at GraphX. Part 20 gives an overview of GraphX:
GraphX is Spark’s API component for graph and graph-parallel computations. GraphX uses Spark RDD and builds a graph abstraction on top of RDD. Graph abstraction is a directed multigraph with properties of edges and vertices.
Part 21 shows off the operators available:
Property graphs have collection of operators, that can take user-defined function and produce new graphs with transformed properties and structure. Core operators are defined in Graph and compositions of core operators are defined as GraphOps, and are automatically available as members of Graph. Each graph representation must provide implementations of the core operations and reuse many of the useful operations that are defined in GraphOps.
Click through for more information on graphs in the Spark ecosystem.
Comments closedAnna Wykes starts off with bad news:
In this blog series I explore a variety of options available for DevOps for Databricks. This blog will focus on working with the Databricks REST API & Python. Why you ask? Well, a large percentage of Databricks/Spark users are Python coders. In fact, in 2021 it was reported that 45% of Databricks users use Python as their language of choice. This is a stark contrast to 2013, in which 92 % of users were Scala coders:
What is wrong with the world today?
Semi-seriously, though, do read Anna’s post, as it covers a variety of things you can do with the Databricks REST API, including cluster management and monitoring. I might be jumping the gun a bit, but I am a big fan of Gerhard Brueckl’s Powershell module for Databricks for this kind of work.
Comments closedNot too long ago, I ended up taking the DP-203 certification exam for sundry reasons. On that exam, they ask a lot about Azure Synapse Analytics, including indexing, distribution, and partitioning strategies. Because these can be a bit different from on-premises SQL Server, I wanted to cover what options are available and when you might choose them. Let’s start with distributions, as that’s the biggest change in thought process.
Read on for the guidance.
Comments closed