Press "Enter" to skip to content

Day: March 1, 2024

Using Apache Spark in Microsoft Fabric

Published 2024-03-01 by Kevin Feasel

Ginger Grant gives us an overview of where we can use Apache Spark in Microsoft Fabric:

If you have used Spark in Azure Synapse, prepare to be pleasantly surprised with the compute experience in Microsoft Fabric as Spark compute starts a lot faster because the underlying technology has changed. The Data Engineering and Data Science Fabric experiences include a managed Spark compute, which like previous Spark compute charges you when it is in use. The difference is the nodes are reserved for you, rather than allocated when you start the compute which results in compute starting in 30 seconds or less versus the 4 minutes of waiting it takes for Azure Synapse compute to start.  If you have different capacity needs that a default managed Spark compute will not provide, you can always create a custom pool.  Custom pools are created in a specific workspace, so you will need Administrator permissions on the workspace to create them. You can choose to make the new pool your default pool as well, so it will be what starts in the workspace.

Read on for more of Ginger’s thoughts on the matter, including how you can use Copilot in Microsoft Fabric (if you pay for it) to help generate Spark code.

Comments closed

Piecemeal Database Restoration

Published 2024-03-01 by Kevin Feasel

Chad Callihan restores an elephant one bite at a time…or something:

The larger a database grows, the more difficult it becomes to restore it in a timely manner. When a database is young, you might be able to manage full restores in seconds. But as it matures and backup sizes go from megabytes to gigabytes to terabytes, those restore times will expand as well.

If you plan ahead, it’s not always a requirement to restore the entire database if only part of the database is necessary. This is where the idea of piecemeal restores can save you time and wasted effort.

I’ve always found piecemeal database restoration more of an interesting idea than something quite practical. The problem is, if your data is so easily separable that you can restore one set and not need the other for some reasonable length of time, why are they in the same database? I understand that there are reasonable answers to this question, but I also rarely see those scenarios pop up.

Comments closed

EXISTS and NOT EXISTS in T-SQL

Published 2024-03-01 by Kevin Feasel

Erik Darling plays Descartes in his spare time:

Let’s talk about two of the most overlooked and undervalued facilities in the SQL language: EXISTS and NOT EXISTS. Perhaps they’d get more traction is they were called THERE and NOT THERE, but but it would be perilously easy to start confusing your WHERE and THERE clause.

Often besmirched by someone who read a blog post about MySQL in 1998 as “subqueries, which are slower than joins”, they’re two of the most useful things you can grasp and use regularly.

This post does a great job of explaining when you’d want to use EXISTS and NOT EXISTS.

Comments closed

Get- cmdlets in dbatools Replication

Published 2024-03-01 by Kevin Feasel

Jess Pomfret continues a series on replication in dbatools:

This post will show off all the Get- commands that are available within dbatools for replication. When you’re using PowerShell, and especially if you’re new to PowerShell, exploring the Get- commands for a certain module, or area is a great way to get started. As it says in the name, these commands get information about something, they aren’t going to change anything, which means they are pretty safe to run in your environment. Of course, I’m always going to say, you should still run these in your test environment first to make sure you understand what they are doing, and how they behave in your specific environment.

Read on to see some of what you can do with it.

Comments closed

Dynamic Subscriptions in Power BI

Published 2024-03-01 by Kevin Feasel

Reza Rad sends an e-mail:

If you ever want to have a general report for sales for all countries, and then you want every morning the report to be sent (as PDF or PowerPoint alongside the link to the report) to representatives of each country, the report with the data filtered for their country only, then Dynamic Subscription in Power BI is the feature you need to use. Previously, this was called a Data-Driven Subscription in SSRS (SQL Server Reporting Services). In this article and video, I’ll explain how to use this feature using an example and what you need to know about it.

Click through for the video and blog post.

Comments closed

Postgres Foreign Data Wrappers and fdw_tuple_cost

Published 2024-03-01 by Kevin Feasel

Umair Shahid provides background info on a recent change:

“Why is DEFAULT_FDW_TUPLE_COST so insanely low?”

That was the subject of the email thread initiated by David Rowley to discuss this topic. I found the subject line amusing, accurate, and fully descriptive of the problem at hand. The discussion resulted in a commit that changed the default value from 0.01 to 0.2. Because I had previously written about postgres_fdw and also mentioned fdw_tuple_cost, I figured it would be good to go deeper into what this parameter is about and why it makes sense to default it to 0.2.

Read on to learn more about Foreign Data Wrappers (the Postgres equivalent to PolyBase in SQL Server), tuple costing, and more.

Comments closed

Microsoft Fabric and Semantic Models

Published 2024-03-01 by Kevin Feasel

Kurt Buhler has a choose-your-own-adventure story:

Semantic models are integral to Microsoft Fabric. They use and are used by many of the different workloads. In Fabric, there’s more items that can connect to and consume your model—such as semantic link in notebooks. Because of these new options and tools, your model is exposed to additional types of users who will use it in different ways. As such, it’s important that you make good models that you manage well throughout their entire lifecycle.

Read on for more information and three separate scenarios

Comments closed