A Fine Slice Of SQL Server
A handy function to delete all Fabric items in a workspace. Run this in a Python notebook in the workspace you want to delete items from. Everything, except that notebook, will be deleted. You need to have contributor+ role in the workspace. Delete the last remaining notebook manually.
Read on for the script. This one’s pretty straightforward, so there isn’t a lot in the way of additional commentary.
Comments closedNicky van Vroenhoven continues a series on governance in Microsoft Fabric:
A short introduction to Domains: they are essentially a way of managing and structuring your data across the organization. You can logically group together data in workspaces. A logical grouping can be business units, areas, fields, solutions or actually whatever works for you. It shouldn’t be something a Fabric Admin decides on his own. Ideally business and / or enterprise architects with the data owners (if any) should implement the design of domains, subdomains and owners. People from you Center of Excellence (again: if available..) would be a good fit to include in this discussion.
In case you need some help or guidance on how to set up your domains, there’s a nice article that can help you get started: Best practices for planning and creating domains.
Click through to learn more about domains and default domain settings.
Comments closedSoheil Bakhshi saves us some cash:
If you work in data and analytics, particularly within the Microsoft Data Platform, you have likely heard of Microsoft Fabric and its many capabilities. However, one of the biggest challenges organisations face is managing costs effectively.
In previous blogs and videos, I have covered how to optimise Microsoft Fabric capacity costs by automating the pause and resume process using Logic Apps. This approach ensures that your Fabric capacity runs only when needed, reducing unnecessary expenses. But how much can this method actually save? In this post, I share the real cost-saving results after applying this automation over the past few months.
Click through for some information on how much Soheil saved over a six-month period.
Comments closedChris Webb gives us the scoop:
There’s a new M function rolling out now that allows you to read metadata from Delta tables (at the time of writing it’s available in Dataflows Gen2 and will be available soon in Desktop). It builds on the DeltaLake.Table M function that allows you to read data from Delta tables and is similar to the Parquet.Metadata function that was released last year.
Click through for an example of how to use it against a Delta table in OneLake.
Comments closedMarc Lelijveld asks, what’s in a name?
In Fabric, you can have many different items in your Workspace. So many, that you easily get lost! Luckily there are tools at hand like Taskflows and Workspace folders. But still, it can be challenging to easily find all your items that ingest data, or find all items that are used for inbetween layers to transform data.
In this blog, I will tell you more about my personal best practice for naming convention of Fabric items that helps me to structure everything in my workspace.
This kind of thing typically doesn’t matter much when you only have a dozen or so items in your workspace. But as that number increases and different teams are working on different sets of items, it gets harder to figure out what’s going on without a proper naming convention.
Comments closedGilbert Quevauvilliers gets an e-mail:
What I have found is that when I created a pipeline in Microsoft Fabric that uses a notebook, when there is an error with the notebook, I do not get an alert that the notebook has failed.
This has happened to me in the past and I have found this pattern below to work consistently to notify me of errors.
In this blog post I will show you how I get notified when a notebook fails in a pipeline.
Read on to learn how.
Comments closedHere is a quick script to copy a semantic model from one workspace to another in the same tenant, assuming you are contributor+ in both the workspaces. I tested this for a Direct Lake model but should work for any more other semantic model. This just copies the metadata (not the data in the model) so be sure to set up other configurations (RLS members, refresh schedule, settings etc.). That can also be changed programmatically, thanks to Semantic Link Labs, but I will cover that in a future post.
Read on for the script, as well as an update from Sandeep on how you can do this even more easily.
Comments closedNikola Ilic builds us a guide:
Once upon a time, handling streaming data was considered an avanguard approach. Since the introduction of relational database management systems in the 1970s and traditional data warehousing systems in the late 1980s, all data workloads began and ended with the so-called batch processing. Batch processing relies on the concept of collecting numerous tasks in a group (or batch) and processing these tasks in a single operation.
On the flip side, there is a concept of streaming data. Although streaming data is still sometimes considered a cutting-edge technology, it already has a solid history. Everything started in 2002, when Stanford University researchers published the paper called “Models and Issues in Data Stream Systems”. However, it wasn’t until almost one decade later (2011) that streaming data systems started to reach a wider audience, when the Apache Kafka platform for storing and processing streaming data was open-sourced. The rest is history, as people say. Nowadays, processing streaming data is not considered a luxury, but a necessity.
This is all part of a book that Nikola and Ben Weissman are writing, and Nikola has an extended excerpt from the book available for us to read.
Comments closedIn a number of previous blogs and in my session on loadtesting Microsoft Fabric, I’ve always questioned the metrics app and one specific point is the timepoint detail. When you click on a graph, you get the option to go to the timepoint detail and read more.
This is all fun and games but looking at the list of active processes at that specific point in time, you’ll quickly see processes that are way out of the selected point in time. For me, it rendered this thing useless because it messed up the things I wanted to see.
Read on to see the right way to handle this app.
Comments closedDennes Torres checks some stats:
We already discovered how to investigate Kusto query history. Let’s discover how to analyse query performance considering the information on this history.
The query history returns 3 fields we can use to make a more detailed analysis of the queries: CachedStatistics, ScannedExtentsStatistics and ResultsetStatistics.
Disclaimer: There are low to no documentation about this content. In this way, the content below may not be 100% precise but will give you good guidance.
Click through to learn more about these three.
Comments closed