A Fine Slice Of SQL Server
Miles Cole picks up from part one:
This is part 2 of my prior post that continues where I left off. I previously showed how you can use Resource folders in either the Notebook or Environment in Microsoft Fabric to do some pretty agile development of Python modules/libraries.
Now, how exactly can you package up your code to distribute and leverage it across multiple Workspaces or Environment items? How could we acomplish something like the below?
Read on for the answer.
Leave a CommentDennes Torres wants to hit a SQL endpoint:
Let’s analyse why we would like to query an SQL Endpoint. Once we understand why, we can dig into how to make a query to a SQL Endpoint.
We use notebooks to connect directly to lakehouse. Except by the T-SQL notebook, the notebooks have a default lakehouse and work directly with it from spark. However, accessing other data object may be more complex
Specifically, this is a Spark notebook in Microsoft Fabric running Scala rather than a pure Python notebook, and is hitting the data warehouse SQL endpoint.
Leave a CommentSince Lakehouse is one of the key items within Microsoft Fabric, it is important to know how to write data into it in various formats and using different tools. One of the most common tools is notebooks, as they provide great flexibility and speed for development and testing with graphical outputs. In this article, I want to focus primarily on the following types of notebooks:
- PySpark
- Python
Click through to see how it works in both notebook types.
Comments closedGilbert Quevauvilliers doesn’t need Spark for this:
This blog below explains how to use a Python only notebook to get all the Fabric items using the Fabric REST API.
NOTE: At the time of this blog post Feb 2025, Dataflow Gen2 is not included in the Fabric items, I am sure it will be there in the future.
NOTE II: This only gets the Fabric Items, which does not include the Power BI Items.
Despite the notes, Gilbert leads off with the main reason why you might want to use this: it takes up approximately 5% of the capacity units that a Spark-based notebook does to perform the same operation.
Comments closedGilbert Quevauvilliers gets an e-mail:
What I have found is that when I created a pipeline in Microsoft Fabric that uses a notebook, when there is an error with the notebook, I do not get an alert that the notebook has failed.
This has happened to me in the past and I have found this pattern below to work consistently to notify me of errors.
In this blog post I will show you how I get notified when a notebook fails in a pipeline.
Read on to learn how.
Comments closedKevin Chant needs a service principal to help:
In this post I want to share one way that you can authenticate as a service principal to run a Microsoft Fabric notebook from Azure DevOps.
Some of you may recall that I previously covered how to run a Microsoft Fabric notebook from Azure DevOps.
I decided to published a newer version of the aforementioned post to amplify the fact that the REST API that runs a notebook on demand now supports service principals.
Service principals are the way to go for this, so long as you’re having one Azure-based service communicate with another Azure-based service. No passwords, no API keys, nothing you need to remember or change every 90 days.
The problem is, this works beautifully for assets inside of Azure, but not so much outside of Azure. But that’s a story for a different day.
Comments closedSandeep Pawar goes on a journey:
Simon Willison is one of my favorite bloggers. In fact, what I blog, how I blog & test, is inspired by him. He wrote a blog a couple of weeks ago about FourSquare Places data that has been open-sourced. I was exploring this dataset and ended up creating a few maps. I love OrgApps in Fabric and I truly believe as it matures, it will be THE way for analysts & data scientists to provide rich insights + traditional reports to business users. Notebooks can augment the Power BI reports to provide insights that are otherwise not possible. I have submitted a session on this topic to FabCon ‘25, let’s see. If it is selected, I hope to show how transformational it is and how businesses can use it.
Click through for a video and the notebook that Sandeep demonstrated.
Comments closedGilbert Quevauvilliers performs an exfiltration:
With the new Schema’s in a Lakehouse, it now is possible to read from Lakehouse A (In Workspace A) and write to Lakehouse B (In Workspace B).
Here are more details about the Schema preview: Lakehouse schemas (Preview) – Microsoft Fabric | Microsoft Learn
This opens a whole new world of possibilities.
I also really like the fact that I can simply use the Names, and I do not need to get the actual GUIDS!
For example, I can use the following as shown below which is WorkspaceName.LakehouseName,SchemaName.TableName
Click through to see it in action.
Comments closedDennes Torres tries out T-SQL notebooks:
T-SQL Notebooks is one of the new features announced during FabCon Europe.
The most distracted could miss the fact this is a new feature at all. Yes, it is. Notebooks were capable to support Spark SQL, but T-SQL is something new.
The main examples being announced are built with data warehouses, but let me confirm and highlight this:
T-SQL Notebooks support lakehouses as well.
There is at least one limitation: DML is not supported with lakehouses.
Saving my rant about lakehouses vs warehouses in Fabric, do read what Dennes has to say about T-SQL notebooks as they exist today.
Comments closedAyman El-Ghazali solves a problem:
A few months ago, an ISV customer approached with a request to have notebooks run across Microsoft Fabric Lakehouse environments dynamically. Initially the first request was to allow pipelines in Fabric to pass parameters for file paths to help with data ingestion. This would allow the customer to use the same notebook across Lakehouse environments for the customers that they are serving. After resolving this, the scope increased to include the notebook execution. The notebooks should be able to run across workspace environments and not have to be attached to a Lakehouse at the time of execution. The solution presented below allows for the customer to run notebooks across environments but also allows them to run SQL queries against existing Lakehouse tables; additionally it allows for access to tables created during the notebook execution run without the notebook being attached to the Lakehouse.
Read on to learn how.
Comments closed