GIlbert Quevauvilliers uses a pure Python notebook:
In this blog post I am going to show you how to use a Fabric Python runtime notebook (This is the notebook which only uses Pure Python functions and consumes significantly lower Capacity Units (CUs)).
The pattern is how to get new data and merge it into an existing Lakehouse table. This ensures that if the notebook is run again data will not be duplicated.
Why I am sharing this is I have found that there is not a lot of useful information about how to use a Python notebook to write to a lakehouse table easily. And then also how to use a Merge statement making it easier to insert or update your lakehouse tables. This simplifies the ingestion process, runs faster and consumes the least amount of CUs
Gilbert doesn’t mention it in the blog post but the notebook does use DuckDB to query the data using SQL.