A Fine Slice Of SQL Server
Thierry Moudiki shows off a survival analysis:
In today’s post, we’ll see how to use rush and the probabilistic survival analysis API provided by techtonique.net (along with R and Python) to plot survival curves . Note that the web app also contains a page for plotting these curves, in 1 click. You can also read this post for more Python examples.
Click through for the demo. H/T R-Bloggers.
Comments closedGilbert Quevauvilliers builds a data frame:
In this blog post I am going to explain how to query multiple Lakehouse tables into a data frame.
The example I am going to use is when you want to load new data into your staging tables, but you need to know the max date from your previous data load.
Read on to see how. The answer, as you might suspect, involves DuckDB.
Comments closedGilbert Quevauvilliers does a bit of writing:
In this blog post I am going to explain how to use a Python Notebook using the Semantic Link module, to run a DAX query and write the output to a Lakehouse table.
I will show you how to install a Python library and then use it within my python notebook.
Read on for a quick primer on Semantic Link Labs, followed by the meat of the article.
Comments closedAida Gjoka and Russ Hyde show off some capabilities in the pytest library:
Writing tests is one of the best ways to keep your code reliable and reproducible. This post builds on our previous blog about Python testing with pytest Part 1, and explores some of the more advanced features it offers. From parametrised fixtures to mocking and other useful pytest plugins, we will show how to make your tests more reproducible, easier to manage and demonstrate how writing simple tests can save you time in the long run.
Click through to learn more. I’m a huge fan of parameterization in pytest—it’s really easy to do. Mocks are a bit harder to pull off in practice, though quite useful.
Comments closedWorking with sensitive data in Microsoft Fabric requires careful handling of secrets, especially when collaborating externally. In a recent customer engagement, I needed to validate access to Azure Key Vault from within a Fabric Notebook, without ever exposing the actual secret values. With only read access granted and no need to manage or update secrets, I focused on confirming that the connection was working as expected.
In this blog, I’ll walk you through the approach, including the setup, code snippets, and logic behind this quick but crucial verification step.
Click through for the full story.
Comments closedLevi Masonde stands up a Redis instance:
Databases play a vital role in software applications—they need to keep updated data or state, which is served by the database acting as the source of truth for the application. How the database performs affects how the entire application performs. Besides obvious factors that affect the performance of a database (hardware, database type, networking infrastructure), there are techniques designed to help you improve the performance of your database and ultimately, your applications. One way to do this is to add caching to your database. But which cache technique works best for which application requirements? This article sheds light on one strategy to implement a cache system.
Click through for one pattern of interaction between cache and database. My preference with the cache-aside pattern is to hide the two data platforms from the calling application as much as possible. In a classic object-oriented language like C#, the actual database + cache calls would be in a separate project and would expose methods on classes that were database-agnostic. With Python, I’d use different .py files in the same project unless I wanted to build a wheel file and deploy it to multiple projects, but the concept would still be the same. I’m not the biggest fan of the way that Levi did it, forcing API developers to have knowledge of both the cache and the database, as that increases the risk of a future developer messing something up.
Comments closedGilbert Quevauvilliers does a bit of writing:
When I was exploring how to easily write to Delta Tables with a Python notebook, it took me a considerable amount of time to find out how to do this.
This is my learnings below, and from my point of view it makes it easy to write to a Lakehouse table, like what is done with a PySpark notebook.
Click through for one very important note, as well as the process.
Comments closedChristian Lorentzen talks diagnostics:
In this post, we show how different use cases require different model diagnostics. In short, we compare (statistical) inference and prediction.
As an example, we use a simple linear model for the Munich rent index dataset, which was kindly provided by the authors of Regression – Models, Methods and Applications 2nd ed. (2021). This dataset contains monthy rents in EUR (
rent) for about 3000 apartments in Munich, Germany, from 1999.
Read on to learn more about this dataset and how the mindset differs if you’re thinking about inference versus prediction.
Comments closedGilbert Quevauvilliers writes a SQL statement:
I come from a TSQL background, so using SQL makes it easy for me to work with data.
There are multiple ways to use SQL in a PySpark notebook, and when I started using a Python notebook it was not so straightforward.
In this blog post I will show you how I use SQL Code.
As mentioned previously I am by no means an expert, I typically find a way that works, is fast and doesn’t consume a lot of capacity. If that works consistently for me then that is how I go about it.
Click through for the solution, which uses DuckDB. As such, the SQL syntax isn’t T-SQL—it’s more like psql. But it does do a great job of interacting with Parquet files and Delta tables.
Comments closedIn this video, I show how to use comprehensions in Python to generate lists, dictionaries, and sets. I also run a quick performance test, comparing a list comprehension to an equivalent for loop.
It can take a little bit of time to get used to the syntax, but once you do, comprehensions are quite powerful.
Comments closed