A Fine Slice Of SQL Server
Christian Lorentzen has released a new package:
Version 1.0.0 of the new Python package for model-diagnostics was just released on PyPI. If you use (machine learning or statistical or other) models to predict a mean, median, quantile or expectile, this library offers tools to assess the calibration of your models and to compare and decompose predictive model performance scores.
This looks like a really useful package, so check it out.
Comments closedJohn Mount shows off a feature:
I’d like to share a great new feature in the wvpy package (available at PyPi).
This package is useful in converting Jupiter notebooks to/from python, and also in rendering many parameterized notebooks. The idea is to make Jupyter notebook easier to use in production.
The latest feature is an extension of notebook parameterization. In addition to the
init_codeandoutput_suffixfeatures, which allow adding arbitrary code to notebooks and saving multiple renders of the same notebook under different (non-coliding!) names. The newsheet_varsfeature allows the insertion of arbitrary data into notebook renders (in addition to the earlier code insertion facility).
Click through for an example on how to use this. Several years ago, I would have considered this to be outstanding. Today, I think it’s cool, but I’ve also gravitated toward using notebooks as an intermediary step rather than a final product, so it’s less critical for me these days.
Comments closedGilbert Quevauvilliers tries out a tool:
I was going through my twitter feed and I came across this tweet where they spoke about the Data Wrangler Calling all #Python users! Have you tried Data Wrangler in #MicrosoftFabric?
I thought I would give this a try and that was the idea for my blog post. I honestly had no idea that firstly was this possible, but second that it is so easy for data wrangler to do all the hard work for me
I am going to demonstrate 2 transformations in this blog post, the first will be changing the d_date from date to datetime and then using the columns from examples I am going to create a new column where it concatenates 2 columns delimited with a double pipe command.
Read on for Gilbert’s thoughts.
Comments closedDustin Vannnoy has two of the three R’s down:
Every Spark pipeline involves reading data from a data source or table. For data engineers we usually end the pipelines by writing the transformed data. In this tutorial we walk through some of the most common format and cloud storage locations for reading and writing with Spark. We’ll save some of the advanced Delta Lake capabilities for another tutorial.
Click through to see how to read from and write to CSV, JSON, and Parquet formats. Dustin has examples of working with Azure Blob Storage, S3, and Google Cloud Storage, and even some database examples with JDBC.
Comments closedMichael Mayer runs an analysis:
This is the next article in our series “Lost in Translation between R and Python”. The aim of this series is to provide high-quality R and Python code to achieve some non-trivial tasks. If you are to learn R, check out the R tab below. Similarly, if you are to learn Python, the Python tab will be your friend.
This post is heavily based on the new {shapviz} vignette.
I appreciate the effort to include both R and Python code in this analysis, and recommend you peruse both sets of code listings.
Comments closedSteven Vaughan-Nichols gives us the story:
The Python Package Index (PyPI), is the most popular Python programming language software repository. It’s also a mess. Earlier this year, the FortiGuard team discovered zero-day malware in three PyPI packages called “colorslib,” “httpslib,” and “libhttps.” Before that, 2022 closed with PyTorch-nightly on Linux being poisoned with a fake dependency. More recently, PyPI had to stop new user registrations and project creations because of a flood of malicious users. PyPI isn’t the only one to notice the user trouble. The Python Software Foundation (PSF) received three subpoenas for PyPI user data. What is going on here!?
Read on to learn more about what’s happening with the most popular Python repository.
Comments closedMyles Mitchell celebrates a year of diffify:
We’ve just passed an important milestone for diffify: our app for tracking Python and R package releases has just turned 1 year old! To mark this exciting occasion we are delighted to announce an “anniversary update” featuring numerous quality of life improvements. This post will outline the latest changes and tease at some exciting developments in the works…
Check out these recent changes and a little bit of what’s on the horizon.
Comments closedDustin Vannoy gives us a primer on Apache Spark:
Get hands on with Python and PySpark to build your first data pipeline. In this video I walk you through how to read, transform, and write the NYC Taxi dataset which can be found on Databricks, Azure Synapse, or downloaded from the web to wherever you run Apache Spark. Once you have watched and followed along with this tutorial, go find a free dataset and try to write your own PySpark application. Pro tip: Search for the Spark equivalent of functions you use in other programming languages (including SQL). Many will exist in the pyspark.sql.functions module.
In addition to the code listing, Dustin has a video walking us through the process.
Comments closedAyodeji Ogunlami mixes approaches:
In developing this hybrid system, sets of rules are required as well as a machine learning model. I would be making use of a vehicle insurance dataset from Kaggle in this demonstration.
The dataset can be downloaded from this link: https://www.kaggle.com/datasets/shivamb/vehicle-claim-fraud-detection
The ML model would be built using a random forest classifier on Azure Databricks using Pyspark.
This seems to be the most sensible approach, especially given how rare actual fraud incidents are and what that imbalance does to classification algorithms.
Comments closedAmlan Patnaik looks at some common implementation problems:
Pyspark has become one of the most popular tools for data processing and data engineering applications. It is a fast and efficient tool that can handle large volumes of data and provide scalable data processing capabilities. However, Pyspark applications also come with their own set of challenges that data engineers face on a day-to-day basis. In this article, we will discuss some of the common challenges faced by data engineers in Pyspark applications and the possible solutions to overcome these challenges.
Read on for five such challenges.
Comments closed