A Fine Slice Of SQL Server
Jon Fletcher has a quick one for us:
In this blog post I will show you how to join two 2D Python lists together.
The code is in the screenshot below.
Nested loops is an easy algorithm to understand, and even here, the whole process is 10 lines of code.
Comments closedJohn Mount has an update on a Python package:
The data algebra is a modern realization of elements of Codd’s 1969 relational model for data wrangling (see also Codd’s 12 rules).
The idea is: most data manipulation tasks can usefully be broken down into a small number of fundamental data transforms plus composition. In Codd’s initial writeup, composition was expressed using standard mathematical operator notation. For “modern” realizations one wants to use a composition notation that is natural for the language you are working in. For Python the natural composition notation is method dispatch.
Click through to see how it works and what’s new in the latest version.
Comments closedDeployment is a crucial move in the ML workflow. It is a mark where we want to implement our ML model into utilization. Later, we can practice the model in practical life.
But how can we design the model as a treatment? We can develop an Application Programming Interface (API). With that, we can reach the model universally, can be a mobile application or web application. In Python, there’s a library that can assist us in building an API. It’s named Flask.
This article will explain how to construct a REST API for our machine learning model utilizing Flask. Without further ado, let’s begun!
Flask is the first step, but then I’d want to reverse proxy it with gunicorn or Nginx afterward.
Comments closedTsuyoshi Matsuzaki sets up Dask as a distributed service:
In my last post, I showed you tutorial for running Apache Spark on managed kubernetes, Azure Kubernetes Service (AKS).
In this post, I’ll show you the tutorial for running distributed workloads of Dask on AKS.By using Dask, you can run Scikit-Learn compliant functions and jobs for data which cannot fit in memory, or run in distributed manners. For simplicity, here I’ll use built-in Dask ML function (
dask_ml.linear_model.LinearRegression) in this tutorial. (With the same manners, you can also run regular sklearn functions.)
Cloud managed kubernetes will make you speed up this large ML workloads.
Click through for the process. I’ve had some positive experiences with Dask as a dashboarding tool. It’s definitely one of the better ones if you’re big into Python.
Comments closedSheldon Hull walks us through the diagrams package in Python:
LucidChart, Draw.io and other tools are great for a quick solution.
Mermaid also provides a nice simple text based diagramming tool that is integrated with many markdown tools.
For me, this just never fit. I like a bit of polish and beauty in a visual presentation and most of these are very utilitarian in their output.
I came across diagrams and found it a perfect fit for intuitive and beautiful diagram rendering of cloud architecture, and figured it would be worth a blog post to share this.
Back when GitPitch was still a viable concern, I had just gotten into using the diagrams package. It takes some getting used to and has very strong preferences on the sorts of relationships diagram elements can have, but it’s good at its job.
Comments closedJon Morisi troubleshoots a performance issue:
I recently had a project in which I needed to transfer a 60 GB SQLite database to SQL Server. After some research I found the sqlite3 and pyodbc modules, and set about scripting connections and insert statements.
The basic form of my script is to import the modules, setup the database connections, and iterate (via cursor) over the rows of the select statement creating insert statements and executing them.
The issue here is that this method results in single inserts being sent one at a time yielding less than satisfactory performance. Inserting 35m+ rows in this fashion takes ~5hrs on my system.
Jon tries out a few different options. It would appear that there is no easy bulk insertion operation with PyODBC.
Comments closedI remembered that I had another blog and actually wrote something technical:
Not too long ago, I worked through an interesting issue with Azure Machine Learning. The question was, what’s the best way to read from Azure SQL Database, perform model processing, and then write results out to Azure SQL Database? Oh, by the way, I want to use a service principal rather than SQL authentication. Here’s what I’ve got.
This turned out to be a lot more work than I first expected.
Comments closedRobert J. Blackburn needs to do some cleanup work:
The function will accept the dataframe and a list of columns to process. Because of syntax restrictions the steps are broken out into multiple statements and a sub-function. Luckily, Spark’s lazy evaluation will optimize it during runtime.
Click through for the function in question.
Comments closedMatt Dancho and Jarrell Chalmers lay out an argument:
The decision can be challenging because they both Python and R have clear strengths.
– R is exceptional for Research – Making visualizations, telling the story, producing reports, and making MVP apps with Shiny. From concept (idea) to execution (code), R users tend to be able to accomplish these tasks 3X to 5X faster than Python users, making them very productive for research.
– Python is exceptional for Production ML – Integrating machine learning models into production systems where your IT infrastructure relies on automation tools like Airflow or Luigi.
They make a pretty solid argument. I’ve launched success R-based projects using SQL Server Machine Learning Services, but outside of ML Services, my team’s much more likely to deploy APIs in Python, and we’re split between Dash and Shiny for visualization. H/T R-Bloggers
Comments closedNiels Berglund troubleshoots some issues:
In the last post, which looks at using Python 3.9 in SQL Server Machine Learning Services, I wrote this at the very end:
It looks like all is good, but maybe not? In a future post we’ll look at an issue we have introduced – but for now, let us bask in the glory of having created a new Python language extension.
In the post, we wrote a new language extension to handle Python 3.9, and that just worked fine. However, when I was doing some other things, I noticed some side effects, and in this post, we look at those side effects and how to solve them.
Click through to learn more.
Comments closed