Press "Enter" to skip to content

Day: December 29, 2022

Join Types in Spark SQL

Published 2022-12-29 by Kevin Feasel

Rituraj Khare makes some connections:

In Apache Spark, we can use the following types of joins in SQL:

Inner join: An inner join in Apache Spark is a type of join that returns only the rows that match a given predicate in both tables. To perform an inner join in Spark using Scala, we can use the join method on a DataFrame.

The set of options is the same as you’d see in a relational database: inner, left outer, right outer, full outer, and cross. The examples here are in Scala, though would apply just as easily to PySpark and, of course, writing classic SQL statements.

Comments closed

Trying out FLAML

Published 2022-12-29 by Kevin Feasel

Gavita Regunath provides an overview of FLAML:

FLAML is short for Fast and Lightweight Automated Machine Learning library. It is an open-source Python library created by Microsoft researchers in 2021 for automated machine learning (AutoML). It is designed to be fast, efficient, and user-friendly, making it ideal for a wide range of applications.

Click through to learn more and to give it a spin with a pair of notebooks.

Comments closed

External Objects in Databricks Unity Catalog

Published 2022-12-29 by Kevin Feasel

Meagan Longoria adds external tables and views to an Azure Databricks Unity Catalog:

I’ve been busy defining objects in my Unity Catalog metastore to create a secure exploratory environment for analysts and data scientists. I’ve found a lack of examples for doing this in Azure with file types other than delta (maybe you’re reading this in the future and this is no longer a problem, but it was when I wrote this). So I wanted to get some more examples out there in case it helps others.

I’m not storing any data in Databricks – I’m leaving my data in the data lake and using Unity Catalog to put a tabular schema on top of it (hence the use of external tables vs managed tables. In order to reference an ADLS account, you need to define a storage credential and an external location.

Read on for examples of what you can do with this.

Comments closed

Importing Excel Files from SharePoint into Power BI

Published 2022-12-29 by Kevin Feasel

Gogula Aryalingam gets some data:

I’ve been asked the question many times, especially when I teach Dashboard in a Day: How do you import data from a bunch of Excel files stored on a SharePoint folder?

It’s a fairly simple process. It’s just that you need to know that you first connect to the SharePoint site (which could have quite a large number of folders and files, and several document libraries. You will need to know the path of your files, and then you keep filtering till you get to your content. I do this in two steps:

Read on for those steps.

Comments closed

SQL ConstantCare Population Analysis

Published 2022-12-29 by Kevin Feasel

Brent Ozar looks at the ConstantCare customer base:

Companies are leapfrogging right past SQL Server 2017. I’m going to hazard a guess that SQL Server 2017 came out too quickly after 2016, and didn’t offer enough features to justify upgrades from 2016.

Does that offer us any lessons for SQL Server 2022? Is 2022 going to be a 2017-style release that people just leapfrog over? Well, as I write this, it’s late December 2022, and I’m not seeing the widespread early adoption that I saw for 2019 where people had it in development environments ahead of the release, learning how to use it.

It’s an interesting analysis of a particular market segment: users of the ConstantCare service.

1 Comment