The Hadoop in Real World team contrasts repartitioning and coalescing in Spark:
The function of repartition and coalesce functions in Spark is to change the number of partitions on a DataFrame.
Read on to see how the two differ.
Comments closedCategory: Spark
Tomaz Kastrun take a look at the original Spark language:
Let us start with Databricks datasets, that are available within every workspace and are here mainly for test purposes. This is nothing new; both Python and R come with sample datasets. For example the Iris dataset that is available with Base R engine and Seaborn Python package. Same goes with Databricks and sample dataset can be found in /databricks-datasets folder.
Click through for the walkthrough and introduction to Scala as it relates to Apache Spark.
Comments closedEuan Garden shows off some Apache Spark functionality in Azure Synapse Analytics:
Apache Spark has been a long-time favorite tool amongst data engineers and data scientists; it is well known for handling large scale data processing and complex machine learning workloads.
Azure Synapse Analytics offers a fully managed and integrated Apache Spark experience. By leveraging Apache Spark in Azure Synapse, you can benefit from integrated security, fully managed provisioning, and tight-coupling to other Azure services, such as SQL databases (dedicated and serverless), Azure Key Vault , ADLS Gen2, and Azure Blob Storage as well as fast starting, high performance compute instances.
Click through for the demo.
Comments closedApache Spark has had a machine learning API for quite some time and this has been partially implemented in .NET for Apache Spark.
In this post we will look at how we can use the Apache Spark ML API from .NET. This is the second version of this post, the first version was written before version 1 of .NET for Apache Spark and there was a vital piece of the implementation missing which meant although we could build the model in .NET, we couldn’t actually use it. The necessary functionality is now available and so I am updating the post. To see the previous version go to: https://the.agilesql.club/2020/07/tf-idf-in-.net-for-apache-spark-using-spark-ml/
Read on for more information, as well as a call to action.
Comments closedTomaz Kastrun takes us through a machine learning scenario using Azure Databricks:
In the past couple of days we looked into configurations and infrastructure and today it is again time to do an analysis, let’s call it end-to-end analysis using R or Python or SQL.
Read on for the process.
Comments closedYitao Li announces version 1.5 of sparklyr:
A large fraction of pull requests that went into the
sparklyr1.5 release were focused on making Spark dataframes work with variousdplyrverbs in the same way that R dataframes do. The full list ofdplyr-related bugs and feature requests that were resolved insparklyr1.5 can be found in here.In this section, we will showcase three new dplyr functionalities that were shipped with
sparklyr1.5.
Read on to learn more about this update. H/T R-Bloggers
Comments closedTsuyoshi Matsuzaki walks us through running Apache Spark on Azure Kubernetes Service:
Apache Spark officially includes Kubernetes support, and thereby you can run a Spark job on your own Kubernetes cluster. (See here for official document. Note that Kubernetes scheduler is currently experimental.)
Especially in Microsoft Azure, you can easily run Spark on cloud-managed Kubernetes, Azure Kubernetes Service (AKS).In this post, I’ll show you step-by-step tutorial for running Apache Spark on AKS. In this tutorial, artifacts, such as, source code, data, and container images are all protected by Azure credentials (keys).
Although managed services for Apache Spark, such as, Azure Databricks, Azure Synapse Analytics, and Azure HDInsight, is the best place to run Spark workloads, you will get much flexibility by running workloads on managed Kubernetes (AKS) – such as, spot VM support, start/stop cluster, confidential computing (Intel SGX) support, so on and so forth.
Read on to see how. Though of these options, I’d probably choose Azure Databricks or Azure Synapse Analytics well before the others.
Comments closedTomaz Kastrun continues a series on Azure Databricks:
So far, we looked into SQL, R and Python and this post will be about Python Koalas package. A special implementation of pandas DataFrame API on Apache Spark. Data Engineers and data scientist love Python pandas, since it makes data preparation with pandas easier, faster and more productive. And Koalas is a direct “response” to make writing and coding on Spark, easier and more familiar. Also follow the official documentation with full description of the package.
Click through for a quick demo.
Comments closedTomaz Kastrun continues opening doors in the advent calendar:
Yesterday we introduced the Databricks CLI and how to upload the file from “anywhere” to Databricks. Today we will look how to use Azure Blob Storage for storing files and accessing the data using Azure Databricks notebooks.
Click through to see how.
Comments closedRobert Blackburn shows us one way to handle bad dates when moving to Spark 3:
Moving from a Spark 2 to a Spark 3 runtime has a lot of benefits including big performance improvements through adaptive query execution, dynamic partition pruning, and other optimizations. Some updates may require you to refactor your code. One of them is Delta tables now use the Proleptic Gregorian Calendar. Isn’t a calendar a calendar? Unfortunately, no. The Julian calendar has discrepancies with old dates. Specifically dates before 1582 and timestamps before 1900. Here we will dynamically update these dates for incoming source files.
If you would like to follow along in detail, I have a sample notebook that uses the community edition of Databricks. The DBC Archive file is here and the source file is here.
Fortunately, this change is unlikely to affect most of us, with perhaps the most common issue being that you used 0001-01-01 as a default date.
Comments closed