A Fine Slice Of SQL Server
Divyansh Jain shows how we can connect to AWS’s S3 using Apache Spark:
Now, coming to the actual topic that how to read data from S3 bucket to Spark. Well, it is not very easy to read S3 bucket by just adding Spark-core dependencies to your Spark project and use spark.read to read you data from S3 Bucket.
So, to read data from an S3, below are the steps to be followed:
This isn’t a built-in source, so there is a little bit of work to do, but it’s not that bad.
Comments closedSaeed Barghi takes us through building an interesting solution:
A few weeks ago I did a talk at AI Bootcamp here in Melbourne on how we can build a serverless solution on Azure that would take us one step closer to powering industrial machines with AI, using the same technology stack that is typically used to deliver IoT analytics use cases. I demoed a solution that received data from an IoT device, in this case a crane, compared the data with the result of a machine learning model that has ran and written its predictions to a repository, in this case a CSV file, and then decided if any actions needs to be taken on the machine, e.g. slowing the crane down if the wind picks up.
This was a really interesting article.
Comments closedDivyansh Jain contrasts repartitioning and coalescing in Spark:
What is Coalesce?
The coalesce method reduces the number of partitions in a DataFrame. Coalesce avoids full shuffle, instead of creating new partitions, it shuffles the data using Hash Partitioner (Default), and adjusts into existing partitions, this means it can only decrease the number of partitions.
What is Repartitioning?
The repartition method can be used to either increase or decrease the number of partitions in a DataFrame. Repartition is a full Shuffle operation, whole data is taken out from existing partitions and equally distributed into newly formed partitions.
Read on to learn good reasons to use both.
Comments closedDmitry Tolpeko walks us through a performance problem in Spark:
I have a Spark job that transforms incoming data from compressed text files into Parquet format and loads them into a daily partition of a Hive table. This is a typical job in a data lake, it is quite simple but in my case it was very slow.
Initially it took about 4 hours to convert ~2,100 input .gz files (~1.9 TB of data) into Parquet, while the actual Spark job took just 38 minutes to run and the remaining time was spent on loading data into a Hive partition.
Let’s see what is the reason of such behavior and how we can improve the performance.
Read on to see how.
Comments closedSarfaraz Hussain explains how some of the pieces of the Hadoop ecosystem fit together:
Once it verifies that everything is in place, it will assign a Job ID to the Job and then allocate the Job ID into a Job Queue.
So, in Job Queue there can be multiple jobs waiting to get processed.
As soon as a job is assigned to the Job Queue, it’s corresponding information about the Job like Input/Output Path, the location of the Jar, etc. are written into the temp location of HDFS.
Read the whole thing.
Comments closedTim van Cann and Daniel van der Ende have an open source project for automatic deployment on Azure:
To give a bit more insight into why we built Schiphol Takeoff, it’s good to take a look at an example use case. This use case ties a number of components together:
– Data arrives in a (near) real-time stream on an Azure Eventhub.
– A Spark job running on Databricks consumes this data from Eventhub, processes the data, and outputs predictions.
– A REST API is running on Azure Kubernetes Service, which exposes the predictions made by the Spark job.Conceptually, this is not a very complex setup. However, there are quite a few components involved:
– Azure Eventhub
– Azure Databricks
– Azure Kubernetes ServiceEach of these individually has some form of automation, but there is no unified way of coordinating and orchestrating deployment of the code to all at the same time. If, for example, you were to change the name of the consumer group for Azure Eventhub, you could script that. However, you’d also need to manually update your Spark job running on Databricks to ensure it could still consume the data.
This looks pretty nice. I’ll need to dive into it some more.
Comments closedRazavi and Michael Johns walk us through examples of processing geospatial data with Databricks:
Earlier, we loaded our base data into a DataFrame. Now we need to turn the latitude/longitude attributes into point geometries. To accomplish this, we will use UDFs to perform operations on DataFrames in a distributed fashion. Please refer to the provided notebooks at the end of the blog for details on adding these frameworks to a cluster and the initialization calls to register UDFs and UDTs. For starters, we have added GeoMesa to our cluster, a framework especially adept at handling vector data. For ingestion, we are mainly leveraging its integration of JTS with Spark SQL which allows us to easily convert to and use registered JTS geometry classes. We will be using the function st_makePoint that given a latitude and longitude create a Point geometry object. Since the function is a UDF, we can apply it to columns directly.
Looks like they have some pretty good functionality here, and they have shared the demos in notebook form.
Comments closedPinku Swargiary shows us how to configure Spark to use Kryo serialization:
If you need a performance boost and also need to reduce memory usage, Kryo is definitely for you. The join operations and the grouping operations are where serialization has an impact on and they usually have data shuffling. Now lesser the amount of data to be shuffled, the faster will be the operation.
Caching also have an impact when caching to disk or when data is spilled over from memory to disk.Also, if we look at the size metrics below for both Java and Kryo, we can see the difference.
Sounds like it’s better overall but requires some custom configuration.
Comments closedBrad Llewellyn starts a new series on Delta Lake in Azure Databricks:
Saving the data in Delta format is as simple as replacing the .format(“parquet”) function with .format(“delta”). However, we see a major difference when we look at the table creation. When creating a table using Delta, we don’t have to specify the schema, because the schema is already strongly defined when we save the data. We also see that Delta tables can be easily queried using the same SQL we’re used to. Next, let’s compare what the raw files look like by examining the blob storage container that we are storing them in.
There are some good demos in this post and it promises to be a nice series.
Comments closedDatabricks natively stores it’s notebook files by default as DBC files, a closed, binary format. A .dbc file has a nice benefit of being self-contained. One dbc file can consist of an entire folder of notebooks and supporting files. But other than that, dbc files are frankly obnoxious.
Read on to see how to convert between these two formats.
Comments closed