A Fine Slice Of SQL Server
Divyansh Jain shows how we can connect to AWS’s S3 using Apache Spark:
Now, coming to the actual topic that how to read data from S3 bucket to Spark. Well, it is not very easy to read S3 bucket by just adding Spark-core dependencies to your Spark project and use spark.read to read you data from S3 Bucket.
So, to read data from an S3, below are the steps to be followed:
This isn’t a built-in source, so there is a little bit of work to do, but it’s not that bad.
Comments closedSwapnil Gosavi walks us through the process of creating a custom partitioner in Apache Kafka:
Assume we are collecting data from different departments. All the departments are sending data to a single topic named department. I planned five partitions for the topic. But, I want two partitions dedicated to a specific department, named IT, and the remaining three partitions for the rest of the departments. How would you achieve this?
You can solve this requirement, and any other type of partitioning needs by implementing a custom partitioner.
This is quite useful when you don’t necessarily want topic explosion but you do want more than what the classic partitioner allows.
Comments closedSaeed Barghi takes us through building an interesting solution:
A few weeks ago I did a talk at AI Bootcamp here in Melbourne on how we can build a serverless solution on Azure that would take us one step closer to powering industrial machines with AI, using the same technology stack that is typically used to deliver IoT analytics use cases. I demoed a solution that received data from an IoT device, in this case a crane, compared the data with the result of a machine learning model that has ran and written its predictions to a repository, in this case a CSV file, and then decided if any actions needs to be taken on the machine, e.g. slowing the crane down if the wind picks up.
This was a really interesting article.
Comments closedRobin Moffatt has a quick demonstration of ksqlDB:
Comments closedI’m going to show you how to use ksqlDB to do the following:
– Configure the live ingest of a stream of data from an external source (in this case, Twitter)
– Filter the stream for certain columns
– Create a new stream populated only by messages that match a given predicate
– Build aggregate materialised views, and use pull queries to directly fetch the state from theseLet’s dive in! As always, you’ll find the full test rig for trying this out yourself on GitHub.
Divyansh Jain contrasts repartitioning and coalescing in Spark:
What is Coalesce?
The coalesce method reduces the number of partitions in a DataFrame. Coalesce avoids full shuffle, instead of creating new partitions, it shuffles the data using Hash Partitioner (Default), and adjusts into existing partitions, this means it can only decrease the number of partitions.
What is Repartitioning?
The repartition method can be used to either increase or decrease the number of partitions in a DataFrame. Repartition is a full Shuffle operation, whole data is taken out from existing partitions and equally distributed into newly formed partitions.
Read on to learn good reasons to use both.
Comments closedDmitry Tolpeko walks us through a performance problem in Spark:
I have a Spark job that transforms incoming data from compressed text files into Parquet format and loads them into a daily partition of a Hive table. This is a typical job in a data lake, it is quite simple but in my case it was very slow.
Initially it took about 4 hours to convert ~2,100 input .gz files (~1.9 TB of data) into Parquet, while the actual Spark job took just 38 minutes to run and the remaining time was spent on loading data into a Hive partition.
Let’s see what is the reason of such behavior and how we can improve the performance.
Read on to see how.
Comments closedSarfaraz Hussain explains how some of the pieces of the Hadoop ecosystem fit together:
Once it verifies that everything is in place, it will assign a Job ID to the Job and then allocate the Job ID into a Job Queue.
So, in Job Queue there can be multiple jobs waiting to get processed.
As soon as a job is assigned to the Job Queue, it’s corresponding information about the Job like Input/Output Path, the location of the Jar, etc. are written into the temp location of HDFS.
Read the whole thing.
Comments closedJukka Karvanen takes us through TopologyTestDriver in Apache Kafka Streams:
Similarly to testing a single record, it is possible to pipe
Valuelists into aTestInputTopic. Validating the output can be done record by record like before, or by using thereadValueToList()method to see the big picture when validating the whole collection at the same time. For our example topology, when the test pipes in the values, it needs to validate the keys and use thereadKeyValueToList()method.
Testing streams of data (regardless of the product) is enough of a mental shift that it’s not an easy problem. For that reason, I welcome any tool which simplifies the process.
Comments closedMatthew Rathbone compares the Parquet and Avro file formats:
JSON improves upon CSV as each row provides some indication of schema, but without a special header-row, there’s no way to derive a schema for every record in the file, and it isn’t always clear what type a ‘null’ value should be interpreted as.
Avro and Parquet on the other hand understand the schema of the data they store. When you write a file in these formats, you need to specify your schema. When you read the file back, it tells you the schema of the data stored within. This is super useful for a framework like Spark, which can use this information to give you a fully formed data-frame with minimal effort.
I was kind of hoping to see ORC in the comparison as well, though even when the Hortonworks-Cloudera competition was at its max, my recollection is that the differences between the two formats were pretty small (where ORC was a little faster for non-nested data and Parquet a little faster for nested data).
Comments closedSagar Kewalramani walks us through the choices for SQL platforms on the Cloudera Data Platform:
CDW on CDP is a new service that enables you to create a self-service data warehouse for teams of Business Intelligence (BI) analysts. You can quickly provision a new data warehouse and share any data set with a specific team or detpartment. Do you remember when you could provision a data warehouse on your own? Without infrastructure and platform teams getting involved? This was never possible. CDW fulfills this mission.
However, CDW makes several SQL engines available, and with more choice comes more opportunities for confusion. Let’s explore the SQL engines available in CDW on CDP and talk about which is the right SQL option for the right use case.
So many choices! Impala? Hive LLAP? Spark? What to use when? Let’s explore.
Infrastructure and platform teams start to get involved approximately two days after the unexpectedly large bill arrives.
That aside, this is a really nice article covering several platform technologies, including Impala, Hive LLAP, and Spark SQL.
Comments closed