A Fine Slice Of SQL Server
The Hadoop in Real World team shows how you can save Spark DataFrames as Hive tables:
It is a very common use case to process the data in Spark and save the processed data or Spark dataframe directly into a Hive table.
There are a couple ways to achieve this.
Click through for those techniques.
Comments closedAndrew Weaver, et al, take us through security practices for running Databricks on AWS:
In this article, we will share a list of cloud security features and capabilities that an enterprise data team can use to harden their Databricks environment on AWS as per their risk profile and governance policy. For more information about how Databricks runs on Amazon Web Services (AWS), view the AWS web page and Databricks security on AWS page for more specific details on security and compliance.
Click through for that list.
Comments closedEuan Garden has some great news for us:
Starting today, the Apache Spark 3.0 runtime is now available in Azure Synapse. This version builds on top of existing open source and Microsoft specific enhancements to include additional unique improvements listed below. The combination of these enhancements results in a significantly faster processing capability than the open-source Spark 3.0.2 and 2.4.
The public preview announced today starts with the foundation based on the open-source Apache Spark 3.0 branch with subsequent updates leading up to a Generally Available version derived from the latest 3.1 branch.
It still won’t be as fast as Databricks, but it should be a good bit faster than the Spark 2 they were running.
Comments closedThe Hadoop in Real World team explains the notion of broadcast variables in Apache Spark:
Broadcast variables are variables which are available in all executors executing the Spark application. These variables are already cached and ready to be used by tasks executing as part of the application. Broadcast variables are sent to the executors only once and it is available for all tasks executing in the executors.
Read on to understand when they are useful and, just as importantly, when not to use them. They seem like the type of thing which a newer developer could easily misuse.
Comments closedThe Hadoop in Real World team shows how to deduplicate rows in a DataFrame in Spark:
It is a pretty common use case to find the list of duplicate elements or rows in a Spark DataFrame and it is very easy to do with a groupBy() and a count()
Where “easy” has as a modifier just how many columns you’re dealing with in the DataFrame.
Comments closedThe Hadoop in Real World team compares two functions against RDDs in Spark:
Let’s examine the below aggregateByKey. The first parameter – 0 is the initial value and also indicates the type of the output.
First _+_ function indicates the function on the map side combine and second _+_ function indicates the reduce side combine. Both functions are the same in this case.
This is a demo-driven post, so check it out.
Comments closedJovan Popovic shows off one integration point between the data services in Azure Synapse Analytics:
Azure Synapse Analytics provides multiple query runtimes that you can use to query in-database or external data. You have the choice to use T-SQL queries using a serverless Synapse SQL pool or notebooks in Apache Spark for Synapse analytics to analyze your data.
You can also connect these runtimes and run the queries from Spark notebooks on a dedicated SQL pool.
In this post, you will see how to create Scala code in a Spark notebook that executes a T-SQL query on a serverless SQL pool.
Read on to see how. You can also query Spark pool and dedicated SQL pool tables from serverless SQL pools.
4 CommentsAntoine Amend uses Databricks to identify financial fraud in a geographical area:
As part of this real-world solution, we are releasing a new open source geospatial library, GEOSCAN, to detect geospatial behaviors at massive scale, track customers patterns over time and detect anomalous card transactions. Finally, we demonstrate how organizations can surface anomalies from an analytics environment to an online data store (ODS) with tight SLA requirements following a Lambda-like infrastructure underpinned by Delta Lake, Apache Spark and MLflow.
Click through for the article, as well as three notebooks.
Comments closedDavid Greenshtein shows how we can build type-2 slowly changing dimensions using Apache Hudi:
Implementing SCD2 in a data lake without using an additional framework like Apache Hudi introduces the challenge of updating data stored on immutable Amazon S3 storage, and as a result requires the implementor to create multiple copies of intermediate results. This situation may lead to a significant maintenance effort and potential data loss or data inconsistency.
Apache Hudi is an open-source data management framework used to simplify incremental data processing and data pipeline development. Hudi enables you to manage data at the record level in Amazon S3 and helps to handle data privacy use cases requiring record-level updates and deletes. Hudi is supported by Amazon EMR starting from version 5.28 and is automatically installed when you choose Spark, Hive, or Presto when deploying your EMR cluster.
Click through for the process.
Comments closedLandon Robinson shows how to sort data in Spark RDDs and DataFrames:
In the analysis section of Spark Starter Guide 4.6: How to Aggregate Data, we asked these questions: “Who is the youngest cat in the data? Who is the oldest?”
Let’s use ordering in Spark as an alternative method to answer those same questions, and achieve the same result. Specifically, let’s again find the youngest and oldest cats in the data.
Click through for plenty of examples.
Comments closed