Engineers at Treselle Systems have put together a two-part series on text normalization using Apache Spark. First, they walk through normalizing the text:
We have used Spark shared variable “broadcast” to achieve distributed caching. Broadcast variables are useful when large datasets need to be cached in executors. “stopwords_en.txt” is not a large dataset but we have used in our use case to make use of that feature.
What are Broadcast Variables?
Broadcast variables in Apache Spark is a mechanism for sharing variables across executors that are meant to be read-only. Without broadcast variables, these variables would be shipped to each executor for every transformation and action, which can cause network overhead. However, with broadcast variables, they are shipped once to all executors and are cached for future reference.
Note: Stage 2 has both reduceByKey() and sortByKey() operations and as indicated in job summary “saveAsTextFile()” action triggered Job 2. Do you have any guess whether Stage 2 will be further divided into other stages in Job 2? The answer is: yes Job 2 DAG: This job is triggered due to saveAsTextFile() action operation. The job DAG clearly indicates the list of operations used before the saveAsTextFile() operations.Stage 2 in Job 1 is further divided into another stage as Stage 2. In Stage 2 has both reduceByKey() and sortByKey() operations and both operations can shuffle the data so that Stage 2 in Job 1 is broken down into Stage 4 and Stage 5 in Job 2. There are three stages in this job. But, Stage 3 is skipped. The answer for the skipped stage is provided below “What does “Skipped Stages” mean in Spark?” section.
There’s some good information here if you want to become more familiar with how Spark works.
So in just 1 minute and 34 seconds you have migrated all of the following from one server to another.
All SP_Configure settings
Any Custom Error Messages
Any SQL Credentials
All Linked Servers
Database Mail along with Configuration and Profiles and Accounts
All user objects in System Databases
All Backup Devices
Any System Triggers
All User Databases
Any Data collectors
Any Security Audits
All Endpoints, Policy Management, Resource Governor, Extended Events
And Finally All SQL Server Agent Jobs, Schedules, Operators, Alerts
These are probably very small databases (as it was a test instance), but dbatools is quite impressive.
Hadley Wickham has two announcements. First, for a slew of tidyverse packages:
Over the couple of months there have been a bunch of smaller releases to packages in the tidyverse. This includes:
- forcats 0.2.0, for working with factors.
- readr 1.1.0, for reading flat-files from disk.
- stringr 1.2.0, for manipulating strings.
- tibble 1.3.0, a modern re-imagining of the data frame.
This blog post summarises the most important new features, and points to the full release notes where you can learn more.
Second, a new version of dplyr is coming:
dplyr 0.6.0 is a major release including over 100 bug fixes and improvements. There are three big changes that I want to touch on here:
- Improved encoding support (particularly for CJK on windows)
- Tidyeval, a new framework for programming with dplyr
You can see a complete list of changes in the draft release notes.
You can already get a tech preview of the new dplyr if you’re interested in trying it out.
Power BI Privacy Levels “specify an isolation level that defines the degree that one data source will be isolated from other data sources”. After working through some testing scenarios and trying to discover the real impact to data security, I was unable to effectively show how this might have any bearing on data security in Power BI. During one test was I shown a warning about using data from a website with data I had marked Organizational and Private. In all cases, I was able to merge the data in the query and in the relationships with no warning or filtering. All of the documentation makes the same statement and most bloggers are restating what is found in the Power BI documentation as were not helpful. My takeaway after reviewing this for a significant amount of time is to not consider these settings when evaluating data security in Power BI. I welcome comments or additional references which actually demonstrate how this isolation actually works in practice. In most cases, we are using organizational data within our Power BI solutions and will not be impacted by this setting and my find improved performance when disabling it.
As Steve notes, this is not really a security feature. Instead, it’s intended to be more a warning to users about which data is confidential and which is publicly-sharable .
The mind-blowing part to me is the <> operator – that seemed utterly crazy to me.
And if you add another table, and join to it, guess what happens:
The answer might be a bit surprising.
Are you tired of right-clicking on your Biml files to Check Biml for Errors or to Generate SSIS Packages? Did you know that you can create your own BimlExpress Keyboard Shortcuts? 🙂
Read on to see how to do this.