PolyBase and Pushdown Limitations

I have a post covering something I learned about predicate pushdown against Hadoop in PolyBase:

Before I start, let’s talk about predicate pushdown for a moment. The gist of it is that when you have data in two sources, you have two options for combining the data:

1. Bring the data in its entirety from your remote source to your local target and work as though everything were in the local target to begin with. I’ll call this the streaming approach.

2. Send as much of your query’s filters, projections, and pre-conditions to the remote source, have the remote source perform some of the work, and then have the remote source send its post-operative data to the local target. Then, the local target once more treats this as though it were simply local data. This is the pushdown approach because you push down those predicates (that is, filters, projections, and pre-conditions).

Click through for the unfortunate finding and also vote up my UserVoice feature request if you want to see string columns as filters.

Related Posts

Connecting PolyBase to Spark

I have a blog post connecting PolyBase to a Spark cluster: If you do define your Spark DataFrames well, you get a much happier result. Here’s me creating a better-looking DataFrame in Spark: import org.apache.spark.sql.functions._ spark.sql(""" SELECT INT(SUMLEV) AS SummaryLevel, INT(COUNTY) AS CountyID, INT(PLACE) AS PlaceID, BOOLEAN(PRIMGEO_FLAG) AS IsPrimaryGeography, NAME AS Name, POPTYPE AS PopulationType, […]

Read More

PolyBase on Linux

I have a post showing how to set up PolyBase on Linux: Now that we have SQL Server on Linux installed, we can begin to install PolyBase. There are some instructions here but because we started with the Docker image, we’ll need to do a little bit of prep work. Let’s get our shell on. First, run docker […]

Read More

Categories

March 2019
MTWTFSS
« Feb Apr »
 123
45678910
11121314151617
18192021222324
25262728293031