Polybase With Compression

I have a post looking at Polybase support for different compression formats:

This is a very interesting set of results.  First, 7Zip archived files do not work with the default encoding.  I’m not particularly surprised by this result, as 7Zip support is relatively scarce across the board and it’s a niche file format (though a very efficient format).

The next failure case is tar.  Tar is a weird case because it missed the first row in the file but was able to collect the remaining 776 records.  Same goes for .tar.gz.  I unpackaged the .tar file and the constituent SecondBasemen.csv file did in fact have all 777 records, so it’s something weird about the codec.

Stick to BZip2 and GZip if you’re using flat files.

Related Posts

Connecting PolyBase to Spark

I have a blog post connecting PolyBase to a Spark cluster: If you do define your Spark DataFrames well, you get a much happier result. Here’s me creating a better-looking DataFrame in Spark: import org.apache.spark.sql.functions._ spark.sql(""" SELECT INT(SUMLEV) AS SummaryLevel, INT(COUNTY) AS CountyID, INT(PLACE) AS PlaceID, BOOLEAN(PRIMGEO_FLAG) AS IsPrimaryGeography, NAME AS Name, POPTYPE AS PopulationType, […]

Read More

PolyBase on Linux

I have a post showing how to set up PolyBase on Linux: Now that we have SQL Server on Linux installed, we can begin to install PolyBase. There are some instructions here but because we started with the Docker image, we’ll need to do a little bit of prep work. Let’s get our shell on. First, run docker […]

Read More

Categories

November 2016
MTWTFSS
« Oct Dec »
 123456
78910111213
14151617181920
21222324252627
282930