PolyBase — SQL to SQL

I have a post covering PolyBase from SQL Server to SQL Server:

Historically, PolyBase has three separate external entities: external data sources, external file formats, and external tables. External data sources tell SQL Server where the remote data is stored. External file formats tell SQL Server what the shape of that data looks like—in other words, CSV, tab-separated, Parquet, ORC, etc. External tables tell SQL Server the structure of some data of a particular external file format at a particular external data source.

With PolyBase V2—connectivity with SQL Server, Cosmos DB, Oracle, Spark, Hive, and a boatload of other external data sources—we no longer need external file formats because we ingest structured data. Therefore, we only need an external data source and an external table. You will need SQL Server 2019 to play along and I’d recommend keeping up on CTPs—PolyBase is under active development so being a CTP behind may mean hitting bugs which have subsequently been fixed.

I want this to get even better, to the point where external tables are a no-brainer over linked servers in terms of performance.

Related Posts

Connecting PolyBase to Spark

I have a blog post connecting PolyBase to a Spark cluster: If you do define your Spark DataFrames well, you get a much happier result. Here’s me creating a better-looking DataFrame in Spark: import org.apache.spark.sql.functions._ spark.sql(""" SELECT INT(SUMLEV) AS SummaryLevel, INT(COUNTY) AS CountyID, INT(PLACE) AS PlaceID, BOOLEAN(PRIMGEO_FLAG) AS IsPrimaryGeography, NAME AS Name, POPTYPE AS PopulationType, […]

Read More

PolyBase on Linux

I have a post showing how to set up PolyBase on Linux: Now that we have SQL Server on Linux installed, we can begin to install PolyBase. There are some instructions here but because we started with the Docker image, we’ll need to do a little bit of prep work. Let’s get our shell on. First, run docker […]

Read More

Categories

May 2019
MTWTFSS
« Apr Jun »
 12345
6789101112
13141516171819
20212223242526
2728293031