Andy Brownsword gives Spark a try:
I’ve spent years shaping data with SQL Server, however after pulling at the threads of Fabric I’m opening notebooks and finding PySpark.
At first glance the difference is stark, but it’s not quite the dramatic shift it appears. If you’re not familiar, let’s look at what’s very similar, and where the true differences are.
There’s plenty of nuance in the syntax differences and behavioral differences between the platforms, but Spark SQL is just as ANSI compliant at this state as pretty much any other platform, and PySpark feels a lot like a chained quasi-functional approach to SQL because of Spark’s Scala heritage.