Richard Swinbank deals with a totally-not-deprecated platform:
The underlying problem here appears to be that the Spark connector is simply not supported in v3.4. There’s very little I can find to officially confirm or deny this, but an answer to this question on Microsoft Q&A backs this up. The answer also suggests a few options, including:
- downgrade to Spark 3.3 – this isn’t an option because it’s end-of-life
- migrate to Fabric – long term this is good idea, but it’s not a quick fix for this problem.
- use alternative data access methods, e.g. using the Azure Data Lake Storage Gen2 connector.
In this article, I take a look at option (3).
Click through for Richard’s workaround.