Press "Enter" to skip to content

Extracting Lineage Information from Microsoft Fabric Spark Notebooks

Gerhard Brueckl retrieves some information:

When building data platforms and data pipelines to populate them, one of the most challenging parts is orchestration. You need to figure out dependencies between your pipelines and activities, know the run times/duration and make sure to run them only when all upstream pipelines succeeded. From a technical point of view, this can be easily accomplished within Microsoft Fabric using runMultiple(). (I really dont know why Databricks doesn’t offer something similar out of the box?!?)

Now the next question that comes up is: “Where do I get those dependencies from?” – and this is exactly what this blog post is about!

Click through to learn how.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.