Jon Lunn checks the status of a data pipeline:
How do you check if a pipeline is running, not from the monitor, but from your Data Pipelines?
Maybe you’re like me and you have a Data Pipeline process that needs to check if some other pipeline else is running. In my case I have to check if a process is running due to Delta tables liking you to have one process writing to them, otherwise you can get concurrency issues as two items are trying to update the same delta table metadata file.
Those tricky metadata items like the process to be exclusive. It’s not just a Delta table issue; this can happen with regular SQL databases tables. So you can use this for anything you want to stop a locking issue or have an exclusive access to an object or just don’t want a process to run while another is doing its thing.
Read on to see how you can check the current status of a data pipeline from within a different data pipeline.