Mark Kromer has a few tips on improving ADF data flow join performance:
When you include literal values in your join conditions, Spark may see that as a requirement to perform a full cartesian product first, then filter out the joined values. But if you ensure that you (1) have column values from both sides of your join condition, you can avoid this Spark-induced cartesian product and improve the performance of your joins and data flows. (2) Avoid use of literal conditions to represent the results of one side of your join condition.
In other words, avoid this for your join condition:
source1@movieId == '1'Instead, implement that with a dummy derived column.
There are several good tips in this post.