Alex Woodie covers some of the improvements available to us in Apache Spark 3.0:
The addition of join hints further enhances the accuracy of the compiler when the built-in algorithms deliver a suboptimal plan. “When the compiler is unable to make the best choice, users can use join hints to influence the optimizer to choose a better plan.”
Ah, join hints—the double-edged sword.