Reynold Xin, et al, debug an interesting test case:
While we were pretty happy with the improvement, we noticed that one of the test cases in Databricks started failing. To simulate a hanging query, the test case performed a cross join to produce 1 trillion rows.
spark.range(1000 * 1000).crossJoin(spark.range(1000 * 1000)).count()On a single node, we expected this query would run infinitely or “hang.” To our surprise, we started seeing this test case failing nondeterministically because sometimes it completed on our Jenkins infrastructure in less than one second, the time limit we put on this query.
You’re not going to get this performance against a real data set, but it was interesting reading their troubleshooting notes.