Miles Cole groups things together:
If there anything that data engineers agree about, it’s that table compaction is important. Often one of the first big lessons that folks will learn early on is that not compacting tables can present serious performance issues: you’ve gotten your lakehouse pilot approved and it’s been running for a couple months in production and you find that both reads and writes are increasingly getting slower and slower while your data volumes have not increased drastically. Guess what, you almost surely have a “small file problem”.
What engineers won’t always sing the same tune on is how and when to perform table compaction.
Read on for a dive into the power of compaction (converting a large number of small files into a small number of large files) and plenty of tips along the way.