Peter Eisentraut lays out the explanation:
Upgrades between PostgreSQL major versions are famously annoying. You can’t just install the server binaries and restart, because the format of the data directory is incompatible.
Why is that? Why can’t we just keep the data format compatible?
Perhaps surprisingly, the data format is actually mostly compatible, but not completely. There are just a few things missing that are very hard to solve.
Perhaps I’m not as sympathetic as I should be to the core developers, but there are other RDBMS platforms that have a direct path for upgrade from version to version, so it’s hardly insurmountable.