Alexander Arvidsson diagnoses an interesting problem:
I ran this several times to see if there was a pattern to the madness, and it turned out it was. All waits were concentrated in database ID 2 – TEMPDB. Many people perk up by now and jump to the conclusion that this is your garden variety SGAM/PFS contention – easily remedied with more TEMPDB files and a trace flag. But, alas- this was further inside the TEMPDB. The output from the query above gave me the exact page number, and plugging that into DBCC PAGE gives the metadata object ID.
His conclusion is to reduce temp table usage and/or use memory-optimized tables. We solved this problem with replacing temp tables with memory-optimized TVPs in our most frequently-used procedures.