Kellyn Pot’vin-Gorman goes through a rough experience:
Disclaimer: I’m not a big fan of benchmark data. I find it doesn’t provide us as much value in the real world as we’d like to think it does. As Cary Milsap says, “You can’t hardware your way out of a software problem” and I find that many folks think that if they just get the fastest hardware, their software problems will go away and this just isn’t true. Sooner or later, it’s going to catch up with you- and it rarely tells you what your real database workload needs to run most efficiently or what might be running in your database that could easily be optimized to save the business time and money.
The second issue is that when comparing different workloads or even worse, different platforms or applications, using the same configuration can be detrimental to the benchmarks collected, which is what we’ll discover in this post.
That said, Kellyn dives into the problem and documents several of the issues in building out this test.