Avoiding Statistical Mistakes

Adrian Sampson explains some common mistakes in statistical analysis, particularly in computer science papers:

It’s tempting to think, when p \ge \alphapα, that you’ve found the opposite thing from the p < \alphap<αcase: that you get to conclude that there is no statistically significant difference between the two averages. Don’t do that!

Simple statistical tests like the tt-test only tell you when averages are different; they can’t tell you when they’re the same. When they fail to find a difference, there are two possible explanations: either there is no difference or you haven’t collected enough data yet. So when a test fails, it could be your fault: if you had run a slightly larger experiment with a slightly larger NN, the test might have successfully found the difference. It’s always wrong to conclude that the difference does not exist.

It’s an interesting read.  H/T Emmanuelle Rieuf.

Related Posts

Analysis Of A Failed Project

Eugene Meidinger looks back at a big project which fell apart: So the first issue was that the software was built in-house by another company in the same industry. Imagine, for example, if a large bakery had created an ERP system and another large bakery wanted to move to that system. Sounds great, right? Well, […]

Read More

What You Can Learn At SQL Saturday

Nate Johnson shares a few things he picked up at the SQL Saturday in San Diego: This was an interesting and even slightly entertaining session presented by Max @ SQLHA. One analogy that really stood out to me was this: SANs have become a bit like the printer industry — You don’t pay a lot for the enclosure, […]

Read More


December 2016
« Nov Jan »