Press "Enter" to skip to content

Avoiding Statistical Mistakes

Adrian Sampson explains some common mistakes in statistical analysis, particularly in computer science papers:

It’s tempting to think, when p \ge \alphapα, that you’ve found the opposite thing from the p < \alphap<αcase: that you get to conclude that there is no statistically significant difference between the two averages. Don’t do that!

Simple statistical tests like the tt-test only tell you when averages are different; they can’t tell you when they’re the same. When they fail to find a difference, there are two possible explanations: either there is no difference or you haven’t collected enough data yet. So when a test fails, it could be your fault: if you had run a slightly larger experiment with a slightly larger NN, the test might have successfully found the difference. It’s always wrong to conclude that the difference does not exist.

It’s an interesting read.  H/T Emmanuelle Rieuf.