Avoiding Statistical Mistakes

Adrian Sampson explains some common mistakes in statistical analysis, particularly in computer science papers:

It’s tempting to think, when p \ge \alphapα, that you’ve found the opposite thing from the p < \alphap<αcase: that you get to conclude that there is no statistically significant difference between the two averages. Don’t do that!

Simple statistical tests like the tt-test only tell you when averages are different; they can’t tell you when they’re the same. When they fail to find a difference, there are two possible explanations: either there is no difference or you haven’t collected enough data yet. So when a test fails, it could be your fault: if you had run a slightly larger experiment with a slightly larger NN, the test might have successfully found the difference. It’s always wrong to conclude that the difference does not exist.

It’s an interesting read.  H/T Emmanuelle Rieuf.

Related Posts

Learning Confluent Kafka

Stephane Maarek has some pointers on where to go to learn Confluent Kafka: I teach online about Apache Kafka, and a very frequent and recurring question I get is: How can I learn Confluent Kafka? Let’s get right to it! I’ve gone through a couple of Stephane’s Kafka courses and they’re excellent. There’s still a […]

Read More

Using Show-Command in Powershell

Kenneth Fisher walks through the Show-Command cmdlet in Powershell: Years ago I blogged about how I like to use the SSMS scripting feature to learn how to do things. Well now I’m starting to learn Powershell and it turns out there is a GUI here as well that will help me learn to script. At least […]

Read More


December 2016
« Nov Jan »