Press "Enter" to skip to content

A Primer on A/B Testing for Engineers

John Mount performs some testing:

I’d like to discuss a simple variation of A/B testing in an engineering style.
By “an engineering style” I mean:

  • We will work a simulated example to see that the system works as claimed.
  • We will exhibit examples of problems before trying to fix them.
  • We will demonstrate all of the top level claims as calculations, and not delegate these to references.
  • We will leave fundamental math to the references, and not try to re-derive it.

In my opinion far too few A/B testing treatments check soundness, even on simulated data. This makes it easy for such articles to leave out important steps. If a relied on reference omits a step, the derived work may have to do the same.
We will implement the experiment design directly, instead of using a canned power calculator so we have a place to discuss some of the design issues in A/B test design.

This is an excellent dive into the topic and I highly recommend taking the time to read it.