John Mount performs some testing:
I’d like to discuss a simple variation of A/B testing in an engineering style.
By “an engineering style” I mean:
- We will work a simulated example to see that the system works as claimed.
- We will exhibit examples of problems before trying to fix them.
- We will demonstrate all of the top level claims as calculations, and not delegate these to references.
- We will leave fundamental math to the references, and not try to re-derive it.
In my opinion far too few A/B testing treatments check soundness, even on simulated data. This makes it easy for such articles to leave out important steps. If a relied on reference omits a step, the derived work may have to do the same.
We will implement the experiment design directly, instead of using a canned power calculator so we have a place to discuss some of the design issues in A/B test design.
This is an excellent dive into the topic and I highly recommend taking the time to read it.