John Cook asks which is clearer, 1 or 2? 3 or 4? 4 or 6?
One problem with A/B testing is that your results may depend on the order of your tests.
Suppose you’re testing three options: X, Y, and Z. Let’s say you have three market segments, equal in size, each with the following preferences.
This is known as the Condorcet paradox of voting.
John also introduces the problem of interaction effects:
Suppose you’re debating between putting a photo of a car or a truck on your web site, and you’re debating between whether the vehicle should be red or blue. You decide to use A/B testing, so you test whether customers prefer a red truck or a blue truck. They prefer the blue truck. Then you test whether customers prefer a blue truck or a blue car. They prefer the blue truck.
Maybe customers would prefer a red car best of all, but you didn’t test that option. By testing vehicle type and color separately, you didn’t learn about the interaction of vehicle type and color.
Click through for both posts as well as some good insights.