Holger von Jouanne-Diedrich walks us through a simple example of A/B testing and analysis using R:
The bad news is, that you have to understand a little bit about statistical hypothesis testing, the good news is that if you read the following post, you have everything you need (plus, as an added bonus R has all the tools you need already at hand!): From Coin Tosses to p-Hacking: Make Statistics Significant Again! (ok, reading it would make it over one minute…).
Check out that article and the example in the blog post as well. R makes it really easy to perform this sort of analysis.