Extended Understanding of Bayesian A/B Testing

How to use Bayesian A/B Testing Efficiently

In my earlier post, I talked about the fundamentals of Bayesian A/B testing and described how it is able to answer one of the most fundamental questions in all A/B tests — Is treatment better than control? Beyond that, I have also left a few unanswered questions that you might be facing when actually applying bayesian A/B testing.

1. What happens if I want to carry out tests with more than 2 groups, e.g. A/B/C/D …?
2. How to reduce the computational cost?
3. Is there any technique to choose a prior distribution?

In this post, I will try to answer the above questions. Now let’s get into it.

Table of Contents

• Multiple Treatments
• Monte Carlo Sampling
• Technique for Selecting a Prior
• Conclusion
• Credit and References

Photo by Minator Yang on Unsplash

Multiple Treatments

In our last post, we derived this final cost function

Where it is only applicable to two groups, e.g. one treatment and one control. Naturally, you would think about what to do with multiple treatments. Actually, it is not…