Extended Understanding of Bayesian A/B Testing

How to use Bayesian A/B Testing Efficiently

Jeremy Zhang
5 min readFeb 16, 2023

In my earlier post, I talked about the fundamentals of Bayesian A/B testing and described how it is able to answer one of the most fundamental questions in all A/B tests — Is treatment better than control? Beyond that, I have also left a few unanswered questions that you might be facing when actually applying bayesian A/B testing.

  1. What happens if I want to carry out tests with more than 2 groups, e.g. A/B/C/D …?
  2. How to reduce the computational cost?
  3. Is there any technique to choose a prior distribution?

In this post, I will try to answer the above questions. Now let’s get into it.

Table of Contents

  • Multiple Treatments
  • Monte Carlo Sampling
  • Technique for Selecting a Prior
  • Conclusion
  • Credit and References
Photo by Minator Yang on Unsplash

Multiple Treatments

In our last post, we derived this final cost function

Where it is only applicable to two groups, e.g. one treatment and one control. Naturally, you would think about what to do with multiple treatments. Actually, it is not…

--

--

Jeremy Zhang
Jeremy Zhang

Written by Jeremy Zhang

Hmm…I am a data scientist looking to catch up the tide…

No responses yet