Extended Understanding of Bayesian A/B Testing

How to use Bayesian A/B Testing Efficiently

In my earlier post, I talked about the fundamentals of Bayesian A/B testing and described how it is able to answer one of the most fundamental questions in all A/B tests — Is treatment better than control? Beyond that, I have also left a few unanswered questions that you might be facing when actually applying bayesian A/B testing.

1. What happens if I want to carry out tests with more than 2 groups, e.g. A/B/C/D …?
2. How to reduce the computational cost?
3. Is there any technique to choose a prior distribution?

In this post, I will try to answer the above questions. Now let’s get into it.

Table of Contents

• Multiple Treatments
• Monte Carlo Sampling
• Technique for Selecting a Prior
• Conclusion
• Credit and References

Photo by Minator Yang on Unsplash

Multiple Treatments

In our last post, we derived this final cost function

Where it is only applicable to two groups, e.g. one treatment and one control. Naturally, you would think about what to do with multiple treatments. Actually, it is not hard to extend this to multiple treatments by tweaking the loss function.

Loss Function

Let’s say we have 3 variants A, B, and C. Then we can define our loss function

This means that by choosing variant A, the maximum loss we could have.

The same goes for cases of choosing B and C.

Now plug this into the expected loss function, we have

Similarly for variants B and C.

The Computational Cost

Note that in the actual computation, in order to calculate the above integrals, you will need to iterate all possible values of each variant. For example, to calculate the loss for 3 variants, each with 100 possible values.

expected_loss = 0
for i in range(100):
  for j in range(100):
    for k in range(100):
        # the loss of select variant A…