Multi-options model

from prayas import *

The experiment consists of multiple variants and in each variant the visitor has one ore more options to choose. A detailed explanation of the methodology is available in Bayesian A/B Testing for Business Decisions by Shafi Kamalbasha and Manuel J. A. Eugster (2020).

In this example, the experiment consists of two variants with each variant having 9 different options from which the visitor can choose:

m = MultiOptionsModel(variants=["Original", "Progressive"],
                      options=[9, 9],
                      baseline="Original")

In addition to the conversion measure, we are also interested in measuring the revenue and the gain both in Euro:

rev_a = [27.95,  47.95,  63.95,
         35.95,  63.95,  79.95,
         79.95, 151.95, 223.95]
rev_b = [34.95,  59.95,  79.95,
         37.95,  67.95,  84.95,
         69.95, 132.95, 195.95]

m.add_measure("revenue",
              success_value=[rev_a, rev_b])

m.add_measure("gain",
              success_value=[rev_a, rev_b],
              nonsuccess_value=[np.repeat(-0.06, 9),
                                np.repeat(-0.04, 9)])

The full model specification for this experiment is:

print(m)

Multi-options model
Variants              : Original, Progressive
Baseline              : Original
Measures              : conversion, revenue, gain
Primary measure       : conversion
Maximum loss threshold: 5

Set the result of the experiment:

m.set_result(successes=[[50, 5, 5, 28, 7, 5, 20, 1, 6],
                        [28, 3, 6, 30, 6, 5, 27, 6, 3]],
             trials=[8067, 8082])

Investigate the result:

m.plot();

The plot provides the posteriors of the measures ‘conversion rate’, ‘revenue’, and ‘gain’. We can already see that for different measures the location of the posteriors to each other is different.

Get details on the result:

m.score_baseline()

	Variant	Measure	ProbabilityToBeBest	ProbabilityToBeatBaseline	UpliftFromBaseline	PotentialLossFromBaseline	MaxUplift	MaxPotentialLoss
0	Progressive	gain	0.89115	0.89045	41.623016	1.981303	42.230377	1.957154
1	Original	conversion	0.79665	0.00000	0.000000	0.000000	10.695682	1.450911
2	Original	revenue	0.50920	0.00000	0.000000	0.000000	0.463493	5.613769
3	Progressive	revenue	0.49080	0.49015	-0.579831	6.336800	-0.461354	6.372550
4	Progressive	conversion	0.20335	0.20305	-9.654056	10.990136	-9.662240	11.121591
5	Original	gain	0.10885	0.00000	0.000000	0.000000	-29.691531	31.152837

The progressive variant has lower conversion rates than the original variant (-10% on average) but the revenue per visitor is almost the same for both variants since the probability to be best is almost 50% for both variants; both variants are equally likely to be the best. The progressive variant has less conversions for the lower priced option compared to the original, but makes up for the lost revenue with higher conversions on the premium product. For the metric gain per visitor, progressive discount had the highest probability to be best with a low expected loss of 2%.