Module experiment

class prayas.experiment.Experiment(name)

Class for experiments

Experiments provide a simple way to continuously monitor the experiment performance and to decide when to stop based on the maximum potential loss of the variants.

Parameters

name – Name of the experiment

property result

Get the latest result of the experiment

add_data(ts, successes, trials)

Add data to the experiment

Parameters

• ts – Timestamp of the data, can be any object

• successes – Successes per variant and option

• trials – Trials per variant and option

Returns

None, used for its side-effect

monitor_decision(days=5)

Compute the decision to continue or stop the experiment.

Parameters

days – Number of days to be lower than threshold

Returns

Data frame with the daily decision to continue or stop based on the estimated loss

monitor_plot()

Plot the loss of the variants.

Returns

Seaborn FacetGrid object

monitor_score_baseline()

Compute the scoring against the baseline for each timestamp of the experiment.

Returns

Data frame with baseline scoring for each timestamp of the experiment