Title: Tools for A/B testing with machine learning: No analysis, programming or stopping decisions required?

This brief episode is a teaser for a scientific paper on a tool for experimentation. It was used at Harvard for people to deploy experiments on copy content (text, links, questions). This was done without the designers having to do any programming, any analysis, or even decision making about when to stop the experiment. That was done with a statistical machine learning algorithm, the Bayesian Bandit algorithm Thompson sampling, which changed the probability of assigning users to variants in real-time, as data was being collected. For example, the randomization probability of assigning variants A1 vs A2, would start at 50/50, change to 60/40, 30/70, 45/55, 80/20, 90/10 and so on. The goal was to phase in the variant in proportion to the probability that it would give the highest score in the target metric.

The paper is at www.josephjaywilliams.com/pape..., search for 'tools for randomized experiments'. Feel free to share questions, skepticisms, opportunities for applying these kinds of techniques or tools! There are >10 more papers Joseph has on this approach, and 100s in the literature, and excited to engage in discussion!


This is an experiment in the next stage of CRO cafe. A professor in experimentation & optimization is interviewing themself, by asking himself questions from listeners and other sources. The professor is www.josephjaywilliams.com and has published over 80 papers since he ran his first experiment in 2004. Like many of you, he's excited to keep innovating in tools and methods for experimentation, personalization, AI, ML until he says goodbye to this lively, lovely world!