Scan QR code to get the slides.

## Beyond Two Groups: Generalized Bayesian A/B[/C/D/E...] Testing

Eric J. Ma

PyCon 2019, Cleveland, OH

 Investigator, Scientific Data Analysis ScD, Biological Engineering, 2017 Bayesian Stats, ML, Network Science

Most data science talks tell you how to do 2 group comparisons, i.e. "A/B" tests, with the t-test.

## I have Three Messages

• Message 1: You don't have to be stuck with two groups and normal distribution assumptions.
• Message 2: PyMC3 is a great language for probabilistic modelling.
• Message 3: With Bayesian modelling, we can move away from canned statistical procedures, and do statistics in a principled fashion.

## Assumed Knowledge

Bayesian inference: updating beliefs with new data.

Some familiarity with PyMC3 syntax.

When I see a lonely number in a news report, it always triggers an alarm: What should this lonely number be compared to?

## Outline

• Vignette 1: Analysis beyond two groups.
• Vignette 2: Non-T-distributed data, beyond two groups.

## Vignette 1: Drug IQ Example

With example PyMC3 code that analyzes >2 groups.

Modified from Kruschke, 2013.

## Not statistically significant!!!

What if I had more groups tested together?

## Violated Assumptions

• Data are normally distributed?
• Standard deviations are equal?
• Independent samples?

Let's stop violating assumptions.

• Let's do this Bayesian style instead.
• Carefully model the data likelihood and its key parameters.

## Two Key Lessons

### Posteriors are all you need

But what if our data were not t-distributed?

## Vignette 2: Clicking on Websites.

With example PyMC3 code that models non-T-distributed data.

Let's talk about clicking on websites.

Which intervention helped our customers find what they were looking for?

• Zero-inflation: Model P(no click).
• Count data: Continuous distribution is inappropriate.

Zero-inflated negative binomial distribution models two processes.

• No intent to click
• "how many clicks before stopping?"

Putting images of rooms increases probability of clicking on anything at all.

Putting images of rooms maintains probability of purchasing an item.

### Key Takeaways

• 3 or more groups possible.
• We didn't need the t-test.
• Carefully constructing the white-box model let us separate out interpretable effects.

## I have Three Messages

• Message 1: You don't have to be stuck with two groups and normal distribution assumptions.
• Message 2: PyMC3 is a great language for probabilistic modelling.
• Message 3: With Bayesian modelling, we can move away from canned statistical procedures, and do statistics in a principled fashion.

Bonus Message: You can better interpret your data with Bayesian methods!