Scan QR code to get the slides.

Beyond Two Groups: Generalized Bayesian A/B[/C/D/E...] Testing

Eric J. Ma

PyCon 2019, Cleveland, OH

About Me


Investigator, Scientific Data Analysis	ScD, Biological Engineering, 2017	Bayesian Stats, ML, Network Science

Most data science talks tell you how to do 2 group comparisons, i.e. "A/B" tests, with the t-test.

I have Three Messages

Message 1: You don't have to be stuck with two groups and normal distribution assumptions.
Message 2: PyMC3 is a great language for probabilistic modelling.
Message 3: With Bayesian modelling, we can move away from canned statistical procedures, and do statistics in a principled fashion.

Assumed Knowledge

Bayesian inference: updating beliefs with new data.

Some familiarity with PyMC3 syntax.

When I see a lonely number in a news report, it always triggers an alarm: What should this lonely number be compared to?

Outline

Vignette 1: Analysis beyond two groups.
Vignette 2: Non-T-distributed data, beyond two groups.

Vignette 1: Drug IQ Example

With example PyMC3 code that analyzes >2 groups.

Modified from Kruschke, 2013.

Not statistically significant!!!

What if I had more groups tested together?

Violated Assumptions

Data are normally distributed?
Standard deviations are equal?
Independent samples?

Let's stop violating assumptions.

Let's do this Bayesian style instead.
Carefully model the data likelihood and its key parameters.

Posteriors are all you need

Two Key Lessons

Posteriors are all you need

### Don't stop at two groups

```python
                        n_groups = len(df['group'].unique())
                        with pm.Model() as model:
                            # Model Parameters
                            mu = pm.Normal('mu', mu=0, sd=10, shape=n_groups)

# Assume each group has its own variance.
                            sigma = pm.HalfCauchy('sigma', beta=5, shape=n_groups)
                            nu = pm.Exponential('nu', lam=1/29.)

# Likelihood
                            # Explicitly encode liklihood as t-distributed.
                            like = pm.StudentT(
                                'like',
                                mu=mu[df['group_enc']],
                                sd=sigma[df['group_enc']],
                                nu=nu,
                                observed=df['iq'].values
                            )

trace = pm.sample(2000)
                        ```

But what if our data were not t-distributed?

Vignette 2: Clicking on Websites.

With example PyMC3 code that models non-T-distributed data.

Let's talk about clicking on websites.

Which intervention helped our customers find what they were looking for?

Zero-inflation: Model P(no click).
Count data: Continuous distribution is inappropriate.

Zero-inflated negative binomial distribution models two processes.

No intent to click
"how many clicks before stopping?"

```python
                        with pm.Model() as model:
                            # Priors for ZINB model: mu and alpha.
                            mu_prior_lam = pm.HalfCauchy('mu_prior_lam', beta=1)
                            mu = pm.HalfNormal('mu', sd=mu_prior_lam, shape=(3,))
                            mu_enc = mu[data['experiment_group_enc']]

alpha_prior_lam = pm.HalfCauchy('alpha_prior_lam', beta=1)
                            alpha = pm.HalfNormal('alpha',sd=alpha_prior_lam, shape=(3,))
                            alpha_enc = alpha[data['experiment_group_enc']]

# Model the probability of purchasing.
                            p_purchase = pm.Deterministic(
                                'p_purchase', alpha / (mu + alpha)
                            )

# Model the probability of being non-zero-inflating group.
                            p = pm.Beta('p', alpha=1, beta=1, shape=(3,))
                            p_enc = p[data['experiment_group_enc']]

like = pm.ZeroInflatedNegativeBinomial(
                                'like',
                                mu=mu_enc,
                                alpha=alpha_enc,
                                psi=p_enc,
                                observed=data['clicks']
                            )
                        ```

Putting images of rooms increases probability of clicking on anything at all.

Putting images of rooms maintains probability of purchasing an item.

Key Takeaways

3 or more groups possible.
We didn't need the t-test.
Carefully constructing the white-box model let us separate out interpretable effects.

I have Three Messages

Message 1: You don't have to be stuck with two groups and normal distribution assumptions.
Message 2: PyMC3 is a great language for probabilistic modelling.
Message 3: With Bayesian modelling, we can move away from canned statistical procedures, and do statistics in a principled fashion.

Bonus Message: You can better interpret your data with Bayesian methods!

Beyond Two Groups: Generalized Bayesian A/B[/C/D/E...] Testing

About Me

I have Three Messages

Assumed Knowledge

Outline

Vignette 1: Drug IQ Example

Not statistically significant!!!

Violated Assumptions

Posteriors are all you need

Two Key Lessons

Posteriors are all you need

Vignette 2: Clicking on Websites.

Key Takeaways

I have Three Messages

Thank you!

Learn More