A/B Testing & Statistical Thinking with 2022 BRFSS Data

2026 GitHub Download Slides

Python SciPy statsmodels

Public health agencies run campaigns all the time: smoking cessation, physical activity, flu shots, cancer screening. Each one is a bundle of decisions about the message, the channel, the timing, and the audience. Should the message use fear or encouragement? Should reminders go out daily or weekly? Should we mail a postcard or send a text?

Good intentions aren't enough here. If an intervention fails, people keep doing the risky thing. If it works, lives improve. A/B testing is how we find out which one we're looking at. It replaces opinion with evidence, and it lets us put limited resources into the approaches that actually work. It also helps with health equity: once you have a well-designed test, you can check whether an intervention works equally well across demographic subgroups or only for some of them.

I taught this material as part of the VADSTI program. I used the CDC's 2022 Behavioral Risk Factor Surveillance System (BRFSS) as the running example because it's one of the best real-world illustrations of sampling and inference at national scale. In this article I'll go through the core statistical ideas you need to design, run, and interpret an A/B test, in the order I'd teach them.

What Is A/B Testing?

An A/B test is a systematic way to compare two versions of an intervention and see which one performs better. The recipe has five steps:

Identify your target audience.
Randomly assign participants to groups.
Expose one group to the control (the current approach).
Expose the other group to the treatment (the new approach).
Measure and compare the outcomes.

Random assignment is what makes the method work. It makes the two groups alike in every way except the intervention, so any difference in outcomes can fairly be attributed to the intervention.

Example: To test cancer screening outreach, send one randomly chosen group a basic postcard (control) and the other a personalized text message with a direct booking link (treatment), then compare screening rates.

You can test almost anything this way: message content, delivery channel, timing, design. You need just two things, random assignment and a clear, measurable outcome.

Population vs. Sample: How BRFSS Talks About 260 Million People

The population is the entire group we care about, for example all ~260 million U.S. adults aged 18 and older. Surveying all of them is impossible on both time and cost.

A sample is a carefully selected subset. BRFSS surveys roughly 400,000 people a year, which is less than 0.2% of U.S. adults, and is designed to represent the full population.

Here's the analogy I like to use. A simple random sample is like stirring a pot of soup and taking one ladle. Every part of the pot has an equal chance of ending up in the ladle, but by pure chance you might get mostly broth or mostly vegetables. BRFSS uses something closer to stratified sampling. It deliberately takes the right proportion of each "ingredient" (age groups, geographic areas, racial and ethnic groups) so the sample mirrors the population.

The common sampling methods you'll meet:

Method	How it works	Watch out for
Simple random	Everyone has an equal chance of selection, like drawing names from a hat	Small subgroups can be under-represented by chance
Stratified	Split the population into strata, then sample within each	Requires knowing the strata in advance
Systematic	Pick every k-th member from an ordered list	Hidden patterns in the list order
Cluster	Randomly select whole clusters (e.g., counties) and sample everyone in them	Clusters can be internally similar
Convenience	Take whoever is easiest to reach	Usually biased; avoid for inference

Key insight: we don't need to survey everyone. We need the right mix of people to make valid inferences about the whole.

Describing the Data: Mean, Variance, and Standard Deviation

The mean is the average value. For example, BRFSS respondents might exercise an average of 3.2 days per week. For a population and a sample, respectively:

$$\mu = \frac{\sum X}{N} \qquad \bar{X} = \frac{\sum X}{n}$$

The mean only tells you where the center is. The standard deviation tells you how spread out values are around that center, and the variance is its square. Variance is handy inside calculations but hard to interpret because it isn't in the original units ("days squared" isn't a thing anyone thinks in).

$$\sigma = \sqrt{\frac{\sum (X - \mu)^2}{N}} \qquad s = \sqrt{\frac{\sum (X - \bar{X})^2}{n-1}}$$

The $n-1$ in the sample formula is Bessel's correction. A sample tends to underestimate the true spread of the population, so we divide by a slightly smaller number to compensate.

Why spread matters for interventions

Imagine two communities that both average 3.2 days of exercise per week:

Community A (low variability): most people exercise 2 to 4 days a week. A single message like "add one more day of activity" might work for nearly everyone.
Community B (high variability): half never exercise and half exercise daily. The average is the same, but these are two very different populations that need completely different interventions.

The mean is the same and the reality is different. That's why I always tell students to look at the spread before designing anything.

Normal Distributions and Z-Scores

Many health measures, including BMI, blood pressure, and cholesterol, roughly follow a normal distribution: the familiar symmetric bell curve, fully described by its mean (center) and standard deviation (spread). The empirical rule says about 68% of values fall within ±1 SD of the mean, about 95% within ±2 SD, and about 99.7% within ±3 SD.

A z-score tells you how many standard deviations a value sits from the mean:

$$z = \frac{X - \mu}{\sigma}$$

Worked example. Say average BMI is 27 with a standard deviation of 3.5. Someone with a BMI of 33 has:

$$z = \frac{33 - 27}{3.5} \approx 1.7$$

That person's BMI is 1.7 standard deviations above average.

Z-scores are useful in two ways:

Comparing apples and oranges. Someone 1.5 SD above average for BMI can be meaningfully compared to someone 1.5 SD above average for blood pressure, even though the units are completely different.
Flagging risk. A z-score beyond about 2 puts someone in the most extreme ~5% of the population, which is often the signal for public health professionals to step in.

Converting every value to a z-score gives the standard normal distribution (mean 0, SD 1). That's the reference curve behind most of the tests that follow.

The Central Limit Theorem: Why Averages Are Predictable

This is the idea that makes the rest of the article possible.

Individual data is often messy. Self-reported alcohol consumption, for example, is heavily skewed: most people drink little or nothing and a few drink heavily. Now try this:

Randomly select 100 people and compute their average alcohol consumption.
Repeat with 50 different random groups.
Plot those 50 averages.

The averages form a bell curve, even though the underlying data was skewed. That's the Central Limit Theorem (CLT): as sample size grows, the distribution of sample means approaches a normal distribution, whatever shape the original data had. A common rule of thumb is that it works well from about $n \geq 30$.

The spread of those sample means is the standard error:

$$SE = \frac{\sigma}{\sqrt{n}} \quad \text{(or } \frac{s}{\sqrt{n}} \text{ when we estimate } \sigma \text{ from the sample)}$$

The CLT is why BRFSS can survey 400,000 people and make confident statements about 260 million. Individual behavior is unpredictable, but group averages are remarkably predictable. It's also why A/B tests on non-normal outcomes (clicks, screenings, vaccinations) can still use normal-based tests.

Confidence Intervals: Being Honest About Uncertainty

Instead of saying "the average is exactly 3.2 days of exercise per week," a confidence interval gives a range: "the average is between 2.9 and 3.5 days per week."

A 95% confidence interval means that if we repeated the study 100 times and built an interval each time, about 95 of those intervals would contain the true population average.

A narrow interval (say 3.1 to 3.3 days) means a precise estimate.
A wide interval (say 2.5 to 3.9 days) means more uncertainty.

Two things control the width:

Variability in the responses. More consistent answers give narrower intervals.
Sample size. Larger samples give narrower intervals, through that $\sqrt{n}$ in the standard error.

Comparing Two Groups: A Flu Vaccination Test

Now let's run an actual A/B test. We compare two flu vaccination reminder strategies:

Group	Strategy	Vaccinated	Total	Rate
A (control)	Standard postcard	225	500	45%
B (treatment)	Personalized text message	260	500	52%
Difference		35		7 percentage points

The critical question: is this 7-point difference real, or could it have happened by chance even if both methods were equally effective? Hypothesis testing gives us a structured way to answer it.

Reading a confidence interval for a difference

If the two methods were truly identical, the true difference would be zero. So the first thing I check is whether the interval for the difference contains zero:

If the 95% CI for the difference is...	Interpretation
2 to 12 points	Excludes zero, so probably a real benefit, plausibly between 2 and 12 points
−1 to 8 points	Includes zero, so we can't rule out "no effect" (or even a small harm)
−8 to −2 points	Excludes zero, but entirely negative, so the new approach probably makes things worse

A confidence interval answers three questions at once: Is there an effect? How big might it be? How precise is our estimate?

Reading a p-value

The process goes like this:

Set the null hypothesis (H₀): there's no real difference between the methods. The alternative (H₁) is that there is one.
Compute the p-value: the probability of seeing a difference at least as large as ours if H₀ were true.
Decide: if $p < \alpha$ (conventionally 0.05), reject H₀.

For our flu example the p-value comes out around 0.03. If texts and postcards were equally effective, we'd see a 7-point gap (or bigger) only about 3% of the time from random assignment alone. That's below our 5% threshold, so the difference is probably not just chance.

As a rough guide, p = 0.01 is very strong evidence, p = 0.03 is good evidence, and p > 0.05 is not significant.

What a p-value is not: it is not the probability that H₀ is true, not the probability you made a mistake, and not the probability the result will replicate. Students mix these up all the time.

Here's how I'd run the test in Python:

two_proportion_test.py

from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

vaccinated = [260, 225]   # texts (B), postcards (A)
totals     = [500, 500]

# Two-sided two-proportion z-test (H0: rates are equal)
z_stat, p_value = proportions_ztest(vaccinated, totals)

# 95% CI for the difference in rates (B - A)
low, high = confint_proportions_2indep(260, 500, 225, 500,
                                       compare="diff", method="wald")

print(f"z = {z_stat:.2f}, p = {p_value:.3f}")
print(f"95% CI for difference: {low*100:.1f} to {high*100:.1f} percentage points")

# z = 2.21, p = 0.027
# 95% CI for difference: 0.8 to 13.2 percentage points

The exact p-value is about 0.027, which is the "about 3%" from above. The exact Wald interval runs from roughly 0.8 to 13.2 points. It's a bit wider than the round "2 to 12" illustration, but the conclusion is the same: zero is outside the interval.

Putting It Together: P-Value + Confidence Interval + Effect Size

I never let students report just one of these. Each answers a different question:

Piece	Our result	What it tells you
P-value	≈ 0.03	The difference is unlikely to be random noise
Confidence interval	Excludes zero	The plausible range of the true effect
Effect size	7 percentage points	Our best estimate of the real-world impact

Combined into a single decision-ready statement:

"We have strong statistical evidence that text message reminders work better than postcards. We observed a 7 percentage point improvement, and the true improvement is plausibly a few points to over ten. Text messages provide a meaningful benefit worth implementing."

Choosing the Test: Z or T?

Both give you p-values and confidence intervals. The choice comes down to two questions: how many people are in the study, and how well do we know the variability?

Use a z-test when you have BOTH	Use a t-test when you have EITHER
Large samples (30+ per group)	Small samples (fewer than 30 per group)
A good estimate of population variability	Uncertainty about population variability
e.g., flu vaccination strategies across Texas with thousands of BRFSS respondents	e.g., diabetes screening in rural Vermont with 25 people per group

The t-distribution has heavier tails to account for the extra uncertainty of estimating the standard deviation from a small sample. Its shape depends on the degrees of freedom ($df = n - 1$). As $n$ grows it converges to the normal curve, so with large samples the two tests give practically identical answers. When in doubt, use the t-test. It's the more conservative choice.

Errors, Power, and Sample Size

Every test can be wrong in two ways:

	Type I error (false positive)	Type II error (false negative)
What happens	Reject a true H₀	Fail to reject a false H₀
In plain words	"Texts work!" when they don't	Miss a real improvement
Cost	Resources wasted on an ineffective change	A valuable intervention never gets adopted
Controlled by	Significance level α (typically 5%)	Sample size and power

Statistical power is the probability your test detects a real effect when one exists: Power = 1 − β, where β is the Type II error rate. The industry standard target is 80%. High power (80 to 90%) means you're very likely to catch real effects. Low power (30 to 40%) means you'll often miss them even when they're there.

Three things drive power:

Sample size: more participants means more power.
Effect size: bigger effects are easier to detect. A 2-point lift needs a far larger study than a 7-point lift.
Variability: more consistent responses make differences easier to see.

Always run the power analysis before collecting data. An underpowered study wastes money and can bury an intervention that actually works. In Python it's a few lines:

power_analysis.py

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# Smallest lift worth detecting: 45% -> 52%
effect = proportion_effectsize(0.52, 0.45)

n_per_group = NormalIndPower().solve_power(
    effect_size=effect, alpha=0.05, power=0.80, alternative="two-sided"
)
print(f"Participants needed per group: {n_per_group:.0f}")

# Or flip it around: what power does a planned design have?
power = NormalIndPower().solve_power(effect_size=effect, nobs1=500, alpha=0.05)
print(f"Power with 500 per group: {power:.0%}")

Effect Size and Practical Significance

Statistical significance tells you an effect is probably real. Effect size tells you whether it's big enough to matter. For the flu example:

Raw difference: 52% − 45% = 7 percentage points
Relative effect: (52 − 45) / 45 ≈ 15.6% increase

In public health, "small" effects can be enormous at population scale. Here's a back-of-the-envelope calculation:

Quantity	Value	Assumption
Eligible U.S. adults	260M
Additional vaccinations	~18M	7-point improvement (45% → 52%)
Hospitalizations prevented	~18K	1 prevention per 1,000 vaccinations
Healthcare savings	~$144M	$8,000 per hospitalization

Context matters, though. A 7-point improvement might be life-changing for cancer screening and trivial for hand-sanitizer use. Weigh health impact, population size, cost, alternatives, and how hard the change is to implement.

Common Mistakes and How to Avoid Them

Confusing statistical significance with practical importance. "p = 0.02, so we must implement it!" Not necessarily. Always look at the effect size too.
Treating non-significant results as "no effect." "p = 0.08, so it doesn't work." Maybe your study just didn't have the power to detect it. Absence of evidence isn't evidence of absence.
P-hacking and cherry-picking. Running analyses until something hits p < 0.05 and reporting only that one. Pre-register your analysis plan, and adjust thresholds when you explore subgroups.
Ignoring confidence intervals. The p-value says nothing about precision. A wide interval is a warning sign.
Forgetting the cost of each error type. For expensive interventions, demand stronger evidence. For low-risk, high-upside ones, you can accept weaker evidence.

A Complete Analysis: Cervical Cancer Screening Reminders

Here's everything together in one case study.

1. Study design. Research question: does a personalized reminder email increase cervical cancer screening rates compared to standard mailed reminders? Before collecting anything, run a power analysis for the smallest lift worth acting on.

2. Implementation.

Control: 550 women receive standard mailed postcards.
Treatment: 550 women receive personalized emails with scheduling links.
Follow-up period: 120 days.

3. Results.

Group	Screened	Total	Rate
Control (mail)	330	550	60%
Treatment (email)	374	550	68%
Difference			8 points

Running the same two-proportion test from earlier on these counts gives p ≈ 0.006 and a 95% interval for the difference of roughly 2.3 to 13.7 points. That's comfortably significant, and zero is well outside the interval.

4. Impact assessment. Email is also much cheaper (about $0.05 per reminder versus $0.75 for mail). Scaled nationally, the lecture's estimate was on the order of 1.2 million additional screenings a year, roughly 1,200 lives saved, and about $10.5 million in annual savings.

Recommendation: make personalized email reminders the new standard for cervical cancer screening outreach, with backup mailed reminders for older women and anyone without reliable email access. The second half of that sentence is where the health-equity lens comes in. A good recommendation names who the intervention might miss.

Key Takeaways

Evidence matters. A/B testing moves public health from opinion-based to evidence-based decisions.
Random assignment is non-negotiable. It's what makes the comparison fair.
The CLT is the engine. Averages of groups are predictable even when individuals aren't, which is why a 400,000-person survey can describe 260 million adults.
Report three things, always: the p-value, the confidence interval, and the effect size.
Plan power before you collect data. Underpowered tests miss real effects.
Context counts. Practical significance depends on health impact, population size, cost, and who might be left out.

With these tools you can design a sound study, separate real effects from random variation, and turn the result into a recommendation that improves health outcomes.