Statistics Foundations: Mean, Spread, Sampling, and Your First Hypothesis Tests

Before we can talk about A/B tests, machine learning, or neural networks, we need a shared vocabulary for describing data and reasoning under uncertainty. This tutorial is adapted from the statistics weeks I taught in the VADSTI data science program, including the lecture on the statistics behind A/B testing. Intuition comes first and formulas second, and the ideas come with small Python examples you can run.

Here's the plan:

1. Describing data: mean, median, mode, and skewness
2. Measuring spread: variance and standard deviation
3. Sampling: why we sample, and how uncertain our estimates are
4. The Central Limit Theorem: why averages behave predictably
5. Hypothesis testing: null and alternative hypotheses, z-scores, p-values, and confidence intervals
6. Errors, power, and effect size: how tests go wrong, and whether a result matters
7. Hypothesis tests in Python: a tour of common tests

The tools are NumPy, pandas, SciPy, and matplotlib/seaborn.

Part 1: Describing Data with Central Tendency

Descriptive statistics summarize data in a meaningful way. The first question is usually: what's a typical value? That's what a measure of central tendency answers. It's a single number meant to represent the whole distribution.

Throughout this section I'll use one small dataset:

data = [1, 3, 5, 2, 3, 7, 8, 4, 10, 0]

The mean

A fun bit of trivia: the word mean comes to English from French around the 1300s, where it referred to the middle point between two musical notes. In statistics it's still the "middle" in a balancing sense: the sum of all values divided by how many there are.

$$\mu = \frac{\sum X}{N} \qquad \bar{X} = \frac{\sum X}{n}$$

The formula is the same for a population (Greek \(\mu\), uppercase \(N\)) and a sample (\(\bar{X}\), lowercase \(n\)). The notation just tells the reader which one you have.

For our data: (1 + 3 + 5 + 2 + 3 + 7 + 8 + 4 + 10 + 0) / 10 = 43 / 10 = 4.3.

A convention worth knowing: report the mean to one more decimal place than the raw data. It isn't a strict rule, but textbooks and journals widely follow it.

Use the mean when the data is interval or ratio scaled and not heavily skewed. Its strength is also its weakness: it's sensitive to every value, so changing one score changes the mean, and a few extreme values can drag it around.

The median

The median is the middle value of the sorted data. If there's an even number of values, it's the average of the middle two. Sorted, our data is 0, 1, 2, 3, 3, 4, 5, 7, 8, 10, so the median is (3 + 4) / 2 = 3.5.

Use the median when the distribution is skewed. A few very large (or very small) values won't pull it the way they pull the mean. That's why you hear about median household income, not mean.

The mode

The mode is the most frequent value. Here it's 3, which appears twice. A distribution with two peaks is called bimodal. The mode is the weakest of the three as a summary, because two very different datasets can share the same mode, but it's the only one that works for purely categorical data.

Two tricks with the mean

The weighted mean. Sometimes values don't count equally. Multiply each value by its weight (or frequency) and divide by the total weight:

$$\bar{X}_w = \frac{\sum (X \times w)}{\sum w}$$

Challenge: Georgia's statistics marks are weighted by credit points:

Her weighted average is (75·3 + 82·4 + 90·2 + 68·3) / (3 + 4 + 2 + 3) = 937 / 12 ≈ 78.1. Course B, with the most credits, pulls hardest.

The mean of a binary variable is a proportion. Take a litter of six puppies: Female, Male, Female, Female, Male, Female. Code Female = 1 and Male = 0, and the mean is (1+0+1+1+0+1)/6 ≈ 0.667, so 66.7% of the puppies are female. This trick shows up everywhere, from conversion rates in A/B tests to accuracy in classification. Every one of them is the mean of a 0/1 column.

Skewness

Skewness measures how lopsided a distribution is. A perfectly symmetric distribution has skewness 0. A long tail to the right gives positive skew, and a long tail to the left gives negative skew. Some rules of thumb:

For our data, stats.skew(data) returns about 0.44, which is approximately symmetric with a slight right tail (that 10 is doing some work). You can see this in the gap between the mean (4.3) and the median (3.5): in a right-skewed distribution the mean sits to the right of the median. Skew isn't bad in itself, but it matters for methods that assume normality, and transformations (log, square root) can help.

Part 2: Measuring Spread

Two datasets can have the same mean and look nothing alike. Variability tells you how far the data points stray from the average and from each other.

Variance

Variance is the average squared distance from the mean:

$$\sigma^2 = \frac{1}{N}\sum_{i=1}^{N}(x_i - \mu)^2 \qquad s^2 = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2$$

Step by step for our data (mean 4.3):

• Treating the data as a population: 92.10 / 10 = 9.21
• Treating it as a sample: 92.10 / 9 ≈ 10.23

Why do we square the deviations? So that negative and positive distances don't cancel out, and so that big deviations count for more than small ones.

Why \(n - 1\)? Bessel's correction

A sample's values cluster around the sample's own mean, which is by construction the point that minimizes the squared distances. So the sample's spread systematically understates the population's. Dividing by \(n - 1\) instead of \(n\) corrects that bias. In NumPy, this is the ddof ("delta degrees of freedom") argument, and its default is 0 (the population formula). That trips up a lot of people.

Standard deviation

The standard deviation is the square root of the variance, which puts spread back in the original units:

$$\sigma = \sqrt{9.21} \approx 3.03$$

A small standard deviation means values cluster near the mean. A large one means they're spread out. Here's a good everyday example: three banks can all have the same average wait time, but the one with the smallest standard deviation is the one where you know what you're getting.

Part 3: Sampling and the Standard Error

If statistics were Captain Planet, sampling would be one of the rings you can't do without. Whenever we measure something, say blood pressure in people with hepatitis C, there's a population behind the measurement (everyone with hepatitis C) and a sample we actually observed.

We sample because it's practical (measuring everyone is usually impossible), efficient (faster and cheaper), and because it still lets us make statistical inferences about the whole. Every sample raises three questions:

1. What's our best estimate of the population mean? (The sample mean.)
2. How confident are we in that estimate?
3. What's our best estimate of the population standard deviation? (The sample SD, with \(n - 1\).)

Question 2 is answered by the standard error of the mean, which measures how much the sample mean would bounce around if we re-sampled:

$$SE_{\bar{X}} = \frac{s}{\sqrt{n}}$$

One sample, three scenarios

Let's measure the heights (in metres) of five basketball players: 2.00, 2.10, 1.98, 2.12, 1.95. The sample mean is 10.15 / 5 = 2.03 m, and the sample standard deviation is about 0.076 m. The question is how uncertain that 2.03 is, and the answer depends on what population we're sampling from.

Scenario 1: an infinite population. We're estimating the average height of basketball players in general, a population so large it's effectively infinite. This is the most common case:

$$SE = \frac{0.0755}{\sqrt{5}} \approx 0.034$$

Scenario 2: a finite population. Now we're estimating the average height of the 14 players on the 1992–93 Chicago Bulls roster, and our 5 are a sample from those 14. We've seen a big chunk of the population, so we should be less uncertain. When sampling without replacement from a finite population, we apply the finite population correction:

$$SE = \frac{s}{\sqrt{n}}\sqrt{\frac{N-n}{N-1}} = 0.034 \times \sqrt{\frac{9}{13}} \approx 0.028$$

Scenario 3: the whole population. If the question is "what's the average height of the five starters?" and we measured all five, then 2.03 m isn't an estimate at all. It's the answer. The standard error is zero, and we'd describe spread with the population formula (dividing by \(N\)), which gives about 0.068 m.

When I first saw this, a lot of small ideas connected at once: the more of the population you've seen, the less uncertainty is left. The standard error is how we put a number on that.

Part 4: The Central Limit Theorem

Every A/B test compares averages from samples. To trust that comparison, we need to know how sample averages behave. That's what the Central Limit Theorem (CLT) tells us, and it's the reason A/B testing works at all.

Imagine a large population: all visitors to a website, all customers of a store, all users of an app. Instead of studying everyone, we take a random sample and calculate its mean. Then we take another sample and calculate its mean. And another, and another.

If we plot all those sample means, something remarkable happens: they form a bell curve (a normal distribution), even if the original data isn't normal at all. Time spent on a page is usually right-skewed: most people leave quickly and a few stay forever. But the averages of many samples of visitors still pile up in a neat, symmetric bell shape around the true mean.

Why does that matter so much?

• It's predictable. In a normal distribution, most values cluster around the middle and extreme values are rare, in known proportions. That lets us put probabilities on what we see.
• The standard tests rely on it. Z-tests and t-tests assume normality. Thanks to the CLT, we can use them on sample means even when the raw data is skewed.
• P-values and confidence intervals work. Both are calculated from the normal curve.
• We can compare groups. The difference between two sample means is also approximately normal, so we can ask whether the difference between control and treatment is real or just chance.

The catch is the sample size. The CLT kicks in once samples are large enough. About 30 or more per group is the usual rule of thumb, and more if the data is very skewed. That's why A/B tests need enough users before you read the results.

Part 5: Hypothesis Testing

Hypothesis testing answers one question: did our change make a real difference, or is the difference we see just random variation?

Two competing claims

Say we change the color of the "Buy Now" (call-to-action) button and want to know whether it increases conversions.

• Null hypothesis (H₀): "Changing the button color does not increase conversions." Any difference we see is due to chance. This is the default: we assume nothing changed until the data gives strong evidence otherwise.
• Alternative hypothesis (H₁): "Changing the button color does increase conversions." The change had a real effect.

At the end of the test, we do one of two things:

• Reject H₀, if the data shows a statistically significant difference. We conclude the change likely had an effect.
• Fail to reject H₀, if it doesn't. We don't have enough evidence to say the change mattered.

Notice the wording. Failing to reject H₀ doesn't prove there's no effect. It means this test couldn't detect one. We never "accept" the null from a single test.

Comparing two means

Now a concrete example. We test a new landing page and measure time on page:

• Control group mean: 30 seconds
• Treatment group mean: 35 seconds
• Difference: 5 seconds

The new page looks more engaging. But could a 5-second gap have happened by chance?

Imagine H₀ is true and the new page does nothing. If we ran this test over and over, we'd still see some difference each time just from which users happened to land in each group: +5 one time, −3 another, +1 another. Those differences would form a bell curve centered at zero. That's the null distribution.

The question becomes: where does our observed 5 fall on that curve?

• If 5 is close to zero, near the middle of the curve, it's the kind of difference chance produces all the time. We fail to reject H₀.
• If 5 is far out in the tail, chance rarely produces a gap that big. We reject H₀.

What counts as "far" depends on how noisy the data is. A 5-second difference is huge if time on page barely varies from person to person, and meaningless if it swings by minutes. The bigger the difference relative to the natural variation, the stronger the evidence. The measure of that variation for a difference in means is the standard error from Part 3.

The z-score: how many standard errors away?

To put the observed difference on the null curve, divide it by the standard error:

$$z = \frac{\text{observed difference} - \text{expected difference under } H_0}{\text{standard error}}$$

Under H₀ the expected difference is 0. Suppose the standard error of the difference is 1.5 seconds:

$$z = \frac{5 - 0}{1.5} \approx 3.33$$

Our result is 3.33 standard errors from zero. At a 95% confidence level, the cutoff (the critical value) is 1.96, because 95% of a normal curve lies within ±1.96 standard errors of the center.

• If \(|z| > 1.96\), the result is statistically significant. Reject H₀.
• If \(|z| \le 1.96\), it isn't. Fail to reject H₀.

3.33 is well past 1.96, so we reject H₀. With small samples we use the t-distribution instead, which gives a t-score with a slightly larger cutoff. For large samples the two are practically identical.

P-values

The p-value answers: if there were really no effect, how often would we see a difference at least this big just by chance?

• Small p-value: a result like ours would be rare if H₀ were true. That's strong evidence of a real effect.
• Large p-value: a result like ours is common even when nothing changed. Not enough evidence.

The usual threshold is 0.05. If \(p < 0.05\), we reject H₀.

For our landing page, \(z = 3.33\) gives \(p \approx 0.001\). If the new page truly did nothing, we'd see a gap of 5 seconds or more only about 0.1% of the time. So we reject H₀.

Now suppose the data had been much noisier, with a standard error of 3 seconds. Then \(z = 5/3 \approx 1.67\) and \(p \approx 0.10\). A gap that big would show up about 10% of the time by chance alone. Since 0.10 > 0.05, we fail to reject H₀. Same 5-second difference, different conclusion, because of the noise.

One common misreading: \(p = 0.001\) does not mean "there's a 0.1% chance the result is due to chance" or "a 99.9% chance the page works." It's a statement about how surprising the data would be if H₀ were true.

Confidence intervals

A confidence interval gives a range of plausible values for the true difference:

$$\text{difference} \pm 1.96 \times SE$$

With a standard error of 1.5:

$$5 \pm 1.96 \times 1.5 = (2.1,\ 7.9)$$

We're 95% confident the true improvement is somewhere between 2.1 and 7.9 seconds. Zero isn't in that range, so "no difference" isn't a plausible value. The result is significant, and we reject H₀. (Strictly, "95% confident" means that if we repeated the experiment many times, about 95% of the intervals we built would contain the true difference.)

With the noisier data (SE = 3):

$$5 \pm 1.96 \times 3 = (-0.9,\ 10.9)$$

The true effect could be a small decrease or a big increase. Zero is inside the interval, so we can't rule out "no effect," and we fail to reject H₀.

The p-value and the confidence interval always agree on significance: \(N\)0 exactly when the 95% interval excludes zero. But the interval tells you more, because it also shows how big the effect plausibly is.

Part 6: Errors, Power, and Effect Size

Two ways to be wrong

Every test ends in a decision, and either decision can be wrong.

Type I error: a false positive. We think there's an effect when there isn't. Say a test shows a new checkout process increases conversions, so we roll it out. But the increase was just chance, and over the following months conversions don't improve at all. We acted on false evidence.

The risk of a Type I error is the significance level, α. With α = 0.05, we accept a 5% chance of a false positive when there's truly no effect. Lowering α to 0.01 makes false positives rarer, but it also makes real effects harder to detect.

Type II error: a false negative. We miss an effect that's really there. Say a new homepage design genuinely increases engagement, but the test shows no significant difference, so we throw the design away. Usually that happens because the sample was too small or the test wasn't sensitive enough.

Statistical power

Power is the probability that a test detects a real effect when one exists:

$$\text{Power} = 1 - \beta$$

where β is the probability of a Type II error. The industry standard is to aim for 80% power. A low-power test will often miss real effects. To increase power:

• Increase the sample size. This is the main lever.
• Test for bigger effects. Large differences are easier to spot than tiny ones.
• Raise α. This works, but it increases the risk of false positives, so it's rarely the right choice.

Sample size and the standard error

Many experiments fail not because the change did nothing, but because the sample was too small to see it.

Recall that the standard error shrinks as the sample grows (\(N\)1). With more users:

• the standard error falls, so estimates are more precise,
• power rises, so real differences are easier to detect, and
• Type II errors become less likely.

Small samples have large standard errors, so even a real effect gets lost in the noise. That's why small tests so often end up "inconclusive."

Minimum detectable effect

The minimum detectable effect (MDE) is the smallest effect a test can reliably detect, given its sample size and power. Anything smaller may be missed even if it's real.

• With a small sample, you might only be able to detect big changes, like a 10% lift in conversions.
• With a large sample, you can detect a 1% lift.

For a large business, a 1% improvement can be worth a lot of money, so choose the MDE that matters to you first, then calculate how many users you need to detect it. (The BRFSS A/B testing tutorial shows that calculation in code.)

Statistically significant isn't the same as important

A p-value tells you whether an effect is likely real. It doesn't tell you whether it's big enough to matter. That's what effect size is for.

A common measure is Cohen's d, the difference between the groups divided by the standard deviation:

$$d = \frac{\bar{x}_{\text{treatment}} - \bar{x}_{\text{control}}}{s}$$

In the landing page test, if individual visitors' time on page has a standard deviation of 10 seconds, then \(N\)2: a medium, meaningful effect.

Now imagine a huge test where a new sign-up page raises the sign-up rate by just 0.5%. With millions of users, that can easily reach \(N\)3. It's real, but is it worth the engineering and design cost? Maybe not.

Always report the effect size along with the p-value. Significance tells you an effect exists; effect size tells you whether to care.

Part 7: Hypothesis Tests in Python

Every test you'll meet has the same structure as the z-score above: (what we observed − what H₀ predicts) / how much it would vary by chance. The difference is which kind of data each one handles.

Always check the test's assumptions (distribution shape, scale of measurement, independence) before you trust its p-value.

Example 1: One-sample t-test on BMI

We have BMI scores from ten people in a region and want to know if their average differs from a known reference of 25.0.

Contrast that with a sample of ages [32, 34, 29, 29, 22, 24, 35, 22, 24, 35] tested against a claimed mean of 30. The sample mean is 28.6, but ttest_1samp gives p ≈ 0.42, so we fail to reject H₀. A 1.4-year gap with this much spread and only ten people is entirely consistent with chance.

Example 2: Chi-square test for categorical data

A survey of 300 people records gender and smoking status. Is gender associated with smoking?

Example 3: When data isn't normal (Mann-Whitney U)

Comparing test scores from two classes without assuming normality:

Example 4: Three or more groups (ANOVA)

For three classrooms' exam scores, one-way ANOVA tests H₀: all group means are equal against H₁: at least one differs. Running stats.f_oneway on classrooms scoring [88, 89, 76, 85, 90, 88], [84, 87, 89, 90, 85, 86], and [91, 88, 85, 90, 92, 86] gives p ≈ 0.45, so there's no significant difference between classrooms.

Example 5: A z-test by hand

When the population SD is known and the sample is large, you can compute the z-statistic directly. It's a nice way to see what's under the hood:

Notice the denominator: that's the standard error from Part 3, and the whole expression is the z-score from Part 5.

Example 6: A confidence interval

The same \(\bar{x} \pm 1.96 \times SE\) recipe from Part 5, applied to a simulated sample:

Interpretation: if we repeated this sampling procedure many times, about 95% of the intervals we built would contain the true population mean.

Two practical notes:

• In real life you rarely know the population SD, so you'd use the sample SD (np.std(sample, ddof=1)) to compute the standard error.
• With small samples (n < 30), swap 1.96 for the critical value from the t-distribution (stats.t.ppf(0.975, df=n-1)). For large samples, z and t are practically identical.

Key Takeaways

• Mean, median, and mode answer "what's typical?" in different ways. Reach for the median when data is skewed.
• The mean of a 0/1 variable is a proportion. That one fact powers conversion rates, accuracy, and much of A/B testing.
• Variance and standard deviation measure spread. Use \(n - 1\) for samples, and remember NumPy defaults to \(n\).
• The standard error measures how uncertain a sample mean is. It shrinks as samples grow.
• The Central Limit Theorem makes sample means (and differences between them) approximately normal once samples are large enough, which is what lets z-tests, t-tests, p-values, and confidence intervals work.
• Hypothesis testing asks whether an observed difference is large relative to chance variation. Start from H₀ (no effect), and reject it only with strong evidence.
• |z| > 1.96, p < 0.05, and a 95% interval that excludes zero are three views of the same decision. The interval also shows how big the effect plausibly is.
• Failing to reject H₀ isn't proof of H₀.
• Type I errors are false positives (controlled by α). Type II errors are false negatives. Power (aim for 80%) mostly comes from sample size.
• Statistically significant isn't the same as important. Always check the effect size.

These are the building blocks for everything that follows: A/B testing, regression, and the evaluation metrics we use for machine learning models.

See it applied: A/B Testing & Statistical Thinking with BRFSS Data