Hypothesis Testing — Deciding With Data

Making decisions with data and p-values

Mathematics Statistics intermediate 8 min

This concept builds on Normal Distribution — The Bell Curve Explained, Z-Score — Distance in Standard Deviations. Consider exploring them first for the best experience.

Step 1 of 9

Start With a Claim to Test

The null hypothesis H₀ is the default assumption (e.g., "the drug has no effect"). We collect data to see if the evidence is strong enough to reject it.

The Null Distribution

If H₀ were true, what would sample results look like? This is the null distribution — typically a normal curve centered on the null value. Results far in the tails are unlikely under H₀.

Compute the Test Statistic

Z = (x̄ − μ₀) / (σ/√n). This converts your sample mean into a Z-score: how many standard errors it sits from the null hypothesis value.

The p-value

The p-value is the probability of getting a result at least as extreme as yours, assuming H₀ is true. p = P(|Z| ≥ |z_obs|). Small p = unlikely under H₀.

Significance Level α

Before collecting data, choose α (typically 0.05). If p < α, reject H₀. The critical region is the shaded tail area. α is the false positive rate you accept.

Decision: Reject or Fail to Reject

p < α → reject H₀ (evidence is significant). p ≥ α → fail to reject H₀ (not enough evidence). Note: failing to reject ≠ proving H₀ is true.

What-If: There Is a Real Effect

Check "What-If: show a real effect (H₁)" to draw a second, dashed curve: where Z lands when the effect is real. Its centre sits d·√n standard errors from zero. The shaded part of it inside the "fail to reject" zone is β, the chance of missing a real effect (a Type II error). Power = 1 − β is the chance of catching it.

How Much Data Is Enough?

Move the "Effect size d" and "Sample size n" sliders. With d = 0.5 and n = 25 the H₁ curve sits 0.5 × √25 = 2.5 standard errors out; at two-tailed α = 0.05 power is about 0.71, so β ≈ 0.29 — almost one real effect in three is missed. The "n for 80% power" card reads 32. Lowering α pushes the critical line out and raises β.

Simulate Many Studies

Press "Run 20 studies" to repeat the experiment 20 times. Each dot along the axis is one study's Z, and red dots land in the rejection region. With H₁ hidden, H₀ is true, so red dots are false positives and their share settles near α. With H₁ shown, red dots are real detections and their share settles near the power.

How It Works — Step by Step

Step 1 of 9
  1. 1

    Start With a Claim to Test

    The null hypothesis H₀ is the default assumption (e.g., "the drug has no effect"). We collect data to see if the evidence is strong enough to reject it.

  2. 2

    The Null Distribution

    If H₀ were true, what would sample results look like? This is the null distribution — typically a normal curve centered on the null value. Results far in the tails are unlikely under H₀.

  3. 3

    Compute the Test Statistic

    Z = (x̄ − μ₀) / (σ/√n). This converts your sample mean into a Z-score: how many standard errors it sits from the null hypothesis value.

  4. 4

    The p-value

    The p-value is the probability of getting a result at least as extreme as yours, assuming H₀ is true. p = P(|Z| ≥ |z_obs|). Small p = unlikely under H₀.

  5. 5

    Significance Level α

    Before collecting data, choose α (typically 0.05). If p < α, reject H₀. The critical region is the shaded tail area. α is the false positive rate you accept.

  6. 6

    Decision: Reject or Fail to Reject

    p < α → reject H₀ (evidence is significant). p ≥ α → fail to reject H₀ (not enough evidence). Note: failing to reject ≠ proving H₀ is true.

  7. 7

    What-If: There Is a Real Effect

    Check "What-If: show a real effect (H₁)" to draw a second, dashed curve: where Z lands when the effect is real. Its centre sits d·√n standard errors from zero. The shaded part of it inside the "fail to reject" zone is β, the chance of missing a real effect (a Type II error). Power = 1 − β is the chance of catching it.

  8. 8

    How Much Data Is Enough?

    Move the "Effect size d" and "Sample size n" sliders. With d = 0.5 and n = 25 the H₁ curve sits 0.5 × √25 = 2.5 standard errors out; at two-tailed α = 0.05 power is about 0.71, so β ≈ 0.29 — almost one real effect in three is missed. The "n for 80% power" card reads 32. Lowering α pushes the critical line out and raises β.

  9. 9

    Simulate Many Studies

    Press "Run 20 studies" to repeat the experiment 20 times. Each dot along the axis is one study's Z, and red dots land in the rejection region. With H₁ hidden, H₀ is true, so red dots are false positives and their share settles near α. With H₁ shown, red dots are real detections and their share settles near the power.

Key Takeaways

p-value Is Not "Probability H₀ Is True"

p = P(data | H₀) — the probability of the data given the null. It is NOT P(H₀ | data). This is the most common misinterpretation in all of statistics.

α, β and Power Trade Off

Setting α = 0.05 means accepting a 5% chance of a false positive (Type I error). Lowering α raises β, the chance of missing a real effect (Type II error). Only a larger effect or a larger sample n raises power = 1 − β without loosening α. Choose α and n before collecting data.

Statistical ≠ Practical Significance

With large n, even trivially small effects become statistically significant. Always report effect size alongside p-values. A drug with p=0.001 but effect size=0.01 may be useless clinically.

Two-Tailed vs One-Tailed

Two-tailed tests check both directions (H₁: μ ≠ μ₀). One-tailed tests check one direction (H₁: μ > μ₀). Two-tailed is more conservative and usually preferred.

Added
Updated
  • — Added a link to Correlation and testing whether a sample r is significant.
  • — Added a what-if mode for statistical power and Type II error, and a repeated-study simulator.