The null hypothesis and p-values, explained

Updated November 2026

A hypothesis test asks one narrow question: if there were really no effect, how surprising would data like ours be? The "no effect" claim is the null hypothesis, and the measure of surprise is the p-value. A p-value is useful, but it is not the probability that the null hypothesis is true, and it says nothing about how big or important an effect is.

The short version

H0 (null): no effect or no difference. H1 or Ha (alternative): there is an effect.
p-value: the probability, calculated assuming H0 and the test's other assumptions are true, of getting a result at least as extreme as the one observed.
If p ≤ α (the significance level, often 0.05), we reject H0; otherwise we fail to reject it, which is not the same as showing it is true.
Type I error: rejecting a true H0 (probability α). Type II error: missing a real effect (probability β). Power = 1 − β.
Always report an effect size with a confidence interval, and check the test's assumptions.

How a test works

Suppose you want to know whether a coin is biased. You flip it 100 times and get 61 heads.

State the hypotheses. H0: the coin is fair, P(heads) = 0.5. H1: it is not, P(heads) ≠ 0.5.
Choose α before looking at the data. Here, α = 0.05.
Pick a test statistic, a number that summarizes how far the data are from what H0 predicts. Here, simply the number of heads; H0 predicts 50.
Find the p-value. Under H0, the number of heads follows a binomial distribution. The probability of 61 or more heads is about 0.018; since we would be just as surprised by 39 or fewer, the two-sided p-value is about 2 × 0.018 ≈ 0.035.
Decide. 0.035 is below 0.05, so we reject H0: the data are not very compatible with a fair coin.

The same logic underlies t-tests, chi-square tests and the rest; only the test statistic and its distribution under H0 change.

What a p-value does not mean

In 2016 the American Statistical Association (ASA) issued a formal statement with six principles on p-values, because misuse had become widespread. In plain words:

A p-value can show how poorly the data fit one particular statistical model.
It is not the probability that the hypothesis is true, nor the probability that the result happened by chance alone. In the coin example, p = 0.035 does not mean there is a 3.5% chance the coin is fair.
Conclusions and decisions should not rest only on whether p crosses a threshold such as 0.05.
Proper inference requires full reporting: all the analyses done, not just the ones that came out "significant".
A p-value does not measure the size of an effect or how important it is. A tiny, useless effect can give a very small p-value in a huge study.
By itself, a p-value is not a good measure of evidence for or against a hypothesis.

Errors, power and effect sizes

The four possible outcomes of a test
	H0 is actually true	H0 is actually false
Reject H0	Type I error (false positive), probability α	Correct (true positive), probability = power
Fail to reject H0	Correct	Type II error (false negative), probability β

Power (1 − β) is the chance a study will detect an effect of a given size if it is really there. Power goes up with larger samples, larger true effects, less noisy measurements and a larger α. Studies are often planned to have about 80% power for the smallest effect worth finding. A non-significant result from a small, low-power study is weak evidence of "no effect"; it may simply have been too small to see one.

An effect size says how big the difference is, in real units (a drug lowers blood pressure by 5 mmHg) or standardized units (the difference divided by the standard deviation). A confidence interval gives a range of effect sizes reasonably compatible with the data. "The drug lowered blood pressure by 5 mmHg (95% CI 1 to 9)" tells a reader much more than "p = 0.02": the effect is probably real, could be small or moderate, and can be weighed against side effects and cost. If the 95% interval excludes zero, a two-sided test at α = 0.05 would reject H0, so the interval contains the test result and more.

Checking the test's assumptions

A p-value is computed from a model of how the data were generated. If the model is badly wrong, the p-value can be badly wrong too.

What are the assumptions?

The common ones, especially for t-tests, ANOVA and regression:

Independence. Each observation is independent of the others: no repeated measurements on the same person treated as separate people, no students from one classroom treated as if they were independent samples from many. This is the most important assumption and is settled by study design, not by a later check. Paired or clustered data need tests built for them.
Random sampling or random assignment. Needed so that results can be generalized to a population or attributed to the treatment.
Normality. For t-tests the data (or, for two groups, each group) should come from a roughly normal distribution. With larger samples the test becomes fairly robust to non-normality, thanks to the central limit theorem, though strong skew and outliers still matter.
Equal variances. The classic pooled two-sample t-test and one-way ANOVA assume the groups have similar spread. Welch's t-test does not, and is often recommended as the default for two groups.
For tests of proportions using the normal approximation, both np and n(1 − p) should be greater than about 5.

How to check: look at the data before testing. Histograms, box plots and normal Q–Q plots reveal skew, outliers and unequal spread better than a formal test of normality, which in small samples misses real problems and in large samples flags trivial ones. If assumptions fail, options include Welch's test, a transformation (such as a log for right-skewed data), a rank-based test such as Mann–Whitney, or resampling methods such as the bootstrap.

Memory aid

"Innocent until proven guilty." H0 is the presumption of innocence. The p-value asks: if the defendant were innocent, how unlikely is evidence this damning? A small p-value is grounds to convict (reject H0). Acquittal (fail to reject) does not prove innocence. A Type I error convicts an innocent person; a Type II error lets a guilty one go free.

And for what a p-value is: "P is for the data, given the null" — not the null, given the data.

A verse to remember it

Tune: Oh My Darling, ClementineSay there's no effect at all: That's the null, our starting line. If it's true, how often would we See results as far out as mine? P's not the chance, p's not the chance, P's not the chance the null is true; It's how strange our data would be If the null were really true. Under alpha? Then reject it, But report the effect size too.

Check yourself

A study reports p = 0.03. Does that mean there is a 3% chance the null hypothesis is true?

No. It means that if H0 (and the model's assumptions) were true, results at least this extreme would occur about 3% of the time. The probability that H0 is true cannot be read from a p-value alone.

What is a Type II error, and how can you make it less likely?

Failing to reject H0 when there really is an effect. Increase power: a larger sample, more precise measurements, or studying a larger effect.

A trial of 100,000 people finds p = 0.0001 for a drug that lowers cholesterol by 0.5%. Is the result important?

It is statistically clear but may be practically trivial. Look at the effect size and its confidence interval, and weigh it against costs and side effects.

You measure the same 20 people before and after a treatment. Why is an ordinary two-sample t-test wrong?

The before and after measurements are not independent; they come from the same people. Use a paired test, which analyzes each person's change.