Hypothesis testing and p-values

Statistics · First-year university statistics · about 10 min

Hypothesis testing checks whether sample evidence is difficult to reconcile with a stated claim about a population. A small difference between a sample mean and the claimed mean may be ordinary sampling variation. A larger difference may be evidence against the claim. The test statistic measures the difference on a standardised scale, while the p-value measures how unusual the data would be if the null hypothesis were true.

The key idea

For a sample mean with a known population standard deviation, use

z=\frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}}

Here $\bar{x}$ is the sample mean, $\mu_0$ the mean under the null hypothesis, $\sigma$ the population standard deviation, and $n$ the sample size.

A horizontal z-axis from -4 to +4 with zero centred. Draw a symmetric smooth standard-normal bell curve above it. Place two vertical dashed guides at -z_obs and +z_obs, both equally far from zero. Shade only the regions under the curve to the left of -z_obs and to the right of +z_obs, each labelled p/2. The central unshaded region is visibly larger than either tail. Show the origin and both threshold marks, but no numerical p-value. — For a two-sided test, the p-value counts both tails beyond equally extreme test statistics.

Try it yourself

Shade a tail of the normal curve to see how a test statistic turns into a p-value.

State the hypotheses

Write the null hypothesis $H_0$ as a statement about a population parameter, such as $H_0:\mu=\mu_0$ . The alternative $H_1$ expresses what would count as evidence against it. For a difference in either direction, use $H_1:\mu\ne\mu_0$ . For an increase only, use $H_1:\mu>\mu_0$ . Decide the direction before looking at the data. Also choose the significance level, commonly $\alpha=0.05$ . A one-sample z-test assumes independent observations and a known population standard deviation; a normally distributed population or a sufficiently large representative sample supports the normal approximation.

Calculate the p-value

Under $H_0$ , the z-statistic follows the standard normal distribution under the stated assumptions. A two-sided test counts outcomes at least as extreme in both directions: $p=2[1-\Phi(|z|)]$ , where $\Phi$ is the standard normal cumulative distribution function. For an upper-tail test, use $p=1-\Phi(z)$ ; for a lower-tail test, use $p=\Phi(z)$ . Check the alternative hypothesis before choosing a tail. A p-value is a probability of data as extreme as those observed, assuming $H_0$ is true. It is not the probability that $H_0$ is true.

One standard-normal bell curve centred over the zero mark on a horizontal z-axis. Draw a single vertical dashed line at a positive observed test statistic z_obs approximately two standard deviations right of zero. Shade only the area to the right of that line. Leave the area to its left unshaded. Draw one curved callout arrow to the shaded tail from the label p. No values or decisions are written. — An upper-tail test includes only the outcomes beyond its observed z-value.

Make and report a decision

Compare $p$ with the previously selected $\alpha$ . If $p\le\alpha$ , reject $H_0$ and report evidence for the alternative. If $p>\alpha$ , fail to reject $H_0$ : this does not establish that $H_0$ is correct. A complete conclusion gives the test direction, statistic, p-value, significance level and real-world interpretation. Statistical significance is not the same as practical importance. A tiny shift can be detectable with a large sample, so discuss the estimated size of the effect as well as the result of the test.

Worked example: Checking the average fill mass

A filling process is claimed to have a population mean of $500\,\mathrm{g}$ . A random independent sample of $64$ packs has mean mass $502.8\,\mathrm{g}$ . The population standard deviation is known to be $8\,\mathrm{g}$ . Test at the $5\%$ significance level whether the population mean differs from the claim. Assume the normal approximation is appropriate.

Show the solution

Hypotheses: $H_0:\mu=500\,\mathrm{g}$ and $H_1:\mu\ne500\,\mathrm{g}$ . This is two-sided.
Standard error: $SE=\frac{8\,\mathrm{g}}{\sqrt{64}}=1\,\mathrm{g}$ .
Test statistic:

z=\frac{502.8-500}{8/\sqrt{64}}=2.80.

Two-sided p-value: $p=2[1-\Phi(2.80)]\approx0.0051$ .
Decision: Since $0.0051<0.05$ , reject $H_0$ .

Answer: $z=2.80$ , $p\approx0.0051$ . There is statistically significant evidence that the population mean fill mass differs from $500\,\mathrm{g}$ .

Common mistakes

Using a one-sided p-value for a two-sided alternative.

Read the sign in $H_1$ first and include both tails when $H_1$ uses $e$ .

Calling the p-value the probability that the null hypothesis is true.

Describe it as a tail probability calculated under the assumption that $H_0$ holds.

Claiming that failing to reject proves no effect.

Say the evidence is insufficient at the chosen significance level.

Substituting the sample standard deviation into a z-test without checking assumptions.

If the population standard deviation is unknown, investigate an appropriate t-test instead.

Check your understanding

1. For a two-sided z-test of $H_0:\mu=100\,\mathrm{g}$ , a sample has $\bar{x}=102\,\mathrm{g}$ , $\sigma=6\,\mathrm{g}$ and $n=36$ . What is the approximate p-value?

2. A valid test gives $p=0.12$ with $\alpha=0.05$ . What is the correct decision?

3. An upper-tail test uses $H_0:\mu=40\,\mathrm{g}$ and $H_1:\mu>40\,\mathrm{g}$ . With $\bar{x}=44\,\mathrm{g}$ , known $\sigma=10\,\mathrm{g}$ and $n=25$ , what is the approximate p-value?

Still not sure? Talk it through.

Waypoint Mentor's tutor explains this out loud, works through your own problems on a whiteboard, and checks you've got it. Free to start.

Explain this to me