Samples and Populations: Drawing Conclusions from Data

Population and sample

The population is everyone or everything you want to know about. The sample is the part you actually measure.

Question	Population	A possible sample
How do students at this school travel in?	all $900$ students	$60$ of them
How long do these batteries last?	every battery made	$200$ tested
Who will win the election?	all voters	$1500$ polled

Measuring the whole population is usually impossible, too slow, or — with the batteries — destroys the thing being measured. So you measure a part and reason about the whole, which is called inference.

What makes a sample random

A sample is random when every member of the population has an equal chance of being chosen.

Method	Random?
draw $60$ names from all $900$	yes
number every student and pick $60$ with a computer	yes
ask the first $60$ students off the bus	no
ask $60$ students in the library	no

The last two are not random. Students who take the bus are certain to be in the first sample, and students who walk cannot be. That is not bad luck. It is a systematic tilt built into the method.

Why bias cannot be fixed by asking more people

This is the point the whole topic turns on.

Suppose you survey travel habits by asking students getting off the bus, and you ask $600$ of them instead of $60$.

The bus riders were already certain to be included and the walkers already certain to be excluded. Asking ten times as many bus riders does not introduce a single walker. The estimate does not get closer to the truth. It gets more precise about the wrong group.

Sample size and bias fix different problems:

Problem	Caused by	Fixed by
the estimate wobbles	small sample	more people
the estimate is systematically off	biased method	a better method

A famous case: in 1936 a magazine polled over two million Americans and predicted the wrong winner of a presidential election, while a poll of a few thousand got it right. The huge sample had been drawn from car and telephone owners, who in 1936 were richer than the country as a whole. Two million of the wrong people lost to a few thousand of the right ones.

Estimating from a sample

Scale the sample proportion up to the population.

In a random sample of $50$ students, $12$ walk to school:

$$\frac{12}{50} = 0.24 = 24\%$$

For all $900$ students:

$$0.24 \times 900 = 216 \text{ students}$$

About $216$. “About” is doing necessary work — a different random sample of $50$ would give a slightly different figure, and the true number is near $216$ rather than exactly it.

That variation is why several samples are better than one. If four samples give $22\%$, $24\%$, $27\%$ and $23\%$, the clustering around the mid-twenties is itself the evidence, and it tells you roughly how much to trust the estimate.

Comparing two populations

To compare two groups, compare both their center and their spread.

Two classes take the same test:

$$\text{A: median } 70, \; \text{IQR } 16 \qquad \text{B: median } 79, \; \text{IQR } 12$$

Class B scored higher and more consistently. Both facts matter, and a report giving only the medians would have missed the second.

How much the two overlap decides how meaningful the gap is. These boxes overlap between $73$ and $78$, so plenty of Class A students outscored plenty of Class B students. B is better on average, not in a different league. Two box plots with no overlap at all would be a much stronger result.

Worked examples

Common mistakes

Practice problems