Statistics & Probability · Grades 10, 11
Surveys, Experiments and Observational Studies
Quick answer
A sample survey measures a population without changing it, an observational study compares groups that formed on their own, and an experiment assigns treatments at random. Randomization appears twice and does two different jobs: random selection from a population lets a result be generalized to it, and random assignment to treatments lets a difference be blamed on the treatment.
What you'll learn
- Tell a survey, an observational study and an experiment apart
- Explain what each design can support as a conclusion
- Distinguish random selection from random assignment
Statistics is a process, not a number
A population is everyone the question is about: every student in a district, every registered voter in a state, every battery coming off a line. A parameter is a number describing that population — the true proportion who walk to school, the true mean charge a battery holds.
Measuring the whole population is almost never possible, so a sample is measured instead, and a number computed from it — a statistic — is used to estimate the parameter.
That arrow is the whole subject. Whether it is trustworthy depends on one thing: how the sample was chosen.
Three designs
| Design | What the researcher does | What it can support |
|---|---|---|
| Sample survey | selects part of a population and measures it | an estimate of a population value |
| Observational study | records groups that already differ | an association between two variables |
| Experiment | assigns treatments to subjects | a claim about cause |
The rows go in order of what they can prove. A survey describes a population, a study finds a pattern, and an experiment explains one.
A sample survey estimates a population value
A district wants to know what fraction of its high school students walk or bike to school. Asking all is impractical, so are chosen at random and asked.
Random selection means every student has the same chance of being picked. It does not make the estimate perfect. It makes the estimate unbiased: too high as often as too low, with no systematic tilt.
A sample chosen any other way risks a tilt that no amount of data removes.
An observational study finds an association
Now a different question: does eating breakfast raise test scores?
An observational study records who ate breakfast and what they scored. Suppose the breakfast eaters average points higher. That gap is real, and it is evidence of an association.
It is not evidence that breakfast caused it. The two groups differ in more than breakfast. Students who eat breakfast are more likely to have gone to bed on time. They are also more likely to live in a household with a steady morning routine, and to have an adult at home. Any of those could raise a score.
A variable like that — one that differs between the groups and also affects the result — is a confounding variable. Its effect and breakfast’s effect arrive mixed together, and no calculation on this data can separate them.
An experiment assigns the treatment
To test cause, stop recording who chose breakfast and start deciding it.
Take volunteers. Flip a coin for each one: heads, they get breakfast; tails, they do not. Then give everyone the same test.
Random assignment is what changes everything. Because a coin decided the groups, they are alike on average in bedtime, household, motivation, prior grades, and every other variable — including ones nobody thought to measure.
So if the breakfast group scores higher by more than chance can explain, the treatment is the only remaining explanation.
Why randomization does two separate jobs
Randomization shows up in both good designs, but it is doing something different in each, and the two are often confused.
Random selection decides who is studied. It draws the sample from the population without tilt, so what is true in the sample is, within a known margin, true in the population. This is what lets a result be generalized.
Random assignment decides who gets which treatment. It makes the groups alike before the treatment starts, so a later difference has one candidate explanation left. This is what supports a claim about cause.
| Random assignment | No random assignment | |
|---|---|---|
| Random selection | cause, for the whole population | association, for the whole population |
| No random selection | cause, for these subjects only | association, for these subjects only |
The top-left cell is the strongest study a person can run, and it is rare, because volunteers for an experiment are seldom a random sample of anybody.
Most real studies sit in one of the other three cells, which is why reading the design matters more than reading the headline.
What each design can claim
| Finding | Design | Fair conclusion |
|---|---|---|
| Walkers are of a random sample of | survey | about of the district walks |
| Breakfast eaters score points higher | observational | breakfast and scores are associated |
| Randomly assigned breakfast group scores points higher | experiment | breakfast raised the scores |
Read the middle column first. It decides what the third column is allowed to say.
Worked examples
Common mistakes
Practice problems
-
A state agency randomly selects licensed drivers and asks how many miles they drove last year. Name the design.
Answer
A sample survey.
Full solution
Drivers were selected at random and measured. Nothing was assigned to them, so it is a survey, and it estimates a state-wide mean.
-
A company gives half its warehouses new scanners, chosen by coin flip, and compares error rates. Name the design.
Answer
An experiment.
Full solution
The coin flip is random assignment of a treatment, which is what makes a study an experiment.
-
A study follows people who already run regularly and people who do not, then compares resting heart rates. Name the design.
Answer
An observational study.
Full solution
The groups formed on their own. The researcher records, and assigns nothing.
-
In problem 3, name a possible confounding variable.
Answer
Diet, age, body weight or smoking — any variable that differs between runners and non-runners and also affects heart rate.
Full solution
Runners also tend to be younger, lighter and less likely to smoke. Each of those lowers a resting heart rate on its own, so the effect of running cannot be separated from them.
-
Which design supports the claim “the new fertilizer increases yield”?
Answer
An experiment, with plots assigned to fertilizers at random.
Full solution
A cause claim needs random assignment. Comparing farms that already use the fertilizer would leave soil, climate and farming practice confounded with it.
-
A radio host asks listeners to call in about a proposed tax. call. Can the result be generalized to the state?
Answer
No. It is a voluntary response sample, not a random one.
Full solution
The people who call are the people with strong opinions and free time, which is not the state. The sample size of does not repair the tilt — the poll had six hundred times as many responses and still missed.
-
A study randomly assigns volunteers to two diets and finds a real weight difference. What can be concluded, and about whom?
Answer
The diet caused the difference, for these volunteers.
Full solution
Random assignment supports the cause claim.
Random selection was not used — they volunteered — so the result does not automatically extend to the wider population. It belongs in the bottom-left cell of the table.
-
Explain why random assignment makes the treatment groups comparable even on variables nobody measured.
Hint
What does the coin know about the subjects?
Answer
The coin ignores every variable equally, so no variable tilts toward one group on average.
Full solution
A coin has no information about bedtime, income, motivation or anything else. Each subject lands in a group for reasons unrelated to every one of those variables.
Over many subjects, that spreads each variable across the two groups in roughly equal measure — including variables the researcher never thought of, and variables that were impossible to measure.
That is the property no other design has. A researcher can adjust for variables they recorded; only random assignment handles the rest.
-
A hospital reports that patients treated in its new wing recover faster than patients in the old wing. Why is this weak evidence that the new wing works?
Answer
Patients were not assigned at random, so the two groups may differ in how sick they were.
Full solution
Assignment to a wing is usually made by staff, by availability, or by the severity of the case.
If the newer wing takes the less urgent cases, faster recovery follows from who was sent there rather than from the wing itself. Severity is confounded with the wing.
Assigning arriving patients at random would separate the two.
-
A news story reports that people who take a daily multivitamin live longer, in a study of adults. Name the design, the fair conclusion, and one question to ask.
Hint
Did anyone assign the vitamins?
Answer
Observational. The fair conclusion is an association. Ask whether the groups differed in other health habits.
Full solution
Nothing in the report says vitamins were assigned, so adults chose for themselves. That makes it an observational study, whatever its size.
The fair conclusion is that taking a multivitamin and living longer go together in this data.
The question to ask is what else separates the two groups. People who take a daily vitamin are more likely to see a doctor regularly, exercise, and have health insurance — each of which extends life on its own.
The size of makes the association precisely measured. It does nothing to turn it into a cause.
Frequently asked questions
What is the difference between an experiment and an observational study?
In an experiment the researcher assigns the treatment. In an observational study the groups already differ, and the researcher only records what happens.
Why can only an experiment show cause?
Random assignment makes the two groups alike on every other variable, on average, so the treatment is the one thing left that could explain a difference.
What is a confounding variable?
A variable that differs between the groups and also affects the result, so its effect cannot be separated from the treatment's.
Does a huge sample fix a biased one?
No. Bias comes from how people were chosen, not how many. The 1936 Literary Digest poll had 2.4 million responses and still picked the wrong winner.
What is random selection versus random assignment?
Random selection chooses who is studied, which lets the result extend to the population. Random assignment decides who gets which treatment, which supports a claim about cause.
Standards alignment
This lesson covers the following Common Core State Standards for Mathematics.
- CCSS.MATH.CONTENT.HSS.IC.A.1Making Inferences and Justifying ConclusionsUnderstand statistics as a process for making inferences about population parameters based on a random sample from that population.
- CCSS.MATH.CONTENT.HSS.IC.B.3Making Inferences and Justifying ConclusionsRecognize the purposes of and differences among sample surveys, experiments, and observational studies; explain how randomization relates to each.