Comparing Two Treatments: Is the Difference Real?

$4$ problems

The standing group averaged $23$ and the regular group $19$, and $23 - 19 = 4$.

That it has no effect, so each subject’s result would be the same in either group.

Under that assumption the only random step was which subjects got which label, so reshuffling the labels reproduces what chance alone could have done.

About $1.1\%$

$\tfrac{10}{924} = 0.0108$, which is $1.1\%$ to one decimal place.

Yes.

$1.1\%$ is below $5\%$, so chance alone rarely produces a gap this large.

No. That is $19.6\%$, well above $5\%$.

$\tfrac{196}{1000} = 0.196$. Chance produces a gap this size about one time in five, so the data does not separate the treatment from chance.

The difference is $7$. A re-randomization is needed to see how often chance reaches $7$.

$48 - 41 = 7$.

A raw gap carries no verdict on its own. With widely varying results, chance reaches $7$ often; with tightly clustered results, it may never. Only the reshuffled distribution settles it.

Were the adults assigned to take the supplement at random?

If a coin decided who took it, the groups started alike and the supplement is the remaining explanation.

If adults chose for themselves, the groups differ in other health habits, and the report supports an association only. The sample size of $2{,}000$ does not decide this.

$4$

$\bar{A} = \tfrac{30 + 34 + 32 + 36}{4} = \tfrac{132}{4} = 33$ and $\bar{B} = \tfrac{27 + 31 + 29 + 29}{4} = \tfrac{116}{4} = 29$.

The difference is $33 - 29 = 4$.

What would a study of four people find?

A small study can miss a real difference, so failing to detect one is not evidence there is none.

Significance asks whether the data is too extreme for chance. A study with few subjects produces gaps so variable that almost nothing looks extreme.

A study of four people would call nearly every difference insignificant, including a large real one. The verdict would describe the study’s size rather than the treatments.

Reporting “no significant difference” is therefore a statement about what this study could detect, not a finding of equality.

What was assumed before the $1.1\%$ was computed?

He reversed the conditions. The $1.1\%$ assumes the desks do nothing and measures the data, not the other way around.

The calculation began by assuming the desks have no effect. Only under that assumption do the $924$ shuffles make sense, since they depend on each result belonging to a student rather than to a desk.

What came out was: given that the desks do nothing, gaps of $4$ or more occur $1.1\%$ of the time.

Leo’s sentence swaps the two halves: given the data, the chance the desks do nothing is $1.1\%$. That is a different quantity, and this method does not produce it — it would also require knowing how plausible the no-effect claim was before the experiment.

The correct reading is the first one, and it is enough. A result that the chance explanation reaches once in ninety tries is strong evidence against that explanation.