Updated 12 September 2026

Regression to the mean

Select people with unusually high scores on a noisy test, then measure the same fixed underlying quantity with independent fresh measurement noise. Their average repeat score may be less extreme because the first selection also favored positive measurement noise. Compare standardized measurements on the same scale; a change in raw height across generations is not the model here.

What to notice

• Orange dots pass the selected cutoff on the first measurement. The larger dot is their observed first/repeat group mean.
• The population conditional means are compared with the actual selected sample. A small selected group can fluctuate above or below its prediction; an empty group has no fabricated sample mean.
• r=1 removes measurement noise and makes both readings identical. r=0 makes the two readings independent, so the selected repeat population mean is zero.
• The red line $E[Y \mid X=x] = r\,x$ is the conditional expectation under the standardized normal model, not a rule requiring every individual point to lie on it.

Why it happens

The math is simple: in a bivariate normal with correlation r, the conditional expectation is $E[Y \mid X=x] = r\,x$. For |r| < 1, the conditional mean of Y is less extreme in magnitude than X on these standardized scales. The linear conditional-mean formula follows from the joint normal model. Imperfect correlation alone is not a universal guarantee of this exact line.

The practical danger

Regression to the mean produces many false causal stories:

• A student scores unusually low on a test, gets tutoring, scores higher — tutoring gets the credit.
• A sports team has an exceptional season, regresses the next year — the coach gets blamed.
• A company tries an unusual intervention when sales are worst, sales recover — intervention gets the credit.

Such changes can occur without an intervention, but this pattern alone cannot establish whether a particular intervention helped. A suitable comparison group is needed. Selection, repeated-measurement noise, and an intervention are different parts of a causal question; the scatterplot alone cannot separate every possible explanation in real data.

Construct repeated measurements of the same quantity

The simulator draws a latent value A with variance r, then independent errors e₁ and e₂ with variance 1−r. It sets X=A+e₁ and Y=A+e₂. The latent value does not change between measurements. Each reading has variance one, and their covariance is Var(A)=r, so their correlation is r as well.

The errors are independent conditional on the shared A; the readings themselves are correlated whenever r>0. The data table exposes the pieces for every selected unit. Selecting high X tends to select positive e₁ as well as high A. The fresh e₂ does not inherit that selection.

Work the conditional prediction

At r=0.6, a first standardized reading x=2 gives predicted repeat reading 1.2, hence a predicted change of −0.8. That does not mean the latent quantity deteriorated. It also does not mean every unit with x near 2 will score 1.2 next time: conditional variance is 1−r²=0.64 in this standardized joint normal model.

For selection above the population upper quartile, the cutoff is approximately 0.674490 and the conditional mean first reading is approximately 1.271106. At r=0.6 the selected repeat population mean is 0.762664. The model table uses this truncated-normal calculation for the selected cutoff, rather than multiplying noisy observed group averages and labeling them exact population values.

References

Random Services: multivariate normal distributions gives the joint-normal conditional expectation and variance. The latent-plus-error construction above verifies its covariance directly. Compare joint and conditional distributions and Simpson’s paradox for other ways aggregation and selection affect interpretation.