Updated 12 September 2026

The bootstrap

Got one sample. Need a standard error. Traditional approach: derive the sampling distribution of your statistic, plug in the variance formula, hope you didn’t mess up the algebra. Bootstrap approach: resample your data with replacement a few thousand times, take the statistic of each resample, and use the spread of those numbers as the standard error.

What to notice

• Bootstrap SE ≈ CLT SE. For the sample mean, the bootstrap standard error approaches √((n−1)/n) × $s/\sqrt{n}$ as the number of resamples grows, when s uses the n−1 variance denominator. The small finite-sample factor comes from treating the empirical distribution as the population. That’s the sanity check: for easy statistics the bootstrap reproduces the analytic answer.
• Skewness does not disappear automatically. A small lognormal sample can miss influential tail observations. Its bootstrap distribution may look smooth while understating the uncertainty in a new sample from the population.
• Percentile intervals have nominal coverage. The 2.5th and 97.5th bootstrap percentiles define a nominal 95% interval. Actual repeated-sample coverage need not be 95%, especially with small samples, bias, or heavy tails.

Work through one resample

Suppose your data are [1, 2, 6], with mean 3. A bootstrap sample takes three independent draws with replacement from those three values. [6, 6, 1] is allowed and has mean 13/3. [2, 2, 2] is also allowed. Repeating this generates a distribution of resampled means around the observed mean, not around an independently known population mean.

The empirical variance is (4+1+9)/3 = 14/3. Therefore the exact conditional variance of a size-three bootstrap mean is 14/9, and its standard error is about 1.247. By comparison s/√3 is √(7/3), about 1.528. More bootstrap repetitions reduce simulation noise; they do not turn three observations into a larger original sample.

Why it matters

The bootstrap shines when analytic SEs don’t exist or are too fragile:

• Median, quantiles, IQR. Resampling can approximate uncertainty under suitable regularity conditions, such as a positive density at the target quantile; extreme quantiles and small samples need particular care.
• Non-linear functions of the data. Correlation, R², ratios, ROC-AUC. Delta-method approximations get hairy; bootstrap can help, but consistency and coverage need checking for the statistic and sampling design.
• Regression coefficients with heteroskedasticity. Cluster resampling preserves within-cluster dependence; a wild bootstrap can address heteroskedastic regression errors. The resampling scheme must match the design.

Variants worth knowing

• BCₐ (bias-corrected and accelerated). Adjusts the percentile CI for skewness and bias in the bootstrap distribution. It can improve coverage under suitable regularity conditions; improvement is not universal.
• Parametric bootstrap. Fit a model, then resample from the fitted model instead of from the data. Relies on the fitted model being an adequate description of the sampling process.
• Block bootstrap. For time series, resample blocks of consecutive observations to preserve autocorrelation.

Read the resampling experiment

Open the original-observation table and select a bootstrap replicate. Every row is assigned mass 1/n in the empirical distribution, even when two rows have the same value. The table gives selection counts and the draw order, making duplicates and omitted rows visible. Changing B keeps the original data and earlier resamples; changing the source or original n changes the dataset being analyzed.

The interval uses the empirical inverse CDF without interpolation: the p-quantile is the sorted value in position ceil(Bp), counted from one. Different percentile conventions can give slightly different endpoints at finite B. The separate repeated-study experiment generates 60 new datasets from the known synthetic source and uses 300 resamples per dataset, so it asks about coverage rather than just conditional simulation precision.

A failure you can prove without simulation

Suppose the target is the unknown upper endpoint θ of Uniform(0, θ), estimated by the sample maximum M. Every ordinary resample has maximum at most M. For observations [0.2, 0.5, 0.8], all 27 ordered size-three resamples have maxima 0.2, 0.5, or 0.8. Their respective counts are 1, 7, and 19, giving a 95% percentile interval [0.2, 0.8]. If the actual endpoint is 1, that interval misses it.

More generally, M is strictly below θ with probability one under this continuous model. Every ordinary percentile bootstrap interval for the endpoint has its upper endpoint at most M, so its coverage of θ is zero. A bootstrap that works for a smooth mean need not work for a support endpoint. A justified parametric or specialized procedure is needed for this different target.

Keep the sampling units intact

Independent row resampling assumes the rows act as independent sampling units. Ten measurements of one person are not ten independent people. Resampling whole people can preserve within-person dependence; time-series settings need an appropriate block or model-based scheme. Selecting a block length or cluster definition is part of the statistical design, not something a larger B resolves.

Reference

CMU Advanced Data Analysis: Bootstrapping explains model-based and empirical resampling and the distinction between percentile and pivotal intervals. The three-value resample and endpoint counterexamples above are finite calculations, rather than claims of universal bootstrap validity.

The empirical CDF lesson makes the resampling distribution explicit, including ties and its inverse-quantile convention.