Part 2: Heteroskedasticity and robust standard errors

Where it comes from

The meat we need is XᵀΩX = Σᵢ σᵢ² xᵢxᵢᵀ, but σᵢ² is unobserved. HC0 replaces each σᵢ² with ê²ᵢ — the squared OLS residual for observation i. By the law of large numbers, (1/n)Σᵢ ê²ᵢ xᵢxᵢᵀ converges in probability to (1/n)Σᵢ σᵢ² xᵢxᵢᵀ as n → ∞, so HC0 is consistent. The problem is finite-sample bias: ê²ᵢ systematically undershoots σᵢ², because OLS residuals are shrunk by the hat matrix — the fit already "saw" observation i. HC0 ignores this and tends to produce SEs that are too small in moderate samples.

Where it comes from — and why it's a heuristic

HC1 transplants the homoskedastic degrees-of-freedom intuition into the sandwich. Under homoskedasticity, HC0 underestimates the true variance by exactly a factor of (n−k)/n — the same reason we divide by n−k rather than n when estimating σ². Multiplying HC0 by n/(n−k) exactly unbiases it in that special case.

Under heteroskedasticity, this is only approximately right. The bias in ê²ᵢ is observation-specific — it depends on hᵢᵢ — so applying a single global correction n/(n−k) is imprecise. HC1 is what you'd write down if you wanted to apply the homoskedastic intuition without thinking too hard. Importantly, HC1 is the global-average version of HC2: since Σᵢ hᵢᵢ = k, the average leverage is k/n, so the average of 1/(1−hᵢᵢ) across observations is approximately 1/(1−k/n) = n/(n−k). HC1 and HC2 agree when leverages are uniform across observations and diverge when a few observations have outsized leverage. It is Stata's default , robust because of historical convention, not because it is theoretically superior.

Where it comes from — a real derivation

Start from the key result established in the leverage section: E[ê²ᵢ | X] = (1−hᵢᵢ)σᵢ². This says that ê²ᵢ is not an unbiased estimate of σᵢ² — it is biased downward by the factor (1−hᵢᵢ). The fix is immediate: divide by (1−hᵢᵢ) to get an approximately unbiased estimate of σᵢ² for each observation separately:

E[ê²ᵢ / (1−hᵢᵢ) | X] ≈ σᵢ²

Plugging this into the meat gives an estimator of XᵀΩX that is approximately unbiased observation by observation:

E[Σᵢ (ê²ᵢ/(1−hᵢᵢ)) xᵢxᵢᵀ | X] ≈ Σᵢ σᵢ² xᵢxᵢᵀ = XᵀΩX

This is a per-observation correction. Each residual is inflated by its own leverage factor, so high-leverage observations — which are pulled closest to the fitted line and therefore have the most downward-biased residuals — receive the largest correction. This is the right behavior, and it comes from an actual derivation rather than analogy with the homoskedastic case.

Where it comes from — the jackknife

HC3 comes from leave-one-out resampling. Define the leave-one-out residual for observation i as the prediction error you would get if you re-estimated β without using observation i:

ê_i⁽⁻ⁱ⁾ = Yᵢ − xᵢᵀ β̂_(−i)

Actually refitting the model n times would be expensive, but there is a beautiful closed-form identity (a consequence of the Sherman–Morrison formula):

ê_i⁽⁻ⁱ⁾ = êᵢ / (1 − hᵢᵢ)

The leave-one-out residual is simply the in-sample residual divided by (1−hᵢᵢ) — no refitting needed. The jackknife variance estimator then uses these honest, leave-one-out residuals in the meat:

V̂_jack(β̂) ≈ (XᵀX)⁻¹ · [Σᵢ (ê_i⁽⁻ⁱ⁾)² xᵢxᵢᵀ] · (XᵀX)⁻¹ =  (XᵀX)⁻¹ · [Σᵢ ê²ᵢ/(1−hᵢᵢ)² xᵢxᵢᵀ] · (XᵀX)⁻¹

That is HC3. The motivation for using leave-one-out residuals is that they are "honest" about how variable β̂ is: observation i did not help fit the model used to predict it, so the residual is not artificially deflated. HC3 is more conservative than HC2 — it overcorrects in large samples but behaves best in small samples or when leverage is concentrated in a few observations.