consistency and the linear probability model

Summary

A while back Twitter once again lost its collective mind and decided to rehash the logistic regression versus linear probability model debate for the umpteenth time. The genesis for this new round of chaos was Gomila (2019), a pre-print by Robin Gomila, a grad student in psychology at Princeton.

So I decided to learn some more about the linear probability model (LPM), which has been on my todo list since Michael Weylandt suggested it as an example for a paper I’m working on. In this post I’ll introduce the LPM, and then spend a bunch of time discussing when OLS is a consistent estimator.

Update: I turned this blog post into somewhat flippant, meme-heavy slides.

What is the linear probability model?

Suppose we have outcomes $Y\in \{0,1\}$ and fixed covariate vectors $X$. The linear probability model is a model, that is, a set of probability distributions that might have produced our observed data. In particular, the linear probability assumes that the data generating process looks like:

$$\begin{array}{r}P(Y=1|X)=\{\begin{array}{ll}1& {\beta }_0+{\beta }_1{X}_1+...+{\beta }_k{X}_k>1\\ {\beta }_0+{\beta }_1{X}_1+...+{\beta }_k{X}_k& {\beta }_0+{\beta }_1{X}_1+...+{\beta }_k{X}_k\in [0,1]\\ 0& {\beta }_0+{\beta }_1{X}_1+...+{\beta }_k{X}_k<0\end{array}\end{array}$$

Essentially we clip $P(Y=1|X)=X\beta$ to $[0,1]$ to make sure we get valid probabilities. We can contrast the LPM with the binomial GLM, where the assumption is that:

$$\begin{array}{r}P(Y=1|X)=\frac{1}{1+\exp (-({\beta }_0+{\beta }_1{X}_1+...+{\beta }_k{X}_k))}\end{array}$$

At points throughout this post we will also be interested in Gaussian GLMs. In slightly different notation, we can write all of these models as follows:

$$\begin{array}{rlr}{Y}_i|X_i& \sim \mathrm{Bernoulli}(min(1,max(0,X_i\beta )))& \text{linear probability model}\\ Y_i|X_i& \sim \mathrm{Bernoulli}({\mathrm{logit}}^{-1}(X_i\beta ))& \text{logistic regression / binomial GLM}\\ Y_i|X_i& \sim \mathrm{Normal}(X_i\beta ,{\sigma }^2)& \text{linear regression / Gaussian GLM}\end{array}$$

The OLS estimator

When people refer to the linear probability model, they are referring to using the Ordinary Least Squares estimator as an estimator for $\beta$, or using $X{\hat{\beta }}_{\text{OLS}}$ as an estimator for $\mathbb{E}(Y|X)=P(Y=1|X)$. The OLS estimator is:

$$\begin{array}{r}{\hat{\beta }}_{\text{OLS}}=(X^{\prime }X{)}^{-1}{X}^{\prime }Y.\end{array}$$

Most people have seen the OLS estimator derived as the MLE of a Gaussian linear model. Here we have binary data, which is definitely non-Gaussian, and this is aesthetically somewhat unappealing. The big question is whether or not using ${\hat{\beta }}_{\text{OLS}}$ on binary data actually works.

A bare minimum for inference

When we do inference, we pretty much always want two things to happen. Given our model (the assumptions we are willing to make about the data generating process):

1. our estimand should be identified, and
2. our estimator should be consistent.

Identification means that each possible value of the estimand should correspond to a distinct distribution in our probability model. When we say that we want a consistent estimator, we mean that our estimator should recover the estimand exactly with infinite data. All the estimands in the LPM debate are identified (to my knowledge), so this isn’t the big deal here. But consistency matters a lot!

Gomila (2019) claims that ${\hat{\beta }}_{\text{OLS}}$ is unbiased and consistent for $\beta$, and attempts to demonstrate this two ways: (1) analytically, and (2) via simulation.

And this is the point that I got pretty confused, because the big question is: consistent under what model? Depending on who you ask, we could conceivably be assuming that the data comes from:

• a Gaussian GLM (linear regression),
• the linear probability model, or
• a binomial GLM (logistic regression).

Consistency of the OLS estimator

The easiest case is when we assume that a Gaussian GLM (linear regression model) holds. In this case, ${\hat{\beta }}_{\text{OLS}}$ is unbiased and consistent. My preferred reference for this is Rencher and Schaalje (2008).

When the linear probability model holds, ${\hat{\beta }}_{\text{OLS}}$ is in general biased and inconsistent (Horrace and Oaxaca (2003)). However, Gomila (2019) claims that OLS is unbiased and consistent. Over Twitter DM I clarified that this claim is with respect to the linear probability model. Gomila referred me to Wooldridge (2001) for an analytic proof, and to his simulation results for empirical confirmation.

At this point the main result of Horrace and Oaxaca (2003) becomes germane. It goes like this: the OLS estimator is consistent and unbiased under the linear probability model if $X_i^T\beta \in [0,1]$ for all $i$, otherwise the OLS estimator is biased and inconsistent. In fact, Wooldridge (2001) makes the same point:

Unless the range of $X$ is severely restricted, the linear probability model cannot be a good description of the population response probability $P(Y=1|X)$.

Here are some simulations demonstrating this bias and inconsistency when the $X_i^T\beta \in [0,1]$ condition is violated. It’s pretty clear that OLS is in general biased and inconsistent under the linear probability model. I was curious why this wasn’t showing up in Gomila’s simulations, so I took a look at his code and it turned out he wasn’t simulating from diverse enough data generating processes.

We discussed this over Twitter DM, and Gomila has since updated the code, but I believe the new simulations still do not seriously violate the $X_i^T\beta \in [0,1]$ condition. I briefly played around with the code but then it was 2 AM and I didn’t understand DeclareDesign particularly well so I gave up. Many kudos to Gomila for posting his code for public review.

Anyway, the gist is that OLS is consistent under the linear probability model if $X_i^T\beta$ is between zero and one for all $i$.

What if logistic regression is the true model?

Another reasonable question is: what happens to ${\hat{\beta }}_{\text{OLS}}$ under a binomial GLM? The story here is pretty much the same: ${\hat{\beta }}_{\text{OLS}}$ is not consistent for $\beta$, but it often does a decent job of estimating $\mathbb{E}(Y|X)$ anyway (Battey, Cox, and Jackson 2019; Cox 1958).

The intuition behind this comes from M-estimation¹, which we digress into momentarily. The idea is to observe that ${\hat{\beta }}_{\text{OLS}}$ is an M-estimator. To see this, recall that OLS is based on minimizing

$$\begin{array}{r}\mathcal{L}(X,Y,\beta )=\frac{1}{2}\Vert Y-X\beta {\Vert }_2^2\end{array}$$

which has a global minimizer ${\beta }_0$. The gradient,

$$\begin{array}{r}\mathrm{\nabla }\mathcal{L}(X,Y,\beta )={(Y-X\beta )}^{\prime }X,\end{array}$$

should be zero at ${\beta }_0$ under any model with linear expectation (provided that $X_i\ne 0$ for all $i$). So we take $\psi =\mathrm{\nabla }\mathcal{L}(X,Y,{\beta }_0)$ as the function in our estimating equation, since $\mathbb{E}(\psi )=0$. This lets us leverage standard results from M-estimation theory.

In particular, M-estimation theory tells us that ${\hat{\beta }}_{\text{OLS}}$ is consistent under any regular distribution $F$ such that ${\beta }_0$ is the unique solution to

$$\begin{array}{r}{\mathbb{E}}_F\left[{(Y_i-X_i^T{\beta }_0)}^{\prime }{X}_i\right]=0.\end{array}$$

We can use this to derive a sufficient condition for consistency; namely OLS is consistent for ${\beta }_0$ if

$$\begin{array}{rl}0& ={\mathbb{E}}_F\left[{(Y-X_i^T{\beta }_0)}^{\prime }{X}_i\right]\\ & ={\mathbb{E}}_F\left[{\mathbb{E}}_F\left[{(Y-X_i^T{\beta }_0)}^{\prime }{X}_i|X_i\right]\right]\\ & ={\mathbb{E}}_F\left[{\mathbb{E}}_F{[Y-X_i^T{\beta }_0|X_i]}^{\prime }{X}_i\right].\end{array}$$

So a sufficient condition for the consistency of OLS is that

$$\begin{array}{r}{\mathbb{E}}_F(Y|X_i)=X_i^T{\beta }_0.\end{array}$$

That is, if the expectation is linear in some model parameter ${\beta }_0$, OLS is consistent for that parameter. If you are an economist you’ve probably seen this written as

$$\begin{array}{r}{\mathbb{E}}_F({\epsilon }_i|X_i)=0,\end{array}$$

where we take ${\epsilon }_i=Y_i-\mathbb{E}(Y_i|X_i)$.

The sufficient condition is also a necessary condition, and if it is violated ${\hat{\beta }}_{\text{OLS}}$ will generally be asymptotically biased.

Returning from our digression into M-estimation, we note the condition that ${\mathbb{E}}_F(Y|X_i)=X_i^T{\beta }_0$ is shockingly weaker than the assumption we used to derive ${\hat{\beta }}_{\text{OLS}}$, where we used the fact that $Y$ followed a Gaussian distribution. For consistency, we only need an assumption on the first moment of the distribution of $Y$, rather than the full parametric form of the model, which specifies every moment $\mathbb{E}(Y^2),\mathbb{E}(Y^3),\mathbb{E}(Y^4),...$ of $Y$.

Anyway, the expectation of a binomial GLM is ${\mathrm{logit}}^{-1}(X_i\beta )$, and the expectation of the LPM is $min(1,max(0,X_i\beta ))$. For certain $X_i$ and $\beta$, these expectations are very close to $X_i\beta$. In these cases, estimates for $\mathbb{E}(Y|X)$ based on ${\hat{\beta }}_{\text{OLS}}$ will do well².

Looking at the expectations of the models, we can see they aren’t wildly different:

Note that, under the LPM, ${\hat{\beta }}_{\text{OLS}}$ can give us good estimates for $\beta$, but under logistic regression, we only get good estimates of $\mathbb{E}(Y|X)$³.

So maybe don’t toss the LPM estimator out with the bath water. Sure, the thing is generally inconsistent and aesthetically offensive, but whatever, it works on occasion, and sometimes there will be other practical considerations that make this tradeoff reasonable.

Where the estimator comes from doesn’t matter

Bayes Twitter in particular had a number of fairly vocal LPM opponents, on the basis that ${\hat{\beta }}_{\text{OLS}}$ was derived under a model that can’t produce the observed data.

This might seem like a dealbreaker, but it doesn’t bother me. Where the estimator comes from doesn’t actually matter. If it has nice properties given the assumptions you are willing to make, you should use it! Estimators derived under unrealistic models⁴ often turn out to be good!

In a bit more detail, here’s how I think of model checking:

1. There’s an estimand I want to know
2. I make some assumptions about my data generating process
3. I pick an estimator that has nice properties given this data generating process

The issue here is that my assumptions about the data generating process can be wrong. And if my modeling assumptions are wrong, then my estimator might not have the nice properties I want it to have, and this is bad.

The concern is that using ${\hat{\beta }}_{\text{OLS}}$ corresponds to making a bad modeling assumption. In particular, using a Gaussian model for binary data isn’t defensible.

That’s not really what’s going on though. Instead, we start by deriving an estimator under the linear regression model. Then, we show this estimator has nice properties under a new, different model. To do model checking, we need to test whether the new, different model holds. Whether or not the data comes from a linear regression is immaterial.

Nobody who likes using ${\hat{\beta }}_{\text{OLS}}$ is arguing that it’s reasonable to assume a Gaussian generative model for binary data. LPM proponents argue that the Gaussian modeling assumption is violated, but it isn’t violated in an important way. Look, they say, the key assumptions of a Gaussian model that we used to derive consistency results can still hold, even if some other assumptions get violated. This is exactly what the M-estimation approach formalizes.

What about uncertainty in our estimator?

So far we have established that ${\hat{\beta }}_{\text{OLS}}$ is at least occasionally a consistent estimator for $P(Y=1|X)$ under the LPM and logistic regression.

In practice, we also care about the uncertainty in $\hat{\beta }$. We might want a confidence interval, or, God forbid, a p-value. So we should think about consistent estimators for $\mathbf{Cov}(\hat{\beta })$. In the grand LPM debate, most people suggest using robust standard errors. For a while, it was unclear to me how these robust standard errors were derived and under what conditions they are consistent. The answer again comes from Boos and Stefanski (2013), and all we need for the consistency of robust standard errors for $\mathbf{Cov}(\hat{\beta })$ is some moment conditions (the existence of (7.5) and (7.6) in the text), which linear regression, logistic regression, and the LPM all satisfy.

It only took several weeks of misreading Boos and Stefanski (2013) and asking dumb questions on Twitter to figure this out. Thanks to Achim Zeileis, James E. Pustejovsky, Cyrus Samii for answering those questions. Note that White (1980) is another canonical paper on robust standard errors, but it doesn’t click for me like the M-estimation framework does.

Takeaways

This post exists because I wasn’t sure when ${\hat{\beta }}_{\text{OLS}}$ was a consistent estimator, and the rest of the LPM debate seemed like a lot of noise until I could figure that out. I have three takeaways.

1. Properties of estimators are always with respect to models, and it’s hard to discuss which estimator is best if you don’t clarify modeling assumptions.

2. If there are multiple reasonable estimators, fit them all. If they result in substantively different conclusions: congratulations! Your data is trash and you can move on to a new project!

3. If you really care about getting efficient, consistent estimates under weak assumptions, you should be doing TMLE or Double ML or burning your CPU to a crisp with BART⁵.

Anyway, this concludes the “I taught myself about the linear probability model and hope to never mention it again” period of my life. I look forward to my mentions being a trash fire.