Applied work often studies the effect of a binary variable (“treatment”) using linear models with additive effects. I study the interpretation of the OLS estimands in such models when treatment effects are heterogeneous. I show that the treatment coefficient is a convex combination of two parameters, which under certain conditions can be interpreted as the average treatment effects on the treated and untreated. The weights on these parameters are inversely related to the proportion of observations in each group. Reliance on these implicit weights can have serious consequences for applied work, as I illustrate with two well-known applications. I develop simple diagnostic tools that empirical researchers can use to avoid potential biases. Software for implementing these methods is available in R and Stata. In an important special case, my diagnostics require only the knowledge of the proportion of treated units.

MANY applied researchers study the effect of a binary variable (“treatment”) on the expected value of an outcome of interest, holding fixed a vector of control variables. As noted by Imbens (2015), despite the availability of a large number of semi- and nonparametric estimators for average treatment effects, applied researchers often continue to use conventional regression methods. In particular, numerous studies use ordinary least squares (OLS) to estimate
(1)
where y denotes the outcome, d denotes the treatment, and X denotes the row vector of control variables, x1,,xK. Usually τ is interpreted as the average treatment effect (ATE). This estimation strategy is used in many influential papers in economics (Voigtländer & Voth, 2012; Alesina, Giuliano, & Nunn, 2013; Aizer et al., 2016), as well as in other disciplines.

The great appeal of the model in equation (1) comes from its simplicity (Angrist & Pischke, 2009). At the same time, however, a large body of evidence demonstrates the importance of heterogeneity in effects (Heckman, 2001; Bitler, Gelbach, & Hoynes, 2006), which is explicitly ruled out by this same model. In this paper, I contribute to the recent literature on interpreting τ, the OLS estimand, when treatment effects are heterogeneous (Angrist, 1998; Humphreys, 2009; Aronow & Samii, 2016). I demonstrate that τ is a convex combination of two parameters, which under certain conditions can be interpreted as the average treatment effects on the treated (ATT) and untreated (ATU). Surprisingly, the weight that is placed by OLS on the average effect for each group is inversely related to the proportion of observations in this group. The more units are treated, the less weight is placed on ATT. One interpretation of this result is that OLS estimation of the model in equation (1) is generally inappropriate when treatment effects are heterogeneous.

It is also possible, however, to present a more pragmatic view of my main result. I derive a number of corollaries of this result that suggest several diagnostic methods that I recommend to applied researchers. These diagnostics are applicable whenever the researcher is (a) studying the effects of a binary treatment, (b) using OLS, and (c) unwilling to maintain that ATT is exactly equal to ATU. Typically, such a homogeneity assumption would be undesirably strong because those choosing or chosen for treatment may have unusually high or low returns from that treatment, which would directly contradict the equality of ATT and ATU.

In deriving my diagnostics, I assume that the researcher is ultimately interested in ATE, ATT, or both and that she wishes to estimate the model in equation (1) using OLS but is concerned about treatment effect heterogeneity. In this case, my diagnostics are able to detect deviations of the OLS weights from the pattern that would be necessary to consistently estimate a given parameter. These diagnostics are easy to implement and interpret; they are bounded between 0 and 1 in absolute value, and they give the proportion of the difference between ATU and ATT (or between ATT and ATU) that contributes to bias. Thus, if a given diagnostic is close to 0, OLS is likely a reasonable choice, but if a diagnostic is far from 0, other methods should be used.

In an important special case, these diagnostics become particularly simple and immediate to report. If we wish to estimate ATT, this rule-of-thumb variant of my diagnostic is equal to the proportion of treated units, Pd=1; if our goal is to estimate ATE, the diagnostic is equal to 2×Pd=1-1, twice the deviation of Pd=1 from 50%. In short, OLS is expected to provide a reasonable approximation to ATE if both groups, treated and untreated, are of similar size. If we wish to estimate ATT, it is necessary that the proportion of treated units is very small.

It follows that OLS might often be substantially biased for ATE, ATT, or both. How common are these biases in practice? In a subset of 37 estimates from Card, Kluve, and Weber (2018), a survey of evaluations of active labor market programs, the mean proportion of treated units is 17.7%.1 Using the rule-of-thumb variants of my diagnostics, I establish that on average the difference between the OLS estimand and ATE is expected to correspond to 64.6% of the difference between ATT and ATU. Similarly, the expected difference between OLS and ATT is on average equal to 17.7% of the difference between ATU and ATT. In other words, these biases might often be large.

The remainder of the paper is organized as follows. Section II presents a leading example and the main theoretical results. Section III discusses two empirical applications. In a study of the effects of a training program (LaLonde, 1986), OLS estimates are very similar to ATT^. Yet in a study of the effects of cash transfers (Aizer et al., 2016), OLS estimates are similar to ATU^. Section IV concludes. Proofs and several extensions are provided in the online appendixes. The main results are implemented in newly developed R and Stata packages, hettreatreg.

A. Leading Example

To illustrate the problem with OLS weights, consider the classic example of the National Supported Work (NSW) program. Because this program originally involved a social experiment, the difference in mean outcomes between the treated and control units provides an unbiased estimate of the effect of treatment. LaLonde (1986) studies the performance of various estimators at reproducing this experimental benchmark when the experimental controls are replaced by an artificial comparison group from the Current Population Survey (CPS) or the Panel Study of Income Dynamics (PSID). Angrist and Pischke (2009) reanalyze the NSW–CPS data and conclude that OLS estimates of the effect of NSW program on earnings in 1978 are similar to the experimental benchmark of $1,794.2 In particular, their richest specification delivers an estimate of $794. As I will show, this conclusion is driven by the small proportion of treated units in these data.

In this example, ATT and ATU are likely to be substantially different. This is because the treated group, unlike the CPS comparison (untreated) group, was highly economically disadvantaged. It is plausible that ATU might be 0 or, due to the opportunity cost of program participation, even negative. Also, only 1.1% of the sample was treated, so ATE and ATU will be similar.

To demonstrate this, I modify the model in equation (1) to include all interactions between d and X. Estimation of this expanded model, again using OLS, allows us to separately compute ATE^, ATT^, and ATU^. This method is usually referred to as “regression adjustment” (Wooldridge, 2010) or “Oaxaca–Blinder” (Kline, 2011; Graham & Pinto, 2022). Using the control variables that deliver the estimate of $794, we obtain ATE^=-$4,930, ATT^=$796, and ATU^=-$4,996. It turns out that since ATE^ and ATU^ are indeed negative, the OLS estimate and ATE^ have different signs. Moreover, if we represent the OLS estimate as a weighted average of ATT^ and ATU^ with weights that sum to unity, we can write $794=w^ATT×$796+1-w^ATT×-$4,996, where w^ATT is the weight on ATT^. Solving for w^ATT yields w^ATT=99.96%. In other words, the hypothetical OLS weight on the effect on the treated is similar to the proportion of untreated units, 98.9%.

This “weight reversal” is not a coincidence. As I demonstrate below, the intuition from this example holds more generally, even though the OLS estimand is not necessarily a convex combination of two parameters from a procedure that controls for the full vector X.

B. Main Result

This section presents my main result, which focuses on the algebra of OLS and descriptive estimands that I define below. A causal interpretation of OLS also requires introducing the notion of potential outcomes as well as certain conditions that I discuss in section IIC, including an ignorability assumption. However, this is not needed for my main result.

If L·· denotes the linear projection, we are interested in the interpretation of τ in the linear projection of y on d and X,
(2)
when this linear projection does not correspond to the (structural) conditional mean. Let
(3)
be the unconditional probability of treatment and let
(4)
be the propensity score from the linear probability model or, equivalently, the best linear approximation to the true propensity score. Generally the specification in equations (2) and (4) can be arbitrarily flexible, so this approximation can be made very accurate; in fact, we can think of equation (2) as partially linear, where we may include powers and cross-products of original control variables.
After defining pX, it is helpful to introduce two linear projections of y on pX, separately for d=1 and d=0, namely,
(5)
and
(6)
Note that equations (4), (5), and (6) are definitional. It is sufficient for my main result that the linear projections introduced so far exist and are unique.
Assumption 1.

(i) E(y2) and E(X2) are finite. (ii) The covariance matrix of d,X is nonsingular.

Assumption 2.

VpXd=1 and VpXd=0 are nonzero, where V·· denotes the conditional variance (with respect to EpXd=j, j=0,1).

Assumption 1 guarantees the existence and uniqueness of the linear projections in equations (2) and (4). Similarly, assumption 2 ensures that the linear projections in equations (5) and (6) exist and are unique.3

The next step is to use the linear projections in equations (5) and (6) to define the average partial linear effect of d as
(7)
as well as the average partial linear effect of d on group j (j=0,1) as
(8)
These estimands are well defined under assumptions 1 and 2 and have a causal interpretation under additional assumptions, as discussed in section IIC.4 When the linear projections in equations (5) and (6) represent the conditional mean of y, the average partial linear effects of d overlap with its average partial effects. It should be stressed, however, that theorem 1, the main result of this paper, is more general and requires only assumptions 1 and 2.
Theorem 1
(Weighted Average Interpretation of OLS). Under assumptions 1 and 2,
where w1=1-ρ×VpXd=0ρ×VpXd=1+1-ρ×VpXd=0 and w0=1-w1=ρ×VpXd=1ρ×VpXd=1+1-ρ×VpXd=0.
Proof.

See online appendix A.

Theorem 1 shows that τ, the OLS estimand, is a convex combination of τAPLE,1 and τAPLE,0. The definition of τAPLE,j makes it clear that τ is equivalent to the outcome of a particular three-step procedure. In the first step, we obtain pX, the propensity score. Next, in the second step, we obtain τAPLE,1 and τAPLE,0, as in equation (8), from two linear projections of y on pX, separately for d=1 and d=0. This is analogous to the regression adjustment procedure in section IIA, although now we control for pX rather than the full vector X. Finally, in the third step, we calculate a weighted average of τAPLE,1 and τAPLE,0. The weight on τAPLE,1, w1, is decreasing in VpXd=1VpXd=0 and ρ, and the weight on τAPLE,0, w0, is increasing in VpXd=1VpXd=0 and ρ.5 This is clearly undesirable, since τAPLE=ρ×τAPLE,1+1-ρ×τAPLE,0.

This weighting scheme is also surprising: the more units belong to group j, the less weight is placed on τAPLE,j, the effect for this group. There are several ways to provide intuition for this result. One is provided in the next section. Another follows from an alternative proof of theorem 1, which is provided with discussion in online appendix B2. It parallels the intuition in Angrist (1998) and Angrist and Pischke (2009) that OLS gives more weight to treatment effects that are better estimated in finite samples.6

C. Causal Interpretation

The fact that theorem 1 requires only the existence and uniqueness of several linear projections makes this result very general. However, one concern about this result might be that τAPLE,1 and τAPLE,0 do not necessarily correspond to the usual (causal) objects of interest. To define these objects, we need two potential outcomes, y(1) and y(0), only one of which is observed for each unit, y=y(d)=y(1)×d+y(0)×1-d. The parameters of interest, ATE, ATT, and ATU, are defined as τATE=E[y(1)-y(0)], τATT=E[y(1)-y(0)d=1], and τATU=E[y(1)-y(0)d=0]. A causal interpretation of OLS also entails the following assumptions.

Assumption 3

(Ignorability in Mean). (i) Ey(1)X,d=Ey(1)X; and (ii) Ey(0)X,d=Ey(0)X.

Assumption 4.

(i) Ey(1)X=α1+γ1×pX; and (ii) Ey(0)X=α0+γ0×pX.

Assumptions 3 and 4 ensure that τ admits a causal interpretation. Assumption 3 is standard in the program evaluation literature (Wooldridge, 2010). Assumption 4 is not commonly used. Sufficient for this assumption, but not necessary, is that the conditional mean of d is linear in X and the conditional means of y(1) and y(0) are linear in the true propensity score, which is now equal to pX. Linearity of EdX is assumed in Aronow and Samii (2016) and Abadie et al. (2020). This assumption is not necessarily strong, since X might include powers and cross-products of original control variables. It is also satisfied automatically in saturated models, as in Angrist (1998) and Humphreys (2009). The linearity assumption for Ey(1)pX and Ey(0)pX dates back to Rosenbaum and Rubin (1983) but is restrictive. See also Imbens and Wooldridge (2009) and Wooldridge (2010) for a discussion.

Corollary 1
(Causal Interpretation of OLS). Under assumptions 1, 2, 3, and 4,
Proof.

Assumption 3 implies that Ey(1)-y(0)X=EyX,d=1-EyX,d=0. Then, assumption 4 implies that Ey(1)-y(0)X=α1-α0+γ1-γ0×pX, which in turn implies that τATT=τAPLE,1 and τATU=τAPLE,0. This, together with theorem 1, completes the proof.

Corollary 1 states that under assumptions 1, 2, 3, and 4, the OLS weights from theorem 1 apply to the causal objects of interest, τATT and τATU. Hence, τ has a causal interpretation. The greater the proportion of treated units, the smaller is the OLS weight on τATT. Again, this is undesirable since τATE=ρ×τATT+1-ρ×τATU.

To aid intuition for this surprising result, recall that an important motivation for using the model in equation (1) and OLS is that the linear projection of y on d and X provides the best linear predictor of y given d and X (Angrist & Pischke, 2009). However, if our goal is to conduct causal inference, then this is not, in fact, a good reason to use this method. Ordinary least squares is “best” in predicting actual outcomes, but causal inference is about predicting missing outcomes, defined as ym=y(1)×1-d+y(0)×d. In other words, the OLS weights are optimal for predicting “what is.” Instead, we are interested in predicting “what would be” if treatment were assigned differently.

Intuition suggests that if our goal were to predict ``what is'' and, without loss of generality, group 1 were substantially larger than group 0, we would like to place a large weight on the linear projection coefficients of group 1 (α1 and γ1), because these coefficients can be used to predict actual outcomes of this group. As noted by Deaton (1997) and Solon et al. (2015), the OLS weights are consistent with this idea. Indeed, theorem 1 apply to the causal objects of interest, τATT and τATU. Hence, τ has a causal interpretation. The greater the proportion of treated units, the smaller is the OLS weight on τATT. Again, this is undesirable since τATE=ρ×τATT+1-ρ×τATU.

To aid intuition for this surprising result, recall that an important motivation for using the model in equation (1) and OLS is that the linear projection of y on d and X provides the best linear predictor of y given d and X (Angrist & Pischke, 2009). However, if our goal is to conduct causal inference, then this is not, in fact, a good reason to use this method. Ordinary least squares is “best” in predicting actual outcomes, but causal inference is about predicting missing outcomes, defined as ym=y(1)×1-d+y(0)×d. In other words, the OLS weights are optimal for predicting “what is.” Instead, we are interested in predicting “what would be” if treatment were assigned differently.

Intuition suggests that if our goal were to predict ``what is'' and, without loss of generality, group 1 were substantially larger than group 0, we would like to place a large weight on the linear projection coefficients of group 1 (α1 and γ1), because these coefficients can be used to predict actual outcomes of this group. As noted by Deaton (1997) and Solon et al. (2015), the OLS weights are consistent with this idea. Indeed, theorem 1 also implies that
(9)
Namely, the OLS estimand is equal to the simple difference in means of y plus an adjustment term that depends on the difference in means of pX and a weighted average of γ1 and γ0. When group one is large, w0, the weight on γ1, is large as well.
Conversely, if group 1 is large but our goal is to predict missing outcomes, we need to place a large weight on α0 and γ0 because these coefficients can be used to predict counterfactual outcomes of group 1. To see this point, note that it follows from the discussion in Imbens and Wooldridge (2009) that when the conditional means of y(1) and y(0) are linear in X, we can write
(10)

where β1 and β0 are the coefficients on X in the conditional means of y(1) and y(0), respectively. Equations (9) and (10) reiterate the point of corollary 1 that τ and τATE have a very similar structure but differ substantially in how they assign weights. Indeed, in the case of τATE, when group 1 is large, the weight on β1 is small, the opposite of what we have seen for OLS.7

D. Implications of Theorem 1

There are several practical implications of my main result. Throughout this section, I assume that the researcher is interested in estimating τATE, τATT, or both, and wishes to use OLS to estimate the model in equation (1) but is concerned about the implications of theorem 1 and corollary 1. In corollaries 2 and 3, I show how to decompose the difference between τ and τATE or τ and τATT into components attributable to (a) the difference between τAPLE,1 and τATT, (b) the difference between τAPLE,0 and τATU (jointly referred to as “bias from nonlinearity”), and (c) the OLS weights on τATT and τATU (“bias from heterogeneity”).8 Because this paper generally focuses on what I now term “bias from heterogeneity,” my discussion below is restricted to this source of bias, which is equivalent to implicitly making assumptions 3 and 4.

Corollary 2.
Under assumptions 1 and 2,
where δ=ρ-w1=ρ2×VpXd=1-1-ρ2×VpXd=0ρ×VpXd=1+1-ρ×VpXd=0. Also, under assumptions 1, 2, 3, and 4,
Corollary 3.
Under assumptions 1 and 2,
Also, under assumptions 1, 2, 3, and 4,

The proofs of corollaries 2 and 3 follow from simple algebra and are omitted. These results show that regardless of whether we focus on τATE or τATT, the bias from heterogeneity is equal to the product of a particular measure of heterogeneity, namely, the difference between τATU and τATT, and an additional parameter that is easy to estimate, δ for τATE and w0 for τATT. While w0 is guaranteed to be positive under assumptions 1 and 2, δ may be positive or negative. Both w0 and δ, however, are bounded between 0 and 1 in absolute value. Thus, w0 and |δ| can be interpreted as the percentage of our measure of heterogeneity, τATU-τATT, which contributes to bias.9 It might be useful to report estimates of w0 and δ in studies that use OLS to estimate the model in equation (1).

As an example, consider the empirical application in section IIA. In this case, w^0=0.017 and δ^=-0.971. The interpretation of these estimates is as follows: if our goal is to estimate τATT, using the model in equation (1) and OLS is expected to bias our estimates by only 1.7% of the difference between τATU and τATT. If instead we wanted to interpret τ as τATE, our estimates would be biased by an estimated 97.1% of the difference between τATT and τATU. Thus, in this application, it might perhaps be acceptable to interpret τ as τATT but clearly not as τATE.

Assumption 5.

VpXd=1=VpXd=0.

The calculation of δ and w0 is further simplified under assumption 5. If we use δ* and w0* to denote the values of δ and w0 in this special case, we can write δ*=2ρ-1 and w0*=ρ. In this setting, the knowledge of δ and w0 requires only information on ρ, the proportion of units with d=1. Of course, the special case where VpXd=1=VpXd=0 is hardly to be expected in practice. Still, δ*=2ρ-1 and w0*=ρ can potentially serve as a rule of thumb.

The practical implications of assumption 5 are particularly clear when ρ is close to 0%, 50%, or 100%. When few units are treated, ττATT. When most of the units are treated, ττATU. Finally, when both groups are of similar size, ττATE. This can also be seen from corollary 4:

Corollary 4.
Under assumptions 1, 2, and 5,
Also, under assumptions 1, 2, 3, 4, and 5,

The proof follows immediately from simple algebra. Corollary 4 provides conditions under which OLS reverses the natural weights on τAPLE,1 and τAPLE,0 (or τATT and τATU). Indeed, under assumption 5, τ is a convex combination of group-specific average effects, with reversed weights attached to these parameters. Namely, the proportion of units with d=1 is used to weight the average effect of d on group 0, and vice versa.

The results in this section allow empirical researchers to interpret the OLS estimand when treatment effects are heterogeneous. Alternatively, it might be sensible to use any of the standard estimators for average treatment effects under ignorability, such as regression adjustment (see section IIA), weighting, matching, and various combinations of these approaches.10 It might also help to estimate a model with homogeneous effects using weighted least squares (WLS). Indeed, in online appendix B3, I demonstrate that when we regress y on d and pX, with weights of 1-ρw0 for units with d=1 and ρw1 for units with d=0, the WLS estimand is equal to τAPLE. In practice, of course, τAPLE can also be obtained directly from equation (7).

E. Related Work

This section discusses the relationship between my main result and those in Angrist (1998) and Humphreys (2009). These papers focus on saturated models with discrete covariates, in which the estimating equation includes an indicator for each combination of covariate values (“stratum”). In particular, Angrist (1998) provides a representation of τn in
(11)
where x1,,xS are stratum indicators. More precisely, Angrist (1998) demonstrates that
(12)

where τs=Eyd=1,xs=1-Eyd=0,xs=1. In online appendix B4, I demonstrate that this result follows from corollary 1 when the model for y is saturated.11 At the same time, the interpretation of OLS in Angrist (1998) is different from theorem 1 and corollary 1. On the one hand, unlike corollary 1 and Humphreys (2009), Angrist (1998) does not restrict the relationship between τs and Pd=1xs=1 in any way. On the other hand, theorem 1 and corollary 1 make it arguably easier to identify whether in a given application the OLS estimand will be close to any of the parameters of interest (cf. corollaries 2 to 4). In particular, Angrist (1998) does not recover a pattern of weight reversal, which is discussed in detail in this paper.

Unlike Angrist (1998), Humphreys (2009) does not derive a new representation of τn, instead presenting further analysis of the result in equation (12). In particular, Humphreys (2009) notes that τn can take any value between min(τs) and max(τs). Then he demonstrates that τn is also bounded by τATT and τATU if we restrict the relationship between τs and Pd=1xs=1 to be monotonic. According to corollary 1, τ is a convex combination of τATT and τATU if, among other things, both potential outcomes are linear in pX, which also implies a linear relationship between τs and Pd=1xs=1 when the model for y is saturated. Of course, this linearity assumption is stronger than the monotonicity assumption in Humphreys (2009). However, in return, we are able to derive a closed-form expression for τ in terms of τATT and τATU, a major advantage over the earlier literature, such as Angrist (1998) and Humphreys (2009).12

This section discusses two empirical illustrations of theorem 1 and its corollaries.13 In online appendixes C and D, I discuss the implementation of these results in Stata and R. Throughout this section, τAPLE, τAPLE,1, and τAPLE,0 are implicitly treated as equivalent to τATE, τATT, and τATU, respectively. Although this might be restrictive, I also demonstrate that in both applications sample analogs of τAPLE, τAPLE,1, and τAPLE,0, reported in the body of the paper, are similar to other estimates of τATE, τATT, and τATU, reported in online appendix E.

A. The Effects of a Training Program on Earnings

I first consider the example from section IIA in more detail. This replication of the study of the effects of NSW program in Angrist and Pischke (2009) constitutes an optimistic scenario for OLS. In this application, as I explained in section IIA, the effect for the treated group (ATT) is likely to be substantially larger than the effect for the CPS comparison group (ATU). Moreover, since the experimental benchmark of $1,794 corresponds to ATT^ and not to ATU^, the researcher should also focus on ATT. It turns out that my diagnostic for estimating ATT, w^0, indicates that this parameter should approximately be recovered by OLS, even if treatment effects are heterogeneous.14

The top and middle panels of table 1 reproduce the estimates from Angrist and Pischke (2009) and report my diagnostics. The specification in column 4 was discussed in section IIA. It turns out that w^0 is between 0.1% and 1.9% for all specifications; similarly, the rule-of-thumb value of this diagnostic, w^0*, is, as always, equal to the proportion of treated units (only 1.1% in this sample). These results are very simple to interpret. As in section IID, we estimate that the difference between the OLS estimand and ATT is less than 2% of the difference between ATU and ATT. In this case, it might indeed be sensible to rely on the OLS estimates of the effect of treatment.

Table 1.

The Effects of a Training Program on Earnings

(1)(2)(3)(4)
 Original estimates 
OLS −3,437*** −78 623 794 
 (612) (596) (610) (619) 
 Diagnostics 
w^0 0.019 0.001 0.017 0.017 
w^0*=ρ^ 0.011 0.011 0.011 0.011 
δ^ −0.970 −0.987 −0.971 −0.971 
δ^*=2ρ^-1 −0.977 −0.977 −0.977 −0.977 
 Decomposition 
ATT^ −3,373*** −69 754 928 
 (620) (595) (619) (630) 
w^1 0.981 0.999 0.983 0.983 
ATU^ −6,753*** −6,289** −6,841*** −6,840*** 
 (1,219) (2,807) (1,294) (1,319) 
w^0 0.019 0.001 0.017 0.017 
ATE^ −6,714*** −6,218** −6,754*** −6,751*** 
 (1,206) (2,777) (1,281) (1,305) 
Demographic controls ✓  ✓ ✓ 
Earnings in 1974    ✓ 
Earnings in 1975  ✓ ✓ ✓ 
ρ^=P^d=1 0.011 0.011 0.011 0.011 
Observations 16,177 16,177 16,177 16,177 
(1)(2)(3)(4)
 Original estimates 
OLS −3,437*** −78 623 794 
 (612) (596) (610) (619) 
 Diagnostics 
w^0 0.019 0.001 0.017 0.017 
w^0*=ρ^ 0.011 0.011 0.011 0.011 
δ^ −0.970 −0.987 −0.971 −0.971 
δ^*=2ρ^-1 −0.977 −0.977 −0.977 −0.977 
 Decomposition 
ATT^ −3,373*** −69 754 928 
 (620) (595) (619) (630) 
w^1 0.981 0.999 0.983 0.983 
ATU^ −6,753*** −6,289** −6,841*** −6,840*** 
 (1,219) (2,807) (1,294) (1,319) 
w^0 0.019 0.001 0.017 0.017 
ATE^ −6,714*** −6,218** −6,754*** −6,751*** 
 (1,206) (2,777) (1,281) (1,305) 
Demographic controls ✓  ✓ ✓ 
Earnings in 1974    ✓ 
Earnings in 1975  ✓ ✓ ✓ 
ρ^=P^d=1 0.011 0.011 0.011 0.011 
Observations 16,177 16,177 16,177 16,177 

The estimates in the top panel correspond to column 2 in table 3.3.3 in Angrist and Pischke (2009, p. 89). The dependent variable is earnings in 1978. Demographic controls include age, age squared, years of schooling, and indicators for married, high school dropout, Black, and Hispanic. For treated individuals, earnings in 1974 correspond to real earnings in months 13 to 24 prior to randomization, which overlaps with calendar year 1974 for a number of individuals. Formulas for w0, w1, and δ are given in theorem 1 and corollary 2. Following these results, OLS=w^1×ATT^+w^0×ATU^. Estimates of ATE, ATT, and ATU are sample analogs of τAPLE, τAPLE,1, and τAPLE,0, respectively. Also, ATE^=ρ^×ATT^+1-ρ^×ATU^. Huber–White standard errors (OLS) and bootstrap standard errors (ATE^, ATT^, and ATU^) are in parentheses. Statistically significant at *10%, **5%, and ***1%.

The bottom panel of table 1 provides an application of corollary 1 to these results. In other words, the estimates from Angrist and Pischke (2009) are now decomposed into two components, ATT^ and ATU^. The difference between these estimates is substantial. In column 4, while the estimate of ATT is $928, ATU is estimated to be −$6,840. In other words, the OLS estimate of $794, reported in Angrist and Pischke (2009) and discussed in section IIA, is actually a weighted average of these two estimates. The fact that it is close to $928, and not to −$6,840, is a consequence of the small proportion of treated units in this sample, 1.1%. The weight on $928, w^1, is 98.3%, and the weight on −$6,840, w^0, is only 1.7%.

We might expect that if the proportion of treated units was larger, the weight on ATT^ would be smaller and the performance of OLS in replicating the experimental benchmark would deteriorate. I confirm this conjecture in online appendix E1 by quasi-discarding random subsamples of untreated units over a range of sample sizes. In particular, I reestimate the model in equation (1) using WLS, with weights of 1 for treated and 1k for untreated units. Figures E1.1 to E1.4 show that in this application WLS estimates become more negative as k increases. This is because larger values of k correspond to greater proportions of untreated units being “discarded,” and hence larger weights on ATU^, which is substantially more negative than ATT^.

Additional extensions of my analysis are also presented in online appendix E1. For each specification in table 1, I provide both a linear and a nonparametric estimate of the conditional mean of the outcome given pX, separately for treated and untreated units (figures E1.5 to E1.8). A visual comparison of both estimates provides an informal test of assumption 4, which is necessary for a causal interpretation of τAPLE, τAPLE,1, and τAPLE,0. The linearity assumption appears to be approximately satisfied for the treated but usually not for the untreated units.

Thus, as a robustness check, I also report a number of alternative estimates of the effects of NSW program in table E1.1. I consider regression adjustment, as in section IIA, as well as matching on pX and on the logit propensity score.15 In each case, I separately estimate ATE, ATT, and ATU. These estimates are consistent with the claim that the general pattern of results in table 1 is driven by the OLS weights. The estimates of ATE and ATU are always negative and large in magnitude; the estimates of ATT are much closer to the experimental benchmark.

Finally, I repeat the following exercise from section IIA. When we match the OLS estimates in table 1 with the corresponding estimates of ATT and ATU in table E1.1, we can write τ^=w^ATT×τ^ATT+1-w^ATT×τ^ATU. Unless τ^ATT and τ^ATU are sample analogs of τAPLE,1 and τAPLE,0, w^ATT does not need to be bounded between 0 and 1. Yet we can solve for w^ATT for each set of estimates. The mean of w^ATT across all sets of estimates in table E1.1 is 98.3%, which is nearly identical to the sample proportion of untreated units, 98.9%. This is reassuring for my claims.

B. The Effects of Cash Transfers on Longevity

In my second application, I replicate a recent paper by Aizer et al. (2016) and study the effects of cash transfers on longevity of the children of their beneficiaries, as measured by their log age at death. In particular, Aizer et al. (2016) analyze the administrative records of applicants to the Mothers' Pension (MP) program, which supported poor mothers with dependent children in pre--World War II United States. In this study, the untreated group consists only of children of mothers who applied for a transfer and were initially deemed eligible but were ultimately rejected. This strategy is used to ensure that treated and untreated individuals are broadly comparable, and hence an ignorability assumption might be plausible. Nevertheless, rejected mothers were slightly older and came from slightly smaller and richer families than accepted mothers. Thus, as before, there is no reason to believe that ATT and ATU are equal, although it is perhaps less clear a priori which is larger. Unlike in section III-A, it seems plausible that the researcher might be interested in either the average effect of cash transfers, ATE, or in their average effect for accepted applicants, ATT.

The top and middle panels of table 2 reproduce the baseline estimates from Aizer et al. (2016) and report my diagnostics. While the OLS estimates are positive and statistically significant, my diagnostics indicate that these results should be approached with caution. Namely, treated units constitute the vast majority (or 87.5%) of the sample. It follows that OLS is expected to place a disproportionately large weight on ATU^, in which case the OLS estimates might be very biased for both ATE and ATT (see corollaries 2 and 3). Indeed, my estimates of δ suggest that the difference between the OLS estimand and ATE is equal to 65.9% to 74.5% of the difference between ATU and ATT. Also, the estimates of w0 suggest that the difference between OLS and ATT corresponds to 78.4% to 87.0% of this measure of heterogeneity. The estimates of δ* and w0* are similar. It turns out that in this application the OLS estimates might be substantially biased for both of our parameters of interest. This would be a pessimistic scenario for OLS.

Table 2.

The Effects of Cash Transfers on Longevity

(1)(2)(3)(4)
 Original estimates 
OLS 0.0157*** 0.0158*** 0.0182*** 0.0167*** 
 (0.0058) (0.0059) (0.0062) (0.0061) 
 Diagnostics 
w^0 0.861 0.870 0.784 0.784 
w^0*=ρ^ 0.875 0.875 0.875 0.875 
δ^ 0.736 0.745 0.659 0.659 
δ^*=2ρ^-1 0.750 0.750 0.750 0.750 
 Decomposition 
ATT^ 0.0129** 0.0149** 0.0097 0.0089 
 (0.0064) (0.0071) (0.0078) (0.0079) 
w^1 0.139 0.130 0.216 0.216 
ATU^ 0.0162*** 0.0160*** 0.0206*** 0.0188*** 
 (0.0057) (0.0059) (0.0063) (0.0064) 
w^0 0.861 0.870 0.784 0.784 
ATE^ 0.0133** 0.0150** 0.0110 0.0102 
 (0.0063) (0.0068) (0.0073) (0.0074) 
State fixed effects ✓    
County fixed effects   ✓ ✓ 
Cohort fixed effects ✓ ✓ ✓ ✓ 
State characteristics  ✓ ✓ ✓ 
County characteristics  ✓   
Individual characteristics  ✓ ✓ ✓ 
ρ^=P^d=1 0.875 0.875 0.875 0.875 
Observations 7,860 7,859 7,859 7,857 
(1)(2)(3)(4)
 Original estimates 
OLS 0.0157*** 0.0158*** 0.0182*** 0.0167*** 
 (0.0058) (0.0059) (0.0062) (0.0061) 
 Diagnostics 
w^0 0.861 0.870 0.784 0.784 
w^0*=ρ^ 0.875 0.875 0.875 0.875 
δ^ 0.736 0.745 0.659 0.659 
δ^*=2ρ^-1 0.750 0.750 0.750 0.750 
 Decomposition 
ATT^ 0.0129** 0.0149** 0.0097 0.0089 
 (0.0064) (0.0071) (0.0078) (0.0079) 
w^1 0.139 0.130 0.216 0.216 
ATU^ 0.0162*** 0.0160*** 0.0206*** 0.0188*** 
 (0.0057) (0.0059) (0.0063) (0.0064) 
w^0 0.861 0.870 0.784 0.784 
ATE^ 0.0133** 0.0150** 0.0110 0.0102 
 (0.0063) (0.0068) (0.0073) (0.0074) 
State fixed effects ✓    
County fixed effects   ✓ ✓ 
Cohort fixed effects ✓ ✓ ✓ ✓ 
State characteristics  ✓ ✓ ✓ 
County characteristics  ✓   
Individual characteristics  ✓ ✓ ✓ 
ρ^=P^d=1 0.875 0.875 0.875 0.875 
Observations 7,860 7,859 7,859 7,857 

The estimates in the top panel correspond to columns 1 to 4 in panel A of table 4 in Aizer et al. (2016, p. 952). The dependent variable is log age at death, as reported in the MP records (columns 1 to 3) or on the death certificate (column 4). State, county, and individual characteristics are listed in table E2.1 in online appendix E2. Formulas for w0, w1, and δ are given in theorem 1 and corollary 2. Following these results, OLS=w^1×ATT^+w^0×ATU^. Estimates of ATE, ATT, and ATU are sample analogs of τAPLE, τAPLE,1, and τAPLE,0, respectively. Also, ATE^=ρ^×ATT^+1-ρ^×ATU^. Huber–White standard errors (OLS) and bootstrap standard errors (ATE^, ATT^, and ATU^) are in parentheses. Statistically significant at *10%, **5%, and ***1%.

The results in the bottom panel of table 2 suggest that these biases are indeed substantial. In this panel, following corollary 1, each OLS estimate from Aizer et al. (2016) is represented as a weighted average of estimates of two effects, on accepted (ATT) and rejected (ATU) applicants. The estimates of ATU are consistently larger than those of ATT. Thus, OLS overestimates both ATE (since δ^>0) and ATT. While the implicit OLS estimates of these parameters remain statistically significant in columns 1 and 2, this is no longer the case in columns 3 and 4, following the inclusion of county fixed effects. Perhaps more importantly, these estimates of ATT are half smaller than the corresponding OLS estimates. Clearly, this difference is economically quite meaningful.

To assess the robustness of these findings, I present several extensions of my analysis in online appendix E2. The informal test of assumption 4, as discussed in section IIIA, appears to suggest that the conditional mean of the outcome given pX is approximately linear for both the treated and untreated units (see figures E2.5 to E2.8). I also report a number of alternative estimates of the effects of cash transfers in table E2.1. These additional results support my conclusion. Only one in twelve estimates of ATT is statistically different from 0, and four of the insignificant estimates are negative. While it is possible that cash transfers increase longevity, the OLS estimates reported in Aizer et al. (2016) are almost certainly too large. Interestingly, this bias appears to be driven by the implicit OLS weights on ATT and ATU, the focus of this paper.16

This paper proposes a new interpretation of the OLS estimand for the effect of a binary treatment in the standard linear model with additive effects. According to the main result of this paper, the OLS estimand is a convex combination of two parameters, which under certain conditions are equivalent to the average treatment effects on the treated (ATT) and untreated (ATU). Surprisingly, the weights on these parameters are inversely related to the proportion of observations in each group, which can lead to substantial biases when interpreting the OLS estimand as ATE or ATT.

One lesson from this result is that it might be preferable, as suggested by a body of work in econometrics, to use any of the standard estimators of average treatment effects under ignorability, such as regression adjustment, weighting, matching, and various combinations of these approaches. Empirical researchers with a preference for OLS might instead want to use the diagnostic tools that this paper also provides. These diagnostics, which are implemented in the hettreatreg package in R and Stata, are applicable whenever the researcher is studying the effects of a binary treatment, using OLS, and unwilling to maintain that ATT is exactly equal to ATU. In an important special case, these diagnostics require only the knowledge of the proportion of treated units.

1

This sample is restricted to studies that Card et al. (2018) coded as “selection on observables” and “regression.”

2

Subsequent to LaLonde (1986), these data were studied by Dehejia and Wahba (1999), Smith and Todd (2005), and many others. Angrist and Pischke (2009) analyze the subsample of the experimental treated units constructed by Dehejia and Wahba (1999), combined with CPS-1 or CPS-3, two of the nonexperimental comparison groups from CPS, constructed by LaLonde (1986). In this replication, I focus on CPS-1.

3

Both assumptions are generally innocuous, although assumption 2 rules out a small number of interesting applications, such as regression adjustments in Bernoulli trials and completely randomized experiments. In these cases, however, OLS is consistent for the average treatment effect under general conditions (Imbens & Rubin, 2015).

4

Moreover, τAPLE is similar to the “average regression coefficient” or “average slope coefficient” in Graham and Pinto (2022), which is also a descriptive estimand in the sense of Abadie et al. (2020).

5

A formal proof that the relationship between ρ and w1 (w0) is indeed always negative (positive) is provided in online appendix B1. This proof additionally assumes that the conditional mean of d is linear in X.

6

This proof uses a result from Deaton (1997) and Solon, Haider, and Wooldridge (2015) as a lemma. The main proof of theorem 1 uses a result on decomposition methods from Elder, Goddeeris, and Haider (2010). See online appendix A for more details.

7

Note that the (infeasible) linear projection of the missing outcome, ym, on d and X would solve our problem of weight reversal. The weights on τATT and τATU would still be different from ρ and 1-ρ if VpXd=1 and VpXd=0 were different; but at least the weight on τATT (τATU) would be increasing (decreasing) in ρ.

8

Because bias from nonlinearity arises when assumptions 3 and/or 4 are violated, it might be more accurate to refer to this component as “bias from endogeneity and nonlinearity.” I use the former term for brevity.

9

To be precise, |δ| can be interpreted as the percentage of sgn(δ)×τATU-τATT that contributes to bias when focusing on τATE. Both δ and w0 also have an intuitive interpretation as the difference between the weight that we should place on τATT when focusing on τATE or τATT and the weight that OLS actually places on this parameter. Indeed, δ is equal to the difference between ρ and w1. Similarly, w0=1-w1.

10

For recent reviews, see Imbens and Wooldridge (2009), Wooldridge (2010), and Abadie and Cattaneo (2018).

11

Also, note that Aronow and Samii (2016) show that this result in Angrist (1998) is not specific to saturated models; instead, it is sufficient to assume that the model for d is linear in X. My analysis in online appendix B4 covers the results in both Angrist (1998) and Aronow and Samii (2016).

12

Humphreys (2009) also provides a brief informal remark that the OLS estimand, as represented in Angrist (1998), is similar to τATT (τATU) if propensity scores are small (large) in every stratum. This is a special case of the rule of thumb derived from corollaries 3 and 4. My rule of thumb does not impose any such restrictions on the propensity score other than the requirement that the unconditional probability of treatment is close to 0 or 1.

13

In a follow-up paper, I apply these results in the study of racial gaps in test scores and wages (Słoczyński, 2020).

14

It is well known that in the NSW–CPS data, there is limited overlap in terms of covariate values between the treated and untreated units (Dehejia & Wahba, 1999; Smith & Todd, 2005). Thus, it is important to note that my theoretical results in section II do not impose the overlap assumption.

15

In particular, the estimates discussed in section IIA are reported in column 4 of the bottom panel of table E1.1.

16

I also repeat two further exercises from section IIIA. First, after I reestimate the model in equation (1) using WLS, with weights of 1 for treated and 1k for untreated units, I demonstrate in figures E2.1 to E2.4 that these estimates become more positive as k increases. As before, larger values of k translate into larger weights on ATU^, which is now greater than ATT^. Second, when I use the estimates of ATT and ATU in table E2.1 to recover the hypothetical OLS weights, I obtain 22.8% as the mean of w^ATT. This is reasonably similar to the proportion of untreated units, 12.5%.

Abadie
,
Alberto
,
Susan
Athey
,
Guido W.
Imbens
, and
Jeffrey M.
Wooldridge
, “
Sampling-Based versus Design-Based Uncertainty in Regression Analysis,
Econometrica
88
(
2020
),
265
296
.
Abadie
,
Alberto
, and
Matias D.
Cattaneo
, “
Econometric Methods for Program Evaluation,
Annual Review of Economics
10
(
2018
),
465
503
.
Aizer
,
Anna
,
Shari
Eli
,
Joseph
Ferrie
, and
Adriana
Lleras-Muney
, “
The Long-Run Impact of Cash Transfers to Poor Families,
American Economic Review
106
(
2016
),
935
971
.
Alesina
,
Alberto
,
Paola
Giuliano
, and
Nathan
Nunn
, “
On the Origins of Gender Roles: Women and the Plough,
Quarterly Journal of Economics
128
(
2013
),
469
530
.
Angrist
,
Joshua D.
, “
Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants,
Econometrica
66
(
1998
),
249
288
.
Angrist
,
Joshua D.
, and
Jörn-Steffen
Pischke
,
Mostly Harmless Econometrics: An Empiricist's Companion
.
Princeton, NJ
:
Princeton University Press
(
2009
).
Aronow
,
Peter M.
, and
Cyrus
Samii
, “
Does Regression Produce Representative Estimates of Causal Effects?
American Journal of Political Science
60
(
2016
),
250
267
.
Bitler
,
Marianne P.
,
Jonah B.
Gelbach
, and
Hilary W.
Hoynes
, “
What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments,
American Economic Review
96
(
2006
),
988
1012
.
Card
,
David
,
Jochen
Kluve
, and
Andrea
Weber
, “
What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations,
Journal of the European Economic Association
16
(
2018
),
894
931
.
Deaton
,
Angus
,
The Analysis of Household Surveys: A Microeconometric Approach to Development Policy
(
Baltimore, MD
:
Johns Hopkins University Press
,
1997
).
Dehejia
,
Rajeev H.
, and
Sadek
Wahba
, “
Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs,
Journal of the American Statistical Association
94
(
1999
),
1053
1062
.
Elder
,
Todd E.
,
John H.
Goddeeris
, and
Steven J.
Haider
, “
Unexplained Gaps and Oaxaca–Blinder Decompositions,
Labour Economics
17
(
2010
),
284
290
.
Graham
,
Bryan S.
, and
Cristine Campos de Xavier
Pinto
, “
Semiparametrically Efficient Estimation of the Average Linear Regression Function
,”
Journal of Econometrics
226
(
2022
),
115
138
.
Heckman
,
James J.
, “
Micro Data, Heterogeneity, and the Evaluation of Public Policy: Nobel Lecture,
Journal of Political Economy
109
(
2001
),
673
748
.
Humphreys
,
Macartan
, “
Bounds on Least Squares Estimates of Causal Effects in the Presence of Heterogeneous Assignment Probabilities
,” unpublished paper (
2009
).
Imbens
,
Guido W.
, “
Matching Methods in Practice: Three Examples,
Journal of Human Resources
50
(
2015
),
373
419
.
Imbens
,
Guido W.
, and
Donald B.
Rubin
,
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
(
New York
:
Cambridge University Press
,
2015
).
Imbens
,
Guido W.
, and
Jeffrey M.
Wooldridge
, “
Recent Developments in the Econometrics of Program Evaluation,
Journal of Economic Literature
47
(
2009
),
5
86
.
Kline
,
Patrick
, “
Oaxaca–Blinder as a Reweighting Estimator,
American Economic Review: Papers and Proceedings
101
(
2011
),
532
537
.
LaLonde
,
Robert J.
, “
Evaluating the Econometric Evaluations of Training Programs with Experimental Data,
American Economic Review
76
(
1986
),
604
620
. https://www.jstor.org/stable/1806062
Rosenbaum
,
Paul R.
, and
Donald B.
Rubin
, “
The Central Role of the Propensity Score in Observational Studies for Causal Effects,
Biometrika
70
(
1983
),
41
55
.
Słoczyński
,
Tymon
, “
A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands
,”
IZA discussion paper
11866
(
2018
).
Słoczyński
,
Tymon
Average Gaps and Oaxaca–Blinder Decompositions: A Cautionary Tale about Regression Estimates of Racial Differences in Labor Market Outcomes,
Industrial and Labor Relations Review
73
(
2020
),
705
729
.
Smith
,
Jeffrey A.
, and
Petra E.
Todd
, “
Does Matching Overcome LaLonde's Critique of Nonexperimental Estimators?
Journal of Econometrics
125
(
2005
),
305
353
.
Solon
,
Gary
,
Steven J.
Haider
, and
Jeffrey M.
Wooldridge
, “
What Are We Weighting For?
Journal of Human Resources
50
(
2015
),
301
316
.
Voigtländer
,
Nico
, and
Hans-Joachim
Voth
, “
Persecution Perpetuated: The Medieval Origins of Anti-Semitic Violence in Nazi Germany
,”
Quarterly Journal of Economics
127
(
2012
),
1339
1392
.
Wooldridge
,
Jeffrey M.
,
Econometric Analysis of Cross Section and Panel Data
, 2nd ed. (
Cambridge, MA
:
MIT Press
,
2010
).

Author notes

This paper is based on portions of my previous working paper (Słoczyński, 2018). I thank the editor and two anonymous referees for their helpful comments. I am very grateful to Alberto Abadie, Max Kasy, Pedro Sant'Anna, and Jeff Wooldridge for many comments and discussions. I also thank Arun Advani, Isaiah Andrews, Josh Angrist, Orley Ashenfelter, Richard Blundell, Stéphane Bonhomme, Carol Caetano, Marco Caliendo, Matias Cattaneo, Gary Chamberlain, Todd Elder, Alfonso Flores-Lagunes, Brigham Frandsen, Josh Goodman, Florian Gunsilius, Andreas Hagemann, James Heckman, Kei Hirano, Peter Hull, Macartan Humphreys, Guido Imbens, Krzysztof Karbownik, Shakeeb Khan, Toru Kitagawa, Pat Kline, Paweł Królikowski, Nicholas Longford, James MacKinnon, Łukasz Marć, Doug Miller, Michał Myck, Mateusz Myśliwski, Gary Solon, Jann Spiess, Michela Tincani, Alex Torgovitsky, Joanna Tyrowicz, Takuya Ura, and Rudolf Winter-Ebmer; seminar participants at BC, Brandeis, Harvard-MIT, Holy Cross, IHS Vienna, Lehigh, MSU, Potsdam, SDU Odense, SGH, Temple, UCL, Upjohn, and WZB Berlin; and many conference participants for useful feedback. I thank Mark McAvoy for his excellent assistance in developing the R package hettreatreg that implements the results in this paper. I also thank David Card, Jochen Kluve, and Andrea Weber for providing me with supplementary data on the articles surveyed in Card, Kluve, and Weber (2018). I acknowledge financial support from the National Science Centre (grant DEC-2012/05/N/HS4/00395), the Foundation for Polish Science (a “Start” scholarship), the ``Weź stypendium—dla rozwoju” scholarship program, and the Theodore and Jane Norman Fund.

A supplemental appendix is available online at https://doi.org/10.1162/rest_a_00953.

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. For a full description of the license, please visit https://creativecommons.org/licenses/by/4.0/legalcode

Supplementary data