- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Outstrumental Variable Regression (OVR), which identifies a linear causal effect under unobserved confounding by exploiting variation in the outcome mechanism across environments, rather than in the treatment as IV does. OVR differences environment-specific conditional means to cancel the causal term, then regresses on after partialling out these contrasts. The authors prove identification and consistency under an inclusion restriction and positivity, and evaluate the method on synthetic data and KuaiRand-Pure.
Critical strengths
-
Original idea. Using heterogeneity in the outcome mechanism across environments to identify causal effects, instead of variation in the treatment as IV does, is new and interesting. It complements existing multi-environment and IV approaches.
-
Clear presentation. The three-step procedure and the worked example (§2.2, Fig. 3) make the intuition easy to follow.
-
Honest treatment of limitations. The paper includes a bias identity and sensitivity bound for IR violations, discusses positivity failures, and supports the smoothing step with ablations.
Critical weaknesses
-
Restrictive inclusion restriction. IR requires the contrasts to fully determine the baseline nuisance. The sufficient settings in §4 rely on special structure: a common low-rank confounding factor with enough environments, an unconfounded environment, or a recoverable latent state. IR also cannot be tested from observational data, and it must hold together with positivity, which rules out injective nuisances such as the scalar linear-Gaussian case. It is unclear how often this holds in practice. Weakening IR, or giving guidance on calibrating in the sensitivity bound, would broaden applicability.
-
Connection to double ML. The estimation step is essentially Robinson's (1988) partialling-out estimator in cross-fitted DML form (Chernozhukov et al., 2018), with as the adjustment variable. Neither is cited.
-
Real-data benchmark. KuaiRand-Pure includes a randomized arm, so the benchmark mainly tests recovery of the randomized-arm slope, not adjustment for unmeasured confounding in observational data. It also reflects an assignment shift, not the outcome-mechanism shift the paper motivates. An observational dataset with outcome-mechanism variation would be more convincing.
-
Lack of baselines. OLS, FE and anchor (γ = 2) nearly coincide when , so they add little. Missing comparisons include DML with an oracle contrast (to separate identification from first-stage error), environment IV/2SLS where , per-environment OLS on KuaiRand, and the method of Kivva et al. (2025) in a scalar linear setting where it applies.
Questions for authors
What does estimate under nonlinear or heterogeneous effects? IV has LATE and its extension to variable treatment intensity (Angrist & Imbens, 1995). For a nonlinear invariant effect with scalar , and assuming IR still holds, appears to be a positively weighted average of . Can the authors characterize this estimand and its weights, including for vector ?
2 comments
W4 "Lack of baselines. OLS, FE and anchor () nearly coincide when , so they add little. Missing comparisons include DML with an oracle contrast (to separate identification from first-stage error), environment IV/2SLS where , per-environment OLS on KuaiRand, and the method of Kivva et al. (2025) in a scalar linear setting where it applies."
A: We have now added all four requested comparisons to the revised manuscript (Appendices E.4 and E.7).
DML with an oracle contrast. This oracle method replaces DOVR's estimated contrast with the population contrast , keeping the sample split and residualization fixed (Appendix E.4, new Figure 8). In settings (a) and (c), at , the oracle contrast reduced the mean coefficient error of DOVR from 0.161 to 0.038 and from 0.330 to 0.108, respectively, so most of the remaining error there comes from estimating the contrast. We did not compute in (b), where it requires numerical integration over the two-dimensional confounder conditional on in each environment, or in (d)–(f), where we would need to approximate for each environment.
Environment IV. In settings (b), (e), and (f), 2SLS had substantially higher mean coefficient error than SOVR (Appendix E.4, Figure 8). In (a) and (d), is independent of , so the population first stage is zero (making population 2SLS inapplicable). In (c), four environments provide at most three independent differences in mean treatment, which are insufficient to identify four treatment coefficients. We have now added a setting where is a valid instrument and OVR's inclusion restriction fails (Figure 9(b)), as expected, 2SLS had the lowest mean absolute error (), compared with for SOVR and for DML with an oracle contrast.
Per-environment OLS on KuaiRand. OLS on the standard-policy data has distance 0.198 to the held-out reference and OLS on the random-policy data 0.010, against 0.019 for SOVR (Table 4).
Kivva et al. (2025). Their estimator assumes a scalar treatment and a linear SCM with independent noise. Where this holds, it is the most accurate non-oracle method: in a scalar experiment we have now added (Appendix E.4, Figure 9(a); mean absolute error , followed by SOVR with ) and in the additive light-tunnel assignment (1.4% against 6.3% for SOVR; Table 2). In the binary and three-level light-tunnel assignments, which are outside its assumptions, its error is 244% and 370%, against 2.3% and 2.9% for SOVR. The six original synthetic settings (a)–(f) have four-dimensional treatments, so it cannot be used there.
Q1 "What does estimate under nonlinear or heterogeneous effects? IV has LATE and its extension to variable treatment intensity (Angrist & Imbens, 1995). For a nonlinear invariant effect with scalar , and assuming IR still holds, appears to be a positively weighted average of . Can the authors characterize this estimand and its weights, including for vector ?"
A: We thank the reviewer for this suggestion. The reviewer's intuition is correct. Let and let be a conditionally independent copy of given . Write and . Under IR, the nuisance is constant given , so the residual regression defining OVR gives
whenever positivity holds. For scalar , this is an average of the secant slopes with weights . If has a bounded continuous derivative on the treatment interval, then
w(t)=\frac{\mathbb E\left[|D|\mathbf{1}\lbrace\min(X,X')\le t<\max(X,X')\rbrace\right]} {\mathbb E[D^2]}.$$ Thus $w(t)\geq 0$ and $\int w(t)dt=1$. A treatment level $t$ receives weight from pairs that straddle it within the same contrast and environment. For vector $X$, the same argument gives matrix weights that average to the identity but can mix partial derivatives with signed coefficients, so the positive-weight interpretation need not hold coordinate by coordinate. On heterogeneous effects: Equation (1) restricts only the average response, so effects may differ between individuals; slopes that differ between environments are treated in the new Appendix H. We have now added the weight formulas and their derivation as the new Appendix G of the revised manuscript.
We thank the reviewer for highlighting the originality of OVR, the clarity of the three-step procedure and worked example, and our discussion of when the assumptions fail. We address their questions below.
W1 "Restrictive inclusion restriction. IR requires the contrasts to fully determine the baseline nuisance. The sufficient settings in §4 rely on special structure: a common low-rank confounding factor with enough environments, an unconfounded environment, or a recoverable latent state. IR also cannot be tested from observational data, and it must hold together with positivity, which rules out injective nuisances such as the scalar linear-Gaussian case. It is unclear how often this holds in practice. Weakening IR, or giving guidance on calibrating δIR in the sensitivity bound, would broaden applicability."
A: We agree that IR is a real restriction, that it cannot be verified from data, and that it excludes the scalar linear-Gaussian case (Sections 3.1 and 5). IR and positivity hold together when the confounding bias depends on X only through a coarser summary than X itself, for example when the confounder takes few values, as in the new light-tunnel experiment (Section 6.3), or when it acts through a low-dimensional summary of a multivariate treatment (Section 4.3). Positivity can be assessed from data (Appendix B.1). In the light tunnel, where a randomized reference is available, we can also check IR empirically: the nuisance is a linear function of the contrast with R2>0.99 in all three assignments (Appendix E.8, Figure 12(c)).
Exact IR is not required. OVR recovers β0 if and only if the nuisance remaining after adjustment is uncorrelated with the residual treatment (Section 3.1, "Relaxing IR"), and otherwise its bias is bounded in terms of δIR. Synthetic settings (d)–(f) and the new airport experiment (Section 6.4) do not impose IR, and OVR still has lower error than all baselines there (in (d)–(f) for n>2,000).
We have now added guidance on calibrating δIR to Section 3.1. It is measured in outcome units: the root-mean-square part of the baseline confounding bias that the contrasts do not explain. For scalar X, the bound implies that the true slope can be zero only if δIR≥∣βOVR∣E[X2]. The right-hand side can be estimated, and a practitioner can compare it with the unexplained confounding they consider plausible, as in standard sensitivity analyses.
W2 "Connection to double ML. The estimation step is essentially Robinson's (1988) partialling-out estimator in cross-fitted DML form (Chernozhukov et al., 2018), with W=(C(X),E) as the adjustment variable. Neither is cited."
A: We agree and apologize for the omission. We have now added both references in Section 5: given W=(C(X),E), the second step of OVR is Robinson's partialling-out estimator, and we use cross-fitting as in double machine learning.
W3 "Real-data benchmark. KuaiRand-Pure includes a randomized arm, so the benchmark mainly tests recovery of the randomized-arm slope, not adjustment for unmeasured confounding in observational data. It also reflects an assignment shift, not the outcome-mechanism shift the paper motivates. An observational dataset with outcome-mechanism variation would be more convincing."
A: We agree that KuaiRand contains a randomized arm and corresponds to the setting of Proposition 2; we have now clarified this in the paper (Section 6.2 and Appendix E.7). We would not describe it as only an assignment shift. The policy changes which video a user sees, and X contains only six attributes of that video. A recommender selects videos using much more than these attributes, so we expect the policy to affect watch time also through video features that are not in X.
To directly test an outcome-mechanism shift, we have now added the light-tunnel experiment: rotating a polarizer changes the hidden light source’s contribution to the outcome while keeping the joint treatment–confounder assignment fixed (Section 6.3; Appendix E.8). We have now also added an observational airport-delay benchmark with taxi-in time withheld from the estimators (Section 6.4; Appendix E.9).