acceptodds
Under review as a conference paper at ICLR 2027

Variance-Efficient Adjustment for High-Dimensional Causal Inference

Abstract

Variable selection for causal inference typically prioritizes unbiasedness by identifying a valid adjustment set. In high-dimensional settings, where there are more covariates than samples, valid adjustment sets can be suboptimal in terms of treatment effect MSE, as they ignore the bias-variance trade-off. To address this, we propose VERA (Variance-Efficient Relaxed Adjustment), a variable selection component designed to wrap around any differentiable causal estimator. VERA selects variables via a differentiable gating mechanism, learning inclusion probabilities for each covariate. The selection is optimized to reduce the variance of the Efficient Influence Function, which prioritizes variables that explain more variation in the outcome than variation in the treatment. In our experiments, VERA improves the finite-sample estimation error of various base estimators in most settings, particularly in high-dimensional and small-sample regimes.

Primary area:
causal reasoning
Keywords:
high-dimensional causal inference, treatment effect estimation, variable selection, covariate adjustment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sdcKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work uses a plug-in estimate of the EIF residual variance VresV_{\mathrm{res}} as a differentiable gating objective to learn a variance-efficient (but not necessarily unbiased) covariate subset in an outer optimization loop around an arbitrary differentiable causal estimator, substantially reducing the base learner’s ATE/CATE estimation errors in high-dimensional, small-sample settings where N/pN/p is small.

Critical strengths

(1) The paper introduces the concept of a variance-efficient adjustment set with an overlap-based worst-case OVB bound.

(2) This work establishes the consistency of the gating objective as a plug-in estimator of VresV_{\mathrm{res}} under a fixed mask and provides a finite-sample decomposition.

Critical weaknesses

(1) Corollary 3.2 fails to justify VERA's central bias–variance claim. Its O(1)O(1) bound on the squared omitted-variable bias includes irreducible outcome noise unrelated to confounding and becomes vacuous as εℓ→0\varepsilon_\ell \to 0, particularly under poor overlap or in high-dimensional settings. Consequently, it cannot establish that VERA's deliberate bias–variance trade-off reduces finite-sample MSE.

(2) The paper does not discuss or compare against the two most relevant lines of work: representation learning methods based on variable disentanglement (DR-CFR and TEDVAE), and TMLE-based methods (Dragonnet and C-TMLE).

Questions for authors

(1) Similar to Weakness (1) (2)

(2) The paper includes the R-learner, which is based on Neyman-orthogonalization as a baseline. However, I would also like to see comparisons with DR-learners (such as DML) to provide a more comprehensive evaluation against doubly robust approaches.

3 comments

Thank you for your careful review. We address your concerns one by one in the following.

New OVB bound

(1) Corollary 3.2 fails to justify VERA's central bias–variance claim. Its O(1)O(1) bound on the squared omitted-variable bias includes irreducible outcome noise unrelated to confounding and becomes vacuous as ϵℓ→0\epsilon_{\ell} \rightarrow 0, particularly under poor overlap or in high-dimensional settings. Consequently, it cannot establish that VERA's deliberate bias–variance trade-off reduces finite-sample MSE.

We agree that the bound in Corollary 3.2 is loose because it bounds omitted outcome information by the full residual variance and uses a worst-case overlap constant. We replace it with a bound that uses Vres(m∗)≤Vres(mℓ)V_{\text{res}}(m^*)\leq V_{\text{res}}(m^\ell) for a valid adjustment mask mℓ⊇m∗m^\ell\supseteq m^*. The new bound is informative for weak confounders, but conservative when an omitted variable strongly predicts both treatment and outcome.

We clarify the connection to MSE in our response to Reviewer w4XD based on the example of the AIPW estimator, where we give a condition for MSE improvement of oracle AIPW (assuming true nuisance functions) based on the increase in outcome-prediction loss.

For the tighter bound, let em=P(A=1∣Xm)e_m=P(A=1 \mid X_m), αm=A/em−(1−A)/(1−em)\alpha_m=A/e_m-(1-A)/(1-e_m) and σℓ2=Var(Y∣A,Xmℓ)\sigma_\ell^2=\text{Var}(Y\mid A,X_{m^\ell}). Under positivity and finite expectations,

\mathbb{E}\left[\sigma_{\ell}^{2}(\alpha_{\ell}^{2}-\alpha_{*}^{2})\right] \mathbb{E}\left[\frac{(e_{\ell}-e_{*})^{2}}{e_{\ell}(1-e_{\ell})}\right].$$ The proof is in Appendix A.2 of the revised manuscript. The variance comparison limits the outcome information that can be lost when covariates are dropped. The first factor bounds this loss after propensity weighting, and the second measures the change in propensities. If the residual variance of the full model is constant with $\sigma^2$, the first factor is $\sigma^2\mathbb E[(\alpha_\ell-\alpha_*)^2]$. **Numerical example.** Consider equally likely treatment groups, Gaussian covariates within each group with common covariance, and a linear outcome with noise SD 1. The kept covariates have modest treatment-group imbalance (a combined standardized mean difference of $\sqrt{1/2}\approx0.71$). We vary the standardized mean difference (SMD) between the treatment groups of one additional confounder which is independent of the other covariates within each treatment group. Dropping it reduces $V_{\mathrm{res}}$ only when the share of residual outcome variance is below the shown threshold, i.e. when it does explain less than a certain amount of outcome variance. The bias is measured in outcome-noise SDs. | Confounder SMD | Variance threshold | Max absolute bias | New absolute-bias bound | |---|---|---|---| | 0.10 | 0.4% | 0.006 | 0.006 | | 0.25 | 2.2% | 0.038 | 0.039 | | 0.50 | 9.2% | 0.16 | 0.17 | | 1.00 | 38% | 0.78 | 1.04 | | 1.50 | 75% | 2.6 | 5.1 | | 2.00 | 95% | 8.7 | 32 | This bound replaces the overlap-based bound in Corollary 3.2 of the revised paper.

Baselines and base learners

(2) The paper does not discuss or compare against the two most relevant lines of work: representation learning methods based on variable disentanglement (DR-CFR and TEDVAE), and TMLE-based methods (Dragonnet and C-TMLE).

Thank you for pointing us towards these lines of work. We consider representation learning methods as complementary to variable selection, since VERA can also select among learned features. We have added DR-CFR, TEDVAE and the TMLE-based methods to the revised related-work section.

We also add Dragonnet and C-TMLE as baselines (see App. G.6), including a version (Dragonnet-TP) that uses the same initialization and training protocol as the other neural learners. VERA + TARNet has the lowest ATE RMSE among these methods at p=100p=100 and full-sample IHDP, while C-TMLE performs best in the synthetic setting with p=10p=10. The table reports ATE RMSE:

SettingRepetitionsDragonnetDragonnet-TPC-TMLEVERA TARNet
Independent, p=10p=1010000.8631.1380.6770.859
Correlated, p=10p=1010000.7601.1150.6350.764
Independent, p=100p=10010001.4341.3261.1590.738
Correlated, p=100p=10010001.4761.7131.1110.754
IHDP, N=50N=5010003.7542.3312.7262.948
IHDP, N=672N=67210000.3420.5830.7480.289

The synthetic settings use N=50N=50, and IHDP uses out-of-sample evaluation.

We also apply VERA to Dragonnet-TP. VERA lowers Dragonnet-TP's CATE error at both sample sizes and it lowers ATE RMSE at N=672N=672 but raises it at N=50N=50. The table below reports out-of-sample ATE RMSE and CATE error (mean root-PEHE) over 100 repetitions.

NNMetricDragonnetDragonnet-TPC-TMLEVERA TARNetVERA Dragonnet-TP
50ATE RMSE3.8332.4963.2682.5813.088
50Root-PEHE6.2224.8205.3124.7624.575
672ATE RMSE0.2620.5380.6850.2400.424
672Root-PEHE1.2791.4182.1520.7471.197

Dragonnet uses the initialization and optimizer schedule in Shi et al. (2019)'s reference implementation. C-TMLE uses a flexible Super Learner outcome model with regression models, random forests and gradient boosting.

(2) The paper includes the R-learner, which is based on Neyman-orthogonalization as a baseline. However, I would also like to see comparisons with DR-learners (such as DML) to provide a more comprehensive evaluation against doubly robust approaches.

Please note that DML (with XGBoost) is already included as a baseline in the submitted paper. We additionally evaluate AIPW with VERA using T-learner and TARNet outcome models and five-fold cross-fitting of the neural nuisance models in App. G.6. VERA improves the base AIPW learner in all four configurations below, but AIPW has higher RMSE than the outcome-model estimates reported in the submitted paper at this sample size.

The following table reports ATE RMSE over 200 repetitions. At N=50N=50, we compare original VERA with the ungated AIPW models with T-Learner and TARNet as outcome models respectively.

ppAIPW outcome modelNo gatingVERA
100T-learner2.1871.927
100TARNet2.0661.665
10T-learner1.5421.479
10TARNet1.5951.473

Thank you for the detailed rebuttal and additional experiments. The derivation of the new OVB bound, together with the supporting experimental results, has adequately addressed my previous concerns. I will therefore raise my score to Weak Accept. Best of luck!

Reviewer_w4XDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VERA (Variance-Efficient Relaxed Adjustment), a differentiable covariate selection framework for improving causal treatment effect estimation in high-dimensional, small-sample settings. The authors argue that conventional adjustment strategies primarily emphasize unbiasedness, while including all relevant covariates may increase estimation variance and lead to suboptimal finite-sample mean squared error (MSE).

To address this issue, the paper introduces a variance-efficient adjustment criterion based on the residual variance component of the Efficient Influence Function (EIF). VERA implements this criterion through differentiable covariate gates and an alternating optimization procedure that updates the base causal estimator and the variable selection mechanism. The framework is evaluated with S-Learner, T-Learner, R-Learner, and TARNet on synthetic, semi-synthetic, and real-world benchmarks, including IHDP, ACIC 2018, and Jobs. The results demonstrate improvements over the corresponding base estimators in many settings, particularly when the number of covariates is large relative to the sample size.

Critical strengths
  1. Well-motivated problem formulation and theoretically grounded selection criterion.

The paper addresses an important limitation of conventional covariate adjustment in high-dimensional causal inference. Instead of exclusively pursuing unbiasedness, it explicitly considers the finite-sample bias-variance trade-off. The proposed objective is derived from the residual EIF variance rather than an arbitrarily weighted combination of prediction objectives. The omitted-variable bias analysis further provides useful theoretical insight into the potential risks of relaxing valid adjustment requirements. These aspects make the proposed approach more principled than purely heuristic feature selection.

  1. Flexible integration with existing estimators and reasonably comprehensive empirical validation.

VERA is designed as a modular variable selection mechanism and is evaluated with four different neural causal estimators. The experiments cover synthetic, semi-synthetic, and real-world benchmarks, multiple sample sizes, and different covariate structures. The supplementary analyses of overlap, confounder omission, and selection stability are particularly valuable. The authors also report unfavorable results in certain settings, which helps clarify the method's limitations and applicability.

Critical weaknesses
  1. Insufficient justification for the connection between the optimization objective and the claimed finite-sample MSE improvement.

Although the proposed criterion is derived from the EIF variance decomposition, VERA optimizes only the residual variance component rather than the complete EIF variance or the actual finite-sample MSE. The objective does not explicitly account for the bias introduced by excluding important confounders. Indeed, the adversarial example in Appendix G.3 demonstrates that minimizing the proposed variance criterion can favor removing a confounder even when retaining it produces a lower MSE. Furthermore, Proposition 4.1 establishes plug-in convergence for a fixed mask but does not guarantee optimality or MSE improvement for the adaptively selected mask. Consequently, the relationship between the proposed optimization objective and the paper's ultimate estimation goal remains insufficiently justified.

  1. Limited methodological novelty and insufficient validation of the necessity of the proposed optimization design.

The main algorithmic components consist of differentiable feature gates and an alternating training procedure, both of which are established techniques. The paper does not sufficiently explain why binary feature selection is preferable to continuous feature weighting or regularized soft gating. Similarly, the necessity of alternating optimization rather than end-to-end joint training is not convincingly demonstrated. The experiments lack direct comparisons with simpler feature selection alternatives under the same base architectures, making it difficult to determine whether the improvements arise specifically from the EIF-based objective or from dimensionality reduction and regularization effects. The additional computational overhead is also not systematically evaluated. These omissions weaken the evidence for the distinct methodological contribution of VERA.

Questions for authors
  1. How does minimizing the residual EIF variance reliably improve finite-sample treatment effect MSE?

The proposed objective balances outcome uncertainty and treatment assignment through the EIF variance decomposition, but it does not explicitly optimize the squared bias introduced by omitted confounders. Appendix G.3 provides a counterexample where the variance-efficient adjustment set is not MSE-optimal. Could the authors clarify the conditions under which minimizing the proposed objective is expected to improve the actual MSE? In particular, can additional theoretical analysis or empirical evidence characterize when variance reduction outweighs the increased bias? A comparison of the proposed objective with alternative criteria that explicitly account for estimation bias would strengthen the central motivation.

  1. Why is binary covariate selection preferable to continuous feature weighting or soft gating?

VERA ultimately converts continuous gate activations into binary masks, completely removing certain covariates. However, a covariate may contribute partially to outcome prediction or confounding adjustment, and hard exclusion can lead to unnecessary information loss or estimation bias. Why not retain continuous weights in the interval [0, 1], potentially combined with appropriate regularization, to control covariate influence more flexibly? Although continuous nonzero scaling does not necessarily remove information in an unrestricted model, it may affect regularization and finite-sample learning. Could the authors compare the proposed hard-gating mechanism with continuous weighting, soft gating, and sparsity-regularized alternatives? Such comparisons would help establish whether binary selection is necessary for the observed improvements.

  1. Why is alternating optimization necessary, and what is its computational cost compared with end-to-end training?

The method repeatedly fits the outcome and propensity models, freezes their parameters, and subsequently updates the gating mechanism. This introduces additional optimization stages and potentially substantial computational overhead. Could the authors explain why the base models and gates cannot be jointly optimized using an appropriate multi-objective formulation or gradient-separation strategy? More importantly, could they provide experimental comparisons between alternating optimization and end-to-end training, including treatment effect estimation accuracy, training time, convergence behavior, and computational resources? Since VERA is presented as a broadly applicable wrapper, its additional training cost is relevant to assessing its practical value.

  1. How robust are the theoretical assumptions and optimization procedure in the actual implementation?

Proposition 4.1 assumes a fixed mask and an independent validation sample, whereas the practical algorithm adaptively updates the gates and reuses the validation split for early stopping, as acknowledged in Appendix C. Could the authors clarify how this discrepancy affects the interpretation of the theoretical guarantee? Would the conclusions remain similar under a strictly independent validation protocol or cross-fitting? Additionally, given the potential mismatch between continuous gate optimization and final binary selection, it would be useful to quantify sensitivity to the gating threshold, initialization, and optimization schedule beyond the limited analyses currently provided.

  1. Can the authors better isolate the contribution of the EIF-based selection objective and strengthen comparisons with related work?

The experiments demonstrate improvements over several base estimators, but they do not establish whether the proposed EIF-based criterion is superior to simpler feature selection strategies. Could the authors evaluate outcome-prediction-based selection, random feature selection with matched subset sizes, and conventional sparsity-regularized gating within the same neural architectures? Furthermore, the paper discusses recent research on finite-sample optimal adjustment sets, particularly Rutsch et al. (2025), but does not provide a direct empirical comparison. Even if exhaustive adjustment-set search is infeasible in high dimensions, a comparison in the low-dimensional synthetic setting would help clarify the methodological improvement over closely related approaches. These experiments could materially strengthen the evidence for the novelty and effectiveness of VERA.

3 comments

Thank you for your careful review. We address your concerns one by one in the following.

Connection between VERA objective and finite-sample MSE improvement

The specific connection between the objective and MSE improvement depends on the estimator. We clarify the connection below based on an example with oracle AIPW (assuming known nuisance functions), where the VresV_{\text{res}} objective matches the estimator’s asymptotic variance under the assumptions stated below. For oracle AIPW, the variance saving outweighs the squared additional bias if the increase in outcome-prediction loss is sufficiently small.

Using the residual variance component is a deliberate choice, as it targets conditional outcome uncertainty and propensity weighting. It excludes the treatment-effect heterogeneity component and can be used with both ATE and CATE estimators.

The criterion can drop a confounder even when retaining it gives lower MSE, as shown in Appendix G.3. A population analysis explains when this becomes a substantial concern. In the linear-outcome model with Gaussian covariates within each treatment group analyzed in our response to sdcK (“OVB bound”, numerical example), limited treatment imbalance restricts the outcome contribution of covariates which the population criterion would drop. This also limits omission bias. In this model, a substantial failure therefore requires a confounder that:

  1. is strongly imbalanced between treatment groups,
  2. has enough outcome influence to introduce substantial bias when omitted, but not enough for the population criterion to prefer keeping it, and
  3. is not known beforehand, as known confounders can be forced into the adjustment set with the constrained version of VERA.

Appendix G.3 illustrates this configuration, but its existence does not mean that it is typical in practice. A single confounder with a substantial outcome effect may also be more likely to be recognized through domain knowledge a priori. The example also shows that other selection methods, including OAL and PDS, can omit the relevant confounder.

How does minimizing the residual EIF variance reliably improve finite-sample treatment effect MSE?

For oracle AIPW, the following condition is sufficient for MSE improvement: the increase in outcome-prediction loss must be small enough relative to the reduction in propensity weighting. Let FF be the reference covariate set (e.g. the full covariate set) and S⊂FS\subset F the selected subset. For either covariate set M∈F,SM\in\\{F,S\\}, let LML_M be the population squared outcome-prediction loss under the true conditional mean. It measures outcome variation left unexplained by treatment and the covariates in MM. After covariates are dropped, it includes their unexplained outcome contribution as well as irreducible noise. The propensity-weighting factor is E[αM2]=E[1/eM+1/(1−eM)]\mathbb E[\alpha_M^2]=\mathbb E[1/e_M+1/(1-e_M)], where αM\alpha_M is the Riesz representer defined in the paper.

For oracle AIPW, the relative loss increase c=(LS−LF)/LFc=(L_S-L_F)/L_F guarantees lower MSE relative to the reference adjustment estimand τF\tau_F when

c<1n+E[αS2]/E[(αF−αS)2].c<\frac{1}{n+\mathbb E[\alpha_S^2]/\mathbb E[(\alpha_F-\alpha_S)^2]}.

This assumes LF>0L_F>0, E[(αF−αS)2]>0\mathbb E[(\alpha_F-\alpha_S)^2]>0, nn independent observations, with the covariate sets fixed in advance or selected using separate data, true nuisance functions, overlap and finite moments. For each set, conditional treatment contrasts must be constant and residual variance must be constant across covariates and treatment arms. Under these assumptions, the score variance is LME[αM2]L_M\mathbb E[\alpha_M^2], and the omitted-variable bound of Chernozhukov et al. bounds the squared additional bias by (LS−LF)E[(αF−αS)2](L_S-L_F)\mathbb E[(\alpha_F-\alpha_S)^2]. Intuitively, if the reduction in the propensity-weighting factor E[αM2]\mathbb E[\alpha_M^2] is small, little outcome loss is allowed. A larger reduction brings the threshold closer to 1/n1/n.

The result uses true nuisance functions. A condition based on fitted models would also need to account for their estimation error. The detailed proof and numerical example are in Appendix A.3 of the revised manuscript.

Reference: Chernozhukov et al., Theorem 2.

Ablations and baselines

Limited methodological novelty and insufficient validation of the necessity of the proposed optimization design.

We agree that differentiable gates and alternating optimization are established techniques. Our contribution is to use residual EIF variance for adjustment-set selection and to characterize the bias this can introduce. Our response to Reviewer CpPv compares three gate objectives within the same architecture: outcome-prediction loss, original VERA, and a new arm-averaged variant. The last estimates residual variance from mean squared outcome errors within each treatment arm and inverse propensities. They show that the performance depends on the practical approximation of the variance criterion. A variance objective that matches the variance of the final estimator and a constraint on outcome-loss increase are directions for further work.

Can the authors better isolate the contribution of the EIF-based selection objective and strengthen comparisons with related work?

VERA outperforms random subsets of the same size in every reported setting, so dimensionality reduction alone does not explain the improvements. The ablation with an outcome-prediction-based objective is reported in our response to CpPv. The outcome-loss ablation shows that much of the improvement can also be achieved without optimizing the variance criterion.

We report the ATE MAE from the random size-matched masks on the synthetic settings with N=50N=50, and on IHDP (out-of-sample error):

Setting, learnerRepetitionsAll covariatesRandom maskVERA
p=10p=10, T10000.8750.8710.665
p=10p=10, TARNet10000.8630.8910.671
p=100p=100, T10001.1571.0920.633
p=100p=100, TARNet10000.9550.9380.593
IHDP N=50N=50, T10001.9852.0411.451
IHDP N=50N=50, TARNet10001.6061.6071.340
IHDP N=672N=672, T3000.2540.3790.235
IHDP N=672N=672, TARNet3000.1780.3370.182

Rutsch et al. (2025) study MSE-optimal adjustment for OLS in linear Gaussian models. The conceptual connection is that dropping covariates can improve finite-sample MSE by trading bias for variance. Their criterion uses the bias and variance of OLS, so its optimal adjustment set is not necessarily optimal for neural learners, even in low dimensions.

Hard versus soft selection

We use binary masks as they give an explicit adjustment set and allow the final estimator to be refitted using only that set. Furthermore, hard selection makes the chosen set easy to inspect, as in OAL and PDS, whereas interpreting soft weights is less straightforward with flexible models. We agree that using the criterion to guide continuous weighting or sparsity regularization is an interesting extension.

Optimization procedure

We use alternating training to separate fitting the outcome and propensity models from learning the covariate selection. In joint training, both the models and the gates would change in response to a combined loss. A lower variance criterion could then reflect changes in the fitted predictions or propensity weights rather than a better covariate set. We agree that gradient separation could be a possible solution for joint training. However, this is beyond the scope of the paper.

As for the computational cost, VERA uses 7.8-8.2 cycles on average (maximum 15) across the synthetic and IHDP settings at N=50N=50. In a separate CPU timing check at N=50N=50 and p∈{10,100}p\in\{10,100\}, a T-learner or TARNet fit with VERA takes 0.67-0.93 seconds versus 0.11-0.12 seconds for the base learner (averaged from 20 repetitions). The overhead is 5.5-7.8 times the fit of the base model, but still less than one second per fit in these settings.

Theoretical assumptions

The fixed-mask guarantee in Proposition 4.1 does not cover adaptive selection or validation reuse. An ablation with fixed nuisance-training budgets gives similar or better results. For this, we held the cycle split fixed and replaced early stopping of the per-cycle nuisance models with fixed training budgets, so those fits do not use the gate-validation split. At N=50N=50 and p=100p=100 (1000 replications), ATE RMSE changes from 0.801 to 0.773 for the T-Learner and from 0.734 to 0.730 for TARNet. However, early stopping provides a practical default, whereas this ablation uses budgets that had to be calibrated separately for each setting and learner, which is less practical.

Sensitivity to hyperparameter choices

Appendix G.4 already analyses selection stability and the mismatch between continuous and binary gates. We have prioritized the requested objective and baseline ablations, which directly test VERA's contribution. We leave a broader tuning study to future work.

My primary concern is why binary covariate selection is preferable to continuous feature weighting or soft gating, and what advantages it offers over end-to-end training. In the current rebuttal, the authors argue that hard selection makes the candidate covariate set easier to inspect. However, I believe that approaches based on feature weights or weighting matrices are equally intuitive and straightforward to interpret.

Regarding end-to-end training, the authors argue that a lower variance criterion may reflect changes in the fitted outcome predictions or propensity weights rather than genuine improvements in the selected covariate set. They also acknowledge that end-to-end training is a feasible alternative, but consider it beyond the scope of this paper.

Reviewer_CpPvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposes VERA, a differentiable covariate-selection tools for treatment-effect estimators. It alternates fitting outcome and propensity models under a binary mask through a gating mechnism using an inverse-propensity-weighted squared-residual criterion. This criterion targets the residual component of an ATE efficient influence function variance. After getting the selected covariates, the new adjustment set can be used for any downstream causal effect estimator. The theory provides an omitted-variable-bias bound and consistency of the criterion for a fixed mask with independent validation data. In the empirical results, improvements are convincing in the main high-dimensional synthetic settings, but not consistently the best in other settings.

Critical strengths
  1. A useful and simple approach to variable selection in covaraintes for causal adjustment.
  2. Empirical results (mainly from Table 1) show the effectiveness of the method using different causal effect estimator.
  3. It works well in small n large p setting.
  4. The authors have a quite comprehensive analysis of the failure modes.
Critical weaknesses
  1. Confounding information may lie in a low-dimensional, dense projection of the covariates rather than a small subset of coordinates. VERA can not handle such situations.
  2. Existing sparse SDR methods already combine variable selection and projection-based nuisance estimation for ATE inference. But totally ignored by the paper.
  3. On small-sample IHDP, VERA improves its neural base learners but remains substantially worse than Lasso/T and BART. ATE improvements at the full sample size are modest or absent for some learners
  4. The method uses a fixed gate threshold without an explicit sparsity penalty, which makes it a bit difficult to control the remaining covariates.
  5. The variance objective does not directly penalize the confounding bias, which may still drop confounds even when retaining them yields lower MSE.
Questions for authors
  1. Many dimension reductions causal methods were ignored and not compared. E.g., sparse SDR methods already combine variable selection in covarites for ATE inference (e.g., Ma et al., 2019, Cheng et al. 2022). They should be compared to.
  2. Appendix G.3 shows that minimizing V_res can favor dropping a confounder even when retaining it yields lower MSE. How to avoid such case in practical use of the algorithm?
  3. The sample size is very small for the experiments ranging from 50 to 3k, the authors should show the results with large sample size, which is very common in modern datasets.
  4. If gates are optimized for validation outcome loss only, how the performance will change? Ablation studies are needed.
  5. VERA variable selection should be combined with AIPW, DML or TMLE, which matches the objective.
  6. How does VERA compare with C-TMLE or outcome-adaptive TMLE on your synthetic and IHDP settings?
3 comments

Thank you for your careful review. We address your concerns one by one in the following.

Comparison with dimensionality reduction methods

Thank you for pointing us towards this line of work, we have added them to the revised paper and show some preliminary comparison here and in the revised paper in App. G.6. In general, we consider projection and selection as complementary, as VERA can also select among learned features. We compare VERA with Ma et al. (2019) and Cheng et al. (2022) in the synthetic settings below. Entries are ATE RMSE at N=50N=50. VERA has lower ATE RMSE than both SDR baselines at p=100p=100.

SettingppRepetitionsMa et al.Cheng et al.VERA T.VERA TARNet
Independent1010000.8030.8380.8510.859
Correlated1010000.8750.7790.7640.764
Independent1002001.1531.3460.7830.735
Correlated1002001.1731.3310.9100.801

Competitiveness on small-sample IHDP

For this comparison, CATE is the most relevant estimand, since the IHDP benchmark focuses on heterogeneous treatment effects. VERA+TARNet has the lowest CATE error among the compared methods at the full IHDP sample size. We agree that at N=50N=50, neural learners are not competitive with Lasso/T and BART. To provide a fairer comparison, here we also evaluate VERA with a differentiable tree-based learner (“soft-tree”), where we report out-of-sample errors over 400 repetition. As can be seen in the table below, VERA improves upon this base learner and approaches BART's CATE performance.

MethodATE RMSERoot-PEHE
BART1.4473.462
Soft-tree1.6493.858
Soft-tree + VERA1.5673.551

Experiments with larger sample sizes

Using small sample sizes was a deliberate choice, because our focus is on limited samples compared to the number of covariates. Standard benchmarks have relatively few covariates, so small NN is needed to study this regime. Despite this, we did already consider larger sample sizes in the original submission in Appendix G.1 (Tables 5-6 of the submitted paper), where we vary both NN and pp. Here, VERA improves both base learners in most of the reported settings up to N=400N=400, including when NN and pp increase together.

Fixed gate threshold without explicit sparsity penalty

We think that using a fixed gate threshold instead of an explicit sparsity penalty is one of the benefits of using VERA, since its objective itself determines how many covariates are kept. We do not aim for a fixed number of retained covariates, since the ideal number is generally unknown beforehand.

Avoiding confounder omission

If we know some variables are confounders, we can use constraints to force VERA to keep them, so it selects only among the remaining variables. Limiting the loss in outcome prediction is another potential protection mechanism. We give a relative-outcome-loss condition in our response to Reviewer w4XD. However, estimating this bound reliably when p>Np>N remains an open problem.

VERA objective - ablations

Outcome-loss gating captures much of the improvement on the synthetic DGP. Since performance depends on the estimation of VresV_{\mathrm{res}}, we also add an ablation using a new arm-averaged plug-in variant, which we introduce below. Overall, no single objective among outcome-only, original VERA and the new arm-averaged variant performs best in all settings, although all of them improve the tested base learners in most settings. Matching the objective to specific learners and settings, possibly combining outcome-loss and variance objectives, is an interesting direction for future work.

For the new arm-averaged variant, we replace the observation-level squared residuals with their averages within each treatment arm:

s_a&=\frac{1}{n_a}\sum_{i:A_i=a}(Y_i-\widehat\mu_a(X_{m,i}))^2, \quad \mathcal L_{\mathrm{arm}}=\log\left[\frac{1}{n_{\mathrm{val}}}\sum_i\left(\frac{s_1}{\widehat e_i}+\frac{s_0}{1-\widehat e_i}\right)+\varepsilon\right]. \end{aligned}$$ Here, $\widehat e_i$ is the clipped propensity and $\varepsilon>0$ is a numerical stability constant. Averaging residuals within arms is intended to reduce the influence of individual noisy observations with large inverse-propensity weights. With true nuisance functions, no propensity clipping, and constant residual variance within each arm, the population quantity inside the logarithm, before adding $\varepsilon$, equals $V_{\mathrm{res}}$. Otherwise, arm averaging is an approximation. The table below shows ATE MAE at $N=50$ based on 200 replications per setting. IHDP uses out-of-sample evaluation. | Setting | Learner | Ungated | Outcome-loss | Original VERA | Arm-averaged (new) | |---|---|---|---|---|---| | Independent, $p=10$ | T | 0.866 | 0.621 | 0.654 | 0.653 | | Independent, $p=10$ | TARNet | 0.853 | 0.653 | 0.657 | 0.668 | | Independent, $p=100$ | T | 1.080 | 0.597 | 0.636 | 0.581 | | Independent, $p=100$ | TARNet | 0.910 | 0.544 | 0.580 | 0.540 | | IHDP | T | 1.998 | 1.381 | 1.432 | 1.409 | | IHDP | TARNet | 1.512 | 1.364 | 1.350 | 1.264 | At $p=100$, the new variant improves on the original VERA objective (Eq. (5)) and performs similarly to outcome-loss. At $p=10$, outcome-loss performs best. In this synthetic DGP, stronger treatment predictors have weaker outcome effects, so outcome-loss and variance-based gating can largely agree on which types of covariates to remove. On IHDP, outcome-loss gives the lowest ATE MAE for T-learner, while the arm-averaged variant gives the lowest for TARNet. All three gate objectives improve CATE estimation over the ungated learners on IHDP at $N=50$. Outcome-loss gating gives the lowest error for both learners. The table reports out-of-sample CATE error (mean root-PEHE over 200 replications): | Learner | Ungated | Outcome-loss | Original VERA | Arm-averaged (new) | |---|---|---|---|---| | T | 4.894 | 4.080 | 4.291 | 4.548 | | TARNet | 5.033 | 4.510 | 4.608 | 4.618 | **Baselines and base learners** We combine VERA with AIPW and TMLE, using the T-learner and TARNet as outcome models, and neural propensity models. We compare the three different objectives (original VERA, arm-averaged variant, outcome-loss based) to the ungated base learners. Original VERA improves on AIPW in all four configurations. At $p=100$, the arm-averaged variant performs best for the T-learner. Outcome-loss gives the lowest RMSE in the other three comparisons. In combination with TMLE, VERA improves both the T-learner and TARNet variants at $p=100$, but not at $p=10$. At $p=100$, the arm-averaged variant gives the lowest RMSE for T-learner, while outcome-loss gating performs best for TARNet. We report the comparison with C-TMLE in our response to sdcK. VERA+TARNet outperforms C-TMLE in both $p=100$ settings and on full-sample IHDP. C-TMLE performs better in both $p=10$ settings and on IHDP with $N=50$. We report the ATE RMSE over 200 repetitions at $N=50$: | $p$ | AIPW outcome model | No gating | Outcome-loss gating | Original VERA | Arm-averaged (new) | |---|---|---|---|---|---| | 100 | T-learner | 2.187 | 1.812 | 1.927 | 1.538 | | 100 | TARNet | 2.066 | 1.544 | 1.665 | 1.571 | | 10 | T-learner | 1.542 | 1.337 | 1.479 | 1.374 | | 10 | TARNet | 1.595 | 1.345 | 1.473 | 1.513 | | $p$ | TMLE outcome model | No gating | Outcome-loss | Original VERA | Arm-averaged (new) | |---|---|---|---|---|---| | 100 | T-learner | 0.981 | 0.803 | 0.822 | 0.771 | | 100 | TARNet | 0.987 | 0.771 | 0.852 | 0.816 | | 10 | T-learner | 0.773 | 0.801 | 0.825 | 0.863 | | 10 | TARNet | 0.800 | 0.791 | 0.837 | 0.811 | AIPW and TMLE are evaluated on the selected masks with five-fold cross-fitting. The final propensities are clipped to $[0.05,0.95]$.

I appreciate the authors for the detailed response. However, these additions do not change my overall assessment.

The fundamental limitation for dense low-dim projections is not solved. Adding SDR methods does not demonstrate that VERA itself can handle such settings. The synthetic settings are still based on variable selection not projection.

Another concern is that a very simple outcome-prediction objective often matches or outperforms the proposed method in the ablation study. So the EIF residual-variance criterion seems not help much.

Overall, I'm sorry that I cannot be more postive and decide to keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.