- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper analyzes structural drift in the R-learner caused by nuisance estimation errors. It proposes generalized R-learner, which replaces the partially linear second-stage model with a nonlinear MLP and adds an orthogonality regularization loss. The method is evaluated on IHDP, Twins, and Jobs data.
Critical strengths
The paper studies an interesting problem where nuisance estimation errors can cause structural drift that standard R-learner cannot fully accommodate. The proposed method is intuitive and straightforward, and is supported with both theoretical analysis and experiments on common CATE estimation benchmarks.
Critical weaknesses
-
While the method includes an orthogonality penalty, it no longer retains the exact Neyman orthogonality property of the R-learner. It seems to me that the paper introduces a trade-off where the more flexible second-stage model can better account for structural drift, but it loses exact orthogonality in return.
-
The meta-learners used for benchmarking and discussed in the related work are mostly methods proposed before 2023, while several improved variants have been published in the past two years. E.g:
[1] Liang, Zhongyuan, Lars van der Laan, and Ahmed Alaa. "Hybrid Meta-learners for Estimating Heterogeneous Treatment Effects."
[2] Nagalapatti, Lokesh, et al. "PairNet: Training with observed pairs to estimate individual treatment effect."
[3] Chauhan, Vinod Kumar, et al. "Dynamic inter-treatment information sharing for individualized treatment effects estimation."
The paper would benefit from benchmarking against more recent variants to provide a more faithful comparison with current SOTA methods.
- The empirical performance gains compared to the baselines seem to be small, particularly on Twins and Jobs.
Questions for authors
Please see my comments in the Weaknesses section. It would be helpful if the authors could revise the paper to include comparisons with these more recent variants, as they have been shown to outperform the meta-learners currently used as baselines in this paper.
3 comments
Thank you to the authors for the clarifications and additional baselines. I think the revised experiments now provide a more faithful comparison with recent state-of-the-art methods.
I also agree with reviewer g8bU that the R-learner is primarily designed for CATE estimation rather than PO estimation. Similarly, other methods, such as the H-learner, may intentionally sacrifice PO estimation accuracy in certain settings to achieve better PEHE. Therefore, I do not think that poor PO estimation performance necessarily indicates a limitation of these methods, particularly when their PEHE is comparable to that of the proposed GRL.
Despite this remaining concern, I appreciate the authors' efforts in revising the paper and I think the current version represents a meaningful improvement over the original submission. I have therefore decided to increase my score to weak accept.
We thank the reviewer for finding our revision a meaningful improvement over the original submission. We would like to bring the reviewer's attention to the two additional revisions added in response to new discussion (outlined in Overall Response O1 and O2), which we hope further strengthen the paper in two aspects related to the reviewer's recent feedback: 1) revisions in O1 now more precisely delineate when and why the structural drift impacts CATE vs. PO estimation differently, and 2) the addition in O2 provides additional empirical evidence for the trade-off between PO vs. CATE estimation, which is consistent with the reviewer's observation that different methods may prioritize one over the other (and thus different methods may be beneficial depending on the use scenarios).
We thank reviewer for the review and constructive feedback.
Neyman orthogonality (G3 of overall response).
We agree that GRL does not inherit Neyman orthogonality automatically. For R-learner, orthogonality follows from its partially linear structure, but for a general nonlinear h, the pathwise derivatives in Eq. 13 need not vanish. The penalty encourages small nuisance gradients, but it can make them vanish only if the second-stage class contains an oracle-correct predictor with this property. We have added this condition to the paper (Assumption 12). Under it, Corollary 1 (Appendix C.9) shows that the minimizer of our orthogonality regularized objective (Equation 14) under oracle nuisances is oracle-correct with vanishing nuisance gradients, such that the estimating function is Neyman-orthogonal at this minimizer and the bound in Proposition 7 becomes second order in the nuisance errors. Without this condition, the reviewer's description of a trade-off is accurate: GRL may lose orthogonality and incur some bias in exchange for the flexibility to absorb structural drift that the partially linear class cannot. We now state this trade-off in the Limitations (Section 7).
Addition of more recent baselines (G4 of overall response).
We thank the reviewer for these references. We have added them to the discussion of related works (Section 2) and as baselines (Section 6, 6.1, & Tables 1-2) . We added all three suggested baselines: HyperITE [3] with TARNet backbone, H-learner [1] with X-learner's pseudo-outcome, and PairNet [2]. On synthetic data, PairNEt achieved the lowest PEHE for all values of λ, but had considerabely higher outcome MSE as its pair loss promotes positive correlation between treatment arms which cancel in CATE, but not in potential outcomes. The H-learner achieved lower PEHE than GRL at only higher values of λ (λ≥0.8), but had higher outcome MSE which also fluctuated across different values of λ. HyperITE outperformed TARNet for λ≥0.2, but remaind behind GRL in both PEHE and Outcome MSE. On real benchmarks, our earlier obserations remain unchanged: GRL had lowest PEHE on IHDP, while differences on Twins and Jobs remained small and not statistically significant across top-performing methods (PairNet achieved lowest policy risk on Jobs).
Performance on Twins and Jobs.
We agree that the differences among top-performers on Twins and Jobs are small and mostly not statistically significant, as we noted in Section 6.2 and the Limitations. We also would like to bring the reviewer's attention to the fact that there was not any method, among the extensive suite of baselines considered, that showed statistically significant improvements over GRL or the next best competitors throughout all datasets. In fact, GRL was the only method that remained among top performers on all three real datasets: top performer with statistically significant difference over the next best competitor on IHDP, and no statistically significant difference to the top performer on Twins and Jobs. In comparison, other baselines fluctuated: e.g., DragonNet is strong on Twins but among the weakest on IHDP and Jobs; the DR-learner and the newly added PairNet showed the reverse pattern. This was the consistency, rather than winning every dataset, that we emphasized in Section 6.2, which is consistent with GRL's expected robustness to nuisance estimation errors that are often unpredictable in real-world datasets.
References:
[1] Liang, Zhongyuan, Lars van der Laan, and Ahmed Alaa. "Hybrid Meta-learners for Estimating Heterogeneous Treatment Effects."
[2] Nagalapatti, Lokesh, et al. "PairNet: Training with observed pairs to estimate individual treatment effect."
[3] Chauhan, Vinod Kumar, et al. "Dynamic inter-treatment information sharing for individualized treatment effects estimation."