acceptodds
Under review as a conference paper at ICLR 2027

Efficient Inference for Distributional Treatment Effects with Partially Observed Outcomes

Abstract

We study the distributional treatment effect when a surrogate outcome is fully observed, but the primary outcome is observed only for a subsample. This setting arises naturally in model or agent evaluation, where automated judges make it inexpensive to score every high-dimensional response while human ratings are often available for only a small subsample. When evaluation scores are discrete or ordinal, a difference between cumulative distribution functions (CDFs) directly measures changes in the probability of low-quality outputs at each rating cutpoint. Building on prediction-powered inference and surrogate-assisted treatment-effect estimation, we develop a cross-fitted estimator that combines conditional CDFs with and without surrogate information. We establish a Gaussian-process limit and uniform confidence bands over outcome thresholds, identify the semiparametric efficiency bound, and characterize variance gains over estimators that reweight the labeled subsample by its known sampling probabilities or adjust for covariates without the surrogate. The construction does not require continuous output density or agreement between the surrogate and primary-outcome scales. Through simulation results and four real data applications, we demonstrate the effectiveness of our approach.

Primary area:
causal reasoning
Keywords:
causal inference, distributional treatment effects, prediction-powered inference, missing outcomes, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ciYoOpenReview ↗
Rating
4(+3)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes an estimator for the distributional treatment effect (DTE) in settings where a surrogate outcome is fully observed but the primary outcome is only partially observed under a missing-at-random (MAR) labeling mechanism. The estimator is built on doubly robust and Neyman-orthogonal (DML) principles: it combines two conditional CDFs (one using only covariates and one augmented with the surrogate) into an AIPW score, estimated via cross-fitting. Experiments on synthetic data and real-world applications demonstrate the method's effectiveness and improved precision over the chosen baselines.

Critical strengths
  1. The empirical evaluation covers four real-world applications spanning LLM evaluation, natural language, and clinical trials, which strengthens the practical relevance of the method.

  2. The theoretical results are detailed.

Critical weaknesses
  1. The role and assumptions of the surrogate outcome are not formally defined. The paper introduces the surrogate only descriptively ("a surrogate available for everyone") without specifying its structural relationship to the primary outcome or the conditions it must satisfy. It is also unclear how to assess whether the surrogate carries useful information beyond covariates. Moreover, the real-world scenarios under which both full surrogate observability and MAR are plausible are not discussed.

  2. The discussion of and experimental comparison with the treatment effect estimators designed for partially observed outcomes are absent. While the paper builds on Kallus & Mao (2025), it does not deeply discuss the broader literature on treatment effect estimation with partially observed outcomes [1-11], nor does it include experimental comparisons against relevant methods from that line of work. This gap makes it difficult to assess the relative novelty and empirical soundness of the proposed approach.

  3. The methodological novelty needs clearer positioning. The estimator combines surrogate-assisted doubly robust estimation with DML-style orthogonalization and distributional inference, which is a combination of well-established ideas. The paper should more explicitly clarify its specific contributions relative to existing doubly robust methods for partially observed outcomes (some of which also handle distributional estimands [10, 11] or even MNAR settings [6, 7, 9, 11]).


Reference

[1] Heckman, J.J.: Sample selection bias as a specification error. Econometrica: Journal of the econometric society, 153–161 (1979).

[2] Robins, J.M., Rotnitzky, A., Zhao, L.P.: Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the american statistical association 90(429), 106–121 (1995).

[3] Rotnitzky, A., Robins, J.M., Scharfstein, D.O.: Semiparametric regression for repeated outcomes with nonignorable nonresponse. Journal of the american statistical association 93(444), 1321–1339 (1998).

[4] Wooldridge, J.M.: Econometric Analysis of Cross Section and Panel Data. MIT press, Cambridge (2010).

[5] Ding, P.: Bayesian robust inference of sample selection using selection-t models. Journal of Multivariate Analysis 124, 451–464 (2014).

[6] Tchetgen Tchetgen, E.J., Wirth, K.E.: A general instrumental variable framework for regression analysis with outcome missing not at random. Biometrics 73(4),1123–1131 (2017).

[7] Sun, B., Liu, L., Miao, W., Wirth, K., Robins, J., Tchetgen, E.J.T.: Semiparametric estimation with data missing not at random using an instrumental variable. Statistica Sinica 28(4), 1965 (2018).

[8] Little, R.J., Rubin, D.B.: Statistical Analysis with Missing Data. John Wiley & Sons, Hoboken (2020).

[9] Li, W., Miao, W., Tchetgen Tchetgen, E.: Non-parametric inference about mean functionals of non-ignorable non-response data without identifying the joint distribution. Journal of the Royal Statistical Society Series B: Statistical Methodology 85(3), 913–935 (2023).

[10] Yifan Cui, Michael R Kosorok, Erik Sverdrup, Stefan Wager, and Ruoqing Zhu. 2023. Estimating heterogeneous treatment effects with right-censored data via causal survival forests. Journal of the Royal Statistical Society Series B: Statistical Methodology 85, 2 (2023), 179–211.

[11] Miao, W., Liu, L., Li, Y., Tchetgen Tchetgen, E.J., Geng, Z.: Identification and semiparametric efficiency theory of nonignorable missing data with a shadow variable. ACM/JMS Journal of Data Science 1(2), 1–23 (2024).

Questions for authors

Please see Weaknesses.

Reviewer_dfJQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the problem of treatment effect estimation when true labels YY are quite rare but we can get surrogate labels Y~\tilde{Y} relatively easily. The paper does a kind of doubly-robust estimator (sometimes called a multi-robust estimator) that (1) corrects the predictions with the surrogate label if the treatment is assigned and then (2) corrects again with the true label if the treatment is assigned and the observation is labeled. Like the doubly-robust estimator,

  • their proposed estimator is robust when either the predictions or the propensities are accurate for both
  • the error is multiplicative so, e.g., n−1/4n^{-1/4} error rate for propensity and prediction error would give an overall n−1/2n^{-1/2} error rate

Since the residuals for both terms are mean zero conditional on the covariates, treatment, (and surrogate), the paper can prove nice results on the convergence of the estimator: namely, a CLT-type result and an optimality result on the convergence rate.

The paper compares the estimator experimentally on several text quality datasets. Notably, the standard doubly robust algorithm (what they call covariate-only) achieves only slightly worse error than their estimator.

Critical strengths

Thank you for writing the introduction!! After reviewing three AI slop papers before this, I'm very grateful that you took the time to make the introduction readable. That said, it's clear AI was used in the other sections ("not" constructions, lots of notation, etc) and it would be better if you could write everything yourself.

The problem investigated is quite interesting: when you have surrogate labels, how can you use these to produce better estimates? The multi-robust estimator is quite natural and the analysis is thorough, i.e., a CLT result and the optimality from the orthogonality property.

Critical weaknesses

I'm genuinely surprised that the proposed multi-robust estimator is so similar to the doubly robust estimator since I would've thought the surrogate labels would help especially in the low ρ\rho setting on e.g., HelpSteer2 where only 2% of samples have true labels. I wonder how much of the similar performance is because of the variance is large because ρ\rho is so small. I'd be very interested to see the variance of each of the three terms for the proposed estimator empirically on each dataset. If the last term has high variance because of the small ρ\rho then I guess both the proposed estimator and standard doubly robust suffer from the same problem.

I have a novelty concern: Up to notation, I think Theorem 2.1 in Kallus and Mao 2020 [https://arxiv.org/pdf/2003.12408] gives a generalized form of the multi-robust estimator used here.

I think there's an algebra mistake in equation 37.

Questions for authors

If most of the variance comes from the true label term, could you compare to a (biased) estimator that is just the first two terms of your proposed estimator? I'm curious empirically whether this would result in lower MSE.

Reviewer_FEwbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the estimation of DTEs when the primary outcome is observed only for a subset of samples, while a cheaper surrogate outcome is available for everyone. Building on PPI, the authors propose a cross-fitted, doubly robust estimator that leverages surrogate information to improve statistical efficiency. They establish asymptotic Gaussian-process convergence, construct uniform confidence bands, and derive the semiparametric efficiency bound. Simulations and real-world applications demonstrate improved estimation precision compared with several existing baselines.

Critical strengths
  • The paper addresses a timely and important problem in LLM evaluation, where human annotations are expensive but automated judge scores are readily available.

  • The theoretical development is rigorous and well presented, with comprehensive results on asymptotic inference and semiparametric efficiency.

  • The method does not require surrogate and primary outcomes to share the same rating scale. Using conditional CDFs to incorporate judge scores is more natural and flexible than directly thresholding surrogate scores.

Critical weaknesses
  • Novelty. The methodological contribution is limited. The causal formulation is exactly the setting of Kallus & Mao (2025). The double-robustness identity, the efficiency bound, and the variance decompositions follow from standard semiparametric arguments, and the uniform bands follow from standard tools (Chernozhukov et al., 2013; Belloni et al., 2017). Distributional effects with surrogate-assisted missing outcomes have also been studied by Wang (2026).

  • The proposed method performs comparably to PPI++ on BiGGen (Table 4), while its MSE improvement over learned-CDF PPI++ in simulations is only 4–15%. These modest gains raise questions about its practical contribution.

  • Although accommodating selective MAR labeling is presented as an important contribution, the real-data experiments do not convincingly demonstrate its practical advantage over existing methods under genuinely selective labeling.

  • Some applications, particularly HelpSteer2 and wine reviews, involve observational comparisons without sufficient justification for causal identification. Consequently, these experiments primarily demonstrate improved statistical precision rather than reliable causal inference.

Questions for authors
  • Could the authors clarify which theoretical or methodological contributions cannot be obtained by directly extending Kallus & Mao (2025) to threshold indicators using standard empirical process techniques? How does the proposed framework fundamentally differ from Wang (2026) and existing PPI methods?

  • The improvements over learned-CDF PPI++ are relatively modest, and the BiGGen results show no consistent advantage. Could the authors provide more detailed comparisons and identify settings where the proposed method offers substantial improvements over PPI++?

  • Accommodating selective MAR labeling is presented as a major contribution, but its practical benefits remain insufficiently demonstrated. Could the authors provide additional experiments, preferably involving real data with surrogate-dependent labeling, to demonstrate this advantage over appropriately adjusted competing methods?

  • Several applications are primarily descriptive rather than causal. Could the authors provide a more convincing real-world causal application that demonstrates the practical value of the proposed framework for treatment effect inference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.