- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper presents a method for estimation of causal effects under attrition/outcome-selection when there is covariate shift between the source and target population. It introduces transport-calibrated prediction-powered inference (TC-PPI) to combine optimally two unbiased estimates of the same prediction mean, one averaged in the target sample (with missing outcomes) and another one transported from the source sample (with observed outcomes). A residual correction term computed from the source preserves an unbiased causal estimator under some misspecification pattens. Oracle weights can guarantee no-worse asymptotic variance than either of the combination points. The paper presents asymptotic properties for estimated weights, the issue of estimation drifts due to the density-ration estimation, and gives a sensitivity analysis for the case of non-transportable conditional average treatment effect (CATE).
Critical strengths
The paper presents a relative new approach for estimation of causal effects under attrition/outcome-selection when there is covariate shift between the source and target population. While very close to recent competing approaches, some modules and elements of the proposed method are innovative on their own. In particular, the sensitivity analysis extension for the case of non-transportable conditional average treatment effects (CATE) is, in my opinion, the most mold-breaking.
Critical weaknesses
Critical comments
- I understand the page limit and the need to be concise, but I feel the paper omits a substantive portion of related literature that can motivate better the setting. For instance, the paper basically deals with the problem of missing outcomes (selection/attrition problem), but fails to sit the studied setup in such context. While many missing outcome frameworks do not work under covariate shift, prior work like [1] has studied the list of independencies/invariances needed to recover a causal effect from missing outcomes. For instance, if there is a mediator causing endogenous selection into the target data, then the approach proposed in this paper would not be valid. Thus, this paper should enlist more clearly the list of independencies/invariances that are sufficient, which means expanding Assumption 2.3 into a more systematic list of conditions (and then update them for section 4.4). Even when one considers the problem of covariate shift, the paper misses some important (although very recent) references such as [2], which presents a very close PPI construction to exploit a discrepancy between target predictions and density-ratio-weighted source predictions, starting from a different baseline estimator; or [3], which applies a similar recipe with a different target and correction structure; or [4], which is relevant for the baseline one-step estimator, even though it does not apply prediction pooling with a tunable mixing parameter. Since papers specializing on PPI under covariate shift are very recent, I do not expect this paper to include them all, but a more encompassing literature review should be given, as it is very reduced now.
- The paper starts focusing on an attrition setup with covariate shift with transportable conditional average treatment effects (CATE), termed “shared causal effects”, which allows for exact transportability. Then, from section 4.4 relaxes the setting to allow shifts on the CATE, and designs and implements a sensitivity analysis. I found this idea compelling and interesting, as it does differentiate enough from competing approaches by breaking the invariance assumptions of no covariate shift and transportable CATE in [1] and of covariate shift and transportable CATE in [2-4]. But the current presentation of this idea is too rushed and compressed, and it is hard to understand without high familiarity with the topic of sensitivity analysis. It should be said carefully what parameters the analyst chooses, how they can be informed, what can and cannot be falsified by the sensitivity analysis, etc. In my opinion this is the most novel and contributing part of the paper, but Corollary 4.1 blends too much together. It think it deserves a more pedagogical presentation. Personally, I would extend it and instead reduce or move the estimation drift issue to the appendix, as that is a common problem with semiparametric estimation of transported or modified policies. Besides, the current paper does not present a novel solution for such drift problem, but an adapted combination of convergence assumptions and of cross-fitting/cross-validation under which such drift is asymptotically negligible.
Stylistic comments
- In some places the paper uses unfit terminology. For instance, the first sentence claims “ML predictions are often available for many more units than causal outcomes”. A more appropriate term, to later convey the utility of PPI, would be gold-standard/observed/labelled outcomes. The adjective causal there does not really add meaning. The same occurs when the paper uses “causal labels”. In my opinion the outcome is not causal, what it is causal is the effect of the treatment on the outcome. Another case is “population shift”. While this term fits, particularly after section 4.4, most of the paper focuses on the special case of “covariate shift”.
- The running examples are supposed to be illustrative but in the paper they are so condensed that it is hard to see what is the idea being clarified. For instance, the first running example mentions “light users” and “heavy users” but does not sit the reader in any application context: users of what?
- Section 4 is hard to follow. Instead of mentioning a rapid succession of theorems and propositions, the section should be broken into clear methodological goals. For instance, (1) finding the optimal weights in the oracle setting, and (2) examining a few asymptotic properties of an estimator using estimated weights from the oracle solution. The way section 4 is currently presented does not allow the reader to understand that there is a progression of methodological steps being advanced. A transition explanation between theorems should be given each time. Moreover, the section uses too many notation changes, which also makes it hard to follow.
References
[1] de Aguas, J., Pensar, J., Varnet Pérez, T., & Biele, G. (2025). Recovery and inference of causal effects with sequential adjustment for confounding and attrition. Journal of Causal Inference, 13(1), 20240009.
[2] Zou, R., Witten, D., & Williamson, B. (2026). Generalized prediction-powered inference, with application to binary classifier evaluation. arXiv preprint arXiv:2602.10332.
[3] Li, W., Jiang, F., & Zhang, X. (2026). Multi-Source Prediction-Powered Inference. arXiv preprint arXiv:2606.21232.
[4] Lal, A., Zheng, W., & Ejdemyr, S. (2023). A Framework for Generalization and Transportation of Causal Estimates Under Covariate Shift. arXiv preprint arXiv:2301.04776.
Questions for authors
- In a couple of spots the authors claim “averaging predictions can become nearly deterministic”. This is a strange thing to say. Under the right conditions for the Law-of-Large-Numbers (or the Central Limit Theorem) to apply, averaging predictions will converge into a deterministic point. I believe the authors meant that ML predictions can converge to a biased point relative to a causal target. Or what was meant by that?
- The density ratio can also be expressed as , where and with encoding that observation comes from the target population. I believe this odds-based expression for is more intuitive and accommodating. Why is it not presented from the start?
5 comments
Thanks for the review and here is the kind response(cont'd)
The Monte Carlo SE is at most 0.20 percentage points. The radius costs exactly in interval width. A separate bounded two-stratum check estimates both the weight and variance, using independent calibration and inference blocks of source records and target records each. Across 20,000 repetitions per design, , and either identical or shifted covariate laws, gives 97.29–97.53% worst-sign coverage. With identical covariate laws and , unexpanded coverage is 0%, versus 97.44% after expansion. This independent validation uses seed 148101313; both departure signs and all three radius values are reported. Estimated-weight sensitivity check. This tests conditional-effect sensitivity separately from observable covariate shift. Original radius sweep
Ratio uncertainty changes the weight as well as the interval. For the fitted center , the total radius is . Under the product-rectangular scalar-bias class and fixed centered noise law of Appendix A.8, the exact worst-case MSE and minimizer are
\qquad \omega_{\mathrm{rob}}^\star= \Pi_{[0,1]}\frac{B_Q/m-C/n-(\rho_c+\rho_0)\rho_r} {B_P/n+B_Q/m+\rho_r^2}.$$ Here $C$ is the source residual–prediction covariance and $B_P,B_Q$ are the two prediction variances. A variance envelope yields a risk bound when departures also change the noise law. Corollary 4.1 is the $\rho_0=0$ specialization. In Appendix A.8.6's numerical example, $(\rho_c,\rho_0,\rho_r)=(0.10,0.08,0.24)$ changes the variance-only weight 0.16383 to 0 and lowers exact rectangular worst-case risk from 0.163438 to 0.154406. Pure effect shift expands inference; ratio uncertainty also changes calibration. When both occur, the cross-term $(\rho_c+\rho_0)\rho_r$ makes the calibration rule respond to their interaction. The ratio imbalance also admits a data-driven correction. For frozen fits and weight, independent validation gives $\widehat b_1=P_v(\widehat r\widehat\tau_f)-Q_v\widehat\tau_f$. Subtracting $\omega\widehat b_1$ leaves conditional mean $\theta_Q^{\mathrm{tr}}+b_0$ and adds $\omega^2\operatorname{Var}(\widehat b_1)$ to the variance. With a joint CLT, consistent variance, and $b_0=o_p(\mathrm{SE}_{\mathrm{rec}})$, the corrected Wald interval is asymptotically valid; a bound $\rho_0$ supplies a sensitivity allowance instead. In 4,000 oracle-outcome repetitions with a saturated fitted ratio and $\omega=0.5$, recentering raises coverage from 89.15% to 95.13%, with widths 0.598 and 0.708. This provides an implemented correction alongside the sensitivity analysis. ### 3. Source identification and mediator-driven attrition (W1) > W1 · “selection/attrition problem” · “list of independencies/invariances” **Source recovery and transportability are separate conditions.** Let $J=0$ denote the source population, $J=1$ the target, and $X$ pretreatment covariates. The sufficient conditions are consistency, $(Y(0),Y(1))\perp T\mid X,J=0$, treatment overlap $0<e_P(X)<1$, target support $Q_X\ll P_X$, and shared conditional effects $\tau_Q=\tau_P$ $Q$-almost surely. Assumption 2.3's equality of both conditional potential-outcome means is stronger than needed for the ATE identity. Section 4.4 retains source identification and support, replacing shared effects by $|E_Q(\tau_Q-\tau_P)|\le\rho_c$. Independent source and target sampling underlies the two-sample variance formula. For follow-up missingness, define the target as the original cohort or a prespecified baseline subgroup, rather than treatment-specific completers. Introduce a separate observation indicator $O$ and distinguish the cohort law from its complete-case law $P_{\mathrm{cc}}$. A sufficient recovery condition is $$E[Y(t)\mid X,T=t,O=1]=E[Y(t)\mid X,T=t], \qquad P(O=1\mid X,T=t)>0,$$ together with cohort treatment exchangeability and consistency. Then the observed arm means identify the cohort's $E[Y(t)\mid X]$. The complete-case implementation uses $r_{\mathrm{cc}}=dQ_X/d(P_{\mathrm{cc}})_X$ and $$e_{\mathrm{cc}}(x)=\frac{e(x)g_1(x)}{e(x)g_1(x)+\{1-e(x)\}g_0(x)}, \qquad g_t(x)=P(O=1\mid X=x,T=t).$$ Thus treatment-dependent follow-up changes the treatment propensity among complete cases, even after randomized assignment. For the reviewer's mediator example, let $T\to M\to Y$ and $M\to O$. With $M$ recorded for the full cohort, mean-ignorable observation given $(X,T,M)$ and observation positivity permit sequential recovery: $$\mu_t(x)=E\!\left[E(Y\mid X=x,T=t,M,O=1)\mid X=x,T=t\right].$$Thanks for the review and here is the kind response(cont'd)
The outer expectation uses the treatment-specific mediator distribution, preserving the total-effect target. This is the source-recovery role of de Aguas et al. (2025). An explicit recovery-score extension replaces the source residual by
\left[\frac{O}{h_t(X,M)}\{Y-\eta_t(X,M)\}+\eta_t(X,M)-f_t(X)\right],$$ where $s_1=1,s_0=-1$, $h_t=P(O=1\mid X,T=t,M)$, and $\eta_t=E(Y\mid X,T=t,M,O=1)$. Conditional expectation gives $E_PZ_f^{\mathrm{sel}}=E_Q(\tau_P-\tau_f)$, so the same target-preserving identity applies after selection correction. An explicit check makes the distinction concrete. Set $P\{M(t)=1\}=(0.2,0.8)$, $Y=M+\varepsilon$ with independent mean-zero noise, and observation probabilities $(0.8,0.2)$ for $M=(0,1)$. The cohort ATE is 0.6, but the complete-case contrast is $0.5-1/17=0.44118$. Selection adjustment restores 0.6. In an independent 20,000-repetition check with $n=1600$, $\varepsilon\sim N(0,1)$, known recovery nuisances and estimated variance, the recovered score gives **94.90% coverage** of the cohort ATE, versus **54.10%** for the complete-case score; mean widths are 0.355 and 0.332. The complete-case interval covers its own selected-population contrast in 95.18% of repetitions. The distinction is the population being recovered: the adjusted score estimates the cohort effect, while complete-case analysis targets the selected contrast. [Protocol, all targets and actual replicate results](https://calibration-figures-q7m4.green-pearl-1123.chatgpt.site/#selection-sensitivity). The STAR analysis follows the same separation: school-clustered split diagnostics concern the observed cohort; the roster analysis varies missing-outcome means. Its lower confidence limit reaches zero at an adverse 0.134 SD per arm, and its point estimate at 0.369 SD. ### 4. Relationship to the cited estimators (W1) > W1 · “related literature” The relevant comparison is the estimand, available labels, correction score, and calibration direction: | Work | Relationship to TC-CPI | | --- | --- | | Zou et al. (2026), §3.2 | Inverse-odds baseline plus a tuned prediction discrepancy for the unlabeled population; directly related calibration architecture | | Li et al. (2026), §§2–4 | Aggregates aligned pseudo-risks from multiple sources, with labeled target data supplying residual correction; includes covariate-shift and transport-map settings | | Lal et al. (2023) | Transported augmented inverse-selection weighting; with matched nuisances and the unnormalized transportation convention, its ATE score is our external endpoint | | de Aguas et al. (2025) | Sequential recovery under confounding and attrition; supplies identification conditions when outcome observation depends on post-treatment variables | Indeed, writing $A=\widehat\theta_1$ and $\Delta_f=Q_m\widehat\tau_f-P_n(\widehat r\widehat\tau_f)$ gives $\widehat\theta_\omega=A+(1-\omega)\Delta_f$. Generalized PPI on this same baseline and direction yields the same path. Our contribution concerns what happens to that causal-transport path when effect invariance is relaxed: exact separation of structural and nuisance displacement, sensitivity coverage, and a robust weight that prices their different consequences. The matched experiments use common final predictors and weights and account for calibration labels. Independent five-fold validation gives 14.88–38.25% width savings over full-budget transported AIPW and 28.82–60.16% over the external endpoint in the 16 signal settings, with TC-CPI coverage 94.00–96.17%. The matched Zou adaptation is narrower in those settings. These comparisons establish gains over fixed endpoints; the sensitivity results above address validity when the target effect itself changes. [Zou](https://arxiv.org/html/2602.10332v1#S3.SS2), [Li](https://arxiv.org/html/2606.21232v1), [Lal](https://arxiv.org/html/2301.04776), [matched results](https://calibration-figures-q7m4.green-pearl-1123.chatgpt.site/#crossfit-budget). ### 5. Terminology, application context, and the prediction limit (W3, W4, Q1) > Q1 · “nearly deterministic” The intended statement is concentration around a potentially biased limit. For a fixed predictor with finite variance, $$Q_m\tau_f\xrightarrow{p}Q\tau_f,\qquad \operatorname{Var}(Q_m\tau_f)=\operatorname{Var}_Q(\tau_f)/m.$$ In the recommendation-platform example, $\tau_f=(0.5,3.5)$ gives $Q\tau_f=2.9$, whereas the shared-effect target is 2.6. The residual correction has expectation $-0.3$. Increasing $m$ shrinks sampling uncertainty around 2.9 without removing its bias. The variance shrinks at the usual $1/m$ rate around the prediction mean.
Thanks for the review and here is the kind response
1. What conditional-effect sensitivity adds (W2)
After balancing the covariates, how much could the treatment response itself change? Section 4.4 separates the transported source effect, θQtr=EQ[τP(X)], from the target effect, θQ=θQtr+Δcond, where Δcond=EQ[τQ(X)−τP(X)]. Under source identification, support, and correct fixed propensity and transport weights, Proposition 4.2 gives
E(\widehat\theta_\omega)-\theta_Q=-\Delta_{\mathrm{cond}}.$$ This identity holds for every fixed predictor and mixing weight: calibration changes precision around the transported effect; sensitivity analysis accounts for its displacement from the target effect. Consider a recommendation-platform trial. Frequent users comprise 20% of the pilot but 80% of the rollout population. Source effects $(1,3)$ give $\theta_Q^{\mathrm{tr}}=2.6$. If the target frequent-user effect increases by 0.5, the target ATE becomes 3.0. These two worlds have identical source observations and target covariates. Consequently, statistics based on those observations have the same distribution in both worlds: covariate balance cannot test the 0.4 effect displacement. For $|\Delta_{\mathrm{cond}}|\le\rho_c$, a confidence interval $[L_\omega,U_\omega]$ for $\theta_Q^{\mathrm{tr}}$ becomes $[L_\omega-\rho_c,U_\omega+\rho_c]$. Coverage inherits the base interval's finite-sample or asymptotic guarantee, uniformly over the allowed effect departures at the fixed observed-data law. Under the stated fixed-noise sensitivity model, pure conditional-effect shift adds the same $\rho_c^2$ to every weight's worst-case MSE. It widens the interval without changing the variance-optimal weight. Additional target covariates improve precision; target outcome evidence informs the departure bound. ### 2. Analyst inputs, decision thresholds, and empirical checks (W2) > W2 · “what parameters the analyst chooses” · “what can and cannot be falsified” **Specify the departure on the outcome scale, then report the conclusion over that range.** For a decision threshold $k$, the smallest symmetric expansion of $[L,U]$ that includes $k$ is $$\rho_{\mathrm{crit}}=\max\{0,L-k,k-U\}.$$ For example, when $L>k$, the positive conclusion survives every $\rho_c<L-k$. This threshold measures robustness to the assumed departure, rather than estimating that departure from covariates. | Input | Interpretation and source | | --- | --- | | $\rho_c$ | Bound on $\lvert \Delta_{\mathrm{cond}}\rvert$ in outcome units; informed by scientific effect-size limits or separately labeled, comparable populations | | $d_X$ | Observable covariate discrepancy using a prespecified feature map and standardization | | $s_{\mathrm{cond}}$ | Source residual scale $\operatorname{sd}_P\{\varphi-\tau_P(X)\}$; in the common-propensity randomized specialization, $\varphi=TY/\pi-(1-T)Y/(1-\pi)$ | | $\Gamma$ | Optional multiplier in $\rho_c=\Gamma d_Xs_{\mathrm{cond}}$, justified by a model linking covariate and effect shifts | | $\rho_r$ | Bound on prediction imbalance $\lvert b_1\rvert=\lvert P(\widehat r\widehat\tau_f)-Q\widehat\tau_f\rvert$; independent covariates provide an estimate and sampling allowance | | $\rho_0$ | Bound on the remaining outcome–propensity/ratio interaction $\lvert b_0\rvert$ | Here $\pi$ is the treatment probability. The standardized specification is motivated by [Jin et al. (2025)](https://arxiv.org/abs/2412.08869); its applicability is a scientific assumption. In particular, identical covariate laws can coexist with different conditional effects. Thus $d_X=0$ does not establish $\Delta_{\mathrm{cond}}=0$; the direct $\rho_c$ grid accommodates such changes. Covariates assess overlap and prediction imbalance, whereas effect transportability requires scientific assumptions or external outcome evidence. Estimated radii carry their own error probabilities: Lemma A.9 gives total error at most $\alpha_{\mathrm{base}}+\alpha_c+\alpha_0+\alpha_r$. A 40,000-repetition two-stratum sweep fixes $|\Delta_{\mathrm{cond}}|=0.4$, correct design weights, $n=100$, $m=1{,}000$, an oracle weight, and oracle variance: | Assumed $\rho_c$ | Worst-sign coverage | Width | | ---: | ---: | ---: | | 0 | 81.18% | 1.462 | | 0.2 | 92.22% | 1.862 | | 0.4 | 97.43% | 2.262 |