acceptodds
Under review as a conference paper at ICLR 2027

Causal Inference Without Trusting the Predictor: Target-Preserving Calibration Under Population Shift

Abstract

Under population shift, averaging machine-learned counterfactual predictions over a large target sample can become nearly deterministic while retaining causal bias. We study target-population average treatment effects when outcomes appear only in a labeled source sample and the target population contributes covariates alone. We introduce transport-calibrated causal prediction-powered inference (TC-CPI). With correct fixed propensity and transport weights, its mixing weight reallocates two unbiased estimates of the same prediction mean, one averaged in the target sample and the other transported from the source, while labeled residual correction preserves the causal estimand for every fixed predictor and weight. The exactly quadratic conditional variance yields a closed-form oracle weight; independent calibration attains asymptotic variance no larger than either endpoint. Population shift also creates two limits: learned density ratios give generic interior weights a first-order imbalance term that cross-fitting does not remove, while the efficiency bound weights source-outcome noise by , so unlimited target covariates cannot compensate for poor source coverage. These limits yield conditional-shift sensitivity intervals and optimal prospective label and treatment allocation. Known design weights and bounds give finite-sample concentration intervals; with learned nuisances, cross-fitted asymptotic inference is valid under an explicit drift condition. Simulations, a 500,000-row randomized advertising study, and Project STAR test the variance geometry and transport penalty.

Primary area:
causal reasoning
Keywords:
Causality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i1wuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper presents a method for estimation of causal effects under attrition/outcome-selection when there is covariate shift between the source and target population. It introduces transport-calibrated prediction-powered inference (TC-PPI) to combine optimally two unbiased estimates of the same prediction mean, one averaged in the target sample (with missing outcomes) and another one transported from the source sample (with observed outcomes). A residual correction term computed from the source preserves an unbiased causal estimator under some misspecification pattens. Oracle weights can guarantee no-worse asymptotic variance than either of the combination points. The paper presents asymptotic properties for estimated weights, the issue of estimation drifts due to the density-ration estimation, and gives a sensitivity analysis for the case of non-transportable conditional average treatment effect (CATE).

Critical strengths

The paper presents a relative new approach for estimation of causal effects under attrition/outcome-selection when there is covariate shift between the source and target population. While very close to recent competing approaches, some modules and elements of the proposed method are innovative on their own. In particular, the sensitivity analysis extension for the case of non-transportable conditional average treatment effects (CATE) is, in my opinion, the most mold-breaking.

Critical weaknesses

Critical comments

  • I understand the page limit and the need to be concise, but I feel the paper omits a substantive portion of related literature that can motivate better the setting. For instance, the paper basically deals with the problem of missing outcomes (selection/attrition problem), but fails to sit the studied setup in such context. While many missing outcome frameworks do not work under covariate shift, prior work like [1] has studied the list of independencies/invariances needed to recover a causal effect from missing outcomes. For instance, if there is a mediator T−YT-Y causing endogenous selection into the target data, then the approach proposed in this paper would not be valid. Thus, this paper should enlist more clearly the list of independencies/invariances that are sufficient, which means expanding Assumption 2.3 into a more systematic list of conditions (and then update them for section 4.4). Even when one considers the problem of covariate shift, the paper misses some important (although very recent) references such as [2], which presents a very close PPI construction to exploit a discrepancy between target predictions and density-ratio-weighted source predictions, starting from a different baseline estimator; or [3], which applies a similar recipe with a different target and correction structure; or [4], which is relevant for the baseline one-step estimator, even though it does not apply prediction pooling with a tunable mixing parameter. Since papers specializing on PPI under covariate shift are very recent, I do not expect this paper to include them all, but a more encompassing literature review should be given, as it is very reduced now.
  • The paper starts focusing on an attrition setup with covariate shift with transportable conditional average treatment effects (CATE), termed “shared causal effects”, which allows for exact transportability. Then, from section 4.4 relaxes the setting to allow shifts on the CATE, and designs and implements a sensitivity analysis. I found this idea compelling and interesting, as it does differentiate enough from competing approaches by breaking the invariance assumptions of no covariate shift and transportable CATE in [1] and of covariate shift and transportable CATE in [2-4]. But the current presentation of this idea is too rushed and compressed, and it is hard to understand without high familiarity with the topic of sensitivity analysis. It should be said carefully what parameters the analyst chooses, how they can be informed, what can and cannot be falsified by the sensitivity analysis, etc. In my opinion this is the most novel and contributing part of the paper, but Corollary 4.1 blends too much together. It think it deserves a more pedagogical presentation. Personally, I would extend it and instead reduce or move the estimation drift issue to the appendix, as that is a common problem with semiparametric estimation of transported or modified policies. Besides, the current paper does not present a novel solution for such drift problem, but an adapted combination of convergence assumptions and of cross-fitting/cross-validation under which such drift is asymptotically negligible.

Stylistic comments

  • In some places the paper uses unfit terminology. For instance, the first sentence claims “ML predictions are often available for many more units than causal outcomes”. A more appropriate term, to later convey the utility of PPI, would be gold-standard/observed/labelled outcomes. The adjective causal there does not really add meaning. The same occurs when the paper uses “causal labels”. In my opinion the outcome is not causal, what it is causal is the effect of the treatment on the outcome. Another case is “population shift”. While this term fits, particularly after section 4.4, most of the paper focuses on the special case of “covariate shift”.
  • The running examples are supposed to be illustrative but in the paper they are so condensed that it is hard to see what is the idea being clarified. For instance, the first running example mentions “light users” and “heavy users” but does not sit the reader in any application context: users of what?
  • Section 4 is hard to follow. Instead of mentioning a rapid succession of theorems and propositions, the section should be broken into clear methodological goals. For instance, (1) finding the optimal weights in the oracle setting, and (2) examining a few asymptotic properties of an estimator using estimated weights from the oracle solution. The way section 4 is currently presented does not allow the reader to understand that there is a progression of methodological steps being advanced. A transition explanation between theorems should be given each time. Moreover, the section uses too many notation changes, which also makes it hard to follow.

References

[1] de Aguas, J., Pensar, J., Varnet Pérez, T., & Biele, G. (2025). Recovery and inference of causal effects with sequential adjustment for confounding and attrition. Journal of Causal Inference, 13(1), 20240009.

[2] Zou, R., Witten, D., & Williamson, B. (2026). Generalized prediction-powered inference, with application to binary classifier evaluation. arXiv preprint arXiv:2602.10332.

[3] Li, W., Jiang, F., & Zhang, X. (2026). Multi-Source Prediction-Powered Inference. arXiv preprint arXiv:2606.21232.

[4] Lal, A., Zheng, W., & Ejdemyr, S. (2023). A Framework for Generalization and Transportation of Causal Estimates Under Covariate Shift. arXiv preprint arXiv:2301.04776.

Questions for authors
  • In a couple of spots the authors claim “averaging predictions can become nearly deterministic”. This is a strange thing to say. Under the right conditions for the Law-of-Large-Numbers (or the Central Limit Theorem) to apply, averaging predictions will converge into a deterministic point. I believe the authors meant that ML predictions can converge to a biased point relative to a causal target. Or what was meant by that?
  • The density ratio r(x)r(x) can also be expressed as q(x)/(1−q(x))×(1−q)/qq(x)/(1-q(x))\times (1-q)/q, where q(x)=P(S=1∣X=x)q(x)=\mathbb{P}(S=1\mid X=x) and q=P(S=1)q=\mathbb{P}(S=1) with Si=1S_i=1 encoding that observation ii comes from the target population. I believe this odds-based expression for rr is more intuitive and accommodating. Why is it not presented from the start?
5 comments
AuthorsOpenReview ↗

Thanks for the review and here is the kind response

1. What conditional-effect sensitivity adds (W2)

W2 · “too rushed and compressed”

After balancing the covariates, how much could the treatment response itself change? Section 4.4 separates the transported source effect, θQtr=EQ[τP(X)]\theta_Q^{\mathrm{tr}}=E_Q[\tau_P(X)], from the target effect, θQ=θQtr+Δcond\theta_Q=\theta_Q^{\mathrm{tr}}+\Delta_{\mathrm{cond}}, where Δcond=EQ[τQ(X)−τP(X)]\Delta_{\mathrm{cond}}=E_Q[\tau_Q(X)-\tau_P(X)]. Under source identification, support, and correct fixed propensity and transport weights, Proposition 4.2 gives

E(\widehat\theta_\omega)-\theta_Q=-\Delta_{\mathrm{cond}}.$$ This identity holds for every fixed predictor and mixing weight: calibration changes precision around the transported effect; sensitivity analysis accounts for its displacement from the target effect. Consider a recommendation-platform trial. Frequent users comprise 20% of the pilot but 80% of the rollout population. Source effects $(1,3)$ give $\theta_Q^{\mathrm{tr}}=2.6$. If the target frequent-user effect increases by 0.5, the target ATE becomes 3.0. These two worlds have identical source observations and target covariates. Consequently, statistics based on those observations have the same distribution in both worlds: covariate balance cannot test the 0.4 effect displacement. For $|\Delta_{\mathrm{cond}}|\le\rho_c$, a confidence interval $[L_\omega,U_\omega]$ for $\theta_Q^{\mathrm{tr}}$ becomes $[L_\omega-\rho_c,U_\omega+\rho_c]$. Coverage inherits the base interval's finite-sample or asymptotic guarantee, uniformly over the allowed effect departures at the fixed observed-data law. Under the stated fixed-noise sensitivity model, pure conditional-effect shift adds the same $\rho_c^2$ to every weight's worst-case MSE. It widens the interval without changing the variance-optimal weight. Additional target covariates improve precision; target outcome evidence informs the departure bound. ### 2. Analyst inputs, decision thresholds, and empirical checks (W2) > W2 · “what parameters the analyst chooses” · “what can and cannot be falsified” **Specify the departure on the outcome scale, then report the conclusion over that range.** For a decision threshold $k$, the smallest symmetric expansion of $[L,U]$ that includes $k$ is $$\rho_{\mathrm{crit}}=\max\{0,L-k,k-U\}.$$ For example, when $L>k$, the positive conclusion survives every $\rho_c<L-k$. This threshold measures robustness to the assumed departure, rather than estimating that departure from covariates. | Input | Interpretation and source | | --- | --- | | $\rho_c$ | Bound on $\lvert \Delta_{\mathrm{cond}}\rvert$ in outcome units; informed by scientific effect-size limits or separately labeled, comparable populations | | $d_X$ | Observable covariate discrepancy using a prespecified feature map and standardization | | $s_{\mathrm{cond}}$ | Source residual scale $\operatorname{sd}_P\{\varphi-\tau_P(X)\}$; in the common-propensity randomized specialization, $\varphi=TY/\pi-(1-T)Y/(1-\pi)$ | | $\Gamma$ | Optional multiplier in $\rho_c=\Gamma d_Xs_{\mathrm{cond}}$, justified by a model linking covariate and effect shifts | | $\rho_r$ | Bound on prediction imbalance $\lvert b_1\rvert=\lvert P(\widehat r\widehat\tau_f)-Q\widehat\tau_f\rvert$; independent covariates provide an estimate and sampling allowance | | $\rho_0$ | Bound on the remaining outcome–propensity/ratio interaction $\lvert b_0\rvert$ | Here $\pi$ is the treatment probability. The standardized specification is motivated by [Jin et al. (2025)](https://arxiv.org/abs/2412.08869); its applicability is a scientific assumption. In particular, identical covariate laws can coexist with different conditional effects. Thus $d_X=0$ does not establish $\Delta_{\mathrm{cond}}=0$; the direct $\rho_c$ grid accommodates such changes. Covariates assess overlap and prediction imbalance, whereas effect transportability requires scientific assumptions or external outcome evidence. Estimated radii carry their own error probabilities: Lemma A.9 gives total error at most $\alpha_{\mathrm{base}}+\alpha_c+\alpha_0+\alpha_r$. A 40,000-repetition two-stratum sweep fixes $|\Delta_{\mathrm{cond}}|=0.4$, correct design weights, $n=100$, $m=1{,}000$, an oracle weight, and oracle variance: | Assumed $\rho_c$ | Worst-sign coverage | Width | | ---: | ---: | ---: | | 0 | 81.18% | 1.462 | | 0.2 | 92.22% | 1.862 | | 0.4 | 97.43% | 2.262 |
AuthorsOpenReview ↗

Thanks for the review and here is the kind response(cont'd)

The Monte Carlo SE is at most 0.20 percentage points. The radius costs exactly 2ρc2\rho_c in interval width. A separate bounded two-stratum check estimates both the weight and variance, using independent calibration and inference blocks of nn source records and 10n10n target records each. Across 20,000 repetitions per design, n∈{400,1600,6400}n\in\{400,1600,6400\}, and either identical or shifted covariate laws, ρc=0.4\rho_c=0.4 gives 97.29–97.53% worst-sign coverage. With identical covariate laws and n=1600n=1600, unexpanded coverage is 0%, versus 97.44% after expansion. This independent validation uses seed 148101313; both departure signs and all three radius values are reported. Estimated-weight sensitivity check. This tests conditional-effect sensitivity separately from observable covariate shift. Original radius sweep

Ratio uncertainty changes the weight as well as the interval. For the fitted center ψω=θQtr+b0+ωb1\psi_\omega=\theta_Q^{\mathrm{tr}}+b_0+\omega b_1, the total radius is ρc+ρ0+ωρr\rho_c+\rho_0+\omega\rho_r. Under the product-rectangular scalar-bias class and fixed centered noise law of Appendix A.8, the exact worst-case MSE and minimizer are

\qquad \omega_{\mathrm{rob}}^\star= \Pi_{[0,1]}\frac{B_Q/m-C/n-(\rho_c+\rho_0)\rho_r} {B_P/n+B_Q/m+\rho_r^2}.$$ Here $C$ is the source residual–prediction covariance and $B_P,B_Q$ are the two prediction variances. A variance envelope yields a risk bound when departures also change the noise law. Corollary 4.1 is the $\rho_0=0$ specialization. In Appendix A.8.6's numerical example, $(\rho_c,\rho_0,\rho_r)=(0.10,0.08,0.24)$ changes the variance-only weight 0.16383 to 0 and lowers exact rectangular worst-case risk from 0.163438 to 0.154406. Pure effect shift expands inference; ratio uncertainty also changes calibration. When both occur, the cross-term $(\rho_c+\rho_0)\rho_r$ makes the calibration rule respond to their interaction. The ratio imbalance also admits a data-driven correction. For frozen fits and weight, independent validation gives $\widehat b_1=P_v(\widehat r\widehat\tau_f)-Q_v\widehat\tau_f$. Subtracting $\omega\widehat b_1$ leaves conditional mean $\theta_Q^{\mathrm{tr}}+b_0$ and adds $\omega^2\operatorname{Var}(\widehat b_1)$ to the variance. With a joint CLT, consistent variance, and $b_0=o_p(\mathrm{SE}_{\mathrm{rec}})$, the corrected Wald interval is asymptotically valid; a bound $\rho_0$ supplies a sensitivity allowance instead. In 4,000 oracle-outcome repetitions with a saturated fitted ratio and $\omega=0.5$, recentering raises coverage from 89.15% to 95.13%, with widths 0.598 and 0.708. This provides an implemented correction alongside the sensitivity analysis. ### 3. Source identification and mediator-driven attrition (W1) > W1 · “selection/attrition problem” · “list of independencies/invariances” **Source recovery and transportability are separate conditions.** Let $J=0$ denote the source population, $J=1$ the target, and $X$ pretreatment covariates. The sufficient conditions are consistency, $(Y(0),Y(1))\perp T\mid X,J=0$, treatment overlap $0<e_P(X)<1$, target support $Q_X\ll P_X$, and shared conditional effects $\tau_Q=\tau_P$ $Q$-almost surely. Assumption 2.3's equality of both conditional potential-outcome means is stronger than needed for the ATE identity. Section 4.4 retains source identification and support, replacing shared effects by $|E_Q(\tau_Q-\tau_P)|\le\rho_c$. Independent source and target sampling underlies the two-sample variance formula. For follow-up missingness, define the target as the original cohort or a prespecified baseline subgroup, rather than treatment-specific completers. Introduce a separate observation indicator $O$ and distinguish the cohort law from its complete-case law $P_{\mathrm{cc}}$. A sufficient recovery condition is $$E[Y(t)\mid X,T=t,O=1]=E[Y(t)\mid X,T=t], \qquad P(O=1\mid X,T=t)>0,$$ together with cohort treatment exchangeability and consistency. Then the observed arm means identify the cohort's $E[Y(t)\mid X]$. The complete-case implementation uses $r_{\mathrm{cc}}=dQ_X/d(P_{\mathrm{cc}})_X$ and $$e_{\mathrm{cc}}(x)=\frac{e(x)g_1(x)}{e(x)g_1(x)+\{1-e(x)\}g_0(x)}, \qquad g_t(x)=P(O=1\mid X=x,T=t).$$ Thus treatment-dependent follow-up changes the treatment propensity among complete cases, even after randomized assignment. For the reviewer's mediator example, let $T\to M\to Y$ and $M\to O$. With $M$ recorded for the full cohort, mean-ignorable observation given $(X,T,M)$ and observation positivity permit sequential recovery: $$\mu_t(x)=E\!\left[E(Y\mid X=x,T=t,M,O=1)\mid X=x,T=t\right].$$
AuthorsOpenReview ↗

Thanks for the review and here is the kind response(cont'd)

The outer expectation uses the treatment-specific mediator distribution, preserving the total-effect target. This is the source-recovery role of de Aguas et al. (2025). An explicit recovery-score extension replaces the source residual by

\left[\frac{O}{h_t(X,M)}\{Y-\eta_t(X,M)\}+\eta_t(X,M)-f_t(X)\right],$$ where $s_1=1,s_0=-1$, $h_t=P(O=1\mid X,T=t,M)$, and $\eta_t=E(Y\mid X,T=t,M,O=1)$. Conditional expectation gives $E_PZ_f^{\mathrm{sel}}=E_Q(\tau_P-\tau_f)$, so the same target-preserving identity applies after selection correction. An explicit check makes the distinction concrete. Set $P\{M(t)=1\}=(0.2,0.8)$, $Y=M+\varepsilon$ with independent mean-zero noise, and observation probabilities $(0.8,0.2)$ for $M=(0,1)$. The cohort ATE is 0.6, but the complete-case contrast is $0.5-1/17=0.44118$. Selection adjustment restores 0.6. In an independent 20,000-repetition check with $n=1600$, $\varepsilon\sim N(0,1)$, known recovery nuisances and estimated variance, the recovered score gives **94.90% coverage** of the cohort ATE, versus **54.10%** for the complete-case score; mean widths are 0.355 and 0.332. The complete-case interval covers its own selected-population contrast in 95.18% of repetitions. The distinction is the population being recovered: the adjusted score estimates the cohort effect, while complete-case analysis targets the selected contrast. [Protocol, all targets and actual replicate results](https://calibration-figures-q7m4.green-pearl-1123.chatgpt.site/#selection-sensitivity). The STAR analysis follows the same separation: school-clustered split diagnostics concern the observed cohort; the roster analysis varies missing-outcome means. Its lower confidence limit reaches zero at an adverse 0.134 SD per arm, and its point estimate at 0.369 SD. ### 4. Relationship to the cited estimators (W1) > W1 · “related literature” The relevant comparison is the estimand, available labels, correction score, and calibration direction: | Work | Relationship to TC-CPI | | --- | --- | | Zou et al. (2026), §3.2 | Inverse-odds baseline plus a tuned prediction discrepancy for the unlabeled population; directly related calibration architecture | | Li et al. (2026), §§2–4 | Aggregates aligned pseudo-risks from multiple sources, with labeled target data supplying residual correction; includes covariate-shift and transport-map settings | | Lal et al. (2023) | Transported augmented inverse-selection weighting; with matched nuisances and the unnormalized transportation convention, its ATE score is our external endpoint | | de Aguas et al. (2025) | Sequential recovery under confounding and attrition; supplies identification conditions when outcome observation depends on post-treatment variables | Indeed, writing $A=\widehat\theta_1$ and $\Delta_f=Q_m\widehat\tau_f-P_n(\widehat r\widehat\tau_f)$ gives $\widehat\theta_\omega=A+(1-\omega)\Delta_f$. Generalized PPI on this same baseline and direction yields the same path. Our contribution concerns what happens to that causal-transport path when effect invariance is relaxed: exact separation of structural and nuisance displacement, sensitivity coverage, and a robust weight that prices their different consequences. The matched experiments use common final predictors and weights and account for calibration labels. Independent five-fold validation gives 14.88–38.25% width savings over full-budget transported AIPW and 28.82–60.16% over the external endpoint in the 16 signal settings, with TC-CPI coverage 94.00–96.17%. The matched Zou adaptation is narrower in those settings. These comparisons establish gains over fixed endpoints; the sensitivity results above address validity when the target effect itself changes. [Zou](https://arxiv.org/html/2602.10332v1#S3.SS2), [Li](https://arxiv.org/html/2606.21232v1), [Lal](https://arxiv.org/html/2301.04776), [matched results](https://calibration-figures-q7m4.green-pearl-1123.chatgpt.site/#crossfit-budget). ### 5. Terminology, application context, and the prediction limit (W3, W4, Q1) > Q1 · “nearly deterministic” The intended statement is concentration around a potentially biased limit. For a fixed predictor with finite variance, $$Q_m\tau_f\xrightarrow{p}Q\tau_f,\qquad \operatorname{Var}(Q_m\tau_f)=\operatorname{Var}_Q(\tau_f)/m.$$ In the recommendation-platform example, $\tau_f=(0.5,3.5)$ gives $Q\tau_f=2.9$, whereas the shared-effect target is 2.6. The residual correction has expectation $-0.3$. Increasing $m$ shrinks sampling uncertainty around 2.9 without removing its bias. The variance shrinks at the usual $1/m$ rate around the prediction mean.
Reviewer_HfkhOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies target-population average treatment effect estimation when outcomes and treatments are observed in a labeled source sample while the target sample contains covariates only. It proposes transport-calibrated causal prediction-powered inference (TC-CPI), a one-parameter family that mixes target-sample and transported source-sample averages of arbitrary treatment-effect predictions while retaining a labeled residual correction. With fixed correct design weights, every point on this path targets the same effect, its conditional variance is exactly quadratic, and an independently calibrated weight can asymptotically match or improve on both endpoint estimators. The paper further characterizes first-order drift from learned transport ratios, an r2r^2-weighted efficiency limit under weak source support, sensitivity intervals for conditional shift, and prospective label and treatment allocation rules. The results are evaluated through extensive simulations, a randomized advertising dataset, and Project STAR.

Critical strengths

S1. The target-preserving interpolation is conceptually clean and technically useful. With correct fixed propensity and transport weights, an arbitrary predictor affects efficiency rather than the estimand, and the exact quadratic variance yields a closed-form oracle weight and an independently calibrated procedure that is asymptotically no worse than either endpoint. The theory, mechanism ablations, and variance-grid experiments align closely.

S2. The paper provides unusually informative negative results about causal prediction under population shift. It proves that estimating the transport ratio creates a generic first-order drift for interior weights that cross-fitting alone does not remove, and that the efficiency bound weights source-outcome noise by r2r^2, so unlimited target covariates cannot repair weak source support. The sensitivity and prospective-design results turn these limitations into explicit inferential and data-collection consequences.

Critical weaknesses

W1. Valid inference with an estimated transport ratio remains conditional on a first-order drift requirement that the proposed workflow does not generally enforce or certify. Theorem 4.5 assumes the foldwise drift is op(n−1/2)o_p(n^{-1/2}), while Proposition 4.1 shows that this condition can fail even with a correct outcome regression and cross-fitting. The sensitivity analysis is useful once a ratio-error radius is supplied, but it does not provide a data-driven correction or observable certificate for the realized drift. This limits the main method's deployability in the common setting where the source-to-target ratio is learned. The Criteo study does not close this gap because its target shift, and hence rr, is known by construction.

W2. The empirical evidence establishes the variance geometry more convincingly than a practical gain over existing alternatives. On Criteo, the adaptive variance ratios relative to transported AIPW are approximately 0.997--1.004, and on STAR the adaptive, labeled, and external split SDs are 5.93, 5.94, and 5.96. These honest near-null results show that calibration avoids losses, but neither real-data study demonstrates a consequential precision improvement. Moreover, the main comparisons are the two endpoints and an oracle rather than matched recent calibrated-PPI or trial-generalization procedures under the same information contract. STAR additionally uses student-level uncertainty despite missing classroom identifiers and attrition information, which weakens its inferential validation.

Questions for authors
  1. Assessment-changing clarification: In an applied dataset with an estimated transport ratio, what observable diagnostic or implemented balancing condition can establish that the realized first-order drift required by Theorem 4.5 is negligible? If the current artifact already contains a balance-enforcing correction or a data-derived upper bound for this drift, please report the corresponding coverage and interval-width results.

  2. Assessment-changing evidence: Do the existing experiments include a matched comparison with the closest applicable calibrated-PPI or prediction-powered trial-generalization estimators under the same source labels, target covariates, nuisance fitting, and calibration budget? Related unreported results, especially in settings with non-negligible real or semi-real precision gains, would materially affect the assessment of practical novelty.

  3. Clarification: For Project STAR, how does the absence of classroom identifiers affect the validity of the student-level method intervals, given clustered class assignment and the school-clustered benchmark uncertainty? Are school-level or otherwise cluster-robust split results available, and how sensitive are the conclusions to plausible attrition adjustments?

6 comments
AuthorsOpenReview ↗

Thanks for the review and here is the kind response

1. A practical correction for learned transport imbalance (W1, Q1)

W1 · “first-order drift requirement” · Q1 · “observable diagnostic”

For a learned transport ratio, subtract the held-out prediction imbalance and include its estimation variance in the standard error. This removes ωb1\omega b_1 from the conditional drift. Validity then depends on the nuisance-interaction term b0b_0.

Condition on all training and calibration information G\mathcal G, freezing f^,e^,r^\widehat f,\widehat e,\widehat r and ω\omega. Write U=Z^fU=\widehat Z_f for the weighted residual correction, R=r^τ^fR=\widehat r\widehat\tau_f and S=τ^fS=\widehat\tau_f. Under shared conditional effects, Appendix A.8 gives

\qquad E(\widehat\theta_\omega\mid\mathcal G)-\theta_Q=b_0+\omega b_1.$$ Validation covariates give $\widehat b_1=P_vR-Q_vS$, with conditional expectation $P[(\widehat r-r)\widehat\tau_f]=b_1$. Computing it requires only the fitted ratio. Write $B_P=\operatorname{Var}_P(R)$, $B_Q=\operatorname{Var}_Q(S)$ and $C=\operatorname{Cov}_P(U,R)$. **Recentring proposition.** For independent validation and inference blocks with finite second moments, $$\widehat\theta_{\omega,\mathrm{rec}}=\widehat\theta_\omega-\omega\widehat b_1, \quad E(\widehat\theta_{\omega,\mathrm{rec}}\mid\mathcal G)=\theta_Q+b_0,$$ $$V_{\mathrm{rec}}(\omega)=V(\omega)+\omega^2\left(B_P/n_v+B_Q/m_v\right).$$ Conditional unbiasedness of the validation moment gives the corrected expectation, and independence adds its variance. Under a joint central limit theorem, consistent variance estimation and $b_0=o_p(\sqrt{V_{\mathrm{rec}}})$, the recentered Wald interval has asymptotic 95% coverage. The proposition extends Theorem 4.5 to the corrected estimator. The term $b_0$ interacts outcome-regression error with propensity or ratio error. Exact regressions $\widehat f_t=\mu_t$ give $b_0=0$. For estimated regressions, Corollary A.1 supplies the sufficient product condition $(a_e+a_r)a_f=o_p(n^{-1/2})$ with bounded weights and score convergence. Correct specification still leaves estimation error. The added variance changes the calibration rule. Minimizing $V_{\mathrm{rec}}$ gives $$\omega^\star_{\mathrm{rec}}=\Pi_{[0,1]} \frac{B_Q/m-C/n}{B_P/n+B_Q/m+B_P/n_v+B_Q/m_v}.$$ We estimate the moments on calibration data and freeze the weight before validation. The added denominator terms reduce the source share when correcting imbalance is costly. Using inference observations to estimate imbalance gives $\widehat\theta_\omega-\omega(P_nR-Q_mS)=P_nU+Q_mS$, exactly the external endpoint. Independent validation retains a nontrivial calibrated path. Its $\omega=1$ endpoint includes the validation correction. **Radius expansion.** Adding $U_\omega=|\omega|(|\widehat b_1|+z_{0.9875}\widehat s_b)$ to a 97.5% base interval allocates 2.5% error each to sampling uncertainty and the imbalance bound. Here $\widehat s_b^2$ estimates $B_P/n_v+B_Q/m_v$. This gives a 95% asymptotic allowance when $b_0$ is negligible. A deterministic $|b_0|$ bound adds to the radius. A random bound needs its own allocation within the total 5% budget. Recentring uses the combined variance and a usual 95% Wald interval. ### 2. Coverage, width, and the remaining drift (W1, Q1) > Q1 · “coverage and interval-width results” With exact outcome regressions ($b_0=0$), known propensity and $\omega=0.5$, the two-stratum check gives saturated learned-ratio coverage 89.15%→95.13% after recentering (width 0.598→0.708), and intercept-only coverage 0→95.25% (0.266→0.277). The 4,000 repetitions use 600 source outcomes, 1,800 target inference covariates and independent training/validation covariates. The four-cell learned-nuisance study uses 8,192 real Criteo covariate vectors with simulated treatment and outcomes, so the target ATE and drift components are known. Rich features contain the generating nonlinear terms. Across 1,200 repetitions per cell, rich outcomes with a linear ratio give ordinary coverage/width 93.00%/0.320 and recentered 95.25%/0.322. The paired coverage gain is 2.25 points (MCSE 0.52). With two linear fits, coverage changes from 51.58% to 52.00%, and the balance-radius interval covers at 69.75%. Its $b_0$ RMS is 0.1634. Rich/rich ordinary coverage is 95.50% at width 0.350. All four settings and eight procedures remain in the linked table. The study uses 2,800 source labels, 800 extra validation covariates and 7,200 target covariates, with every allocation in the supplement. Rich fitted models retain sampling error. Persistent ratio misspecification combined with root-$n$ outcome error generally leaves an $O_p(n^{-1/2})$ product. The little-$o$ remainder condition needs a separate rate argument.
AuthorsOpenReview ↗

Thanks for the review and here is the kind response(cont'd)

Pooling all 1,200 post-training source labels and 6,000 target covariates gives external coverage/width 94.58%/0.284 with rich fits, versus 95.25%/0.350 on inference blocks. With two linear fits, pooled coverage falls to 36.33%. The complete learned-nuisance results retain every method, allocation and drift component.

We additionally run a fixed 15-cell sample-size study that learns both nuisances in every repetition. It crosses weak, moderate and strong signal with correctly specified rich fits, a persistent ratio-error control, and a two-model misspecification control at inference n=400,800,1600n=400,800,1600, with m=3nm=3n. Each cell has 1,000 repetitions. Source training/calibration/inference labels are 2n/0.5n/n2n/0.5n/n, with nn further validation covariates. Target training/calibration/validation/inference covariates are 1.5n/1.5n/3n/3n1.5n/1.5n/3n/3n. Thus total labels are 3.5n3.5n and target covariates 9n9n.

Learned-nuisance settingOrdinary coverage at n=400/800/1600n=400/800/1600Variance-aware recentered coveragePooled external coverage
Weak, rich/rich94.0/94.2/95.9%93.5/94.2/95.8%93.6/93.6/96.0%
Moderate, rich/rich94.6/95.0/95.1%95.0/95.4/95.1%94.1/94.8/95.2%
Strong, rich/rich92.4/92.2/92.5%95.2/95.2/95.7%95.1/95.2/94.5%
Strong, rich/linear ratio58.8/35.7/10.2%93.5/95.0/94.8%92.9/94.7/94.3%
Moderate, linear/linear21.6/3.0/0.0%22.1/2.8/0.0%9.2/0.3/0.0%

For strong rich/rich fits, the RMS of b0/SEtrueb_0/\mathrm{SE}_{\mathrm{true}} falls from 0.134 to 0.088 to 0.047. Empirical SD/mean reported SE is about 1.01 throughout. Under two-model misspecification, standardized b0b_0 grows from 2.72 to 5.58, explaining the worsening coverage. The pooled external estimator uses all 1.5n1.5n post-training source labels and 7.5n7.5n post-training target covariates. Recentered widths are 22.5–39.2% larger across all 15 settings. This study measures correction effectiveness and its actual data cost together. All 15 settings, widths and drift components.

In a 200-repetition observed-outcome Criteo replay, outcome models use 80,000 source records. Ratio fitting uses 20,000 of those covariates and 20,000 target covariates. Calibration, covariate-only validation and inference each contain 6,000 source and 18,000 target records. Inference is compared with a separate estimated target benchmark.

Fitted ratioOrdinary inclusion / widthRecentered inclusion / widthRadius-expanded inclusion / width
Logistic odds93.5% / 0.0285094.0% / 0.0285297.5% / 0.03411
Intercept-only odds94.5% / 0.0230790.5% / 0.0231098.0% / 0.03088

Mean ωb^1\omega\widehat b_1 is 0.000036 for logistic odds and 0.001705 for intercept-only odds. Evaluating the population moment using the known design ratio gives 0.000031 and 0.001694. The estimator itself uses learned odds. The intercept-only inclusion rate falls after recentering, illustrating that removal of one drift component can increase total error. Benchmark uncertainty remains part of these inclusion results.

3. Matched alternatives and the source of precision gains (W2, Q2)

W2 · “consequential precision improvement” · Q2 · “calibration budget”

Where the gain is substantial. With informative, over-amplified treatment-effect predictions, five-fold TC-CPI gives more than 5% shorter intervals than all three full-budget endpoints simultaneously in 17 of the 20 fixed settings; every paired 95% Monte Carlo lower bound exceeds 5%. The 16 signal settings yield 14.88–38.25% savings over transported AIPW. The comparison uses known design weights and charges every method for its source labels.

We study which prediction-based precision gains survive learning transport weights. The contribution joins the causal score's variance, first-order transport drift and correction cost. The scalar path has the generalized-PPI form. With A=θ^1A=\widehat\theta_1, Δ=Qmτ^f−Pn(r^τ^f)\Delta=Q_m\widehat\tau_f-P_n(\widehat r\widehat\tau_f) and a=1−ωa=1-\omega,

θ^ω=A+aΔ.\widehat\theta_\omega=A+a\Delta.

Matching baseline, control direction, coefficient range and calibration gives identical point estimates. Zou et al. also treat covariate shift and the first-order contribution of estimating the labeling mechanism. Our added result identifies ωb1\omega b_1 for this frozen causal score, even with exact outcome regressions, and provides the correction above. Zou et al. (2026), Sections 3.2 and 4

AuthorsOpenReview ↗

Thanks for the review and here is the kind response(cont'd)

Matched-score comparisons share final arm regressions, propensity, ratio and blocks. Demirel's DR-PA and Lal's unstabilized transportation AISW coincide with external after aligning domain-prior scaling. This matches their final scores. Demirel's native training also augments outcome modeling with external predictions. Our Zou adaptation uses the augmented causal score in a transported Hájek baseline, posterior-domain prediction pooling and the same calibration observations as TC-CPI. Demirel et al., Appendix A.4, Lal et al., Table 1

On observed Criteo outcomes, these methods have similar precision. Across 100 matched splits, mean widths are 0.028688 for TC-CPI, 0.028695 for the external score, 0.028711 for Zou, 0.028683 for transported AIPW and 0.028678 for self-normalized AIPW. The first four include the separate estimated benchmark in 96 splits. Across the 17 original recorded-outcome replay settings, the largest mean width saving over transported AIPW is 0.38%. Observed-outcome comparisons

The mechanism is covariance cancellation. Symmetric arm construction gives U=rτ−τf+2(2T−1)εU=r\\{\tau-\tau_f+2(2T-1)\varepsilon\\}. An amplified, aligned predictor can make C=Cov⁡P(U,rτf)<0C=\operatorname{Cov}_P(U,r\tau_f)<0, so an interior share cancels residual variation. Available gain over the better endpoint is [min⁡V(0),V(1)−min⁡ω∈[0,1]V(ω)]/min⁡V(0),V(1)[\min\\{V(0),V(1)\\}-\min_{\omega\in[0,1]}V(\omega)]/\min\\{V(0),V(1)\\}. This gain must cover coefficient-estimation and label-allocation costs.

SettingMoment evidenceResidual–prediction correlationAvailable variance gain
Criteo, four predictorsCalibration projections−0.084 to −0.0180–0.0116%
STAR, original student splitsCalibration projections−0.0660.226%
Sixteen signal mechanismsExact generating law−0.74541.29%

Real-data entries summarize medians over calibration splits. The Criteo range spans four predictor medians. The mechanism row gives medians over 16 laws, with available gains ranging from 27.61% to 61.86%. These are variance opportunities at fixed inference allocation before calibration cost. The real-data projections are consistent with the observed near-null gains.

The real-covariate mechanism study uses known propensity, ratio and ATE. Its 16 signal cells cross amplitudes 1.5/2, contrast correlations 0.75/0.95, two bounded shifts and target/source ratios 1/4, with effect/noise SD 1/0.25 and fixed common prognosis. Four controls retain weak, uninformative, negatively aligned and no-shift cases. Total budgets are 1,000 source labels and 1,000/4,000 target covariates. The original independent-split methods reserve 20% for calibration. Five-fold calibration lets each label contribute once to inference, with its coefficient trained on the other folds. Zou receives matching folds. This extension was defined after the original study and validated on all 20 unchanged cells with 1,200 new repetitions each.

The table reports native performance across the same 16 five-fold signal cells. Entries are ranges except maximum absolute bias. SD/SE is empirical SD divided by mean reported SE.

MethodCoverageMean widthMaximum absolute biasRMSESD/SE
TC-CPI, five-fold94.00–96.17%0.0927–0.13860.001310.0234–0.03690.966–1.044
Transported AIPW, full budget94.17–96.50%0.1500–0.16310.002340.0370–0.04300.966–1.034
Self-normalized AIPW, full budget93.92–95.92%0.1439–0.15120.001900.0356–0.03980.968–1.034
External endpoint, full budget93.33–96.08%0.1386–0.31400.003190.0356–0.07950.980–1.063
Zou, matched five-fold93.67–96.50%0.0919–0.13090.001170.0232–0.03500.962–1.050

Five-fold TC-CPI saves 14.88–38.25% width against full-budget AIPW, 8.04–35.62% against its normalized version and 28.82–60.16% against external. Paired 95% Monte Carlo lower bounds exceed 5% in 17/20, 17/20 and 19/20 settings, respectively. TC-CPI is 0.85–7.40% wider than Zou in all 16 signal cells, with native coverage displayed alongside its width. Same-direction bounded generalized PPI agrees with TC-CPI. The original split results, unfavorable controls and all error metrics remain in the complete comparisons.

For fixed predictors and correct known weights, each fold preserves the target because its prediction difference has conditional mean zero given the other folds. With fixed fold count and consistent coefficients, coefficient error times this root-nn difference is lower order, yielding the full-sample path's first-order variance. This foldwise extension retains the drift conditions above when weights are learned.

Reviewer_3Cg8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies estimation of the target-population average treatment effect under population shift, using observed outcomes from a source population and covariates alone from the target population. It introduces TC-CPI, which adapts PPI-style calibration to transported causal estimation. Under the identification assumptions and correct propensity and density-ratio weights, the estimator is unbiased even when the outcome predictor is misspecified. The paper derives a variance-minimizing mixing weight and shows that independent calibration asymptotically matches the best weight within this estimator family. With oracle outcome regression and the external-PPI endpoint, the estimator attains the semiparametric efficiency bound. It also characterizes the additional bias that can arise when the density ratio is estimated. Simulations support the theoretical predictions, including adaptation to poor predictors, but experiments on Project STAR and Criteo show little to no practically meaningful variance reduction relative to transported AIPW.

Critical strengths
  1. TC-CPI connects prediction-powered inference with causal transport estimation and provides a unified analysis of target-population ATE estimation when target outcomes are unavailable under covariate shift.
  2. The paper carefully analyzes mixing-weight calibration and distinguishes its variance benefits from the bias introduced by density-ratio estimation. The simulation experiments are closely aligned with the theoretical claims and illustrate both the benefits and limitations of the method.
Critical weaknesses
  1. The incremental theoretical contribution relative to existing work is not sufficiently clear. PPI++ already develops prediction-weight tuning for robust variance reduction, while prior transport and covariate-shift methods combine outcome regression with density-ratio-weighted residual correction. In particular, Uehara et al. (2020), "Off-Policy Evaluation and Learning for External Validity under a Covariate Shift", derive a closely related estimator structure and an efficiency bound containing an r2r^2-weighted outcome-noise term. The paper would benefit from a more explicit distinction between results inherited or adapted from these frameworks and genuinely new results concerning the proposed calibration path.
  2. The real-data experiments show negligible variance improvements over transported AIPW. Although the authors acknowledge this, further explanation would help understand the method’s practical relevance.
Questions for authors
  1. Which results go beyond an application or combination of existing PPI++ calibration and transported doubly robust estimation? In particular, could you clarify the additional contributions of Theorems 3.1, 4.1–4.3, and 4.6 relative to these precedents?
  2. Could you elaborate more on why the proposed method doesn't improve the variance in real applications?
15 comments
AuthorsOpenReview ↗

Thanks for the review and here is the kind response

1. Why the target population changes the calibration problem (W1, Q1)

“incremental theoretical contribution”

“Theorems 3.1, 4.1–4.3, and 4.6 relative to these precedents”

A1. Theoretical novelty: from transported identification to the efficiency–robustness trade-off

We thank the reviewer for highlighting the connections to PPI++ and Uehara et al. (2020). Prediction-power tuning and transported doubly robust estimation are established tools. Our contribution addresses a distinct statistical question: when can predictive information measured under two different population laws improve the precision of a target causal effect, and what transport information is required to realize that improvement without compromising valid inference? Prior work establishes how to identify and estimate the target effect under covariate shift. The remaining question is whether the two sampling representations of its predictive component contain exploitable covariance information beyond the established transported endpoint, and whether the resulting efficiency gains survive uncertainty about the transport mechanism. Our results connect these questions through a target-preserving construction, optimal calibration theory, and a characterization of the robustness costs introduced by calibration.

1. A unified calibration path that contains the established estimators (Theorem 3.1). Our estimator admits the decomposition

θ^ω=Qmτf+PnZf⏟external transported DR endpoint+ωPn(rτf)−Qmτf⏟cross-population control variate.\widehat\theta_\omega = \underbrace{Q_m\tau_f+P_nZ_f}_{\text{external transported DR endpoint}} + \omega\underbrace{\\{P_n(r\tau_f)-Q_m\tau_f\\}}_{\text{cross-population control variate}}.

The first component is the ATE specialization of the transported doubly robust estimator of Uehara et al. (2020). The second arises because EP[rτf]=EQ[τf]E_P[r\tau_f]=E_Q[\tau_f]: the source and target populations provide two sampling representations of the same predictive functional, with different variances and dependence structures. Their difference is therefore a zero-mean control variate under correct transport weights. Theorem 3.1 establishes that incorporating this term preserves the target ATE for every fixed predictor and calibration weight when the propensity and density ratio are correct. This identity fixes the estimand across the entire path, allowing a direct analysis of its precision and transport sensitivity. The construction also recovers prior approaches as special cases: ω=0\omega=0 gives the external transported DR endpoint, while PX=QXP_X=Q_X yields the PPI++ power-tuning formulation on the labeled causal score (Proposition 3.1). The contribution lies in analyzing what becomes statistically possible, and what becomes fragile, when these two sampling representations arise from different populations.

2. Exploiting information beyond the transported endpoint (Theorems 4.1–4.3). The calibration problem asks how much precision the two representations of the prediction mean can provide under the specified transport model. In our construction, the source-weighted prediction and labeled residual correction share observations. Their covariance enters the exact variance together with the source and target prediction variances:

V(ω)=A+2ωCUR+ω2BPn+(1−ω)2BQm.V(\omega) = \frac{A+2\omega C_{UR}+\omega^2B_P}{n} + \frac{(1-\omega)^2B_Q}{m}.

Theorem 4.1 characterizes this two-law variance geometry and its oracle-optimal calibration weight. An interior weight can strictly outperform both endpoints: in our two-stratum example, ω⋆=0.157\omega^\star=0.157 yields variance 0.13920.1392, compared with 0.14680.1468 for the external endpoint and 0.35900.3590 for transported labeled AIPW. More revealingly, the optimal source weight converges to approximately 0.1530.153 even as the target sample becomes arbitrarily large. Thus, increasingly precise target prediction averaging need not make source predictions redundant: their value can persist through covariance with the labeled residual correction, rather than through improved estimation of the prediction mean alone. The residual–prediction covariance determines when this persistent source contribution is useful.

Theorem 4.1 establishes the oracle opportunity; Theorems 4.2–4.3 establish its statistical attainability and cost. Independent calibration preserves conditional unbiasedness under the stated correct-weight assumptions and achieves oracle first-order variance, while the finite-sample excess-variance guarantee is of order O{log⁡(1/δ)/(nNc)}O\{\log(1/\delta)/(nN_c)\}. Together, these results answer three connected questions for the specified calibration path: when does additional predictive information improve efficiency, can the optimal combination be learned, and how much variance is lost by learning it? The guarantees quantify this causal path under two different covariate laws.

AuthorsOpenReview ↗

Thanks for the review and here is the kind response(cont'd)

3. The intrinsic conflict between calibration efficiency and transport robustness (Proposition 4.1 and Theorem 4.5). The oracle gain comes with a statistical qualification: the control variate is mean-zero only under correct transport information, precisely the quantity that must generally be learned under population shift. This creates a tension not resolved by the oracle variance calculation. Even when the outcome regression is correct, a normalized density-ratio perturbation r~ε=r(1+εg)\widetilde r_\varepsilon=r(1+\varepsilon g) induces the exact drift

D(ω)=εωEQ[g(X)τ(X)].D(\omega) = \varepsilon\omega E_Q[g(X)\tau(X)].

Consequently, with heterogeneous treatment effects, every positive interior weight admits a density-ratio perturbation direction that produces first-order bias, whereas the external DR endpoint remains ratio-orthogonal when the outcome regression is correct. Cross-fitting can control dependence between nuisance estimation and evaluation, but it cannot remove this population-level drift. The same cross-population control variate that improves oracle precision can therefore reintroduce first-order sensitivity to transport error, even with an oracle outcome model.

This is the central efficiency–robustness tension revealed by our analysis. Substituting a learned ratio changes the estimator’s center as well as its variance. Theorem 4.5 identifies the drift control required for valid inference, while Corollary 4.1 incorporates explicit bounds on transport and conditional-shift errors into sensitivity-aware calibration. The resulting principle is that the amount of source prediction information worth using depends jointly on its variance benefit and the reliability of the transport mechanism. This yields an operational rule for this estimator family: price transport uncertainty alongside the covariance benefit when choosing the source share.

4. Why transport information changes the efficiency comparison (Theorem 4.6). The preceding tension also explains why our oracle variance improvements must be interpreted relative to the appropriate statistical information model. The unrestricted external-target semiparametric efficiency bound follows from Uehara et al. (2020). Their policy-evaluation result specializes to the ATE through the always-treat versus never-treat contrast, yielding the familiar r2r^2-weighted source-outcome-noise term and the target-covariate variance term. Theorem 4.6 records this established ATE specialization and its attainment by the external endpoint; the revision attributes the bound where it is stated.

The distinction is that Theorem 4.1 optimizes calibration with the correct density ratio treated as fixed transport information, whereas the unrestricted efficiency bound allows the source and target covariate distributions to vary independently. Knowing the density ratio imposes the restriction dQX=r dPXdQ_X=r\,dP_X, which changes the admissible distributional perturbations and potentially the efficiency bound. The two comparisons consequently use different information models. The apparent conflict is resolved by distinguishing the statistical value of knowing the transport mechanism from what can be achieved when that mechanism must itself be estimated. This distinction also motivates the robustness analysis above: additional transport information can make calibration gains attainable, while uncertainty about that information can compromise their inferential validity. Corollary 4.2 further translates the inherited source-noise structure into prospective labeling and treatment-allocation rules.

The resulting contribution is a theory of precision and transport reliability along the causal calibration path. Theorem 3.1 fixes its target; Theorems 4.1–4.3 quantify attainable covariance gains and calibration cost; Proposition 4.1 and Theorem 4.5 identify the first-order sensitivity created by learning transport weights; and Corollary 4.1 converts uncertainty bounds into a calibration rule. Theorem 4.6 supplies the established efficiency benchmark. Sections 2–4 now state these roles and the Uehara attribution explicitly. The design principle extends to future prediction sources: assess their covariance benefit and the reliability of the transport information that makes their contribution mean-zero.

2. When prediction calibration improves precision (W2, Q2)

Where the gain is substantial. With informative, over-amplified treatment-effect predictions, five-fold TC-CPI gives more than 5% shorter intervals than all three full-budget endpoints simultaneously in 17 of the 20 fixed settings; every paired 95% Monte Carlo lower bound exceeds 5%. The 16 signal settings yield 14.88–38.25% savings over transported AIPW. The comparison uses known design weights and charges every method for its source labels.

“negligible variance improvements over transported AIPW”

“improve the variance in real applications”

AuthorsOpenReview ↗

Thanks for the review and here is the kind response(cont'd)

The gain depends on predictable treatment-effect variation, its alignment with the prediction contrast, the source outcome-noise share, and the labels used for calibration. We tested these mechanisms with an independently confirmed grid and matched total data budgets.

Observed trials. The submitted Criteo and STAR results, and the additional Hillstrom replays, show little precision change with the evaluated predictors. Across the 17 original observed-outcome replay settings, adaptive variance savings relative to transported AIPW range from −0.45% to 1.31%. Mean interval-width savings are at most 0.38%, and at most 0.11% against self-normalized AIPW. These are repeated-split diagnostics on recorded trials. All 17 observed-outcome settings

The covariance mechanism. Let AA be transported AIPW, Δ=Qm[τf]−Pn[rτf]\Delta=Q_m[\tau_f]-P_n[r\tau_f], and a=1−ωa=1-\omega. With fixed predictions and correct propensity and transport weights, Theorem 4.1 gives

c=-\operatorname{Cov}(A,\Delta)=\operatorname{Cov}_P(r\tau,r\tau_f)/n,\qquad v=\operatorname{Var}(\Delta).$$ For $v>0$, $a^\star=\Pi_{[0,1]}(c/v)$. Predictable weighted effect variation gives a useful control variate. The source residual correction and weighted prediction share observations, so their covariance can also make an interior weight more precise than the external endpoint. Conditional mean-zero outcome noise remains in every score. The new experiment separates that covariance from common prognosis. It uses $\mu_t=m+(2t-1)\tau/2$ and $f_t=m+(2t-1)\tau_f/2$. Under randomized propensity $1/2$, $$U=r\{\tau-\tau_f+2(2T-1)\varepsilon\},\qquad \operatorname{Cov}_P(U,r\tau_f)=\operatorname{Cov}_P(r\tau,r\tau_f)-\operatorname{Var}_P(r\tau_f).$$ An aligned predictor with an overstated effect scale can therefore create negative residual–prediction covariance. The calibrated source share uses this covariance to reduce variation while preserving the target under the correct fixed weights. **Independent confirmation with total label budgets.** The grid was motivated by the earlier experiments. We fixed all 20 cells, budgets, seeds and reporting rules before running 200 exploratory repetitions per cell, then confirmed every cell using 1,200 repetitions with seed 148101213. The 16 signal settings cross prediction amplitudes 1.5 and 2, correlations 0.75 and 0.95, target/source budget ratios 1 and 4, and two bounded shifts. Effect SD is 1 and outcome-noise SD is 0.25. Four additional controls test weak signal, uninformative prediction, negative alignment and a shared covariate law. These are simulated randomized outcomes on a fixed empirical Criteo covariate support, with known target ATE and known design ratio. Each setting has 1,000 source labels and 1,000 or 4,000 target covariates. The independent-split implementation uses 20% for calibration and 80% for inference. Source-only and external full-budget comparators use all 1,000 labels and their available target covariates. Zou's augmented-score adaptation uses the same calibration/inference allocation as TC-CPI. Positive width savings favor TC-CPI. | Comparator | Width saved in the 16 signal settings | Narrower, signal settings | Narrower, all 20 settings | More than 5% narrower, all 20 | | --- | ---: | ---: | ---: | ---: | | Transported AIPW, all labels | +4.80% to +30.91% | 16/16 | 17/20 | 14/20 | | Self-normalized AIPW, all labels | -2.85% to +27.95% | 14/16 | 15/20 | 13/20 | | External one-step, all labels | +20.16% to +55.41% | 16/16 | 19/20 | 19/20 | | Zou, augmented-score adaptation | -7.31% to -0.86% | 0/16 | 1/20 | 0/20 | | Generalized PPI, same path and coefficient range | 0.00% (same estimate) | 0/16 | 0/20 | 0/20 | The more-than-5% column requires the lower endpoint of the paired 95% Monte Carlo interval to exceed 5%. These intervals measure simulation precision within the specified design. In the 16 signal settings, TC-CPI coverage is 93.83–95.83%. Its RMSE reductions relative to the full-budget source, normalized source and external estimators range from 1.39–31.19%, −3.56–28.22%, and 16.56–55.23%. Thus the width differences also track estimation error. Zou's adaptation has shorter intervals throughout this signal domain. Generalized PPI using exactly the same baseline, control direction and coefficient range gives the same estimate. [Every confirmation cell, comparator and metric](https://calibration-figures-q7m4.green-pearl-1123.chatgpt.site/#mechanism-confirmation) **Boundary settings and calibration cost.** All four controls are included below. The weak-signal case assigns 98.93% of source-estimator variance to irreducible outcome noise. In the uninformative and negatively aligned cases, the constrained optimum puts most or all weight on the source endpoint. Reserving labels for calibration then reduces precision relative to endpoints using every label.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks for the review and here is the kind response

A Kind General Response to All Reviewers and Dear AC

Dear Reviewers and Area Chair,

Thank you for the careful reviews and constructive suggestions. They helped us sharpen the scientific question, clarify the assumptions and add direct empirical checks. The revised manuscript marks the changes in blue; its main text remains nine pages.

  1. Identification and sensitivity to population changes (i1wu–W1–W5, Q1–Q2).

    • Section 2 and Appendix A.8 separate source-outcome recovery, covariate transport and conditional-effect changes. The mediator-driven attrition example now includes a recovery score and simulation.
    • Section 4.4 explains outcome-scale sensitivity inputs and decision thresholds. The revision also clarifies the prediction-average limit, membership odds, terminology and methodological sequence.
  2. Calibration contributions and learned transport (3Cg8–W1, Q1, Hfkh–W1, Q1).

    • Sections 2–4 distinguish inherited PPI tuning and transported estimation from the two-population covariance analysis, calibration cost and transport-induced drift. The endpoint and efficiency benchmark explicitly credit Uehara et al.
    • The added validation-correction proposition removes held-out prediction imbalance and includes its variance. Fitted-nuisance experiments report coverage, width and the remaining interaction drift together.
  3. Matched evidence and STAR inference (3Cg8–W2, Q2, Hfkh–W2, Q2–Q3).

    • The complete known-weight grid compares identical budgets and recent alternatives. Five-fold calibration gives more than 5% width savings over all three full-budget endpoints simultaneously in 17/20 settings. The 16 signal settings give 14.88–38.25% savings over transported AIPW, with 94.00–96.17% coverage. Full comparisons and covariance diagnostics identify the conditions supporting these gains.
    • STAR now reports school-clustered uncertainty on unchanged estimates, followed by a separate roster attrition sensitivity analysis. Anonymous links provide the complete protocols, figures and reproducible outputs.

Our detailed replies are organized under i1wu, Hfkh, and 3Cg8, with continuations in each discussion chain.

Thank you again for your time and thoughtful guidance. We warmly welcome further questions and discussion with the reviewers and Area Chair, and look forward to clarifying any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.