acceptodds
Under review as a conference paper at ICLR 2027

DRIFT: Causal Inference for Generated Treatments When Replay Is Not Provenance

Abstract

Causal effects of generated treatments depend not only on which candidates were shown, but also on the historical mechanism that selected among them. When that mechanism is no longer reproducible, current replay estimates today's selector rather than reconstructing yesterday's assignment law. We prove that even unlimited replay and exact aggregate calibration can leave distinct historical causal effects observationally indistinguishable. This provenance boundary motivates Drift-Robust Inference From Treatment traces (DRIFT), which turns a prespecified or independently audited historical-to-current log-odds bridge into inverse-selection envelopes, observable calibration restrictions, and dual causal certificates. The resulting interval is sharp for the specified marginal trace-weight model and outer-valid for every compatible latent historical effect; at compatible zero drift it reduces to the ordered binary-pair logged counterfactual exposure functional. With bounded held-out dual scores and independent population-valid replay bands, cross-fitting yields finite-sample outer confidence sets even under approximate optimization. Regular endpoints separate historical and replay uncertainty, whereas nonregular endpoints need not admit Gaussian inference; on an explicit compact class, the minimax expected squared Hausdorff risk is of order . The same dual geometry quantifies where historical audits can most reduce causal ambiguity and yields a first-order cost-aware allocation rule. A controlled HelpSteer2 study tests certificates with real ratings, constructed selectors, and potential-rating coupling; predicted inclusions are observed in valid conditions. A fixed-action real-log diagnostic shows operation under injected mismatch without validating causal coverage. Eight synthetic studies probe identification, inference, and audits: replay estimates today's selector but need not recover the historical selection law.

Primary area:
causal reasoning
Keywords:
causal inference, generated treatments, partial identification, selection mechanism drift, sensitivity analysis, finite-sample inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uTYHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies causal identification and inference in a setting that builds on the logged counterfactual exposure (LCE) framework developed by Athey et al. (2026). It considers a practically relevant departure from the original framework: the available replay mechanism differs from the historical mechanism that generated the observed assignments, and the historical assignment probabilities are unknown. The paper adapts existing sensitivity-analysis and partial-identification techniques to construct bounds on the causal effect of interest under restrictions on the discrepancy between the two mechanisms. It also develops confidence sets that account for uncertainty in historical observations and replay data.

Critical strengths
  1. Well-motivated problem. The paper addresses an important practical limitation of replay-based LCE. In deployed applications, the selection mechanism may change over time, and the information needed to reproduce historical assignments may no longer be available. Consequently, replaying the current mechanism need not recover the probabilities relevant to historical outcomes. I find this problem meaningful and well motivated.

  2. Coverage guarantees do not require exact optimization. An appealing property of Theorem 5.1 is that the proposed confidence-set construction does not require exact optimization for its coverage guarantee. Numerical suboptimality may widen the interval without invalidating coverage. This is a useful feature for an optimization-based inference procedure and strengthens its practical appeal.

  3. Timely research direction. LCE is a recent framework, and studying its limitations and extensions is a worthwhile research direction. The paper engages with questions relevant to evaluating generated treatments, where conventional approaches to defining and estimating treatment effects may be difficult to apply.

Critical weaknesses
  1. Limited novelty of the central identification approach. My impression is that the paper’s core identification method is primarily an adaptation of existing sensitivity-analysis and partial-identification techniques to the LCE setting. Although this adaptation may be useful, the manuscript would benefit from a clearer separation between established methodological ingredients and genuinely new contributions. In particular, it should explain more explicitly which results go beyond applying existing calibrated-weight methods to the ordered-pair comparison.

  2. The practical resolution of the historical-information problem remains unclear. The proposed interval depends on a prespecified or independently justified bound on the discrepancy between the historical and current assignment mechanisms. However, when information about the historical mechanism has been lost—the setting that motivates the paper—it is not clear how an informative bound can be credibly chosen or certified. Sensitivity analysis is a reasonable response to this problem, but the paper should explain more concretely what additional historical information is needed and when that information is realistically available.

  3. The exposition could be clearer. I found the presentation difficult to follow in several places, particularly when terminology is introduced without sufficiently explicit definitions. For example, the second paragraph of the introduction refers to “n interactions over K_n identities. The meanings of “interactions” and “identities” are not immediately clear to readers familiar with the causal inference literature. More direct terminology, or explicit definitions in terms of observations and distinct treatment variants, would improve accessibility. More generally, the paper would benefit from clearer explanations of the statistical objects before presenting the associated results.

  4. The sharpness claim applies to a relaxation. The interval is sharp for the specified marginal trace-weight model, but it is only an outer bound for the original latent historical model. The paper acknowledges that separate armwise optimization can widen the interval. This distinction limits the scope of the sharpness result. It would be helpful to explain when the relaxation is exact and how much additional width it can introduce, so that readers can distinguish uncertainty inherent in the original model from conservativeness introduced by the relaxation.

Questions for authors
  1. How does your estimand relate to that of Athey et al.? Your target is E[Y(1)-Y(0) | E=1], whereas Athey et al. consider a formulation of the form E[tau | h(V,V^N)>0]. Could you clarify the relationship between these quantities and the conditions under which they coincide? In particular, when assignment probabilities vary across candidate pairs, does conditioning on the positive display orientation change the population over which effects are averaged?

  2. How does the proposed framework relate to current LLM evaluation methods? Both this paper and LCE develop causal approaches relevant to evaluating LLM-generated outputs. It would be helpful to explain how these approaches relate to evaluation methods currently used in leading research and practical deployments. What substantive questions do existing methods fail to answer, and what additional information or decision-making value does the causal approach provide?

  3. Could you give a more substantive interpretation of the estimand? I have some reservations about the interpretation of this estimand, including the related estimand in Athey et al. In particular, how does an average comparison between interaction-specific candidate treatments relate to conventional causal estimands comparing two common, well-defined interventions? Does the estimand describe the effect of choosing between complete generated treatments, rather than the effect of changing a particular treatment characteristic while holding other characteristics fixed? I appreciate that the setting may prevent identification of more narrowly defined causal effects without stronger assumptions. Nevertheless, clearly explaining this conceptual distinction—and what counterfactual question the proposed estimand does and does not answer—would substantially improve the paper.

4 comments
AuthorsOpenReview ↗

The causal target, its evaluation value, and the sharpness result

Thank you for your feedback.

Q1: Positive-display conditioning changes the target population.

DRIFT averages over all eligible pairs. Let E=1E=1 denote a non-tied pair, D=1D=1 higher-answer display, pp its historical probability, and Δ=Y(1)−Y(0)\Delta=Y(1)-Y(0). For the same ordered pairs and contrast, historical conditional randomization gives

\tau_+=\frac{\mathbb E[p\Delta\mid E=1]}{\mathbb E[p\mid E=1]},\quad \tau_+-\tau_h=\frac{\operatorname{Cov}(p,\Delta\mid E=1)}{\mathbb E[p\mid E=1]}.$$ Thus positive-display conditioning reweights pairs and historical states by $p$. The targets coincide precisely when this covariance is zero; constant $p$ is sufficient. Even with $q=p$, two equally likely types with $(p,\Delta)=(1/4,1)$ and $(3/4,0)$ have $\tau_h=1/2$ and $\tau_+=1/4$. Revised Equation (2) gives this identity and example. DRIFT retains $\tau_h$ throughout, including at compatible zero drift. **Q3: The interventions are two common answer-selection rules.** Each pair uses the same prespecified rule: display its higher-ranked versus lower-ranked complete answer. Texts vary; the rules remain common. Pair generation precedes display; the historical generator and eligible-pair population stay fixed. The effect compares applying these rules to all eligible pairs. For example, ranking tutoring answers by explanatory detail and measuring subsequent retention estimates the effect of selecting the complete more-detailed answer, including its accompanying content differences. Isolating detail alone requires controlled edits or additional identifying assumptions. Revised Section 3 and Appendix A separate the intervention, population, and outcome explicitly. **Q2: Decision-making value alongside existing LLM evaluation.** MT-Bench measures LLM-judge scores, Chatbot Arena measures human preferences, and randomized online A/B tests identify effects of assigned systems or policies on measured outcomes. DRIFT adds a causal certificate for historical answer-selection decisions when assignment probabilities are unavailable. A positive lower confidence endpoint supports selecting the higher-ranked answer for that historical population under the drift and coverage assumptions. Measured retention, correctness, or helpfulness determines what improvement means; the ranking rule defines the comparison. Section 2 clarifies this value and the transport assumption needed for another generator or population. The added controlled HelpSteer2 study demonstrates this decision with the full inference procedure, fixed selectors, $\Gamma=0.6$, $K=4$, and $B=10{,}000$. Entries below count positive lower confidence endpoints out of 100 repetitions. | Constructed outcome law | $\tau_h$ | Population lower endpoint | $n=2{,}000$ | $n=10{,}000$ | $n=50{,}000$ | | --- | --- | --- | --- | --- | --- | | Original | 0.034546 | −0.070074 | 0 | 0 | 0 | | Null | 0 | −0.079521 | 0 | 0 | 0 | | $s=0.1$ | 0.064614 | −0.014333 | 0 | 0 | 0 | | $s=0.4$ | 0.258457 | 0.182187 | 0 | 100 | 100 | | $s=0.8$ | 0.516914 | 0.462335 | 87 | 100 | 100 | All 1,500 intervals cover truth. At $s=0.4$, DRIFT certifies a positive effect in 100/100 runs at $n=10,000$; the null yields none. These controlled alternatives demonstrate useful sign decisions; the original population remains sign-ambiguous under the marginal model (Figure 4(g), Appendix F.5; [full results](https://anonymous.4open.science/r/ICLR2027-rebuttal-01-F15F/fig3.pdf). **W4: Sharpness and the gains from calibration.** Theorem 4.1 establishes sharpness within the calibrated marginal weight model and containment of all compatible latent effects. Separate armwise weights can enlarge the interval because a latent completion additionally requires one common historical state, assignment probability, and joint potential-outcome law. Section 4 now quantifies this distinction. For $Z\sim\mathrm{Bernoulli}(1/2)$, $p=q=1/2$, $Y(1)=Z$, $Y(0)=0$, and $\Gamma=\log3$, the weight boxes are $[4/3,4]$: | Balance | Marginal interval | Latent set | Extra width | | --- | --- | --- | --- | | Intercept | $[1/3,2/3]$ | $\{1/2\}$ | $1/3$ | | Intercept and $Z$ | $\{1/2\}$ | $\{1/2\}$ | 0 | Positivity and the observed outcomes force $Y(1)=Z,Y(0)=0$ in every latent completion. Adding $Z$ fixes $\mathbb E[DwZ]=1/2$, closing the entire gap. Compatible zero drift is another exact case. Generally, nested valid moments tighten the marginal interval; latent equality requires compatibility. At the HelpSteer2 anchor, $K=1\to4$ reduces population/final widths by 10.7%/5.1%, retaining all 1,000 chains per condition (Appendix F.2). Section 7 also reports EXP5's wider final certificates: more balance can reduce population ambiguity without ensuring a narrower final confidence interval, making the moment-resolution tradeoff explicit. We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.
AuthorsOpenReview ↗

Additional contributions, historical evidence, and clearer exposition

W1: Beyond calibrated-weight methods.

DRIFT recovers the calibrated population program for matched P,q,ΓP,q,\Gamma, boxes, and balance constraints. It adds a two-source finite-sample certificate and a way to value historical audits when replay measures a changed selector. Section 2 clarifies these contributions and credits established optimization and concentration tools.

Theorem 3.2 shows why historical selection–outcome dependence must be recovered or constrained: effects of 1/21/2 and 1/61/6 yield identical observable laws and exact replay, even under aggregate calibration.

Under iid history, fixed folds and radius, bounded outcomes and basis, nonempty weight classes, and deterministic bounds on inflated weights and scores, Theorem 5.1 gives Pr⁡(I⊆I~⊆C)≥1−α.\Pr\left(\mathcal I\subseteq\widetilde{\mathcal I}\subseteq\mathcal C\right)\ge 1-\alpha. An independent population-valid replay band enlarges the marginal population interval I\mathcal I to I~\widetilde{\mathcal I}; held-out historical corrections give C\mathcal C. Approximate optimization preserves coverage but may widen the interval.

Experiments check both inclusions separately; all 22,000 repetitions across 22 valid HelpSteer2 conditions satisfy the full chain. Component checks at n=B=10,000n=B=10{,}000, K=4K=4, and η=Γ=0.6\eta=\Gamma=0.6 are:

ConstructionComplete chainFinal-set coverageScalar coverage
Full DRIFT100%100%100%
Replay point estimate without band14.6%100%100%
Without Bernstein corrections25.9%95.5%100%
Coarse-grid dual optimization100%100%100%

The ablations separate protection of the whole identified set from coverage of one causal value, showing why scalar coverage alone cannot verify the full chain. Each row has 1,000 runs; final-set coverage is I⊆C\mathcal I\subseteq\mathcal C. The link diagnostics show failed inclusions and violation sizes. Coarse-grid training preserves all chains at widths 0.5390 versus 0.5218 (Appendices F.2/J.2).

Under Theorem 5.2's regularity conditions, uninflated estimates have asymptotic covariance VO/n+VR/BV_O/n+V_R/B; its stated compact class has minimax squared-Hausdorff risk n−1+B−1n^{-1}+B^{-1}. At Proposition 5.3's nonregular endpoint, the limit is A+∣H∣A+|H| for independent centered Gaussians; the finite-sample certificate still applies. Section 6 further differentiates population width, sj=∂jWs_j=\partial_jW, to value audits. Radius uncertainty cj/mjc_j/\sqrt{m_j} and cost κj\kappa_j yield mj∝(sjcj/κj)2/3m_j\propto(s_jc_j/\kappa_j)^{2/3}. These width sensitivities make the standard allocation rule responsive to causal precision. Exact-width optimality needs differentiability; its plug-in form needs a growing independent consistent pilot. Sections 5–6 thus guide investment in history, replay, or audits.

W2: Partial historical information can support an informative radius.

Separate old-version or overlap-period choice records estimate historical cell probabilities even after the executable system and trace-level propensities are lost. This requires matched pairs and contexts, representative sampling, a stable historical mechanism, and homogeneous cells. Recovered configuration or state supports envelopes when its effect on selection probabilities is justified; full propensity recovery permits direct weighting.

Appendix E.2 now specifies these information regimes. Given simultaneous historical and replay bands [pjL,pjU][p_j^L,p_j^U], [qjL,qjU][q_j^L,q_j^U], homogeneous-cell certification uses

Γ^jU=max⁡{∣logit⁡pjL−logit⁡qjU∣,;∣logit⁡pjU−logit⁡qjL∣}.\widehat{\Gamma}_{j}^{U}=\max\left\lbrace\left|\operatorname{logit}p_{j}^{L}-\operatorname{logit}q_{j}^{U}\right|,\\;\left|\operatorname{logit}p_{j}^{U}-\operatorname{logit}q_{j}^{L}\right|\right\rbrace.

Heterogeneous cells require additional justified bounds on within-cell log-odds variation. Pilot, audit, and replay information must be jointly independent of history reserved for training and evaluation, with audit failure included in the error budget. Current-only replay supports assumed-radius sensitivity analysis. The radius study demonstrates the value of a valid historical restriction: with true maximum drift 0.48, radius 0.25 yields the feasible interval [0.039988,0.089684][0.039988,0.089684], missing truth 0.03454556; all five tested valid radii contain it (Figure 4(f), Appendix F.4; radius results).

W3: Terminology and exposition.

The introduction uses “historical observations” and “exact candidate answers.” Section 3/Table 2 define n,B,Kn,K^n,Kn,B,K_n,\widehat K_n,K before results. Section 4 and Figure 2 distinguish latent effects, the marginal interval, replay enlargement, and final confidence interval, separating population ambiguity from sampling uncertainty.

We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.

Thank you for the detailed response. Your explanations substantially address my concerns about the estimand, the scope of sharpness, and the historical information needed to justify the drift restriction. I also better understand the contributions of the two-source inference procedure and audit analysis beyond the existing population weighting framework. Although I retain some reservations about the extent of methodological novelty, the response has improved my assessment, and I will increase my score accordingly.

AuthorsOpenReview ↗

Thank you again for the increased score and evaluation

Thank you again for increasing your score, and thank you again for the time and care invested in these reviews. Very happy and honoured to contribute to the community together.

Reviewer_7hi5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author investigates how to estimate the causal effect of AI-generated outputs (e.g., which of two candidate answers was shown to a user) from historical logs, when the system that originally chose between them can no longer be reproduced. The authors argue that re-running today’s selection system does not recover how choices were made in the past, and show with a simple example that the historical effect can then be unidentifiable. They propose DRIFT, a sensitivity-analysis method that assumes the historical and current selectors differ by at most a bounded amount. Under that assumption, DRIFT gives bounds on the effect with finite-sample confidence guarantees and a rule for deciding where to spend effort auditing the historical system. The method is evaluated on synthetic data, a semi-synthetic study built on HelpSteer2, and a diagnostic on Open Bandit logs.

Critical strengths
  • The distinction between replaying the current selector and recovering the historical assignment law is real and practically relevant in real-world systems.
  • The authors clearly state the scope of what is claimed, and are candid about the limits of its evidence. The held-out evaluation is honest and transparent.
  • The fact that the finite-sample construction propagates both historical and replay certainty is a remarkable point of the overall design. The experiments do show that each of these two factors matter.
Critical weaknesses
  • Once Γ\Gamma is fixed, the identification step is literally the marginal sensitivity model with calibration constraints. The only change here is that the anchor is an externally measured qq, rather than eP(z)e_P(z). What remains new here is mostly inferential plumbing: the boxes are widened with a replay band, then the held-out dual scores are obtained with Bernstein bounds. These are pretty standard techniques. The paper needs to say clearly what DRIFT identifies or certifies, that MSM with an uncertain answer does not.
  • Presenting Theorem 3.1 and Theorem 3.2 as "two information boundaries" seems like overstating in my opinion. Both are quite easy to derive and do not carry much depth.
  • I can not get the idea behind Γ\Gamma. All guarantees treat it as fixed/pre-specified, or certified by audits independent of outcomes. I understand that the motivating scenario is one in which historical selection information is not there. However, the suggested sources for certification either makes the problem disappear or measure the current mechanism. Moreover, the audit framework assumes Bernoulli audits of theoretical selection in each cell. There is never a setting where these exist while pp itself is lost. In practice, Γ\Gamma is a parameter for sensitivity, and the method reduces to a standard sensitivity curve.
  • The baselines are weak. Both replay-IPW and cell-AIPW are point estimators, which are invalid under drift by construction, so beating them on coverage is not that informative. Try the percentile bootstrap (Zhao et al., 2019) or doubly-valid/sharp procedures (Dorn, Guo & Kallus, 2025).
  • EXP1-8 are, as the authors state, "analyzed from retained outputs because the historical runner is unavailable." This is a serious reproducibility problem.
  • The paper in this state is too compressed and relies heavily on coined terms ("information relocation", "provenance boundary", etc.) I understand that this is a consequence of LLM assistance. Most results are also deferred to the appendix, which is 65-page long. The abstract alone makes too many distinct technical claims. Much more human writing effort have to be put in a resubmission, if possible. To be fair, this is actually my main problem with the paper.
Questions for authors
  • What does DRIFT provide that the marginal sensitivity model with calibration does not, beyond the replay band, if qq is substituted for ePe_P? Can the authors give a setting where the two population intervals differ?
  • How should one choose Γ\Gamma without knowledge of the true drift? How sensitive are the conclusions to Γ\Gamma below the true shift (0.480.48 in HelpSteer2)?
  • The paper reports that every final certificate contains zero. Is there any setting in your experiments where the final certification excludes zero? Under what conditions would DRIFT be expected to decide the sign of the effect?
  • The target compares complete generated treatments within a randomized pre-display binary pair. How common is genuine within-pair randomization in real-world systems? How does the framework extend to larger slates which are deferred, beyond a designated pre-display pair and its selection law?
4 comments
AuthorsOpenReview ↗

Contribution, historical evidence, and sensitivity to drift

Thank you for your thoughtful feedback.

W1/Q1: What does DRIFT add to calibrated sensitivity analysis?

DRIFT provides a finite-sample certificate when outcomes come from a historical selector and replay measures a changed selector. Theorem 4.1 gives sharp bounds within the marginal weight model and outer validity for compatible latent effects. With identical observed law, anchor, radius, boxes, and balance restrictions, its population interval equals calibrated sensitivity bounds; Section 2 and Tables 1 and 20 state this equivalence.

Theorem 3.2 gives effects 1/21/2 and 1/61/6 with identical observable laws, exact q=1/2q=1/2, and aggregate calibration. The HelpSteer2 diagnostic complements this motivation: hidden association leaves Replay-IPW/Cell-AIPW mean errors 0.048/0.047 under marginal selector agreement. Theorem 5.1 propagates both data sources into

Pr⁡{I⊆I~⊆C}≥1−α,\Pr\left\lbrace\mathcal{I}\subseteq\widetilde{\mathcal{I}}\subseteq\mathcal{C}\right\rbrace\ge 1-\alpha,

where I\mathcal I is the marginal population interval, I~\widetilde{\mathcal I} its replay-band enlargement, and C\mathcal C the final confidence interval. Independent replay bands protect the weights; weak duality gives valid bounds for every returned bounded multiplier; held-out evaluation and Bernstein corrections protect score means. Conditions include iid history, fixed folds and radius, bounded outcomes/basis, nonempty population classes, independent population-valid replay bands, and deterministic bounds on inflated weights/scores. Approximate optimization preserves coverage, with a possible cost in width.

Proposition 5.3 gives a strictly feasible kink with a non-Gaussian A+∣H∣A+|H| endpoint limit; the finite-sample certificate still applies. DVDS also allows non-Gaussian estimators under its conditions. Under regularity, Theorem 5.2 separates covariance as VO/n+VR/BV_O/n+V_R/B, identifying the gains from history and replay. Section 6 derives audit value from width sensitivity sj=∂jWs_j=\partial_jW: radius uncertainty cj/mjc_j/\sqrt{m_j} and cost κj\kappa_j yield mj∝(sjcj/κj)2/3m_j\propto(s_jc_j/\kappa_j)^{2/3}. The convex allocation solution is standard; the contribution is its causal width criterion. Exact-width optimality needs differentiability; the plug-in rule needs a growing consistent independent pilot.

All 22,000 repetitions across 22 valid HelpSteer2 conditions satisfy the complete chain. The original anchor gives the following results (1,000 repetitions per row):

ConstructionComplete chainTruth coverageMean width
Full DRIFT100%100%0.5218
Replay point estimate only14.6%100%0.4823
No Bernstein corrections25.9%100%0.3055
Coarse-grid optimization100%100%0.5390

Coarse optimization retains the full-chain guarantee; scalar coverage alone hides failures (Appendices F.2/J.2). The ablations separately test replay and historical uncertainty corrections; coarse-grid training also tests robustness to approximate dual optimization.

W2: Role of Theorems 3.1–3.2.

Theorem 3.1 motivates the local contrast despite sparse replication of unstructured exact-answer means; Theorem 3.2 establishes historical nonidentification. Section 3 clarifies these roles, EXP1–3 illustrate them, and Sections 4–6 give the main inference and audit results.

W3/Q2: Choosing Γ\Gamma from historical evidence.

Separate old-version or overlap-period choice records can estimate historical cell probabilities after per-trace propensities or the executable system are lost, under representative sampling and mechanism stability. In homogeneous cells, simultaneous historical/replay bands upper-bound their log-odds gap; heterogeneous cells also require justified bounds on within-cell logit variation. Full recovery permits direct weighting; partial records support envelopes; current-only replay supports sensitivity analysis. Section 6 and Appendix E.2 specify these regimes. Pilot, audit, and replay data must be jointly independent of reserved history, with audit failure budgeted.

The added below-truth analysis fixes the original HelpSteer2 population and varies only Γ\Gamma. True drift is 0.48 and τh=0.034546\tau_h=0.034546. All five tested valid radii (0.48, 0.50, 0.60, 0.80, 1.00) contain truth. Radii 0, 0.10, and 0.20 are infeasible; 0.25 gives the feasible interval [0.039988, 0.089684], excluding truth; tested radii 0.30–0.45 contain truth despite an invalid bridge. Thus neither feasibility nor incidental truth containment certifies the radius. Figure 4(f), Appendix F.4, and Table 8 report all 13 deterministic population calculations: radius sensitivity.

We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.

AuthorsOpenReview ↗

Requested comparisons, sign decisions, practical scope, and reproducibility

Thank you for your thoughtful feedback.

W4: Zhao bootstrap and DVDS comparisons.

DRIFT retains the calibrated population bound while adding finite-sample protection for both history and replay. We added six matched settings (1,000 paired repetitions each; 18,000 method results), sharing historical samples, q=ePq=e_P, K=4K=4, and Λ=eΓ\Lambda=e^\Gamma. DRIFT additionally uses 10,000 independent replay draws. At nominal 95%, Zhao uses 999 refitted-propensity row bootstraps; DVDS uses asymptotic inference retaining arm covariance. Entries give common-set coverage (%) / mean untruncated final width.

OutcomennΓ\GammaDRIFTZhaoDVDS
Binary helpfulness ≥32,0000.6100/1.3458100/0.598795.7/0.4788
Binary helpfulness ≥310,0000.6100/0.7023100/0.554496.6/0.4277
Binary helpfulness ≥310,0001.0100/0.9326100/0.867695.1/0.6478
Helpfulness/42,0000.6100/1.108999.0/0.384096.1/0.3571
Helpfulness/410,0000.6100/0.540099.9/0.350795.4/0.3210
Helpfulness/410,0001.0100/0.7602100/0.564296.5/0.4982

DRIFT matches DVDS's population width and reduces Zhao's globally normalized population width by 9.2–26.8%. Its finite-sample construction attains 100% observed common-set coverage with wider final intervals; DVDS is narrower and near nominal here. All three cover every scalar truth. Figure 4(d,e), Appendix F.3, and Table 7 give Monte Carlo intervals and DVDS discrete-quantile checks: matched comparison. The matched comparison uses the common anchor q=ePq=e_P across methods; the original ablations evaluate the logit-shift setting. These comparisons distinguish population ambiguity from sampling uncertainty: the population interval describes the former, while the final certificate also accounts for estimating the selector and using finite history.

Q3: When can the final certificate decide the sign?

A positive population lower bound must exceed replay and historical-sampling corrections. The added direction study retains the full inference procedure and changes potential-rating distributions through prespecified strength ss. Pooling each pair's original distributions into FiF_i, we use Fi0=(1−s/2)Fi+(s/2)δ0F_{i0}=(1-s/2)F_i+(s/2)\delta_0, Fi1=(1−s)Fi+sδ1F_{i1}=(1-s)F_i+s\delta_1, coupled by a shared uniform rank. Entries count positive final certificates out of 100 at each nn, with B=10,000B=10{,}000, Γ=0.6\Gamma=0.6, K=4K=4.

LawTrue effectPopulation lower boundn=2,000n=2{,}000n=10,000n=10{,}000n=50,000n=50{,}000
Original0.034546−0.070074000
s=0s=00−0.079521000
s=0.1s=0.10.064614−0.014333000
s=0.4s=0.40.2584570.1821870100100
s=0.8s=0.80.5169140.46233587100100

All 1,500 intervals cover truth, including fallback runs. At s=0.4s=0.4, positive certificates rise from 0/100 at n=2,000n=2,000 to 100/100 at n=10,000n=10,000; the null yields none. Figure 4(g) and Appendix F.5 report the study and Monte Carlo uncertainty: sign decisions.

Q4: Randomized pairs and larger slates.

Bo et al., Sections 3.1–3.2 provide a precedent for randomized exposure to pre-generated LLM advice and downstream outcomes. This demonstrates feasibility. Our design requires a retained pre-display pair and a valid restriction on historical conditional selection; Section 3 now states this scope.

For m>2m>2, the target becomes a fixed policy value. Appendix I.4 assumes consistency, conditional randomization, and overlap; it supplies actionwise log-ratio envelopes and a normalized calculation. Extending the certificate would sum actionwise bounds, requiring simultaneous replay bands, bounded held-out scores, independence, and error allocation across actions/folds/tails. Ordered slates require joint propensities and overlap. This is an auxiliary extension; τh\tau_h requires a pre-display pair and its conditional selection law (Appendix A.2).

W5/W6: Reproducibility and presentation.

The release contains rerunnable primary and added HelpSteer2 studies, protocols, inputs or pinned reconstruction, all records, validators, and figure code. EXP8 includes archived source and executed data; EXP1–7 support saved-output audits. Hashes, Appendices F/J, and the evidence map connect claims to artifacts. The revised abstract introduces the target directly; Section 3/Table 2 define observations and notation; Section 4 separates the four interval objects; Figure 2 connects the results. Main Figure 4 presents the requested additional studies alongside the theory.

We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.

Thank you for your detailed response. I think that the additional explanations and experiments have, in parts, addressed the concerns I have for this paper. However, I am still with concerns regarding the methodological novelty of this model, as well as the writing (which I do not see can be fixed within a short time). Thus, I am inclined to keep my original score.

AuthorsOpenReview ↗

Thanks again for your review

Thank you again for the time and care invested in these reviews. Very happy and honoured to contribute to the community together. We fully implement these suggested revisions in the final version.

Reviewer_YdVVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

To the best of my understanding, the paper suggests a method to estimate treatment effects based on data generated by stochastic algorithms (namely, Drift-Robust Inference From Treatment Traces (DRIFT)). DRIFT tackles unobserved confounding in the data and, hence, yields a partial identification region. As a human (and not an LLM), I cannot provide further summary, as I could not understand the setting and the context of the work.

Critical strengths

The paper tries to develop a new method in a relatively new and relevant causal setting with data generated by stochastic algorithms [1].



  • [1] Susan Athey, Guido Imbens, and Zoe Ji. Estimating causal effects from data generated by stochastic algorithms. arXiv preprint arXiv:2607.05792, 2026.
Critical weaknesses

The paper suffers from severe clarity issues and seems to be low-quality LLM-generated content. I stopped reading carefully after Sec. 4 (as I got too frustrated), and here is a short list of critical issues:


  • Inconsistent terminology (almost every new sentence in both the abstract and the introduction (and beyond) introduces new terms, without referencing previous terminology)
  • The structure of the methods section is chaotic. First, the introduction already starts explaining tiny details of the method. Furthermore, the paper does not introduce the setting and immediately throws theoretical results in Sec. 3/4, without proper explanation of how all connect to each other
  • A lot of notation is not properly introduced: What are B−1B^{-1}, KnK_n, E=1E=1, qq, …?.
  • Multiple other issues. To name a few: too small plots to read from the printed version, odd terminology (what are provenance, audit, yesterday’s assignment law?)

Given the issues above, the paper does not meet the basic requirements for a scientific publication.


P.S. Just to demonstrate the scale of my confusion (and frustration): here is a copy of the abstract, where I marked the parts that were not clear to me (in bold):


Causal effects of generated treatments (generated by whom?) depend not only on which candidates (candidates as in observations?) were shown, but also on the historical mechanism that selected among them. When that mechanism is no longer reproducible (why is it not reproducible?), current replay (formal definition?) estimates today's selector (selector = assignment law?) rather than reconstructing yesterday's assignment law. We prove that even unlimited replay and exact aggregate calibration (how is calibration relevant here?) can leave distinct historical causal effects observationally indistinguishable. This provenance boundary (what is referred to here?) motivates Drift-Robust Inference From Treatment traces (DRIFT), which turns a prespecified or independently audited (what does this mean?) historical-to-current log-odds bridge into inverse-selection envelopes (every term here requires definition), observable calibration restrictions (definition?), and dual causal certificates (definition?). The resulting interval (= envelope?) is sharp for the specified marginal trace-weight model (first-time mentioned, requires explanation) and outer-valid (definition?) for every compatible latent historical effect (was it introduced before); at compatible zero drift (definition?) it reduces to the ordered binary-pair logged counterfactual exposure functional (how does this relate to the preceding part of the abstract). With bounded held-out dual scores (are those new, or they relate to DRIFT?) and independent population-valid replay bands, cross-fitting yields finite-sample outer confidence sets even under approximate optimization. Regular endpoints (endpoints = bounded scores = interval = envelope?) separate historical and replay uncertainty, whereas nonregular endpoints need not admit Gaussian inference (what is the meaning behind it?); on an explicit compact class, the minimax expected squared Hausdorff risk (why is this important?) is of order B−1+n−1B^{-1} + n^{-1} (what are those terms?). The same dual geometry (what is it? first time I hear about it) quantifies where historical audits can most reduce causal ambiguity (definition?) and yields a first-order cost-aware allocation rule (what does it mean?). A controlled HelpSteer2 study tests certificates with real ratings, constructed selectors, and potential-rating coupling (how does this relate to DRIFT?); predicted inclusions are observed in valid conditions. A fixed-action real-log diagnostic (I don’t know what that means) shows operation under injected mismatch without validating causal coverage (what is causal coverage?). Eight synthetic studies probe identification, inference, and audits: replay estimates today's selector (selector = assignment law?) but need not recover the historical selection law.

Questions for authors

It would be nice if the authors could answer the questions that I had regarding the abstract.

4 comments
AuthorsOpenReview ↗

The causal question, replay, and population bounds

Thank you for your thoughtful feedback.

DRIFT gives causal bounds under selector drift, with finite-sample coverage under approximate optimization.

W1–W3, Q1: Treatments, observations, and replay.

Consider a tutoring system that generates, ranks, and randomly displays two complete answers, then records learning. Each observation retains context, both answers, display, and outcome. We keep non-tied pairs (E=1E=1), label higher/lower answers 1/0, and denote display by DD. Thus

τh=E[Y(1)−Y(0)∣E=1]\tau_h=\mathbb E[Y(1)-Y(0)\mid E=1]

Here Y(1)Y(1) and Y(0)Y(0) are the two potential learning outcomes. This asks how outcomes change under the two fixed selection rules on the same historical pair population. Text varies, so the effect includes all content differences; a feature-specific effect needs additional design or assumptions.

The selector is the assignment mechanism. Given saved context/pair ZZ and unretained state UU, historical display is randomized independently of potential outcomes with probability p(Z,U)p(Z,U). Replay estimates q(Z)q(Z) by rerunning the current selector; updates, filters, or missing state may block recovery. Here h,E,Dh,E,D denote ordering, eligibility, and display; n,Bn,B are history/replay sizes; Kn,K^nK_n,\widehat K_n are possible/observed candidates, and KK is the number of balance functions.

Theorem 3.1 shows that the local contrast achieves O(1/n)O(1/n) mean squared error with known historical probabilities and fixed overlap, even when KnK_n scales with nn and the unrestricted mean surface is inconsistent; Figure 3(a,b) illustrates this separation: synthetic evidence.

Q1: Aggregate calibration and the historical information limit.

Aggregate calibration, E[D∣q]=q\mathbb E[D\mid q]=q, matches average display frequencies to replay probabilities but leaves their association with potential outcomes unresolved. Theorem 3.2 constructs two calibrated worlds with the same observable law and exact q=1/2q=1/2, but effects 1/21/2 and 1/61/6; every estimator therefore has worst-world error at least 1/61/6, even with unlimited replay. Historical evidence or sensitivity must constrain this missing relationship.

Q1: Scope of the target and its identifying information.

The intervention changes which answer from a retained pair is displayed. Conditioning on E=1E=1 fixes the eligible population before display. Causal interpretation uses consistency for the two complete-answer policies and conditional randomization under the historical selector. If contexts or assignment traces are missing, current replay cannot recover them; Γ\Gamma or independently recovered propensities must bridge the historical and current selectors.

Q1: What the balanced interval certifies.

The balance equations match weighted arm observations to prespecified basis averages. Theorem 4.1 is sharp for the marginal feasible set and contains effects from every compatible latent law. Additional scientifically justified moments can tighten both armwise sets. Thus KK determines moment resolution, while outer validity retains compatible effects.

Q1: Drift, weights, balance, and sharpness.

The historical-to-current log-odds restriction is

∣logit⁡p−logit⁡q∣≤Γ.|\operatorname{logit}p-\operatorname{logit}q|\le\Gamma.

It gives pL=expit⁡(logit⁡q−Γ)p_L=\operatorname{expit}(\operatorname{logit}q-\Gamma) and pU=expit⁡(logit⁡q+Γ)p_U=\operatorname{expit}(\operatorname{logit}q+\Gamma). The inverse-weight envelopes are (1/pU,1/pL)(1/p_U,1/p_L) for arm 1 and (1/(1−pL),1/(1−pU))(1/(1-p_L),1/(1-p_U)) for arm 0. Observable calibration requires

E[IawϕK(Z)]=E[ϕK(Z)],Ia=1D=a.\mathbb E[I_aw\phi_K(Z)]=\mathbb E[\phi_K(Z)],\quad I_a=\mathbf{1}_{D=a}.

The basis includes an intercept and context functions, so weighted arm observations recover eligible-population averages. Optimizing E[IawY]\mathbb E[I_awY] over bounded, balanced weights gives arm-mean bounds; subtracting opposite bounds gives I\mathcal I.

Theorem 4.1 gives sharp bounds for this marginal weight model: both endpoints of I\mathcal I are attainable. Every compatible latent historical law has its effect inside I\mathcal I (outer validity). Separate arm weights may admit additional effects, so latent-model sharpness needs compatibility; one additional valid balance moment contracts [1/3,2/3][1/3,2/3] to {1/2}\{1/2\}.

At zero drift, p=qp=q, the weights collapse and Corollary 4.2 recovers eligible-pair IPW. The positive-display LCE target additionally conditions on D=1D=1; it equals τh\tau_h when Cov⁡(p,Y(1)−Y(0)∣E=1)=0\operatorname{Cov}(p,Y(1)-Y(0)\mid E=1)=0, including constant assignment probability. Revised Equation (2) states this relationship.

We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.

AuthorsOpenReview ↗

Confidence guarantees, historical audits, and experimental evidence

Q1: Dual scores, endpoints, and causal coverage.

Dual scores are observable functions certifying interval endpoints (lower or upper boundaries), even under approximate optimization. Write the arm's weight limits as ℓa,ua\ell_a,u_a. For r=Y−λ⊤ϕK(Z)r=Y-\lambda^\top\phi_K(Z), r+=max⁡(r,0)r_+=\max(r,0), and r−=min⁡(r,0)r_-=\min(r,0), Equation (6) gives the lower-arm score

ga,λ−=λ⊤ϕK+Ia(ℓar++uar−).g^{-}_{a,\lambda}=\lambda^{\top}\phi_{K}+I_{a}(\ell_{a}r_{+}+u_{a}r_{-}).

Exchanging ℓa,ua\ell_a,u_a gives the upper score. Their expectations bound the arm endpoints for every multiplier λ\lambda. DRIFT learns bounded multipliers on training folds, freezes them, and evaluates the scores on held-out historical observations. Independent replay confidence bands enlarge the weight limits; empirical Bernstein corrections protect the held-out score means.

ObjectMeaning
I\mathcal IPopulation effect interval using true replay probability.
I~\widetilde{\mathcal I}Population interval after enlarging weights for replay uncertainty.
C\mathcal CFinal interval including historical sampling corrections.

Theorem 5.1 gives Pr⁡{I⊆I~⊆C}≥1−α\Pr\left\lbrace\mathcal{I}\subseteq\widetilde{\mathcal{I}}\subseteq\mathcal{C}\right\rbrace\ge 1-\alpha. These nested inclusions protect the entire marginal population interval, beyond coverage of a single causal effect. Its conditions include iid history, fixed folds and radius, bounded outcomes and basis, nonempty population weight classes, independent population-valid replay bands, and deterministic bounds on inflated weights and returned scores. Weak duality preserves coverage under approximate optimization, with suboptimality affecting tightness. Revised Section 5 makes the resulting causal coverage explicit.

Q1: Regularity and the two sample sizes.

Under Theorem 5.2's smoothness and optimizer-stability conditions, endpoint covariance is VO/n+VR/BV_O/n+V_R/B. On the stated compact class, squared-Hausdorff minimax risk is n−1+B−1n^{-1}+B^{-1}; both samples matter even when population width remains positive. At a kink, Proposition 5.3 yields non-Gaussian A+∣H∣A+|H| for independent centered Gaussians. Section 5 explains why normal approximations can fail while concentration and weak duality retain finite-sample coverage.

Q1: Independently audited drift and cost-aware allocation.

A historical audit uses a restored selector with relevant state or separate matched records of its choices. Simultaneous bands bound the historical/current log-odds gap in homogeneous cells; heterogeneous cells additionally need bounds on within-cell variation. Pilot, audit, and replay information must be jointly independent of the reserved historical inference sample, with audit failure budgeted. Section 6 and Appendix E.2 now specify these requirements. With current-only replay, Γ\Gamma is a sensitivity parameter; recovered historical probabilities also permit direct weighting.

The dual geometry gives sj=∂W/∂Γjs_j=\partial W/\partial\Gamma_j: the sensitivity of interval width (causal ambiguity) to cell jj's radius. If radius uncertainty falls as cj/mjc_j/\sqrt{m_j} at unit cost κj\kappa_j, the first-order width-minimizing allocation is mj∝(sjcj/κj)2/3m_j\propto(s_jc_j/\kappa_j)^{2/3}. Section 6 states the differentiability and growing-pilot conditions: the rule prioritizes causal width, uncertainty, and cost.

W4; Q1: Experimental meaning and readable figures.

DRIFT achieves 22,000/22,000 chains across 20 valid positive-radius conditions and two zero-drift checks. HelpSteer2 supplies 10,626 prompts with two responses and human ratings each; histories sample 6,376 confirmation pairs with replacement. Constructed selectors, hidden state, and potential-rating coupling make truth known.

At Γ=0.6,K=4,η=0.6\Gamma=0.6,K=4,\eta=0.6, each setting below has 1,000 repetitions and 100% chain success. Widths are untruncated.

nnBBPopulation widthMean final width
2,00010,0000.26691.1102
10,0001,0000.26690.5899
10,00010,0000.26690.5218
50,00010,0000.26690.3741

Increasing n:2,000→50,000n:2,000\to50,000 at B=10,000B=10,000 cuts mean width by 66.3%; increasing B:1,000→10,000B:1,000\to10,000 at n=10,000n=10,000 cuts it by 11.5%, with unchanged population width. The width decomposition separates the sources. A coarse-grid optimizer preserves all 1,000 chains at mean width 0.5390 versus 0.5218. The controlled direction study gives 100/100 positive lower endpoints at n=10,000n=10,000, τh=0.258457\tau_h=0.258457, and 0/100 under the null; all 1,500 truths are covered.

Open Bandit uses fixed actions, injected mismatch, and simulated audits; HelpSteer2 tests generated pairs. Section 7 explains these roles and the semi-synthetic scope. Full-width Figures 3–4 separate synthetic and HelpSteer2 results.

We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.

Thank you for your response!

Unfortunately, the provided answers seem to cherry-pick my initial comments and only address them (I even struggle to match the weaknesses mentioned in the review to the provided answers). Also, I looked over the newly updated abstract, and I am still as confused as I initially was.

I don't think this paper is accessible to anyone (who is not an LLM); and, therefore, I definitely retain my current score.

AuthorsOpenReview ↗

Thanks you again for the review

Thank you again for the time and care invested in these reviews. Very happy and honoured to contribute to the community together. We fully implement these suggested revisions in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewer Concerns

General Response to the Area Chair and Reviewers

We thank the AC and reviewers for their careful evaluation and constructive suggestions. We have provided point-by-point responses and revised the manuscript, with changes marked in blue. The main revisions are summarized below.

1. Improving clarity and organization

We rewrote the abstract, introduced the historical experiment and key definitions before the theoretical results, added a notation table, and reorganized the figures to clarify the connections among identification, inference, and experiments.

Related comments: YdVV–W1–W4/Q1, 7hi5–W6, uTYH–W3.

2. Clarifying contributions relative to existing methods

We acknowledged the equivalence to calibrated sensitivity bounds when the population inputs and constraints match. We clarified DRIFT’s contributions in inference with separate historical and replay samples, finite-sample coverage under approximate optimization and nonregular endpoints, uncertainty decomposition, and audit allocation.

Related comments: 7hi5–W1–W2/Q1, uTYH–W1.

3. Clarifying the estimand and scope of sharpness

We explained the intervention as selecting between complete generated answers over the historical eligible-pair population, derived its relationship to the positive-display LCE target, and stated when the targets coincide. We also distinguished latent causal effects, marginal bounds, replay enlargement, and confidence intervals, and demonstrated how additional balance closes an illustrative relaxation gap.

Related comments: uTYH–W4/Q1/Q3, YdVV–Q1.

4. Making historical evidence and audit requirements explicit

We specified when matched historical choice records or recoverable selectors and states can support a drift bound Γ\Gamma, including coverage, within-cell variation, independence, and audit-error requirements. We clarified when Γ\Gamma remains a sensitivity parameter, when recovered propensities permit direct inference, and the assumptions underlying audit-allocation optimality.

Related comments: 7hi5–W3/Q2, uTYH–W2, YdVV–Q1.

5. Strengthening empirical evaluation

We added comparisons with Zhao’s percentile bootstrap and DVDS, reporting coverage and width under a common evaluation target. We also added a drift-radius misspecification study and controlled effect-direction experiments showing when final confidence intervals exclude zero. The revised discussion explains both the coverage–width tradeoff and the scope of the semi-synthetic evidence.

Related comments: 7hi5–W4/Q2–Q3, uTYH–Q2, YdVV–W4.

6. Addressing practical relevance and reproducibility

We clarified the relationship to existing LLM evaluation methods and discussed the additional requirements for larger candidate sets in our responses. We also expanded the documentation of experimental code, validation procedures, and reproduction commands.

Related comments: 7hi5–W5/Q4, uTYH–Q2.

We appreciate the opportunity to clarify these points and welcome any further questions from the AC and reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.