- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies causal identification and inference in a setting that builds on the logged counterfactual exposure (LCE) framework developed by Athey et al. (2026). It considers a practically relevant departure from the original framework: the available replay mechanism differs from the historical mechanism that generated the observed assignments, and the historical assignment probabilities are unknown. The paper adapts existing sensitivity-analysis and partial-identification techniques to construct bounds on the causal effect of interest under restrictions on the discrepancy between the two mechanisms. It also develops confidence sets that account for uncertainty in historical observations and replay data.
Critical strengths
-
Well-motivated problem. The paper addresses an important practical limitation of replay-based LCE. In deployed applications, the selection mechanism may change over time, and the information needed to reproduce historical assignments may no longer be available. Consequently, replaying the current mechanism need not recover the probabilities relevant to historical outcomes. I find this problem meaningful and well motivated.
-
Coverage guarantees do not require exact optimization. An appealing property of Theorem 5.1 is that the proposed confidence-set construction does not require exact optimization for its coverage guarantee. Numerical suboptimality may widen the interval without invalidating coverage. This is a useful feature for an optimization-based inference procedure and strengthens its practical appeal.
-
Timely research direction. LCE is a recent framework, and studying its limitations and extensions is a worthwhile research direction. The paper engages with questions relevant to evaluating generated treatments, where conventional approaches to defining and estimating treatment effects may be difficult to apply.
Critical weaknesses
-
Limited novelty of the central identification approach. My impression is that the paper’s core identification method is primarily an adaptation of existing sensitivity-analysis and partial-identification techniques to the LCE setting. Although this adaptation may be useful, the manuscript would benefit from a clearer separation between established methodological ingredients and genuinely new contributions. In particular, it should explain more explicitly which results go beyond applying existing calibrated-weight methods to the ordered-pair comparison.
-
The practical resolution of the historical-information problem remains unclear. The proposed interval depends on a prespecified or independently justified bound on the discrepancy between the historical and current assignment mechanisms. However, when information about the historical mechanism has been lost—the setting that motivates the paper—it is not clear how an informative bound can be credibly chosen or certified. Sensitivity analysis is a reasonable response to this problem, but the paper should explain more concretely what additional historical information is needed and when that information is realistically available.
-
The exposition could be clearer. I found the presentation difficult to follow in several places, particularly when terminology is introduced without sufficiently explicit definitions. For example, the second paragraph of the introduction refers to “n interactions over K_n identities. The meanings of “interactions” and “identities” are not immediately clear to readers familiar with the causal inference literature. More direct terminology, or explicit definitions in terms of observations and distinct treatment variants, would improve accessibility. More generally, the paper would benefit from clearer explanations of the statistical objects before presenting the associated results.
-
The sharpness claim applies to a relaxation. The interval is sharp for the specified marginal trace-weight model, but it is only an outer bound for the original latent historical model. The paper acknowledges that separate armwise optimization can widen the interval. This distinction limits the scope of the sharpness result. It would be helpful to explain when the relaxation is exact and how much additional width it can introduce, so that readers can distinguish uncertainty inherent in the original model from conservativeness introduced by the relaxation.
Questions for authors
-
How does your estimand relate to that of Athey et al.? Your target is E[Y(1)-Y(0) | E=1], whereas Athey et al. consider a formulation of the form E[tau | h(V,V^N)>0]. Could you clarify the relationship between these quantities and the conditions under which they coincide? In particular, when assignment probabilities vary across candidate pairs, does conditioning on the positive display orientation change the population over which effects are averaged?
-
How does the proposed framework relate to current LLM evaluation methods? Both this paper and LCE develop causal approaches relevant to evaluating LLM-generated outputs. It would be helpful to explain how these approaches relate to evaluation methods currently used in leading research and practical deployments. What substantive questions do existing methods fail to answer, and what additional information or decision-making value does the causal approach provide?
-
Could you give a more substantive interpretation of the estimand? I have some reservations about the interpretation of this estimand, including the related estimand in Athey et al. In particular, how does an average comparison between interaction-specific candidate treatments relate to conventional causal estimands comparing two common, well-defined interventions? Does the estimand describe the effect of choosing between complete generated treatments, rather than the effect of changing a particular treatment characteristic while holding other characteristics fixed? I appreciate that the setting may prevent identification of more narrowly defined causal effects without stronger assumptions. Nevertheless, clearly explaining this conceptual distinction—and what counterfactual question the proposed estimand does and does not answer—would substantially improve the paper.
4 comments
Additional contributions, historical evidence, and clearer exposition
W1: Beyond calibrated-weight methods.
DRIFT recovers the calibrated population program for matched , boxes, and balance constraints. It adds a two-source finite-sample certificate and a way to value historical audits when replay measures a changed selector. Section 2 clarifies these contributions and credits established optimization and concentration tools.
Theorem 3.2 shows why historical selection–outcome dependence must be recovered or constrained: effects of and yield identical observable laws and exact replay, even under aggregate calibration.
Under iid history, fixed folds and radius, bounded outcomes and basis, nonempty weight classes, and deterministic bounds on inflated weights and scores, Theorem 5.1 gives An independent population-valid replay band enlarges the marginal population interval to ; held-out historical corrections give . Approximate optimization preserves coverage but may widen the interval.
Experiments check both inclusions separately; all 22,000 repetitions across 22 valid HelpSteer2 conditions satisfy the full chain. Component checks at , , and are:
| Construction | Complete chain | Final-set coverage | Scalar coverage |
|---|---|---|---|
| Full DRIFT | 100% | 100% | 100% |
| Replay point estimate without band | 14.6% | 100% | 100% |
| Without Bernstein corrections | 25.9% | 95.5% | 100% |
| Coarse-grid dual optimization | 100% | 100% | 100% |
The ablations separate protection of the whole identified set from coverage of one causal value, showing why scalar coverage alone cannot verify the full chain. Each row has 1,000 runs; final-set coverage is . The link diagnostics show failed inclusions and violation sizes. Coarse-grid training preserves all chains at widths 0.5390 versus 0.5218 (Appendices F.2/J.2).
Under Theorem 5.2's regularity conditions, uninflated estimates have asymptotic covariance ; its stated compact class has minimax squared-Hausdorff risk . At Proposition 5.3's nonregular endpoint, the limit is for independent centered Gaussians; the finite-sample certificate still applies. Section 6 further differentiates population width, , to value audits. Radius uncertainty and cost yield . These width sensitivities make the standard allocation rule responsive to causal precision. Exact-width optimality needs differentiability; its plug-in form needs a growing independent consistent pilot. Sections 5–6 thus guide investment in history, replay, or audits.
W2: Partial historical information can support an informative radius.
Separate old-version or overlap-period choice records estimate historical cell probabilities even after the executable system and trace-level propensities are lost. This requires matched pairs and contexts, representative sampling, a stable historical mechanism, and homogeneous cells. Recovered configuration or state supports envelopes when its effect on selection probabilities is justified; full propensity recovery permits direct weighting.
Appendix E.2 now specifies these information regimes. Given simultaneous historical and replay bands , , homogeneous-cell certification uses
Heterogeneous cells require additional justified bounds on within-cell log-odds variation. Pilot, audit, and replay information must be jointly independent of history reserved for training and evaluation, with audit failure included in the error budget. Current-only replay supports assumed-radius sensitivity analysis. The radius study demonstrates the value of a valid historical restriction: with true maximum drift 0.48, radius 0.25 yields the feasible interval , missing truth 0.03454556; all five tested valid radii contain it (Figure 4(f), Appendix F.4; radius results).
W3: Terminology and exposition.
The introduction uses “historical observations” and “exact candidate answers.” Section 3/Table 2 define before results. Section 4 and Figure 2 distinguish latent effects, the marginal interval, replay enlargement, and final confidence interval, separating population ambiguity from sampling uncertainty.
We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.
Thank you for the detailed response. Your explanations substantially address my concerns about the estimand, the scope of sharpness, and the historical information needed to justify the drift restriction. I also better understand the contributions of the two-source inference procedure and audit analysis beyond the existing population weighting framework. Although I retain some reservations about the extent of methodological novelty, the response has improved my assessment, and I will increase my score accordingly.
Thank you again for the increased score and evaluation
Thank you again for increasing your score, and thank you again for the time and care invested in these reviews. Very happy and honoured to contribute to the community together.
The causal target, its evaluation value, and the sharpness result
Thank you for your feedback.
Q1: Positive-display conditioning changes the target population.
DRIFT averages over all eligible pairs. Let E=1 denote a non-tied pair, D=1 higher-answer display, p its historical probability, and Δ=Y(1)−Y(0). For the same ordered pairs and contrast, historical conditional randomization gives
\tau_+=\frac{\mathbb E[p\Delta\mid E=1]}{\mathbb E[p\mid E=1]},\quad \tau_+-\tau_h=\frac{\operatorname{Cov}(p,\Delta\mid E=1)}{\mathbb E[p\mid E=1]}.$$ Thus positive-display conditioning reweights pairs and historical states by $p$. The targets coincide precisely when this covariance is zero; constant $p$ is sufficient. Even with $q=p$, two equally likely types with $(p,\Delta)=(1/4,1)$ and $(3/4,0)$ have $\tau_h=1/2$ and $\tau_+=1/4$. Revised Equation (2) gives this identity and example. DRIFT retains $\tau_h$ throughout, including at compatible zero drift. **Q3: The interventions are two common answer-selection rules.** Each pair uses the same prespecified rule: display its higher-ranked versus lower-ranked complete answer. Texts vary; the rules remain common. Pair generation precedes display; the historical generator and eligible-pair population stay fixed. The effect compares applying these rules to all eligible pairs. For example, ranking tutoring answers by explanatory detail and measuring subsequent retention estimates the effect of selecting the complete more-detailed answer, including its accompanying content differences. Isolating detail alone requires controlled edits or additional identifying assumptions. Revised Section 3 and Appendix A separate the intervention, population, and outcome explicitly. **Q2: Decision-making value alongside existing LLM evaluation.** MT-Bench measures LLM-judge scores, Chatbot Arena measures human preferences, and randomized online A/B tests identify effects of assigned systems or policies on measured outcomes. DRIFT adds a causal certificate for historical answer-selection decisions when assignment probabilities are unavailable. A positive lower confidence endpoint supports selecting the higher-ranked answer for that historical population under the drift and coverage assumptions. Measured retention, correctness, or helpfulness determines what improvement means; the ranking rule defines the comparison. Section 2 clarifies this value and the transport assumption needed for another generator or population. The added controlled HelpSteer2 study demonstrates this decision with the full inference procedure, fixed selectors, $\Gamma=0.6$, $K=4$, and $B=10{,}000$. Entries below count positive lower confidence endpoints out of 100 repetitions. | Constructed outcome law | $\tau_h$ | Population lower endpoint | $n=2{,}000$ | $n=10{,}000$ | $n=50{,}000$ | | --- | --- | --- | --- | --- | --- | | Original | 0.034546 | −0.070074 | 0 | 0 | 0 | | Null | 0 | −0.079521 | 0 | 0 | 0 | | $s=0.1$ | 0.064614 | −0.014333 | 0 | 0 | 0 | | $s=0.4$ | 0.258457 | 0.182187 | 0 | 100 | 100 | | $s=0.8$ | 0.516914 | 0.462335 | 87 | 100 | 100 | All 1,500 intervals cover truth. At $s=0.4$, DRIFT certifies a positive effect in 100/100 runs at $n=10,000$; the null yields none. These controlled alternatives demonstrate useful sign decisions; the original population remains sign-ambiguous under the marginal model (Figure 4(g), Appendix F.5; [full results](https://anonymous.4open.science/r/ICLR2027-rebuttal-01-F15F/fig3.pdf). **W4: Sharpness and the gains from calibration.** Theorem 4.1 establishes sharpness within the calibrated marginal weight model and containment of all compatible latent effects. Separate armwise weights can enlarge the interval because a latent completion additionally requires one common historical state, assignment probability, and joint potential-outcome law. Section 4 now quantifies this distinction. For $Z\sim\mathrm{Bernoulli}(1/2)$, $p=q=1/2$, $Y(1)=Z$, $Y(0)=0$, and $\Gamma=\log3$, the weight boxes are $[4/3,4]$: | Balance | Marginal interval | Latent set | Extra width | | --- | --- | --- | --- | | Intercept | $[1/3,2/3]$ | $\{1/2\}$ | $1/3$ | | Intercept and $Z$ | $\{1/2\}$ | $\{1/2\}$ | 0 | Positivity and the observed outcomes force $Y(1)=Z,Y(0)=0$ in every latent completion. Adding $Z$ fixes $\mathbb E[DwZ]=1/2$, closing the entire gap. Compatible zero drift is another exact case. Generally, nested valid moments tighten the marginal interval; latent equality requires compatibility. At the HelpSteer2 anchor, $K=1\to4$ reduces population/final widths by 10.7%/5.1%, retaining all 1,000 chains per condition (Appendix F.2). Section 7 also reports EXP5's wider final certificates: more balance can reduce population ambiguity without ensuring a narrower final confidence interval, making the moment-resolution tradeoff explicit. We are eager to know whether we have addressed your questions, and we would be happy to discuss further at any time to assist with your re-evaluation.