- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether action effects estimated through trajectory replay accurately reflect those observed when an LLM agent naturally reaches the same intervention point. The authors formalize the discrepancy as "replay transport bias" and show that, under an action–response factorization assumption, a three-arm observation setting without natural policy control leaves the bias partially unidentified. To address this, they propose Replay Transport Audit (RTA), a four-arm evaluation protocol that directly measures repair and policy-control outcomes under both replay and natural arrival. Experiments on ALFWorld and WebShop show that complete replay generally agrees with natural-arrival effects at the aggregate level, while deliberately omitting policy history or hidden environment state can introduce substantial discrepancies.
Critical strengths
- The paper studies the reliability of replay-based action-effect estimation, which is relevant to failure attribution and counterfactual evaluation of LLM agents. This is a meaningful research question.
- The assumptions in the paper are stated explicitly and honestly. The authors are generally careful about the scope of their claims and acknowledge their theoretical and empirical findings.
- The paper includes a substantial number of experiments across environments, policies, and reconstruction conditions, along with detailed documentation of statistical analysis and experimental protocols.
Critical weaknesses
- The paper explicitly defines a three-arm setting where the natural policy-control outcomes are missing. This is a mathematically legitimate observation setting, but I'm not convinced that this is a common restriction in existing agent evaluation workflows. Given that the paper is heavily built on top of this setting, its practical necessity and prevalence need to be better motivated.
- The largest experimental discrepancies result from deliberately introduced omissions. In the resettable agent environment discussed, collecting natural policy-control outcomes is conceptually straightforward.
- The method proposed, RTA, appears to be an experimental protocol fix (direct four-arm intervention comparison), rather than a technically novel method. Therefore, the novelty lies mostly in the replay-to-natural transport problem formulation. However, the problem formulation appears to be incremental relative to existing replay fidelity studies (https://arxiv.org/abs/2608.08239).
- The practical benefits attributable specifically to RTA remain unclear. While the downstream correction experiments show improvements over RL-only training, Live RTA does not establish a statistically significant advantage over uniform correction weighting (Table 2; Table 19). Comparisons with Raw Replay are confounded by differences in correction weight mass and coverage, making it difficult to isolate the contribution of live-effect estimation.
Questions for authors
See weaknesses
Thank you for the detailed assessment. Your comments raise important questions about when the three-arm setting applies and what practical benefit can be attributed to the audit.
Section 2.2 defines an observation setting, not a restriction that all agent-evaluation workflows face. We observe natural repair outcomes F_N, replay repair/control outcomes F_R,C_R, and natural action probabilities, but not natural policy-control outcomes C_N.
One example in that section is a live procedure that queries the policy's action but always executes the repair. It records action probabilities without observing the unmodified continuation. Existing logs can supply C_N if they represent the same policy, arrival rule, complete-context law, and continuation protocol. The result therefore depends on what has been observed; it does not require C_N to be difficult to collect or assume that three-arm workflows are common.
Under Proposition 2.1's factorization assumption, with alternative-action responses unrestricted in [0,1], the sharp interval has width 1-alpha_N. This describes the uncertainty left by the missing control. If suitable live control is already available, that restriction no longer applies.
The WebShop probe tests sensitivity to reconstruction. With a backend option omitted, the aggregate repair-versus-alternative effect differs from natural arrival by -0.1651, even though the observation, action menu, prompt, and token IDs remain identical (Section 3.4; Table 11). This does not measure how often such omissions occur in ordinary workflows.
In contrast, complete replay agrees with natural effects at the measured ALFWorld panel level: Table 1 gives biases of +0.0056 and -0.0034, with joint intervals inside +/-0.10. The experiments therefore show both agreement under complete reconstruction and sensitivity to particular omissions, rather than a general failure of replay.
The theorem does not assume that collecting C_N is difficult. When suitable live repair/control outcomes are available, F_N-C_N identifies the natural effect under the stated assumptions. RTA adds a comparison with replay; it does not offer an established efficiency advantage over direct live evaluation (Sections 2.2, 2.4, and 5).
RTA is an audit protocol, not a new optimizer. Under Assumption A.2, the four-arm comparison measures b=(F_R-C_R)-(F_N-C_N) without requiring the factorization used in the three-arm result. The contribution here is specifying which observations identify the replay-to-natural discrepancy, rather than proposing a generally better learning algorithm.
We do not claim that reconstruction fidelity is a new concern. Our formulation focuses on whether an action effect in a reconstructed context agrees with the effect conditional on natural arrival. The paper defines that target, derives a sharp missing-control identification interval under explicit assumptions, and measures the discrepancy with four arms on supported populations (Sections 2.1-2.5). This is the scope of our contribution, not a claim of technical superiority over the related study you cite.
In the calibrated-pool study, Table 2 reports gains over RL-only of 3.47 points for Gemma and 14.35 for Qwen. However, both live-minus-uniform 95% intervals include zero (Table 19), so these results do not establish an added benefit from live-effect weighting.
Raw Replay has higher point estimates, but its correction weight mass and coverage differ. That comparison cannot isolate the effect of the weighting rule. The correction pipeline improves over RL-only in this setting, but the evidence does not show that live weighting is better than simpler alternatives. Checking whether replay agrees with natural execution and improving the downstream training signal remain separate questions.