acceptodds
Desk rejected at ICLR 2027

From Replay to Natural Arrival: Identifying Agent Action Effects

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Replay enables researchers to evaluate agent action repairs by reconstructing a trajectory and comparing a replacement action with the agent's own continuation. Whether this comparison captures the repair's benefit during natural execution, however, depends on what reconstruction preserves and what the experiment observes. We study when replay evidence identifies natural-arrival action effects. For an observation setting containing natural repair outcomes and replay repair/control outcomes, but no natural policy-control outcomes, we derive a sharp bias interval of width 1-α_N under an action–response factorization assumption, where α_N is the natural probability of selecting the repair. Additional replay samples cannot resolve this missing-control uncertainty. We introduce Replay Transport Audit (RTA), which adds independent live-control measurements to identify the bias under stated within-context sampling conditions, without requiring that factorization. In ALFWorld, complete replay agrees with natural-arrival effects within an absolute panel-level tolerance of 0.10. In WebShop, removing a backend option shifts the repair-versus-alternative effect by −0.165 on a [0,1] score scale despite identical visible inputs. In a separate calibrated-pool training study, live-audited corrections improve success over RL-only by 3.47 and 14.35 percentage points on Gemma4-31B and Qwen3.6-27B, respectively. These findings clarify which action-effect claims replay supports and which require additional live observations or assumptions about reconstruction.

Primary area:
causal reasoning
Keywords:
Agent evaluation, Counterfactual replay, Action effects, Partial identification, Transportability, Language-model agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9aYTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether action effects estimated through trajectory replay accurately reflect those observed when an LLM agent naturally reaches the same intervention point. The authors formalize the discrepancy as "replay transport bias" and show that, under an action–response factorization assumption, a three-arm observation setting without natural policy control leaves the bias partially unidentified. To address this, they propose Replay Transport Audit (RTA), a four-arm evaluation protocol that directly measures repair and policy-control outcomes under both replay and natural arrival. Experiments on ALFWorld and WebShop show that complete replay generally agrees with natural-arrival effects at the aggregate level, while deliberately omitting policy history or hidden environment state can introduce substantial discrepancies.

Critical strengths
  • The paper studies the reliability of replay-based action-effect estimation, which is relevant to failure attribution and counterfactual evaluation of LLM agents. This is a meaningful research question.
  • The assumptions in the paper are stated explicitly and honestly. The authors are generally careful about the scope of their claims and acknowledge their theoretical and empirical findings.
  • The paper includes a substantial number of experiments across environments, policies, and reconstruction conditions, along with detailed documentation of statistical analysis and experimental protocols.
Critical weaknesses
  • The paper explicitly defines a three-arm setting where the natural policy-control outcomes are missing. This is a mathematically legitimate observation setting, but I'm not convinced that this is a common restriction in existing agent evaluation workflows. Given that the paper is heavily built on top of this setting, its practical necessity and prevalence need to be better motivated.
  • The largest experimental discrepancies result from deliberately introduced omissions. In the resettable agent environment discussed, collecting natural policy-control outcomes is conceptually straightforward.
  • The method proposed, RTA, appears to be an experimental protocol fix (direct four-arm intervention comparison), rather than a technically novel method. Therefore, the novelty lies mostly in the replay-to-natural transport problem formulation. However, the problem formulation appears to be incremental relative to existing replay fidelity studies (https://arxiv.org/abs/2608.08239).
  • The practical benefits attributable specifically to RTA remain unclear. While the downstream correction experiments show improvements over RL-only training, Live RTA does not establish a statistically significant advantage over uniform correction weighting (Table 2; Table 19). Comparisons with Raw Replay are confounded by differences in correction weight mass and coverage, making it difficult to isolate the contribution of live-effect estimation.
Questions for authors

See weaknesses

1 comment

Thank you for the detailed assessment. Your comments raise important questions about when the three-arm setting applies and what practical benefit can be attributed to the audit.

  1. When the three-arm setting applies

Section 2.2 defines an observation setting, not a restriction that all agent-evaluation workflows face. We observe natural repair outcomes F_N, replay repair/control outcomes F_R,C_R, and natural action probabilities, but not natural policy-control outcomes C_N.

One example in that section is a live procedure that queries the policy's action but always executes the repair. It records action probabilities without observing the unmodified continuation. Existing logs can supply C_N if they represent the same policy, arrival rule, complete-context law, and continuation protocol. The result therefore depends on what has been observed; it does not require C_N to be difficult to collect or assume that three-arm workflows are common.

Under Proposition 2.1's factorization assumption, with alternative-action responses unrestricted in [0,1], the sharp interval has width 1-alpha_N. This describes the uncertainty left by the missing control. If suitable live control is already available, that restriction no longer applies.

  1. What the deliberately introduced omissions show

The WebShop probe tests sensitivity to reconstruction. With a backend option omitted, the aggregate repair-versus-alternative effect differs from natural arrival by -0.1651, even though the observation, action menu, prompt, and token IDs remain identical (Section 3.4; Table 11). This does not measure how often such omissions occur in ordinary workflows.

In contrast, complete replay agrees with natural effects at the measured ALFWorld panel level: Table 1 gives biases of +0.0056 and -0.0034, with joint intervals inside +/-0.10. The experiments therefore show both agreement under complete reconstruction and sensitivity to particular omissions, rather than a general failure of replay.

  1. Collecting natural control in resettable environments

The theorem does not assume that collecting C_N is difficult. When suitable live repair/control outcomes are available, F_N-C_N identifies the natural effect under the stated assumptions. RTA adds a comparison with replay; it does not offer an established efficiency advantage over direct live evaluation (Sections 2.2, 2.4, and 5).

  1. RTA as an audit protocol

RTA is an audit protocol, not a new optimizer. Under Assumption A.2, the four-arm comparison measures b=(F_R-C_R)-(F_N-C_N) without requiring the factorization used in the three-arm result. The contribution here is specifying which observations identify the replay-to-natural discrepancy, rather than proposing a generally better learning algorithm.

  1. Relation to replay-fidelity studies

We do not claim that reconstruction fidelity is a new concern. Our formulation focuses on whether an action effect in a reconstructed context agrees with the effect conditional on natural arrival. The paper defines that target, derives a sharp missing-control identification interval under explicit assumptions, and measures the discrepancy with four arms on supported populations (Sections 2.1-2.5). This is the scope of our contribution, not a claim of technical superiority over the related study you cite.

  1. Whether live-effect weighting improves downstream training

In the calibrated-pool study, Table 2 reports gains over RL-only of 3.47 points for Gemma and 14.35 for Qwen. However, both live-minus-uniform 95% intervals include zero (Table 19), so these results do not establish an added benefit from live-effect weighting.

Raw Replay has higher point estimates, but its correction weight mass and coverage differ. That comparison cannot isolate the effect of the weighting rule. The correction pipeline improves over RL-only in this setting, but the evidence does not show that live weighting is better than simpler alternatives. Checking whether replay agrees with natural execution and improving the downstream training signal remain separate questions.

Reviewer_cADaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Replay is often used to test whether a different action (a "repair") would have helped an agent. They rebuild a recorded state in a new environment, then compare forcing the repair against letting the agent choose. This paper asks when replay gives the same answer as testing in the real run. The gap between the two is the replay error or bias. The paper makes three contributions: They show that replay without a live control cannot pin down how biased replay is. An audit that estimates the replay error by also running the real, live control as well and then comparing the two. The paper calls this check the Replay Transport Audit (RTA). Deliberately leaving out information during reconstruction, can lead to an increase in replay error. Experiments show that the error depends on how completely the state is rebuilt. In ALFWorld, thorough replay matches live results closely. In WebShop, deliberately removing a piece of information changes the result.

Critical strengths

The conceptual point is correct, the paper correctly shows that matching the agents action distribution and the forced repair’s outcome between replay and execution is not enough to trust replay and matching the visible input is not enough either (Figure 1).

Critical weaknesses

Weak contribution. The value of the audit is not clear to me, it requires running the agent live, which defeats the purpose of replay. The authors acknowledge the audit offers no efficiency advantage (Sec. 5). It is also not clear to me how often the failures of replay highlighted in the paper occur in practice.

The presentation makes the paper hard to evaluate. The text is dense, it introduces many shorthand terms with little explanation and a lot of the evidence that the main text relies on is in the appendix.

Questions for authors

What does the audit add? For the audit you must run the agent live, but the live runs already tell you the true effect of the repair. So why would you use replay at all in this case? Could the replay error you measure at a few points tell you whether replay is safe at other points? Showing this would address my main concern. Does this problem happen in practice? In WebShop replay only went wrong because you removed information, but can you show replay going wrong in an existing method?

2 comments

Thank you for raising the central practical question: what does the audit add when live measurements are already available?

  1. What the audit adds to direct live measurement

If suitable live repair and control outcomes are available and the only goal is the natural action effect, their difference already answers the question. We do not claim that RTA is a more efficient way to estimate that effect (Section 5).

The audit asks whether a replay estimate gives the same answer. With natural repair/control means F_N,C_N and replay means F_R,C_R, it measures b=(F_R-C_R)-(F_N-C_N) under Assumption A.2. Its purpose is to check a claim made from replay, not to replace direct live evaluation when that already suffices.

The three-arm result explains what is missing when C_N is not observed. Under Proposition 2.1's factorization assumption, with alternative-action responses unrestricted in [0,1], the sharp bias interval has width 1-alpha_N. Here alpha_N is the probability of choosing the repair under natural arrival. More replay samples cannot recover that missing response. This identifies an information gap; it does not imply that replay must be biased.

  1. Whether a few audited points can certify other points

The current audit does not provide that guarantee. Its conclusions apply to the declared supported population (Sections 2.5 and 5.1). Agreement in a panel average does not even guarantee small errors at every included cell, let alone at candidates outside the audit. We therefore cannot use these results as a general safety certificate for replay elsewhere.

  1. How often mismatches occur in practice

The controlled interventions do not measure this frequency. As discussed in Section 5, estimating it requires a sampling frame of actual replay workflows. The experiments test sensitivity to reconstruction changes, not how often ordinary implementations omit relevant state.

  1. Failure in an existing method versus a controlled omission

The WebShop experiment is a controlled omission, not a failure case from an unmodified existing method. Across its 12 cells, omitting a backend option gives an aggregate repair-versus-alternative effect that differs from natural arrival by -0.1651, while the observation, action menu, prompt, and token IDs remain identical (Section 3.4; Table 11). This shows why matching visible inputs can miss an effect difference. It does not supply the existing-method example you request.

  1. Clarifying the terminology and main result

By "natural arrival," we mean that the agent reaches a candidate during a fresh, uninterrupted run. "Replay" means executing a recorded prefix in a fresh environment. Within either context, the action effect compares forcing the repair with letting the policy choose.

The main empirical results are that complete replay agrees with natural effects at the measured ALFWorld panel level (Table 1), while the specified WebShop omissions change relative action effects (Section 3.4). RTA measures that agreement on supported candidates. These results concern when the measured replay effects agree with live effects, not computational savings or a claim that replay generally fails.

Thank you for your candid response. My main concerns remain, the audit requires the measurements that make replay unnecessary at those points, it gives no guarantee for unaudited points and the paper doesn't show that such mismatches occur in real, existing methods. Even though the conceptual point is correct, the practical contribution is limited and I am keeping my rating.

Reviewer_5UKROpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The work studies whether the benefits of replacing an agent's action can be reliably estimated using replay-based rollouts, where execution is resumed from a reconstructed trajectory prefix. The authors formalize the uncertainty arising when the agent's natural continuation is not observed and propose Replay Transport Audit (RTA), which adds live execution measurements to estimate the discrepancy between replay and natural execution.

Critical strengths
  1. The formalism provides a useful characterization of what can and cannot be inferred from replay outcomes. In particular, the authors derive a sharp identification interval for the missing natural-control response and show how additional live measurements resolve this uncertainty under stated assumptions.

  2. The authors demonstrate that relative action benefits can change even when the visible agent input remains identical, due to differences in hidden environment state (e.g., an omitted backend product option in WebShop). They also show that live-audited corrections improve performance over RL-only training in their calibrated-pool experiments.

Critical weaknesses
  1. The paper is extremely difficult to follow, particularly for readers without a background in the area. I found myself spending considerable effort reconstructing the main argument rather than evaluating the contributions.

a. The abstract introduces technical expressions such as "panel-level tolerance" without sufficient intuition.

b. The paper repeatedly presents results alongside numerous qualifications without clearly prioritizing their implications. For example, Lines 357–366 discuss how reconstruction omissions change repair decisions, only to subsequently state that the proposed certificate abstains on all 12 WebShop cells. The practical takeaway is difficult to extract.

c. Lines 92 and 109: What exactly are the "within-context common laws" and execution "laws"? A more intuitive explanation of these assumptions would help.

d. The term "natural" is used extensively in expressions such as "natural arrival," "natural outcome," and "natural execution." A clear, consistent definition early in the paper would improve readability.

e. Line 269: Statements such as "Neither overall policy success nor a task-population average is estimated" appear rather abruptly, without enough explanation of why this distinction matters.

f. It is difficult to understand what constitutes a "cell" in the experimental setup and how cells, checkpoints, and panels relate. A simple running example would be useful.

  1. Why are only ALFWorld and WebShop considered for the empirical evaluation?

  2. The practical contribution of RTA is somewhat unclear. Although Section 2.4 defines the four-arm comparison, a concrete description of the complete audit procedure, its costs, and its advantages over directly measuring live outcomes would help. Furthermore, in Table 2, live RTA performs worse than raw replay weighting for both models. The authors acknowledge that the experiments do not establish an additional benefit from live-effect weighting. What, then, is the practical advantage of RTA over simpler replay-based corrections, especially given the additional cost of collecting live measurements?

Questions for authors

See weaknesses.

1 comment

Thank you for pointing out where the terminology and presentation obscure the argument. Below we clarify the six terms and distinctions you raised, then discuss the experiments and practical value.

  1. What "panel-level tolerance" means

A panel is a fixed weighted average over candidate-level results. Here, each result is the action effect under replay minus the effect under natural arrival. The +/-0.10 tolerance applies to that average, not to every candidate. Table 1 reports ALFWorld panel biases of +0.0056 for GRPO and -0.0034 for GiGPO, with joint intervals inside this range.

  1. Why the omission result and the certificate give different messages

The omission experiment shows that hidden state can matter even when visible inputs match. With a backend option omitted, the aggregate repair-versus-alternative effect differs from natural arrival by -0.1651 (Section 3.4; Table 11).

The certificate asks a stricter question: can the data justify a repair decision under the specified uncertainty bounds? It abstains on all 12 cells (Section 3.5). It uses weighted-Hoeffding intervals and a different error allocation from Figure 3's bootstrap families. Under that rule, complete-replay bias intervals extend beyond +/-0.10, and aggregate live-effect intervals cross the 0.05 benefit threshold (Appendix C.2). Thus, the experiment demonstrates sensitivity to the omission, but the certificate does not resolve the repair decisions. Differences between point-estimate decisions are not certified errors.

  1. Execution laws and within-context common laws

Here, "law" means a probability distribution. An execution law describes the states before intervention, including hidden environment and policy state. The common-law assumption requires repair and control attempts to represent the same distribution within Natural, and separately within Replay. It does not require Natural and Replay to have the same distribution. Matching visible inputs alone is not enough to verify this assumption (Sections 2.1 and 2.4).

  1. What "natural" means

Natural arrival means that the policy reaches the candidate during a fresh, uninterrupted rollout. A natural outcome is measured after an intervention at that arrival; it need not be an outcome of the unmodified policy. Replay instead executes a recorded prefix in a fresh environment (Section 2.1).

  1. Why this is not overall policy success

The effects are conditional on reaching the selected candidates. We do not average over all tasks or count non-arrivals as failed outcomes in these means. The question is whether changing an action helps after the agent reaches that point, not how often the deployed policy succeeds overall (Sections 2.5 and 3.1).

  1. Cells, checkpoints, and panels

In ALFWorld, a cell fixes a checkpoint-template candidate. For example, choose one checkpoint and one task template, then use separate attempts to measure repair and control outcomes under natural arrival and replay. The cells are combined with fixed weights into a GRPO or GiGPO panel. A cell is therefore the candidate-level unit; a panel summarizes multiple cells (Sections 2.5 and 3.1).

  1. What ALFWorld and WebShop each test

ALFWorld measures agreement between complete replay and natural arrival on supported candidates. WebShop tests controlled changes to hidden backend state while keeping visible inputs fixed (Sections 3.1-3.4). These are different empirical checks, but they do not establish validity in other agent domains.

  1. Audit procedure and cost

The procedure fixes candidate-matching rules and repairs before observing intervention outcomes, allocates separate attempts to the four arms before rollout, and estimates natural repair/control means F_N,C_N and replay means F_R,C_R. Under Assumption A.2, the replay bias is b=(F_R-C_R)-(F_N-C_N) (Section 2.4). Live attempts that do not reach the candidate still cost resources, although they do not enter the conditional outcome means.

If suitable live repair/control results already exist and only the natural effect is needed, F_N-C_N is enough. RTA additionally checks whether replay agrees with that effect. Section 5 does not establish an efficiency advantage over direct live measurement.

  1. Practical value when Raw Replay scores higher

Live RTA improves success over RL-only by 3.47 points for Gemma and 14.35 for Qwen (Table 2). However, both live-minus-uniform intervals include zero (Table 19). Raw Replay has higher point estimates, but differs in weight mass and coverage. These comparisons do not isolate an advantage of RTA's weighting rule. The audit provides a measurement of replay-to-natural disagreement on declared support; the downstream results do not show that its estimates yield better training than simpler correction schemes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.