acceptodds
Under review as a conference paper at ICLR 2027

When Harnesses Lose the Signal: Causal Evaluation of Recovery in LLM Agents

Abstract

Large language model agents rely on external harnesses to pass information between the model and its environment and to recover from execution errors. Yet recovery is usually judged only by average task success. This hides an important tension. The same operation can rescue a failing trajectory or disrupt one that would otherwise succeed. We frame recovery as a causal decision problem. Starting from the same execution state, we compare what happens with and without recovery, separate rescue from harm, and study how the value of recovery changes over time. We then introduce the Causal Intervention Router (CIR), a lightweight policy that uses information available before recovery to decide when intervention is worthwhile. On long-horizon ALFWorld tasks with Qwen3-14B, CIR raises success from 70.33% to 73.33%, a gain of \(3.00\) percentage points. It leaves all evaluated trajectories with correct observations untouched. Additional controls show that the benefit of recovery cannot be explained solely by the new observation returned by the environment. These results provide a practical way to evaluate recovery and apply it selectively. Code and artifacts are available at https://anonymous.4open.science/r/Causal-Evaluation-of-Recovery-in-LLM-Agents-308D/.

Primary area:
interpretability and explainable AI
Keywords:
Agent harness, causality, recovery analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5UQMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses an under-explored problem in LLM agent engineering. External recovery operations (like refreshing or querying the environment) are typically evaluated solely on aggregate task success rates. This obscures the dual nature of recovery: while an operation can rescue a failing trajectory, it can disrupt a successful one. To address this problem, the paper proposes the Causal Intervention Router (CIR), which uses pre-intervention features to estimate whether refreshing the environment observation is worthwhile. The method is evaluated using Qwen3-14B agents in ALFWorld under clean, stale, and missing-observation conditions and reports that CIR improves overall success. Additional experiments examine recovery timing and whether the benefit of recovery comes from the returned observation content or other aspects of the intervention.

Critical strengths
  • I find the paper's framing of harness recovery as a causal decision problem useful and clearly motivated. Evaluating recovery solely through aggregate success rates can obscure the cases where an intervention helps or harms an agent. The method (CIR) is lightweight, model-agnostic at inference time.
  • I appreciate the use of matched executions to compare recovery and no-recovery outcomes. This provides a useful way to study intervention effects under controlled conditions.
Critical weaknesses
  • I appreciate the controlled experiments across different observation conditions and the use of complete cohorts to evaluate selective recovery. The results provide useful evidence that CIR can improve performance in the studied setting. However, all experiments use Qwen3-14B on ALFWorld. Since models can differ substantially in reasoning ability, model size, and sensitivity to self-correction, I am not yet convinced that the findings generalize to other architectures or agent environments.
  • The reported 3% improvement over never refreshing is statistically significant, and CIR intervenes in 16.3% of evaluated episodes. However, each intervention may introduce additional model calls, latency, or environment interactions. I would like to see an analysis of these costs alongside the success-rate improvements. This would help establish whether the performance gains justify the additional policy and engineering complexity.
  • I would encourage the authors to strengthen the direct baseline comparisons and provide either broader evaluations or more carefully bounded conclusions. The paper would be more convincing if it positioned the rescue-versus-harm evaluation framework as its primary contribution and presented CIR as a promising initial policy rather than a demonstrated general solution. I think the following works is particularly relevant because it provides a controlled benchmark for dynamic replanning and anomaly recovery under tool failures. I suggest discussing it in Related Works and, if feasible, evaluating CIR on these benchmarks:

(1) Zhu et al., “When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents” (2026) (2) Liu et al., “PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems” (2026).

Questions for authors
  • What specific factor explains why content-ablated refresh rescues many of the same tasks as full refresh? Can the authors distinguish the effects of an extra environment interaction, replanning, and fresh observation content?
  • How sensitive are the results to the choice of model, benchmark, observation corruption, and policy thresholds?
  • What is the additional inference and interaction cost of CIR, and does the success-rate improvement remain attractive when this overhead is considered?
3 comments
AuthorsOpenReview ↗

Response to W1: Additional Evaluation Across Model Families and Environments & Response to W2: Recovery Gains and Execution Costs

Response to W1: Additional Evaluation Across Model Families and Environments

Thank you for your feedback. We added experiments with Seed-OSS-36B-Instruct on ALFWorld and Qwen3-14B on WebShop, extending evaluation to another model family and a second interactive environment. For each setting, the router was fitted on its training data, and its parameters and thresholds were frozen before held-out evaluation.

On ALFWorld, we evaluated Seed-OSS-36B-Instruct on the same 75 held-out tasks under four observation conditions, yielding 300 episodes. CIR increased success from 43.67% (131/300) to 47.33% (142/300), a gain of 3.67 percentage points (95% CI [0.00, 8.33]), comprising 18 rescues and 7 harms. Gains were 9.33 points under missing observations and 5.33 points under two-step stale observations, with no harms under clean observations. This provides a positive gain estimate in another model family and shows that recovery opportunities vary across models.

On WebShop, recovery rereads the current page observation. We used separate sets of 200 training, 50 development, and 100 test goals, with four observation conditions producing 400 test episodes. Immediate forced refresh increased success from 12.25% (49/400) to 15.00% (60/400), a gain of 2.75 points (95% CI [0.50, 5.50]), comprising 15 rescues and 4 harms. The WebShop-adapted CIR used a single-checkpoint decision rule and performed 25 page rereads, producing 1 rescue and no harms, with 12.50% success.

These experiments extend paired recovery evaluation across model families and environments. They separately quantify the value of the recovery operation and how effectively a learned policy selects interventions. We will report these results together and present CIR as a lightweight policy fitted to the model and recovery operation in each setting.

Response to W2: Recovery Gains and Execution Costs

Thank you for the suggestion. We added paired cost analyses for the original Qwen3-14B and the new Seed-OSS-36B experiments. The accounting covers execution from the shared checkpoint to termination, including execution before a delayed refresh and any additional environment queries.

CIR’s routing requires no additional LLM requests or retraining of the underlying agent. Three lightweight logistic regression models use information available before intervention to decide whether to refresh. Each intervention adds one look query. On the original 300 Qwen3-14B test episodes, CIR increased success from 70.33% to 73.33%, yielding nine additional successes, while reducing model calls from 6,466 to 6,030—a 6.7% reduction. This corresponds to 1.45 fewer calls per episode (95% CI: 0.69–2.28 fewer). Input and output tokens decreased by approximately 1.21 million and 39,000, respectively. Including the 49 refresh queries, total environment interactions decreased from 3,954 to 3,832.

The new Seed-OSS-36B experiment also produced more successes with fewer model calls. Across 300 test episodes, CIR increased success from 43.67% to 47.33%, yielding 11 additional successes, while reducing model calls from 7,898 to 7,614—a 3.6% reduction. The average reduction was 0.95 calls per episode (95% CI: 0.01–2.04 fewer). Input and output tokens decreased by 314,710 and 3,162, respectively. Including 151 refresh queries, total environment interactions decreased by 170. These queries took approximately 16.28 seconds in total, or 0.108 seconds per query; this measures the environment-query overhead itself.

We also added a comparison using AgentDebug’s official two-stage diagnostic module adapted to our ReAct executor, followed by one checkpoint retry after failure. On the same 300 episodes, this adaptation achieved 76.67% success using 14,016 model calls, compared with CIR’s 73.33% success using 6,030 calls—approximately 57.0% fewer calls. This illustrates the budget trade-off: diagnosis followed by retry achieves more successes, whereas CIR provides recovery during execution with substantially fewer model requests.

Together, the original 14B and new Seed results show increased task success alongside reduced model-call consumption. The additional queries can be offset by savings in subsequent execution, so recovery costs should be assessed over the complete continuation. CIR’s lightweight implementation and measured resource savings provide practical support for selective recovery in long-horizon agents. We will report success, model calls, token consumption, and environment interactions together in the revision.

AuthorsOpenReview ↗

Response to W3: Expanded Comparisons and Evaluation & Q1: Understanding Refresh Effects

Response to W3: Expanded Comparisons and Evaluation

Thank you for the suggestion. We added direct baseline comparisons and experiments across model families and environments. We will clarify that our primary contribution is paired rescue–harm evaluation of observation recovery, together with analyses of recovery timing and mechanisms. CIR serves as a lightweight policy that uses this evidence for selective recovery.

For direct comparisons, we added AgentDebug-based recovery and generic failure-feedback retry. We adapted AgentDebug’s official two-stage diagnostic module to our ReAct executor and allowed one checkpoint retry for each of the 89 episodes that failed without refresh. Generic retry controls for the benefit of an additional attempt. Across the same 300 episodes, never-refresh, CIR, generic retry, and the AgentDebug adaptation achieved 70.33%, 73.33%, 77.67%, and 76.67% success, respectively. CIR used 6,030 model calls, compared with 9,505 for generic retry and 14,016 for the AgentDebug adaptation. These results characterize the success–cost trade-off: post-failure retry achieves more successes, while CIR provides recovery during execution with fewer model requests—approximately 57.0% fewer calls than the AgentDebug adaptation.

To broaden evaluation, we added Seed-OSS-36B-Instruct and a second environment, WebShop. On Seed, CIR increased success from 43.67% to 47.33%, comprising 18 rescues and 7 harms. On WebShop, recovery rereads the current page observation. Across 100 held-out goals under four observation conditions, yielding 400 episodes, immediate forced refresh produced a 2.75-percentage-point gain, comprising 15 rescues and 4 harms. The WebShop-adapted CIR performed 25 page rereads, producing one rescue and no harms. These experiments separately examine the value of the recovery operation and how effectively a policy selects recovery opportunities.

We will also expand the discussion of ToolMaze and PlanBench-XL. ToolMaze controls tool-failure types and task topology to study dynamic path discovery, replanning, and recovery costs. PlanBench-XL examines tool retrieval, long-horizon composition, and alternative paths in large tool ecosystems. These benchmarks complement our evaluation: they provide tool-failure scenarios, while paired execution can measure the rescues, harms, and costs caused by retrying, retrieving alternatives, or replacing tools relative to continued execution.

The revision will present these comparisons and expanded experiments together, emphasizing how paired evaluation identifies recovery opportunities and risks across settings. CIR’s empirical conclusions will be tied to the specific model and environment in which its router was independently fitted, frozen, and evaluated on held-out tasks.

Response to Q1: Understanding Refresh Effects

Thank you for the suggestion. We combine the original mechanism experiments with an additional control to examine the roles of extra environment queries, replanning, and fresh observation content.

Under stale observations with a four-action recovery delay, the original 67-task experiment shows success-rate gains of 13.43, 10.45, and 1.49 percentage points for full refresh, content-ablated refresh, and replanning alone, respectively, relative to no recovery. Among the 65 tasks where both refresh operations were executed, full refresh rescued 10 tasks and content-ablated refresh rescued nine, with eight shared rescues. Thus, fresh observation content was not necessary for these shared rescues.

To examine the effect of the additional environment interaction, we added a prompt-matched query/no-query control. Both conditions present exactly the same text to the model; the only difference is whether a look query is executed with its returned content hidden. Across 65 eligible task pairs, both conditions succeeded on 46 tasks, and their recorded continuations were identical. Therefore, in this experiment, the additional hidden query itself provided no recovery benefit.

One possible explanation for the shared rescues is that content ablation still presents the query record and the message “Current observation is unavailable.” These signals may prompt the agent to reconsider its existing history and adjust subsequent actions. The two tasks rescued only by full refresh also illustrate how fresh observations can help individual tasks. Together, these comparisons help distinguish the roles of information acquisition, prompt changes, and subsequent replanning, providing a more specific account of where recovery benefits may arise.

AuthorsOpenReview ↗

Response to Q2: Sensitivity Across Settings and Thresholds & Q3: Recovery Benefits and Costs

Response to Q2: Sensitivity Across Settings and Thresholds

Thank you for pointing this out. We added comparisons across models, environments, observation corruptions, injection positions, and policy thresholds, and will report these results separately.

1. Model sensitivity. On the same 75 held-out ALFWorld tasks under four observation conditions, CIR achieves net gains of +3.00 percentage points on Qwen3-14B and +3.67 points on Seed-OSS-36B. The Qwen3-32B experiment yields −1.67 points (95% CI [−4.33, 0.33]). These results show that model scale alone does not determine recovery-policy effectiveness. We separately report forced-refresh and CIR outcomes to identify the recovery opportunities available to each model and how effectively the policy selects them.

2. Benchmark sensitivity. On WebShop, 100 held-out goals under four observation conditions yield 400 episodes. Immediate forced refresh produces a +2.75-point gain (95% CI [0.50, 5.50]), comprising 15 rescues and 4 harms. The WebShop-adapted CIR uses a single-checkpoint decision rule and performs 25 page rereads, producing one rescue and no harms, for a +0.25-point gain. Paired evaluation thus distinguishes the value of the recovery operation from the effectiveness of its selection policy.

3. Observation corruption and injection position. We evaluate clean, one-step stale, two-step stale, and missing observations. On Qwen3-14B, CIR gains 9.33 points under two-step stale observations and 1.33 points under both one-step stale and missing observations, while leaving clean episodes untouched. On Seed, the corresponding gains are 9.33 points for missing observations, 5.33 points for two-step stale observations, and −2.67 points for one-step stale observations, showing that recovery opportunities also vary across models. Replacing the fixed injection position with a randomly sampled eligible position, while keeping the original 14B policy frozen, yields a +1.67-point gain estimate. Removing the observation-novelty feature yields +2.33 points, showing that positive gains in this setting can be retained without that explicit feature.

4. Threshold sensitivity. The original 14B results are stable at nearby threshold values. Varying the anomaly and continuation-failure thresholds individually by ±0.05 preserves the +3.00-point gain. Across the tested utility thresholds from −0.2 to +0.2, gain estimates range from +1.00 to +3.00 points. Higher utility thresholds reduce intervention frequency while forgoing some recovery opportunities. These sweeps characterize sensitivity; the thresholds used for the main test evaluation were selected on training data and frozen beforehand.

Response to Q3: Recovery Benefits and Costs

Thank you for the suggestion. We added cost accounting from the shared checkpoint to task termination, reporting additional queries, model calls, and token consumption alongside success rates.

CIR uses three lightweight logistic regression models on existing execution records, without additional LLM calls or agent retraining; each refresh adds one observation query.

In the original Qwen3-14B experiment, success increased from 70.33% to 73.33%, yielding nine additional successes, while model calls decreased from 6,466 to 6,030—a reduction of approximately 6.7%. The average reduction was 1.45 calls per episode (95% CI: 0.69–2.28 fewer). Input and output tokens decreased by approximately 1.21 million and 39,000, respectively. Including the 49 refresh queries, total environment interactions decreased by 122.

The additional experiments also produced more successes with fewer model calls. When observation errors were injected at random positions, with the original CIR parameters unchanged, CIR yielded five additional successes while reducing model calls by 189, or approximately 3.9%. On Seed-OSS-36B, CIR yielded 11 additional successes while reducing model calls by 284, or approximately 3.6%. Input and output tokens decreased by 314,710 and 3,162, respectively, and total environment interactions decreased by 170 after including refresh queries. The 151 look queries took approximately 16.28 seconds in total, averaging 0.108 seconds per query.

We also compared against an AgentDebug adaptation using its official two-stage diagnostic module followed by one retry from the shared checkpoint after failure. On the same 300 episodes, CIR achieved 73.33% success with 6,030 model calls, while the AgentDebug adaptation achieved 76.67% with 14,016 calls. CIR used approximately 57.0% fewer calls, offering a recovery option when model-call budgets are limited.

These results show that an additional refresh query can reduce the model calls and tokens needed for subsequent execution. In the original 14B, random-position, and Seed experiments, CIR completed more tasks while using fewer model calls.

Thank you for your thoughtful comments, which have helped us clarify the contribution and strengthen the evaluation.

Best regards,

The Authors

Reviewer_DgGVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper measures a harness "refresh" after an injected stale or missing observation by branching paired runs from one state, splits the effect into rescues and harms, and trains CIR, a thresholded T-learner that refreshes when predicted gain exceeds predicted harm. On 75 held-out ALFWorld tasks with Qwen3-14B, CIR lifts success from 70.33% to 73.33%. CIR's edge over anomaly detection is not established; the evidence covers one model, one benchmark, and one injection point, and the two closest works are uncited.

Critical strengths
  • The paired counts reproduce every reported effect (13.98 pp = 13/93, p.7); the cluster bootstrap fits 300 episodes in 75 tasks.
  • The content-ablated refresh (Sec. 5.3, Fig. 5) is a real finding: 8 of 10 rescues at delay 4 survive with the returned observation hidden.
  • Eq. 4's clean-harm budget and Table 1's rescue/harm counts make the claims checkable.
Critical weaknesses
  • p.9: CIR's margins over the sequential (1.33 pp, CI [-0.33, 3.33]) and rate-matched (1.00 pp, CI [-0.67, 2.67]) anomaly baselines include zero, and the rate-matched one has no harms; yet the Discussion says they "show why anomaly detection is insufficient". Soften it or add tasks.
  • p.6 and p.8: CIR's training conditions are never stated (the primary cohort holds only clean and two-step stale), and the forced-refresh effect under one-step stale or missing observations is never reported (Table 1 gives only CIR's +1.33 pp there), so "concentrates recovery where the observed benefit is largest" has no evidence; CIR fires on 1 of 75 missing episodes though 28% fail. Report forced-refresh effects and trigger rates per condition.
  • p.6: one model, one benchmark, one injection point (25% of the trajectory), and errors the detector was fitted on; a two-step stale observation repeats an earlier one verbatim, an easy target for the observation-novelty feature (p.7). Inject at a random point and repeat Table 1.
  • p.7: "task progress" is undefined; if it comes from the simulator's subgoal counter, it is privileged at deployment.
  • p.8: the hidden-content message is never shown, and "the returned content is unnecessary for most shared rescues" is circular (shared rescues are those rescued without content). Give the message and a CI for ablated minus replan.
  • Abstract: 70.33% and 73.33% summarize 9 net episodes of 300; report the gain with its interval.
Questions for authors

Two uncited works make the central argument first. Vasudev et al. (arXiv 2602.03338, Feb 2026) define the same 2x2 table of recoveries and disruptions (rescue and harm here), gain +2.8 pp on ALFWorld, and show that a critic with AUROC 0.94 can cut success by 26 pp on HotPotQA. Zhang et al. (arXiv 2606.21399, June 2026; NeurIPS 2026) branch prefixes on ALFWorld and train controllers on intervention utility in place of failure risk; their Section 2 makes this paper's Section 4 argument in a more general form. Under ICLR policy their absence is no ground for rejection, but the contribution list (p.2) must be rewritten against them. New here: the refresh operation, the delay axis, and the content-ablated control. The policy (Eq. 6 to 8) is a T-learner uplift score behind three thresholds; ThriftyDAgger (Hoque et al., CoRL 2021) gates interventions on novelty and risk thresholds in the same spirit.

5 comments
AuthorsOpenReview ↗

Response to W1: Comparison with Anomaly-Detection Baselines

Thank you for your suggestion. We have added direct comparisons on Seed-OSS-36B and will revise the discussion to present the policies’ performance through success rates, rescues and harms, and intervention counts.

In the additional Seed-OSS-36B experiment, CIR produces 18 rescues and seven harms, yielding an observed rescue-to-harm ratio of 2.57—the highest among the intervention policies evaluated. Sequential anomaly detection and rate-matched anomaly detection achieve ratios of 2.40 (24/10) and 2.38 (19/8), respectively, while immediate forced refresh achieves 1.24 (26/21). This descriptive ratio complements success rate by making the balance between effective recovery and intervention-induced harm more explicit.

On this model, CIR increases successes from 131 to 142 relative to no refresh, yielding 11 additional successes and a gain of 3.67 percentage points (95% CI [0.00, 8.33]). Compared with standard sequential anomaly detection, CIR reduces interventions from 202 to 151, a reduction of 25.2%. Furthermore, compared with no refresh, CIR reduces model calls from 7,898 to 7,614, showing that selective refresh can also save subsequent model calls in this evaluation.

In the original Qwen3-14B experiment, CIR achieves a net gain of 3.00 percentage points over no refresh (95% CI [0.67, 5.67]). We will replace the statement that these results “show why anomaly detection is insufficient” with: “Anomaly detection can support effective recovery decisions. Paired evaluation further reveals the rescues, harms, and intervention burden associated with different policies. In the additional Seed-OSS-36B experiment, CIR achieves the highest observed rescue-to-harm ratio among the evaluated intervention policies, while reducing interventions relative to standard sequential anomaly detection. These results illustrate the practical value of using paired benefit estimates to guide selective recovery.”

AuthorsOpenReview ↗

Response to W2: Refresh Effects and Trigger Rates

Thank you for this suggestion. We will clarify CIR’s training conditions and report forced-refresh effects, rescue/harm counts, and policy trigger rates for each observation condition. The additional model experiments also provide further evidence under missing observations.

For the original Qwen3-14B evaluation, CIR’s training data combine the primary paired records with extended control records. The former provide clean and two-step stale observations, while the latter provide one-step stale and missing observations. Training therefore covers all four conditions, with 93 source tasks and 1,426 valid decision examples. Thresholds are selected through five-fold cross-validation grouped by source task, followed by evaluation on 75 independent held-out tasks.

The following table reports results for the original model, with 75 episodes per condition. Forced refresh applies one refresh immediately at the checkpoint; CIR selects whether to refresh at subsequent candidate decision points, with at most one intervention per episode. All gains are relative to no refresh and expressed in percentage points (pp).

Observation conditionSuccesses: no refresh → forced refreshForced-refresh gain, pp [95% CI]Forced-refresh rescues / harmsCIR interventions (rate)CIR rescues / harmsCIR gain, pp
Clean61 → 59−2.67 [−10.67, 5.33]3 / 50/75 (0%)0 / 00.00
One-step stale51 → 49−2.67 [−9.33, 4.00]2 / 417/75 (22.67%)1 / 0+1.33
Two-step stale45 → 51+8.00 [−1.33, 17.33]10 / 431/75 (41.33%)9 / 2+9.33
Missing54 → 56+2.67 [−5.33, 10.67]6 / 41/75 (1.33%)1 / 0+1.33

In the original Qwen3-14B experiment, two-step stale observations have the highest point estimate of net benefit from immediate forced refresh. CIR also intervenes most frequently in this condition, producing nine rescues and two harms, for seven additional successes. We will replace the original general statement with this specific finding to make the relationship between intervention frequency and recovery benefit clearer.

Under missing observations, the 28% failure rate corresponds to 21 episodes that fail without refresh. The paired results show that immediate refresh rescues six of these episodes but also causes four otherwise successful episodes to fail. Failure rate and net refresh benefit therefore need to be measured separately. CIR selects interventions jointly based on the anomaly score, persistent failure risk, and expected net utility. Of the 204 valid candidate decision points in this condition, only one satisfies all trigger criteria, resulting in one rescue and no harms. We will report forced-refresh effects alongside actual trigger rates to distinguish available recovery opportunities from those captured by the policy.

The additional Seed-OSS-36B experiment further demonstrates CIR’s recovery effect under missing observations. CIR is fitted using this model’s own training-task execution records and evaluated on the same 75 held-out tasks. Results under missing observations are as follows:

MethodSuccessesRefreshes (rate)Rescues / harmsGain over no refresh, pp [95% CI]
No refresh27/750 (0%)——
Immediate forced refresh32/7575 (100%)8 / 3+6.67 [−1.33, 16.00]
CIR34/7540 (53.33%)8 / 1+9.33 [2.67, 17.33]

In this condition, CIR increases successes from 27 to 34. Compared with immediate forced refresh, CIR uses 46.7% fewer refreshes. Both methods produce eight rescues, while CIR produces one harm compared with three for forced refresh. These results show that CIR can capture recovery opportunities under missing observations on another model, achieving a positive net gain with fewer refreshes through selective intervention in this evaluation.

The revised manuscript will report recovery effects and trigger rates separately by model and observation condition, allowing readers to see the benefits and harms of refresh, together with the recovery opportunities actually captured by CIR.

AuthorsOpenReview ↗

Response to W3: Random Injection and Novelty Ablation

Thank you for raising this point. We have completed the random-checkpoint injection experiment and added an ablation that removes the observation-novelty feature. We have also added experiments with Qwen3-32B, Seed-OSS-36B, and WebShop, extending the evaluation to three base models, two model families, and two environments.

The random-injection experiment uses the original 75 held-out tasks under four observation conditions. For each trajectory, we uniformly sample one injection point from its valid nonterminal positions and use the same position across all four conditions, yielding 300 episodes. The sampled positions span 14.3%–96.3% of trajectory action progress, and all 75 tasks are included.

To examine the role of observation novelty, we remove this feature from CIR’s anomaly detector and both success-probability predictors, refit the policy on the original training set, and evaluate it on the same random-injection tasks. The following table summarizes the results. All gains are relative to no refresh. The 95% confidence intervals are computed by resampling source tasks, retaining all four observation conditions for each sampled task.

PolicySuccess rate (count)Gain, pp [95% CI]Interventions (rate)Rescues / harms
No refresh68.67% (206/300)—0 (0%)—
Sequential anomaly detection72.00% (216/300)+3.33 [1.33, 5.67]41 (13.67%)13 / 3
Rate-matched anomaly detection72.00% (216/300)+3.33 [1.00, 5.67]51 (17.00%)13 / 3
CIR70.33% (211/300)+1.67 [0.00, 3.33]27 (9.00%)6 / 1
CIR without observation novelty71.00% (213/300)+2.33 [0.67, 4.00]19 (6.33%)8 / 1

Under random injection, CIR performs 27 refreshes and yields five additional successful episodes. Compared with the two anomaly-detection baselines, it reduces interventions by 34.1% and 47.1%, respectively, with one observed harm compared with three for each baseline. These results illustrate the policies’ practical trade-offs: the anomaly-detection baselines achieve more successes, while CIR obtains additional recovery benefits with fewer interventions and fewer observed harms.

Recovery benefits remain after removing the observation-novelty feature. The ablated policy performs 19 refreshes, producing eight rescues and one harm, for a net gain of seven successes. Specifically, under the two-step stale condition highlighted by the reviewer, this variant performs 12 refreshes and increases successes from 41/75 to 46/75, with six rescues and one harm. The net gain is 6.67 percentage points (95% CI [0.00, 13.33]). This provides direct evidence that the recovery benefits do not depend entirely on the explicit novelty feature.

The fixed-checkpoint experiment provides a complementary comparison: full CIR and the ablated variant achieve net gains of 3.00 and 2.00 percentage points, respectively, compared with 1.67 and 2.33 percentage points under random injection. Under random injection, the paired difference between the ablated and full policies is +0.67 percentage points (95% CI [−0.67, 2.00]). The conclusion supported by this ablation is therefore that positive net gains remain without the explicit novelty feature under both injection-position settings.

The additional experiments also broaden the model and environment coverage. On Seed-OSS-36B, from a different model family, CIR increases successes from 131/300 to 142/300, a net gain of 3.67 percentage points (95% CI [0.00, 8.33]). On WebShop, with 100 independent test tasks and 400 episodes, refreshing once at the checkpoint in every episode increases successes from 49 to 60, a gain of 2.75 percentage points (95% CI [0.50, 5.50]); CIR performs 25 refreshes and increases successes from 49 to 50. These results provide additional evidence on selective recovery across models and on the benefits of refresh in another environment.

In the revised manuscript, we will report the random-injection results by observation condition, following the structure of the original Table 1. Together with the feature ablation and additional model and environment experiments, these results will show how recovery performance varies with injection position, feature use, and evaluation setting.

Reviewer_ZNMrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames harness-level recovery (specifically, re-querying the environment, or "refresh") as a causal decision problem. From the same pre-recovery state, it compares continuations with and without refresh, separating rescue (failure → success) from harm (success → failure), and studies how this changes with observation condition and recovery delay.

Critical strengths
  1. Focusing on the refresh decision within recovery, and separating rescue from harm through paired continuations, is somewhat novel.
  2. Failure recovery in agentic systems is an important problem.
Critical weaknesses
  1. Missing related work discussion and few baselines.
    • AgentDebug and AdaMASt are not cited or discussed.
    • AgentRx (Barke et al., 2026) is cited but not compared against.
    • The only baselines are never-refresh and two sequential anomaly-detection variants. There is no comparison with failure-diagnosis or recovery methods such as AgentDebug and AgentRx, or even with the cited intervention-based methods (e.g., CausalFlow, DoVer).
  2. Only ALFWorld is used, and CIR is evaluated on just 75 held-out tasks. Many other benchmarks are available, such as WebShop and GAIA, as well as failure-focused benchmarks such as Who&When and AgentErrorBench.
  3. Only one base model is evaluated: Qwen3-14B, which is a relatively old open-source model. Since the value of refresh likely depends on the model, results with more and stronger models are needed.
  4. The writing should be improved. Sections 3.3, 4.1 and 4.2 use many paragraphs and heavy notation (indicator functions, rescue/harm scores, multiple thresholds) to explain fairly simple ideas. This overuse makes the paper harder to understand than necessary. Several of these definitions could be stated in a sentence or moved to the appendix.
Questions for authors
  1. The paper states (Sec. 3.3) that "a method should not be judged only by how often it succeeds after intervention, because many of those trajectories may already have succeeded without intervention."
    • Can you give concrete examples of prior work that judges recovery only by post-intervention success?
    • Is this the only work that evaluates changes in outcome after intervention? Several works you cite (Causal Agent Replay, CausalFlow, DoVer) also compare outcomes with and without an intervention.
    • Please also compare with related work such as AgentProcessBench, and clarify what the paired rescue/harm evaluation adds beyond these.
4 comments
AuthorsOpenReview ↗

Response to W1: Missing related work discussion and few baselines

Thank you for this suggestion. Our original experiments used two anomaly-detection baselines to compare intervention decisions while keeping the refresh operation fixed. Following your recommendation, we have completed additional comparisons with an AgentDebug-based recovery method and a generic retry baseline. We will also expand Related Work to clarify how each method relates to our approach.

AgentDebug identifies root-cause errors and generates corrective feedback to help agents recover from failures, making it directly relevant to our work. Its recovery process involves diagnosis followed by another execution attempt guided by feedback. CIR decides whether to obtain a fresh environment observation during the current execution. We therefore included AgentDebug in the additional experiments to compare the recovery benefits and model-call costs of these approaches.

AgentRx uses constraint checking and evidence logs to locate critical failure steps and classify errors in failed trajectories. Its diagnoses can inform recovery, but constructing an execution policy comparable to CIR additionally requires specifying which recovery action to take and when to take it. Our work focuses on a concrete operation—refreshing the current observation—and directly evaluates its benefits and harms relative to continuing without intervention. We will clarify this connection between failure diagnosis and recovery decisions in the revised manuscript.

AdaMAST derives domain-adaptive failure taxonomies from execution traces and uses them for system optimization, runtime feedback, and trajectory selection. Both approaches seek to improve agent behavior using execution experience. AdaMAST provides structured failure feedback, while our approach learns when observation refresh is worthwhile from the paired outcomes of refreshing and continuing without refresh. Evaluating AdaMAST within the same recovery setting would require constructing a domain-specific taxonomy and specifying how its feedback translates into recovery actions. We will discuss its runtime applications and its complementary relationship with our approach.

CausalFlow uses step-level counterfactual interventions to identify failure causes and generate and validate localized repairs. It evaluates repair effectiveness through intervention outcomes. Our work focuses on a fixed observation-refresh operation, systematically varying observation corruption and intervention delay while measuring effects on trajectories that would otherwise succeed as well as those that would fail. The approaches are therefore closely related in their use of intervention-based evaluation, while studying different recovery operations and decision problems.

DoVer verifies failure hypotheses through targeted interventions, such as editing messages or modifying plans, and evaluates debugging through task recovery or progress. It focuses on finding and validating repairs for specific failures. Our work examines whether and when a single environment-observation query is worthwhile while keeping the underlying agent unchanged. A direct comparison would require aligning the available intervention actions and verification budgets. We will clarify this distinction and discuss the shared use of actual execution to validate intervention effects.

In the additional comparison, we used AgentDebug’s official two-stage diagnosis module, integrated its feedback into our existing ReAct agent, and allowed one retry from the checkpoint. The experiment used Qwen3-14B on the same 75 held-out tasks under four observation conditions, totaling 300 episodes. For the 89 episodes that initially failed, we ran both diagnosis-guided and generic retries. Model calls are counted from the common checkpoint to termination, including the continuation of the initial attempt, diagnosis, and retry.

The success rates were 70.33% (211/300) for never-refresh, 73.33% (220/300) for CIR, 76.67% (230/300) for AgentDebug-guided retry, and 77.67% (233/300) for generic retry. Their total model-call counts were 6,466, 6,030, 14,016, and 9,505, respectively. These results further demonstrate CIR’s resource efficiency. Measured as total model calls divided by the number of successful episodes, CIR achieved the best model-call efficiency among the four methods evaluated: 27.4 calls per successful episode, compared with 30.6 for never-refresh, 40.8 for generic retry, and 60.9 for AgentDebug-guided retry. This calculation includes calls consumed by unsuccessful episodes. With 49 online refreshes, CIR achieved 9 additional successes while reducing model calls by 436 relative to never-refresh. It also used approximately 57.0% fewer model calls than AgentDebug-guided retry. In this evaluation, CIR’s lightweight online interventions therefore yielded the lowest average model-call expenditure per successful episode.

We will incorporate these methodological comparisons and experimental results into the revised manuscript.

AuthorsOpenReview ↗

Response to W2: Only ALFWorld is used

Thank you for this suggestion. We have added experiments on WebShop as a second environment, using 200 training tasks, 50 validation tasks, and 100 held-out test tasks. Each test task is evaluated under four observation conditions—clean, one-step stale, two-step stale, and missing observations yielding 400 episodes. Together with the original ALFWorld evaluation, the two environments now cover 175 test tasks and 700 episodes.

In WebShop, we implement recovery by rereading the current page and execute refresh and no-refresh branches from the same execution prefix. This environment allows us to examine observation recovery in a different interaction setting while continuing to distinguish rescues from intervention-induced harms.

The additional experiments show measurable recovery benefits from observation refresh in WebShop. Applying one refresh at the checkpoint in every episode increases the number of successes from 49/400 to 60/400, raising the success rate from 12.25% to 15.00%. The net gain is 2.75 percentage points (95% CI: [0.50, 5.50]), comprising 15 rescues and 4 harms. These results also illustrate the value of paired evaluation: the 11 additional successes can be decomposed into recovery benefits and intervention risks.

We also evaluated CIR on WebShop. It performed 25 refreshes across 400 episodes, corresponding to an intervention rate of 6.25%, and increased the number of successes from 49 to 50. The net gain was 0.25 percentage points (95% CI: [0.00, 0.75]), with one observed rescue and no cases in which intervention caused an otherwise successful episode to fail. We will report the refresh-operation results separately from the selection-policy results so that readers can distinguish the recovery benefits available in the environment from those realized by the policy.

These additional experiments address the concern about evaluation in a single environment and provide evidence for the framework across environments. For both ALFWorld environment observations and WebShop page observations, paired execution allows us to measure recovery benefits, harms, and intervention burden. The revised manuscript will present the empirical findings from both environments and position CIR as an example of how the evaluation framework can support selective recovery.

AuthorsOpenReview ↗

Response to W3: Only one base model is evaluated

Thank you for this suggestion. We have added experiments with Qwen3-32B and Seed-OSS-36B, extending the evaluation to three model sizes—14B, 32B, and 36B and two model families. These experiments further examine how observation refresh and its selection policy perform across different base models.

For each model, CIR is fitted and its thresholds are selected using execution records from that model’s training tasks. Evaluation then uses the same 75 held-out tasks under four observation conditions, totaling 300 episodes per model. We report the effects of the refresh operation and CIR separately. Confidence intervals are computed by resampling source tasks, retaining all four observation conditions for each sampled task when calculating aggregate results.

On Qwen3-32B, immediate refresh under two-step stale observations increases the number of successes from 41/75 to 50/75, raising the success rate from 54.67% to 66.67%. The net gain is 12.00 percentage points (95% CI: [1.33, 22.67]). This shows that refresh can recover some tasks affected by stale observations in the larger Qwen model. Across all four conditions, the net change is +2.00 percentage points for immediate refresh (95% CI: [−3.33, 6.67]) and −1.67 percentage points for CIR (95% CI: [−4.33, 0.33]). These comparisons distinguish the recovery opportunities provided by refresh from the selection policy’s ability to use those opportunities.

On Seed-OSS-36B, which belongs to a different model family, CIR increases the number of successes from 131/300 to 142/300, raising the success rate from 43.67% to 47.33%. This corresponds to 11 additional successes and a net gain of 3.67 percentage points (95% CI: [0.00, 8.33]). Together with the original gain of +3.00 percentage points on Qwen3-14B (95% CI: [0.67, 5.67]), this provides additional evidence for applying selective recovery across model families.

The paired results on Seed-OSS-36B also show that CIR produces 18 rescues and 7 harms, yielding an observed rescue-to-harm ratio of 2.57, the highest among the four intervention strategies compared in this experiment. The corresponding ratios are 2.40 (24/10) for sequential anomaly detection, 2.38 (19/8) for intervention-rate-matched anomaly detection, and 1.24 (26/21) for immediate forced refresh. This descriptive measure provides information beyond success rate. Compared with standard sequential anomaly detection, CIR also reduces the number of interventions from 202 to 151, a reduction of 25.2%.

In terms of resource use, CIR on Seed-OSS-36B reduces total model calls from 7,898 to 7,614 relative to never-refresh—a reduction of 284 calls, or approximately 3.6%. Thus, in this setting, CIR increases the number of successful episodes while reducing the model calls required for subsequent execution.

We will present these results across all three models in the revised manuscript, emphasizing the significant recovery benefit of immediate refresh under two-step stale observations on Qwen3-32B, alongside CIR’s positive net gain, favorable observed rescue-to-harm ratio, and model-call efficiency on Seed-OSS-36B. These experiments broaden the model evaluation and clarify how the effects of the recovery operation and its selection policy vary with the base model.

As for W4: Simplifying the Presentation, we thank you for this detailed suggestion. In Section 3.3, we will replace the detailed notation for rescues, harms, and the two unchanged-outcome cases with an intuitive paragraph, moving the indicator-function definitions to the appendix. Section 4.1 will first explain the objective in plain language: improving final success while limiting the proportion of clean-observation trajectories that fail because of refresh. The formal objective will then be moved to the appendix. Section 4.2 will directly explain that CIR predicts success probabilities with and without refresh and weights potential harm in its decision rule, removing repetitive explanations of the scores. The appendix will retain the complete formulas, threshold definitions, and implementation details for reproducibility. These changes will make the central idea clearer: use paired outcomes to measure rescues and harms, then use this evidence to decide when to refresh.

AuthorsOpenReview ↗

Response to Questions

Response to Q1: Clarifying the Role of Paired Evaluation

Thank you for this follow-up question. We will clarify in Section 3.3 that this statement concerns the information provided by final success rate, rather than a general characterization of prior work. Reflexion includes baseline comparisons and ablation studies, while CRITIC explicitly analyzes both successful corrections and cases where correction turns an initially correct answer into an incorrect one. These works are therefore not appropriate examples of evaluating recovery solely through post-intervention success.

Final success rate alone does not identify tasks rescued or harmed by recovery. We measure these changes by comparing recovery and no-recovery continuations from the same environment state and execution prefix.

For example, in our new WebShop experiment that refreshes once at the checkpoint in every episode, 60 of 400 episodes succeed after refresh. Of these, 45 also succeed without refresh, while 15 are rescued by refresh. Meanwhile, four episodes that succeed without refresh fail after refresh. The net gain is therefore 15 − 4 = 11 additional successes. This breakdown shows both recovery opportunities and intervention risks, beyond the total number of successes after refresh.

We will revise the original sentence as follows: “To characterize the changes attributable to recovery, we report final success rate alongside paired comparisons of recovery and no-recovery continuations from the same execution prefix. We separately count tasks rescued by recovery and otherwise successful tasks harmed by recovery, and report the resulting net gain.”

Response to Q2: Comparison with Prior Intervention Studies

The key distinction between our work and these approaches lies in the decision that intervention evaluation supports. Our goal is to determine, during task execution and before the final outcome is known, whether and when to apply a fixed recovery operation, while avoiding disruption to trajectories that would otherwise succeed.

Causal Agent Replay primarily estimates the causal contribution of individual steps to failure; CausalFlow generates minimal counterfactual repairs from failed trajectories; and DoVer tests failure-diagnosis hypotheses through targeted interventions. These approaches do compare outcomes after intervention. However, results on failure attribution or repair do not directly establish an online recovery policy’s net benefit across both successful and unsuccessful trajectories. We therefore measure both rescues and harms, systematically vary observation errors and intervention delays, and use paired execution evidence to train a recovery router that relies only on currently observable information.

This design has practical deployment benefits: CIR’s online scoring requires no additional LLM requests and no search over candidate repairs at test time. In the original Qwen3-14B held-out evaluation, CIR refreshed in only 16.33% of episodes and increased success rate from 70.33% to 73.33%, a net gain of 3.00 percentage points (95% CI [0.67, 5.67]). It produced 11 rescues and two harms, with no interventions in clean-observation episodes. Additional analysis shows that it also reduced model calls by an average of 1.45 per episode in this setting.

Response to Q3:What Paired Rescue/Harm Evaluation Adds

Thank you for this suggestion. AgentProcessBench uses step-level quality labels to evaluate process assessment and error localization, and demonstrates how process signals support trajectory selection. Our work further measures which final task outcomes change when a recovery operation is applied, compared with continuing execution from the same state. CIR performs 27 refreshes, producing six rescues and one harm, for a net gain of five successes. Sequential anomaly detection performs 41 refreshes, producing 13 rescues and three harms, for a net gain of ten successes. Paired outcomes break down changes in success rate into rescues and harms, while intervention counts help characterize each policy’s benefits and costs.

We present the model with identical prompts and vary only whether a background look query is executed with its returned content hidden. Across 65 valid paired tasks, both conditions achieve 46 successes, and their recorded subsequent trajectories are identical in every pair. Thus, in this experiment, the additional query itself provides no recovery benefit.

Beyond process-quality evaluation, our work therefore adds two types of execution evidence: changes in final outcomes attributable to recovery, and the actual contribution of an individual component of the operation. These paired outcomes can also train a recovery policy to decide whether to refresh at runtime using information available before intervention.

Thank you for your thoughtful suggestions and the time you have devoted to reviewing our paper. We wish you continued success in your research.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.