acceptodds
Withdrawn from ICLR 2027

The Cost of Failure: Quantifying and Mitigating the Impact of Trajectory Errors on Agent Robustness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Model (LLM) agents solve long-horizon tasks by repeatedly acting, observing feedback, and conditioning later decisions on the accumulated interaction history. This trial-and-error process creates a Cost of Failure: invalid actions, execution errors, and low-value feedback persist in the context, polluting later decision making, though models try to recover from mistakes. We quantify this effect with Error Token Density (ETD), controlled context interventions, and failure-conditioned evaluations. On natural ALFWorld rollouts from Qwen3-8B, success drops from 71% with no error-token accumulation to 6.7% when error tokens occupy more than 40% of the trajectory context. Controlled interventions provide causal evidence that accumulated failure context substantially reduces task success, while removing it improves performance. Across ReTool, OpenThinkImg, and ALFWorld, the same failure-conditioned gap appears even for advanced closed-source models. To reduce this failure-induced context pollution, we propose Tool-Feedback Advantage Shaping (TFAS), which combines an accuracy advantage with a tool-feedback advantage estimated from trajectory-level tool-use quality. Unlike outcome-only reinforcement learning, this objective preserves final task credit while separately discouraging invalid intermediate actions. Experiments across three environments and both 4B/8B text and vision-language models show that TFAS achieves the best overall accuracy in all six model-environment settings; notably, on ALFWorld with Qwen3-8B, it improves outcome-only RL from 31.2% to 41.2% overall accuracy and from 21.0% to 52.4% accuracy on error-containing trajectories.

Primary area:
reinforcement learning
Keywords:
Agent, Tool Use, Reinforcement Learning, Agent Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PuA6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how failed actions and their associated feedback affect subsequent decision-making in LLM agents. It introduces Error Token Density (ETD) to quantify failure-related content in interaction histories and conducts controlled insertion and deletion experiments to examine its impact. The proposed Tool-Feedback Advantage Shaping (TFAS) method combines separately normalized task-success and tool-validity advantages during reinforcement learning. Experiments across ReTool, OpenThinkImg, and ALFWorld report improvements in overall task success over outcome-only RL and an auxiliary tool-validity reward baseline.

Critical strengths
  1. The paper studies a relevant problem and includes controlled interventions.
  2. The reported overall performance improvements extend across multiple settings.
  3. The implementation is simple and uses readily available supervision.
Critical weaknesses
  1. The definition of an error does not adequately distinguish harmful failures from informative exploration.\textbf{The definition of an error does not adequately distinguish harmful failures from informative exploration.} In partially observed or unfamiliar environments, an unsuccessful action may reveal an unknown constraint and improve subsequent planning. Leaving the environment state unchanged does not imply that an action provides no useful information. Although Section 3.1 acknowledges that error feedback can provide negative constraints, this distinction is not operationalized in ETD or the tool-validity reward. Consequently, lower error frequency does not necessarily indicate better decision-making. TFAS could discourage informative failed probes, and the paper provides insufficient analysis of whether useful exploration is preserved. This limits the interpretation of error reduction as evidence of improved robustness.
  2. The optimization signal is insufficiently aligned with the diagnosed failure mechanism.\textbf{The optimization signal is insufficiently aligned with the diagnosed failure mechanism.} The analysis attributes degradation to accumulated failure content, whereas TFAS optimizes the fraction of tool invocations that are valid and executable. Execution validity is a limited proxy for action correctness and contextual interference: an executable action may be strategically incorrect, irrelevant, or repetitive. Moreover, the validity ratio does not capture the length, timing, or informational content of error feedback. Trajectories with identical validity ratios can therefore impose different contextual burdens. The credit-assignment interpretation is also overstated. Section 4.1 broadcasts a single trajectory-level tool advantage to all response tokens. This provides a preference between trajectories but does not explicitly assign different credit to valid and invalid actions within the same trajectory. The authors should clarify this limitation and demonstrate that TFAS specifically mitigates the contextual mechanism identified in Section 3.
  3. Prior work on learning calibration and self-correction capabilities is insufficiently discussed.\textbf{Prior work on learning calibration and self-correction capabilities is insufficiently discussed.} Existing research has already investigated training agents to identify deviations and correct their behavior during execution, thereby limiting error accumulation. STeCa [1] studies timely step-level trajectory calibration, while Agent-R [2] uses revision trajectories and iterative self-training to develop on-the-fly reflection and error correction. These are directly relevant to the paper’s motivation and robustness claims.
  4. The evaluation does not clearly establish improved recovery from errors.\textbf{The evaluation does not clearly establish improved recovery from errors.} ErrAcc conditions on trajectories containing errors, but the membership of this subset changes with the policy. An increase in ErrAcc can therefore reflect changes in which tasks produce errors, rather than improved recovery from comparable failures. The results are also not uniform: on OpenThinkImg with Qwen3-VL-4B, TFAS improves overall accuracy but has lower ErrAcc than outcome-only RL, at 10.7% versus 19.4%. A comparison from matched erroneous prefixes would more directly assess recovery. Measurements of post-training failure-context accumulation would also help connect the performance gains to the proposed mechanism. The current results support improved overall performance more clearly than they support a general improvement in robustness after errors.

[1] Wang H, Wang J, Leong C T, et al. Steca: Step-level trajectory calibration for llm agent learning. ACL2025.

[2] Yuan S, Chen Z, Xi Z, et al. Agent-r: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425, 2025.

Questions for authors
  1. What evidence connects execution validity to harmful contextual interference? How does broadcasting a trajectory-level advantage substantiate the claimed distinction between valid and invalid actions within a trajectory?
  2. How does TFAS differ from GDPO [3] and OTC-PO [4] at the optimization level?

[3] Liu S Y, Dong X, Lu X, et al. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization[J]. arXiv preprint arXiv:2601.05242, 2026.

[4] Wang H, Qian C, Zhong W, et al. Acting less is reasoning more! teaching model to act efficiently[J]. arXiv preprint arXiv:2504.14870, 2025.

Reviewer_pMRUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how erroneous interaction histories affect subsequent decision-making in LLM agents. The authors introduce the ETD metric to analyze erroneous contexts and conduct controlled experiments in ALFWorld to study their effects. They further propose Tool-Feedback Advantage Shaping (TFAS), which combines task success signals with tool-call validity signals for reinforcement learning. Experiments across three environments and six model–environment settings show that TFAS achieves the highest overall accuracy in all settings, improving Qwen3-8B's accuracy on ALFWorld from 31.2% to 41.2%.

Critical strengths
  1. Reasonable controlled experimental design. The authors compare the effects of erroneous content and length-matched neutral content, and examine the impact of removing failed interactions. These experiments provide empirical evidence that erroneous histories may impair subsequent decision-making.

  2. Simple method with promising results. TFAS does not require an additional reward model and achieves the highest overall accuracy across six settings in three environments, demonstrating its potential practical value.

Critical weaknesses
  1. Insufficient causal evidence across domains. Controlled interventions are conducted primarily in ALFWorld, while results from other environments mainly provide correlational evidence. Therefore, whether erroneous histories consistently cause performance degradation across different environments remains insufficiently validated.

  2. The proposed mechanism is not sufficiently validated. TFAS improves overall accuracy and generally reduces the proportion of error-containing trajectories, but the paper does not directly demonstrate that it improves recovery from the same erroneous history. Thus, the link between performance gains and mitigation of context pollution remains unclear.

Questions for authors
  1. How are errors defined and identified across different environments?

  2. How is the mixing weight in TFAS selected? Are the baselines tuned with comparable effort?

  3. Are the main experiments repeated across independent runs? Are the reported performance gains consistent across runs?

  4. In Figure 7, the task is to put a cellphone in a drawer, but the displayed trajectory involves potatoes and plates, yet the environment returns TASK_SUCCESS. How do the authors explain this inconsistency?

Reviewer_eUsXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how failed actions and their feedback affect subsequent decisions in long-horizon LLM agents. The authors introduce Error Token Density (ETD) to measure accumulated error context and use controlled interventions to study its impact. They further propose Tool-Feedback Advantage Shaping (TFAS), which separately normalizes and combines task-outcome and tool-validity advantages to reduce invalid actions and improve task success.

Critical strengths

The problem is practically relevant. Beyond analyzing the correlation between failed trajectories and task failures, the paper conducts controlled error insertion and deletion experiments to provide more direct evidence.

TFAS is simple and easy to integrate into existing RL frameworks. It achieves the highest overall accuracy across all six model-environment settings.

Critical weaknesses

The methodological novelty is relatively limited. TFAS essentially combines two trajectory-level advantages, assigning the same tool-feedback advantage to all steps within a trajectory. It therefore cannot directly distinguish which actions caused errors. Moreover, tool-call validity does not necessarily reflect whether an action contributes to task completion.

The controlled causal experiments on error context are limited to Qwen3-8B in ALFWorld, while the other environments mainly provide correlational evidence. Furthermore, the training experiments do not directly report how TFAS changes ETD, leaving its claimed mechanism of reducing context pollution insufficiently validated.

TFAS does not consistently improve all metrics. For example, on OpenThinkImg with Qwen3-VL-4B, accuracy on error-containing trajectories drops from 19.4% to 10.7% compared with outcome-only RL. In addition, ReTool and OpenThinkImg lack the same comprehensive repeated-run confidence intervals reported for ALFWorld, making the stability of some improvements unclear.

Figure 7 in the appendix contains a clear inconsistency. The stated task is to put a cellphone in a drawer, but the supposedly successful trajectory operates on a potato and a plate without demonstrating completion of the stated task. This discrepancy needs clarification.

Questions for authors

Since all steps share the same trajectory-level advantage, how does TFAS distinguish incorrect actions from correct ones? Have step-level feedback signals been considered?

Could the authors report ETD before and after training and explain the decrease in error-conditioned accuracy on OpenThinkImg?

Why does the successful trajectory in Figure 7 not match the stated task? Is this a figure-labeling error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.