- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how failed actions and their associated feedback affect subsequent decision-making in LLM agents. It introduces Error Token Density (ETD) to quantify failure-related content in interaction histories and conducts controlled insertion and deletion experiments to examine its impact. The proposed Tool-Feedback Advantage Shaping (TFAS) method combines separately normalized task-success and tool-validity advantages during reinforcement learning. Experiments across ReTool, OpenThinkImg, and ALFWorld report improvements in overall task success over outcome-only RL and an auxiliary tool-validity reward baseline.
Critical strengths
- The paper studies a relevant problem and includes controlled interventions.
- The reported overall performance improvements extend across multiple settings.
- The implementation is simple and uses readily available supervision.
Critical weaknesses
- In partially observed or unfamiliar environments, an unsuccessful action may reveal an unknown constraint and improve subsequent planning. Leaving the environment state unchanged does not imply that an action provides no useful information. Although Section 3.1 acknowledges that error feedback can provide negative constraints, this distinction is not operationalized in ETD or the tool-validity reward. Consequently, lower error frequency does not necessarily indicate better decision-making. TFAS could discourage informative failed probes, and the paper provides insufficient analysis of whether useful exploration is preserved. This limits the interpretation of error reduction as evidence of improved robustness.
- The analysis attributes degradation to accumulated failure content, whereas TFAS optimizes the fraction of tool invocations that are valid and executable. Execution validity is a limited proxy for action correctness and contextual interference: an executable action may be strategically incorrect, irrelevant, or repetitive. Moreover, the validity ratio does not capture the length, timing, or informational content of error feedback. Trajectories with identical validity ratios can therefore impose different contextual burdens. The credit-assignment interpretation is also overstated. Section 4.1 broadcasts a single trajectory-level tool advantage to all response tokens. This provides a preference between trajectories but does not explicitly assign different credit to valid and invalid actions within the same trajectory. The authors should clarify this limitation and demonstrate that TFAS specifically mitigates the contextual mechanism identified in Section 3.
- Existing research has already investigated training agents to identify deviations and correct their behavior during execution, thereby limiting error accumulation. STeCa [1] studies timely step-level trajectory calibration, while Agent-R [2] uses revision trajectories and iterative self-training to develop on-the-fly reflection and error correction. These are directly relevant to the paper’s motivation and robustness claims.
- ErrAcc conditions on trajectories containing errors, but the membership of this subset changes with the policy. An increase in ErrAcc can therefore reflect changes in which tasks produce errors, rather than improved recovery from comparable failures. The results are also not uniform: on OpenThinkImg with Qwen3-VL-4B, TFAS improves overall accuracy but has lower ErrAcc than outcome-only RL, at 10.7% versus 19.4%. A comparison from matched erroneous prefixes would more directly assess recovery. Measurements of post-training failure-context accumulation would also help connect the performance gains to the proposed mechanism. The current results support improved overall performance more clearly than they support a general improvement in robustness after errors.
[1] Wang H, Wang J, Leong C T, et al. Steca: Step-level trajectory calibration for llm agent learning. ACL2025.
[2] Yuan S, Chen Z, Xi Z, et al. Agent-r: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425, 2025.
Questions for authors
- What evidence connects execution validity to harmful contextual interference? How does broadcasting a trajectory-level advantage substantiate the claimed distinction between valid and invalid actions within a trajectory?
- How does TFAS differ from GDPO [3] and OTC-PO [4] at the optimization level?
[3] Liu S Y, Dong X, Lu X, et al. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization[J]. arXiv preprint arXiv:2601.05242, 2026.
[4] Wang H, Qian C, Zhong W, et al. Acting less is reasoning more! teaching model to act efficiently[J]. arXiv preprint arXiv:2504.14870, 2025.