- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether a multi-turn LLM agent can return to correct behavior after a failed step or instead continues making errors. The authors define failure persistence, P(f|f), as the probability that a failed step is followed by another failed step, and sequential consistency, P(p|p), as the probability that a passing step is followed by another passing step. They vary the coefficient β of a KL-motivated training penalty and evaluate these transition statistics on BFCL, HotPotQA, InterCode, ALFWorld, and WebShop, mainly using GRPO-trained Qwen models. In four of the five main settings, failure persistence is higher at β=1 than at β=0, although the best coefficient varies across tasks, models, and training algorithms. The paper further validates and analyzes these observations using within-trajectory label shuffling, held-out HotPotQA evaluation, PPO and model-scale comparisons, gradient diagnostics, observation replacement, and tool-call error analysis.
Critical strengths
-
The paper addresses an important limitation of standard agent evaluation: overall accuracy and final task success do not distinguish maintaining correct behavior from returning to a passing step after failure.
-
The experiments cover several types of agent tasks and include useful controls and diagnostics, such as label shuffling, held-out evaluation, bootstrap confidence intervals, Holm-corrected permutation tests, PPO comparisons, observation replacement, and error-type analysis.
-
The paper reports negative and inconsistent results rather than claiming that a single value of β is universally optimal across tasks, algorithms, and model families. This makes its description of the empirical boundaries relatively careful.
Critical weaknesses
My primary concern is that the paper's central metric does not capture what it claims to. Each step's pass/fail label is, in essence, a position-wise match between the agent's action and the reference sequence at that position; adjacent steps are scored independently, with no causal link between labels. The f→p transition that the authors count as a "recovery" therefore only indicates that the next action again matched the reference, not that the agent corrected its previous mistake. The paper's own ALFWorld example makes this concrete: the agent issues look instead of the second expert action (scored f), yet the third step is scored p simply because its text matches the third expert action. This p results purely from position-wise text matching and is independent of whether the second-step error was ever repaired. A genuine notion of recovery requires that the earlier error actually affected the current step and that the agent overcame that effect; a label assigned independently at each position cannot express this. The §3.1 qualification that observed recovery is defined "under the scoring rule" and is "distinct from completing the task" does not rescue the stronger framing, since the title ("Recover or Cascade") and the abstract and introduction all discuss recovery and error cascades as genuine behavioral phenomena. Because the metric reflects position-wise agreement with a reference rather than true error recovery, the evidence for whether KL regularization affects the agent's ability to recover is, in my view, insufficient.
Table 2 also reveals severe and unexplained degradation of every evaluated RL checkpoint relative to the base model. Answer-overlap outcomes collapse from 28/100 for the base model to 0–5/100 after RL, while step-pass rates decrease from 17.4% to 2.0%–9.4%. Although the paper acknowledges that low pass rates explain most of the near-one failure persistence, it neither diagnoses this degradation nor shows that the transition effects persist among performance-matched, non-degenerate policies. The observed persistence may therefore reflect broad training failure rather than a selective impairment of post-error recovery.
Finally, the proposed mechanism is not directly tested. Appendix G.2 explicitly states that the current observation-swap and transition analyses do not provide the required matched-state controlled-error comparison. Moreover, the gradient diagnostics, response-change rates, and transition statistics come from separate experiments and do not establish a common causal pathway from the penalty to reduced feedback sensitivity and then to impaired recovery. The evidence therefore supports descriptive associations among the implemented penalty, response sensitivity, and label persistence, but not the proposed recovery mechanism.
Questions for authors
See above weaknesses.
2 comments
Thank you for the response. To clarify, my concern is not that recovery should require undoing the erroneous action or restoring the previous state. My concern is the definition of recovery itself. First, once a step has failed, the next action is not necessarily correct even if it matches the reference sequence, because the earlier error has not been addressed. Yet the paper counts it as recovery. Second, a genuine correction may be labeled as a failure. In the ALFWorld example in Appendix A.3, the agent issues look instead of the second expert action and then matches the third, giving (p, f, p). Because the second action was never performed, the third action is premature, yet it counts as recovery. Conversely, if the agent had performed the missed second action at step 3, it would not match the third reference action and would be labeled f. If the authors can convince me that this definition captures recovery from errors, I would consider raising my score. Otherwise, since this issue affects the paper's central claim, I maintain my rating.
Thank you for the valuable review.
W1: Does returning to a passing label mean recovery from the previous error?
A1: We define step-level recovery as returning to reference-consistent behavior after a failed step. P(p|f) measures this step-level return, while P(p|p) measures continued correctness. Recovery here concerns resuming the expected behavior; it does not require undoing the erroneous action or restoring the exact previous state.
We additionally test whether injected errors affect task outcomes through 100 paired interventions per policy in live ALFWorld. Each pair starts from the same reachable state and executes either a correct or reversible erroneous action, followed by a 30-step continuation budget under the same policy. Actions change the environment, and success requires completing the task goal. We also evaluate 100 natural tasks.
Success loss is correct-branch success minus error-branch success for the same policy; pp denotes percentage points.
Across the tested β values, injected errors reduce task success by 10–17 percentage points relative to the paired correct actions. This establishes that these errors affect subsequent task completion, complementing our step-level measurement of whether reference-consistent behavior resumes after failure.
W2: Does high failure persistence reflect poor recovery or overall degradation?
A2: We measure overall task performance and the success loss caused by an injected error separately, using the natural tasks and paired interventions in A1.
Natural task success ranges from 35% to 40%. At β=1, correct-branch success is highest (80%), yet the error-induced loss is largest (17 pp versus 10 pp at β=0). Thus, stronger performance after correct actions can coexist with a larger loss after errors.
W3: Do the separate diagnostics establish a causal mechanism?
A3: We derive the implemented loss, replace observations, and classify action errors to examine specific parts of the proposed mechanism. The derivation shows that increasing β weakens reinforcement of positive-advantage trajectories through the effective advantage A−β (§5.3). Replacing the latest observation while retaining the rest of the history reveals lower response-change rates after training (0.067–0.263 versus 0.716 for the base model; §6.1). Error analysis identifies missing tool calls as the dominant BFCL failure at β=1 (89.2% of outputs versus 6.9% at β=0.1; §6.2).
We also train with verified standard KL under matched conditions and evaluate controlled errors in live ALFWorld. At β=1, correct-branch success is higher, but the measured error-induced success loss is larger: 17 percentage points versus 10 at β=0. These experiments quantify error consequences under standard KL, alongside the loss analysis of training updates and the behavioral diagnostics of observation use and tool calling.