acceptodds
Under review as a conference paper at ICLR 2027

Recover or Cascade? The Hidden Role of KL Regularization in Multi-Turn LLM Agents

Abstract

Preventing errors from propagating through a multi-turn language-model agent is important for reliable task execution. However, evaluations of agent training commonly report accuracy and final task success, which do not distinguish avoiding errors from recovering after them. We study how on-policy reinforcement learning (RL) shapes behavior after errors by varying the coefficient in penalties motivated by Kullback–Leibler (KL) regularization. The intended constraint on departures from a reference policy provides a way to examine whether training preserves existing behavior or allows corrective actions to be learned. We measure failure persistence, the probability of failing after a failed step, and sequential consistency, the probability of passing after a passed step. Across five main benchmark settings, failure persistence is higher at than at in four. Our analyses further reveal that, as changes, higher sequential consistency can coexist with a lower probability of returning to a passing step after failure, distinguishing gains in continued correctness from gains in recovery.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, RL, KL divergence, Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nG55OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether a multi-turn LLM agent can return to correct behavior after a failed step or instead continues making errors. The authors define failure persistence, P(f|f), as the probability that a failed step is followed by another failed step, and sequential consistency, P(p|p), as the probability that a passing step is followed by another passing step. They vary the coefficient β of a KL-motivated training penalty and evaluate these transition statistics on BFCL, HotPotQA, InterCode, ALFWorld, and WebShop, mainly using GRPO-trained Qwen models. In four of the five main settings, failure persistence is higher at β=1 than at β=0, although the best coefficient varies across tasks, models, and training algorithms. The paper further validates and analyzes these observations using within-trajectory label shuffling, held-out HotPotQA evaluation, PPO and model-scale comparisons, gradient diagnostics, observation replacement, and tool-call error analysis.

Critical strengths
  1. The paper addresses an important limitation of standard agent evaluation: overall accuracy and final task success do not distinguish maintaining correct behavior from returning to a passing step after failure.

  2. The experiments cover several types of agent tasks and include useful controls and diagnostics, such as label shuffling, held-out evaluation, bootstrap confidence intervals, Holm-corrected permutation tests, PPO comparisons, observation replacement, and error-type analysis.

  3. The paper reports negative and inconsistent results rather than claiming that a single value of β is universally optimal across tasks, algorithms, and model families. This makes its description of the empirical boundaries relatively careful.

Critical weaknesses

My primary concern is that the paper's central metric does not capture what it claims to. Each step's pass/fail label is, in essence, a position-wise match between the agent's action and the reference sequence at that position; adjacent steps are scored independently, with no causal link between labels. The f→p transition that the authors count as a "recovery" therefore only indicates that the next action again matched the reference, not that the agent corrected its previous mistake. The paper's own ALFWorld example makes this concrete: the agent issues look instead of the second expert action (scored f), yet the third step is scored p simply because its text matches the third expert action. This p results purely from position-wise text matching and is independent of whether the second-step error was ever repaired. A genuine notion of recovery requires that the earlier error actually affected the current step and that the agent overcame that effect; a label assigned independently at each position cannot express this. The §3.1 qualification that observed recovery is defined "under the scoring rule" and is "distinct from completing the task" does not rescue the stronger framing, since the title ("Recover or Cascade") and the abstract and introduction all discuss recovery and error cascades as genuine behavioral phenomena. Because the metric reflects position-wise agreement with a reference rather than true error recovery, the evidence for whether KL regularization affects the agent's ability to recover is, in my view, insufficient.

Table 2 also reveals severe and unexplained degradation of every evaluated RL checkpoint relative to the base model. Answer-overlap outcomes collapse from 28/100 for the base model to 0–5/100 after RL, while step-pass rates decrease from 17.4% to 2.0%–9.4%. Although the paper acknowledges that low pass rates explain most of the near-one failure persistence, it neither diagnoses this degradation nor shows that the transition effects persist among performance-matched, non-degenerate policies. The observed persistence may therefore reflect broad training failure rather than a selective impairment of post-error recovery.

Finally, the proposed mechanism is not directly tested. Appendix G.2 explicitly states that the current observation-swap and transition analyses do not provide the required matched-state controlled-error comparison. Moreover, the gradient diagnostics, response-change rates, and transition statistics come from separate experiments and do not establish a common causal pathway from the penalty to reduced feedback sensitivity and then to impaired recovery. The evidence therefore supports descriptive associations among the implemented penalty, response sensitivity, and label persistence, but not the proposed recovery mechanism.

Questions for authors

See above weaknesses.

2 comments

Thank you for the valuable review.

W1: Does returning to a passing label mean recovery from the previous error?

A1: We define step-level recovery as returning to reference-consistent behavior after a failed step. P(p|f) measures this step-level return, while P(p|p) measures continued correctness. Recovery here concerns resuming the expected behavior; it does not require undoing the erroneous action or restoring the exact previous state.

We additionally test whether injected errors affect task outcomes through 100 paired interventions per policy in live ALFWorld. Each pair starts from the same reachable state and executes either a correct or reversible erroneous action, followed by a 30-step continuation budget under the same policy. Actions change the environment, and success requires completing the task goal. We also evaluate 100 natural tasks.

βNatural task successCorrect-branch successError-branch successSuccess loss
039%72%62%10 pp
0.135%77%67%10 pp
0.540%77%65%12 pp
135%80%63%17 pp

Success loss is correct-branch success minus error-branch success for the same policy; pp denotes percentage points.

Across the tested β values, injected errors reduce task success by 10–17 percentage points relative to the paired correct actions. This establishes that these errors affect subsequent task completion, complementing our step-level measurement of whether reference-consistent behavior resumes after failure.

W2: Does high failure persistence reflect poor recovery or overall degradation?

A2: We measure overall task performance and the success loss caused by an injected error separately, using the natural tasks and paired interventions in A1.

Natural task success ranges from 35% to 40%. At β=1, correct-branch success is highest (80%), yet the error-induced loss is largest (17 pp versus 10 pp at β=0). Thus, stronger performance after correct actions can coexist with a larger loss after errors.

W3: Do the separate diagnostics establish a causal mechanism?

A3: We derive the implemented loss, replace observations, and classify action errors to examine specific parts of the proposed mechanism. The derivation shows that increasing β weakens reinforcement of positive-advantage trajectories through the effective advantage A−β (§5.3). Replacing the latest observation while retaining the rest of the history reveals lower response-change rates after training (0.067–0.263 versus 0.716 for the base model; §6.1). Error analysis identifies missing tool calls as the dominant BFCL failure at β=1 (89.2% of outputs versus 6.9% at β=0.1; §6.2).

We also train with verified standard KL under matched conditions and evaluate controlled errors in live ALFWorld. At β=1, correct-branch success is higher, but the measured error-induced success loss is larger: 17 percentage points versus 10 at β=0. These experiments quantify error consequences under standard KL, alongside the loss analysis of training updates and the behavioral diagnostics of observation use and tool calling.

Thank you for the response. To clarify, my concern is not that recovery should require undoing the erroneous action or restoring the previous state. My concern is the definition of recovery itself. First, once a step has failed, the next action is not necessarily correct even if it matches the reference sequence, because the earlier error has not been addressed. Yet the paper counts it as recovery. Second, a genuine correction may be labeled as a failure. In the ALFWorld example in Appendix A.3, the agent issues look instead of the second expert action and then matches the third, giving (p, f, p). Because the second action was never performed, the third action is premature, yet it counts as recovery. Conversely, if the agent had performed the missed second action at step 3, it would not match the third reference action and would be labeled f. If the authors can convince me that this definition captures recovery from errors, I would consider raising my score. Otherwise, since this issue affects the paper's central claim, I maintain my rating.

Reviewer_7H9EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how a training penalty coefficient affects error persistence in multi-turn LLM agents. It distinguishes failure persistence, (P(f\mid f)), from sequential consistency, (P(p\mid p)), and uses within-episode label shuffling to assess whether temporal ordering adds information beyond failure frequency. Experiments primarily use Qwen3 on five benchmark settings, with supplementary model and algorithm comparisons. Additional analyses examine training gradients, responses to replaced observations, and tool-call errors. Failure persistence is higher at (\beta=1) than at (\beta=0) in four main settings, although coefficient rankings vary. An important qualification is that the implemented penalty in most experiments has a reference-independent gradient, while several evaluation settings measure transitions between proxy labels rather than recovery in an action-dependent environment.

Critical strengths

1.The paper separates continued correctness from returning to a passing step. Reporting both conditional rates can reveal behavior that aggregate accuracy obscures. The within-episode shuffle control is particularly useful: it preserves episode lengths and failure counts, and shows that most of the high HotPotQA failure persistence survives random reordering. This is a worthwhile diagnostic for interpreting agent evaluations. 2.The implementation and statistical audits are concrete and candid. Appendix D.1 derives and numerically checks the implemented gradient, while Appendix B reports transition counts, episode-level uncertainty, paired comparisons, and corrected permutation tests. These analyses help establish what the recorded results actually support. I view this diagnostic work as the strongest informative contribution of the submission.

Critical weaknesses

1.In most experiments, the implemented penalty has a gradient that is independent of the reference policy, as acknowledged in Appendix D.1. These results therefore do not establish how strengthening a KL constraint affects recovery. The PPO comparisons use the same penalty, and the separate BFCL objective remains insufficiently verified. This limits the central contribution of the paper. 2.The evaluations do not clearly establish actual error recovery. ALFWorld and WebShop replay fixed observations regardless of the agent’s actions, while HotPotQA’s overlap-based labels can mark useful retrievals as failures. All trained HotPotQA checkpoints also perform substantially worse than the base model. Consequently, the reported transitions are difficult to distinguish from scoring artifacts and general training deterioration.

Questions for authors

1.Which reported experiments, if any, use a verified reference-dependent KL gradient? In particular, what is the exact BFCL training objective, and does any comparison establish that the observed effect survives replacing the penalty analyzed in Appendix D.1 with a verified KL implementation? 2.Is there evidence that the coefficient-dependent differences persist in live, action-dependent environments, using task-progress or task-success criteria? A comparison from the same reachable state after a controlled error would materially strengthen the interpretation of these transitions as recovery. 3.What explains the substantial held-out HotPotQA deterioration across all trained checkpoints? Can retained trajectories and configurations distinguish failed learning from parsing, decoding, or checkpoint-evaluation mismatches?

2 comments

Thank you for the valuable review.

W1 & Q1: Is the KL gradient verified, including for BFCL, and does the effect persist under standard KL?

A1: We implement full-vocabulary conditional KL with a frozen reference, covering reasoning and action tokens. We compare its computed gradients with finite-difference estimates and test how they change with the reference distribution. The gradients agree with the numerical estimates, change when the reference changes, and become zero when policy and reference distributions coincide.

We run LoRA policy-gradient continuation across β with the same training initialization, optimizer state, task order, and budget. Each policy faces the same 100 controlled-error pairs in ALFWorld: a correct or reversible erroneous action from the same reachable state, followed by a 30-step continuation budget.

βCorrect-branch successError-branch successSuccess loss
072%62%10 pp
0.177%67%10 pp
0.577%65%12 pp
180%63%17 pp

Success loss is correct-branch minus error-branch success for the same policy; pp denotes percentage points.

Under verified standard KL, β=1 shows a larger error-induced success loss than β=0 (17 versus 10 pp), despite higher correct-branch success. Thus, the association between higher β and a larger post-error loss also appears in this standard-KL comparison.

W2 & Q2: Do matched-state tests in environments show β-dependent recovery beyond proxy labels and overall degradation?

A2: We run the matched-state intervention in ALFWorld and score task completion after the correct or erroneous action. Subsequent observations follow the actions actually executed, and success requires achieving the environment goal. This directly evaluates task completion after an error, including recovery through alternative valid paths.

We also evaluate 100 natural tasks, obtaining success rates of 39%, 35%, 40%, and 35% for β=0, 0.1, 0.5, and 1. In the paired tests, injected errors reduce task success by 10–17 pp, with the largest loss at β=1 despite its highest correct-branch success. These results show β-dependent differences in the measured cost of errors in an action-dependent environment, using actual task completion as the outcome.

Q3: Does HotPotQA deterioration reflect failed learning or evaluation mismatches?

A3: We apologize for inadequately validating Table 2’s evaluation setup. The test shows that enabling thinking with only 256 tokens truncates responses and produces invalid actions. We reevaluate the available checkpoints on 100 HotPotQA tasks with thinking off, 256 tokens per turn, and at most eight turns:

ConditionTable 2New evaluation
Base28/10054/100
β=01/10040/100
β=0.15/10033/100
β=15/10050/100

Entries count answers meeting the overlap threshold. In a future revision, we will include the reevaluation and controlled diagnostic results, document the evaluation settings explicitly, and update the discussion of Table 2 accordingly.

Thank you for the response and the additional experiments. However, post-error success in the new ALFWorld test is nearly flat across β\beta, it remains unclear which reported experiments (including BFCL) use a verified KL gradient, and the HotPotQA re-evaluation means that Table 2 and the related held-out analyses need to be recomputed. I will therefore maintain my rating 2(weak rejection).

Reviewer_oj97OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how training penalties motivated by KL regularization affect error recovery in multi-turn LLM agents, distinguishing failure persistence from the ability to remain correct after a successful step. Experiments across five benchmarks show that increasing the penalty coefficient raises failure persistence in four settings, although the best coefficient varies by task and algorithm. Label-shuffling controls indicate that low overall accuracy explains much of the persistence, while observation-replacement tests suggest reduced responsiveness to feedback after training. Crucially, the penalty implemented in four main settings effectively shifts the reward advantage rather than providing the intended KL gradient, limiting conclusions about standard KL regularization. The main contribution is an evaluation framework that separates sustained correctness from recovery, supported by findings whose generalizability is constrained by short training runs and simplified environments.

Critical strengths
  1. Separates error recovery from sustained correctness, revealing behavior that aggregate accuracy can obscure.
  2. Combines benchmark comparisons, label-shuffling controls, gradient analysis, and observation-replacement tests.
Critical weaknesses
  1. Table 2 reports only 0–5/100 answers meeting the overlap threshold for trained models, versus 28/100 for the base model. Step pass rates also fall from 17.4% to 2.0–9.4%, making recovery difficult to separate from general performance degradation.
  2. Section 5.2 and Figure 2 (pp. 6–7) show that shuffling reduces failure persistence by only 0.53–1.34 percentage points. At beta=1, recovery is 3.8% versus 5.1% after shuffling, limiting the evidence for substantial cascade effects beyond low accuracy.
Questions for authors
  1. Given the substantial drop in answer success and step pass rates after training in Table 2, how can the paper distinguish impaired recovery from general performance degradation?
  2. Given the small differences between observed and shuffled failure persistence in Section 5.2 and Figure 2, how strong is the evidence for error cascades beyond what low overall accuracy would explain?
1 comment

Thank you for the feedback.Thank you for the feedback.

W1 & Q1: How is impaired recovery distinct from the overall degradation?

A1: We evaluate each policy on 100 natural tasks and 100 controlled-error pairs in ALFWorld to separate overall performance from the cost of an error. Each pair starts from the same reachable state, executes a correct or reversible erroneous action, and allows 30 further steps. Actions change the environment; success requires completing the goal. All β conditions use gradient-verified standard KL and share the training initialization, optimizer state, task order, and budget.

βNatural task successCorrect-branch successError-branch successSuccess loss
039%72%62%10 pp
0.135%77%67%10 pp
0.540%77%65%12 pp
135%80%63%17 pp

Success loss is correct-branch success minus error-branch success for the same policy; pp denotes percentage points.

At β=1, correct-branch success is highest (80%), yet the error-induced loss is largest (17 pp versus 10 pp at β=0). Natural task success ranges from 35% to 40%. Thus, the larger post-error loss occurs despite stronger correct-branch performance, showing why overall task performance and sensitivity to errors require separate evaluation.

W2 & Q2: Do the small shuffle effects support cascades beyond low accuracy?

A2: We shuffle labels within each HotPotQA episode while preserving its length and pass/fail counts to test for failure persistence beyond overall failure frequency. At β=1, P(p|f) is 3.8% in the original sequences versus 5.1% after shuffling, corresponding to 1.3 percentage points of excess failure persistence (Holm-adjusted p=0.0048). This shows a statistically significant effect of label order on failure persistence: with exactly the same numbers of passes and failures, the original sequences show a lower probability of returning to a passing step after failure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.