acceptodds
Under review as a conference paper at ICLR 2027

Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

Abstract

Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for improving these capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. We identify a recurring failure mode, termed structural collapse, in which performance drops sharply as valid tool calls become malformed control-token sequences. We analyze this failure at the token level and find that a small set of control tokens appears repeatedly across trajectories, accumulating reward-weighted training signals. They accumulate greater absolute advantage per token type than ordinary tokens, and their probabilities change much more sharply before collapse. Changing the invocation format recovers some performance, suggesting that part of the tool-use capability remains accessible. We further show that masking the direct policy-gradient (PG) terms at control-token positions can mitigate collapse. Building on these findings, we systematically compare supervisory signals, including off-policy supervision, hint-based guidance, and erroneous trajectory supervision, under synchronous and interleaved training schemes. We also introduce process reflection supervision, which uses teacher-generated error analyses and related examples to address the model's evolving failures. Across the evaluated models, interleaving supervised fine-tuning (SFT) with RL often improves stability, although the most effective supervision strategy varies by model. These improvements do not consistently transfer to format and content out-of-distribution (OOD) evaluation. Further analyses examine learning-rate effects and generalization across settings. Together, our findings identify a concrete failure mode in tool-use RL, provide token-level evidence about its development, and show how supervisory signals affect stability and generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool Learning, Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8aPbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates structural collapse in multi-turn tool-use reinforcement learning, where valid tool calls degenerate into malformed control-token sequences. Through token-level analyses and policy-gradient masking experiments, the authors identify repeated updates to control tokens as a potential source of training instability. To address this issue, they systematically compare different supervisory signals and propose Process Reflection Supervision (PRS), which incorporates error analysis, synthetic scenarios, and corrective examples into interleaved SFT and RL training. Experiments on BFCL-V3 demonstrate improved training stability and in-distribution performance, while evaluations on ACEBench reveal limited out-of-distribution generalization.

Critical strengths
  1. In-depth analysis of training instability. The paper provides a detailed empirical investigation of training instability across different model families under both vanilla RL and SFT+RL settings. The trajectory-level and token-level analyses, together with control-token masking experiments, offer valuable insights into the potential causes of structural collapse, making the findings more informative than simply observing performance degradation.

  2. Intuitive and practical solutions. The proposed supervision-based approaches are straightforward and well-motivated by the observed failure modes. In particular, interleaving corrective SFT with RL provides a clear and practical strategy for improving training stability, with empirical results demonstrating its effectiveness across several model settings.

Critical weaknesses
  1. Insufficient evidence for the proposed collapse mechanism. Although the token-level analysis and control-token masking experiments provide useful insights, the evidence is not sufficient to establish control-token updates as a primary cause of structural collapse. The higher accumulated advantage mainly reflects the frequent occurrence of a small set of control tokens, rather than unusually large per-token gradients. Moreover, the masking experiments are limited, and alternative factors such as prompt-format differences may also contribute to instability. Stronger controlled experiments are needed to substantiate the proposed mechanism.

  2. Limited connection between the proposed method and the identified failure mechanism. While PRS improves training stability, it does not directly address the abnormal control-token updates identified in the analysis. Furthermore, PRS combines process reflection, synthetic data augmentation, and corrective SFT without sufficient ablations to isolate their individual contributions. It therefore remains unclear whether the improvements come from process reflection or simply additional supervision and teacher-generated data. The limited OOD generalization further raises questions about whether PRS fundamentally improves tool-use learning rather than primarily stabilizing in-distribution training.

Questions for authors
  1. Generalization to stronger and more recent models. Does the observed structural collapse also occur in newer models with stronger inherent agentic and tool-use capabilities, such as Qwen3.5-4B? Evaluating such models under both vanilla RL and SFT+RL settings would help clarify whether structural collapse is a general issue in multi-step tool-use RL or is primarily associated with models that have relatively weak initial tool-use capabilities.

  2. Effectiveness and underlying mechanism of PRS. Can the authors provide further ablation studies to disentangle the contributions of process reflection, synthetic data augmentation, and corrective SFT? In particular, does PRS actually mitigate the abnormal control-token probability shifts identified in the analysis, or are its improvements mainly attributable to additional supervised training? Clarifying this would strengthen the connection between the proposed method and the identified failure mechanism.

1 comment

Thank you for your carefully responses.

Response to W1. The evidence is insufficient to establish control-token updates as a primary cause of structural collapse. Higher accumulated advantage may mainly reflect frequent occurrence, masking experiments are limited, and prompt-format differences may contribute to instability.

We consider the accumulated-advantage analysis, token-probability measurements, and masking experiments to support our interpretation that control-token updates contribute to collapse.

Greater accumulated advantage can produce larger changes through RL updates if the rewards are consistent, as discussed in Appendix A. Figure 3 further shows that the probabilities of control tokens change more sharply during training. Thus, our explanation concerns the accumulated effect of repeated occurrences, rather than requiring unusually large gradients at each individual occurrence.

Regarding prompt-format differences, the masking experiments use the same prompts across the compared settings. Prompt differences therefore do not explain the observed benefit in these experiments. Taken together, these results support our attribution of collapse to changes involving the special control tokens.

Response to W2. PRS does not directly address the abnormal control-token updates, and its combined use of reflection, synthetic augmentation, and corrective SFT lacks ablations. Limited OOD generalization also leaves unclear whether PRS improves tool-use learning or mainly stabilizes in-distribution training.

Compared with ETS, PRS adds textual SFT data. Although the generated text contains process reflection, related scenarios, and corrective examples, these are produced together through a single prompt. They are not separate data-generation modules. Isolating one element would require changing the prompt and therefore changing the PRS procedure. The complete prompt and an example are provided in Appendix F.

Regarding generalization, PRS improves over the other evaluated training methods in content OOD and exhibits the smallest decline relative to the vanilla model. We view these results as supporting the effectiveness of the method.

Our design adds textual knowledge to ETS, whose supervision consists of multi-turn trajectories in a consistent tool-call format. The additional PRS supervision consists of single-turn textual responses; it does not add further multi-turn tool-call trajectories. Introducing external knowledge through teacher-generated supervision is also a common approach in distillation. We therefore interpret PRS as improving the tool-use learning process through this additional textual supervision.

Response to Q1. Does structural collapse also occur in stronger and newer models, such as Qwen3.5-4B, under vanilla RL and SFT+RL? Is the phenomenon general, or mainly associated with weaker initial tool-use capabilities?

Larger or stronger models may not exhibit the same collapse, although they may encounter other forms of format-related failure. Some problems observed in smaller models disappear at larger scales; nevertheless, models around the 7B scale remain a common setting for this type of investigation.

Our conclusion is that structural collapse occurs in the evaluated settings, and that faster probability changes in frequently repeated control tokens provide a possible explanation. We also present supervision-based methods for mitigating the observed problem.

Response to Q2. Can further ablations disentangle process reflection, synthetic augmentation, and corrective SFT? Does PRS mitigate abnormal control-token probability shifts, or do its gains mainly come from additional supervised training?

SFT directly trains the model to assign higher probability to ground-truth sequences, rather than relying on model-sampled trajectories as the training targets. This provides a mechanism through which corrective supervision can influence token-probability changes.

As discussed in Response to W2, the reflection, related scenarios, and corrective examples are generated together through an integrated prompt. They are not implemented as separate modules. Removing individual components would require modifying that prompt and would therefore change the method being evaluated. Our explanation of the benefit is that the added SFT supervision directly guides token probabilities toward correct outputs.

Reviewer_JkopOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Authors study instability issue of multi-turn tool-use RL and present a failure mode called structural collapse, where tool-calls gradually degrade into malformed control-token sequences. According to authors claim, the link is the repeated reinforcement of a small set of control tokens and masking these tokens can delay collapse. They then compare several ways of combining supervision with RL and introduce Process Reflection Supervision (PRS), which combines teacher-generated error analyses, synthetic related scenarios, and corrective tool-use examples. The analysis is evaluated on BFCL-V3 for in-distribution performance and on ACEBench for out-of-distribution generalization, using Qwen2.5-1.5B, Qwen2.5-7B, and Qwen3-1.7B.

Critical strengths
  • Authors focus on a important problem in multi-step tool-use RL where the finding with error taxonomy can be useful for other agent researchers.
  • The comparison of different supervisory strategies within a common framework, reporting both positive and negative results, and distinguishing content from format shifts helps clarify where supervision is beneficial and where its benefits fail to generalize.
Critical weaknesses
  • The main limitation of current version is the lack of clarity in the experimental setup and controls, which makes it difficult to judge the robustness of the observations and the strength of the conclusions. Although authors provide basic/high-level RL hyperparameters in Appendix, differences in prompt formatting and training duration complicate comparisons between the baselines and proposed methods. Moreover, the reliance on small Qwen models and BFCL-based RL training leaves open whether the findings generalize across other model families and training distributions or reflect properties of the particular setup. Last but not least, the overall presentation of the work would benefit from a clearer flow with connection between the research questions, experimental design choices, and supporting evidence, so that readers can distinguish well-supported findings from interpretations that require further validation. It is hard to follow current version of the presentation of the work. Citation style should also be fixed (e.g., cite->citep in overleaf)
  • It is unclear how much of PRS’s gain comes from process reflection itself rather than from the additional supervision it introduces. For instance, compared with ETS, PRS adds teacher-generated error analyses, synthetic scenarios, corrective examples, and a different trajectory-selection rule, all produced by a stronger model (i.e., gpt-5-mini). Authors acknowledge that the comparison evaluates this whole package, but as a result it is again unclear whether reflection adds anything beyond more teacher-generated training data. That said, a comparison with augmentation alone with the same selection strategy and supervision budget would be helpful to understand what actually drives the improvement.
  • On the other hand, in Table 2, PRS remains below the vanilla model in aggregate ACEBench performance under both OOD settings, which limits the evidence that improved training stability translates into more generalizable tool-use ability.
Questions for authors
  1. Could the authors repeat the core analysis with at least one additional training source, for example xLAM [1] (single-turn) or CoALM [2] (multi-turn), and with a model outside the Qwen family, such as Olmo-8b or Llama-3.1-8b? Relatedly, would the findings hold in a more complex environment such as tau2-bench?
  2. In the authors' own words, since it is difficult to trace in the current version of the manuscript: why does RL collapse in these settings, and why does SFT, which reduces the entropy available to the subsequent RL stage, fix it? Does the paper offer a theoretical account, supported by experimental evidence, that actually answers the question posed in the title?

References

[1] Zhang et al. “xLAM: A Family of Large Action Models to Empower AI Agent Systems.” NAACL 2025

[2] Acikgoz et al. “Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model.” ACL 2025

3 comments

Thank you for your carefully responses and suggestions.

Response to W1. The experimental setup and controls are insufficiently clear. Differences in prompt formatting and training duration complicate comparisons, model and training-distribution coverage are limited, and the presentation should better connect research questions, experimental choices, and supporting evidence. Citation formatting also needs correction.

We clarify the experimental setup and presentation as follows.

  • Prompt formatting. We use consistent prompts during training and follow each model’s default chat template.
  • Training duration. Different methods inherently incur different training costs, which complicates comparisons based on elapsed time. Enforcing identical training durations would also disadvantage some methods because their supervision procedures require different amounts of computation.
  • Model coverage. Our experiments include Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-1.7B. Although we do not evaluate larger models, we consider the inclusion of a 7B model sufficient for the scale of the present investigation. The evaluated models differ in size and architecture, and the two model series may also differ in their training data. Observing collapse in all three supports our interpretation that the phenomenon is not unique to a single experimental configuration.
  • Presentation. We will revise the manuscript to make the connections between the research questions, experimental design choices, and supporting evidence easier to follow, and to distinguish supported findings from interpretations requiring further validation. The explanation in Response to Q2 also addresses this concern.
  • Citation formatting. We will correct the citation commands, including the use of \citep where appropriate.

Response to W2. It is unclear how much of PRS’s gain comes from process reflection rather than additional supervision. A comparison with augmentation alone, using the same selection strategy and supervision budget, would help identify the source of improvement.

We would like to clarify the form of the additional supervision in PRS. Compared with ETS, PRS adds textual supervision. ETS uses trajectories in the tool-call format, whereas PRS additionally asks the teacher to generate error analyses, related scenarios, and corrective examples. These elements are presented together in a textual response; they are not introduced as separate collections of additional training trajectories or independently expanded samples. The prompt and an example are provided in Appendix F.

For example, the prompt includes the following instruction: Provide one scenario that is similar but requires a different solution

  • Explain the similarity in reasoning or decision features.
  • Explain clearly why different tools are required.
  • Provide correct tool calls in the same <tool_call> format.
  • Provide a rationale.

We do not introduce an additional data-selection method. The central design of PRS is to add this textual supervision on top of the ETS data. Under a fixed supervision budget, retaining the added text would require removing part of the ETS supervision, which would change the method as currently defined.

Response to W3. PRS remains below the vanilla model in aggregate ACEBench performance under both OOD settings, limiting the evidence that improved training stability produces more generalizable tool-use ability.

Our OOD experiments are intended to examine two points.

  1. The role of tool-call format in collapse. The recovery of tool-use performance after changing the invocation format supports our interpretation that collapse is associated with the special control tokens used in the original format.
  2. The relative generalization of PRS. PRS performs better than the other evaluated training methods on OOD evaluation. Our interpretation is that the added textual reflection helps reduce overfitting. Because the OOD tasks involve unseen data and scenarios, a decline relative to the vanilla model can occur. We will revise the wording to state the intended comparative claim more precisely: “Improved training stability translates into more generalizable tool-use ability compared with the other evaluated training methods.”

Response to Q1. Could the core analysis be repeated with an additional training source, such as xLAM or CoALM, a model outside the Qwen family, and a more complex environment such as tau2-bench?

The suggested xLAM and CoALM datasets provide SFT data, whereas our experiments require an executable environment for multi-turn RL training. We therefore cannot directly use these datasets to add equivalent RL experiments, but we will discuss the corresponding papers in the manuscript.

Additional experiments with Llama are beyond our available computational resources because of the training time required. Our current experiments demonstrate collapse across three models with different sizes and architectures, whose training data may also differ.

We use the OOD evaluation on ACEbench and the result show that if same tool call format are use the result will be zero on τ-bench.

Response to Q2. Why does RL collapse in these settings, and why can SFT mitigate it despite reducing the entropy available to subsequent RL? Does the paper provide a theoretical account supported by experimental evidence?

Why RL collapses. We use a standard RL training procedure and the models’ default templates. Our explanation focuses on the special control tokens used in tool calls: their repeated occurrence concentrates accumulated training signals, and their probabilities change sharply during training, eventually producing malformed tool-call sequences.

Why SFT helps. SFT trains the model against correct target sequences, directly increasing the probability of the ground-truth outputs. In contrast, RL learns from trajectories sampled by the model and weighted by their rewards. These sampled trajectories are not constrained to remain correct, and errors in them can contribute to unstable training. Corrective SFT provides an explicit target distribution that guides the model back toward valid outputs.

Theoretical account. The supporting analysis is provided in Appendix A.

Reviewer_EbN5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies structural collapse in multi-step tool-use reinforcement learning, where valid tool calls deteriorate into malformed control-token sequences. Token-level diagnostics associate collapse with repeated exposure of control tokens and disproportionate probability changes. Masking their direct policy-gradient contributions prevents collapse in one trial and delays it in another. The paper also compares several supervisory strategies and introduces Process Reflection Supervision (PRS), which combines teacher-generated analysis, synthetic scenarios, and corrective examples. Results on three Qwen models show that supervision can improve training stability, but the best strategy varies by model, and in-distribution gains do not consistently transfer to out-of-distribution evaluation.

Critical strengths
  1. Meaningful mechanistic contribution. The experiments show that masking the direct policy-gradient contributions at control-token positions can mitigate structural collapse and improve training stability.
  2. Useful negative results. Larger models can still collapse, PRS does not consistently outperform ETS, and improved training stability does not guarantee better OOD performance.
Critical weaknesses
  1. Insufficient experiments for causal attribution. It remains unclear whether control-token masking helps because these tokens are particularly susceptible to training–inference mismatch, or because of the repeated-exposure concentration mechanism emphasized in the paper.
  2. The empirical scope is narrow. Training uses 300 BFCL questions and only Qwen models. OOD results remain weak, and variability across repeated runs is not sufficiently characterized.
Questions for authors
  1. During LLM training, a small subset of tokens, such as commas and periods, can account for disproportionately large activation magnitudes or accumulated gradient contributions. This is not necessarily pathological, and such outlier behavior can depend on context and be associated with amplified numerical discrepancies between training and inference. Although masking control-token gradients mitigates collapse, could this benefit primarily reflect the suppression of updates affected by training–inference mismatch, rather than the mechanism proposed in the paper? What evidence distinguishes these explanations?
  2. Could training–inference mismatch be the underlying cause of the observed collapse? In a zero-mismatch setting, would masking the direct policy-gradient contributions at control-token positions still provide additional benefits? How does this intervention compare with IcePop, and does it offer clear advantages in stability or downstream performance under matched experimental conditions?
1 comment

Thank you for your carefully responses and suggestions.

Response to W1. Insufficient experiments for causal attribution. It remains unclear whether control-token masking helps because these tokens are particularly susceptible to training–inference mismatch, or because of the repeated-exposure concentration mechanism emphasized in the paper.

We analyze the cause of structural collapse from two perspectives.

  • Output behavior and invocation format. After collapse, the model tends to produce malformed sequences involving special control tokens such as <tool_call>, rather than other frequent tokens such as braces or commas. Our format-OOD experiments further show that changing the tool-invocation format can recover tool-use performance. These observations support our interpretation that the collapse concerns the tool-call format, particularly its control tokens, rather than frequent vocabulary in general.
  • Token probabilities and accumulated advantages. The same small set of control tokens repeatedly occurs across trajectories and accumulates greater reward-weighted advantage per token type. Their probabilities also change more sharply during training, which we identify as a mechanism contributing to collapse.

Regarding training–inference mismatch, we understand the concern to refer to numerical discrepancies between the vLLM inference backend and FSDP training backend. We consider this an implementation-related explanation that is unlikely to account for our observations for the following reasons.

  1. We use verl, a widely used RL training framework. Many studies investigating training collapse or related phenomena use this framework without separately validating numerical discrepancies between vLLM and the training backend.
  2. The mismatch issue examined in IcePop arose in the training of mixture-of-experts models, whereas our experiments use dense models.

Response to W2. The empirical scope is narrow. Training uses 300 BFCL questions and only Qwen models. OOD results remain weak, and variability across repeated runs is not sufficiently characterized.

Thank you for these suggestions. We address the concerns about training data, model coverage, OOD evaluation, and repeated runs below.

  • Training data. Publicly available environments supporting multi-turn tool-use RL remain limited. It is difficult to obtain a substantially larger training set that also provides executable environments and an appropriate level of difficulty.
  • Model coverage. We evaluate Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-1.7B. These models differ in scale and architecture, and the two model series may also differ in their pretraining data. Although we do not include a model from another family, such as Llama, observing collapse across these three models supports the relevance of the phenomenon beyond a single model configuration.
  • OOD evaluation. These experiments serve two purposes. First, recovering performance after changing the invocation format supports our analysis that collapse is associated with the tool-call format, as discussed in Response to W1. Second, PRS demonstrates advantages over the other evaluated training methods. Our intended claim is that process information provides relative benefits, rather than that PRS necessarily improves OOD performance over the vanilla model. Because the training and evaluation domains differ, improvements in tool-use generalization are not guaranteed.
  • Repeated runs. Multi-turn RL training is computationally expensive, making repeated runs of every experiment impractical within our resources. Nevertheless, we have repeatedly observed the collapse process and verified that the phenomenon recurs.

Questions for Authors

Response to Q1. Could the benefit of control-token masking primarily reflect the suppression of updates affected by training–inference mismatch, rather than the proposed mechanism? What evidence distinguishes these explanations?

The output-level observations, token-probability analysis, and measurements of training–inference probability differences are provided in Response to W1.

Response to Q2. Could training–inference mismatch be the underlying cause? Would control-token masking still help in a zero-mismatch setting, and how does it compare with IcePop under matched conditions?

Our discussion of training–inference mismatch and the applicability of IcePop is provided in Response to W1.

Comments on the paper

1 comment

Dear AC and PC,

Thank you for considering our submission and the rebuttal discussion. Our paper studies structural collapse in multi-step tool-use reinforcement learning, where models begin producing malformed tool calls, and examines how corrective supervision can improve training stability. We summarize the strengths highlighted by the reviewers and our responses to their main concerns.

  1. Strengths highlighted by the reviewers
  • A concrete failure mode and a detailed analysis of how it develops (EbN5, Jkop, and 8aPb). Jkop highlighted the relevance of the problem and the usefulness of the error taxonomy for agent research. 8aPb valued the trajectory- and token-level analyses, and EbN5 highlighted the masking experiments showing that suppressing direct policy-gradient updates at control-token positions can mitigate collapse.
  • A practical approach to stabilizing training (8aPb). 8aPb highlighted interleaving corrective supervised fine-tuning with reinforcement learning as a straightforward intervention motivated by the observed failures and supported across the evaluated model settings.
  • Systematic comparisons and informative negative results (EbN5 and Jkop). Jkop valued the comparison of supervision strategies within a common framework and the distinction between format and content shifts. EbN5 highlighted that collapse can also occur in larger evaluated models, that process reflection supervision (PRS) does not consistently outperform erroneous trajectory supervision (ETS), and that improved training stability does not guarantee better out-of-distribution performance.
  1. Reviewer concerns addressed in the rebuttal
  • Evidence for the role of control tokens (EbN5 and 8aPb). We explained how repeated occurrences can concentrate the accumulated training signal on control tokens, whose probabilities shift sharply before collapse. For 8aPb, we clarified that the masking comparisons used the same prompts. For EbN5, the observed probability shifts, partial recovery after changing the invocation format, and benefits of masking support a contributing role for control-token updates.
  • Experimental protocol and scope (EbN5, Jkop, and 8aPb). For Jkop, we clarified the prompt templates and why training costs differ across supervision methods. In response to EbN5 and 8aPb, we explained that collapse recurred across the evaluated Qwen configurations. Our conclusions apply to those settings.
  • What PRS adds and why corrective supervision helps (Jkop and 8aPb). We clarified that PRS adds teacher-generated error analyses, related scenarios, and corrective examples to ETS through an integrated prompt. .
  • Stability and out-of-distribution generalization (EbN5, Jkop, and 8aPb). We clarified that PRS compared favorably with the other evaluated training methods on out-of-distribution tasks, while the untrained baseline retained an advantage in aggregate performance. The evidence supports a relative benefit among the evaluated training methods. Thank you for considering these clarifications and the evidence presented in the rebuttal.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.