- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates structural collapse in multi-turn tool-use reinforcement learning, where valid tool calls degenerate into malformed control-token sequences. Through token-level analyses and policy-gradient masking experiments, the authors identify repeated updates to control tokens as a potential source of training instability. To address this issue, they systematically compare different supervisory signals and propose Process Reflection Supervision (PRS), which incorporates error analysis, synthetic scenarios, and corrective examples into interleaved SFT and RL training. Experiments on BFCL-V3 demonstrate improved training stability and in-distribution performance, while evaluations on ACEBench reveal limited out-of-distribution generalization.
Critical strengths
-
In-depth analysis of training instability. The paper provides a detailed empirical investigation of training instability across different model families under both vanilla RL and SFT+RL settings. The trajectory-level and token-level analyses, together with control-token masking experiments, offer valuable insights into the potential causes of structural collapse, making the findings more informative than simply observing performance degradation.
-
Intuitive and practical solutions. The proposed supervision-based approaches are straightforward and well-motivated by the observed failure modes. In particular, interleaving corrective SFT with RL provides a clear and practical strategy for improving training stability, with empirical results demonstrating its effectiveness across several model settings.
Critical weaknesses
-
Insufficient evidence for the proposed collapse mechanism. Although the token-level analysis and control-token masking experiments provide useful insights, the evidence is not sufficient to establish control-token updates as a primary cause of structural collapse. The higher accumulated advantage mainly reflects the frequent occurrence of a small set of control tokens, rather than unusually large per-token gradients. Moreover, the masking experiments are limited, and alternative factors such as prompt-format differences may also contribute to instability. Stronger controlled experiments are needed to substantiate the proposed mechanism.
-
Limited connection between the proposed method and the identified failure mechanism. While PRS improves training stability, it does not directly address the abnormal control-token updates identified in the analysis. Furthermore, PRS combines process reflection, synthetic data augmentation, and corrective SFT without sufficient ablations to isolate their individual contributions. It therefore remains unclear whether the improvements come from process reflection or simply additional supervision and teacher-generated data. The limited OOD generalization further raises questions about whether PRS fundamentally improves tool-use learning rather than primarily stabilizing in-distribution training.
Questions for authors
-
Generalization to stronger and more recent models. Does the observed structural collapse also occur in newer models with stronger inherent agentic and tool-use capabilities, such as Qwen3.5-4B? Evaluating such models under both vanilla RL and SFT+RL settings would help clarify whether structural collapse is a general issue in multi-step tool-use RL or is primarily associated with models that have relatively weak initial tool-use capabilities.
-
Effectiveness and underlying mechanism of PRS. Can the authors provide further ablation studies to disentangle the contributions of process reflection, synthetic data augmentation, and corrective SFT? In particular, does PRS actually mitigate the abnormal control-token probability shifts identified in the analysis, or are its improvements mainly attributable to additional supervised training? Clarifying this would strengthen the connection between the proposed method and the identified failure mechanism.
Thank you for your carefully responses.
We consider the accumulated-advantage analysis, token-probability measurements, and masking experiments to support our interpretation that control-token updates contribute to collapse.
Greater accumulated advantage can produce larger changes through RL updates if the rewards are consistent, as discussed in Appendix A. Figure 3 further shows that the probabilities of control tokens change more sharply during training. Thus, our explanation concerns the accumulated effect of repeated occurrences, rather than requiring unusually large gradients at each individual occurrence.
Regarding prompt-format differences, the masking experiments use the same prompts across the compared settings. Prompt differences therefore do not explain the observed benefit in these experiments. Taken together, these results support our attribution of collapse to changes involving the special control tokens.
Compared with ETS, PRS adds textual SFT data. Although the generated text contains process reflection, related scenarios, and corrective examples, these are produced together through a single prompt. They are not separate data-generation modules. Isolating one element would require changing the prompt and therefore changing the PRS procedure. The complete prompt and an example are provided in Appendix F.
Regarding generalization, PRS improves over the other evaluated training methods in content OOD and exhibits the smallest decline relative to the vanilla model. We view these results as supporting the effectiveness of the method.
Our design adds textual knowledge to ETS, whose supervision consists of multi-turn trajectories in a consistent tool-call format. The additional PRS supervision consists of single-turn textual responses; it does not add further multi-turn tool-call trajectories. Introducing external knowledge through teacher-generated supervision is also a common approach in distillation. We therefore interpret PRS as improving the tool-use learning process through this additional textual supervision.
Larger or stronger models may not exhibit the same collapse, although they may encounter other forms of format-related failure. Some problems observed in smaller models disappear at larger scales; nevertheless, models around the 7B scale remain a common setting for this type of investigation.
Our conclusion is that structural collapse occurs in the evaluated settings, and that faster probability changes in frequently repeated control tokens provide a possible explanation. We also present supervision-based methods for mitigating the observed problem.
SFT directly trains the model to assign higher probability to ground-truth sequences, rather than relying on model-sampled trajectories as the training targets. This provides a mechanism through which corrective supervision can influence token-probability changes.
As discussed in Response to W2, the reflection, related scenarios, and corrective examples are generated together through an integrated prompt. They are not implemented as separate modules. Removing individual components would require modifying that prompt and would therefore change the method being evaluated. Our explanation of the benefit is that the added SFT supervision directly guides token probabilities toward correct outputs.