- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why the OPSD method fails when applied to retrieval-interleaved search agents. The authors propose "Feedback-Augmented Self-Distillation" (FA-SD) and link training degradation to a phenomenon termed "decoding collapse", a state where the trajectories generated by a model may appear diverse on the surface, but in reality, increasingly rely on fixed reasoning and search templates, becoming insensitive to the specifics of individual problems. Furthermore, the study finds that employing an EMA approach can lead to some improvement.
Critical strengths
- The authors' research objective was very clear: to investigate why search agents fail when using the OPSD method. They explored the possible causes of this failure (which they termed "decoding collapse") and conducted a series of experiments on the Qwen2.5-3B-base model.
- Furthermore, the authors propose a mitigation strategy that employs the EMA method to alleviate this collapse phenomenon.
Critical weaknesses
- The experiments mainly use Qwen2.5-3B-Base as the student, so it is unclear whether the observed decoding collapse is model-specific or a broader issue with FA-SD. Base and instruction-tuned models can differ significantly in instruction following, tool use, and multi-step interaction. Could the authors evaluate additional models, including instruction-tuned models or newer reasoning models, to test whether the same failure patterns persist across different model types?
- The analysis of decoding collapse is not sufficiently comprehensive. The paper lacks a comparison of the initial model, GRPO, MOPD, and other compared methods using the same metrics. As a result, it remains unclear whether the observed collapse is specific to FA-SD or also occurs with other training methods.
- The paper may lacks a simple successful-trajectory SFT baseline, which could help clarify whether the observed failure stems from the distillation mechanism or the demonstrations themselves. Comparing SFT and FA-SD using the same behavioral metrics would help determine whether decoding collapse is specific to online self-distillation or also occurs when learning directly from successful trajectories.
Questions for authors
See Weaknesses.
3 comments
2. Behavioral diagnostics across different training methods
We thank the reviewer for raising this important concern. We agree that a systematic comparison using consistent behavioral metrics across different training methods is essential to determine whether decoding collapse is specific to FA-SD or represents a more general training failure mode. To address this concern, we conduct additional analyses of the initial policy, MOPD, FA-MOPD, and successful-trajectory SFT under a unified diagnostic framework.
Unified diagnostic protocol. We evaluate decoding behavior along three complementary dimensions, computed offline from per-step training rollouts: (1) trajectory similarity, measured by mean cross-question similarity, near-duplicate trajectory fraction (similarity ≥ 0.5), and answer-mode share; (2) query diversity, measured by search rate, first-query entropy, and degenerate-query rate; and (3) input sensitivity, measured by question–query overlap and the trajectory similarity gap between identical and different questions. We focus on training steps 25–50, where FA-SD first exhibits clear collapse signatures, and use the earliest available rollouts as a reference for initial policy behavior.
Initial policy. Under these metrics, the initial Qwen2.5-3B-Base search policy does not exhibit decoding collapse. Early rollouts demonstrate diverse search behavior and meaningful input sensitivity. In contrast, as FA-SD training progresses, the model increasingly converges toward a dominant answer mode, accompanied by reduced query diversity and diminishing input sensitivity. These observations indicate that collapse emerges during FA-SD optimization rather than being an inherent property of the initial policy.
MOPD and FA-MOPD (same metrics and student model). To investigate whether decoding collapse is a general consequence of on-policy distillation, we apply identical diagnostics to Multi-rollout On-Policy Distillation (MOPD) and Feedback-Augmented MOPD (FA-MOPD), both of which train the same Qwen2.5-3B-Base student using an external Qwen2.5-7B-Instruct teacher. The corresponding results are presented in Appendix C.5 ("Collapse Diagnostics for MOPD and FA-MOPD"). During steps 25–50, when FA-SD already exhibits collapse, neither method shows comparable failure signatures:
- Trajectory diversity remains high: Mean cross-question similarity stays around 0.1, near-duplicate trajectories are virtually absent, and answer-mode share decreases rather than converging toward a dominant response pattern.
- Query diversity is preserved: First-query entropy exceeds 10 bits, search rates remain high, and degenerate queries are rare.
- Input sensitivity is maintained or improved: Both question–query overlap and the conditioned trajectory gap increase during training.
These results demonstrate that early decoding collapse is not an inevitable consequence of on-policy distillation and distinguish FA-SD from external-teacher distillation under comparable training settings.
Successful-trajectory SFT (same metrics). We further examine an online successful-trajectory SFT baseline to determine whether collapse arises simply from learning from successful search demonstrations. The corresponding results are presented in Appendix E ("Successful-Trajectory SFT Baseline"). Under the same diagnostic protocol, SFT does not exhibit FA-SD's collapse pattern. Instead, training pass rate improves and remains stable, near-duplicate trajectories remain rare, answer-mode share decreases, search behavior remains diverse, and input sensitivity strengthens throughout training. This provides additional evidence that decoding collapse is not a general consequence of training on successful trajectories.
GRPO. Although not explicitly reported in the original paper, GRPO does not exhibit the decoding-collapse behavior observed in FA-SD, providing further evidence that this failure mode is specific to FA-SD rather than a general consequence of training.
Conclusion. Our cross-method analysis shows that the characteristic decoding-collapse signatures emerge in FA-SD but are not observed in the initial policy, MOPD, FA-MOPD, successful-trajectory SFT, or GRPO. In particular, the unified behavioral diagnostics demonstrate that MOPD, FA-MOPD, and SFT preserve trajectory diversity, query diversity, and input sensitivity, while GRPO provides complementary evidence through its stable training behavior. Together, these findings support the conclusion that the observed collapse is associated with FA-SD-specific training dynamics rather than being an artifact of the diagnostic metrics or a failure mode shared across different training methods.
3. Successful-trajectory SFT baseline
We thank the reviewer for this suggestion. We agree that a successful-trajectory SFT baseline is the right control for separating demonstration quality from the distillation mechanism.
We therefore train an online SFT baseline that imitates successful search trajectories mined from rollouts in the same retrieval-interleaved environment, and we evaluate it with the same behavioral diagnostics used for FA-SD: trajectory similarity, query diversity, and input sensitivity.
Under SFT, training pass rate rises from about 0.07 to about 0.40 and remains stable, while FA-SD collapses. Behaviorally, SFT does not exhibit decoding collapse: the cross-question near-duplicate trajectory rate stays near zero, answer-mode share decreases rather than locking onto a dominant answer, search rate approaches 1 with high first-query entropy, and the same–different question trajectory gap increases, indicating improved input sensitivity. By contrast, FA-SD shows the opposite signatures: answer-mode locking, trajectory homogenization, and loss of input sensitivity.
These results indicate that the observed failure is not caused by unusable demonstrations; the same successful trajectories support stable learning under direct imitation. The failure is specific to online feedback-augmented self-distillation, consistent with our analysis of inconsistent self-teacher supervision. We have added this successful-trajectory SFT control, with the corresponding reward and behavioral-diagnostic plots, to Appendix E ("Successful-Trajectory SFT Baseline").
Response to Reviewer xCPp
We thank the reviewer for the careful evaluation and constructive feedback. We appreciate the recognition of our research motivation, the behavioral investigation of decoding collapse in retrieval-interleaved search agents, and the EMA-based teacher-stabilization experiments. We address the concerns regarding backbone generalizability, behavioral comparisons, and the successful-trajectory SFT baseline below.
1. Model scale and backbone generalizability
We thank the reviewer for highlighting the limited backbone coverage in our original experiments. To examine whether the observed FA-SD decoding collapse is specific to Qwen2.5-3B-Base, we ran the same FA-SD training protocol and behavioral diagnostics on two additional students, covering both model scale and instruction tuning: Qwen2.5-3B-Instruct and Qwen2.5-7B. Results are reported in Appendix D ("Decoding Collapse Across Model Scale and Instruction Tuning").
Pass-rate dynamics. Both additional students exhibit FA-SD training failure rather than sustained improvement. Qwen2.5-3B-Instruct starts from a stronger initial pass rate but collapses to near zero by roughly step 110. Qwen2.5-7B collapses earlier: pass rate drops sharply after ~step 25 and reaches near zero by ~step 75. In the same comparison plot, an SFT run under the same environment continues to improve, indicating that the collapse is associated with FA-SD training dynamics rather than an inability of larger/instruct students to learn the task.
Behavioral signatures match decoding collapse. Using the same three axes as in the main paper (trajectory similarity, query diversity, and input sensitivity):
Conclusion. These results indicate that the observed decoding collapse is not an artifact of a single small base model: the same failure pattern persists for a larger base model (7B) and an instruction-tuned backbone (3B-Instruct). These additional experiments strengthen the evidence that FA-SD decoding collapse extends beyond the original Qwen2.5-3B-Base configuration. Nevertheless, our conclusions remain limited to the evaluated retrieval-interleaved setting and backbones. We acknowledge that our additional evidence is diagnostic (training dynamics + behavioral metrics), without full multi-benchmark final tables or evaluations on 14B / newer reasoning models.