acceptodds
Under review as a conference paper at ICLR 2027

Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?

Abstract

On-policy self-distillation (OPSD) offers a promising approach for training large language models without relying on a separate teacher model. However, its effectiveness on complex agentic tasks remains largely unexplored. In this work, we instantiate Feedback-Augmented Self-Distillation (FA-SD), a self-distillation algorithm for agentic search that leverages successful demonstrations as privileged information. We observe that models can rely on recurring reasoning-and-search output templates, producing trajectories that appear diverse but are largely agnostic to the input question, making the KL-based self-distillation signal less informative. We term this phenomenon decoding collapse, a failure mode that can be missed by aggregate evaluation metrics. To understand why this occurs, we show that although the self-teacher achieves stronger performance, learning remains unstable under inconsistent supervision signals. We further analyze two sources of inconsistency, model inconsistency and prompt inconsistency, and find that prompt-conditioned feedback can degrade the quality of the supervision signal, limiting the effectiveness of self-teacher learning. To mitigate this inconsistency, we introduce an exponential moving average (EMA) teacher to stabilize the self-teacher and provide more consistent supervision signals. Although the EMA teacher requires a warm-up phase during which performance may temporarily regress, it partially improves FA-SD by providing more stable supervision, while leaving a gap to stronger RL and teacher-distillation baselines.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Self-Distillation, Search Agents, Agentic Reinforcement Learning, Feedback Augmentation, Teacher Stability, Decoding Collapse

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xCPpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why the OPSD method fails when applied to retrieval-interleaved search agents. The authors propose "Feedback-Augmented Self-Distillation" (FA-SD) and link training degradation to a phenomenon termed "decoding collapse", a state where the trajectories generated by a model may appear diverse on the surface, but in reality, increasingly rely on fixed reasoning and search templates, becoming insensitive to the specifics of individual problems. Furthermore, the study finds that employing an EMA approach can lead to some improvement.

Critical strengths
  1. The authors' research objective was very clear: to investigate why search agents fail when using the OPSD method. They explored the possible causes of this failure (which they termed "decoding collapse") and conducted a series of experiments on the Qwen2.5-3B-base model.
  2. Furthermore, the authors propose a mitigation strategy that employs the EMA method to alleviate this collapse phenomenon.
Critical weaknesses
  1. The experiments mainly use Qwen2.5-3B-Base as the student, so it is unclear whether the observed decoding collapse is model-specific or a broader issue with FA-SD. Base and instruction-tuned models can differ significantly in instruction following, tool use, and multi-step interaction. Could the authors evaluate additional models, including instruction-tuned models or newer reasoning models, to test whether the same failure patterns persist across different model types?
  2. The analysis of decoding collapse is not sufficiently comprehensive. The paper lacks a comparison of the initial model, GRPO, MOPD, and other compared methods using the same metrics. As a result, it remains unclear whether the observed collapse is specific to FA-SD or also occurs with other training methods.
  3. The paper may lacks a simple successful-trajectory SFT baseline, which could help clarify whether the observed failure stems from the distillation mechanism or the demonstrations themselves. Comparing SFT and FA-SD using the same behavioral metrics would help determine whether decoding collapse is specific to online self-distillation or also occurs when learning directly from successful trajectories.
Questions for authors

See Weaknesses.

3 comments

Response to Reviewer xCPp

We thank the reviewer for the careful evaluation and constructive feedback. We appreciate the recognition of our research motivation, the behavioral investigation of decoding collapse in retrieval-interleaved search agents, and the EMA-based teacher-stabilization experiments. We address the concerns regarding backbone generalizability, behavioral comparisons, and the successful-trajectory SFT baseline below.

1. Model scale and backbone generalizability

We thank the reviewer for highlighting the limited backbone coverage in our original experiments. To examine whether the observed FA-SD decoding collapse is specific to Qwen2.5-3B-Base, we ran the same FA-SD training protocol and behavioral diagnostics on two additional students, covering both model scale and instruction tuning: Qwen2.5-3B-Instruct and Qwen2.5-7B. Results are reported in Appendix D ("Decoding Collapse Across Model Scale and Instruction Tuning").

Pass-rate dynamics. Both additional students exhibit FA-SD training failure rather than sustained improvement. Qwen2.5-3B-Instruct starts from a stronger initial pass rate but collapses to near zero by roughly step 110. Qwen2.5-7B collapses earlier: pass rate drops sharply after ~step 25 and reaches near zero by ~step 75. In the same comparison plot, an SFT run under the same environment continues to improve, indicating that the collapse is associated with FA-SD training dynamics rather than an inability of larger/instruct students to learn the task.

Behavioral signatures match decoding collapse. Using the same three axes as in the main paper (trajectory similarity, query diversity, and input sensitivity):

  • For Qwen2.5-3B-Instruct, search rate declines early; by mid-to-late training the policy locks into a near-unique answer mode (answer-mode share → ~1), cross-question trajectory homogenization becomes extreme (fraction of pairs with traj-sim ≥ 0.5 → ~0.97), first-query entropy collapses to 0, and the same–different question trajectory gap approaches 0.
  • For Qwen2.5-7B, the early-training window already shows answer-mode locking together with loss of input sensitivity (question↔query overlap and same–diff trajectory gap fall toward 0 by ~step 50), i.e., the same early collapse family observed for Qwen2.5-3B-Base.

Conclusion. These results indicate that the observed decoding collapse is not an artifact of a single small base model: the same failure pattern persists for a larger base model (7B) and an instruction-tuned backbone (3B-Instruct). These additional experiments strengthen the evidence that FA-SD decoding collapse extends beyond the original Qwen2.5-3B-Base configuration. Nevertheless, our conclusions remain limited to the evaluated retrieval-interleaved setting and backbones. We acknowledge that our additional evidence is diagnostic (training dynamics + behavioral metrics), without full multi-benchmark final tables or evaluations on 14B / newer reasoning models.

2. Behavioral diagnostics across different training methods

We thank the reviewer for raising this important concern. We agree that a systematic comparison using consistent behavioral metrics across different training methods is essential to determine whether decoding collapse is specific to FA-SD or represents a more general training failure mode. To address this concern, we conduct additional analyses of the initial policy, MOPD, FA-MOPD, and successful-trajectory SFT under a unified diagnostic framework.

Unified diagnostic protocol. We evaluate decoding behavior along three complementary dimensions, computed offline from per-step training rollouts: (1) trajectory similarity, measured by mean cross-question similarity, near-duplicate trajectory fraction (similarity ≥ 0.5), and answer-mode share; (2) query diversity, measured by search rate, first-query entropy, and degenerate-query rate; and (3) input sensitivity, measured by question–query overlap and the trajectory similarity gap between identical and different questions. We focus on training steps 25–50, where FA-SD first exhibits clear collapse signatures, and use the earliest available rollouts as a reference for initial policy behavior.

Initial policy. Under these metrics, the initial Qwen2.5-3B-Base search policy does not exhibit decoding collapse. Early rollouts demonstrate diverse search behavior and meaningful input sensitivity. In contrast, as FA-SD training progresses, the model increasingly converges toward a dominant answer mode, accompanied by reduced query diversity and diminishing input sensitivity. These observations indicate that collapse emerges during FA-SD optimization rather than being an inherent property of the initial policy.

MOPD and FA-MOPD (same metrics and student model). To investigate whether decoding collapse is a general consequence of on-policy distillation, we apply identical diagnostics to Multi-rollout On-Policy Distillation (MOPD) and Feedback-Augmented MOPD (FA-MOPD), both of which train the same Qwen2.5-3B-Base student using an external Qwen2.5-7B-Instruct teacher. The corresponding results are presented in Appendix C.5 ("Collapse Diagnostics for MOPD and FA-MOPD"). During steps 25–50, when FA-SD already exhibits collapse, neither method shows comparable failure signatures:

  • Trajectory diversity remains high: Mean cross-question similarity stays around 0.1, near-duplicate trajectories are virtually absent, and answer-mode share decreases rather than converging toward a dominant response pattern.
  • Query diversity is preserved: First-query entropy exceeds 10 bits, search rates remain high, and degenerate queries are rare.
  • Input sensitivity is maintained or improved: Both question–query overlap and the conditioned trajectory gap increase during training.

These results demonstrate that early decoding collapse is not an inevitable consequence of on-policy distillation and distinguish FA-SD from external-teacher distillation under comparable training settings.

Successful-trajectory SFT (same metrics). We further examine an online successful-trajectory SFT baseline to determine whether collapse arises simply from learning from successful search demonstrations. The corresponding results are presented in Appendix E ("Successful-Trajectory SFT Baseline"). Under the same diagnostic protocol, SFT does not exhibit FA-SD's collapse pattern. Instead, training pass rate improves and remains stable, near-duplicate trajectories remain rare, answer-mode share decreases, search behavior remains diverse, and input sensitivity strengthens throughout training. This provides additional evidence that decoding collapse is not a general consequence of training on successful trajectories.

GRPO. Although not explicitly reported in the original paper, GRPO does not exhibit the decoding-collapse behavior observed in FA-SD, providing further evidence that this failure mode is specific to FA-SD rather than a general consequence of training.

Conclusion. Our cross-method analysis shows that the characteristic decoding-collapse signatures emerge in FA-SD but are not observed in the initial policy, MOPD, FA-MOPD, successful-trajectory SFT, or GRPO. In particular, the unified behavioral diagnostics demonstrate that MOPD, FA-MOPD, and SFT preserve trajectory diversity, query diversity, and input sensitivity, while GRPO provides complementary evidence through its stable training behavior. Together, these findings support the conclusion that the observed collapse is associated with FA-SD-specific training dynamics rather than being an artifact of the diagnostic metrics or a failure mode shared across different training methods.

3. Successful-trajectory SFT baseline

We thank the reviewer for this suggestion. We agree that a successful-trajectory SFT baseline is the right control for separating demonstration quality from the distillation mechanism.

We therefore train an online SFT baseline that imitates successful search trajectories mined from rollouts in the same retrieval-interleaved environment, and we evaluate it with the same behavioral diagnostics used for FA-SD: trajectory similarity, query diversity, and input sensitivity.

Under SFT, training pass rate rises from about 0.07 to about 0.40 and remains stable, while FA-SD collapses. Behaviorally, SFT does not exhibit decoding collapse: the cross-question near-duplicate trajectory rate stays near zero, answer-mode share decreases rather than locking onto a dominant answer, search rate approaches 1 with high first-query entropy, and the same–different question trajectory gap increases, indicating improved input sensitivity. By contrast, FA-SD shows the opposite signatures: answer-mode locking, trajectory homogenization, and loss of input sensitivity.

These results indicate that the observed failure is not caused by unusable demonstrations; the same successful trajectories support stable learning under direct imitation. The failure is specific to online feedback-augmented self-distillation, consistent with our analysis of inconsistent self-teacher supervision. We have added this successful-trajectory SFT control, with the corresponding reward and behavioral-diagnostic plots, to Appendix E ("Successful-Trajectory SFT Baseline").

Reviewer_Ahb9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether Feedback-Augmented Self-Distillation (FA-SD) can provide effective dense token-level supervision for retrieval-interleaved search agents. By conditioning the policy on successful rollouts from the same group as privileged feedback, FA-SD constructs a self-teacher to supervise the unconditioned student via reverse KL divergence. Across seven open-domain QA benchmarks using Qwen2.5-3B-Base, FA-SD fails to sustain improvement. The authors show that while the feedback-augmented self-teacher achieves stronger inference-time accuracy, the student suffers from decoding collapse, relying on recurring reasoning-and-search templates that are largely agnostic to the input question. They attribute this failure to model inconsistency (drifting self-teacher weights) and prompt inconsistency (conditioning mismatch from privileged feedback).

Critical strengths
  1. Timely investigation and rigorous behavioral diagnostics of decoding collapse. Extending on-policy self-distillation (OPSD) to retrieval-interleaved search agents is a natural and well-motivated direction. Rather than merely reporting negative aggregate metrics, the authors design an insightful suite of fine-grained behavioral analysis.
  2. Comprehensive experimental comparisons across multiple distillation variants and ablations. The paper provides detailed empirical analyses covering diverse settings, including unclipped, PPO-clipped, and joint FA-SD+GRPO variants; inference-time comparisons between the student and self-teacher.
Critical weaknesses
  1. All trainable student experiments are restricted exclusively to a single small base model (Qwen2.5-3B-Base), without evaluating larger scales (e.g., 7B/14B) or instruction-tuned backbones.
  2. As shown in Appendix D, the self-teacher prompt directly includes the entire successful rollout—including the retrieved <information> passages and final <answer>. Consequently, while the student must figure out what to search from scratch, the self-teacher already has access to the retrieved evidence and answer in its context. Forcing token-level alignment across search actions under such severe information asymmetry is an ill-posed setup. The authors do not explore more principled feedback designs (e.g., abstract hints without leaked observations, or selective distillation only on post-retrieval tokens).
Questions for authors

See the weaknesses above.

2 comments

Response to Reviewer Ahb9

We thank the reviewer for the thoughtful evaluation and constructive feedback. We appreciate the recognition of the value of studying FA-SD in retrieval-interleaved search agents, particularly our fine-grained behavioral diagnostics and comparisons across different training variants. We address the concerns regarding backbone generalizability and privileged feedback conditioning below.

1. Model scale and backbone generalizability

We thank the reviewer for raising the concern that the FA-SD decoding-collapse findings might be specific to Qwen2.5-3B-Base. To test generality across model scale and instruction tuning, we ran the same FA-SD training protocol and the same behavioral diagnostics on two additional students: Qwen2.5-3B-Instruct and Qwen2.5-7B. Results are reported in Appendix D ("Decoding Collapse Across Model Scale and Instruction Tuning").

Pass-rate dynamics. Both additional students exhibit FA-SD training failure rather than sustained improvement. Qwen2.5-3B-Instruct starts from a stronger initial pass rate but collapses to near zero by roughly step 110. Qwen2.5-7B collapses earlier: pass rate drops sharply after ~step 25 and reaches near zero by ~step 75. In the same comparison plot, an SFT run under the same environment continues to improve, indicating that the collapse is associated with FA-SD training dynamics rather than an inability of larger/instruct students to learn the task.

Behavioral signatures match decoding collapse. Using the same three axes as in the main paper (trajectory similarity, query diversity, and input sensitivity):

  • For Qwen2.5-3B-Instruct, search rate declines early; by mid-to-late training the policy locks into a near-unique answer mode (answer-mode share → ~1), cross-question trajectory homogenization becomes extreme (fraction of pairs with traj-sim ≥ 0.5 → ~0.97), first-query entropy collapses to 0, and the same–different question trajectory gap approaches 0.
  • For Qwen2.5-7B, the early-training window already shows answer-mode locking together with loss of input sensitivity (question↔query overlap and same–diff trajectory gap fall toward 0 by ~step 50), i.e., the same early collapse family observed for Qwen2.5-3B-Base.

Conclusion. These results indicate that the observed decoding collapse is not an artifact of a single small base model: the same failure pattern persists for a larger base model (7B) and an instruction-tuned backbone (3B-Instruct). We therefore view the phenomenon as a broader issue with FA-SD in this retrieval-interleaved setting. We acknowledge that our additional evidence is diagnostic (training dynamics + behavioral metrics), and we have not yet included full multi-benchmark final tables for these students, nor 14B / newer reasoning models; we will note this scope clearly in the appendix.

2. Privileged feedback, information asymmetry, and alternative feedback designs

We thank the reviewer for the comment. We respectfully disagree that providing the self-teacher with successful rollouts, retrieved evidence, and final answers makes token-level distillation inherently ill-posed. Leveraging privileged information unavailable to the student is a well-established paradigm in knowledge distillation, as demonstrated by Learning Using Privileged Information (Vapnik and Izmailov, 2015) and generalized distillation (Lopez-Paz et al., 2016).

More directly, STaR (Zelikman et al., NeurIPS 2022) leverages ground-truth answers to generate improved reasoning trajectories, while Self-Distilled Reasoner (Zhao et al., ICML 2026) explicitly performs token-level self-distillation using a teacher conditioned on privileged verified solutions. These works demonstrate that teacher–student information asymmetry is an intentional mechanism for transferring knowledge rather than an inherent methodological flaw.

We acknowledge that privileged feedback may influence search-action supervision, particularly when the teacher has access to retrieved evidence that the student must independently discover. However, this concerns the quality and transferability of supervision rather than the fundamental validity of the distillation objective.

We appreciate the suggested alternatives, including abstract hints and selective post-retrieval distillation. Our study specifically investigates the effectiveness and failure modes of full-rollout feedback augmentation, rather than claiming that this is the optimal feedback design. Our negative findings therefore apply to the evaluated formulation and do not imply that alternative feedback designs would exhibit the same failures.

Indeed, our Conclusion explicitly identifies determining what to distill from successful rollouts and when feedback-augmented teachers should be trusted as important directions for future work. We will clarify this experimental scope and acknowledge that evaluating alternative feedback representations and selective distillation objectives is a valuable extension of our study.

Reviewer_yR4dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates feedback-augmented self-distillation for retrieval-interleaved search agents. The proposed instantiation, FA-SD, conditions a self-teacher on successful peer rollouts and distills its token-level predictions into a student that does not receive this additional context. Retrieved observations are excluded from the distillation loss.

The experiments show that unregularized FA-SD can suffer severe training degradation. Behavioral diagnostics examine trajectory similarity, answer concentration, query diversity, and input sensitivity. The authors associate the observed collapse with changing teacher parameters and feedback-conditioned supervision. Fixed-reference and EMA teachers partially stabilize training, with EMA improving over the initial model while remaining below the reported RL and external-teacher baselines. Additional experiments investigate feedback augmentation with an external teacher.

Critical strengths

The paper tests whether successful-rollout feedback, as used in SDPO [1], transfers effectively to retrieval-interleaved agents. Documenting failure and partial recovery is useful for practitioners considering this training approach, particularly because stronger prompted inference does not automatically imply effective student learning.

The trajectory, query, and input-sensitivity diagnostics provide a useful complement to aggregate QA scores and connect the observations to template-collapse concerns in RAGEN-2 [2]. Testing clipped updates, joint RL training, and stabilized teachers also gives the study more diagnostic value than a single failed training run.

Critical weaknesses

Effective-sample availability is mechanically linked to successful-rollout availability, so its correlation with pass rate is not independent evidence of supervision inconsistency. Likewise, higher teacher accuracy with a correct solution in context does not establish reliable guidance on student prefixes. Stabilization helps, but sampled-token optimization problems identified by Fu et al. [4] and conditioning effects discussed by Kim et al. [3] remain plausible alternative explanations.

Original MOPD [5] already uses peer-conditioned teacher supervision, making the distinction between this paper’s MOPD and FA-MOPD unclear. The stated training budgets differ across teacher settings, and Figure 5 shows late collapse even for ordinary MOPD. Without clearer implementation details, checkpoint selection, and reliability estimates, the evidence supports a configuration-specific failure more strongly than a general explanation.

[1] Reinforcement Learning via Self-Distillation.

[2] RAGEN-2: Reasoning Collapse in Agentic RL.

[3] Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

[4] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes.

[5] Multi-Rollout On-Policy Distillation via Peer Successes and Failures.

Questions for authors

NA

2 comments

Response to Reviewer yR4d

We thank the reviewer for the careful evaluation and constructive feedback. We appreciate the recognition of the value of studying FA-SD failures in retrieval-interleaved search agents, particularly our behavioral diagnostics and teacher-stabilization experiments. We would like to clarify the following points.

1. Effective-sample availability and supervision inconsistency

We agree that effective-sample availability is mechanically related to successful-rollout availability. Our paper does not claim that its correlation with pass rate constitutes independent causal evidence of supervision inconsistency. Rather, the effective sample ratio measures the availability of usable distillation signals, which becomes increasingly sparse as training performance deteriorates.

Importantly, our analysis goes beyond this correlation. Figure 6 provides independent behavioral diagnostics of decoding collapse, including increasing cross-question trajectory similarity, answer-mode concentration, reduced query diversity, and declining input sensitivity. These measurements directly characterize the emergence of question-agnostic behavior. Thus, the effective sample ratio measures supervision availability, while the behavioral diagnostics characterize decoding collapse.

2. Self-teacher accuracy versus supervision quality

We agree that higher self-teacher inference-time accuracy does not necessarily imply reliable token-level guidance on student-generated prefixes. This is not a claim made in our paper.

The purpose of Figure 2 is to examine whether FA-SD fails simply because the self-teacher lacks sufficient capability. Our results show that feedback-conditioned self-teachers can achieve stronger inference-time performance, yet this advantage is not reliably transferred to the unconditioned student.

This discrepancy motivates our investigation rather than contradicting it. As discussed in Section 4.8, privileged feedback may allow the self-teacher to behave as though useful evidence has already been identified, while the student must independently discover that evidence through search. Our findings highlight the difficulty of transferring feedback-conditioned capabilities, without assuming that the resulting token-level supervision is necessarily optimal.

3. Alternative explanations for training instability

We appreciate the reviewer highlighting the sampled-token optimization issues identified by Fu et al. [4] and the teacher-conditioning effects discussed by Kim et al. [3], both of which are acknowledged in our Related Work.

We do not claim that model inconsistency and prompt inconsistency are the exclusive causes of FA-SD failure. Instead, our experiments investigate two plausible contributing factors.

First, Figure 4 demonstrates that both fixed-reference and EMA teachers improve training stability by reducing teacher parameter drift. In particular, EMA-regularized FA-SD achieves an average evaluation score of 0.206, compared with 0.114 for the initial model, although it remains below stronger GRPO and external-teacher OPD baselines.

Second, our external-teacher experiments investigate whether successful-rollout feedback provides additional benefits when teacher parameters are fixed. The observed degradation under feedback augmentation is consistent with potentially detrimental prompt conditioning, although it does not establish a unique causal mechanism.

These findings support the relevance of teacher stability and feedback conditioning without ruling out other optimization-related explanations. Our contribution is to characterize decoding collapse and investigate plausible mechanisms, rather than provide an exhaustive causal decomposition.

4. Clarification of MOPD and FA-MOPD

We thank the reviewer for identifying an ambiguity in our terminology.

Our experimental MOPD baseline refers to naive multi-rollout on-policy distillation with an external teacher, not the peer-conditioned MOPD algorithm proposed by Yu et al. [5].

Specifically, our experimental baseline samples multiple student trajectories per question (group size = 5) and uses Qwen2.5-7B-Instruct to provide token-level supervision on each student-generated trajectory, without incorporating other peer trajectories into the teacher's prompt.

In contrast, Yu et al. [5] explicitly incorporate successful and/or failed peer trajectories into teacher conditioning.

Our FA-MOPD variant extends our naive multi-rollout OPD baseline by explicitly augmenting the external teacher's prompt with a complete successful rollout, following the same feedback template used in FA-SD (Appendix A). The student remains unconditioned.

Therefore, our comparison isolates the effect of adding successful-rollout feedback to naive external-teacher OPD. It is neither a reproduction nor an ablation of Yu et al.'s peer-conditioned MOPD algorithm.

We acknowledge that the shared acronym may cause confusion and will revise the terminology to clearly distinguish our naive multi-rollout OPD baseline from the original MOPD method [5]. Our conclusions concern feedback augmentation in the evaluated external-teacher setting, not the general effectiveness of peer-conditioned MOPD.

5. Training budgets

We would also like to clarify the concern regarding different training budgets. Although Appendix B reports different maximum-step configurations (200 for self-teacher experiments and 1,000 for external-teacher experiments), all experiments used in the reported comparisons are evaluated over the same 200 training steps.

6. Mechanistic Evidence Beyond Configuration-Specific Failures

We thank the reviewer for raising concerns about implementation details, checkpoint selection, and reliability estimates. While we agree that additional reporting would strengthen reproducibility, we respectfully disagree that our findings merely reflect a configuration-specific failure.

First, our conclusions are supported by multiple experimental configurations, including unclipped FA-SD, PPO-style clipped FA-SD, joint FA-SD+GRPO, and fixed-reference/EMA teacher variants, with additional decoding-collapse diagnostics across different model initializations and scales.

Second, our mechanistic explanation is supported by direct behavioral evidence, including answer-mode locking, cross-question trajectory homogenization, reduced query diversity, and diminished input sensitivity (Section 4.5, Figure 6), rather than performance degradation alone.

Third, controlled interventions provide further evidence for the proposed supervision-inconsistency mechanism: fixed-reference and EMA teachers stabilize training, while feedback augmentation degrades external-teacher MOPD performance relative to standard MOPD.

We acknowledge that additional reliability estimates and clearer checkpoint-selection criteria would strengthen our empirical characterization. Nevertheless, the consistent evidence across configurations, behavioral diagnostics, and targeted interventions supports a mechanistic explanation beyond a single unsuccessful training recipe.

Importantly, our contribution is not to claim that FA-SD universally fails, but to identify and characterize an important failure mode in feedback-augmented self-distillation for retrieval-interleaved agents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.