- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses the optimization instability and collapse of feedback-based on-policy self-distillation for LLMs. The authors propose FIRE, a dual-branch framework that routes responses by correctness and bounds both branches with a token-level radius derived from the student’s softmax Fisher trace and learning rate. Two key designs are proposed: 1) Fisher-informed re-weighted on-policy SFT for correct responses; 2) leave-one-feedback-field-out attribution with radial projection to recalibrate the teacher target for incorrect responses. Experiments on Qwen2.5 across seven benchmarks show higher accuracy, more stable gradients, and less response-length collapse than eight baselines.
Critical strengths
-
The idea of separating which direction feedback should move the model from how far it should move is clear and natural for stabilizing self-distillation.
-
The experiments are comprehensive, covering three model scales, seven benchmarks, and eight recent baselines, along with analyses of gradient norms, response length, efficiency, ablations, and learning-rate sensitivity.
Critical weaknesses
-
Most design choices lack clear motivation.
- Key components, such as the Fisher-trace radius, the LR divisor, and the excess-energy gate, are introduced as heuristics without theoretical or empirical justification. The paper even states that a large does not imply a harmful field, yet Eq. 35 attenuates fields in proportion to their influence, likely suppressing the field that contains the correct answer ().
- The attribution mechanism is motivated by the possibility that some feedback components are unreliable and should be down-weighted. However, in the experiments all four feedback fields are automatically constructed from a ground-truth verifier, so the feedback is always reliable, and the scenario that motivates the design never actually arises.
-
The experiments are problematic.
- Ablations are insufficient. Table 3 covers only the incorrect branch, and most gaps are within std; No Attribution (72.22 on 3B) already outperforms all baselines in Table 1. Ablations of the correct-branch , of the Fisher radius vs. a constant radius, and of are missing.
- The only hyperparameter, , is inactive in the main results and appears sensitive. It equals the peak LR in all 1.5B/3B settings (Table 5), so throughout Table 1.
- The evaluation has limited statistical reliability. Only 192 test samples are used, and many gains are within std (e.g., WinoGrande vs. ).
-
The theoretical analysis is overclaimed.
- "Fisher-informed" uses only the scalar with a Euclidean norm, not the Fisher metric.
- The derivations in App. F are elementary softmax/log-linear identities; e.g., Eq. 21 is simply the definition of the categorical Fisher matrix. The unified bound (Eq. 43) only restates that both branches are explicitly clipped to , yet it is listed as a contribution.
-
The presentation needs improvement.
- Sec. 3.2 introduces many coupled quantities (, , , , , , , ) through equations without first explaining why each is needed. Key rationales, such as why the Fisher trace is used, are deferred to App. F.3–F.4.
- FIRE's results differ across Tables 1, 3, and 4 because of different LR/ settings, and the paper does not clearly explain why.
Questions for authors
See above
4 comments
Authors Response to Reviewer aBwc (#2)
- (W3)
- We would like to respectfully clarify the reason behind calling our method “Fisher-informed”. As we noted a few times throughout the manuscript (abstract, Sec. 1.1, Sec 3.2, and in the Appendix), we claim we utilize the square root of the softmax Fisher trace, which is informed from the Fisher matrix.
- Many of the derivations in Appendix F are presented to show how each equation in the main text is derived while keeping the main text of the manuscript readable. Importantly, we do not claim standard Fisher and softmax identities as new results (e.g., "The softmax Fisher matrix is....." in Appendix F.3). For Appendix F.7, we have updated the wording to make it more clear that Eq. 43 to 44 demonstrates that FIRE's losses and teacher targets enforce a shared gradient budget. Lastly, the original wording for the last contribution bullet point was intended to highlight an empirical observation of FIRE, namely its improved training stability. However, since the bound that accompanied it was presented only in the Appendix, we have removed it from the contribution list. We have instead revised it to emphasize our empirical validation and stability improvements. Both changes can be found in the updated manuscript highlighted in blue.
- (W4)
-
For our presentation, when each mathematical definition is presented for the first time, they are defined around their equations, and often within the same sentence. For example, immediately after Eq. 5, we define and explain that the resulting counterfactual reveals how removing field changes the teacher’s supervision. Similarly, the paragraph following Eq. 7 explains how student uncertainty and learning rate determine the radius. The text preceding Eq. 10 motivates activating attribution only when the full-teacher gradient exceeds this budget. These explanations are already included in the main text, and Appendix F provides the detailed derivations to keep the main presentation readable.
-
We would like to direct the reviewer to Appendix A.4, where for the results in Table 3, we explicitly state: "For Qwen2.5-1.5B-Instruct, we use a uniformly higher learning-rate setting across all variants to better emphasize component behavior advantages of FIRE in more unstable optimization landscapes." We also have similar explanations in Appendix A.5 of the original submitted manuscript with regards to Table 4 (now Table 2 in Sec. 4.1 of the updated manuscript), but have added the following sentence to make it even more explicit: " is purposely set low so that contraction occurs in all settings." We have also moved the varying LR experiment from the Appendix to the main text to better emphasize its importance. It can be found right before the conclusion in the updated manuscript, highlighted in blue.
To isolate the effect of the LR divisor and give greater support of our usage of it, we present an ablation of LR divisor vs. no LR divisor on 2x the learning rate in the main text below in Table C. We demonstrate from this the importance of the divisor, as without it, the performance is lower.
-
Table C: Ablation on FIRE vs. No LR Divisor at 2x LR on Full Test Data
| Method / Ablation | GSM8K | ARC-Challenge | ASDiv | WinoGrande | Avg. |
|---|---|---|---|---|---|
| No LR Divisor | 52.81±0.95 | 81.05±0.12 | 76.71±0.87 | 63.35±0.10 | 68.46 |
| FIRE | 53.53±1.52 | 81.00±0.15 | 80.37±1.04 | 63.93±0.24 | 69.71 |
[1] Zhao et al., Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
[2] Lee et al., Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning
[3] Xing et al., Trust Region On-Policy Distillation
We appreciate your time and effort in reviewing our paper. We would be happy to address any remaining questions or concerns the reviewer might have.
Reviewer aBwc,
We thank you for taking the time to review our work. As the discussion deadline approaches, we kindly request that you consider our rebuttal in response to your concerns and hope that you can reconsider your score in light of our clarifications and additional experimental evidence.
Reviewer aBwc,
Once again, we thank you for taking the time to review our work. As the rebuttal will end soon, we wanted to follow up again and check whether we have addressed your concerns. If so, we hope you can reconsider your score in light of our clarifications and additional experimental evidence. We would be grateful for an opportunity to address any additional questions you have.
Authors Response to Reviewer aBwc
We thank the reviewer for the thoughtful review and suggestions. Below, we answer each concern.
(W1)
We would to clarify that we justify that the Fisher-trace radius has a specific mathematical interpretation, provided in Appendix F.3 of the manuscript. For the LR divisor, we provided empirical justification in Appendix A.5 of the original submitted manuscript (now moved to the end of Sec. 4.1), where FIRE is able to train on a range of differing learning rate schedules due to constraining conducted by a0. To address concerns about the radius and excess-energy gate, we provide an ablation in Table A, (W2), where we compare FIRE against a tuned constant radius value and no gating taking place, and find that FIRE performs better, justifying our methodology's selection empirically. With regards to calling influence "harmful", we have updated this in the manuscript to clarify that the intuition is to scale back large updates, which might make updates more unstable. We have removed the "harmful" terminology.
We thank the reviewer for the comment. Ground-truth feedback does not guarantee the quality of the teacher’s token-level predictions along an incorrect student trajectory. Sec. 1 of the manuscript (Introduction, Challenges paragraph) identifies that the feedback-conditioned teacher can still produce incorrect predictions and poorly directed updates. Therefore, despite the fact that the feedback fields are automatically constructed from a ground-truth verifier, it does not eliminate the stabilization problem motivating FIRE.
These observations also appear in existing work. For example, OPSD [1] gives the solution to the teacher, and such methods are demonstrated in [2] to still be conducive to performance collapse. However, to make this more clear, we have added short clarification of this characteristic in the "Challenges" section of the introduction, which is highlighted in blue.
(W2)
In regards to the concern that the ablations are insufficient, we have conducted ablations on the requested settings. More specifically, below we consider only standard SFT on correct traces (SFT Without Re-Weighting), the FIRE-based Fisher radius vs. a tuned constant radius value (Tuned Constant Radius), and removal of the excess energy gate (No Gate) in Table A. From the results, we can see that FIRE performs better than any one ablation.
In addition, while "No Attribution" outperforms the baselines in Table 1 of the manuscript for 3B, FIRE's methodology further increases the performance, bringing it up to 73.03. Moreover, "No Attribution" does not outperform all the baselines on 1.5B, in contrast with FIRE, which means FIRE's performance is more consistent.
In Sec. 4 of the manuscript, we set the nominal rate and peak learning rate the same to demonstrate that even without meticulous tuning of a0, FIRE still outperforms the baselines. In addition, we would like to respectfully highlight that our method is rather robust to the choice of a0. From Table 4 in Appendix A.5.1 of the original submitted manuscript (now Table 2 at the end of Sec. 4.1 in the updated manuscript), we can see that while setting the learning rate much higher than values commonly utilized in distillation settings [1, 3] can lower the performance, it successfully prevents collapse in performance, in contrast to some baselines, whose decrease in performance occurs even under the default settings, as seen with Appendices A.1 and A.2.
To address the reviewer's concern on the evaluation limitations, we present evaluation results with respect to the full test sets in Table B. We picked Full Context and SCOPE due to their roles as standard feedback-based on-policy self-distillation and the best performing on-average baseline. In addition, we tuned the baseline learning rates. All results presented in our response, including Tables A/C, are evaluated on the full test sets.
Table A: Additional Ablations
Table B: FIRE vs. LR Tuned Full-Context and SCOPE on full test data