- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies activation-space on-policy self-distillation (A-OPSD) and proposes a Fisher/Gauss–Newton objective obtained by pulling back the OPSD KL through the frozen LM head. It compares this objective with OPSD and with direct hidden-state MSE matching, and introduces a per-token loss cap to address instability under longer training.
Critical strengths
The Fisher formulation is clearly motivated, and the paper shows that it is consistently more effective than naive MSE matching. The experimental effort is substantial: four model configurations across two families, three seeds for the main results, and analyses of top-k approximation, supervision memory, and training dynamics. The paper also reports unfavorable results, including longer-horizon degradation and the lack of improvement in thinking mode, which I appreciate.
Critical weaknesses
My central concern is that the evidence does not establish a reliable advantage over OPSD, which is the paper’s headline claim.
-
Substantial empirical heterogeneity without a clear explanation. Despite a meaningful experimental effort involving multiple models and training seeds, the improvements of Fisher A-OPSD over OPSD are highly inconsistent, even within the same Qwen3 family. Moreover, simple MSE matching outperforms Fisher on Qwen3-8B, further questioning the added value of the proposed geometry. The paper emphasizes the average improvement but does not explain why the method succeeds in some settings and fails to improve in others, or identify conditions under which it should be preferred. Consequently, the experiments, while valuable, have not yet yielded a clear, generalizable finding about the proposed method. I believe that a convincing explanation of this heterogeneity, either through additional experiments or a deeper analysis of the existing results, would substantially strengthen the contribution.
-
Zhao et al. (2026) report consistent OPSD gains under thinking-mode evaluation, whereas this paper's main results use non-thinking mode. Its appendix finds no thinking-mode improvement for Fisher A-OPSD, but provides no matched OPSD comparison, leaving the discrepancy with the original findings unresolved. Thus, the reported gains do not establish whether Fisher improves over OPSD in its original evaluation regime. Moreover, uncapped Fisher substantially degrades under longer training, while the proposed loss cap is validated over longer horizons on only one model and one seed.
Questions for authors
-
What explains the substantial heterogeneity in Fisher A-OPSD's improvements, even within the same Qwen3 family? Can the authors identify any measurable factors that explain or predict when the proposed method is beneficial, either through additional experiments or deeper analysis of the existing results?
-
reports no improvement from Fisher A-OPSD in that setting. Can the authors reconcile these findings and provide a direct OPSD comparison under the same thinking-mode evaluation protocol?
-
Given the substantial performance degradation of uncapped Fisher under longer training, can the authors provide stronger evidence that the proposed loss cap consistently resolves this instability across multiple seeds and model configurations, rather than only one model and one seed?
-
On Qwen3-8B, OPSD underperforms its 4B counterpart (50.0 vs. 58.1) with much larger variance (±7.6 on MATH-500), while even naive MSE matching reaches 69.9. Was OPSD tuned separately for each model? Was the per-token KL clipping from the original OPSD work enabled for this baseline?
4 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the matched thinking-mode evaluation, the additional seeds and model settings used to evaluate the loss cap, and the comparisons with OPRD and top-k OPSD.
These additions substantially strengthen the empirical evaluation and provide useful insights into activation-space self-distillation. In particular, the results suggest that Fisher A-OPSD is more robust to top-k approximation than token-space OPSD, while the extended training experiments provide stronger evidence for the stabilizing effect of the loss cap. I also appreciate the authors' clarification of the method's scope, especially the limitations under thinking-mode evaluation.
My main concern regarding the heterogeneous improvements over OPSD remains. Although Fisher A-OPSD achieves substantial improvements on some models, its relative advantage varies considerably across model sizes. Attributing this variation primarily to OPSD's behavior does not fully explain the observed differences or clarify when Fisher A-OPSD should be preferred.
Nevertheless, I find that the additional experiments and analyses provide meaningful empirical contributions beyond the performance comparisons alone. I appreciate the stronger evidence and the more balanced discussion of the method's strengths and limitations.
We thank the reviewer for the follow-up, and for recognizing that the additional experiments substantially strengthen the empirical evaluation. We would like to address the remaining concern about heterogeneous improvements directly, because we believe the existing results answer it clearly.
1. The heterogeneity is in the baselines, not in Fisher A-OPSD
The varying margin between Fisher A-OPSD and OPSD does not come from Fisher behaving inconsistently. It comes from OPSD and MSE swinging widely across model sizes, while Fisher stays at or near the top on every model. The table below reports each method's average score (Table 1) and, in parentheses, its gap to the best method on that model:
| Method | Qwen3-1.7B | Qwen3-4B | Qwen3-8B | OLMo-3-7B | Worst-case gap | Mean gap |
|---|---|---|---|---|---|---|
| Fisher A-OPSD | 50.2 (best) | 57.3 (−0.8) | 68.9 (−1.0) | 57.9 (best) | 1.0 | 0.5 |
| OPSD | 38.6 (−11.6) | 58.1 (best) | 50.0 (−19.9) | 57.4 (−0.5) | 19.9 | 8.0 |
| A-OPSD MSE | 40.1 (−10.1) | 42.4 (−15.7) | 69.9 (best) | 56.1 (−1.8) | 15.7 | 6.9 |
Fisher A-OPSD is the best method on two models. On the other two it is within 1.0 point of the best, which is within seed noise (e.g., ±6.5 on Qwen3-4B AIME24). It is the only method that is never more than 1 point from the best. By contrast:
- OPSD falls 11.6 and 19.9 points below the best method on Qwen3-1.7B and Qwen3-8B.
- MSE falls 10.1 and 15.7 points below on Qwen3-1.7B and Qwen3-4B.
Each baseline wins only on the model where it happens to work well, while Fisher remains consistently close to whichever method is best.
Fisher also improves over its base model on every model (+20.2, +17.5, +28.5, +2.3). The smaller gain on OLMo-3-7B reflects limited headroom rather than a failure: its base model already scores 91.3 on MATH-500, and Fisher still achieves the largest gain of any method on that model.
The one model where OPSD leads at step 100 is Qwen3-4B, and that lead does not survive longer training. In our extended Qwen3-4B runs, OPSD peaks at step 100 (55.4 AIME24) and declines to 46.8 by step 500. Capped Fisher rises to 58.1, above OPSD's best checkpoint (Table 14).
2. When should Fisher A-OPSD be preferred?
We argue that Fisher A-OPSD should be the default objective for on-policy self-distillation, because it is the strongest option on every axis we measured:
- Most robust across models. It has the smallest worst-case gap to the best method (1.0 vs. 19.9 for OPSD and 15.7 for MSE) and the highest mean score (58.6 vs. 51.0 for OPSD and 52.1 for MSE).
- Most stable under longer training. Capped Fisher keeps improving through 500 steps across three seeds on Qwen3-1.7B (34.1 → 44.4 AIME24) and on Qwen3-4B (43.9 → 58.1). Over the same horizon, uncapped Fisher and MSE collapse, and OPSD itself declines on Qwen3-4B (55.4 → 46.8).
- Most robust to approximation. Fisher varies by at most 1.4 AIME24 points for k between 16 and 64, while restricting OPSD to its top-64 tokens costs 1.2–13.6 average points across the four models.
- Most efficient. Supervision memory is as follows:
| Model | OPSD, full KL (MiB) | Fisher, top-64 (MiB) | Reduction |
|---|---|---|---|
| Qwen3-1.7B | 10,692 | 2,213 | 4.8× |
| Qwen3-4B | 10,692 | 2,765 | 3.9× |
| Qwen3-8B | 10,692 | 4,421 | 2.4× |
Fisher also uses less memory at the full training-step level (47.4 vs. 55.8 GiB on Qwen3-1.7B).
- A better optimizer of OPSD's own objective. On held-out problems, training with the Fisher objective reduces the token-space KL by 51.1%, compared with 40.8% for OPSD, which minimizes that KL directly (Appendix B).
Finally, on the question of predictive factors: Table 1 shows that whether OPSD works well on a given model is hard to predict, even within a single model family. Because Fisher's worst case across all settings is within 1 point of the best method, choosing it does not require predicting in advance whether OPSD will succeed. This is precisely what makes it a reliable default.
We thank the reviewer again for the constructive engagement. The reviewer noted that the additional experiments substantially strengthen the empirical evaluation. We hope the analysis above also resolves the remaining concern: Fisher A-OPSD is consistent across models, and the heterogeneity comes from the baselines. If so, we kindly ask the reviewer to reconsider their score.
Thank you for the additional experiments, which provide valuable insights and address my main concerns. I recommend the authors to add these findings and results to the manuscript and I think it will be very helpful. With these new results, I will raise my score from 2 to 3.
Response to Reviewer Rf3z
We thank the reviewer for the thorough review and feedback, and for appreciating our clear formulation and our extensive experiments and ablations.
We would like to take this opportunity to address your concerns and answer your questions.
W1 / Q1. What explains the heterogeneous gains?
Much of the variation in the comparison with Fisher comes from how well OPSD works at each model size. To separate this from the effect of activation-space supervision, we compare every method against its own untrained base model:
Change in average MATH-500 / AIME24 / HMMT25 score relative to base (Table 1; points).
Fisher improves over base in every setting, although the size of the gain varies. OPSD is especially effective at Qwen3-4B (+18.4) but much weaker at 1.7B (+8.6) and 8B (+9.7). At 8B it also has high seed variance (±7.6 on MATH-500). In our results, we ran each method on 3 seeds for 100 steps each and evaluated each model at the final checkpoint. In the original OPSD paper, they evaluated a single seed across intermediate checkpoints and picked the best checkpoint for each method and model. Since we ran more training runs and more models, we could not afford to evaluate every checkpoint. This explains the large variance in some of the results.
We also agree that MSE is better on Qwen3-8B, but only by 1.0 point on average (69.9 vs. 68.9), which is within noise and is also affected by the limited hyperparameter tuning we did.
W2 / Q2. Matched thinking-mode results
We ran a matched thinking-mode evaluation of the Qwen3 base, OPSD, and Fisher A-OPSD checkpoints. The results do not show a consistent improvement in thinking mode:
Thinking-mode evaluation (MATH-500 / AIME24 / HMMT25; %).
OPSD is stronger at 1.7B; the methods are close to the base at 4B; and both are below base at 8B. We therefore limit our performance claim to non-thinking evaluation rather than suggesting a thinking-mode improvement. This is consistent with recent evidence that privileged-context self-distillation can disrupt long reasoning traces (Kaur et al., 2026; Kim et al., 2026), so it is not an artifact of activation space distillation. These works do not establish the cause of our specific results, so fixing the limitation of think-mode training and evaluation is left for future work.
W2 / Q3. Does the cap generalize across seeds and model sizes?
We extended the Qwen3-1.7B capped Fisher run to two additional seeds (c = 0.1, 500 steps). All three capped seeds improve on AIME24 after step 100; the uncapped run collapses:
Qwen3-1.7B, AIME24 (%).
Across the three capped seeds, AIME24 increases from 34.1 at step 100 to 44.4 at step 500 on average. We also extended training on Qwen3-4B:
Qwen3-4B extended-run evaluation score on AIME24.
On this second model, the capped score rises from 43.9 at step 100 to 58.1 at step 500, while uncapped Fisher falls from 49.6 to 40.0. These runs support the cap's stability benefit across more than one seed and model.
Q4. Was OPSD tuned per model, and was KL clipping enabled?
Clipping was enabled in every OPSD run. We used the original OPSD repository's recommended model-specific configurations, including its point-wise KL clipping, without additional tuning by us.
Kaur et al. (2026), Rethinking On-Policy Self-Distillation for Thinking Models. https://arxiv.org/abs/2607.05184
Kim et al. (2026), Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? https://arxiv.org/abs/2603.24472
We thank the reviewer once again for their time and effort. We have added all of these results, analyses, and much more to the revised version of the paper. We hope we have addressed all of their questions and concerns, and if so, we kindly ask you to increase your score.