acceptodds
Under review as a conference paper at ICLR 2027

A-OPSD: Activation-Space On-Policy Self-Distillation for Large Language Models

11 views

Abstract

On-policy self-distillation (OPSD) improves reasoning models by using a privileged teacher —the same model conditioned on both a problem and its reference solution— to supervise a student that observes only the problem. Existing OPSD methods transfer this privileged signal through dense token-level distribution matching over the student's own rollouts. While effective, this supervision acts only after the internal reasoning of the student and teacher has been projected through the language modeling head, and its cost scales with the full vocabulary. We ask whether the same signal can be transferred directly in the model's hidden-state space, and study a family of activation-space objectives for OPSD—from plain hidden-state matching to metrics derived from the local geometry of the OPSD loss. We find that even naively matching the teacher and student's hidden-states can match token-level OPSD, while a Gauss–Newton (Fisher) pullback of the OPSD KL through the LM head can outperform it. Across Qwen3 and OLMo-3-7B models with sizes ranging from 1.7B to 8B, A-OPSD (Activation-Space On-Policy Self-Distillation) matches or exceeds token-level OPSD on mathematical reasoning benchmarks (MATH-500, AIME24, HMMT25) while requiring less supervision memory.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
On-policy distillation, self-distillation, activation-space, reasoning, language models.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Rf3zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies activation-space on-policy self-distillation (A-OPSD) and proposes a Fisher/Gauss–Newton objective obtained by pulling back the OPSD KL through the frozen LM head. It compares this objective with OPSD and with direct hidden-state MSE matching, and introduces a per-token loss cap to address instability under longer training.

Critical strengths

The Fisher formulation is clearly motivated, and the paper shows that it is consistently more effective than naive MSE matching. The experimental effort is substantial: four model configurations across two families, three seeds for the main results, and analyses of top-k approximation, supervision memory, and training dynamics. The paper also reports unfavorable results, including longer-horizon degradation and the lack of improvement in thinking mode, which I appreciate.

Critical weaknesses

My central concern is that the evidence does not establish a reliable advantage over OPSD, which is the paper’s headline claim.

  1. Substantial empirical heterogeneity without a clear explanation. Despite a meaningful experimental effort involving multiple models and training seeds, the improvements of Fisher A-OPSD over OPSD are highly inconsistent, even within the same Qwen3 family. Moreover, simple MSE matching outperforms Fisher on Qwen3-8B, further questioning the added value of the proposed geometry. The paper emphasizes the average improvement but does not explain why the method succeeds in some settings and fails to improve in others, or identify conditions under which it should be preferred. Consequently, the experiments, while valuable, have not yet yielded a clear, generalizable finding about the proposed method. I believe that a convincing explanation of this heterogeneity, either through additional experiments or a deeper analysis of the existing results, would substantially strengthen the contribution.

  2. Zhao et al. (2026) report consistent OPSD gains under thinking-mode evaluation, whereas this paper's main results use non-thinking mode. Its appendix finds no thinking-mode improvement for Fisher A-OPSD, but provides no matched OPSD comparison, leaving the discrepancy with the original findings unresolved. Thus, the reported gains do not establish whether Fisher improves over OPSD in its original evaluation regime. Moreover, uncapped Fisher substantially degrades under longer training, while the proposed loss cap is validated over longer horizons on only one model and one seed.

Questions for authors
  1. What explains the substantial heterogeneity in Fisher A-OPSD's improvements, even within the same Qwen3 family? Can the authors identify any measurable factors that explain or predict when the proposed method is beneficial, either through additional experiments or deeper analysis of the existing results?

  2. reports no improvement from Fisher A-OPSD in that setting. Can the authors reconcile these findings and provide a direct OPSD comparison under the same thinking-mode evaluation protocol?

  3. Given the substantial performance degradation of uncapped Fisher under longer training, can the authors provide stronger evidence that the proposed loss cap consistently resolves this instability across multiple seeds and model configurations, rather than only one model and one seed?

  4. On Qwen3-8B, OPSD underperforms its 4B counterpart (50.0 vs. 58.1) with much larger variance (±7.6 on MATH-500), while even naive MSE matching reaches 69.9. Was OPSD tuned separately for each model? Was the per-token KL clipping from the original OPSD work enabled for this baseline?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Rf3z

We thank the reviewer for the thorough review and feedback, and for appreciating our clear formulation and our extensive experiments and ablations.

We would like to take this opportunity to address your concerns and answer your questions.


W1 / Q1. What explains the heterogeneous gains?

Much of the variation in the comparison with Fisher comes from how well OPSD works at each model size. To separate this from the effect of activation-space supervision, we compare every method against its own untrained base model:

Change in average MATH-500 / AIME24 / HMMT25 score relative to base (Table 1; points).

MethodQwen3-1.7BQwen3-4BQwen3-8BOLMo-3-7B
Fisher A-OPSD+20.2+17.5+28.5+2.3
OPSD+8.6+18.4+9.7+1.8
A-OPSD MSE+10.1+2.7+29.5+0.4

Fisher improves over base in every setting, although the size of the gain varies. OPSD is especially effective at Qwen3-4B (+18.4) but much weaker at 1.7B (+8.6) and 8B (+9.7). At 8B it also has high seed variance (±7.6 on MATH-500). In our results, we ran each method on 3 seeds for 100 steps each and evaluated each model at the final checkpoint. In the original OPSD paper, they evaluated a single seed across intermediate checkpoints and picked the best checkpoint for each method and model. Since we ran more training runs and more models, we could not afford to evaluate every checkpoint. This explains the large variance in some of the results.

We also agree that MSE is better on Qwen3-8B, but only by 1.0 point on average (69.9 vs. 68.9), which is within noise and is also affected by the limited hyperparameter tuning we did.

W2 / Q2. Matched thinking-mode results

We ran a matched thinking-mode evaluation of the Qwen3 base, OPSD, and Fisher A-OPSD checkpoints. The results do not show a consistent improvement in thinking mode:

Thinking-mode evaluation (MATH-500 / AIME24 / HMMT25; %).

ModelMethodMATH-500AIME24HMMT25Avg
Qwen3-1.7BBase90.545.621.752.6
OPSD92.052.226.757.0
Fisher A-OPSD91.350.024.255.2
Qwen3-4BBase95.874.243.371.1
OPSD95.773.943.671.1
Fisher A-OPSD95.074.445.371.6
Qwen3-8BBase96.076.145.672.6
OPSD96.275.643.971.9
Fisher A-OPSD95.069.742.869.2

OPSD is stronger at 1.7B; the methods are close to the base at 4B; and both are below base at 8B. We therefore limit our performance claim to non-thinking evaluation rather than suggesting a thinking-mode improvement. This is consistent with recent evidence that privileged-context self-distillation can disrupt long reasoning traces (Kaur et al., 2026; Kim et al., 2026), so it is not an artifact of activation space distillation. These works do not establish the cause of our specific results, so fixing the limitation of think-mode training and evaluation is left for future work.

W2 / Q3. Does the cap generalize across seeds and model sizes?

We extended the Qwen3-1.7B capped Fisher run to two additional seeds (c = 0.1, 500 steps). All three capped seeds improve on AIME24 after step 100; the uncapped run collapses:

Qwen3-1.7B, AIME24 (%).

RunStep 100Step 500Change
Capped, original seed35.345.3+10.0
Capped, new seed 132.843.1+10.3
Capped, new seed 234.244.9+10.7
Uncapped42.526.5−16.0

Across the three capped seeds, AIME24 increases from 34.1 at step 100 to 44.4 at step 500 on average. We also extended training on Qwen3-4B:

Qwen3-4B extended-run evaluation score on AIME24.

MethodStep 50Step 100Step 250Step 500
OPSD42.555.451.146.8
Fisher, uncapped43.549.640.640.0
Fisher + cap (c = 0.1)43.443.954.458.1

On this second model, the capped score rises from 43.9 at step 100 to 58.1 at step 500, while uncapped Fisher falls from 49.6 to 40.0. These runs support the cap's stability benefit across more than one seed and model.

Q4. Was OPSD tuned per model, and was KL clipping enabled?

Clipping was enabled in every OPSD run. We used the original OPSD repository's recommended model-specific configurations, including its point-wise KL clipping, without additional tuning by us.

Kaur et al. (2026), Rethinking On-Policy Self-Distillation for Thinking Models. https://arxiv.org/abs/2607.05184

Kim et al. (2026), Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? https://arxiv.org/abs/2603.24472


We thank the reviewer once again for their time and effort. We have added all of these results, analyses, and much more to the revised version of the paper. We hope we have addressed all of their questions and concerns, and if so, we kindly ask you to increase your score.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the matched thinking-mode evaluation, the additional seeds and model settings used to evaluate the loss cap, and the comparisons with OPRD and top-k OPSD.

These additions substantially strengthen the empirical evaluation and provide useful insights into activation-space self-distillation. In particular, the results suggest that Fisher A-OPSD is more robust to top-k approximation than token-space OPSD, while the extended training experiments provide stronger evidence for the stabilizing effect of the loss cap. I also appreciate the authors' clarification of the method's scope, especially the limitations under thinking-mode evaluation.

My main concern regarding the heterogeneous improvements over OPSD remains. Although Fisher A-OPSD achieves substantial improvements on some models, its relative advantage varies considerably across model sizes. Attributing this variation primarily to OPSD's behavior does not fully explain the observed differences or clarify when Fisher A-OPSD should be preferred.

Nevertheless, I find that the additional experiments and analyses provide meaningful empirical contributions beyond the performance comparisons alone. I appreciate the stronger evidence and the more balanced discussion of the method's strengths and limitations.

We thank the reviewer for the follow-up, and for recognizing that the additional experiments substantially strengthen the empirical evaluation. We would like to address the remaining concern about heterogeneous improvements directly, because we believe the existing results answer it clearly.

1. The heterogeneity is in the baselines, not in Fisher A-OPSD

The varying margin between Fisher A-OPSD and OPSD does not come from Fisher behaving inconsistently. It comes from OPSD and MSE swinging widely across model sizes, while Fisher stays at or near the top on every model. The table below reports each method's average score (Table 1) and, in parentheses, its gap to the best method on that model:

MethodQwen3-1.7BQwen3-4BQwen3-8BOLMo-3-7BWorst-case gapMean gap
Fisher A-OPSD50.2 (best)57.3 (−0.8)68.9 (−1.0)57.9 (best)1.00.5
OPSD38.6 (−11.6)58.1 (best)50.0 (−19.9)57.4 (−0.5)19.98.0
A-OPSD MSE40.1 (−10.1)42.4 (−15.7)69.9 (best)56.1 (−1.8)15.76.9

Fisher A-OPSD is the best method on two models. On the other two it is within 1.0 point of the best, which is within seed noise (e.g., ±6.5 on Qwen3-4B AIME24). It is the only method that is never more than 1 point from the best. By contrast:

  • OPSD falls 11.6 and 19.9 points below the best method on Qwen3-1.7B and Qwen3-8B.
  • MSE falls 10.1 and 15.7 points below on Qwen3-1.7B and Qwen3-4B.

Each baseline wins only on the model where it happens to work well, while Fisher remains consistently close to whichever method is best.

Fisher also improves over its base model on every model (+20.2, +17.5, +28.5, +2.3). The smaller gain on OLMo-3-7B reflects limited headroom rather than a failure: its base model already scores 91.3 on MATH-500, and Fisher still achieves the largest gain of any method on that model.

The one model where OPSD leads at step 100 is Qwen3-4B, and that lead does not survive longer training. In our extended Qwen3-4B runs, OPSD peaks at step 100 (55.4 AIME24) and declines to 46.8 by step 500. Capped Fisher rises to 58.1, above OPSD's best checkpoint (Table 14).

2. When should Fisher A-OPSD be preferred?

We argue that Fisher A-OPSD should be the default objective for on-policy self-distillation, because it is the strongest option on every axis we measured:

  1. Most robust across models. It has the smallest worst-case gap to the best method (1.0 vs. 19.9 for OPSD and 15.7 for MSE) and the highest mean score (58.6 vs. 51.0 for OPSD and 52.1 for MSE).
  2. Most stable under longer training. Capped Fisher keeps improving through 500 steps across three seeds on Qwen3-1.7B (34.1 → 44.4 AIME24) and on Qwen3-4B (43.9 → 58.1). Over the same horizon, uncapped Fisher and MSE collapse, and OPSD itself declines on Qwen3-4B (55.4 → 46.8).
  3. Most robust to approximation. Fisher varies by at most 1.4 AIME24 points for k between 16 and 64, while restricting OPSD to its top-64 tokens costs 1.2–13.6 average points across the four models.
  4. Most efficient. Supervision memory is as follows:
ModelOPSD, full KL (MiB)Fisher, top-64 (MiB)Reduction
Qwen3-1.7B10,6922,2134.8×
Qwen3-4B10,6922,7653.9×
Qwen3-8B10,6924,4212.4×

Fisher also uses less memory at the full training-step level (47.4 vs. 55.8 GiB on Qwen3-1.7B).

  1. A better optimizer of OPSD's own objective. On held-out problems, training with the Fisher objective reduces the token-space KL by 51.1%, compared with 40.8% for OPSD, which minimizes that KL directly (Appendix B).

Finally, on the question of predictive factors: Table 1 shows that whether OPSD works well on a given model is hard to predict, even within a single model family. Because Fisher's worst case across all settings is within 1 point of the best method, choosing it does not require predicting in advance whether OPSD will succeed. This is precisely what makes it a reliable default.


We thank the reviewer again for the constructive engagement. The reviewer noted that the additional experiments substantially strengthen the empirical evaluation. We hope the analysis above also resolves the remaining concern: Fisher A-OPSD is consistent across models, and the heterogeneity comes from the baselines. If so, we kindly ask the reviewer to reconsider their score.

Thank you for the additional experiments, which provide valuable insights and address my main concerns. I recommend the authors to add these findings and results to the manuscript and I think it will be very helpful. With these new results, I will raise my score from 2 to 3.

Reviewer_YuxbOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Activation-Space On-Policy Self-Distillation (A-OPSD), which uses the final hidden representations as supervision when distilling a privileged teacher to a student. Following OPSD, the teacher is the frozen base model supplied with privileged information like the reference answer, while the student tries to close the gap between itself and the teacher. Rather than using the KL divergence between token distributions as the loss objective, this paper studies using hidden-state matching and Fisher-weighted matching hidden-state matching as the objectives. A top-k approximation and a per-token loss cap are also applied to reduce memory usage and stabilize training.

Experiments on four models with different sizes and from different families show that Fisher A-OPSD improves average accuracy, especially for Qwen3-1.7B and 8B models. The method also substantially reduces the memory required for training with the small-k approximation.

Critical strengths

The motivation is clear, and the implementation of the method is simple yet effective. Self-distillation provides a natural setting for matching the hidden states because the shapes of the hidden states match each other. The proposed objectives are therefore straightforward and require no additional mappings between diffusion models.

The authors did a comprehensive profiling of the performance and memory across different sizes of k, helping the authors to better understand the scaling trend of each method.

Critical weaknesses

It seems the memory advantage of the proposed method depends on the choice of k. At a higher k, the savings can diminish substantially, as the trend in Table 2 suggests. Since the memory usage of A-OPSD also seems to scale with model sizes, the scalability of A-OPSD is questionable.

The motivation for the loss cap feels disconnected from its design. The authors motivate the cap by arguing that privileged information creates an irreducible student–teacher representation gap. However, the proposed mechanism removes gradients from all tokens above a loss threshold, without distinguishing irreducible discrepancies from difficult but learnable ones. Also, Figure 2 seems to suggest that the degradation issue is specific to activation matching methods, but not OPSD. Therefore, the reason why this loss cap works remains vague to me.

Missing related work and baseline: OPRD [1] can be interpreted as activation-space MSE matching and should be included as a baseline. It should also be included in related work as prior work that explores representation/activation space distillation.

[1] Yang, Shenzhi, et al. "OPRD: On-Policy Representation Distillation." arXiv preprint arXiv:2606.06021 (2026).

Questions for authors
  1. From Table1, it seems A-OPSD’s advantage over vanilla OPSD is not consistent, even within the same model family, Qwen3. Why does A-OPSD fail at the middle size, and what is the failure pattern?

  2. For controlled comparison, Table 1 should also include an OPSD baseline with k=64, despite k=64 not necessarily outperforming the full-vocabulary OPSD.

  3. Does the OPSD baseline also use stabilization mechanisms like the loss cap used in A-OPSD, such as KL clipping?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Yuxb

We thank the reviewer for the constructive comments and for recognizing the simplicity and effectiveness of the method and the scope of the experiments.

We would like to take this chance to address your questions and raised concerns.


W1. Memory savings across model sizes

The memory advantage does depend on k and the hidden dimension. We agree that it should not be presented as independent of model size. However, at k = 64, Fisher requires substantially less supervision memory than full-vocabulary OPSD across all evaluated Qwen3 sizes:

Supervision-memory comparison at 4,096 decision tokens (MiB).

ModelOPSD, full KLFisher, top-64Reduction
Qwen3-1.7B10,6922,2134.8×
Qwen3-4B10,6922,7653.9×
Qwen3-8B10,6924,4212.4×

Even at 8B, Fisher uses less than half the supervision memory. Reducing k to 32 or 16 can lower it further without large performance changes over the tested range (Table 2). We also tested top-64 KL for OPSD; unlike Fisher, truncating OPSD causes substantial accuracy drops, as shown under Q2.

W2. What does the loss cap actually remove?

We agree that a loss threshold cannot distinguish irreducible student–teacher differences from difficult but learnable ones. Instead, it's meant to prevent a small number of high-residual positions from dominating optimization. We measured its effect on fixed Qwen3-1.7B rollouts to validate this interpretation:

Effect of the Fisher loss cap on fixed rollouts.

MeasureStep 0Step 100
Positions above the cap19.8%9.9%
Their share of squared hidden-state gradient98.6%87.2%
p90 / p99 per-position loss0.34 / 4.260.10 / 0.65

The fraction of excluded positions falls from 19.8% to 9.9% during training, while the tail of the loss distribution shrinks. This is consistent with some high-loss positions becoming trainable as their loss falls; it is not evidence that the cap identifies irreducible errors. Instead, the goal of the cap is to stop a small number of outliers from destabilizing longer training.

W3. Missing OPRD baseline

Thank you for pointing out OPRD (Yang et al., 2026). We had missed this work. We ran additional 100-step comparisons (two runs per method) on Qwen3-1.7B and OLMo-3-7B and added OPRD to the related work and baseline discussion.

New OPRD comparison (reported means; all values in %).

ModelMethodMATH-500AIME24HMMT25Avg
Qwen3-1.7BBase71.713.84.530.0
OPSD77.126.313.238.9
OPRD85.140.220.048.4
Fisher A-OPSD88.942.520.650.7
OLMo-3-7BBase91.350.525.155.6
OPSD92.454.026.657.7
OPRD91.153.126.757.0
Fisher A-OPSD92.654.927.358.3

OPRD improves substantially over OPSD at 1.7B and is competitive at 7B, but Fisher A-OPSD is stronger in both tested settings (+2.3 and +1.3 average points over OPRD). Due to the time and cost of these experiments, we will include the other two models in the final camera-ready version if accepted.

AuthorsOpenReview ↗

Response to Reviewer Yuxb - Part 2

Q1. Why is Fisher not better than OPSD on Qwen3-4B?

We would like to push back on calling this a failure. On Qwen3-4B, Fisher improves over the base model by 17.5 average points, while OPSD improves by 18.4; their final averages differ by only 0.8 points (57.3 vs. 58.1). The gap is also small relative to the variation reported across seeds in Table 1. OPSD is unusually strong for this model, rather than Fisher failing to learn.

Improvement in average score over each model's untrained baseline (Table 1; points).

MethodQwen3-1.7BQwen3-4BQwen3-8BOLMo-3-7B
Fisher A-OPSD+20.2+17.5+28.5+2.3
OPSD+8.6+18.4+9.7+1.8

So the gains over base remain substantial across Qwen3 sizes, even though the advantage over OPSD varies.

Q2. Matched top-64 OPSD results

We ran top-64 OPSD on all four models, using the same evaluation protocol.

Single-seed OPSD results. Each cell shows MATH-500 / AIME24 / HMMT25 → Avg (%).

ModelFull KLTop-64 KLΔ Avg
Qwen3-1.7B77.0 / 26.9 / 13.6 → 39.276.0 / 18.3 / 6.9 → 33.7-5.4
Qwen3-4B89.0 / 53.1 / 30.3 → 57.586.4 / 29.4 / 15.8 → 43.9-13.6
Qwen3-8B88.5 / 48.1 / 17.9 → 51.585.2 / 39.2 / 15.0 → 46.5-5.0
OLMo-3-7B92.2 / 53.7 / 25.8 → 57.291.6 / 52.4 / 24.2 → 56.1-1.2

Top-64 KL underperforms full KL on every model, losing 1.2–13.6 average points. By comparison, Fisher A-OPSD tolerates top-k approximation much better (Table 2).

Q3. Was stabilization enabled for the OPSD baseline?

Yes. Every OPSD run uses the original implementation's point-wise KL clipping with its recommended model-specific thresholds from the original OPSD repo. We did not turn clipping off for OPSD or tune it in favor of A-OPSD.


We thank the reviewer once again for their time and effort. We have added all of these results, analyses, and much more to the revised version of the paper. We hope we have addressed all of their questions and concerns, and if so, we kindly ask you to reconsider your score.

I thank the authors for the additional explanation and experimental details. This helped address my W1, Q1, Q2, and Q3. On W2, the loss cap should be interpreted as a technique to stabilize the inherent instability of self-distillation training, rather than as a way to address the gap caused by the teacher model's privileged information. For W3, thanks for including OPRD. However, based on the results of OPRD, it seems the improvement of Fisher A-OPSD over OPRD is marginal. Since A-OPSD MSE version is conceptually similar to OPRD (please correct me if I missed any differences) and it significantly underperforms A-OPSD Fisher version, it raises two questions: 1) whether the algorithmic workflow of A-OPSD MSE is sufficiently optimized, and 2) whether the geometric information, as well as the algorithm-level complexity introduced by the Fisher version, is really necessary since it only brings marginal improvement.

Reviewer_JcLYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores activation-space distillation for OPSD. It is motivated by the fact that, in self-distillation, the teacher and the student are the same model sharing the same weights. Under LoRA, the LM head is frozen and shared, so the teacher’s and the student’s hidden states lie in the same hidden space. This makes self-distillation more suitable for activation-space distillation than on-policy distillation with a separate teacher model. The paper explores an important design choice here by selecting a Fisher distance over an MSE distance. It then points out another important question for self-distillation stability: the teacher has access to more information than the student, so it is hard for the optimization to truly minimize the loss to zero. In light of this, it introduces a capping mechanism that ignores some outlier tokens. Its performance curves show that this capping leads to much better stability when extending training to more steps. In the results across four models, the method outperforms OPSD on most tasks.

Critical strengths
  1. The paper is well motivated. I think self-distillation is well motivated for activation-based distillation. OPSD in token space which might have issues due to a limitation of reasoning-style compression: if too much privileged information is given, it will heavily suppress the probability of some tokens, leading to instability. And this paper is a first-step toward using activation space distillation to mitigate this challegende. Further, this paper also mitigates it with activation-space distillation and the cap mechanism, which is shown to work more reliably and stably than without it in figure 2, and better scaling than OPSD.
  2. In terms of experimental results, it outperforms OPSD on three of the four models on average and is comparable on Qwen3-4B, across different tasks. In Figure 2, the evaluation curves show that A-OPSD with the cap keeps improving with more training steps, while the uncapped variants degrade.
  3. It carefully compares MSE and Fisher to find which one is better, and also identifies the stability issue in the naive form of the objective. Recognizing this, it introduces a capping mechanism, which mitigates the issue a lot, judging from Figure 2.
  4. the paper provides extensive ablations and analysis around the Fisher objective. The derivation is clean under the forward KL: since the LM head is frozen and shared between the teacher and the student, the Fisher objective is an exact second-order expansion of the OPSD KL
Critical weaknesses
  1. For the memory saving, the paper compares against OPSD with full-vocabulary distillation, but their method uses Fisher with top-k. I wonder if the saving largely comes from the top-k selection instead of the activation-space distillation. Although Table 3 includes OPSD with a top-k KL in vocabulary space, it only reports AIME24 accuracy and does not report its memory. So the claimed memory saving might not be that reliable.
  2. OPSD also recognized that some tokens can dominate training in KL losses and uses per-token, point-wise clipping. This paper does not discuss the connection with its capping mechanism. What is the connection, and how similar are they?
Questions for authors

please see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer JcLY

We thank the reviewer for the careful review and for recognizing the motivation, experimental effort, and stability results. We address both concerns below.


W1. How much of the memory saving comes from top-k?

Direct MSE activation matching saves memory by avoiding vocabulary-space supervision. For the Fisher objective, top-k is necessary: without truncation, the loss still requires vocabulary-scale projections. We therefore added a matched top-64 OPSD comparison.

Memory comparison on Qwen3-1.7B.

ObjectiveSupervision memory (MiB)Peak training step (GiB)
OPSD, full-vocabulary KL10,69255.8
OPSD, top-64 KL3,58748.9
Fisher A-OPSD, top-642,21347.4

At the same k = 64, Fisher still uses 1.6× less supervision memory than top-64 OPSD (2,213 vs. 3,587 MiB), with a smaller reduction in total step memory (47.4 vs. 48.9 GiB). The more important difference is the accuracy cost of truncating OPSD:

New top-64 OPSD comparison; all scores are percentages.

ModelOPSD objectiveMATH-500AIME24HMMT25Avg
Qwen3-1.7BFull KL77.026.913.639.2
Top-64 KL76.018.36.933.7
Qwen3-4BFull KL89.053.130.357.5
Top-64 KL86.429.415.843.9

Top-64 OPSD loses 5.4 average points on Qwen3-1.7B and 13.6 on Qwen3-4B. In contrast, Fisher A-OPSD remains comparatively stable over the top-k choices in Table 2. Top-k enables the memory savings, but the Fisher objective retains accuracy under this approximation much better than KL. We compare the rest of the models in Appendix C.4 in the revised paper, where the same pattern holds.

W2. How does our cap differ from OPSD clipping?

Our OPSD baselines already use the point-wise KL clipping from Zhao et al. (2026), with their recommended per-model thresholds. The two caps operate at different levels: OPSD clips each vocabulary-coordinate KL contribution before summing over the vocabulary, whereas ours clips the total Fisher loss at each rollout position.

We measured the cap's effect on fixed Qwen3-1.7B rollouts after 100 steps of training:

  • It removes 9.9% of positions.
  • Those positions carry 87% of the squared hidden-state gradient.
  • The capped gradient has cosine 0.36 with the uncapped one, so the cap mainly changes the direction of the update.

So the cap prevents a small set of high-loss positions from dominating the update. We do not claim that every excluded position is irreducible, instead the cap acts as a stability mechanism.

[1] Zhao et al, Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, 2026.


We once again thank the reviewer for their time and effort. We have added all of these results, analyses, and much more to the revised version of the paper. We hope our response has answered all of your questions; if so, we kindly ask the reviewer to reconsider their score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their time and constructive feedback. We are glad the reviewers found the motivation clear (JcLY, Rf3z), the method simple and effective (Yuxb), and the experiments and ablations extensive (JcLY, Yuxb, Rf3z). Below we summarize the new experiments we ran, the main findings, and the changes to the paper.


New experiments

  1. Matched top-64 OPSD (JcLY, Yuxb). We restricted the OPSD KL to the teacher's top-64 tokens and trained it on all four models. We also measured supervision memory and peak training-step memory at matched k = 64.
  2. OPRD baseline (Yuxb). We added OPRD (Yang et al., 2026) as a baseline on Qwen3-1.7B and OLMo-3-7B, using 100-step runs with two runs per method.
  3. Loss-cap diagnostics (JcLY, Yuxb). We measured what the per-token Fisher cap removes on fixed Qwen3-1.7B rollouts at initialization and after 100 steps.
  4. Loss-cap robustness (Rf3z). We ran two additional 500-step capped seeds on Qwen3-1.7B. We also ran extended 500-step training on Qwen3-4B for OPSD, uncapped Fisher, and capped Fisher.
  5. Matched thinking-mode evaluation (Rf3z). We evaluated the base, OPSD, and Fisher A-OPSD checkpoints with thinking enabled on Qwen3-1.7B, 4B, and 8B.
  6. Gains relative to base (Yuxb, Rf3z). We compared every method against its own untrained base model to separate the effect of activation-space supervision from OPSD's variability across model sizes.

Main findings

  • Fisher tolerates top-k truncation; the KL does not. Top-64 OPSD loses 1.2–13.6 average points relative to full-vocabulary OPSD, while Fisher A-OPSD is stable across k. At matched k = 64, Fisher still uses 1.6× less supervision memory than top-64 OPSD. Against full-vocabulary OPSD it uses 4.8×, 3.9×, and 2.4× less on Qwen3-1.7B, 4B, and 8B.
  • Fisher A-OPSD outperforms OPRD by +2.3 average points on Qwen3-1.7B and +1.3 on OLMo-3-7B.
  • The cap is a stability mechanism. After 100 steps, it removes 9.9% of positions that carry 87% of the squared hidden-state gradient, and it mainly changes the direction of the update (cosine 0.36 with the uncapped gradient). The fraction of capped positions falls from 19.8% to 9.9% during training. We therefore make the narrower claim that the cap prevents a few outliers from dominating optimization, rather than that it identifies irreducible errors.
  • The cap's benefit generalizes across seeds and models. All three capped Qwen3-1.7B seeds improve AIME24 from 34.1 to 44.4 on average between steps 100 and 500, while the uncapped run falls from 42.5 to 26.5. On Qwen3-4B, capped Fisher rises from 43.9 to 58.1, while uncapped Fisher falls from 49.6 to 40.0.
  • Fisher improves over base in every setting (+20.2, +17.5, +28.5, and +2.3 average points on Qwen3-1.7B, 4B, 8B, and OLMo-3-7B). The varying margin over OPSD largely reflects how well OPSD works at each size.
  • Thinking mode. Neither OPSD nor Fisher A-OPSD consistently improves over the base model in thinking mode. We therefore limit our performance claims to non-thinking evaluation, consistent with concurrent reports on privileged-context self-distillation (Kaur et al., 2026; Kim et al., 2026).
  • OPSD baselines are not weakened. Every OPSD run uses the original implementation's point-wise KL clipping with its recommended model-specific configurations.

Paper revisions

We have uploaded a revised manuscript in which all new or modified text is highlighted in blue. Main changes:

  • Section 2.3 (Related Work): added OPRD.
  • Section 3.3 (Per-Token Loss Cap): quantified how the high-loss positions concentrate the gradient, and added a paragraph on how our cap differs from OPSD's point-wise KL clipping.
  • Section 4.1 (Setup): stated the OPSD clipping configuration and our evaluation protocol (three seeds, final checkpoint), and how it differs from the original OPSD paper.
  • Section 4.2 (Main Results): added discussions of gains relative to the base model and of the OPRD comparison.
  • Section 4.3 (Longer Horizon Training): added the cap-robustness results across seeds and on Qwen3-4B.
  • Section 4.4 (Memory Efficiency): discussed how the memory advantage depends on k and the hidden dimension, and added the matched top-64 OPSD memory comparison.
  • Section 5 (Top-k Ablations): extended the top-64 OPSD comparison to all four models.
  • Limitations: performance claims are now explicitly scoped to non-thinking evaluation.
  • Appendix A.4: matched thinking-mode evaluation (Table 6).
  • New Appendix C (Additional Comparisons and Analyses):
    • Gains relative to the base model (C.1, Table 7).
    • OPRD comparison (C.2, Table 8).
    • Top-64 OPSD accuracy and memory (C.3, Tables 9–11).
    • Loss-cap diagnostics (C.4, Table 12).
    • Cap robustness across seeds and on Qwen3-4B (C.5, Tables 13–14).

We hope these additions address the reviewers' concerns, and we are happy to answer any further questions during the discussion period.

AuthorsOpenReview ↗

Gentle Reminder: Author–Reviewer Discussion

We thank all reviewers again for their time and thoughtful feedback.

We have posted detailed responses to each review, along with a revised manuscript in which all new or modified text is highlighted in blue.

As the discussion period is coming to a close, we would be grateful if reviewers could let us know whether our responses address their concerns. We are happy to answer any remaining questions or provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.