- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates capability degradation during on-policy self-distillation (OPSD), where previously correct responses become incorrect despite improvements in overall performance. The authors associate this phenomenon with deteriorating hidden-state geometry and propose Representation Geometry Regularization (RGR) to preserve intra-cluster compactness and inter-cluster separation through Gram matrix matching. The paper provides theoretical analyses of the relationship between representation geometry and distillation risk. Experiments across four OPSD methods and three mathematical reasoning benchmarks show improvements in accuracy and reductions in capability degradation, with additional evaluation of cross-task forgetting.
Critical strengths
-
Interesting and novel approach. The paper proposes an interesting perspective on mitigating capability degradation by preserving hidden-state representation geometry, offering an alternative to conventional output-distribution regularization.
-
Substantial theoretical analysis and broad methodological applicability. The paper provides theoretical motivation for the proposed regularization objective and evaluates its integration with four different OPSD methods, demonstrating generally consistent performance improvements.
Critical weaknesses
-
Disconnect between the motivating observations, theoretical analysis, and optimization objective. The paper observes deteriorating hidden-state geometry on queries that transition from correct to incorrect, but RGR uniformly regularizes the geometry of current training queries, which may not exhibit the same deterioration. The paper does not establish whether these geometric changes are specific to capability degradation or also occur during successful learning. Furthermore, the theoretical analysis focuses on excess distillation risk rather than directly establishing a connection to correct-to-incorrect transitions. These gaps weaken the justification for the proposed regularization objective.
-
Insufficient experimental validation and missing critical baselines. The paper does not demonstrate that preserving hidden-state geometry is more effective than simpler alternatives, such as reference-policy KL regularization or direct hidden-state matching. The evaluation is also largely restricted to mathematical reasoning benchmarks and two models from the same family. The cross-task experiment does not report performance on the newly learned task, making it difficult to assess the stability–plasticity trade-off. Consequently, the experiments do not sufficiently establish whether RGR specifically addresses capability degradation or primarily acts as a generic regularizer.
Questions for authors
-
Geometry deterioration and prompt selection: The motivating analysis focuses on correct-to-incorrect queries, while RGR regularizes all training queries. Does similar geometric deterioration also occur for correct-to-correct or incorrect-to-correct queries? Can the authors provide quantitative comparisons of intra-cluster compactness and inter-cluster separation across these groups?
-
Justification of the regularization objective: Why should the reference model's representation geometry necessarily be preserved? How does the method distinguish harmful geometric deterioration from beneficial representation changes required for learning new capabilities? Would selectively regularizing queries exhibiting geometric deterioration be more appropriate?
-
Comparison with simpler regularization methods: How does RGR compare with reference-policy KL regularization and direct hidden-state matching under comparable training settings? Are the improvements specifically attributable to preserving cluster geometry rather than generally restricting representation drift?
-
Stability–plasticity trade-off: In the cross-task forgetting experiment, how does RGR affect performance on the newly learned tool-use task? Does reducing forgetting come at the cost of slower or weaker acquisition of new capabilities?
-
Computational overhead: What additional training time and peak GPU memory does RGR require compared with the original OPSD methods, particularly for long reasoning trajectories?
-
Typos and presentation: The manuscript contains numerous typographical and editorial errors that should be carefully corrected, including:
- Section 6: “EXEPERIMENT” should be “EXPERIMENT.”
- Section 6.1: “ImplementationDetails” is missing a space, and “model training..” contains a duplicated period.
- Section 5.1: The reference to “Eq. 17” appears incorrect or unclear in the context of the main-text derivation.
- Page 8: Figure 4 appears before Figure 3, making the figure ordering confusing.
- Section headings: Inconsistent capitalization and spacing are observed across sections and subsections.
The authors should carefully proofread the manuscript and correct these issues to improve readability and presentation quality.
8 comments
Responses to Reviewer isJU (Q3-Q4)
Q3: How does RGR compare with reference-policy KL regularization and direct hidden-state matching under comparable training settings?
R3: Thank you for your comment. We compare RGR with five alternatives targeting different aspects of alignment: (1) hidden-state MSE for direct token-level representation matching, (2) inter-layer hidden-state MSE for matching intermediate-layer representations, (3) reference-policy KL regularization [1] for output-distribution alignment, (4) SPKD [2] for preserving inter-sample relationships, and (5) FDD [3] for aligning layer-wise feature dynamics.
As shown below, RGR achieves the highest average score among the evaluated methods under this experimental setting.
| Method | AIME24 (%) | AIME25 (%) | HMMT25 (%) | Avg. (%) |
|---|---|---|---|---|
| OPSD | 56.67 | 43.06 | 28.89 | 42.87 |
| OPSD + Hidden MSE | 55.56 | 40.83 | 28.06 | 41.48 |
| OPSD + Inter-layer Hidden MSE | 59.17 | 40.56 | 26.67 | 42.13 |
| OPSD + Reference KL | 55.56 | 40.83 | 26.67 | 41.02 |
| OPSD + FDD | 51.39 | 37.22 | 25.83 | 38.15 |
| OPSD + SPKD | 56.67 | 40.56 | 24.44 | 40.56 |
| OPSD + RGR | 59.89 | 43.60 | 30.00 | 44.50 |
[1] Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback[J]. Advances in neural information processing systems, 2022, 35: 27730-27744.
[2] Tung F, Mori G. Similarity-preserving knowledge distillation[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1365-1374.
[3] Gong G, Wang J, Xu J, et al. Beyond logits: Aligning feature dynamics for effective knowledge distillation[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 23067-23077.
Q4: The evaluation is also largely restricted to mathematical reasoning benchmarks and two models from the same family.
R4: Thanks for your suggestion. Following OPSD [1], we conducted experiments on Qwen3-1.7B and Qwen3-4B. To address your concern, we additionally evaluate RGR on VibeThinker-3B [2], Llama-3.2-3B-Instruct [3], and Phi-4-mini-instruct [4]. The results are reported in the table below. These results support its applicability beyond the Qwen3 family.
| Model | AIME24 (%) | AIME25 (%) | HMMT25 (%) | Avg. (%) |
|---|---|---|---|---|
| VibeThinker-3B | 79.44 | 72.50 | 50.28 | 67.41 |
| VibeThinker-3B + RGR | 80.56 | 73.06 | 51.67 | 68.43 |
| Llama-3.2-3B-Instruct | 3.33 | 0.00 | 0.00 | 1.11 |
| Llama-3.2-3B-Instruct + RGR | 6.11 | 0.56 | 0.00 | 2.22 |
| Phi-4-mini-instruct | 5.83 | 4.17 | 2.78 | 4.26 |
| Phi-4-mini-instruct + RGR | 6.67 | 4.44 | 2.78 | 4.63 |
We further extend our evaluation beyond mathematical reasoning to science question answering on SciQ[5] and tool-use tasks[6], covering all four self-distillation methods considered in our paper. The results are reported in the table below. These results provide additional empirical support for RGR's applicability beyond mathematical reasoning.
| Method | SciQ (%) | SciQ + RGR (%) | Tool use (%) | Tool use + RGR (%) |
|---|---|---|---|---|
| OPSD | 89.17 | 90.28 | 7.23 | 8.46 |
| RLSD | 91.67 | 91.94 | 44.36 | 47.55 |
| SDPO | 93.33 | 93.33 | 50.00 | 50.98 |
| SRPO | 93.06 | 94.17 | 42.03 | 44.36 |
| Avg. | 91.81 | 92.43 | 35.91 | 37.84 |
[1] Zhao S, Xie Z, Liu M, et al. Self-distilled reasoner: On-policy self-distillation for large language models[J]. arXiv preprint arXiv:2601.18734, 2026.
[2] Xu S, Liu S, Wang W, et al. VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models[J]. arXiv preprint arXiv:2606.16140, 2026.
[3] Grattafiori A, Dubey A, Jauhri A, et al. The llama 3 herd of models[J]. arXiv preprint arXiv:2407.21783, 2024.
[4] Abouelenin A, Ashfaq A, Atkinson A, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras[J]. arXiv preprint arXiv:2503.01743, 2025.
[5] Welbl J, Liu N F, Gardner M. Crowdsourcing multiple choice science questions[C]//Proceedings of the 3rd Workshop on Noisy User-generated Text. 2017: 94-106.
[6] Tang Q, Deng Z, Lin H, et al. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases[J]. arXiv preprint arXiv:2306.05301, 2023.
Responses to Reviewer isJU (Q5-Q8)
Q5: how does RGR affect performance on the newly learned tool-use task?
R5: Thanks for your comment. To address your concern, we conducted additional experiments. For each method, we select the checkpoint with the best mathematical performance and then continue training it on the tool-use
training set. We evaluate the checkpoints before and after tool-use training on both the original mathematical benchmarks and the tool-use evaluation set.
As shown below, adding RGR yields higher post-training tool-use accuracy than the corresponding baseline across all four methods. Together with the mathematical retention results, where RGR reduces average forgetting across the four methods from 3.36 to 2.82 percentage points, these findings suggest that RGR mitigates forgetting of previously acquired mathematical capabilities while achieving better final tool-use performance than training without RGR.
| Method | Before training (%) | After training (%) |
|---|---|---|
| RLSD | 45.10 | 44.98 |
| RLSD + RGR | 45.71 | 47.06 |
| SDPO | 45.83 | 50.00 |
| SDPO + RGR | 45.34 | 51.47 |
| SRPO | 45.59 | 52.08 |
| SRPO + RGR | 45.34 | 52.70 |
| OPSD | 39.83 | 5.00 |
| OPSD + RGR | 40.93 | 9.00 |
Q6: Why does RGR impose constraints on all training samples?
R6: We believe that degradation of representation geometry is a gradual, cumulative process rather than something that occurs only when a model's answer changes from correct to incorrect. In other words, even when the model still answers a question correctly, its internal representation geometry may already have begun to drift from its original structure. Constraining only Correct → Incorrect examples could therefore overlook examples whose geometry has started to degrade but whose predictions remain correct. For this reason, we apply RGR to all training examples, aiming to limit potential geometric degradation before it leads to a loss of performance.
At the same time, we recognize that applying geometric regularization to all samples could constrain the representation changes needed to learn new capabilities, particularly for Incorrect → Correct examples. To mitigate this risk, we do not optimize the geometric regularization objective in isolation. Instead, we optimize it jointly with the original OPSD training objective, (Eq. 12). The distillation constraint in the original OPSD objective continues to guide the model toward the teacher policy, providing a learning signal for new capabilities alongside the preservation of existing representation structure. Through this joint optimization, we aim to balance the stability of existing capabilities with the flexibility needed to learn new ones.
Q7: What additional training time and peak GPU memory does RGR require compared with the original OPSD methods, particularly for long reasoning trajectories?
R7: We thank the reviewer for this comment. We compare training time and peak GPU memory with and without RGR. Across the four methods, RGR increases training time per step by an average of 18.96% and peak GPU memory by an average of 2.32%. Although RGR adds some training-time overhead, the additional memory overhead is relatively small. Given the performance improvements observed in our experiments, we believe this is a worthwhile trade-off.
Training-time overhead of RGR.
| Method | Base (s/step) | +RGR (s/step) | Overhead |
|---|---|---|---|
| OPSD | 18 | 21 | +16.67% |
| RLSD | 22 | 26 | +18.18% |
| SRPO | 25 | 29 | +16.00% |
| SDPO | 24 | 30 | +25.00% |
| Average | — | — | +18.96% |
Peak GPU memory overhead of RGR.
| Method | Base (MiB) | +RGR (MiB) | Overhead |
|---|---|---|---|
| OPSD | 94,656 | 97,966 | +3.50% |
| RLSD | 50,860 | 51,876 | +2.00% |
| SRPO | 49,504 | 50,552 | +2.12% |
| SDPO | 49,518 | 50,350 | +1.68% |
| Average | — | — | +2.32% |
Q8: The manuscript contains numerous typographical and editorial errors that should be carefully corrected.
R8: Thank you for your comment. We have corrected this in the revised manuscript.
Responses to Reviewer isJU (Q1-Q2)
Q1: The paper does not establish whether these geometric changes are specific to capability degradation or also occur during successful learning.
R1:We thank the reviewer for this insightful comment.
We additionally illustrate the geometric changes in correct-to-correct cases in Figure 2(b) of the manuscript. As shown, the boundaries between different clusters remain well-defined throughout successful learning.
To further investigate whether geometric degradation is associated with the models' capabilities, we analyze 30 AIME24 problems at five OPSD training checkpoints (steps 25, 50, 75, 100, and 125), yielding 150 response-level observations. Specifically, we first cluster normalized token representations using KMeans. Then, following Sun et al. [1], we evaluate the resulting clustering (geometry) structure using the cosine silhouette score (S)[2]. A higher score indicates more distinct cluster boundaries, whereas a lower score indicates less distinct (i.e., more blurred) cluster boundaries in hidden state geometry.
The table below reports the numbers of incorrect and correct responses, along with their ratio (incorrect/correct), at different cosine silhouette score thresholds. We provide the score and correctness label for each individual sample in Appendix G.
As shown in the table, the incorrect-to-correct ratio generally decreases as the cosine silhouette score threshold increases. Since a higher score indicates more distinct cluster boundaries in geometry, this trend suggests that clearer cluster boundaries are associated with fewer incorrect responses and more correct responses. Conversely, geometric degradation, characterized by less distinct cluster boundaries, is more likely to be associated with incorrect responses. This relationship is also theoretically supported by Theorem 4.1 in our paper.
[1] Sun S, Cai M, He H, et al. Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 21990-22006.
[2] Rousseeuw P J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis[J]. Journal of computational and applied mathematics, 1987, 20: 53-65.
Q2: The theoretical analysis focuses on excess distillation risk rather than directly establishing a connection to correct-to-incorrect transitions.
R2: The excess distillation risk measures the capability gap between the optimal student model and the model learned through on-policy self-distillation. Our theorem aims to provide a deeper theoretical understanding of the relationship between the observed geometric degradation and capability loss, particularly the correct-to-incorrect transitions observed during OPSD training. Specifically, we prove that less distinct boundaries between hidden-state clusters (i.e., more severe geometric degradation) lead to higher excess distillation risk, indicating a larger deviation from the optimal student model. Since the optimal model can generate correct responses to a given query, a larger deviation from the optimal model implies a greater risk of failing to preserve the correct response behavior. This provides a theoretical explanation for why queries that were initially answered correctly may become incorrect as representation geometry deteriorates during OPSD training.