acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization

Abstract

On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts. However, we find that OPSD methods can progressively degrade previously acquired capabilities: queries that are correctly responded at earlier training stages may become incorrectly responded as optimization proceeds, even the overall performance continues to improve. Further analysis reveals that this degradation coincides with a deterioration of the geometry of hidden representations, characterized by reduced intra-cluster compactness and diminished inter-cluster separation. Motivated by this observation, we propose Representation Geometry Regularization (RGR), which preserves representation geometry during post-training by encouraging compactness within cluster and separation across cluster. RGR can be seamlessly integrated into existing OPSD methods without altering their core optimization procedures. Extensive experiments on multiple OPSD methods and benchmarks show that RGR generally alleviates capability degradation while retaining capability improvements. Our code is available in https://anonymous.4open.science/r/RGR-782F

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training, on-policy self-distillation, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_isJUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates capability degradation during on-policy self-distillation (OPSD), where previously correct responses become incorrect despite improvements in overall performance. The authors associate this phenomenon with deteriorating hidden-state geometry and propose Representation Geometry Regularization (RGR) to preserve intra-cluster compactness and inter-cluster separation through Gram matrix matching. The paper provides theoretical analyses of the relationship between representation geometry and distillation risk. Experiments across four OPSD methods and three mathematical reasoning benchmarks show improvements in accuracy and reductions in capability degradation, with additional evaluation of cross-task forgetting.

Critical strengths
  1. Interesting and novel approach. The paper proposes an interesting perspective on mitigating capability degradation by preserving hidden-state representation geometry, offering an alternative to conventional output-distribution regularization.

  2. Substantial theoretical analysis and broad methodological applicability. The paper provides theoretical motivation for the proposed regularization objective and evaluates its integration with four different OPSD methods, demonstrating generally consistent performance improvements.

Critical weaknesses
  1. Disconnect between the motivating observations, theoretical analysis, and optimization objective. The paper observes deteriorating hidden-state geometry on queries that transition from correct to incorrect, but RGR uniformly regularizes the geometry of current training queries, which may not exhibit the same deterioration. The paper does not establish whether these geometric changes are specific to capability degradation or also occur during successful learning. Furthermore, the theoretical analysis focuses on excess distillation risk rather than directly establishing a connection to correct-to-incorrect transitions. These gaps weaken the justification for the proposed regularization objective.

  2. Insufficient experimental validation and missing critical baselines. The paper does not demonstrate that preserving hidden-state geometry is more effective than simpler alternatives, such as reference-policy KL regularization or direct hidden-state matching. The evaluation is also largely restricted to mathematical reasoning benchmarks and two models from the same family. The cross-task experiment does not report performance on the newly learned task, making it difficult to assess the stability–plasticity trade-off. Consequently, the experiments do not sufficiently establish whether RGR specifically addresses capability degradation or primarily acts as a generic regularizer.

Questions for authors
  1. Geometry deterioration and prompt selection: The motivating analysis focuses on correct-to-incorrect queries, while RGR regularizes all training queries. Does similar geometric deterioration also occur for correct-to-correct or incorrect-to-correct queries? Can the authors provide quantitative comparisons of intra-cluster compactness and inter-cluster separation across these groups?

  2. Justification of the regularization objective: Why should the reference model's representation geometry necessarily be preserved? How does the method distinguish harmful geometric deterioration from beneficial representation changes required for learning new capabilities? Would selectively regularizing queries exhibiting geometric deterioration be more appropriate?

  3. Comparison with simpler regularization methods: How does RGR compare with reference-policy KL regularization and direct hidden-state matching under comparable training settings? Are the improvements specifically attributable to preserving cluster geometry rather than generally restricting representation drift?

  4. Stability–plasticity trade-off: In the cross-task forgetting experiment, how does RGR affect performance on the newly learned tool-use task? Does reducing forgetting come at the cost of slower or weaker acquisition of new capabilities?

  5. Computational overhead: What additional training time and peak GPU memory does RGR require compared with the original OPSD methods, particularly for long reasoning trajectories?

  6. Typos and presentation: The manuscript contains numerous typographical and editorial errors that should be carefully corrected, including:

    • Section 6: “EXEPERIMENT” should be “EXPERIMENT.”
    • Section 6.1: “ImplementationDetails” is missing a space, and “model training..” contains a duplicated period.
    • Section 5.1: The reference to “Eq. 17” appears incorrect or unclear in the context of the main-text derivation.
    • Page 8: Figure 4 appears before Figure 3, making the figure ordering confusing.
    • Section headings: Inconsistent capitalization and spacing are observed across sections and subsections.

The authors should carefully proofread the manuscript and correct these issues to improve readability and presentation quality.

8 comments
AuthorsOpenReview ↗

Responses to Reviewer isJU (Q1-Q2)

Q1: The paper does not establish whether these geometric changes are specific to capability degradation or also occur during successful learning.
R1:We thank the reviewer for this insightful comment.
We additionally illustrate the geometric changes in correct-to-correct cases in Figure 2(b) of the manuscript. As shown, the boundaries between different clusters remain well-defined throughout successful learning.
To further investigate whether geometric degradation is associated with the models' capabilities, we analyze 30 AIME24 problems at five OPSD training checkpoints (steps 25, 50, 75, 100, and 125), yielding 150 response-level observations. Specifically, we first cluster normalized token representations using KMeans. Then, following Sun et al. [1], we evaluate the resulting clustering (geometry) structure using the cosine silhouette score (S)[2]. A higher score indicates more distinct cluster boundaries, whereas a lower score indicates less distinct (i.e., more blurred) cluster boundaries in hidden state geometry.
The table below reports the numbers of incorrect and correct responses, along with their ratio (incorrect/correct), at different cosine silhouette score thresholds. We provide the score and correctness label for each individual sample in Appendix G.

ThresholdIncorrect responsesCorrect responsesIncorrect/correct ratio
S>0.25S>0.2573750.97
S>0.26S>0.2671740.96
S>0.27S>0.2766730.90
S>0.28S>0.2857710.80
S>0.29S>0.2952700.74
S>0.30S>0.3048670.72
S>0.31S>0.3141600.68
S>0.32S>0.3236560.64
S>0.33S>0.3331480.65
S>0.34S>0.3422430.51
S>0.35S>0.3515400.38

As shown in the table, the incorrect-to-correct ratio generally decreases as the cosine silhouette score threshold increases. Since a higher score indicates more distinct cluster boundaries in geometry, this trend suggests that clearer cluster boundaries are associated with fewer incorrect responses and more correct responses. Conversely, geometric degradation, characterized by less distinct cluster boundaries, is more likely to be associated with incorrect responses. This relationship is also theoretically supported by Theorem 4.1 in our paper.

[1] Sun S, Cai M, He H, et al. Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 21990-22006.
[2] Rousseeuw P J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis[J]. Journal of computational and applied mathematics, 1987, 20: 53-65.

Q2: The theoretical analysis focuses on excess distillation risk rather than directly establishing a connection to correct-to-incorrect transitions.
R2: The excess distillation risk measures the capability gap between the optimal student model and the model learned through on-policy self-distillation. Our theorem aims to provide a deeper theoretical understanding of the relationship between the observed geometric degradation and capability loss, particularly the correct-to-incorrect transitions observed during OPSD training. Specifically, we prove that less distinct boundaries between hidden-state clusters (i.e., more severe geometric degradation) lead to higher excess distillation risk, indicating a larger deviation from the optimal student model. Since the optimal model can generate correct responses to a given query, a larger deviation from the optimal model implies a greater risk of failing to preserve the correct response behavior. This provides a theoretical explanation for why queries that were initially answered correctly may become incorrect as representation geometry deteriorates during OPSD training.

AuthorsOpenReview ↗

Responses to Reviewer isJU (Q3-Q4)

Q3: How does RGR compare with reference-policy KL regularization and direct hidden-state matching under comparable training settings?
R3: Thank you for your comment. We compare RGR with five alternatives targeting different aspects of alignment: (1) hidden-state MSE for direct token-level representation matching, (2) inter-layer hidden-state MSE for matching intermediate-layer representations, (3) reference-policy KL regularization [1] for output-distribution alignment, (4) SPKD [2] for preserving inter-sample relationships, and (5) FDD [3] for aligning layer-wise feature dynamics.
As shown below, RGR achieves the highest average score among the evaluated methods under this experimental setting.

MethodAIME24 (%)AIME25 (%)HMMT25 (%)Avg. (%)
OPSD56.6743.0628.8942.87
OPSD + Hidden MSE55.5640.8328.0641.48
OPSD + Inter-layer Hidden MSE59.1740.5626.6742.13
OPSD + Reference KL55.5640.8326.6741.02
OPSD + FDD51.3937.2225.8338.15
OPSD + SPKD56.6740.5624.4440.56
OPSD + RGR59.8943.6030.0044.50

[1] Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback[J]. Advances in neural information processing systems, 2022, 35: 27730-27744.
[2] Tung F, Mori G. Similarity-preserving knowledge distillation[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1365-1374.
[3] Gong G, Wang J, Xu J, et al. Beyond logits: Aligning feature dynamics for effective knowledge distillation[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 23067-23077.

Q4: The evaluation is also largely restricted to mathematical reasoning benchmarks and two models from the same family.
R4: Thanks for your suggestion. Following OPSD [1], we conducted experiments on Qwen3-1.7B and Qwen3-4B. To address your concern, we additionally evaluate RGR on VibeThinker-3B [2], Llama-3.2-3B-Instruct [3], and Phi-4-mini-instruct [4]. The results are reported in the table below. These results support its applicability beyond the Qwen3 family.

ModelAIME24 (%)AIME25 (%)HMMT25 (%)Avg. (%)
VibeThinker-3B79.4472.5050.2867.41
VibeThinker-3B + RGR80.5673.0651.6768.43
Llama-3.2-3B-Instruct3.330.000.001.11
Llama-3.2-3B-Instruct + RGR6.110.560.002.22
Phi-4-mini-instruct5.834.172.784.26
Phi-4-mini-instruct + RGR6.674.442.784.63

We further extend our evaluation beyond mathematical reasoning to science question answering on SciQ[5] and tool-use tasks[6], covering all four self-distillation methods considered in our paper. The results are reported in the table below. These results provide additional empirical support for RGR's applicability beyond mathematical reasoning.

MethodSciQ (%)SciQ + RGR (%)Tool use (%)Tool use + RGR (%)
OPSD89.1790.287.238.46
RLSD91.6791.9444.3647.55
SDPO93.3393.3350.0050.98
SRPO93.0694.1742.0344.36
Avg.91.8192.4335.9137.84

[1] Zhao S, Xie Z, Liu M, et al. Self-distilled reasoner: On-policy self-distillation for large language models[J]. arXiv preprint arXiv:2601.18734, 2026.
[2] Xu S, Liu S, Wang W, et al. VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models[J]. arXiv preprint arXiv:2606.16140, 2026.
[3] Grattafiori A, Dubey A, Jauhri A, et al. The llama 3 herd of models[J]. arXiv preprint arXiv:2407.21783, 2024.
[4] Abouelenin A, Ashfaq A, Atkinson A, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras[J]. arXiv preprint arXiv:2503.01743, 2025.
[5] Welbl J, Liu N F, Gardner M. Crowdsourcing multiple choice science questions[C]//Proceedings of the 3rd Workshop on Noisy User-generated Text. 2017: 94-106.
[6] Tang Q, Deng Z, Lin H, et al. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases[J]. arXiv preprint arXiv:2306.05301, 2023.

AuthorsOpenReview ↗

Responses to Reviewer isJU (Q5-Q8)

Q5: how does RGR affect performance on the newly learned tool-use task?
R5: Thanks for your comment. To address your concern, we conducted additional experiments. For each method, we select the checkpoint with the best mathematical performance and then continue training it on the tool-use training set. We evaluate the checkpoints before and after tool-use training on both the original mathematical benchmarks and the tool-use evaluation set.
As shown below, adding RGR yields higher post-training tool-use accuracy than the corresponding baseline across all four methods. Together with the mathematical retention results, where RGR reduces average forgetting across the four methods from 3.36 to 2.82 percentage points, these findings suggest that RGR mitigates forgetting of previously acquired mathematical capabilities while achieving better final tool-use performance than training without RGR.

MethodBefore training (%)After training (%)
RLSD45.1044.98
RLSD + RGR45.7147.06
SDPO45.8350.00
SDPO + RGR45.3451.47
SRPO45.5952.08
SRPO + RGR45.3452.70
OPSD39.835.00
OPSD + RGR40.939.00

Q6: Why does RGR impose constraints on all training samples?
R6: We believe that degradation of representation geometry is a gradual, cumulative process rather than something that occurs only when a model's answer changes from correct to incorrect. In other words, even when the model still answers a question correctly, its internal representation geometry may already have begun to drift from its original structure. Constraining only Correct → Incorrect examples could therefore overlook examples whose geometry has started to degrade but whose predictions remain correct. For this reason, we apply RGR to all training examples, aiming to limit potential geometric degradation before it leads to a loss of performance.
At the same time, we recognize that applying geometric regularization to all samples could constrain the representation changes needed to learn new capabilities, particularly for Incorrect → Correct examples. To mitigate this risk, we do not optimize the geometric regularization objective in isolation. Instead, we optimize it jointly with the original OPSD training objective, LOPSD\mathcal{L} _ {\mathrm{OPSD}} (Eq. 12). The distillation constraint in the original OPSD objective continues to guide the model toward the teacher policy, providing a learning signal for new capabilities alongside the preservation of existing representation structure. Through this joint optimization, we aim to balance the stability of existing capabilities with the flexibility needed to learn new ones.

Q7: What additional training time and peak GPU memory does RGR require compared with the original OPSD methods, particularly for long reasoning trajectories?
R7: We thank the reviewer for this comment. We compare training time and peak GPU memory with and without RGR. Across the four methods, RGR increases training time per step by an average of 18.96% and peak GPU memory by an average of 2.32%. Although RGR adds some training-time overhead, the additional memory overhead is relatively small. Given the performance improvements observed in our experiments, we believe this is a worthwhile trade-off.
Training-time overhead of RGR.

MethodBase (s/step)+RGR (s/step)Overhead
OPSD1821+16.67%
RLSD2226+18.18%
SRPO2529+16.00%
SDPO2430+25.00%
Average——+18.96%

Peak GPU memory overhead of RGR.

MethodBase (MiB)+RGR (MiB)Overhead
OPSD94,65697,966+3.50%
RLSD50,86051,876+2.00%
SRPO49,50450,552+2.12%
SDPO49,51850,350+1.68%
Average——+2.32%

Q8: The manuscript contains numerous typographical and editorial errors that should be carefully corrected.
R8: Thank you for your comment. We have corrected this in the revised manuscript.

Reviewer_i8tYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors observe that clusters of hidden representations become less distinct during OPSD training, accompanied by previously correct responses becoming incorrect. Motivated by this observation, they propose Representation Geometry Regularization (RGR), which encourages the model to preserve the reference model’s representation geometry while learning new capabilities. Experiments on Qwen3-1.7B and Qwen3-4B show notable performance improvements when RGR is incorporated into OPSD and SDPO, supporting the effectiveness of the method.

Critical strengths

The authors provide intuitive visualizations of the correlation between answer degradation and increasingly blurred cluster boundaries during OPSD training. Their experiments demonstrate that regularization aimed at preserving representation geometry can improve training outcomes. The method is also relatively general and can be integrated into multiple self-distillation methods.

Critical weaknesses

The experimental scope is limited, covering only two relatively small Qwen3 models, with evaluation datasets primarily focused on mathematical reasoning.

Questions for authors

Do similar transitions from well-defined to blurred cluster boundaries also occur during other training procedures, such as SFT, RL, KD? Could RGR also be applied to these settings?

5 comments
AuthorsOpenReview ↗

Responses to Reviewer i8tY (Q1)

Q1: The experimental scope is limited, covering only two relatively small Qwen3 models, with evaluation datasets primarily focused on mathematical reasoning.
R1: We thank the reviewer for highlighting the limited experimental scope. Following OPSD [1], our original experiments focused on Qwen3-1.7B and Qwen3-4B and mathematical reasoning benchmarks. To address your concern, we additionally evaluate RGR on VibeThinker-3B [2], Llama-3.2-3B-Instruct [3], and Phi-4-mini-instruct [4]. The results are reported in the table below. These results support RGR applicability beyond the Qwen3 family, although the improvements are not uniform across models and benchmarks.

ModelAIME24 (%)AIME25 (%)HMMT25 (%)Avg. (%)
VibeThinker-3B79.4472.5050.2867.41
VibeThinker-3B + RGR80.5673.0651.6768.43
Llama-3.2-3B-Instruct3.330.000.001.11
Llama-3.2-3B-Instruct + RGR6.110.560.002.22
Phi-4-mini-instruct5.834.172.784.26
Phi-4-mini-instruct + RGR6.674.442.784.63

We further extend our evaluation beyond mathematical reasoning to science question answering on SciQ and tool-use tasks, covering all four self-distillation methods considered in our paper. The results are reported in the table below. These results provide additional empirical support for RGR's applicability beyond mathematical reasoning.

MethodSciQ (%)SciQ + RGR (%)Tool use (%)Tool use + RGR (%)
OPSD89.1790.287.238.46
RLSD91.6791.9444.3647.55
SDPO93.3393.3350.0050.98
SRPO93.0694.1742.0344.36
Avg.91.8192.4335.9137.84

[1] Zhao S, Xie Z, Liu M, et al. Self-distilled reasoner: On-policy self-distillation for large language models[J]. arXiv preprint arXiv:2601.18734, 2026.
[2] Xu S, Liu S, Wang W, et al. VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models[J]. arXiv preprint arXiv:2606.16140, 2026.
[3] Grattafiori A, Dubey A, Jauhri A, et al. The llama 3 herd of models[J]. arXiv preprint arXiv:2407.21783, 2024.
[4] Abouelenin A, Ashfaq A, Atkinson A, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras[J]. arXiv preprint arXiv:2503.01743, 2025.

AuthorsOpenReview ↗

Responses to Reviewer i8tY (Q2)

Q2: Do similar transitions from well-defined to blurred cluster boundaries also occur during other training procedures, such as SFT, RL, KD? Could RGR also be applied to these settings?
R2: We thank the reviewer for this insightful question. We conducted a quantitative geometric analysis to answer the question. Specifically, we analyzed 30 AIME24 problems at five training checkpoints (steps 25, 50, 75, 100, and 125) for each of OPSD, SFT, GRPO, and KD, yielding 150 response-level observations per training procedure. Specifically, we first cluster normalized token representations using KMeans. Then, following Sun et al. [1], we evaluate the resulting clustering structure (geometry) using the cosine silhouette score (S)[2]. A higher score indicates more distinct cluster boundaries, whereas a lower score indicates less distinct (i.e., more blurred) cluster boundaries in hidden state geometry. For each given cosine silhouette score threshold xx, we report samples with S>xS>x and report the numbers of incorrect and correct responses, along with their ratio (incorrect/correct). (The details of cosine silhouette score and the correctness of each response are provided in Appendix H.3)

MethodS>0.25S>0.25S>0.26S>0.26S>0.27S>0.27S>0.28S>0.28S>0.29S>0.29S>0.30S>0.30S>0.31S>0.31S>0.32S>0.32S>0.33S>0.33S>0.34S>0.34S>0.35S>0.35
OPSD0.97 (73/75)0.96 (71/74)0.90 (66/73)0.80 (57/71)0.74 (52/70)0.72 (48/67)0.68 (41/60)0.64 (36/56)0.65 (31/48)0.51 (22/43)0.38 (15/40)
GRPO1.03 (70/68)0.95 (62/65)0.95 (57/60)0.79 (42/53)0.70 (32/46)0.68 (25/37)0.42 (14/33)0.27 (7/26)0.27 (6/22)0.29 (4/14)0.15 (2/13)
KD0.92 (67/73)0.85 (57/67)0.75 (48/64)0.71 (42/59)0.63 (34/54)0.70 (28/40)0.74 (25/34)0.53 (17/32)0.38 (11/29)0.35 (8/23)0.42 (8/19)
SFT3.35 (114/34)3.47 (111/32)3.42 (106/31)3.81 (103/27)3.92 (98/25)4.04 (93/23)4.63 (88/19)5.18 (88/17)6.21 (87/14)6.23 (81/13)8.67 (78/9)

As shown in the table, the incorrect-to-correct ratio generally decreases as the cosine silhouette score threshold increases for OPSD, GRPO, and KD, whereas SFT exhibits the opposite trend. Since a higher score indicates more distinct cluster boundaries, these results suggest that clearer cluster boundaries are associated with fewer incorrect responses and more correct responses in OPSD, GRPO, and KD. In contrast, this association is not observed in SFT.
To directly test whether RGR can be applied to these settings, we incorporated it into SFT, GRPO, and KD and evaluated performance on AIME24, AIME25, and HMMT25. These results support RGR 's applicability to SFT, GRPO, and KD, extending its benefits beyond self-distillation. Interestingly, although SFT exhibits a different association between representation geometry and response correctness compared with GRPO and KD, RGR still improves its performance. We conjecture that this is because RGR regularizes the preservation of the overall representation geometry during training, rather than directly optimizing cluster separation or enforcing a specific relationship between geometric structure and answer correctness. Therefore, its benefits are not restricted to training paradigms in which clearer cluster boundaries are positively associated with answer correctness.

MethodAIME24 (%)AIME25 (%)HMMT25 (%)Avg. (%)
SFT48.4035.8322.7035.64
SFT + RGR49.7237.5023.6136.94
GRPO51.1137.5023.7037.44
GRPO + RGR51.1138.0626.1138.43
KD45.2835.5623.0634.63
KD + RGR48.3337.7823.0636.39

[1] Sun S, Cai M, He H, et al. Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 21990-22006.
[2] Rousseeuw P J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis[J]. Journal of computational and applied mathematics, 1987, 20: 53-65.

AuthorsOpenReview ↗

Do Our Responses Resolve Your Main Concerns?

Dear Reviewer i8tY,

I hope this message finds you well. We have provided detailed responses to your insightful comments, which we hope address them satisfactorily. Some of the key clarifications include the following:

  1. Generalizability. We added experiments on three additional models, science QA, and tool use, as well as SFT, GRPO, and KD, providing broader evidence of RGR's applicability.
  2. Quantitative analysis. We analyzed 150 responses per training procedure. Clearer cluster boundaries are associated with higher answer correctness in OPSD, GRPO, and KD, whereas SFT shows the opposite trend.
  3. Applicability beyond self-distillation. We incorporated RGR into SFT, GRPO, and KD, improving average accuracy in all three settings.

We sincerely hope that these clarifications have resolved your concerns. If so, we would greatly appreciate it if you could consider updating your evaluation score in light of these clarifications. If there are any additional points or feedback you would like us to consider, please let us know. Your insights are invaluable to us, and we are eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,

The Authors

Reviewer_M1ikOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the capability degradation phenomenon in large language models during on-policy self-distillation, where previously correct responses may become incorrect as training progresses, even when overall performance improves. The authors associate this phenomenon with changes in hidden representation geometry, particularly reduced intra-cluster compactness and inter-cluster separation. Based on this observation, the paper proposes Representation Geometry Regularization (RGR), which introduces a geometry-preserving objective based on Gram matrix alignment, together with an entropy-related regularization term, into existing OPSD methods. The authors provide theoretical analyses relating representation geometry to distillation risk and evaluate RGR on four self-distillation methods using Qwen3-1.7B and Qwen3-4B. Experiments on three mathematical reasoning benchmarks examine overall performance, capability degradation during training, and cross-task forgetting.

Critical strengths
  1. Instead of focusing exclusively on overall performance improvements, the authors examine correct-to-incorrect transitions during training and analyze their relationship with changes in hidden representations. This provides an additional perspective for evaluating the stability of self-distillation methods.
  2. RGR is integrated into four existing methods, including OPSD, SDPO, RLSD, and SRPO. The paper reports comparisons of reasoning accuracy, component ablations, training dynamics, and cross-task forgetting, providing empirical evidence of its applicability to different self-distillation objectives.
Critical weaknesses
  1. The core idea of RGR is to preserve hidden representation geometry through regularization, which is closely related to existing representation alignment, relational knowledge distillation, and feature similarity preservation techniques. In particular, the proposed geometry-preserving objective is essentially based on matching Gram matrices between student and reference representations, a relatively established technique in representation learning. Although the authors provide a theoretical motivation for this objective, it remains unclear what fundamentally distinguishes the proposed method from conventional feature-level distillation or representation consistency regularization. The paper should clarify its methodological novelty and compare it with related regularization approaches.
  2. The paper argues that deteriorating representation geometry contributes to the loss of previously acquired capabilities. However, the empirical evidence relies heavily on visualizations of hidden representations, which primarily demonstrate correlation rather than causation. Moreover, the theoretical analysis establishes bounds on excess distillation risk rather than directly characterizing correct-to-incorrect prediction transitions. Therefore, the connection between the theoretical results and the claimed capability degradation mechanism remains insufficiently established. More quantitative analyses and controlled experiments are needed to demonstrate that representation geometry distortion is a primary cause of capability degradation.
  3. The evaluation is restricted to two relatively small models from the Qwen3 family, namely Qwen3-1.7B and Qwen3-4B. Furthermore, the main experiments focus exclusively on mathematical reasoning benchmarks, including AIME 2024, AIME 2025, and HMMT 2025. These settings are insufficient to establish whether RGR generalizes to larger language models, different model families, or other capabilities such as coding, instruction following, and general knowledge. Additional experiments across model architectures, scales, and task domains would strengthen the generalizability claims.
  4. The experimental comparisons primarily focus on applying RGR to existing OPSD methods and evaluating the performance differences. However, the paper does not compare RGR against alternative approaches designed to stabilize representations or mitigate forgetting, such as feature matching, hidden-state consistency regularization, or conventional knowledge distillation with intermediate representation alignment. Without these comparisons, it is difficult to determine whether the observed improvements originate from the proposed geometry-preserving mechanism or from the more general benefits of adding representation-level regularization.
  5. The ablation studies are insufficient to isolate the contributions of the proposed components. Although the authors evaluate variants without geometry regularization or entropy regularization, the analysis does not sufficiently distinguish the individual contributions of intra-cluster compactness and inter-cluster separation. Since these two properties constitute the central motivation of RGR, separate evaluations of their effects would help validate the proposed design. Furthermore, the paper introduces two regularization coefficients but does not provide a systematic sensitivity analysis of these hyperparameters. It remains unclear how sensitive the method is to the regularization strength and whether the same configuration consistently works across different models and tasks.
Questions for authors

See the weakness.

6 comments
AuthorsOpenReview ↗

Responses to Reviewer M1ik (Q1)

Q1: What fundamentally distinguishes RGR from conventional feature-level distillation or representation consistency regularization?
R1: Thank you for your comment. To be honest, we acknowledge that RGR is closely related to existing work on feature-level distillation and representation consistency regularization. However, RGR differs from these methods:
(1) Unlike conventional feature-level distillation, which conducts distillation by aligning teacher and student hidden states at corresponding token positions, RGR does not require token-wise feature vectors to match. Instead, it constrains their overall geometric structure across the generated sequence.
(2) Compared with classical Relational Knowledge Distillation and Feature Similarity Preservation methods, such as SPKD [1], which mainly focus on relationships between different samples, RGR focuses on the representation geometry within a single sequence to mitigate its degradation during OPSD training.
To further assess RGR's advantages over existing feature-level distillation and representation-consistency approaches, we conduct additional experiments.
The results are reported in the table below, where hidden MSE matches final-layer hidden states, Inter-layer Hidden MSE matches intermediate-layer hidden states, and Reference KL [2] aligns output distributions. SPKD matches relationships between samples, while FDD [3] matches changes in representations across layers. The results suggest that RGR is more effective on average than feature-matching and representation consistency regularization methods.

MethodAIME24 (%)AIME25 (%)HMMT25 (%)Avg. (%)
OPSD56.6743.0628.8942.87
OPSD + Hidden MSE55.5640.8328.0641.48
OPSD + Inter-layer Hidden MSE59.1740.5626.6742.13
OPSD + Reference KL55.5640.8326.6741.02
OPSD + FDD51.3937.2225.8338.15
OPSD + SPKD56.6740.5624.4440.56
OPSD + RGR59.8943.6030.0044.50

[1] Tung F, Mori G. Similarity-preserving knowledge distillation[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1365-1374.
[2] Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback[J]. Advances in neural information processing systems, 2022, 35: 27730-27744.
[3] Gong G, Wang J, Xu J, et al. Beyond logits: Aligning feature dynamics for effective knowledge distillation[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 23067-23077.

AuthorsOpenReview ↗

Responses to Reviewer M1ik (Q2)

Q2: The paper argues that deteriorating representation geometry contributes to the loss of previously acquired capabilities. However, the empirical evidence relies heavily on visualizations of hidden representations, which primarily demonstrate correlation rather than causation.
R2: We thank the reviewer for this insightful comment. To further investigate whether geometric degradation is associated with the models' capabilities, we analyze 30 AIME24 problems at five OPSD training checkpoints (steps 25, 50, 75, 100, and 125), yielding 150 response-level observations. Specifically, we first cluster normalized token representations using KMeans. Then, following Sun et al. [1], we evaluate the resulting clustering (geometry) structure using the cosine silhouette score (S)[2]. A higher score indicates more distinct cluster boundaries, whereas a lower score indicates less distinct (i.e., more blurred) cluster boundaries in hidden state geometry.
The table below reports the numbers of incorrect and correct responses, along with their ratio (incorrect/correct), at different cosine silhouette score thresholds. We provide the score and correctness label for each individual sample in Appendix G.

ThresholdIncorrect responsesCorrect responsesIncorrect/correct ratio
S>0.25S>0.2573750.97
S>0.26S>0.2671740.96
S>0.27S>0.2766730.90
S>0.28S>0.2857710.80
S>0.29S>0.2952700.74
S>0.30S>0.3048670.72
S>0.31S>0.3141600.68
S>0.32S>0.3236560.64
S>0.33S>0.3331480.65
S>0.34S>0.3422430.51
S>0.35S>0.3515400.38

As shown in the table, the incorrect-to-correct ratio generally decreases as the cosine silhouette score threshold increases. Since a higher score indicates more distinct cluster boundaries in geometry, this trend suggests that clearer cluster boundaries are associated with fewer incorrect responses and more correct responses. Conversely, geometric degradation, characterized by less distinct cluster boundaries, is more likely to be associated with incorrect responses. This relationship is also theoretically supported by Theorem 4.1 in our paper.

[1] Sun S, Cai M, He H, et al. Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 21990-22006.
[2] Rousseeuw P J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis[J]. Journal of computational and applied mathematics, 1987, 20: 53-65.

AuthorsOpenReview ↗

Responses to Reviewer M1ik (Q3-Q4)

Q3:Additional experiments across model architectures, scales, and task domains would strengthen the generalizability claims.
R3: Thank you for this valuable suggestion. To further assess RGR’s generalizability, we have conducted additional experiments across model families and scales, training paradigms, and task domains and datasets.
Generalization across model families. Beyond Qwen3-1.7B and Qwen3-4B in the original submission, we additionally evaluate RGR on VibeThinker-3B [1], Llama-3.2-3B-Instruct [2], and Phi-4-mini-instruct [3]. The results are reported in the table below. RGR improves the average score for all three additional models, providing further evidence for RGR applicability across model families.

ModelAIME24 (%)AIME25 (%)HMMT25 (%)Avg. (%)
VibeThinker-3B79.4472.5050.2867.41
VibeThinker-3B + RGR80.5673.0651.6768.43
Llama-3.2-3B-Instruct3.330.000.001.11
Llama-3.2-3B-Instruct + RGR6.110.560.002.22
Phi-4-mini-instruct5.834.172.784.26
Phi-4-mini-instruct + RGR6.674.442.784.63

Generalization across training paradigms. To examine whether RGR extends beyond on-policy self-distillation, we additionally integrate it into supervised fine-tuning (SFT), GRPO, and knowledge distillation (KD). The results are reported in the table below. RGR improves the average score under SFT, GRPO, and KD, suggesting that its benefits extend beyond self-distillation to supervised fine-tuning, reinforcement learning, and knowledge distillation.

MethodAIME24 (%)AIME25 (%)HMMT25 (%)Avg. (%)
SFT48.4035.8322.7035.64
SFT + RGR49.7237.5023.6136.94
GRPO51.1137.5023.7037.44
GRPO + RGR51.1138.0626.1138.43
KD45.2835.5623.0634.63
KD + RGR48.3337.7823.0636.39

Generalization across task domains and datasets. We further evaluate RGR on science question answering using SciQ [4] and tool-use tasks [5], covering all four self-distillation methods considered in our paper. The results are reported in the table below. RGR increases the average score across the four methods on both SciQ and tool-use tasks, supporting its applicability beyond mathematical reasoning.

MethodSciQ (%)SciQ + RGR (%)Tool use (%)Tool use + RGR (%)
OPSD89.1790.287.238.46
RLSD91.6791.9444.3647.55
SDPO93.3393.3350.0050.98
SRPO93.0694.1742.0344.36
Avg.91.8192.4335.9137.84

We will incorporate these results and the corresponding experimental details into the revised manuscript.

[1] Xu S, Liu S, Wang W, et al. VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models[J]. arXiv preprint arXiv:2606.16140, 2026.
[2] Grattafiori A, Dubey A, Jauhri A, et al. The llama 3 herd of models[J]. arXiv preprint arXiv:2407.21783, 2024.
[3] Abouelenin A, Ashfaq A, Atkinson A, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras[J]. arXiv preprint arXiv:2503.01743, 2025.
[4] Welbl J, Liu N F, Gardner M. Crowdsourcing multiple choice science questions[C]//Proceedings of the 3rd Workshop on Noisy User-generated Text. 2017: 94-106.
[5] Tang Q, Deng Z, Lin H, et al. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases[J]. arXiv preprint arXiv:2306.05301, 2023.

Q4:The ablation study does not sufficiently distinguish the individual contributions of intra-cluster compactness and inter-cluster separation.
R4: Thanks for your comment. We would like to clarify that intra-cluster compactness and inter-cluster separation are not implemented as two independent regularization terms in our framework.
As explained in Section 5.3, the constraints on intra-cluster compactness (Eq. 5) and inter-cluster separation (Eq. 10) share the same underlying optimization term. Therefore, we formulate a unified geometry-preserving objective, Lgeo\mathcal{L} _ {\mathrm{geo}} (Eq. 11), which simultaneously captures both geometric properties.
Consequently, unlike methods that combine separate compactness and separation losses, our ablation study does not contain two independently removable components. It evaluates the contribution of the unified geometry-preserving objective by comparing models trained with and without Lgeo\mathcal{L} _ {\mathrm{geo}}.

Comments on the paper

2 comments

Dear reviewers, please review the authors’ rebuttal and provide your responses as soon as possible. Your timely feedback is important for moving the discussion forward. Thank you!

AuthorsOpenReview ↗

General Response

We thank all three reviewers for their time and constructive feedback. Below we summarize our main contribution, the strengths noted by the reviewers, and the revisions made in response to their concerns.

Main Contribution

During on-policy self-distillation (OPSD), a model can make mistakes on questions it previously answered correctly (Figure 1). Our quantitative analysis showed that responses with better-defined token clusters were more often correct (Appendix G). Our theoretical analysis further links representation geometry degradation, characterized by blurred cluster boundaries, to increased deviation from optimal responses and a higher risk of errors. Motivated by these findings, we propose Representation Geometry Regularization (RGR), a plug-and-play regularizer that preserves token representation geometry to mitigate capability degradation during OPSD. Across all four self-distillation methods, RGR reduced correct-to-incorrect transitions and improved average accuracy (Figure 3; Table 1).

Strengths Noted by the Reviewers

The reviewers highlighted three strengths of our paper: stability analysis, theoretical motivation, and broad applicability.

  1. Stability beyond aggregate accuracy. M1ik highlighted the value of tracking correct-to-incorrect transitions during training, and I8ty noted that the visualizations make the association between answer degradation and less distinct token clusters intuitive.
  2. Novelty and theoretical motivation. isJU highlighted the use of hidden-state geometry as an alternative to output-distribution regularization and the theoretical motivation for the objective.
  3. Applicability and empirical breadth. All three reviewers noted that RGR integrates with multiple self-distillation methods. M1ik also highlighted the ablations, training dynamics, and cross-task forgetting analyses.

Main Concerns and Our Responses

We clarified how token geometry relates to answer correctness and motivates RGR, compared RGR with alternative regularizers, and evaluated its generalization, retention of prior skills, performance on new tasks, sensitivity, and computational cost.

  1. Geometric degradation and correctness. Figure 2 now contrasts two cases: token clusters become less distinct in the correct-to-incorrect example but remain stable in the correct-to-correct example. We quantified the association across 150 responses from five OPSD checkpoints: responses with better-defined token clusters were more often correct (Appendix G). We clarified that Theorem 4.1 shows that blurred token-cluster boundaries can lead to greater deviations from optimal responses and an increased likelihood of generating incorrect responses. Applying RGR to all responses yielded higher average accuracy than applying it only to correct responses (Appendix H.4).
  2. Alternative regularizers. RGR achieved higher accuracy than all five alternatives, including feature-level regularizers and reference-policy KL (Appendix F, Table 8).
  3. Generalization. We observed gains on three additional base models and on science question answering and tool use (Appendices H.1–H.2). We also found that RGR improved average accuracy on mathematics benchmarks for model trained with GRPO, KD, and SFT (Appendix H.3).
  4. Retention and new-task performance. RGR reduced average mathematical forgetting while improving final tool-use accuracy for each of the four methods (Appendix E, Table 7).
  5. Sensitivity, cost, and presentation. We analyzed sensitivity to both regularization coefficients, reported training-time and memory overhead, and corrected the presentation issues noted by the reviewers (Appendices B and D).

We appreciate the reviewers' and chairs' consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.