acceptodds
Under review as a conference paper at ICLR 2027

Geometric Self-Distillation for Reasoning Generalization

33 views

Abstract

On-policy distillation provides dense teacher supervision on a language model's own trajectories. In self-distillation with privileged context, this supervision comes from the model itself, conditioned on a hint or solution trace hidden from the student. When the teacher’s preferences hinge on privileged information, it can assign higher probability to continuations the student cannot infer from its own context. Matching these preferences throughout training can induce predictive drift and degrade out-of-distribution (OOD) reasoning. We propose GeoSD, a self-distillation method that controls this drift through two complementary geometric terms. A Hellinger loss weights each teacher preference by the student–teacher overlap, reducing the influence of tokens to which the student assigns low probability. Because these influences can still accumulate, a Fisher–Rao penalty regulates predictive distance from a copy of the student refreshed periodically during training. Both terms compare next-token distributions in Fisher–Rao geometry and are jointly optimized with a preconditioner motivated by the natural gradient. Across three model families, GeoSD retains strong in-distribution gains while improving average mathematical OOD accuracy by – points over the base model. OOD gains hold across five model scales from B to B and transfer to code generation, where GeoSD improves code accuracy by points on average despite distilling on mathematics alone. Our analysis of mathematical reasoning shows that standard matching rapidly concentrates probability mass at high-entropy states and that its samples confidently agree on incorrect answers. In contrast, GeoSD preserves alternative token mass and reduces false consensus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-distillation, large language models, post-training, reasoning, out-of-distribution generalization, information geometry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_q5sROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the distribution drift issue in on-policy self-distillation where the self-teacher is provided with privileged context that leads to supports rarely visited by the student policy itself. Hellinger loss was employed based on geometric mean of student-teacher overlap to reweigh teacher gradients. Fisher–Rao distance is further adopted to penalize the predictive drift across updated model checkpoints. In-domain and OOD domain performance gain were demonstrated comparing to methods using KL style distillation without reweighing.

Critical strengths

The paper explores an important challenge of covariate drift, where the self-teacher provides dramatically deviant distribution that the student, leading to improved in-distribution performance while degrading generalization.

The experiments cover multiple model families and sizes, with component ablations and analyses of predictive drift and probability concentration.

Critical weaknesses
  • Novelty: I don't think the distribution drift issue is new or specific to on-policy self-distillation. Learning on samples from distributions of behavior policy's πβ\pi_\beta that differ from the target policy πθ\pi_\theta is well-studied in offline or offpolicy RL. The paper frames this problem in self-distillation setting, which is fine. But even within the distillation literature, reweighting teacher signals by student probability mass has been actively studied. Authors should compare with these work in experiments. The reweighting scheme (Eq3 and Eq4) also relies on established density distance measurement. What is the real novelty of this work is worth more elaboration.
  • Lacking baselines: Baselines or related work do not include reweighting schemes such as using student probability as dynamic importnace weights https://proceedings.iclr.cc/paper_files/paper/2026/hash/2e53c02ea028cbf603f4b6b47fef3d97-Abstract-Conference.html or its variants https://arxiv.org/pdf/2606.09396 https://arxiv.org/html/2509.23753v1 which are directly relevant to the paper’s core reweighting contribution.
  • Generalization: Code transfer is evaluated only on Qwen3-8B, and no evaluation of general language understanding (e.g., MMLU) or instruction-following quality is reported.
Questions for authors

1 How this methods are comparable to work with reweighting schemes?

2 How much impact this method has to model's general utility, compared with other work?

1 comment

Thank you for the positive assessment and the pointers to the reweighting literature. We address your questions below. We have added new experiments and clarifications to the revised manuscript.

Q1. How this methods are comparable to work with reweighting schemes?

We agree that importance weighting is classical. The works you point to apply it to supervised fine-tuning: DFT scales each token's loss by the model's own probability, Anchored SFT adds a KL anchor to the base model, and PriFT weights tokens by a frozen pretrained model's probability. Within on-policy distillation, reverse KL is a natural analog, since it also weights by student probability, and it degrades OOD accuracy in all three families (Table 1).

Our contribution is not the reweighting itself, but showing why it is crucial in privileged-context OPSD and what it requires there. Empirically, with the same penalty and optimizer, changing only the per-token weighting moves OOD accuracy from 50.6 (forward KL) to 56.1 (JSD) and 60.5 (Hellinger) (Table 2). Combined with the moving anchor, GeoSD improves OOD accuracy over the base model by 5.7--8.6 points across three families.

Reweighting matters especially here because the teacher can be confident simply because it sees a solution the student never sees: at 27% of high-entropy states, the teacher assigns at least 0.8 to a token the student gives at most 0.1 (Figure 8). Because teacher and student also share parameters, reweighting alone is not sufficient: without the proximal penalty, Hellinger matching reaches only 53.8 OOD (Table 2). This motivated our geometric approach, in which the matching loss and the drift penalty both measure differences between predictive distributions relative to each token's probability, so the two terms are compatible.

Q2. How much impact this method has to model's general utility, compared with other work?

Prompted by your remark, we conducted additional experiments on general knowledge and scientific reasoning. We have also extended code-transfer evaluation to all three model families. We added this to the revised manuscript (Appendix C.9, Tables 11 and 12) and we report the results below.

Cross-domain transfer to code holds across all three families: GeoSD achieves the best code accuracy in every family (+1.9, +1.7 and +2.7 over π0\pi_0), while forward and reverse KL degrade code accuracy in every family.

MethodQwen3-8B: OOD codeOlmo-3-7B-Think: OOD codeDS-R1-Llama-8B: OOD code
π0\pi_046.643.541.1
FwdKL42.741.239.4
RevKL44.341.438.7
JSD46.543.041.5
TrOPD47.243.940.7
TIP46.843.938.4
GeoSD48.545.243.8

On two additional benchmarks that cover general knowledge (MMLU-Pro) and scientific reasoning (GPQA-Diamond), GeoSD is the only evaluated method that improves on both datasets. The results in the table below are with Qwen3-8B.

MethodMMLU-ProGPQA-Diamond
π0\pi_066.451.0
FwdKL64.347.7
RevKL65.648.4
JSD66.250.6
TrOPD65.951.4
TIP65.151.7
GeoSD68.253.8
Reviewer_ronMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoSD, a geometric self-distillation method that combines Hellinger divergence, Fisher–Rao regularization, and K-FAC optimization to mitigate predictive drift in privileged-context self-distillation. The method aims to preserve reasoning diversity and improve out-of-distribution generalization while maintaining in-distribution performance.

Critical strengths

S1. The paper presents a coherent framework that integrates Hellinger divergence, Fisher–Rao regularization, and K-FAC optimization from an information-geometric perspective.

S2. The reduction in reasoning diversity during self-distillation has been reported in several prior studies[1, 2] and is an important issue to address for distilling more diverse and complex reasoning capabilities.

[1] Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?, https://arxiv.org/abs/2603.24472

[2] On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity, https://arxiv.org/abs/2606.26091

Critical weaknesses

W1. The goal of self-distillation is to improve the student through teacher supervision. However, the motivation for suppressing teacher–student disagreement and predictive drift is unclear. Tokens assigned high probability by the teacher but low probability by the student can be precisely those the student needs to learn, yet the proposed method reduces their learning signal.

Moreover, the training setup appears particularly favorable to such conservative updates. The privileged context consists only of verified solutions generated by the base model, and problems the base model cannot solve are excluded. Thus, the teacher does not introduce solutions beyond those the base model can already generate. It is unclear whether the method is effective more generally or benefits from a setting where substantial changes to the student's predictions may be unnecessary. Evaluation on problems the base model cannot solve and on larger, more diverse training sets would help clarify this.

W2. The goal of this paper is to reduce the teacher's influence when teacher–student disagreement is large and to mitigate accumulated predictive drift during training. However, these objectives could also be achieved using simpler approaches, such as conventional KL regularization. It is unclear what theoretical motivation or necessity justifies introducing three more complex components (Hellinger loss, Fisher–Rao regularization, K-FAC).

Questions for authors

Q1. The paper argues that a teacher conditioned on a solution trace assigns excessive probability to specific reasoning continuations, reducing the student's reasoning diversity during distillation. However, it is unclear why this effect primarily harms OOD performance rather than ID performance, where the privileged supervision is directly applied during training.

Q2. The overconfidence of a teacher and the resulting reduction in reasoning diversity could also occur in general on-policy distillation with a stronger teacher model. It is therefore unclear why this issue is framed specifically as a problem of privileged-context self-distillation rather than on-policy distillation more broadly.

2 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for the feedback and for raising these points. We address each of them below with clarifications and additional experiments that we have added to the revised manuscript.

W1. Tokens assigned high probability by the teacher but low probability by the student can be precisely those the student needs to learn, yet the proposed method reduces their learning signal.

Hellinger does not remove these signals, it scales them by pq\sqrt{pq}, so the weight is reinforced by the student probability (as soon as assigned). Learning is preserved, which is demonstrated by GeoSD’s in-distribution accuracy (within 1.4 points of FwdKL). GeoSD also increases OOD accuracy monotonically as the teacher gets more of the solution (Figure 3), so it extracts more from stronger privileged information. Notably, it outperforms GRPO as well, which uses no teacher, so the gain is not explained by simply staying near the base model.

Moreover, the training setup appears particularly favorable to such conservative updates. [...] It is unclear whether the method is effective more generally or benefits from a setting where substantial changes to the student's predictions may be unnecessary.

Excluding unsolvable problems follows the standard OPSD protocol used in other work (Zhao et al., 2026; Kim et al., 2026), and we apply it to all baselines in the same way, and FwdKL still degrades OOD accuracy in this setting. More conservative variants of GeoSD perform worse than the default (fixed anchor: 55.6, λ=10\lambda = 10: 58.2; Tables 8 and 10), and lower learning rates for forward KL never exceed the base model (Figure 9). In contrast, GeoSD gains 8.6 points in OOD accuracy over the base model. Nevertheless, we agree that learning beyond the base model is an open question (Appendix E). The current results show that GeoSD delays rather than blocks learning from low-support tokens: the teacher's top choice still becomes the student's most probable token at high-entropy states (Figure 5), and ID accuracy improves by 21.3 points. Moreover, although all privileged solutions come from π0\pi_0 within a budget of 32 samples, GeoSD's OOD pass@32 surpasses π0\pi_0 by 13 points (Figure 6A).

W2. These objectives could also be achieved using simpler approaches, such as conventional KL regularization. It is unclear what theoretical motivation or necessity justifies introducing three more complex components (Hellinger, Fisher–Rao regularization, K-FAC).

Our goal is to control two quantities: how strongly the student follows the teacher where it has little support, and how far its predictions drift during training. Both are differences between predictive distributions, which we measure relative to each token's probability. Suppressing a low-probability alternative is a small change in raw probability, but a significant one relative to that token's mass (Section 2.2). This geometric view motivates all three components, although it does not dictate a unique method, and each specific choice is supported empirically.

Why Hellinger? We hypothesized that attenuating the influence of teacher-preferred tokens with low student support would improve generalization, which motivated Hellinger, whose gradient weights each token by pq\sqrt{pq}. With the same penalty and optimizer, Hellinger outperforms JSD and forward KL (60.5 vs. 56.1 and 50.6 OOD, Table 2).

Why Fisher--Rao? Conventional KL regularization anchors the policy to a fixed reference, and our fixed-anchor variant shows the cost: anchoring at π0\pi_0 restricts useful learning (55.6 OOD / 71.8 ID, Table 8). Because of this, we anchor to a recent checkpoint, and measure the distance with Fisher--Rao, which compares distributions through the same overlap as the Hellinger loss (∑ipiqi\sum_i \sqrt{p_i q_i}, Eqs. 1--2), so the two terms are geometrically compatible. A KL penalty to the same moving checkpoint recovers most of the gain (59.1 vs. 60.5 OOD), as expected since both induce the same local Fisher geometry (App. A.3), and Fisher--Rao adds a further 1.4 points, likely because it is bounded and KL is not, so a few states with large divergence cannot dominate the penalty. No penalty (53.8) or a parameter-space penalty (57.2) are clearly worse (see below or in Appendix C.11 in the revision).

Proximal term (Hellinger + K-FAC)AnchorOOD avg@16ID avg@16
None--53.883.3
L2 parameter distancemoving ckpt57.283.0
Fisher--Raofixed π0\pi_055.671.8
KLmoving ckpt59.181.9
Fisher--Rao (GeoSD)moving ckpt60.582.7

Why K-FAC? The Fisher metric also motivates natural-gradient preconditioning, which scales each update by its effect on the model's predictions rather than by its size in parameter space. K-FAC makes such preconditioning feasible at LLM scale and improves every configuration in Table 2 (56.7 →\to 60.5 OOD for GeoSD).

AuthorsOpenReview ↗

Rebuttal (2/2)

Q1. It is unclear why this effect primarily harms OOD performance rather than ID performance, where the privileged supervision is directly applied during training.

Strong teacher supervision can improve ID accuracy because the reasoning patterns it reinforces remain useful on problems similar to those seen during training, even if it suppresses alternative paths. On shifted problems, the same concentrated choices at uncertain decision points are more often wrong, and exploring alternatives becomes more important, which aggressive matching discourages. This pattern is reported in both references you provide: Kim et al. (2026) find that conditioning the teacher on rich information suppresses uncertainty, enabling rapid in-domain optimization but harming OOD performance, and Nicolicioiu et al. (2026) find that self-distillation concentrates mass on dominant modes, flattening pass@k curves and failing in OOD settings that require diverse strategies. Harne et al. (2026) further show that privileged teachers penalize exploratory tokens. Our analysis in Section 4 gives evidence consistent with this explanation, though we do not claim to have established causality: forward-KL samples agree strongly on wrong answers far more often than the base model (Figure 6B), while GeoSD retains alternatives and gains more from additional sampling (Figure 6A).

Q2. The overconfidence of a teacher and the resulting reduction in reasoning diversity could also occur in general on-policy distillation with a stronger teacher model. It is therefore unclear why this issue is framed specifically as a problem of privileged-context self-distillation rather than on-policy distillation more broadly.

We agree that over-confident supervision is a more general problem: teacher--student mismatch and predictive drift also arise in regular OPD with an external teacher, and both GeoSD terms can be applied there unchanged. However, we believe that GeoSD is particularly relevant to privileged-context OPSD, where the teacher can be overconfident simply because it has access to a solution the student never sees, which makes controlling the teacher's influence especially important. This mismatch comes from missing information rather than limited capacity, and it affects 27% of high-entropy states (Figure 8). Parameter sharing adds a second difficulty, since the teacher's targets move along with the student. In Table 9, we show this feedback effect: without the penalty, a teacher that tracks the current student reaches 53.8 OOD, compared with 56.4 for an EMA teacher. With a frozen base teacher, the closest analog to regular OPD in our experiments, the student still benefits from the proximal term (55.7 →\to 57.9 OOD, Table 9). We have not yet tested this with a separate, stronger teacher; we leave it to future work.


Harne, S., Karkar, C., Pandya, Y., Awadallah, A. and Nambi, A., 2026. Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation.

Reviewer_7MgLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper combines three (geometry-aware) components in loss for on-policy self-distillation (OPSD):

  • Hellinger loss (instead of reverse KL) that reduce influence from the teacher on tokens that are unlikely under the student distribution
  • Fischer-Rao proximal regularisation that keep the model close to a periodically refreshed checkpoint
  • K-FAC preconditioning as an approximation to the natural gradient update

The method shows improved performance on near-OOD (out of distribution) and OOD tasks with a minor sacrifice on in-distribution (ID) tasks. For example, compared to the strongest model on Qwen3-8B, the proposed method GeoSD has −1.4 pp on ID, +5.7 pp on near-OOD (other math tasks), and +1.2 pp on cross-domain OOD (code).

The approach is motivated by the argument that a teacher with privileged information (e.g. verified solution) can be overconfident about tokens that the student cannot justify since the student has no access to the privileged information.

Critical strengths
  • The empirical evaluation is reasonably broad, including across scales, with decent to strong near-OOD results across models.
  • The approach and each of the components are well-motivated and the component-wise ablations are informal.
  • The analysis (probability concentration, false consensus, etc.) is insightful.
Critical weaknesses
  • Most of the gains are on math benchmarks after training on different math problems. I would consider this near-OOD rather than OOD. Coding is bigger shift, but there the gains are more modest. This is fine, even just near-OOD improvement is great to have, but the claims and limitations need to be made appropriately softened.
  • The paper does not really explain why the proposed approach, and in particular the geometric aspects, should specifically improve OOD or near-OOD generalisation. Based on my reading, one reason could be that less "aggressive" distilation and reduced drift somehow preserve alternative reasoning paths, which maybe is helpful on unfamiliar problems. However, this connection is mostly based on what I could infer, it is not clearly spelled out in the paper, and there are no direct experiments that support it. See also Q1 and Q2.
  • Some closely related baselines seem to be missing: (i) Trust Region Policy Distillation (TOP-D; Xie et al., 2026) addresses instability caused by teacher–student mismatch using proximal teacher updates, which seems related; (ii) Entropy-Aware OPD (EOPD, Jin et al., 2026) tries to prevent probability concentration and preserve diversity which would match the proposed explanation above for improved generalization; (iii) Trust-Region Behavior Blending (TRB, Plyusov et al., 2026) controls policy updates through a KL trust region similar to the Fisher-Rao checkpoint. I do acknowledge that TrOPD has similar ideas and is already included.
  • Related to the above, something like a moving-checkpoint KL regularizer would be a useful comparison to the proposed Fisher–Rao term, so we can see if the gains come from the geometry or simply from controlling drift. There are a few possible variants (forward or revese KL, symmetric KL, etc.). Note this is different from L2 and has similar local geometry to Fisher–Rao.
  • More generally, related to the points above, it's not clear if GeoSD is "just" trading a little bit of ID performance for better generalization simply because it has stronger regularisation.
  • To me it is not entirely clear which parts of the method and motivation are specific to self-distillation vs. regular distillation. For example, teacher–student mismatch and predictive drift can also occur with a separate teacher. I acknowledge the experiments in C.7 comparing different teacher configurations, including a fixed teacher, but these do not directly address my comment.
Questions for authors
  1. Can you explain why GeoSD benefits OOD performance so much more and provide direct experiments that support whatever mechanism explains these gains?
  2. Could Hellinger/Fisher–Rao geometry simply act as a good regularizer that prevents overfitting (which also preserves pretrained capabilities), rather than offering something specific to OOD generalization? Is there an experiment you could run to distinguish the different potential explanations?
  3. Can you compare to TOP-D, EOPD, TRB (see weakness)?
  4. Can you compare with a moving-checkpoint KL regularizer?
  5. Can you clarify what is specific to self-distillation, and if GeoSD can provide similar benefits for regular OPD?
2 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for the positive assessment and the concrete suggestions. We address your points below with references to clarifications and new experiments in the revised manuscript.

Most of the gains are on math benchmarks after training on different math problems. I would consider this near-OOD rather than OOD. [...] the claims and limitations need to be made appropriately softened.

We agree with this characterization, which matches our statement in Section 3.1 that the mathematics benchmarks "mainly test shifts in problem source and difficulty rather than domain". We made this terminology consistent in the revised manuscript, referring to the mathematics results as within-domain (cross-benchmark) OOD generalization to distinguish them from cross-domain transfer to code (lines 250--251). To complement these results, we have extended the code evaluation to all three families and added general knowledge and scientific reasoning benchmarks as well. We report the results in tables below and in Appendix C.9 (Tables 11 and 12) of the revised manuscript.

Cross-domain transfer to code holds across all three families: GeoSD achieves the best code accuracy in every family (+1.9, +1.7 and +2.7 over π0\pi_0), while forward and reverse KL degrade code accuracy in every family.

Qwen3-8B: OOD codeOlmo-3-7B-Think: OOD codeDS-R1-Llama-8B: OOD code
π0\pi_046.643.541.1
FwdKL42.741.239.4
RevKL44.341.438.7
JSD46.543.041.5
TrOPD47.243.940.7
TIP46.843.938.4
GeoSD48.545.243.8

We evaluated Qwen3-8B on two general benchmarks, where GeoSD improves on both MMLU-Pro (+1.8) and GPQA-Diamond (+2.8) and outperforms all evaluated baselines.

MMLU-ProGPQA-Diamond
π0\pi_066.451.0
FwdKL64.347.7
RevKL65.648.4
JSD66.250.6
TrOPD65.951.4
TIP65.151.7
GeoSD68.253.8

Q1. Can you explain why GeoSD benefits OOD performance so much more and provide direct experiments that support whatever mechanism explains these gains?

Our intuition is that a privileged teacher can be overconfident in continuations that are justified only by the solution trace. Matching it closely makes the student commit to these continuations at uncertain decision points and drains probability from alternatives. On new problems, these commitments are more often wrong and fewer alternatives remain available. We believe that controlling the teacher--student agreement and predictive drift is important for OOD generalization in privileged-context OPSD. Concretely, we regulate how strongly the student follows the teacher where it has little support, and how far its predictions drift during training. We choose a geometric approach because both are differences between predictive distributions: we measure them relative to each token's probability, not in parameter space, so that suppressing a low-probability alternative counts as a significant change (Section 2.2). The geometric view does not dictate a unique method. Our specific choices reflect additional hypotheses (reducing teacher influence where student support is low motivates Hellinger) or practical constraints (K-FAC makes preconditioning motivated by the natural gradient feasible at scale).

The empirical evidence supports this intuition. Section 4 shows the association between retained alternatives and OOD performance: unlike forward KL, GeoSD retains alternative token mass (Figure 5), benefits more from additional sampling, and avoids confident agreement on wrong answers (Figure 6).

Prompted by your remark, we also ran a new branching experiment, where we found that a lower-ranked alternative recovers a majority-correct answer on 22% of OOD problems where the rank-1 branch fails under GeoSD, compared to 9% under forward KL and 12% for π0\pi_0. As stated in the revised manuscript at the end of Section 4 (lines 471--475) and in Appendix E, we present this as an empirically supported hypothesis, rather than a causal relationship.

Q2. Could Hellinger/Fisher–Rao geometry simply act as a good regularizer that prevents overfitting (which also preserves pretrained capabilities), rather than offering something specific to OOD generalization?

If GeoSD's benefit came only from preserving pretrained ability, its OOD accuracy could at best match π0\pi_0 (51.9). Instead, GeoSD improves over π0\pi_0 by 8.6 points. Stronger regularization alone does not reproduce this: for forward KL, lower learning rates, early stopping and teacher softening never exceed π0\pi_0 (Figure 9, Table 8), and more conservative variants of GeoSD fall below the default (fixed anchor: 55.6, λ=10\lambda = 10: 58.2; Tables 8 and 10).

AuthorsOpenReview ↗

Rebuttal (2/2)

Q3. Can you compare to TOP-D, EOPD, TRB?

We evaluated EOPD, TOP-D and TRB on Qwen3-8B. All three remain well below GeoSD on OOD accuracy. EOPD also tests the diversity explanation, since it is designed to preserve diversity by a different route (entropy-aware mixing of KL directions), yet its OOD accuracy stays at the base model's level (−0.5). Adding our Fisher--Rao penalty and K-FAC improves EOPD to 55.7, but this is still 4.8 points below our method. We report the results below and in Table 13 in the revised manuscript (Appendix C.10).

MethodOOD avg@16ID avg@16ΔOOD vs. base
π0\pi_051.961.4--
Forward KL43.884.1−8.1
EOPD51.481.9−0.5
EOPD + FR + K-FAC55.782.0+3.8
TOP-D54.982.3+3.0
TRB48.783.9−3.2
GeoSD60.582.7+8.6

Q4. Something like a moving-checkpoint KL regularizer would be a useful comparison to the proposed Fisher--Rao term, so we can see if the gains come from the geometry or simply from controlling drift. ... Can you compare with a moving-checkpoint KL regularizer?

The results indicate that the improvement comes from controlling drift in the right geometry, not from controlling drift alone. A penalty on parameter distance to the same moving checkpoint also limits drift, but reaches only 57.2 OOD, while penalties measured in predictive space perform best: 59.1 for KL and 60.5 for Fisher--Rao. KL comes close, which is expected since both distances induce the same local Fisher geometry (App. A.3), and Fisher--Rao adds a further 1.4 points, likely because it is bounded and KL is not, so a few states with large divergence cannot dominate the penalty (Table 14, Appendix C.11). Removing the penalty (53.8) or fixing the anchor at π0\pi_0 (55.6) is clearly worse (Tables 2 and 8), and the geometry of the matching loss matters too: with the same penalty and optimizer, Hellinger outperforms forward KL (60.5 vs. 50.6, Table 2). ID accuracy is preserved with either moving predictive anchor, with a slight edge for Fisher--Rao (82.7 for FR vs. 81.9 for KL), whereas fixing the anchor at π0\pi_0 reduces it to 71.8.

Q5. Can you clarify what is specific to self-distillation, and if GeoSD can provide similar benefits for regular OPD?

Teacher--student mismatch and predictive drift also arise in regular OPD, and both GeoSD terms can be applied there unchanged. However, GeoSD is particularly relevant to privileged-context OPSD, where the teacher can be overconfident simply because it sees a solution the student will never see, which makes controlling the teacher's influence especially important. The mismatch between the teacher and student predictions comes from missing information rather than limited capacity, and it affects 27% of high-entropy states (Figure 8). Parameter sharing adds a second difficulty, since the teacher’s targets move along with the student. In Table 9, we show this feedback effect: without the penalty, a teacher that tracks the current student reaches 53.8 OOD, compared with 56.4 for an EMA teacher. With a frozen base teacher, the closest analog to regular OPD in our experiments, the student still benefits from the proximal term (55.7 →\to 57.9 OOD, Table 9). Testing GeoSD with separate, stronger teachers is an interesting direction for future work.

Reviewer_C8e6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GeoSD, a geometric self-distillation method designed to improve reasoning generalization in privileged-context on-policy self-distillation. The method combines three components: a Hellinger divergence loss that weights teacher preferences according to student–teacher probability overlap, a squared Fisher–Rao proximal penalty measured against a periodically refreshed student checkpoint, and a K-FAC preconditioner motivated by natural-gradient optimization.The central motivation is that privileged-context teachers may assign high probability to continuations that the student cannot confidently infer from its own context. Standard distillation can exert substantial pressure toward these continuations, potentially leading to excessive predictive concentration and degraded out-of-distribution (OOD) performance. GeoSD aims to attenuate such local teacher influence while regulating the accumulation of predictive changes over training.Then paper evaluates GeoSD on three model families and five Qwen3 model sizes, using competition-mathematics benchmarks as the primary evaluation setting. It reports improvements of 5.7–8.6 percentage points in average mathematical OOD accuracy over the base models, while retaining strong in-distribution performance. It also reports an average improvement of 1.9 percentage points on code-generation benchmarks for Qwen3-8B after training exclusively on mathematics. Ablation studies examine the contributions of the loss, proximal penalty, and optimizer, while Section 4 analyzes token-level probability concentration and answer-level false consensus as potential explanations for the observed gains.

Critical strengths

1. Consistent empirical improvements across mathematical benchmarks and model families.

GeoSD improves average mathematical OOD accuracy across three model families, with reported gains of 5.7–8.6 percentage points over the base models. The improvements are not confined to a single benchmark, and the scale experiments further show positive gains across five Qwen3 model sizes. The main comparisons also include multiple training and evaluation runs and statistical testing. This is the paper's strongest contribution: despite my reservations about the interpretation of the results, the empirical performance is sufficiently consistent to warrant serious consideration.

2. A useful component ablation study and a relevant problem formulation.

The ablation grid in Table 2 provides evidence that the full method's performance cannot be attributed to Hellinger matching alone. For example, removing the proximal penalty or replacing K-FAC with AdamW reduces OOD accuracy, and the full combination outperforms several matched alternatives. The paper also addresses a practically relevant problem: how to exploit privileged teacher information without degrading the student's ability to solve related but different problems. These experiments provide a useful starting point for understanding the trade-offs in privileged-context self-distillation, even if they do not fully establish the proposed theoretical interpretation.

Critical weaknesses

1. The methodological novelty and the claimed unifying geometric principle are insufficiently established.

Hellinger divergence, Fisher–Rao distances, proximal regularization, and K-FAC are all established tools. Their combination is not inherently unoriginal, and I acknowledge that the ablations provide evidence of empirical complementarity. My concern is more specific: the paper presents the components as a coherent geometric solution, but the mathematical development does not establish why this particular combination is principled or uniquely appropriate for the problem.

The Hellinger gradient does down-weight teacher-preferred tokens to which the student assigns little probability. However, this property alone does not imply that the overall parameter gradient is smaller or more stable, nor does it distinguish useful alternative continuations from incorrect ones. Likewise, the Fisher–Rao proximal term penalizes predictive displacement from a recent checkpoint, but this does not establish that the penalized displacement is harmful or that controlling it improves generalization.

2. The OOD and mechanistic claims extend beyond what the current evidence establishes.

The mathematical evaluation primarily measures transfer across competition-mathematics benchmarks, which may differ in source, difficulty, and problem distribution but do not establish broad distributional generalization. The code-generation results are encouraging, but are limited to Qwen3-8B and show a smaller gain than the mathematical results. The paper provides no theoretical generalization result connecting the proposed objective to OOD risk.

More importantly, Section 4 does not close the explanatory gap. Figure 5 shows that GeoSD retains more probability mass on lower-ranked teacher tokens at selected high-entropy states, while Figure 6 reports lower false-consensus rates and different relationships between consensus strength and answer correctness. These are useful observations, but they do not establish that the retained alternatives are useful reasoning paths, that local token concentration causes false consensus, or that reducing false consensus mediates the OOD gains. The main token-level comparison also contrasts Forward KL with the full GeoSD method, making it difficult to isolate the contribution of individual components.

Questions for authors

1.methodological novelty and component selection. I understand the motivation for Hellinger matching: it attenuates the direct influence of teacher-preferred tokens when the student assigns them little probability. I am less clear about why the Fisher–Rao proximal term and K-FAC should be viewed as part of the same principled solution, rather than as two additional techniques that happen to improve the empirical results.

The ablations establish that the components are complementary in the tested configurations, but do not by themselves explain why this particular combination follows from the proposed geometric perspective. Could the authors clarify which parts of the method are derived from a common principle and which are empirical design choices? In particular, what justifies pairing Hellinger for local matching with Fisher–Rao for the proximal penalty, given that several divergences induce the same local Fisher geometry?

2. what does the Fisher–Rao penalty actually control? I am still not fully convinced by the claim that the Fisher–Rao term addresses the accumulation of small updates. Since the checkpoint is periodically refreshed, the penalty constrains the student relative to a moving local anchor, not relative to the initial model. Small displacements across successive windows could therefore accumulate into substantial long-term drift.

Could the authors clarify the precise claim they intend to make about drift control? Does the method aim to limit local predictive changes, cumulative displacement from the initial policy, or specifically harmful changes caused by privileged supervision? I would also appreciate clarification on how the local natural-gradient argument in Appendix A.6 translates to the practical damped K-FAC update, given that the current and checkpoint Fisher matrices need not be similar.

3.The scope of the OOD claim. I appreciate the breadth of the mathematical benchmark evaluation, and the code-transfer experiment is a useful addition. However, I am uncertain whether the current evidence supports the broader framing of reasoning generalization. The mathematics benchmarks all belong to competition mathematics, while cross-domain transfer is evaluated only on Qwen3-8B.

Could the authors more precisely characterize what distribution shifts the current experiments establish robustness to, and distinguish cross-benchmark transfer from broader OOD generalization? I am not asking for a complete generalization theorem; I would like to understand the scope of the central claim and which parts are directly supported by the current experiments.

4. The causal interpretation of Section 4. Section 4 is potentially the most interesting part of the paper, but I find the interpretation stronger than the evidence currently warrants. Figure 5 shows that GeoSD retains more mass on lower-ranked teacher tokens, but teacher rank is not a measure of the correctness or usefulness of a continuation. Figure 6 then shows differences in false consensus and consensus calibration, but does not establish that the token-level behavior causes these answer-level differences or mediates the OOD gains.

How do the authors distinguish the proposed mechanism from the alternative explanation that GeoSD simply produces a better-calibrated or otherwise different answer distribution? Also, why should retaining lower-ranked teacher tokens be interpreted as preserving useful reasoning alternatives rather than merely retaining probability mass on less-preferred continuations? I would appreciate a more explicit account of what the current figures establish and what remains a hypothesis.

Overall, if the authors can convincingly resolve these concerns about the method's underlying principle, the scope of its claims, and the interpretation of Section 4, I would be open to reconsidering my rating.

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for the thorough review and the thoughtful questions. We address your questions below, with clarifications and new experiments that we have added to the revised manuscript.

Q1. Could the authors clarify which parts of the method are derived from a common principle and which are empirical design choices?

Why Hellinger? Both loss terms of our objective (Hellinger matching and Fisher--Rao proximal penalty) measure changes of next-token distributions relative to each token’s probability mass. As you note, all ff-divergences induce the Fisher metric locally (Appendix A.3), so this does not single out Hellinger over other ff-divergences. Our hypothesis was that attenuating the influence of teacher-preferred tokens with low student support would improve generalization in OPSD. This motivated our choice of Hellinger, whose gradient weights each token by pq\sqrt{pq}. The empirical evidence supports this: cases where the teacher assigns at least 0.8 (q∗≥0.8q^* \geq 0.8), and the student gives at most 0.1 (p∗≤0.1p^* \leq 0.1) covers 27% of high-entropy states (Figure 8). At such low-overlap states, the gradient norm grows under forward KL but stays nearly flat under Hellinger, attenuating the pressure toward continuations that the student barely supports (Figure 2A). On top of this, the ablations showed that Hellinger outperforms JSD and KL under the same penalty and optimizer (60.5 vs. 56.1 and 50.6 OOD, Table 2).

Why Fisher--Rao? For the proximal term, our goal was to anchor the student to its own recent predictions instead of the base model, with distance measured in predictive space. We measure this distance with Fisher--Rao, which shares the geometry of the Hellinger loss. Both compare distributions through the same overlap (∑ipiqi\sum_i \sqrt{p_i q_i}, Equations 1 and 2), so the two terms of the objective are geometrically compatible. This is supported with empirical results: with Hellinger matching and K-FAC fixed, the moving Fisher--Rao anchor (60.5 OOD) outperforms no penalty (53.8), a parameter-space penalty (57.2) and a fixed anchor at π0\pi_0 (55.6) (Tables 2 and 8). Replacing Fisher--Rao with KL to the same moving checkpoint keeps most of this gain (59.1; see new results in Table 14, Appendix C.11), which is expected since the two distances agree locally (App. A.3). Fisher--Rao still adds 1.4 points on top of this, likely because it is bounded while KL is not, so a few states with large divergence cannot dominate the penalty. Most of the benefit therefore comes from the moving anchor itself, with Fisher--Rao as the more robust choice of distance.

Why K-FAC? The same geometry also applies to the parameter update. The Fisher metric motivates natural-gradient preconditioning, which scales each update by its effect on the model's predictions rather than by its size in parameter space. K-FAC is a practical, curvature-aware preconditioner motivated by this principle, since the full Fisher is infeasible at LLM scale. Empirically, it improves every loss and penalty configuration in Table 2 (e.g., 56.7 →\to 60.5 OOD for GeoSD) at 1.10x wall-clock cost.

Q2. Could the authors clarify the precise claim they intend to make about drift control?

We agree that the moving anchor does not bound displacement from π0\pi_0 (Appendix E). Our claim is narrower, and we clarify this in the main text of the revised manuscript (lines 196--200). The motivation is similar to trust-region methods in RL such as TRPO and PPO: rather than anchoring the student to its initialization, we regulate predictive changes relative to a recent checkpoint. This discourages large changes within each refresh window while still allowing the student to keep adapting over training. The penalty therefore controls the rate of predictive drift, not its cumulative amount. With the penalty, cumulative drift stabilizes during training (Figure 2B), but this is an empirical observation rather than a guarantee, and we do not claim that the penalty targets harmful changes specifically.

Appendix A.6 only motivates the update form and the algorithm does not rely on Fk≈FckptF_k \approx F_{\text{ckpt}}. Finally, the gain is not explained by only being more conservative: stronger regularization of GeoSD lowers OOD accuracy (fixed anchor: 55.6, λ=10\lambda = 10: 58.2, vs. 60.5; Tables 8 and 10), and for forward KL, lower learning rates, early stopping and teacher softening never cross above π0\pi_0's OOD accuracy of 51.9 (Figure 9, Table 8).

AuthorsOpenReview ↗

Rebuttal (2/2)

Q3. Could the authors more precisely characterize what distribution shifts the current experiments establish robustness to?

Section 3.1 characterizes the OOD scope: the mathematics benchmarks "mainly test shifts in problem source and difficulty rather than domain", and only code generation is treated as cross-domain transfer (Section 3.1, Table 6). For consistency, we refer to the mathematics results as within-domain (cross-benchmark) OOD generalization in the revised manuscript (lines 250--251). We chose these benchmarks because prior work reports that privileged-context OPSD degrades performance on them (Kim et al., 2026), so they directly test whether that degradation can be avoided.

To test how far the gains extend beyond competition mathematics, we extended the code evaluation, previously limited to Qwen3-8B, to all three model families, and added benchmarks for general knowledge (MMLU-Pro) and scientific reasoning (GPQA-Diamond). We report the results in tables below and in Appendix C.9 (Tables 11 and 12) of the revised manuscript.

Cross-domain transfer to code holds across all three families: GeoSD achieves the best code accuracy in every family (+1.9, +1.7 and +2.7 over π0\pi_0), while FwdKL and RevKL degrade code accuracy in every family.

Qwen3-8B: OOD codeOlmo-3-7B-Think: OOD codeDS-R1-Llama-8B: OOD code
π0\pi_046.643.541.1
FwdKL42.741.239.4
RevKL44.341.438.7
JSD46.543.041.5
TrOPD47.243.940.7
TIP46.843.938.4
GeoSD48.545.243.8

We also evaluated Qwen3-8B on two general benchmarks, where GeoSD improves on both, namely +1.8 on MMLU-Pro and +2.8 on GPQA-Diamond, outperforming all evaluated baselines.

MMLU-ProGPQA-Diamond
π0\pi_066.451.0
FwdKL64.347.7
RevKL65.648.4
JSD66.250.6
TrOPD65.951.4
TIP65.151.7
GeoSD68.253.8

Q4. I would appreciate a more explicit account of what the current figures establish and what remains a hypothesis.

We agree that the analyses establish associations rather than a causal mechanism, which we state explicitly in the revised manuscript (lines 471--475): forward KL concentrates mass on the teacher's top token much faster, while GeoSD shows less false consensus and better consensus calibration. We use these as diagnostics supporting the hypothesis that retained alternatives contribute to the OOD improvement. We also support this with a new analysis. When branching rollouts at high-entropy positions, a lower-ranked alternative recovers a majority-correct answer on 22% of OOD problems where the rank-1 branch fails under GeoSD, versus 9% under forward KL and 12% for π0\pi_0.

Thank you for the detailed rebuttal and the additional experiments. I appreciate that the authors have directly addressed the main concerns rather than simply reiterating the original claims.

I find the clarification regarding the Fisher–Rao proximal penalty convincing. The revised interpretation as local predictive-drift regulation, rather than a guarantee on cumulative drift, appropriately aligns the claim with the method. I also appreciate the more precise characterization of OOD generalization and, particularly, the additional cross-domain experiments across three model families. These results meaningfully strengthen the empirical contribution.

I still have two reservations. First, the unified geometric interpretation remains primarily a design motivation rather than a principled derivation. The new KL proximal comparison is useful, but the relatively small difference (60.5 vs. 59.1) suggests that much of the benefit may come from the moving anchor itself. Second, the new alternative-branch analysis provides encouraging evidence, but it remains correlational rather than establishing the proposed causal mechanism. I would also appreciate uncertainty estimates for the newly reported cross-domain improvements.

Nevertheless, the authors have addressed several of my original concerns, narrowed their claims appropriately, and provided meaningful new empirical evidence. On balance, I am inclined to update my rating from 2 to 3 (Weak Accept), although I still view the methodological novelty and mechanistic explanation as limitations.

Comments on the paper

1 comment

We thank the reviewers for their feedback. Below we summarize clarifications and new experiments that we have added to the revised manuscript.

Revisions

1. OOD scope (C8e6, 7MgL, q5sR)

  • Clarification: Mathematics results are described as within-domain (cross-benchmark) OOD generalization, and code as cross-domain transfer (Section 1, Section 3.1).
  • New experiments: Code transfer is extended from Qwen3-8B to all three families. GeoSD is best in every family (+1.9 / +1.7 / +2.7 over the base model), while forward and reverse KL degrade code accuracy in every family. On MMLU-Pro and GPQA-Diamond (Qwen3-8B), GeoSD is the only method that improves on both, +1.8 and +2.8, respectively (Appendix C.9, Tables 11 and 12).

2. Geometry vs. regularization (7MgL, ronM, C8e6)

  • New experiment: KL penalty to the same moving checkpoint (Appendix C.11, Table 14). Penalties in the predictive space perform best (KL 59.1, Fisher--Rao 60.5 OOD), ahead of a parameter-space penalty (57.2), no penalty (53.8), and a fixed anchor at the base model (55.6 OOD / 71.8 ID).
  • Clarification: Simply applying stronger regularization does not explain the improvements. More conservative GeoSD variants perform worse (fixed anchor: 55.6; λ=10\lambda = 10: 58.2), and forward KL never surpasses the base model with lower learning rates, early stopping, or teacher softening (Tables 8, 10; Figure 9).

3. Additional baselines (7MgL, q5sR)

  • New experiments: EOPD, TOP-D, and TRB on Qwen3-8B (Appendix C.10, Table 13). All remain below GeoSD on OOD (51.4 / 54.9 / 48.7 vs. 60.5). EOPD with our Fisher--Rao penalty and K-FAC reaches 55.7.
  • Clarification: With the penalty and optimizer fixed, changing only the per-token weighting moves OOD accuracy from 50.6 (forward KL) to 56.1 (JSD) and 60.5 (Hellinger) (Table 2). Reverse KL, which also weights by student probability, degrades OOD accuracy in all three families.

4. Drift control and design principle (C8e6, ronM)

  • Clarification: Our proximal term regulates the rate of predictive drift relative to a recent checkpoint, similar to trust-region methods, rather than cumulative displacement from the base model. We have made this explicit in Section 2.3. Figure 2B shows that cumulative drift also stabilizes empirically, though this is not guaranteed.
  • In our responses, we distinguish what follows from the shared geometry from choices motivated by specific hypotheses (Hellinger's pq\sqrt{pq} weighting) or practical constraints (K-FAC at scale), each supported by matched ablations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.