acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Distillation Guided by Evolution of Internal Predictions

Abstract

On-policy distillation lets a student generate the trajectories on which it receives dense token-level supervision, adapting the training contexts to its own policy and outperforming off-policy post-training methods on reasoning tasks. This student-centric view motivates us to investigate another source of model-specific information: how student predictions evolve across depth. Inspired by prior work on internal policy changes in large language models and depth-contrast decoding, we propose Internal Guidance Policy Distillation (IGPD), which combines a frozen teacher with the detached change from a calibrated shallow readout to the final policy. At each sampled prefix, the resulting logit offset modifies the teacher anchor to form an adaptive target. Training jointly aligns the shallow readout with the teacher and distills the final policy toward this target. Because the offset is derived from the current model, it evolves throughout training without requiring a separate reference-policy forward pass. Across two strong-to-weak settings and competition-mathematics benchmarks, IGPD consistently improves over matched OPD and achieves the highest macro-average among the evaluated methods. We hope these findings encourage further exploration of internal prediction dynamics in on-policy distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, layer-wise predictions, strong-to-weak learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AN23OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Internal Guidance Policy Distillation (IGPD), an on-policy distillation method that modifies the teacher’s logits using the difference between the student’s final and calibrated shallow predictions. During training, an auxiliary loss aligns the shallow prediction with the teacher, while the final prediction is trained toward the modified target. The internal contrast shares the student’s backbone forward pass, avoiding the separate reference-model forward pass used by ExOPD. The auxiliary readout is removed at inference.

Experiments cover two teacher–student configurations within the Qwen3 family, using mathematical training data and both competition-math and additional transfer evaluations. IGPD consistently improves over matched OPD and obtains higher mathematical macro-averages than ExOPD. Ablations examine guidance strength, shallow alignment weight, and exit depth.

Critical strengths

The paper shows that the contrast between a calibrated shallow prediction and the final student prediction can guide on-policy distillation effectively. Although the construction is closely related to existing target-extrapolation approaches, obtaining this signal from within the current student is an interesting design choice. It avoids maintaining and evaluating a separate frozen reference model, offering a practical advantage over ExOPD. The auxiliary components are removed after training, leaving ordinary student inference unchanged.

IGPD improves over matched OPD on every reported main competition-math benchmark, with macro-average gains of 6.48 and 4.04 percentage points across the two configurations. It also exceeds ExOPD’s mathematical macro-average in both settings. The additional evaluations and ablations provide useful support for the method. In particular, the 4B student improves accuracy while generating shorter responses, showing that the benefits are not uniformly explained by increased response length.

Critical weaknesses

The contribution is primarily empirical, with limited understanding of why the internal contrast helps. The target construction is a relatively simple extension of existing contrastive and extrapolative approaches. The experiments establish its usefulness, but do not fully explain when the shallow-to-final difference represents a beneficial correction rather than a change in confidence or style. The coefficient and exit-depth ablations are informative, but the interaction between shallow teacher alignment and final-target modification remains insufficiently understood.

The practical efficiency and generality claims need stronger evidence. Avoiding a separate reference-model forward pass is attractive, but IGPD introduces offline calibration and auxiliary training computation. The paper does not report a direct end-to-end timing comparison with ExOPD. Moreover, the 1.7B accuracy gains coincide with substantially longer responses, making matched-token-budget comparisons relevant. Both training configurations use Qwen3 models and mathematical data; experiments with another model family or training domain would strengthen the broader applicability claim.

Questions for authors

Efficiency clarification. What are the measured training time and peak memory usage of IGPD relative to ExOPD, both excluding and including calibration and calibration-data generation? A matched-token comparison would help isolate savings from the internal-reference construction.

Mechanism clarification. How does the shallow prediction’s agreement with the teacher evolve during training, and how does this relate to the strength of the final-layer learning signal? With guidance strength set to one, exact shallow–teacher agreement makes the guided target equal to the current final prediction. Does training approach this regime, and does it help explain the observed gains?

Compute-controlled evaluation. Does the 1.7B model retain its advantage over OPD and ExOPD under matched inference-token budgets? Its longer responses make this relevant to interpreting the accuracy improvement.

5 comments
AuthorsOpenReview ↗

Response to Reviewer AN23 (1/3) — Efficiency and Preparation Cost

Thank you for your careful review and constructive questions. We respond in three parts.

Q1 & W2. Measured computation and preparation cost

(a) Non-rollout training time

We use identical cached sequences, batch size 128, and four A100 GPUs, with matched gradient checkpointing and actor parameter/optimizer offload. Times include recurring teacher/reference transfers, scoring, student probability recomputation, backward computation, optimizer updates, and weight synchronization. Generation, initialization, evaluation, and checkpoint saving are excluded. The scoring pool uses four teacher replicas for OPD/IGPD and two replicas per scoring model for ExOPD.

Here and below, 1.7B denotes Qwen3-1.7B non-thinking and 4B denotes Qwen3-4B-Instruct-2507.

Method1.7B, 5,034 tokens1.7B, 8,192 tokens4B, 8,192 tokens
OPD120.15 ± 0.48184.37 ± 0.24443.50 ± 2.54
ExOPD, top-K adaptation133.57 ± 0.65203.81 ± 1.43466.13 ± 0.92
IGPD, SwiGLU125.79 ± 0.73191.43 ± 0.97445.00 ± 0.79
IGPD, linear125.60 ± 0.83190.69 ± 0.31449.76 ± 2.62

Values are seconds, mean ± sample standard deviation. The 5,034-token measurements use two warmups and five measured updates. At 8,192 tokens, we use one warmup and two measured updates, except 4B OPD/ExOPD with three measured updates.

(b) Readout preparation

The lightweight variant uses a residual linear readout, 1,280 student-generated responses, and cached hidden states.

StudentReadoutParametersResponsesFitting updatesPreparation, minutesAIME mean
1.7BSwiGLU18.87M12,80030086.0238.18
1.7BLinear4.19M1,28020015.8838.65
4BSwiGLU23.59M12,800300328.8060.26
4BLinear6.55M1,28020050.2760.31

Preparation uses four GPUs. SwiGLU times sum generation critical-path and fitting durations, excluding startup and file operations. Linear times include generation, hidden-state extraction, fitting, and process overhead. The 1.7B estimate subtracts the logged final 100 fitting updates from a 300-update run. The 4B time is measured end to end.

This recipe reduces preparation time by 81.5% and 84.7%, with AIME24/25 mean Avg@32 differences of +0.47 and +0.05 points. Preparation is a one-time cost reusable across runs with the same readout.

(c) Cumulative online training time

StudentMethodTo best evaluated checkpoint, minutesTo ExOPD's best score or best checkpoint, minutes
1.7BExOPD139.99139.99
1.7BIGPD (SwiGLU)122.18122.18
1.7BIGPD (linear)159.50122.07
4BExOPD169.34169.34
4BIGPD (SwiGLU)227.83227.83
4BIGPD (linear)151.78151.78

These historical totals include rollout and all logged online training phases at their actual response lengths. They exclude preparation, evaluation, saving, initialization, idle time, and hyperparameter search. The last column uses the first evaluated checkpoint reaching ExOPD's best AIME mean, or the method's own best checkpoint if unreached.

For 1.7B, SwiGLU IGPD reaches its best evaluated checkpoint with 12.7% less online training time than ExOPD and a higher AIME mean. For 4B, linear IGPD matches ExOPD's mean in 151.78 versus 169.34 minutes.

The practical benefit is an evolving internal reference that improves supervision without maintaining a separate reference model. The linear variant improves on ExOPD's AIME mean by 3.96 points for 1.7B and matches its 4B mean with 22.0% shorter responses. The lightweight readout reduces preparation cost while retaining comparable accuracy.


AuthorsOpenReview ↗

Response to Reviewer AN23 (2/3) — Shallow Alignment and the Learning Signal

Q2 & W1. Interaction between the shallow policy and the guided target

“How does the shallow prediction’s agreement with the teacher evolve during training, and how does this relate to the strength of the final-layer learning signal? With guidance strength set to one, exact shallow–teacher agreement makes the guided target equal to the current final prediction.”

Let pp, ss, and tt denote the final student, calibrated shallow student, and teacher distributions on the same support. The additive target can be written as

\qquad q_j=\frac{t_j(p_j/s_j)^\alpha}{\sum_v t_v(p_v/s_v)^\alpha}.$$ The ratio $p_j/s_j$ measures a token's relative probability change across depth. It increases or decreases that token's weight in the teacher-anchored target. We study this adjustment empirically, motivated by the distinction between a preliminary shallow prediction and a final prediction produced by additional computation. The reviewer is correct that, at $\alpha=1$, exact equality $s=t$ gives $q=p$, and both losses vanish even if $p\ne t$. With the target detached, the direct final-logit gradient at target construction is $$\frac{\partial D_{\mathrm{KL}}(p\Vert q)}{\partial z_{f,j}} =p_j\left[\log\frac{s_j}{t_j}-\sum_v p_v\log\frac{s_v}{t_v}\right].$$ Thus token-dependent shallow–teacher disagreement, weighted and centered under the final policy, determines this direct update. Exact distributional matching would require the shallow branch to reproduce the teacher's entire distribution on the support, which is much stronger than predicting the same top token. The following table summarizes the raw training curves for both students. Each entry averages five logged updates on the on-policy batches. Let $t$, $s$, $p$, and $q$ denote the teacher, shallow, final, and guided-target distributions on the shared top-$K$ support. | Student | Training updates | Shallow FKL $D_{\mathrm{KL}}(t\Vert s)$ | Final RKL $D_{\mathrm{KL}}(p\Vert q)$ | | --- | --- | ---: | ---: | | Qwen3-1.7B, non-thinking | 1–5 | 1.701 | 0.597 | | Qwen3-1.7B, non-thinking | 21–25 | 1.792 | 0.360 | | Qwen3-1.7B, non-thinking | 46–50 | 2.918 | 0.219 | | Qwen3-4B-Instruct-2507 | 1–5 | 1.792 | 0.391 | | Qwen3-4B-Instruct-2507 | 21–25 | 2.094 | 0.257 | | Qwen3-4B-Instruct-2507 | 46–50 | 2.546 | 0.178 | The two loss trajectories clearly separate later in training. Shallow–teacher FKL increases while final–guided-target RKL decreases, and shallow–teacher disagreement remains substantial in both settings. The shallow policy therefore maintains a distinct internal reference. The columns track each branch's alignment with its respective target. We additionally separate shallow supervision and target modification. The following results use AIME Avg@32 in the 1.7B non-thinking setting. | Configuration | AIME24 | AIME25 | Mean | | --- | ---: | ---: | ---: | | OPD | 37.92 | 24.27 | 31.09 | | IGPD with raw logit lens (no calibrated readout) | 36.88 | 28.44 | 32.66 | | Shallow alignment only ($\alpha=0,\lambda=1$) | 37.19 | 24.69 | 30.94 | | Depth offset only ($\alpha=1,\lambda=0$) | 45.21 | 31.15 | 38.18 | | Full IGPD ($\alpha=1,\lambda=1$) | 46.46 | 29.90 | 38.18 | | Stronger shallow alignment ($\alpha=1,\lambda=2$) | 47.29 | 30.94 | 39.11 | Auxiliary supervision alone gives a mean close to OPD. The offset alone improves it by 7.08 points, and stronger shallow alignment adds a further 0.93 points. Removing readout calibration lowers the mean by 5.52 points relative to full IGPD. Moving the exit from layer 6 to layers 14 and 22 lowers it by 2.81 and 6.56 points. These component and exit-depth comparisons provide empirical support for the calibrated internal offset and its interaction with shallow supervision. ---
AuthorsOpenReview ↗

Response to Reviewer AN23 (3/3) — Accuracy under Matched Generation Limits

Q3 & W2. Compute-controlled inference

“Does the 1.7B model retain its advantage over OPD and ExOPD under matched inference-token budgets?”

We regenerated responses under four common maximum response lengths, keeping each method's checkpoint fixed across budgets. All methods use identical questions, temperature 1, and top-pp 1 in non-thinking mode. AIME24 and AIME25 each contain 30 problems with 32 responses per problem. AMC23 contains 40 problems and HMMT-Feb25 contains 30, each with 16 responses per problem. The table reports the equally weighted macro-average across these four benchmarks.

Maximum response tokensOPDExOPDIGPD
4,09624.2324.8225.33
8,19232.6032.6033.44
16,38436.9739.0041.11
32,76837.3239.6243.20

IGPD retains a higher observed mean at every tested limit. Its advantage is small at 4K and 8K and grows at larger limits. At 16K, the gains over OPD and ExOPD are 4.14 and 2.11 points, increasing to 5.89 and 3.58 points at 32K. These are newly generated evaluations, so their scores differ from the original main-table results.

This comparison fixes the maximum tokens available per response, with actual response lengths varying across methods. For Qwen3-4B-Instruct-2507, IGPD improves the mathematical macro-average over OPD by 4.04 points while reducing average response length by 5.5%. The accuracy–length relationship therefore differs across the two students.

Our main comparisons cover two Qwen3 students trained on mathematics, with additional transfer evaluations demonstrating gains beyond the training domain.

Reviewer_LJx7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Internal Guidance Policy Distillation (IGPD), an extension of on-policy distillation (OPD) that incorporates the evolution of the student's internal predictions into the distillation target. Specifically, the method introduces a calibrated shallow readout and constructs a teacher-anchored target by adding a detached logit difference between the student's final and shallow predictions. The training objective combines a forward-KL loss that aligns the shallow policy with the teacher and a reverse-KL loss that aligns the final policy with the guided target.

The method is evaluated on two teacher-student configurations using Qwen3 models, primarily on competition mathematics benchmarks, with additional out-of-distribution evaluations. IGPD improves the macro-average accuracy over matched OPD by 6.48 and 4.04 percentage points in the two settings, respectively. The paper also provides ablations on guidance strength, shallow alignment weight, and intermediate exit depth.

Critical strengths
  1. A practical and interesting extension of on-policy distillation.

The paper explores an interesting direction: using the student's own intermediate predictions to construct adaptive distillation targets instead of simply imitating the teacher distribution. The method is relatively straightforward to implement and does not require an additional full-model reference forward pass during online training. Furthermore, the auxiliary readout is removed after training, leaving inference-time model architecture unchanged. This makes the approach practically attractive.

  1. Consistent empirical improvements over matched OPD.

The reported improvements are encouraging, particularly the 6.48-point and 4.04-point gains in mathematical reasoning macro-average accuracy across two different model configurations. The inclusion of ExOPD as a stronger baseline, additional out-of-distribution benchmarks, and coefficient and exit-depth ablations strengthens the empirical evaluation. The results suggest that intermediate-layer information can be useful for improving on-policy distillation.

Critical weaknesses
  1. The mechanism underlying the proposed guidance is not sufficiently isolated.

The main methodological concern is whether the improvements genuinely arise from exploiting the evolution of internal predictions or primarily from the auxiliary shallow-policy supervision and the resulting modification of the optimization gradient.

In particular, when α=1\alpha = 1, the guided target satisfies

qG(v)∝exp⁡(zT(v)+z+(v)−z−(v)).q_G(v) \propto \exp(z_T(v) + z_+(v) - z_-(v)).

Since p+(v)∝exp⁡(z+(v))p_+(v) \propto \exp(z_+(v)), the log ratio appearing in the final-policy reverse-KL gradient simplifies, up to a token-independent constant, to z−(v)−zT(v)z_-(v) - z_T(v). Thus, with the guided target detached, the direct final-policy update is determined by the teacher–shallow disagreement, weighted and centered under the final policy.

  1. The empirical evidence lacks uncertainty estimates and leaves important efficiency questions unresolved.

The experiments report single-run results without variability across training seeds or confidence intervals. Since some of the principal evaluations rely on relatively small competition-mathematics benchmarks, it is difficult to determine the robustness of the reported improvements, particularly the smaller differences over ExOPD.

Additionally, in the Qwen3-1.7B setting, IGPD increases average response length from 7.65k tokens under OPD to 12.93k tokens. This raises the question of how much of the accuracy improvement remains under a matched inference-token budget. The opposite length trend for the 4B student is encouraging, but does not resolve the first setting's efficiency concern.

Questions for authors
  1. Clarification of the optimization mechanism.

For α = 1, the direct final-policy reverse-KL gradient can be expressed in terms of the teacher–shallow logit difference, centered under the final student distribution. Could the authors discuss this equivalence and clarify how their interpretation in terms of internal prediction evolution differs from teacher–shallow gradient shaping? In particular, what evidence supports the necessity of using a depth-derived auxiliary policy, beyond the auxiliary supervision and gradient modification themselves?

  1. Statistical reliability.

Could the authors provide results across multiple training seeds, ideally with mean and standard deviation for the main macro-average metrics? This would help assess whether the improvements, especially the smaller margins over ExOPD, are statistically robust.

  1. Accuracy versus computational cost.

For Qwen3-1.7B, IGPD substantially increases average response length. How do the methods compare under a fixed inference-token budget? It would also be useful to report total training overhead relative to OPD, including shallow-readout calibration, preparation of its calibration corpus, and the additional computations during online training.

3 comments
AuthorsOpenReview ↗

Response to Reviewer LJx7 (1/3) — Optimization and Component Evidence

Thank you for your careful review and constructive questions. We appreciate your positive assessment of the method's practicality and empirical improvements. Our responses are organized into three parts.

Q1 & W1. Optimization equivalence and the role of the depth-derived policy

“Could the authors discuss this equivalence and clarify how their interpretation in terms of internal prediction evolution differs from teacher–shallow gradient shaping?”

(a) Optimization equivalence

We agree with the reviewer's interpretation and thank them for offering this interesting perspective on the optimization mechanism. The two formulations are equivalent at α=1\alpha=1. Our motivation is to maintain an evolving internal negative reference by continually aligning the shallow policy with the teacher as the student changes. Contrasting this preliminary prediction with the final prediction gives the depth-derived correction used in our target construction. This motivates our teacher-anchored formulation

q=softmax⁡(sg⁡[zT+α(zf−zs)]),q=\operatorname{softmax}\left(\operatorname{sg}[z_T+\alpha(z_f-z_s)]\right),

Here pp, ss, and tt denote the final student, calibrated shallow student, and teacher distributions on the shared support. At α=1\alpha=1, the same target can be written with the current student as its center

\qquad q_j\propto p_j\frac{t_j}{s_j}.$$ Under this interpretation, teacher–shallow disagreement supplies the correction to the current student. With the target detached, the direct final-logit gradient at target construction is $$\frac{\partial D_{\mathrm{KL}}(p\Vert q)}{\partial z_{f,j}} =p_j\left(r_j-\sum_v p_vr_v\right), \qquad r=z_s-z_T.$$ The shallow policy makes a more preliminary prediction using fewer layers of computation. Even with continual teacher alignment, it may therefore retain informative disagreements with the teacher as training progresses. In our experiments, the shallow policy remains substantially different from the teacher, as the loss trajectories below show. From the reviewer's perspective, these persistent disagreements may help prevent the final-policy learning signal from diminishing prematurely, allowing further improvement during training. ### (b) Component evidence > “In particular, what evidence supports the necessity of using a depth-derived auxiliary policy, beyond the auxiliary supervision and gradient modification themselves?” We compare the offset and shallow-supervision components separately. All scores below are AIME Avg@32 percentages for the Qwen3-1.7B non-thinking setting. | Configuration | AIME24 | AIME25 | Mean | | --- | ---: | ---: | ---: | | OPD | 37.92 | 24.27 | 31.09 | | IGPD with raw logit lens (no calibrated readout) | 36.88 | 28.44 | 32.66 | | Shallow alignment only ($\alpha=0,\lambda=1$) | 37.19 | 24.69 | 30.94 | | Depth offset only ($\alpha=1,\lambda=0$) | 45.21 | 31.15 | 38.18 | | Full IGPD ($\alpha=1,\lambda=1$) | 46.46 | 29.90 | 38.18 | | Stronger shallow alignment ($\alpha=1,\lambda=2$) | 47.29 | 30.94 | 39.11 | Auxiliary supervision alone gives a mean close to OPD, while the offset alone improves it by 7.08 points. Stronger shallow alignment adds a further 0.93 points. Thus auxiliary supervision alone does not account for the observed gain. The layer-6 exit also outperforms layers 14 and 22 by 2.81 and 6.56 points in mean accuracy. Replacing the calibrated SwiGLU readout with a raw logit lens lowers the mean by 5.52 points. Together, the exit-depth and component comparisons provide empirical support for using a calibrated internal policy to construct the offset. ---
AuthorsOpenReview ↗

Response to Reviewer LJx7 (2/3) — Statistical Reliability

Q2 & W2. Uncertainty in the reported improvements

“Could the authors provide results across multiple training seeds, ideally with mean and standard deviation for the main macro-average metrics?”

Due to limited compute resources, we have not yet completed experiments across multiple training seeds. The current training results are from single runs, and we will make every effort to supplement them with multi-seed results as resources permit. To quantify evaluation uncertainty, we performed a paired problem bootstrap on the new response-budget evaluations described below. We used 20,000 resamples, resampling problems within each benchmark and retaining all responses belonging to a problem together. This preserves the dependence among repeated responses to the same question.

The following intervals concern differences in the macro-average across AIME24, AIME25, AMC23, and HMMT-Feb25 for Qwen3-1.7B in non-thinking mode. Part 3 gives the corresponding accuracy results and evaluation protocol.

Maximum response tokensComparisonDifference in percentage points95% bootstrap interval
16,384IGPD − OPD+4.14[1.97, 6.32]
16,384IGPD − ExOPD+2.11[0.14, 4.13]
32,768IGPD − OPD+5.89[3.61, 8.32]
32,768IGPD − ExOPD+3.58[1.72, 5.49]

These intervals support the improvements on this evaluation set at the two larger budgets. They quantify problem-sampling uncertainty conditional on the trained checkpoints and generated responses. At the shorter 4K and 8K budgets, the intervals include zero for both baseline comparisons.


AuthorsOpenReview ↗

Response to Reviewer LJx7 (3/3) — Token Budget and Training Cost

Q3 & W2. Accuracy versus computational cost

(a) Matched maximum response lengths

“How do the methods compare under a fixed inference-token budget?”

We regenerated responses at four common maximum lengths, keeping each method's checkpoint fixed across limits. All methods use the same questions, temperature 1, and top-pp 1. Scores are macro-averages over AIME24/25 Avg@32 and AMC23/HMMT-Feb25 Avg@16.

Maximum response tokensOPDExOPDIGPD
4,09624.2324.8225.33
8,19232.6032.6033.44
16,38436.9739.0041.11
32,76837.3239.6243.20

IGPD retains higher accuracy at every tested limit. At 16K, it improves over OPD and ExOPD by 4.14 and 2.11 points. These limits cap each response, while actual lengths vary. Newly generated responses account for differences from the main-table scores.

(b) Non-rollout training time

“It would also be useful to report total training overhead relative to OPD, including shallow-readout calibration, preparation of its calibration corpus, and the additional computations during online training.”

We use identical cached sequences, batch size 128, and four A100 GPUs, with matched gradient checkpointing and actor parameter/optimizer offload. Times include recurring teacher/reference transfers, scoring, student probability recomputation, backward computation, optimizer updates, and weight synchronization. Generation, initialization, evaluation, and checkpoint saving are excluded. The scoring pool uses four teacher replicas for OPD/IGPD and two replicas per scoring model for ExOPD.

Here and below, 1.7B denotes Qwen3-1.7B non-thinking and 4B denotes Qwen3-4B-Instruct-2507.

Method1.7B, 5,034 tokens1.7B, 8,192 tokens4B, 8,192 tokens
OPD120.15 ± 0.48184.37 ± 0.24443.50 ± 2.54
ExOPD, top-K adaptation133.57 ± 0.65203.81 ± 1.43466.13 ± 0.92
IGPD, SwiGLU125.79 ± 0.73191.43 ± 0.97445.00 ± 0.79
IGPD, linear125.60 ± 0.83190.69 ± 0.31449.76 ± 2.62

Values are seconds, mean ± sample standard deviation. The 5,034-token measurements use two warmups and five measured updates. At 8,192 tokens, we use one warmup and two measured updates, except 4B OPD/ExOPD with three measured updates.

(c) Readout preparation

The lightweight variant uses a residual linear readout, 1,280 student-generated responses, and cached hidden states.

StudentReadoutParametersResponsesFitting updatesPreparation, minutesAIME mean
1.7BSwiGLU18.87M12,80030086.0238.18
1.7BLinear4.19M1,28020015.8838.65
4BSwiGLU23.59M12,800300328.8060.26
4BLinear6.55M1,28020050.2760.31

Preparation uses four GPUs. SwiGLU times sum generation critical-path and fitting durations, excluding startup and file operations. Linear times include generation, hidden-state extraction, fitting, and process overhead. The 1.7B estimate subtracts the logged final 100 fitting updates from a 300-update run. The 4B time is measured end to end.

This recipe reduces preparation time by 81.5% and 84.7%, with AIME24/25 mean Avg@32 differences of +0.47 and +0.05 points. Preparation is a one-time cost reusable across runs with the same readout.

(d) Cumulative online training time

StudentMethodTo best evaluated checkpoint, minutesTo ExOPD's best score or best checkpoint, minutes
1.7BExOPD139.99139.99
1.7BIGPD (SwiGLU)122.18122.18
1.7BIGPD (linear)159.50122.07
4BExOPD169.34169.34
4BIGPD (SwiGLU)227.83227.83
4BIGPD (linear)151.78151.78

These historical totals include rollout and all logged online training phases at their actual response lengths. They exclude preparation, evaluation, saving, initialization, idle time, and hyperparameter search. The last column uses the first evaluated checkpoint reaching ExOPD's best AIME mean, or the method's own best checkpoint if unreached.

For 1.7B, SwiGLU IGPD reaches its best evaluated checkpoint with 12.7% less online training time than ExOPD and a higher AIME mean. For 4B, linear IGPD matches ExOPD's mean in 151.78 versus 169.34 minutes.

The practical benefit is an evolving internal reference that improves supervision without maintaining a separate reference model. The linear variant improves on ExOPD's AIME mean by 3.96 points for 1.7B and matches its 4B mean with 22.0% shorter responses. The lightweight readout reduces preparation cost while retaining comparable accuracy.

Reviewer_3mJbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IGPD, which uses internal guidance in addition to teacher guidance during OPD training. This internal guidance uses the detached change from a calibrated shallow policy to the final policy. IGPD improves accuracy over OPD across two teacher-student configurations.

Critical strengths
  • Borrowing insights from contrasting model outputs at two different layers like DoLa, is novel in the OPD setting, although not well justified.
  • On average, the proposed IGPD improves performance.
Critical weaknesses

I think an important part of the paper is how the authors justify their proposed method theoretically, intuitively, or empirically. But in general, it feels like it just applies an existing idea and shows that it works.

  • First of all, it is hard to understand the motivation. Inference-time contrast works and makes sense because such contrasts could reveal signals of uncertainty or differences in predictions across layers. But that does not mean that adding this contrast to teacher guidance will improve OPD training. There is no clear connection here. There is also little intuition behind the design choices for each component of the method.
  • Also, the authors apply a forward KL loss to the shallow policy. But there is no clear intuition or justification for why the method needs this term. In fact, when the shallow policy perfectly matches the teacher distribution, the guided target becomes identical to the final student policy (with the default guidance coefficient). In this case, the reverse KL loss becomes zero even if the final student does not match the teacher. This raises the question of whether aligning the shallow policy with the teacher could weaken the training signal for the final policy, and how these two losses are supposed to work together.
  • The evaluation is focused mostly on math and Qwen models. There are also only a few baselines. More relevant baselines, such as OPD2\text{OPD}^2 [1] and TrOPD [2], are missing.
  • There are no error bars or standard deviations in the tables, even though AIME and AMC benchmarks can be very noisy.
  • The analysis is relatively weak. It is hard to understand why IGPD works and through what mechanism. Although there is a coefficient ablation for the shallow forward KL loss, more component-level ablations are needed, such as removing calibration or separating the effects of the shallow loss and depth offset. Also, in the 1.7B setting, the response length increases significantly, and it is unclear how much this contributes to the performance improvement.

References

[1] Heo et al. “On-Policy Delta Distillation.”

[2] Xing et al. “Trust Region On-Policy Distillation.”

Questions for authors
  • Why does applying contrast to the training signal in addition to teacher guidance work?
  • Why do the authors apply forward KL for the shallow policy, and what happens if it is removed?
  • What happens without calibration of the shallow policy?
  • Why are more recent OPD baselines not included in the comparison?
9 comments
AuthorsOpenReview ↗

Response to Reviewer 3mJb (1/3) — Motivation and Target Construction

Thank you for your careful review and constructive questions. We address the four questions below with additional component ablations and baseline comparisons. Our responses are organized into three parts.

Q1. Motivation for applying depth contrast during training

“Why does applying contrast to the training signal in addition to teacher guidance work?”

“Inference-time contrast works and makes sense because such contrasts could reveal signals of uncertainty or differences in predictions across layers. But that does not mean that adding this contrast to teacher guidance will improve OPD training.”

Our motivation follows the depth-contrast intuition of DoLa. The shallow policy provides a preliminary prediction, while the final policy reflects additional computation. Their relative change provides a student-specific direction for adjusting teacher supervision. We study this idea empirically by incorporating the contrast into the training target.

Let pp, ss, and tt denote the final student, calibrated shallow student, and teacher distributions on the same support. Writing their logits as zfz_f, zsz_s, and zTz_T, the additive target is

q=softmax⁡(sg⁡[zT+α(zf−zs)]).q=\operatorname{softmax}\left(\operatorname{sg}[z_T+\alpha(z_f-z_s)]\right).

Since zf,j=log⁡pj+log⁡Zfz_{f,j}=\log p_j+\log Z_f, and similarly for ss and tt, the log-normalizers contribute only a token-independent constant CC. Consequently,

=\log t_j+\alpha(\log p_j-\log s_j)+C.$$ Softmax cancels $C$, giving $$q_j\propto\exp\left(\log t_j+\alpha(\log p_j-\log s_j)\right) =t_j(p_j/s_j)^\alpha.$$ Detachment affects gradients without changing these target probabilities. The ratio $p_j/s_j$ captures the relative change in token $j$'s probability across depth. Tokens with larger ratios are promoted relative to those with smaller ratios. After normalization, these token-dependent factors redistribute probability around the teacher's supervision. This provides a student-specific adjustment based on how its own predictions evolve. The resulting supervision updates the model during training, and standard decoding is used at inference. The evidence for this choice is empirical. In the Qwen3-1.7B non-thinking setting, the layer-6 exit improves AIME24/25 mean Avg@32 over layers 14 and 22 by 2.82 and 6.57 percentage points. The component ablations below further show that the offset improves accuracy even without the auxiliary shallow loss, whereas shallow supervision alone gives a mean close to OPD. ---
AuthorsOpenReview ↗

Response to Reviewer 3mJb (2/3) — Shallow Alignment and Calibration

Q2. Role of shallow forward KL

“Why do the authors apply forward KL for the shallow policy, and what happens if it is removed?”

(a) Design rationale

We introduce shallow alignment to keep the intermediate prediction connected to teacher supervision as the shared student backbone changes. The shallow branch learns from the teacher distribution tt, and the final branch learns from the teacher-anchored target qq. This maintains an evolving internal reference after the initial readout calibration. The two losses update the shared backbone through their respective branches.

We choose forward KL to provide teacher-weighted supervision to this preliminary policy.

Lshallow=DKL(t∥s)=∑jtjlog⁡tjsj.\mathcal{L}_{\mathrm{shallow}} = D_{\mathrm{KL}}(t \Vert s) = \sum_{j} t_{j}\log\frac{t_{j}}{s_{j}}.

It weights tokens by teacher probability, providing supervision for teacher-supported tokens even when their shallow probabilities are small. Reverse KL weights its centered log-ratio gradient by the shallow policy, which can weaken learning for tokens the shallow policy underestimates.

(b) Component ablations

The following table compares the components in the Qwen3-1.7B non-thinking setting. All scores are AIME Avg@32 percentages.

ConfigurationAIME24AIME25Mean
OPD37.9224.2731.09
IGPD with raw logit lens (no calibrated readout)36.8828.4432.66
Shallow alignment only (α=0,λ=1\alpha=0,\lambda=1)37.1924.6930.94
Depth offset only (α=1,λ=0\alpha=1,\lambda=0)45.2131.1538.18
Full IGPD (α=1,λ=1\alpha=1,\lambda=1)46.4629.9038.18
Stronger shallow alignment (α=1,λ=2\alpha=1,\lambda=2)47.2930.9439.11

Without shallow alignment, the depth offset still improves over OPD by 7.08 points. Increasing the alignment weight to λ=2\lambda=2 yields a further 0.93-point improvement over the default, supporting the potential benefit of stronger shallow supervision.

(c) Shallow–teacher matching and the training signal

“This raises the question of whether aligning the shallow policy with the teacher could weaken the training signal for the final policy, and how these two losses are supposed to work together.”

At α=1\alpha=1, exact equality s=ts=t implies q=pq=p. This would require the shallow policy to reproduce the teacher's entire distribution on the shared support. In our experiments, the shallow policy remains substantially different from the teacher throughout training, as shown by the recorded loss curves below.

The following table summarizes the raw training curves for both students. Each entry averages five logged updates on the changing on-policy batches, using the distributions defined above on the shared top-KK support.

StudentTraining updatesShallow FKL DKL(t∥s)D_{\mathrm{KL}}(t\Vert s)Final RKL DKL(p∥q)D_{\mathrm{KL}}(p\Vert q)
Qwen3-1.7B, non-thinking1–51.7010.597
Qwen3-1.7B, non-thinking21–251.7920.360
Qwen3-1.7B, non-thinking46–502.9180.219
Qwen3-4B-Instruct-25071–51.7920.391
Qwen3-4B-Instruct-250721–252.0940.257
Qwen3-4B-Instruct-250746–502.5460.178

The two loss trajectories clearly separate later in training. Shallow–teacher FKL increases while final–guided-target RKL decreases in both settings. Thus, the shallow policy continues to provide a distinct internal reference instead of converging to exact teacher matching in these runs.

Q3. Calibration of the shallow policy

“What happens without calibration of the shallow policy?”

Performance decreases when the calibrated readout is replaced by a raw logit lens. The final LM head is trained on output-layer representations, so intermediate hidden states can be distribution-shifted inputs to it. A calibrated SwiGLU or linear readout helps map these states into a representation space compatible with the shared head.

Replacing the calibrated SwiGLU readout with a raw logit lens lowers the AIME24/25 mean by 5.52 points.

A simpler calibrated readout also works well. With a residual linear map trained on 1,280 student-generated responses for 200 fitting updates, the AIME24/25 mean is within 0.5 points of the original SwiGLU readout in both student settings, supporting a lightweight calibration approach.


AuthorsOpenReview ↗

Response to Reviewer 3mJb (3/3) — Additional Baselines and Response Length

Q4. Comparison with recent OPD methods

“Why are more recent OPD baselines not included in the comparison?”

The original experiments were constrained by available resources. We have now added OPD² [1] and TOP-D [2] under our training and evaluation setup. OPD² requires the teacher's base checkpoint, which is not always available. Nevertheless, we obtained the required Qwen3-4B base checkpoint and included OPD² in the comparison. IGPD maintains an evolving reference within the current student and does not require this additional model.

The following comparison uses the Qwen3-1.7B non-thinking student and global token-mean reduction shared with our training setup. OPD² and TOP-D retain their sampled-token objectives. Scores are AIME Avg@32 percentages.

MethodAIME24AIME25Mean
OPD37.9224.2731.09
OPD² [1] (global token mean)43.5432.4037.97
TOP-D [2]37.7127.4032.55
IGPD (SwiGLU)46.4629.9038.18
IGPD (linear)44.7932.5038.65

IGPD with the SwiGLU readout exceeds OPD² and TOP-D in mean accuracy by 0.21 and 5.63 points, respectively. The lightweight linear variant improves the mean by another 0.47 points.

Due to limited compute resources, the corresponding OPD² and TOP-D evaluations for Qwen3-4B-Instruct-2507 are still in progress, and evaluation of the added baselines on the remaining benchmarks is not yet complete. We will supplement these results as the evaluations are completed.

Additional concern. Accuracy under a fixed response-length limit

“Also, in the 1.7B setting, the response length increases significantly, and it is unclear how much this contributes to the performance improvement.”

We evaluated the Qwen3-1.7B non-thinking models under four common maximum response lengths. Each method uses a fixed checkpoint across all four limits, with the same questions, temperature 1, and top-pp 1. We generated 32 responses per problem on AIME24 and AIME25 and 16 on AMC23 and HMMT-Feb25. The table reports the equally weighted macro-average of Avg@K across these four benchmarks, in percent.

Maximum response tokensOPDExOPDIGPD
4,09624.2324.8225.33
8,19232.6032.6033.44
16,38436.9739.0041.11
32,76837.3239.6243.20

IGPD achieves higher accuracy at every tested response-length limit. At 16,384 tokens, it improves over OPD and ExOPD by 4.14 and 2.11 points, respectively. Its advantage grows as more generation tokens become available. These evaluations use newly generated responses, so the scores differ from the original main-table results.

References

[1] Byeongho Heo, Jaehui Hwang, Sangdoo Yun, and Dongyoon Han. On-Policy Delta Distillation. 2026. https://arxiv.org/abs/2607.15161

[2] Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, and Mao Yang. Trust Region Policy Distillation. 2026. https://arxiv.org/abs/2607.04751

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.