- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Internal Guidance Policy Distillation (IGPD), an on-policy distillation method that modifies the teacher’s logits using the difference between the student’s final and calibrated shallow predictions. During training, an auxiliary loss aligns the shallow prediction with the teacher, while the final prediction is trained toward the modified target. The internal contrast shares the student’s backbone forward pass, avoiding the separate reference-model forward pass used by ExOPD. The auxiliary readout is removed at inference.
Experiments cover two teacher–student configurations within the Qwen3 family, using mathematical training data and both competition-math and additional transfer evaluations. IGPD consistently improves over matched OPD and obtains higher mathematical macro-averages than ExOPD. Ablations examine guidance strength, shallow alignment weight, and exit depth.
Critical strengths
The paper shows that the contrast between a calibrated shallow prediction and the final student prediction can guide on-policy distillation effectively. Although the construction is closely related to existing target-extrapolation approaches, obtaining this signal from within the current student is an interesting design choice. It avoids maintaining and evaluating a separate frozen reference model, offering a practical advantage over ExOPD. The auxiliary components are removed after training, leaving ordinary student inference unchanged.
IGPD improves over matched OPD on every reported main competition-math benchmark, with macro-average gains of 6.48 and 4.04 percentage points across the two configurations. It also exceeds ExOPD’s mathematical macro-average in both settings. The additional evaluations and ablations provide useful support for the method. In particular, the 4B student improves accuracy while generating shorter responses, showing that the benefits are not uniformly explained by increased response length.
Critical weaknesses
The contribution is primarily empirical, with limited understanding of why the internal contrast helps. The target construction is a relatively simple extension of existing contrastive and extrapolative approaches. The experiments establish its usefulness, but do not fully explain when the shallow-to-final difference represents a beneficial correction rather than a change in confidence or style. The coefficient and exit-depth ablations are informative, but the interaction between shallow teacher alignment and final-target modification remains insufficiently understood.
The practical efficiency and generality claims need stronger evidence. Avoiding a separate reference-model forward pass is attractive, but IGPD introduces offline calibration and auxiliary training computation. The paper does not report a direct end-to-end timing comparison with ExOPD. Moreover, the 1.7B accuracy gains coincide with substantially longer responses, making matched-token-budget comparisons relevant. Both training configurations use Qwen3 models and mathematical data; experiments with another model family or training domain would strengthen the broader applicability claim.
Questions for authors
Efficiency clarification. What are the measured training time and peak memory usage of IGPD relative to ExOPD, both excluding and including calibration and calibration-data generation? A matched-token comparison would help isolate savings from the internal-reference construction.
Mechanism clarification. How does the shallow prediction’s agreement with the teacher evolve during training, and how does this relate to the strength of the final-layer learning signal? With guidance strength set to one, exact shallow–teacher agreement makes the guided target equal to the current final prediction. Does training approach this regime, and does it help explain the observed gains?
Compute-controlled evaluation. Does the 1.7B model retain its advantage over OPD and ExOPD under matched inference-token budgets? Its longer responses make this relevant to interpreting the accuracy improvement.
5 comments
Response to Reviewer AN23 (2/3) — Shallow Alignment and the Learning Signal
Q2 & W1. Interaction between the shallow policy and the guided target
“How does the shallow prediction’s agreement with the teacher evolve during training, and how does this relate to the strength of the final-layer learning signal? With guidance strength set to one, exact shallow–teacher agreement makes the guided target equal to the current final prediction.”
Let , , and denote the final student, calibrated shallow student, and teacher distributions on the same support. The additive target can be written as
\qquad q_j=\frac{t_j(p_j/s_j)^\alpha}{\sum_v t_v(p_v/s_v)^\alpha}.$$ The ratio $p_j/s_j$ measures a token's relative probability change across depth. It increases or decreases that token's weight in the teacher-anchored target. We study this adjustment empirically, motivated by the distinction between a preliminary shallow prediction and a final prediction produced by additional computation. The reviewer is correct that, at $\alpha=1$, exact equality $s=t$ gives $q=p$, and both losses vanish even if $p\ne t$. With the target detached, the direct final-logit gradient at target construction is $$\frac{\partial D_{\mathrm{KL}}(p\Vert q)}{\partial z_{f,j}} =p_j\left[\log\frac{s_j}{t_j}-\sum_v p_v\log\frac{s_v}{t_v}\right].$$ Thus token-dependent shallow–teacher disagreement, weighted and centered under the final policy, determines this direct update. Exact distributional matching would require the shallow branch to reproduce the teacher's entire distribution on the support, which is much stronger than predicting the same top token. The following table summarizes the raw training curves for both students. Each entry averages five logged updates on the on-policy batches. Let $t$, $s$, $p$, and $q$ denote the teacher, shallow, final, and guided-target distributions on the shared top-$K$ support. | Student | Training updates | Shallow FKL $D_{\mathrm{KL}}(t\Vert s)$ | Final RKL $D_{\mathrm{KL}}(p\Vert q)$ | | --- | --- | ---: | ---: | | Qwen3-1.7B, non-thinking | 1–5 | 1.701 | 0.597 | | Qwen3-1.7B, non-thinking | 21–25 | 1.792 | 0.360 | | Qwen3-1.7B, non-thinking | 46–50 | 2.918 | 0.219 | | Qwen3-4B-Instruct-2507 | 1–5 | 1.792 | 0.391 | | Qwen3-4B-Instruct-2507 | 21–25 | 2.094 | 0.257 | | Qwen3-4B-Instruct-2507 | 46–50 | 2.546 | 0.178 | The two loss trajectories clearly separate later in training. Shallow–teacher FKL increases while final–guided-target RKL decreases, and shallow–teacher disagreement remains substantial in both settings. The shallow policy therefore maintains a distinct internal reference. The columns track each branch's alignment with its respective target. We additionally separate shallow supervision and target modification. The following results use AIME Avg@32 in the 1.7B non-thinking setting. | Configuration | AIME24 | AIME25 | Mean | | --- | ---: | ---: | ---: | | OPD | 37.92 | 24.27 | 31.09 | | IGPD with raw logit lens (no calibrated readout) | 36.88 | 28.44 | 32.66 | | Shallow alignment only ($\alpha=0,\lambda=1$) | 37.19 | 24.69 | 30.94 | | Depth offset only ($\alpha=1,\lambda=0$) | 45.21 | 31.15 | 38.18 | | Full IGPD ($\alpha=1,\lambda=1$) | 46.46 | 29.90 | 38.18 | | Stronger shallow alignment ($\alpha=1,\lambda=2$) | 47.29 | 30.94 | 39.11 | Auxiliary supervision alone gives a mean close to OPD. The offset alone improves it by 7.08 points, and stronger shallow alignment adds a further 0.93 points. Removing readout calibration lowers the mean by 5.52 points relative to full IGPD. Moving the exit from layer 6 to layers 14 and 22 lowers it by 2.81 and 6.56 points. These component and exit-depth comparisons provide empirical support for the calibrated internal offset and its interaction with shallow supervision. ---Response to Reviewer AN23 (3/3) — Accuracy under Matched Generation Limits
Q3 & W2. Compute-controlled inference
“Does the 1.7B model retain its advantage over OPD and ExOPD under matched inference-token budgets?”
We regenerated responses under four common maximum response lengths, keeping each method's checkpoint fixed across budgets. All methods use identical questions, temperature 1, and top- 1 in non-thinking mode. AIME24 and AIME25 each contain 30 problems with 32 responses per problem. AMC23 contains 40 problems and HMMT-Feb25 contains 30, each with 16 responses per problem. The table reports the equally weighted macro-average across these four benchmarks.
| Maximum response tokens | OPD | ExOPD | IGPD |
|---|---|---|---|
| 4,096 | 24.23 | 24.82 | 25.33 |
| 8,192 | 32.60 | 32.60 | 33.44 |
| 16,384 | 36.97 | 39.00 | 41.11 |
| 32,768 | 37.32 | 39.62 | 43.20 |
IGPD retains a higher observed mean at every tested limit. Its advantage is small at 4K and 8K and grows at larger limits. At 16K, the gains over OPD and ExOPD are 4.14 and 2.11 points, increasing to 5.89 and 3.58 points at 32K. These are newly generated evaluations, so their scores differ from the original main-table results.
This comparison fixes the maximum tokens available per response, with actual response lengths varying across methods. For Qwen3-4B-Instruct-2507, IGPD improves the mathematical macro-average over OPD by 4.04 points while reducing average response length by 5.5%. The accuracy–length relationship therefore differs across the two students.
Our main comparisons cover two Qwen3 students trained on mathematics, with additional transfer evaluations demonstrating gains beyond the training domain.
Response to Reviewer AN23 (1/3) — Efficiency and Preparation Cost
Thank you for your careful review and constructive questions. We respond in three parts.
Q1 & W2. Measured computation and preparation cost
(a) Non-rollout training time
We use identical cached sequences, batch size 128, and four A100 GPUs, with matched gradient checkpointing and actor parameter/optimizer offload. Times include recurring teacher/reference transfers, scoring, student probability recomputation, backward computation, optimizer updates, and weight synchronization. Generation, initialization, evaluation, and checkpoint saving are excluded. The scoring pool uses four teacher replicas for OPD/IGPD and two replicas per scoring model for ExOPD.
Here and below, 1.7B denotes Qwen3-1.7B non-thinking and 4B denotes Qwen3-4B-Instruct-2507.
Values are seconds, mean ± sample standard deviation. The 5,034-token measurements use two warmups and five measured updates. At 8,192 tokens, we use one warmup and two measured updates, except 4B OPD/ExOPD with three measured updates.
(b) Readout preparation
The lightweight variant uses a residual linear readout, 1,280 student-generated responses, and cached hidden states.
Preparation uses four GPUs. SwiGLU times sum generation critical-path and fitting durations, excluding startup and file operations. Linear times include generation, hidden-state extraction, fitting, and process overhead. The 1.7B estimate subtracts the logged final 100 fitting updates from a 300-update run. The 4B time is measured end to end.
This recipe reduces preparation time by 81.5% and 84.7%, with AIME24/25 mean Avg@32 differences of +0.47 and +0.05 points. Preparation is a one-time cost reusable across runs with the same readout.
(c) Cumulative online training time
These historical totals include rollout and all logged online training phases at their actual response lengths. They exclude preparation, evaluation, saving, initialization, idle time, and hyperparameter search. The last column uses the first evaluated checkpoint reaching ExOPD's best AIME mean, or the method's own best checkpoint if unreached.
For 1.7B, SwiGLU IGPD reaches its best evaluated checkpoint with 12.7% less online training time than ExOPD and a higher AIME mean. For 4B, linear IGPD matches ExOPD's mean in 151.78 versus 169.34 minutes.
The practical benefit is an evolving internal reference that improves supervision without maintaining a separate reference model. The linear variant improves on ExOPD's AIME mean by 3.96 points for 1.7B and matches its 4B mean with 22.0% shorter responses. The lightweight readout reduces preparation cost while retaining comparable accuracy.