acceptodds
Under review as a conference paper at ICLR 2027

Influence-Directed Distillation: Solving the Pass@k Bottleneck in Sampled-Token On-Policy Distillation

Abstract

Sampled-token on-policy distillation (OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teacher probabilities only for sampled tokens. Yet gains in pass@1 do not always translate into comparable gains in pass@, leaving a pass@ bottleneck in distillation. To explain this, we introduce First-Order Local Entropy Influence, a signed first-order proxy that decouples each update's entropy effect into the teacher–student log-probability gap and the student's local probability structure, and empirically links entropy contraction to negative-influence positions. Motivated by this, we propose Influence-Directed Adaptive On-Policy Distillation (IDA-OPD): rather than relying on costly full-vocabulary Forward-KL objectives, it preserves entropy-expanding updates while replacing entropy-contracting ones with divergence-adaptive advantage shrinkage, using only the teacher's sampled-token log-probability. Experiments on reasoning-oriented distillation show IDA-OPD consistently improves pass@, inheriting the teacher's diversity through distillation, matches the strongest teacher-informed methods at strictly lower cost, and broadly maintains vanilla OPD's pass@1, all without full-vocabulary teacher information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nY6ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents IDA-OPD, which mitigates the pass@k bottleneck of sampled-token on-policy distillation through adaptive weight control. The authors derive the first-order entropy influence, where a negative value predicts an entropy decrease under an isolated logit-space gradient update. They also propose a symmetric relative teacher-student disagreement factor, which quantifies the degree of disagreement between the teacher and student on a single sampled token. For all negative-influence tokens, IDA-OPD attenuates the advantage signal. Experiments show that IDA-OPD generally improves pass@16 performance across math and coding tasks compared with other OPD variants.

Critical strengths
  • The paper investigates the pass@k bottleneck through the lens of entropy (Figure 2), analyzing how the advantage and the student’s distribution jointly contribute to the entropy changes. Using a first-order approximation of the local entropy change, IDA-OPD attenuates predicted entropy-decreasing updates. The entropy analysis provides empirical evidence that it steers the student’s policy entropy in the intended direction.
  • The experiments show consistent improvements in pass@16 over vanilla OPD across math and coding tasks in Qwen3-4B and 8B distillation with IDA-OPD, also outperforming some OPD variants with entropy control. Moreover, the pass@k curves in the Qwen3-4B setting provide evidence that IDA-OPD successfully mitigates the pass@k bottleneck.
Critical weaknesses
  • The proposed first-order local entropy influence, a core part of adaptive entropy control in IDA-OPD, has already been established in prior work on RLVR [1,2]. This limits the novelty of the theoretical contribution, even though the paper applies the method to on-policy distillation.

[1] On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models, https://openreview.net/forum?id=Sj8NBI6aV0

[2] Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control, https://arxiv.org/pdf/2605.11775

  • Although IDA-OPD consistently outperforms vanilla OPD in pass@16, its gains in pass@1 and the coding tasks are marginal, particularly at Qwen3-8B distillation compared with other OPD variants. Moreover, the evaluations are limited to a single model-family (Qwen3-Non-Thinking), which leaves generalization beyond this model family untested.
Questions for authors
  • The proposed first-order influence metric has already been proposed in prior work on RLVR. Are there differences between IDA-OPD and those in [1,2]?

[1] On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models, https://openreview.net/forum?id=Sj8NBI6aV0

[2] Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control, https://arxiv.org/pdf/2605.11775

  • Figure 3 shows that the cumulative entropy loss concentrates in regions where the teacher and student largely agree and the token counts are also high. How is this cumulative entropy loss calculated and aggregated? Could you also report the mean entropy loss per token in each bin?

  • The experimental setup uses 128 responses per problem for two model sizes and their respective baselines. Could you provide pass@k curves for the Qwen3-8B models? Also, have you conducted any additional experiments on other model families?

  • As discussed in Appendix B, IDA-OPD uses the student’s full-vocabulary probability distribution to compute the first-order local entropy influence. Does this introduce any memory or runtime overhead compared with baselines such as AOPD or EOPD? Any quantitative measurements would help clarify the practical trade-off.

2 comments

Relation to RLVR work [1,2]. Prior RLVR work [1,2] indeed uses a similar first-order analysis and derives the same student-side coefficient DyD_y. However, First-order Taylor expansion is a standard analytical tool, and its use in prior work does not diminish the novelty of our contribution. Our contribution is to establish how this coefficient interacts with the teacher signal in sampled-token OPD. Specifically, the OPD objective induces the local logit update Δz=ηAy(ey−p)\Delta z=\eta A_y(e_y-p), where Ay=log⁡qy−log⁡pyA_y=\log q_y-\log p_y, yielding ΔH=ηAyDy+O(η2)\Delta H=\eta A_yD_y+O(\eta^2). This result reveals that teacher-directed updates can either increase or decrease student entropy: their effect depends jointly on the teacher–student discrepancy AyA_y and the student's current distribution through DyD_y.

Based on our derivation, IDA-OPD scales only updates with AyDy<0A_yD_y<0 by tanh⁡(∣Ay∣/2)\tanh(|A_y|/2), suppressing contraction near agreement while preserving stronger teacher corrections at larger gaps. This differs from coefficient-guided clipping in [1] and entropy-trajectory-based weighting of two RL branches in [2]. We have expanded this comparison in Section 6 (Related Work).

Figure 3 and per-token normalization. In the Qwen3-4B training setup, we compute Hbefore−HafterH_{\rm before}-H_{\rm after} at the same sampled positions before and after each recorded minibatch update. We group positions by δ=(qy−py)/(qy+py)\delta=(q_y-p_y)/(q_y+p_y), sum the differences across minibatches, and divide by the corresponding token count to obtain the mean. For pooled intervals we sum losses and counts before dividing. The estimates below use the totals and counts plotted in Figure 3.

Discrepancy intervalMean loss per tokenShare of total net loss
[−1.0,−0.9)[-1.0,-0.9)0.0934.3%
[−0.6,−0.4)[-0.6,-0.4)0.0523.4%
[−0.1,0.1)[-0.1,0.1)0.03968.1%

Near agreement, the mean loss is smaller, but about 78% of positions fall in that interval, so it contributes about 68% of cumulative net loss. This explains why the aggregate peak motivates targeting many small corrections there. Table 6 tests the weighting rule with the influence gate held fixed. Adaptive shrinkage improves pass@16 over constant shrinkage from 79.2 to 83.3 on AIME24 and from 66.7 to 70.0 on AIME25.

8B curves and model choice. The revised Figure 5 reports AIME24, AIME25, MBPP++, and LiveCodeBench v6 at k=1,2,4,8,16,32,64k=1,2,4,8,16,32,64. IDA-OPD leads vanilla OPD at every plotted budget. The pass@16 gains are +4.2, +6.0, +1.7, and +1.5 points, respectively. Table 2 shows mixed pass@1 rankings and smaller coding margins against other OPD variants. We use the same Qwen3-Non-Thinking backbones at 4B and 8B and a common evaluation protocol for fair comparison with the baselines.

Compute and memory. Appendix B quantifies O(LV)O(LV) reductions on existing student logits and one teacher log-probability per sampled position for IDA-OPD, compared with top-KK teacher values at selected positions for AOPD/EOPD. Actual runtime and peak GPU memory depend on the implementation and training infrastructure.

Thank you for the response. I appreciate the acknowledgement of similarity to prior works and clarification of the difference in OPD objective. Although it takes a first-order Taylor expansion approach yielding the same coefficient, the expanded comparison of related work largely resolves the concern.

Also, additional pass@k curves for Qwen3-8B-Non-Thinking strengthen the comparison with vanilla OPD, but the generalization beyond the Qwen3 family remains untested.

The per-token statistics confirms the peak concentration of the cumulative loss which is consistent with the motivation in section 4, but the mean loss per-token near agreement seems to be lower than in the other reported intervals. It would be better if section 4 clarify this.

I understand that the actual runtime and peak GPU memory depend on the implementation and infrastructure, though providing quantitative results would help answer the question.

Clarifications are appreciated, but still there are some remaining concerns, leading me to maintain the overall rating.

Reviewer_kY67OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IDA-OPD, a method for sampled-token on-policy distillation that aims to prevent student entropy collapse. It uses a first-order approximation to estimate the entropy influence of each sampled-token update. The training updates predicted to increase entropy are kept, while entropy-decreasing updates are attenuated based on the teacher-student log-prob gap. The goal is to preserve student diversity while still maintaining the teacher's learning signal.

Critical strengths
  • The authors propose a simple solution that can preserve student entropy by using the advantage and a student-side entropy factor.
  • The authors provide theoretical justification for the method.
  • The experiments show that the method improves pass@16 performance compared to vanilla OPD.
Critical weaknesses
  • Paper [1] came out about four months before this submission, so it may not be a concurrent work. It also uses a first-order approximation and analyzes entropy collapse. Although this paper focuses on RLVR rather than OPD, some of the theoretical analysis and insights are not new.
  • The authors acknowledge this in the paper, but it is still a weakness that these theorems rely on idealized logit-space updates and may not directly apply to deep neural networks trained using practical optimizers in a real setting. The effect of attenuation may also differ under AdamW, which the authors use.
  • I think the main weakness is in the experimental results. The gains over the strongest baselines look marginal in several cases. AIME has only 30 questions, so a 3.3 percentage-point difference is roughly equivalent to one question in a single-sample evaluation. Also, the number of training seeds is not reported, and AIME results could vary substantially across seeds. Furthermore, the authors should emphasize the gains over the best baseline, not just OPD. EOPD and other papers also have similar results and analyses like entropy histograms, so the actual performance gains over these methods matter more.
  • Also, I believe the core part is how the proposed method updates the student while preserving student entropy. Although Figure 5(a) shows the entropy trajectory during training, I think there is limited analysis of how the proposed method affects token-level entropy during OPD training, not just the final entropy distribution. Having an empirical efficiency analysis might be beneficial too.
  • The writing is hard to read in some parts, especially the abstract. References to specific appendices are not always clear. In the tables, some columns are missing bold marks for the best results.

References

[1] Zhang et al. "Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control." (NeurIPS 2026)

Questions for authors
  1. Given the current results, how can the authors conclude that IDA-OPD outperforms the other baselines?
  2. How many independent training seeds were used for the experiments for the main table?
  3. How does IDA-OPD affect student entropy at the token level throughout training, beyond the aggregate entropy trajectories and final entropy distributions?
2 comments

Thank you for raising the comparison with prior RLVR work. IDA-OPD turns a local entropy diagnostic into a teacher-disagreement rule for sampled-token distillation. With one teacher scalar per position, it leads the distilled baselines in pass@16 on all eight math benchmark–scale comparisons in Tables 1–2.

Relation to RLVR work. We agree that Zhang et al.’s Entropy Polarity predates our submission and derives the same student-side coefficient DyD_y. However, First-order Taylor expansion is a standard analytical tool, and its use in prior work does not diminish the novelty of our contribution. Our contribution is to establish how it interacts with the teacher signal in sampled-token OPD. Specifically, the OPD objective induces the local logit update Δz=ηAy(ey−p)\Delta z=\eta A_y(e_y-p), where Ay=log⁡qy−log⁡pyA_y=\log q_y-\log p_y, yielding ΔH=ηAyDy+O(η2)\Delta H=\eta A_yD_y+O(\eta^2). This result reveals that teacher-directed updates can either increase or decrease student entropy: their effect depends jointly on the teacher–student discrepancy AyA_y and the student's current distribution through DyD_y.

Based on our derivation, IDA-OPD scales only updates with AyDy<0A_yD_y<0 by tanh⁡(∣Ay∣/2)\tanh(|A_y|/2), suppressing contraction near agreement while preserving stronger teacher corrections at larger gaps. Wang et al. use this coefficient to guide clipping, while Zhang et al.’s PAPO uses the training entropy trajectory to weight two RL branches. We have expanded this comparison in Section 6 (Related Work).

Performance against the strongest baselines. We agree that some gains may look marginal in absolute terms, but they can be substantial relative to the remaining gap. At pass@16, IDA-OPD brings the student closer to teacher performance than competing distillation methods do and surpasses its own teacher on several benchmarks. On AIME24, for example, the 2.2-point gain over the strongest 4B baseline closes the entire remaining gap to the 8B teacher. IDA-OPD leads the strongest baseline at pass@16 on all four math benchmarks at both model sizes (Tables 1–2).

The benefit is clearest when the model has multiple attempts. On both 8B AIME benchmarks, the advantage over vanilla OPD widens from pass@1 to pass@16 and persists through pass@64 (Figure 5 in the revision). This is the improvement IDA-OPD targets: a higher chance of finding a correct solution as the sampling budget grows.

Applicability to AdamW. All reported training runs use AdamW. The masking ablations shift entropy in the directions predicted by Theorem 1 (Fig. 6a in the revision), and IDA-OPD improves pass@16 under the same optimizer, showing that the first-order approximation can guide updates in practice.

Token-level entropy during training. The student generates fresh rollouts during on-policy training, so there is no fixed set of prefixes and token positions to track across batches. Figure 6a tracks mean token entropy on the student's own rollouts over training and shows that IDA-OPD maintains higher entropy than vanilla OPD. On each model's AIME24 rollouts, near-deterministic positions account for 55% under IDA-OPD, compared with 74% under vanilla OPD and 57% for the teacher (Figure 5b, Figure 6b in the revision).

Cost and presentation. Appendix B and Table 4 give the analytical trade-off: one teacher scalar per position and a student-side reduction over the vocabulary. An end-to-end runtime and memory comparison remains a separate empirical question. Appendix A gives the proofs. We agree that the abstract should foreground the OPD observation and training rule, and the tables should mark each column's best distilled result consistently.

Reviewer_kY67OpenReview ↗

Response to the Rebuttal

Thank you for the response and revisions. I appreciate the authors' efforts. However, my main concerns regarding the entropy analysis and experimental results remain.

While Figure 6a shows that the proposed method maintains higher aggregate entropy and Figure 6b shows differences in the final token-level entropy distributions, these analyses do not directly address how token-level entropy evolves throughout training. More importantly, it is still unclear whether the proposed first-order entropy influence reliably predicts the actual direction of entropy changes under practical optimization like AdamW in this work. I think it would have been helpful to compare the predicted entropy influence with the actual entropy changes during training, or probably to track token-level entropy on a fixed set of prefixes across training checkpoints. Again, this is because the method is mainly use these changes in entropy.

I also appreciate the additional pass@k results and the reporting of three seeds. However, I still find the improvements over the strongest baselines relatively marginal. Also, the results across three training seeds are reported only for IDA-OPD so it's not fully resolved. So it seems like the results suggest that the proposed method does provide improvements, but the gains appear relatively small.

Reviewer_2MiNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IDA-OPD to mitigate the pass@k bottleneck in sampled-token on-policy distillation. It derives a first-order local entropy-influence criterion and uses its sign to retain or attenuate the sampled-token advantage. The method requires only sampled-token teacher probabilities and reports improved pass@k on mathematics and code benchmarks at 4B and 8B scales.

Critical strengths
  1. Addresses the entropy problem while retaining sampled-token-only teacher supervision.
  2. Connects a clearly scoped first-order analysis to an understandable, sign-preserving update rule.
  3. Evaluates two model scales and two task domains, with ablations examining both the influence gate and shrinkage design.
Critical weaknesses
  1. The reliability of small improvements is unclear. Some margins over competing methods are only 0.3 percentage points, such as 4B AIME25 pass@16 (70.0 versus AOPD's 69.7). Training-run counts and confidence intervals are not reported. The 128 decoding samples per problem help estimate checkpoint performance but do not quantify training variability.

  2. Figure 3 does not separate token frequency from per-token entropy effects. The figure reports cumulative entropy loss across bins containing different numbers of tokens. Please clarify whether the entropy loss in Figure 3 is normalized by the token count in each bin. If not, reporting the per-token mean would help distinguish token-frequency effects from per-token entropy changes

  3. Figure 5b compares models on different prefix distributions. Each model generates its own responses, so histogram differences may reflect both changes in predictive uncertainty and changes in the contexts visited. Evaluating all models on shared prefixes would better isolate the entropy effect of IDA-OPD.

Questions for authors

How is the cumulative entropy loss in Figure 3 computed?

2 comments

Reliability of small improvements. We agree that some gains may look marginal in absolute terms, but they can be substantial relative to the remaining gap. On AIME24, the 2.2-point improvement over the strongest 4B baseline is enough for our 4B student to match the 8B teacher. On AIME25, we now report a pass@16 standard deviation of 0.052 percentage points across three independent IDA-OPD training runs (Table 1). The 0.3-point margin over AOPD’s reported score is larger than the variation we observe across our three seeds.

Token frequency and entropy loss in Figure 3. For each recorded minibatch update, we measure Hbefore−HafterH_{\rm before}-H_{\rm after} at the same sampled token positions. Figure 3 sums those differences within each teacher–student discrepancy bin across minibatches. Dividing each sum by its token count gives the mean loss per token. We report estimates for three representative bins below.

Discrepancy binMean loss per tokenShare of cumulative net loss
[−1.0,−0.9)[-1.0,-0.9)0.0934.3%
[−0.6,−0.4)[-0.6,-0.4)0.0523.4%
[−0.1,0.1)[-0.1,0.1)0.03968.1%

The near-agreement bin contains about 78% of sampled positions. Its mean loss is lower than in the other bins shown, but it accounts for more than two-thirds of the cumulative loss. This explains why IDA-OPD applies stronger attenuation near agreement.

Entropy on each model’s own rollouts. To measure the effect of IDA-OPD on entropy during generation, we need to compute entropy on each model’s own rollouts. Using continuations generated by another model would introduce off-policy prefixes and change the distribution over which entropy is measured. On the same AIME24 prompts, nearly deterministic positions account for 55% of IDA-OPD’s rollout positions, compared with 74% for OPD (Figure 6b).

Thank you for the clarification. Some of my concerns have been addressed. However, I remain concerned about the marginal gains of IDA-OPD, leading me to maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.