- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents IDA-OPD, which mitigates the pass@k bottleneck of sampled-token on-policy distillation through adaptive weight control. The authors derive the first-order entropy influence, where a negative value predicts an entropy decrease under an isolated logit-space gradient update. They also propose a symmetric relative teacher-student disagreement factor, which quantifies the degree of disagreement between the teacher and student on a single sampled token. For all negative-influence tokens, IDA-OPD attenuates the advantage signal. Experiments show that IDA-OPD generally improves pass@16 performance across math and coding tasks compared with other OPD variants.
Critical strengths
- The paper investigates the pass@k bottleneck through the lens of entropy (Figure 2), analyzing how the advantage and the student’s distribution jointly contribute to the entropy changes. Using a first-order approximation of the local entropy change, IDA-OPD attenuates predicted entropy-decreasing updates. The entropy analysis provides empirical evidence that it steers the student’s policy entropy in the intended direction.
- The experiments show consistent improvements in pass@16 over vanilla OPD across math and coding tasks in Qwen3-4B and 8B distillation with IDA-OPD, also outperforming some OPD variants with entropy control. Moreover, the pass@k curves in the Qwen3-4B setting provide evidence that IDA-OPD successfully mitigates the pass@k bottleneck.
Critical weaknesses
- The proposed first-order local entropy influence, a core part of adaptive entropy control in IDA-OPD, has already been established in prior work on RLVR [1,2]. This limits the novelty of the theoretical contribution, even though the paper applies the method to on-policy distillation.
[1] On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models, https://openreview.net/forum?id=Sj8NBI6aV0
[2] Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control, https://arxiv.org/pdf/2605.11775
- Although IDA-OPD consistently outperforms vanilla OPD in pass@16, its gains in pass@1 and the coding tasks are marginal, particularly at Qwen3-8B distillation compared with other OPD variants. Moreover, the evaluations are limited to a single model-family (Qwen3-Non-Thinking), which leaves generalization beyond this model family untested.
Questions for authors
- The proposed first-order influence metric has already been proposed in prior work on RLVR. Are there differences between IDA-OPD and those in [1,2]?
[1] On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models, https://openreview.net/forum?id=Sj8NBI6aV0
[2] Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control, https://arxiv.org/pdf/2605.11775
-
Figure 3 shows that the cumulative entropy loss concentrates in regions where the teacher and student largely agree and the token counts are also high. How is this cumulative entropy loss calculated and aggregated? Could you also report the mean entropy loss per token in each bin?
-
The experimental setup uses 128 responses per problem for two model sizes and their respective baselines. Could you provide pass@k curves for the Qwen3-8B models? Also, have you conducted any additional experiments on other model families?
-
As discussed in Appendix B, IDA-OPD uses the student’s full-vocabulary probability distribution to compute the first-order local entropy influence. Does this introduce any memory or runtime overhead compared with baselines such as AOPD or EOPD? Any quantitative measurements would help clarify the practical trade-off.
2 comments
Thank you for the response. I appreciate the acknowledgement of similarity to prior works and clarification of the difference in OPD objective. Although it takes a first-order Taylor expansion approach yielding the same coefficient, the expanded comparison of related work largely resolves the concern.
Also, additional pass@k curves for Qwen3-8B-Non-Thinking strengthen the comparison with vanilla OPD, but the generalization beyond the Qwen3 family remains untested.
The per-token statistics confirms the peak concentration of the cumulative loss which is consistent with the motivation in section 4, but the mean loss per-token near agreement seems to be lower than in the other reported intervals. It would be better if section 4 clarify this.
I understand that the actual runtime and peak GPU memory depend on the implementation and infrastructure, though providing quantitative results would help answer the question.
Clarifications are appreciated, but still there are some remaining concerns, leading me to maintain the overall rating.
Relation to RLVR work [1,2]. Prior RLVR work [1,2] indeed uses a similar first-order analysis and derives the same student-side coefficient Dy. However, First-order Taylor expansion is a standard analytical tool, and its use in prior work does not diminish the novelty of our contribution. Our contribution is to establish how this coefficient interacts with the teacher signal in sampled-token OPD. Specifically, the OPD objective induces the local logit update Δz=ηAy(ey−p), where Ay=logqy−logpy, yielding ΔH=ηAyDy+O(η2). This result reveals that teacher-directed updates can either increase or decrease student entropy: their effect depends jointly on the teacher–student discrepancy Ay and the student's current distribution through Dy.
Based on our derivation, IDA-OPD scales only updates with AyDy<0 by tanh(∣Ay∣/2), suppressing contraction near agreement while preserving stronger teacher corrections at larger gaps. This differs from coefficient-guided clipping in [1] and entropy-trajectory-based weighting of two RL branches in [2]. We have expanded this comparison in Section 6 (Related Work).
Figure 3 and per-token normalization. In the Qwen3-4B training setup, we compute Hbefore−Hafter at the same sampled positions before and after each recorded minibatch update. We group positions by δ=(qy−py)/(qy+py), sum the differences across minibatches, and divide by the corresponding token count to obtain the mean. For pooled intervals we sum losses and counts before dividing. The estimates below use the totals and counts plotted in Figure 3.
Near agreement, the mean loss is smaller, but about 78% of positions fall in that interval, so it contributes about 68% of cumulative net loss. This explains why the aggregate peak motivates targeting many small corrections there. Table 6 tests the weighting rule with the influence gate held fixed. Adaptive shrinkage improves pass@16 over constant shrinkage from 79.2 to 83.3 on AIME24 and from 66.7 to 70.0 on AIME25.
8B curves and model choice. The revised Figure 5 reports AIME24, AIME25, MBPP++, and LiveCodeBench v6 at k=1,2,4,8,16,32,64. IDA-OPD leads vanilla OPD at every plotted budget. The pass@16 gains are +4.2, +6.0, +1.7, and +1.5 points, respectively. Table 2 shows mixed pass@1 rankings and smaller coding margins against other OPD variants. We use the same Qwen3-Non-Thinking backbones at 4B and 8B and a common evaluation protocol for fair comparison with the baselines.
Compute and memory. Appendix B quantifies O(LV) reductions on existing student logits and one teacher log-probability per sampled position for IDA-OPD, compared with top-K teacher values at selected positions for AOPD/EOPD. Actual runtime and peak GPU memory depend on the implementation and training infrastructure.