acceptodds
Withdrawn from ICLR 2027

Block-Level Credit Assignment in On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) transfers reasoning behavior through teacher feedback on student-generated trajectories. Common sampled-token methods assign this feedback independently to each token, although a local reasoning decision may span several tokens. We introduce Block3 Mean, which averages teacher–student log-probability differences within three-token blocks and optimizes a clipped objective with a joint block importance ratio and valid-length weighting. This gives adjacent tokens a shared learning signal without requiring an additional reward model or process annotations. We analyze how the update couples neighboring credit, changes gradient scale and alters clipping, and evaluate mathematical reasoning across teacher–student pairs and saved checkpoints. In the historical Qwen3-1.7B comparison at Step200, Block3 improves Avg@8 over sampled-token OPD by 8.90 percentage points on MATH500 and 5.42 on AMC23. The official Qwen3-8B-to-1.7B Instruct pair also has positive Step200 Avg@8 differences on all four evaluated benchmarks. Trajectory diagnostics connect these results to entropy, teacher–student overlap and local credit redistribution, while distinguishing numerical stability from generation quality. Our study makes feedback granularity an explicit design dimension in reasoning distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, credit assignment, language model reasoning, training stability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_byk2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Block3 Mean, which averages teacher–student log-probability differences within 3-token blocks and optimizes a clipped PPO objective with a joint block importance ratio and valid-length weighting. The motivation is that token-level feedback may be too noisy for reasoning decisions spanning multiple tokens. The paper provides gradient analysis, evaluates on math reasoning benchmarks across multiple teacher–student pairs, and includes extensive diagnostics. Results are mixed: Block3 shows gains in the historical Qwen3-1.7B setting and small positive differences for official Qwen3-8B→1.7B Instruct, but the 4B Base comparison favors Token on 3/4 tasks, and Block3 catastrophically fails on 0.6B and Llama students.

Critical strengths

The paper's transparency about negative results is exemplary.

Critical weaknesses
  1. Block3 catastrophically fails on 2 of 4 historical student types (0.6B and Llama), produces mostly negative results on the controlled 4B comparison, and yields statistically insignificant improvements on the official Instruct comparison. A method that dramatically improves one setting while destroying another is not ready for a contribution claim. The paper acknowledges this honestly but does not resolve it.

  2. Block3 simultaneously changes advantage sharing, the importance ratio, and the loss reduction. Without ablating these components independently, we cannot determine whether the observed effects come from the signal smoothing that motivates the method, the gradient scale change, or the altered clipping behavior.

  3. At its core, the proposed intervention is relatively straightforward: average teacher–student log-probability differences within fixed 3-token blocks and feed the result into a PPO-style clipped objective. The paper does not yet offer a principled justification for why this particular block construction should be expected to generalize beyond the intuitive smoothing argument. I am not yet convinced the current contribution clears the novelty/insight bar for ICLR, though I would be happy to be convinced by a clearer articulation of the underlying principle and a more decisive empirical separation.

Questions for authors

Clarification

  1. Can you characterize what distinguishes the teacher–student pairs where Block3 works (1.7B, DeepSeek 1.5B) from those where it fails catastrophically (0.6B, Llama 1B)? Is there a teacher–student compatibility condition that predicts success?

  2. Have you tried Block3 with the token-level ratio (instead of the joint block ratio) to isolate the effect of advantage averaging from the ratio change?

Reviewer_aaP2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates block-level credit assignment in on-policy distillation (OPD). The proposed method, Block3 Mean, replaces individual token advantages with their average over non-overlapping three-token blocks and optimizes a PPO-style objective using joint block importance ratios. The authors evaluate the method on mathematical reasoning benchmarks.

Critical strengths
  1. The paper explores feedback granularity as a design choice in OPD, which is not fully explored for LLM post-training.
Critical weaknesses

1. The proposed method combines multiple optimization changes without isolating their individual contributions.

  • The central motivation is to improve credit assignment by averaging teacher-student feedback across neighboring tokens. However, the proposed method simultaneously changes (1) advantage aggregation, (2) importance ratio computation, and (3) effective gradient scaling.

  • In particular, Equation (3) uses the product of token-level importance ratios rather than individual ratios. As Proposition 1 demonstrates, this causes the block-level gradient to be (k) times larger than the token-level gradient when all token advantages are identical, even though advantage averaging has no effect in this case. Furthermore, the joint ratio changes PPO clipping behavior independently of advantage aggregation.

  • Therefore, it is unclear whether the observed performance differences arise from improved credit assignment, altered gradient magnitudes, or different clipping behavior.

2. The motivation for fixed-length block-level credit assignment is insufficiently supported.

  • The paper motivates block-level aggregation by arguing that a reasoning decision may span multiple adjacent tokens. However, the proposed method simply groups three consecutive tokens without considering reasoning boundaries, semantic structure, or the relevance of individual tokens.

  • It is unclear why three adjacent tokens should constitute a meaningful unit of reasoning. A block may contain unrelated tokens with conflicting teacher preferences, and averaging their advantages can suppress or even reverse informative token-level feedback.

3. The empirical results are inconsistent and do not demonstrate reliable improvements over token-level OPD.

  • In Table 3, Block3 underperforms token-level OPD on three of the four benchmarks in Avg@8. Figure 3 also shows substantial checkpoint-dependent fluctuations, with the relative ranking reversing during training.

  • Figure 4 also demonstrates no superficial performance of Block3.

4. The paper's presentation and clarity need substantial improvement.

  • Figures 1 and 2 provide largely overlapping illustrations of advantage averaging, but neither clearly explains the complete optimization objective, particularly the joint block importance ratio and its implications.

  • The experimental results are poorly organized. Table 1 highlights selected positive historical results, whereas Tables 2 and 3 present more rigorously controlled comparisons with marginal or negative improvements. The main empirical conclusions are therefore difficult to identify.

  • The manuscript contains excessive implementation details, historical experimental descriptions, and auxiliary diagnostics that interrupt the main narrative. The connection between these analyses and the central claim of improved credit assignment is often unclear.

Questions for authors

Please check the weaknesses above.

Reviewer_YQyqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Block3 Mean for on-policy distillation, sharing teacher feedback within three-token blocks and using joint block importance ratios. Experiments on mathematical reasoning benchmarks show mixed results across model settings.

Critical strengths
  1. The paper studies a meaningful credit-assignment question and provides useful analysis of cross-token gradients and clipping behavior.
  2. The authors transparently report negative results and experimental limitations.
Critical weaknesses
  1. Limited empirical evidence. As acknowledged in Section 4.2, "the positive estimates do not resolve a statistical advantage at this endpoint." The well-documented 4B comparison favors Token OPD on three of four benchmarks. The strongest historical 1.7B gains also come from a protocol with acknowledged training–evaluation differences and sampling limitations. Overall, the evidence does not establish a reliable advantage for Block3.
  2. Missing key component ablations. Section 3.2 states that "the intervention changes advantage sharing, ratio granularity and loss reduction together." Appendix A further notes that "the measured performance differences concern the complete update rather than advantage averaging alone." Without isolating these changes, it is unclear which component drives the reported improvements.
Questions for authors
  1. Can the substantial historical 1.7B improvements be reproduced using the same teacher–student pair under a corrected and consistent training–evaluation protocol?
  2. Can the authors isolate the effects of advantage sharing, joint importance ratios, and the effective gradient scale?
  3. Presentation suggestion : Could the authors use vector figures with clearer labels and streamline the extensive historical experiment records to improve readability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.