- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Block3 Mean, which averages teacher–student log-probability differences within 3-token blocks and optimizes a clipped PPO objective with a joint block importance ratio and valid-length weighting. The motivation is that token-level feedback may be too noisy for reasoning decisions spanning multiple tokens. The paper provides gradient analysis, evaluates on math reasoning benchmarks across multiple teacher–student pairs, and includes extensive diagnostics. Results are mixed: Block3 shows gains in the historical Qwen3-1.7B setting and small positive differences for official Qwen3-8B→1.7B Instruct, but the 4B Base comparison favors Token on 3/4 tasks, and Block3 catastrophically fails on 0.6B and Llama students.
Critical strengths
The paper's transparency about negative results is exemplary.
Critical weaknesses
-
Block3 catastrophically fails on 2 of 4 historical student types (0.6B and Llama), produces mostly negative results on the controlled 4B comparison, and yields statistically insignificant improvements on the official Instruct comparison. A method that dramatically improves one setting while destroying another is not ready for a contribution claim. The paper acknowledges this honestly but does not resolve it.
-
Block3 simultaneously changes advantage sharing, the importance ratio, and the loss reduction. Without ablating these components independently, we cannot determine whether the observed effects come from the signal smoothing that motivates the method, the gradient scale change, or the altered clipping behavior.
-
At its core, the proposed intervention is relatively straightforward: average teacher–student log-probability differences within fixed 3-token blocks and feed the result into a PPO-style clipped objective. The paper does not yet offer a principled justification for why this particular block construction should be expected to generalize beyond the intuitive smoothing argument. I am not yet convinced the current contribution clears the novelty/insight bar for ICLR, though I would be happy to be convinced by a clearer articulation of the underlying principle and a more decisive empirical separation.
Questions for authors
Clarification
-
Can you characterize what distinguishes the teacher–student pairs where Block3 works (1.7B, DeepSeek 1.5B) from those where it fails catastrophically (0.6B, Llama 1B)? Is there a teacher–student compatibility condition that predicts success?
-
Have you tried Block3 with the token-level ratio (instead of the joint block ratio) to isolate the effect of advantage averaging from the ratio change?