Block-Level Credit Assignment in On-Policy Distillation

On-policy distillation (OPD) transfers reasoning behavior through teacher feedback on student-generated trajectories. Common sampled-token methods assign this feedback independently to each token, although a local reasoning decision may span several tokens. We introduce Block3 Mean, which averages teacher–student log-probability differences within three-token blocks and optimizes a clipped objective with a joint block importance ratio and valid-length weighting. This gives adjacent tokens a shared learning signal without requiring an additional reward model or process annotations. We analyze how the update couples neighboring credit, changes gradient scale and alters clipping, and evaluate mathematical reasoning across teacher–student pairs and saved checkpoints. In the historical Qwen3-1.7B comparison at Step200, Block3 improves Avg@8 over sampled-token OPD by 8.90 percentage points on MATH500 and 5.42 on AMC23. The official Qwen3-8B-to-1.7B Instruct pair also has positive Step200 Avg@8 differences on all four evaluated benchmarks. Trajectory diagnostics connect these results to entropy, teacher–student overlap and local credit redistribution, while distinguishing numerical stability from generation quality. Our study makes feedback granularity an explicit design dimension in reasoning distillation.