Prefix-Outcome Validation for Reliable On-Policy Distillation of Reasoning Models

On-policy distillation (OPD) provides dense token-level supervision by evaluating student-generated trajectories with a teacher model. However, a globally stronger teacher does not guarantee beneficial local supervision: teacher preferences can conflict with response outcomes, and student-generated prefixes can deviate from teacher-supported reasoning paths. We introduce Prefix-Outcome Validation (POV), a selective correction mechanism for dense OPD supervision. POV uses group-relative outcomes to identify conflicting supervision and relative teacher support across consecutive windows to regulate attenuation strength. Only outcome-conflicting supervision is attenuated, with lower prefix support inducing stronger attenuation; all other supervision remains unchanged. The correction requires no tunable mixing coefficient and preserves the sign of the original token-level supervision without increasing its magnitude. We evaluate DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and Qwen3-4B as students on AMC23, AIME24, AIME25, and GPQA-Diamond. Across all evaluated student–teacher configurations, POV-OPD improves both four-benchmark macro-averaged mean and best-of-$k$ performance over matched raw OPD. In the GPQA experiment where the teacher underperforms the student, POV-OPD also mitigates the loss of pre-distillation capability. Component ablations demonstrate the complementary value of outcome validation and prefix support. Further token-level analysis shows that POV's selective correction reaches key reasoning functions, including logical structuring, self-checking, and reasoning revision. Together, these findings highlight POV as a simple and effective approach to improving on-policy distillation: without additional process annotations, it uses response outcomes and teacher support to selectively weight dense supervision, reducing conflicting signals while retaining teacher guidance.