- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This study investigates On-Policy Self-Distillation (OPSD) for Large Language Models' (LLMs) reasoning; specifically, the authors focus on Self-Distillation via Policy Optimization (SDPO). The authors argue that SDPO imposes too much distillation pressure on the valid prefix of incorrect rollouts, which reduces diversity and causes late-stage collapse. To address these issues, the authors introduce ROSD. In ROSD, for each incorrect rollout, a single (frozen) model serves as a "reflector." The reflector compares the incorrect rollout to the shortest correct rollout within the group, generates a brief corrective explanation, and identifies the sentence in which the first error occurred. Instead of providing a complete solution, the teacher is conditioned on the explanation generated by the reflector. Additionally, prior to the sentence identified by the reflector as containing the first error, the Jensen-Shannon Divergence (JSD) loss is masked. For correct rollouts, the reflector provides a summary of why the rollout was correct, and the entire response is distilled. The authors evaluate ROSD using four SciKnowEval L3 subsets and ToolAlpaca, using Qwen3-4B/8B and Llama-3.2-3B-Instruct. The authors report that ROSD achieves higher average mean@16 than both GRPO and SDPO, is more stable during training than SDPO, and is more generalizable across domains than SDPO.
Critical strengths
- The problem addressed is important. Figures 3/4 and Table 5 demonstrate that SDPO can experience significant degradation over extended periods of training. For example, Qwen3-4B trained on ToolUse experiences a drop to a 2.88 cross-domain average. This is a valuable finding for researchers working on self-distillation.
- The proposed method is easy to implement on top of SDPO. No additional models or trainable parameters are required. The concept of masking everything prior to the first error is logical.
- The ablation study separates the two components (reflection and localized masking) and evaluates them on all three backbones. Code is provided.
Critical weaknesses
- The performance gains are small. On Qwen3-8B, ROSD gets 73.45 compared with 72.50 for SDPO. On Llama, it gets 53.38 compared with 52.69 for GRPO. Per task, ROSD is often not the best. The test sets are also small: 50 Biology, 68 ToolUse, 80 Physics, and 94 Material questions.
- The quality of error localization is never evaluated, even though the entire method relies on the quote being the first error. Fig. 9 only indicates that quotes can be matched to the original text via string matching 57-83% of the time. All other quotes are used in full response distillation, which means a large portion of incorrect rollouts are trained in a manner similar to SDPO with reflection. The paper does not evaluate whether matched quotes are accurate, for example by comparing them to a stronger judge or a small human-labeled set.
- One of paper’s motivations is that distilling on already-valid reasoning hurts diversity. However, for correct rollouts, ROSD distills every token, with a teacher conditioned on a summary of that same rollout. This seems to go against the paper’s own argument.
Questions for authors
- The Llama baseline scores of 0.00-0.01 need to be explained.
- A breakdown of the training times shown in Fig. 7 into rollout, reflection, and teacher forward would provide more detail.
7 comments
Response to Reviewer W7Sy (2/4)
2. Quality Evaluation of Error Localization and Reflection Accuracy (Weakness 2)
We appreciate the reviewer pointing out the necessity of validating the semantic correctness of error localization beyond lexical string matching. To directly evaluate the semantic correctness of the reflector during training, we sampled incorrect rollouts and their corresponding reflector outputs across checkpoints throughout training on Material and Chemistry across Qwen3-4B and Qwen3-8B. We conducted two decoupled evaluation tasks using an independent judge model (Claude-Opus-5-5):
- Explanation Accuracy (Overall): Evaluated on the diagnostic explanations () generated for all sampled incorrect rollouts to determine whether the explanation correctly identifies the core reasoning failure and provides scientifically valid guidance.
- First-Error Hit Rate (on Matched): Evaluated on the extracted error quotes () that successfully matched the student rollout to determine whether the quoted sentence genuinely marks the exact first error, thereby ensuring all preceding prefix reasoning is fully sound.
Table: Semantic Correctness Evaluation of Reflector During Training
| Model Backbone | Benchmark | String Match Rate | First-Error Hit Rate (on Matched) | Explanation Accuracy (Overall) |
|---|---|---|---|---|
| Qwen3-4B | Material | 56.4% | 89.5% | 90.2% |
| Qwen3-4B | Chemistry | 67.7% | 90.3% | 92.5% |
| Qwen3-8B | Material | 71.7% | 89.2% | 92.5% |
| Qwen3-8B | Chemistry | 82.4% | 91.5% | 93.0% |
Quantitative Findings on Reflector Quality: The empirical results in the table above demonstrate two critical conclusions regarding the self-reflector's diagnostic quality during training:
- High Semantic Precision of Error Localization: Across both model backbones and domains, whenever the reflector's quoted error string successfully matches the original rollout, it accurately pinpoints the exact first logical or factual mistake in 89.2% to 91.5% of cases (e.g., 89.5% on Qwen3-4B Material and 91.5% on Qwen3-8B Chemistry). This high hit rate directly confirms our core hypothesis: masking the distillation loss prior to the diagnosed error boundary reliably protects reasoning prefixes that are genuinely logically sound and scientifically valid, rather than performing arbitrary truncation.
- High Diagnostic Accuracy Across All Rollouts: The diagnostic explanation () achieves between 90.2% and 93.0% accuracy across all sampled rollouts (regardless of substring matching success). This confirms that the reflector consistently understands the core reasoning flaw and supplies scientifically valid corrective guidance, providing dependable semantic conditioning for the self-teacher throughout optimization.
Why Reflection Accuracy is High: The reflector operates under a contrastive reflection formulation (Section 3.1). Conditioning on the problem, the shortest verified correct rollout (), and the incorrect rollout () heavily anchors the diagnostic reasoning to the reference solution, making error boundary identification and corrective insight extraction natural and precise.
Response to Reviewer W7Sy (3/4)
Evaluation Prompts Used for the Independent Judge
Prompt 1: Explanation Accuracy Evaluation (Evaluated on the diagnostic explanations () generated for all sampled incorrect rollouts)
[System]
You are an expert scientific reasoning evaluator. Your task is to evaluate whether a diagnostic explanation produced by a tutor model correctly identifies a student's reasoning mistake and provides valid corrective logic.
[Problem]
{problem_text}
[Verified Correct Solution]
{correct_solution}
[Incorrect Student Solution]
{student_solution}
[Diagnostic Explanation]
{reflector_explanation}
Task:
Determine whether the [Diagnostic Explanation] is accurate.
- TRUE: It correctly identifies the core reasoning mistake in [Incorrect Student Solution], and its corrective reasoning is scientifically sound and aligned with [Verified Correct Solution].
- FALSE: It misidentifies the mistake, hallucinates an error, or provides flawed corrective logic.
Respond strictly in valid JSON format:
{
"is_explanation_accurate": true/false
}
Prompt 2: First-Error Hit Evaluation (Evaluated on the extracted error quotes () that successfully matched the student rollout)
[System]
You are an expert academic evaluator. Your task is to determine whether an extracted quote marks the EXACT FIRST logical or factual error in an incorrect student reasoning chain.
[Problem]
{problem_text}
[Verified Correct Solution]
{correct_solution}
[Incorrect Student Solution]
{student_solution}
[Extracted Error Quote]
{reflector_error_quote}
Task:
Determine whether [Extracted Error Quote] is the FIRST error in [Incorrect Student Solution].
- TRUE: The quoted sentence genuinely contains a reasoning/factual mistake, AND all reasoning strictly BEFORE this quote in [Incorrect Student Solution] is completely valid and correct.
- FALSE: The student already made an earlier mistake BEFORE this quote, OR the quote itself is not actually an error.
Respond strictly in valid JSON format:
{
"is_first_error_hit": true/false
}
3. Distillation on Correct Rollouts vs. Preserving Prefix Diversity (Weakness 3)
We thank the reviewer for raising this thoughtful conceptual question. We clarify that distilling correct rollouts does not contradict our motivation, but rather reflects a fundamental distinction between harmful external trajectory imitation and beneficial positive self-reinforcement.
For an incorrect rollout (), standard SDPO conditions the teacher on an external, separate reference solution (). Because carries its own specific phrasing and reasoning order, forcing the student to distill over its already-valid prefix compels the model to abandon its own valid derivation and rigidly conform to the external reference template. This overwrites viable alternative reasoning paths and leads to mode collapse.
In contrast, for a correct rollout (), the student model has already generated an intrinsically sound reasoning path. The self-reflector extracts a high-level summary () of the student's own successful reasoning. Distilling under reinforces the student's own discovered reasoning logic rather than forcing it to conform to an alien external template. Crucially, the positive reflection acts as an information bottleneck (Section 3.1) by capturing abstract conceptual principles rather than full token sequences, providing semantic consolidation of the underlying scientific logic without imposing rigid lexical constraints.
Response to Reviewer W7Sy (1/4)
We sincerely thank the reviewer for the positive and encouraging assessment of our work. We greatly appreciate the reviewer's recognition that our problem is important, our approach is concrete, logical, and easy to implement without extra trainable parameters, and our component ablations across three backbones are comprehensive. Below, we address each weakness and question point by point with full respect and thoroughness.
1. Clarification on Performance and Benchmark Scales (Weakness 1)
We appreciate the reviewer's careful examination of the numerical margins across benchmarks in Table 1. On Qwen3-4B, ROSD achieves substantial average improvements of 5.81 percentage points over SDPO (72.83% vs. 67.02%) and 2.97 percentage points over GRPO (72.83% vs. 69.86%), achieving or approaching the top score on all individual tasks.
For all backbones, Table 1 reports the best observed accuracy across checkpoints evaluated during training. Importantly, this selection protocol is favorable to SDPO compared with final-checkpoint evaluation: because SDPO exhibits substantial late-stage performance degradation in Figures 3 and 4, selecting its best score credits it with its strongest observed performance before degradation, rather than evaluating it only after that decline. Even under this favorable comparison, ROSD's advantages extend beyond isolated differences in best-checkpoint accuracy:
Regarding test set scales, the five evaluation benchmarks in our work (SciKnowEval reasoning subsets and ToolAlpaca) are directly adopted from prior on-policy distillation literature [1, 2, 3], enabling comparison under established benchmark settings. Furthermore, every test question is evaluated with 16 independently sampled rollouts (mean@16), reducing generation-level sampling variance relative to single-rollout evaluation on the same question sets. This complements the evidence from three model backbones and five diverse domains. Taken together, the consistent macro-average improvements, sustained advantages across training checkpoints, and mitigation of SDPO's late-stage degradation provide substantive support for ROSD's effectiveness beyond the smaller best-score margins in individual settings.