acceptodds
Under review as a conference paper at ICLR 2027

Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning

Abstract

On-policy self-distillation (OPSD) improves the reasoning capabilities of large language models (LLMs) by providing dense token-level supervision for on-policy rollouts. However, existing OPSD methods often yield limited gains on complex reasoning tasks and suffer from severe training instability. We identify two key causes: conditioning the self-teacher on a complete verified solution encourages imitation of complete reference trajectories rather than extraction of transferable reasoning insights, while indiscriminate full-response distillation imposes superfluous supervision on already-valid reasoning prefixes. Together, these issues suppress reasoning diversity and contribute to late-stage mode collapse. We propose Reflective On-policy Self-Distillation (ROSD), which distills transferable reasoning insights rather than complete reference trajectories. For each erroneous rollout, a self-reflector contrasts it with a correct rollout from the same group to derive a corrective idea and identify the sentence containing the first reasoning error. The corrective idea provides the self-teacher with targeted guidance, while the diagnosed error boundary allows ROSD to mask out the distillation loss over the valid prefix and apply token-level distillation only from the first erroneous sentence onward. Experiments across multiple reasoning benchmarks and model backbones show that ROSD consistently outperforms standard OPSD and reinforcement learning baselines, better preserves reasoning diversity, stabilizes training, and mitigates late-stage mode collapse. Code is available at https://anonymous.4open.science/r/ROSD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-policy distillation, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_W7SyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This study investigates On-Policy Self-Distillation (OPSD) for Large Language Models' (LLMs) reasoning; specifically, the authors focus on Self-Distillation via Policy Optimization (SDPO). The authors argue that SDPO imposes too much distillation pressure on the valid prefix of incorrect rollouts, which reduces diversity and causes late-stage collapse. To address these issues, the authors introduce ROSD. In ROSD, for each incorrect rollout, a single (frozen) model serves as a "reflector." The reflector compares the incorrect rollout to the shortest correct rollout within the group, generates a brief corrective explanation, and identifies the sentence in which the first error occurred. Instead of providing a complete solution, the teacher is conditioned on the explanation generated by the reflector. Additionally, prior to the sentence identified by the reflector as containing the first error, the Jensen-Shannon Divergence (JSD) loss is masked. For correct rollouts, the reflector provides a summary of why the rollout was correct, and the entire response is distilled. The authors evaluate ROSD using four SciKnowEval L3 subsets and ToolAlpaca, using Qwen3-4B/8B and Llama-3.2-3B-Instruct. The authors report that ROSD achieves higher average mean@16 than both GRPO and SDPO, is more stable during training than SDPO, and is more generalizable across domains than SDPO.

Critical strengths
  • The problem addressed is important. Figures 3/4 and Table 5 demonstrate that SDPO can experience significant degradation over extended periods of training. For example, Qwen3-4B trained on ToolUse experiences a drop to a 2.88 cross-domain average. This is a valuable finding for researchers working on self-distillation.
  • The proposed method is easy to implement on top of SDPO. No additional models or trainable parameters are required. The concept of masking everything prior to the first error is logical.
  • The ablation study separates the two components (reflection and localized masking) and evaluates them on all three backbones. Code is provided.
Critical weaknesses
  • The performance gains are small. On Qwen3-8B, ROSD gets 73.45 compared with 72.50 for SDPO. On Llama, it gets 53.38 compared with 52.69 for GRPO. Per task, ROSD is often not the best. The test sets are also small: 50 Biology, 68 ToolUse, 80 Physics, and 94 Material questions.
  • The quality of error localization is never evaluated, even though the entire method relies on the quote being the first error. Fig. 9 only indicates that quotes can be matched to the original text via string matching 57-83% of the time. All other quotes are used in full response distillation, which means a large portion of incorrect rollouts are trained in a manner similar to SDPO with reflection. The paper does not evaluate whether matched quotes are accurate, for example by comparing them to a stronger judge or a small human-labeled set.
  • One of paper’s motivations is that distilling on already-valid reasoning hurts diversity. However, for correct rollouts, ROSD distills every token, with a teacher conditioned on a summary of that same rollout. This seems to go against the paper’s own argument.
Questions for authors
  • The Llama baseline scores of 0.00-0.01 need to be explained.
  • A breakdown of the training times shown in Fig. 7 into rollout, reflection, and teacher forward would provide more detail.
7 comments
AuthorsOpenReview ↗

Response to Reviewer W7Sy (1/4)

We sincerely thank the reviewer for the positive and encouraging assessment of our work. We greatly appreciate the reviewer's recognition that our problem is important, our approach is concrete, logical, and easy to implement without extra trainable parameters, and our component ablations across three backbones are comprehensive. Below, we address each weakness and question point by point with full respect and thoroughness.


1. Clarification on Performance and Benchmark Scales (Weakness 1)

We appreciate the reviewer's careful examination of the numerical margins across benchmarks in Table 1. On Qwen3-4B, ROSD achieves substantial average improvements of 5.81 percentage points over SDPO (72.83% vs. 67.02%) and 2.97 percentage points over GRPO (72.83% vs. 69.86%), achieving or approaching the top score on all individual tasks.

For all backbones, Table 1 reports the best observed accuracy across checkpoints evaluated during training. Importantly, this selection protocol is favorable to SDPO compared with final-checkpoint evaluation: because SDPO exhibits substantial late-stage performance degradation in Figures 3 and 4, selecting its best score credits it with its strongest observed performance before degradation, rather than evaluating it only after that decline. Even under this favorable comparison, ROSD's advantages extend beyond isolated differences in best-checkpoint accuracy:

  1. Sustained Advantage Across Consecutive Steps: In the step-wise learning curves (Figures 3 and 4), ROSD's advantage over SDPO persists across multiple consecutive checkpoints in later training stages. The observed improvement therefore extends beyond a single favorable evaluation point.
  2. Cross-Backbone Consistency: Unlike SDPO, which falls behind GRPO on Qwen3-4B and Llama-3.2-3B, ROSD is the only compared method that achieves the highest in-domain macro-average accuracy on all three backbones spanning two model families. Its improvements over GRPO are 2.97, 1.46, and 0.69 percentage points on Qwen3-4B, Qwen3-8B, and Llama-3.2-3B, respectively.
  3. Superior Optimization Stability and Mitigation of Degradation: Beyond static best scores, Figures 3 and 4 show that ROSD maintains comparatively stable late-stage performance on Chemistry and Material with both Qwen3 backbones, while SDPO exhibits substantial fluctuations or degradation. Complementing these training dynamics, Table 6 demonstrates markedly better cross-domain retention than SDPO: for example, after training Qwen3-4B on ToolUse, ROSD achieves 41.31% cross-domain average accuracy, compared with 2.88% for SDPO.

Regarding test set scales, the five evaluation benchmarks in our work (SciKnowEval reasoning subsets and ToolAlpaca) are directly adopted from prior on-policy distillation literature [1, 2, 3], enabling comparison under established benchmark settings. Furthermore, every test question is evaluated with 16 independently sampled rollouts (mean@16), reducing generation-level sampling variance relative to single-rollout evaluation on the same question sets. This complements the evidence from three model backbones and five diverse domains. Taken together, the consistent macro-average improvements, sustained advantages across training checkpoints, and mitigation of SDPO's late-stage degradation provide substantive support for ROSD's effectiveness beyond the smaller best-score margins in individual settings.

AuthorsOpenReview ↗

Response to Reviewer W7Sy (2/4)

2. Quality Evaluation of Error Localization and Reflection Accuracy (Weakness 2)

We appreciate the reviewer pointing out the necessity of validating the semantic correctness of error localization beyond lexical string matching. To directly evaluate the semantic correctness of the reflector during training, we sampled incorrect rollouts and their corresponding reflector outputs across checkpoints throughout training on Material and Chemistry across Qwen3-4B and Qwen3-8B. We conducted two decoupled evaluation tasks using an independent judge model (Claude-Opus-5-5):

  1. Explanation Accuracy (Overall): Evaluated on the diagnostic explanations (e−e^-) generated for all sampled incorrect rollouts to determine whether the explanation correctly identifies the core reasoning failure and provides scientifically valid guidance.
  2. First-Error Hit Rate (on Matched): Evaluated on the extracted error quotes (qq) that successfully matched the student rollout to determine whether the quoted sentence genuinely marks the exact first error, thereby ensuring all preceding prefix reasoning is fully sound.

Table: Semantic Correctness Evaluation of Reflector During Training

Model BackboneBenchmarkString Match RateFirst-Error Hit Rate (on Matched)Explanation Accuracy (Overall)
Qwen3-4BMaterial56.4%89.5%90.2%
Qwen3-4BChemistry67.7%90.3%92.5%
Qwen3-8BMaterial71.7%89.2%92.5%
Qwen3-8BChemistry82.4%91.5%93.0%

Quantitative Findings on Reflector Quality: The empirical results in the table above demonstrate two critical conclusions regarding the self-reflector's diagnostic quality during training:

  1. High Semantic Precision of Error Localization: Across both model backbones and domains, whenever the reflector's quoted error string successfully matches the original rollout, it accurately pinpoints the exact first logical or factual mistake in 89.2% to 91.5% of cases (e.g., 89.5% on Qwen3-4B Material and 91.5% on Qwen3-8B Chemistry). This high hit rate directly confirms our core hypothesis: masking the distillation loss prior to the diagnosed error boundary reliably protects reasoning prefixes that are genuinely logically sound and scientifically valid, rather than performing arbitrary truncation.
  2. High Diagnostic Accuracy Across All Rollouts: The diagnostic explanation (e−e^-) achieves between 90.2% and 93.0% accuracy across all sampled rollouts (regardless of substring matching success). This confirms that the reflector consistently understands the core reasoning flaw and supplies scientifically valid corrective guidance, providing dependable semantic conditioning for the self-teacher throughout optimization.

Why Reflection Accuracy is High: The reflector operates under a contrastive reflection formulation (Section 3.1). Conditioning on the problem, the shortest verified correct rollout (y∗y^*), and the incorrect rollout (y−y^-) heavily anchors the diagnostic reasoning to the reference solution, making error boundary identification and corrective insight extraction natural and precise.

AuthorsOpenReview ↗

Response to Reviewer W7Sy (3/4)

Evaluation Prompts Used for the Independent Judge

Prompt 1: Explanation Accuracy Evaluation (Evaluated on the diagnostic explanations (e−e^-) generated for all sampled incorrect rollouts)

[System]
You are an expert scientific reasoning evaluator. Your task is to evaluate whether a diagnostic explanation produced by a tutor model correctly identifies a student's reasoning mistake and provides valid corrective logic.

[Problem]
{problem_text}

[Verified Correct Solution]
{correct_solution}

[Incorrect Student Solution]
{student_solution}

[Diagnostic Explanation]
{reflector_explanation}

Task:
Determine whether the [Diagnostic Explanation] is accurate.
- TRUE: It correctly identifies the core reasoning mistake in [Incorrect Student Solution], and its corrective reasoning is scientifically sound and aligned with [Verified Correct Solution].
- FALSE: It misidentifies the mistake, hallucinates an error, or provides flawed corrective logic.

Respond strictly in valid JSON format:
{
  "is_explanation_accurate": true/false
}

Prompt 2: First-Error Hit Evaluation (Evaluated on the extracted error quotes (qq) that successfully matched the student rollout)

[System]
You are an expert academic evaluator. Your task is to determine whether an extracted quote marks the EXACT FIRST logical or factual error in an incorrect student reasoning chain.

[Problem]
{problem_text}

[Verified Correct Solution]
{correct_solution}

[Incorrect Student Solution]
{student_solution}

[Extracted Error Quote]
{reflector_error_quote}

Task:
Determine whether [Extracted Error Quote] is the FIRST error in [Incorrect Student Solution].
- TRUE: The quoted sentence genuinely contains a reasoning/factual mistake, AND all reasoning strictly BEFORE this quote in [Incorrect Student Solution] is completely valid and correct.
- FALSE: The student already made an earlier mistake BEFORE this quote, OR the quote itself is not actually an error.

Respond strictly in valid JSON format:
{
  "is_first_error_hit": true/false
}

3. Distillation on Correct Rollouts vs. Preserving Prefix Diversity (Weakness 3)

We thank the reviewer for raising this thoughtful conceptual question. We clarify that distilling correct rollouts does not contradict our motivation, but rather reflects a fundamental distinction between harmful external trajectory imitation and beneficial positive self-reinforcement.

For an incorrect rollout (y−y^-), standard SDPO conditions the teacher on an external, separate reference solution (y∗y^*). Because y∗y^* carries its own specific phrasing and reasoning order, forcing the student to distill over its already-valid prefix compels the model to abandon its own valid derivation and rigidly conform to the external reference template. This overwrites viable alternative reasoning paths and leads to mode collapse.

In contrast, for a correct rollout (y+y^+), the student model has already generated an intrinsically sound reasoning path. The self-reflector extracts a high-level summary (e+e^+) of the student's own successful reasoning. Distilling y+y^+ under e+e^+ reinforces the student's own discovered reasoning logic rather than forcing it to conform to an alien external template. Crucially, the positive reflection e+e^+ acts as an information bottleneck (Section 3.1) by capturing abstract conceptual principles rather than full token sequences, providing semantic consolidation of the underlying scientific logic without imposing rigid lexical constraints.

Reviewer_bmG1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reflective On-policy Self-Distillation (ROSD) for improving LLM reasoning. For an incorrect on-policy rollout, a self-reflector compares it with a verified-correct rollout to generate a corrective insight and identify the first erroneous sentence. The insight conditions the self-teacher, while an error-dependent mask restricts distillation to the suffix beginning at that sentence. Correct rollouts receive full-response distillation conditioned on a summary of their successful reasoning. Experiments on three model backbones across five scientific reasoning and tool-use tasks report improved average in-domain accuracy and more stable training than SDPO. Validation includes component ablations, diversity measurements, cross-domain evaluation, and training-efficiency analysis.

Critical strengths

A concrete and well-motivated approach to selective supervision. ROSD combines rollout-specific corrective guidance with error-localized distillation. These components address complementary questions: what the teacher should communicate and where the student should receive supervision. The method uses the same base model for reflection and teaching, without requiring a larger external teacher or human-written reference solutions. This makes the proposed training design practically relevant.

Encouraging results across multiple backbones, supported by component ablations. ROSD achieves the highest average in-domain accuracy among the compared methods on all three backbones. On Qwen3-4B, it improves the average score from 69.86% with GRPO and 67.02% with SDPO to 72.83%. Removing reflection or localized distillation lowers this score to 67.56% and 69.73%, respectively. These ablations provide useful evidence that both components contribute to the reported improvement.

Critical weaknesses
  1. Teacher-update differences confound the central stability comparison. Appendix B states that SDPO uses an EMA teacher with update rate 0.05, whereas ROSD freezes its teacher and reflector throughout training. Teacher dynamics could independently affect feedback amplification, overfitting, and late-stage degradation. Consequently, the comparison does not isolate how much of ROSD’s stability advantage comes from reflective guidance and localized distillation. The component ablations are informative, but they do not establish whether the headline comparison with SDPO persists under matched teacher-update policies. This matters because explaining and mitigating self-distillation collapse is a central contribution.

  2. The robustness and scope of the empirical advantage remain insufficiently established. The five in-domain test sets contain only 50–210 questions each, and independent training-seed variability is not reported. The uncertainty bands in Figure 3 describe variation across sampled responses, which does not establish reproducibility across training runs. This is particularly consequential for the smaller average improvements over GRPO on Qwen3-8B and Llama-3.2-3B-Instruct: 1.46 and 0.69 percentage points, respectively. Moreover, GRPO achieves higher cross-domain average accuracy in every setting in Table 5, limiting claims about improved generalizable reasoning. The nearly zero untuned Llama scores also require explanation to assess the evaluation protocol.

Questions for authors
  1. Implementation clarification and controlled comparison: Are ROSD’s teacher and reflector initialized from the base checkpoint and then kept fixed throughout training? How should this be reconciled with the shared-parameter description in Section 3? Does the stability advantage remain when SDPO and ROSD use the same teacher-update policy, particularly when both use frozen teachers?

  2. Validation clarification: How many independent training runs support the reported results, and how were the checkpoints in Table 1 selected? Can you provide uncertainty estimates across training seeds and evaluation questions for the main comparisons, especially the smaller gains over GRPO?

  3. Evaluation clarification: Why does untuned Llama-3.2-3B-Instruct have approximately zero accuracy across all five tasks? Is this primarily due to answer-format or parsing failures? Given that training retains only mixed-outcome groups, how frequently does this model initially generate groups containing a verified-correct response?

7 comments
AuthorsOpenReview ↗

Response to Reviewer bmG1 (1/6)

We sincerely thank the reviewer for the positive evaluation of our motivation, method design, and empirical results across multiple model backbones. We address each concern point by point below.


1. Implementation Clarification, Parameter Sharing, and Teacher Update Policy (W1 + Q1)

1.1 Parameter Initialization and Shared-Parameter Clarification

Addressed Reviewer Comments:

  • "Are ROSD’s teacher and reflector initialized from the base checkpoint and then kept fixed throughout training? How should this be reconciled with the shared-parameter description in Section 3?" (Question 1)

We clarify the exact parameter setup of ROSD:

  • Initialization and Freezing: Both the self-teacher and the self-reflector are initialized directly from the pretrained base checkpoint and remain fixed (frozen) throughout training. The student policy is initialized from the same base checkpoint and updates its parameters via gradient descent.
  • Reconciliation with Section 3: In Section 3, "shared parameters / reusing model weights" refers to the fact that the student, self-teacher, and self-reflector all share the identical base model architecture and initial parameters. ROSD requires no auxiliary models and no external larger teacher model.

To eliminate any potential ambiguity, we have refined the wording in Section 3 and Appendix B of the revised manuscript:

  • Section 3 (Introductory paragraph, Lines 210–211):
    • Original phrasing: "ROSD uses the same base model for student model, reflection and self-teaching, requiring no auxiliary models or additional trainable parameters."
    • Revised phrasing (highlighted in blue): "ROSD reuses the frozen pretrained base model for reflection and self-teaching (sharing the initial model architecture and weights with the initial student model), requiring no auxiliary models."
  • Section 3.1 (Concluding paragraph, Lines 256–257):
    • Original phrasing: "The self-reflector shares the same model weights as the policy and self-teacher, requiring no auxiliary model or additional trainable parameters."
    • Revised phrasing (highlighted in blue): "The self-reflector reuses the frozen pretrained base model weights (sharing the architecture and initial parameters with the initial student model), requiring no auxiliary model."
  • Appendix B (Implementation Details, Lines 731–734):
    • Original phrasing: "In ROSD, the self-teacher and self-reflector remain frozen throughout each training run."
    • Revised phrasing (highlighted in blue): "In ROSD, the self-teacher and self-reflector are initialized directly from the pretrained base checkpoint and remain frozen throughout each training run, ensuring a constant diagnostic and supervisory standard without requiring auxiliary models."

AuthorsOpenReview ↗

Response to Reviewer bmG1 (2/6)

1.2 Ablation Study Under Matched Teacher Update Policies

Addressed Reviewer Comments:

  • "Does the stability advantage remain when SDPO and ROSD use the same teacher-update policy, particularly when both use frozen teachers?" (Question 1)
  • "Teacher-update differences confound the central stability comparison. Appendix B states that SDPO uses an EMA teacher with update rate 0.05, whereas ROSD freezes its teacher and reflector throughout training... The component ablations are informative, but they do not establish whether the headline comparison with SDPO persists under matched teacher-update policies." (Weakness 1)

To address whether ROSD's stability and performance advantages persist under matched teacher update policies, we conducted an ablation study comparing SDPO and ROSD under both frozen teacher and EMA teacher configurations across the representative Chemistry and Materials datasets on Qwen3-4B and Qwen3-8B. The EMA teacher adopts an update rate of 0.05 per step (matching SDPO), while the frozen teacher remains fixed throughout training. All other hyperparameters and training protocols are identical across conditions.

Table: Ablation of Teacher Update Policy on Material and Chemistry (all methods are trained to convergence and scores are reported as mean@16(%))

MethodTeacherQwen3-4B (Material)Qwen3-4B (Chemistry)Qwen3-8B (Material)Qwen3-8B (Chemistry)Average
SDPOFrozen73.4276.6778.0381.0377.29
SDPOEMA (0.05)73.6076.9378.4680.7177.43
ROSDFrozen80.1882.4779.5283.0781.31
ROSDEMA (0.05)79.8782.6479.2382.5781.08

We provide the complete training dynamics in Figure 5 (Appendix C) of the revised manuscript. Based on the quantitative results and training trajectories, we highlight the following key findings:

  1. Teacher update policy has negligible impact on SDPO's performance: SDPO with frozen teacher achieves nearly identical results to the EMA variant (77.29% vs. 77.43% on average, a difference of only 0.14%). In Figure 5, the two SDPO variants show similar overall training trends and both exhibit performance degradation, suggesting that freezing the teacher alone does not resolve SDPO's instability in the evaluated settings. We analyze that this is because SDPO's fundamental limitation stems from full-response trajectory imitation: conditioning the teacher on a complete reference solution forces dense token-level distillation over the entire sequence. This inevitably penalizes valid reasoning prefixes regardless of whether the teacher weights are frozen or updated via EMA, leaving the underlying cause of mode collapse and instability unaddressed.
  2. ROSD maintains consistently superior and stable performance under both teacher policies: ROSD achieves 81.31% with a frozen teacher and 81.08% with an EMA teacher (a negligible difference of only 0.23%), substantially outperforming SDPO under both matched configurations (by 4.02% under frozen teacher and 3.65% under EMA teacher). In Figure 5, both ROSD variants exhibit smooth, stable learning trajectories across training steps without late-stage performance collapse. Crucially, the consistency of this about 3.8% performance advantage across different teacher update rules demonstrates that ROSD's stability and performance gains stem fundamentally from our reflective guidance and error-localized distillation, rather than whether the teacher is updated via EMA or kept frozen. We adopt a frozen teacher as our default design because we consider that keeping both the teacher and self-reflector fixed provides a more consistent evaluation and supervision reference throughout training.
AuthorsOpenReview ↗

Response to Reviewer bmG1 (3/6)

2. Validation Protocol, Checkpoint Selection, and Cross-Domain Generalization (W2 + Q2)

2.1 Checkpoint Selection Protocol and Baseline Fairness

Addressed Reviewer Comments:

  • "How were the checkpoints in Table 1 selected, was an independent validation set used..." (Question 2)

As detailed in Section 4.1 and Appendix B of our manuscript, following the standard evaluation setup on these exact benchmarks in prior on-policy distillation studies [1, 2, 5], no separate validation split is carved out from the target datasets, as the official domain test splits have compact sizes (50–210 questions per domain) where further splitting would compromise evaluation reliability. Instead, all methods across all benchmarks are evaluated periodically at regular intervals (every 10 training steps) on the held-out test sets. For each test question, we draw 16 independent rollout samples and compute the average accuracy (mean@16). The scores reported in Table 1 represent the best performance achieved when models are trained to convergence.

This periodic evaluation protocol directly adheres to the standard practice in LLM post-training and on-policy RL literature [1, 2, 3, 4, 5, 7], where checkpoint performance is tracked periodically throughout optimization.

Importantly, this selection protocol is strictly conservative and highly favorable to baseline methods such as SDPO. Because standard SDPO undergoes severe performance degradation and mode collapse in later training stages (as shown in Figures 3 and 4), selecting its best score captures SDPO at its best operating point before collapse granting SDPO an oracle early-stopping advantage; if evaluated strictly at the final checkpoint, ROSD's margin over SDPO would be substantially larger. Even under this favorable protocol for SDPO, ROSD achieves the highest five-domain average accuracy on all three model architectures (with an average gain of about 5.81% over SDPO on Qwen3-4B), and Figure 3 shows sustained late-stage test-performance advantages over SDPO across consecutive checkpoints in several evaluated settings.

Reviewer_39JgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy self-distillation for reasoning models, arguing that conditioning on complete reference solutions and distilling entire responses may impose unnecessary constraints on reasoning that is already correct. The proposed method, ROSD, selects the shortest correct response among samples for the same question and uses a reflector to contrast it with an incorrect response, produce corrective guidance, and quote the first erroneous sentence. The teacher receives this guidance as additional context, and distillation is applied only from the first erroneous sentence onward. For correct responses, the method extracts the successful reasoning insight and retains full-response distillation. Training does not use a larger external teacher. Experiments cover three models, four scientific multiple-choice domains, and a tool-use task. ROSD achieves the highest average accuracy across the five tasks on all three models. Component ablations support contributions from both reflection and localized distillation, while the appendices also report cross-domain transfer, training time, and localization behavior.

Critical strengths
  1. The method is concrete, with component-level evidence across models. ROSD separates what corrective information to provide from where distillation should begin, without requiring additional trainable parameters. In Table 3, removing either component reduces the five-task average on all three models; for Qwen3-4B, the score drops from 72.83 to 67.56 or 69.73. This provides stronger evidence for the practical value of both components than the main results alone.

  2. The evaluation is reasonably comprehensive, with particularly notable improvements on the smaller model. In Table 1, Qwen3-4B improves from an average of 67.02 with SDPO and 69.86 with GRPO to 72.83 with ROSD; ROSD also obtains the best average on the other two models. Beyond final scores, the authors provide training curves, transfer results, and timing analysis. The cross-domain results in Appendix F show that ROSD substantially alleviates some of SDPO's transfer degradation. For example, Qwen3-4B trained on ToolUse achieves a cross-domain average of 41.31 with ROSD, compared with 2.88 with SDPO. This is a practically meaningful improvement, although GRPO still reaches 43.57 in this setting.

Critical weaknesses
  1. Teacher updating, which is directly relevant to the central stability claim, is not controlled consistently. Appendix B (page 14) explicitly states that SDPO updates its teacher through EMA at a rate of 0.05 per step, whereas ROSD freezes both its teacher and reflector throughout training. Whether the teacher continually tracks the student can itself affect feedback, drift, and late-stage degradation. Consequently, Figures 3–4 and the cross-domain results do not establish that SDPO's degradation is primarily caused by complete reference trajectories and full-response distillation, or that ROSD's stability primarily comes from reflection and error localization. The component ablations in Table 3 are useful, but they do not include standard SDPO with a frozen teacher or ROSD with an EMA teacher, leaving this alternative explanation unresolved. This is my main concern affecting the overall assessment.

  2. The evidence that the method accurately protects valid reasoning prefixes and thereby preserves reasoning diversity is not sufficiently direct. Figure 9 in Appendix I (page 19) measures whether quoted text can be matched to the original response, rather than whether the first-error boundary is semantically correct. The match rate for Qwen3-4B on Material appears to be only about 56%, and the method falls back to full-response distillation when matching fails (page 6). The external-model diagnosis in Appendix D is performed at step 0 and does not substitute for validating the accuracy of the method's own reflector during training. The ablation without localization in Table 3 shows that masking helps, but does not distinguish a semantically correct boundary from generic suffix masking or a reduction in the amount of supervision. Self-BLEU and Distinct-2 in Table 2 measure lexical differences and may be affected by the response-length differences reported in Appendix H; they do not establish that the diversity of correct reasoning paths is preserved. In particular, I find the causal language in Appendix D, which presents larger prefix KL as direct evidence of mode collapse, too strong.

Questions for authors
  1. Under otherwise identical training settings, could you provide standard SDPO with a frozen teacher and ROSD with the same EMA update rule, including full training curves for at least one setting with clear degradation? Do both ablation variants in Table 3 use the same frozen teacher as ROSD? These results would directly affect my interpretation of the stability and performance improvements.

  2. Has the semantic correctness of the error boundaries produced by the method's own reflector during training been independently checked, including the proportion for which the prefix before the identified first error is indeed correct? Does error localization still provide a benefit over fixed-fraction or random suffix masks retaining the same number of tokens? Please also clarify how the loss is normalized after masking, to rule out an explanation based solely on changing the effective amount of supervision.

  3. How were the checkpoints in Table 1 selected, was an independent validation set used, and are the results based on multiple independent training seeds? The caption of Figure 3 describes the shading as the standard deviation across sampled responses, rather than variability across training seeds. Given settings such as the Biology test set with only 50 questions (Table 4), could you provide uncertainty estimates paired at the question level and clarify the discrepancy between step 140 in Table 2 and the reference to the final checkpoint? I would particularly like to establish whether the average gains of 0.95 percentage points for Qwen3-8B over SDPO and 0.69 percentage points for Llama over GRPO are stable.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 39Jg (1/7)

We sincerely thank the reviewer for the thorough and constructive feedback. We address each concern point by point below.


1. Teacher Update Policy and Ablation (W1 + Q1)

We appreciate the reviewer identifying the question regarding teacher update policy. The reviewer notes that SDPO uses an exponential moving average (EMA) teacher while ROSD freezes both the teacher and reflector, and asks whether this difference accounts for the observed stability differences.

To isolate the effect of our proposed method from teacher update strategies, we conducted an ablation study comparing SDPO and ROSD under both frozen teacher and EMA teacher configurations. Consistent with our analysis of training dynamics in the main text (Section 4.3, Figures 3 and 4), we evaluate on the representative Chemistry and Materials datasets across Qwen3-4B and Qwen3-8B. The EMA teacher adopts an update rate of 0.05 per step (matching SDPO), while the frozen teacher remains fixed throughout training. All other hyperparameters and training protocols are identical across conditions.

Table: Ablation of Teacher Update Policy on Material and Chemistry (all methods are trained to convergence and scores are reported as mean@16(%))

MethodTeacherQwen3-4B (Material)Qwen3-4B (Chemistry)Qwen3-8B (Material)Qwen3-8B (Chemistry)Average
SDPOFrozen73.4276.6778.0381.0377.29
SDPOEMA (0.05)73.6076.9378.4680.7177.43
ROSDFrozen80.1882.4779.5283.0781.31
ROSDEMA (0.05)79.8782.6479.2382.5781.08

We provide the complete training dynamics in Figure 5 (Appendix C) of the revised manuscript. Based on the quantitative results and training trajectories, we highlight the following key findings:

  1. Teacher update policy has negligible impact on SDPO's performance: SDPO with frozen teacher achieves nearly identical results to the EMA variant (77.29% vs. 77.43% on average). In Figure 5, the two SDPO variants show similar overall training trends and both exhibit performance degradation, suggesting that freezing the teacher alone does not resolve SDPO's instability in the evaluated settings. We analyze that this is because SDPO's fundamental limitation stems from full-response trajectory imitation: conditioning the teacher on a complete reference solution forces dense token-level distillation over the entire sequence. This inevitably penalizes valid reasoning prefixes regardless of whether the teacher weights are frozen or updated via EMA, leaving the underlying cause of mode collapse and instability unaddressed.
  2. ROSD maintains consistently superior and stable performance under both teacher policies: ROSD achieves 81.31% with a frozen teacher and 81.08% with an EMA teacher (a negligible difference of only 0.23%), substantially outperforming SDPO under both matched configurations (by 4.02% under frozen teacher and 3.65% under EMA teacher). In Figure 5, both ROSD variants exhibit smooth, stable learning trajectories across training steps without late-stage performance collapse. Crucially, the consistency of this about 3.8% performance advantage across different teacher update rules demonstrates that ROSD's stability and performance gains stem fundamentally from our reflective guidance and error-localized distillation, rather than whether the teacher is updated via EMA or kept frozen. We adopt a frozen teacher as our default design because we consider that keeping both the teacher and self-reflector fixed provides a more consistent evaluation and supervision reference throughout training.

Response to Q1 Questions

Q1a: "Do both ablation variants in Table 3 use the same frozen teacher as ROSD?"

Yes. Both ablation variants in Table 3 ("w/o Reflection" and "w/o Localized Distillation") use the same frozen teacher policy as the full ROSD method. As demonstrated by the ablation results above, teacher update policy has minimal effect on either method, and a frozen teacher is simply better aligned with ROSD's frozen self-reflector to maintain consistent reasoning evaluation standards. We have added this clarification to the caption of Table 3 in the revised manuscript.

Q1b: "Under otherwise identical training settings, could you provide standard SDPO with a frozen teacher and ROSD with the same EMA update rule, including full training curves?"

Yes. The complete quantitative results and training dynamics under matched teacher update policies are detailed in the table above and in Appendix C (Table 5 and Figure 5) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 39Jg (2/7)

2. Error Localization Quality, Semantic Accuracy, and Diversity (W2 + Q2)

We appreciate the reviewer's insightful questions regarding the semantic accuracy of the self-reflector, the comparison against positional masking baselines, the loss normalization mechanism, and the interpretation of diversity metrics. We address each point thoroughly below.


2.1 Clarification on Appendix E and Semantic Accuracy of Reflector During Training

Addressed Reviewer Comments:

  • "Has the semantic correctness of the error boundaries produced by the method's own reflector during training been independently checked, including the proportion for which the prefix before the identified first error is indeed correct?" (Question 2)
  • "Figure 9 in Appendix I measures whether quoted text can be matched... rather than whether the first-error boundary is semantically correct... The external-model diagnosis in Appendix D is performed at step 0 and does not substitute for validating the accuracy of the method's own reflector during training." (Weakness 2)

Clarification of the Role of Appendix E: We first clarify the distinct objective of Appendix E. Appendix E is a motivation-oriented pilot study conducted at Step 0 using an impartial adjudicator (Claude-Opus-5) to diagnose standard SDPO. Its sole purpose is to reveal the structural flaw of full-response distillation: standard SDPO exerts excessively high KL penalties on valid prefixes, frequently exceeding the supervision on the actual mistake. Appendix E was never intended as an evaluation of the training-time self-reflector.

Why the Self-Reflector Accurately Pinpoints Errors: During ROSD training, the self-reflector is initialized from the same base checkpoint as the student and remains frozen throughout training, while the student policy is updated. The reflector operates under a contrastive reflection formulation (Section 3.1): given the problem, the shortest rollout with a verified-correct final answer from the current student's rollout group (y∗y^*), and an incorrect rollout (y−y^-), it generates corrective guidance and identifies the first erroneous sentence in y−y^-. This reference-conditioned setup provides additional context for diagnosis; we evaluate the semantic accuracy of the resulting explanations and error boundaries below.

Independent Verification via LLM-as-a-Judge: To directly verify the semantic correctness of the reflector during training, we sampled incorrect rollouts and their corresponding reflector outputs across checkpoints throughout training on Material and Chemistry across Qwen3-4B and Qwen3-8B. We conducted two decoupled evaluation tasks using an independent judge model (Claude-Opus-5-5):

  1. Explanation Accuracy (Overall): Evaluated on the diagnostic explanations (e−e^-) generated for all sampled incorrect rollouts to determine whether the explanation correctly identifies the core reasoning failure and provides scientifically valid guidance.
  2. First-Error Hit Rate (on Matched): Evaluated on the extracted error quotes (qq) that successfully matched the student rollout to determine whether the quoted sentence genuinely marks the exact first error, thereby ensuring all preceding prefix reasoning is fully sound.

Table: Semantic Correctness Evaluation of Reflector During Training

Model BackboneBenchmarkString Match RateFirst-Error Hit Rate (on Matched)Explanation Accuracy (Overall)
Qwen3-4BMaterial56.4%89.5%90.2%
Qwen3-4BChemistry67.7%90.3%92.5%
Qwen3-8BMaterial71.7%89.2%92.5%
Qwen3-8BChemistry82.4%91.5%93.0%

Quantitative Findings on Reflector Quality: The empirical results in the table above demonstrate two critical conclusions regarding the self-reflector's diagnostic quality during training:

  1. High Semantic Precision of Error Localization: Across both model backbones and domains, whenever the reflector's quoted error string successfully matches the original rollout, it accurately pinpoints the exact first logical or factual mistake in 89.2% to 91.5% of cases (e.g., 89.5% on Qwen3-4B Material and 91.5% on Qwen3-8B Chemistry). This high hit rate directly confirms our core hypothesis: masking the distillation loss prior to the diagnosed error boundary reliably protects reasoning prefixes that are genuinely logically sound and scientifically valid, rather than performing arbitrary truncation.
  2. High Diagnostic Accuracy Across All Rollouts: The diagnostic explanation (e−e^-) achieves between 90.2% and 93.0% accuracy across all sampled rollouts (regardless of substring matching success). This confirms that the reflector consistently understands the core reasoning flaw and supplies scientifically valid corrective guidance, providing dependable semantic conditioning for the self-teacher throughout optimization.
AuthorsOpenReview ↗

Response to Reviewer 39Jg (3/7)

Robustness via Fallback Mechanism: When string matching fails (e.g., due to minor whitespace or formatting discrepancies), ROSD conservatively falls back to full-response distillation (k=1k=1) to prevent improper prefix truncation. Crucially, as shown by the high Explanation Accuracy (90.2% to 93.0%), even in fallback cases, the self-teacher is still conditioned on a semantically accurate corrective idea (e−e^-), ensuring robust learning across all samples.

Evaluation Prompts Used for the Independent Judge

Prompt 1: Explanation Accuracy Evaluation (Evaluated on reflections generated for all sampled incorrect rollouts)

[System]
You are an expert scientific reasoning evaluator. Your task is to evaluate whether a diagnostic explanation produced by a tutor model correctly identifies a student's reasoning mistake and provides valid corrective logic.

[Problem]
{problem_text}

[Verified Correct Solution]
{correct_solution}

[Incorrect Student Solution]
{student_solution}

[Diagnostic Explanation]
{reflector_explanation}

Task:
Determine whether the [Diagnostic Explanation] is accurate.
- TRUE: It correctly identifies the core reasoning mistake in [Incorrect Student Solution], and its corrective reasoning is scientifically sound and aligned with [Verified Correct Solution].
- FALSE: It misidentifies the mistake, hallucinates an error, or provides flawed corrective logic.

Respond strictly in valid JSON format:
{
  "is_explanation_accurate": true/false
}

Prompt 2: First-Error Hit Evaluation (Evaluated on rollouts where string match succeeded)

[System]
You are an expert academic evaluator. Your task is to determine whether an extracted quote marks the EXACT FIRST logical or factual error in an incorrect student reasoning chain.

[Problem]
{problem_text}

[Verified Correct Solution]
{correct_solution}

[Incorrect Student Solution]
{student_solution}

[Extracted Error Quote]
{reflector_error_quote}

Task:
Determine whether [Extracted Error Quote] is the FIRST error in [Incorrect Student Solution].
- TRUE: The quoted sentence genuinely contains a reasoning/factual mistake, AND all reasoning strictly BEFORE this quote in [Incorrect Student Solution] is completely valid and correct.
- FALSE: The student already made an earlier mistake BEFORE this quote, OR the quote itself is not actually an error.

Respond strictly in valid JSON format:
{
  "is_first_error_hit": true/false
}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.