acceptodds
Under review as a conference paper at ICLR 2027

ReTeach: Building a Better Self-Teacher through Multi-Round Reflection and Retry

Abstract

Self-distillation can improve reasoning without a separately trained, more capable teacher, but its effectiveness depends on how the self-teacher gains an advantage over the student. Conditioning the teacher on reference answers or solutions can provide such an advantage, but this information may be unavailable. Reflection offers a way to derive explicit error diagnoses and revision guidance from self-generated attempts, yet existing reflection-based methods often combine it with reference information, rich task feedback, or persistent memory. We introduce ReTeach, a Reflective self-distillation framework that constructs its self-Teacher through multi-round reflection and retry using only self-generated attempts and outcome-level verification. Starting from an unsuccessful student rollout, the teacher alternates explicit reflection with renewed attempts until success or the retry budget is exhausted, without reference answers or solutions, external diagnostic feedback, or cross-example memory. Each failed retry informs subsequent reflection, while successful correction provides outcome-level evidence for the potential utility of the resulting teacher context. An outcome-aware selection and weighting strategy distinguishes initially correct, reflection-corrected, and unresolved examples, assigning separate weights to their category-normalized distillation losses. Through on-policy distillation, the student matches the teacher's context-conditioned token-level predictive distributions at prefixes of its own rollouts, transferring the benefits of iterative correction while retaining single-pass inference. Across six benchmarks spanning mathematical reasoning, science question answering, and tool use, ReTeach improves average accuracy over GRPO by 1.39 percentage points. Among all compared methods, including those using privileged information, ReTeach achieves the highest average accuracy on five benchmarks and ranks second on the remaining one. These results demonstrate the effectiveness of constructing reflective self-teachers from self-generated experience and outcome-level verification to improve single-pass reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Distillation, Large Language Model Reasoning, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hNhGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduced ReTeach, a multi-round reflection-and-retry process for self-teacher construction. Unlike related works that discard wrong trails or stop at a retrospective diagnosis, it asks the model to keep generating new responses until it reaches the correct answer or cost limit. This enables the student model to learn from reflective contexts without access to reference answers or solutions.

Critical strengths
  1. The paper is well-written and uses a straightforward way to explain its methods and analysis. Figure 1 and 2 are clear in showing the related work comparison and its own method.
  2. The motivation sounds interesting, combining self-evolution and self-distillation. One big problem of OPSD is deciding what to give as the hint in the teacher prompt, which normally tends to be gold answer or solution. ReTeach uses student model's evolving path to replace it and provide reflection signals.
  3. The experiments are sound. Results cover mathematics, science QA, and tool use, comparing against GRPO, OPSD, RLSD, RESD, and CEPO.
Critical weaknesses
  1. ReTeach requires multiple rounds of teacher generations and explicit reflection. However, the paper does not report training time, GPU-hours, or token consumption against the baselines. This analysis will make it easier to tell whether the improvements justify the additional cost.
  2. More analysis about reflection versus additional responses would be great. For example, what if we only ask the teacher model to generate more inferences without any reflection? Now it is unclear how much improvement comes from reflection rather than additional inference opportunities.
  3. I understand that designing ablation studies that change Wa, Wb, Wc in equation 11 at the same time would be difficult. And Figure 3 already shows the results when fixing Wa, Wb and changing Wc. However, it'd be great to see more discussion about them. For example, why fix both Wa, Wb=0.5? Is there any limitation on the relationship between these 3 weights, as the sum should be <= 1? Similarly, now λ in equation 9 also only covers 0.9999 (default), 0 (tracking), 1(frozen). It would be interesting to see, like λ=0.5 curves in Figure 4.
Questions for authors
  1. How much of the improvement comes from the teacher's original thinking mode, rather than reflection?
  2. If the work claims it doesn't need access to the gold answer, how can the verifier check whether the current teacher-generated reflection response is correct or not?
5 comments

We sincerely thank Reviewer hNhG for the detailed feedback and for recognizing our paper “the motivation sounds interesting” and “the experiments are sound”. We also appreciate the reviewer’s acknowledgment that "the paper is well written". For weaknesses and questions, we address each point below.

W1: ReTeach requires multiple rounds of teacher generations and explicit reflection. However, the paper does not report training time, GPU-hours, or token consumption against the baselines. This analysis will make it easier to tell whether the improvements justify the additional cost.

1. [W1]: Computational Cost of ReTeach

We thank the reviewer for the opportunity to clarify the computational cost of ReTeach and its relationship to the performance gains. Following each method’s original setup, the methods are implemented using different frameworks. For a fair comparison, we evaluate all methods on the SciKnowEval-Physics dataset and report the actual generated tokens and rollouts per training step. For ReTeach, the token count includes the initial student responses, the teacher’s native thinking, explicit reflection, and all retries, including unsuccessful ones. The GRPO-based baselines use eight student rollouts per question with a global batch size of 32.

MethodsRollouts / StepGenerated Tokens / StepAvg@16 (%)
ReTeach53139,34364.24±1.0564.24 \pm 1.05
GRPO256408,52661.96±1.2561.96 \pm 1.25
CEPO256170,46563.39±1.4563.39 \pm 1.45
RLSD256253,83161.90±0.8361.90 \pm 0.83
RESD4021,644,27161.09±0.5461.09 \pm 0.54

The results demonstrate that ReTeach achieves the highest Avg@16 with substantially fewer rollouts and a comparable overall generation budget.

W2: More analysis about reflection versus additional responses would be great. For example, what if we only ask the teacher model to generate more inferences without any reflection? Now it is unclear how much improvement comes from reflection rather than additional inference opportunities.

2. [W2]: Improvement from Explicit Reflections

We thank the reviewer for the opportunity to clarify the contribution of explicit reflection. We respectfully clarify that our "w/o Reflection" ablation exactly implements the setting suggested by the reviewer, retaining additional teacher retries in thinking mode while removing the explicit <reflect>...</reflect> tags and their enclosed reflection content. On the SciKnowEval-Physics dataset, the w/o Reflection configuration in Figure 4 achieves an Avg@16 of 62.91±0.6762.91 \pm 0.67. Adding explicit reflection improves performance to 64.24, an increase of 1.33 percentage points. The results suggest that explicit reflection provides more useful teaching context by locating errors and reasoning toward a correct solution, helping improve the model’s single-attempt accuracy through self-distillation.

W3: I understand that designing ablation studies that change Wa, Wb, Wc in equation 11 at the same time would be difficult. And Figure 3 already shows the results when fixing Wa, Wb and changing Wc. However, it’d be great to see more discussion about them. For example, why fix both Wa, Wb=0.5? Is there any limitation on the relationship between these 3 weights, as the sum should be <= 1? Similarly, now λ in equation 9 also only covers 0.9999 (default), 0 (tracking), 1(frozen). It would be interesting to see, like λ=0.5 curves in Figure 4.

3. [W3]: Hyperparameter Analysis

We thank the reviewer for these suggestions regarding our hyperparameter settings, which help us make the analysis more rigorous. Below, we discuss the two types of parameters in turn and will incorporate the corresponding discussion into the revised manuscript.

3.1 Sample weights.

Equation 10 first averages the loss within each category, and Equation 11 combines the category-level losses as

= w_A\mathcal{L}_A + w_B\mathcal{L}_B + w_C\mathcal{L}_C.$$ The weights $w_A$, $w_B$, and $w_C$ are nonnegative loss coefficients, not category probabilities. Their sum therefore need not equal 1 or be bounded by 1. Our default choice of $w_A=w_B=0.5$ assigns equal category-level weight to preserving initially correct behavior and learning from successfully corrected attempts. Choosing $w_B \geq w_A$ is a reasonable option when prioritizing the supervision obtained through teacher reflection. #### 3.2 EMA update coefficient. We respectfully clarify that $\lambda=0.5$ assigns half of the weight to the current student at every update, providing limited temporal smoothing for the teacher. Following the reviewer’s suggestion, and given the time available for additional training, we report Avg@16 on SciKnowEval-Physics for $\lambda=0.95$, averaged over two random seeds. We extend the analysis in Figure 4 with the results below. | $\lambda$ Value | Step 100 | Step 200 | Step 300 | Step 400 | | --- | :---: | :---: | :---: | :---: | | $\lambda=0$ (Tracking Teacher) | $58.80 \pm 1.67$ | $59.27 \pm 0.73$ | $56.56 \pm 1.05$ | $55.83 \pm 0.63$ | | $\lambda=1$ (Frozen Teacher) | $60.94 \pm 0.68$ | $62.71 \pm 0.27$ | $63.02 \pm 0.44$ | $63.31 \pm 0.23$ | | $\lambda=0.95$ | $60.24 \pm 0.22$ | $60.55 \pm 1.22$ | $60.82 \pm 0.72$ | $61.73 \pm 0.21$ | | $\lambda=0.9999$ (Default Setting) | $62.00 \pm 0.74$ | $63.12 \pm 0.64$ | $63.39 \pm 1.23$ | $64.24 \pm 1.36$ | With $\lambda=0.95$, ReTeach reaches 61.73, improving on the Qwen3-8B base model’s 59.84 by 1.89 percentage points. It shows sustained improvement across the evaluated training steps compared with the tracking teacher. Our default setting further increases $\lambda$ to balance the stability of the teacher’s reflection with gradual adaptation to student updates, achieving the best final performance of 64.24. We will extend the evaluation to additional random seeds and update Figure 4 in the revised manuscript.

Q1: How much of the improvement comes from the teacher’s original thinking mode, rather than reflection?

4. [Q1]: Improvement by Thinking Mode in Teacher

We thank the reviewer for the opportunity to clarify our ablation setup. In the w/o Reflection experiment, we retain the teacher’s native thinking mode and iterative retries based on previous failed trajectories. We remove only the explicit <reflect>...</reflect> block and its enclosed content. The teacher reasons over the history in thinking mode, and then outputs the solution and final answer without a separate explicit reflection block. The table below reports Avg@16 (%) on SciKnowEval-Physics.

DatasetQwen3-8BGRPORLSDRESDCEPOReTeach w/o ReflectionReTeach
SciKnowEval-Physics59.84±0.3259.84 \pm 0.3261.96±1.2561.96 \pm 1.2561.90±0.8361.90 \pm 0.8361.09±0.5461.09 \pm 0.5463.39±1.4563.39 \pm 1.4562.91±0.6762.91 \pm 0.6764.24±1.0564.24 \pm 1.05

With native thinking and history-based retries, ReTeach w/o Reflection already achieves an Avg@16 of 62.91, outperforming GRPO, RLSD, and RESD. Its improvement over the initial model suggests that multi-round teacher reasoning can provide useful supervision for improving single-attempt responses. Adding the explicit <reflect>...</reflect> block further structures this process by asking the teacher to identify the causes of previous errors and articulate how to correct them before producing a solution. The resulting teaching context further improves performance to 64.24, a gain of 1.33 percentage points.

Q2: If the work claims it doesn’t need access to the gold answer, how can the verifier check whether the current teacher-generated reflection response is correct or not?

5. [Q2]: The Verifier Design

We thank the reviewer for the opportunity to clarify the verifier design. Our claim is about what the model observes during training. In our experiments, the verifier compares every generated response, from the initial student responses to each teacher reflection-and-retry response, against the reference answer and returns a binary judgment (correct or incorrect). The student and the teacher receive only this correctness signal, and the reference answers remain private to the verifier throughout training.

For comparison, in RESD the reflector receives the failed response together with feedback that explicitly contains the correct answer, which lets the model work backward from the given answer to build the solution steps. ReTeach keeps the reference answer private to the verifier. The teacher therefore has to reason through each correction on its own, guided only by whether each attempt is judged correct.

Our verifier design offers two benefits. First, the student learns from corrections developed through the teacher’s own problem-solving process without reference-answer leaking. Second, ReTeach is compatible with any black-box verifier that returns binary judgments, including rule-based answer matching, unit tests, and external grading services. Training can therefore proceed while reference answers remain confidential. We will clarify this distinction in the revised manuscript.

Reviewer_CBQMOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies self-distillation by constructing a stronger teacher from the model itself. Specifically, the student first generates an answer. If verification fails, the teacher reflects on previous unsuccessful attempts and retries, up to KK rounds. The samples are then divided into three categories: initially correct, corrected through reflection, and unresolved, with different training weights. The teacher uses the reflection history to provide token-level supervision, and the student is trained using Jensen–Shannon divergence. Experiments on Qwen3-8B show an average accuracy improvement of approximately 1.4 percentage points over GRPO across six benchmarks.

Critical strengths
  1. The experiments report results across multiple random seeds, showing relatively consistent performance improvements across several benchmarks.

  2. The proposed method is simple and straightforward, and constructing stronger self-teachers without relying on reference answers or solutions is an interesting and worthwhile research direction.

Critical weaknesses
  1. Reflection, retry, outcome-based filtering, EMA teachers, and self-distillation are all established techniques. The main contribution appears to be their combination. The paper should more clearly distinguish its methodological novelty from closely related approaches, particularly ROSD [1] and RESD [2], and explain what new insights this combination provides.

  2. Multi-round reflection and retry introduce substantial additional computational costs, which are not adequately accounted for in the comparisons. Could simpler alternatives, such as increasing the number of GRPO rollouts or using best-of-N self-training, achieve comparable improvements? Compute-matched baseline comparisons are needed to establish the benefits of the proposed approach.

  3. The improvements over CEPO are relatively modest, and ReTeach underperforms CEPO on SciKnowEval-Material. The paper provides limited analysis of why the method improves performance in some settings but not others. A deeper investigation into the sources of these gains and the conditions under which the method fails would strengthen the empirical findings.

References:

[1] Zhao Z, Ma X, Yang L, et al. Rosd: Reflective on-policy self-distillation for language model reasoning across domains[J]. arXiv preprint arXiv:2605.28014, 2026.

[2] Zhang Y, Li S, Yu C, et al. Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation[J]. arXiv preprint arXiv:2605.12741, 2026.

Questions for authors
  1. Could the authors provide additional ablations and compute-matched comparisons, as discussed in Weaknesses 1 and 2, to better establish the effectiveness of the proposed method?

  2. Could the authors provide more intuition and analysis explaining where the performance gains come from, why the method works, and under what conditions it may fail?

4 comments

We sincerely thank Reviewer CBQM for the thoughtful feedback and for recognizing that our work explores "an interesting and worthwhile research direction". We also appreciate the reviewer’s acknowledgment that our experimental results show "relatively consistent performance improvements across several benchmarks". For weaknesses and questions, we address each point below.

W1: Reflection, retry, outcome-based filtering, EMA teachers, and self-distillation are all established techniques. The main contribution appears to be their combination. The paper should more clearly distinguish its methodological novelty from closely related approaches, particularly ROSD [1] and RESD [2], and explain what new insights this combination provides.

1. [W1] Novelty and New Insights

We thank the reviewer for encouraging us to clarify our novelty. We agree that the individual components are established; our contribution is not their mere combination, but using multi-round, per-question reflection and retries to construct a self-teacher’s advantage under binary-only feedback, then transferring this advantage to single-attempt responses.

Compared with ROSD and RESD, ReTeach requires less external information and a simpler teacher-side structure: ROSD uses reference CoTs and answers to guide reflection, while RESD leverages richer environmental feedback and maintains a persistent playbook with an optional successful-rollout cache. Although both methods use reflection, their reflection-based correction is single-round, whereas ReTeach performs multiple rounds of reflection and retries on the same question. Each failed attempt informs the next round of reflection and correction, progressively developing useful corrective context on the same question. Thus, ReTeach uses iterative problem solving to build the teacher’s instructional advantage, reducing reliance on reference solutions and persistent memory modules.

Our key insight is that multi-round reflection and retries can enable a model to solve questions it fails to solve in its initial attempt, thereby creating a more capable self-teacher. Verified successful retries then provide distillation signals that transfer this multi-round problem-solving advantage to the student’s single-attempt responses.

We thank the reviewer for recognizing the value of our work. We will incorporate the above analysis of our methodological novelty and key insight into the revised manuscript.

W2: Multi-round reflection and retry introduce substantial additional computational costs, which are not adequately accounted for in the comparisons. Could simpler alternatives, such as increasing the number of GRPO rollouts or using best-of-N self-training, achieve comparable improvements? Compute-matched baseline comparisons are needed to establish the benefits of the proposed approach.

Q1: Could the authors provide additional ablations and compute-matched comparisons, as discussed in Weaknesses 1 and 2, to better establish the effectiveness of the proposed method?

2. [W2&Q1] Compute-Matched Comparison

We thank the reviewer for raising this point, which gives us the opportunity to clarify the computational cost of ReTeach. We report the per-step generation cost on SciKnowEval-Physics for ReTeach and all baselines. For ReTeach, the token count covers every generation within a training step: The initial student responses, the teacher’s native thinking, explicit reflection, and all retries, including unsuccessful ones. For the GRPO-based baselines, we sample 8 student rollouts per question with a global batch size of 32.

MethodsTotal Rollouts / StepGenerated Tokens / Step
ReTeach53139,343
GRPO256408,526
CEPO256170,465
RLSD256253,831
RESD4021,644,271
  • Cost allocation for reflection and retry: To directly address the reviewer’s concern about the cost of reflection and retry, we further break down the tokens generated by ReTeach by source. Tokens generated during the teacher’s thinking, reflection, and retries account for 117,520 of the 139,343 tokens generated per step, i.e., 84.34%. The remaining 21,823 tokens come from the initial student responses.

  • Cost comparison with other baselines: Even with all teacher-generated tokens included, ReTeach generates fewer tokens per step on average than all baselines. ReTeach also requires substantially fewer generation requests, averaging 53 per step compared with hundreds of rollouts for the baselines.

Overall, ReTeach achieves its reported gains with a generation budget comparable to existing methods. Its total token consumption is even slightly lower than that of most baselines, which sample multiple rollouts for every question.

W3: The improvements over CEPO are relatively modest, and ReTeach underperforms CEPO on SciKnowEval-Material. The paper provides limited analysis of why the method improves performance in some settings but not others. A deeper investigation into the sources of these gains and the conditions under which the method fails would strengthen the empirical findings.

Q2: Could the authors provide more intuition and analysis explaining where the performance gains come from, why the method works, and under what conditions it may fail?

3. [W3&Q2] Deep Analysis of ReTeach’s Applicability

We thank the reviewer for the opportunity to clarify the sources of ReTeach’s advantages and limitations. Following your suggestion, we explain these results through the teacher’s knowledge and its ability to correct errors.

3.1 Sources of improvement.

ReTeach benefits from a self-teacher strengthened through multi-round, per-question reflection and retries, enabling it to turn initial failures into successful corrections and provide more useful supervision. CEPO, and RESD when reference answers are included in its feedback, construct supervision with the correct answer already known, using that answer to identify the reasoning that should support it. ReTeach instead starts from the question and its own failed attempts, requiring the teacher to discover corrections and reach the answer through its own reasoning, guided only by binary verification. The example in Section 3.3 further illustrates ReTeach’s improved problem-solving behavior.

3.2 Why the benefits vary across tasks.

ReTeach is expected to work best when the teacher’s knowledge and capabilities match the task’s requirements, enabling reflection and self-distillation to improve single-attempt accuracy. On Material, it scores 69.17 versus CEPO’s 70.83. This gap may reflect domain knowledge that the teacher struggles to recover through reflection alone. reference-answer conditioning in CEPO provides additional guidance in such cases.

3.3 Advantages of ReTeach on Cross-dataset generalization.

ReTeach exceeds CEPO in Avg@12 by 2.50 percentage points on AIME24, 1.94 on AIME25, and 3.51 on HMMT25, supporting stronger cross-dataset generalization in our evaluation. To complement these aggregate results, we present an AIME24 example comparing CEPO and ReTeach. Bold annotations in the table were added by us for clarity.

Problem. Among the 900 residents of Aimeville, 195 own a diamond ring, 367 own a set of golf clubs, and 562 own a garden spade. Every resident owns candy hearts. There are 437 residents who own exactly two of these items and 234 who own exactly three. How many residents own all four?

CEPO: incorrect answerReTeach: correct answer
[1. Assert 113 before deriving it.] … “This value, 113\boxed{113}, is not arbitrary or purely numerical.” … [2. Assign 447 to the candidate answer.] Unique Individuals who own all four items: 900−453=447900 - 453 = 447. [3. Immediately reassert 113 without resolving the contradiction.] “The Final and Unifying Consistent Value: 113\boxed{113}” … “Hence, 113\boxed{113} is the only correct answer, as it is the only value that is logically and mathematically aligned with the original constraints.”… [1. Define the problem.] A resident who owns exactly two items must have candy hearts plus one of the other three items. A resident who owns exactly three items must have candy hearts plus two of the other three items. … gg: number of residents owning all four items. … [2. Solve it step by step.] 437 residents own exactly two items: a+b+c=437a+b+c=437; 234 residents own exactly three items: d+e+f=234d+e+f=234. … Total diamond ring owners: a+d+e+g=195a+d+e+g=195; total golf club owners: b+d+f+g=367b+d+f+g=367. … Left: a+b+c+2d+2e+2fa+b+c+2d+2e+2f, which equals 437+2×234=905437+2\times234=905. Right: 195+367+562−3g=1124−3g195+367+562-3g=1124-3g. Set equal: 905=1124−3g905=1124-3g, so 3g=2193g=219 and g=73g=73. … 73\boxed{73}

ReTeach explicitly accounts for the fact that every resident owns candy hearts, translates the remaining ownership conditions into counting equations, and derives the correct answer of 7373 step by step. The response illustrates how ReTeach can elicit detailed reasoning before producing the final answer.

3.4 Failure conditions and practical implications.

ReTeach may struggle when the teacher lacks essential domain knowledge, causing reflection to repeat errors rather than correct them. Stronger base models with better domain knowledge and error-diagnosis capabilities may provide better supervision. Balanced training data and a progression from foundational to harder tasks may also help establish the knowledge needed for effective correction.

We will incorporate the above analysis into the revised manuscript.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer CBQM,

We have posted our response to your initial review and updated the manuscript to address your concerns. As the discussion period progresses, we would greatly appreciate it if you could take a moment to check our rebuttal. We are happy to answer any further questions or conduct additional clarifications if needed.

Your insights will be invaluable in refining the final version of our work. Thank you once again for your time and effort!

Thanks,

Authors

Dear Reviewer CBQM,

As the discussion period closes in a few hours, we would like to send a brief reminder. We addressed each of your questions in our earlier replies, and the corresponding changes are included in the rebuttal revision PDF. If you feel that any of your concerns remain unresolved, we would be glad to discuss them before the deadline :)

Thanks,

Authors

Reviewer_UbxROpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReTeach builds a self-teacher by reflecting on failed student attempts and retrying them with binary correctness feedback. It then uses the failed attempts and final deliberation to supervise the student’s original response at the token level. The teacher is updated through an exponential moving average, with different weights for initially correct, corrected, and unresolved examples. The student answers without reflection or retries at inference time. Experiments with Qwen3-8B span six math, science QA, and structured tool-call benchmarks. ReTeach reports the highest mean score on five of them and an average gain of 1.39 percentage points over GRPO.

Critical strengths
  1. The idea of using failed attempts to build a better teacher is interesting. Reflection gives the teacher context, and retries check whether that context helps it reach a correct answer. This provides token-level supervision without giving the teacher reference answers or solutions, while keeping the student’s inference process simple.
  2. The ablations helped me understand which parts of the method matter. They cover reflection, teacher updates, retry rounds, and unresolved examples, and the main results span multiple domains with three seeds. Keeping retries but removing explicit reflection drops Physics accuracy from 64.24 to 62.91, which supports including reflection.
Critical weaknesses
  1. My main concern is that the evaluation results were used to choose the category weights. Appendix A says that w_C was selected using the benchmark results in Figure 3, without a separate validation split. On ToolAlpaca, changing w_C from 0 to the selected value of 0.2 raises the score from 60.69 to 62.72. Given the average gain of 1.35 points over CEPO, I am not convinced that the reported improvement would hold on data that was not used to choose the weights.
  2. It would be great to see a direct comparison with SDPO. It already supports self-distillation from binary feedback using successful model-generated rollouts, including Qwen3-8B experiments on SciKnowEval and ToolAlpaca. Yet Figure 1 and Appendix A suggest that SDPO requires reference answers, and it is absent from Table 1. Without this comparison, I cannot tell how much reflection and retry add over an existing self-teacher with access to the same feedback.
Questions for authors
  1. Do Figure 3 and Table 1 use the same evaluation examples? Do you have results where w_C was chosen on separate data? Were these benchmarks also used to select checkpoints or other hyperparameters?
  2. Why was SDPO’s binary-feedback variant left out, especially since you use its datasets and evaluation protocol? Do you have a comparison under the same training setup?
  3. In the “w/o Reflection” ablation, does the teacher’s distillation prefix still include the native <think> block, and is thinking mode unchanged? The full method includes both <think> and <reflect>, so I want to understand exactly what this ablation removes.
  4. How many tokens were generated, or how many GPU-hours were used, for ReTeach and each baseline, including teacher reflection and retries? With different rollout schedules, I cannot tell how the methods compare at a similar training cost.
6 comments

We sincerely thank the reviewer UbxR for the detailed feedback and for finding our idea of “using failed attempts to build a better teacher” interesting. For weaknesses and questions, we address each point below.

W1: My main concern is that the evaluation results were used to choose the category weights. Appendix A says that w_C was selected using the benchmark results in Figure 3, without a separate validation split. On ToolAlpaca, changing w_C from 0 to the selected value of 0.2 raises the score from 60.69 to 62.72. Given the average gain of 1.35 points over CEPO, I am not convinced that the reported improvement would hold on data that was not used to choose the weights.

1. [W1] Category Weight Parameters

Thank you for raising this concern. wC=0w_C=0 is our default setting, used on all datasets other than ToolAlpaca. The hyperparameter search over category weights in Figure 3 was motivated by data quality issues we encountered in ToolAlpaca, which can cause useful rollouts to be incorrectly rejected and assigned to Category C. We provide an example below. The question refers to “today,” but the reference call hard-codes the dataset construction date, 2020-09-14.

Question: I want to know what's airing on Czech TV channels this week. Can you fetch the weekly TV program for me starting from today's date?

Reference Answer:

[{"Action": "getWeeklyTVProgram", "Action_Input": "{\"startDate\": \"2020-09-14\"}"}]

The dataset also contains other issues, such as reference tool calls with varying optional parameter values that cannot be inferred directly from the prompt. These issues therefore cannot be fully resolved by simply modifying the verification function used during rollouts. In our original submission, we followed the experimental setup of SDPO on the dataset to compare with baselines. We have now cleaned the ToolAlpaca training set, retaining 1,823 examples (45.1% of the original dataset), and are rerunning the baseline comparisons with ReTeach’s wCw_C fixed at its default value of 0. Given the limited time available, we have so far completed experiments with a single random seed and report the avg@16 results in the table below. We are running additional seeds and will update the table with multi-seed means and standard deviations once these experiments are complete.

MethodQwen3-8BGRPORLSDRESDCEPOReTeach (ours)
Tooluse-ToolAlpaca-Clean57.960.5760.6660.8561.0361.76

We observe that ReTeach outperforms all evaluated baselines on ToolAlpaca-Clean with wC=0w_C=0, indicating that its strong performance does not rely on fine-tuning this weight. This result supports wC=0w_C=0 as a simple, effective default for datasets whose reference answers can be reliably determined from their prompts.

W2:It would be great to see a direct comparison with SDPO. It already supports self-distillation from binary feedback using successful model-generated rollouts, including Qwen3-8B experiments on SciKnowEval and ToolAlpaca. Yet Figure 1 and Appendix A suggest that SDPO requires reference answers, and it is absent from Table 1. Without this comparison, I cannot tell how much reflection and retry add over an existing self-teacher with access to the same feedback.

Q2: Why was SDPO’s binary-feedback variant left out, especially since you use its datasets and evaluation protocol? Do you have a comparison under the same training setup?

2. [W2&Q2] Adding the SDPO Baseline and Clarifying Its Configuration

2.1. SDPO configuration.

Thank you for pointing out this detail about SDPO. We respectfully note that SDPO can also expose reference answers through verifier feedback. When no successful rollout is available, its feedback includes the actual output and the expected reference output, making the correct answer explicitly visible to the teacher. More details can be found in the feedback implementation in the official SDPO GitHub repository. We show the relevant code below.

if not actions_correct:
    feedback_parts.append(f"Actions mismatch: predicted {pred_actions}, expected {gt_actions}")
if not action_inputs_correct:
    feedback_parts.append(f"Action inputs mismatch: predicted {pred_action_inputs}, expected {gt_action_inputs}")

In contrast, ReTeach’s verifier returns only binary feedback indicating whether the answer is correct, without exposing reference answers to the model through prompts or any other input.

2.2 SDPO baseline.

Following the reviewer's suggestion, we compare ReTeach with SDPO to evaluate the teacher built through reflection and retry. The table below reports SDPO’s performance across tasks with a single random seed. We are running additional seeds and will update the table with multi-seed means and standard deviations in the revised manuscript once these experiments are complete.

MethodSDPOReTeach
SciKnowEval-Physics62.6164.24
SciKnowEval-Material68.8269.17
Tooluse-ToolAlpaca60.6162.72
Math-AIME2466.3978.61
Math-AIME2555.2870.46
Math-HMMT2534.7247.78

We observe that ReTeach consistently outperforms SDPO across all tasks, with particularly pronounced advantages on cross-dataset Math benchmarks. In SDPO configurations where feedback reveals the reference answer, supervision may be shaped around explaining an already known target. ReTeach instead encourages the model to develop corrections from the question and its own historical errors. Through self-distillation, the model can internalize how to reconsider intermediate steps and apply its existing knowledge more effectively. These reusable reasoning procedures may transfer more readily to unfamiliar tasks, supporting ReTeach’s stronger cross-dataset generalization.

Q1: Do Figure 3 and Table 1 use the same evaluation examples? Do you have results where w_C was chosen on separate data? Were these benchmarks also used to select checkpoints or other hyperparameters?

3. [Q1] Evaluation Setup and Hyperparameter Settings

3.1 Evaluation examples.

We thank the reviewer for prompting us to describe the evaluation setup more precisely. Figure 3 and Table 1 use exactly the same evaluation examples. Each task uses publicly available data with separate training and evaluation splits. Within each task, all methods and ablations use the same evaluation set.

3.2. Default setting for wCw_C.

We adopt wC=0w_C=0 as the default across datasets, because trajectories that remain unresolved after multiple rounds of reflection often lack reliable corrective supervision. Figure 3 examines this choice by varying wCw_C. The experiments revealed an exception on ToolAlpaca due to data quality issues. We find that 65% of its training samples contain ambiguous reference fields or invalid reference calls. Keeping the default wC=0w_C=0 would therefore discard useful supervision from incorrectly rejected rollouts. For datasets whose prompts provide sufficient information to infer the correct answer, we can use wC=0w_C=0 as the default without an additional search over wCw_C to find the best-performing checkpoint.

Q3: In the “w/o Reflection” ablation, does the teacher’s distillation prefix still include the native <think> block, and is thinking mode unchanged? The full method includes both <think> and <reflect>, so I want to understand exactly what this ablation removes.

4. [Q3] Details of the “w/o Reflection” Ablation

Thank you for highlighting this implementation detail. In the w/o Reflection ablation, we retain the teacher’s thinking mode and the native <think> block in the distillation prefix, following the OPSD training configuration. The ablation removes the explicit <reflect> and </reflect> tags and the enclosed reflection content, while retaining iterative retries based on the history of failed attempts. In the revised manuscript, we will add a more detailed description of the experimental configuration to the “Effectiveness of Explicit Reflection” ablation in Section 4.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.