acceptodds
Under review as a conference paper at ICLR 2027

MENTOR-SDFT: Making Explicit Knowledge Transferable via On-policy References for Continual Self-Distillation Fine-Tuning

Abstract

Continual fine-tuning requires large language models (LLMs) to acquire new capabilities while retaining existing knowledge. Self-Distillation Fine-Tuning (SDFT) offers a practical approach to this problem by using a reference-conditioned self-teacher to provide on-policy soft targets. However, we find that its supervision is often weak: reference conditioning alone does not guarantee that the self-teacher can transform the reference into transferable supervision, thereby undermining the plasticity–stability trade-off in continual learning. To address this limitation, we propose MENTOR-SDFT (Making Explicit kNowledge Transferable via On-policy References for SDFT), a framework that strengthens self-teacher guidance by improving both supervision construction and utilization. MENTOR-SDFT introduces Agent-Guided Prompt Evolution to optimize teacher conditioning and Confidence-guided Soft Trajectory Distillation to adaptively regulate supervision according to teacher confidence and trajectory position. Experiments across continual-learning sequences and multiple LLM backbones show that MENTOR-SDFT improves final average accuracy and reduces forgetting compared with sequential SDFT baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Large Language Models, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QXGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MENTOR-SDFT for continual fine-tuning without replay. Before each task, Agent-Guided Prompt Evolution (AGPE) uses GPT-5.4-mini to optimize a teacher prompt while keeping the target model frozen. Subsequent on-policy distillation uses a reference-conditioned EMA self-teacher, with Confidence-guided Soft Trajectory Distillation (CSTD) weighting supervision according to teacher entropy and trajectory position. Experiments on four tasks and three Qwen backbones report improved final average accuracy and reduced forgetting relative to sequential SDFT.

Critical strengths

1.The paper addresses a meaningful question: whether reference conditioning alone provides sufficiently useful supervision for self-distillation. The controlled prompting experiment provides initial evidence that teacher conditioning deserves further investigation.

2.The evolved prompts and reference answers are required only during training. The resulting student incurs no additional prompt optimization or privileged-information requirements at inference time.

3.Improvements over sequential SDFT are consistent across the three evaluated backbones. The progressive ablations provide initial evidence that both AGPE and CSTD contribute to the overall gains.

Critical weaknesses

1.The offline prompt selection metric is not validated as a predictor of downstream distillation gains. Section 4.1 selects prompts using the frozen model’s answer accuracy without reference answers. During distillation, however, the teacher receives references and supplies distributions along student-generated prefixes. The paper does not establish whether prompts ranked highly under the first condition also yield larger student gains under the second. Comparing candidate scores with outcomes after equal-budget distillation would test this assumption. Furthermore, the proposed net gain is equivalent to the accuracy increase on a fixed dataset. Although individual regression cases may inform reflection, the scalar ranking offers no additional preference for fewer regressions at the same accuracy.

2.The evolved prompts provide limited support for the proposed explanation of the gains. The motivation emphasizes helping the teacher interpret reference answers and reconstruct their derivations. However, Appendix E largely shows formatting instructions, exact-matching requirements, and general task constraints. The Hendrycks prompt makes explicit reasoning optional and requests brevity. These examples leave unclear whether improvements reflect more transferable supervision or better compliance with output requirements. A baseline that only strengthens formatting instructions, together with an error breakdown, would help distinguish these explanations. The complete teacher input should also be provided, particularly to clarify what “reference” denotes in the ToolUse prompt when candidate selection excludes reference answers.

3.The necessity of iterative reflection and evolution remains unverified. AGPE generates three candidates per generation and runs for two generations, leaving one refinement round after initial generation. The paper does not report results across evolution depths or compare against simpler alternatives, such as a single prompt rewrite or independently generating the same number of candidates and selecting the best. Comparisons using the same optimizer and evaluation budget would isolate the contribution of iterative feedback from the benefit of prompt generation and selection alone.

4.The continued suitability of a fixed prompt during training is not examined. AGPE optimizes the prompt before each task and keeps it fixed throughout distillation. Meanwhile, both the student and EMA teacher evolve, potentially changing their trajectories and failure patterns. The initially selected prompt may therefore become less suitable as training progresses. Comparing optimization performed entirely before training with periodic refinement based on updated student errors, under the same total search budget, would help justify this design choice.

5.Dependence on the external prompt optimizer and its computational cost are insufficiently characterized. All prompt evolution uses GPT-5.4-mini. The paper does not compare this setup with using the target model itself for diagnosis and refinement, leaving unclear how much the gains depend on external model capabilities. Such a comparison would clarify the requirements of the prompt optimization stage. Reporting API usage, candidate evaluation costs, and total training time would also make the practical trade-off relative to sequential SDFT easier to assess.

Questions for authors

See weaknesses.

2 comments

Q3.1: Does offline prompt accuracy reflect downstream distillation performance?

A3.1:

Thank you for raising this concern. We conducted additional experiments on Qwen3-1.7B to examine the relationship between offline prompt accuracy and downstream distillation performance. We selected three prompts with the highest, median, and lowest offline accuracy from all evolved candidates and performed distillation using identical training settings and budgets. As shown below, higher offline prompt accuracy consistently corresponds to better post-distillation performance, supporting the effectiveness of our offline prompt selection criterion.

Prompt TierOffline Accuracy (%)Post-Distillation Accuracy (%)
Worst16.7347.57
Medium24.7348.50
Best32.5251.31

Q3.2: Is the net-gain criterion equivalent to the accuracy?

A3.2:

Yes. As defined in Eq. 6, the net-gain G(p)G(p) is equivalent to accuracy. G(p)=∣I+(p)∣−∣I−(p)∣G(p)=|I^{+}(p)|-|I^{-}(p)|.

We adopt this form because the reflection process considers not only G(p)G(p) but also I+(p)I^{+}(p) and I−(p)I^{-}(p), ensuring that regressions are taken into account during optimization.

The detailed implementation is available in the supplementary code repository at src/prompt_agent (lines 1177–1183).

We have added the corresponding description to the Method section of the manuscript.

Q3.3: Do the evolved prompts improve supervision beyond formatting compliance?

A3.3:

Thank you for this observation.

As shown below, the two methods exhibit similar formatting error rates on Qwen2.5-3B, indicating that the gains are not solely due to formatting compliance.

Notably, the original task prompts already impose format constraints. Such constraints therefore become important content during evolution. Nevertheless, the experimental results show that the evolved prompt still helps self-distillation beyond formatting issues.

DatasetSDFT (using original prompts)MENTOR-SDFT (using evolved prompts)
AccuracyFormat ErrorAccuracyFormat Error
ToolUse48.456.1964.956.19
Science63.912.1766.071.78
MBPP37.800.4040.000.60
Hendrycks78.451.4378.851.52

We have included the above results in Table 6 of the Appendix in the manuscript.

Q3.4: What information is provided to the teacher during distillation, and what does “reference” mean in ToolUse?

A3.4:

Thank you for this question. During distillation, the complete self-teacher input during distillation can be represented as follows:

Teacher Input:

Teacher Prompt:  [AGPE-evolved prompt]
Task Input:      [Task question]
Reference:       [Ground-truth answer]
Student Prefix:  [Student-generated prefix]

For ToolUse, the reference denotes the ground-truth answer. For example:

I need to find the most recent changes to travel restrictions in Japan. Action: getLatestPolicyUpdates Action Input: {"limit": 10}.

Q3.5: Is iterative reflection necessary for effective prompt optimization?

A3.5:

Thank you for this suggestion. We compare two strategies under the same budget:

  • Independent Search: Generates and selects prompts without iterative feedback.
  • Iterative Refinement (Ours): Refines prompts using error feedback from previous evaluations.

As shown in the figure below, iterative refinement consistently outperforms independent search across all four tasks, demonstrating the benefits of iterative reflection for prompt optimization.

DatasetIndependent SearchIterative Refinement (Ours)
ToolUse30.8631.05
Science38.8744.14
MBPP40.0040.50
Hendrycks69.1474.22
Average44.7247.48

We have included the above results in Table 4 of the Experiments in the manuscript.

Q3.6: Does periodic refinement of teacher prompts during distillation improve performance compared to keeping them fixed?

A3.6:

Thank you for this suggestion. We compared fixed prompts with periodic refinement. The results support keeping teacher prompts fixed for two reasons:

  • Effectiveness: Periodic refinement does not provide performance improvement (table below).
  • Efficiency: Fixed prompts avoid the cost of repeated prompt optimization during distillation.
ModelPrompt StrategyToolUseScienceMBPPHendrycksAverage
Qwen2.5-3BPeriodic Refinement59.7968.4441.2080.1262.39
Fixed Prompt(Ours)64.9566.0740.0078.8562.47
Qwen3-4BPeriodic Refinement61.8659.5747.4091.8365.17
Fixed Prompt(Ours)62.8965.4847.8091.6666.96

We have included the above results in Table 9 of the Appendix in the manuscript.

Q3.7: How does AGPE's performance change when the target model replaces GPT-5.4-mini as the prompt optimizer?

A3.7:

Thank you for this question. We replaced GPT-5.4-mini with the target model itself for prompt evolution. As shown below, self-evolution improves SDFT on both prompt evolvers. The gains are smaller for Qwen2.5-3B due to its weaker prompt-optimization capability, whereas Qwen3-4B already has sufficient ability.

ModelPrompt OptimizerToolUseScienceMBPPHendrycksAverage
Qwen2.5-3BSDFT48.4563.9137.8078.4557.15
Target Model49.4864.3041.4078.2658.36
GPT-5.4-mini64.9566.0740.0078.8562.47
Qwen3-4BSDFT58.7660.3647.8092.1764.77
Target Model56.7062.7248.8092.0065.06
GPT-5.4-mini62.8965.4847.8091.6666.96

We have included the above results in Table 7 of the Appendix in the manuscript.

Q3.8: What is the cost of AGPE?

A3.8:

Thank you for this question. For four-task continual learning, AGPE requires 63 GPT-5.4-mini API calls and approximately 1.16 million tokens, totaling only $1.03.

For training time, sequential SDFT takes approximately 5.5 hours (Qwen2.5-3B on a single H100 GPU), while AGPE adds only 4 minutes per task on average for prompt optimization and candidate evaluation. Overall, AGPE introduces only a small additional cost.

Reviewer_rXcQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to achieve continuous fine-tuning of large language models (LLMs) without replay. SDFT employs a model replica based on reference conditions as a teacher for on-policy distillation. The authors note that its supervision mechanism is weak (its performance lags behind OPD with an external teacher, and the performance of the self-teacher is limited by the student’s own knowledge) and demonstrate that manually crafting teacher prompts that explain how to interpret the reference information can improve teacher accuracy (from an average of 72.1 to 78.4).

MENTOR-SDFT consists of two components:

  • AGPE: An LLM agent diagnoses student errors, proposes candidate teacher prompts, optimizes them based on net gain (newly corrected portions minus newly regressed portions), and selects the best ones on the holdout set.
  • CSTD: A token-level KL divergence weighted by the teacher’s normalized confidence within the trajectory, with a positional gating mechanism ensuring that answer tokens retain full weight.

Results: In experiments covering four task sequences (ToolUse, Science, MBPP, and Hendrycks), using the Qwen2.5-3B and Qwen3-1.7B/4B models, the final average accuracy improved by 5.31, 3.70, and 2.19 percentage points, respectively, compared to sequential SDFT, while exhibiting lower forgetting rates; We report the results of ablation experiments as well as sensitivity analyses regarding prompts and task sequences.

Critical strengths
  1. A clear diagnosis based on controlled experiments: By fixing the teacher weights, inputs, and privilege information while varying only the teacher prompts, we isolated teacher conditioning as the bottleneck, thereby providing motivation for the development of these two components.

  2. A practical and well-defined method (using mutually exclusive partitions for regression-aware prompt selection; simple confidence/position weighting) that improves final accuracy and mitigates the forgetting problem across three backbone networks, supported by ablation experiments and robustness validation (varying prompts and task order resulted in error variations of less than 0.6 points).

Critical weaknesses

This comparison is not made under equivalent conditions and has a narrow scope.

  • AGPE uses an external large language model (gpt-5.4-mini) to generate teacher prompts based on student errors, thereby injecting external knowledge and computational resources into the “self” distillation method; none of the baseline models received comparable information or resources.
  • Baseline methods include only sequential SFT/SDFT, layer freezing, and OSFT (excluding replay, O-LoRA-style, or external teacher OPD); furthermore, they cover only four tasks.
Questions for authors
  1. What is the cost of AGPE (number of proxy calls/number of tokens)? How does MENTOR-SDFT perform compared to baselines that use the same external model and have comparable costs (e.g., knowledge distillation from that model)?

  2. How many seeds were used in Table 2? Do these performance improvements persist as the task sequence lengthens, and when using replay or parameter-isolation baselines?

  3. Can you provide some examples of the evolved prompts and confirm that they do not contain task-specific solution knowledge?

2 comments

Q2.1: Given that AGPE employs an external LLM to optimize prompts, is the comparison fair?

A2.1:

Thank you for raising this concern.

The external LLM is used only for prompt refinement. Our goal is to test the core hypothesis that SDFT’s effectiveness hinges on whether the self-teacher can transform references into transferable supervision. We find that Qwen3-4B already has sufficient prompt-refinement capability (Qwen2.5-3B is somewhat weaker). Since this capability is readily accessible in practice, we do not believe its use undermines the fairness of the comparison.

To further rule out any fairness concerns, we conduct two additional experiments:

Experiment 1: We replace GPT-5.4-mini with the target model itself for prompt evolution. As shown below, self-evolution improves SDFT with both prompt evolvers. The gains are smaller for Qwen2.5-3B due to its weaker prompt-optimization capability, whereas Qwen3-4B already has sufficient ability.

ModelMethodToolUseScienceMBPPHendrycksAverage
Qwen2.5-3BSDFT48.4563.9137.8078.4557.15
MENTOR-SDFT (Qwen2.5-3B as prompt optimizer)49.4864.3041.4078.2658.36
MENTOR-SDFT (GPT-5.4-mini as prompt optimizer)64.9566.0740.0078.8562.47
Qwen3-4BSDFT58.7660.3647.8092.1764.77
MENTOR-SDFT (Qwen3-4B as prompt optimizer)56.7062.7248.8092.0065.06
MENTOR-SDFT (GPT-5.4-mini as prompt optimizer)62.8965.4847.8091.6666.96

Experiment 2: We apply GPT-5.4-mini-refined prompts to two main baselines (Layer Freezing and OSFT). As shown in the results on Qwen3-1.7B below, MENTOR-SDFT still achieves the highest average accuracy, outperforming both baselines under the same prompt guidance.

MethodToolUseScienceMBPPHendrycksAverage
Layer Freezing + refined prompts52.5868.0540.2082.9060.93
OSFT + refined prompts49.4866.6737.2083.2459.15
MENTOR-SDFT (Ours)58.7669.4337.8083.4062.35

We have included the above results in Table 8 of the Appendix in the manuscript.

Q2.2: Why are replay, O-LoRA, and external-teacher OPD baselines not included?

A2.2:

Thank you for this suggestion. We added Replay and O-LoRA as additional baselines. As shown in the table below, MENTOR-SDFT achieves the highest average accuracy.

MethodToolUseScienceMBPPHendrycksAverage
Sequential SDFT52.5862.5236.0083.4958.65
Replay54.6465.4836.0082.6559.69
O-LoRA55.6731.1630.2044.9940.51
MENTOR-SDFT58.7669.4337.8083.4062.35

Q2.3: Does MENTOR-SDFT remain effective over longer continual-learning sequences?

A2.3:

Thank you for this suggestion. We extended the learning sequence to seven tasks. As shown below, MENTOR-SDFT maintains its overall advantage over Sequential SDFT, demonstrating its effectiveness across longer task sequences.

MethodToolUseScienceMBPPHendrycksTRACE-2TRACE-5TRACE-7Average
Sequential SDFT41.2459.3736.2081.3055.8582.0043.3857.05
MENTOR-SDFT52.5870.0236.4080.7156.6580.8048.0060.74

We have included the above results in Table 5 of the Experiments in the manuscript.

Q2.4: What is the cost of AGPE?

A2.4:

Thank you for this question. For four-task continual learning, AGPE requires 63 GPT-5.4-mini API calls and approximately 1.16 million tokens, totaling only $1.03, making AGPE practical.

Q2.5: How many seeds were used in Table 2?

A2.5:

The original results were obtained from a single run. We reran the experiments with three random seeds on Qwen3-1.7B. As shown in the table below, MENTOR-SDFT exhibits small standard deviations. We will include standard deviations for all results in the final version.

MethodToolUseScienceMBPPHendrycksAverage
Sequential SDFT50.17 ± 2.1562.85 ± 0.9437.20 ± 1.1183.24 ± 0.3058.37 ± 0.35
MENTOR-SDFT57.04 ± 1.5768.71 ± 1.2536.93 ± 0.7682.98 ± 0.3861.42 ± 0.81

Q2.6: Can you provide some examples of the evolved prompts?

A2.6:

Thank you for this question. We provide some representative evolved teacher prompts below. The evolved prompts generally provide task-level guidance.

Evolved Teacher Prompt (TRACE-2)

Determine the task answer using the most direct evidence available. For classification tasks, prefer the most conservative label supported by the text: if the passage mainly describes conditions, uncertainty, monitoring, or mixed signals without a clear policy direction, choose neutral; use restrictive or accommodative labels only when the wording clearly indicates them. Then provide exactly one final answer block at the end of the response, with only the final task output inside it and no extra text after it.

<answer>

your final task answer

</answer>

Put only the final task output inside <answer>.

Evolved Teacher Prompt (MBPP)

You are solving a Python programming problem for an evaluator that extracts Python from the final answer and runs tests.

If the prompt is an executable coding task, return exactly this wrapper and nothing else:

</answer>

```python
# complete Python program
```

</answer>

The code block must contain only valid, executable Python code. Do not include explanations, prose, markdown, or any non-code text inside the code block. Do not place any text outside the wrapper.

If the prompt explicitly specifies a function name, parameters, or return behavior, match them exactly; otherwise, implement only what is stated and avoid guessing unstated behavior.

The solution must be self-contained, use only the Python standard library, and run directly.

If the task has clear semantics, handle common edge cases and include needed imports. If the prompt is ambiguous, stay as close as possible to the stated requirement and do not over-infer missing behavior.

Keep the final answer limited to the wrapper and the code block.

Reviewer_h4cuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a refinement on top of the original SDFT, where they automatically evolve the prompts with an LLM (GPT 5.4 mini) and distill the teacher’s diagnosis of what possibly went wrong in the student’s generations and then apply SDFT with these prompts. There is an additional component of confidence-guided distillation, where they quantify the confidence using the entropy of the teacher model to selectively trust the teacher. Across the 4 tasks learned in sequential order, Menhtor-SDFT on average improves upon the other SDFT variants, while maintaining a similar or higher level of forgetting.

Critical strengths
  • It’s a simple idea that a better privilege/context for the teacher in SDFT will improve the learning of the student, and can be easily applied to different settings.
  • The results seem to be consistently better on the task performance although a small degradation in forgetting (this seems marginal).
Critical weaknesses
  • There is a gap between what the paper describes in terms of the problem and claims to solve versus the method that is presented at hand. The paper motivates from the perspective that SDFT is limiting b/c it is hard to go beyond the knowledge of the base model and a model has to really understand and follow through the reasoning in order to provide more fine-grained feedback as to what reasoning is correct and incorrect. However, the method, at heart, is simply a way to distill a bigger/better teacher’s knowledge and ability to give feedback on the generated responses from the student into the privilege and, given the optimized prompt examples shown in the Appendix, they mostly focus on the syntax of how to answer than the fundamental problems described earlier in the paper.
  • The experiments are rather small scale (1B-4B) and except for the tool use dataset, other results seem rather comparable
  • No variance reported across multiple seeds, so robustness of the method is unclear yet
Questions for authors
  • How does it fare with different teachers used for the prompt optimization stage? e.g. use a better or smaller model than GPT 5.4 mini
  • Could you run multiple seeds of the experiments to show that these improvements are indeed statistically significant? (given that the improvements are not drastic)
4 comments

Q1.1: Is there a gap between the paper's motivation and the proposed method?

A1.1:

This is an interesting question. We would like to clarify three points:

  • Our observation: In self-distillation, self-teachers may provide limited guidance despite having access to reference answers.
  • Our motivation: The effectiveness of SDFT hinges on whether the self-teacher can transform references into transferable supervision.
  • Our solution: We use an agent system powered by GPT-5.4-mini to generate better prompts that guide the self-teacher toward more effective supervision.

Crucially, GPT-5.4-mini optimizes the shared prompt, and the training supervisions are still generated by the self-teacher. We therefore view this as improving how the self-teacher uses references, rather than distilling supervision from an external teacher. Our motivation and method are thus aligned.

Q1.2: Are the improvements primarily driven by formatting instructions in evolved prompts?

A1.2:

Thank you for this comment.

As shown below, the two methods exhibit similar formatting error rates on Qwen2.5-3B, indicating that the gains are not solely due to formatting compliance.

Notably, the original task prompts already impose format constraints. Such constraints therefore become important content during evolution. Nevertheless, the experimental results show that the evolved prompt still helps self-distillation beyond formatting issues.

DatasetSDFTMENTOR-SDFT (Ours)
AccuracyFormat ErrorAccuracyFormat Error
ToolUse48.456.1964.956.19
Science63.912.1766.071.78
MBPP37.800.4040.000.60
Hendrycks78.451.4378.851.52

We have included the above results in Table 6 of the Appendix in the manuscript.

Q1.3: Does MENTOR-SDFT generalize to larger models?

A1.3:

Thank you for this suggestion. We are currently conducting experiments on larger models. The results are not yet available due to time constraints. We will provide them as soon as possible.

Q1.4: Are the improvements mainly limited to ToolUse?

A1.4:

We thank the reviewer for this observation. The improvement is not limited to ToolUse. ToolUse exhibits the largest gain because it is the earliest task in the sequence and thus most exposed to forgetting during subsequent updates.

We extended the learning sequence to seven tasks. As shown below, MENTOR-SDFT consistently improves across multiple tasks.

MethodToolUseScienceMBPPHendrycksTRACE-2TRACE-5TRACE-7Average
Sequential SDFT41.2459.3736.2081.3055.8582.0043.3857.05
MENTOR-SDFT52.5870.0236.4080.7156.6580.8048.0060.74

We have included the above results in Table 5 of the Experiments in the manuscript.

Q1.5: Robustness Across Random Seeds

A1.5:

Thank you for this suggestion. We reran the experiments with three random seeds on Qwen3-1.7B. As shown in the table below, MENTOR-SDFT exhibits small standard deviations. We will include standard deviations for all results in the final version.

MethodToolUseScienceMBPPHendrycksAverage
Sequential SDFT50.17 ± 2.1562.85 ± 0.9437.20 ± 1.1183.24 ± 0.3058.37 ± 0.35
MENTOR-SDFT57.04 ± 1.5768.71 ± 1.2536.93 ± 0.7682.98 ± 0.3861.42 ± 0.81

Q1.6: How does it fare with different teachers used for the prompt optimization stage?

A1.6:

Thank you for this question. We replaced GPT-5.4-mini with the target model itself for prompt evolution. As shown below, self-evolution improves SDFT with both prompt evolvers. The gains are smaller for Qwen2.5-3B due to its weaker prompt-optimization capability, whereas Qwen3-4B already has sufficient ability.

ModelMethodToolUseScienceMBPPHendrycksAverage
Qwen2.5-3BSDFT48.4563.9137.8078.4557.15
MENTOR-SDFT (Qwen2.5-3B as prompt optimizer)49.4864.3041.4078.2658.36
MENTOR-SDFT (GPT-5.4-mini as prompt optimizer)64.9566.0740.0078.8562.47
Qwen3-4BSDFT58.7660.3647.8092.1764.77
MENTOR-SDFT (Qwen3-4B as prompt optimizer)56.7062.7248.8092.0065.06
MENTOR-SDFT (GPT-5.4-mini as prompt optimizer)62.8965.4847.8091.6666.96

We sincerely appreciate your patience. We have now completed additional experiments on Qwen2.5-7B to address your concern about model scalability.

Q1.3: Does MENTOR-SDFT generalize to larger models?

A1.3:

Thank you for this suggestion. We further evaluate MENTOR-SDFT on Qwen2.5-7B using a seven-task sequence. As shown below, MENTOR-SDFT maintains its overall advantage over Sequential SDFT, demonstrating its effectiveness on larger models.

MethodToolUseScienceMBPPHendrycksTRACE-2TRACE-5TRACE-7Avg.
Sequential SDFT61.8662.1342.6078.1855.4484.6054.4662.75
MENTOR-SDFT60.9266.0746.2084.0852.6286.7060.3165.27

We have included the above results in Table 10 of the Appendix in the manuscript.

Thank you for the details and experiments.

On the point about the mismatch between the paper's motivation and the method, I do agree that the authors' description of the observation of the limitation and the method match. However, the paper in the introduction present it from this perpsective:

"This gap is intuitive. Firstly, although the self-teacher is conditioned on the reference answer, it is instantiated from the same underlying model and thus cannot provide supervision beyond the model’s own knowledge; reference conditioning may sharpen existing behavior, but it cannot introduce genuinely new capabilities. Secondly, prior work suggests that reference-conditioned self-distillation often encourages over-reliance on the reference, suppressing exploratory and reasoning-related tokens and curtailing the model’s own deliberation (Kim et al., 2026).

But, these problems that the introduction start off with is about how reliance on the reference solution does not guarantee or give you capabilities beyond the model's own knowledge. This has nothing to do with the proposed method. The proposed method is about how to identify the errors using a different teacher and to provide better privilege so that the student can learn better.

The new seeds and new experiments resolve most of my concerns on robustness and generalization and are encouraging.

The most important concern I still have is on the new Qwen 3B and 4B experiments with varying teachers. It seems that most of the benefit is coming from using a stronger teacher to identify the errors and optimize the prompt. I'm not sure we can necessarily say that this is self-distillation anymore. I wonder what the authors think about this point and the other reviewers.

Once this concern is addressed, I would be happy to raise my score.

Q1.1: Is the motivation in the Introduction aligned with the proposed method?

A1.1:

Thank you for this question. Our method was indeed designed based on the revised motivation, but the original Introduction contained an overclaim, which we have now corrected.

Original:

Firstly, although the self-teacher is conditioned on the reference answer, it is instantiated from the same underlying model and thus cannot provide supervision beyond the model's own knowledge; reference conditioning may sharpen existing behavior, but it cannot introduce genuinely new capabilities.

Revised:

Firstly, although the self-teacher is instantiated from the same underlying model and conditioned on the reference answer, merely including the reference in its context does not guarantee a substantial improvement in its supervisory capability or reliable guidance for the student.

Q1.2: Does MENTOR-SDFT qualify as self-distillation?

A1.2:

Thank you for raising this important concern. We would like to compare our objective function with standard self-distillation:


Def. (On-Policy Self-Distillation). On-policy self-distillation involves a student and a self-teacher derived from the same underlying model, where the self-teacher leverages additional privileged context to guide student learning. A representative distillation objective is:

LSD=DKL(πθ(⋅∣x,y^<t);∥;πθ(⋅∣x,c,y^<t))\mathcal{L}_{\mathrm{SD}} = D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid x,\hat{y}_{< t}) \\;\Vert\\; \pi_\theta(\cdot\mid x,c,\hat{y}_{< t})\right)

Our Self-Distillation Objective :

LMENTOR=DKL(πθ(⋅∣x,y^<t);∥;πθ(⋅∣x,cAGPE,y^<t))\mathcal{L}_{\mathrm{MENTOR}} = D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid x,\hat{y}_{< t}) \\;\Vert\\; \pi_\theta(\cdot\mid x,\mathbf{c}_{\mathrm{AGPE}},\hat{y}_{< t})\right)

where xx denotes the task input, y^<t\hat{y}_{< t} is the student-generated prefix, and πθ\pi_\theta represents the shared underlying model. Here, cc denotes the original privileged context, while cAGPE\mathbf{c}_{\mathrm{AGPE}} represents the task-level privileged context optimized by AGPE.


Compared with standard self-distillation, MENTOR-SDFT shares the key properties:

  1. Same teacher model. Both the student and the self-teacher are the training model itself; no independent external teacher is introduced.
  2. Same supervision generation. The self-teacher uses reference answers and context to produce token-level supervision, i.e., the supervision signal is still generated by the target model itself.
  3. Different only in privileged contexts. MENTOR-SDFT optimizes this context via an external LLM (e.g., GPT-5.4-mini) to help the self-teacher better exploit reference answers.

We do not agree that the difference in privileged context makes our method deviate from self-distillation.

Firstly, the context design is open in self-distillation. Existing work largely varies how context is generated, including expert demonstrations, reference answers, and environmental feedback[1-3].

Furthermore, external-model involvement in MENTOR-SDFT is deliberately restrained. We generate one context per task rather than per sample, and it remains fixed once created at task start. In other words, the external model does not generate sample-level or training-process-level supervision. It is designed to replace manual context design and make our project more practical.

Moreover, introducing an external model does not limit the applicability of our method. Empirically, Qwen3-4B-level models already generate sufficiently capable contexts, and such models can be easily deployed locally or accessed via low-cost APIs.

We acknowledge that a small amount of external knowledge is introduced, but this does not affect the core contribution of this paper. We have clarified this in the Related Work of the manuscript.

Reference

[1] Shenfeld I, Damani M, Hübotter J, et al. Self-distillation enables continual learning[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 111542–111558.

[2] Zhao S, Xie Z, Liu M, et al. Self-distilled reasoner: On-policy self-distillation for large language models[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 162433–162448.

[3] Hübotter J, Lübeck F, Behric L, et al. Reinforcement learning via self-distillation[C]//Proceedings of the 43rd International Conference on Machine Learning. PMLR, 2026, 306: 48677–48721.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to AC and All Reviewers

We thank the Area Chair and all reviewers for their constructive feedback and valuable suggestions. We are pleased that reviewers rXcQ and QXGZ recognize the importance of improving teacher conditioning in SDFT and the supporting controlled experiments. Reviewers h4cu and rXcQ highlight the simplicity and practicality of our approach, while QXGZ further appreciates its absence of additional inference-time overhead. All three reviewers acknowledge the consistent performance improvements across different backbone models.

We have revised the manuscript accordingly and marked all changes in blue. We provide point-by-point responses to each reviewer below and look forward to further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.