acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Teach: Adaptive Hint Generation for On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) turns a language model into its own teacher by supplying it with privileged context (a hint) and distilling the resulting token-level guidance back into the unconditioned student. Existing methods often supply task-level hints that capture what a task generally requires, rather than what the current student gets wrong. We argue that the hint should instead be learned. We introduce EvoHint, which trains a dedicated hint generator to produce reflective hints: guidance conditioned on the student’s own attempt and optimized by whether it helps the student succeed. The generator draws on a persistent memory of effective hints, adapting retrieved hints to the observed attempt. Same-task student rollouts provide outcome feedback for generator training and select successful hints for memory updates, allowing guidance to adapt as the student learns. Experiments on agentic tasks (Search-QA and WebShop) and six mathematical reasoning benchmarks show consistent improvements over a fixed-hint self-distillation baseline across student model scales, with gains of up to 9.8 percentage points on agentic tasks and 1.8–3.1 points in average mathematical reasoning accuracy. Ablations further show that EvoHint’s gains hold across student objectives, that learned hints outperform frontier-model hints by up to 5.4 points, and that reusing generated hints as memory performs on par with frontier-model-built memory without a separate memory writer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving agent, Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1HxkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EvoHint, an RL + on-policy self-distillation training algorithm for LLMs. The method features alternatively updating a hint generator and a task solver, while accumulating a persistent memory that can be re-used in test-time. The authors empirically validate their method on web-search and mathematical reasoning benchmarks. I can see marginal improvement but it is not highly consistent.

Critical strengths

The method generally shows 1~3 points improvements over existing baselines in the provided experiments.

Critical weaknesses
  1. Novelty is limited. The method appears as an interpolation of existing ideas to me. (1) RL + OPSD from RLSD/SDAR; (2) RL + ideas bank + test-time reusing like [1], which also discusses proactive/reflective hint. (3) co-training hint generation & solver is a form of hierachical reasoning models [2]

  2. Statistical significance of improvements. None of the tables shown in this paper provide variance/standarad error. And RL training is known to have large variance. I believe it is necessary to at least repeat the main table for several times and report steady improvment.

  3. End-to-end time overhead. The method seems complicated and to have large time/computational overhead. The authors should report end-to-end time/computational overhead. And if possible, compare with baselines in matched compute setting.

  4. Ablated effectiveness of each design components. Current, the contribution of each component (co-training, hint memory) is not very clear to me.Main message of Section 5.2 is not clear.

[1] Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs; https://openreview.net/forum?id=K9GxQKEJzS

[2] Reinforce LLM Reasoning through Multi-Agent Reflection; https://icml.cc/virtual/2025/poster/46364

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1Hxk (1 / N)

We thank the reviewer for their constructive suggestions and address the concerns below.

W1: Limited Novelty

We thank the reviewer for highlighting these related works. Building upon prior RL + OPSD methods (e.g., RLSD and SDAR), our key novelty lies in learning adaptive teacher conditioning that addresses the student's specific mistakes and is optimized through outcome feedback. We do not claim hint memory, reflection, or co-training as standalone contributions. Rather, our contribution is to make teacher conditioning adaptive and learnable within the OPSD framework, enabling more effective self-distillation.We also conduct extensive experiments across both agentic tasks and mathematical reasoning benchmarks to demonstrate the effectiveness and generality of our approach.

W2: Statistical significance and run-to-run variability.

We have added three-run results across WebShop, SearchQA, and Math Reasoning, reporting mean ± standard deviation. For the two agentic tasks, we compare RLSD, SDAR, and EvoHint (RLSD-based) across three model backbones; for Math Reasoning, we compare RLSD and EvoHint across two backbones. Compared with RLSD, EvoHint achieves average gains of approximately 3–5 points on WebShop, 2–3 points on Math Reasoning, and 1-3 points on SearchQA, supporting consistent improvements.

WebShop

StudentMethodScore (mean ± SD)Accuracy (%) (mean ± SD)
Qwen3-1.7BRLSD75.2 ± 1.257.0 ± 0.7
SDAR77.0 ± 0.458.8 ± 0.4
EvoHint (RLSD-based)83.2 ± 2.462.6 ± 0.6
Qwen2.5-3B-InstructRLSD84.0 ± 1.266.6 ± 0.8
SDAR84.8 ± 1.667.6 ± 0.9
EvoHint (RLSD-based)86.6 ± 1.069.0 ± 0.6
Qwen2.5-7B-InstructRLSD87.5 ± 0.177.6 ± 0.4
SDAR89.2 ± 0.282.1 ± 0.6
EvoHint (RLSD-based)91.0 ± 0.280.7 ± 1.0

Search-QA

StudentMethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Qwen3-1.7BRLSD38.8 ± 0.357.9 ± 0.843.8 ± 1.235.1 ± 0.831.6 ± 3.511.4 ± 0.265.5 ± 0.340.6 ± 0.0
SDAR41.5 ± 2.558.1 ± 1.145.1 ± 0.335.8 ± 0.134.6 ± 1.313.1 ± 0.765.1 ± 0.041.9 ± 0.0
EvoHint (RLSD-based)43.0 ± 0.760.3 ± 0.547.2 ± 0.138.1 ± 0.136.7 ± 0.613.0 ± 0.367.7 ± 1.143.7 ± 0.1
Qwen2.5-3B-InstructRLSD41.6 ± 0.658.6 ± 0.142.3 ± 0.740.1 ± 0.440.1 ± 1.816.3 ± 0.867.0 ± 0.243.7 ± 0.4
SDAR44.5 ± 0.458.0 ± 0.143.6 ± 1.138.9 ± 0.436.3 ± 0.115.8 ± 0.166.5 ± 0.543.3 ± 0.1
EvoHint (RLSD-based)44.7 ± 0.161.1 ± 0.145.7 ± 0.539.1 ± 0.140.2 ± 0.114.6 ± 0.467.0 ± 0.144.6 ± 0.1
Qwen2.5-7B-InstructRLSD46.1 ± 0.663.6 ± 0.943.9 ± 0.844.7 ± 0.948.6 ± 0.321.2 ± 0.372.9 ± 0.148.7 ± 0.3
SDAR45.6 ± 1.163.4 ± 0.248.2 ± 0.143.5 ± 0.448.7 ± 0.419.4 ± 0.373.2 ± 0.248.8 ± 0.2
EvoHint (RLSD-based)47.4 ± 0.364.0 ± 0.349.1 ± 0.145.6 ± 0.149.1 ± 0.121.4 ± 0.173.4 ± 0.350.0 ± 0.0

Math Reasoning

MethodMATH500AIME24AIME25AMC23MinervaOlyBenchAvg.
RLSD (1.7B)75.9 ± 0.824.8 ± 2.724.5 ± 1.060.0 ± 0.822.8 ± 0.350.5 ± 0.943.1 ± 0.2
EvoHint (RLSD-based, 1.7B)76.7 ± 0.127.5 ± 0.428.8 ± 0.666.6 ± 0.625.4 ± 0.353.0 ± 0.346.3 ± 0.1
RLSD (8B)83.5 ± 0.650.7 ± 3.441.7 ± 2.389.7 ± 1.332.3 ± 1.462.1 ± 2.260.0 ± 0.3
EvoHint (RLSD-based, 8B)84.4 ± 0.355.1 ± 0.143.8 ± 0.391.0 ± 0.332.8 ± 0.764.3 ± 1.861.9 ± 0.5
AuthorsOpenReview ↗

Response to Reviewer 1Hxk (2 / N)

W3: End-to-End Computational Overhead

We thank the reviewer for this suggestion. To assess the computational overhead, we conduct additional experiments on WebShop by doubling the training steps of RLSD (from 150 to 300), resulting in roughly the same total GPU hours and number of training rollouts as EvoHint. As shown below, doubling RLSD's training budget yields only marginal improvements (75.2 → 76.6 for Qwen3-1.7B and 84.0 → 84.5 for Qwen2.5-3B). In contrast, EvoHint consistently outperforms RLSD under comparable computational budgets, achieving improvements of +6.6 and +2.1 Score points, respectively.

BackboneMethodBszStepsStudent training rolloutsTeacher training rolloutsScore ↑Acc ↑
Qwen3-1.7BRLSD1615019,200075.2 ± 1.257.0 ± 0.7
RLSD (2× steps)1630038,400076.657.0
EvoHint (RLSD-based)1615019,20019,20083.2 ± 2.462.6 ± 0.6
Qwen2.5-3B-InstructRLSD1615019,200084.0 ± 1.266.6 ± 0.8
RLSD (2× steps)1630038,400084.567.0
EvoHint (RLSD-based)1615019,20019,20086.6 ± 1.069.0 ± 0.6
AuthorsOpenReview ↗

Response to Reviewer 1Hxk (3 / N)

W4: Effectiveness of Individual Components

We thank the reviewer for this suggestion. To better clarify the contributions of individual components, we separate our ablation analysis into training-time and inference-time evaluations, using Qwen3-1.7B on WebShop.

Training-Time Ablation

We compare the following configurations to evaluate the contributions of co-training and evolving memory.

MethodScore ↑Acc. (%) ↑
RLSD (baseline)75.856.8
EvoHint (Fixed Memory + Co-Training)79.860.0
EvoHint (Evolving Memory + Co-Training)85.663.2

This comparison isolates the benefit of learning adaptive hints through co-training and the additional contribution of evolving the hint memory during training.

Inference-Time Ablation (Clarification of Section 5.2)

We further evaluate whether the learned memory should be retained during inference, using the same trained EvoHint student.

MethodScore ↑Acc. (%) ↑
EvoHint (w/o Memory)85.663.2
EvoHint (w/ Memory)87.266.8

The results show that inference-time memory retrieval further improves WebShop performance by +1.6 Score points and +3.6 accuracy points. Importantly, memory retrieval is optional at inference, and the student can operate independently without the hint generator or memory.

We will revise Section 5.2 to clarify the distinction between the role of evolving memory during training and optional memory retrieval during inference.

Thank you for the additional results. However, I still cannot see a clear improvement over SDAR, especially under matched-time comparison. The method's novelty is still limited, and the it does not offer a strong enough reason to add up so much additional compute and implementation complexity. I maintain my score of clear rejection.

Reviewer_Z4GSOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to learn more effective teacher hints for on-policy self-distillation. EvoHint uses a separate hint generator that conditions on the task, the student's previous attempts, and retrieved memories to produce reflective hints. The generator is trained using outcomes from fresh student rollouts conditioned on these hints, and useful hints are stored in memory. The student receives token-level supervision from a hint-conditioned copy of itself, and trajectories used to evaluate hints can also be reused for student training. The authors report improvements across multiple model scales on Search-QA, WebShop, and six mathematical benchmarks, with analyses of attempt conditioning, student objectives, and memory use. In the main results, student inference requires neither the hint generator nor memory.

Critical strengths
  1. Hints target the student's actual errors. EvoHint generates guidance based on the student's attempts and trains the hint generator using task outcomes. In Table 5, reflective hints substantially outperform hints generated without observing student attempts, supporting the value of tailoring guidance to specific errors.

  2. Empirical support across tasks. Experiments cover search-based question answering, shopping, and six mathematical benchmarks, with improvements across model sizes and two student objectives. The students in the main tables require neither the generator nor memory at inference time, indicating that the training benefits are retained in the student model. (Tables 1–2, Figure 2)

Critical weaknesses
  1. The experiments lack an assessment of randomness. EvoHint's reported results are relatively close to those of the baselines, but the paper does not report independent training seeds or confidence intervals for the main results. It is therefore difficult to determine whether these improvements are consistent.

  2. The generated hints lack empirical analysis. The paper mainly reports task scores and ablations, without examining which hinting strategies the generator learns, which types of hints benefit the student most, or examples of how hints evolve during training. This makes it difficult to understand the specific guidance capabilities behind the performance gains.

  3. I also think scaling this to more “non-toy” training and evaluations would improve the paper’s significance (although not necessary since this is proposing a new method).

Questions for authors
  1. Could the authors provide more empirical analysis? This would help readers understand why EvoHint works.

  2. Training methods that combine memory and hints are not new, and I am interested in whether OPSD-based methods outperform RL-based alternatives. Could the authors add comparisons with memory-augmented RL methods such as EMPO² [1] to establish whether OPSD provides additional benefits?

Reviewer_h2ywOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoHint, a framework for learning the privileged hints used in on-policy self-distillation. A separately trained hint generator conditions on the task, the student's current attempt, and retrieved memory to produce reflective hints. A detached copy of the student uses these hints to score the student's original, hint-free trajectories, providing supervision through RLSD or SDAR. The generator is optimized with a GRPO-style objective using the outcomes of fresh, hint-conditioned student attempts on the same task, while outcome-selected hints update a persistent memory. Experiments on Search-QA, WebShop, and six mathematical reasoning benchmarks report aggregate improvements over RLSD across the evaluated model sizes. The main results use standalone student inference without the generator or memory.

Critical strengths
  1. A well-motivated focus on teacher conditioning. Rather than only modifying how teacher scores enter the student loss, the paper studies how to generate guidance tailored to the student's current attempt. This is a meaningful direction for improving self-distillation.

  2. Evaluation across tasks and model sizes. The experiments cover both interactive agent tasks and mathematical reasoning. EvoHint improves over RLSD on the reported aggregate metrics across the evaluated scales, and the main results do not require additional hint generation or retrieval at inference.

  3. Useful evidence on attempt conditioning and objective compatibility. The WebShop ablations show a substantial advantage for reflective over proactive hints and improvements when EvoHint is combined with either RLSD or SDAR. These results support the usefulness of attempt-conditioned guidance, although they do not fully isolate all components.

Critical weaknesses

1. Insufficient justification of the generator's proxy objective

The generator maximizes the student's return when the student directly receives a hint and retries the task (Section 3.3, Eqs. 2-3). However, the intended use of the hint is to improve the self-teacher's scoring of the original trajectory and ultimately improve the student's hint-free performance after distillation. These are different objectives, and better hint-conditioned execution does not necessarily imply more useful distillation supervision. The final performance gains establish the utility of the overall system but do not establish this alignment. Can the authors justify the proxy objective or directly test whether hint-conditioned task returns predict improvements from subsequent distillation? This concern remains even if hints contain no answers.

2. Unclear rollout-budget alignment and contribution of trajectory reuse

Under the training loop that updates both policies each iteration, EvoHint collects N plain student trajectories and an additional H times K trajectories per task to evaluate candidate hints. The latter are required even when rollout reuse is disabled. If reuse is enabled, these hinted trajectories also contribute ordinary policy-gradient updates to the student, whereas Appendix A.5 states that the RLSD/SDAR baselines do not use the EvoHint replay batch. The manuscript does not clearly specify the reuse settings and quantities for each main experiment or establish matched sampling and compute budgets. Please report N, H, K, training iterations, reuse settings, total student rollouts, generated tokens, environment interactions, and end-to-end training cost. How do RLSD/GRPO perform with the same sampling budget? Reuse ablations and a control retaining the same hinted-trajectory updates but removing teacher-based reweighting would help separate improved teacher conditioning from additional RL training.

3. Unclear presence and handling of answers in hints

The manuscript does not clearly establish whether generated hints contain final answers or near-complete solutions, or whether such content is restricted. Please clarify whether the generator can access reference answers or solutions, including through its inputs, and how answer-bearing content in student attempts or memory is handled. This matters because hints are supplied directly to the student during generator evaluation: a hint that reveals the answer can receive a high reward without demonstrating useful strategic guidance. This is not an allegation of test-data leakage, nor an objection to privileged information for the teacher. Rather, the actual information content of hints needs clarification. Representative examples, an analysis of answer-containing hints, and comparisons with answer-bearing content removed would help explain what the generator learns and how it contributes to the final hint-free performance.

4. Missing comparisons with more closely related adaptive guidance methods

The paper's main contribution concerns hint generation, but its performance comparisons largely involve methods with different RL/self-distillation objectives. The related-work section discusses closer approaches to adaptive skill generation and memory evolution, including SEED, UCOB, and PMD, yet these are not directly compared. The RLSD baseline is informative about the benefit of the overall added framework, but does not establish its advantage over existing adaptive guidance methods; replacing the generator with GPT-5.5 does not resolve this gap either. Why were these closer methods omitted? A comparison with compatible representatives under matched student models, tasks, and budgets, controlling the student objective where feasible, would better establish the incremental value of the proposed hint-learning mechanism. If such comparisons are infeasible, the authors should explain the limitations.

Questions for authors

Please refer to the weakness part.

7 comments
AuthorsOpenReview ↗

Response to Reviewer h2yw (1 / N)

We thank the reviewer for recognizing the motivation behind our work, the comprehensive evaluation across diverse tasks and model scales. We also appreciate the reviewer's constructive suggestions and address the concerns in detail below.

W1: Justification of the Generator's Proxy Objective

We thank the reviewer for raising this important point. We agree that hint-conditioned task returns and downstream distillation improvements are not necessarily equivalent. Our motivation is that hints that help the current student solve a task provide potentially useful guidance that the student has not yet fully internalized, making them promising candidates for teacher conditioning in self-distillation. Importantly, the generator is not directly provided with ground-truth answers or reference solutions; its inputs consist of the task, the student's attempt, and retrieved memory. We provide further clarification and analysis of answer-related information in our response to W3.

Directly optimizing downstream distillation improvement would require repeatedly updating and evaluating the student for each candidate hint, introducing substantial computational overhead. Moreover, improvements from individual distillation updates may not reliably reflect a hint's long-term utility as the student continuously evolves. Therefore, we adopt hint-conditioned task returns as a practical, outcome-driven proxy that allows the generator to adapt its guidance to the student's current capabilities. While the comprehensive evaluation within this paper does not establish a strict alignment between the two objectives, they provide empirical support for the effectiveness of our proxy objective.

AuthorsOpenReview ↗

Response to Reviewer h2yw (2 / N)

W2: Unclear rollout-budget alignment and contribution of trajectory reuse

We thank the reviewer for highlighting these concerns. We conduct additional experiments on WebShop to evaluate EvoHint under comparable computational budgets and disentangle the contributions of trajectory reuse and teacher-based token-level supervision.

1. Compute-Matched Comparison.

To assess whether EvoHint's improvements primarily result from additional computation, we double the training steps of RLSD from 150 to 300, resulting in the same total number of training rollouts (38,400) and roughly the same total GPU hours as EvoHint. As shown below, doubling RLSD's training budget yields only marginal improvements, while EvoHint consistently outperforms RLSD under comparable computational budgets.

BackboneMethodTask bszStepsStudent training rolloutsTeacher training rolloutsScore ↑Acc. (%) ↑
Qwen3-1.7BRLSD1615019,200075.2 ± 1.257.0 ± 0.7
RLSD (2× steps)1630038,400076.657.0
EvoHint (RLSD-based)1615019,20019,20083.2 ± 2.462.6 ± 0.6
Qwen2.5-3B-InstructRLSD1615019,200084.0 ± 1.266.6 ± 0.8
RLSD (2× steps)1630038,400084.567.0
EvoHint (RLSD-based)1615019,20019,20086.6 ± 1.069.0 ± 0.6

Specifically, doubling RLSD's training budget improves WebShop Score by only +1.4 and +0.5 points for Qwen3-1.7B and Qwen2.5-3B, respectively. In contrast, EvoHint achieves improvements of +6.6 and +2.1 points over the corresponding compute-matched RLSD baselines. These results suggest that EvoHint's gains cannot be explained solely by increased training computation.

2. Contribution of Trajectory Reuse and Teacher-Based Reweighting.

We further conduct ablation experiments to distinguish the benefits of adaptive teacher conditioning from those of additional RL training through trajectory reuse. Specifically, we compare EvoHint with and without trajectory reuse, as well as a variant that retains hint-conditioned trajectory reuse but removes teacher-based token-level reweighting.

MethodScore ↑Acc. (%) ↑
RLSD (Baseline)75.2 ± 1.257.0 ± 0.7
EvoHint (w/o Trajectory Reuse)81.660.8
EvoHint (w/ Reuse, w/o Teacher Reweighting)76.857.8
EvoHint (Full)83.2 ± 2.462.6 ± 0.6

All experiments use Qwen3-1.7B on WebShop. The baseline and full EvoHint results are averaged over three runs (mean ± std). Due to the limited time available during the rebuttal period, the two additional ablation configurations were evaluated with a single run each.*

The results reveal two key observations:

  • Teacher-based token-level supervision is the primary source of improvement. Even without trajectory reuse, EvoHint achieves a Score of 81.6, substantially outperforming RLSD (75.2). In contrast, retaining trajectory reuse while removing teacher-based reweighting yields only a modest improvement (76.8), suggesting that additional RL training alone cannot account for EvoHint's gains.
  • Trajectory reuse provides an additional benefit. Full EvoHint achieves a Score of 83.2, compared with 81.6 without reuse, suggesting a modest additional benefit from reusing hint-conditioned trajectories.

Overall, these results suggest that EvoHint's improvements primarily stem from adaptive teacher conditioning and token-level self-distillation, rather than additional computation or trajectory reuse alone.

AuthorsOpenReview ↗

Response to Reviewer h2yw (3 / N)

W3: Unclear presence and handling of answers in hints.

Thank you for raising this point. Our goal is to learn high-level guidance that generalizes across tasks, rather than task-specific answers or complete solutions. The generator receives the task, the student’s attempt, and retrieved experience; ground-truth answers and reference solutions are not explicitly provided. The prompt asks it to identify issues in the student’s approach and explicitly instructs: “Give strategy guidance only; do not solve the task.”

For example, hints on SearchQA include:

Check if the top results mention the same person for both conditions before finalizing your answer.

Apply disambiguation add-ons: If your initial search yields multiple terms, refine by adding disambiguation qualifiers to pinpoint the most accurate term.

On WebShop, examples include:

After clicking on a result, carefully review the details to confirm if it fits all criteria before making a decision.

Utilize the back button strategically to revisit previous pages of results if an item with the desired specifications is not immediately found.

These examples encourage checking evidence, resolving ambiguity, verifying constraints, and revising unsuccessful approaches. The student must still identify the answer or select an appropriate product and execute the task. We will clarify the generator’s inputs and prompting instructions and include these examples in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.