- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes EvoHint, an RL + on-policy self-distillation training algorithm for LLMs. The method features alternatively updating a hint generator and a task solver, while accumulating a persistent memory that can be re-used in test-time. The authors empirically validate their method on web-search and mathematical reasoning benchmarks. I can see marginal improvement but it is not highly consistent.
Critical strengths
The method generally shows 1~3 points improvements over existing baselines in the provided experiments.
Critical weaknesses
-
Novelty is limited. The method appears as an interpolation of existing ideas to me. (1) RL + OPSD from RLSD/SDAR; (2) RL + ideas bank + test-time reusing like [1], which also discusses proactive/reflective hint. (3) co-training hint generation & solver is a form of hierachical reasoning models [2]
-
Statistical significance of improvements. None of the tables shown in this paper provide variance/standarad error. And RL training is known to have large variance. I believe it is necessary to at least repeat the main table for several times and report steady improvment.
-
End-to-end time overhead. The method seems complicated and to have large time/computational overhead. The authors should report end-to-end time/computational overhead. And if possible, compare with baselines in matched compute setting.
-
Ablated effectiveness of each design components. Current, the contribution of each component (co-training, hint memory) is not very clear to me.Main message of Section 5.2 is not clear.
[1] Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs; https://openreview.net/forum?id=K9GxQKEJzS
[2] Reinforce LLM Reasoning through Multi-Agent Reflection; https://icml.cc/virtual/2025/poster/46364
Questions for authors
Please see weaknesses.
4 comments
Response to Reviewer 1Hxk (2 / N)
W3: End-to-End Computational Overhead
We thank the reviewer for this suggestion. To assess the computational overhead, we conduct additional experiments on WebShop by doubling the training steps of RLSD (from 150 to 300), resulting in roughly the same total GPU hours and number of training rollouts as EvoHint. As shown below, doubling RLSD's training budget yields only marginal improvements (75.2 → 76.6 for Qwen3-1.7B and 84.0 → 84.5 for Qwen2.5-3B). In contrast, EvoHint consistently outperforms RLSD under comparable computational budgets, achieving improvements of +6.6 and +2.1 Score points, respectively.
| Backbone | Method | Bsz | Steps | Student training rollouts | Teacher training rollouts | Score ↑ | Acc ↑ |
|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | RLSD | 16 | 150 | 19,200 | 0 | 75.2 ± 1.2 | 57.0 ± 0.7 |
| RLSD (2× steps) | 16 | 300 | 38,400 | 0 | 76.6 | 57.0 | |
| EvoHint (RLSD-based) | 16 | 150 | 19,200 | 19,200 | 83.2 ± 2.4 | 62.6 ± 0.6 | |
| Qwen2.5-3B-Instruct | RLSD | 16 | 150 | 19,200 | 0 | 84.0 ± 1.2 | 66.6 ± 0.8 |
| RLSD (2× steps) | 16 | 300 | 38,400 | 0 | 84.5 | 67.0 | |
| EvoHint (RLSD-based) | 16 | 150 | 19,200 | 19,200 | 86.6 ± 1.0 | 69.0 ± 0.6 |
Response to Reviewer 1Hxk (3 / N)
W4: Effectiveness of Individual Components
We thank the reviewer for this suggestion. To better clarify the contributions of individual components, we separate our ablation analysis into training-time and inference-time evaluations, using Qwen3-1.7B on WebShop.
Training-Time Ablation
We compare the following configurations to evaluate the contributions of co-training and evolving memory.
| Method | Score ↑ | Acc. (%) ↑ |
|---|---|---|
| RLSD (baseline) | 75.8 | 56.8 |
| EvoHint (Fixed Memory + Co-Training) | 79.8 | 60.0 |
| EvoHint (Evolving Memory + Co-Training) | 85.6 | 63.2 |
This comparison isolates the benefit of learning adaptive hints through co-training and the additional contribution of evolving the hint memory during training.
Inference-Time Ablation (Clarification of Section 5.2)
We further evaluate whether the learned memory should be retained during inference, using the same trained EvoHint student.
| Method | Score ↑ | Acc. (%) ↑ |
|---|---|---|
| EvoHint (w/o Memory) | 85.6 | 63.2 |
| EvoHint (w/ Memory) | 87.2 | 66.8 |
The results show that inference-time memory retrieval further improves WebShop performance by +1.6 Score points and +3.6 accuracy points. Importantly, memory retrieval is optional at inference, and the student can operate independently without the hint generator or memory.
We will revise Section 5.2 to clarify the distinction between the role of evolving memory during training and optional memory retrieval during inference.
Thank you for the additional results. However, I still cannot see a clear improvement over SDAR, especially under matched-time comparison. The method's novelty is still limited, and the it does not offer a strong enough reason to add up so much additional compute and implementation complexity. I maintain my score of clear rejection.
Response to Reviewer 1Hxk (1 / N)
We thank the reviewer for their constructive suggestions and address the concerns below.
W1: Limited Novelty
We thank the reviewer for highlighting these related works. Building upon prior RL + OPSD methods (e.g., RLSD and SDAR), our key novelty lies in learning adaptive teacher conditioning that addresses the student's specific mistakes and is optimized through outcome feedback. We do not claim hint memory, reflection, or co-training as standalone contributions. Rather, our contribution is to make teacher conditioning adaptive and learnable within the OPSD framework, enabling more effective self-distillation.We also conduct extensive experiments across both agentic tasks and mathematical reasoning benchmarks to demonstrate the effectiveness and generality of our approach.
W2: Statistical significance and run-to-run variability.
We have added three-run results across WebShop, SearchQA, and Math Reasoning, reporting mean ± standard deviation. For the two agentic tasks, we compare RLSD, SDAR, and EvoHint (RLSD-based) across three model backbones; for Math Reasoning, we compare RLSD and EvoHint across two backbones. Compared with RLSD, EvoHint achieves average gains of approximately 3–5 points on WebShop, 2–3 points on Math Reasoning, and 1-3 points on SearchQA, supporting consistent improvements.
WebShop
Search-QA
Math Reasoning