acceptodds
Desk rejected at ICLR 2027

Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Outcome based reinforcement learning provides a simple and effective recipe for training search augmented reasoning agents. Recent work seeks further improvements through stronger external teachers, auxiliary process supervision, specialized reward shaping, and increasingly elaborate training pipelines. This raises a natural question: do search agents really need this additional machinery to continue improving? We present Search-E1, a simple self evolution framework that alternates vanilla GRPO with on policy self distillation (OPSD). After each GRPO round, Search-E1 mines sibling trajectories generated by the policy itself and exposes the search skeleton of an efficient successful trajectory, that is, the sequence of queries it issued, as privileged context. A token level forward KL objective then aligns the policy under its standard inference context with its own distribution under this privileged context. In this way, the policy converts contrasts among its own rollouts into dense token level supervision without an external teacher, auxiliary reward model, or additional annotation. On seven single hop and multi hop QA benchmarks, Search-E1 reaches an average Exact Match of with Qwen2.5-3B and with Qwen2.5-7B, achieving the best average performance among all compared methods at both scales. These results show that alternating outcome based exploration with self distillation provides a simple and effective path to iterative improvement in search augmented reasoning.

Primary area:
reinforcement learning
Keywords:
Search-augmented Reasoning, On-policy Self-distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xRifOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Search-E1, an iterative self-improvement framework for search-augmented reasoning agents. The core idea is to alternately apply standard GRPO and on-policy self-distillation (OPSD) over multiple training rounds: GRPO is used to explore different search and reasoning trajectories, while OPSD leverages successful trajectories generated by the current policy itself to consolidate the effective search behaviors within them. Specifically, the authors select relatively efficient successful trajectories from multiple sibling trajectories of the same question, and retain only the search queries within them to construct a search skeleton that serves as the privileged context. The teacher and the student are initialized from the same policy checkpoint, but the teacher additionally observes this privileged context; the teacher's distribution is then distilled into the student via token-level forward KL on the student's own on-policy trajectory. To preserve the exploration capability needed for subsequent GRPO rounds, the authors further propose an entropy-aware handoff to prevent OPSD from making the policy distribution overly peaked; they also adopt question recycling, which removes questions that have already been consistently solved, concentrating the training budget on samples that still offer learning value. The experiments cover Qwen2.5-3B/7B in both Base and Instruct variants, as well as 7 single-hop and multi-hop QA benchmarks. The best average EM reported by the paper is 0.440 (3B) and 0.482 (7B). The paper also conducts fairly systematic ablations, including GRPO-only, SFT on mined trajectories, one-time distillation, different privileged contexts, different reference selections, and settings without the entropy-aware handoff or question recycling.

Critical strengths
  • The experiments are fairly thorough, and the effectiveness of the core method is validated convincingly. Search-E1 does not merely demonstrate performance gains on the final benchmarks, but rules out several simple explanations through fairly systematic controlled ablations.

  • The analysis of the form of privileged supervision has good methodological value. The result shows that in agent distillation, both the form of privileged information and the way reference trajectories are selected significantly affect the distillation outcome—providing "more information" is not necessarily better.

Critical weaknesses
  • Insufficient transparency and consistency in external baseline comparisons. The paper states that baseline results are taken from prior work under matching evaluation protocols rather than reproduced under unified experimental conditions. However, cross-checking the Search-R1 [1] results reveals inconsistencies: the reported numbers appear to correspond to different RL variants (PPO/GRPO) across model scales, while some results, particularly those for 7B-Base, do not match either variant in the original paper. This raises questions about the provenance of these results and the consistency of the baseline selection criteria. Furthermore, although some retrieval settings are aligned with Search-R1, the paper does not systematically establish whether all external baselines share comparable retrieval configurations, search budgets, and training conditions. Consequently, the main comparisons are not demonstrably strict apples-to-apples evaluations, limiting the strength of the evidence supporting the paper's SOTA claims.

  • Limited experimental scope relative to the broad framing of "self-evolving search agents." First, although OPSD does not require an external teacher, the overall training pipeline still relies on verifiable ground-truth answers for GRPO outcome rewards and successful-trajectory selection in OPSD. Thus, the extent to which the proposed framework enables self-evolution without external supervision should be more clearly characterized. Second, the experiments are limited to Qwen2.5 3B/7B Base and Instruct models, focusing exclusively on QA/search tasks with a fixed retriever and Wikipedia corpus. Given the availability of newer backbones such as Qwen3 and Qwen3.5, as well as alternative model families such as Llama, it remains unclear whether the reported improvements generalize across different model generations and architectures. These limitations leave the broader applicability of the proposed mechanism insufficiently validated.

Reference

[1] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning, July 2025. arXiv:2503.09516 [cs].

Questions for authors
  1. Please clarify the specific data sources and selection rules for each external baseline in Tables 1–2. In particular, some Search-R1 results do not appear to fully match the corresponding results in its original paper. Please indicate whether each set of results comes from the original paper, a reproduction from public checkpoints, or a reproduction reported in another paper, and explain the selection criteria applied when multiple public results exist.

  2. Does Search-E1 retain stable gains on newer or different model families? Could you add experiments on at least one Qwen3/Qwen3.5 model, or provide results on the Llama series? This would help determine whether the method's gains are a Qwen2.5-specific phenomenon or generalize to a broader range of backbones.

Reviewer_vd83OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose Search-E1, an RL training method that alternates between GRPO and OPSD training to slowly improve a model's performance over several rounds. Succesful traces from the prior GRPO stage are reused as privileged information during OPSD training, retaining only the correct search commands (as opposed to the full trace). Both policy entropy and validation performance are used to select checkpoints during OPSD rounds. Experiments show strong performance across both single and multi-hop QA benchmarks, with Search-E1 performing best overall, and ablations validate the need for the alternating setup.

Critical strengths
  1. The evaluation setup and method seems solid. Using prior correct rollouts as OPSD hints is a simple and clear idea, and all reasonable aspects of the design are ablated, with lots of details in the appendix including seed robustness.
  2. Results are strong and compared to a good number of baselines, with some minor concerns (see below).

I think generally, the paper is very robust empirically, and the core method is explained well, which leans me to acceptance.

Critical weaknesses
  1. The authors empirically show that alternating OPSD and GRPO does well, but the argument for why seems weak, mainly that OPSD can 'consolidate' behaviours discovered during GRPO. But isn't this already the purpose of the GRPO training itself? One confounder is the privileged reference selection, which emphasises some behaviours, but I would be curious if these can be emulated via reward shaping, e.g. a reward for using fewer tool calls. I felt the argument was a bit vague in the paper.
  2. The overall best average performance hides Search-E1's weak multi-hop performance. For example, in Table 2, GiGPO-instruct gets 0.4325 avg on multihop, while Search-E1 gets 0.4235 avg! Averaging multi-hop separately would make it clearer and easier to determine if Search-E1 is indeed beating a reasonable number of baselines in multi-hop, especially since the authors claim that "The gains [of Search-E1] are particularly pronounced on multi hop benchmarks" in lines 70-71. This claim seems too strong, especially also considering Search-E1 is not the best method for any multi-hop benchmark at 3B size, and only 1/4 of them at 7b size!

As it stands, I think the empirics of the paper are robust, and clearing up the above issues would make me lean accept (current reject).

edit: the authors addressed these concerns in their rebuttal and as such I have raised my score.

Questions for authors
  1. Did the authors rerun or retrain any baselines themselves? I worry that directly using results from other papers may have confounding issues with e.g. hardware or software version changes. For example, changes in which GPU is used can result in large differences in training and evaluation results. (https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda)
  2. Why is Search-E1 essentially always the top-1 model for single-hop, but rarely the best model for any one multi-hop eval? At 3b size, it is never the best model in multi-hop (only on average), and at 7b size it is only the best on MuSiQue.
1 comment
AuthorsOpenReview ↗

W1: Reward shaping and alternation

We also test your suggestion that GRPO might achieve similar gains without OPSD by directly rewarding successful trajectories with fewer search calls. For this comparison, we added a GRPO baseline whose reward follows the same reference-selection rule. For each question, we use the same group size of 5 and rank successful rollouts from fewest to most search calls. If 2 rollouts use the same number of search calls, the one with the shorter search skeleton ranks higher. Let MqM_q be the number of distinct (search count, skeleton length) pairs among these successful rollouts, and let kik_i be the rank of rollout ii, starting from 0 for the best pair. Rollouts tied on both criteria have the same rank. The reward is

OutcomeReward
Incorrect (ci=0c_i = 0)ri=0r_i = 0
Correct (ci=1c_i = 1)ri=1+λ(1−kimax⁡(Mq−1,1))r_i = 1 + \lambda(1 - \frac{k_i}{\max(M_q - 1, 1)})

Here, ci=EM(a^i,a)c_i = \mathrm{EM}(\hat{a}_i, a) and λ>0\lambda > 0.

Incorrect rollouts always receive 0, while correct rollouts receive a reward between 1 and 1+λ1 + \lambda. Among correct rollouts, fewer search calls always give a higher reward, and skeleton length matters only when the call counts are equal. For any given rollout group containing a correct answer, the highest-reward trajectories are exactly those eligible to be selected as the Search-E1 reference. If all rollouts are incorrect, all rewards are 0. The reward therefore preserves the same priority: correctness first, then search count, then search-skeleton length.

We believe the difference is how this preference becomes a training signal. In GRPO, the reward is converted into a group-normalized advantage shared across the policy-generated positions of the trajectory (Section 2.1). This reinforces the preferred trajectory as a whole, but does not provide a separate teaching signal for each search decision. OPSD conditions the teacher on the selected search skeleton and aligns the student to the teacher distribution along the student's own rollout. The supervision depends on the current student prefix and covers what to search for, the query order, and the reasoning between searches. This provides a more detailed signal for credit assignment.

We use Qwen2.5-3B-Instruct with Wikipedia retrieval and a maximum of 4 search calls during both training and evaluation. This is the original setting also included for reference in our response to Reviewer jcX8. Each reward-shaping run uses 700 optimizer steps, following the step-matched GRPO control in Appendix E and matching the executed optimizer-step budget of Search-E1. We keep the initial checkpoint, training data, question-recycling rule, hardware and software environment, and decoding settings fixed across the GRPO controls. Question recycling remains based on answer correctness rather than the shaped reward. We sweep λ\lambda over 0.05, 0.10, 0.20 and select the coefficient and checkpoints using the held-out validation set.

MethodRanking bonus coefficient λ7-benchmark avg. EM4-benchmark multi-hop avg. EMMean search calls per rollout
GRPO-only (Appendix E, peak within 700 steps)00.427——
GRPO + ranking reward0.050.4300.3692.05
GRPO + ranking reward0.100.4330.3721.98
GRPO + ranking reward0.200.4310.3701.84
Search-E1 (Tables 3 and 20)00.4400.3802.12

The GRPO-only baseline is the 0.427 peak reported in Appendix E, rather than the 0.426 result under the original GRPO schedule in Table 3. Mean search calls are measured over the same 7 evaluation benchmarks. The multi-hop column averages HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle; Search-E1's 0.380 is the same original multi-hop result reported in our response to Reviewer jcX8.

With the validation-selected coefficient of 0.10, the ranking reward reaches 0.433 average EM with 1.98 searches per rollout. Increasing the coefficient to 0.20 further reduces searches to 1.84, while EM is 0.431. Search-E1 reaches 0.440 EM with 2.12 searches, exceeding the selected reward-shaping baseline by 0.7 pp overall and 0.8 pp on multi-hop QA. The reward improves over GRPO-only, while Search-E1 retains a gain under the same trajectory preference. This comparison supports the value of OPSD's token-level supervision for credit assignment. The query-order ablation (Appendix D, Table 6) further shows that the order of the selected queries carries useful information beyond search count.

Alternating the stages lets the next GRPO round start from the distilled policy, while fresh rollouts provide updated references. Table 12 in Appendix G compares alternating distillation with distillation once after all GRPO rounds. Applying the full distillation budget after all GRPO rounds gives 0.432 EM, compared with 0.440 when the stages alternate.

Reviewer_jcX8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Search-E1, which alternates GRPO with on-policy self-distillation. Successful sibling trajectories provide search skeletons as privileged teacher context, while the student uses the standard inference context. Question recycling and an entropy-based handoff rule support repeated training cycles. Experiments cover seven QA benchmarks with Qwen2.5-3B and 7B.

Critical strengths
  1. The framework reuses the policy's own successful rollouts to obtain dense supervision without an external teacher or process reward model.
  2. The ablations on privileged context, alternating training, and entropy-aware handoff provide useful evidence about when self-distillation helps further exploration.
Critical weaknesses

All experiments use Qwen2.5 and a fixed Wikipedia retrieval environment with at most four searches. Whether the approach remains effective for longer-horizon or web-based search is unclear.

Questions for authors

See the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.