- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Search-E1, an iterative self-improvement framework for search-augmented reasoning agents. The core idea is to alternately apply standard GRPO and on-policy self-distillation (OPSD) over multiple training rounds: GRPO is used to explore different search and reasoning trajectories, while OPSD leverages successful trajectories generated by the current policy itself to consolidate the effective search behaviors within them. Specifically, the authors select relatively efficient successful trajectories from multiple sibling trajectories of the same question, and retain only the search queries within them to construct a search skeleton that serves as the privileged context. The teacher and the student are initialized from the same policy checkpoint, but the teacher additionally observes this privileged context; the teacher's distribution is then distilled into the student via token-level forward KL on the student's own on-policy trajectory. To preserve the exploration capability needed for subsequent GRPO rounds, the authors further propose an entropy-aware handoff to prevent OPSD from making the policy distribution overly peaked; they also adopt question recycling, which removes questions that have already been consistently solved, concentrating the training budget on samples that still offer learning value. The experiments cover Qwen2.5-3B/7B in both Base and Instruct variants, as well as 7 single-hop and multi-hop QA benchmarks. The best average EM reported by the paper is 0.440 (3B) and 0.482 (7B). The paper also conducts fairly systematic ablations, including GRPO-only, SFT on mined trajectories, one-time distillation, different privileged contexts, different reference selections, and settings without the entropy-aware handoff or question recycling.
Critical strengths
-
The experiments are fairly thorough, and the effectiveness of the core method is validated convincingly. Search-E1 does not merely demonstrate performance gains on the final benchmarks, but rules out several simple explanations through fairly systematic controlled ablations.
-
The analysis of the form of privileged supervision has good methodological value. The result shows that in agent distillation, both the form of privileged information and the way reference trajectories are selected significantly affect the distillation outcome—providing "more information" is not necessarily better.
Critical weaknesses
-
Insufficient transparency and consistency in external baseline comparisons. The paper states that baseline results are taken from prior work under matching evaluation protocols rather than reproduced under unified experimental conditions. However, cross-checking the Search-R1 [1] results reveals inconsistencies: the reported numbers appear to correspond to different RL variants (PPO/GRPO) across model scales, while some results, particularly those for 7B-Base, do not match either variant in the original paper. This raises questions about the provenance of these results and the consistency of the baseline selection criteria. Furthermore, although some retrieval settings are aligned with Search-R1, the paper does not systematically establish whether all external baselines share comparable retrieval configurations, search budgets, and training conditions. Consequently, the main comparisons are not demonstrably strict apples-to-apples evaluations, limiting the strength of the evidence supporting the paper's SOTA claims.
-
Limited experimental scope relative to the broad framing of "self-evolving search agents." First, although OPSD does not require an external teacher, the overall training pipeline still relies on verifiable ground-truth answers for GRPO outcome rewards and successful-trajectory selection in OPSD. Thus, the extent to which the proposed framework enables self-evolution without external supervision should be more clearly characterized. Second, the experiments are limited to Qwen2.5 3B/7B Base and Instruct models, focusing exclusively on QA/search tasks with a fixed retriever and Wikipedia corpus. Given the availability of newer backbones such as Qwen3 and Qwen3.5, as well as alternative model families such as Llama, it remains unclear whether the reported improvements generalize across different model generations and architectures. These limitations leave the broader applicability of the proposed mechanism insufficiently validated.
Reference
[1] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning, July 2025. arXiv:2503.09516 [cs].
Questions for authors
-
Please clarify the specific data sources and selection rules for each external baseline in Tables 1–2. In particular, some Search-R1 results do not appear to fully match the corresponding results in its original paper. Please indicate whether each set of results comes from the original paper, a reproduction from public checkpoints, or a reproduction reported in another paper, and explain the selection criteria applied when multiple public results exist.
-
Does Search-E1 retain stable gains on newer or different model families? Could you add experiments on at least one Qwen3/Qwen3.5 model, or provide results on the Llama series? This would help determine whether the method's gains are a Qwen2.5-specific phenomenon or generalize to a broader range of backbones.
W1: Reward shaping and alternation
We also test your suggestion that GRPO might achieve similar gains without OPSD by directly rewarding successful trajectories with fewer search calls. For this comparison, we added a GRPO baseline whose reward follows the same reference-selection rule. For each question, we use the same group size of 5 and rank successful rollouts from fewest to most search calls. If 2 rollouts use the same number of search calls, the one with the shorter search skeleton ranks higher. Let Mq be the number of distinct (search count, skeleton length) pairs among these successful rollouts, and let ki be the rank of rollout i, starting from 0 for the best pair. Rollouts tied on both criteria have the same rank. The reward is
Here, ci=EM(a^i,a) and λ>0.
Incorrect rollouts always receive 0, while correct rollouts receive a reward between 1 and 1+λ. Among correct rollouts, fewer search calls always give a higher reward, and skeleton length matters only when the call counts are equal. For any given rollout group containing a correct answer, the highest-reward trajectories are exactly those eligible to be selected as the Search-E1 reference. If all rollouts are incorrect, all rewards are 0. The reward therefore preserves the same priority: correctness first, then search count, then search-skeleton length.
We believe the difference is how this preference becomes a training signal. In GRPO, the reward is converted into a group-normalized advantage shared across the policy-generated positions of the trajectory (Section 2.1). This reinforces the preferred trajectory as a whole, but does not provide a separate teaching signal for each search decision. OPSD conditions the teacher on the selected search skeleton and aligns the student to the teacher distribution along the student's own rollout. The supervision depends on the current student prefix and covers what to search for, the query order, and the reasoning between searches. This provides a more detailed signal for credit assignment.
We use Qwen2.5-3B-Instruct with Wikipedia retrieval and a maximum of 4 search calls during both training and evaluation. This is the original setting also included for reference in our response to Reviewer jcX8. Each reward-shaping run uses 700 optimizer steps, following the step-matched GRPO control in Appendix E and matching the executed optimizer-step budget of Search-E1. We keep the initial checkpoint, training data, question-recycling rule, hardware and software environment, and decoding settings fixed across the GRPO controls. Question recycling remains based on answer correctness rather than the shaped reward. We sweep λ over 0.05, 0.10, 0.20 and select the coefficient and checkpoints using the held-out validation set.
The GRPO-only baseline is the 0.427 peak reported in Appendix E, rather than the 0.426 result under the original GRPO schedule in Table 3. Mean search calls are measured over the same 7 evaluation benchmarks. The multi-hop column averages HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle; Search-E1's 0.380 is the same original multi-hop result reported in our response to Reviewer jcX8.
With the validation-selected coefficient of 0.10, the ranking reward reaches 0.433 average EM with 1.98 searches per rollout. Increasing the coefficient to 0.20 further reduces searches to 1.84, while EM is 0.431. Search-E1 reaches 0.440 EM with 2.12 searches, exceeding the selected reward-shaping baseline by 0.7 pp overall and 0.8 pp on multi-hop QA. The reward improves over GRPO-only, while Search-E1 retains a gain under the same trajectory preference. This comparison supports the value of OPSD's token-level supervision for credit assignment. The query-order ablation (Appendix D, Table 6) further shows that the order of the selected queries carries useful information beyond search count.
Alternating the stages lets the next GRPO round start from the distilled policy, while fresh rollouts provide updated references. Table 12 in Appendix G compares alternating distillation with distillation once after all GRPO rounds. Applying the full distillation budget after all GRPO rounds gives 0.432 EM, compared with 0.440 when the stages alternate.