- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper argues that cheap simulated environments for training search agents with RL can be factually correct and still teach the wrong search behavior. They can withhold evidence that a good query would find on the real web, or hand decisive evidence to a vague query. The authors propose SEA, a trained model that acts as the search environment. For each tool call, it either builds a response by selecting and reranking pages from a local cache and a memory of past real-web results, or it calls the real web when it predicts a decision-relevant gap it cannot fix locally. The results are strong. SEA-RL-35B improves over its base model by about 23 points on average across six benchmarks (e.g., 31.0→74.0 on BrowseComp). It outperforms all of the open research agents reported in Table 1, beats the much larger Qwen3.5-122B on every benchmark, and is competitive with DeepSeek-V4-Flash. In the controlled 4B comparison, SEA beats Cached-Web RL on all four benchmarks and Real-Web RL on three, at about 5% of the latter's API cost and 17% less training time. The evaluation setup is sound: SEA and Oracle Memory are removed at test time, so the policy is judged purely on the real web. The environment is also evaluated directly through continuation success from fixed decision points (60.7% Pure Local vs. 85.3% SEA vs. 90.2% Real-Web), which tests the environment more meaningfully than overall recall.
The main concerns are with how parts of the evaluation are supported. First, the same LLM, DeepSeek-V4-Flash, creates all 5,000 SEA training labels and is also the primary annotator for the Figure 2 feedback-quality metrics. SEA is trained to match that model's judgments, so these metrics may partly reward agreement with the teacher. Annotator disagreement rates and results without DeepSeek are not reported. Second, Table 1 reports single runs with no mean, standard deviation, or pass@k. Third, key information is missing: the Real-Web branch (90.2%), which anchors the "83.4% of the gap closed" claim, appears only in the text and not in Figure 2, and the annotation prompt for response selection is not provided. Fourth, the evaluation setup is described for the authors' own models, but not for the baselines, so it is unclear whether those numbers are comparable or how much the agent harness itself contributes. Finally, some ablations are missing: how often the untrained base model chooses the real web versus local evidence and how training changes this, and whether a different annotator model would change SEA or the final policy. Table 1 would also be easier to read with best results highlighted and an average column.
Due to the above reasons, I am giving this paper a weak accept, and will consider changing the rating to a strong accept if the authors can address above concerns.
Critical strengths
- S1 Results are strong - With just 2000 and 3000 training samples for picking and routing & ~2500 QA pairs, they were able to improve the results by more than ~10-40% points on an average (across benchmarks) over base model and very comparable results with much larger models like DeepSeek-v4-Flash, Qwen-3.5-122B models and all the open research agent models, falling behind only Kimi-2.6 and GPT5.4 (both significantly larger, so to be expected).
- S2 Eval settings are highly competitive, honest and relevant in current AI landscape For Example
- Evaluated both AI search policy as well as the SEA environment system
- For both evals, Picked strong benchmarks to showcase their results.
- For inference, removed SEA AI system and Oracle Memory – so the search AI policy’s true merit can be measured
- Significantly closed the Sim-to-Real gap (more in S3) with a SEA-driven environment while achieving 95% lower API cost and 17% lower time cost than a real-web RL environment.
- S3 The RL environment itself is AI-driven The RL environment is driven be a model to dynamically determine whether to use local search from local docs or real web API. Besides that, the AI model driving the search environment is now trained on objectives to rank as closely to real search APIs as possible. They were successfully able to prove the gap between sim2real is closed as per ablation study in Figure 2. Instead of using dataset-level success or recall, they measured continuation success i.e. given a decision point, continue with a fixed policy. SEA-optimized closed the gap with Pure Local by a lot: continuation success of PureLocal=60.7%, SEA-optimized =85.3% and Real-Web=90.2%.
Critical weaknesses
-
W1 (Readability) Not a weakness per say, but a request for better readability of results. In Table 1, please highlight the best performing model. Also a column showing an average performance across all benchmarks is missing.
-
W2 (Strengthening Experimental Results and Evaluation Settings)
- Same model builds SEA's labels and grades SEA. DeepSeek-V4-Flash writes all 5,000 SEA training labels: the 3,000 routing examples and the 2,000 response-selection examples (Appendix G). It is also the main annotator for the feedback-quality numbers in Figure 2 (left): lead/fact recall, extra evidence, and answer jump (Appendix B.1). Since SEA is trained to copy DeepSeek's judgments, a DeepSeek-led evaluation may favor SEA for agreeing with its teacher, not for being closer to real search. The second annotator (Qwen3.5-122B) and the Gemini-3.1-Pro tie-breaker reduce this risk but don't remove it, and the paper doesn't report how often the annotators disagreed or how the results change without DeepSeek.
- On top of that, DeepSeek-V4-Flash is also a baseline in Table 1 (73.2 on BrowseComp), so a model being compared against also shapes the training signal. The authors should report annotator agreement for Figure 2, show the Figure 2 metrics graded by Qwen and Gemini alone, and explain why DeepSeek-V4-Flash was chosen as the annotator.
- Authors should run experiments in Table 1 atleast 3 or more times and report mean / standard deviation. Alternate can be to run it N times and report Pass@3 results or Worst@3 results.
-
W3 (Missing Agent Harness settings for Table 1 and suggestion to rerun experiments under different agent harnesses)
- The paper doesn't say which agentic harness (custom or open-source) was used for experiment results in Table 1. This should be specified. The abstract calls them "reported" results, which suggests the numbers are taken from other papers, each with its own agent harness, tools, turn limits, and prompts. If so, the comparison mixes the effect of the model with the effect of the harness. The authors should state which baseline numbers they ran themselves and which are taken from other papers. Ideally, they would also re-run key baselines under the same harness as SEA-RL.
- Furthermore, An interesting extra experiment would be to evaluate the reported models under several agent harnesses, such as Claude Code, Codex, and open-source options like OpenHands. This would show how much a strong harness improves search performance on its own, and whether SEA-RL's gains still hold when every model runs under the same harness.
-
W4 (Major missing results from plots and information)
- From Section 3.4, Continuation Outcomes, it is clear that real-web got a score of 90.2% but the entire 4th branch is missing from Figure 2 (left). Request to authors to please update the figure with results from Real-Web ablation.
- The judge AI's (the 3rd model) instructions for picking, routing and grading are not provided in the paper.
Questions for authors
Q1. What is the general routing tendency of untrained, base model between picking google API or local docs and how does training the model change it’s behavior. When AI policy is trained with SEA + w/o routing supervision, how well does the model do on routing specifically? How well does the base model do on routing?
Q2. Does the choice of Judge AI (which was critical in creating the picking and routing dataset for SEA) affects the performance of SEA environment model and SEA-RL? This could serve as a very interesting ablation.
4 comments
Response to Reviewer yomH (2)
W3 (Missing Agent Harness settings for Table 1 and suggestion to rerun experiments under different agent harnesses):
The paper doesn't say which agentic harness (custom or open-source) was used for experiment results in Table 1. This should be specified. The abstract calls them "reported" results, which suggests the numbers are taken from other papers, each with its own agent harness, tools, turn limits, and prompts. If so, the comparison mixes the effect of the model with the effect of the harness. The authors should state which baseline numbers they ran themselves and which are taken from other papers. Ideally, they would also re-run key baselines under the same harness as SEA-RL.
Furthermore, An interesting extra experiment would be to evaluate the reported models under several agent harnesses, such as Claude Code, Codex, and open-source options like OpenHands. This would show how much a strong harness improves search performance on its own, and whether SEA-RL's gains still hold when every model runs under the same harness.
W2: Authors should run experiments in Table 1 at least 3 or more times and report mean / standard deviation. Alternate can be to run it N times and report Pass@3 results or Worst@3 results.
- Thank you for highlighting the need to clarify the evaluation setup and result sources. Our own evaluations use the MiroThinker-1.7 deep research agent harness. We have revised Table 1 to distinguish these results from scores taken from the respective models’ official technical reports, which are marked with an asterisk (*).
- We agree that differences in harnesses, tools, prompts, and interaction budgets can affect performance. To provide a controlled comparison and assess evaluation variability, we have added experiments for the five models below under the same MiroThinker-1.7 harness. Each model is evaluated in three independent runs, and we report mean ± standard deviation.
| Model | BrowseComp | GAIA | DSQA | xBench | WideSearch-EN | AR-Deep |
|---|---|---|---|---|---|---|
| LiteResearcher-4B | 29.4 ± 1.3 | 70.6 ± 1.5 | 18.3 ± 1.0 | 12.0 ± 1.2 | 22.2 ± 1.4 | 4.5 ± 0.5 |
| Qwen3.5-122B-A10B | 38.4 ± 1.1 | 82.1 ± 1.2 | 66.5 ± 0.9 | 52.4 ± 1.3 | 67.5 ± 1.1 | 9.4 ± 0.7 |
| DeepSeek-V4-Flash | 72.0 ± 0.9 | 83.0 ± 1.0 | 89.4 ± 0.7 | 67.5 ± 1.1 | 75.8 ± 0.9 | 22.4 ± 0.8 |
| SEA-RL-4B | 52.6 ± 0.8 | 78.4 ± 1.1 | 67.4 ± 0.6 | 51.0 ± 1.0 | 68.8 ± 0.8 | 14.3 ± 0.8 |
| SEA-RL-35B | 74.0 ± 0.7 | 85.4 ± 0.8 | 80.6 ± 0.6 | 60.0 ± 0.9 | 73.8 ± 0.7 | 39.0 ± 0.8 |
W4 (Major missing results from plots and information):
- Thank you for pointing out the omission. We added the Real-Web continuation success rate of 90.2%, previously reported in §3.4, to Figure 2 (left). The Lead Recall, Fact Recall, Extra Evidence, and Answer Jump metrics all use the Real-Web response to the same action as the reference. We add the prompts in the paper.
Q1: What is the general routing tendency of untrained, base model between picking google API or local docs and how does training the model change it’s behavior. When AI policy is trained with SEA + w/o routing supervision, how well does the model do on routing specifically? How well does the base model do on routing?
- Thank you for this question. We directly evaluate SEA’s routing behavior on held-out examples excluded from SEA training and model selection. Prompt-based SEA uses the base model without SEA-specific fine-tuning.
| SEA variant | REAL selection rate (%) | REAL recall (%) ↑ |
|---|---|---|
| Prompt-based SEA | 4.3 | 67.8 |
| Trained SEA | 5.1 | 89.5 |
SEA Training helps SEA identify necessary real-tool calls while preserving predominantly local execution.
Post-rebuttal comments
I thank the authors for their detailed responses and additional experiments. The clarified harness setup, the independent re-grading of Figure 2, the added routing-accuracy results, and the Real-Web continuation result address several of my concerns. Using a learned agent to manage the training environment is a neat and practical idea for reducing the cost of search RL. However, it builds on established principles the paper itself cites, namely selective real-environment use (as in MBPO) and decision-aware model learning, so I view its novelty and impact as solid but not high enough to warrant a higher score. I am keeping my score at weak accept for the following reasons:
-
Routing baselines: Building on the concern raised by Reviewer uGeG and Reviewer 9kF1 about missing routing baselines, I appreciate the authors' new comparison with random and retrieval-confidence routers at matched real-tool cost. However, the comparison is not like-for-like: each router is only used to serve a fixed SFT policy, while SEA's claimed benefit comes from RL training inside its environment, and no policy was RL-trained with the alternative routers and then evaluated on the real-web benchmarks as in Table 2. Without that experiment, I cannot attribute the real-web gains to the learned routing rather than to cheap evidence reuse plus occasional real calls.
-
Controlled baselines and failure analysis: I appreciate the controlled re-evaluation under the MiroThinker-1.7 harness with three runs per model. However, no analysis shows whether the harness itself (e.g., tool-call formatting, turn limits, or context-length cutoffs) contributes to failures for SEA-RL or the baselines. More broadly, failure analysis remains limited: Appendix J offers two hand-picked case studies and a brief qualitative note on the 25 remaining BrowseComp errors, but no systematic breakdown of failure types or their frequency across benchmarks.
-
Annotator dependence: The independent re-grading of Figure 2 by Qwen3.5-122B and Gemini-3.1-Pro shows that the evaluation does not depend on DeepSeek's vote. However, my Q2 asked whether the choice of annotator for SEA's training data affects SEA and SEA-RL, and this ablation was not run. All 5,000 routing and response-selection labels still come from DeepSeek-V4-Flash, so it remains unclear whether SEA's behavior, and the downstream policy gains, depend on which model served as the teacher.
Overall, the paper addresses a relevant problem with strong empirical results and substantial cost savings, but the remaining gaps prevent me from raising my score.
Thank you for the follow-up.
Q4: However, no analysis shows whether the harness itself (e.g., tool-call formatting, turn limits, or context-length cutoffs) contributes to failures for SEA-RL or the baselines.
We evaluated SEA-RL-4B on 100 randomly sampled questions from each of BrowseComp and AR-Deep, varying the tool-call format, turn limit, and context limit.
| Tool-call format | Max. turns | Context limit | BrowseComp | AR-Deep |
|---|---|---|---|---|
| Native XML | 100 | 256K | 49/100 | 12/100 |
| OpenAI | 100 | 256K | 53/100 | 14/100 |
| Native XML | 25 | 256K | 28/100 | 5/100 |
| Native XML | 50 | 256K | 42/100 | 11/100 |
| Native XML | 75 | 256K | 49/100 | 11/100 |
| Native XML | 100 | 64K | 30/100 | 3/100 |
| Native XML | 100 | 128K | 47/100 | 10/100 |
- Although the model was trained with native XML tool calls, the OpenAI format yields slightly higher scores on both samples. However, Native XML is more stable and requires far fewer tool-call regenerations (13 vs. 1,323; each tool call is regenerated at most 3 times when invaild). On AR-Deep, the OpenAI-style interface issues fewer search queries and visits more pages, with a lower median peak prompt length (180K vs. 234K tokens) and fewer runs terminated by the context limit (17/100 vs. 47/100). Shorter tool outputs may help explain its lower context usage.
- Increasing the budget from 25 to 75 turns improves performance on both benchmarks. Increasing the limit from 64K to 256K improves scores from 30/100 to 49/100 on BrowseComp and from 3/100 to 12/100 on AR-Deep. Many unsuccessful runs exhaust their turn budget.
- These results show that harness settings affect SEA-RL-4B's measured performance. We will extend this analysis to the baselines.
Q3 and Q5: However, my Q2 asked whether the choice of annotator for SEA's training data affects SEA and SEA-RL, and this ablation was not run. the comparison is not like-for-like: each router is only used to serve a fixed SFT policy, while SEA's claimed benefit comes from RL training inside its environment, and no policy was RL-trained with the alternative routers and then evaluated on the real-web benchmarks as in Table 2.
- Thanks for the question. We added end-to-end RL experiments in Prompt SEA and Trained SEA and evaluated the resulting policies on the real-web benchmarks.
| Training setting | BC | GAIA | DSQA | AR-Deep |
|---|---|---|---|---|
| Cached-Web RL | 47.3 | 69.6 | 63.7 | 8.1 |
| Real-Web RL | 51.7 | 80.6 | 66.2 | 10.1 |
| Prompt SEA RL | 50.2 | 74.4 | 65.2 | 11.0 |
| Trained SEA RL | 52.6 | 78.4 | 67.4 | 14.3 |
- Trained SEA outperforms Prompt SEA on all four benchmarks. On BrowseComp, it improves the score from 50.2 to 52.6 while reducing the average number of turns from 30.7 to 29.0. These results support the benefit of the trained environment for policy RL.
Given the limited rebuttal period and the substantial time required for RL training, we will extend our experiments in follow-up work. This will include end-to-end RL training in additional search environments (e.g., those using random and retrieval-confidence routers), evaluation and analysis on more benchmarks, and an investigation of how the choice of model used to annotate SEA’s training data affects search policy RL.
Thanks again for your constructive feedback! Your constructive comments have been invaluable in helping us strengthen the empirical rigor and clarity of our work.
Response to Reviewer yomH (1)
Both independent judges rank trained SEA best on all four metrics, preserving Figure 2’s conclusions without using DeepSeek’s judgments.