acceptodds
Under review as a conference paper at ICLR 2027

Search Environment Agent for Efficient Deep Search Reinforcement Learning

10 views

Abstract

Training deep search agents with reinforcement learning (RL) on the real web is costly, motivating cheaper simulated environments. Yet we observe **reward–transfer decoupling**: training reward in the local environment improves without corresponding real-web gains. Even factually correct feedback can omit evidence obtainable by effective queries or over-assist underspecified ones, distorting which search behaviors RL reinforces. We introduce **Search Environment Agent (SEA)**, a memory-augmented environment manager that calibrates action-dependent access to evidence. Guided by **Oracle Memory**, its internal record of real-tool experience, SEA selects and reranks locally supported evidence and reserves real execution for predicted decision-relevant discrepancies that cannot be repaired locally. Paired local and real feedback supervise response selection and routing, with counterfactual policy continuations informing routing labels. Verified real observations update the memory and webpage index. Evaluated exclusively with real-web tools without SEA or Oracle Memory, SEA-RL-35B surpasses reported specialized-agent baselines across six benchmarks. In controlled 4B comparisons, SEA outperforms Cached-Web RL on all four benchmarks and Real-Web RL on three, using approximately 4.8% of the latter's measured training API expenditure. Diagnostics further show improved evidence recall alongside reduced over-assistance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deep Research, Agentic RL, Search Agent

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_yomHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that cheap simulated environments for training search agents with RL can be factually correct and still teach the wrong search behavior. They can withhold evidence that a good query would find on the real web, or hand decisive evidence to a vague query. The authors propose SEA, a trained model that acts as the search environment. For each tool call, it either builds a response by selecting and reranking pages from a local cache and a memory of past real-web results, or it calls the real web when it predicts a decision-relevant gap it cannot fix locally. The results are strong. SEA-RL-35B improves over its base model by about 23 points on average across six benchmarks (e.g., 31.0→74.0 on BrowseComp). It outperforms all of the open research agents reported in Table 1, beats the much larger Qwen3.5-122B on every benchmark, and is competitive with DeepSeek-V4-Flash. In the controlled 4B comparison, SEA beats Cached-Web RL on all four benchmarks and Real-Web RL on three, at about 5% of the latter's API cost and 17% less training time. The evaluation setup is sound: SEA and Oracle Memory are removed at test time, so the policy is judged purely on the real web. The environment is also evaluated directly through continuation success from fixed decision points (60.7% Pure Local vs. 85.3% SEA vs. 90.2% Real-Web), which tests the environment more meaningfully than overall recall.

The main concerns are with how parts of the evaluation are supported. First, the same LLM, DeepSeek-V4-Flash, creates all 5,000 SEA training labels and is also the primary annotator for the Figure 2 feedback-quality metrics. SEA is trained to match that model's judgments, so these metrics may partly reward agreement with the teacher. Annotator disagreement rates and results without DeepSeek are not reported. Second, Table 1 reports single runs with no mean, standard deviation, or pass@k. Third, key information is missing: the Real-Web branch (90.2%), which anchors the "83.4% of the gap closed" claim, appears only in the text and not in Figure 2, and the annotation prompt for response selection is not provided. Fourth, the evaluation setup is described for the authors' own models, but not for the baselines, so it is unclear whether those numbers are comparable or how much the agent harness itself contributes. Finally, some ablations are missing: how often the untrained base model chooses the real web versus local evidence and how training changes this, and whether a different annotator model would change SEA or the final policy. Table 1 would also be easier to read with best results highlighted and an average column.

Due to the above reasons, I am giving this paper a weak accept, and will consider changing the rating to a strong accept if the authors can address above concerns.

Critical strengths
  • S1 Results are strong - With just 2000 and 3000 training samples for picking and routing & ~2500 QA pairs, they were able to improve the results by more than ~10-40% points on an average (across benchmarks) over base model and very comparable results with much larger models like DeepSeek-v4-Flash, Qwen-3.5-122B models and all the open research agent models, falling behind only Kimi-2.6 and GPT5.4 (both significantly larger, so to be expected).
  • S2 Eval settings are highly competitive, honest and relevant in current AI landscape For Example
    • Evaluated both AI search policy as well as the SEA environment system
    • For both evals, Picked strong benchmarks to showcase their results.
    • For inference, removed SEA AI system and Oracle Memory – so the search AI policy’s true merit can be measured
    • Significantly closed the Sim-to-Real gap (more in S3) with a SEA-driven environment while achieving 95% lower API cost and 17% lower time cost than a real-web RL environment.
  • S3 The RL environment itself is AI-driven The RL environment is driven be a model to dynamically determine whether to use local search from local docs or real web API. Besides that, the AI model driving the search environment is now trained on objectives to rank as closely to real search APIs as possible. They were successfully able to prove the gap between sim2real is closed as per ablation study in Figure 2. Instead of using dataset-level success or recall, they measured continuation success i.e. given a decision point, continue with a fixed policy. SEA-optimized closed the gap with Pure Local by a lot: continuation success of PureLocal=60.7%, SEA-optimized =85.3% and Real-Web=90.2%.
Critical weaknesses
  • W1 (Readability) Not a weakness per say, but a request for better readability of results. In Table 1, please highlight the best performing model. Also a column showing an average performance across all benchmarks is missing.

  • W2 (Strengthening Experimental Results and Evaluation Settings)

    • Same model builds SEA's labels and grades SEA. DeepSeek-V4-Flash writes all 5,000 SEA training labels: the 3,000 routing examples and the 2,000 response-selection examples (Appendix G). It is also the main annotator for the feedback-quality numbers in Figure 2 (left): lead/fact recall, extra evidence, and answer jump (Appendix B.1). Since SEA is trained to copy DeepSeek's judgments, a DeepSeek-led evaluation may favor SEA for agreeing with its teacher, not for being closer to real search. The second annotator (Qwen3.5-122B) and the Gemini-3.1-Pro tie-breaker reduce this risk but don't remove it, and the paper doesn't report how often the annotators disagreed or how the results change without DeepSeek.
    • On top of that, DeepSeek-V4-Flash is also a baseline in Table 1 (73.2 on BrowseComp), so a model being compared against also shapes the training signal. The authors should report annotator agreement for Figure 2, show the Figure 2 metrics graded by Qwen and Gemini alone, and explain why DeepSeek-V4-Flash was chosen as the annotator.
    • Authors should run experiments in Table 1 atleast 3 or more times and report mean / standard deviation. Alternate can be to run it N times and report Pass@3 results or Worst@3 results.
  • W3 (Missing Agent Harness settings for Table 1 and suggestion to rerun experiments under different agent harnesses)

    • The paper doesn't say which agentic harness (custom or open-source) was used for experiment results in Table 1. This should be specified. The abstract calls them "reported" results, which suggests the numbers are taken from other papers, each with its own agent harness, tools, turn limits, and prompts. If so, the comparison mixes the effect of the model with the effect of the harness. The authors should state which baseline numbers they ran themselves and which are taken from other papers. Ideally, they would also re-run key baselines under the same harness as SEA-RL.
    • Furthermore, An interesting extra experiment would be to evaluate the reported models under several agent harnesses, such as Claude Code, Codex, and open-source options like OpenHands. This would show how much a strong harness improves search performance on its own, and whether SEA-RL's gains still hold when every model runs under the same harness.
  • W4 (Major missing results from plots and information)

    • From Section 3.4, Continuation Outcomes, it is clear that real-web got a score of 90.2% but the entire 4th branch is missing from Figure 2 (left). Request to authors to please update the figure with results from Real-Web ablation.
    • The judge AI's (the 3rd model) instructions for picking, routing and grading are not provided in the paper.
Questions for authors

Q1. What is the general routing tendency of untrained, base model between picking google API or local docs and how does training the model change it’s behavior. When AI policy is trained with SEA + w/o routing supervision, how well does the model do on routing specifically? How well does the base model do on routing?

Q2. Does the choice of Judge AI (which was critical in creating the picking and routing dataset for SEA) affects the performance of SEA environment model and SEA-RL? This could serve as a very interesting ablation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yomH (1)

W1 (Readability): Not a weakness per say, but a request for better readability of results. In Table 1, please highlight the best performing model. Also a column showing an average performance across all benchmarks is missing.

  • We revised Table 1 to bold the highest score and underline the second-highest score in each results column. We also added an Avg. column reporting the unweighted mean across all six benchmarks.

W2 (Strengthening Experimental Results and Evaluation Settings):

The second annotator (Qwen3.5-122B) and the Gemini-3.1-Pro tie-breaker reduce this risk but don't remove it, and the paper doesn't report how often the annotators disagreed or how the results change without DeepSeek. The authors should report annotator agreement for Figure 2, show the Figure 2 metrics graded by Qwen and Gemini alone, and explain why DeepSeek-V4-Flash was chosen as the annotator.

Q2. Does the choice of Judge AI (which was critical in creating the picking and routing dataset for SEA) affects the performance of SEA environment model and SEA-RL?

  • Thank you for highlighting this potential evaluator (Judge AI) bias. We agree that using Gemini only to adjudicate disagreements does not establish evaluation independence. We therefore re-evaluated the same frozen histories, actions, and feedback using Qwen3.5-122B-A10B and Gemini-3.1-Pro independently. Each independently extracted reference evidence from the raw real-tool responses and scored anonymized environment responses, without access to DeepSeek’s annotations.
Independent judgeEnvironmentLead / Fact Recall ↑Extra Evidence / Answer Jump ↓
Qwen3.5-122B-A10BPure Local81.3 / 89.914.3 / 10.0
SEA before optimization83.6 / 92.710.0 / 5.7
Trained SEA89.4 / 94.96.0 / 2.3
Gemini-3.1-ProPure Local82.7 / 91.213.0 / 9.0
SEA before optimization84.8 / 93.89.0 / 5.0
Trained SEA91.1 / 96.05.3 / 2.0

Both independent judges rank trained SEA best on all four metrics, preserving Figure 2’s conclusions without using DeepSeek’s judgments.

  • We also computed pre-adjudication agreement across all three judge pairs. For retention judgments on jointly extracted evidence, agreement was 92.8–94.2% for leads. Agreement was 96.1–97.1% for Extra Evidence (26–35 disagreements out of 900) and 97.8–98.2% for Answer Jump (16–20 disagreements). We will include the full pairwise breakdown together with the confusion matrices and annotation prompts in Appendix B.1.
  • DeepSeek was selected for stable structured outputs, reliable API availability and throughput, and cost efficiency. Its role as an offline annotator is separate from its inclusion as a benchmark baseline.
AuthorsOpenReview ↗

Response to Reviewer yomH (2)

W3 (Missing Agent Harness settings for Table 1 and suggestion to rerun experiments under different agent harnesses):

The paper doesn't say which agentic harness (custom or open-source) was used for experiment results in Table 1. This should be specified. The abstract calls them "reported" results, which suggests the numbers are taken from other papers, each with its own agent harness, tools, turn limits, and prompts. If so, the comparison mixes the effect of the model with the effect of the harness. The authors should state which baseline numbers they ran themselves and which are taken from other papers. Ideally, they would also re-run key baselines under the same harness as SEA-RL.

Furthermore, An interesting extra experiment would be to evaluate the reported models under several agent harnesses, such as Claude Code, Codex, and open-source options like OpenHands. This would show how much a strong harness improves search performance on its own, and whether SEA-RL's gains still hold when every model runs under the same harness.

W2: Authors should run experiments in Table 1 at least 3 or more times and report mean / standard deviation. Alternate can be to run it N times and report Pass@3 results or Worst@3 results.

  • Thank you for highlighting the need to clarify the evaluation setup and result sources. Our own evaluations use the MiroThinker-1.7 deep research agent harness. We have revised Table 1 to distinguish these results from scores taken from the respective models’ official technical reports, which are marked with an asterisk (*).
  • We agree that differences in harnesses, tools, prompts, and interaction budgets can affect performance. To provide a controlled comparison and assess evaluation variability, we have added experiments for the five models below under the same MiroThinker-1.7 harness. Each model is evaluated in three independent runs, and we report mean ± standard deviation.
ModelBrowseCompGAIADSQAxBenchWideSearch-ENAR-Deep
LiteResearcher-4B29.4 ± 1.370.6 ± 1.518.3 ± 1.012.0 ± 1.222.2 ± 1.44.5 ± 0.5
Qwen3.5-122B-A10B38.4 ± 1.182.1 ± 1.266.5 ± 0.952.4 ± 1.367.5 ± 1.19.4 ± 0.7
DeepSeek-V4-Flash72.0 ± 0.983.0 ± 1.089.4 ± 0.767.5 ± 1.175.8 ± 0.922.4 ± 0.8
SEA-RL-4B52.6 ± 0.878.4 ± 1.167.4 ± 0.651.0 ± 1.068.8 ± 0.814.3 ± 0.8
SEA-RL-35B74.0 ± 0.785.4 ± 0.880.6 ± 0.660.0 ± 0.973.8 ± 0.739.0 ± 0.8

W4 (Major missing results from plots and information):

  • Thank you for pointing out the omission. We added the Real-Web continuation success rate of 90.2%, previously reported in §3.4, to Figure 2 (left). The Lead Recall, Fact Recall, Extra Evidence, and Answer Jump metrics all use the Real-Web response to the same action as the reference. We add the prompts in the paper.

Q1: What is the general routing tendency of untrained, base model between picking google API or local docs and how does training the model change it’s behavior. When AI policy is trained with SEA + w/o routing supervision, how well does the model do on routing specifically? How well does the base model do on routing?

  • Thank you for this question. We directly evaluate SEA’s routing behavior on held-out examples excluded from SEA training and model selection. Prompt-based SEA uses the base model without SEA-specific fine-tuning.
SEA variantREAL selection rate (%)REAL recall (%) ↑
Prompt-based SEA4.367.8
Trained SEA5.189.5

SEA Training helps SEA identify necessary real-tool calls while preserving predominantly local execution.

Reviewer_yomHOpenReview ↗

Post-rebuttal comments

I thank the authors for their detailed responses and additional experiments. The clarified harness setup, the independent re-grading of Figure 2, the added routing-accuracy results, and the Real-Web continuation result address several of my concerns. Using a learned agent to manage the training environment is a neat and practical idea for reducing the cost of search RL. However, it builds on established principles the paper itself cites, namely selective real-environment use (as in MBPO) and decision-aware model learning, so I view its novelty and impact as solid but not high enough to warrant a higher score. I am keeping my score at weak accept for the following reasons:

  1. Routing baselines: Building on the concern raised by Reviewer uGeG and Reviewer 9kF1 about missing routing baselines, I appreciate the authors' new comparison with random and retrieval-confidence routers at matched real-tool cost. However, the comparison is not like-for-like: each router is only used to serve a fixed SFT policy, while SEA's claimed benefit comes from RL training inside its environment, and no policy was RL-trained with the alternative routers and then evaluated on the real-web benchmarks as in Table 2. Without that experiment, I cannot attribute the real-web gains to the learned routing rather than to cheap evidence reuse plus occasional real calls.

  2. Controlled baselines and failure analysis: I appreciate the controlled re-evaluation under the MiroThinker-1.7 harness with three runs per model. However, no analysis shows whether the harness itself (e.g., tool-call formatting, turn limits, or context-length cutoffs) contributes to failures for SEA-RL or the baselines. More broadly, failure analysis remains limited: Appendix J offers two hand-picked case studies and a brief qualitative note on the 25 remaining BrowseComp errors, but no systematic breakdown of failure types or their frequency across benchmarks.

  3. Annotator dependence: The independent re-grading of Figure 2 by Qwen3.5-122B and Gemini-3.1-Pro shows that the evaluation does not depend on DeepSeek's vote. However, my Q2 asked whether the choice of annotator for SEA's training data affects SEA and SEA-RL, and this ablation was not run. All 5,000 routing and response-selection labels still come from DeepSeek-V4-Flash, so it remains unclear whether SEA's behavior, and the downstream policy gains, depend on which model served as the teacher.

Overall, the paper addresses a relevant problem with strong empirical results and substantial cost savings, but the remaining gaps prevent me from raising my score.

Thank you for the follow-up.

Q4: However, no analysis shows whether the harness itself (e.g., tool-call formatting, turn limits, or context-length cutoffs) contributes to failures for SEA-RL or the baselines.

We evaluated SEA-RL-4B on 100 randomly sampled questions from each of BrowseComp and AR-Deep, varying the tool-call format, turn limit, and context limit.

Tool-call formatMax. turnsContext limitBrowseCompAR-Deep
Native XML100256K49/10012/100
OpenAI100256K53/10014/100
Native XML25256K28/1005/100
Native XML50256K42/10011/100
Native XML75256K49/10011/100
Native XML10064K30/1003/100
Native XML100128K47/10010/100
  • Although the model was trained with native XML tool calls, the OpenAI format yields slightly higher scores on both samples. However, Native XML is more stable and requires far fewer tool-call regenerations (13 vs. 1,323; each tool call is regenerated at most 3 times when invaild). On AR-Deep, the OpenAI-style interface issues fewer search queries and visits more pages, with a lower median peak prompt length (180K vs. 234K tokens) and fewer runs terminated by the context limit (17/100 vs. 47/100). Shorter tool outputs may help explain its lower context usage.
  • Increasing the budget from 25 to 75 turns improves performance on both benchmarks. Increasing the limit from 64K to 256K improves scores from 30/100 to 49/100 on BrowseComp and from 3/100 to 12/100 on AR-Deep. Many unsuccessful runs exhaust their turn budget.
  • These results show that harness settings affect SEA-RL-4B's measured performance. We will extend this analysis to the baselines.

Q3 and Q5: However, my Q2 asked whether the choice of annotator for SEA's training data affects SEA and SEA-RL, and this ablation was not run. the comparison is not like-for-like: each router is only used to serve a fixed SFT policy, while SEA's claimed benefit comes from RL training inside its environment, and no policy was RL-trained with the alternative routers and then evaluated on the real-web benchmarks as in Table 2.

  • Thanks for the question. We added end-to-end RL experiments in Prompt SEA and Trained SEA and evaluated the resulting policies on the real-web benchmarks.
Training settingBCGAIADSQAAR-Deep
Cached-Web RL47.369.663.78.1
Real-Web RL51.780.666.210.1
Prompt SEA RL50.274.465.211.0
Trained SEA RL52.678.467.414.3
  • Trained SEA outperforms Prompt SEA on all four benchmarks. On BrowseComp, it improves the score from 50.2 to 52.6 while reducing the average number of turns from 30.7 to 29.0. These results support the benefit of the trained environment for policy RL.

Given the limited rebuttal period and the substantial time required for RL training, we will extend our experiments in follow-up work. This will include end-to-end RL training in additional search environments (e.g., those using random and retrieval-confidence routers), evaluation and analysis on more benchmarks, and an investigation of how the choice of model used to annotate SEA’s training data affects search policy RL.

Thanks again for your constructive feedback! Your constructive comments have been invaluable in helping us strengthen the empirical rigor and clarity of our work.

Reviewer_9kF1OpenReview ↗
Rating
4(+1)
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Search Environment Agent (SEA), a memory-augmented training environment for efficient deep-search reinforcement learning. SEA calibrates local tool feedback by selecting evidence and invokes real-web tools when consequential discrepancies cannot be repaired locally. Policies trained with SEA achieve competitive real-web performance at substantially lower RL-stage tool API cost, without relying on SEA or its memory during evaluation.

Critical strengths
  1. The paper identifies an important problem: factually correct simulated feedback can still teach ineffective search behavior.
  2. The proposed SEA combines grounded local feedback with selective real-web calls in a well-motivated framework.
  3. The paired-feedback supervision and component ablations provide useful support for the proposed design.
  4. Experiments show competitive real-web performance with substantially lower training-stage tool API costs.
Critical weaknesses
  1. The benefit of the proposed routing criterion is not fully isolated. Although the component ablations support the overall design, the paper lacks a hybrid baseline that uses the same memory resources and matches SEA’s real-tool API expenditure through random or retrieval-confidence-based routing. Such a comparison would clarify how much improvement specifically comes from learning decision relevance and local repairability, beyond evidence reuse and occasional real-web interaction.
  2. Some experimental reporting requires clarification. Appendix E describes 100 RL updates, whereas the training dynamics cover only 40 updates, and the checkpoint-selection procedure for Table 2 is unclear. Additionally, the reported three independent runs should distinguish independent RL training runs from repeated evaluations of a fixed checkpoint. These details matter when interpreting the relatively small gains over Real-Web RL on several benchmarks.
Questions for authors

see Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9kF1

W1: The benefit of the proposed routing criterion is not fully isolated. Although the component ablations support the overall design, the paper lacks a hybrid baseline that uses the same memory resources and matches SEA’s real-tool API expenditure through random or retrieval-confidence-based routing. Such a comparison would clarify how much improvement specifically comes from learning decision relevance and local repairability, beyond evidence reuse and occasional real-web interaction.

  • Thank you for identifying this missing control. Our added random and retrieval-confidence baselines share SEA’s Oracle Memory, local index, and response selector, with closely matched real-tool expenditure and a fixed SFT policy. SEA’s advantage under these controls supports a contribution from learned routing beyond evidence reuse. The separate prompt-based comparison further supports the benefit of SEA supervision. These fixed-policy diagnostics complement the policy-RL ablations in §3.5.
Environmentcost ($) ↓Lead / Fact Recall ↑Extra Evidence / Answer Jump ↓Continuation success ↑
Random routing0.6584.5 / 92.06.4 / 3.070.8
Prompt-based SEA0.6084.2 / 93.59.6 / 5.279.4
Trained SEA0.6490.6 / 95.85.4 / 2.185.3
Real-Web reference13.10100.0 / 100.00.0 / 0.090.2

Budget matching applies to random, retrieval-confidence, and trained SEA routing. Feedback metrics and continuations follow §3.4 and Appendix B, with each continuation retaining its assigned environment.

Task success measures end-to-end environment inference on 100 questions held out from our constructed QA dataset in Sec 3.4.

W2: Some experimental reporting requires clarification. Appendix E describes 100 RL updates, whereas the training dynamics cover only 40 updates, and the checkpoint-selection procedure for Table 2 is unclear. Additionally, the reported three independent runs should distinguish independent RL training runs from repeated evaluations of a fixed checkpoint. These details matter when interpreting the relatively small gains over Real-Web RL on several benchmarks.

  • Thank you for highlighting these reporting ambiguities. All RL runs were trained for 100 updates, with the same learning-rate and warm-up schedules within each policy scale. We save and evaluate checkpoints every five updates and select the checkpoint with the best evaluation performance as the final policy reported in Table 2. We apply this selection procedure consistently across the training-environment comparisons. The original training-dynamics plot displayed the first 40 updates. We have now extended it to cover the full 100-update training horizon. (Table 3) Periodic evaluation followed by performance-based checkpoint selection is also used in recent RL reasoning studies. [1], [2]
  • The “three independent runs” in Table 2 refer to three repeated evaluations of a fixed selected checkpoint. The reported standard deviations therefore quantify evaluation-time variability conditional on that checkpoint.
  • We have additionally evaluated the following five models under the same MiroThinker-1.7 harness. Each model uses a fixed checkpoint or model version across three repeated evaluation runs, and we report the mean and standard deviation below.
ModelBrowseCompGAIADSQAxBenchWideSearch-ENAR-Deep
LiteResearcher-4B29.4 ± 1.370.6 ± 1.518.3 ± 1.012.0 ± 1.222.2 ± 1.44.5 ± 0.5
Qwen3.5-122B-A10B38.4 ± 1.182.1 ± 1.266.5 ± 0.952.4 ± 1.367.5 ± 1.19.4 ± 0.7
DeepSeek-V4-Flash72.0 ± 0.983.0 ± 1.089.4 ± 0.767.5 ± 1.175.8 ± 0.922.4 ± 0.8
SEA-RL-4B52.6 ± 0.878.4 ± 1.167.4 ± 0.651.0 ± 1.068.8 ± 0.814.3 ± 0.8
SEA-RL-35B74.0 ± 0.785.4 ± 0.880.6 ± 0.660.0 ± 0.973.8 ± 0.739.0 ± 0.8
  • [1] Dong et al., Generalized Parallel Scaling with Interdependent Generations.
  • [2] Wang et al., SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models.
Reviewer_uGeGOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper tries to at first establish the challenges with training an LLM for deep search capabilities using cheaper alternatives such as locally saved environment. Specifically, it highlights reward-transfer decoupling that shows training not generalising to real web performance. It points to the problem of under support and over-assistance that occur while training the policy that might hinder the development of correct and promote incorrect action taking abilities. To this end, they propose SEA agent that is trained to identify the need to escalate to real-web for search while prioritising local environment as much as possible. SEA is trained to construct action-consistent feedback from cached webpages and Oracle Memory for local evidence grounding wherever possible before invoking real tools. The authors present a simple way to create the routing supervision labels for SEA's training. Further, the paper shows the policy trained via SEA's assistance, shows strong uplift on real web benchmarks. Validation is done on 6 benchmarks with 4B and 35B policies, all evaluated on the real web without SEA.

Critical strengths
  • Substantial performance uplight across multiple benchmarks and across 2 model sizes. The results are very competitive w.r.t other open-weights. Also SEA, Oracle Memory and the cache are removed during inference, so the gains are purely from the policy.. Paper details out the training strategies both SFT and RL are mentioned in detail which makes it a valuable recipe contribution in this research area.

  • The work is well ablated and studied. The authors identify key limitations with local environment training and develop an intuitive approach to fix it. Fig 2 shows SEA reduced Extra Evidence and Answer Jump by 60.6% and 77.7%, respectively. SEA improves it to 85.3%. Also all three training settings start from the same SFT checkpoint and only the environment changes, so the comparison in Table 2 is fair. SEA matches or beats Real-Web RL on 3 of 4 benchmarks at both sizes with ~5% of the API cost. So the dependency on Real-Web is minimised as possible.

  • The authors mention that they plan to release the training questions, SFT trajectories and redistributable cached-web resources. This could be an important contribution.

Critical weaknesses
  • Given training cost efficiency is central to the motivation, just comparing API pricing is not enough. Specifically, cost of data prep and training (in some form) SEA is not included. Costs around SEA SFT, continuations to get d labels, DeepSeek-V4-Flash annotations for 5K examples, etc should be disclosed and used in comparisons.

  • Error Analysis on where the tuned policy is failing is missing and it is unclear how the search and browsing behaviour evolves during RL. Table 2 reports lower token usage for SEA RL but does not establish whether this comes from more effective evidence gathering, shorter reasoning, or reduced exploration.This is different than Fig 2 as it requires comparing tool usage and failure patterns.

Questions for authors
  • The 92% in G.3 only shows human agreement with the LLM annotator. However, local/real split is not given, so 92% could be close to just always choosing local, and only 100 of 5K examples are audited. Or is it 100 samples for each of the routing labels?

  • Are there any direct evaluation of SEA's ability for routing and response selection ? While SEA's primary aim is to assist the policy and that's how it is evaluated, it would be great to have some direct measurement.

  • Any comparison with a random or simple router at the same real-web budget? How is it established that the gains come from d(1−c) and not just from having some real-web calls?

  • When a mismatch is labeled fixable (c = 1), is the fixed response tested by continuing the policy from it? How many continuations are run per branch for d?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uGeG (1)

W1: Given training cost efficiency is central to the motivation, just comparing API pricing is not enough. Specifically, cost of data prep and training (in some form) SEA is not included. Costs around SEA SFT, continuations to get d labels, DeepSeek-V4-Flash annotations for 5K examples, etc should be disclosed and used in comparisons.

  • Thank you for highlighting the need to make preparation costs more explicit. Appendix H already reports approximately USD 3,400 in cumulative search-and-browsing expenditure during question construction, annotation, SFT trajectory collection, and RL development, together with an amortization analysis.
  • Our updated accounting identifies USD 200 for the web-tool calls used in continuation-based routing supervision and less than USD 100 for the 5,000-example annotation and associated judging, including DeepSeek and Gemini. The overall preparation API expenditure remains below USD 3,700. With the per-run costs of USD 32.1 for SEA versus USD 675.8 for Real-Web RL, cumulative API savings begin at the sixth additional comparable 4B training run.
  • SEA SFT uses eight H100 GPUs for three epochs and makes no external web-tool calls. Table 2’s end-to-end step times include SEA inference and environment processing. GPU computation is not monetized in this comparison; the 95.3% saving refers specifically to web-tool API expenditure during policy RL, rather than total training expenditure. We will make this breakdown and accounting scope explicit alongside Table 2.

W2: Error Analysis on where the tuned policy is failing is missing and it is unclear how the search and browsing behaviour evolves during RL. Table 2 reports lower token usage for SEA RL but does not establish whether this comes from more effective evidence gathering, shorter reasoning, or reduced exploration. This is different than Fig 2 as it requires comparing tool usage and failure patterns.

  • Thank you for highlighting this gap. We add two paired case studies comparing the 35B SFT and SEA-RL policies with real-web tools (Appendix J, Table 5 and Figure 8), reporting tool use, generated tokens, and non-tool narration separately.
  • In the museum case, SEA-RL verifies the identifying evidence with fewer repeated queries, reducing generated tokens from 14,435 to 2,306. In the fort case, it resolves the distinguishing historical clue using more assistant turns (17 to 22) and the same seven URL requests, while generated tokens decrease from 9,728 to 7,023. These cases illustrate reduced redundant search and more concise narration; lower output does not necessarily imply fewer evidence checks.
  • We also discuss remaining distractor-anchoring and evidence-resolution failures. This retrospective endpoint comparison complements the environment-feedback analysis in §3.4, but does not establish how behavior evolves across RL updates.

Q1: The 92% in G.3 only shows human agreement with the LLM annotator. However, local/real split is not given, so 92% could be close to just always choosing local, and only 100 of 5K examples are audited. Or is it 100 samples for each of the routing labels?

  • Thank you for highlighting this limitation. We randomly sampled 50 routing and 50 response-selection examples to audit annotation quality, totaling 100 examples rather than 100 per routing label. The reported 92% is routing-label agreement with the majority vote of three label-blinded human experts. We agree that aggregate agreement alone is insufficient under class imbalance.
  • Including additional expert-review records, we have audited 30 automatically REAL-labeled examples, of which 27 were confirmed as REAL by human review (90% REAL precision). We report this class-specific result separately, preserving the original random-sample statistics.
Audit setHuman reference labelAutomatic: LOCALAutomatic: REALTotal
Original random routing auditLOCAL43245
REAL235
Total45550
Cumulative REAL-labeled auditLOCAL—33
REAL—2727
Total—3030
AuthorsOpenReview ↗

Response to Reviewer uGeG (2)

Q2: Are there any direct evaluation of SEA's ability for routing and response selection? While SEA's primary aim is to assist the policy and that's how it is evaluated, it would be great to have some direct measurement.

  • Thank you for this suggestion. Section 3.4 evaluates SEA’s feedback quality under fixed policy histories and actions, including evidence recall, extra evidence, and answer leakage. We agree that these environment-level metrics do not separately measure routing and response selection.
  • We therefore add direct evaluations on held-out examples excluded from SEA training and model selection. For LOCAL/REAL routing, we report recall of actions requiring real execution. For response selection, we bypass routing and evaluate the selected candidate set against reference selections using identical, fixed candidate pools.
MethodREAL Recall ↑Selection F1 ↑
Prompt-based SEA67.853.6
Trained SEA89.578.9

Trained SEA improves both routing and response selection, with particularly higher recall of actions requiring real tools. These results directly assess SEA’s two components and complement the feedback-quality evaluation in Section 3.4.

Q3: Any comparison with a random or simple router at the same real-web budget? How is it established that the gains come from d(1−c) and not just from having some real-web calls?

  • Thank you for suggesting this comparison. We added random and retrieval-confidence routers with nearly identical real-tool expenditure, holding Oracle Memory, the local index, the response selector, and the 4B SFT policy fixed. SEA achieves better feedback fidelity and higher continuation/task success under these controls. This supports the value of allocating real-tool calls through routing learned with z⋆=d(1−c)z^\star=d(1-c), beyond simply including occasional real-web calls.
Environmentcost ($) ↓Lead / Fact Recall ↑Extra Evidence / Answer Jump ↓Continuation success ↑E2E Task success ↑
Random routing0.6584.5 / 92.06.4 / 3.070.832.0
Retrieval-conf routing0.6586.4 / 93.26.1 / 2.775.635.0
Prompt-based SEA0.6084.2 / 93.59.6 / 5.279.437.0
Trained SEA0.6490.6 / 95.85.4 / 2.185.341.0
Real-Web reference13.10100.0 / 100.00.0 / 0.090.243.0

Budget matching applies to random, retrieval-confidence, and trained SEA routing. Feedback metrics and continuations follow §3.4 and Appendix B, with each continuation retaining its assigned environment.

Task success measures end-to-end environment inference on 100 questions held out from our constructed QA dataset in Sec 3.4.

Q4: When a mismatch is labeled fixable (c = 1), is the fixed response tested by continuing the policy from it? How many continuations are run per branch for d?

  • For annotating dd, we ran three policy continuations per branch. To directly evaluate repaired responses, we added an ablation comparing original and repaired local feedback using the same fixed SFT policy. Repair increases continuation success from 82.6% to 85.3%, while reducing Extra Evidence from 7.5% to 5.4% and Answer Jump from 5.0% to 2.1%.
  • These results provide behavioral evidence for the benefit of local repair: it reduces over-assistance while improving downstream task completion, supporting more faithful calibration of local feedback.
Current Response ConditionContinuation Success (%) ↑Extra Evidence / Answer Jump ↓
SEA w.o local repair82.67.5 / 5.0
SEA w. local repair85.35.4 / 2.1
Reviewer_uGeGOpenReview ↗

Official Reply by Reviewer uGeG

Thank you for your response. The clarifications have addressed my main concerns, and I will raise the score accordingly.

Thank you very much for taking the time to read our responses and for raising your score. Your constructive comments helped us improve both the experiments and the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.