- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SEEDS, a preview-guided evidence admission framework for multi-round search agents. After retrieval, the agent reads lightweight previews and selects an arbitrary subset of candidates for full inspection, conditioned on its current reasoning state. The framework is evaluated with both local retrieval and live web search, with an additional trainable variant based on supervised fine-tuning and reinforcement learning.
Critical strengths
-
The distinction between retrieving candidates and deciding which ones to inspect is practically meaningful. The framework is simple and can be integrated into existing search agents without additional training.
-
The experiments cover multiple QA benchmarks and both local and live-web environments. The reduction in full-document consumption is substantial: SEEDS-ReAct reduces average reads from 7.7 to 3.1 relative to ReAct dyn-k while improving average EM from 36.9 to 41.6.
-
The appendix provides useful implementation details, preview ablations, and an explicit discussion of the limitations of answer occurrence as an evidence-quality proxy.
Critical weaknesses
-
The core mechanism is not sufficiently validated. The paper emphasizes evidence selection conditioned on the evolving reasoning state. However, ReAct dyn-k determines the reading depth before observing retrieval results, whereas SEEDS first reads all previews and then selects an arbitrary subset. This comparison changes both the available information and the selection mechanism. It remains unclear whether the improvement comes from preview access, non-prefix selection, or reasoning-state conditioning. Please include a preview-aware prefix baseline and a selector using only the question, current query, and previews without the reasoning history.
-
The main efficiency evaluation is incomplete. The document-token statistics in Table 4 exclude previews, selection reasoning, and other generated outputs. This overhead is potentially substantial because local passages contain approximately 100 words, while each search exposes ten previews of up to 60 tokens each. Although Table 13 includes preview overhead for SEEDS-R1, equivalent accounting is missing for the main QwQ-32B experiments. Please report total inference tokens, including selection and document-refinement calls, together with end-to-end latency, to establish whether fewer full-document reads translate into practical computational savings.
-
The contribution of evidence expansion is unclear. All candidate previews enter the reasoning context and may already contain sufficient information to answer the question. Appendix E.3 also explicitly accommodates correct answers without expanding the relevant document. Therefore, reduced full-document consumption does not by itself establish the benefit of selecting documents for expansion. Please include a preview-only baseline that retains iterative search but disables expansion, and report how often SEEDS answers correctly without opening any documents. This would distinguish the benefit of compact previews from the additional value of selective expansion.
-
The learning experiment does not isolate the source of improvement. SEEDS-R1 introduces supervised warm-start training and an auxiliary selection reward based on gold-answer occurrence or supporting-fact annotations. These provide additional supervision beyond the admission interface itself. It is therefore unclear how much of the improvement over Search-R1 comes from the proposed interaction mechanism versus the training recipe. Please compare SFT-only, SFT followed by EM-only PPO, and PPO with the selection reward under matched budgets, and clarify whether the Search-R1 baseline receives comparable demonstrations and evidence-level supervision.
Questions for authors
See Weakness
4 comments
W3: Contribution of Evidence Expansion.
R3: Following the reviewer's advice, we conduct two more analyses to separate the contribution of lightweight previews from that of selective evidence expansion.
(1) Preview-only Ablation. We introduce a preview-only baseline that retains iterative search but disables full-document expansion, along with an expand-all variant that expands every candidate. Exact Match (EM) and Avg.#Doc per question are reported.
| Variant | NQ | TriviaQA | HotpotQA | Bamboogle | Avg. | #Doc |
|---|---|---|---|---|---|---|
| Preview-only | 36.6 | 66.6 | 33.6 | 55.2 | 48.0 | 0 |
| Preview + expand-all | 39.2 | 68.0 | 33.0 | 52.8 | 48.3 | 12.1 |
| SEEDS | 39.6 | 69.6 | 35.6 | 56.8 | 50.4 | 3.1 |
Selective expansion improves accuracy beyond preview exposure alone. SEEDS outperforms the preview-only baseline by 2.4 average EM, which shows that expansion adds value beyond what previews provide. SEEDS also outperforms expand-all by 2.1 average EM while reading 74% fewer full documents (3.1 vs. 12.1), which shows that the gain comes from selecting which documents to expand and not from reading more evidence.
(2) Correct Answers Without Expansion. We further examine how often SEEDS-ReAct answers correctly without full-document expansion. Across all six datasets, only 4.5% of all questions are answered correctly without opening any document.
Together, these results show that SEEDS benefits from selective full-evidence expansion and does not rely on preview information alone.
W4: Source of Improvement in the Learning Experiment.
To separate the supervised warm-start, outcome-only RL, and the auxiliary selection reward, we compare SFT-only, SFT followed by EM-only RL, and the full SEEDS-R1 with the selection reward, alongside Search-R1.
| Method | NQ | TriviaQA | PopQA | 2Wiki | MuSiQue | Overall |
|---|---|---|---|---|---|---|
| Search-R1 | 43.6 | 61.0 | 40.0 | 28.2 | 14.9 | 40.5 |
| SEEDS SFT-only | 40.6 | 59.3 | 41.0 | 33.7 | 10.8 | 40.8 |
| SEEDS SFT + EM-only RL | 44.5 | 62.5 | 43.5 | 37.9 | 13.0 | 44.0 |
| SEEDS-R1 (SFT + RL with selection reward) | 45.2 | 61.8 | 44.4 | 38.4 | 12.7 | 44.4 |
Without the selection reward, the gain remains. Trained with the same EM-only reward as Search-R1, SEEDS SFT + EM-only RL reaches 44.0 overall EM, 3.5 points above Search-R1. Adding the selection reward contributes only a further 0.4 points.
The warm-start alone gives no advantage. SEEDS SFT-only is on par with Search-R1 (40.8 vs. 40.5), so the demonstrations do not by themselves improve accuracy. The improvement therefore arises when outcome-based RL, the same training signal used by Search-R1, is applied to the admission interface.
Regarding the baseline, we reproduce Search-R1 following its original training protocol, which uses only the final-answer EM reward without evidence-level supervision. We totally agree that adding evidence-level supervision would provide a stronger comparison and will include this enhanced Search-R1 baseline in the revised paper.
My issue has been resolved. Thank you for your reply.
Dear Reviewer prVn,
Thank you so much for your encouraging feedback and support! Your thoughtful suggestions have helped us substantially improve the paper. We are truly grateful for your recognition of our work!
Best regards, The Authors
W1: Isolating the Core Mechanism.
R1: We thank the reviewer for this suggestion. To disentangle the contributions of preview access, non-prefix selection, and reasoning-state conditioning, we add two baselines under the same QwQ-32B local-corpus setting:
Non-prefix selection and reasoning-state conditioning each contribute. (1) Preview access alone is insufficient: with previews but prefix-only selection, ReAct dyn-k-preview underperforms SEEDS by 4.1 average EM. (2) Reasoning-state conditioning adds gains: SEEDS improves average EM by another 2.1 points over the non-history selector while reading fewer documents (3.1 vs. 3.8). SEEDS achieves the highest EM on all four datasets. We will include these controls in the revised paper.
W2: Computational Efficiency and Overhead.
R2: We extend our efficiency analysis to report total inference tokens, including previews, admission decisions, and document-refinement calls, together with end-to-end latency per question. All values are averages per question as the setting in Table4.
Local retrieval: the overhead is small. Passages are short, so previews and admission decisions add some cost, but the overall cost of SEEDS-ReAct stays within the range of the ReAct baselines: its total tokens are below ReAct top-10, and its latency is within 3 s of it. At this cost, SEEDS-ReAct achieves the highest EM among all methods.
Live-web search: clear savings. Each page must be fetched and summarized, so skipping pages saves much more than admission costs. Compared with Search-o1, SEEDS-o1 uses 23.2% fewer total tokens and is 21.4% faster, with higher accuracy by 2.5 points.
These results highlight the benefits of selective evidence admission, particularly when full-evidence acquisition is computationally expensive.