- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes VeriSearch, a prompt-driven search agent with four modules: a Planner that dynamically decomposes the question into atomic sub-questions, a Searcher that rewrites each sub-question into L = 3 keyword queries, an AnswerabilityPruner that scores every search result on a 0–10 scale from its snippet, crawls only results with score ≥ θ = 4, and triggers up to K = 3 reflective query-rewriting retries when no result passes, and a ConflictResolver that detects cross-source factual conflicts among crawled pages and resolves them with a fixed arbitration score Φ = 0.6·Auth + 0.4·Rec, where Auth is an LLM-estimated authority of the source URL and Rec is a linear recency score with a floor of 0.5. The lower-scoring evidence is discarded; ties fall back to search-engine rank. With a training-free Qwen3.5-9B backbone, VeriSearch reports 67.3 / 45.1 / 89.1 (GPT-4o-judged accuracy) on xbench-DeepSearch, SEAL-0, and Bamboogle. These scores exceed reimplemented same-backbone Search-o1 and WebThinker-Base, as well as numbers reported elsewhere for Claude-4-Sonnet and OpenAI-o3. Ablations (Table 2) and a 48-question conflict-triggered subset of SEAL-0 (Table 3) are used to argue that both modules contribute.
Critical strengths
- A practically relevant problem. Open-web agents ingest pages that are uninformative or mutually contradictory. Making evidence control an explicit stage before reasoning (Sec. 1, Fig. 1) is a sensible design goal, and SEAL-0 is an appropriate stress test for it.
- The method is simple, training-free, and clearly described. The pipeline (Fig. 2, Eqs. 1–5) is easy to follow and could be implemented, and the paper states its key hyperparameters (θ, K, α, L, T_max, top-6 Serper results).
- Strong absolute numbers for a 9B backbone, with some care in reporting. VeriSearch reports means and standard deviations over three runs, adds EM/F1 alongside the LLM judge (Table 5), tests three other backbones (Table 4), and documents honestly why MindSearch could not be faithfully ported to Qwen3.5-9B (Appendix E) instead of reporting a broken baseline as a fair comparison.
Critical weaknesses
-
The headline gains are mostly not attributable to the two proposed modules. The paper's contributions are AnswerabilityPruner and ConflictResolver, but most of the improvement over the same-backbone baselines comes from the rest of the framework. On xbench-DeepSearch, VeriSearch without AnswerabilityPruner still scores 62.5 versus 52.0 for Search-o1 (Table 2 vs. Table 1); without ConflictResolver it scores 66.3. On SEAL-0, the corresponding numbers are 38.7 and 39.6 versus 30.6. The paper never reports the key control, Planner + Searcher with neither module. So the claim that "the improvements stem from the framework design" (Sec. 3.2) cannot be credited to the contributions, as opposed to dynamic decomposition, multi-query search (L = 3, up to T_max = 10 iterations), or extra search and LLM calls. Search budget is not matched or reported for any system: no counts of search calls, crawled pages, LLM calls, tokens, or latency are given, even though the Pruner adds per-snippet scoring and up to three retries per sub-question.
-
The comparative and statistical evidence is weak. (a) The comparisons with Claude-4-Sonnet, OpenAI-o3, GLM-4.5-Air, and Kimi-K2 (abstract, Sec. 3.2) use numbers copied from prior work. Those systems ran with different search tools, at different times, and possibly with different judges, so "exceeds Claude-4-Sonnet by 5.0 points" is not a controlled claim and should not appear in the abstract. (b) The same-backbone baselines are single runs. Their search API, top-k, crawler, context length, and prompts are not stated to match VeriSearch. (c) The benchmarks are small (100 / 111 / 125 questions), so one xbench question equals one point. Several reported effects lie within the stated run-to-run noise: Bamboogle +1.1 over Search-o1 with std 1.9, and ConflictResolver's −1.0 on xbench with std 1.5. Yet these are described as gains or as "consistent" contributions. No significance tests are reported.
-
The mechanism-level claims are not validated, and the heuristics are not justified. The paper argues that the modules work through better answerability estimation and better conflict adjudication, but it never measures either. There is no precision or recall for answerability scores (whether the crawled page actually contains the answer). There is no accuracy for conflict detection, and, most importantly, no accuracy for arbitration (how often the retained claim is the correct one). The arbitration rule assumes that an LLM-guessed URL authority and recency indicate correctness: Φ = 0.6·Auth + 0.4·Rec with a 0.01/day slope and a 0.5 floor, and the losing evidence is discarded outright. Neither assumption is analyzed. Recency is misleading for static facts, and SEAL-0 is built around evolving and conflicting results, which may favor this heuristic. Further gaps: (i) the conflict-triggered subset (Table 3) is selected by the module's own detector; (ii) the "w/o AnswerabilityPruner" ablation removes scoring, gating, and retries together, so the effect of filtering cannot be separated from the effect of extra retries; (iii) there is no sensitivity analysis for θ, K, or α, and no stated development split for choosing them. This leaves open whether they were tuned on the test benchmarks.
-
The novelty is limited relative to existing evidence-filtering and conflict-handling work. Snippet-level LLM filtering before page reading is close to SelectLink in ReST-meets-ReAct, which the paper cites, and to retrieval-evaluator gating with corrective re-search as in CRAG (Yan et al., 2024). Weighing conflicting retrieved sources by credibility has also been studied in RAG, e.g., Astute RAG (Wang et al., 2024) and the source-background approach of Ge et al. (2025), which the paper cites. The paper does not compare with these mechanisms empirically. A threshold on an LLM relevance score plus a hand-weighted authority/recency rule is a reasonable engineering combination, but it is not shown to offer insight or generality beyond these prior designs.
Questions for authors
- Please report Planner + Searcher with both modules removed, under the same Qwen3.5-9B setup, on xbench-DeepSearch and SEAL-0. What fraction of the gain over Search-o1 is due to the two proposed modules?
- For VeriSearch, Search-o1, and WebThinker-Base, what are the average numbers of search calls, crawled pages, LLM calls, total tokens, and wall-clock time per question? Do the same-backbone baselines use the same Serper top-6 results, crawler, cleaning pipeline, context length, and GPT-4o judge prompt?
- On a manually labeled sample, what are the precision and recall of conflict detection, and how often does arbitration retain the correct claim? How often does the recency term overturn the authority term, and is that decision correct?
- How were θ = 4, K = 3, and α = 0.6 selected? Please provide a sensitivity curve for each and state whether any test-set results guided the choice.