- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LiveSearchVQA, a new benchmark for evaluating multimodal search agents. The authors point out that simply measuring final answer accuracy is not enough. A model might answer correctly without searching, or it might find the right evidence but still give a wrong answer.
To address this, the authors build a benchmark with 6,000 questions across 30 time-based releases. The questions are based on recent news, and each question must pass two checks: models should fail to answer it without searching, but succeed when given the correct evidence. This helps ensure that the questions actually require external information and are answerable.
The paper also proposes a diagnostic framework that compares model performance under three settings: closed-book (CB), with-search (WS), and oracle-evidence (OR). By checking whether the correct evidence appears in the search trajectory, the framework can better distinguish between failure to retrieve evidence and failure to use evidence correctly.
For validation, the authors conduct human evaluations, test the benchmark on models outside the admission panel, and run several controlled experiments. The results show that the benchmark has high question quality and that models with similar overall accuracy can have quite different evidence retrieval and reasoning behaviors.
Critical strengths
-
I think the main strength of this paper is that it goes beyond final answer accuracy and tries to understand why a search agent succeeds or fails. By separating evidence coverage from answer correctness, the proposed framework can distinguish between models that fail to find the right information and models that find it but fail to use it correctly. The experiments also show that models with similar overall accuracy can have very different behaviors. I find this quite meaningful, since it provides more useful information than a standard benchmark leaderboard and could help guide future agent development.
-
The paper puts considerable effort into making sure the questions actually require searching and can be answered with the correct evidence. The joint admission process filters questions using repeated closed-book and oracle-evidence tests. The authors also conduct human evaluation on 600 questions and report a very high validity rate. In addition, the results generalize reasonably well to models outside the admission panel. These experiments make the benchmark quality more convincing and support its usefulness for evaluating multimodal search agents.
Critical weaknesses
- There are already several related works, such as MTA, REDSearcher, OpenSearch-VL, BC-VL, and MMSearch-Plus, that focus on evaluating or improving multimodal search agents. Some of these methods also use verification mechanisms to generate better training data and show improvements through SFT or RL. In comparison, LiveSearchVQA mainly proposes a new evaluation and diagnostic framework, without showing whether it can actually help improve model performance. This makes the novelty and practical impact of the paper less convincing.
- The paper only allows agents to use text search and page opening, while excluding common visual tools such as OCR, image cropping, and reverse image search. However, these tools are important for real-world multimodal search tasks, especially when agents need to identify small visual details or search directly using images. This makes the evaluation setting somewhat limited and raises questions about whether the benchmark can fully reflect multimodal search capabilities.
Questions for authors
- Add more tools and run the evaluation again.
- Table 6 reports a 99.83% pass rate in the human evaluation, with no confirmed errors among 600 sampled questions. This seems surprisingly high given that the dataset is built from dynamic web news using an automatic pipeline. It is unclear whether any questions were manually filtered or corrected before the human evaluation. Also, the strict admission process only keeps 47.9% of candidate questions, which raises concerns about whether the benchmark favors simpler or less diverse questions. B/c the agent is not equipped with enough tools, hard to tell the difficulty.
5 comments
Response to W1
Response to Weakness 1: Value for agent development
We agree that the benchmark should demonstrate value for improving agents. The new four-system controlled replay study tests a concrete development question: does an initial support diagnosis distinguish errors that benefit from adding evidence from errors that benefit from selecting evidence already observed?
The study uses GLM-5V-turbo, qwen3.7-flash, Qwen3-VL-32B and Llama-4-Maverick, including three configurations absent from the original intervention-selection experiment. Original correctness and independently adjudicated support states are fixed before replay. Among the 200 original errors, the gains are:
| Controlled contrast | Initially unsupported errors (G=0; n=100) | Initially supported errors (G=1; n=100) |
|---|---|---|
| Add reference support versus a matched irrelevant addition | +32.00 pp [20.79, 42.99] | +12.00 pp [3.81, 21.11] |
| Select existing observations versus replaying all original observations | +2.00 pp [−4.35, 8.08] | +23.00 pp [12.37, 35.42] |
Values are paired accuracy changes with 95% build-bootstrap intervals. Each intervention has its own fresh control; the original incorrect answer is not treated as a zero-accuracy replay baseline. These results give developers evidence for testing different repairs according to the initial evidence state. Because the controls differ, the two rows do not rank intervention policies. The replay uses adjudicated labels and reference-assisted selection, so it assesses diagnostic utility under controlled conditions rather than a deployed automatic selector.
The original held-out selection study provides complementary evidence: a rule fixed on development builds yields 46 additional correct answers on 1,200 test items for Qwen3-VL-8B over the score-only choice; the selectors agree for the other three configurations. Retained records also improve an existing evaluator: RAGChecker BA rises from 74.5% to 93.0% on incorrect answers and 82.0% to 91.0% on correct answers. Thus the benchmark provides both reusable evaluation information and controlled evidence for choosing what to test when improving an agent, complementing data and training methods such as MTA-Agent, REDSearcher and OpenSearch-VL.
Manuscript locations: §5.2 and Appendix I.2 (evaluator comparison); §5.4 and Appendix I.5, Tables S6b–S6c (controlled replay); Appendix G.1, Table 20 (held-out intervention selection).
Kind Reminder of the Discussion Period Ending Soon
Dear Reviewer 1ycj,
We hope you are doing well.
As the discussion period will end on Nov 18, 11.59pm AoE, we would like to know whether our response has addressed your further concerns, and we would really appreciate it if you could take some time to read our responses and provide any further feedback that you have. Your feedback is important to us, and we would like to have the opportunity to discuss them with you before the discussion period ends.
Thank you again for helping us, we believe our paper has been standing much stronger after incorporating your constructive suggestions.
Best regards,
Response to Q1
Dear Reviewer 1ycj,
Thanks for your constructive comments. Below we will address your concerns.
Q1: Add visual tools and rerun the evaluation.
A1: We added OCR, image cropping and reverse-image search, and evaluated text tools versus the extended tool set under paired conditions. The pilot uses Qwen3.5-27B in thinking mode on 180 items from 30 builds, with two items per build in each of three visual strata. Every item is evaluated under both tool sets at ten and twenty actions, giving 720 episodes. The original twelve-system evaluation and this separate tool-extension pilot address different scopes.
At ten actions, accuracy increases from 50.00% to 55.56%, comprising 31 corrections and 21 new errors; at twenty, it increases from 56.11% to 61.67%, with 24 corrections and 14 new errors. Intervals resample whole builds 2,000 times. Both pooled intervals include zero. The clearest observed benefit is for small text/charts at ten actions; the negative point estimate for other items shows why the aggregate gain alone is insufficient. These stratum results describe this cohort and do not identify the reason for regressions.
The comparisons hold item versions, model settings, action limits, visible-text caps and time limits fixed. Every search, page-open, OCR, crop or reverse-search call consumes one action, including failures. Full tool implementations, per-return limits and paired outcomes are reported in Appendix I.6. Equal action budgets do not imply equal computation cost.
This pilot directly tests whether adding the requested visual tools changes performance and which item strata are affected. It does not establish a tool-rich ranking for the original twelve systems or a release-wide gain. Qwen3.5-27B is a separate evaluation configuration, not a substitute for the original Qwen3-VL-32B result.
Manuscript locations: §5.4, “Visual-tool scope,” and Appendix I.6, Table S3a; §6 states the remaining tool and model scope.
Q2: Was the audit manually cleaned, and does admission favor easier or less diverse questions?
A2: We separate validity of the released questions, residual difficulty, and changes in composition.
No pre-audit manual cleaning. No questions were manually filtered or corrected before the audit. The domain-stratified sample was drawn from the frozen release, and all 600 sampled items remain in Table 6: 599 joint passes, zero confirmed defects and one unresolved case, giving 599/600=99.833%. This estimates validity under the audit rubric for the selected release; it is not the success rate of the unfiltered generator.
Residual difficulty. On the four-configuration visual-control evaluation, the original image condition gives 9.4% CB and 56.7% WS accuracy. Naming the referent in text, without giving the requested fact, changes these to 11.8% and 73.0% (Table 13). Target identification helps search, while most CB answers remain incorrect even after naming. These results characterize difficulty in the declared tool setting, not under every possible multimodal tool interface; the paired tool pilot above addresses that scope separately.
Selection and diversity. We compare the 16,401 proposed, 7,861 admitted and 6,000 released items using consistent topic and answer-type labels (Appendix I.1, Table S1b). In this experiment, science/technology increases from 5,000/16,401 proposals (30.49%) to 2,600/7,861 admitted items (33.07%), then decreases to the reported release share of 1,800/6,000 (30.00%). The corresponding share changes are +2.59 and −3.07 percentage points. Among answer types, Temporal decreases from 15.85% to 13.99% during admission (−1.86 points), before reaching 15.00%. These are the largest absolute admission shifts within their respective topic and answer-type classifications. The stage-wise comparison identifies category-dependent selection and distinguishes admission from subsequent composition-based release selection. It does not establish unchanged semantic diversity or equal difficulty across stages. Appendix I.1 separately measures the gate's quality–yield tradeoff.
Manuscript locations: §5.1 and Appendix D, Table 6 (frozen validity audit); Appendix E, Table 13 (visual controls); Appendix F (construction flow); Appendix I.1, Tables S1 and S1b (gate comparison and stage-wise composition); Appendix I.6 (visual-tool pilot).