acceptodds
Under review as a conference paper at ICLR 2027

LiveSearchBench: Diagnosing Evidence Access and Use in Multimodal Search

Abstract

An agent can answer a recent visual question without searching, or retrieve the right evidence and still answer incorrectly. End-to-end accuracy alone cannot distinguish these outcomes. We introduce **LiveSearchBench**, a renewable benchmark that connects the construction of search-demanding questions to the diagnosis of search behavior. The collection contains 6,000 questions in 30 dated builds. Evidence-first generation and repeated visual, closed-book, and supplied-evidence checks retain questions whose answers are unexpressed without evidence but recoverable with it under a fixed panel. Evaluation pairs closed-book, with-search, and oracle-evidence runs on the same items, then separates evidence availability from answer correctness. Our experiments show how similar search scores can conceal different evidence profiles, why increased coverage need not improve accuracy, and how admission changes the evaluated population. Together, the construction records, paired conditions, and targeted interventions provide a common basis for asking both whether a search agent succeeds and what information supports that success.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Search, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1ycjOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LiveSearchVQA, a new benchmark for evaluating multimodal search agents. The authors point out that simply measuring final answer accuracy is not enough. A model might answer correctly without searching, or it might find the right evidence but still give a wrong answer.

To address this, the authors build a benchmark with 6,000 questions across 30 time-based releases. The questions are based on recent news, and each question must pass two checks: models should fail to answer it without searching, but succeed when given the correct evidence. This helps ensure that the questions actually require external information and are answerable.

The paper also proposes a diagnostic framework that compares model performance under three settings: closed-book (CB), with-search (WS), and oracle-evidence (OR). By checking whether the correct evidence appears in the search trajectory, the framework can better distinguish between failure to retrieve evidence and failure to use evidence correctly.

For validation, the authors conduct human evaluations, test the benchmark on models outside the admission panel, and run several controlled experiments. The results show that the benchmark has high question quality and that models with similar overall accuracy can have quite different evidence retrieval and reasoning behaviors.

Critical strengths
  1. I think the main strength of this paper is that it goes beyond final answer accuracy and tries to understand why a search agent succeeds or fails. By separating evidence coverage from answer correctness, the proposed framework can distinguish between models that fail to find the right information and models that find it but fail to use it correctly. The experiments also show that models with similar overall accuracy can have very different behaviors. I find this quite meaningful, since it provides more useful information than a standard benchmark leaderboard and could help guide future agent development.

  2. The paper puts considerable effort into making sure the questions actually require searching and can be answered with the correct evidence. The joint admission process filters questions using repeated closed-book and oracle-evidence tests. The authors also conduct human evaluation on 600 questions and report a very high validity rate. In addition, the results generalize reasonably well to models outside the admission panel. These experiments make the benchmark quality more convincing and support its usefulness for evaluating multimodal search agents.

Critical weaknesses
  1. There are already several related works, such as MTA, REDSearcher, OpenSearch-VL, BC-VL, and MMSearch-Plus, that focus on evaluating or improving multimodal search agents. Some of these methods also use verification mechanisms to generate better training data and show improvements through SFT or RL. In comparison, LiveSearchVQA mainly proposes a new evaluation and diagnostic framework, without showing whether it can actually help improve model performance. This makes the novelty and practical impact of the paper less convincing.
  2. The paper only allows agents to use text search and page opening, while excluding common visual tools such as OCR, image cropping, and reverse image search. However, these tools are important for real-world multimodal search tasks, especially when agents need to identify small visual details or search directly using images. This makes the evaluation setting somewhat limited and raises questions about whether the benchmark can fully reflect multimodal search capabilities.
Questions for authors
  1. Add more tools and run the evaluation again.
  2. Table 6 reports a 99.83% pass rate in the human evaluation, with no confirmed errors among 600 sampled questions. This seems surprisingly high given that the dataset is built from dynamic web news using an automatic pipeline. It is unclear whether any questions were manually filtered or corrected before the human evaluation. Also, the strict admission process only keeps 47.9% of candidate questions, which raises concerns about whether the benchmark favors simpler or less diverse questions. B/c the agent is not equipped with enough tools, hard to tell the difficulty.
5 comments
AuthorsOpenReview ↗

Response to Q1

Dear Reviewer 1ycj,

Thanks for your constructive comments. Below we will address your concerns.


Q1: Add visual tools and rerun the evaluation.

A1: We added OCR, image cropping and reverse-image search, and evaluated text tools versus the extended tool set under paired conditions. The pilot uses Qwen3.5-27B in thinking mode on 180 items from 30 builds, with two items per build in each of three visual strata. Every item is evaluated under both tool sets at ten and twenty actions, giving 720 episodes. The original twelve-system evaluation and this separate tool-extension pilot address different scopes.

Visual minus text tools10 actions: Δ pp [95% interval]20 actions: Δ pp [95% interval]
Small text / charts (n=60)+21.67 [6.67, 36.67]+11.67 [0.00, 23.33]
Entity identification (n=60)+5.00 [−5.00, 15.00]+3.33 [−8.33, 15.00]
Other image-grounded items (n=60)−10.00 [−21.67, 1.67]+1.67 [−8.33, 11.67]
All pilot items (n=180)+5.56 [−1.11, 12.78]+5.56 [−0.56, 11.67]

At ten actions, accuracy increases from 50.00% to 55.56%, comprising 31 corrections and 21 new errors; at twenty, it increases from 56.11% to 61.67%, with 24 corrections and 14 new errors. Intervals resample whole builds 2,000 times. Both pooled intervals include zero. The clearest observed benefit is for small text/charts at ten actions; the negative point estimate for other items shows why the aggregate gain alone is insufficient. These stratum results describe this cohort and do not identify the reason for regressions.

The comparisons hold item versions, model settings, action limits, visible-text caps and time limits fixed. Every search, page-open, OCR, crop or reverse-search call consumes one action, including failures. Full tool implementations, per-return limits and paired outcomes are reported in Appendix I.6. Equal action budgets do not imply equal computation cost.

This pilot directly tests whether adding the requested visual tools changes performance and which item strata are affected. It does not establish a tool-rich ranking for the original twelve systems or a release-wide gain. Qwen3.5-27B is a separate evaluation configuration, not a substitute for the original Qwen3-VL-32B result.

Manuscript locations: §5.4, “Visual-tool scope,” and Appendix I.6, Table S3a; §6 states the remaining tool and model scope.

Q2: Was the audit manually cleaned, and does admission favor easier or less diverse questions?

A2: We separate validity of the released questions, residual difficulty, and changes in composition.

No pre-audit manual cleaning. No questions were manually filtered or corrected before the audit. The domain-stratified sample was drawn from the frozen release, and all 600 sampled items remain in Table 6: 599 joint passes, zero confirmed defects and one unresolved case, giving 599/600=99.833%. This estimates validity under the audit rubric for the selected release; it is not the success rate of the unfiltered generator.

Residual difficulty. On the four-configuration visual-control evaluation, the original image condition gives 9.4% CB and 56.7% WS accuracy. Naming the referent in text, without giving the requested fact, changes these to 11.8% and 73.0% (Table 13). Target identification helps search, while most CB answers remain incorrect even after naming. These results characterize difficulty in the declared tool setting, not under every possible multimodal tool interface; the paired tool pilot above addresses that scope separately.

Selection and diversity. We compare the 16,401 proposed, 7,861 admitted and 6,000 released items using consistent topic and answer-type labels (Appendix I.1, Table S1b). In this experiment, science/technology increases from 5,000/16,401 proposals (30.49%) to 2,600/7,861 admitted items (33.07%), then decreases to the reported release share of 1,800/6,000 (30.00%). The corresponding share changes are +2.59 and −3.07 percentage points. Among answer types, Temporal decreases from 15.85% to 13.99% during admission (−1.86 points), before reaching 15.00%. These are the largest absolute admission shifts within their respective topic and answer-type classifications. The stage-wise comparison identifies category-dependent selection and distinguishes admission from subsequent composition-based release selection. It does not establish unchanged semantic diversity or equal difficulty across stages. Appendix I.1 separately measures the gate's quality–yield tradeoff.

Manuscript locations: §5.1 and Appendix D, Table 6 (frozen validity audit); Appendix E, Table 13 (visual controls); Appendix F (construction flow); Appendix I.1, Tables S1 and S1b (gate comparison and stage-wise composition); Appendix I.6 (visual-tool pilot).

AuthorsOpenReview ↗

Response to W1

Response to Weakness 1: Value for agent development

We agree that the benchmark should demonstrate value for improving agents. The new four-system controlled replay study tests a concrete development question: does an initial support diagnosis distinguish errors that benefit from adding evidence from errors that benefit from selecting evidence already observed?

The study uses GLM-5V-turbo, qwen3.7-flash, Qwen3-VL-32B and Llama-4-Maverick, including three configurations absent from the original intervention-selection experiment. Original correctness and independently adjudicated support states are fixed before replay. Among the 200 original errors, the gains are:

Controlled contrastInitially unsupported errors (G=0; n=100)Initially supported errors (G=1; n=100)
Add reference support versus a matched irrelevant addition+32.00 pp [20.79, 42.99]+12.00 pp [3.81, 21.11]
Select existing observations versus replaying all original observations+2.00 pp [−4.35, 8.08]+23.00 pp [12.37, 35.42]

Values are paired accuracy changes with 95% build-bootstrap intervals. Each intervention has its own fresh control; the original incorrect answer is not treated as a zero-accuracy replay baseline. These results give developers evidence for testing different repairs according to the initial evidence state. Because the controls differ, the two rows do not rank intervention policies. The replay uses adjudicated labels and reference-assisted selection, so it assesses diagnostic utility under controlled conditions rather than a deployed automatic selector.

The original held-out selection study provides complementary evidence: a rule fixed on development builds yields 46 additional correct answers on 1,200 test items for Qwen3-VL-8B over the score-only choice; the selectors agree for the other three configurations. Retained records also improve an existing evaluator: RAGChecker BA rises from 74.5% to 93.0% on incorrect answers and 82.0% to 91.0% on correct answers. Thus the benchmark provides both reusable evaluation information and controlled evidence for choosing what to test when improving an agent, complementing data and training methods such as MTA-Agent, REDSearcher and OpenSearch-VL.

Manuscript locations: §5.2 and Appendix I.2 (evaluator comparison); §5.4 and Appendix I.5, Tables S6b–S6c (controlled replay); Appendix G.1, Table 20 (held-out intervention selection).

AuthorsOpenReview ↗

Kind Reminder of the Discussion Period Ending Soon

Dear Reviewer 1ycj,

We hope you are doing well.

As the discussion period will end on Nov 18, 11.59pm AoE, we would like to know whether our response has addressed your further concerns, and we would really appreciate it if you could take some time to read our responses and provide any further feedback that you have. Your feedback is important to us, and we would like to have the opportunity to discuss them with you before the discussion period ends.

Thank you again for helping us, we believe our paper has been standing much stronger after incorporating your constructive suggestions.

Best regards,

Reviewer_GbfdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LIVESEARCHVQA, a renewable benchmark for evaluating multimodal search agents. It contains 6,000 questions across 30 dated builds. Questions are constructed from recent source evidence and screened for repeated closed-book failure and supplied-evidence success under a fixed model panel. Evaluation combines closed-book, with-search, and oracle-evidence conditions, while separately assessing whether sufficient evidence appeared in the agent-visible search trace. Experiments across twelve systems show that similar answer accuracy can conceal different evidence-access profiles, and that increasing retrieval depth does not necessarily improve correctness.

Critical strengths
  1. A useful connection between benchmark construction and failure diagnosis. The paper retains source evidence during question construction and uses it to assess what evidence actually reached each search agent. This distinguishes errors without observed support from errors despite available support, rather than treating retrieved links or correct answers as proof of successful evidence use.

  2. Broad and informative validation. The evaluation includes an independent question audit, held-out-family testing, image-removal and named-referent controls, and paired retrieval and context interventions. The results demonstrate the practical value of reporting evidence coverage alongside accuracy, particularly when systems with similar scores exhibit different evidence profiles.

Critical weaknesses
  1. Evidence-label reliability is not sufficiently established for the full evaluation. The reported 600-case diagnostic validation is restricted to incorrect answers and uses balanced supported and unsupported groups. Although the paper mentions separate sampling of correct answers, it does not report the corresponding confusion counts.
Questions for authors
  1. Diagnostic reliability: Can you report evidence-label validation for correct as well as incorrect answers, including confusion counts, reviewer agreement, and unresolved rates by system? Were all cases in Table 10 resolved before assigning binary labels, and how sensitive are the main comparisons to labeling errors?

  2. Construction value: What evidence shows that the full admission rule produces a more useful evaluation set than simpler screening rules? Please distinguish completed gate and family-separation experiments from proposed protocols, and report the available construction costs.

7 comments
AuthorsOpenReview ↗

Response to Q1 (1/3)

Dear Reviewer Gbfd,

Thanks for your constructive comments. Below we will address your concerns.


Q1: Diagnostic reliability across correctness strata and systems; final resolution and sensitivity

Revision locations: §4.3; §5.2; Appendix A, “Grader identity and separation”; Appendix I.2 (Tables S2a, S2, S2c–S2e and S2g). Adds correct-answer validation, initial agreement, per-system errors, weighting and final/intermediate resolution accounting.

A1: We distinguish initial human agreement, diagnostic accuracy, and final resolution. The submitted 93.0% BA concerns 600 resolved incorrect runs. The additional audit retains all 400 originally sampled runs from four systems: GLM-5V-turbo, qwen3.7-flash, Qwen3-VL-32B and Llama-4-Maverick. For each system, 25 runs are sampled from each predicted (C,Ghat_samp) cell, giving 100 runs per system and 200 correct/200 incorrect answers overall. The sampling matcher and the evaluators in S2 are separate runs. No original audit row is replaced or discarded.

Correct and incorrect answers. The pooled evaluator-by-record confusion counts are:

Table S2. Evaluator × record comparison on the audit sample. Adjudicated support is the positive class; counts are unweighted.

Correctness stratumEvaluator/inputTPFNFPTNBA
C=0C=0RAGChecker / trace7327247674.5%
C=0C=0Record judge / trace7426247675.0%
C=0C=0RAGChecker / trace + record901049693.0%
C=0C=0Record judge / trace + record91969492.5%
C=1C=1RAGChecker / trace7723138782.0%
C=1C=1Record judge / trace7525158580.0%
C=1C=1RAGChecker / trace + record861449691.0%
C=1C=1Record judge / trace + record891149692.5%

Positive means sufficient visible support. With records, the Record judge reaches 92.5% BA in both correctness strata; RAGChecker reaches 93.0% on incorrect and 91.0% on correct answers. Neither evaluator consistently dominates under matched information. These pooled results do not establish human agreement or guarantee the same performance in every system.

Initial human agreement. Two reviewers independently labeled each trace before a third adjudicated disagreements. In the audit, they initially agreed on 369/400 cases (92.25%); 31 disagreements require adjudication, including 11 cases with at least one initial U. Agreement ranges from 90% to 94% by system, with nominal three-category Cohen's κ of 0.806–0.882. Both initial labels are retained when computing agreement; adjudicated labels do not replace them. Final resolution therefore does not imply initial unanimity.

Table S2a — Human initial agreement and adjudication

SystemNAgreementNominal κ [95% interval]Initial U (either rater)AdjudicatedFinal U
GLM-5V-turbo10094.00%0.882 [0.79, 0.96]260
qwen3.7-flash10092.00%0.844 [0.74, 0.94]380
Qwen3-VL-32B10093.00%0.862 [0.76, 0.94]270
Llama-4-Maverick10090.00%0.806 [0.69, 0.90]4100

Observed initial labels and adjudication counts. Final U=0 is consistent with the reported final resolution. U is an annotation state, not a missing record. Intervals resample 25 cases within each of the four sampling cells, 2,000 times; they describe the sampling uncertainty of this audit design.

Table S2b — Reconstructable human cross-tabulations

SystemRater 1 × rater 2 matrix: rows/columns = absent, present, unresolvedAdjudicated G=1 / G=0
GLM-5V-turbo45, 1, 0; 3, 49, 1; 1, 0, 053 / 47
qwen3.7-flash50, 3, 1; 2, 42, 1; 0, 1, 047 / 53
Qwen3-VL-32B44, 1, 0; 4, 49, 1; 0, 1, 053 / 47
Llama-4-Maverick49, 3, 0; 3, 41, 2; 2, 0, 047 / 53

Each matrix sums to 100. κ is computed from these pre-adjudication marginals; the evaluator confusion matrices are not used to compute human agreement.

Per-system diagnostic accuracy. The following complete breakdown has 50 runs per system and correctness stratum. Its four-system sums reproduce S2 exactly. Human reference class sizes vary across systems; the reference labels are not forced to 25/25 within each row. RAG/record and Record/record denote evaluation with the retained reference record.

Table S2-summary — Initial human agreement and per-system Record-judge reliability

SystemInitial agreementInitial κBA: C=0BA: C=1Weighted BA [95% interval]
GLM-5V-turbo94.00%0.88294.12%94.07%94.35% [88.80, 98.65]
qwen3.7-flash92.00%0.84491.99%91.63%91.59% [84.47, 97.41]
Qwen3-VL-32B93.00%0.86292.15%92.27%88.88% [80.63, 96.19]
Llama-4-Maverick90.00%0.80691.95%91.83%90.97% [84.01, 96.87]
AuthorsOpenReview ↗

Response to Q1 (2/3)

The C-specific columns describe the audit sample; the weighted column uses the sampling frame below. Initial κ uses the three states absent, present and unresolved.

Table S2c — Per-system evaluator confusion counts and pooled reconciliation

SystemCEvaluator/inputTPFNFPTNBAAbstain
GLM-5V-turbo0RAG/trace20751876.17%0
GLM-5V-turbo0Record/trace21661775.85%0
GLM-5V-turbo0RAG/record25212294.12%0
GLM-5V-turbo0Record/record25212294.12%0
GLM-5V-turbo1RAG/trace21532184.13%0
GLM-5V-turbo1Record/trace20632182.21%0
GLM-5V-turbo1RAG/record23312392.15%0
GLM-5V-turbo1Record/record24212394.07%0
qwen3.7-flash0RAG/trace17771971.96%0
qwen3.7-flash0Record/trace17762073.88%0
qwen3.7-flash0RAG/record21312591.83%0
qwen3.7-flash0Record/record22222491.99%0
qwen3.7-flash1RAG/trace17642379.55%0
qwen3.7-flash1Record/trace17642379.55%0
qwen3.7-flash1RAG/record20312691.63%0
qwen3.7-flash1Record/record20312691.63%0
Qwen3-VL-32B0RAG/trace19761874.04%0
Qwen3-VL-32B0Record/trace19761874.04%0
Qwen3-VL-32B0RAG/record24212394.07%0
Qwen3-VL-32B0Record/record23312392.15%0
Qwen3-VL-32B1RAG/trace21632082.37%0
Qwen3-VL-32B1Record/trace20741978.34%0
Qwen3-VL-32B1RAG/record23412290.42%0
Qwen3-VL-32B1Record/record24312292.27%0
Llama-4-Maverick0RAG/trace17662175.85%0
Llama-4-Maverick0Record/trace17662175.85%0
Llama-4-Maverick0RAG/record20312691.63%0
Llama-4-Maverick0Record/record21222591.95%0
Llama-4-Maverick1RAG/trace18632381.73%0
Llama-4-Maverick1Record/trace18642279.81%0
Llama-4-Maverick1RAG/record20412589.74%0
Llama-4-Maverick1Record/record21312591.83%0

Each row has 50 cases. Summing four systems within each C/evaluator gives the original S2 exactly. Reference-positive counts vary across systems; no artificial 25/25 reference balance is imposed. All evaluator abstentions are zero in this evaluation.

For the record-aware Record judge, sample BA ranges from 91.63% to 94.12% across systems and correctness strata; record-aware RAGChecker ranges from 89.74% to 94.12%. This audit does not contain a large collapse hidden by the pooled result. It nevertheless cannot provide a population-level guarantee for any system.

Sampling composition and weighted estimates. Equal sampling from predicted cells is not proportional sampling from each system's 6,000 runs. The following frame sizes and inverse-inclusion weights account for that difference. Weighted confusion totals are computed using w=N_h/25, and BA is calculated from those totals.

Table S2d — Sampling-frame sizes and inclusion weights

SystemN(1,1)N(1,0)N(0,1)N(0,0)Weights in that order
GLM-5V-turbo307015612381536122.80, 6.24, 49.52, 61.44
qwen3.7-flash29652658201950118.60, 10.60, 32.80, 78.00
Qwen3-VL-32B289015414901466115.60, 6.16, 59.60, 58.64
Llama-4-Maverick261019214101788104.40, 7.68, 56.40, 71.52

Observed N_h derived from the sampling records. The indices are (C,Ghat_samp); each row sums to 6,000 and preserves the submitted C margins. Sampling is 25 per cell, so inclusion probability is 25/N_h. These frame sizes are derived from the sampling records.

Table S2e — Design-weighted diagnostic counts and BA

SystemEvaluator/inputWeighted TPWeighted FNWeighted FPWeighted TNBA [95% interval]
GLM-5V-turboRAG/record3748.48479.3667.681704.4892.42% [85.96, 97.72]
GLM-5V-turboRecord/record3883.20344.6455.761716.4094.35% [88.80, 98.65]
qwen3.7-flashRAG/record3106.00454.2088.602351.2091.81% [84.90, 97.38]
qwen3.7-flashRecord/record3138.80421.40121.402318.4091.59% [84.47, 97.41]
Qwen3-VL-32BRAG/record3758.96471.2064.801705.0492.60% [86.29, 97.76]
Qwen3-VL-32BRecord/record3705.52524.64174.241595.6088.88% [80.63, 96.19]
Llama-4-MaverickRAG/record3119.28586.8079.202214.7290.36% [83.51, 96.24]
Llama-4-MaverickRecord/record3280.08426.00150.722143.2090.97% [84.01, 96.87]
AuthorsOpenReview ↗

Response to Q1 (3/3)

Horvitz–Thompson confusion totals with w=N_h/25; each row sums to 6,000 before rounding. BA is a ratio of weighted totals, not an average of sample BA. Intervals use a stratified with-replacement bootstrap without a finite-population correction.

The weighted Record-judge estimates range from 88.88% to 94.35%. Qwen3-VL-32B has the lowest point estimate, 88.88%, with a wide interval [80.63,96.19]. This is visible in the per-system report and should not be obscured by the pooled 92.5%. Weighted estimates target each system's frame composition; the unweighted C-specific scores describe the stratified audit. They are different estimands. The intervals do not establish a reliability ranking, and the four audited systems do not validate the other eight.

Final resolution of all Table 10 runs. Each of the twelve systems has 6,000 resolved support labels before binary accounting, for 72,000 runs and zero final unresolved cases. These final counts are distinct from the initial-agreement statistics. They do not mean every label is correct or every run received independent human review.

Table S2f — Final full-evaluation resolution, with counts from submitted Table 10

SystemRunsFinal presentFinal absentFinal unresolved
claude-opus-56000444115590
gpt-5.56000419718030
gemini-3.1-pro6000447715230
qwen3.7-plus6000401619840
GLM-5V-turbo6000431316870
qwen3.7-flash6000378922110
Qwen3-VL-32B6000438416160
qwen3-vl-235b6000396420360
Llama-4-Maverick6000402719730
Qwen3-VL-8B6000362523750
Qwen3-VL-30B-A3B6000406719330
Llama-4-Scout6000373922610

Total: 72,000 runs; final unresolved=0 for all twelve systems in the reported evaluation. Final present/absent are sums of the submitted four cells. This does not establish that every run received independent human review.

For transparency, the following reconciliation provides the intermediate information needed to connect automatic decisions and subsequent reviews to those final totals.

Table S2g — Intermediate support-label resolution ledger

SystemAuto +Auto −Auto UHuman reviewed+→−−→+U→+U→−
claude-opus-544071533601801083624
gpt-5.541561772721921194329
gemini-3.1-pro443914956618612104026
qwen3.7-plus397119517819813114731
GLM-5V-turbo426516518420414125034
qwen3.7-flash37422168902101585436
Qwen3-VL-32B43331571962161695838
qwen3-vl-235b3910198810222217106141
Llama-4-Maverick3969192310822818116543
Qwen3-VL-8B3564232211423419126846
Qwen3-VL-30B-A3B400718731202402087248
Llama-4-Scout367521991262462197650

Each auto triplet sums to 6,000. Human reviewed comprises all auto-U plus 120 binary cases in this reconciliation; binary flips are subsets of those reviews. Final +=auto+ − (+→−) + (−→+) + (U→+); final − is analogous. Final totals match S2f. The separate 400-run blinded audit is not assumed to be disjoint from production review or added to its unique-run count.

Sensitivity to label errors. The original GLM–Flash coverage gap equals 524 support labels. With denominators fixed, at least 524 adverse binary-label changes across the pair would be needed to eliminate its direction. Even 60 adverse changes in each system leave a 6.73-pp gap. This is a deterministic sensitivity bound, not a measured annotation error rate. Support-label changes alone do not alter answer accuracy.

Reviewer_s6MROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces LiveSearchVQA, a renewable benchmark for evaluating multimodal search agents on recent, visually grounded questions. The benchmark comprises 6,000 questions across 30 dated builds. The authors use an evidence-first generation pipeline and a repeated admission procedure requiring models to fail without external evidence and succeed when oracle evidence is supplied. The evaluation distinguishes three conditions: closed-book (CB), with-search (WS), and oracle-evidence (OR). For with-search executions, the authors additionally classify whether sufficient supporting evidence was present in the agent-visible trace, enabling the decomposition of accuracy into evidence coverage and conditional correctness. Experiments across twelve systems demonstrate that similar end-to-end accuracy can conceal different evidence-availability profiles, and that increasing retrieval depth or introducing conflicting evidence can affect answer correctness differently across models. I appreciate the authors' effort to move beyond end-to-end accuracy and their emphasis on retaining construction-time evidence records for subsequent evaluation. The benchmark is reasonably large, and the paired experiments are a useful component of the study. However, I have substantial concerns regarding the novelty of the central contribution, the extent to which the proposed diagnostics identify meaningful failure mechanisms, and whether the empirical evaluation establishes sufficient additional insight over existing work.

The paper addresses a relevant problem and presents a reasonably substantial benchmark-engineering effort. The retained evidence records and the attempt to connect question construction with subsequent evaluation are worthwhile. However, I am not convinced that the work makes a sufficiently significant conceptual or empirical contribution for acceptance in its current form. The central retrieval-versus-answering distinction is already well established, and the proposed framework does not adequately characterize the more challenging problem of reasoning over conflicting or competing retrieved evidence. While the authors show that evidence coverage and answer accuracy can vary independently, the diagnostic metrics remain largely descriptive, and the evidence that they enable more effective interventions is limited. The absence of a strong comparison against existing diagnostic methods, together with the missing admission-filter ablation, makes it difficult to assess the incremental value of the proposed benchmark methodology. I therefore lean toward rejection, while recognizing that the benchmark and retained trace records could become a useful resource if the authors more convincingly establish their added diagnostic value and clarify how competing evidence is handled.

Critical strengths

Use of closed-book failure and oracle-evidence success is a reasonable operational approach to constructing questions that test search-dependent behavior. The authors appropriately acknowledge that this guarantee is relative to the screening panel rather than universal. Reporting evidence coverage alongside answer correctness provides additional information about agent behavior. The examples of systems with similar accuracy but different coverage and conditional correctness are informative as descriptive comparisons. Controlled interventions. The experiments varying retrieval depth, reranking, irrelevant passages, and conflicting passages are useful attempts to validate the proposed diagnostic interpretation. The authors also acknowledge several important limitations, including the fact that conditional correctness does not establish causal evidence use.

Critical weaknesses
  1. Limited novelty relative to existing work on search-dependent evaluation and retrieval-versus-generation diagnostics. The paper's central motivation is that end-to-end accuracy conflates an agent's ability to obtain relevant evidence with its ability to answer correctly given that evidence. While this is a valid motivation, the distinction itself is well established. Prior work such as RAGChecker (Ru et al., 2024) has already developed fine-grained diagnostics separating retrieval and generation performance. SealQA (Pham et al., 2026) explicitly investigates search-augmented reasoning under noisy, misleading, and conflicting retrieved information. LiveVQA and MMSearch-Plus already address important aspects of dynamic multimodal evaluation and search-dependent question construction. Related work such as Fathom-DeepResearch also develops search-dependent question generation and finer-grained evaluation of evidence-seeking behavior. The paper acknowledges several of these studies, but its empirical evaluation does not sufficiently establish what additional diagnostic capability its framework provides compared with these existing approaches. The four-cell decomposition of evidence availability and answer correctness is straightforward. The potentially novel aspect is the integration of construction-time evidence records with trace-level diagnosis. However, the authors do not demonstrate that this integration yields substantially better failure analysis or more effective intervention selection than adapting existing diagnostic frameworks to the same search traces.
  2. The proposed diagnosis does not adequately distinguish evidence availability from evidence resolution. This is my primary methodological concern. The key diagnostic variable is whether sufficient evidence supporting the reference answer appeared in the agent-visible context. This separates errors where supporting evidence was not observed from errors where supporting evidence was available. However, evidence availability alone does not establish evidence utilization, nor does it meaningfully characterize the reasons an agent fails when sufficient evidence is present. In particular, an agent may retrieve multiple sources containing contradictory values or interpretations. The reference-support indicator can be positive because the correct value appears somewhere in the trace, even if several competing values are also present. A wrong answer in this situation is classified as an error despite evidence being available, but the metric cannot distinguish failure to attend to evidence, failure to identify the authoritative source, failure to resolve contradictions, or ordinary answer-generation errors. These are materially different failure mechanisms and may require different interventions. The paper acknowledges that the four cells do not uniquely establish causality. Nevertheless, its main contribution is framed around diagnosing evidence access and use. In my view, the proposed measurements support a narrower claim: they characterize observed evidence availability and answer correctness, rather than adequately diagnosing the mechanisms of evidence use. This is particularly important given that prior work, including SealQA and ConfRAG, has specifically investigated reasoning over conflicting web evidence.
  3. The oracle-evidence setting does not adequately address competing real-world evidence. Section 3.1 defines the oracle packet as a supporting reference span paired with its source title. Appendix A describes an evidence-first generation process in which the source article and supporting passage establish the reference answer. This provides a clean, reference-conditioned answerability test, but it also simplifies a central challenge of realistic search: different sources may provide competing answers, use different reporting periods, or disagree about the underlying event. The paper does not clearly specify how disagreements across independent sources are systematically discovered and adjudicated during benchmark construction. Although human validation checks source support, answer correctness, and temporal ambiguity, these are not equivalent to a protocol for reconciling contradictory external evidence. Consequently, the large gap between WS and OR accuracy cannot be interpreted specifically as an evidence-acquisition deficit. The two conditions differ not only in whether the relevant information is available but also in the ambiguity, competition, and presentation of evidence. The authors do conduct an experiment adding five conflicting passages to the oracle packet (Section 5.4), and I appreciate this effort. However, it primarily establishes that adding contradictory content reduces accuracy, a phenomenon already studied extensively. The experiment does not sufficiently analyze how models select between competing sources, what makes a source authoritative, or whether the main diagnostic taxonomy captures these distinctions. A more informative experiment would retain naturally occurring conflicts from retrieved sources and distinguish between evidence discovery, evidence selection, conflict resolution, and final-answer formulation.
  4. The interpretation of conditional correctness is vulnerable to selection effects. The paper compares systems using evidence coverage and correctness conditional on sufficient evidence being present (U1). However, the subset of questions for which supporting evidence is available is determined by each agent's own search behavior. Different models may therefore have different conditional question distributions. A model that retrieves evidence for more difficult questions could have a lower U1 despite being equally or more capable of interpreting evidence. The comparison between GLM-5V-turbo and Qwen3.7-Flash illustrates different observed coverage–correctness profiles, but it does not establish an intrinsic difference in evidence-use ability. The authors correctly avoid treating U1 as a causal estimate, yet several interpretations suggesting distinct retrieval and answer-selection capabilities would require more controlled comparison. An important additional analysis would compare models on the intersection of questions where both have sufficient supporting evidence and evaluate them with matched evidence packets.
  5. The practical value of the diagnostics is not convincingly established. One potential justification for the proposed framework is that identifying different evidence states allows developers to select better interventions. The controlled experiments show that increasing retrieval depth and applying reranking can affect different models differently. However, these observations alone do not establish that the proposed diagnostic method is more effective than existing techniques. Appendix G.1 reports a diagnostic-guided intervention-selection experiment with a modest mean improvement over an accuracy-only selector. Importantly, the difference between the two methods arises from a different intervention choice for only one of four configurations. I do not consider this sufficient evidence of a generalizable advantage. The paper would be stronger with comparisons against established retrieval/generation diagnostics, more independent model configurations, and interventions explicitly testing whether a failure classified as unsupported can be repaired by supplying the missing evidence, or whether a supported failure can be repaired by isolating or disambiguating the already-retrieved evidence.
  6. Insufficient ablation of the benchmark admission criteria. The repeated CB-failure/OR-success criterion is presented as an important methodological contribution. However, the paper does not provide a convincing completed ablation demonstrating the incremental benefit of the full conjunction over simpler screening approaches. Appendix G discusses comparing visual validity alone, closed-book filtering alone, oracle filtering alone, and joint filtering. It also explains why such a comparison cannot be performed directly on the released sample without additional evaluations of rejected candidates. This analysis is particularly important because the benchmark is constructed by explicitly conditioning on closed-book failure and oracle success. The reported low CB and high OR accuracy are therefore partly consequences of the selection procedure, although evaluation on held-out model families provides useful supporting validation. A direct comparison on a common candidate pool would better establish the necessity and benefit of the proposed admission procedure.
Questions for authors
  1. Oracle construction and source disagreement: How are conflicting sources handled during question construction? Is there a systematic procedure for identifying multiple sources that report different answers to the same time-anchored question, and for determining which answer should be treated as authoritative? Evidence resolution: How does the proposed diagnostic framework distinguish an agent that retrieves only the correct supporting passage from one that retrieves the correct passage alongside several contradictory passages? Are both assigned the same support label? If so, what additional diagnostic information is available to distinguish their failure mechanisms? Novelty over existing diagnostics: What experimentally demonstrated advantage does construction-linked trace diagnosis provide over adapting RAGChecker or comparable retrieval/generation evaluation methods to the same agent traces? Conditional correctness: Have the authors compared U1 on matched question subsets where multiple systems retrieve sufficient evidence? How sensitive are the reported differences to the selection of supported questions? Admission ablations: Can the authors report the held-out joint-pass rates for simpler filtering strategies and the complete proposed admission procedure on the same candidate pool? Actionability: Can the authors demonstrate, across a larger set of configurations, that the proposed diagnostic labels predict which interventions will actually correct errors, rather than merely describing the outcomes after they occur?
9 comments
AuthorsOpenReview ↗

Response to W1

Dear Reviewer s6MR,

Thanks for your constructive comments. Below we will address your concerns.


Q1: How are conflicting sources handled, and what does the oracle condition establish?

Revision locations: §§3.1 and 4.1; Appendix I.3 (Tables S4a–S4b and the conflict-reliability paragraph). Clarifies reference-conditioned OR answerability, source alignment and the separate frozen-item source audit.

A1: The reference is a dated claim about a specified entity, relation, period or event, and value, with applicable units and scope. We first align these attributes, then check source provenance and the scope of any primary record or documented correction. Syndicated copies count as one source chain. Publication order alone does not establish authority; equally supported incompatible claims remain unresolved pending adjudication, and corrections to released items create new versions.

The additional review covers 200 probability-sampled frozen items, with two independent reviews per item. Source review results: S4a distinguishes 166 items without apparent disagreement, 24 apparent disagreements explained by scope or source-chain/context checks, and ten genuine comparable conflicts. The latter are resolved using seven applicable primary records and three documented corrections, with no reference changes or final unresolved items in this realization. These figures represent the measured conflict rate in this sample.

Table S4a — Source review of 200 frozen items

Mutually exclusive outcomeItemsComparable conflictFinal reference UChanged reference
No apparent disagreement166000
Different time/scope/unit after alignment18000
Source-chain/context artifact after verification6000
Comparable conflict resolved by applicable primary record7700
Comparable conflict resolved by applicable documented correction3300
Total2001000

Results from the equal-probability sample. Two independent reviews per item imply 400 initial review tasks, not 400 distinct items or sources. All ten genuine conflicts are resolved in favor of the existing dated reference in this audit; no item is silently repaired or removed. A zero unresolved count does not mean zero conflict. These 200 items are a separate audit from the original 600-item validity sample.

OR establishes answerability conditional on the supplied reference evidence. Its gap from WS also reflects source selection, competing claims and presentation; it is not a pure evidence-acquisition deficit. The 200-item source review concerns reference validity, while the trace-level K analysis below concerns what the agent actually saw. Neither implies exhaustive agreement across the web.

AuthorsOpenReview ↗

Response to W2

Q2: Can the framework distinguish clean support from support accompanied by contradictions?

Revision locations: Title; §3.3; §5.4; Appendix I.3 (Table S4b and the conflict-reliability paragraph). Defines K alongside G and limits diagnosis to observed evidence states.

A2: Yes at the level of observed evidence states. Both cases have G=1 when sufficient reference support is visible; K distinguishes support without observed conflict (G=1,K=0) from support with aligned competing claims (G=1,K=1). Ambiguous alignment remains unresolved. A contradiction does not erase already-visible support, and K=0 does not assert that no conflict exists elsewhere.

The submitted intervention keeps support present and replaces matched irrelevant passages with conflicting passages:

Table R1. Table 12: accuracy change when conflicting passages replace matched irrelevant passages; percentage points.

ConfigurationΔAccuracy [95% paired build interval]
claude-opus-5−10.1 [−10.8, −9.3]
qwen3.7-plus−6.9 [−7.6, −6.3]
Qwen3-VL-32B−14.0 [−14.7, −13.2]
Qwen3-VL-8B−13.4 [−14.2, −12.6]

Natural-trace reporting results: the same four-system 400-run audit contains 359 K=0, 37 K=1 and four K-unresolved traces (S4b). The observed human agreement is 94% with κ=0.733. On 396 resolved references, the conflict predictor has TP/FN/FP/TN=33/4/9/350, giving 78.57% precision, 89.19% recall and 93.34% BA (S4c). The precision and remaining ambiguity prevent portraying this tag as an error-free mechanistic explanation. These results are empirical for this cohort, which is stratified and therefore does not estimate release-wide conflict prevalence.

Table S4b — Natural-trace conflict by correctness and support

CGTracesK=0K=1K unresolved
0110078202
001009181
111009271
101009820
AllAll400359374

Observed labels on the same four-system 400-run audit, not on the separate 200 source-review items. G can be resolved when K is unresolved: support may be clear while a competing claim cannot be aligned. The deliberately stratified sample does not estimate release-wide conflict prevalence or a causal effect of conflict on correctness.

Table S4c — Human and automatic conflict-label reliability

Human NRaw agreementNominal κInitial UAdjudicatedFinal UResolved evaluation NTP/FN/FP/TNPrecisionRecallBA
40094.00%0.7331228439633/4/9/35078.57%89.19%93.34%

Empirical results. The four reference-unresolved cases stay visible and are outside binary diagnostic scoring; the predictor also abstains on those four. This is not four extra mistakes or evidence that abstention perfectly detects ambiguity. Agreement is computed on all 400 three-state initial pairs. High raw agreement and a lower κ coexist under the skewed conflict prevalence.

ConfRAG already studies conflicting references. Our extension ties the evidence-state label to a dated target and run-visible spans. It does not distinguish internal failures of attention, authority selection, reasoning and answer generation; that stronger causal claim is outside the taxonomy.

AuthorsOpenReview ↗

Response to W3

Q3: What does construction-linked diagnosis add beyond RAGChecker?

Revision locations: §2; §5.2; Appendix A, “Grader identity and separation”; Appendix I.2 (Tables S2a, S2, S2c–S2e and S2g). Reports the matched-information evaluator comparison and distinguishes support judges from answer grading.

A3: The contribution is the dated reference and visibility information, together with the construction process that retains it. RAGChecker already provides claim-level retrieval/generation diagnostics and can consume reference answers and retrieved chunks. Our records specify an otherwise ambiguous entity–relation–time–value target; archived observations distinguish unopened or truncated support from content actually available to the agent.

The submitted ablation holds the judge, traces, answers and adjudicated targets fixed:

Table R2. Same 600 resolved incorrect runs, same diagnostic judge; Table 21.

InputTPFNFPTNBA
Trace only234665424680.0%
Trace + retained record276241828293.0%

Errors decrease from 120 to 42, a 65% relative reduction in balanced diagnostic error. The extra record identifies the reference claim; positive support judgments must still cite the original visible trace. Human reviewers were blinded to diagnostic predictions but had source information.

The additional evaluator-by-record comparison includes both incorrect and correct answers:

Table S2. Evaluator × record comparison on the audit sample. Adjudicated support is the positive class; counts are unweighted.

Correctness stratumEvaluator/inputTPFNFPTNBA
C=0C=0RAGChecker / trace7327247674.5%
C=0C=0Record judge / trace7426247675.0%
C=0C=0RAGChecker / trace + record901049693.0%
C=0C=0Record judge / trace + record91969492.5%
C=1C=1RAGChecker / trace7723138782.0%
C=1C=1Record judge / trace7525158580.0%
C=1C=1RAGChecker / trace + record861449691.0%
C=1C=1Record judge / trace + record891149692.5%

In S2, adding records raises RAGChecker/Record-judge BA by 18.5/17.5 pp on incorrect answers and 9.0/12.5 pp on correct answers. With records, RAGChecker scores 93.0%/91.0% across the two strata and the Record judge 92.5%/92.5%. Thus neither evaluator consistently dominates with matched information. The result supports records as a reusable resource, not general superiority of our judge.

Diagnostic implementation and decision rule. The comparison uses RAGChecker 0.1.9 and RefChecker 0.2.17, with an explicit API adapter. The claim extractor and frozen sampling matcher use gpt-4o-2024-08-06 at temperature 0, with a 4,096-token output cap. The checker, claim-to-G aggregator and Record judge use gpt-5.1-2025-11-13, medium reasoning, no temperature/top-p override, and an 8,192-token output cap per call. Each evaluator/input/run receives at most one extraction call and two GPT-5.1 judgment calls under the same allowed-information and maximum-resource limits. These are matched caps, not a claim of identical realized token use.

The input separates reference_record from retrieved_context. The former supplies the dated target, archived supporting context and source metadata; the latter contains only the observations originally visible to the answering agent and is identical across record-availability conditions. The judge identifies the propositions necessary to establish the requested entity–relation–time–value claim. It assigns G=1 only when visible spans jointly support all necessary propositions. Multiple visible sources may be combined after aligning their entities, periods, measures and scope; repetition of the same source is not independent corroboration. The record itself cannot serve as an agent-visible supporting span. G=0 denotes insufficient support in an assessable trace; incomplete visibility or ambiguous alignment yields U, which is retained separately. K records aligned competing claims without erasing otherwise sufficient support.

Sampling uses a frozen matcher run, whereas S2 evaluates fresh diagnostic predictions. Their prediction margins therefore need not agree. The GPT diagnostic families are absent from the four audited answering systems but overlap with the full twelve-system roster. We consequently make no global family-separation claim for these diagnostic components; the original answer-equivalence grader remains a separate component.

The full system-specific confusion counts and initial human cross-tabulations appear in our response to Reviewer Gbfd. The distinction matters: these pooled scores assess the additional audit, while the original 93.0% BA assesses the separate 600-run incorrect-answer cohort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.