acceptodds
Under review as a conference paper at ICLR 2027

Pix2Fact: When Vision Is Not Enough — Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes

Abstract

Despite progress on general tasks, vision-language models (VLMs) still struggle with challenges that demand both fine-grained visual grounding and external knowledge, a synergy overlooked by existing benchmarks that evaluate these abilities in isolation. To fill this void, we introduce Pix2Fact, a visual question-answering benchmark designed to assess expert-level visual perception and knowledge search. Pix2Fact comprises 1,000 high-resolution (4K+) images spanning eight scenarios. Its questions and answers are meticulously crafted by PhD-holding annotators from top global universities across diverse disciplines. Each question requires detailed visual grounding and the integration of external knowledge. Evaluating ten state-of-the-art VLMs, including proprietary models such as Gemini-3.1-Pro and GPT-5.4, we find that Pix2Fact poses a formidable challenge: the most advanced model (Gemini-3.1-Pro) achieves only 51.7% average accuracy, even with access to visual ground truth and search tools. Our analysis attributes this low accuracy to three factors, frequent visual grounding errors even with visual ground truth, shallow search harnessing, and VLM's inability to retrieve long-tail, unstructured local information. This striking gap exposes the limitations of current models in assisting humans with real-world scenarios that demand overwhelming visual comprehension. We believe Pix2Fact will serve as a critical benchmark to drive the next generation of language-vision agents that seamlessly integrate fine-grained perception with robust knowledge search.

Primary area:
datasets and benchmarks
Keywords:
Vision Large Language Model, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bYSXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Pix2Fact, a visual question-answering benchmark designed to evaluate the integration of expert-level visual perception and knowledge search in high-resolution real-world scenes. The benchmark comprises 1,000 high-resolution images and corresponding knowledge-intensive question–answer pairs across eight real-world scenarios. The authors construct the dataset through expert-driven annotation and evaluate ten vision-language models (VLMs) under four conditions combining original images, expert-cropped visual ground truth, and web search. The results show that even under the strongest setting (crop + search), the best-performing model achieves only 51.7% accuracy. The authors further conduct an error analysis to investigate the limitations of current VLMs in visual grounding, query construction, and external knowledge retrieval.

Critical strengths
  1. The paper addresses a practically meaningful problem with a potentially valuable benchmark design. The paper focuses on integrating fine-grained visual grounding with open-world knowledge search in complex real-world scenes. Unlike visual question-answering tasks that rely solely on image content, Pix2Fact requires models to extract critical visual cues from high-resolution images and subsequently retrieve external knowledge to obtain the final answer. This design reflects the multi-stage information-seeking process encountered in real-world applications and provides a potentially valuable testbed for evaluating the integrated capabilities of VLMs.

  2. The benchmark involves substantial annotation efforts and a systematic experimental design. The authors construct a benchmark comprising 1,000 high-resolution images across eight real-world scenarios, employing a multi-stage expert review process to improve annotation quality. Furthermore, ten VLMs are evaluated under different combinations of image inputs and web-search availability, allowing the authors to examine how visual grounding and external knowledge retrieval affect answering performance. These experiments provide useful empirical evidence regarding the limitations of current VLMs on knowledge-intensive visual question-answering tasks.

Critical weaknesses
  1. The distinctive evaluation value of Pix2Fact remains insufficiently demonstrated, and the current experiments do not adequately substantiate its central contribution. The paper emphasizes that Pix2Fact uniquely evaluates the conjunction of fine-grained visual grounding and open-world knowledge search. However, the discussion and comparison with closely related benchmarks remain insufficient. For example, MMSearch has investigated multi-stage multimodal search evaluation, while MMSearch-Plus further involves fine-grained visual cue extraction, iterative search, and external knowledge retrieval. These studies are closely related to the stated objectives of Pix2Fact, yet the paper does not adequately clarify its substantive differences or additional evaluation value relative to these benchmarks. Moreover, the main experimental findings primarily highlight limitations in visual grounding and knowledge retrieval, together with accuracy improvements obtained through expert cropping and web search. Although these observations are meaningful, they do not sufficiently establish failure modes unique to the integration of these capabilities. More importantly, the evaluation relies primarily on final VQA accuracy. While the dataset includes target-region annotations, reference search queries, and supporting evidence, these annotations are not systematically utilized to evaluate intermediate stages such as visual cue identification, query construction, evidence retrieval, and knowledge integration. Consequently, the current experiments cannot reliably distinguish failures of individual capabilities from failures arising from their integration, nor do they fully substantiate the claimed synergy between visual grounding and knowledge search. This limitation directly weakens the benchmark's distinctive evaluation value and central contribution.

  2. The stability of the benchmark, annotation quality, and reliability of the evaluation results require further validation. Pix2Fact relies on open-world knowledge search under search-enabled conditions, and some of the required information may be time-sensitive. Moreover, different models employ different web-search tools, potentially introducing variations in retrieval coverage and search effectiveness. Consequently, evaluation results may be influenced by knowledge updates, search-tool capabilities, and changes in the external search environment. The paper does not sufficiently clarify how these factors are controlled or how they affect the fairness and reproducibility of model comparisons. Furthermore, Appendix F reports that 30 out of 152 samples selected for detailed error analysis were excluded due to LLM-as-judge errors or annotation issues, with exclusion rates exceeding 40% in certain subsets. Although these statistics do not directly represent the error rate of the entire benchmark, and the authors provide additional validation of judge reliability, the paper lacks a detailed breakdown of these issues. It also remains unclear whether the identified annotation problems have been corrected in the original benchmark or whether they affect the main experimental results. For a benchmark intended to provide standardized evaluation, these uncertainties raise important concerns regarding the reliability of the reported findings.

Questions for authors
  1. Regarding the distinctive evaluation value of Pix2Fact: Compared with existing multimodal search benchmarks such as MMSearch and MMSearch-Plus, what are the substantive differences in the evaluation objectives and failure modes captured by Pix2Fact? Can the authors provide more direct experimental evidence demonstrating that Pix2Fact reveals unique failure mechanisms that existing benchmarks cannot adequately capture, rather than simply increasing the difficulty of a combined task?

  2. Regarding the evaluation of integrated capabilities: Given that the dataset already provides target-region annotations, reference search queries, and supporting evidence, why does the evaluation primarily rely on final VQA accuracy instead of systematically measuring the correctness of query construction, evidence retrieval, and knowledge integration? Furthermore, expert cropping does not necessarily guarantee correct entity recognition. How do the current experiments reliably distinguish visual understanding failures from subsequent knowledge search failures, and demonstrate that the observed performance gains reflect effective integration rather than improvements in individual capabilities?

  3. Regarding annotation quality and LLM-as-judge reliability: Among the 30 excluded samples reported in Appendix F, how many were caused by LLM-as-judge errors, incorrect ground-truth answers, or other annotation issues? Have the confirmed annotation errors been corrected in the original benchmark? If so, were the main evaluation results recomputed, and did these corrections affect model rankings or the main conclusions? Additionally, please clarify the inconsistency between Section 4.3 and Appendix F regarding the number of valid samples (121 versus 122).

  4. Regarding benchmark stability and evaluation fairness: For time-sensitive external knowledge, do the authors preserve the reference timestamps and evidence snapshots associated with the ground-truth answers to ensure evaluation consistency over time? Given that different models use different web-search services, are the cross-model comparisons intended to assess the intrinsic capabilities of VLMs or the performance of complete model–search-tool systems? Have the authors investigated how differences in search environments affect the main experimental findings?

Reviewer_e7wpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Pix2Fact is a visual question-answering benchmark comprising 1,000 image–question–answer examples across eight everyday scenarios. It aims to evaluate whether vision-language models can extract fine-grained visual cues from high-resolution images and use these cues to retrieve external information from the web. The main contributions are the manually constructed dataset, annotations of relevant visual regions and supporting evidence, and an evaluation examining the interaction between visual grounding and web search. The authors evaluate ten models under four conditions combining original versus expert-cropped images with search enabled versus disabled. Evaluation uses an LLM judge, supplemented by human audits and qualitative error analysis. Gemini-3.1-Pro achieves 18.4% accuracy with original images without search, 42.4% with original images and search, and 51.7% with cropped images and search. However, issues with annotation consistency, grading rules, and experimental reporting limit the strength of the conclusions.

Critical strengths
  1. The benchmark addresses a practically meaningful problem and provides potentially valuable annotations. Extracting a small visual cue from a real-world scene and using it to retrieve opening hours, contact information, or product attributes is an important capability for visual assistants.
  2. The factorial evaluation provides a useful starting point for diagnosis. Crossing original versus cropped images with search availability is more informative than reporting a single end-to-end accuracy. For example, Gemini-3.1-Pro gains 2.6 percentage points from cropping without search, compared with 9.3 points when search is enabled. This supports an interaction between visual input quality and retrieval effectiveness and motivates further investigation.
Critical weaknesses
  1. Concrete problems in annotations, grading, and error analysis undermine confidence in the benchmark’s measurements.
  • The grading prompt in Appendix C.3 explicitly rejects some semantically equivalent answers. For example, page 16 treats “Monday through Friday” as different from an explicit enumeration of the five weekdays. It also rejects additional factual claims even when those claims may be correct. Consequently, the evaluation can penalize answer formulation rather than factual correctness.
  • There is also a verifiable unit inconsistency in example 160-2. The question requests consumption in tons, rounded to the nearest million, but the ground-truth answer is 39, while its supporting evidence states approximately 39 million tons. The answer should therefore specify approximately 39,000,000 tons or explicitly state 39 million tons. Combined with the strict numerical matching rules, this inconsistency could cause a correctly scaled answer to be rejected.
  • Appendix F further reports excluding 30 of 152 deliberately selected cases because of judge or data issues. This proportion should not be extrapolated to the entire dataset, but it demonstrates that evaluation problems are substantial within the inspected difficult cases. The paper does not sufficiently establish how these issues affect the full benchmark scores and subsequent error distributions. High agreement between two automated judges does not resolve this concern, particularly when both follow the same problematic grading rules.
  1. The contribution is insufficiently distinguished from closely related benchmarks, weakening the central novelty claim. The paper repeatedly argues that existing benchmarks assess fine-grained visual understanding and external knowledge retrieval separately. However, it does not discuss closely related work that directly addresses their combination:
  • MMSearch-Plus requires extracting localized visual cues, propagating them through iterative image–text retrieval, and obtaining facts outside the input image. It also studies cropping and retrieval failures.
  • VDR-Bench contains 2,000 examples and uses a construction pipeline involving local-region cropping, web-scale visual search, entity verification, and multi-hop question expansion. These works substantially overlap with the gap Pix2Fact claims to fill. Combining fine-grained perception with open-web retrieval is therefore insufficient, by itself, to establish the contribution. Pix2Fact may offer meaningful additional value through higher-resolution everyday photographs, different target-region characteristics, or different factual information needs. However, the paper does not sufficiently quantify these distinctions or demonstrate that they reveal capabilities inadequately assessed by existing benchmarks.
Questions for authors
  1. Annotation and grading reliability: After correcting the semantic-equivalence rules and the unit inconsistency in example 160-2, do the main accuracies and model rankings remain stable? Can you provide a randomly sampled human audit representative of the full benchmark, including sample sizes, false-positive and false-negative rates, and a separation between ground-truth errors and judge errors?

  2. Experimental-record consistency: Was Claude evaluated with search? If not, what experiments support the C4 retrieval-failure claims in Section 4.3 and Appendix F? Please also reconcile the 121 versus 122 retained cases and explain the question–response mismatch in Case 635-2.

  3. Bottleneck attribution: Expert cropping still requires the model to recognize the target; it does not provide the correct entity identity. Do you have results that supply the correct entity name, or additionally supply the gold external evidence, to distinguish visual recognition, query construction, retrieval coverage, and answer-generation failures? Such evidence would materially affect the claim that external knowledge is the primary bottleneck.

  4. Distinctive contribution: What measurable properties distinguish Pix2Fact from MMSearch-Plus and VDR-Bench in task dependencies, visual difficulty, or retrieval structure? Which findings specifically depend on those properties, beyond demonstrating that current models struggle with multimodal search?

Reviewer_5gXJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Pix2Fact, a visual question-answering benchmark comprising 1,000 high-resolution images and expert-authored questions across eight everyday scene categories. The tasks connect localized visual cues with external facts obtained through web search. The authors evaluate ten vision-language models using original or expert-cropped images, with and without search where supported. The strongest reported result is 51.7% accuracy under the crop-plus-search condition. The paper also provides gain decomposition and manually inspected case studies intended to distinguish visual grounding, query construction, retrieval, and knowledge-integration failures.

Critical strengths

1.The benchmark focuses on a practically relevant problem. Answering questions about small signs, logos, products, and local businesses often requires both recognizing details in an image and finding information online. The use of high-resolution everyday scenes makes this a useful setting for evaluating how well VLMs combine these abilities.

2.The evaluation setup helps separate the roles of perception and search. Comparing original and cropped images, with and without search, makes the contribution of each component easier to assess. I also appreciate the effort put into annotation and validation, including three stages of review, supporting evidence, and human checks of automated judgments. Gemini-3.1-Pro, for example, improves from 18.4% in C1 to 42.4% with search in C2, and then to 51.7% with both search and expert crops in C4. These results suggest that both retrieval and accurate visual localization matter for this benchmark.

Critical weaknesses
  1. The scoring rules may penalize factually correct answers. Appendix C.3 rejects equivalent expressions such as “Monday through Friday” and an explicit list of those weekdays, as well as additional facts beyond the reference. Human audits also found judge or data issues in 14/32 “vg_rescue” and 10/24 “pix2fact_hard” cases. Although these selected subsets do not establish the overall error rate, they raise concerns about the resulting analysis. A representative human audit and rescoring with revised equivalence rules would help establish whether the main findings hold.

  2. Inconsistencies and unclear sampling weaken the failure analysis. Table 1 excludes Claude from search-enabled evaluation, yet later sections discuss its C4 search behavior. The main text and appendix also disagree on sample counts and error proportions, and case 635-2 pairs a car-purchase question with an unrelated answer about novels. Moreover, Gemini’s 92.5% accuracy and the 100% cropping-rescue rate come from selected subsets. The authors should clarify their selection criteria before drawing broader conclusions about model saturation or the effectiveness of cropping.

Questions for authors

1.Evaluation validity: After correcting the semantic-equivalence rules and independently adjudicating a representative sample across models and conditions, how much do the reported accuracies and rankings change? Please report audit sample sizes and false-positive/false-negative rates, and clarify whether discovered errors were corrected in Table 1 or only excluded from the case studies.

2.Experimental consistency: Was Claude actually run with search for any analysis? Please identify the configurations and source runs for Appendix F, reconcile the conflicting sample counts and percentages, and verify the question–response pairing in case 635-2.

3.Diagnostic claims: How were the disagreement, rescue, and residual-error groups selected? Were successful cropping outcomes part of the rescue-group selection criterion? Please distinguish observations conditional on these selected groups from estimates intended to characterize the full benchmark.

4.Contribution and benchmark validity: How does Pix2Fact differ from MMSearch-Plus, which already studies fine-grained visual cues and retrieval of out-of-image facts? A direct comparison would clarify the contribution beyond image resolution and scene coverage. Also, do question-only/search baselines verify that visual input is necessary, and how are changing facts such as prices and business hours anchored to dated evidence? [1] MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents, ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.