- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Pix2Fact, a visual question-answering benchmark designed to evaluate the integration of expert-level visual perception and knowledge search in high-resolution real-world scenes. The benchmark comprises 1,000 high-resolution images and corresponding knowledge-intensive question–answer pairs across eight real-world scenarios. The authors construct the dataset through expert-driven annotation and evaluate ten vision-language models (VLMs) under four conditions combining original images, expert-cropped visual ground truth, and web search. The results show that even under the strongest setting (crop + search), the best-performing model achieves only 51.7% accuracy. The authors further conduct an error analysis to investigate the limitations of current VLMs in visual grounding, query construction, and external knowledge retrieval.
Critical strengths
-
The paper addresses a practically meaningful problem with a potentially valuable benchmark design. The paper focuses on integrating fine-grained visual grounding with open-world knowledge search in complex real-world scenes. Unlike visual question-answering tasks that rely solely on image content, Pix2Fact requires models to extract critical visual cues from high-resolution images and subsequently retrieve external knowledge to obtain the final answer. This design reflects the multi-stage information-seeking process encountered in real-world applications and provides a potentially valuable testbed for evaluating the integrated capabilities of VLMs.
-
The benchmark involves substantial annotation efforts and a systematic experimental design. The authors construct a benchmark comprising 1,000 high-resolution images across eight real-world scenarios, employing a multi-stage expert review process to improve annotation quality. Furthermore, ten VLMs are evaluated under different combinations of image inputs and web-search availability, allowing the authors to examine how visual grounding and external knowledge retrieval affect answering performance. These experiments provide useful empirical evidence regarding the limitations of current VLMs on knowledge-intensive visual question-answering tasks.
Critical weaknesses
-
The distinctive evaluation value of Pix2Fact remains insufficiently demonstrated, and the current experiments do not adequately substantiate its central contribution. The paper emphasizes that Pix2Fact uniquely evaluates the conjunction of fine-grained visual grounding and open-world knowledge search. However, the discussion and comparison with closely related benchmarks remain insufficient. For example, MMSearch has investigated multi-stage multimodal search evaluation, while MMSearch-Plus further involves fine-grained visual cue extraction, iterative search, and external knowledge retrieval. These studies are closely related to the stated objectives of Pix2Fact, yet the paper does not adequately clarify its substantive differences or additional evaluation value relative to these benchmarks. Moreover, the main experimental findings primarily highlight limitations in visual grounding and knowledge retrieval, together with accuracy improvements obtained through expert cropping and web search. Although these observations are meaningful, they do not sufficiently establish failure modes unique to the integration of these capabilities. More importantly, the evaluation relies primarily on final VQA accuracy. While the dataset includes target-region annotations, reference search queries, and supporting evidence, these annotations are not systematically utilized to evaluate intermediate stages such as visual cue identification, query construction, evidence retrieval, and knowledge integration. Consequently, the current experiments cannot reliably distinguish failures of individual capabilities from failures arising from their integration, nor do they fully substantiate the claimed synergy between visual grounding and knowledge search. This limitation directly weakens the benchmark's distinctive evaluation value and central contribution.
-
The stability of the benchmark, annotation quality, and reliability of the evaluation results require further validation. Pix2Fact relies on open-world knowledge search under search-enabled conditions, and some of the required information may be time-sensitive. Moreover, different models employ different web-search tools, potentially introducing variations in retrieval coverage and search effectiveness. Consequently, evaluation results may be influenced by knowledge updates, search-tool capabilities, and changes in the external search environment. The paper does not sufficiently clarify how these factors are controlled or how they affect the fairness and reproducibility of model comparisons. Furthermore, Appendix F reports that 30 out of 152 samples selected for detailed error analysis were excluded due to LLM-as-judge errors or annotation issues, with exclusion rates exceeding 40% in certain subsets. Although these statistics do not directly represent the error rate of the entire benchmark, and the authors provide additional validation of judge reliability, the paper lacks a detailed breakdown of these issues. It also remains unclear whether the identified annotation problems have been corrected in the original benchmark or whether they affect the main experimental results. For a benchmark intended to provide standardized evaluation, these uncertainties raise important concerns regarding the reliability of the reported findings.
Questions for authors
-
Regarding the distinctive evaluation value of Pix2Fact: Compared with existing multimodal search benchmarks such as MMSearch and MMSearch-Plus, what are the substantive differences in the evaluation objectives and failure modes captured by Pix2Fact? Can the authors provide more direct experimental evidence demonstrating that Pix2Fact reveals unique failure mechanisms that existing benchmarks cannot adequately capture, rather than simply increasing the difficulty of a combined task?
-
Regarding the evaluation of integrated capabilities: Given that the dataset already provides target-region annotations, reference search queries, and supporting evidence, why does the evaluation primarily rely on final VQA accuracy instead of systematically measuring the correctness of query construction, evidence retrieval, and knowledge integration? Furthermore, expert cropping does not necessarily guarantee correct entity recognition. How do the current experiments reliably distinguish visual understanding failures from subsequent knowledge search failures, and demonstrate that the observed performance gains reflect effective integration rather than improvements in individual capabilities?
-
Regarding annotation quality and LLM-as-judge reliability: Among the 30 excluded samples reported in Appendix F, how many were caused by LLM-as-judge errors, incorrect ground-truth answers, or other annotation issues? Have the confirmed annotation errors been corrected in the original benchmark? If so, were the main evaluation results recomputed, and did these corrections affect model rankings or the main conclusions? Additionally, please clarify the inconsistency between Section 4.3 and Appendix F regarding the number of valid samples (121 versus 122).
-
Regarding benchmark stability and evaluation fairness: For time-sensitive external knowledge, do the authors preserve the reference timestamps and evidence snapshots associated with the ground-truth answers to ensure evaluation consistency over time? Given that different models use different web-search services, are the cross-model comparisons intended to assess the intrinsic capabilities of VLMs or the performance of complete model–search-tool systems? Have the authors investigated how differences in search environments affect the main experimental findings?