- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Unlike existing VQAs, the authors introduce NoteVQA, a new benchmark customized to real-world consumer life.
Critical strengths
- In this paper, the authors introduce NoteVQA, a novel domain benchmark related to the real world. NoteVQA is designed to capture real-world scenarios, incorporating actual user community examples.
- The authors introduce a new rubric, IVR-12, to assess the quality of VQA answers. This covers content, presentation, and image categories.
- The authors conduct extensive experiments with several VLMs (including strong model).
Critical weaknesses
- Citation
- line 047, visual questions as challenges for VLMs [?]
- I think the authors made a mistake in citing FVQA. Given the number of samples and the domain, I believe the authors should cite [1], not [2].
- The authors explain NoteVQA as related to consumer life. But I'm curious about whether this data can solve by a real human. Especially, can a human respond in a similar way to a GT response when given a picture and information?
- The authors' target domain is real-world consumer life, but I'm worried that the LLM may introduce bias when selecting the data, which may not fully capture the diversity of human life (like culture).
- The authors conducted their experiments using ReAct, but I think it's also worth comparing it to other frameworks. Since this paper only proposes benchmarks, I think it's necessary to compare with other frameworks as well. Additionally, reporting on responses to the base VLM in a zero-shot setting would also be helpful.
- How does the SimpleQA style judge work? How does the judge determine the result, and what model are using? How does it align with humans? (for this point, the authors also need to report for IVR-12) The lack of these details undermines the reliability of the reported scores.
- Qwen model does everything. It would be good to use other models for cross-validated or report human evaluation for alignment.
- The authors don't provide detailed instructions for the judge and evaluation method, so it is challenging to fully reproduce the evaluation metrics.
[1] MMSearch-R1: Incentivizing LMMs to Search (Wu et al., ACL 2026) [2] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(10):2413–2427, 2017.
Questions for authors
- How were the weights for the overall score of IVR-12 determined?
- I think the authors should explain the details about the auditing stage (especially the human reviewer)
- Where is the expected answer for the examples? (Figure 5) The authors may use the Appendix for this.
- The authors don't provide detailed instructions, making it challenging to fully replicate the evaluation metrics. I would like to see this included in the final version.