acceptodds
Under review as a conference paper at ICLR 2027

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

Abstract

Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains challenging. Existing benchmarks often target predefined capabilities, such as multi-hop retrieval or long-form synthesis, whereas users ask photo-grounded questions spanning a long tail of everyday scenarios. Despite advances in VLMs, users continue to turn to other people for help with everyday visual questions on image-sharing platform. Motivated by this behaviour, we curate NoteVQA from these questions, yielding 252 items across 12 topical categories and 7 user intents. Each item includes a concise reference distilled from expert community responses and a human-audited interleaved reference answer that combines textual explanations with supporting visual evidence. We evaluate short-answer correctness and interleaved-answer quality. To support the latter, we introduce AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, together with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved references and model outputs. Across 9 frontier VLMs, the highest short-answer accuracy is 52.8%, while adding agentic search to Qwen3.5-397B-A17B improves accuracy by only 2.0%. For interleaved answers, the same model running AgenticInterleave scores 3.52 under IVR-12, compared with 4.65 for the human-audited references, with the largest gap in content quality. These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, benchmark, real-life question

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MTTgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Unlike existing VQAs, the authors introduce NoteVQA, a new benchmark customized to real-world consumer life.

Critical strengths
  1. In this paper, the authors introduce NoteVQA, a novel domain benchmark related to the real world. NoteVQA is designed to capture real-world scenarios, incorporating actual user community examples.
  2. The authors introduce a new rubric, IVR-12, to assess the quality of VQA answers. This covers content, presentation, and image categories.
  3. The authors conduct extensive experiments with several VLMs (including strong model).
Critical weaknesses
  1. Citation
  • line 047, visual questions as challenges for VLMs [?]
  • I think the authors made a mistake in citing FVQA. Given the number of samples and the domain, I believe the authors should cite [1], not [2].
  1. The authors explain NoteVQA as related to consumer life. But I'm curious about whether this data can solve by a real human. Especially, can a human respond in a similar way to a GT response when given a picture and information?
  2. The authors' target domain is real-world consumer life, but I'm worried that the LLM may introduce bias when selecting the data, which may not fully capture the diversity of human life (like culture).
  3. The authors conducted their experiments using ReAct, but I think it's also worth comparing it to other frameworks. Since this paper only proposes benchmarks, I think it's necessary to compare with other frameworks as well. Additionally, reporting on responses to the base VLM in a zero-shot setting would also be helpful.
  4. How does the SimpleQA style judge work? How does the judge determine the result, and what model are using? How does it align with humans? (for this point, the authors also need to report for IVR-12) The lack of these details undermines the reliability of the reported scores.
  5. Qwen model does everything. It would be good to use other models for cross-validated or report human evaluation for alignment.
  6. The authors don't provide detailed instructions for the judge and evaluation method, so it is challenging to fully reproduce the evaluation metrics.

[1] MMSearch-R1: Incentivizing LMMs to Search (Wu et al., ACL 2026) [2] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(10):2413–2427, 2017.

Questions for authors
  1. How were the weights for the overall score of IVR-12 determined?
  2. I think the authors should explain the details about the auditing stage (especially the human reviewer)
  3. Where is the expected answer for the examples? (Figure 5) The authors may use the Appendix for this.
  4. The authors don't provide detailed instructions, making it challenging to fully replicate the evaluation metrics. I would like to see this included in the final version.
Reviewer_6527OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing multimodal QA benchmarks are "capability-first" — researchers define a target ability and then write queries for it — and that this produces questions unlike what users actually ask. NoteVQA instead mines real image-sharing-community posts (a Xiaohongshu-style UGC platform), keeping the user's photo and natural question. The curation funnel (Figure 3, Section 3.1) shrinks 5.24M raw notes to 5.84K candidates via a VLM vision-necessity gate and image–text consistency check, then to 338 high-confidence samples via a five-axis rubric (clarity, image quality, alignment, interleaved fit, objectivity) plus per-tag top-K selection, and finally to 252 items after query rewriting (removing answer-revealing domain terms) and a difficulty probe that runs Qwen3.5-397B-A17B ten times per item and uses response consistency to prune. The final set spans 12 topics (Figure 5; Household Pets 19.84%, Exotic Wildlife 16.27%, Plants/Gardening 13.49%) and 7 intents; queries average 22.7 words, short references 107.5 words, interleaved references 219.4 words with 378 embedded images (1.5/item).

Two secondary artifacts are introduced. AgenticInterleave (Section 3.3) is a single-agent ReAct loop with four tools (text search, page visit, reverse image search, forward image search), Serper backend, max 10 rounds, using a SUBJECT/BACKGROUND entity protocol that appends traceable image tags to entity descriptions. IVR-12 is a 12-dimension, 1–5 rubric (5 content, 3 presentation, 4 image) plus three binary gates (timeliness, safety redlines, refusal correctness), aggregated as S = 0.50·C̄ + 0.20·P̄ + 0.30·Ī if all gates pass, else 0. I verified this aggregation against Table 3 and it reproduces 4.65 and 3.52 exactly.

Experiments show 9 frontier VLMs (thinking mode) topping out at Gemini-3.1-Pro's 52.8% short-answer accuracy (Table 2; Doubao-Seed-2.0-Pro 49.6%, GLM-5.3-Flash 48.0%, Kimi-K2.6 47.6%, Claude-Opus-4.7 43.7%, Claude-Sonnet-4.6 35.3%, Qwen3.5-397B-A17B 35.7%). Adding agentic retrieval moves Qwen3.5-397B-A17B only 35.7% → 37.7% (+2.0). The instance-level decomposition (Table 4: 129 both wrong, 62 both correct, 33 search helps, 28 search hurts) is the paper's most interesting result — retrieval flips 33 errors but creates 28 new ones — and a judge-assigned failure taxonomy (Table 5) attributes 70.7% of the 157 incorrect runs to "confirmation/anchoring loops." Under IVR-12 (Table 3), the human-audited references score 4.65 vs 3.52 for AgenticInterleave, with content the widest category gap (4.82 vs 2.95; accuracy alone differs by 2.68). Finally, a controlled image ablation (Tables 6–7) shows image-critical probe accuracy rising 20.7% → 31.3% → 97.0% across text / placeholder / full-image conditions, with text-answerable controls flat at 97.9–98.4%.

Critical strengths
  • Real, non-synthetic query provenance with a documented funnel. The 5.24M → 5.84K → 338 → 252 pipeline (Section 3.1) is specified stage by stage, and the anti-shortcut measures are real design work: a vision-necessity gate, query rewriting to strip answer-revealing terms, and consistency-based difficulty filtering. Most capability benchmarks skip all three.
  • Genuinely dual-format references. Pairing a 107.5-word short reference with a 219.4-word human-audited interleaved reference on the same 252 items lets correctness and explanation quality be measured on identical inputs — a real gap the authors correctly identify (Section 2(3)).
  • Honest negative results, reported at instance level. Table 4's "search hurts" bucket (28 items, 11.1%) and Table 6's placeholder condition are the kind of findings most benchmark papers suppress. The "Search hurts" group even makes more searches (7.11) than "Search helps" (6.91), which the authors explicitly refuse to over-interpret ("query volume cannot prove failure origins") — good epistemics.
  • A well-designed image ablation with a control. Using both image-critical and text-answerable probes under three matched conditions, with bootstrap CIs, is the strongest experiment in the paper. The gain is positive across all 7 intents (+66.7 to +85.7, Table 7), and the internal arithmetic is consistent (probe counts sum to 367; intent cells sum to 367).
  • Arithmetic is verifiable and correct. I checked every aggregate the paper reports: Table 4's buckets reproduce the 35.7%/37.7% row accuracies, Table 5's columns sum to 129/28/157, Figure 5's percentages sum to 100.01, and the IVR-12 weighted formula reproduces both headline scores.
Critical weaknesses
  • A single model occupies every evaluator role.** Qwen3.5-397B-A17B writes the interleaved reference drafts (Stage 4), runs AgenticInterleave, judges short-answer correctness, scores IVR-12, and labels the Table 5 failures. It is also the baseline model whose +2.0 gain is the paper's headline, so that row is self-evaluated. No human-vs-judge agreement study, no second judge family, and no inter-annotator agreement for the PhD audit is reported anywhere — the accept/minor-fix/reject distribution, annotator count, and rejection count are all absent.
  • No human ceiling for the accuracy number. 52.8% is the paper's central claim about difficulty, but no number is reported for expert annotators, community respondents, or original post commenters on the short-answer track. Without it, 52.8% is uninterpretable: it is equally consistent with "VLMs are weak here" and "these community questions are hard for humans too." Note also that difficulty was defined by consistency under Qwen3.5-397B-A17B (Section 3.1, Step 3), so the benchmark is filtered toward one model's failures.
  • The interleaved evaluation has n=1 system and no component ablation. Table 3 compares exactly one output (AgenticInterleave + Qwen3.5-397B-A17B) against gold. No frontier model (Gemini, Claude) produces interleaved output, and there is no plain-text baseline, no images-without-ReAct baseline, and no ablation of the SUBJECT/BACKGROUND placement policy or image-count budget. It is impossible to attribute the 3.52 to the model, the loop, the image policy, or the 1.5-images-per-item budget.
  • The §2 retrieval claim contradicts Figure 2. Section 2(2) states agentic retrieval "lifts accuracy 14–43 points across ten testbeds," but the values rendered in Figure 2 include clear drops (77.0 vs 42.1, 65.5 vs 34.8, 77.5 vs 72.2). Figure 2 is also effectively unparseable as rendered — 11 numeric labels for ~10 benchmarks × 2 settings — so a reader cannot verify the claim. As written, the claim is false.
  • Reporting inconsistencies and missing uncertainty. Figure 5's 12 categories (Household Pets, …, Supplies, Electronics, Furnishing, Others) do not map onto Table 2's 12 (Pet, Exo, Pln, Hlt, Fas, Jwl, App=Appliance/Kitchen/Car, …, CE, Hom, Oth), with no mapping given. Table 2's smallest cells hold ~7 items yet report one decimal (e.g., Claude-Sonnet-4.6 at 57.1% on "Oth" is 4/7), and no CIs accompany Tables 2 or 3. Section 1 also contains an unresolved citation marker "[ ? ]", and §3.3's prose enumerates "five content dimensions" while listing six items and "three presentation dimensions" while listing four.
  • Single retrieval backend confounds the core negative claim. Both reference construction and AgenticInterleave run on Serper. The reported useful-evidence hit rate of 0.12% across 1623 web searches (≈2 useful results total) is very hard to reconcile with 33 items flipped to correct, and I would ask the authors to verify this figure. Since search coverage of niche UGC/vertical communities is named as a bottleneck in §4.2 but never tested against a second backend, "NoteVQA is retrieval-resistant" is not yet separable from "Serper cannot index this content."
Questions for authors
  1. What is the human ceiling on NoteVQA short-answer accuracy (expert annotators and/or original commenters), and how were these questions distributed in difficulty relative to human respondents? Given that difficulty was probed using Qwen3.5-397B-A17B, how much does the model ranking in Table 2 depend on that choice?
  2. What is the agreement (κ or equivalent) between human annotators and the Qwen3.5-397B-A17B judge on SimpleQA-style correctness and on each IVR-12 dimension? How many items were labeled accept / minor fix / reject in the Stage 4 audit, and how many annotators worked on them?
  3. Can you show AgenticInterleave ablations — plain text-only answer, images without the ReAct loop, without the SUBJECT/BACKGROUND protocol, and without forward image search — so the 3.52 can be attributed to a component? Would a stronger frontier model as the interleaved backbone close the content gap?
  4. How do you reconcile "14–43 point lifts across ten testbeds" (Section 2(2)) with the decreases visible in Figure 2, and is the marginal 0.12% useful-evidence hit rate over 1623 searches the intended unit?
  5. The placeholder condition already lifts image-critical accuracy 20.7% → 31.3%. How much of the remaining 31.3% → 97.0% jump is genuine image content versus the positioning signal that "[image N]" tags give the reader, and could this be tested with content-free but image-like stimuli?
Reviewer_SSeJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NoteVQA, a novel VQA benchmark containing 252 real-world, photo grounded questions sourced from online user generated content on an image-sharing community. Rather than using a synthesized prompts, NoteVQA captures authentic human visual info needs across 12 categories. This benchmark provides two evaluation tracks: short answer accuracy and interleaved multimodal generation. To support the interleaved track, the authors propose 1. AgenticInterleave which is a single-agent ReAct framework that coordinates web-search, page-visit, and forward/reverse image-search tools to synthesize answers embedded with traceable source-image tags; 2. IVR-12 which is a 12-dimensional evaluation rubric spanning Content, Presentation, and Image Quality)

Critical strengths
  1. Highly authentic and demanding evaluation setting.
    Different from other benchmarks that come from synthetic, research templated datasets, this benchmark leverages real and community driven visual inquiries. These represent real life scenarios that any model must face in the production phase.
  2. High-Quality, Audited Curation Pipeline.
    The curation pipeline is rigorous. It filters from an initial pool of 5.24 million notes using vision necessity filters, metrics, probes and human annotators to manually audit the items. This ensures a high quality.
Critical weaknesses
  1. Small dataset scale (252 items).
    The benchmark's size is relatively small for modern benchmark standards. With only ~20 items per category on average, even a minor performance fluctuations during VLM evaluation could lead to high variance.
  2. Single backbone for the evaluation suite.
    For the AgenticInterleave generator, IVR-12 judge, and short-answer correctness judge, this paper only evaluates the Qwen3.5-397B-A17B. Using the same model family as both the generator and the evaluator introduces potential self evaluation bias.
  3. Potential translation bias.
    The data curation pipeline notes that raw notes were translated into English and cross checked. Since the source community are likely East Asian (like RedNote), translating into English might introduce cultural mismatch during Western centric web retrieval.
Questions for authors
  1. Do you plan to scale up this benchmark?
  2. Given the NoteVQA is highly dynamic, how do you prevent it from date contamination as future frontier models are trained on more recent web crawls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.