- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark for evaluating whether visual question answering models answer supported questions correctly and abstain when evidence is insufficient. It constructs 3,000 question groups across PlotQA, CLEVR, and GQA, pairing original images with controlled edits and evidence removal. For chart missing-information cases, executable witnesses demonstrate that different valid answers are compatible with identical observed pixels. Evaluation distinguishes per-view answerability decisions, complete-group decision success, and complete-group task success. Experiments across six model configurations reveal false acceptance of insufficient evidence, incorrect answers despite correct answerability decisions, and differences between per-view and grouped evaluation.
Critical strengths
-
Clear problem formulation and rigorous chart-label validation. The paper clearly distinguishes answerability from answer correctness. Its chart ambiguity witnesses provide concrete, executable justification for missing-information labels, strengthening the evaluation beyond simply assuming that masking relevant content makes a question unanswerable.
-
Well-motivated evaluation levels. Separating per-view decisions, complete-group decision success, and complete-group task success provides a useful account of model reliability. The accompanying state-level diagnostics distinguish unnecessary refusal, acceptance of insufficient evidence, and incorrect supported answers.
-
Substantial and transparent evaluation. The experiments cover three visual domains and six model configurations, with detailed error decompositions, paired comparisons, and uncertainty estimates. Reporting both individual and grouped performance helps explain model differences that a single aggregate score would obscure.
Critical weaknesses
-
Uncertain validity of photographic labels. GQA missing-information labels rely on program dependencies and bounding-box masks, which may preserve answer-relevant cues. The residual-location analysis acknowledges this limitation but does not resolve it. Consequently, some reported false acceptances may reflect remaining evidence rather than genuine failures to recognize unanswerability.
-
Limited analysis of failure mechanisms. The results clearly describe which response obligations fail, but provide limited explanation of why. In particular, false acceptance on certified chart inputs could involve overlooked occlusion, unsupported extrapolation, or misunderstanding of the abstention criterion. Distinguishing these mechanisms would strengthen the benchmark’s diagnostic value.
Questions for authors
-
Could alternative masking strategies, targeted human review, or a separate “potentially answerable from residual cues” category improve the GQA labels? Reporting results on a more carefully validated subset would help assess how much the current findings depend on uncertain missing-information annotations.
-
Could you analyze a representative sample of false acceptances on certified chart inputs and test targeted controls to distinguish their causes? For example, do models overlook the missing region, extrapolate from visible trends, or misunderstand the requirement that the image must determine a unique answer?
3 comments
Thank you for the additional validation and targeted controls. These additions address my concerns to some extent, and I will maintain my current score.
Could you clarify how much of the remaining GQA group failure under permissive scoring comes from F/S errors rather than M responses? Also, the interpolation results have very few numerical answer pairs, particularly Qwen’s 0/1. Please keep the corresponding mechanism claims limited to what these controls directly establish.
Thank you for these two clarification requests. We have made the GQA failure attribution and the scope of the chart evidence explicit in the revised manuscript.
GQA: most residual group failures involve supported states. Across six configurations, 71.8–93.9% of groups remaining incomplete under permissive M scoring fail at least one F/S obligation, including simultaneous F/S and M failures. For Qwen Flash, the 380 incomplete groups comprise 262 F/S-only, 68 simultaneous, and 50 M-only failures; 330/380 (86.8%) therefore involve F/S. Supported-state failures include wrong answers, unnecessary abstentions, and invalid outputs.
Permissive M scoring credits valid abstentions and reference-compatible or unresolved answers. The residual 36.5–48.8% measures incomplete joint performance with F/S references fixed; it is not an insufficiency-recognition error rate. The disjoint decomposition for all six configurations is in §6.2 and Appendix I.7, Table 21.
Chart controls: observed neighbor dependence with a fixed interpolant. Gemma tracks the edited neighbor in 11/74 M pairs versus 2/74 F controls, a paired difference of +12.2 percentage points (95% interval [4.1, 20.3]). Nine M tracks also satisfy exact rational equality, and 7/63 remain among pairs answered correctly on both F controls. These observations support visible-neighbor dependence in the eligible cohort.
Neighbor tracking uses all 74 eligible pairs. Interpolant compatibility uses only pairs with numerical answers on both M views: Molmo2 0/12, Gemma 26B 0/23, and Qwen 0/1. These conditional counts do not rule out interpolation use, particularly for Qwen. We limit the mechanism claim to the observed response dependence; an internal copying algorithm is not identified. §7 and Appendix I.10, Tables 25–26, state the denominators and interpretation.
The revised presentation keeps the central finding explicit: correct answerability decisions can conceal wrong supported answers, motivating evaluation of all response obligations for the same question.
Thank you for requesting direct checks of photographic labels and mechanisms behind chart false acceptance.
GQA labels. We report a scoring bracket with F/S labels fixed: J = 22.3–33.7% under the original M labels and 51.2–63.5% when M accepts abstention or a reference-compatible answer, including uncertain comparisons (Section 6.2, Fig. 5b, App. I.7). Even under this permissive rule, 36.5–48.8% of groups remain incomplete. Because GQA rankings change with the rule, the decision-versus-task ranking comparison rests on certified charts.
On 200 fixed GQA groups, Qwen 3.8 Flash Next's answering rate falls from 41.5% to 29.5% when mask sides are tripled, with clipping at image boundaries, and to 1.0% on blank images. Forced-guess source agreement is 59.0% on M versus 45.5% on blank images, identifying residual predictive information. Blinded model judgments flag residual-evidence cases; subsequent source-aware checking identifies one grounding error, corrected only in the reviewed targets (App. I.1). The 183 retained GQA views yield 85.25–97.81% task accuracy across the six configurations (App. I.1). Only 14/200 M views and 3/200 complete groups pass the filter, so these per-view results characterize the selected subset; the full-cohort bracket separately measures scoring sensitivity.
Chart failure mechanisms. Section 7 and Table 5 report controls tied to the proposed explanations.
On CLEVR, M acceptance peaks on counting questions in five of six configurations. The witnesses establish ambiguity under hidden-object semantics; the review also finds literal counting of the visible black cube (Fig. 5a, App. I.12).