acceptodds
Under review as a conference paper at ICLR 2027

When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

Abstract

Visual question-answering systems should answer when evidence is sufficient and abstain when it is not. Evaluating this behavior requires justified labels, because removing image content can leave the answer recoverable. We introduce a benchmark spanning PlotQA charts, CLEVR rendered scenes, and GQA photographs. Each question groups original and edited images, presented independently; success requires correct answers on all supported versions and abstention on all versions labelled unanswerable. Complementing human or model validation, chart missing-information labels include an executable check: two complete charts with permitted values give different answers but identical pixels after masking. This demonstrates missing information under the specified chart rules. CLEVR and GQA labels instead follow programs operating on scene descriptions and designated edits; photographic masks are analyzed for residual location cues without the same ambiguity guarantee. Across 72,000 responses from six model configurations, the highest observed group success rates are 57.0%, 43.5%, and 33.7%, respectively. The benchmark separates failures in deciding whether to answer from failures to supply the correct answer.

Primary area:
datasets and benchmarks
Keywords:
visual question answering, vision-language models, answerability, abstention, benchmark evaluation, label verification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fkraOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for evaluating whether visual question answering models answer supported questions correctly and abstain when evidence is insufficient. It constructs 3,000 question groups across PlotQA, CLEVR, and GQA, pairing original images with controlled edits and evidence removal. For chart missing-information cases, executable witnesses demonstrate that different valid answers are compatible with identical observed pixels. Evaluation distinguishes per-view answerability decisions, complete-group decision success, and complete-group task success. Experiments across six model configurations reveal false acceptance of insufficient evidence, incorrect answers despite correct answerability decisions, and differences between per-view and grouped evaluation.

Critical strengths
  1. Clear problem formulation and rigorous chart-label validation. The paper clearly distinguishes answerability from answer correctness. Its chart ambiguity witnesses provide concrete, executable justification for missing-information labels, strengthening the evaluation beyond simply assuming that masking relevant content makes a question unanswerable.

  2. Well-motivated evaluation levels. Separating per-view decisions, complete-group decision success, and complete-group task success provides a useful account of model reliability. The accompanying state-level diagnostics distinguish unnecessary refusal, acceptance of insufficient evidence, and incorrect supported answers.

  3. Substantial and transparent evaluation. The experiments cover three visual domains and six model configurations, with detailed error decompositions, paired comparisons, and uncertainty estimates. Reporting both individual and grouped performance helps explain model differences that a single aggregate score would obscure.

Critical weaknesses
  1. Uncertain validity of photographic labels. GQA missing-information labels rely on program dependencies and bounding-box masks, which may preserve answer-relevant cues. The residual-location analysis acknowledges this limitation but does not resolve it. Consequently, some reported false acceptances may reflect remaining evidence rather than genuine failures to recognize unanswerability.

  2. Limited analysis of failure mechanisms. The results clearly describe which response obligations fail, but provide limited explanation of why. In particular, false acceptance on certified chart inputs could involve overlooked occlusion, unsupported extrapolation, or misunderstanding of the abstention criterion. Distinguishing these mechanisms would strengthen the benchmark’s diagnostic value.

Questions for authors
  1. Could alternative masking strategies, targeted human review, or a separate “potentially answerable from residual cues” category improve the GQA labels? Reporting results on a more carefully validated subset would help assess how much the current findings depend on uncertain missing-information annotations.

  2. Could you analyze a representative sample of false acceptances on certified chart inputs and test targeted controls to distinguish their causes? For example, do models overlook the missing region, extrapolate from visible trends, or misunderstand the requirement that the image must determine a unique answer?

3 comments

Thank you for requesting direct checks of photographic labels and mechanisms behind chart false acceptance.

GQA labels. We report a scoring bracket with F/S labels fixed: J = 22.3–33.7% under the original M labels and 51.2–63.5% when M accepts abstention or a reference-compatible answer, including uncertain comparisons (Section 6.2, Fig. 5b, App. I.7). Even under this permissive rule, 36.5–48.8% of groups remain incomplete. Because GQA rankings change with the rule, the decision-versus-task ranking comparison rests on certified charts.

On 200 fixed GQA groups, Qwen 3.8 Flash Next's answering rate falls from 41.5% to 29.5% when mask sides are tripled, with clipping at image boundaries, and to 1.0% on blank images. Forced-guess source agreement is 59.0% on M versus 45.5% on blank images, identifying residual predictive information. Blinded model judgments flag residual-evidence cases; subsequent source-aware checking identifies one grounding error, corrected only in the reviewed targets (App. I.1). The 183 retained GQA views yield 85.25–97.81% task accuracy across the six configurations (App. I.1). Only 14/200 M views and 3/200 complete groups pass the filter, so these per-view results characterize the selected subset; the full-cohort bracket separately measures scoring sensitivity.

Chart failure mechanisms. Section 7 and Table 5 report controls tied to the proposed explanations.

  • Overlooked location: an x-position marker does not reduce M acceptance: Qwen 8 → 10%, Molmo2-8B 46 → 46% on 50 groups.
  • Interpolation: both neighbors move while the interpolant stays fixed in 74 disjoint eligible pairs selected without consulting responses. No pair of numerical M answers matches that interpolant: Qwen 0/1, Molmo2 0/12, Gemma 26B 0/23.
  • Neighbor copying: Gemma 26B tracks the edited neighbor in 11/74 M pairs versus 2/74 F controls (+12.2 points, [4.1, 20.3]); corresponding M/F counts are 3/0 for Molmo2 and 0/1 for Qwen.
  • Instruction sensitivity: a uniqueness-and-examples prompt lowers Molmo2's acceptance from 46% to 24% ([−34, −12] points). Qwen changes from 8% to 2% ([−14, 0]). Molmo2's full-image accuracy rises from 38% to 42%; Qwen's remains 74%.
  • Reported evidence: 11/12 sampled false-acceptance explanations for each of Qwen and GLM claim to read the covered target's label.

On CLEVR, M acceptance peaks on counting questions in five of six configurations. The witnesses establish ambiguity under hidden-object semantics; the review also finds literal counting of the visible black cube (Fig. 5a, App. I.12).

Thank you for the additional validation and targeted controls. These additions address my concerns to some extent, and I will maintain my current score.

Could you clarify how much of the remaining GQA group failure under permissive scoring comes from F/S errors rather than M responses? Also, the interpolation results have very few numerical answer pairs, particularly Qwen’s 0/1. Please keep the corresponding mechanism claims limited to what these controls directly establish.

Thank you for these two clarification requests. We have made the GQA failure attribution and the scope of the chart evidence explicit in the revised manuscript.

GQA: most residual group failures involve supported states. Across six configurations, 71.8–93.9% of groups remaining incomplete under permissive M scoring fail at least one F/S obligation, including simultaneous F/S and M failures. For Qwen Flash, the 380 incomplete groups comprise 262 F/S-only, 68 simultaneous, and 50 M-only failures; 330/380 (86.8%) therefore involve F/S. Supported-state failures include wrong answers, unnecessary abstentions, and invalid outputs.

Permissive M scoring credits valid abstentions and reference-compatible or unresolved answers. The residual 36.5–48.8% measures incomplete joint performance with F/S references fixed; it is not an insufficiency-recognition error rate. The disjoint decomposition for all six configurations is in §6.2 and Appendix I.7, Table 21.

Chart controls: observed neighbor dependence with a fixed interpolant. Gemma tracks the edited neighbor in 11/74 M pairs versus 2/74 F controls, a paired difference of +12.2 percentage points (95% interval [4.1, 20.3]). Nine M tracks also satisfy exact rational equality, and 7/63 remain among pairs answered correctly on both F controls. These observations support visible-neighbor dependence in the eligible cohort.

Neighbor tracking uses all 74 eligible pairs. Interpolant compatibility uses only pairs with numerical answers on both M views: Molmo2 0/12, Gemma 26B 0/23, and Qwen 0/1. These conditional counts do not rule out interpolation use, particularly for Qwen. We limit the mechanism claim to the observed response dependence; an internal copying algorithm is not identified. §7 and Appendix I.10, Tables 25–26, state the denominators and interpretation.

The revised presentation keeps the central finding explicit: correct answerability decisions can conceal wrong supported answers, motivating evaluation of all response obligations for the same question.

Reviewer_emoqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark of 3,000 question groups, 1,000 each from PlotQA charts, CLEVR rendered scenes and GQA photographs. Each group holds one fixed question and several image versions, shown to the model one at a time: F: the original. S: an edit that keeps the answer. C: an edit that changes the answer. M: missing evidence, where the model should abstain. I: an invalid reference, charts only. A group counts as a success (J) only if every supported answer and every required abstention is correct. The authors also report a decisions-only variant (B) and per-view decision error (E), and they split errors by state. Their main conceptual contribution is label evidence. Charts: for each M label, they build an executable witness. It is a pair of admissible complete charts with different answers and identical pixels after masking. CLEVR and GQA: labels come from source programs and edits. A residual-cue sensitivity analysis covers the GQA questions about left, right, top and bottom.

Critical strengths
  1. A useful evaluation unit. Scoring a question as one unit across evidence changes, and splitting J into B and the decision-to-answer gap, shows a failure mode that per-view accuracy hides. The 96.2% → 83.5% → 57.0% decomposition is clear.
  2. A principled definition of "unanswerable" for charts. The possible-worlds formulation (Eqs. 1–2) with a machine-checked witness for every chart M view is stronger than annotator or heuristic labels. The audit shows that two intuitive rules are wrong: "any hidden value is used" and "check only min and max".
  3. Fine-grained diagnostics. Reporting supported rejection, missing-information acceptance and invalid-reference acceptance separately gives real insight. Examples are Gemma 26B (427 M failures vs 32 I failures) and GLM and Molmo (rarely refuse, rarely abstain correctly).
  4. Rigorous, transparent reporting.
    • Invalid outputs count as failures.
    • The answer parser is fully specified.
    • Intervals come from a group bootstrap, with paired comparisons.
    • Post-hoc analyses are labelled as such.
    • The numbers I cross-checked are internally consistent, for example 835 = 570 + 265, 399 − 337 = 62, and λ = 4/11.
    • The authors disclose that their first chart certificates were flawed (785 out-of-axis values in 503 groups) and fixed them.
  5. Honest about scope. The paper states that photographic ambiguity is not certified and quantifies one residual-cue concern. It also states that the guarantee holds only under the declared program, world and renderer.
Critical weaknesses
  1. The altered labels can fully be verified only on PlotQA dataset, therefore, two-thirds of the benchmark has weak or unverified label evidence.
  2. There are only a few possible edits tested. The paper should also test natural cropping, blur, counterfactual edits etc
  3. The benchmark is scoped to simpler questions, which can question its applicability.
Questions for authors
  1. Is J measuring a genuinely new phenomenon, or mostly the expected consequence of multiplying several probabilities together?
  2. How robust are the results to prompt variations and abstention instructions? Did reasoning modes, generation limits, or quantization materially affect any reported conclusions?
1 comment

Thank you for asking what J adds and how inference settings affect the findings.

What J measures. J requires all response obligations to hold for the same question. It is not the product of state marginals: observed J exceeds that product in all 18 model–source cells by 2.2–11.7 points. Permuting states across questions while preserving the marginals changes J (Section 6.3, App. I.4). Separately, including answer correctness changes a chart ranking with the question groups fixed: Gemma 4 26B A4B has more per-view decision errors than Qwen 3.8 27B (+2.54 points, [1.66, 3.46]) and lower B₅ (−12.6, [−16.3, −9.0]), yet higher J₅ (+6.6, [2.9, 10.3]). The contribution combines established joint-success scoring with controlled response obligations and verified label evidence.

Prompts, reasoning, and generation limits. Complete-question failures persist with native reasoning and longer generation budgets (Section 7, App. I.11–I.12). These follow-ups request JSON without constrained decoding; the archived Qwen chart run enforced JSON syntax.

  • On 60 fixed chart groups, Qwen 3.8 Flash Next's native reasoning at 2,048 tokens gives J₅ = 56.7%, versus 58.3% under the archived benchmark protocol (paired difference −1.7 points, [−11.7, +10.0]).
  • On 60 CLEVR groups, the corresponding scores are 45.0% and 43.3% (+1.7 points, [−8.3, +11.7]).
  • Molmo2-8B's standard-prompt scores are identical at 512 and 2,048 tokens.
  • Output compliance explains much of Qwen's standard-prompt chart difference: post-hoc JSON extraction raises nonthinking J₅ from 33.3% to 55.0% at 512 tokens and from 30.0% to 55.0% at 2,048 tokens. With native reasoning on CLEVR, the larger budget raises J from 35.0% to 45.0% as truncations fall from 42 to 8.
  • Qwen is strongly prompt-sensitive: the uniqueness-and-examples arm, with reasoning off at 512 tokens, gives J₅ = 0.0% with 153/300 invalid outputs, and 18.3% after JSON extraction, versus 55.0% for the standard prompt under the same extraction rule.
  • The uniqueness-and-examples prompt raises Molmo2's M abstention from 56.7% to 70.0% but lowers I abstention from 56.7% to 35.0%, with J₅ changing from 13.3% to 11.7%. Joint scoring exposes this trade-off.

Quantization was not separately ablated. Appendix A identifies the FP8 configuration and documents its two serving deployments.

Label evidence, edits, and question scope. The revision adds executable CLEVR witnesses and a GQA scoring bracket to the chart witnesses. The tested edits include answer-changing states, color variants, and a nested-mask ladder with side-length factors 1×, 1.5×, 2×, and 3×, plus blank images. CLEVR programs span 3–23 recorded steps; missing-state acceptance peaks on counting questions in five of six configurations (Fig. 5a). Section 8 identifies natural corruptions and multi-step chart reasoning as extensions of the same protocol.

Reviewer_WUzfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark of 3,000 question groups across PlotQA, CLEVR, and GQA, where the question is fixed and the image is edited to preserve the answer, change it, or remove the required evidence. Models are evaluated with joint success, which requires every supported answer and every required abstention within a group to be correct. Missing-information labels for charts are backed by executable witnesses. Across six models, the best joint success is 57.0%, 43.5%, and 33.7%, and many groups with fully correct answerability decisions still contain wrong answers.

Critical strengths
  • The motivation is clear, and the paper proposes a well-suited benchmark for an important problem.
  • The paper proposes an appropriate evaluation metric to assess whether models simultaneously satisfy both answering and required-abstention obligations.
  • The experimental design separating E_M and E_I is interesting and helps identify the sources of failure, providing useful insight.
Critical weaknesses
  • The evaluation covers only six open-weight models, with no closed frontier models and no human baseline.
  • The validity of scene labels is not verified. The CLEVR occluder (which appears as a black cube) and GQA masks may allow alternative interpretations or leave residual location cues, so it is unclear whether M-state failures reflect model errors or label ambiguity.
  • There is no control for appearance confounds. Since visible occlusion appears only in the M state, it is unclear whether high M accuracy reflects judgment of evidence sufficiency or simple detection of occlusion.
Questions for authors

If possible, it would be helpful to evaluate a wider range of models, including closed models.

2 comments

Thank you for distinguishing evidence sufficiency from occluder appearance. The revision addresses both directly.

Scene labels. CLEVR now has executable completion witnesses for all 1,000 groups: two hidden-attribute completions give different program answers under identical masked renderer inputs. Independent renders of 100 pairs are pixel-identical and match the released M images (Section 3, App. I.6). These witnesses establish ambiguity under hidden-object semantics. The review also identifies a literal reading: both review models judge 17/27 count-M views answerable, most often counting the black cube as a scene object (App. I.12). Section 6.1 now distinguishes these interpretations. For GQA, we report a scoring bracket with F/S labels fixed. J rises from 22.3–33.7% under the original M labels to 51.2–63.5% when M accepts abstention or a reference-compatible answer, including uncertain comparisons. Even under this permissive rule, 36.5–48.8% of groups remain incomplete. The decision-versus-task ranking comparison rests on certified charts.

On 200 fixed GQA groups, Qwen 3.8 Flash Next's answering rate falls from 41.5% to 29.5% when mask sides are tripled, with clipping at image boundaries, and to 1.0% on blank images. Forced-guess source agreement is 59.0% on M versus 45.5% on blank images, identifying residual predictive information (Section 6.2, App. I.7).

Appearance confound. Answer-preserving controls use the same CLEVR black-cube edit and GQA masks matched in width, height, color, and count. Correct-answer counts for Qwen and Molmo2-8B are respectively 38/42 → 38/42 and 39/42 → 39/42 on GQA, and 67/74 → 65/74 and 38/74 → 36/74 on CLEVR (Section 7, App. I.2). These correct answers contradict an unconditional “occlusion implies abstention” rule. On identical masked CLEVR images, switching from the original question to a visible-target question yields the correct abstain/answer pair in 5/5 cases for Qwen and 2/5 for Molmo2. Abstention follows the question's evidence, not the occluder.

Model coverage and human baseline. The performance evaluation comprises six open-weight configurations across four families. Open weights support controlled comparisons of decoding, native reasoning, and generation budgets; archived responses and scoring code support exact score replay. Appendix A documents checkpoint and deployment provenance. On the same unconstrained Qwen chart responses, strict scoring gives J₅ = 33.3%, while post-hoc extraction of a unique valid JSON object gives 55.0% (App. I.11).

Executable witnesses establish the chart and CLEVR abstention targets under the declared task semantics; blinded model reviews separately audit scene labels (App. I.1, I.9). These are label checks, not closed-model or human-performance baselines. The same protocol extends directly to further model families, as stated in Section 8.

I appreciate the authors' efforts to improve the manuscript. While the additional experiments have addressed many of my concerns, some remain regarding the scope of model evaluation and the semantic validity of the labels. Therefore, I will maintain my original score.

My concern regarding the scope of model evaluation was not about reproducibility, but rather the generalizability of the findings and the reliability of the evaluation. I believe that a broader evaluation, including performance analyses across different model scales, comparisons with closed frontier models, and human baselines, would further strengthen the paper by providing stronger evidence for the generalizability of its findings. However, as these evaluations remain insufficient, my concern has not been fully resolved.

The executable witness strengthens the structural validity of CLEVR, but semantic ambiguity regarding the interpretation of the black cube remains. Similarly, the additional analysis on GQA quantifies label uncertainty but does not fully resolve it.

I appreciate the authors' efforts in revising the manuscript. If I have missed or misunderstood any of the changes, I would be happy to reconsider these points with further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.