acceptodds
Under review as a conference paper at ICLR 2027

VisAtom: Do Multimodal Models Truly See Before They Think?

Abstract

Multimodal large language models (MLLMs) have achieved impressive scores on established reasoning benchmarks, yet recent evidence shows that many such benchmarks can be solved through textual shortcuts rather than genuine visual interpretation. This gap is critical because answer accuracy alone can overestimate capabilities and reward models that fail when visual conditions change. We introduce **VISATOM**, a methodology and benchmark for evaluating visual perception, reasoning with reduced textual cues, and generalization to modified problems. Starting from existing image-based problems, we extract *atomic facts*: elemental, directly observable statements describing content supported by an image, and use them to construct and check diagnostic tasks. These facts connect three evaluation tiers: **Perception Probing** tests basic visual observation, **Pruned Problems** remove image-redundant text, and **Edited Variants** programmatically alter problem instances to test generalization beyond public benchmark examples. Sourced from 2,711 challenging problems across 12 existing benchmarks spanning 8 domains, VISATOM provides a unified, multi-granularity diagnostic system on top of existing community benchmarks. Across ten frontier and open-weight MLLMs, VISATOM reveals persistent weaknesses in visual estimation (81–92%) and universal degradation after text pruning (4–11%). Edited variants further expose a 12.4% generalization gap for the strongest model (GPT-5.4), suggesting that strong reasoning performance does not always transfer to new problem variants.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language model, multimodal reasoning, benchmark, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GKx6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

VisAtom is a benchmark and construction methodology for diagnosing whether multimodal LLMs genuinely use visual information. It extracts “atomic facts” (entities, topology, visual estimation) from images of existing hard problems drawn from 12 benchmarks, and uses them to build three tiers: (1) perception-probing questions, (2) pruned problems with image-redundant text removed, and (3) human-verified edited variants. Evaluating ten MLLMs, the authors find that visual estimation is the weakest perception skill, that pruning lowers accuracy for every model, and that the strongest model (GPT-5.4) shows a notable drop on edited variants.

Critical strengths

Careful quality control and reasonably broad coverage. The paper uses multi-model verification (unanimous agreement for Tier 1, recovery and coverage checks for Tier 2, an auditor and verifier loop for Tier 3), followed by human review. Tier 3 also includes a difficulty-preservation check (75.3% of variants keep the original difficulty). The benchmark spans 12 sources and 8 domains with 10 models, and the modality ablation (text only, captions, image, recovered question) gives some mechanistic support for the shortcut interpretation.

Critical weaknesses

The findings largely confirm what the community already knows, and offer little new insight. The headline conclusions are: models exploit textual shortcuts, visual estimation is weak, and performance drops on perturbed problem variants. The first two were already established by prior work the paper itself cites (MMMU-Pro, MathVerse, EMMA and related studies on text-only solvability and caption substitution). The paper positions itself as a “complementary” construction workflow rather than as a source of new understanding, and the analysis stays at the level of reporting accuracy deltas. It does not explain why models fail, for example by examining which kinds of shortcuts or visual-reasoning errors drive the drops.

No path from diagnosis to improvement. The benchmark is not used to do anything beyond evaluation. There is no post-training, no test-time method, no reward-design experiment showing that the atomic-fact signal can be used to make models perceive or generalize better. Without this, the contribution is a benchmark and a set of observations. That is a thin result for ICLR, where benchmark papers are generally expected either to reveal something surprising about model behavior or to enable measurable progress.

Questions for authors

Please refer to the weakness section.

4 comments

Thank you for recognizing the careful quality control, broad coverage across sources, domains, and models, and modality ablations that support the textual-shortcut interpretation. We address your concerns about diagnostic insight and the path from diagnosis to improvement below.

W1. New insight beyond familiar phenomena

VisAtom contributes a reusable construction methodology, controlled findings about the use of visual information, and a diagnosis of a specific visual-reasoning failure.

1. Reusable unified workflow. The same image-grounded atomic facts turn existing problems into source-linked perception probes, text-pruned reasoning problems, and verified edited variants. This shared representation provides control over individual conditions and supports joint diagnosis across the three tiers; the workflow can also be reused on future datasets (§1 of the revised paper).

2. Findings beyond the headline limitations. MMMU-Pro filters text-solvable questions, MathVerse varies textual and visual information across six versions, and EMMA targets integrated multimodal reasoning. By contrast, VisAtom connects local perception measurements with controlled pruning and edits of the same sources. The original submission showed 4–11-point pruning drops across all ten models (§4.2) and a steeper perception–reasoning relationship after pruning (Figure 3, §4.3.1). These linked comparisons characterize how textual support compensates for perception weaknesses and whether success survives changes to a problem’s givens or target.

3. A specific failure mechanism and diagnostic tool. Our fine-grained analysis (see W2 in our response to Reviewer ouiL) examines Tier 3 cases where a model solves the original but fails on the edited problem. We separate these failures into visual misreading, reasoning errors, no final answer, and other causes, revealing different dominant causes across models. The figure-misreading analysis traces errors to a specific mechanism: treating approximate visual cues as exact premises despite conflicting givens.

To quantify this mechanism, we use the atomic-fact inventory and the solution’s fact chain to construct verified trap/twin pairs: the trap invalidates the visually suggested premise while retaining the misleading appearance; the twin restores that premise with consistent givens. On 33 human-verified pairs, the four evaluated models show 59.5–75.6-point twin–trap gaps. Together, the case analysis and paired results connect an observed error to verified edits that test its mechanism. Our response to Reviewer ouiL W2 and Appendix Figures 7–8 provide the diagnosis and construction details.

W2. From diagnosis to useful interventions

The same atomic facts used for diagnosis can also help models recover visual context during problem solving. Figure 4 and §4.3.2 in our paper already examine this as a simple test-time method that offloads part of the perception burden to atomic-fact extraction and problem text supplementation: use the image and text-informed atomic-fact inventory to recover visual context in a pruned question, then give the recovered question and the same image to the answering model. On the 200-item subset evaluated with four models, adding atomic-fact captions to pruned text + image improves average accuracy by 4.4 points; using recovered question + image improves it from 68.0% to 75.5%.

The village-map example in Figure 6 (Appendix E.7) illustrates this mechanism. Kimi K2.5 incorrectly chooses C with the pruned question and image. The recovered statement makes the four straight streets, four circular streets, and eleven existing houses explicit. With that statement and the same image, Kimi identifies the missing third house on the northwest–southeast street and correctly chooses D, illustrating how explicit visual context helps it apply the relevant constraint.

Dear Reviewer GKx6,

Thank you again for your constructive feedback and for recognizing our quality control, broad evaluation coverage, and modality analyses. We wanted to gently follow up on our posted rebuttal, which addresses your concerns about new diagnostic insight and the path from diagnosis to improvement. Our expanded failure analysis identifies models’ use of approximate visual cues as exact premises, and the 33 human-verified trap/twin pairs quantify this mechanism through 59.5–75.6-point accuracy gaps across four models.

We also clarified how the atomic-fact recovery experiment in the original submission serves as a test-time method: supplying recovered visual context improves average accuracy from 68.0% to 75.5% on the 200-item comparison. The revised paper makes this practical use explicit and includes a worked example, with changes highlighted in blue.

We would be very grateful if you could let us know whether these results and clarifications resolve your concerns, or if there are further points you would like us to address.

Best regards,

Authors of Submission 35787

Reviewer_GKx6OpenReview ↗

Official Comment By Reviewer

Thank you for the detailed rebuttal and for the follow-up message.

I have read the revised paper, including Figure 4, §4.3.2, and the village-map example in Figure 6. These are interesting additions, but I continue to have reservations. The atomic-fact recovery experiment (68.0% → 75.5%) and the trap/twin gap analysis are presented as evidence of a "path from diagnosis to improvement," but I view them as preliminary supporting experiments rather than technical or methodological contributions in their own right. The recovery method is a straightforward prompting intervention, rather than a novel technique, and the trap/twin pairs serve primarily to quantify an already-identified failure mode rather than to enable or demonstrate an improvement method.

I don't doubt the benchmark or the empirical findings. But the gap between diagnosis and actionable improvement remains. I will keep my current rating of 2: Weak Rejection.

AuthorsOpenReview ↗

Response to GKx6 — follow-up

Thank you for reading the revised paper and for clarifying your remaining reservations. We appreciate your confidence in the benchmark and the empirical findings. Your original review noted that benchmark papers are expected either to reveal something surprising about model behavior or to enable measurable progress. We address both briefly below; full details are in our earlier response and the revised paper.

1. A specific failure pattern, identified and then tested. Your follow-up notes that the trap/twin pairs quantify an already-identified failure mode. Identifying it was itself a result of our analysis (Table R4): models treat approximate visual cues as exact premises, even when the stated givens contradict them. The trap/twin pairs then test this diagnosis under controlled changes: on 33 human-verified pairs, all four evaluated models show twin–trap gaps of 59.5–75.6 points (Table R5; Appendix Figures 7–8). The analysis and the pairs turn an observed error pattern into a controlled, reusable diagnostic test. This addresses your original request to identify which visual-reasoning errors drive the observed drops. It also identifies a concrete development target: premise verification, i.e., checking whether a visually suggested relation is justified by the givens.

2. The methodological contribution: one representation for linked evaluation. VisAtom contributes a reusable methodology for benchmark construction and model diagnosis, rather than a model-training algorithm. The same image-grounded atomic facts turn heterogeneous existing datasets into linked perception probes, text-pruned reasoning problems, and verified edited variants. This shared representation supports joint analysis of perception and reasoning under controlled changes to the same source problems, including the diagnosis above (§1 and §4). The construction workflow can also be applied to future datasets.

3. Demonstrated utility, kept separate from future work. We agree that atomic-fact recovery is straightforward; the framework's methodological contribution does not depend on recovery being a standalone algorithm. What recovery shows is that the representation built for evaluation also helps during problem solving: it raises average accuracy from 68.0% to 75.5% on a 200-item subset across four models (§4.3.2, Figure 4). The trap/twin construction also yields verified contrastive examples that could support targeted training. We do not claim training gains from these examples, or that recovery resolves the trap/twin failure.

4. Where this contribution sits. VisAtom shares the diagnostic aim of DynaMath (ICLR 2025) [1] and GeoBench (ICLR 2026) [2], which build systematic evaluations to expose specific weaknesses in existing models. DynaMath, for instance, leaves training-based improvements to future work.

We appreciate your careful engagement throughout the discussion. VisAtom combines a reusable construction methodology, a controlled diagnostic test of a specific failure pattern, and demonstrated test-time utility. We hope these contributions can be considered together in the overall assessment.

References

[1] Zou et al. DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. ICLR 2025.

[2] Feng et al. GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation. ICLR 2026.

Reviewer_ouiLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the VISATOM benchmark, which consists of 2711 problems from 12 benchmarks. It extracts atomic facts from the problems and evaluates the models for diagnostic tasks. It provides three stages of evaluation: perception probing for visual understanding, pruned problems for eliminating extra text, and edited variants for generalization testing. The results show that visual estimation is the main weakness among the models, and text pruning affects performance by a 4- 11% decrease. Generalization among the reasoning problems shows a 12% gap for the best-performing model.

Critical strengths
  • The deduplication process removes similar questions from the dataset.
  • Filtering easy problems to focus on more challenging questions.
  • The granularity of the diagnosis evaluates perception, reasoning, and generalization capabilities.
  • The creation of the generalization task
Critical weaknesses
  • The study shows a correlation between perception and reasoning problems, the informative text pruning problem, and the generalization problem; however, they have already been covered by some papers, seperately. The contribution of the paper is a combination of them, but nothing more.
  • The evaluations are based on original scores vs. created tasks, but not on diagnostic analysis of why the model fails. It only provides overall scores but no underlying causes. So, I found the study limited in its discoveries.
Questions for authors
  • As the VisAtom was created using problems from the existing benchmarks, what did the authors do to prevent the data leakage problem?

  • Although the Related Work section includes reasoning benchmarks and shortcuts in multimodal evaluation, similar benchmarks aiming to extract atomic facts from previous benchmarks are not mentioned. The authors should explain how their methodology differs from other benchmarks' methods for evaluating the underlying features. Some benchmarks are PerceptionBench, Atomic Visual Entailment (AVE), and AVA-Bench.

  • As I understand the authors used GPT-5.4 for extracting atomic fact list from images and used Gemini 3 and GPT-5.4 for checking the list. How are these models trustworthy for visual data extraction, as they already show flaws during perception tests? Additional human review for verification or error analysis of portions of these extracted data is required for the validation of the fact lists.

  • For Tier 2 performances, can authors analyze whether the wrong answers were reviewed by humans or passed from the verification stage without human review?

  • How do authors evaluate the generalization problem? As I understand they just run full test without looking at specific questions which can't explain the generalization ability as models can answer wrong to the original question but correct to the edited question. Besides, the calculation includes wrong original answers, too. How can the models transfer knowledge if they do not answer the original question accurately? The generalization score should be calculated separately for each question. Can the authors compute the score where the models only answer the original question correctly and then check whether they can solve the edited questions created for this problem? Otherwise, the overall score does not provide any meaningful result regarding to the generalization score.

  • In Table 3 the authors evaluate the perception task separate than the original task. I suggest comparing the original task score with the perception score for each problem to understand the contribution of the perception to the task and diagnose whether the model failures are caused by perception or reasoning steps.

8 comments

Thank you for recognizing the deduplication and difficulty filtering, the diagnostic coverage of perception, reasoning, and generalization, and the construction of generalization tasks. We address your concerns and questions below.

W1. Contribution beyond combining known observations

VisAtom’s contribution has three complementary aspects.

1. Reusable benchmark construction. VisAtom consolidates existing datasets through a shared, image-grounded atomic-fact representation that supports perception probes, selective text pruning, and verified problem edits. It provides control over which textual or visual conditions vary within a question while retaining source correspondences, and the workflow can be reused on future datasets.

2. Findings from controlled, matched comparisons. The pruning results expose dependence on overlapping text even when the relevant visual information remains available. The generalization results show that success on an original problem does not consistently transfer to a verified variant with changed givens or targets. These comparisons characterize sensitivity to specific changes within individual source problems, beyond a broad perception–reasoning correlation or an aggregate generalization score.

3. A diagnostic tool enabled by the workflow. Our unified framework uses atomic facts to trace an observed error to a specific unsupported visual premise, then construct verified trap/twin comparisons that probe the same failure mechanism. This links qualitative diagnosis to quantitative testing within the same workflow. The more detailed figure-misreading analysis in W2 below illustrates this connection through the source example, the failure breakdown (Table R4), and the 33-pair comparison (Table R5).

We have clarified this contribution in the revised Introduction (§1).

W2. Diagnose why models fail

Tier 3 failures have distinct causes across models, including a recurring failure to check visually suggested premises. Table R4 below (Table 4 in §4.3.3 of the revised paper) breaks down original-correct → edited-wrong outcomes: figure misreading dominates GPT-5.4’s losses; output cutoffs are the largest category for Opus and Sonnet; Qwen’s losses mainly split between figure misreading and reasoning errors.

Table R4. Causes of original-correct → edited-wrong outcomes

ModelFigure misreadReasoning errorNo final answerOtherTotal losses
GPT-5.42640838
Claude Opus 4.620828460
Claude Sonnet 4.613429349
Qwen3-VL-235B18200442

A specific figure-misreading failure is treating an approximate visual cue as an exact premise despite conflicting givens. In the chemistry example (Figure 7 in Appendix E.7 of the revised paper), assuming exact half-neutralization leads to the incorrect ABC answer; the stated conditions instead require AC, exposing the unsupported coordinate assumption.

The atomic-fact workflow makes this mechanism testable: identify the unsupported premise, construct a trap that invalidates it, and build an internally consistent twin that restores it while adjusting conflicting givens. The tangency trap/twin pair built by our workflow in Figure 8 (Appendix E.7) shows the same error: the diagram suggests that CD is tangent, but only the twin licenses that premise. Both answers and the pair’s consistency are then manually verified to test whether models check a theorem’s premise before applying it.

The 33 human-verified pairs yield twin–trap gaps of 59.5–75.6 points across the four models with hand-graded answers (Table R5 below; Table 5 in §4.3.4 of the revised paper). Majority grading gives 19–26 twin-correct/trap-wrong pairs and no reverse pairs. Even the latest frontier model GPT-6.1 Sol still drops from 99.4% to 50.3%. Together with the shortcut answers, these results quantify the failure to check visually suggested premises and show how our unified workflow uses atomic facts to trace an observed error and links it to verified edits that probe its mechanism.

Table R5. Trap/twin results on 33 human-verified pairs

ModelTwin (%)Trap (%)Gap (%)Discordant pairsMcNemar p
GPT-5.496.020.475.626 / 03 × 10⁻⁸
Claude Opus 4.681.814.167.723 / 02 × 10⁻⁷
Claude Sonnet 4.678.86.172.724 / 01 × 10⁻⁷
Qwen3-VL-235B64.65.159.519 / 04 × 10⁻⁶
GPT-6.1 Sol99.450.349.116 / 03 × 10⁻⁵

Q1. Leakage from existing benchmarks

Data leakage is a general concern for benchmarks but does not apply to ours: possible leakage or overfitting is itself a motivation for VisAtom’s diagnostic evaluation. Tier 3 therefore asks whether a model that succeeds on a source problem can also solve a verified variant with changed conditions or a different target quantity. The original problems provide the reference for this comparison, so possible gains from prior exposure on them are less important to this diagnostic objective than whether performance transfers to the edited instances, as clarified in §3.2.5 of the revised paper.

Since proprietary training corpora are not available to us, we cannot guarantee that the public source problems were unseen during training. Tier 3 changes the problem instances and revalidates their answers, enabling us to examine performance beyond the unchanged source problems even when prior exposure to those sources cannot be ruled out.

Q2. Relation to PerceptionBench, AVE and AVA-Bench

VisAtom differs in both the atomic representation and its use: it extracts a per-problem inventory of image-grounded facts about entities, topology, and visual estimates, then reuses it to construct linked perception, pruning, and editing tasks. The cited works use “atomic” for different representations and purposes: PerceptionBench and AVA-Bench isolate perceptual abilities, while AVE decomposes a textual hypothesis to predict entailment.

PerceptionBench [1] treats individual perceptual capabilities as atomic units: its error taxonomy from 42 benchmarks defines ten capabilities, and its questions isolate one capability at a time to diagnose MLLM perception. VisAtom’s atomic units are facts extracted for a source problem; they support perception assessment and also specify which textual conditions to prune or which problem conditions to edit.

Atomic Visual Entailment (AVE) [2] decomposes a supplied textual hypothesis into atomic claims and checks them against an image. It combines claim-level and full-hypothesis predictions to determine an entailment label for that hypothesis. VisAtom extracts an image-grounded fact inventory to construct and transform benchmark questions across three tiers, rather than to predict entailment for a supplied hypothesis.

AVA-Bench [3] defines 14 atomic visual abilities, such as counting, localization, and depth estimation. It evaluates these abilities in frozen vision encoders using a connector/LLM readout trained for each ability. VisAtom instead evaluates complete MLLMs and uses per-problem fact inventories to construct source-linked perception, pruning, and editing tasks.

We have added these distinctions to §2 of the revised paper.

Q3. Reliability of model-extracted atomic facts

To address extraction errors, we combine cross-model validation with a dedicated workflow that checks and revises facts, and apply human verification to the generated evaluation questions.

Cross-model checking. GPT-5.4 already scores over 97% on the two precise observations and scores 89.6% on the approximate visual estimation. Checking fact lists with both Gemini 3 and GPT-5.4 further allows one model’s errors to be challenged by another before consolidation. Therefore, the precise visual perception of the extraction model is already fairly reliable, while the less accurate visual estimates are marked explicitly as approximate during construction and usage.

Structured workflow. Each fact is checked for visual accuracy, and the list for completeness; omissions, errors, and unsupported inferences are then revised. These dedicated checks go beyond a model’s single, casual reading of an image while answering a question, supporting more accurate atomic facts.

Human validation. We used the extracted atomic facts to generate questions and focused human review on the resulting questions used to evaluate model performance. This review covered flagged Tier 1 and main Tier 2 items, every retained conservative Tier 2 problem, and all 283 retained Tier 3 variants.

Together, these safeguards combine error checking during construction with human validation of the evaluation questions that support our findings.

Q4. Tier 2 errors with and without human review

The main (“aggressive”) Tier 2 evaluation combines automatically accepted and human-reviewed items; the conservative evaluation contains only human-reviewed items. Reviewers check the pruned statement, image, solvability and reference answer, correcting or rejecting construction errors before retention. Appendix C’s Table 6 and conservative-review-pool paragraph report the workflow statistics summarized below.

Table R6. Tier 2 construction and human-review statistics (Appendix C)

Tier 2 modeAuto acceptedReview poolAccepted from reviewNot retainedEvaluated
Aggressive (main)1,8038124243881,035
Conservative—885387498387

On this fully human-reviewed conservative set, accuracy remains below original accuracy for 9/10 models on both matched sets (see our response to Reviewer EmPv, Q2). This consistent finding supports our conclusion that models benefit from textual restatements of visual information even when every evaluated item has been reviewed for construction errors.

Q5. Conditional generalization and paired outcomes

Table R7 reports accuracy on edited problems only when the same model solves the original, following the reviewer’s suggestion. We manually graded the four models’ saved answers to exclude LLM-judge errors. Conditional accuracy is 60.4–85.7%, a 14.3–39.6-point drop from the 100% original baseline, showing substantial transfer failures across all four models.

Table R7. Accuracy on edited problems when the original is solved

ModelConditional accuracy (%)
GPT-5.485.7
Claude Opus 4.667.9
Claude Sonnet 4.671.8
Qwen3-VL-235B60.4

W2’s fine-grained breakdown (Table R4) analyzes these same saved-result transfer failures.

The original submission reports the full set of verified edits to evaluate the construction protocol systematically across both losses from pass→fail and gains from fail→pass. This preserves a complete comparison under the same editing rules. Success on an edited instance whose original was missed can arise from model randomness, a changed reasoning path, or changed difficulty. Conditional accuracy and the failure breakdown are reported in revised-paper Table 4 (§4.3.3).

Reviewer_EmPvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VisAtom, a diagnostic benchmark for evaluating visual perception, reliance on textual shortcuts, and generalization in multimodal language models. Using image-grounded atomic facts, it constructs perception probes, text-pruned problems, and edited variants from 2,711 problems across 12 benchmarks. Experiments on ten models reveal persistent visual estimation weaknesses, consistent performance drops after text pruning, and varying robustness to edited problems, highlighting limitations obscured by conventional answer accuracy.

Critical strengths
  1. The atomic-fact framework coherently links perception, text pruning, and instance editing, enabling more informative diagnosis than final-answer accuracy alone.
  2. Paired comparisons across ten models provide useful evidence of textual shortcut reliance. Modality ablations and conservative pruning experiments further strengthen this finding.
Critical weaknesses
  1. Tier 3 changes both problem content and rendering, making the causes of performance changes difficult to isolate. The difficulty breakdown helps, but a redraw-only control would better separate rendering effects from generalization failures.
  2. The main pruning policy permits removing conditions supported by diagram conventions, introducing potential ambiguity about information preservation. Conservative pruning partly addresses this concern, but its evaluation covers only four models.
Questions for authors
  1. Could you add a redraw-only control that preserves the original problem semantics to separate rendering effects from generalization failures in Tier 3?

  2. How do you ensure that pruning convention-dependent conditions preserves solvability and answer validity? Could conservative pruning be evaluated across all ten models?

3 comments

Thank you for recognizing the value of linking perception, text pruning, and edited instances through atomic facts, and the evidence for textual shortcuts from paired comparisons across ten models, modality ablations, and conservative pruning. We address your concerns about rendering effects and information preservation below.

Q1 / W1. Separate rendering effects from problem edits

Our redraw-only control separates rendering effects from changes to the problem itself. The original question text, options, givens, and reference answer are kept fixed; only the image is replaced by a programmatic reproduction. We use a set of 188 items with reproduced original diagrams and evaluate them on the original problems. Table R1 below summarizes the original and redraw-only accuracy on these 188 items (Table 14 in Appendix E.6 of the revised paper).

Table R1. Redraw control on the 188 items

ModelOriginalRedraw-only
GPT-5.489.989.4
Claude Opus 4.667.067.0
Claude Sonnet 4.659.660.1
Qwen3-VL-235B58.555.9

Original and redraw-only accuracies are comparable across all four models. This control indicates that rendering is not the cause of the reported Tier 3 performance drop, supporting the conclusion that the drop reflects failures to generalize to edited problems.

Q2 / W2. Information preservation and conservative pruning for all ten models

Information preservation. Both pruning policies retain necessary text-only givens, the target quantity, and answer options. Conservative pruning removes only explicitly encoded information or unambiguous visible topology, such as a printed value or a right-angle mark. Convention-licensed (“aggressive”) pruning additionally permits a visually suggested relation only when the question text implies that it holds; appearance alone does not license removal.

Recovery and coverage checks compare the pruned problem against the image and atomic-fact inventory (Appendix E.1). Failed checks in convention-licensed mode trigger human review; every retained conservative variant is human-verified for the removal criterion, solvability, and reference answer.

All ten models. We provide the six additional model evaluations below. The first table compares original and conservative accuracy on all 387 conservative questions. The second uses the 232 questions shared by the 387-question conservative set and the 1,035-question aggressive set, enabling a matched three-way comparison. O, A, and C denote original, aggressive, and conservative, respectively.

Table R2. Conservative evaluation — 387 matched questions

ModelOriginal (%)Conservative (%)C−O (%)
GPT-5.477.0072.09−4.91
GPT-5.4 mini57.1153.49−3.62
Gemini 3.1 Pro78.8173.13−5.68
Gemini 3.1 Flash-Lite45.7437.47−8.27
Claude Opus 4.663.3159.43−3.88
Claude Sonnet 4.658.1450.90−7.24
Qwen3-VL-235B41.8637.47−4.39
Qwen3.5-397B64.8665.12+0.26
Gemma 4 31B49.1044.96−4.13
Kimi K2.557.3654.52−2.84

Table R3. Policy comparison — 232 shared questions

ModelOriginal (%)Aggressive (%)Conservative (%)C−A (%)
GPT-5.479.3159.0573.28+14.22
GPT-5.4 mini56.9047.4155.60+8.19
Gemini 3.1 Pro81.0369.8373.71+3.88
Gemini 3.1 Flash-Lite46.9832.7637.07+4.31
Claude Opus 4.667.2450.8662.50+11.64
Claude Sonnet 4.664.2249.1453.88+4.74
Qwen3-VL-235B41.8131.9039.22+7.33
Qwen3.5-397B64.6656.4766.38+9.91
Gemma 4 31B48.7139.6644.40+4.74
Kimi K2.558.1946.1254.74+8.62

Conservative accuracy remains below original accuracy for 9/10 models on both sets, with drops of 2.84–8.27 points on the 387 questions. On the 232 shared questions, all ten models perform better under conservative than aggressive pruning, by 3.88–14.22 points, since conservative pruning retains more convention-dependent textual support; the remaining gap is consistent with models benefiting from textual restatements even under the stricter removal criterion. The ten-model results are now in Appendix E.1, Tables 8–9, with the policy clarified in §3.2.4 and its interpretation in §4.2 of the revised paper.

Dear Reviewer EmPv,

Thank you again for your positive evaluation and for recognizing the value of our unified atomic-fact framework. We wanted to gently follow up on our posted rebuttal, where we addressed your concerns about rendering effects and information preservation. We added a redraw-only control on 188 items, showing comparable original and redraw-only accuracy across four models, and extended the conservative-pruning evaluation to all ten models. Under this stricter policy, accuracy remains below the original for nine of the ten models. We also clarified the pruning criteria and human verification of solvability and answer validity.

These additions are included in the revised paper, with changes highlighted in blue. We would be very grateful if you could let us know whether these results resolve your concerns, or if you have any remaining questions we can answer.

Best regards,

Authors of Submission 35787

Comments on the paper

1 comment

We thank the reviewers for their feedback and the AC and SAC for their time and consideration. We appreciate the reviewers’ recognition of our unified diagnostic design, broad evaluation coverage, and careful quality control. We have responded to each concern and question in the point-by-point responses below. Revisions are highlighted in blue in the revised paper. The additional analyses deepen the diagnostic evidence and clarify the practical utility of the same atomic-fact framework, strengthening the contribution within the paper’s original scope of benchmark construction and model diagnosis. Below, we summarize the main shared concerns and corresponding revisions.

Scope and reusable benchmark construction. VisAtom’s central contribution is a reusable methodology for benchmark construction and model diagnosis. It consolidates heterogeneous existing datasets through a shared, image-grounded atomic-fact representation. The same per-problem fact inventory supports linked perception probes, text-pruned reasoning problems, and verified edited variants, enabling control over individual problem conditions and reuse on future datasets. The methodological contribution lies in this shared representation and the linked construction and diagnosis it enables. The revised Introduction makes this contribution explicit, while §2 distinguishes our per-problem fact inventories from capability-level taxonomies and hypothesis decomposition in related work. This diagnostic emphasis is consistent with the aims of DynaMath [1] and GeoBench [2].

Findings from linked tasks. Our matched comparisons reveal how model performance changes when specific information within the same source problem is removed or altered. In the original submission, pruning overlapping text reduced accuracy by 4–11 points across all ten models while preserving the relevant visual information (§4.2). Figure 3 (§4.3.1) also shows a steeper perception–reasoning relationship after pruning, indicating that textual support can compensate for perception weaknesses. In the added conditional evaluation, the four models with manually graded responses achieve 60.4–85.7% accuracy on edited variants whose originals they answered correctly. These linked evaluations connect dependence on textual restatements, local perception, and failures to transfer original successes to controlled changes within individual source problems.

Diagnostic insight enabled by the workflow. Our shared atomic-fact representation connects qualitative diagnosis with targeted quantitative testing. The expanded analysis separates Tier-3 failures by cause and identifies a recurring pattern within figure-misreading errors: treating approximate visual cues as exact premises despite conflicting givens. Using the same atomic facts and the solution’s fact chain, we construct verified trap/twin pairs that invalidate or restore the implicated premise. The 33 human-verified pairs, drawn from eight source benchmarks, show twin–trap gaps of 59.5–75.6 percentage points for the four hand-graded models. This turns an observed error pattern into a controlled, reusable diagnostic test and identifies premise verification – checking whether a visually suggested relation is justified by the givens – as a concrete development target.

Practical utility and implications for model improvement. Figure 4 in the original submission showed that atomic-fact recovery improves average accuracy from 68.0% to 75.5% across four models on the 200-item comparison. The revised §4.3.2 clarifies this test-time application, with a worked example in Appendix Figure 6. This demonstrates that the representation used for construction and diagnosis can also support problem solving. The same framework synthesizes verified trap/twin pairs as contrastive examples that could support targeted training for premise verification. These examples provide a concrete resource for pursuing the diagnosed weakness; evaluating their effectiveness for training remains future work.

Broader implications. By turning existing source problems into linked probes and verified variants, VisAtom provides a reusable way to extend diagnostic evaluation across existing and future benchmarks without collecting new source problems from scratch. Its findings show that high accuracy on text-rich problems can mask weaknesses in using visual information and does not necessarily imply successful transfer to variants with changed givens or targets.

References

[1] Zou et al. DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. ICLR 2025.

[2] Feng et al. GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation. ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.