- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
VisAtom is a benchmark and construction methodology for diagnosing whether multimodal LLMs genuinely use visual information. It extracts “atomic facts” (entities, topology, visual estimation) from images of existing hard problems drawn from 12 benchmarks, and uses them to build three tiers: (1) perception-probing questions, (2) pruned problems with image-redundant text removed, and (3) human-verified edited variants. Evaluating ten MLLMs, the authors find that visual estimation is the weakest perception skill, that pruning lowers accuracy for every model, and that the strongest model (GPT-5.4) shows a notable drop on edited variants.
Critical strengths
Careful quality control and reasonably broad coverage. The paper uses multi-model verification (unanimous agreement for Tier 1, recovery and coverage checks for Tier 2, an auditor and verifier loop for Tier 3), followed by human review. Tier 3 also includes a difficulty-preservation check (75.3% of variants keep the original difficulty). The benchmark spans 12 sources and 8 domains with 10 models, and the modality ablation (text only, captions, image, recovered question) gives some mechanistic support for the shortcut interpretation.
Critical weaknesses
The findings largely confirm what the community already knows, and offer little new insight. The headline conclusions are: models exploit textual shortcuts, visual estimation is weak, and performance drops on perturbed problem variants. The first two were already established by prior work the paper itself cites (MMMU-Pro, MathVerse, EMMA and related studies on text-only solvability and caption substitution). The paper positions itself as a “complementary” construction workflow rather than as a source of new understanding, and the analysis stays at the level of reporting accuracy deltas. It does not explain why models fail, for example by examining which kinds of shortcuts or visual-reasoning errors drive the drops.
No path from diagnosis to improvement. The benchmark is not used to do anything beyond evaluation. There is no post-training, no test-time method, no reward-design experiment showing that the atomic-fact signal can be used to make models perceive or generalize better. Without this, the contribution is a benchmark and a set of observations. That is a thin result for ICLR, where benchmark papers are generally expected either to reveal something surprising about model behavior or to enable measurable progress.
Questions for authors
Please refer to the weakness section.
4 comments
Dear Reviewer GKx6,
Thank you again for your constructive feedback and for recognizing our quality control, broad evaluation coverage, and modality analyses. We wanted to gently follow up on our posted rebuttal, which addresses your concerns about new diagnostic insight and the path from diagnosis to improvement. Our expanded failure analysis identifies models’ use of approximate visual cues as exact premises, and the 33 human-verified trap/twin pairs quantify this mechanism through 59.5–75.6-point accuracy gaps across four models.
We also clarified how the atomic-fact recovery experiment in the original submission serves as a test-time method: supplying recovered visual context improves average accuracy from 68.0% to 75.5% on the 200-item comparison. The revised paper makes this practical use explicit and includes a worked example, with changes highlighted in blue.
We would be very grateful if you could let us know whether these results and clarifications resolve your concerns, or if there are further points you would like us to address.
Best regards,
Authors of Submission 35787
Official Comment By Reviewer
Thank you for the detailed rebuttal and for the follow-up message.
I have read the revised paper, including Figure 4, §4.3.2, and the village-map example in Figure 6. These are interesting additions, but I continue to have reservations. The atomic-fact recovery experiment (68.0% → 75.5%) and the trap/twin gap analysis are presented as evidence of a "path from diagnosis to improvement," but I view them as preliminary supporting experiments rather than technical or methodological contributions in their own right. The recovery method is a straightforward prompting intervention, rather than a novel technique, and the trap/twin pairs serve primarily to quantify an already-identified failure mode rather than to enable or demonstrate an improvement method.
I don't doubt the benchmark or the empirical findings. But the gap between diagnosis and actionable improvement remains. I will keep my current rating of 2: Weak Rejection.
Response to GKx6 — follow-up
Thank you for reading the revised paper and for clarifying your remaining reservations. We appreciate your confidence in the benchmark and the empirical findings. Your original review noted that benchmark papers are expected either to reveal something surprising about model behavior or to enable measurable progress. We address both briefly below; full details are in our earlier response and the revised paper.
1. A specific failure pattern, identified and then tested. Your follow-up notes that the trap/twin pairs quantify an already-identified failure mode. Identifying it was itself a result of our analysis (Table R4): models treat approximate visual cues as exact premises, even when the stated givens contradict them. The trap/twin pairs then test this diagnosis under controlled changes: on 33 human-verified pairs, all four evaluated models show twin–trap gaps of 59.5–75.6 points (Table R5; Appendix Figures 7–8). The analysis and the pairs turn an observed error pattern into a controlled, reusable diagnostic test. This addresses your original request to identify which visual-reasoning errors drive the observed drops. It also identifies a concrete development target: premise verification, i.e., checking whether a visually suggested relation is justified by the givens.
2. The methodological contribution: one representation for linked evaluation. VisAtom contributes a reusable methodology for benchmark construction and model diagnosis, rather than a model-training algorithm. The same image-grounded atomic facts turn heterogeneous existing datasets into linked perception probes, text-pruned reasoning problems, and verified edited variants. This shared representation supports joint analysis of perception and reasoning under controlled changes to the same source problems, including the diagnosis above (§1 and §4). The construction workflow can also be applied to future datasets.
3. Demonstrated utility, kept separate from future work. We agree that atomic-fact recovery is straightforward; the framework's methodological contribution does not depend on recovery being a standalone algorithm. What recovery shows is that the representation built for evaluation also helps during problem solving: it raises average accuracy from 68.0% to 75.5% on a 200-item subset across four models (§4.3.2, Figure 4). The trap/twin construction also yields verified contrastive examples that could support targeted training. We do not claim training gains from these examples, or that recovery resolves the trap/twin failure.
4. Where this contribution sits. VisAtom shares the diagnostic aim of DynaMath (ICLR 2025) [1] and GeoBench (ICLR 2026) [2], which build systematic evaluations to expose specific weaknesses in existing models. DynaMath, for instance, leaves training-based improvements to future work.
We appreciate your careful engagement throughout the discussion. VisAtom combines a reusable construction methodology, a controlled diagnostic test of a specific failure pattern, and demonstrated test-time utility. We hope these contributions can be considered together in the overall assessment.
Thank you for recognizing the careful quality control, broad coverage across sources, domains, and models, and modality ablations that support the textual-shortcut interpretation. We address your concerns about diagnostic insight and the path from diagnosis to improvement below.
W1. New insight beyond familiar phenomena
VisAtom contributes a reusable construction methodology, controlled findings about the use of visual information, and a diagnosis of a specific visual-reasoning failure.
1. Reusable unified workflow. The same image-grounded atomic facts turn existing problems into source-linked perception probes, text-pruned reasoning problems, and verified edited variants. This shared representation provides control over individual conditions and supports joint diagnosis across the three tiers; the workflow can also be reused on future datasets (§1 of the revised paper).
2. Findings beyond the headline limitations. MMMU-Pro filters text-solvable questions, MathVerse varies textual and visual information across six versions, and EMMA targets integrated multimodal reasoning. By contrast, VisAtom connects local perception measurements with controlled pruning and edits of the same sources. The original submission showed 4–11-point pruning drops across all ten models (§4.2) and a steeper perception–reasoning relationship after pruning (Figure 3, §4.3.1). These linked comparisons characterize how textual support compensates for perception weaknesses and whether success survives changes to a problem’s givens or target.
3. A specific failure mechanism and diagnostic tool. Our fine-grained analysis (see W2 in our response to Reviewer ouiL) examines Tier 3 cases where a model solves the original but fails on the edited problem. We separate these failures into visual misreading, reasoning errors, no final answer, and other causes, revealing different dominant causes across models. The figure-misreading analysis traces errors to a specific mechanism: treating approximate visual cues as exact premises despite conflicting givens.
To quantify this mechanism, we use the atomic-fact inventory and the solution’s fact chain to construct verified trap/twin pairs: the trap invalidates the visually suggested premise while retaining the misleading appearance; the twin restores that premise with consistent givens. On 33 human-verified pairs, the four evaluated models show 59.5–75.6-point twin–trap gaps. Together, the case analysis and paired results connect an observed error to verified edits that test its mechanism. Our response to Reviewer ouiL W2 and Appendix Figures 7–8 provide the diagnosis and construction details.
W2. From diagnosis to useful interventions
The same atomic facts used for diagnosis can also help models recover visual context during problem solving. Figure 4 and §4.3.2 in our paper already examine this as a simple test-time method that offloads part of the perception burden to atomic-fact extraction and problem text supplementation: use the image and text-informed atomic-fact inventory to recover visual context in a pruned question, then give the recovered question and the same image to the answering model. On the 200-item subset evaluated with four models, adding atomic-fact captions to pruned text + image improves average accuracy by 4.4 points; using recovered question + image improves it from 68.0% to 75.5%.
The village-map example in Figure 6 (Appendix E.7) illustrates this mechanism. Kimi K2.5 incorrectly chooses C with the pruned question and image. The recovered statement makes the four straight streets, four circular streets, and eleven existing houses explicit. With that statement and the same image, Kimi identifies the missing third house on the northwest–southeast street and correctly chooses D, illustrating how explicit visual context helps it apply the relevant constraint.