- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ReGap-Bench to investigate whether external visual references, including numbering, shading, and grid lines, can improve MLLMs' multi-step visual reasoning. The authors attribute such improvements to a “reference gap,” defined as the failure to maintain stable bindings between language and visual entities. Although the controlled intervention design is interesting, I have significant concerns about the novelty of the problem formulation, the generalizability of the benchmark, and the strength of the supporting evidence.
Critical strengths
- The paired intervention design is intuitive and helps isolate the effects of different visual scaffolds. Evaluating multiple models across several tasks also provides useful empirical observations about when external references help or hurt performance.
Critical weaknesses
-
Limited novelty The proposed “reference gap” substantially overlaps with existing problems in visual grounding, object tracking, and entity binding. The paper does not convincingly establish that it represents a fundamentally new failure mode rather than a reframing of known limitations.
-
Insufficient causal evidence Adding numbering, shading, or grids also changes visual saliency and provides additional information. The current experiments do not sufficiently distinguish improved reference maintenance from simply making the visual input easier to process. Stronger controls are needed to support the proposed explanation.
-
Reproducibility concerns The paper does not provide a dataset link or access to the generation and evaluation code. Since the benchmark is a central contribution, the lack of accessible resources makes independent verification and reproduction difficult.
Questions for authors
-
How does reference gap fundamentally differ from existing visual grounding and entity binding problems?
-
Can the authors demonstrate that the observed gains arise from reference structures rather than easier visual perception?
-
Will the benchmark and its evaluation code be publicly released?
2 comments
Thank you for the follow-up, and for checking the released dataset.
Novelty
Although the general phenomenon of visual marks can help MLLMs is known, our diagnostic analysis has not been conducted in prior work. We (1) systematically quantify how much of a model's error an external reference recovers, without changing the model's visual perception capability; (2) explain the reference mechanism through the theory of visual routines, and use behavioral analysis to test whether the reasoning process itself improves with references; and (3) offer a training-free solution that improves accuracy in real-world scenarios such as map and chart understanding. Our contribution is therefore not only the benchmark, but a reusable diagnostic framework and a plug-and-play visual skill that can benefit the community.
To our knowledge, the following results have not been reported before:
- The model's own reference system matters. Models that already build a reference system internally gain less: on shape counting, Gemini numbers the shapes by itself in 52.0% of its plain responses and gains +6.0, whereas GPT uses a systematic reference in only 15.5% and gains +17.5. A supplied reference also helps only when the model uses it: on transformation prediction, Gemini measures with the grid in 51% of its responses and gains +5.5, whereas GPT never does and gains +2.0.
- The benefit of the reference depends on the visual routines a task requires. The benefit comes from supporting the visual routine a task requires, not from adding marks. Identical "(•)" marks placed at the positions of the numbering labels give no gain (mean −3.2, vs. +11.3 for numbering), and mismatched references give little or negative gain on average: the same grid lines help size ranking (+4.5) and transformation prediction (+3.9) but hurt shape counting (−4.1).
If there is prior work that reports these results, we would be grateful if you provide the specific reference and we will discuss it in the paper.
Templates
Template-based questions over generated scenes are standard for diagnostic benchmarks (e.g., CLEVR [1], BlindTest [2]), and our diagnostic design requires them. The variation lies in the scenes. In shape counting, each scene has 5 to 47 shapes from 11 classes at random sizes and orientations; angle ranking includes acute, obtuse, and reflex angles that open clockwise or anticlockwise; and transformation prediction chains three steps from four transformation types (translation, reflection, rotation, and dilation), with various distractor options.
More importantly, the findings carry over beyond our templates: the visual skill built on our synthetic tasks improves accuracy on two real-world benchmarks with real maps and charts, CartoMapQA (Table 2) and ChartQAPro (71.6 → 78.4, App D).
References: [1] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR 2017.
[2] Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision Language Models Are Blind. ACCV 2024.
The three points raised in the response—isolation of the effect of visual references, comparison of which references help which tasks, and implementation as a training-free visual skill—do not constitute a substantive novel contribution. As for the dataset, the dataset contains many examples, but its templates are highly repetitive.