acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning

Abstract

Multimodal large language models can recognize visual content yet struggle to reason reliably over it across multiple steps. In counting, a model may see all objects but lose track of which ones it has counted. We argue that such failures, commonly attributed to perception, reveal a reference gap: the failure to bind language to the relevant visual entities, regions, or locations. Motivated by the theory of visual routines, we test whether external references such as numbering can narrow this gap. We introduce ReGap-Bench, a paired benchmark of five visual reasoning tasks in which the scene, question, and answer are fixed and only the reference varies. Results show that task-matched references improve accuracy by up to 25%, while mismatched references provide little benefit or even hurt performance. These findings identify the reference gap as a bottleneck distinct from visual perception and show that constructing task-appropriate visual references can serve as a training-free visual skill for improving multimodal reasoning.

Primary area:
datasets and benchmarks
Keywords:
multimodal reasoning, visual reasoning, visual routines

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4cdzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ReGap-Bench to investigate whether external visual references, including numbering, shading, and grid lines, can improve MLLMs' multi-step visual reasoning. The authors attribute such improvements to a “reference gap,” defined as the failure to maintain stable bindings between language and visual entities. Although the controlled intervention design is interesting, I have significant concerns about the novelty of the problem formulation, the generalizability of the benchmark, and the strength of the supporting evidence.

Critical strengths
  1. The paired intervention design is intuitive and helps isolate the effects of different visual scaffolds. Evaluating multiple models across several tasks also provides useful empirical observations about when external references help or hurt performance.
Critical weaknesses
  1. Limited novelty The proposed “reference gap” substantially overlaps with existing problems in visual grounding, object tracking, and entity binding. The paper does not convincingly establish that it represents a fundamentally new failure mode rather than a reframing of known limitations.

  2. Insufficient causal evidence Adding numbering, shading, or grids also changes visual saliency and provides additional information. The current experiments do not sufficiently distinguish improved reference maintenance from simply making the visual input easier to process. Stronger controls are needed to support the proposed explanation.

  3. Reproducibility concerns The paper does not provide a dataset link or access to the generation and evaluation code. Since the benchmark is a central contribution, the lack of accessible resources makes independent verification and reproduction difficult.

Questions for authors
  1. How does reference gap fundamentally differ from existing visual grounding and entity binding problems?

  2. Can the authors demonstrate that the observed gains arise from reference structures rather than easier visual perception?

  3. Will the benchmark and its evaluation code be publicly released?

2 comments

The three points raised in the response—isolation of the effect of visual references, comparison of which references help which tasks, and implementation as a training-free visual skill—do not constitute a substantive novel contribution. As for the dataset, the dataset contains many examples, but its templates are highly repetitive.

Thank you for the follow-up, and for checking the released dataset.

Novelty

Although the general phenomenon of visual marks can help MLLMs is known, our diagnostic analysis has not been conducted in prior work. We (1) systematically quantify how much of a model's error an external reference recovers, without changing the model's visual perception capability; (2) explain the reference mechanism through the theory of visual routines, and use behavioral analysis to test whether the reasoning process itself improves with references; and (3) offer a training-free solution that improves accuracy in real-world scenarios such as map and chart understanding. Our contribution is therefore not only the benchmark, but a reusable diagnostic framework and a plug-and-play visual skill that can benefit the community.

To our knowledge, the following results have not been reported before:

  1. The model's own reference system matters. Models that already build a reference system internally gain less: on shape counting, Gemini numbers the shapes by itself in 52.0% of its plain responses and gains +6.0, whereas GPT uses a systematic reference in only 15.5% and gains +17.5. A supplied reference also helps only when the model uses it: on transformation prediction, Gemini measures with the grid in 51% of its responses and gains +5.5, whereas GPT never does and gains +2.0.
  2. The benefit of the reference depends on the visual routines a task requires. The benefit comes from supporting the visual routine a task requires, not from adding marks. Identical "(•)" marks placed at the positions of the numbering labels give no gain (mean −3.2, vs. +11.3 for numbering), and mismatched references give little or negative gain on average: the same grid lines help size ranking (+4.5) and transformation prediction (+3.9) but hurt shape counting (−4.1).

If there is prior work that reports these results, we would be grateful if you provide the specific reference and we will discuss it in the paper.

Templates

Template-based questions over generated scenes are standard for diagnostic benchmarks (e.g., CLEVR [1], BlindTest [2]), and our diagnostic design requires them. The variation lies in the scenes. In shape counting, each scene has 5 to 47 shapes from 11 classes at random sizes and orientations; angle ranking includes acute, obtuse, and reflex angles that open clockwise or anticlockwise; and transformation prediction chains three steps from four transformation types (translation, reflection, rotation, and dilation), with various distractor options.

More importantly, the findings carry over beyond our templates: the visual skill built on our synthetic tasks improves accuracy on two real-world benchmarks with real maps and charts, CartoMapQA (Table 2) and ChartQAPro (71.6 → 78.4, App D).

References: [1] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR 2017.

[2] Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision Language Models Are Blind. ACCV 2024.

Reviewer_FxC7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies if MLLM failures in multi-step visual reasoning come from a reference gap rather than perception. It creates ReGap-Bench with five synthetic tasks, where each scene is rendered plain and with numbering, shading, or grid lines, while the question and answer stay fixed. Task-matched references improve all five models, mismatched ones help little or hurt, and the gain stays from 2B to 235B. A training-free visual skill is also tested on CartoMapQA.

Critical strengths
  1. The paired design is clean. Only the reference changes, so its effect is isolated. The matched vs mismatched comparison and the partial numbering study make the main claim much more convincing than just showing gains.
  2. The response-level analysis is useful, e.g. how models refer to shapes, clock reading accuracy, and the conjugate angle error. It shows how the reference changes the reasoning process, not only the final accuracy.
Critical weaknesses

1.. Reference vs perception is not fully separated. For clocks, the paper itself shows dial numbers raise single clock reading accuracy (e.g. Claude 10.6% to 28.6%), so numbering there looks more like a perception aid. 2. Novelty is moderate given the previous Set-of-Mark. The CartoMapQA experiment is small (only 100 tasks, only 3 for shading).

Questions for authors

In the CartoMapQA experiment, does the no-skill agent have the same tools (e.g. can it draw on the image), so that only the skill differs?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer FxC7,

Thank you again for your comments. With the discussion deadline approaching, we would like to check whether our responses have addressed your concerns. If there are any remaining questions, we would be happy to respond.

If you find our responses satisfactory, we would appreciate your consideration in updating the evaluation.

Sincerely, Authors

Reviewer_wKnaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that failures of MLLMs in multi-step visual reasoning (e.g., double counting or omissions when counting) stem not primarily from perception but from a "reference gap": language-based reasoning fails to stay bound to the relevant visual entities or regions (Sec 2.1). Inspired by Ullman's visual routines, the authors propose three external references, namely numbering, shading, and grid lines (Sec 2.2), and build ReGap-Bench:

  • Tasks: five synthetic tasks: shape counting, angle ranking, size ranking, clock ranking, and transformation prediction.
  • Design: 200 scenes per task. Each scene is rendered under different reference conditions while the prompt and answer are held fixed (Sec 2.3, Sec 3, App B/C.1).

Main results on five MLLMs:

  • Matched references improve exact-match accuracy by +0.5 to +25.0 percentage points (Fig 3–5).
  • On counting and clock ranking, mismatched references yield small or negative gains (Fig 6).
  • On clock ranking, accuracy increases as the dial numerals become more complete (Fig 7).
  • The numbering gain persists across Qwen3-VL models from 2B to 235B (Fig 8).

Finally, the authors package the three overlays as a training-free visual skill for an agent and evaluate it on 100 CartoMapQA tasks (Table 2).

Critical strengths
  1. A clean paired-intervention design, complemented by mechanism-level behavioral analysis. All conditions are rendered from the same scene specification with identical prompts (Sec 2.3, App B, C.1). Compared with cross-model or cross-prompt comparisons, this design better isolates the effect of the image-side intervention. Beyond accuracy, the authors also analyze the following, and they honestly report negative results (grid lines reduce shape-counting accuracy by 4.1 points on average, Fig 6; GPT's wrong-side rate on angle ranking does not decrease, 13.3% → 13.8%, Table 6):

    • manual annotation of reference strategies on shape counting (Sec 4.1);
    • per-clock reading accuracy on clock ranking (Sec 4.1);
    • wrong-side error rates on angle ranking, with a random baseline (Table 6).

    Why this matters: even if the conceptual claim does not hold, the controlled benchmark and analysis framework are reusable, and this is the paper's main positive contribution. However, they support the weaker conclusion of which overlays help which tasks, not the diagnosis promised by the title.

Critical weaknesses
  1. The core claim to "identify the reference gap as a bottleneck distinct from visual perception" (Abstract) is not actually diagnosed: on most tasks, the effect of the scaffolds cannot be separated from aiding perception or adding task information.

    • Clock ranking: the so-called "numbering" prints the numerals 1–12 on the dial (App B.3); it is not an object address. The authors' own analysis shows that the gain comes with more accurate readings of individual clocks (Sec 4.1), which is at least as naturally interpreted as improved perception. For Claude and InternVL in particular, the bottleneck lies in reading individual clocks: among clocks with a stated time on plain images, only 10.6% and 2.9% of their readings are correct, rising to 28.6% and 17.4% with dial numerals (Sec 4.1), and Claude reaches only 8.0% exact match on plain clock ranking (Fig 3). The dial numerals thus fix precisely this reading step.
    • Angle ranking: shading fills the side indicated by the red arc (App B.3), and the authors identify reading the wrong side as the main difficulty of this task (Sec 3.1). The red arc already marks which side is intended; shading makes that side more salient. Such a salience boost cannot be distinguished from the mechanism the paper proposes, namely remedying "the failure to maintain stable bindings between language-based reasoning and the relevant visual entities" (Sec 1).
    • Size ranking / transformation prediction: grid lines are themselves coordinates and a ruler, which help measurement and localization; this is likewise hard to distinguish from maintaining stable bindings.
    • Ranking tasks: the plain images of all three ranking tasks already identify each object by "numerical labels in parentheses" such as (1), (2) (App C.1, Fig 2). Shape counting is the only task that truly lacks object addresses.
    • App A: the only direct test that models perceive the relevant content comes from App A, but it tests only one model (Claude) and only one task (shape counting). Moreover, the single-shape probe ("Recognizing single shapes", App A) crops one shape and asks about it in isolation, which is much easier than recognizing it within the full scene; recognizing a shape in isolation does not show that the model perceives every shape correctly in the full image.
    • Missing controls: a prompt-only control (asking the model on plain images to "number the shapes first, then count"), a higher-resolution or zoom control, and marks that carry comparable information but are not addressable.
  2. The evidence that "the gains depend on matching the reference to the task, not on adding marks" (Sec 1) is incomplete and statistically weak.

    • The controls are insufficient to support shading and grid lines as "matched" references. The mismatched controls in Sec 4.2 are run only on shape counting and clock ranking (Fig 6), and numbering is the matched reference for both. Fig 6 therefore shows at most that numbering outperforms shading and grid lines on these two tasks. For shading and grid lines, the paper compares the matched condition only against the plain condition (Fig 4, Fig 5), and against each other only on size ranking, where both are designated as matched (Sec 4.1). On their matched tasks, they are never compared with a mismatched reference: grid lines are not reported on angle ranking, and shading is not reported on transformation prediction. If grid lines on angle ranking or shading on transformation prediction yielded similar gains, the gains would only show that adding any marks helps, not that matching is what matters. So far, the claim that the gains depend on matching has been tested only for numbering.
    • The claim that mismatched references "provide little benefit or even hurt performance" (Abstract) is not uniformly supported and does not rule out visual interference. The "little benefit" part does not hold uniformly: on shape counting, shading improves Claude and GPT by 5.0 points each, close to Claude's numbering gain of 6.0 (Fig 6). The "hurt" part comes mainly from grid lines on shape counting, where GPT drops by 11.0 points; on clock ranking, grid lines average only −0.5, which is close to neutral. With light gray grid lines every 15 px overlaid on outlined shapes, the negative effect may also come from visual clutter rather than mismatch, and the paper does not separate the two.
    • Missing statistics. Each task has n = 200, with no confidence intervals and no paired tests (McNemar's test or flip counts). The exact-match gains on angle ranking of +0.5 for GPT and +1.0 for Qwen (Fig 4) correspond to 1–2 questions, yet they are counted toward the claim that task-matched references "improve all evaluated models on all five tasks" (Sec 1, contribution 2).
Questions for authors
  1. Table 4 shows 600 images each for angle ranking and transformation prediction (200 scenes × 3 conditions), and Fig 2 shows example images for angle × grid lines and transformation × shading, indicating that these two mismatched conditions were generated. Why are their results reported neither in the main text nor in the appendix? What are the results? Could you complete the full reference × task matrix to test whether shading and grid lines also achieve their largest gains only on their matched tasks?

  2. Could you provide a prompt-only control (plain images + "number the shapes row by row first, then count"), as well as higher-resolution or cropping controls? If these controls achieve similar gains, how would the "reference gap" interpretation still hold?

  3. Could you report paired tests, the numbers of 0→1 and 1→0 flips, invalid-output rates per condition (as defined in App C.3), and each model's temperature and reasoning settings (e.g., whether thinking mode is enabled for the proprietary models, and the reasoning budget)? In Table 6, only 95 of the 200 angle-ranking scenes have valid rankings for Qwen under both conditions; how much of the exact-match gain comes from changes in the rate of validly formatted outputs?

  4. App A contains two numerical inconsistencies; please confirm which value is correct:

    • The text states that "all six errors in scenes with at most 10 shapes overcount by exactly one", but Table 3 shows that the 1–10 bin is 38/38 correct and that there are 7 errors in total (1 in the 21–30 bin, 4 in the 31–40 bin, and 2 in the 41–50 bin). Should this read "more than 30 shapes" (the 31–50 bins contain exactly 6 errors)?
    • App A reports that Claude counts the target class correctly in 72.0% of plain scenes, whereas Fig 3 reports 68.0% for Claude on plain shape counting. Do the two numbers come from different runs or subsets?

    In addition, what is the Yes/No ratio of the single-shape probe? Could this probe be extended to other models and tasks?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer wKna,

Thank you again for your comments. With the discussion deadline approaching, we would like to check whether our responses have addressed your concerns. If there are any remaining questions, we would be happy to respond.

If you find our responses satisfactory, we would appreciate your consideration in updating the evaluation.

Sincerely, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear Reviewers and Area Chair,

Thank you for your time and efforts in evaluating our paper. We are encouraged to see the reviewers' recognition of our paper's strengths, including the clean paired-intervention design that isolates the effect of the reference (wKna, FxC7, 4cdz), the mechanism analysis of how references change the reasoning process (wKna, FxC7), the analysis of when references help or hurt (FxC7, 4cdz), the reusable benchmark and analysis framework (wKna), and the clear presentation (wKna, FxC7). We summarize the new results and revisions below and answer every point in the individual responses.

Key Contributions

  • A controlled diagnostic benchmark. We systematically quantify how much of a model's error an external reference recovers, without changing the model's visual perception capability.
  • A reusable analysis framework. We explain the reference mechanism through the theory of visual routines and use behavioral analysis to test whether the reasoning process itself improves with references. We additionally analyze task-matched versus mismatched references and whether the reference gap persists at larger model scales.
  • A training-free visual skill. An agent draws task-matched references at inference time and improves accuracy on real maps and charts (CartoMapQA, ChartQAPro).

Key Revisions

  • Reference vs. perception (wKna, FxC7, 4cdz). We clarified that the model's visual capability stays fixed and only the image changes: like a ruler, a reference is an external aid that does not change the eyesight of its user (Abstract, Sec. 1, 2.1). New controls show that the gain comes from the reference itself, not from adding marks. Identical "(•)" marks at the numbering positions give no gain (−3.2 vs. +11.3 for numbering). On the unchanged plain image, asking GPT to number the shapes matches drawn numbering (43.0 → 61.0). With the full reference × task matrix, the matched reference gives the largest gain on every task (Sec. 4.2, App. A, D).

  • Generalization and release (FxC7, 4cdz). The visual skill built on our synthetic tasks improves accuracy on two real-world benchmarks (Sec. 5, App. D). Data and code are released at the anonymous link.

  • Statistics (wKna). Every matched pairing is significant at the 5% level. We also report confidence intervals, flip counts, invalid outputs, and decoding settings (App. C.6, D).

Regarding Reviewer 4cdz's follow-up concerns about novelty and the question templates, we respectfully note that visual marks for grounding and entity binding are known to help, but to our knowledge, prior work has not systematically quantified the problem and shown when and why. We show that the gain depends on the model's own reference system and on the visual routine the task requires. Our benchmark is template-based by design. Template questions are standard for diagnostic benchmarks such as CLEVR and BlindTest, and our findings translate into real gains in map reading and chart understanding.

We have incorporated the reviewers' feedback into the revised PDF. Thank you again for your time and consideration.

Sincerely,

Authors of Submission 44375

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.