acceptodds
Under review as a conference paper at ICLR 2027

Visual Scaffolding: Toward Native Multimodal Reasoning

Abstract

The success of chain-of-thought (CoT) in complex reasoning can be attributed to a textual scaffolding effect: explicit intermediate results can be examined, reused, and composed to support subsequent reasoning, much as scaffolding supports a building under construction. However, mainstream multimodal LLMs lack an equivalent scaffold for visual distinctions that are difficult to express in concise verbal descriptions. We therefore ask: Can native visual representations become first-class citizens alongside language, providing a visual scaffold for reasoning? We define native multimodal reasoning as reasoning in which native visual representations and language jointly carry reusable intermediate states. As a concrete implementation, the visual scaffolding mechanism we proposed lets the MLLM autonomously initiate a retrieval action at any reasoning step. A lightweight retriever selects native visual features, which are then incorporated directly into the evolving CoT and remain available to subsequent steps. To evaluate this mechanism under controlled visual demands, we introduce NMR-Bench, comprising Burden Counting and Path Tracing. With 88M added parameters (approximately 2%) on Qwen3-VL-4B-Thinking, our model achieves 66.43% and 72.50% accuracy, respectively, surpassing the evaluated GPT-5.5 and Gemini 3.1 Pro baselines. Controlled interventions show that changing selected visual content affects task accuracy and can redirect subsequent visual selection even when intervening text is held fixed. These findings show that native visual states can function as intermediate reasoning material and point toward reusable scaffolds for additional modalities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Reasoning, Visual Scaffolding, Chain-of-Thought, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j5kcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a multimodal reasoning framework called Visual Scaffolding, which aims to address the limitations of existing multimodal large language models (MLLMs) that primarily rely on textual Chain-of-Thought (CoT) reasoning and struggle to preserve and utilize fine-grained visual information throughout multi-step reasoning processes.

Specifically, the proposed method allows the model to actively trigger visual retrieval during reasoning through a special [ref] token. A reasoning-conditioned Visual Retriever selects relevant regions from high-resolution visual features, which are then inserted into the subsequent reasoning sequence through an Evidence Adaptor. This enables the model to continuously access visual evidence during language generation.

To evaluate the proposed method, the authors introduce NMR-Bench, consisting of two tasks: Burden Counting and Path Tracing, which assess the model's ability to maintain visual instance identity and preserve spatial path continuity, respectively. Experimental results show that VScaffold-4B achieves accuracies of 66.43% and 72.50% on these two tasks, respectively, substantially outperforming the No-Ref baseline trained on the same task-specific data. Furthermore, visual feature replacement experiments and subsequent retrieval analyses demonstrate that intermediate visual features can influence both the model's final predictions and its subsequent visual selections.

Critical strengths
  • The paper proposes dynamically incorporating native visual features into CoT, enabling the model to continuously utilize visual information throughout multi-step reasoning. This is a meaningful and potentially valuable research direction.

  • The proposed method achieves notable improvements on two visual reasoning tasks. The visual feature replacement experiments further demonstrate that intermediate visual information plays a functional role in subsequent reasoning.

Critical weaknesses
  • The proposed method is closely related to existing approaches to interleaved visual reasoning and latent visual reasoning, such as ICoT [1], MINT-CoT [2], and LVR [3]. However, these works are not sufficiently discussed or compared. In addition, I suggest including relevant studies on multimodal reasoning data and training, such as Bee [4] and Zebra-CoT [5], to better clarify the unique contributions of this work.
  • The experiments are primarily conducted on two synthetic visual reasoning tasks, with limited evaluation on real-world images or other established multimodal reasoning benchmarks. Therefore, the generalizability of the proposed method remains unclear.
  • The proposed framework simultaneously introduces high-resolution visual encoding, visual retrieval, and feature insertion mechanisms. However, it lacks a baseline with comparable high-resolution visual inputs. Consequently, it is difficult to determine whether the performance improvements primarily result from access to finer-grained visual information or from the proposed Visual Scaffolding mechanism itself.
  • The paper compares a task-specifically trained model against general-purpose MLLMs that have not undergone equivalent task-specific training, making it difficult to directly attribute the observed performance advantages to the proposed architecture. Furthermore, the paper mentions that the standard evaluation sets were used to inform the selection of the inference evidence cap. This may introduce test-set-dependent model selection and potentially affect the objectivity of the reported results. Further clarification and validation on an independent held-out test set would be beneficial.

[1] Gao, J., et al. Interleaved-Modal Chain-of-Thought. CVPR, 2025. https://openaccess.thecvf.com/content/CVPR2025/html/Gao_Interleaved-Modal_Chain-of-Thought_CVPR_2025_paper.html

[2] Chen, X., et al. MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning. NeurIPS, 2025. https://arxiv.org/abs/2506.05331

[3] Li, B., et al. Latent Visual Reasoning. ICLR, 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/efa0939caa8a906fd0a61a9c60e46e8b-Abstract-Conference.html

[4] Zhang, Y., et al. Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs. ICLR, 2026. https://arxiv.org/abs/2510.13795

[5] Li, A., et al. Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning. ICLR, 2026. https://openreview.net/forum?id=c6XIVI3TiQ

Questions for authors

same with weaknesses.

5 comments
AuthorsOpenReview ↗

Response Part (1/2)

Thank you for your thoughtful and constructive review. We particularly appreciate your recognition of dynamically incorporating native visual features into CoT as a valuable research direction. We also appreciate your positive assessment of our results on both tasks and of the feature replacement experiments, which show that intermediate visual information plays a functional role in subsequent reasoning. We address your concerns point by point below.

w1: Visual Scaffolding has a distinct objective, mechanism, evaluation setting, and experimental analysis relative to the cited work.

We thank the reviewer for identifying these relevant studies. The table below provides NMR-Bench results for their available models and an ICoT adaptation. All methods are evaluated on the original NMR-Bench test sets. The ICoT row is an adaptation to Qwen3-VL-4B-Thinking, not an official ICoT checkpoint. The released MINT-CoT, LVR, Bee, and Zebra-CoT models are official checkpoints.

MethodCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
Qwen3-VL-4B-Thinking2.8610.494.7218.45
ICoT (Qwen3-VL-4B-Thinking)5.718.3213.7926.44
MINT-CoT-7B3.108.830.5612.27
LVR-7B5.2411.780.2815.05
Bee-8B-RL3.5710.332.5021.57
Anole-Zebra-CoT0.4814.730.000.00
Bagel-Zebra-CoT3.5710.042.508.96
No-Ref-4B7.384.6453.0668.17
VScaffold-4B66.430.5072.5086.98

We clarify the distinction at three levels: the research objective, the mechanism, and the evaluation setting and evidence.

  • Research objective. ICoT and MINT-CoT bring visual tokens into reasoning to address details that textual steps may miss, particularly in general visual questions and mathematical figures. LVR studies reasoning through generated latent states that reconstruct relevant visual semantics. We take a broader view. We ask whether native visual tokens can serve as first-class citizen alongside language, so that both modalities carry information that later reasoning steps can inspect and reuse. Recovering details that are difficult to describe in words is one consequence of this view, rather than its full scope. In the longer term, such a framework may support operations in visual tokens analogous to the composition and editing of linguistic concepts, including counterfactual changes to intermediate visual states. We present Visual Scaffolding as a concrete first step toward this goal.

  • Mechanism. ICoT selects original image tokens using existing attention scores when a predefined textual signal appears. MINT-CoT uses an Interleave Token and learned similarity scores to select tokens associated with mathematical reasoning steps. LVR generates latent visual thoughts trained to reconstruct relevant visual features. Our model generates [ref] to request evidence, explicitly combines the current request with recent text and previously inserted visual states, and selects multiple locations from a separate dense feature bank. Spatial context guides selection, while the selected features enter the causal sequence as individual states. The contribution is this combination of active requests, history-conditioned selection, fine spatial access, and continued availability of the selected evidence.

  • Evaluation setting and evidence. Bee primarily contributes multimodal training data, a curation pipeline, and trained models. Zebra-CoT contributes interleaved text and image reasoning data and models trained on it. NMR-Bench instead provides controlled tasks that test sustained object identity and path continuity, with exact answers and spatial annotations. Beyond the accuracy results in the table, we intervene on inserted features and observe changes in later answers and visual selections, including when the intervening text is fixed. These experiments directly test the functional role of visual intermediate states, an aspect the reviewer recognized as a strength.

We will incorporate these discussions and results into the final version of our paper.

AuthorsOpenReview ↗

Response Part (2/2)

w2: Real-world images generalization

NMR-Bench uses procedural images to isolate sustained instance tracking and spatial continuity, provide exact answers, and support controlled interventions. We acknowledge that VScaffold-4B's generalization to real-world scenes remains limited, which is an important direction for future work. A key challenge lies in annotating appropriate visual evidence and corresponding CoT reasoning for non-procedural natural images. In this work, our primary goal is to establish the perspective that visual tokens should serve as first-class citizens alongside textual tokens in CoT reasoning, and to take an initial step toward this goal using synthetic data. Nevertheless, we provide several successful examples on natural images to illustrate the potential of our approach. Due to the plain-text limitations of OpenReview responses, we have included these examples in Appendix H of our revised manuscript.

w3: Effects of higher-resolution visual access

We conducted an additional experiment to address your questions:

  • 2xResolution No-Ref-4B: Similar to No-Ref-4B, this variant does not retain persistent visual evidence but uses images upsampled by 2×, matching the resolution scaling used for visual evidence in VScaffold-4B.

The results are presented below:

MethodCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
No-Ref-4B7.384.6453.0668.17
2xResolution No-Ref-4B16.433.2851.6772.19
VScaffold-4B66.430.5072.5086.98

Increasing the input image resolution improves No-Ref-4B to some extent, but its performance remains substantially below that of VScaffold-4B. This indicates that higher resolution alone is not the primary source of improvement. The Visual Scaffolding mechanism plays a more significant role.

w4: Task-specific training and test-set-informed selection of K

Task-specific training. We evaluated a diverse set of baselines, including frontier MLLMs, Coordinate-Mediated Reasoning Models, Multimodal Visual-Tool Agents, and the interleaved visual reasoning models discussed in W1, to demonstrate the broad relevance of the challenges and the advantages of our mechanism. To ensure a fair comparison, we also trained No-Ref-4B under identical settings, with the only difference from VScaffold-4B being the absence of visual evidence in the CoT. VScaffold-4B improves accuracy from 7.38% to 66.43% on Counting and from 53.06% to 72.50% on Path Tracing. Furthermore, the fixed-checkpoint visual-content interventions provide complementary evidence that the inserted visual features directly influence subsequent computation.

MethodCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
No-Ref-4B (trained on the same data)7.384.6453.0668.17
VScaffold-4B66.430.5072.5086.98

Selection of K. We acknowledge that the original test set informed our choice of K=8K=8. Its reported results cannot be described as performance on an untouched test set. We therefore generated disjoint validation and test sets with new visual assets. We evaluated different values of K on the new validation set and confirmed that K = 8 remains optimal. We then evaluated K = 8 on the new test set.

Evaluation setKCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
Original866.430.5072.5086.98
Disjoint val set870.950.4576.3988.78
Disjoint val set1666.190.5264.5880.45
Disjoint val set3266.670.5562.5078.22
New test set868.570.4869.4485.79

Thank you again for your valuable and helpful review. We would be happy to address any further questions and look forward to continued discussion.

Thank you again for your time and valuable feedback. We have posted our author response and wanted to kindly flag it here in case it was missed. We completely understand that this is a busy period. If any part of our response is unclear, we would be happy to provide a brief clarification. Thank you for your consideration.

Reviewer_uEepOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Visual Scaffolding, which allows an MLLM to retrieve native visual features during chain-of-thought reasoning and reuse them as persistent intermediate states. It further introduces NMR-Bench and uses controlled interventions to show that the inserted visual content affects subsequent reasoning and retrieval.

Critical strengths
  1. The paper presents a clear and potentially important formulation of visual intermediate reasoning. Rather than expressing all intermediate information through text or external tools, the proposed mechanism directly inserts native visual features into the autoregressive reasoning context, providing a concrete implementation of persistent multimodal intermediate states.
  2. The intervention experiments provide unusually strong evidence that the visual states are functionally used. Replacing selected visual features substantially degrades task performance, and changing an earlier visual block can alter later visual selections even when the intervening text is fixed. These experiments support the claim that the inserted features participate in subsequent computation rather than merely serving as auxiliary annotations.
Critical weaknesses
  1. The main comparison does not fully isolate visual scaffolding from increased high-resolution visual access. VScaffold additionally constructs a 2×2\times enlarged visual feature bank, which provides four times as many candidate locations. Since both Counting and Path Tracing strongly depend on fine spatial details, the large gain over No-Ref may partly come from improved visual resolution rather than from persistent visual intermediate states themselves. A matched high-resolution baseline, ideally together with a non-persistent retrieval variant, is needed to isolate the claimed contribution.
  2. The evaluation protocol uses the standard test set for inference-cap selection. The paper states that the standard evaluation set informs the choice of the evidence cap KK, while changing KK from 3232 to 88 improves accuracy by approximately 1111--1313 percentage points. This introduces test-set tuning into the main reported results and should be corrected using a separate validation set.
Questions for authors
  1. Can the authors provide a matched control with the same 2×2\times high-resolution visual encoding and comparable training budget, but without persistent visual-feature insertion? Ideally, a variant where retrieved features are available only to the current reasoning step would help determine whether the gain comes from finer visual access or from the proposed persistent visual scaffolding itself.

  2. The paper states that the standard evaluation set is used to inform the choice of the inference evidence cap KK, while this choice changes accuracy by roughly 1111--1313 percentage points. Can the authors select KK on a disjoint validation set and report the resulting untouched test performance?

4 comments

Thank you for your thoughtful and constructive review. We particularly appreciate your recognition of our formulation of persistent visual intermediate states and your positive assessment of our intervention experiments, especially the evidence that earlier visual content can affect later retrieval even when the intervening text is fixed. We address your questions point by point below.

w1: The contribution of persistent visual evidence should be separated from the benefit of higher visual resolution.

We conducted two additional experiments to address your questions:

  • 2xResolution No-Ref-4B: Similar to No-Ref-4B, this variant does not retain persistent visual evidence but uses images upsampled by 2×, matching the resolution scaling used for visual evidence in VScaffold-4B.
  • VScaffold-4B, latest-only: This variant follows VScaffold-4B but retains only the most recently retrieved visual evidence rather than accumulating evidence across reasoning steps.

The results are presented below:

MethodCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
No-Ref-4B7.384.6453.0668.17
2xResolution No-Ref-4B16.433.2851.6772.19
VScaffold-4B66.430.5072.5086.98
VScaffold-4B, latest-only0.24238.4666.1180.53

1. No-Ref-4B vs. 2xResolution No-Ref-4B vs. VScaffold-4B. Increasing the input image resolution improves No-Ref-4B to some extent, but its performance remains substantially below that of VScaffold-4B. This indicates that higher resolution alone is not the primary source of improvement. The Visual Scaffolding mechanism plays a more significant role.

2. VScaffold-4B vs. VScaffold-4B, latest-only. Retaining only the latest visual evidence leads to different outcomes across tasks: Counting accuracy drops dramatically to near-random levels, whereas Path Tracing exhibits only a modest decline. This difference reflects how the two tasks utilize visual evidence. Counting requires accumulating and preserving previous counting records, so discarding earlier evidence disrupts the accumulated count and causes a severe performance degradation. In contrast, Path Tracing primarily aims to match endpoints, where each step largely depends on the immediately preceding visual evidence, resembling a Markovian process. Consequently, retaining only the latest evidence has a relatively limited impact.

These findings further validate the effectiveness of Visual Scaffolding while revealing task-dependent mechanisms of visual evidence utilization.

w2: The inference cap was informed by the original test set and requires evaluation on disjoint data.

We acknowledge that the original test set informed our choice of K=8K=8. Its reported results cannot be described as performance on an untouched test set. We additionally constructed new validation and test sets with strictly disjoint generation seeds and rigorous post-generation filtering to prevent data overlap. Incorporating feedback from other reviewers, both sets were generated using entirely new visual assets. We evaluated different values of K on the new validation set and confirmed that K = 8 remains optimal. We then evaluated K = 8 on the new test set, with the results presented below:

Evaluation setKCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
Original866.430.5072.5086.98
Disjoint val set870.950.4576.3988.78
Disjoint val set1666.190.5264.5880.45
Disjoint val set3266.670.5562.5078.22
New test set868.570.4869.4485.79

Thank you again for your valuable and helpful review. We would be happy to address any further questions and look forward to continued discussion.

Thank you again for your time and valuable feedback. We have posted our author response and wanted to kindly flag it here in case it was missed. We completely understand that this is a busy period. If any part of our response is unclear, we would be happy to provide a brief clarification. Thank you for your consideration.

Thank you for the additional experiments. The resolution-matched baseline and latest-only ablation provide useful evidence that the gains cannot be explained solely by higher visual resolution and that retaining historical visual evidence is important, particularly for counting.

The new validation and disjoint test evaluation also adequately address my concern regarding test-set tuning.

One minor clarification would be whether the 2×Resolution baseline matches the visual feature bank and candidate-location budget of VScaffold, and whether the extreme Counting degradation in the latest-only setting has been verified.

Overall, my main concerns have been substantially addressed. I appreciate the additional effort and am inclined to increase my overall rating.

Thank you very much for your positive feedback. Your review has been extremely helpful in improving our work. Thank you again for your time and constructive comments.

Reviewer_NYq5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces visual scaffolding, a mechanism that interleaves textual reasoning with retrieved native visual features. Built on Qwen3-VL-4B-Thinking, the model emits a special reference token to request visual evidence. A context-conditioned retriever selects features from a dense image representation, and these features are inserted into the autoregressive sequence, remaining available to subsequent reasoning steps.

It also introduces NMR-Bench, consisting of Burden Counting and Path Tracing, and report substantial improvements over a task-trained No-Ref baseline. They also conduct controlled interventions on inserted visual content, including experiments that hold intervening text fixed, to investigate whether visual evidence influences subsequent reasoning and retrieval.

Critical strengths

The paper provides substantive evidence that inserted visual content has a functional role in subsequent computation. The authors go beyond comparing accuracy with and without retrieval. They replace visual content while controlling the reference interface and evidence-block length, include same-image donors and norm-matched controls, and examine multiple intervention positions and donor seeds. Particularly useful are the fixed-text experiments: subsequent retrieval can change even when the intervening textual sequence is held constant. These results support the claim that the visual features are computationally consequential and that their influence is not fully mediated by the emitted words.

Critical weaknesses

The novelty needs clearer positioning against closely related approaches. ICoT already selects original-image visual tokens using attention and inserts them into autoregressive reasoning, while TVI-CoT uses learnable control tokens, reasoning-conditioned visual grounding, and visual-feature injection with language-modeling and grounding supervision. These methods overlap substantially with the proposed mechanism, although their feature selection and representation differ.

The context-conditioned retriever, spatial feature selection, and intervention-based analysis may offer meaningful advances, but these contributions should be explicitly distinguished and supported by direct comparisons. Comparisons with coordinate-based interfaces and external visual tools alone do not establish the advance over native feature-interleaving methods.

Questions for authors
  1. How does the proposed mechanism differ substantively from ICoT and TVI-CoT? Can the authors provide a comparison on the same backbone and benchmark that controls, as far as possible, for task adaptation and visual-information access?

  2. Beyond disjoint example identifiers, which generator components or visual assets are held out between training and evaluation? This would clarify whether the reported results primarily measure new-instance generalization within the same procedural distribution or transfer to unfamiliar visual structures.

4 comments
AuthorsOpenReview ↗

Response Part (1/2)

Thank you for your thoughtful and constructive review. We particularly appreciate your recognition of our work and your positive assessment of our controlled visual-content interventions, especially the fixed-text experiments, which provide substantive evidence that the inserted visual features influence subsequent reasoning and retrieval even when the intervening text is held constant. We address your questions point by point below.

w1: Innovations of Visual Scaffolding over ICoT and TVI-CoT and Quantitative Comparisons

We provide additional quantitative comparisons with closely related methods. Since TVI-CoT's implementation remains unavailable (https://github.com/hulianyuyy/TVI-CoT) and could not be reliably reproduced within the rebuttal period, we focus on comparing Visual Scaffolding with ICoT on NMR-Bench.

While ICoT originally used Chameleon-7B and Qwen2-VL-7B, we adopt the stronger Qwen3-VL-4B-Thinking backbone for a fair comparison and verify the correctness of our reproduction. The results below show that Visual Scaffolding consistently outperforms ICoT by a substantial margin.

MethodCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
Qwen3-VL-4B-Thinking2.8610.494.7218.45
ICoT (Qwne3-VL-4B-T)5.718.3213.7926.44
VScaffold-4B66.430.5072.5086.98

Then we clarify the distinction at four levels: the research objective, the mechanism, the evaluation setting, and the experimental evidence.

  • Research objective. ICoT and TVI-CoT both identify a limitation of text-only visual reasoning: language may fail to preserve fine-grained information from an image. Their interleaved methods address this limitation by making visual information available during reasoning. We take a broader view. We ask whether native visual tokens can serve as first-class citizen alongside language, so that both modalities carry information that later reasoning steps can inspect and reuse. Recovering details that are difficult to describe in words is one consequence of this view, rather than its full scope. In the longer term, such a framework may support operations in visual tokens analogous to the composition and editing of linguistic concepts, including counterfactual changes to intermediate visual states. We present Visual Scaffolding as a concrete first step toward this goal.

  • Mechanism. This reasearch objective above leads to three design choices. First, the model initiates visual participation by generating [ref] as an explicit request for evidence, much as it chooses the next step in a textual CoT. In ICoT, visual insertion follows a predefined textual signal, and the inserted features are selected using the model’s attention scores. Second, each request is guided by the reasoning developed so far. Our retriever explicitly combines the current request state with recent decoder states, including previously inserted visual evidence, to determine which image locations are needed next. TVI-CoT uses the hidden state at <Look> to guide grounding, whereas our explicit aggregation lets earlier visual states directly inform the next retrieval query. Third, the retrieved evidence remains individually available for subsequent reasoning. TVI-CoT formulates its injected evidence as a single weighted visual representation. We instead insert multiple native features as separate states in the causal sequence, allowing later steps to use them for both reasoning and further visual requests.

  • Two additional architectural choices make this interface practical for current MLLMs. Dual-resolution encoding preserves the backbone's standard image input while providing denser candidate features for retrieval, distinguishing nearby objects or competing path segments without expanding the full image prefix. Spatially informed selection incorporates local neighborhood information into retrieval keys to support object localization and spatial continuity, while independently scored locations enable variable-sized feature selection. Together, these components provide a reusable design template for integrating native visual states into reasoning traces.

AuthorsOpenReview ↗

Response Part (2/2)

  • Evaluation setting. To make this evaluation more comprehensive and controlled, we introduce NMR-Bench. It comprises Burden Counting and Path Tracing, two settings designed to expose systematic limitations when intermediate reasoning is confined to text. Burden Counting requires the model to maintain visual object identities while counting, and Path Tracing requires it to follow spatial continuity through competing connections. The benchmark provides a standardized protocol for evaluating native visual representations as part of CoT, and we hope it will support future research in this direction.

  • Experimental evidence. Beyond final-answer accuracy and component ablations, we intervene directly on the visual states inserted during reasoning. We test whether changing their content affects the answer and whether it changes the model’s next visual selection even when the intervening text is held fixed. These experiments examine the causal role of visual evidence in the evolving reasoning process, rather than relying solely on accuracy gains or illustrative traces. We appreciate the reviewer’s recognition of this aspect of our work.

w2: What is held out between training and evaluation, and what kind of generalization do the results establish?

Thank you for asking us to clarify the scope of generalization. In our original experiments, the training and test sets shared the same visual assets but used strictly disjoint generation seeds. We also performed programmatic post-generation checks to ensure no overlap between the two sets.

To further address concerns about generalization, we generated an additional test set using entirely new visual assets (Google Noto Emoji), while maintaining unique generation seeds and the same programmatic verification procedure. As shown below, the model exhibits only minor performance fluctuations, suggesting that Visual Scaffolding does not overfit to specific visual assets.

Evaluation setCounting Acc. ↑Counting MAE ↓Path Acc. ↑Path CRA ↑
Original66.430.5072.5086.98
New-artwork68.570.4869.4485.79

Additionally, in response to other reviewers, we have included examples on natural images in Appendix H of our revised manuscript, which has already been uploaded. Due to the plain-text limitations of OpenReview responses, we cannot present these visual examples here.

Thank you again for your valuable and helpful review. We would be happy to address any further questions and look forward to continued discussion.

I thank the authors for their responses. I am inclined to maintain this positive score.

Thank you again for your recognition of our work and for your thorough review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.