- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a multimodal reasoning framework called Visual Scaffolding, which aims to address the limitations of existing multimodal large language models (MLLMs) that primarily rely on textual Chain-of-Thought (CoT) reasoning and struggle to preserve and utilize fine-grained visual information throughout multi-step reasoning processes.
Specifically, the proposed method allows the model to actively trigger visual retrieval during reasoning through a special [ref] token. A reasoning-conditioned Visual Retriever selects relevant regions from high-resolution visual features, which are then inserted into the subsequent reasoning sequence through an Evidence Adaptor. This enables the model to continuously access visual evidence during language generation.
To evaluate the proposed method, the authors introduce NMR-Bench, consisting of two tasks: Burden Counting and Path Tracing, which assess the model's ability to maintain visual instance identity and preserve spatial path continuity, respectively. Experimental results show that VScaffold-4B achieves accuracies of 66.43% and 72.50% on these two tasks, respectively, substantially outperforming the No-Ref baseline trained on the same task-specific data. Furthermore, visual feature replacement experiments and subsequent retrieval analyses demonstrate that intermediate visual features can influence both the model's final predictions and its subsequent visual selections.
Critical strengths
-
The paper proposes dynamically incorporating native visual features into CoT, enabling the model to continuously utilize visual information throughout multi-step reasoning. This is a meaningful and potentially valuable research direction.
-
The proposed method achieves notable improvements on two visual reasoning tasks. The visual feature replacement experiments further demonstrate that intermediate visual information plays a functional role in subsequent reasoning.
Critical weaknesses
- The proposed method is closely related to existing approaches to interleaved visual reasoning and latent visual reasoning, such as ICoT [1], MINT-CoT [2], and LVR [3]. However, these works are not sufficiently discussed or compared. In addition, I suggest including relevant studies on multimodal reasoning data and training, such as Bee [4] and Zebra-CoT [5], to better clarify the unique contributions of this work.
- The experiments are primarily conducted on two synthetic visual reasoning tasks, with limited evaluation on real-world images or other established multimodal reasoning benchmarks. Therefore, the generalizability of the proposed method remains unclear.
- The proposed framework simultaneously introduces high-resolution visual encoding, visual retrieval, and feature insertion mechanisms. However, it lacks a baseline with comparable high-resolution visual inputs. Consequently, it is difficult to determine whether the performance improvements primarily result from access to finer-grained visual information or from the proposed Visual Scaffolding mechanism itself.
- The paper compares a task-specifically trained model against general-purpose MLLMs that have not undergone equivalent task-specific training, making it difficult to directly attribute the observed performance advantages to the proposed architecture. Furthermore, the paper mentions that the standard evaluation sets were used to inform the selection of the inference evidence cap. This may introduce test-set-dependent model selection and potentially affect the objectivity of the reported results. Further clarification and validation on an independent held-out test set would be beneficial.
[1] Gao, J., et al. Interleaved-Modal Chain-of-Thought. CVPR, 2025. https://openaccess.thecvf.com/content/CVPR2025/html/Gao_Interleaved-Modal_Chain-of-Thought_CVPR_2025_paper.html
[2] Chen, X., et al. MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning. NeurIPS, 2025. https://arxiv.org/abs/2506.05331
[3] Li, B., et al. Latent Visual Reasoning. ICLR, 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/efa0939caa8a906fd0a61a9c60e46e8b-Abstract-Conference.html
[4] Zhang, Y., et al. Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs. ICLR, 2026. https://arxiv.org/abs/2510.13795
[5] Li, A., et al. Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning. ICLR, 2026. https://openreview.net/forum?id=c6XIVI3TiQ
Questions for authors
same with weaknesses.
5 comments
Response Part (2/2)
w2: Real-world images generalization
NMR-Bench uses procedural images to isolate sustained instance tracking and spatial continuity, provide exact answers, and support controlled interventions. We acknowledge that VScaffold-4B's generalization to real-world scenes remains limited, which is an important direction for future work. A key challenge lies in annotating appropriate visual evidence and corresponding CoT reasoning for non-procedural natural images. In this work, our primary goal is to establish the perspective that visual tokens should serve as first-class citizens alongside textual tokens in CoT reasoning, and to take an initial step toward this goal using synthetic data. Nevertheless, we provide several successful examples on natural images to illustrate the potential of our approach. Due to the plain-text limitations of OpenReview responses, we have included these examples in Appendix H of our revised manuscript.
w3: Effects of higher-resolution visual access
We conducted an additional experiment to address your questions:
- 2xResolution No-Ref-4B: Similar to No-Ref-4B, this variant does not retain persistent visual evidence but uses images upsampled by 2×, matching the resolution scaling used for visual evidence in VScaffold-4B.
The results are presented below:
| Method | Counting Acc. ↑ | Counting MAE ↓ | Path Acc. ↑ | Path CRA ↑ |
|---|---|---|---|---|
| No-Ref-4B | 7.38 | 4.64 | 53.06 | 68.17 |
| 2xResolution No-Ref-4B | 16.43 | 3.28 | 51.67 | 72.19 |
| VScaffold-4B | 66.43 | 0.50 | 72.50 | 86.98 |
Increasing the input image resolution improves No-Ref-4B to some extent, but its performance remains substantially below that of VScaffold-4B. This indicates that higher resolution alone is not the primary source of improvement. The Visual Scaffolding mechanism plays a more significant role.
w4: Task-specific training and test-set-informed selection of K
Task-specific training. We evaluated a diverse set of baselines, including frontier MLLMs, Coordinate-Mediated Reasoning Models, Multimodal Visual-Tool Agents, and the interleaved visual reasoning models discussed in W1, to demonstrate the broad relevance of the challenges and the advantages of our mechanism. To ensure a fair comparison, we also trained No-Ref-4B under identical settings, with the only difference from VScaffold-4B being the absence of visual evidence in the CoT. VScaffold-4B improves accuracy from 7.38% to 66.43% on Counting and from 53.06% to 72.50% on Path Tracing. Furthermore, the fixed-checkpoint visual-content interventions provide complementary evidence that the inserted visual features directly influence subsequent computation.
| Method | Counting Acc. ↑ | Counting MAE ↓ | Path Acc. ↑ | Path CRA ↑ |
|---|---|---|---|---|
| No-Ref-4B (trained on the same data) | 7.38 | 4.64 | 53.06 | 68.17 |
| VScaffold-4B | 66.43 | 0.50 | 72.50 | 86.98 |
Selection of K. We acknowledge that the original test set informed our choice of . Its reported results cannot be described as performance on an untouched test set. We therefore generated disjoint validation and test sets with new visual assets. We evaluated different values of K on the new validation set and confirmed that K = 8 remains optimal. We then evaluated K = 8 on the new test set.
| Evaluation set | K | Counting Acc. ↑ | Counting MAE ↓ | Path Acc. ↑ | Path CRA ↑ |
|---|---|---|---|---|---|
| Original | 8 | 66.43 | 0.50 | 72.50 | 86.98 |
| Disjoint val set | 8 | 70.95 | 0.45 | 76.39 | 88.78 |
| Disjoint val set | 16 | 66.19 | 0.52 | 64.58 | 80.45 |
| Disjoint val set | 32 | 66.67 | 0.55 | 62.50 | 78.22 |
| New test set | 8 | 68.57 | 0.48 | 69.44 | 85.79 |
Thank you again for your valuable and helpful review. We would be happy to address any further questions and look forward to continued discussion.
Thank you again for your time and valuable feedback. We have posted our author response and wanted to kindly flag it here in case it was missed. We completely understand that this is a busy period. If any part of our response is unclear, we would be happy to provide a brief clarification. Thank you for your consideration.
Response Part (1/2)
Thank you for your thoughtful and constructive review. We particularly appreciate your recognition of dynamically incorporating native visual features into CoT as a valuable research direction. We also appreciate your positive assessment of our results on both tasks and of the feature replacement experiments, which show that intermediate visual information plays a functional role in subsequent reasoning. We address your concerns point by point below.
w1: Visual Scaffolding has a distinct objective, mechanism, evaluation setting, and experimental analysis relative to the cited work.
We thank the reviewer for identifying these relevant studies. The table below provides NMR-Bench results for their available models and an ICoT adaptation. All methods are evaluated on the original NMR-Bench test sets. The ICoT row is an adaptation to Qwen3-VL-4B-Thinking, not an official ICoT checkpoint. The released MINT-CoT, LVR, Bee, and Zebra-CoT models are official checkpoints.
We clarify the distinction at three levels: the research objective, the mechanism, and the evaluation setting and evidence.
Research objective. ICoT and MINT-CoT bring visual tokens into reasoning to address details that textual steps may miss, particularly in general visual questions and mathematical figures. LVR studies reasoning through generated latent states that reconstruct relevant visual semantics. We take a broader view. We ask whether native visual tokens can serve as first-class citizen alongside language, so that both modalities carry information that later reasoning steps can inspect and reuse. Recovering details that are difficult to describe in words is one consequence of this view, rather than its full scope. In the longer term, such a framework may support operations in visual tokens analogous to the composition and editing of linguistic concepts, including counterfactual changes to intermediate visual states. We present Visual Scaffolding as a concrete first step toward this goal.
Mechanism. ICoT selects original image tokens using existing attention scores when a predefined textual signal appears. MINT-CoT uses an Interleave Token and learned similarity scores to select tokens associated with mathematical reasoning steps. LVR generates latent visual thoughts trained to reconstruct relevant visual features. Our model generates
[ref]to request evidence, explicitly combines the current request with recent text and previously inserted visual states, and selects multiple locations from a separate dense feature bank. Spatial context guides selection, while the selected features enter the causal sequence as individual states. The contribution is this combination of active requests, history-conditioned selection, fine spatial access, and continued availability of the selected evidence.Evaluation setting and evidence. Bee primarily contributes multimodal training data, a curation pipeline, and trained models. Zebra-CoT contributes interleaved text and image reasoning data and models trained on it. NMR-Bench instead provides controlled tasks that test sustained object identity and path continuity, with exact answers and spatial annotations. Beyond the accuracy results in the table, we intervene on inserted features and observe changes in later answers and visual selections, including when the intervening text is fixed. These experiments directly test the functional role of visual intermediate states, an aspect the reviewer recognized as a strength.
We will incorporate these discussions and results into the final version of our paper.