acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Visual Thoughts: From Inert Context to Privileged Supervision

Abstract

Interleaved visual chain-of-thought (CoT) enables multimodal models to generate intermediate images during reasoning, under the premise that visual states capture what text poorly describes. We investigate a fundamental question: do these models actually use the images they generate? Through a targeted intervention framework, we reveal that self-generated visual thoughts remain largely question-invariant, and replacing them with noise or the original input causes little performance drop. Instead, subsequent reasoning is overwhelmingly dominated by language, treating visual tokens as inert context. Building on this insight, we propose treating visual thoughts as privileged supervision during training to transfer visual knowledge directly into the language pathway. We explore two distinct strategies: explicitly verbalizing visual states into text (Verbalized CoT, V-CoT) and implicitly aligning the hidden states of the student's own reasoning with a teacher that reads the visual thoughts (Visual-State Distillation, VSD). Extensive evaluations across diverse model backbones show that both strategies yield highly effective text-only reasoners, outperforming standard interleaved baselines on average in every recipe across varied visual reasoning domains while substantially improving overall inference efficiency at test time. All code, data, and model weights will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Reasoning, Visual Chain-of-Thought, Privileged Supervision, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PkYpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether intermediate visual thoughts in multimodal chain-of-thought (CoT) reasoning models are genuinely used to support reasoning or merely serve as inert context. Through analyses of visual-thought similarity, perturbation-based interventions, and attention patterns, the authors argue that existing models often fail to effectively assimilate their generated visual thoughts into the reasoning process. To address this issue, the paper proposes two training strategies that treat visual thoughts as privileged supervision. The first, Visual Chain-of-Thought (V-CoT), converts intermediate visual states into textual descriptions and incorporates them into training traces. The second, Visual State Distillation (VSD), uses a teacher model with access to visual thoughts to guide a student model that does not observe them, aligning their hidden representations through on-policy distillation. Experiments across multiple model-training recipes, backbones, and visual reasoning benchmarks suggest that these approaches can improve reasoning performance while encouraging more effective use of visual information. Overall, the paper reframes intermediate visual thoughts from inference-time context into training-time supervision and presents practical methods for leveraging them.

Critical strengths

The paper questions a central assumption in multimodal chain-of-thought reasoning: that generating intermediate visual thoughts necessarily means the model uses them for subsequent reasoning. This distinction between producing visual information and effectively assimilating it is meaningful and motivates a focused investigation. Rather than introducing training techniques without examining the underlying problem, the authors first analyze whether visual thoughts influence model predictions through similarity measurements, perturbation-based interventions, and attention patterns. These analyses provide motivation for the proposed methods and help connect the empirical diagnosis to the training design. V-CoT provides an intuitive way to expose models to textual interpretations of intermediate visual states, while VSD transfers information from a teacher with access to visual thoughts to a student that does not. The two methods offer complementary perspectives on how visual thoughts can be exploited during training without necessarily requiring the student to consume them at inference time.

Critical weaknesses

The claim that interleaved visual thoughts are largely ignored is supported by intervention experiments on three open-source models: ThinkMorph, Zebra-CoT, and MathCanvas. However, it remains unclear whether this finding generalizes to other multimodal architectures, stronger models, or tasks in which intermediate visual states are essential. The current results may reveal a limitation of the evaluated models and training recipes rather than a general property of interleaved visual chain-of-thought.

The analysis of task relevance relies partly on CLIP embedding similarity, which may not adequately capture fine-grained structural or task-specific differences between intermediate visual states. Similarly, replacing a visual thought with Gaussian noise or the original input is informative but does not exhaustively test how the model uses visual information. These interventions may not fully distinguish between a genuinely irrelevant visual thought and a useful representation that the model cannot robustly process under the tested conditions.

Both V-CoT and VSD improve average performance over the evaluated baselines, but their effectiveness may depend on the quality and informativeness of the generated visual thoughts. V-CoT relies on an external captioner to verbalize intermediate states, while VSD depends on a teacher that can extract useful information from those states. The paper does not fully establish how robust these methods are when visual thoughts are noisy, incomplete, misleading, or difficult to describe in language.

The paper includes shuffled-description and blinded-teacher controls, which provide useful evidence that the content of visual supervision matters. Nevertheless, it would be helpful to further separate the benefits of visual information from those of additional supervision, changes in training trajectories, and optimization effects. In particular, the comparison between V-CoT, VSD, and TextCoT does not by itself establish which components account for the gains across different tasks and backbones.

Limited evaluation of statistical significance and robustness. The reported results average four training seeds and include standard deviations, which is helpful. However, the paper would benefit from more explicit statistical testing or confidence intervals for the improvements, especially on individual benchmarks where gains are small or inconsistent.

Questions for authors

The paper concludes that interleaved visual thoughts are largely ignored by downstream reasoning. How robust is this finding across a broader range of multimodal architectures, model scales, and visual reasoning tasks? Could the authors provide evidence that the observed phenomenon is not specific to the three evaluated model families or their training recipes?

The task-relevance analysis relies on CLIP embedding similarity, while the causal analysis replaces visual thoughts with noise or the original input image. How do the authors ensure that these measurements and interventions adequately capture the functional utility of intermediate visual states? In particular, could the conclusions change when using task-specific structural metrics or alternative interventions that preserve more of the visual thought's information?

Both V-CoT and VSD rely on informative intermediate visual thoughts, either through textual descriptions or teacher representations. How sensitive are the proposed methods to errors, omissions, and hallucinations in these visual thoughts? Could the authors systematically evaluate performance under controlled levels of visual-supervision noise and across tasks with different degrees of dependence on spatial or geometric information?

Although the shuffled-description and blinded-teacher experiments support the importance of visual information, how much of the improvement can be attributed specifically to the visual supervision rather than additional textual supervision or optimization effects? Could the authors include stronger controls that match the amount and structure of training supervision while varying only the availability of task-relevant visual information?

The experiments average results over four training seeds, but some benchmark-level improvements are relatively small and performance varies across datasets. Have the authors conducted paired significance tests or reported confidence intervals for the main comparisons? Are the improvements consistent across individual seeds, especially on benchmarks where V-CoT or VSD does not outperform TextCoT?

The paper motivates a shift from inference-time visual thoughts to training-time privileged supervision. Are there task categories or architectural conditions under which directly consuming generated visual states remains more effective than V-CoT or VSD? A more systematic characterization of these conditions would help clarify the scope of the proposed paradigm and avoid overgeneralizing the diagnosis.

3 comments
AuthorsOpenReview ↗

Response to Reviewer PkYp

We thank Reviewer PkYp for the thorough review and for recognizing the value of separating visual generation from downstream assimilation. Several requested controls and statistical tests are already in our Appendix, and we realize they were not prominent enough in the main text. Below we present them alongside new experiments and commit to moving the key tables (Tables 9 and 10) into the main paper.

W1 and Q1. Generality across architectures, scales, and tasks.

Architectures and training recipes. Our three diagnosed models span two backbones (Bagel-7B and Anole-7B), two re-entry interfaces (continuous VAE and ViT features versus discrete VQ tokens), and three domains (spatial, geometric, and multi-domain). We further apply our full protocol to Omni-R1 and Omni-R1-Zero, which add RL with a perception-calibrated reward after SFT.

ModelTrainingChanged: z←ε / z←x (%)Changed: Resampling (%)Δ Acc.: z←ε / z←x
Omni-R1SFT + RL68.1 / 63.544.8+0.66 / −1.01
Omni-R1-ZeroSFT + RL45.4 / 50.015.2−0.97 / −1.27

Although RL increases answer sensitivity to z beyond seed resampling, net accuracy stays within 1.3 points, showing that z steers the trajectory without improving accuracy. Training our Zebra-CoT (Bagel) Interleave model for more epochs likewise leaves the diagnosis unchanged.

Zebra-CoT (Bagel) InterleaveΔ Acc. (z←ε)Δ Acc. (z←x)
1 epoch−0.38+0.29
2 epochs−0.46−0.35
3 epochs−0.39−0.25

Tasks where intermediate states are essential. Appendix D.3 (Table 6) already evaluates ThinkMorph's in-domain jigsaw assembly, spatial navigation, and visual search benchmarks, whose training traces are built directly around visual intermediates.

ThinkMorph, In-Domain Tasks (410 questions)Native (%)Δ BlindΔ x←εΔ z←εΔ z←x
Accuracy and changes (points)73.9−16.2−28.8−0.12+0.08

Removing the input image costs 16.2 points on these tasks, whereas replacing the generated visual thought changes accuracy by less than 0.2 points.

Scale. The open interleaved reasoners trained for visual reasoning, including ThinkMorph, Zebra-CoT, MathCanvas, and Omni-R1, are all built on 7B backbones. We will explicitly scope our claims to the models studied and revise the Abstract and Conclusion accordingly.

W2 and Q2. Adequacy of CLIP similarity and causal interventions.

Similarity metrics. Appendix D.2 (Table 5) calibrated global CLIP against this exact concern. Ground-truth intermediate states, differing only in task-relevant regions, lower CLIP similarity from 1.00 to 0.82 (−0.18), whereas generated S_q never falls below S_seed for any model. Following the reviewer's suggestion, we further evaluate similarity strictly inside the task-relevant region using three fine-grained metrics across sampled constructed sets (60, 60, and 39 images for ThinkMorph, Zebra-CoT (Anole), and MathCanvas), testing whether each generated thought best matches its own question's ground-truth state among the four questions per image.

Matching Accuracy (%), Chance = 25%ThinkMorphZebra-CoT (Anole)MathCanvasGround-Truth States
Global CLIP (Table 5)29.431.227.8–
Task-relevant region, pixel MSE24.423.926.7100.0
Task-relevant region, SSIM24.021.624.6100.0
Task-relevant region, DINOv2 patch features26.222.727.1100.0

Across all localized structural metrics, generated visual thoughts match their target question near chance, whereas the same metrics identify ground-truth states perfectly. Thus, question invariance is not an artifact of coarse global embeddings.

Interventions. Our framework spans full removal (noise ε), removing only information beyond x (input x; Section 3.3), and partial corruption (64-token perturbation; Section 3.4). Following the reviewer's suggestion, we add a question-swap intervention on a paired subset of the constructed tasks, replacing z_k with a thought z_k′ generated for another question on the same image to preserve visual style while altering only the task step, alongside the ground-truth oracle z* (which also changes accuracy by at most 0.5 points on the full sets of n=1,976 and 1,173 in Appendix D.6, Table 7).

Constructed Sets Subset, Accuracy (%)nNative zz←z_k′ (Question Swap)z←z* (Oracle)
ThinkMorph24040.841.740.4
MathCanvas9036.735.631.1

Neither swapping across questions nor supplying drawn ground-truth states yields accuracy gains. To test whether the reasoners can exploit any visual thought, we construct oracles on VStar from its annotated boxes. For the 115 attribute questions, z* is a high-resolution target crop revealing detail lost in downsampled x. For the 76 relative-position questions, z* outlines and labels targets on x without adding unseen pixels.

VStarNative (%)Δ z←εΔ z←xΔ z*, Attribute Crop (n=115)Δ z*, Marked Input (n=76)
ThinkMorph62.8+0.5−1.1+24.40.0
Zebra-CoT (Anole)28.4+1.1+1.6+15.7−3.9

The visual re-entry slot is functional when z* supplies new high-resolution perceptual evidence, but not when z* encodes reasoning annotations on existing content, and self-generated thoughts provide neither. We will refine Section 3 accordingly.

W3 and Q3. Sensitivity to noisy supervision and task dependence. Complementing the 100%-corrupted controls in Table 2, we evaluate 50% supervision corruption and open-weight captioners on MathCanvas (averaged over its five benchmarks).

MathCanvas Recipe, Avg. (%)Clean50% Corrupted100% Corrupted (Table 2)
V-CoT (descriptions shuffled)49.647.845.5
VSD (teacher reads noise)48.545.844.2
V-CoT w/ Qwen3.5-27B / Qwen3.5-8B captioner49.3 / 48.7––
TextCoT / Interleave46.6 / 46.8––

V-CoT degrades gracefully, still beating TextCoT and Interleave under 50% corruption or with an 8B open captioner. VSD is more sensitive to heavy noise, dropping below TextCoT at 50% because a noise-conditioned teacher pulls student hidden states toward uninformative targets. Thus, V-CoT is preferable when reference visual states are noisy.

Grouping all 25 recipe–benchmark pairs in Table 1 by domain further clarifies task-level behavior.

Median Gain over TextCoTV-CoTVSD
Symbolic/geometric/logical benchmarks (13 pairs)+1.4+0.7
Perceptual/spatial benchmarks (12 pairs)+1.3+1.9

Both methods improve across both categories, with V-CoT excelling on discrete, nameable elements (e.g., auxiliary lines and paths) and VSD excelling on perceptual states that resist verbalization.

W4 and Q4. Isolating visual information from supervision structure and optimization. Appendix F.5 (Table 9) provides the exact structure-matched controls requested, holding supervision position, length, format, and optimization constant while removing only z.

Variant, Average Accuracy (%)ThinkMorphZebra-CoT (Bagel)
TextCoT66.343.4
V-CoT, captioner without z (matched position, format, length)66.043.5
V-CoT68.244.9
VSD, teacher without z (matched reader, rollouts, EMA, loss weight)66.343.0
VSD, teacher with repeated x (matched image count, no new info)65.241.8
VSD68.444.6

None of these controls improves over TextCoT. Combined with Table 2 (where shuffled descriptions and blinded teachers drop accuracy by 4.1–6.9 and 2.7–6.5 points), this confirms that gains stem specifically from task-relevant information in z rather than extra text, extra images, or on-policy distillation mechanics.

W5 and Q5. Statistical significance and seed consistency. Appendix F.6 (Table 10) evaluates each recipe-level gain using a paired bootstrap (10,000 resamples drawn jointly over training seeds and questions) alongside per-seed win counts.

RecipeVariantGain over TextCoT95% CISeed Wins
ThinkMorphV-CoT+1.9[+0.9, +3.1]4/4
ThinkMorphVSD+2.1[+1.2, +3.4]4/4
Zebra-CoT (Bagel)V-CoT+1.5[+0.6, +2.8]4/4
Zebra-CoT (Bagel)VSD+1.2[+0.5, +1.8]4/4
Zebra-CoT (Anole)V-CoT+0.6[−0.4, +1.7]3/4
Zebra-CoT (Anole)VSD+2.0[+1.1, +3.3]4/4
MathCanvasV-CoT+3.0[+1.8, +4.5]4/4
MathCanvasVSD+1.9[+1.1, +2.6]4/4

Seven of the eight gains are statistically significant and hold across all 4/4 seeds; on the weaker Anole backbone where self-generating descriptions is harder for V-CoT, VSD achieves a significant +2.0 (4/4 seeds). Across individual benchmarks, V-CoT beats TextCoT on 21/25 pairs and VSD on 20/25, including every benchmark on ThinkMorph (6/6) and MathCanvas (5/5), while the remaining Zebra-CoT pairs are ties or differ by at most 0.8 (V-CoT) and 1.1 points (VSD). We will move Table 10 to the main text.

Q6. When does consuming visual thoughts directly remain preferable? Across all 25 pairs in Table 1, Interleave never outperforms the better of V-CoT and VSD, even where visual intermediates matter strongly (e.g., ThinkMorph MMVP, where Interleave beats TextCoT by +4.4, and both V-CoT and VSD retain this full gain without generating images). Still, direct consumption may remain preferable under three conditions. First, intermediate states may supply genuinely new perceptual detail via deterministic tools (consistent with our VStar crop oracle in W2), or future architectures may explicitly enforce visual-state assimilation. Second, multi-step physical or pixel-precise simulations may exceed the capacity of brief text descriptions or hidden-state alignment. Third, downstream tasks may require rendered visual outputs directly. We will add this scope discussion to frame our conclusions around current native interleaved reasoners rather than all visual intermediates.

Reviewer_X855OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether interleaved multimodal reasoners use the intermediate images they generate. It diagnoses three released models through question-invariance measurements, interventions replacing visual thoughts, token perturbations, and attention analysis. The reported interventions produce little change in average accuracy. The paper then uses intermediate visual states as training-only supervision through Verbalized CoT (V-CoT), which converts them into textual descriptions, and Visual-State Distillation (VSD), which aligns student hidden states with an image-informed teacher on student-generated trajectories. At inference, the models still observe the original image but generate only textual reasoning. Across three training recipes and four recipe/backbone configurations, both methods improve average accuracy over interleaved and text-only baselines, while reducing inference latency relative to image-generating reasoning.

Critical strengths
  1. The paper investigates an important and underexamined question: whether generated visual thoughts actually contribute to reasoning. Interventions across three released models reveal limited average accuracy benefits from these images, providing useful evidence for reassessing the role of visual intermediates in multimodal reasoning.

  2. Using visual states as training-only supervision yields consistent average accuracy gains across four recipe/backbone configurations while eliminating intermediate-image generation at inference. The practical benefit is substantial: on ThinkMorph, VSD improves average accuracy over TextCoT by 2.1 percentage points and reduces latency from 61.6 to 23.3 seconds relative to Interleave.

Critical weaknesses
  1. Average accuracy alone is insufficient to conclude that the reasoner ignores visual content. The measured benefit also depends on the quality of the generated intermediate images, so limited gains may reflect inadequate image-generation capability rather than a failure to use visual information.

  2. Additional training and annotation costs remain insufficiently quantified. V-CoT requires model-generated descriptions across the three recipes . VSD adds a separate one-epoch reader-training stage, plus student rollouts and teacher forward passes at every update . Without total training GPU-hours and annotation costs, the practical trade-off between these upfront expenses and inference savings remains unclear.

  3. The methodological novelty is limited. V-CoT builds on visual-to-text and synthetic reasoning supervision explored in recent work such as LLaVA-CoT (Xu et al., 2025) and R1-Onevision (Yang et al., 2025), with the distinction of verbalizing generated intermediate states. VSD applies established representation-distillation ideas within an on-policy framework. The contribution therefore rests more on the empirical diagnosis and the use of intermediate visual thoughts as privileged supervision than on fundamentally new training methods.

Questions for authors
  1. Visual utilization versus generation quality: Beyond average accuracy, what evidence shows that the reasoner ignores intermediate images? Given that the oracle controls use constructed tasks, can you assess whether the limited benefit on the main benchmarks reflects poor intermediate-image quality or a failure to use informative visual content?

  2. Training and annotation costs: What are the total costs of generating V-CoT's descriptions and of VSD's additional reader training, student rollouts, and teacher forward passes? Please report annotation expenses and training GPU-hours, and indicate how much inference use is needed to offset these upfront costs relative to the interleaved baseline.

  3. Methodological contribution: What distinguishes V-CoT from previous visual-to-text and synthetic reasoning supervision approaches such as LLaVA-CoT (Xu et al., 2025) and R1-Onevision (Yang et al., 2025), and VSD from established on-policy distillation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer X855

We thank Reviewer X855 for the positive assessment and insightful questions, which helped us sharpen both our empirical evidence and core claim.

W1 and Q1. Visual utilization versus generation quality.

Evidence beyond average accuracy. Per-sample diagnostics on the released checkpoints confirm that generated visual thoughts exert minimal influence on final predictions.

Measurement (Figure 3)Text ThoughtVisual Thought
Answers changed when 64 tokens are corrupted26–52%6–16%
Mean attention per token (relative to visual)9–39×1×

Moreover, replacing the entire visual thought changes fewer answers than simply re-decoding the native condition with a different random seed.

Answers Changed (%), Diagnosis SubsetsThinkMorphZebra-CoT (Anole)
z replaced by noise (z←ε)13.362.1
z replaced by input image (z←x)11.957.7
Native condition re-decoded with another seed20.568.2

Generation quality versus utilization (Oracle interventions). Oracle visual thoughts z* separate generation quality from assimilation, and their effect depends critically on what z* adds. On our constructed tasks (Appendix D.6, Table 7), where z* renders the ground-truth intermediate reasoning state (e.g., highlighted bars or drawn maze paths), accuracy barely moves.

Constructed Sets, Accuracy (%)nGenerated zInput xOracle z*
ThinkMorph1,97663.262.762.9
Zebra-CoT (Anole)1,72043.643.043.2
MathCanvas1,17325.926.025.4

Following the reviewer's suggestion, we construct oracles on a main benchmark, VStar, using its ground-truth target boxes. For the 115 attribute questions, z* is a high-resolution crop of the target revealing fine detail lost in the model's downsampled view of x. For the 76 relative-position questions, z* outlines and labels each target on x without adding unseen pixels.

VStarNative (%)Δ z←εΔ z←xΔ z*, Attribute Crop (n=115)Δ z*, Marked Input (n=76)
ThinkMorph62.8+0.5−1.1+24.40.0
Zebra-CoT (Anole)28.4+1.1+1.6+15.7−3.9

Thus, the visual re-entry pathway itself is not broken. Models exploit z* when it supplies new high-resolution perceptual evidence, but fail to benefit when z* encodes visual reasoning annotations on existing content.

Stratifying by generated-thought quality. Using GPT-4.1 (given x, q, and z, blinded to answers and model identity), 35.0% of Zebra-CoT (Anole) visual thoughts across the full diagnosis set depict a valid task-relevant step. Re-evaluating interventions per stratum across the six non-paired benchmarks (excluding MMVP, whose pair metric precludes subsetting) yields the following results.

Zebra-CoT (Anole)NΔ z←εΔ z←x
Task-correct z101−0.28+9.16
Other z295−1.020.00

Even visually valid self-generated thoughts do not help, since self-generation conditioned on x cannot recover detail the model does not see. Replacing such a thought with noise changes nothing, whereas re-feeding x raises accuracy by 9.16 points, so the model gains more from a second view of the input than from its own intermediate state.

Refined claim. Current interleaved reasoners can read newly injected perceptual detail, but at inference they neither recover unseen pixels on their own nor assimilate visual reasoning annotations from image tokens. Because curated intermediate states in training traces do encode valid task structure, they are far more effective as training-time privileged supervision via V-CoT and VSD. We will refine the Abstract and Section 3 accordingly and incorporate both new analyses.

W2 and Q2. Training and annotation costs. Below we report end-to-end training, annotation, and inference accounting (benchmarked on H200 GPUs for unified training and inference comparison).

Cost ItemInterleaveTextCoTV-CoTVSD
Training GPU-hrs, ThinkMorph15.811.812.030.8 (17.3 reader + 13.5 student)
Training GPU-hrs, Zebra-CoT (Bagel)14.411.311.144.2 (31.0 reader + 13.2 student)
One-time caption API cost (~$0.008/desc.)––~$200 (ThinkMorph) / ~$341 (Zebra-CoT); $632 for all 78,954 across 3 recipes–
Inference speedup (ThinkMorph / Zebra-CoT (Bagel))1.0×–2.4× / 1.6×2.6× / 2.3×

By dropping the image-generation objective, V-CoT trains faster than Interleave across all four configurations (0.42–0.78× its GPU-hours), saving compute from query one after a modest one-time API cost. VSD incurs higher upfront training cost, which its faster inference amortizes rapidly.

VSD Break-Even vs. InterleaveThinkMorphZebra-CoT (Bagel)
Extra training GPU-hrs (incl. reader)15.029.8
Inference queries to break even3,3107,643
Equivalent passes over evaluation suite0.6× (5,330 qs)1.0× (7,484 qs)

W3 and Q3. Methodological contribution. We agree that our core contribution lies in the empirical diagnosis and reframing visual thoughts as privileged supervision. Both methods are intentionally clean instantiations, and once supervision targets the language pathway where the reasoner actually attends, simple mechanisms suffice to surpass interleaved generation. Our structure-matched controls (Table 9) isolate how both differ from prior work by holding supervision format, length, and optimization constant while removing only z.

Variant, Average Accuracy (%)ThinkMorphZebra-CoT (Bagel)
TextCoT66.343.4
V-CoT, captioner without z (input verbalization)66.043.5
V-CoT68.244.9
VSD, teacher without z (on-policy self-distillation)66.343.0
VSD, teacher with repeated x (extra images, no new info)65.241.8
VSD68.444.6
  • V-CoT vs. LLaVA-CoT and R1-Onevision. Prior methods verbalize the static input x into captions or formal descriptions to structure information already visible in x. V-CoT verbalizes dynamic intermediate states z absent from x (e.g., joined table layouts or auxiliary lines). Verbalizing (x, q, t0) without z at the exact same locus yields no gain, showing that the benefit stems from the intermediate visual state rather than input verbalization.
  • VSD vs. standard on-policy distillation. Rather than matching output logits from a larger teacher on identical inputs (Agarwal et al., 2024), VSD performs cross-modal privileged distillation (Vapnik & Vashist, 2009). An equal-sized EMA teacher, initialized from a z-conditioned reader (Table 2), aligns hidden states on student rollouts using privileged access to z. Removing z from the teacher eliminates the gain, showing that the improvement comes from the privileged visual states rather than from on-policy EMA regularization.

We will cite LLaVA-CoT and R1-Onevision in Section 2 and make these distinctions explicit.

Reviewer_AfszOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a dissenting perspective on the interleaved visual Chain-of-Thought (CoT) paradigm for Multimodal Large Language Models (MLLMs), empirically identifying a phenomenon termed the "Visual CoT Illusion." Specifically, the intermediate images generated by the model during reasoning lack task relevance and exert negligible causal impact on the final answer; instead, subsequent reasoning is overwhelmingly dominated by text.

Building on this insight, the authors propose a paradigm shift that visual thoughts from inference-time inputs into privileged supervision during training. Two strategies are designed to achieve this: 1) V-CoT, which explicitly converts visual states into textual descriptions; and 2) VSD, which implicitly distills visual knowledge into the language pathway via on-policy hidden-state alignment. Extensive experiments demonstrate that both methods not only outperform traditional interleaved image-text baselines in accuracy but also substantially reduce computational overhead by eliminating image generation and re-encoding during inference.

The core contribution of this work lies in its critical empirical analysis exposing the flaws of the interleaved visual CoT paradigm, along with the proposal and design of two superior model training strategies.

Critical strengths
  1. Through a diagnostic-first approach, the paper provides a comprehensive discussion of the issues surrounding interleaved visual reasoning methods; the diagnostic experiments are designed and extensive.

  2. Furthermore, through algorithmic design, the paper proposes two improved methods based on privileged supervision, which significantly enhance the model's reasoning performance. This research paradigm is well-conceived.

Critical weaknesses
  1. The paper lacks a comparative analysis and discussion regarding the applicability of the V-CoT and VSD methods. Since two improved strategies are proposed, the authors should discuss their relationship, whether one is superior to the other, and which method should be selected for specific tasks. Merely presenting two solutions without diagnosing and analyzing them makes the work feel incomplete. This stands in stark contrast to the systematic critique of visual reasoning presented earlier, ultimately diminishing the perceived contribution of the paper.

  2. The authors critique the interleaved visual reasoning paradigm solely from the perspective of final reasoning outcomes, and subsequently shift to improving these results via training-time interventions. In doing so, they may have overlooked other essential requirements of visual reasoning models, particularly interleaved ones, namely, providing visualized and interpretable reasoning processes.

  3. The current experiments are limited to small-sized models (7B) and rely exclusively on SFT. Deriving a universal claim such as the "Visual CoT Illusion" from this restricted setting is somewhat one-sided.

  4. When surveying related work on visual reasoning, the authors overlook the analysis of latent-space reasoning approaches. For example:

    • Monet: Reasoning in Latent Visual Space Beyond Images and Language

    • Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs

Questions for authors
  1. Have the experiments in Figures 2 and 3 been reproduced on interleaved visual CoT models trained with RL? Could the current results simply stem from insufficient SFT optimization rather than an inherent flaw of the model itself?

  2. The analysis in Section 3.3, particularly Lines 206–211, does not fully capture the phenomena shown in Figure 2 (Right). Beyond the conclusion guided by the authors, several other observations warrant further analysis and interpretation: 1) For MathCanvas, replacing the visual thought with the original image also degrades performance; 2) For ThinkMorph, replacing it with a noise image actually improves performance. What accounts for these discrepancies?

  3. Is V-CoT highly dependent on the captioner? Could this dependency negatively impact its effectiveness on complex spatial reasoning tasks that require fine-grained descriptions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Afsz

We thank Reviewer Afsz for recognizing our extensive diagnostic experiments and well-conceived paradigm. Below we address each point.

W1. Relationship between V-CoT and VSD. Systematically comparing both methods clarifies their complementary strengths across Table 1, Figure 5, and our new corruption study:

DimensionV-CoTVSD
Median gain over TextCoT, symbolic/geometric benchmarks (13 recipe–benchmark pairs in Table 1)+1.4+0.7
Median gain over TextCoT, perceptual/spatial benchmarks (12 recipe–benchmark pairs in Table 1)+1.3+1.9
Weaker backbone gain (Anole)+0.6+2.0
Gain w/ 50% noisy supervision (MathCanvas)+1.2 (47.8%)−0.8 (45.8%)
Inference speedup (ThinkMorph)2.4×2.6×
Readable intermediate stateYesNo

Selection guideline: Choose V-CoT when intermediate states consist of discrete, nameable elements (e.g., auxiliary lines or paths), when visual supervision is noisy, or when human-readable traces are desired. Choose VSD for fine-grained perceptual states that resist verbalization, weaker backbones, or strict latency budgets. We will add a dedicated comparison subsection presenting these guidelines.

W2. Interpretability of interleaved reasoning. While visualized reasoning is an important goal, an intermediate trace provides genuine interpretability only if it is causally faithful, that is, if it actually drives the prediction (Turpin et al., 2023; Lanham et al., 2023). Because replacing z with noise leaves interleaved accuracy largely unchanged, displayed images offer plausible rather than faithful explanations.

Conversely, V-CoT provides faithful, human-readable state tracking. Training on shuffled descriptions reduces accuracy by 4.1–6.9 points across recipes (Table 2). At inference, we take our V-CoT and Interleave models trained with the ThinkMorph recipe (Bagel-7B) and replace each model's intermediate step with the one it generated for another question, on the ThinkMorph diagnosis subset without MMVP:

ThinkMorph (183 questions)Native (%)InterventionΔ Acc.Changed (%)
V-CoT61.1Description of another question−9.126.8
V-CoT61.1Description deleted−9.726.3
Interleave63.4Image of another question−3.319.5

Swapping V-CoT's description degrades accuracy nearly as much as deleting it altogether, confirming strong reliance on its semantic content, whereas Interleave is far less sensitive to swapping its image. We do not argue against visual interfaces; rather, visualizations can be rendered for display without injecting inert image tokens into the reasoning loop.

W3 and Q1. RL-trained models, SFT optimization, and model scale.

RL-trained reasoners: We apply our full diagnostic protocol to Omni-R1 and Omni-R1-Zero, interleaved models trained via SFT followed by RL with a perception-calibrated reward:

ModelChanged: z←ε / z←x (%)Changed: Resampling (%)Δ Acc.: z←ε / z←x
Omni-R168.1 / 63.544.8+0.66 / −1.01
Omni-R1-Zero45.4 / 50.015.2−0.97 / −1.27

Although RL increases answer sensitivity to z beyond seed resampling, net accuracy stays within 1.3 points, so z steers the trajectory without improving reasoning accuracy.

SFT optimization: All diagnosed models use official, full-recipe checkpoints. Training our Zebra-CoT (Bagel) Interleave model for more epochs likewise leaves the diagnosis unchanged:

Zebra-CoT (Bagel) InterleaveΔ Acc. (z←ε)Δ Acc. (z←x)
1 epoch−0.38+0.29
2 epochs−0.46−0.35
3 epochs−0.39−0.25

Scale and scope: The open interleaved reasoners trained for visual reasoning, including ThinkMorph, Zebra-CoT, MathCanvas, and Omni-R1, are all built on 7B backbones. We will explicitly scope our claims to the models studied and revise the wording of "Visual CoT Illusion".

W4. Latent-space reasoning. Section 2 briefly cited continuous latent approaches including Monet (Wang et al., 2026b; alongside Bigverdi et al., 2025; Yu et al., 2025); we will expand this discussion to include Sketch-in-Latents (Tong et al., 2025). Like our paradigm, Monet and Sketch-in-Latents restrict reference visual states to training-time supervision (with Sketch-in-Latents discarding its auxiliary sketch-reconstruction module at test time). However, both still emit dedicated continuous latent tokens during inference, adding decoding steps and raising the same downstream assimilation question we study. In contrast, VSD distills visual knowledge directly into standard text hidden states with no latent visual tokens at inference. Applying our intervention protocol to continuous latent tokens is a natural extension that we will highlight.

Q2. Discrepancies in Figure 2 (right). Functional reliance requires both Δ z←ε ≪ 0 and Δ z←x < 0. All observed shifts are minor relative to our input-corruption positive controls (Table 6):

ModelΔ BlindΔ x←εΔ z←εΔ z←x
ThinkMorph (out-of-domain)−18.0−32.6+0.72+1.17
MathCanvas−12.8−20.9−0.61−0.58

(1) MathCanvas (Δ z←x < 0): Replacing z with x yields virtually the same slight dip as pure noise (Δ z←ε), showing that z offers no utility beyond noise; both lie well within one standard error (2.1–2.7 points; Appendix D.3). (2) ThinkMorph (Δ z←ε > 0): This slight gain corresponds to roughly two net questions out of 302, so removing the visual thought costs nothing. We will clarify these cases in Section 3.3.

Q3. Dependence of V-CoT on the captioner. V-CoT relies on the visual content of z rather than captioner reasoning: a captioner given (x, q, t0) without z yields no gain over TextCoT on ThinkMorph (66.0% vs. 66.3%; Table 9), and the prompt strictly restricts descriptions to visible evidence without access to t1 or y (Appendix H). We further test open-weight captioners and controlled description corruption on the MathCanvas recipe (Bagel-7B), reporting the average over its five benchmarks (MathCanvas-Bench, MathVision, MathVista-GPS, MathVerse-D and MathVerse-L) against our paper baselines (Interleave 46.8%, TextCoT 46.6%; Tables 1–2):

MathCanvas recipeCaptioner / supervisionAvg. (%)
V-CoT (paper)DeepSeek-V4.1-Flash49.6
V-CoTQwen3.5-27B49.3
V-CoTQwen3.5-8B48.7
V-CoT50% shuffled47.8
V-CoT (Table 2)100% shuffled45.5

Switching to an 8B open captioner reduces accuracy only slightly while remaining well above Interleave and TextCoT, whereas shuffling descriptions degrades accuracy monotonically. V-CoT therefore relies on faithful state descriptions rather than on a specific captioner. Even on fine-grained spatial tasks where ThinkMorph descriptions average just 34 tokens, V-CoT outperforms TextCoT on SAT (+1.3), MMVP (+4.6), and BLINK-Jigsaw (+2.8) (Table 1). Where continuous spatial details resist verbalization, VSD provides a captioner-free alternative.

I appreciated authors for their response. While it addresses some of my concerns, I maintain my original rating.

I suggest that the authors provide a more systematic presentation of the relationship between V-COT and VSD, as well as their respective applicability, within the analysis and discussion sections in the revision.

Dear Reviewer Afsz,

Thank you for your follow-up and the concrete suggestion. We have revised the paper to present the relationship between V-CoT and VSD more systematically:

  • Section 5.2 (Main Results), last paragraph: we compare the two methods along the type of visual state, backbone capacity, robustness to noisy supervision, and inference cost, and give a selection guideline. V-CoT is preferable for discrete, nameable states, noisy supervision, or when an inspectable record of the visual state is needed, while VSD is preferable for fine-grained perceptual states, weaker backbones, or tight latency budgets.
  • Appendix H (new, Table 11): a side-by-side comparison of the two methods, including the benchmark grouping and the noisy-supervision experiment on MathCanvas.

All changes are highlighted in blue in the revised PDF. We hope this addresses your remaining concern, and we would be grateful if you could take the revision into account when reassessing your score. If any concerns remain unresolved, we would be glad to continue the discussion and address them.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.