- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether intermediate visual thoughts in multimodal chain-of-thought (CoT) reasoning models are genuinely used to support reasoning or merely serve as inert context. Through analyses of visual-thought similarity, perturbation-based interventions, and attention patterns, the authors argue that existing models often fail to effectively assimilate their generated visual thoughts into the reasoning process. To address this issue, the paper proposes two training strategies that treat visual thoughts as privileged supervision. The first, Visual Chain-of-Thought (V-CoT), converts intermediate visual states into textual descriptions and incorporates them into training traces. The second, Visual State Distillation (VSD), uses a teacher model with access to visual thoughts to guide a student model that does not observe them, aligning their hidden representations through on-policy distillation. Experiments across multiple model-training recipes, backbones, and visual reasoning benchmarks suggest that these approaches can improve reasoning performance while encouraging more effective use of visual information. Overall, the paper reframes intermediate visual thoughts from inference-time context into training-time supervision and presents practical methods for leveraging them.
Critical strengths
The paper questions a central assumption in multimodal chain-of-thought reasoning: that generating intermediate visual thoughts necessarily means the model uses them for subsequent reasoning. This distinction between producing visual information and effectively assimilating it is meaningful and motivates a focused investigation. Rather than introducing training techniques without examining the underlying problem, the authors first analyze whether visual thoughts influence model predictions through similarity measurements, perturbation-based interventions, and attention patterns. These analyses provide motivation for the proposed methods and help connect the empirical diagnosis to the training design. V-CoT provides an intuitive way to expose models to textual interpretations of intermediate visual states, while VSD transfers information from a teacher with access to visual thoughts to a student that does not. The two methods offer complementary perspectives on how visual thoughts can be exploited during training without necessarily requiring the student to consume them at inference time.
Critical weaknesses
The claim that interleaved visual thoughts are largely ignored is supported by intervention experiments on three open-source models: ThinkMorph, Zebra-CoT, and MathCanvas. However, it remains unclear whether this finding generalizes to other multimodal architectures, stronger models, or tasks in which intermediate visual states are essential. The current results may reveal a limitation of the evaluated models and training recipes rather than a general property of interleaved visual chain-of-thought.
The analysis of task relevance relies partly on CLIP embedding similarity, which may not adequately capture fine-grained structural or task-specific differences between intermediate visual states. Similarly, replacing a visual thought with Gaussian noise or the original input is informative but does not exhaustively test how the model uses visual information. These interventions may not fully distinguish between a genuinely irrelevant visual thought and a useful representation that the model cannot robustly process under the tested conditions.
Both V-CoT and VSD improve average performance over the evaluated baselines, but their effectiveness may depend on the quality and informativeness of the generated visual thoughts. V-CoT relies on an external captioner to verbalize intermediate states, while VSD depends on a teacher that can extract useful information from those states. The paper does not fully establish how robust these methods are when visual thoughts are noisy, incomplete, misleading, or difficult to describe in language.
The paper includes shuffled-description and blinded-teacher controls, which provide useful evidence that the content of visual supervision matters. Nevertheless, it would be helpful to further separate the benefits of visual information from those of additional supervision, changes in training trajectories, and optimization effects. In particular, the comparison between V-CoT, VSD, and TextCoT does not by itself establish which components account for the gains across different tasks and backbones.
Limited evaluation of statistical significance and robustness. The reported results average four training seeds and include standard deviations, which is helpful. However, the paper would benefit from more explicit statistical testing or confidence intervals for the improvements, especially on individual benchmarks where gains are small or inconsistent.
Questions for authors
The paper concludes that interleaved visual thoughts are largely ignored by downstream reasoning. How robust is this finding across a broader range of multimodal architectures, model scales, and visual reasoning tasks? Could the authors provide evidence that the observed phenomenon is not specific to the three evaluated model families or their training recipes?
The task-relevance analysis relies on CLIP embedding similarity, while the causal analysis replaces visual thoughts with noise or the original input image. How do the authors ensure that these measurements and interventions adequately capture the functional utility of intermediate visual states? In particular, could the conclusions change when using task-specific structural metrics or alternative interventions that preserve more of the visual thought's information?
Both V-CoT and VSD rely on informative intermediate visual thoughts, either through textual descriptions or teacher representations. How sensitive are the proposed methods to errors, omissions, and hallucinations in these visual thoughts? Could the authors systematically evaluate performance under controlled levels of visual-supervision noise and across tasks with different degrees of dependence on spatial or geometric information?
Although the shuffled-description and blinded-teacher experiments support the importance of visual information, how much of the improvement can be attributed specifically to the visual supervision rather than additional textual supervision or optimization effects? Could the authors include stronger controls that match the amount and structure of training supervision while varying only the availability of task-relevant visual information?
The experiments average results over four training seeds, but some benchmark-level improvements are relatively small and performance varies across datasets. Have the authors conducted paired significance tests or reported confidence intervals for the main comparisons? Are the improvements consistent across individual seeds, especially on benchmarks where V-CoT or VSD does not outperform TextCoT?
The paper motivates a shift from inference-time visual thoughts to training-time privileged supervision. Are there task categories or architectural conditions under which directly consuming generated visual states remains more effective than V-CoT or VSD? A more systematic characterization of these conditions would help clarify the scope of the proposed paradigm and avoid overgeneralizing the diagnosis.
3 comments
W2 and Q2. Adequacy of CLIP similarity and causal interventions.
Similarity metrics. Appendix D.2 (Table 5) calibrated global CLIP against this exact concern. Ground-truth intermediate states, differing only in task-relevant regions, lower CLIP similarity from 1.00 to 0.82 (−0.18), whereas generated S_q never falls below S_seed for any model. Following the reviewer's suggestion, we further evaluate similarity strictly inside the task-relevant region using three fine-grained metrics across sampled constructed sets (60, 60, and 39 images for ThinkMorph, Zebra-CoT (Anole), and MathCanvas), testing whether each generated thought best matches its own question's ground-truth state among the four questions per image.
| Matching Accuracy (%), Chance = 25% | ThinkMorph | Zebra-CoT (Anole) | MathCanvas | Ground-Truth States |
|---|---|---|---|---|
| Global CLIP (Table 5) | 29.4 | 31.2 | 27.8 | – |
| Task-relevant region, pixel MSE | 24.4 | 23.9 | 26.7 | 100.0 |
| Task-relevant region, SSIM | 24.0 | 21.6 | 24.6 | 100.0 |
| Task-relevant region, DINOv2 patch features | 26.2 | 22.7 | 27.1 | 100.0 |
Across all localized structural metrics, generated visual thoughts match their target question near chance, whereas the same metrics identify ground-truth states perfectly. Thus, question invariance is not an artifact of coarse global embeddings.
Interventions. Our framework spans full removal (noise ε), removing only information beyond x (input x; Section 3.3), and partial corruption (64-token perturbation; Section 3.4). Following the reviewer's suggestion, we add a question-swap intervention on a paired subset of the constructed tasks, replacing z_k with a thought z_k′ generated for another question on the same image to preserve visual style while altering only the task step, alongside the ground-truth oracle z* (which also changes accuracy by at most 0.5 points on the full sets of n=1,976 and 1,173 in Appendix D.6, Table 7).
| Constructed Sets Subset, Accuracy (%) | n | Native z | z←z_k′ (Question Swap) | z←z* (Oracle) |
|---|---|---|---|---|
| ThinkMorph | 240 | 40.8 | 41.7 | 40.4 |
| MathCanvas | 90 | 36.7 | 35.6 | 31.1 |
Neither swapping across questions nor supplying drawn ground-truth states yields accuracy gains. To test whether the reasoners can exploit any visual thought, we construct oracles on VStar from its annotated boxes. For the 115 attribute questions, z* is a high-resolution target crop revealing detail lost in downsampled x. For the 76 relative-position questions, z* outlines and labels targets on x without adding unseen pixels.
| VStar | Native (%) | Δ z←ε | Δ z←x | Δ z*, Attribute Crop (n=115) | Δ z*, Marked Input (n=76) |
|---|---|---|---|---|---|
| ThinkMorph | 62.8 | +0.5 | −1.1 | +24.4 | 0.0 |
| Zebra-CoT (Anole) | 28.4 | +1.1 | +1.6 | +15.7 | −3.9 |
The visual re-entry slot is functional when z* supplies new high-resolution perceptual evidence, but not when z* encodes reasoning annotations on existing content, and self-generated thoughts provide neither. We will refine Section 3 accordingly.
W3 and Q3. Sensitivity to noisy supervision and task dependence. Complementing the 100%-corrupted controls in Table 2, we evaluate 50% supervision corruption and open-weight captioners on MathCanvas (averaged over its five benchmarks).
| MathCanvas Recipe, Avg. (%) | Clean | 50% Corrupted | 100% Corrupted (Table 2) |
|---|---|---|---|
| V-CoT (descriptions shuffled) | 49.6 | 47.8 | 45.5 |
| VSD (teacher reads noise) | 48.5 | 45.8 | 44.2 |
| V-CoT w/ Qwen3.5-27B / Qwen3.5-8B captioner | 49.3 / 48.7 | – | – |
| TextCoT / Interleave | 46.6 / 46.8 | – | – |
V-CoT degrades gracefully, still beating TextCoT and Interleave under 50% corruption or with an 8B open captioner. VSD is more sensitive to heavy noise, dropping below TextCoT at 50% because a noise-conditioned teacher pulls student hidden states toward uninformative targets. Thus, V-CoT is preferable when reference visual states are noisy.
Grouping all 25 recipe–benchmark pairs in Table 1 by domain further clarifies task-level behavior.
| Median Gain over TextCoT | V-CoT | VSD |
|---|---|---|
| Symbolic/geometric/logical benchmarks (13 pairs) | +1.4 | +0.7 |
| Perceptual/spatial benchmarks (12 pairs) | +1.3 | +1.9 |
Both methods improve across both categories, with V-CoT excelling on discrete, nameable elements (e.g., auxiliary lines and paths) and VSD excelling on perceptual states that resist verbalization.
W4 and Q4. Isolating visual information from supervision structure and optimization. Appendix F.5 (Table 9) provides the exact structure-matched controls requested, holding supervision position, length, format, and optimization constant while removing only z.
| Variant, Average Accuracy (%) | ThinkMorph | Zebra-CoT (Bagel) |
|---|---|---|
| TextCoT | 66.3 | 43.4 |
| V-CoT, captioner without z (matched position, format, length) | 66.0 | 43.5 |
| V-CoT | 68.2 | 44.9 |
| VSD, teacher without z (matched reader, rollouts, EMA, loss weight) | 66.3 | 43.0 |
| VSD, teacher with repeated x (matched image count, no new info) | 65.2 | 41.8 |
| VSD | 68.4 | 44.6 |
None of these controls improves over TextCoT. Combined with Table 2 (where shuffled descriptions and blinded teachers drop accuracy by 4.1–6.9 and 2.7–6.5 points), this confirms that gains stem specifically from task-relevant information in z rather than extra text, extra images, or on-policy distillation mechanics.
W5 and Q5. Statistical significance and seed consistency. Appendix F.6 (Table 10) evaluates each recipe-level gain using a paired bootstrap (10,000 resamples drawn jointly over training seeds and questions) alongside per-seed win counts.
| Recipe | Variant | Gain over TextCoT | 95% CI | Seed Wins |
|---|---|---|---|---|
| ThinkMorph | V-CoT | +1.9 | [+0.9, +3.1] | 4/4 |
| ThinkMorph | VSD | +2.1 | [+1.2, +3.4] | 4/4 |
| Zebra-CoT (Bagel) | V-CoT | +1.5 | [+0.6, +2.8] | 4/4 |
| Zebra-CoT (Bagel) | VSD | +1.2 | [+0.5, +1.8] | 4/4 |
| Zebra-CoT (Anole) | V-CoT | +0.6 | [−0.4, +1.7] | 3/4 |
| Zebra-CoT (Anole) | VSD | +2.0 | [+1.1, +3.3] | 4/4 |
| MathCanvas | V-CoT | +3.0 | [+1.8, +4.5] | 4/4 |
| MathCanvas | VSD | +1.9 | [+1.1, +2.6] | 4/4 |
Seven of the eight gains are statistically significant and hold across all 4/4 seeds; on the weaker Anole backbone where self-generating descriptions is harder for V-CoT, VSD achieves a significant +2.0 (4/4 seeds). Across individual benchmarks, V-CoT beats TextCoT on 21/25 pairs and VSD on 20/25, including every benchmark on ThinkMorph (6/6) and MathCanvas (5/5), while the remaining Zebra-CoT pairs are ties or differ by at most 0.8 (V-CoT) and 1.1 points (VSD). We will move Table 10 to the main text.
Q6. When does consuming visual thoughts directly remain preferable? Across all 25 pairs in Table 1, Interleave never outperforms the better of V-CoT and VSD, even where visual intermediates matter strongly (e.g., ThinkMorph MMVP, where Interleave beats TextCoT by +4.4, and both V-CoT and VSD retain this full gain without generating images). Still, direct consumption may remain preferable under three conditions. First, intermediate states may supply genuinely new perceptual detail via deterministic tools (consistent with our VStar crop oracle in W2), or future architectures may explicitly enforce visual-state assimilation. Second, multi-step physical or pixel-precise simulations may exceed the capacity of brief text descriptions or hidden-state alignment. Third, downstream tasks may require rendered visual outputs directly. We will add this scope discussion to frame our conclusions around current native interleaved reasoners rather than all visual intermediates.
Response to Reviewer PkYp
We thank Reviewer PkYp for the thorough review and for recognizing the value of separating visual generation from downstream assimilation. Several requested controls and statistical tests are already in our Appendix, and we realize they were not prominent enough in the main text. Below we present them alongside new experiments and commit to moving the key tables (Tables 9 and 10) into the main paper.
W1 and Q1. Generality across architectures, scales, and tasks.
Architectures and training recipes. Our three diagnosed models span two backbones (Bagel-7B and Anole-7B), two re-entry interfaces (continuous VAE and ViT features versus discrete VQ tokens), and three domains (spatial, geometric, and multi-domain). We further apply our full protocol to Omni-R1 and Omni-R1-Zero, which add RL with a perception-calibrated reward after SFT.
Although RL increases answer sensitivity to z beyond seed resampling, net accuracy stays within 1.3 points, showing that z steers the trajectory without improving accuracy. Training our Zebra-CoT (Bagel) Interleave model for more epochs likewise leaves the diagnosis unchanged.
Tasks where intermediate states are essential. Appendix D.3 (Table 6) already evaluates ThinkMorph's in-domain jigsaw assembly, spatial navigation, and visual search benchmarks, whose training traces are built directly around visual intermediates.
Removing the input image costs 16.2 points on these tasks, whereas replacing the generated visual thought changes accuracy by less than 0.2 points.
Scale. The open interleaved reasoners trained for visual reasoning, including ThinkMorph, Zebra-CoT, MathCanvas, and Omni-R1, are all built on 7B backbones. We will explicitly scope our claims to the models studied and revise the Abstract and Conclusion accordingly.