acceptodds
Under review as a conference paper at ICLR 2027

What Is a Visual Thought Worth? When Continuous Visual States Help Vision-Language Models Reason

Abstract

Recent vision-language models augment the visual features of a pretrained backbone with continuous intermediate states intended to support reasoning. These “visual thoughts” can be trained to encode additional visual structure, such as segmentation or depth, or can emerge through iterative latent computation. The goal is to provide the model with a more task-relevant representation of the scene than the backbone features alone may offer. But does this added representation actually work as intended? In particular, can the answer be decoded from it, does the model itself use it, can training make it useful, and does it provide value beyond ordinary image features? We study these questions in two released models, Chain-of-Visual-Thought (CoVT) and Latent Visual Reasoning (LVR). We save each model’s visual states from the original image and progressively weaken or remove the image during answering, while keeping the states fixed. This lets us distinguish three properties that are often conflated: whether an answer can be decoded from a state, whether the released model uses its image-specific content, and whether an answer model explicitly trained to read the states becomes more accurate because of them, relative to an otherwise identical model trained with a non-image-specific mean state. We find that these properties can differ sharply. LVR states become increasingly useful as direct visual evidence disappears, improving counting by 14.2 points and left-or-right accuracy by 12.9 points when the image is removed. CoVT states reveal a different failure mode: the answer is decodable from them even when the answering model initially derives almost no benefit from them, while stronger readout training yields an 8.6-point counting gain without the image. On CLEVR, CoVT states are highly decodable but unused by the released model; after training the answer model to read them, their value grows dramatically as the image degrades, reaching a 71.3-point counting gain with a blank image. Yet at an equal-size interface, ordinary pooled image features outperform the latent states. Because the states are written after the question, part of these gains may reflect an answer already formed, which our runs do not rule out. Thus, adding structured visual states is not sufficient: their value depends on what information is written into them, how that information is exposed to the backbone, and whether subsequent computation learns to use it. Our results motivate latent-reasoning architectures that jointly optimize the representation and its readout, rather than treating the presence of richer intermediate features as evidence that they contribute to reasoning.

Primary area:
interpretability and explainable AI
Keywords:
visual thoughts, latent visual reasoning, continuous visual tokens, vision language models, multimodal reasoning, visual memory, representation analysis, hidden state interventions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dYNnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether continuous latent visual states genuinely contribute to reasoning in vision–language models. It proposes a unified framework that distinguishes whether these states are decodable, actually used by the released model, useful after matched readout training, and more effective than equal-sized pooled image features. Evaluations on CoVT and LVR keep states generated from the original image and question fixed while progressively degrading or removing the image, with several controls used to isolate the contribution of state content. The results show that LVR states become more useful as visual evidence weakens, whereas CoVT states exhibit a readout gap: they are decodable but initially provide little benefit, although stronger training recovers some utility. On CLEVR, matched training makes otherwise unused CoVT states highly useful. Nevertheless, at an equal-budget portable interface, pooled image features outperform the latent states.

Critical strengths

The paper’s main strength is its careful unified diagnostic framework. By separating decodability, actual use, and trained utility, and by combining image-degradation sweeps with matched controls and an equal-budget pooled-feature baseline, the paper avoids treating probe accuracy as evidence that a model causally relies on a latent state.

Critical weaknesses
  1. Limited external validity. The paper studies only two released checkpoints, CoVT and LVR, both based on the Qwen2.5-VL backbone. Evaluation focuses mainly on counting, left–right relations, and synthetic CLEVR tasks. It is therefore unclear whether the findings generalize to other architectures or more complex visual reasoning tasks.

  2. The experiments cannot identify what kind of information the latent states carry. Because the states are generated after the model has seen the question, the experiments cannot rule out the possibility that they encode an answer already formed for that question, rather than a reusable visual representation or a genuine visual reasoning process. The results show that these states can be decoded or used under certain conditions, but do not support stronger claims about their “visual reasoning” properties.

Questions for authors
  1. Scope and generalizability. Which of the paper’s conclusions are intended to generalize beyond the two Qwen2.5-VL-based checkpoints examined here?

  2. Answer code versus reusable visual representation. Do any of the reported or existing experiments distinguish a reusable visual representation from a question-specific answer code?

2 comments

1. Scope and generalization. Four conclusions are existence claims, which one case settles.

  • A replacement test on a released model cannot show that a state is useless (CoVT on CLEVR: accuracy moves by 0.0 points when the states are replaced, and a trained reader gains 71.3).

  • A near-zero gain after training cannot show it either (CoVT on photographs: within half a point under one recipe, 8.6 points under a longer one).

  • A gain over a mean-state control does not show that the state holds more than an answer (LVR: 71.2% falls to 43.5% under a mirrored question's state).

  • Nor does it show an advantage over pooled image features of equal size.

The sizes of the gains are measurements on two checkpoints, and the paper now says so. We make no claim about more complex reasoning tasks.

We also added a second backbone. The one public checkpoint of this class outside Qwen2.5-VL is CoVT on LLaVA-1.5 13B. It writes its depth tokens only when asked for the depth map, so we ask for them in a first turn and pose the question in a second. The writer therefore never sees the question (Appendix I).

  • Decodable: a reader gives the object count from the tokens on 66.9% of 1,000 held-out CLEVR images (floor 13.0%, Table 26).

  • Unused: on 3,000 examples, replacing the tokens with another picture's changes 2.0% of answers, and accuracy moves by -0.2 [-0.7, +0.3] points.

  • Useful: a trained answer model gives the right count on 36.6% of 2,000 images without the picture, against 12.7% for the mean-state control, a gain of +23.9 [+21.4, +26.4] points over three pairs (Table 27, Figure 3b). Shape and color gain only 4.5 and 3.3 points, against our prediction.

2. Answer code or reusable representation. The new experiments separate them (Section 4.3, Figure 4, Table 1).

  • Photographs: an answer code, for both models. Under a mirrored question's state, LVR's answer model is right on 43.5% of 719 pictures, against 71.2% under the asked question's state and 53.0% under the mean state. When the writer is given no question, the gain is +2.0 points on counting and +0.4 on left-or-right, against +14.3 and +13.2. For CoVT the count falls from 61.3% to 49.7% under another counting question's state (mean state 45.2%).

  • CLEVR: content of the scene. The count is decodable from tokens written with no question (97.5% of 4,999 images), and the trained answer model gives it under any other question's state (76.0 to 81.5%, against 83.6% under its own state and 12.3% under the mean state).

We agree that none of this supports a claim of visual reasoning, and the paper makes none.

Thank you to the authors for their detailed response and clarifications. The authors have provided additional experiments and further clarified the scope of the study, but these additions and clarifications do not fully resolve my concerns about the paper’s external validity. I maintain my original rating.

Reviewer_cVheOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether the continuous visual states produced by CoVT and LVR help answer questions. It saves states generated from a clear image, then blurs or removes the image and tests whether the model can still use those states. It distinguishes information being present in a state from the model actually using it.

Critical strengths
  1. State replacement, shuffled-state training and repeated experiments make the findings more convincing.

  2. I also appreciate the clear reporting of both successful and unsuccessful predictions.

Critical weaknesses
  1. Compared to existing studies, it remains unclear precisely which new problem this paper solves. [1] Prior work has already examined whether models use continuous states by replacing them; [2] prior work has also investigated how models are trained to read such states after image occlusion. Moreover, Appendix G.3 in version 2 of [3] already includes multiple image clarity conditions—not merely two endpoints.

  2. Current gains may still stem primarily from preserving the answer to the original question. The model generating the states has already seen the question. For instance, it may have already counted objects, and the subsequent model needs only retrieve that number to answer correctly even after image occlusion. This differs fundamentally from preserving visual information usable for other questions.

  3. Two comparisons in the reader experiments remain insufficiently clear. First, the shared reader used in Section 4.3 and Appendix E receives no question input, whereas continuous states are generated after the question is observed, while standard image features are not. Consequently, performance differences between these representations may reflect unequal access to question information rather than intrinsic representational superiority.

[1] Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?

[2] What's Holding Back Latent Visual Reasoning?

[3] Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning

Questions for authors

Please see weaknesses

3 comments

1. What is new relative to [1], [2], [3]. [1], [2] and [3] show that released models keep their answers when latent content is replaced. Our replacement test agrees and adds nothing, and Section 2 now says so. A fixed model cannot show whether an unread state is empty or only unread. We add four things (Table 19 sets them against prior work):

  • A reader trained on the released model's own generated states, at eight picture levels, against a control trained on a mean state. The gain that [3] does not find on fixed models appears as the picture degrades (+14.2 and +12.9 points for LVR without the picture). [2] finds usefulness for oracle latents only.

  • A reader fitted to the tensor the answer model receives, which separates an empty state from an unread one.

  • A comparison with pooled image features of the same size.

  • The question swap on a model trained to read the state. With the full picture we agree with [3]: 2.9% of answers change. Without it, the trained reader gives the other question's answer on 60.6% of pictures.

Our related work understated [3]. Sections 2 and 4.3 now describe its Appendix G.3 (state injection that recovers 7.8% of the gap, and the six-step image-quality sweep on fixed models) and its Appendix G.4 (the cross-question swap on fixed models).

2. The gain may be the answer to the original question. On photographs it is, for both models. On CLEVR it is not. We wrote the state under another question about the same image and asked the trained answer model the original one (Section 4.3, Figure 4). The rules and predictions were fixed before any state was written.

  • LVR, mirrored question (same two objects, opposite answer, 719 pictures): accuracy without the picture falls from 71.2% to 43.5%, a change of -27.7 [-32.1, -23.5] points, below the mean state (53.0%). The model gives the mirrored answer on 60.6% of pictures, against 27.3% under its own state. Counting behaves the same way (63.6% to 35.0%). We had predicted the opposite.

  • LVR, no question at writing time: answer models trained on such states gain +2.0 [+0.7, +3.4] points on counting and +0.4 [-1.0, +1.8] on left-or-right, against +14.3 and +13.2 (Table 24).

  • CoVT on photographs: the count falls from 61.3% to 49.7% under another counting question's state, a change of -11.6 [-15.6, -7.5] points (mean state 45.2%, Table 23).

  • CoVT on CLEVR: under the states of the five other questions the count is right on 76.0 to 81.5% of images, against 83.6% under its own state and 12.3% under the mean state. The models give the writer's other answer on at most 1.4% of images.

  • A CoVT checkpoint on LLaVA-1.5 13B writes its state before any question, and a trained answer model gains +23.9 [+21.4, +26.4] points on the count from it (Table 27).

The abstract, Section 4.1 and the conclusion now say this.

3. The common readers receive no question. We ran the test you describe, the same reader with the question for both sources. The order does not change: the reader gives the right count from LVR's vectors on 73.6% of pictures and from the pooled features on 50.9% (pooled minus latent -22.7 [-25.2, -20.2] points). Your objection holds in another way. When the writer is given no question, the same reader gets 55.2% from LVR's vectors, against 51.6% from the question alone and 73.3% from vectors written under the question (CoVT: 53.5%, 52.2% and 74.6%). So what the readers found was written under the question. Section 4.4 no longer uses the common readers to rank the sources (Table 25). The ranking rests on the test in which both answer models receive the question, where pooled features lead by 7.3 to 20.9 points (Figure 5a, b).

The review announces two comparisons and describes one. If the second concerns the question-aware reader of the submission's Table 8, the same result applies to it, and Section 4.1 says so. We will answer it directly if we misread.

Area_Chair_TsjaOpenReview ↗

AC reminder to Reviewer cVhe

Dear Reviewer cVhe, this is a reminder that the author-reviewer discussion period ends in a few hours.

Thanks, AC for #49456

I thank the authors for their rebuttal. However, the response does not resolve my concerns about the significance of the paper’s contribution, and I therefore maintain my original rating.

Reviewer_srBCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Recent studies have augmented vision-language models with continuous intermediate states for better performance. However, the improved accuracy alone does not reveal the internal mechanism of how the models decode or use these states. To solve this problem, this paper proposes a framework that investigates whether models could decode, use these states and whether these states could be useful. This paper further conducts diverse experiments on CLEVR and photograph data and reveal several interesting findings.

Critical strengths
  • This paper focuses on the internal mechanism of how models decode or use continuous visual states, which is an important task.
  • This paper proposes a framework from three perspectives: decodable, used, and useful. These three aspects are complementary.
  • This paper conducts experiments on multiple datasets and reveal several insightful findings.
Critical weaknesses
  • The paper writing needs to be improved a lot.

For example, the abstract and conclusion are very long, making it difficult to identify the key points.

Additionally, many key results appear in the appendix, while analysis apepar in the main body, such as Table 5 and Table 8, making it inconvenient for clear understanding.

Moreover, it might make the reader feel confused since the related work section is between the protocol section and the experiment section.

  • Discussion about why the conclusion differs across datasets.

This paper conducts experiments on CLEVR and photograph samples, and achieves quite different conclusions. What might be the reason for these differences? Is there high-level or general understanding across datasets?

  • The conclusion of whether the states could be made useful highly depends on the training recipe of the answer model.

As discussed in Section 4.1, the answer model trained longer starts to read CoVT's states. This raises concerns about the stability of the conclusions and methodology design.

Questions for authors

Please refer to the weaknesses.

2 comments

1. Writing. The abstract has 246 words (409 before), the conclusion is one paragraph, Tables 5 and 8 of the submission are now Tables 2 and 3 in the main text, and related work is Section 2.

2. Why the conclusions differ across datasets. CoVT's tokens hold object-level content of the scene whether or not a question is asked. CLEVR's count question asks for that content. The photograph questions do not, so there a state helps only through an answer written under the question. The new Table 1 shows this side by side. The evidence:

  • With no question, a reader recovers the object count from CoVT's CLEVR tokens (97.5%) and the count per shape (83.9 to 88.6%), but not colors (58.8% against a floor of 46.6%).

  • On CLEVR the trained pairs gain 71.4 points on the count, 8.0 on shape, 5.1 on parity, 2.3 on color, and nothing on color with shape or on the relation (0.0 and 0.1). The count gain survives under any other question's state (76.0 to 81.5% against 83.6%).

  • On photographs, when the writer is given no question, a reader finds almost nothing in CoVT's vectors (53.5% and 50.0%, against 52.2% and 57.2% from the question alone). Under another counting question's state the count falls from 61.3% to 49.7%, near the mean state (45.2%). LVR's states behave the same way.

This account fits the measurements. We do not claim it as a tested mechanism.

3. Dependence on the training recipe. Correct, and the protocol now says so: useful is defined relative to a reader and its training budget. A gain is stable evidence: all three trained pairs agree at every degraded level, and an LVR answer model trained on other pictures' states gains nothing (it ends 14.1 and 13.7 points below the state-trained model). A near-zero gain is bound to its recipe: for CoVT on photographs it is within half a point of zero at 1,500 steps and 8.6 points on counting at 4,500 steps and twice the rank. We therefore report it as a readout gap, not as a property of the state.

Thanks to the authors for the rebuttal, which addressed most of my concerns. However, I still have concerns regarding to the stability of the training recipe and its generality. Therefore, I will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers. The revision addresses three concerns.

1. The state is written after the question, so the gain may be an answer already formed (cVhe, dYNn). We tested this by writing the state under one question and asking the trained answer model another question about the same image (new Section 4.3, Figure 4).

  • LVR on photographs: the state holds an answer. Under the state of a mirrored question (same two objects, opposite answer), accuracy without the picture falls from 71.2% to 43.5%, below the mean state (53.0%). When the writer is given no question, the gain is +2.0 points on counting and +0.4 on left-or-right, against +14.3 and +13.2.

  • CoVT on photographs: also an answer. The count falls from 61.3% to 49.7% under another counting question's state (mean state 45.2%).

  • CoVT on CLEVR: the state holds the count. Under the states of the five other questions the count is right on 76.0 to 81.5% of images, against 83.6% under its own state and 12.3% under the mean state.

2. Two checkpoints on one backbone (dYNn). We added a CoVT checkpoint on LLaVA-1.5 13B whose state is written before any question. The count is decodable from it (66.9%, floor 13.0%), the released model does not use it (at most 2.0% of answers change when it is replaced), and a trained answer model gains +23.9 [+21.4, +26.4] points on the count without the picture (Figure 3b, Appendix I).

3. Presentation (srBC). The abstract has 246 words (409 before), the conclusion is one paragraph, related work is Section 2, and the two tables the text relied on are in the main text (Tables 2 and 3). The new Table 1 gives the four answers for each setting.

The predictions of every study were fixed before its runs were read. Tables 18 and 20 list each one with its outcome, including the ones that failed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.