- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether continuous latent visual states genuinely contribute to reasoning in vision–language models. It proposes a unified framework that distinguishes whether these states are decodable, actually used by the released model, useful after matched readout training, and more effective than equal-sized pooled image features. Evaluations on CoVT and LVR keep states generated from the original image and question fixed while progressively degrading or removing the image, with several controls used to isolate the contribution of state content. The results show that LVR states become more useful as visual evidence weakens, whereas CoVT states exhibit a readout gap: they are decodable but initially provide little benefit, although stronger training recovers some utility. On CLEVR, matched training makes otherwise unused CoVT states highly useful. Nevertheless, at an equal-budget portable interface, pooled image features outperform the latent states.
Critical strengths
The paper’s main strength is its careful unified diagnostic framework. By separating decodability, actual use, and trained utility, and by combining image-degradation sweeps with matched controls and an equal-budget pooled-feature baseline, the paper avoids treating probe accuracy as evidence that a model causally relies on a latent state.
Critical weaknesses
-
Limited external validity. The paper studies only two released checkpoints, CoVT and LVR, both based on the Qwen2.5-VL backbone. Evaluation focuses mainly on counting, left–right relations, and synthetic CLEVR tasks. It is therefore unclear whether the findings generalize to other architectures or more complex visual reasoning tasks.
-
The experiments cannot identify what kind of information the latent states carry. Because the states are generated after the model has seen the question, the experiments cannot rule out the possibility that they encode an answer already formed for that question, rather than a reusable visual representation or a genuine visual reasoning process. The results show that these states can be decoded or used under certain conditions, but do not support stronger claims about their “visual reasoning” properties.
Questions for authors
-
Scope and generalizability. Which of the paper’s conclusions are intended to generalize beyond the two Qwen2.5-VL-based checkpoints examined here?
-
Answer code versus reusable visual representation. Do any of the reported or existing experiments distinguish a reusable visual representation from a question-specific answer code?
2 comments
Thank you to the authors for their detailed response and clarifications. The authors have provided additional experiments and further clarified the scope of the study, but these additions and clarifications do not fully resolve my concerns about the paper’s external validity. I maintain my original rating.
1. Scope and generalization. Four conclusions are existence claims, which one case settles.
A replacement test on a released model cannot show that a state is useless (CoVT on CLEVR: accuracy moves by 0.0 points when the states are replaced, and a trained reader gains 71.3).
A near-zero gain after training cannot show it either (CoVT on photographs: within half a point under one recipe, 8.6 points under a longer one).
A gain over a mean-state control does not show that the state holds more than an answer (LVR: 71.2% falls to 43.5% under a mirrored question's state).
Nor does it show an advantage over pooled image features of equal size.
The sizes of the gains are measurements on two checkpoints, and the paper now says so. We make no claim about more complex reasoning tasks.
We also added a second backbone. The one public checkpoint of this class outside Qwen2.5-VL is CoVT on LLaVA-1.5 13B. It writes its depth tokens only when asked for the depth map, so we ask for them in a first turn and pose the question in a second. The writer therefore never sees the question (Appendix I).
Decodable: a reader gives the object count from the tokens on 66.9% of 1,000 held-out CLEVR images (floor 13.0%, Table 26).
Unused: on 3,000 examples, replacing the tokens with another picture's changes 2.0% of answers, and accuracy moves by -0.2 [-0.7, +0.3] points.
Useful: a trained answer model gives the right count on 36.6% of 2,000 images without the picture, against 12.7% for the mean-state control, a gain of +23.9 [+21.4, +26.4] points over three pairs (Table 27, Figure 3b). Shape and color gain only 4.5 and 3.3 points, against our prediction.
2. Answer code or reusable representation. The new experiments separate them (Section 4.3, Figure 4, Table 1).
Photographs: an answer code, for both models. Under a mirrored question's state, LVR's answer model is right on 43.5% of 719 pictures, against 71.2% under the asked question's state and 53.0% under the mean state. When the writer is given no question, the gain is +2.0 points on counting and +0.4 on left-or-right, against +14.3 and +13.2. For CoVT the count falls from 61.3% to 49.7% under another counting question's state (mean state 45.2%).
CLEVR: content of the scene. The count is decodable from tokens written with no question (97.5% of 4,999 images), and the trained answer model gives it under any other question's state (76.0 to 81.5%, against 83.6% under its own state and 12.3% under the mean state).
We agree that none of this supports a claim of visual reasoning, and the paper makes none.