- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addressed the problem behind the intermediate generated visual latent of VLM to help VLM make decisions. To deeply investigate how these generated latents help the final answer, this paper tries to utilize several counterfactual methods such as zero replacement, inserting noise into the generated latents, and repeated tokens. It tests LLaVA-13B and Qwen 2.5VL-3B on different tasks like Mirage, Mull-Tokens, and CoVT to prove the observation.
Critical strengths
- The problem definition is clear and easy to follow.
- It tests several backbones on different benchmarks, which proves the proposed observation.
Critical weaknesses
- The main weakness of this paper is that this problem is not new. Such a phenomenon has already been reported in recent publications [1], which used similar methods, such as replacement, shuffling, repetition, and oracle substitution, to test the latent visual reasoning ability of VLMs. Thus, this will heavily limit this paper's contribution and novelty. Additionally, [2] also introduces the same evaluation strategy for latent reasoning. Based on this, it could be hard to recognize the contribution of this paper in extending a similar method to other datasets.
- The tested models are limited. It is strange that LLaVA-13B is tested while Qwen-2.5-VL is tested only with 3B. Why not use the same-size model for Qwen? Also, Qwen-2.5 and LLaVA-13B are both old versions; it should discuss newer models like Qwen 3 and InternVL.
- The evaluation protocol is not substantially proving the unusage of original visual tokens. Since the proposed method only change the intermediate visual latent, the original visual token can still be accessed and will complement such information when the intermediate visual latent is changed. This could be the reason that the performance doesn't change a lot when the intermediate latent is heavily changed.
- The exploration of negligible parts and generality remains limited. The paired analyses help, but insignificant differences don't show practically negligible effects. For example, Table 9 reports a continuous-model oracle improvement of +1.61 points with a 95% range of [−1.24, +4.48], which is still a meaningful benefit.
[1] Zhu, D., Wang, Z., Xiao, X., Jiang, H., Vahidian, S., Chao, W. L., ... & Gu, J. (2026). Leveraging Latent Visual Reasoning in Silence. arXiv preprint arXiv:2605.18641.
[2] Jin, H., Yang, S., & Wang, D. (2026). Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories. arXiv preprint arXiv:2607.06648.
Questions for authors
Please see the weakness, and especially answer the difference with existing works, which could be the main point of the novelty
Table numbers refer to the submitted PDF.
We thank the reviewer.
The reviewer asks how our study differs from Zhu et al. and Jin et al. Both were arXiv preprints at the submission deadline, concurrent with our work. We did not find oracle injection, the slot shuffle or our attention-value test (noise in the attention values of the visual tokens, keys kept; Table 12) in either paper, nor the following.
A comparison between discrete and continuous visual tokens in a controlled testbed. We train discrete and continuous models on the same depth task, on two backbones (LLaVA-13B and Qwen2.5-VL-3B). We know each model's target visual tokens, so we can inject them (oracle injection). Under the same tests, the discrete models are more sensitive to the content of their visual tokens than the continuous models, on both backbones (Tables 6 and 7). So the tests detect a model that uses its visual tokens.
The reviewer asks about the model sizes and newer models. The released models of the three off-the-shelf methods (Tables 1 to 3) are all built on Qwen2.5-VL-7B. LLaVA-13B is the backbone of Aurora, the method that our discrete models follow. Qwen2.5-VL-3B adds a second model family at the size our compute budget allows. We also trained a discrete model on Qwen3-VL-4B, with the data and schedule of our Qwen2.5-VL-3B discrete model. Change from the model's own visual tokens (points, 95% CI over images):
So the discrete Qwen3-VL-4B model also uses its visual tokens. This is one training run. Due to the time limit of the discussion period, we do not have a continuous Qwen3-VL model yet. The final version will include one.
The reviewer notes that the model can still use the image when the visual tokens are replaced. We ran two tests in which the model writes its visual tokens from the real image and then answers without the image, with its own visual tokens or with random ones.
A fully blank image or no image is too far from the training data: even the discrete models fall to chance, so we do not interpret those two settings.
Gain of own visual tokens over random visual tokens (points, 95% CI):
Without the image, the discrete models still gain from their own visual tokens. The continuous models show no detectable gain. So the image does not explain the flat result of the continuous models.
The reviewer notes that a change that is not significant can still be meaningful. The reviewer cites the oracle gain of +1.61 points [-1.24, +4.48] for the matched-budget continuous model (Table 9). The same table has a test with no oracle content: first-repeat, where every slot repeats one vector. It gives the same +1.61. So this gain does not come from the content of the oracle. For each change that is not significant, the final version will report the largest effect its interval allows.
We will add these tests and results and adjust the wording in the final version.