acceptodds
Under review as a conference paper at ICLR 2027

Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?

Abstract

Vision-language models (VLMs) are increasingly augmented with continuous or latent non-textual tokens intended to support "visual reasoning." Improved task accuracy alone does not show that models use these tokens: gains may instead arise from added context length, special-token anchoring, or training-time regularization. We formalize a diagnostic principle, Ablate-to-Validate, and instantiate it as the Token Replacement Test (TRT), a standardized suite of content-replacement ablations that measures how sensitive task accuracy is to a latent span's content while holding span placement, token budget, prompt, image and decoding fixed. We apply TRT to three off-the-shelf visual reasoning methods, Mirage, Mull-Tokens and CoVT. As a controlled testbed, we also train our own LLaVA-13B and Qwen2.5-VL-3B models to estimate relative depth by predicting and consuming continuous or discrete depth tokens. Our results show that accuracy gains can be a misleading proxy for latent-token reasoning: across all methods with continuous visual tokens, accuracy changes little when latent-token content is replaced by distribution-matched noise or shuffled across slots, and injecting an oracle yields no measurable gain, revealing a gap between "having a latent channel" and relying on its content. This gap is bolder for continuous spans: under the same test, accuracy with discrete depth tokens is more sensitive to their content on both backbones. TRT therefore shows how much and where the content of visual reasoning tokens matters for accuracy, and we recommend reporting such diagnostics as standard practice. Code will be released upon publication.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, latent tokens, visual reasoning, ablation, diagnostics, interpretability, depth reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PSmYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addressed the problem behind the intermediate generated visual latent of VLM to help VLM make decisions. To deeply investigate how these generated latents help the final answer, this paper tries to utilize several counterfactual methods such as zero replacement, inserting noise into the generated latents, and repeated tokens. It tests LLaVA-13B and Qwen 2.5VL-3B on different tasks like Mirage, Mull-Tokens, and CoVT to prove the observation.

Critical strengths
  1. The problem definition is clear and easy to follow.
  2. It tests several backbones on different benchmarks, which proves the proposed observation.
Critical weaknesses
  1. The main weakness of this paper is that this problem is not new. Such a phenomenon has already been reported in recent publications [1], which used similar methods, such as replacement, shuffling, repetition, and oracle substitution, to test the latent visual reasoning ability of VLMs. Thus, this will heavily limit this paper's contribution and novelty. Additionally, [2] also introduces the same evaluation strategy for latent reasoning. Based on this, it could be hard to recognize the contribution of this paper in extending a similar method to other datasets.
  2. The tested models are limited. It is strange that LLaVA-13B is tested while Qwen-2.5-VL is tested only with 3B. Why not use the same-size model for Qwen? Also, Qwen-2.5 and LLaVA-13B are both old versions; it should discuss newer models like Qwen 3 and InternVL.
  3. The evaluation protocol is not substantially proving the unusage of original visual tokens. Since the proposed method only change the intermediate visual latent, the original visual token can still be accessed and will complement such information when the intermediate visual latent is changed. This could be the reason that the performance doesn't change a lot when the intermediate latent is heavily changed.
  4. The exploration of negligible parts and generality remains limited. The paired analyses help, but insignificant differences don't show practically negligible effects. For example, Table 9 reports a continuous-model oracle improvement of +1.61 points with a 95% range of [−1.24, +4.48], which is still a meaningful benefit.

[1] Zhu, D., Wang, Z., Xiao, X., Jiang, H., Vahidian, S., Chao, W. L., ... & Gu, J. (2026). Leveraging Latent Visual Reasoning in Silence. arXiv preprint arXiv:2605.18641.
[2] Jin, H., Yang, S., & Wang, D. (2026). Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories. arXiv preprint arXiv:2607.06648.

Questions for authors

Please see the weakness, and especially answer the difference with existing works, which could be the main point of the novelty

1 comment

Table numbers refer to the submitted PDF.

We thank the reviewer.

The reviewer asks how our study differs from Zhu et al. and Jin et al. Both were arXiv preprints at the submission deadline, concurrent with our work. We did not find oracle injection, the slot shuffle or our attention-value test (noise in the attention values of the visual tokens, keys kept; Table 12) in either paper, nor the following.

A comparison between discrete and continuous visual tokens in a controlled testbed. We train discrete and continuous models on the same depth task, on two backbones (LLaVA-13B and Qwen2.5-VL-3B). We know each model's target visual tokens, so we can inject them (oracle injection). Under the same tests, the discrete models are more sensitive to the content of their visual tokens than the continuous models, on both backbones (Tables 6 and 7). So the tests detect a model that uses its visual tokens.

The reviewer asks about the model sizes and newer models. The released models of the three off-the-shelf methods (Tables 1 to 3) are all built on Qwen2.5-VL-7B. LLaVA-13B is the backbone of Aurora, the method that our discrete models follow. Qwen2.5-VL-3B adds a second model family at the size our compute budget allows. We also trained a discrete model on Qwen3-VL-4B, with the data and schedule of our Qwen2.5-VL-3B discrete model. Change from the model's own visual tokens (points, 95% CI over images):

replacementchange in accuracy
oracle injection+7.80 [+4.35, +11.47]
slot shuffle-12.10 [-17.31, -6.85]
random tokens-12.90 [-17.75, -8.03]
zero tokens-3.76 [-7.94, +0.39]

So the discrete Qwen3-VL-4B model also uses its visual tokens. This is one training run. Due to the time limit of the discussion period, we do not have a continuous Qwen3-VL model yet. The final version will include one.

The reviewer notes that the model can still use the image when the visual tokens are replaced. We ran two tests in which the model writes its visual tokens from the real image and then answers without the image, with its own visual tokens or with random ones.

  • Blank image. We replace the real image with a blank image. Only the circles and letters that mark the points stay.
  • No attention to the image. The real image stays in the prompt, but the model cannot attend to the image tokens while it writes the answer. The tokens before the answer could still attend to the image.

A fully blank image or no image is too far from the training data: even the discrete models fall to chance, so we do not interpret those two settings.

Gain of own visual tokens over random visual tokens (points, 95% CI):

testmodeldiscrete visual tokenscontinuous visual tokens
Blank imageQwen2.5-VL-3B+30.38 [+23.4, +36.9]0.00 [-2.2, +2.2]
Blank imageLLaVA-13B+12.20 [+7.3, +16.9]-0.54 [-3.2, +2.1]
No attention to the imageQwen2.5-VL-3B+28.76 [+21.6, +35.4]-1.34 [-3.9, +1.2]

Without the image, the discrete models still gain from their own visual tokens. The continuous models show no detectable gain. So the image does not explain the flat result of the continuous models.

The reviewer notes that a change that is not significant can still be meaningful. The reviewer cites the oracle gain of +1.61 points [-1.24, +4.48] for the matched-budget continuous model (Table 9). The same table has a test with no oracle content: first-repeat, where every slot repeats one vector. It gives the same +1.61. So this gain does not come from the content of the oracle. For each change that is not significant, the final version will report the largest effect its interval allows.

We will add these tests and results and adjust the wording in the final version.

Reviewer_6L6oOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether vision-language models genuinely rely on the content of continuous visual reasoning tokens. The authors propose the Token Replacement Test, which replaces intermediate tokens with different alternatives while keeping other inference conditions fixed. Experiments on three existing visual reasoning methods and a controlled depth reasoning testbed suggest that continuous tokens have limited influence on final prediction accuracy, whereas discrete depth tokens exhibit stronger content sensitivity.

Critical strengths
  1. The paper investigates an important question regarding the faithfulness and effectiveness of latent visual reasoning.

  2. The experiments cover multiple existing methods and model backbones. The use of distribution-matched perturbations, oracle injection, and controlled depth reasoning experiments provides useful empirical observations.

Critical weaknesses
  1. My primary concern is that the central research question, experimental approach, and main findings substantially overlap with existing work. In particular, Imagination Helps Visual Reasoning, But Not Yet in Latent Space[1] has already investigated whether latent visual reasoning tokens causally contribute to model predictions. Using causal mediation analysis and interventions on intermediate representations, they identify a latent-answer disconnect, showing that perturbing latent tokens often has little effect on final predictions. It further analyzes the information encoded by latent tokens and proposes CapImagine as an alternative approach. In comparison, the proposed TRT primarily organizes several token-replacement ablations into a standardized evaluation protocol. While additional controls such as distribution-matched noise, slot shuffling, and matched token budgets are useful, the paper does not sufficiently demonstrate what fundamentally new methodological insight TRT provides beyond previous studies. The closely related papers are also missing from the discussion of prior work, making the novelty and positioning of this submission unclear.

  2. The paper provides limited actionable insights or solutions beyond confirming an already identified problem. Another major concern is that the paper primarily diagnoses a failure mode without substantially advancing our understanding of how to address it. The main conclusion is that continuous visual reasoning tokens often have little measurable influence on downstream accuracy. However, similar observations have already been reported in previous research. Given this overlap, I would expect the paper to go further by identifying the specific mechanisms responsible for this failure, establishing conditions under which continuous tokens become useful, or proposing a concrete method to improve their utilization. Instead, the paper mainly recommends TRT as a standard diagnostic practice. Although evaluation-focused contributions can certainly be valuable without introducing a new model, the diagnostic itself should offer sufficiently novel insights or capabilities. Here, the combination of limited methodological novelty, largely confirmatory findings, and the absence of a concrete mechanism-level explanation or improvement strategy makes the overall contribution appear incremental.

  3. The conclusions about continuous versus discrete tokens are not fully supported by a sufficiently controlled comparison. The paper emphasizes that discrete depth tokens exhibit stronger content sensitivity than continuous tokens. However, these two representations differ in several important aspects beyond their discrete or continuous nature, including training objectives, token representations, prediction mechanisms, and decoding procedures. Although the additional matched-budget experiment in Appendix A.1 controls for some of these differences, it does not isolate the representation type as the sole experimental variable. Consequently, it remains unclear whether the observed difference arises from token discretization itself or from other architectural and optimization factors. The results support a difference in sensitivity between the implemented systems, but they are insufficient to establish a general advantage of discrete representations over continuous ones.

  4. The empirical evidence remains limited in scope, and some negative findings are difficult to interpret. The controlled testbed focuses exclusively on relative depth reasoning, with 372 evaluation questions derived from 124 images. It is unclear whether the same conclusions extend to more complex multi-step visual reasoning tasks where intermediate representations may provide information that cannot be easily recovered from the original input.Additionally, some interventions cause decoding collapse rather than ordinary prediction errors, suggesting that the measured effects may partly reflect distributional or decoding instability. The paper appropriately reports these failures, but they complicate the interpretation of token-content sensitivity. Finally, several comparisons involve relatively small accuracy differences and single training runs. The absence of statistically significant improvements should not be interpreted as evidence that the true effects are necessarily negligible.

[1] Imagination Helps Visual Reasoning, But Not Yet in Latent Space. ICML 2026

Questions for authors

See weakness

1 comment

Table numbers refer to the submitted PDF.

We thank the reviewer.

The reviewer asks what our study adds to Li et al. We did our study independently, over the same period. We will discuss Li et al. in the final version. Li et al. show that perturbing the visual tokens of Monet and Mirage barely changes the answer. We find the same for the continuous visual tokens we test. Our study adds two things.

  1. A comparison between discrete and continuous visual tokens in a controlled testbed. We train discrete and continuous models on the same depth task, on two backbones (LLaVA-13B and Qwen2.5-VL-3B). We know each model's target visual tokens, so we can inject them (oracle injection). Under the same tests, the discrete models are more sensitive to the content of their visual tokens than the continuous models, on both backbones (Tables 6 and 7). So the tests detect a model that uses its visual tokens.
  2. Released methods and two more tests. We test Mirage, Mull-Tokens, Mull-Tokens-GRPO and CoVT as released (Tables 1 to 3). Two of our tests are not in Li et al.: the slot shuffle, and a test that replaces the attention values of the visual tokens with noise and keeps their keys (Table 12).

The reviewer asks for a mechanism, conditions or a remedy beyond the diagnosis. We do not propose a new method. Our contribution is a test of the faithfulness of visual tokens: does the model use their content when it answers? Papers that introduce visual tokens should run it along with their accuracy metrics. The paper reports conditions and a mechanism.

  • Conditions. We trained 24 continuous models, with three encoders (SigLIP2, CLIP, DINOv2) and four numbers of visual tokens on each backbone (Table 5). Accuracy does not rise steadily with the number of visual tokens for any backbone and encoder.
  • Mechanism. The oracle visual tokens of the Qwen2.5-VL-3B continuous model hold answer information. A probe on them recovers the answer on 80.6% of the questions (Appendix D.2). Yet injecting them does not detectably raise accuracy (Table 13).

The reviewer notes that our comparison does not isolate the token type (discrete or continuous) as the only variable. We do not claim that discrete visual tokens are better in general (Appendix A.2). The final version will use the wording of Section 4.3 throughout: "We report these as differences in measured sensitivity between implemented systems."

The reviewer raises the scope, decoding collapse and single training runs. We answer each below.

  • Scope. Beyond the depth testbed, the paper tests the released methods on four more benchmarks: VSP, BLINK, SAT and CV-Bench (Tables 1 to 3). VSP is multi-step: it asks for a path through a maze. There, random visual tokens from Mirage's own distribution give the same accuracy as Mirage's own visual tokens.
  • Decoding collapse. Zero and i.i.d. random tokens push the visual tokens off their training distribution, so we do not read a drop under them alone as content use (Section 3). Our content tests are distribution-matched noise and the slot shuffle, which keep the scale and statistics of the visual tokens.
  • Single training runs. We trained the Qwen2.5-VL-3B discrete model three more times. Pooled over the three runs, oracle injection adds 8.96 points [+5.93, +11.99] (Table 7: +9.41). The slot shuffle, random tokens and zero tokens still lower accuracy.

We will add these results in the final version.

Reviewer_K1ecOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether the content of intermediate visual thought tokens contributes to task accuracy. Its Token Replacement Test (TRT) perturbs the intermediate span while holding the image, prompt, placement, token budget, and decoding configuration fixed. The authors evaluate existing visual reasoning methods and controlled continuous/discrete depth-token systems. Several continuous implementations show little accuracy sensitivity to content replacement, whereas the discrete depth systems are more sensitive. The main contribution is an intervention-based diagnostic of the relationship between intermediate representations and downstream performance.

Critical strengths
  1. The paper asks an important question about evidence for latent reasoning. Improved accuracy after introducing thought tokens does not, by itself, establish that their semantic content produces the improvement. Testing content sensitivity while controlling span placement and budget is a useful step beyond reporting benchmark gains. This diagnostic perspective has value independently of whether continuous representations are ultimately preferable to discrete ones.

  2. The controls reveal a meaningful difference between the evaluated systems. The study covers existing methods as well as a controlled depth task, and includes matched-budget VQ-VAE and attention-value interventions (Table 9; Appendix A.4). In particular, the discrete system remains content-sensitive even with the original image available. This positive control shows that image access does not make TRT universally insensitive, and supports a real difference in reliance on the intermediate channel across the implemented systems.

Critical weaknesses
  1. The experiments do not fully distinguish an unused channel from redundant visual evidence. The original image remains available when the thought-token content is perturbed. Low accuracy sensitivity therefore establishes limited marginal contribution under that condition, but does not determine whether the representation contains usable task information that overlaps with the direct visual pathway. The discrete positive control partly addresses this concern, yet it does not rule out different redundancy or information-routing patterns in the continuous system. The authors acknowledge that the span need not be a strict causal bottleneck (Section 5), but the stronger “Content unused” interpretation in Section 4.4 remains insufficiently supported. This distinction affects how much mechanistic understanding the paper contributes, rather than invalidating its measured accuracy results.

  2. The connection between content insensitivity and the gains attributed to continuous tokens needs to be clearer. In the controlled depth task, the best continuous model is below the no-auxiliary baseline on LLaVA (74.46% versus 75.54%) and only modestly above it on Qwen (68.28% versus 66.40%; Table 4). These experiments diagnose the implemented channels, but are less direct evidence that an established benefit of continuous reasoning does not depend on content. Existing-method results are consequently central to the paper’s broader significance. Table 11 contextualizes their reproduced scores, but it would help to establish the no-thought baseline, intact-channel gain, and residual gain after content replacement under a consistent evaluation setup. Without that connection, it is harder to distinguish an explanation of successful latent reasoning methods from a diagnosis of systems that did not establish a useful intermediate channel in the first place.

Questions for authors
  1. [Decision-relevant evidence] Can you distinguish redundancy with the original image from lack of usable content in the continuous span? One targeted control would generate the span from the intact image, then compare intact versus perturbed span content with and without direct image access during answer generation. Any such control should account for image information retained in other prefix/cache states and for distribution shift caused by masking. If this is not available, would you limit the conclusion to marginal accuracy sensitivity with the image accessible?

  2. [Clarification and decision-relevant evidence] For the existing methods, which continuous-token configurations exhibit a clear gain over their corresponding no-thought baselines in your evaluation, and how much of that gain survives content replacement under the same protocol? Please distinguish reproduced matched comparisons from baseline numbers taken from prior papers. This would clarify whether the main result diagnoses the source of an established gain or the failure to learn a useful channel.

1 comment

Table numbers refer to the submitted PDF.

We thank the reviewer.

The reviewer notes that the visual tokens may be redundant with the image rather than unused. We ran the control the reviewer proposed. The model writes its visual tokens from the real image. Then it answers without the image, with its own visual tokens or with random ones. We remove the image in two ways.

  • Blank image. We replace the real image with a blank image. Only the circles and letters that mark the points stay, because the question asks about them. The answer comes from a new forward pass. No key, value or hidden state of the first pass is reused.
  • No attention to the image. The real image stays in the prompt, but the model cannot attend to the image tokens while it writes the answer. The tokens before the answer could still attend to the image.

In four stronger tests (a fully blank image, no image at all, and two where the answer can attend only to the visual tokens), even the discrete models gain nothing from their visual tokens or give no readable answer. These four tests are too far from the training data, so we do not interpret them.

Gain of own visual tokens over random visual tokens (points, 95% CI):

testmodeldiscrete visual tokenscontinuous visual tokens
Blank imageQwen2.5-VL-3B+30.38 [+23.4, +36.9]0.00 [-2.2, +2.2]
Blank imageLLaVA-13B+12.20 [+7.3, +16.9]-0.54 [-3.2, +2.1]
No attention to the imageQwen2.5-VL-3B+28.76 [+21.6, +35.4]-1.34 [-3.9, +1.2]

Without the image, the discrete models still gain from their own visual tokens, so removing the image does not break the tests. The continuous models show no detectable gain. So the image does not explain the flat result of the continuous models.

The reviewer asks how much each method gains over a no-thought baseline, and how much of that gain remains with random visual tokens. We measured both under our protocol, on the same questions. The "Paper" column of Table 11 is copied from the original papers.

In every row, the baseline is the released base model, Qwen2.5-VL-7B-Instruct, run with the method's prompt and without visual tokens. It is not fine-tuned. We did not train a no-thought model for any method. So each gain includes the method's fine-tuning as well as its visual tokens.

For every method, gain = Acc(method) - Acc(base model), and change = Acc(Random, distribution-matched) - Acc(Identity), with the replacements of Tables 1 to 3 (all values in points, 95% CI over questions):

method, benchmarkgainchange
Mull-Tokens, SAT+16.33 [+10.6, +21.9]-0.33 [-2.3, +1.6]
Mull-Tokens-GRPO, SAT+17.67 [+11.9, +23.3]-0.67 [-2.0, +0.6]
CoVT, CV-Bench-3D+7.42 [+4.6, +10.2]-1.92 [-3.2, -0.7]
Mirage, VSP+68.00 [+63.0, +72.3]0.00 [-1.7, +1.7]

For Mull-Tokens and CoVT, the base model is also the baseline that Table 11 lists. The 95% CI of each gain is above 0. With random visual tokens, 98% and 96% of the two Mull-Tokens gains remain, and 74% of the CoVT gain. So most of each gain does not come from the content of the visual tokens. On BLINK and CV-Bench-2D, the gains are small or not significant.

The Mirage row is different. The base model was never trained on VSP and scores 8.25%, so the +68.00 is mostly task fine-tuning, not a gain of the visual tokens. The baseline of the Mirage paper is a model fine-tuned without visual tokens (Table 11). Its checkpoint is not public, so we could not run it. Random visual tokens leave Mirage's accuracy unchanged.

We will add these tests and baselines and adjust the wording in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.