acceptodds
Withdrawn from ICLR 2027

Do Latent States Really Reason? Making It Great Again through Information Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Textual chain-of-thought (CoT) improves multimodal reasoning but incurs substantial autoregressive decoding cost. Latent reasoning reduces this overhead by compressing intermediate reasoning into a fixed number of latent states. Despite recent progress, existing methods often implicitly assume that supervised latent states necessarily encode useful reasoning information. Our empirical results challenge this assumption, showing that latent states do not necessarily encode meaningful semantic information and that even skipping them during inference can have little impact on performance. These findings call into question whether current latent reasoning methods truly learn informative and interpretable latent representations. To address this, we propose ICLR (**I**nformation **C**ontrol for **L**atent **R**easoning), which explicitly controls information flow to make latent states both necessary for prediction and effective for reasoning. Specifically, we design a two-stage framework. In the first stage, we explicitly route all information through the latent states, making them necessary intermediates in the reasoning pathway. In the second stage, we further extract the visual evidence and textual semantics required for step-level reasoning and reinforce the latent states to encode this information, thereby enabling effective latent reasoning. Extensive experiments show that ICLR achieves state-of-the-art performance across multimodal benchmarks, surpassing existing latent reasoning baselines and latent visual reasoning approaches requiring auxiliary images and costly annotations. Compared to textual CoT methods, ICLR achieves an average performance gain of **3.6%** while delivering a **20.9×** generation speedup. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Chain-of-Thought Reasoning, Latent Reasoning, Computer Vision, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V5YzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper first demonstrates, through empirical analyses, that latent states in existing multimodal latent-reasoning methods may not encode meaningful semantic information. To address this limitation, it further proposes Information Control for Latent Reasoning (ICLR), a two-stage framework designed to make latent states both necessary for prediction and effective for reasoning. In particular, the first stage blocks direct access to the original visual input during answer generation, thereby routing task-relevant visual information through the latent states, after which the second stage restores visual access and distills step-relevant visual evidence and textual semantics into these states. Experiments show that ICLR outperforms existing latent-reasoning baselines, achieving state-of-the-art performance across various benchmarks.

Critical strengths
  • The paper investigates the important question of whether latent states in existing multimodal latent-reasoning methods genuinely encode meaningful semantic information and contribute effectively to prediction.
  • The paper proposes Information Control for Latent Reasoning (ICLR), a well-motivated two-stage framework that makes latent states both necessary for prediction and effective for reasoning.
  • Extensive experiments across diverse multimodal benchmarks demonstrate consistent improvements over existing latent-reasoning baselines, together with substantially lower generation overhead than textual CoT.
Critical weaknesses
  • The paper overclaims the scope and generality of its contributions. While it claims to systematically analyze existing latent-reasoning methods and presents ICLR as a general solution, all analyses and experiments are confined to multimodal latent reasoning, leaving its applicability to language-only latent reasoning unverified.
  • The construction of step-level supervision is highly heuristic. Each CoT is divided into three approximately equal-length spans, while the visual targets are extracted from eight fixed attention heads. The paper provides no ablation on these settings, leaving it unclear whether the reported gains arise from genuine step-level alignment or from specific heuristic design choices.
  • The experimental validation remains insufficient. Despite covering eight benchmarks, ICLR is evaluated using only one vision-language backbone. Moreover, the paper provides no budget-matched comparison between the complete method and its single-stage variants. These limitations make it difficult to assess the generalizability of ICLR and isolate the contributions of its two stages.
Questions for authors
  • How can ICLR be extended to language-only latent reasoning, where visual-path masking and patch-level visual distillation are unavailable?
  • What criteria are used to select the three latent steps, the approximately equal-length CoT segmentation, and the eight fixed attention heads?
  • Are all latent-reasoning baselines in Table 1 trained and evaluated with the same backbone, data, and training budget as ICLR? If not, how can the reported gains be attributed to the proposed method rather than differences in experimental setup?
Reviewer_dpGnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that existing multimodal latent reasoning models largely bypass their latent states, and proposes a two-stage training framework (ICLR) built on CoLT. Stage 1 zeroes out visual tokens during answer prediction to force information through the final latent state, while Stage 2 restores visual access and adds visual-attention and textual-semantic distillation, achieving a 79.3% average score across eight benchmarks.

Critical strengths
  1. Direct and important motivation. Investigating whether latent tokens are actually used for prediction or bypassed by direct visual attention addresses a central question in multimodal latent reasoning.
  2. Clean, annotation-free pipeline with fast inference. The method requires only standard CoT text annotations without extra bounding boxes or auxiliary images, and keeps a ~20× generation speedup over explicit text CoT at inference time.
Critical weaknesses
  1. In Table 1, all † rows are copied from the CoLT paper (Hu et al., 2026), except CoLT itself, which is replaced by a reproduced version (‡). The reproduction is 12.3 points lower on TextVQA (69.0 vs. 81.3) and 6.5 points lower on ChartQA (68.2 vs. 74.7), exactly the benchmarks where the proposed method claims its largest gains; against CoLT's published numbers, the +3.0 average improvement shrinks to +0.2. The reproduction protocol is not described at all. Section 4.5 shows that ICLR itself retains only Lfwd\mathcal{L}_{\text{fwd}} and Lbwd\mathcal{L}_{\text{bwd}} from CoLT and is trained with SFT only, i.e., without CoLT's internal loss Lint\mathcal{L}_{\text{int}} and GRPO stage. If the reproduction also omits these, it is an ablated CoLT and should be labeled as such; if it does not, the 12.3-point TextVQA gap needs to be explained. Either way, the claimed +3.0 over CoLT, which is the main empirical evidence for the method, cannot be interpreted.
  2. In Stage 1, the visual tokens are zeroed and the answer sees only the single final latent vector hKh_K (Uans(0)=[0;T]U_{\text{ans}}^{(0)} = [\mathbf{0}; T] with hKh_K, Eq. 2-3), which seems a flawed setup for learning step-by-step reasoning. Although compressing the visual information of an entire image into a single vector appears overly restrictive, Stage 1 is easy to optimize: hKh_K is already computed with full access to both the image VTV_T and the question TT in the first pass, and Stage 1 only optimizes the final answer loss. The model can therefore solve the question during the first pass and simply use hKh_K to carry the answer into the second pass, instead of encoding compressed visual evidence or multi-step reasoning. Moreover, h1,…,hK−1h_1, \dots, h_{K-1} are not directly visible to the answer and receive no supervision of their own in Stage 1. Furthermore, once Stage 2 restores full visual access, the model can attend to the image directly again.
  3. The two-stage training idea (blocking direct visual access during answer prediction in Stage 1 so that information must pass through latent tokens, then restoring full visual access in Stage 2) was already proposed in LIVR (Latent Implicit Visual Reasoning; Li et al., CVPR 2026, arXiv:2512.21218), which is neither cited nor discussed. Unlike LIVR, which masks attention within a single forward pass over K=16K=16 latent tokens at every layer, this paper passes only a single last-layer vector hKh_K into a zeroed-vision pass, which arguably makes it more prone to the answer-shortcut issue above.
Questions for authors

Q1: Why does the CoLT reproduction differ from CoLT’s reported results by 2.8 points on average and 12.3 points on TextVQA under the same backbone, data and K? Was it trained with the same recipe as ICLR (SFT only, without the internal loss and the GRPO stage)? Please describe the reproduction protocol, report CoLT’s published numbers alongside yours, and compare against them.

Q2: At inference, what is ICLR’s accuracy when the image is visible only through the latents (the Stage-1 bottleneck), and with the question alone (no image, no latents)? Please also report the Stage-1 model in these settings; this would instantiate the δ in Appendix A.1.

Q3: Does applying the LM head directly to h_K (logit lens) already yield the answer token? For the Stage-1 model, does replacing h_1 and h_2 change accuracy?

Q4: How does ICLR differ from LIVR’s bottleneck-then-restore training, and how does it compare empirically?

Q5:How were the 8 teacher heads (L23H4, L21H10, …) selected, and was any evaluation data used? How are the heatmaps of LVR and CoLT in Fig. 5b and Fig. 7 extracted, given that they have no refocus module? Fig. 7(a) shows a CLEVR example while its caption describes airway/trachea anatomy.

Reviewer_1f41OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper noticed an interesting problem in latent reasoning: replacing or skipping latent states often has little impact on performance, and the states remain similar across reasoning steps, lacking step-specific semantics. To solve this problem, the paper proposed ICLR (Information Control for Latent Reasoning), a two-stage framework to establish both the predictive necessity and the semantic effectiveness of latent states. In the first stage, ICLR restricts direct access to images, explicitly routing all related information through the latent states. In the second stage, access to visual evidence is restored, and visual and textual semantic distillation are applied to enhance step-specific representations. Extensive experiments have been done across various multimodal benchmarks, and ICLR outperforms latent reasoning baselines oThe paper demonstrates an important problem in latent reasoning: latent states can be bypassed and exhibit highly similar semantic representations, with evidence shown in Fig. 1. The paper also proposes an interesting direction for effectively leveraging latent reasoning in a more general way. Extensive experiments across eight multimodal benchmarks show consistent improvements over CoLT, the strongest baseline trained on the same data.n most benchmarks.

Critical strengths
  1. The paper demonstrates an important problem in latent reasoning: latent states can be bypassed and exhibit highly similar semantic representations, with evidence shown in Fig. 1. The paper also proposes an interesting direction for effectively leveraging latent reasoning in a more general way.

  2. Extensive experiments across eight multimodal benchmarks show consistent improvements over other baselines.

Critical weaknesses
  1. In the first stage, direct access to the image is blocked, but it is enabled again in the second stage. Since Stage 2 trains on more data (~89K vs. ~33K) with image access restored (Eq. 9), the model may relearn the shortcut, and there seems to be no explicit mechanism to prevent this.
Questions for authors
  1. Section 5.3 reports the intervention test for the final model. Could the authors also report the analyses in Fig. 1(a) and (b) for the model after Stage 1, to check whether the dependency is preserved after Stage 2?

  2. Some qualitative examples of decoded latent reasoning steps and visual attention maps are provided in the paper. Could the authors evaluate whether the decoded steps contain distinct and meaningful reasoning content, and whether the corresponding attention maps align with that text content?

  3. In Table 1, are Qwen3-VL (Direct answer) and Qwen3-VL (Textual reasoning) also trained on the same dataset (OneThinker)? If not, could the authors compare ICLR with direct-answer and textual-CoT models fine-tuned on the same OneThinker data?

  4. The authors use A^tea_k as the teacher distribution for distillation. How do the authors make sure A^tea_k is reliable and accurate? Similarly, for textual semantic distillation, how do the authors verify that <EOS> is a real semantic summary token?

  5. In Eq. 2, the authors replace visual input embeddings with zeros to block direct access to the original visual content. However, since Qwen3-VL uses DeepStack, where visual features are injected into intermediate layers, are these features also removed? Otherwise, the image may not actually be blocked.

  6. In Eq. 7, it seems z_k is aligned with μ_{k−1}. Why do the authors align z_k with span k−1 rather than span k? Does this mean z_1 and the final span are supervised only through L_fwd/L_bwd?

  7. What does "Neither" mean in Fig. 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.