acceptodds
Withdrawn from ICLR 2027

Functionally Grounded Latent Distillation for Visual Latent Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual latent reasoning offers a promising alternative to explicit multimodal reasoning by performing intermediate computation in continuous representation space. Existing approaches commonly learn latent states by aligning them with perceptual features or privileged intermediate representations, yet representational similarity alone does not guarantee that the learned states preserve the reasoning computation they are intended to replace. We propose HCLR, a hierarchical latent distillation framework for visual latent reasoning that moves beyond representation alignment by learning latent states according to the downstream reasoning behavior they preserve. We formulate this principle as functional equivalence and progressively internalize explicit multimodal reasoning through hierarchical latent distillation. HCLR first learns compact latent substitutes for intermediate visual observations and then compresses the remaining textual and local-latent reasoning scaffold into a compact global latent trajectory. Privileged visual observations and reasoning scaffolds are used only during training and are subsequently removed through distillation, enabling the final model to perform intermediate reasoning entirely in continuous latent space using only the original image and question. Empirical results demonstrate that HCLR achieves strong performance across diverse real-world visual perception and reasoning settings, while exhibiting strong out-of-distribution generalization on challenging abstract visual reasoning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent Reasoning, Multimodal Large Language Models, Visual Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hnEiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a novel multimodal latent reasoning framework, HCLR, which replaces explicit multimodal reasoning (text interleaved with intermediate images) with a short sequence of latent vectors. Instead of aligning latent vectors to visual features, it trains them so that the model's downstream outputs match those produced with the explicit reasoning ("functional equivalence"). After an SFT warm-up on the interleaved reasoning dataset Monet-SFT-125K, this is done in two rounds of distillation: first the intermediate images are replaced by latent vectors; then, the remaining textual reasoning is compressed into 8 latent vectors, so that at inference the model reasons from only the image and question.

HCLR is compared with vanilla SFT and visual latent reasoning methods (LVR, SkiLa, Monet) on V*, HRBench4K/8K, and MME-RealWorld-Lite, and on the out-of-distribution VisualPuzzles. It achieves the best overall scores among the compared latent reasoning methods on V*, HRBench4K, and MME-RealWorld-Lite, and reports results comparable to tool-based models (DeepEyes, PixelReasoner).

Critical strengths
  1. The proposed approach, HCLR, is novel. Rather than aligning latent states to a predefined target representation, HCLR supervises them through the downstream behavior they induce, and progressively removes both intermediate visual observations and textual reasoning through a two-level distillation. This is a distinct direction compared to previous works.

  2. HCLR shows competitive results against the compared baselines. It outperforms the compared latent reasoning baselines on three of four benchmarks and performs comparably to tool-based models

Critical weaknesses

W1. The core motivation is not validated. The paper claims that aligning latent states to representations does not guarantee they preserve the reasoning they replace, but never shows this. There is no example of alignment-based latents failing in this way, and no measurement of whether HCLR's latents preserve downstream behavior better (e.g., output agreement with the explicit-reasoning model). Fig. 4 shows only that the latents matter. HCLR itself also relies on alignment: removing the alignment loss hurts more than removing behavioral KD (79.06 vs. 81.68 on V*, Table 3). Only one baseline per alignment family is compared (LVR, Monet; see Q1). So the gains cannot be attributed to functional equivalence.

W2. The paper is hard to follow. The core idea is simple but buried under overlapping terms ("functional equivalence," "local/global functional equivalence," "functional induction," "functional latent learning"). Table 3 uses names never defined in the method ("Behavioral KD," "Latent Alignment," "CE Only"), so the ablation is hard to map to Eqs. 1-4. The same motivation is repeated across the Abstract and Sec. 1-3.

W3. Performance may be bounded by the teacher. Each stage distills from a teacher trained on Monet-SFT-125K, but no teacher accuracy is reported, so it is unclear how much each stage loses. See Q3.

Questions for authors
  1. The paper contrasts HCLR with perceptual-alignment and privileged-representation-alignment methods but compares against only one of each (LVR, Monet). Could the authors include more recent methods they cite that use the same Qwen2.5-VL-7B base, such as VaLR (Jeon et al., 2026), CoVT (Qin et al., 2025), and UniVLR (Jiang et al., 2026), under the same protocol? These report higher numbers on overlapping benchmarks (e.g., VaLR 86.4 on V*; UniVLR 73.3 / 68.8 on HRBench4K/8K).

  2. Could the authors report (1) training cost (e.g., GPU hours per stage and in total) compared with LVR, SkiLa, and Monet, and (2) inference cost (latency, generated tokens, or FLOPs per sample) compared with both latent (LVR, Monet) and tool-based (DeepEyes, PixelReasoner) models?

  3. What is the accuracy of the explicit-reasoning model (M_warm) and of each intermediate teacher? This would show how much each distillation stage loses and whether HCLR is bounded by its teacher.

  4. Table 2 includes only LVR among the latent baselines and no tool-based models. How do Monet, SkiLa, DeepEyes, and PixelReasoner perform on VisualPuzzles?

Reviewer_mh95OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the representational similarity objectives commonly used in existing visual latent reasoning methods do not guarantee that latent states preserve the reasoning computation they are intended to replace. In this case, it proposes HCLR, a hierarchical latent distillation framework, to address this limitation. HCLR formulates functional equivalence as preserving downstream reasoning behavior rather than reproducing the prescribed intermediate representation, and progressively internalizes explicit multimodal reasoning through local functional latent learning followed by global functional latent reasoning. Experimental results show that HCLR performs competitively on diverse benchmarks and generalizes well to out-of-distribution abstract visual reasoning.

Critical strengths
  • This paper offers a well-motivated formulation of functional equivalence, characterizing latent states by the downstream reasoning behavior they preserve rather than by their similarity to the prescribed intermediate representation. This criterion is well aligned with the intended role of latent states as functional substitutes for explicit reasoning components.
  • HCLR adopts the coherent local-to-global hierarchy that progressively internalizes explicit multimodal reasoning. By separating the substitution of intermediate visual observations from the compression of the complete reasoning scaffold, this framework makes the learning problem more structured and tractable.
  • Evaluation combines diverse visual reasoning benchmarks with component ablations and latent intervention analyses. These complementary experiments provide evidence for both the overall effectiveness of HCLR and the functional contribution of its learned latent trajectory.
Critical weaknesses
  • The motivation of this paper is insufficiently substantiated. This paper argues that representational similarity does not guarantee that latent states preserve the reasoning computation they replace, but provides neither a theoretical analysis nor a controlled empirical comparison between representation-based and behavior-based supervision under matched settings.
  • The training procedure of HCLR is extremely complex, which requires a four-epoch warm-up followed by multiple local and global induction and distillation stages. Without compute-matched comparisons, it is unclear whether the reported gains stem from the proposed learning principle or simply from the additional training stages and computational budget.
  • The experimental evaluation is limited in scope. HCLR is evaluated using only a single backbone and a single training corpus, without experiments across different model families or scales. It is therefore difficult to determine whether the proposed method generalizes beyond the specific experimental setting.
Questions for authors
  1. Can the authors provide a controlled comparison between representation-based and behavior-based supervision under the same architecture, data, and training budget?
  2. Why are the MME-RealWorld-Lite results for HCLR inconsistent across Table 1, Table 5, and Figure 4?
  3. What are the training cost of HCLR and various baselines?
Reviewer_E7fAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HCLR, a hierarchical latent distillation framework for visual latent reasoning. Unlike existing methods that focus on representation alignment, HCLR introduces functional equivalence to preserve downstream reasoning behavior. The framework first learns local latent states to replace intermediate visual observations. It then compresses the remaining textual and latent reasoning steps into a compact global latent trajectory. Through progressive distillation, the final model performs reasoning without auxiliary visual observations or explicit reasoning traces. Experiments on multiple visual perception and reasoning benchmarks demonstrate competitive performance and improved out-of-distribution generalization.

Critical strengths
  1. HCLR introduces functional equivalence as an alternative to conventional representation alignment. Instead of forcing latent states to match predefined visual features or teacher representations, the method aligns downstream predictive distributions through KL divergence. This allows latent states to learn the functional role of intermediate visual observations rather than merely reproducing their representations.
  2. The hierarchical distillation framework separates the compression of intermediate visual observations from the compression of the complete reasoning process. Local functional latent learning first replaces auxiliary visual observations with compact latent blocks. Global functional latent reasoning then compresses the remaining textual and local-latent reasoning into a fixed-length latent trajectory. The bottleneck attention mechanism prevents subsequent reasoning from directly accessing the intermediate information being replaced.
Critical weaknesses
  1. HCLR uses KL divergence to align the downstream predictive distributions of explicit and latent reasoning paths. This is closely related to conventional behavioral knowledge distillation. The paper does not clearly distinguish the proposed functional equivalence from existing distillation objectives. It also lacks comparisons with alternative distillation strategies. This makes the methodological novelty of functional equivalence less convincing.
  2. HCLR involves five training stages, including explicit reasoning warm-up, local functional induction, local latent distillation, global functional induction, and global latent distillation. However, the ablation study does not compare this pipeline with direct global latent distillation from explicit multimodal reasoning traces. Such a comparison is necessary to establish whether the local learning stage is essential or whether a simpler training pipeline could achieve similar results.
  3. HCLR achieves only modest improvements over Monet on several benchmarks. The gains are 1.13 points on HRBench4K and 1.98 points on MME-RealWorld-Lite. Moreover, HCLR underperforms Monet on HRBench8K (65.00 vs. 66.88). The paper does not investigate this performance degradation. It also lacks multiple-seed experiments and statistical significance analysis. These omissions make the reliability of the reported improvements difficult to assess.
Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.