acceptodds
Under review as a conference paper at ICLR 2027

CrystaL: Seeing Less to Learn Better Visual Latents

Abstract

Learning visual latent states that effectively support answer generation remains challenging for multimodal large language models (MLLMs). Existing approaches often rely on explicit visual supervision or staged restrictions on image access, introducing dependencies on predefined visual targets or multi-stage training. We introduce CrystaL, a single-stage dual-path framework that learns task-relevant visual latent states through their utility for answer generation. An intact path derives latent states from the original image, while a parameter-shared corrupted path leverages the transferred states to predict the same answer under degraded visual input. Joint answer supervision and cross-path consistency encourage the latent states to preserve information useful for prediction when direct visual evidence is weakened. CrystaL requires neither external visual feature targets nor intermediate visual annotations, and retains only the intact path at inference. Across nine benchmarks, CrystaL achieves relative gains of 6.0% and 4.6% over Qwen2.5-VL-7B and Qwen3-VL-4B,respectively, supporting the effectiveness of the proposed dual-path training framework in learning visual latents that support answer generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_34iAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CrystaL, a single-stage dual-path framework for learning useful visual latent states in MLLMs without external visual teachers or intermediate visual annotations. The intact path processes the original image and generates latent states, while a parameter-shared corrupted path receives degraded visual input and uses the intact-path latent states through layer-wise latent replacement to predict the same answer. The model is trained with answer supervision together with predictive and attention consistency losses, while only the intact path is retained at inference.

Critical strengths
  1. Well-motivated information-routing formulation. CrystaL addresses an important weakness of latent visual reasoning: answer supervision alone does not guarantee that intermediate latent states become functionally useful. The dual-path design provides a simple way to increase dependence on latent information without prescribing what the latent states must encode or requiring a staged visual bottleneck. The layer-wise transfer of intact-conditioned latent states to the corrupted predictor is particularly well aligned with this objective.
  2. Strong functional evidence for latent utility. The controlled intervention experiments are a notable strength. Supplying intact-conditioned latent states consistently improves performance under clean, blurred, and shuffled-option conditions, providing stronger evidence than simply observing higher final benchmark accuracy. The broad improvements across high-resolution perception, spatial reasoning, and visual search, together with transfer to a second backbone, further support the robustness of the approach.
Critical weaknesses

The contribution of the CrystaL-specific routing mechanism is not fully disentangled from latent fine-tuning itself. The paper compares CrystaL with Direct-FT, but this control does not appear to provide the same latent computation slots. More importantly, the corruption ablation shows that training without corruption already reaches an average score of 70.9, compared with 72.0 for the default moderate blur setting. Likewise, corrupted-path CE alone reaches 71.0, while the full combination of KL and attention consistency reaches 72.0. Thus, a substantial portion of the total improvement may arise from introducing latent computation and task-specific fine-tuning rather than from the proposed corruption-based information-routing mechanism itself. This attribution issue is important because the latter is the central methodological claim.

Questions for authors
  1. Can the authors provide an intact-only latent fine-tuning baseline using the same number of latent tokens, training data, optimization budget, and autoregressive latent generation, but without the corrupted path, latent transfer, KL loss, or attention consistency? This comparison would directly clarify how much of the gain comes from latent computation itself versus the proposed dual-path routing mechanism.
  2. In the no-corruption setting, where the two paths receive the same visual input, which parts of the objective remain non-trivial in practice? In particular, how large are the KL and attention-consistency losses in this setting, and what explains the substantial improvement over the base model?
3 comments

(1) Disentangling CrystaL's routing mechanism from latent fine-tuning.

We clarify that Direct-FT in the paper actually corresponds to Intact-Only Latent Fine-Tuning under the 8-diverse-token setting, trained with the exact same data mixture and optimization recipe as CrystaL (4,000 optimizer steps, per-device batch size 4 across 4$\times$A40 GPUs, cosine schedule with 2×10−42\times10^{-4} peak LR and 0.05 warmup ratio). As shown in Figure 3, CrystaL consistently outperforms Intact-Only Latent Fine-Tuning across all 8 benchmarks.

Furthermore, inspired by the reviewer's comment, we added an Answer-Only Direct Fine-Tuning (Answer-Only FT) baseline (directly supervising final answers without latent tokens) to cleanly separate the benefit of latent tokens from our dual-path routing mechanism:

SettingCV-2DCV-3DBLINKHR-4KHR-8KRWQAAVG
Qwen2.5-VL-7B75.073.355.768.664.968.667.68
Answer-Only FT74.374.854.1368.565.168.967.62
CrystaL76.684.455.973.471.170.672.00

As shown above, Answer-Only FT yields virtually no gain over the base Qwen2.5-VL-7B model (67.62% vs. 67.68% AVG), indicating that naively fine-tuning on final answers does not meaningfully improve multimodal reasoning. In contrast, CrystaL boosts the average score to 72.00% (+4.38 points over Answer-Only FT), confirming the critical role of our latent reasoning paradigm.

(Q2) Objective behavior, loss magnitudes, and gains in the no-corruption setting.

In the no-corruption setting, because both pathways receive identical clean visual inputs and share weights, they produce identical latent representations and output distributions. Consequently, both the KL divergence and attention-consistency losses collapse to zero and become trivially inactive. Under this degenerate condition, the dual-path objective mathematically reduces to the Intact-Only Latent Fine-Tuning baseline discussed in (1) (effectively scaling the cross-entropy gradient by 2×2\times). Thus, the initial gain of no-corruption over the base model stems entirely from supervised fine-tuning with extra latent computation steps on clean images.

We also apologize for an imprecise wording regarding the "CE alone (71.0)" ablation. As defined in Eq. (3), the CE objective actually comprises the cross-entropy losses from both the intact and corrupted pathways (LCE=LCEintact+LCEcorrupt\mathcal{L}_{\text{CE}} = \mathcal{L}_{\text{CE}}^{\text{intact}} + \mathcal{L}_{\text{CE}}^{\text{corrupt}}), rather than the corrupted path alone. Because the intact-path term directly optimizes generation under the full visual context, the performance under "CE alone" naturally aligns closely with that of Intact-Only Latent Fine-Tuning. We will revise the text to make this formulation explicit.

Crucially, introducing visual corruption to activate our dual-path alignment (CrystaL) yields substantial further gains beyond this no-corruption baseline. We additionally evaluated no-corruption on V*Bench and POPE, alongside the overall average across all 8 benchmarks in Table 1:

SettingV*BenchPOPE8-Benchmark AVG (Table 1)
Base Model76.486.571.1
no-corruption (Intact-Only Latent FT)77.487.073.7
CrystaL (Ours)82.788.775.4

While no-corruption improves the 8-benchmark average from 71.1 to 73.7 via standard latent fine-tuning, activating our dual-path corruption and alignment mechanism pushes performance to 75.4 overall—including a notable +5.3 point surge on V*Bench (77.4 →\to 82.7) and +1.7 points on POPE (87.0 →\to 88.7). This confirms that the improvements over no-corruption are far from marginal, directly validating our dual-path alignment mechanism over naive latent fine-tuning.

Thank you for your response.

I will maintain my rating of 3. Unfortunately, I am not very familiar with this research area, so my confidence score is 2.

I am also somewhat puzzled as to why several papers assigned to me for ICLR this year fall outside my areas of expertise. I would appreciate it if paper assignments could be more closely aligned with my broader research area, as they have been in previous years.

AuthorsOpenReview ↗

Best wishes !

Dear Reviewer,

Thank you for your time and thoughtful feedback. We sincerely appreciate your consideration of our work and your transparency regarding your familiarity with this research area.

We also understand your concerns about the reviewer assignment process and hope that future assignments will be better aligned with your research expertise.

Thank you again for your valuable time and effort throughout the review process.

Reviewer_oPTcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to learn visual latent states that support multimodal question answering without external visual teachers or intermediate visual annotations. The authors propose CrystaL, a single-stage dual-path training framework with shared parameters. The intact path generates latent states from the original image and transfers them at each decoder layer to an auxiliary path that receives a degraded image. Results show improvements over the base model and Direct-FT using the same backbone and task data.

Critical strengths

The method has a clear motivation, and its training design aligns well with its objective. To address the possibility that models answer directly from images without using latent states, CrystaL combines representation generation from intact images with auxiliary prediction from degraded images. It requires neither external visual feature targets nor intermediate visual annotations, and the auxiliary path is removed at inference. These properties give the approach practical value.

Critical weaknesses
  1. The contribution of the core transfer mechanism has not been sufficiently supported. The paper does not include a control that removes latent state replacement across paths. The existing evidence supports the effectiveness of the overall method, but provides limited evidence for attributing the gains specifically to its information routing mechanism.
  2. Key experimental protocols and uncertainty estimates are insufficiently reported. The paper does not adequately specify the number of training steps, effective batch size, or modules to which LoRA is applied. Table 5 does not explain how latent access is disabled. The paper states that ablation results are averaged over repeated runs, but reports neither the number of runs nor standard deviations.
Questions for authors
  1. How does performance change when latent state replacement is removed while all other settings remain unchanged?
  2. Please clarify the experimental protocol for Table 5, including the backbone used and whether the rows differ only in interventions applied during evaluation. How exactly is the “Latent ✗” condition implemented?
  3. Please provide the missing experimental details identified in Weakness 2.
5 comments

(1) The contribution of the core transfer mechanism has not been sufficiently supported.

Consistent with our response to Reviewer m2de (Q2), we conducted an ablation study removing the state replacement operation (No-transfer). The results are summarized below:

SettingCV-2DCV-3DBLINKHR-4KHR-8KRWQAAVG
Qwen2.5-VL-7B75.073.355.768.664.968.667.68
No-transfer71.5174.2549.8766.564.069.4167.02
CrystaL76.684.455.973.471.170.672.00

We attribute this degradation (dropping from 72.00% to 67.02% AVG, falling below the base model at 67.68%) to representation decoupling between the two pathways: without state replacement, the latent tokens in the two branches evolve independently and diverge in semantics. Overall, our state replacement mechanism explicitly bridges the two pathways, forcing the clean-image latent tokens to internalize restorative visual semantics and thereby directly strengthening reasoning on standard, uncorrupted inputs.

(2) Protocol for Table 5.

Consistent with our response to Reviewer m2de (Q1), the backbone evaluated in Table 5 is our fully trained CrystaL model, where the "no latent" setting refers to an attention-masking strategy (i.e., masking subsequent answer tokens from attending to the generated latent positions). Furthermore, the results under the three latent perturbation settings are as follows:

Setting2D · ADE20K2D · COCO2D · All3D
latent-zero1.58%1.37%1.46%8.08%
latent-position26.22%23.23%24.55%17.67%
latent-noise67.93%82.24%75.94%84.83%

Replacing the latent states with zeros (latent-zero) or positional constants (latent-position) causes severe degradation, whereas injecting minor noise (latent-noise, ϵ\epsilon) leaves performance largely intact—confirming both the semantic necessity and the structural robustness of the learned latent representations.

(3) Key experimental protocols and uncertainty estimates are insufficiently reported.

Consistent with our response to Reviewer m2de (Q3), all main experiments and Direct-FT follow the exact same optimization protocol, differing only in the training objective: we train for 4,000 updates on 4$\times$A40 GPUs with a per-GPU batch size of 4 (effective batch size 16), using a cosine learning rate schedule (peak LR 2×10−42\times10^{-4}, 5% warmup). LoRA (r=16,α=32r=16, \alpha=32) is applied to the LLM backbone and multimodal projector while keeping the vision encoder frozen. Finally, every setting is run three times (seed 42) to mitigate hardware-level non-determinism, and all reported scores are three-run means; we will include the full run-level statistics in the revision and release all code and checkpoints.

Thank you for the detailed responses and additional experiments, which address many of my concerns. Could you please upload the revised manuscript incorporating these results and clarifications? I would appreciate the opportunity to review the changes before updating my assessment.

AuthorsOpenReview ↗

Revised manuscript uploaded!

Thank you for your prompt feedback and for acknowledging the additional experiments conducted during the rebuttal phase. We have uploaded the revised manuscript for your consideration.

Reviewer_m2deOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CrystaL, which trains a shared MLLM on intact and corrupted images. At each decoder layer, intact-path latent states replace corrupted-path states. Training combines answer supervision and consistency losses. Only the intact path is used at inference. The reported averages improve from 71.1 to 75.4 across eight metrics on Qwen2.5-VL-7B.

Critical strengths

The method uses one training stage and needs no external visual teacher or intermediate visual annotations. Experiments include several visual benchmarks, a Direct-FT control with the same backbone and data, and a second backbone.

Critical weaknesses
  1. The role of latent-state content is unclear. Section 3.1 uses fixed latent-token identities, but Table 5 does not explain the “Latent-x” setting. Removing the whole interface would change content, length, and computation together. Recent studies show that latent-token benefits can remain after strong content changes [1]. Please keep length and format fixed, replace states with zeros, position-specific constants, matched-scale noise, or states from other examples, and report answer changes.

  2. State replacement is not isolated. Figure 3 compares Direct-FT with the full method. Table 7 already includes replacement and only removes consistency losses. Please keep both image views, both answer losses, and answer-level KL fixed while disabling replacement. A length-matched ordinary-token or pause-token control would also help.

  3. The evaluation needs clearer reporting. Section 4.5 mentions repeated runs but gives neither the run count nor variation, despite several differences being about one point or less. Appendix A.1 also omits the Direct-FT update budget, batch size, and schedule. Training includes ADE20K-derived data, while evaluation includes CVBench 2D_ADE. This does not prove overlap, but warrants image-level checks. Please report matched budgets, seed-level results with uncertainty, and train–test overlap checks.

[1] Imagination Helps Visual Reasoning, But Not Yet in Latent Space, ICML, 2026

Questions for authors

Please see the Weakness part.

1 comment

(1) Latent-state content ablation.

We thank the reviewer for this constructive suggestion. In Table 5, to strictly control for sequence length and compute, our "no latent" baseline used an attention-masking strategy: the model still generates latent tokens, but subsequent answer tokens are masked from attending to them. This closely aligns with the reviewer's intuition of blocking latent information flow. Following the reviewer's advice, we further evaluated three direct perturbations on the latent hidden states h∈Rdh \in \mathbb{R}^d—replacing them with all zeros (latent-zero), position-dependent constants (latent-position), or injecting slight Gaussian noise (latent-noise, where ϵ∼N(0,0.01 RMS(h)2)\epsilon \sim \mathcal{N}(0, 0.01\,\mathrm{RMS}(h)^2) with RMS(h)=1d∑j=1dhj2\mathrm{RMS}(h)=\sqrt{\frac{1}{d}\sum_{j=1}^{d}h_j^2}):

Setting2D · ADE20K2D · COCO2D · All3D
latent-zero1.58%1.37%1.46%8.08%
latent-position26.22%23.23%24.55%17.67%
latent-noise67.93%82.24%75.94%84.83%

As shown above, substituting latent states with zeros (latent-zero) or positional constants (latent-position) severely degrades accuracy—with zero-replacement shifting hidden states far out-of-distribution (1.46% on 2D All). This proves that our latent tokens encode essential visual-semantic information rather than acting as mere computational or positional placeholders. Meanwhile, performance remains stable under slight perturbations (latent-noise), confirming the robustness of the learned latent representations. We will add both the clarification and these ablations to the revision.

(2) State replacement is not isolated.

We thank the reviewer for raising this insightful question. To rigorously evaluate the necessity of the state replacement operation, we conducted an ablation study comparing our full framework (CrystaL) against a variant without state replacement (No-transfer):

SettingCV-2DCV-3DBLINKHR-4KHR-8KRWQAAVG
Qwen2.5-VL-7B75.073.355.768.664.968.667.7
No-transfer71.5174.2549.8766.564.069.4167.02
CrystaL76.684.455.973.471.170.672.0

As shown in the table, removing state replacement (No-transfer) causes a substantial performance drop across benchmarks, reducing the average accuracy from 72.0% to 67.02%—even falling below the base Qwen2.5-VL-7B model (67.7%).

We attribute this degradation to representation decoupling between the two pathways: without state replacement, the latent tokens in the two branches evolve independently and diverge in semantics. Consequently, the training supervision on the corrupted-image path merely optimizes the model to compensate for degraded visual inputs in isolation, failing to transfer these learned visual-compensation priors back to the clean-image inference pathway. In contrast, our state replacement mechanism explicitly bridges the two pathways, forcing the clean-image latent tokens to internalize restorative visual semantics and thereby directly strengthening reasoning on standard, uncorrupted inputs.

(3) Clearer evaluation reporting.

Training budget (Direct-FT). Direct-FT and all main experiments share an identical setup: 4,000 optimizer steps, batch size 4 on 4$\times$A40 GPUs, cosine schedule (2×10−42\times10^{-4} LR, 0.05 warmup ratio), and LoRA (r=16,α=32r=16, \alpha=32). Only the training objective differs.

Repeated runs. Each configuration is trained three times (seed 42) to account for training non-determinism, and all reported numbers are three-run averages. We will include the run-level statistics in the revision.

Train–test overlap check. We use the exact training mixture from CoVT (github.com/Wakals/CoVT), where ADE20K-Depth is sourced strictly from the ADE20K training split. Since CV-Bench 2D_ADE is built from the ADE20K validation split, the training and test sets have little image overlap.

Release. Full training code and checkpoints will be open-sourced.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of our rebuttal

Global Response

We sincerely thank all reviewers (m2de, oPTc, 34iA) for their constructive feedback and recognition of CrystaL's dual-path visual latent learning paradigm. As several comments share common themes regarding our core mechanisms, latent representations, and experimental protocols, we summarize the primary concerns, our new empirical evidence, and the corresponding revision actions below. Detailed experimental tables and step-by-step analyses are provided in the individual responses.

ReviewerPoint RaisedOur Response & New EvidenceRevision Action
m2de (Q2), oPTc (Q1)Necessity of the cross-path state replacement mechanismConducted a No-transfer ablation removing state replacement. 6-benchmark average accuracy drops from 72.00% to 67.02% (falling below the 67.68% base model), proving state replacement is critical to prevent representation decoupling between branches.Add No-transfer ablation results and representation decoupling analysis.
m2de (Q1), oPTc (Q2)Semantic content of latent states & Table 5 "no latent" protocolClarified "no latent" masks answer-to-latent attention on the trained CrystaL model. Added three hidden-state perturbations: latent-zero (1.46% on 2D All) and latent-position (24.55%) collapse accuracy, while latent-noise (75.94%) remains robust.Clarify Table 5 evaluation setup and incorporate the three latent perturbation ablations.
34iA (Q1)Disentangling dual-path routing from standard latent fine-tuningClarified Direct-FT is Intact-Only Latent FT under identical budgets. Added an Answer-Only FT baseline (no latent tokens), which yields no gain over the base model (67.62% vs. 67.68% AVG), whereas CrystaL reaches 72.00% (+4.38%).Add Answer-Only FT baseline comparison and explicitly define Direct-FT.
34iA (Q2)Objective behavior and empirical gains over no-corruptionExplained that without corruption, alignment losses collapse to zero (reducing to Intact-Only Latent FT) and clarified "CE alone" optimizes $ L_{\text{CE}}^{\text{intact}} + L_{\text{CE}}^{\text{corrupt}}$. Evaluated no-corruption on V*Bench (+5.3% gain for CrystaL) and POPE (+1.7%), raising 8-benchmark AVG from 73.7% to 75.4%.Clarify Eq. (3) loss formulation for "CE alone" and include full 8-benchmark no-corruption results.
m2de (Q3), oPTc (Q3)Training budget clarity, multi-run statistics, and train–test overlapConfirmed unified protocol across all settings (4,000 steps, batch size 16 on 4$\times$A40, LoRA r=16,α=32r=16, \alpha=32, cosine LR 2×10−42\times 10^{-4}), 3-run averaging (seed 42), and strict train/val split separation on ADE20K following CoVT.Add complete hyperparameter details, 3-run statistics, split verification, and open-source code/checkpoints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.