acceptodds
Under review as a conference paper at ICLR 2027

TraViS: Learning Latent Visual Trajectories with Context-Adaptive Visual Targeting

Abstract

Multimodal large language models (MLLMs) often rely on localized visual evidence, yet existing supervision provides limited guidance on what intermediate latent states should visually predict. We introduce TraViS, a framework for learning latent visual trajectories with context-adaptive visual targets. Instead of using predefined intermediate supervision, TraViS enables each latent state to identify task-relevant visual evidence from the evolving multimodal context. Specifically, we propose a state-conditioned target construction mechanism, where a frozen visual teacher encodes the selected region into continuous visual targets, and a target feedback mechanism that injects these targets into subsequent latent computation during training. At inference, TraViS removes the teacher and performs self-feedback latent rollout with only a single global image encoding, avoiding additional visual processing. Extensive experiments on multiple backbones and eight multimodal reasoning benchmarks demonstrate that TraViS consistently improves performance, achieving average gains of over 2.5% while maintaining a favorable accuracy-efficiency trade-off.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Latent Visual Reasoning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LDfkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TraViS, a framework that learns latent visual trajectories with context-adaptive visual targets for multimodal reasoning. During training, each latent state selects task-relevant visual evidence, which a frozen visual teacher encodes into a continuous target to align with the state and guide subsequent latent computation. At inference, TraViS uses self-feedback latent rollout over a single global image encoding, eliminating the need for the crop teacher and per-step target construction. Across multiple backbones and eight multimodal reasoning benchmarks, TraViS achieves an average improvement of over 2.5% while reducing inference latency.

Critical strengths
  1. Context-adaptive visual supervision. TraViS allows intermediate latent states to select task-relevant visual targets from the evolving multimodal context, addressing the limited flexibility of predefined supervision in latent visual reasoning.
  2. Efficient inference. By restricting the visual teacher and target construction to training, TraViS supports self-feedback latent reasoning with a single global image encoding, improving performance while reducing inference latency.
  3. Improvement of empirical results. Experiments across multiple backbones and benchmarks demonstrate consistent improvements exceeding 2.5% on average, while the controlled fixed-crop ablation supports the effectiveness of context-adaptive targeting.
Critical weaknesses
  1. Fixed rollout length. Using eight latent steps for every question may limit efficiency and flexibility, as discussed in A.9 Limitations.
  2. The performance improvement compared with PixelReasoner is marginal.
Questions for authors

Please refer to the weaknesses.

Reviewer_frFHOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the general visual reasoning of MLLMs. The paper proposes the TraVis framework, which injects multiple sequential latent steps. A frozen teacher encoder is introduced to generate the clipped representation, acting as the alignment objective for the current latent state and the input to the last latent state. During inference, the model uses its own latent state as the visual objective. The results show that TraVis can improve visual reasoning capabilities across multiple benchmarks and backbones.

Critical strengths
  1. Building latent reasoning trajectories that span from coarse to fine is reasonable for improving fine-grained visual reasoning. The method adopts an external encoder as the supervision signal, avoiding collapsed representations that come from itself.
  2. Table 1 shows stable improvements on multiple benchmarks that vary across different spectra of visual tasks.
Critical weaknesses
  1. How the clipped regions at the beginning of the latent visual sequences are determined is unclear. The potential region should be closely related to the query, while the query can vary across different tasks.
  2. The latent sequence requires each hidden state to focus on a specific region of the visual input, which may introduce bias for the reasoning tasks, as not all visual tasks require reasoning from region to region. Limiting the reasoning trajectories with strong correlations can potentially undermine the model's originally emerged reasoning trajectories. The problem can especially severe during inference, as the model has to bootstrap the latent trajectories. The mechanism of TraVis should be clearly analyzed.
  3. Table 1 is confusing. The authors state that TraVis improves the model over the vanilla model, which leaves me wondering is the vanilla is the original ckpts or fine-tuned ckpts (SFT baseline) on the same amount of data.
Questions for authors

Please see the Critical Weaknesses above.

Reviewer_YtzdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed TraViS, a method for improving latent visual reasoning via context-adaptive visual targets. At each latent step, the model uses its current hidden state to select a relevant image crop based on attention scores over visual tokens. The selected crop is then encoded using a frozen visual encoder and mean-pooled to construct a visual target. Each latent hidden state is supervised to match its corresponding visual target through an alignment loss, while the target is also fed into the next latent step during training.

The paper trains TraViS using Visual-CoT 363K and evaluates on 8 multimodal reasoning benchmarks. The paper demonstrates the gains of TraViS vs base model across 4 models (Qwen2.5-VL-7B , Qwen3-VL-4B , Qwen3-VL-8B and LLaVA-OneVision-1.5-4B). The paper then compares TraViS (both SFT and GRPO variants) with existing multimodal reasoning methods, on Qwen2.5-VL-7B, demonstrating that TraViS outperforms these existing methods on average across the 8 benchmarks. The paper also provide ablations showing that adaptive visual targeting outperforms fixed targeting, along with qualitative visualizations of the selected crops across latent steps and inference efficiency comparisons.

Critical strengths

S1. Interesting and Efficient Method

The paper proposes an interesting and novel way to supervise intermediate latents to dynamically learn where to look. The inference is also efficient, as the crop encoding and supervision are only required during training.

S2. Broad Evaluation and Qualitative Visualizations

The authors evaluate TraViS across four different VLM backbones and eight multimodal benchmarks, showing consistent improvements over the base models. The qualitative visualizations in Figure 4 are also useful, as they show how the selected visual regions change across latent steps and help visualize the proposed mechanism.

Critical weaknesses

W1. Missing controlled baselines.

An important missing baseline is a data-matched direct SFT baseline (eg text only SFT on Visual-COT 363K) without latent tokens. This baseline will isolate the gains of training data vs the method design. A secondary baseline is text only SFT with latents added (ie add latents, but not supervising the latents). This will help isolate the gains of simply introducing extra latent compute space. Table 2 compares TraViS against existing multimodal reasoning methods, but the comparison of their training data and compute is limited. Without these baselines or a clearer comparison of the training data and compute used by existing methods, it is difficult to determine how much of TraViS's improvements come from the proposed method or the training data.

W2. Adaptive vs fixed target ablation

A core contribution of the paper is the use of adaptive crop targets. Therefore the fixed-target ablation in Table 5 is particularly important to test this contribution. However, the paper does not clearly explain how the fixed target is constructed (eg is it taken from the regions in Visual-COT? or is it random or based on some heuristic?). This is important because different choices of fixed targets could lead to substantially different interpretations of the results. For example, if the fixed target is randomly selected, the improvement could simply come from selecting more relevant visual regions, rather than adapting the targets across latent steps. This clarification is important for interpreting the results of Table 5 and understanding whether the improvements come specifically from adapting the visual targets across latent steps.

Questions for authors

Q1. Have the authors evaluated a data matched direct SFT baseline? This will clarify how much of the improvement comes from the method design vs the training data (refer to W1).

Q2. Could the authors clarify how the fixed target in Table 5 is constructed? (refer to W2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.