acceptodds
Withdrawn from ICLR 2027

InnerEye: Giving Multimodal Agents a Pre-Action Visual State

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal agents increasingly operate in interactive environments by interpreting visual inputs, maintaining interaction context, and generating executable actions. In many such agents, the explicit decision trajectory is progressively represented through autoregressive token sequences, while visual features remain available within the underlying multimodal model. However, agent decisions are inherently step-dependent: as interaction history evolves, the visual evidence relevant to the next action may change even when the visual input itself remains unchanged. This raises an agent-level question beyond visual perception itself: how should visual information be represented for the current decision before it is expressed as an action? We introduce InnerEye, a multimodal agent formulation that explicitly constructs a step-specific pre-action visual state before action generation. At each step, this transient state is freshly formed from the task instruction, visual input, and current interaction history, providing a latent representation that supports the upcoming action. InnerEye progressively trains this state from visual grounding toward supporting the upcoming action. Stage 1 performs visually grounded supervised fine-tuning by aligning latent hidden states with features from a frozen vision encoder while learning expert actions. Stage 2 further optimizes the grounded state through reinforcement learning with Online Latent Action Distillation, which transfers the pre-action visual states from a full-vision branch to an auxiliary branch without direct access to image tokens on successful rollouts, providing an action-oriented training signal without altering the standard full-vision inference process. Across two Qwen3-VL scales and three benchmarks spanning multi-step visual tool use and single-step visual action prediction, InnerEye consistently improves over both standard post-training baselines and a latent visual reasoning baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9eBBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on how multimodal agents should organize visual information prior to generating actions. The authors argue that even if the image remains unchanged, the visual information genuinely relevant to different steps varies as the interaction history evolves. Therefore, they propose InnerEye: regenerating a transient, step-specific latent visual state before each action and discarding it once the action is completed. This method first endows the latent state with visual grounding via visual feature alignment, and subsequently integrates GRPO and OLAD to further gear this state toward next-action prediction.

Critical strengths
  1. The problem formulation is clear, and the method design aligns well with the motivation. The paper explicitly defines the latent state prior to each action as a transient, step-specific visual state that is reconstructed after the action is completed, rather than treating it as a continuously propagated latent memory. This setup nicely matches the motivation that "the same visual input may require attention to different information across different interaction stages."
  2. The validation of the latent state's effectiveness is relatively comprehensive. Through ablation and intervention experiments, the authors demonstrate that these latent states do not merely serve as extra compute slots, but are indeed utilized by the model for current action decision-making.
Critical weaknesses
  1. The novelty is relatively limited, and the distinction from existing latent visual reasoning works is not sufficiently established. Designs utilized in InnerEye—such as latent visual tokens, visual feature alignment, and multi-stage training—have become fairly common in recent literature. The primary modification here is placing the latent state immediately before each action and leveraging OLAD to gear it more directly toward action prediction. While this idea is reasonable, I am not yet fully convinced that it constitutes a substantial enough methodological innovation.
  2. Empirical validation of the core setting remains somewhat constrained. The paper emphasizes that distinct pre-action visual states should be formed at different interaction stages; however, in the multi-step experiments, the image itself remains static throughout, with only the interaction history varying. While this setup does provide a clean testbed to validate the authors' hypothesis, it still diverges from real-world GUI, Web, or embodied agents, where executing an action typically alters the visual scene observed in the subsequent step.
Questions for authors
  1. Could the authors provide a fair comparison with more closely related methods such as VaLR and Mirage, ideally using the same backbone, training data, and benchmarks? Such results would help clarify whether the observed gains primarily originate from the action-specific latent state and OLAD.
  2. Are there multi-step experiments where visual observations change alongside the executed actions? If available, this would more directly bolster the paper's motivation regarding general multimodal agents.
Reviewer_a9ZDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the question of how visual information should be represented for the current decision before it is expressed as an action. The authors introduce InnerEye, a multimodal agent framework that explicitly inserts a fixed-capacity pre-action visual state before discrete action generation. Experimental results show that InnerEye consistently improves performance across the controlled settings, and extensive ablation studies further support the effectiveness of its key design choices.

Critical strengths
  1. The method consistently improves over SFT, GRPO, and LIVR across 2 model scales and 3 benchmarks. The ablation studies and latent-state interventions provide useful evidence that the learned representations contribute to action prediction.
Critical weaknesses
  1. Appendix A.7 shows that its Stage 2 training requires approximately 1.97× the training time of GRPO. The paper does not investigate whether standard GRPO could achieve comparable improvements under the same total computational budget.

  2. Prior methods such as VaLR [1] have introduced vision-aligned latent tokens before reasoning steps. It remains unclear whether the pre-action formulation provides sufficient conceptual novelty beyond existing vision-aligned latent reasoning methods.

[1] Byungwoo Jeon, et al. Vision-aligned Latent Reasoning for Multi-modal Large Language Model. ICML 2026.

Questions for authors
  1. Given the limited performance degradation from previous-step and step-shuffled latent substitutions on ToolVQA, to what extent do the learned states capture step-specific visual information instead of information shared across decision steps?

  2. Could the authors compare against GRPO trained under the same total computational budget?

Reviewer_9VLhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InnerEye, a framework that introduces a step-specific latent visual state before action generation in multimodal agents. The method uses a two-stage training approach, where the latent states are first aligned with visual features through SFT and then further optimized with GRPO and Online Latent Action Distillation (OLAD). The authors evaluate the method on three benchmarks using Qwen3-VL 2B and 8B models, showing consistent improvements over the baselines.

Critical strengths

The paper is generally well written and easy to follow. The proposed OLAD provides an interesting way to encourage latent representations to preserve information useful for action prediction. The authors also provide a reasonable set of ablation studies to analyze the effectiveness of different components.

Critical weaknesses

My main concern is the limited novelty and unclear necessity of the proposed latent states, as standard Transformers can already generate context-dependent representations based on the interaction history. The motivation for using RL is also not sufficiently justified, and it is unclear whether similar improvements could be achieved through supervised distillation without GRPO. The improvements over the baselines are relatively marginal considering the additional training cost, and the experiments are limited to fixed visual inputs or single-step action prediction. Overall, I do not find the current experimental results sufficiently convincing to support the significance of the proposed method.

Questions for authors
  1. What is the main advantage of introducing additional latent tokens? Since the Transformer can already generate context-dependent representations based on the visual input and interaction history, it is unclear why these additional latent tokens are necessary. Have the authors considered applying OLAD directly to the existing hidden states instead?

  2. Why is RL necessary for training the proposed latent representations? Since OLAD is essentially a self-distillation objective, could similar improvements be achieved by combining SFT with OLAD without GRPO? I think this would be an important baseline to justify the proposed training strategy.

  3. How do the authors justify the importance of step-specific latent states? According to Table 3, replacing the current latent states with those from the previous step only slightly decreases the performance. This seems to suggest that the step-specific property may not be as important as claimed.

  4. How do the authors justify the additional training cost given the relatively modest improvements? The proposed method nearly doubles the Stage 2 training time compared to GRPO, while most of the improvements are around 2 percentage points. It would be helpful to further discuss whether the benefits justify the additional complexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.