- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper focuses on how multimodal agents should organize visual information prior to generating actions. The authors argue that even if the image remains unchanged, the visual information genuinely relevant to different steps varies as the interaction history evolves. Therefore, they propose InnerEye: regenerating a transient, step-specific latent visual state before each action and discarding it once the action is completed. This method first endows the latent state with visual grounding via visual feature alignment, and subsequently integrates GRPO and OLAD to further gear this state toward next-action prediction.
Critical strengths
- The problem formulation is clear, and the method design aligns well with the motivation. The paper explicitly defines the latent state prior to each action as a transient, step-specific visual state that is reconstructed after the action is completed, rather than treating it as a continuously propagated latent memory. This setup nicely matches the motivation that "the same visual input may require attention to different information across different interaction stages."
- The validation of the latent state's effectiveness is relatively comprehensive. Through ablation and intervention experiments, the authors demonstrate that these latent states do not merely serve as extra compute slots, but are indeed utilized by the model for current action decision-making.
Critical weaknesses
- The novelty is relatively limited, and the distinction from existing latent visual reasoning works is not sufficiently established. Designs utilized in InnerEye—such as latent visual tokens, visual feature alignment, and multi-stage training—have become fairly common in recent literature. The primary modification here is placing the latent state immediately before each action and leveraging OLAD to gear it more directly toward action prediction. While this idea is reasonable, I am not yet fully convinced that it constitutes a substantial enough methodological innovation.
- Empirical validation of the core setting remains somewhat constrained. The paper emphasizes that distinct pre-action visual states should be formed at different interaction stages; however, in the multi-step experiments, the image itself remains static throughout, with only the interaction history varying. While this setup does provide a clean testbed to validate the authors' hypothesis, it still diverges from real-world GUI, Web, or embodied agents, where executing an action typically alters the visual scene observed in the subsequent step.
Questions for authors
- Could the authors provide a fair comparison with more closely related methods such as VaLR and Mirage, ideally using the same backbone, training data, and benchmarks? Such results would help clarify whether the observed gains primarily originate from the action-specific latent state and OLAD.
- Are there multi-step experiments where visual observations change alongside the executed actions? If available, this would more directly bolster the paper's motivation regarding general multimodal agents.