acceptodds
Under review as a conference paper at ICLR 2027

EgoR1: Interaction-Centric Latent Reasoning for Egocentric Future Interaction Forecasting

Abstract

Future interaction forecasting is a key capability for predictive multimodal intelligence, with broad applications in egocentric AI systems such as AR/VR assistance, human–robot collaboration, and embodied world models. However, despite the rapid progress of multimodal large language models (MLLMs) on general video understanding, their ability to predict imminent interaction events remains largely underexplored, and existing egocentric forecasting benchmarks are primarily designed for specialist models rather than MLLM-based reasoning. To bridge this gap, we introduce EgoR1-Bench1K, a new benchmark for evaluating future interaction forecasting in MLLMs. Given historical egocentric video frames and a multiple-choice query, the model must select the most likely next interaction event and localize the interacted object in the last observed frame. On top of this benchmark, we propose EgoR1, an interaction-centric latent reasoning framework that transforms latent reasoning from generic future summarization into progressive interaction-state grounding. Specifically, EgoR1 first distills semantically complete future interaction state through a Multimodal State Teacher, and then performs object-centric latent reasoning to progressively ground the predicted interaction onto the target object. We further apply a GRPO-based reinforcement learning stage to improve robustness and generalization. Extensive experiments show that future interaction forecasting remains challenging for current MLLMs, while EgoR1 consistently improves both future event prediction and object grounding over strong MLLM and latent reasoning baselines. These results suggest that structured latent reasoning is a promising route toward predictive egocentric intelligence in multimodal foundation models.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.