acceptodds
Under review as a conference paper at ICLR 2027

EgoR1: Interaction-Centric Latent Reasoning for Egocentric Future Interaction Forecasting

Abstract

Future interaction forecasting is a key capability for predictive multimodal intelligence, with broad applications in egocentric AI systems such as AR/VR assistance, human–robot collaboration, and embodied world models. However, despite the rapid progress of multimodal large language models (MLLMs) on general video understanding, their ability to predict imminent interaction events remains largely underexplored, and existing egocentric forecasting benchmarks are primarily designed for specialist models rather than MLLM-based reasoning. To bridge this gap, we introduce EgoR1-Bench1K, a new benchmark for evaluating future interaction forecasting in MLLMs. Given historical egocentric video frames and a multiple-choice query, the model must select the most likely next interaction event and localize the interacted object in the last observed frame. On top of this benchmark, we propose EgoR1, an interaction-centric latent reasoning framework that transforms latent reasoning from generic future summarization into progressive interaction-state grounding. Specifically, EgoR1 first distills semantically complete future interaction state through a Multimodal State Teacher, and then performs object-centric latent reasoning to progressively ground the predicted interaction onto the target object. We further apply a GRPO-based reinforcement learning stage to improve robustness and generalization. Extensive experiments show that future interaction forecasting remains challenging for current MLLMs, while EgoR1 consistently improves both future event prediction and object grounding over strong MLLM and latent reasoning baselines. These results suggest that structured latent reasoning is a promising route toward predictive egocentric intelligence in multimodal foundation models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLLM, Ego-centric vision, STA, Computer Vision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8dc8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript proposes a benchmark called EgoR1-Bench1K and builds a baseline on this dataset.

Critical strengths
  1. The structure of this manuscript is easy to follow.
Critical weaknesses
  1. The differences and necessity of the introduced benchmark, EgoR1-Bench1K, are not adequately described. The paper does not clarify how EgoR1-Bench1K compares with other related datasets, nor does it explain what specifically distinguishes this benchmark from existing ones. A clearer comparison would help justify the need for a new benchmark.
  2. The effectiveness of the proposed Ego-R1 method is not sufficiently demonstrated. The comparisons are conducted only on the proposed benchmark. It is recommended that the authors compare Ego-R1 with other methods on existing datasets (e.g., Ego4D and Ego-Exo4D), and further include comparisons between Ego-R1 and other anticipation methods. Such experiments would provide stronger evidence for the method's effectiveness.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Benchmark Contributions and Effectiveness of EgoR1

1. Differences and contributions of EgoR1-Bench1K.

EgoR1-Bench1K is designed to evaluate future interaction forecasting in MLLMs, with several key differences from existing benchmarks.

  • Open-vocabulary interaction prediction: Unlike conventional anticipation benchmarks that use predefined verb/noun categories, EgoR1-Bench1K uses sample-specific, open-vocabulary interaction phrases. This supports more diverse and fine-grained descriptions of future interactions.
  • Challenging and visually grounded distractors: All distractor objects must appear in the last observed frame, and their associated interactions must be plausible given the historical context. This prevents models from relying on simple object-presence cues or eliminating obviously irrelevant options.
  • Joint event prediction and spatial grounding: Beyond selecting the future interaction, models must localize the corresponding target object in the last observed frame. This requires both semantic anticipation and object-level spatial reasoning.

Compared with existing benchmarks, EgoPlan-Bench focuses on MCQ-based next-step prediction, Ego4D STA evaluates future active-object localization and interaction prediction through structured outputs, and Ego-Exo4D emphasizes skilled activities across first- and third-person perspectives. EgoR1-Bench1K provides a unified MLLM-compatible setting that combines open-vocabulary future interaction selection, challenging distractors, and target-object grounding.

The benchmark also uses source-video-level data splits and removes contact-frame leakage, near-duplicate observations, and ambiguous samples to support reliable evaluation. We will make these distinctions more explicit in the benchmark comparison.

2. Effectiveness and contributions of EgoR1.

EgoR1 introduces progressive interaction-state grounding, which differs from generic latent visual reasoning by explicitly modeling the transition from a predicted future interaction state to its target object.

Specifically, Future Interaction State Distillation uses future contact frames and paired interaction text to construct interaction-aware latent supervision. Object-Centric Latent Reasoning then conditions on the predicted state and uses region alignment and contrastive grounding to distinguish the target object from other plausible candidates.

The experimental results support this design:

MethodAcc.mIoU
Qwen2.5-VL-7B CoT-SFT0.62690.3216
Mirage0.77570.3615
LVR0.79000.3959
EgoR10.84850.4219

EgoR1 improves over LVR by 5.85 percentage points in accuracy and 0.0260 mIoU. The component ablations further show that combining future interaction state distillation and object-centric reasoning achieves 0.8222 Acc. and 0.4023 mIoU before GRPO. GRPO then improves these results to 0.8485 Acc. and 0.4219 mIoU.

These results highlight the complementary roles of state-level prediction and object-level grounding, showing that the gains come from task-aligned latent supervision and structured reasoning rather than simply introducing additional latent tokens.

3. Relationship to specialist anticipation methods.

EgoR1 and specialist anticipation methods address related prediction problems through different modeling approaches. Traditional methods typically use dedicated prediction heads for verbs, nouns, bounding boxes, or temporal offsets. EgoR1 instead performs future interaction prediction and object grounding through a unified generative MLLM, using interaction-centric latent representations as intermediate reasoning states.

Our goal is not to replace specialist anticipation architectures, but to investigate how general-purpose MLLMs can acquire interaction forecasting capabilities through structured latent reasoning. We will further clarify this distinction and include comparisons with representative anticipation methods under compatible evaluation settings.

EgoR1-Bench1K provides a challenging evaluation setting for joint semantic and spatial forecasting, while EgoR1 demonstrates how interaction-centric latent reasoning can improve both capabilities within a unified MLLM framework.

Thanks for the clarification. However, W1 remains qualitative, i.e., no direct comparison with existing benchmarks is given. W2 is unresolved: all results are still on EgoR1-Bench1K only, with no Ego4D/Ego-Exo4D experiments and no comparison with specialist anticipation methods (Mirage/LVR are not such methods). Promised future comparisons do not provide current evidence.

Reviewer_kPjKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EgoR1-Bench1K, a benchmark that was constructed from Ego4D and EPIC-KITCHENS for multimodal language models (MLLMs) future interaction forecasting. The task is to take 16 second past video and multi-choice options to select the upcoming interaction and bounding box that localizes the target object. The author also proposes EgoR1 that employs a three-stage latent reasoning pipeline with two specialized latent tokens to encode future contact frame/text and object regions. The experiments show that EgoR1 can have improved accuracy and mIOU over fine-tuned VLMs and latent visual reasoning baselines.

Critical strengths
  • The coarse-to-fine design of latent visual tokens align the global interaction state and local object target, and the ablation also shows the effectiveness.
  • The main results show good empirical performance gain on the proposed benchmark.
Critical weaknesses
  • The novelty of the paper is limited. The benchmark is a reformatting of the existing egocentric dataset into MLLM multi-choice format. The EgoR1 pipeline is very similar to other latent reasoning works but with special tokens.
  • While the reformulation of egocentric video tasks to multi-choice matches what MLLM expects, this simplifies the anticipation task which makes it unclear about how the benchmark can improve model’s interaction anticipation performance.
Questions for authors
  • It will be important if the authors can show how the benchmark is not merely a reformulation of the existing dataset, and how the proposed method/model shows new insights for interaction anticipation compared with the prior methods in Ego4D and EPIC-KITCHENS.
2 comments
AuthorsOpenReview ↗

Clarification on Benchmark Novelty and Interaction-Centric Latent Reasoning

1. Benchmark novelty beyond MCQ reformulation.

EgoR1-Bench1K is not simply a conversion of existing datasets into multiple-choice questions. Its main contribution lies in three aspects:

  • Open-vocabulary interactions: Unlike conventional closed-set anticipation tasks based on predefined verb/noun categories, our candidate options are open-vocabulary interaction phrases generated for each sample. This allows more diverse and fine-grained interaction descriptions, although the final prediction is selected from five candidates.
  • Challenging distractors: All distractor objects must appear in the last observed frame, and their associated actions must be plausible given the video history. The model therefore needs to distinguish among realistic future interactions rather than eliminate obviously incorrect options.
  • Joint semantic and spatial prediction: The model must predict both the future interaction and its target-object bounding box. This combines event-level reasoning and object grounding in a unified MLLM-compatible output.

The benchmark builds on Ego4D and EPIC-KITCHENS rather than replacing them. Its contribution is a controlled evaluation protocol that tests future interaction discrimination and target grounding together, instead of treating them as separate predictions.

2. Novelty and insights of EgoR1.

EgoR1 is not simply existing latent reasoning with additional special tokens. The key difference is how the latent representations are supervised and connected.

Generic latent visual reasoning mainly learns compact future visual representations. EgoR1 instead introduces two task-specific stages:

  • Future Interaction State Distillation (FISD) uses both the future contact frame and interaction text to supervise a semantically grounded interaction state, rather than a generic visual summary.
  • Object-Centric Latent Reasoning (OCLR) conditions object latents on the predicted interaction state and uses region alignment with contrastive negatives to distinguish the actual target from other plausible objects.

The ablation results support the complementary roles of these components:

MethodAcc.mIoU
FISD only0.80100.3660
OCLR only0.79000.3959
FISD + OCLR0.82220.4023
Full EgoR1 (+ GRPO)0.84850.4219

FISD provides stronger event prediction, while OCLR mainly improves object grounding. Combining them improves both metrics. Compared with LVR, the full EgoR1 improves accuracy from 79.00% to 84.85% and mIoU from 0.3959 to 0.4219.

These results suggest that future interaction forecasting benefits from explicitly modeling what will happen next and which object will be involved, rather than relying only on generic future visual representations. We will further clarify this distinction from existing latent reasoning and specialist anticipation methods in the revised paper.

3. Does the MCQ formulation oversimplify anticipation?

MCQ provides a controlled evaluation setting, but does not fully replace open-ended interaction prediction. Our design reduces this limitation by using open-vocabulary interaction phrases and visually grounded, semantically plausible distractors instead of a fixed global action list.

More importantly, selecting the correct option is only part of the task. The model must also localize the target object, which requires spatial understanding beyond semantic option matching.

We will clarify this limitation and add candidate-only and open-generation evaluations to better measure option bias and forecasting ability beyond the MCQ setting.

Tighten claims about open vocabularyClarify distractor construction and evaluation

I thank the authors to clarify the novelty. However, it remains descriptive about the design. It will be a more direct comparison to show the models work well for the original Ego4D and EPIC-KITCHENS question design, but fail in the newly constructed questions to justify it is not just merely reformulate the dataset.

Reviewer_pYzbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EgoR1 predicts the next interaction and localizes its target object from historical egocentric frames. A teacher trained on future contact frames and interaction text supervises context latents, which are followed by object latents trained with region alignment and contrastive grounding. The model then undergoes GRPO post-training. The paper also introduces EgoR1-Bench1K, with 1,296 test examples, and reports improvements over supervised MLLMs and latent-reasoning baselines.

Critical strengths
  1. Combining next-interaction prediction with localization is a useful task formulation. The supplement gives sensible safeguards: source videos are separated across splits, contact and near-duplicate frames are excluded from the observed input, and future frames and phrases are used as training targets only. These details make the evaluation protocol clearer.

  2. The improvements on the proposed benchmark are substantial. EgoR1 reaches 84.85% option accuracy versus 79.00% for LVR, with mIoU increasing from 0.3959 to 0.4219 (main Table 2). The component ablations and the comparison with a learned visual-only teacher are worthwhile experiments and support the effectiveness of the training recipe.

Critical weaknesses
  1. The experiments do not isolate the benefit of the proposed state-to-object reasoning sequence. EgoR1 combines privileged future interaction text, region-level alignment, contrastive supervision, and GRPO. The existing ablations vary these ingredients, but do not compare sequential latents with a direct or parallel predictor given the same targets and optimization. This distinction matters in the reported results: FISD+OCLR reaches 0.4023 mIoU before GRPO, compared with LVR’s 0.3959, and GRPO then raises it to 0.4219 (supplementary Table 3). The baseline protocol matches frames, data splits, and answer targets, but does not establish that privileged supervision and the GRPO schedule are also matched. The results support better task-aligned training more clearly than they establish that progressive latent reasoning is necessary.

  2. The benchmark needs stronger controls for temporal forecasting. All candidate objects are required to appear in the last observed frame, yet there is no last-frame-only, shuffled-history, or candidate-only evaluation. Excluding contact and near-duplicate frames is a useful safeguard, but does not establish how much sequential history contributes beyond current hand–object cues and option matching. The 16-second window specifies the amount of observed history, not the anticipation lead time; results at controlled observation-to-contact gaps are also missing. Since all main experiments use this custom benchmark, these controls are important for interpreting the high accuracy as evidence of future-state prediction.

Questions for authors

Can LVR or a direct predictor be trained with the same privileged targets and GRPO budget, and can the sequential design be compared with a token- and supervision-matched parallel variant? How much accuracy is retained with only the last frame or with the historical order shuffled? Please report the actual anticipation lead-time distribution and performance when a fixed gap is inserted before contact.

2 comments
AuthorsOpenReview ↗

Clarification on Progressive Latent Reasoning and Temporal Forecasting

1. Effectiveness of progressive state-to-object reasoning.

The key design of EgoR1 is the sequential dependency between two latent groups: context latents predict the future interaction state, and object latents are then conditioned on this state to identify the target object. Future frames and interaction text are used only as training supervision and are never available during inference.

The existing ablations show the contribution of each component:

MethodAcc.mIoU
FISD only0.80100.3660
OCLR only0.79000.3959
FISD + OCLR0.82220.4023
FISD + OCLR + GRPO0.84850.4219

Combining FISD and OCLR improves both metrics over either component alone. GRPO further adds 2.63 percentage points in accuracy and 0.0196 mIoU. This shows that the improvement is not solely due to reinforcement learning.

To isolate the benefit of sequential reasoning, we will add two controlled variants: (1) a direct predictor trained with the same privileged supervision and (2) a parallel-latent variant where state and object latents are predicted independently. Both will use the same backbone, latent-token budget, supervision targets, losses, and training schedule. We will also compare LVR under a matched GRPO budget. These comparisons will separate the effect of the reasoning structure from additional supervision and optimization.

2. Temporal forecasting and anticipation lead time.

Our benchmark already applies several leakage controls. Training and test samples are separated by source video, future contact frames and near-duplicate observations are excluded from the input, and future annotations are used only for supervision.

To further measure the contribution of temporal information, we will add three input ablations:

  • Last-frame-only: Tests whether the final observed frame alone is sufficient for prediction.
  • Shuffled-history: Tests whether the temporal order of historical observations contributes to forecasting.
  • Candidate-only: Measures whether the model can exploit language or option-design biases without visual evidence.

All variants will use the same questions, candidate options, and evaluation protocol for fair comparison.

We will also clarify the difference between observation length and anticipation lead time. The 16-second window defines the available history, while the actual lead time is the interval between the last observed frame and the annotated contact frame:

Δt=tcontact−tlast.\Delta t = t_{\mathrm{contact}} - t_{\mathrm{last}}.

We will report the lead-time distribution and evaluate controlled 1 s, 2 s, and 4 s anticipation gaps on matched samples. This will show how performance changes as the interaction becomes more distant.

These additional experiments will help determine whether EgoR1's improvements come from progressive state-to-object reasoning and temporal interaction evidence, rather than stronger supervision, current-frame cues, or candidate-option biases.

Thank you for the response, which partially addresses my concerns; however, I still consider the paper below the acceptance bar and will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.