acceptodds
Under review as a conference paper at ICLR 2027

EgoWorldMem: 4D World Memory for Long-Horizon Egocentric Video Understanding

Abstract

Egocentric videos contain substantial redundancy as the same physical environment is repeatedly observed from changing viewpoints, while brief object interactions often carry critical evidence. Existing memory mechanisms typically compress visual history based on temporal proximity or feature similarity, making it difficult to distinguish camera motion from actual changes in the world. We introduce **EgoWorldMem**, a 4D world-memory framework that compacts redundant observations of the physical world rather than merely historical tokens. EgoWorldMem reprojects an evolving geometric map into each incoming view to compensate for ego motion, retains sparse world-grounded visual evidence, and organizes dynamic observations into object-centric event segments through persistent tracking. The resulting memory is serialized into a compact visual-token sequence for VLM inference. Across five video-understanding benchmarks, EgoWorldMem improves the average accuracy of Qwen3-VL and LLaVA-NeXT-Video by 10.0 and 23.6 percentage points while removing 73.96% and 78.65% of visual tokens, respectively. An online variant further reduces time-to-first-token by 57.39% on EgoSchema, demonstrating the effectiveness of world-grounded memory for efficient long-horizon egocentric video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Video Understanding, 4D World Memory, Geometry-Aware Token Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bFuPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EgoWorldMem, a 4D world memory for long-term first-person video understanding. The method identifies redundant regions across viewpoints through 3D reconstruction and geometric re-projection, simultaneously performs sparse storage of static scene information, organizes dynamic information into object-centric event segments, and finally serializes them into compact visual token inputs for VLM.

Critical strengths
  1. By combining geometric re-projection with object-centric temporal memory, it is used to handle the significant visual redundancy caused by camera movement in egocentric videos.
  2. The paper writing is standard, and it contains all the necessary sections of a paper.
Critical weaknesses
  1. The degree of innovation is relatively limited. The main difference of the paper seems to be the combination of geometric filtering and object tracking/event serialization. At present, it is not clear whether this combination at the method level has a significantly more innovative aspect compared to the existing spatial memory work.
  2. The experiment is not sufficient to fully support that object-centric event organization is a strong and universal core contribution in this method, because the performance improvement brought by this module in the ablation experiment was not significant.
Questions for authors
  1. Can we conduct a comparison with existing memory-based methods under exactly the same training data, fine-tuning protocol, and token budget, rather than just random token retention?
  2. Does 4D World Memory explicitly establish a complete and continuous 4D scene representation?
  3. The pipeline strongly relies on accurate camera pose, 3D point map, and SAM2 persistent tracking. If there are any errors in these components, how would they affect the experimental results? It is hoped that a quantitative analysis for these typical failure modes can be provided.
  4. In the efficiency experiment, it is hoped to further report cumulative FLOPs, total latency or GPU-hour and other relevant metrics.
Reviewer_2K9sOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EgoWorldMem compresses egocentric video history by compacting redundant observations of the world rather than temporal token sequences: an evolving geometric map reprojected into each view separates static background, redundant re-observations, and dynamic objects, whose tokens are grounded in shared 3D coordinates and organized into object-centric event segments for prefill. On five MCQ benchmarks it reaches 80.5 (Qwen3-VL-8B) and 73.0 (LLaVA-NeXT-Video) while retaining only 26.0% and 21.4% of visual tokens, with an online variant cutting TTFT by 57.4%.

Critical strengths
  1. Reprojecting the evolving map into the current view and comparing two renderings per token patch is a direct way to separate ego motion from world change, and the resulting three-way partition feeds naturally into static retention, object tracking, and event segmentation.

  2. Memory-augmented baselines are budget-matched; a fine-tuned random-retention control and a training-free comparison isolate the selection mechanism; the threshold sweep gives an accuracy–compression curve; three geometry backbones including causal CUT3R are tested.

  3. The readout replaces video tokens at prefill without architectural change, with M-RoPE reconstruction for Qwen3-VL; gains hold across two very different backbones; and the paper reports where the spatial position embedding helps and where it hurts.

Critical weaknesses
  1. EgoPrune and Geo3DPruner (geometry-aware token pruning) and 3D-Mem and Reasmory (geometry-grounded memory for VLMs) appear only in related work, yet they attack the same redundancy-disambiguation problem. Without budget-matched comparisons, the advantage of the 4D world memory over these closest alternatives remains unestablished.

  2. All five benchmarks are three-minute clips or QA over short segments; memory growth over hour-scale egocentric streams is uncharacterized, the visual memory has no fixed bound, and the proposed token-merging extension is explicitly left unevaluated. Construction cost (779 ms/frame, of which SAM2 takes 341 ms) and failure modes when reconstruction or tracking errs (motion blur, textureless surfaces, fast ego motion) are likewise unanalyzed.

Questions for authors
  1. How does EgoWorldMem compare against EgoPrune, Geo3DPruner, 3D-Mem, and Reasmory under matched token budgets and, where applicable, matched training protocols?

  2. What happens on hour-scale egocentric streams: memory size over time, accuracy decay, and the effect of the token-merging extension? And how does performance degrade when the 3D reconstruction or SAM2 tracks fail?

Reviewer_VQqMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper wants to make long egocentric videos cheaper for a VLM to read. It keeps a 3D map of the scene (built with VGGT-Ω, or CUT3R for the online version). For every new frame, it projects the map into the current view and compares them patch by patch. If the map already explains a patch, the patch is thrown away. If the map has never seen that area, the patch is kept. If the map has seen it but it looks different now, it's treated as something moving, tracked with SAM2, and grouped into a small "event" block marked with <moving> tags. Only the kept tokens go into Qwen3-VL-8B or LLaVA-NeXT-Video, and both models are then fine-tuned on NExT-QA, QaEgo4Dv2 and about 1K auto-labeled VLM4D-style questions on EgoIT videos. The paper reports +10.0 and +23.6 points on average over the two base models across five benchmarks, while keeping only about 21 to 26% of the tokens. The online version also cuts time-to-first-token by 57%.

Critical strengths
  1. Using the 3D map to tell "the camera moved" apart from "something in the scene changed" is a good idea for egocentric video.

  2. The random-token baseline in Table 2 is the right control. Same training, same token budget, and the geometry-based selection still wins. The no-training results in Table 5 say the same thing.

  3. The paper is open about cost. Table 8 shows where the time goes, and swapping the 3D model doesn't change the results much.

Critical weaknesses
  1. Most of the gain comes from fine-tuning. The training data (NExT-QA, QaEgo4Dv2, VLM4D-style questions) matches four of the five test sets, and none of the Table 1 baselines got this training. Random tokens plus the same fine-tuning already beats the original full-token Qwen3-VL-8B (73.1 vs 70.5). So the fair gain is +5.8 to +7.4 over random tokens, not +10. LLaVA-NeXT-Video has no such control at all.

  2. The test videos are only minutes long (EgoSchema clips are 3 minutes), and NExT-QA / NExT-GQA aren't even egocentric. There's no hour-long or streaming benchmark, even though the title says long-horizon.

  3. Please add other pruning methods with the same training and budget, especially the geometry-based ones the paper cites. a simple frame-difference pruner like TimeChat-Online, and uniform frame sampling (App. D.1 argues against it but never runs it).

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.