- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the problems of memory accumulation and retrieval-context misalignment in long-horizon embodied reasoning. Existing approaches face challenges associated with redundant multi-view observations and accumulated long-term memories, while memory organization and retrieval can lead to context mismatches. To address these issues, the paper proposes View-Transition Egocentric Memory (VTEM). It infers scene, viewpoint, and cross-view object changes from consecutive observations, combines semantic value and visual novelty to determine which observations should be retained, and decomposes user questions into scene constraints, visual targets, and association constraints. It then performs hierarchical retrieval and conditionally conducts multi-hop retrieval along memory-transition edges when the initial evidence is insufficient. The paper evaluates VTEM primarily on OpenEQA, ECBench-MultiScene, and EnvQA, reporting improvements of 7.89, 3.44, and 4.57 score points, respectively, over the strongest baselines: 3D-Mem, GeLM, and Qwen3-VL-8B. The main contributions are:
- Constructing memory using view-transition relationships.
- Designing hierarchical multi-hop retrieval over the memory chain.
- Conducting extensive experiments on three major embodied question-answering benchmarks.
Critical strengths
- The problem is important and well motivated. Long-horizon embodied tasks can lead to substantial memory redundancy, while retrieval based on semantic or visual similarity alone has limitations and may result in context mismatches. The paper addresses a meaningful problem with a reasonable motivation.
- The method is clearly presented and reasonably designed. The paper incorporates view-transition relationships into memory construction to assess the value of observations, then reuses these relationships during retrieval. The overall framework is coherent and well structured.
- The experimental results demonstrate promising performance. The paper evaluates the method on major embodied question-answering benchmarks and reports consistent performance improvements. It also evaluates the approach on real-world observation videos, providing some evidence of its practical applicability.
Critical weaknesses
- The novelty of the memory construction mechanism could be further clarified. Although view-transition-based memory construction is a key contribution, the observation selection process partly relies on CLIP feature similarity, which has also been explored in prior work. It would be helpful to clarify more explicitly what distinguishes the proposed mechanism from existing approaches and what additional benefits it provides.
- The question decomposition strategy may pose some challenges for generalization. The retrieval stage decomposes questions into three components: scene constraints, visual targets, and association constraints. This design relies on the assumption that questions can be effectively mapped to these predefined components. It would be useful to discuss how the method handles questions that do not naturally fit this decomposition and whether its performance remains robust across diverse question formulations.
- The breadth of the fine-grained component evidence is limited. Table 2 and Appendix A.4 do provide module, transition, and retrieval ablations, including temporal-chain and shuffled-transition controls. However, the detailed appendix comparisons use only 50 OpenEQA questions from four trajectories. It remains unclear whether the individual component gains generalize across the full benchmarks and diverse environments.
- The end-to-end efficiency trade-off remains unclear. Appendix A.3 reports construction and question-time model calls: Full VTEM uses 4.68 calls per question versus 1.00 for flat retrieval, while reducing retrieved memories from 12.00 to 8.38 on the controlled subset. This analysis is useful, but wall-clock latency and exact token costs are not reported, so it is unclear when memory compression offsets the additional reasoning overhead.
Questions for authors
Could the authors report the end-to-end inference latency and the offline memory-construction cost? Specifically, under what conditions can the benefits of memory compression offset the additional overhead introduced by retrieval?
3 comments
Response to Reviewer E4Y5 (2)
Weakness 2
The retrieval stage decomposes questions into three components: scene constraints, visual targets, and association constraints. This design relies on the assumption that questions can be effectively mapped to these predefined components. It would be useful to discuss how the method handles questions that do not naturally fit this decomposition and whether its performance remains robust across diverse question formulations.
Thank you for your valuable suggestions. Our question decomposition is designed for a general embodied egocentric reasoning task: As characterized in previous studies [1–3], this task involves an agent answering natural-language task questions grounded in its first-person visual observations, requiring it to identify relevant objects and their states, locate related scenes or events, and reason about their spatio-temporal relationships. Existing studies [1–3] on embodied and egocentric reasoning cover these question elements, including locations, objects, states, actions, events, and their spatio-temporal associations. Based on these elements, we further organize the question information into three retrieval components: scene constraints (locations and spatio-temporal localization), visual targets (objects and states), and association constraints (events, actions, and spatio-temporal associations). For questions lacking certain elements, the corresponding components are left empty and their retrieval operations are skipped.
Experiments on three established benchmarks featuring diverse question types in embodied egocentric reasoning further demonstrate that VTEM maintains robust performance across different question formulations: OpenEQA [1], Env-QA [4], and ECBench-MS [5]. Specifically, OpenEQA evaluates embodied question answering over prior embodied experience, Env-QA focuses on understanding dynamic environments, including object states, temporal orders, and events, while ECBench-MS involves long-horizon multi-scene reasoning. VTEM achieves consistent performance improvements across all three benchmarks, further supporting the applicability and robustness of our question decomposition across diverse embodied egocentric reasoning tasks.
References:
[1] Majumdar et al. OpenEQA: Embodied Question Answering in the Era of Foundation Models. CVPR 2024.
[2] Grauman et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022.
[3] Jia et al. EgoTaskQA: Understanding Human Tasks in Egocentric Videos. NeurIPS 2022.
[4] Gao et al. Env-QA: A Video Question Answering Benchmark for Comprehensive Understanding of Dynamic Environments. ICCV 2021.
[5] Dang et al. ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? 2025.
Response to Reviewer E4Y5 (3)
Weakness 3
Table 2 and Appendix A.4 do provide module, transition, and retrieval ablations, including temporal-chain and shuffled-transition controls. However, the detailed appendix comparisons use only 50 OpenEQA questions from four trajectories. It remains unclear whether the individual component gains generalize across the full benchmarks and diverse environments.
Thank you for your valuable suggestions. Our original fine-grained ablations used a representative subset, following established experimental practices in prior works (VideoAgent, Embodied VideoAgent, and 3D-Mem). To further address your concern about generalization, we expand the fine-grained ablations in Appendix A.4 to all OpenEQA-HM3D tasks, consistent with the main-text ablations.
As shown below, Full VTEM achieves 62.61% LLM-Match, outperforming TMC association only, TMC value only, THR anchor only, and THR multi-hop only by 7.36, 9.15, 8.44, and 7.54 percentage points, respectively. These results demonstrate that the component gains persist across the full OpenEQA-HM3D evaluation set, rather than being limited to the original 50-question subset.
| Method | LLM-Match (%) |
|---|---|
| no TMC & no THR | 48.52 |
| TMC association only | 55.25 |
| TMC value only | 53.46 |
| THR anchor only | 54.17 |
| THR multi-hop only | 55.07 |
| Full VTEM | 62.61 |
We will include these expanded ablation results in the revised manuscript.
Weakness 4 and Question 1
W4. Appendix A.3 reports construction and question-time model calls: Full VTEM uses 4.68 calls per question versus 1.00 for flat retrieval, while reducing retrieved memories from 12.00 to 8.38 on the controlled subset. This analysis is useful, but wall-clock latency and exact token costs are not reported, so it is unclear when memory compression offsets the additional reasoning overhead.
Q1. Could the authors report the end-to-end inference latency and the offline memory-construction cost? Specifically, under what conditions can the benefits of memory compression offset the additional overhead introduced by retrieval?
Thank you for your valuable suggestions. Following your suggestions, we conduct additional efficiency experiments to evaluate VTEM's offline memory-construction cost and end-to-end inference latency, including token counts and peak GPU memory usage. The results are reported below.
Memory Construction. The table below compares the memory-construction time of VTEM and 3D-Mem. VTEM requires 1.10 s per sampled candidate view, substantially faster than 3D-Mem (2.47 s/frame), reducing construction time by approximately 55.5%. This is a one-time offline cost, and the constructed memory can be reused across multiple questions.
| Method | Construction Time per Sampled Candidate View |
|---|---|
| 3D-Mem | 2.47 s/frame |
| VTEM (Ours) | 1.10 s/frame |
Query Inference. We further compare VTEM with Flat Retrieval in terms of VLM calls, retrieved memories, token counts, peak GPU memory, and end-to-end inference latency. As shown below, despite increasing VLM calls from 1.00 to 4.68, VTEM reduces retrieved memories from 12.00 to 8.38, total tokens by 7.1%, and inference latency from 8.77 to 7.76 s (11.5%), while improving the score from 0.51 to 0.57.
| Method | VLM Calls | Retrieved Memories | Input Tokens | Output Tokens | Total Tokens | Peak GPU Memory | Latency | Score |
|---|---|---|---|---|---|---|---|---|
| Flat Retrieval | 1.00 | 12.00 | 9947.84 | 12.16 | 9960.00 | 18.45 GiB | 8.77 s | 0.51 |
| VTEM (Ours) | 4.68 | 8.38 | 9090.88 | 165.62 | 9256.50 | 17.46 GiB | 7.76 s | 0.57 |
These results suggest that for long-horizon tasks with substantial memory evidence, reducing the amount of retrieved evidence can offset the additional retrieval overhead. In our experiments, VTEM achieves lower end-to-end inference latency despite more VLM calls. Moreover, the offline construction cost can be amortized when the same memory is reused across multiple questions.
We will include these efficiency results and corresponding analyses in the revised manuscript.
Response to Reviewer E4Y5 (1)
Thank you for your valuable feedback. We have carefully revised the manuscript in response to your comments, with the major revisions highlighted in blue for ease of reference.
Weakness 1
Thank you for your valuable suggestions. VTEM identifies the semantic value of new observations by capturing new scene, object, or state information across views, and combines it with visual value for observation selection; CLIP similarity is only one replaceable implementation of visual value estimation (e.g., VLMo-based similarity). Therefore, VTEM does not rely heavily on CLIP similarity for observation selection. Specifically, VTEM infers scene changes, viewing-direction changes, and cross-view object changes between the current observation and the retained observation history without explicit 3D reconstruction. It then combines the resulting semantic value with visual novelty to determine the overall memory value and selectively retain high-value observations.
To further validate our design, we conduct two complementary experiments: (1) an ablation study comparing CLIP-based visual novelty alone with its combination with view-transition-based semantic value, to evaluate the contribution of semantic value assessment; and (2) a visual feature replacement experiment, replacing CLIP features with VLMo features while keeping the remaining components unchanged, to examine whether VTEM's effectiveness depends on a specific visual feature encoder. As shown in the table below, incorporating semantic value increases the normalized score from 0.400 to 0.530. Furthermore, replacing CLIP with VLMo achieves a normalized score of 0.535. These results demonstrate that semantic value assessment improves observation selection beyond visual novelty alone, while VTEM maintains comparable performance with different visual feature encoders.