- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for long egocentric video question answering (EgoTrace). The key motivation is that retrieving isolated records based on semantic similarity may fail to recover the complete context needed to answer a question. To address this, EgoTrace constructs a scene trace that captures how situations evolve over time while preserving the original caption and ASR records. A Controller iteratively performs Search, Reinstate, and Open operations to retrieve relevant evidence, which is then used by a separate Reader to answer questions. Experiments on EgoLifeQA and MM-Lifelong demonstrate improvements over existing methods.
Critical strengths
- The motivation of this paper makes sense. In particular, reinstating a more complete context after the initial retrieval step is a reasonable approach to addressing the limitations of retrieving isolated records.
- The paper focuses on QA over extremely long egocentric videos, where retrieval based on embedding models remains important. This is a challenging setting for end-to-end agentic video understanding methods relying on cropped visual input.
- Figure 4 provides a clear illustration of the proposed retrieval process and helps explain how contextual reinstatement contributes to answering questions.
Critical weaknesses
- I am not entirely clear about the fundamental difference between the proposed Scene Trace and event captions. Could similar benefits be achieved through an appropriate captioning prompt or harness design, without introducing a separate
Scene Traceterm? - The proposed framework seems to focus primarily on generating more informative/continous captions and designing a retrieval and reasoning procedure over textual information. However, stronger agentic models may not necessarily benefit from being constrained to a predefined set of operations (Search, Reinstate, Open, and Stop). It is unclear whether this particular interaction paradigm is necessary or offers advantages over more flexible agentic reasoning.
- For writing, the paper does not sufficiently highlight or clearly explain its key innovations and how they differ from prior methods. In addition, the improvement from 68.4% to 74.4% in Table 1 corresponds to approximately 30 additional correctly answered questions, given that EgoLifeQA contains just 500 questions. The relatively small benchmark size raises some concerns about the robustness and significance of the reported improvements.
Questions for authors
- I do not think the teaser in Figure 1 clearly illustrates the intended motivation. For the example question, "Why did I leave the cafe abruptly?", why couldn't the answer simply explain the reasons for leaving the cafe on each occasion? (I understand it is a multiple-choice question). If the intention is to identify a particular departure based on the word "abruptly," why not use this distinguishing cue to rerank the retrieved candidates? It is unclear why contextual reinstatement is necessary in this example, rather than a more targeted retrieval or reranking strategy.
- This is the first paper I've encountered that modifies the original ICLR template. The original template does not seem to require keywords after the abstract.