acceptodds
Withdrawn from ICLR 2027

EgoTrace: Coupling Semantic Proximity and Temporal Continuity for Egocentric Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Long-horizon video question answering often begins with an incomplete cue: a question names part of an experience without specifying the evidence needed to answer it. Retrieving isolated records by similarity is fragile here, because similar encounters recur and the record that matches a question may not be the one that answers it. Inspired by contextual reinstatement in episodic memory, we introduce EgoTrace, which keeps original captions and aligned speech unchanged and links them to a scene trace: source-linked descriptions of how the surrounding situation develops. A Controller iteratively chooses among searching for records, reinstating the context around a retrieved record, and opening original evidence; a separate Reader answers from the opened sources and their linked context. Because the trace guides navigation rather than defining retrieval units, the extent of recovered context adapts to each question instead of being fixed by a window or pre-segmented events, and opened evidence can turn an incomplete match into a more precise cue. The trace is built in a single chronological pass and can be extended as footage arrives. EgoTrace achieves the highest reported scores of 74.6% on EgoLifeQA and 33.4% on MM-Lifelong Month validation, exceeding prior results by 3.4 and 8.9 percentage points, respectively. Ablations show that trace descriptions add value beyond temporal access alone, while direct source retrieval outperforms tested graph and semantic-memory additions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric video QA, long-horizon memory, contextual reinstatement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_17VAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for long egocentric video question answering (EgoTrace). The key motivation is that retrieving isolated records based on semantic similarity may fail to recover the complete context needed to answer a question. To address this, EgoTrace constructs a scene trace that captures how situations evolve over time while preserving the original caption and ASR records. A Controller iteratively performs Search, Reinstate, and Open operations to retrieve relevant evidence, which is then used by a separate Reader to answer questions. Experiments on EgoLifeQA and MM-Lifelong demonstrate improvements over existing methods.

Critical strengths
  1. The motivation of this paper makes sense. In particular, reinstating a more complete context after the initial retrieval step is a reasonable approach to addressing the limitations of retrieving isolated records.
  2. The paper focuses on QA over extremely long egocentric videos, where retrieval based on embedding models remains important. This is a challenging setting for end-to-end agentic video understanding methods relying on cropped visual input.
  3. Figure 4 provides a clear illustration of the proposed retrieval process and helps explain how contextual reinstatement contributes to answering questions.
Critical weaknesses
  1. I am not entirely clear about the fundamental difference between the proposed Scene Trace and event captions. Could similar benefits be achieved through an appropriate captioning prompt or harness design, without introducing a separate Scene Trace term?
  2. The proposed framework seems to focus primarily on generating more informative/continous captions and designing a retrieval and reasoning procedure over textual information. However, stronger agentic models may not necessarily benefit from being constrained to a predefined set of operations (Search, Reinstate, Open, and Stop). It is unclear whether this particular interaction paradigm is necessary or offers advantages over more flexible agentic reasoning.
  3. For writing, the paper does not sufficiently highlight or clearly explain its key innovations and how they differ from prior methods. In addition, the improvement from 68.4% to 74.4% in Table 1 corresponds to approximately 30 additional correctly answered questions, given that EgoLifeQA contains just 500 questions. The relatively small benchmark size raises some concerns about the robustness and significance of the reported improvements.
Questions for authors
  • I do not think the teaser in Figure 1 clearly illustrates the intended motivation. For the example question, "Why did I leave the cafe abruptly?", why couldn't the answer simply explain the reasons for leaving the cafe on each occasion? (I understand it is a multiple-choice question). If the intention is to identify a particular departure based on the word "abruptly," why not use this distinguishing cue to rerank the retrieved candidates? It is unclear why contextual reinstatement is necessary in this example, rather than a more targeted retrieval or reranking strategy.
  • This is the first paper I've encountered that modifies the original ICLR template. The original template does not seem to require keywords after the abstract.
Reviewer_L1RWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EgoTrace to address the recurrence of similar situations in similarity-based retrieval. Specifically, it introduces a scene trace that links the original records and describes how the surrounding situation develops. To support the newly introduced scene trace, the controller is also extended to reinstate the context around a retrieved record and open original evidence. Experiments show that trace descriptions provide value beyond simply expanding temporal access.

Critical strengths

Organizing narratives at the scene level is a sensible design that helps address the lack of context in individual records.

Critical weaknesses
  1. The datasets used are relatively niche. Moreover, the accuracies achieved with different backbone models (Qwen3.8-27B, GPT-5, and DeepSeek-4.1-Flash) are similar, with Qwen3.8-27B even outperforming GPT-5. This is surprising, and the paper does not sufficiently explain this phenomenon.
  2. The paper lacks efficiency comparisons and does not report average latency or offline construction time.
Questions for authors
  1. Regarding Weakness 1, please evaluate on datasets such as Video-MME and compare against state-of-the-art agent-based methods to strengthen the evidence.
  2. Regarding Weakness 2, please provide additional efficiency evaluations.
Reviewer_GhMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a harness to perform long-form egocentric video QA. There are two main components: A memory, and a navigation harness to search for the answer.

Memory: The idea is to retain a detailed memory of all events that happened (30 second captions + ASR text --- a record), and develop a scene trace (one of the key contributions). The scene trace tracks the progression of events through time and segments the video by high-level events with text summaries of what happened in those segments. The scene trace is mapped to the original 30 second records and vice versa.

Navigation harness: Given the memory and the question, a "Controller" is used to search for matching records, pick the most promising records based on the scene trace and the evidence, and expand the search context to include nearby records (30 second segments) to search for the answer.

Experiments are performed on MM-Lifelong and EgoLifeQA benchmarks and show improvements over prior models / harnesses. Ablation studies are performed to evaluate the utility of visual inputs for the harness, expansion to surrounding context around a record, direct access to captions + ASR within a record and memory organization.

Critical strengths

Strength 1: Sensible, well-motivated approach

The paper motivates its problem well. Over long recordings, similar encounters recur and many records mention the same people, objects or activities. Near-duplicates are therefore hard to tell apart, and a strict top-k cutoff may discard the decisive record. Linking unchanged caption/ASR records to a navigable scene trace is a sensible response: the Controller can tell which activity a candidate belongs to and follow the surrounding context, rather than trusting a single ranking.

Strength 2: Good baseline selection and ablation studies

The paper compares against a relevant set of recent long-horizon video memory systems. The ablations are informative. The "access to surrounding experience" study (Table 4) separates the value of temporal access from the value of the trace descriptions, and the results look promising. The semantic-organization ablation (Table 5) shows that adding PPR graph retrieval or precomputed semantic memory hurts direct source retrieval, a useful finding that simple works better.

Strength 3: Good improvements on EgoLifeQA

On EgoLifeQA (Table 1), EgoTrace improves over prior methods consistently across all four backbones tested. The numbers are reasonably good overall, though not spectacular.

Critical weaknesses

Weakness 1: Lack of clarity in writing

Several core components and experiments are hard to follow.

  • It is unclear from the overview (Fig. 2) whether EgoTrace is a harness over existing models or whether a model is trained.

  • Scene-trace construction (§3.2) has no pseudocode or figure giving the exact steps needed for reproducibility, even in the appendix.

  • Many terms are introduced without explanation:

    • BM25 (§3.3);
    • the "cards" exposed to the Controller, and how many candidates are retained (App. B);
    • the "temporal coverage and visual-change cues" used for frame selection (App. B);
    • the "call limit" in the retrieval-budget study.
  • The "+ temporal access" ablation variant is also described vaguely.

  • The visual-input study (Table 3) leaves several questions open: what is being compared, what "broad frames" and "sparse vision" mean, whether "no visual channel" is the default, how visual context is provided, and why the study is only preliminary.

  • In the budget-sensitivity study (Fig. 3), the paper draws conclusions without explaining the experiment clearly.

Weakness 2: Several experiments are weak or inconclusive

L292 - 297: The headline EgoLifeQA comparison is unfair. It pits EgoTrace on DeepSeek-4.1-Flash (74.6%) against MERIT on GPT-5 (71.2%). With GPT-5 as the backbone for both, EgoTrace's gain is only 2.6 points. The analysis also doesn't explain cases where the method fails to help, for example GPT-5 against MERIT on RelationMap. It is also not clear whether the design choices are overfit to the evaluation set.

L299 - 305: The backbone results also undercut the claim that EgoTrace benefits from stronger backbones. A relatively small model, Qwen3.8-27B, outperforms GPT-5 and is comparable to DeepSeek-4.1-Flash, which reads more as a negative result. This goes against the claim "Benefiting from stronger backbones".

Table 2: On MM-Lifelong, there is no comparable-backbone study, so it is unclear how much of the gain comes from the backbone versus the harness. MAGIC-Video is not compared using Qwen3.8-27B, and EgoTrace is not evaluated across different base models.

Central claim is not validated: Most importantly, the paper never directly tests its central claim. EgoTrace is motivated by cases with multiple highly similar matches, where near-duplicate encounters are hard to tell apart and the scene trace is needed to identify the right one (L042 - 045). Yet there is no evidence about how often this situation occurs in the benchmarks, or whether EgoTrace actually resolves it better than the baselines. The only qualitative example (Case 38, Fig. 4) is a fairly straightforward query. It does not show the competing candidates, does not involve resolving several highly similar situations, and does not say whether the baselines got it wrong or why. Without measuring how often such cases arise, what fraction EgoTrace resolves correctly, and how baselines do on the same cases, it remains unclear whether the reported gains come from the mechanism the paper proposes.

Failure cases are not analyzed: EgoTrace improves over baselines under some conditions, but the improvements are not very high. What remains to be unsolved? What are the failure cases that aren't addressed yet by the approach?

Cost trade-off is unclear. The paper should state whether the approach is costlier than prior methods and characterize the memory/compute versus performance trade-off. In particular, it should explain why EgoTrace costs 1.55× the baseline despite having fewer uncached input tokens (Table 6).

Questions for authors
  1. How often do the benchmark questions involve multiple highly similar candidate matches? Does EgoTrace resolve these cases better than the baselines? Quantitative and qualitative evidence to substantiate this claim is critical.
  2. Can you compare EgoTrace and the baselines using the same backbone on both EgoLifeQA and MM-Lifelong? This would separate the contribution of the harness from that of the backbone.
  3. How do stronger backbones actually benefit EgoTrace? The results presented so far are largely negative.
  4. How are the design choices and hyperparameters selected?
  5. What are the failure cases of the method?
  6. Clarity is a huge concern here, especially for reproducibility. Could you clarify the points raised in the weakness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.