acceptodds
Withdrawn from ICLR 2027

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Egocentric video provides a continuous first-person record of everyday activities, making it a natural foundation for wearable AI assistants. Unlike offline video understanding, streaming systems must process observations as they arrive, while queries may refer to experiences from the distant past. Recent work has begun to explore whether information can be recalled across long temporal gaps. Yet remembering how far is only part of the challenge: recurring objects, actions, and locations also require models to remember exactly which detail belongs to which event. We introduce EgoProbe, a benchmark that evaluates streaming egocentric memory along these two complementary dimensions: long-range retention, i.e., how far, and event-specific recall under interference, i.e., how exact. Built on long-form egocentric videos, EgoProbe uses an event-grounded data engine to extend recall intervals, identify related events, and construct event-grounded distractors, followed by human verification. It contains two subsets: EgoProbe-Base, which evaluates long-range recall without intervening related events, and EgoProbe-Echo, which further challenges models with similar intervening events and event-grounded distractors. EgoProbe-Echo additionally adopts coarse-to-fine multi-turn probing: the first turn evaluates coarse-grained event recall, while the second tests whether the model can retrieve the target-specific detail associated with the designated event. Extensive evaluation across diverse models reveals substantial limitations in long-range recall, with further challenges in recovering target-event details among related experiences. Even when models correctly answer coarse-grained questions, they often fail to recall the specific details of the designated event. EgoProbe exposes this underexplored failure mode and provides a testbed for advancing long-range and exact memory in streaming egocentric video understanding. Code and data will be released.

Primary area:
datasets and benchmarks
Keywords:
Egocentric Video Understanding, Streaming Video Understanding, Video Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uqb1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies three limitations of previous works: 1. designed for generic video rather than on egocentric streams; 2. does not imply long-range memory dependence and many queries may be answered from recent observations; 3. largely focus on how far information can be recalled rather than how precisely model recall details. Thus, the authors propose EgoProbe which evaluates in egocentric streams that requires long-range memory and evaluates how exact the model can recall details of target event. The dataset is built from Ego4D and EgoLife dataset and build two EgoProbe-Base, questions without intervening related events and EgoProbe-Echo, questions with contrastive related events.

Critical strengths
  • The paper addresses an important problem for egocentric assistants which evaluates how well it retains information over long periods and can distinguish details from similar experiences.
  • The authors conduct wide experiment over 17 models of broad model families and provide analysis.
Critical weaknesses
  • The distinction from previous works (table 1) is insufficiently discussed. How is the benchmark different from existing benchmarks such as MemLife[1], EgoMemReason[2], and EgoCITE[3]?
  • Can the authors provide more analysis or discussion on why model rankings varies across evaluation settings?

[1] MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories [2] EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding [3] EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory

Questions for authors
  • How are the annotators recruited? How many annotators reviewed each instance and what was the inter-annotator agreement? I wonder if there was any biases introduced from automatic multiple-choice generation.
  • How would the number of distractor affect on the performance? Can the authors also conduct analysis by matching the number?
5 comments

We thank the reviewer for the feedback and for recognizing the importance of evaluating long-term memory and event-specific recall for egocentric videos, as well as our extensive evaluation across 17 models from diverse model families. Below, we address your concerns with clarifications, further analyses, and additional controlled experiments.

W1: Relationship to MemLife, EgoMemReason, and EgoCITE.

Thank you for bringing these related works to our attention. We first summarize their release dates, publication/submission status, and primary contributions below.

WorkFirst arXiv submissionPublication / submission statusPaper typeComparable to ours?
MemLifeSeptember 30, 2026 (later than ICLR 2027 submission deadline)ICLR 2027 submission; arXiv preprintMethodNo
EgoMemReasonMay 11, 2026COLM 2026; published October 15, 2026BenchmarkYes
EgoCITEAugust 12, 2026ICLR 2027 submission; arXiv preprintMethodNo

According to the ICLR 2027 Reviewer Guidelines, authors are not required to compare against contemporaneous work published at peer-reviewed venues on or after July 17, 2026, or against unpublished arXiv preprints. The guidelines explicitly state that the absence of such comparisons cannot be a basis for rejection. MemLife and EgoCITE fall under the provision for unpublished arXiv preprints, while EgoMemReason’s official COLM proceedings metadata records its publication date as October 15, 2026, placing it within the contemporaneous-work window. In particular, MemLife’s first arXiv version appeared after the ICLR 2027 full-paper submission deadline. We therefore sincerely hope that the reviewer will reconsider treating the absence of these comparisons as a weakness of our original submission, in light of the publication timeline and the ICLR policy.

Nevertheless, we appreciate these suggestions and have cited and discussed all three works in the revised paper. Below, we provide a direct comparison with EgoMemReason, the benchmark among these works, focusing on its evaluation setting and benchmark design.

Comparison with EgoMemReason:

  • Different research scopes: offline video versus streaming setting. EgoMemReason evaluates offline long-video reasoning with query-time access to sampled historical frames. EgoProbe targets strict streaming memory: models process video and update memory without knowing future questions, then answer using retained memory without reloading historical video. EgoMemReason itself acknowledges in Appendix D that its evaluation relies on sampled frames and optional auxiliary inputs, and identifies streaming-based evaluation as future work. These are fundamentally different evaluation settings, imposing different requirements on memory construction and retention.

  • Different question designs and diagnostic goals. EgoMemReason emphasizes reasoning over multiple pieces of evidence through entity tracking, counting, event ordering and linking, and behavior-pattern inference. EgoProbe focuses on how exactly a model recalls a designated event after a long interval, particularly when similar intervening experiences contain competing details. Its coarse-to-fine dialogues further distinguish coarse event recall from target-specific detail recall.

  • Event-linked diagnostics beyond accuracy. EgoMemReason primarily reports QA accuracy. EgoProbe-Echo additionally links event-grounded distractors to verified related events, enabling Event err. and Other err. analyses to distinguish event confusion from other errors.

We also added EgoMemReason to the benchmark comparison in Table 1.

W2: Model rankings across evaluation settings.

Thank you. EgoProbe provides a diagnostic testbed for evaluating different dimensions of streaming memory. Some ranking variations are expected because different settings place different demands on memory capabilities. Following your suggestion, we further explain these variations through overall ranking consistency, the different capabilities required across settings, and model-specific memory mechanisms.

  • Overall ranking consistency across evaluation settings. We quantify consistency using Spearman’s rank correlation coefficient (ρ), computed from models’ relative rankings across two settings. It ranges from −1 to 1, with values closer to 1 indicating stronger agreement. Across all eight streaming models in Table 2, all three correlations are high, indicating broadly consistent rankings across settings, particularly between Base and Echo single-turn evaluation (ρ = 0.976).
Evaluation settings comparedSpearman’s ρ
Base single-turn Acc. vs. Echo single-turn Acc.0.976
Base single-turn Acc. vs. Echo multi-turn R2 Acc.0.905
Echo single-turn Acc. vs. Echo multi-turn R2 Acc.0.833
  • Different dimensions of memory across evaluation settings. Although all settings evaluate memory, they emphasize different capabilities. Base focuses on long-range detail retention without intervening related events; Echo single-turn emphasizes recalling the correct target-event detail among similar experiences; and Echo multi-turn additionally requires successful coarse event recall before precise detail recall. A model’s strength in one dimension does not necessarily imply equal strength in another, helping explain local ranking differences. This reflects EgoProbe’s design to diagnose complementary dimensions of memory.

  • Model-specific memory mechanisms. First, access to distant evidence explains stable performance differences. SimpleStream’s four-frame recent window in our configuration and StreamingVLM’s bounded visual and textual windows restrict access to distant details, consistent with their lower rankings across all three settings.

    Second, event-level organization supports long-range recall. StreamForest maintains event trees with selective compression, while EventMemAgent stores retrievable event descriptions, visual anchors, and change records. Both remain among the top three across settings, with StreamForest consistently ranking first.

    Third, memory representations align with different recall strengths. VST records streaming thoughts, while EventMemAgent additionally preserves visual anchors and change records. This distinction aligns with VST’s higher coarse R1 accuracy (95.28% vs. 88.36%) and EventMemAgent’s higher standalone detail accuracy (37.75% vs. 35.48%). Under two-turn evaluation, which requires both stages to succeed, VST ranks higher (33.93% vs. 32.63%), illustrating how different recall requirements change model rankings.

Q1: Annotator qualifications, agreement, and answer-option bias.

Thank you for these questions. We clarify the annotators’ qualifications, review procedure, inter-annotator agreement, and checks for potential bias in answer options below.

  • Annotator qualifications and review procedure. Our annotation team consists of 3 members recruited from the project team, all holding at least a master’s degree and having research experience in video understanding, VQA, and related areas. Before formal review, we developed detailed guidelines and examples and conducted a demonstration to align the review criteria. Each of the 2,555 candidate QA items was independently reviewed by 2 annotators using four categories: Accepted, Needs revision, Recommend exclusion, and Cannot determine. Disagreements were examined and resolved by the third annotator, who also serves as the project lead.

  • Inter-annotator agreement. The two independent reviewers assigned identical judgments to 2,421 of the 2,555 items, yielding 94.76% agreement and Cohen’s κ = 0.831. Both metrics were computed before adjudication and reflect the consistency of the independent review judgments.

  • Checks for bias in automatically generated answer options. We address this concern in two ways:

  1. Human verification and correction. Reviewers assess whether the options are reasonable, distinct, and appropriate for the question, with exactly one correct answer. They verify the correct answer against the video and confirm that each event-grounded distractor is supported by another related event but remains incorrect for the target event. Identified issues are corrected, and revised items are rechecked before acceptance.

  2. Text-only evaluation. In Appendix E.2, Table 7, we examine potential bias in the final questions and options. Without video input, Qwen3.5 achieves 25.51% on Base and 24.49% on Echo, close to the 25% random-choice baseline, compared with 45.91% and 35.31% with video input. These results show no evidence of answer-option bias in this text-only evaluation.

Reviewer_qfLJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper raises a benchmark called EgoProbe. It is a streaming egocentric video benchmark aiming at memory evaluation. The core of this benchmark is built around a two-dimensional remembering ability: long-range retention (how far), and event-specific recall under interference. They also build an event-grounded data engine based on Ego4D / QaEgo4D and the EgoLife dataset, which includes four automatic stages: related-event discovery, recall-interval extension, query disambiguation (a detector–editor–verifier loop), and multiple-choice and conversation generation, followed by manual checking in the end. This benchmark includes two subsets: EgoProbe-Base and EgoProbe-Echo. EgoProbe-Base is one-round question answering with no related events within the recall interval. EgoProbe-Echo is one-round question answering plus two-round question answering based on event-grounded distractors. The intervals of the memories are classified into four ranges: short, medium, long, and ultra-long, and the longest one is 138.54 hours.

For the evaluation, there are 17 models. The main discoveries are:

  1. The best model, GPT-5.6-Sol, only reaches about 55.32% on Base, and its two-round completion rate is 43.69%.
  2. All the streaming models reach their lowest accuracy in the ultra-long group, which is similar to a forgetting curve.
  3. When the number of distractors (similar events) rises, the overall accuracy decreases and the event-linked error rate rises.
  4. Accuracy at the coarse level does not imply accuracy at the detail level. For example, VST's R1 accuracy is about 95.28%, but its two-round completion rate is only about 33.93%.
Critical strengths
  1. The dimension of "how exact" is the main contribution. Current streaming benchmarks mainly focus on asking how far away the models can remember. But EgoProbe-Echo adds a second question: whether the model can separate the details from similar situations and similar memories. To test this, the wrong options are not randomly set, but are extracted from related events, which gives the event-grounded distractors. The paper also reports the event-linked error rate. With this specific design, this benchmark can separate two kinds of failures: the model could not remember it, or the model remembered it as something else. This specific diagnostic signal is missing in current benchmarks, which is the main reason that I lean toward accepting it.

  2. The construction pipeline and evaluation protocol are very strict and careful. Under the strict streaming protocol, each query is answered within an isolated branch without changing the persistent memory. The query disambiguation loop makes sure that prolonging the recall interval causes no ambiguity, and the paper gives a full example as well. Each piece of data goes through human verification, and the retention rate is about 93.5%. The text-only baseline is only about 25%, which means that most of the data used here cannot be answered without the video. All these details make the results reliable and make the benchmark easier to be adopted by others.

Critical weaknesses
  1. Base and Echo are not matched controls. The two subsets are different in data source, recall-interval distribution, and question wording, because the questions from Echo have extra cues added in the query disambiguation loop. Therefore, the lower accuracy on the Echo subset cannot be explained only by the interference of similar events or distractor events. The paper admits this in Section 4.2, but still uses the difference between subset Base and subset Echo as the main evidence. The data engine already has the ability to control the distractor events, including whether they are placed before or after the query. I believe that adding a controlled comparison, which asks about the same target event before and after the distractor events, would be more useful.
  2. All the answers to the R1 questions are "Yes", according to Appendix D.4. Note that a model which answers "Yes" all the time would get a 100% result on R1. Therefore, the R1 accuracy tests for a yes-bias rather than coarse-grained recall. Because of this, the conclusion that "coarse-grained recall is successful but detailed recall fails" should not be considered as a main finding. For example, VST having 95.28% on R1 does not mean that VST actually remembers this event. Also, Ego-R1, EgoRAG, and SimpleStream have lower R1 scores, which probably come from output parsing rather than memory. In order to make the two-round question answering more meaningful, I believe that adding negative-answer R1 questions would be important.
Questions for authors
  1. Can you provide a controlled subset where the same target event is queried before and after the related events, and report the accuracy difference? This would directly support the "how exact" claim.
  2. What is the "Yes" rate of each model on R1? Would you consider adding negative R1 questions, where the event did not happen?
  3. Ultra-long intervals come only from EgoLife, and the input budgets are fixed (e.g., 128 prefix frames for the general-purpose models, fixed memory budgets for the streaming models). Does the accuracy drop with recall interval still hold within each source dataset? How much of the drop comes from the target event not being sampled at all?
  4. How many annotators reviewed each item, and what is the inter-annotator agreement?
  5. Are all results from a single run? Some gaps in Table 2 are within one or two points.
5 comments

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of our event-grounded distractor design and our carefully designed construction and evaluation protocols. Below, we address your concerns with clarifications, additional controlled experiments, and further analyses.

W1 & Q1: Controlled comparisons of related-event interference.

Thank you for this constructive suggestion. Below, we clarify our design and provide two complementary paired comparisons to further examine this effect.

  • Why Base and Echo are not strictly paired controls. As described in Section 3.2, our data engine assigns each QA to Base or Echo according to the absence or presence of intervening related events, with each QA appearing in only one subset. Base and Echo therefore contain different QA instances, rather than the same questions evaluated with and without related events. Constructing perfectly matched video pairs for every QA is impractical with real-world videos, as this would require keeping the target event and all other video content unchanged while changing only the intervening related events. We explicitly acknowledge this limitation in Section 4.2. The two subsets also focus on different aspects of memory: Base evaluates how far models recall without intervening related events, while Echo evaluates both how far and how exact they recall target-event details among related experiences.

  • Event confusion is diagnosed within Echo. Our evidence for event confusion does not rely solely on comparing Base and Echo. Within Echo, each event-grounded distractor is linked to a verified related event, allowing us to identify errors that select details belonging to another observed experience rather than the designated target event. Table 4 and Appendix E.3, Table 8 analyze these event-linked errors, with Table 4 showing Event err. consistently above the group-specific random-choice references.

Experiment:

To directly examine the effect of related-event interference, we additionally conduct two complementary paired comparisons.

Following your suggestion, we conduct a paired comparison of recall before versus after a related event. We independently sample 300 QA groups from EgoProbe-Echo videos, each containing one annotated related event. Queries are placed 10 seconds before and after the related event. The video, question, options, and correct answer remain unchanged, and the target evidence precedes both queries. We report single-turn Acc. before and after the event, together with the accuracy change.

ModelAcc.: Query before a related eventAcc.: Query after a related eventAcc. change (Query After Event - Query Before Event)
StreamingVLM41.6734.67-7.00
HERMES40.0034.33-5.67
StreamForest50.3345.00-5.33
OASIS50.0037.67-12.33
EventMemAgent44.6739.00-5.67
VST44.3337.67-6.66

The results above show that accuracy decreases across all models when queries are asked after the related event, with drops of 5.33–12.33 percentage points. This highlights the challenge of recalling target-event details after related experiences and supports EgoProbe’s diagnostic value.

To further strengthen this analysis, we evaluate the same target-event questions with and without related-event content. We sample 300 QA groups from EgoProbe-Echo videos, covering both single-event and multi-event cases. For each QA, we compare Original (with related events) against Interference removed (without related events). The latter replaces frames covering each annotated related event with repeated copies of a pre-event frame. The target evidence, question, options, correct answer, query time, sampling rules, and video timeline remain unchanged. We report single-turn Acc. for both versions and the accuracy change after removing related-event content.

ModelAcc.: Original video with related eventsAcc.: Same video without related eventsAcc. change (Removed - Original)
StreamingVLM30.0039.33+9.33
HERMES31.6742.67+11.00
StreamForest38.6748.00+9.33
OASIS32.0041.67+9.67
EventMemAgent34.6746.67+12.00
VST36.3341.33+5.00

The results above show that removing related-event content increases single-turn accuracy across all evaluated models by 5.00–12.00 percentage points. These consistent increases under controlled conditions further support our conclusion that intervening related events interfere with target-specific detail recall.

W2 & Q2: All the answers to the R1 questions are "Yes".

Thank you for these comments. We first clarify our evaluation protocol, then report parsing success rates and Yes-response rates, followed by a paired positive–negative evaluation to further assess coarse event recall within our two-turn coarse-to-fine design.

  • Independent QA evaluation. EgoProbe-Echo is an evaluation testbed; we do not train or adapt models on its QA pairs. Each QA is answered in an isolated branch using the video evidence or retained memory available at query time. Answers from other QA pairs are neither included in the current context nor written back to persistent memory. Thus, although all original R1 answers are Yes, models cannot learn this label pattern through cross-QA interactions during evaluation.

  • Parsing success rates. An R1 output is successfully parsed if our evaluation parser extracts a valid Yes/No answer from the returned text. We compute parsing success rate over all returned textual outputs. 14 of 16 text-generating models achieve 100%, with Qwen3.5 (95.93%) and LLaVA-OneVision-2 (99.84%) being the only exceptions. In particular, all returned outputs from Ego-R1, EgoRAG, and SimpleStream are successfully parsed. Their incorrect returned answers therefore correspond to No responses rather than parsing failures, indicating that unsuccessful text parsing does not explain their low R1 scores. StreamingVLM scores answer options directly using logits and does not require text parsing.

  • Yes-response rates. The Yes rate measures the proportion of R1 questions for which a model answers Yes. Since all original R1 questions have Yes as the correct answer, each model’s Yes rate equals its R1 Acc. reported in Table 2. Both use all 1,229 questions as the denominator; missing or unparseable outputs contribute to neither the Yes count nor the correct-answer count. Specifically, Ego-R1, EgoRAG, and SimpleStream have Yes rates of 43.53%, 44.18%, and 25.71%, respectively.

Experiment:

To further assess whether R1 reflects coarse event recall, we construct 300 paired Yes/No probes based on a subset of EgoProbe-Echo. Each original Yes question is paired with a candidate negative by replacing only its object noun phrase with a scene-plausible but semantically and visually distinct object, preserving the action, relation, and sentence structure. To construct negative examples, we retain only replacement objects with no recorded occurrence in the pre-query video. Each pair shares the same video and query time, providing a paired comparison with the visual evidence and query conditions held fixed. We report accuracy separately on the 300 positive questions (Yes Acc.) and the 300 negative questions (No Acc.).

ModelPositive questions: Yes Acc. (%) ↑Negative questions: No Acc. (%) ↑
StreamingVLM40.0041.00
HERMES58.0054.33
StreamForest90.6788.33
OASIS69.6776.00
EventMemAgent89.0073.67
VST95.0093.33

The results above show that models can answer both positive and negative R1 questions, with negative accuracy ranging from 41.00% to 93.33%. This indicates that high R1 accuracy cannot be explained solely by Yes-response bias. Their differing Yes and No accuracies further reveal variation in coarse event recall performance, supporting R1’s value in evaluating and distinguishing this capability across models.

Q3:

Thank you for these questions. We address whether the accuracy decline persists within each data source and the role of target-evidence coverage separately below.

Q3-1: Within-source analysis of recall over time.

  • Why we combine QaEgo4D and EgoLife in Base. EgoProbe-Base evaluates how long models retain past experiences across a broad range of recall intervals. QaEgo4D’s video durations limit the intervals it can support, so we incorporate EgoLife to extend evaluation to ultra-long intervals.

Experiment:

To validate the accuracy drop over time in each source dataset, we conduct strictly controlled same-question evaluations at different recall delays within QaEgo4D and EgoLife separately. We use source-specific delays to accommodate their different temporal coverage. We sample 100 QaEgo4D questions and query each at 30 seconds, 3 minutes, 10 minutes, and 30 minutes after the end of the target evidence. For EgoLife, we independently sample 100 questions and use delays of 1 minute, 3 minutes, 30 minutes, and 12 hours. Within each source, all delays use the same questions, options, option order, and correct answers; only the query time changes. We also verify through our data engine that each question remains valid at the revised query times. We retain the original streaming evaluation protocol and model configurations and report Acc. at each delay separately for each source.

QaEgo4D (Acc. (%) ↑)

Model30 s3 min10 min30 min
StreamingVLM59.0054.0050.0042.00
HERMES52.0051.0046.0039.00
StreamForest57.0052.0051.0040.00
OASIS62.0059.0053.0041.00
EventMemAgent55.0052.0047.0044.00
VST56.0054.0049.0042.00

EgoLife (Acc. (%) ↑)

Model1 min3 min30 min12 h
StreamingVLM56.0053.0039.0031.00
HERMES44.0042.0029.0023.00
StreamForest57.0050.0042.0038.00
OASIS51.0047.0041.0032.00
EventMemAgent55.0049.0039.0034.00
VST53.0050.0044.0035.00

As shown in the tables above, accuracy decreases monotonically with longer recall intervals within both datasets, showing that the decline is not solely explained by differences between data sources.

Q3-2: Target-evidence coverage.

  • Recall-interval analysis focuses on streaming models. Table 3 evaluates streaming models, rather than general-purpose models using a fixed 128-frame prefix. Streaming models continuously process the video, including the segment containing the target event, before the query arrives. Importantly, the target-event input remains unchanged across delays in the controlled experiment above, so the decline cannot be attributed to reduced target-evidence coverage at longer delays under streaming setting. These results support EgoProbe’s evaluation of how long models retain and recall previously processed experiences under their streaming memory constraints.

Experiment:

Although our recall-interval analysis focuses on streaming models, following your suggestion, we additionally examine target-evidence coverage for general-purpose models under offline evaluation. We compare Uniform sampling with Target-coverage sampling, which ensures that the input includes target-event evidence, on 300 EgoProbe-Base questions. Both settings use 128 frames, with the question, options, query time, and model configuration unchanged.

ModelUniform Acc. (%)Target-coverage Acc. (%)Acc. change (Target - Uniform)
InternVL3.5-8B50.3353.67+3.34
LLaVA-OneVision2-8B57.3361.00+3.67

The results above show that target-evidence coverage increases accuracy by 3.34–3.67 percentage points. However, during standard inference, models do not directly know where the target evidence is located. This accuracy gap therefore reflects a limitation of prefix sampling under a fixed frame budget for general-purpose models.

Reviewer_qQAzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoProbe benchmarks how long streaming egocentric video models retain information and whether they recall details from the correct event when similar events recur. It introduces event-grounded distractors and evaluates 17 models, revealing substantial recall errors.

Critical strengths
  • Event-grounded distractors make wrong answers more informative.
  • The benchmark includes human verification and a broad model evaluation.
Critical weaknesses
  • Every first-turn answer is “Yes,” so first-turn accuracy cannot establish coarse event recall. An always-Yes model would score 100% without using the video. Consequently, the gap between first-turn and detail-question accuracy does not demonstrate a gap between coarse and precise memory.
  • The comparisons across subsets and recall intervals use different questions. Differences in question difficulty may therefore explain part of the observed interference and forgetting effects. Without matched comparisons, these results establish benchmark difficulty but do not isolate its proposed causes.
Questions for authors
  • Can you add negative first-turn questions and an always-Yes baseline? This would show whether the first turn measures event recall rather than a response bias.
  • How does accuracy change on the same question with and without intervening related events? A paired comparison would directly test the effect of interference.
  • How does accuracy change when the same question is asked at different delays? Holding the question fixed would provide stronger evidence for forgetting over time.
  • Do event-grounded errors persist when the number and type of answer options are held fixed? This would help separate event confusion from the increased chance of selecting an event-grounded option.
5 comments

We thank the reviewer for recognizing the value of our event-grounded distractor design and the quality of our benchmark. We address the reviewer’s concerns below and please kindly let us know if our response addresses your concerns.

W1 & Q1: Every first-turn correct answer is “Yes”

Thank you for raising this concern. All original R1 questions have Yes as the correct answer. We clarify the design motivation and evaluation protocol below:

  • Why all original R1 answers are Yes. We construct R1 questions about observed experiences so that each dialogue can proceed from event existence to event details. R1 asks whether a coarse event occurred, and R2 then asks for a specific detail of the designated target event. The positive R1 labels thus reflect the construction of these coarse-to-fine dialogues, which are designed to probe coarse event recall followed by event-specific detail recall.

  • Independent QA evaluation. EgoProbe-Echo is an evaluation testbed; we do not train or adapt models on its QA pairs. Each QA is answered in an isolated branch using the video evidence or retained memory available at query time. Answers from other QA pairs are neither included in the current context nor written back to persistent memory. Although all original R1 answers are Yes, models cannot learn this label pattern from preceding QA pairs under our evaluation protocol. Uniform random guessing over Yes/No still yields an expected accuracy of 50%. Therefore, the all-Yes ground-truth distribution does not by itself imply that a model has a Yes-response bias.

Experiment:

To further assess whether R1 reflects coarse event recall, we construct 300 paired Yes/No probes based on a subset of EgoProbe-Echo. Each original Yes question is paired with a candidate negative by replacing only its object noun phrase with a scene-plausible but semantically and visually distinct object, preserving the action, relation, and sentence structure. To construct negative examples, we retain only replacement objects with no recorded occurrence in the pre-query video. Each pair shares the same video and query time, providing a paired comparison with the visual evidence and query conditions held fixed. We report accuracy separately on the 300 positive questions (Yes Acc.) and the 300 negative questions (No Acc.).

ModelPositive questions: Yes Acc. (%) ↑Negative questions: No Acc. (%) ↑
StreamingVLM40.0041.00
HERMES58.0054.33
StreamForest90.6788.33
OASIS69.6776.00
EventMemAgent89.0073.67
VST95.0093.33

The above results show that models can answer both positive and negative R1 questions, with negative accuracy ranging from 41.00% to 93.33%. This shows that none of the tested models simply follows an Always-Yes strategy (Yes Acc. = 100%; No Acc. = 0%). Their differing Yes and No accuracies reveal substantial variation in coarse event recall performance, supporting R1’s value in evaluating and distinguishing these capabilities across models.

W2 & Q2: Controlled comparisons of related-event interference.

Thank you for raising this concern. Below, we clarify our design and present additional controlled experiments.

  • Why Base and Echo are not strictly paired controls. As described in Section 3.2, our data engine assigns each QA to Base or Echo according to the absence or presence of intervening related events, with each QA appearing in only one subset. Base and Echo therefore contain different QA instances, rather than the same questions evaluated with and without related events. Constructing perfectly matched video pairs for every QA is impractical with real-world videos, as this would require keeping the target event and all other video content unchanged while changing only the intervening related events. We explicitly acknowledge this limitation in Section 4.2. The two subsets also focus on different aspects of memory: Base evaluates how far models recall without intervening related events, while Echo evaluates both how far and how exact they recall target-event details among related experiences.

  • Event confusion is diagnosed within Echo. Our evidence for event confusion does not rely solely on comparing Base and Echo. Within Echo, each event-grounded distractor is linked to a verified related event, allowing us to identify errors that select details belonging to another observed experience rather than the designated target event. Table 4 and Appendix E.3, Table 8 analyze these event-linked errors, with Table 4 showing Event err. consistently above the group-specific random-choice references.

Experiment:

Following your suggestion, we evaluate the same target-event questions with and without related-event content. We sample 300 QA groups from EgoProbe-Echo videos, covering both single-event and multi-event cases. For each QA, we compare Original (with related events) against Interference removed (without related events). The latter replaces frames covering each annotated related event with repeated copies of a pre-event frame. The target evidence, question, options, correct answer, query time, sampling rules, and video timeline remain unchanged. We report single-turn Acc. for both versions and the accuracy change after removing related-event content.

ModelAcc.: Original video with related eventsAcc.: Same video without related eventsAcc. change (Removed - Original)
StreamingVLM30.0039.33+9.33
HERMES31.6742.67+11.00
StreamForest38.6748.00+9.33
OASIS32.0041.67+9.67
EventMemAgent34.6746.67+12.00
VST36.3341.33+5.00

The results above show that removing related-event content increases single-turn accuracy across all evaluated models, with gains of 5.00–12.00 percentage points. These consistent increases under controlled conditions support our conclusion that intervening related events interfere with target-specific detail recall.

To further validate our benchmark’s diagnosis of interference in unedited videos, we conduct a second paired comparison of recall before versus after a related event. We independently sample 300 QA groups from EgoProbe-Echo videos, each containing one annotated related event. Queries are placed 10 seconds before and after the related event. The video, question, options, and correct answer remain unchanged, and the target evidence precedes both queries. We report single-turn Acc. before and after the event, together with the accuracy change.

ModelAcc.: Query before a related eventAcc.: Query after a related eventAcc. change (Query After Event - Query Before Event)
StreamingVLM41.6734.67-7.00
HERMES40.0034.33-5.67
StreamForest50.3345.00-5.33
OASIS50.0037.67-12.33
EventMemAgent44.6739.00-5.67
VST44.3337.67-6.66

The results above show that accuracy decreases across all models when queries are asked after the related event, with drops of 5.33–12.33 percentage points. This again highlights the challenge of recalling target-event details after related experiences and supports EgoProbe’s diagnostic value.

W2 & Q3: Same-question evaluation at different recall delays.

Thank you for your suggestion. We present additional controlled experiments below.

Experiment:

To further validate our analysis of recall over time while controlling for question difficulty, we evaluate the same questions at multiple recall delays. Since Ego4D and EgoLife differ in temporal coverage, we evaluate their corresponding EgoProbe-Base subsets separately. We sample 100 QaEgo4D questions and query each at 30 seconds, 3 minutes, 10 minutes, and 30 minutes after the end of the target evidence; for EgoLife, we independently sample 100 questions and use delays of 1 minute, 3 minutes, 30 minutes, and 12 hours. Within each source, all delays use the same questions, options, option order, and correct answers; only the query time changes. We also verify through our data engine that each question remains valid at the revised query times. We retain the original streaming evaluation protocol and model configurations and report Acc. at each delay for each source separately.

QaEgo4D (Acc. (%) ↑)

Model30 s3 min10 min30 min
StreamingVLM59.0054.0050.0042.00
HERMES52.0051.0046.0039.00
StreamForest57.0052.0051.0040.00
OASIS62.0059.0053.0041.00
EventMemAgent55.0052.0047.0044.00
VST56.0054.0049.0042.00

EgoLife (Acc. (%) ↑)

Model1 min3 min30 min12 h
StreamingVLM56.0053.0039.0031.00
HERMES44.0042.0029.0023.00
StreamForest57.0050.0042.0038.00
OASIS51.0047.0041.0032.00
EventMemAgent55.0049.0039.0034.00
VST53.0050.0044.0035.00

As shown in the tables above, model accuracy exhibits a monotonic decline as the recall interval increases across both datasets. Accuracy decreases by 11–21 percentage points on QaEgo4D and 18–25 percentage points on EgoLife between the shortest and longest delays. This consistent pattern supports EgoProbe’s evaluation of how long models retain and recall past experiences under their streaming memory constraints.

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We sincerely thank the reviewers for their constructive feedback. In response to these comments, we have revised the paper and uploaded an updated PDF, with changes highlighted in blue. The main updates are:

  1. Added discussion of EgoCITE and MemLife in Section 2.2 and comparison with EgoMemReason in Table 1. (Reviewer uqb1)
  2. Added annotator qualifications, independent review procedures, and inter-annotator agreement statistics in Appendix C.3. (Reviewers qfLJ, uqb1)
  3. Added same-question evaluations at different recall delays within QaEgo4D and EgoLife in Appendix E.1, referenced in Section 4.3. (Reviewers qQAz, qfLJ)
  4. Added an evaluation using 300 positive–negative R1 question pairs to examine coarse event recall and Yes-response bias in Appendix E.4. (Reviewers qQAz, qfLJ)
  5. Added paired related-event removal and before–after query experiments in Appendix E.5, referenced in Section 4.4. (Reviewers qQAz, qfLJ)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.