- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper identifies three limitations of previous works: 1. designed for generic video rather than on egocentric streams; 2. does not imply long-range memory dependence and many queries may be answered from recent observations; 3. largely focus on how far information can be recalled rather than how precisely model recall details. Thus, the authors propose EgoProbe which evaluates in egocentric streams that requires long-range memory and evaluates how exact the model can recall details of target event. The dataset is built from Ego4D and EgoLife dataset and build two EgoProbe-Base, questions without intervening related events and EgoProbe-Echo, questions with contrastive related events.
Critical strengths
- The paper addresses an important problem for egocentric assistants which evaluates how well it retains information over long periods and can distinguish details from similar experiences.
- The authors conduct wide experiment over 17 models of broad model families and provide analysis.
Critical weaknesses
- The distinction from previous works (table 1) is insufficiently discussed. How is the benchmark different from existing benchmarks such as MemLife[1], EgoMemReason[2], and EgoCITE[3]?
- Can the authors provide more analysis or discussion on why model rankings varies across evaluation settings?
[1] MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories [2] EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding [3] EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory
Questions for authors
- How are the annotators recruited? How many annotators reviewed each instance and what was the inter-annotator agreement? I wonder if there was any biases introduced from automatic multiple-choice generation.
- How would the number of distractor affect on the performance? Can the authors also conduct analysis by matching the number?
5 comments
W2: Model rankings across evaluation settings.
Thank you. EgoProbe provides a diagnostic testbed for evaluating different dimensions of streaming memory. Some ranking variations are expected because different settings place different demands on memory capabilities. Following your suggestion, we further explain these variations through overall ranking consistency, the different capabilities required across settings, and model-specific memory mechanisms.
- Overall ranking consistency across evaluation settings. We quantify consistency using Spearman’s rank correlation coefficient (ρ), computed from models’ relative rankings across two settings. It ranges from −1 to 1, with values closer to 1 indicating stronger agreement. Across all eight streaming models in Table 2, all three correlations are high, indicating broadly consistent rankings across settings, particularly between Base and Echo single-turn evaluation (ρ = 0.976).
| Evaluation settings compared | Spearman’s ρ |
|---|---|
| Base single-turn Acc. vs. Echo single-turn Acc. | 0.976 |
| Base single-turn Acc. vs. Echo multi-turn R2 Acc. | 0.905 |
| Echo single-turn Acc. vs. Echo multi-turn R2 Acc. | 0.833 |
-
Different dimensions of memory across evaluation settings. Although all settings evaluate memory, they emphasize different capabilities. Base focuses on long-range detail retention without intervening related events; Echo single-turn emphasizes recalling the correct target-event detail among similar experiences; and Echo multi-turn additionally requires successful coarse event recall before precise detail recall. A model’s strength in one dimension does not necessarily imply equal strength in another, helping explain local ranking differences. This reflects EgoProbe’s design to diagnose complementary dimensions of memory.
-
Model-specific memory mechanisms. First, access to distant evidence explains stable performance differences. SimpleStream’s four-frame recent window in our configuration and StreamingVLM’s bounded visual and textual windows restrict access to distant details, consistent with their lower rankings across all three settings.
Second, event-level organization supports long-range recall. StreamForest maintains event trees with selective compression, while EventMemAgent stores retrievable event descriptions, visual anchors, and change records. Both remain among the top three across settings, with StreamForest consistently ranking first.
Third, memory representations align with different recall strengths. VST records streaming thoughts, while EventMemAgent additionally preserves visual anchors and change records. This distinction aligns with VST’s higher coarse R1 accuracy (95.28% vs. 88.36%) and EventMemAgent’s higher standalone detail accuracy (37.75% vs. 35.48%). Under two-turn evaluation, which requires both stages to succeed, VST ranks higher (33.93% vs. 32.63%), illustrating how different recall requirements change model rankings.
Q1: Annotator qualifications, agreement, and answer-option bias.
Thank you for these questions. We clarify the annotators’ qualifications, review procedure, inter-annotator agreement, and checks for potential bias in answer options below.
-
Annotator qualifications and review procedure. Our annotation team consists of 3 members recruited from the project team, all holding at least a master’s degree and having research experience in video understanding, VQA, and related areas. Before formal review, we developed detailed guidelines and examples and conducted a demonstration to align the review criteria. Each of the 2,555 candidate QA items was independently reviewed by 2 annotators using four categories: Accepted, Needs revision, Recommend exclusion, and Cannot determine. Disagreements were examined and resolved by the third annotator, who also serves as the project lead.
-
Inter-annotator agreement. The two independent reviewers assigned identical judgments to 2,421 of the 2,555 items, yielding 94.76% agreement and Cohen’s κ = 0.831. Both metrics were computed before adjudication and reflect the consistency of the independent review judgments.
-
Checks for bias in automatically generated answer options. We address this concern in two ways:
-
Human verification and correction. Reviewers assess whether the options are reasonable, distinct, and appropriate for the question, with exactly one correct answer. They verify the correct answer against the video and confirm that each event-grounded distractor is supported by another related event but remains incorrect for the target event. Identified issues are corrected, and revised items are rechecked before acceptance.
-
Text-only evaluation. In Appendix E.2, Table 7, we examine potential bias in the final questions and options. Without video input, Qwen3.5 achieves 25.51% on Base and 24.49% on Echo, close to the 25% random-choice baseline, compared with 45.91% and 35.31% with video input. These results show no evidence of answer-option bias in this text-only evaluation.
We thank the reviewer for the feedback and for recognizing the importance of evaluating long-term memory and event-specific recall for egocentric videos, as well as our extensive evaluation across 17 models from diverse model families. Below, we address your concerns with clarifications, further analyses, and additional controlled experiments.
W1: Relationship to MemLife, EgoMemReason, and EgoCITE.
Thank you for bringing these related works to our attention. We first summarize their release dates, publication/submission status, and primary contributions below.
According to the ICLR 2027 Reviewer Guidelines, authors are not required to compare against contemporaneous work published at peer-reviewed venues on or after July 17, 2026, or against unpublished arXiv preprints. The guidelines explicitly state that the absence of such comparisons cannot be a basis for rejection. MemLife and EgoCITE fall under the provision for unpublished arXiv preprints, while EgoMemReason’s official COLM proceedings metadata records its publication date as October 15, 2026, placing it within the contemporaneous-work window. In particular, MemLife’s first arXiv version appeared after the ICLR 2027 full-paper submission deadline. We therefore sincerely hope that the reviewer will reconsider treating the absence of these comparisons as a weakness of our original submission, in light of the publication timeline and the ICLR policy.
Nevertheless, we appreciate these suggestions and have cited and discussed all three works in the revised paper. Below, we provide a direct comparison with EgoMemReason, the benchmark among these works, focusing on its evaluation setting and benchmark design.
Comparison with EgoMemReason:
Different research scopes: offline video versus streaming setting. EgoMemReason evaluates offline long-video reasoning with query-time access to sampled historical frames. EgoProbe targets strict streaming memory: models process video and update memory without knowing future questions, then answer using retained memory without reloading historical video. EgoMemReason itself acknowledges in Appendix D that its evaluation relies on sampled frames and optional auxiliary inputs, and identifies streaming-based evaluation as future work. These are fundamentally different evaluation settings, imposing different requirements on memory construction and retention.
Different question designs and diagnostic goals. EgoMemReason emphasizes reasoning over multiple pieces of evidence through entity tracking, counting, event ordering and linking, and behavior-pattern inference. EgoProbe focuses on how exactly a model recalls a designated event after a long interval, particularly when similar intervening experiences contain competing details. Its coarse-to-fine dialogues further distinguish coarse event recall from target-specific detail recall.
Event-linked diagnostics beyond accuracy. EgoMemReason primarily reports QA accuracy. EgoProbe-Echo additionally links event-grounded distractors to verified related events, enabling Event err. and Other err. analyses to distinguish event confusion from other errors.
We also added EgoMemReason to the benchmark comparison in Table 1.