- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ProactEgoMem, a benchmark designed to evaluate proactive memory acquisition and maintenance in streaming egocentric videos. Unlike conventional video question answering benchmarks that focus on responding to user queries, ProactEgoMem aims to assess whether multimodal agents can autonomously identify relevant events, preserve useful information, and maintain memories without explicit user requests.The benchmark consists of three tasks: Information Capture (IC), State Tracking (ST), and Location Tracking (LT). It further distinguishes between two levels of proactiveness: the Notification Level, which evaluates timely event-triggered notifications, and the Intervention Level, which requires agents to manage memory tools through a start–maintain–stop lifecycle. The dataset includes 400 video clips and 1,500 QA pairs across daily-life and scientific laboratory scenarios.The authors evaluate 13 models from different model families and report substantial limitations in proactive event detection, temporal precision, tool lifecycle management, and memory-supported question answering. The results highlight several challenges in developing reliable proactive multimodal assistants.
Critical strengths
S1. A meaningful benchmark formulation for proactive memory in streaming egocentric videos. The paper addresses an important research problem by shifting the evaluation focus from reactive video understanding to proactive memory acquisition. In particular, the proposed Intervention Level extends conventional event-triggered notification by requiring agents to initiate, maintain, and terminate memory tools autonomously. This formulation introduces a meaningful temporal decision-making dimension that is insufficiently captured by conventional video QA benchmarks. I consider this task formulation the paper's strongest contribution, as it provides a potentially useful evaluation setting for future research on proactive multimodal agents. S2. A comprehensive evaluation framework across different tasks, scenarios, and model families. The benchmark covers three complementary memory tasks and includes both daily-life and scientific laboratory scenarios. The evaluation of 13 models, together with the analysis of temporal accuracy, event detection, tool lifecycle management, and downstream QA performance, provides useful empirical insights into current model limitations. The distinction between memory acquisition and subsequent memory utilization is particularly valuable. These results could help identify research directions for developing more reliable proactive memory systems.
Critical weaknesses
W1. Concerns about the temporal evaluation protocol and the interpretation of model failures. A major concern is the potential mismatch between the primary temporal metric and the streaming evaluation protocol. Section 5.1 adopts [email protected], which requires a predicted action to occur within 200 milliseconds of the ground-truth event. However, Section 5.2 evaluates models at 1 FPS, allowing tool calls only at discrete one-second intervals. Although Appendix E reports results under relaxed temporal thresholds, the impact of this discretization on the primary metric remains insufficiently clarified. Further, the evaluation does not fully disentangle failures in proactive event understanding from failures in tool-call formatting and execution. As discussed in Section 5.6, some models produce free-form descriptions rather than valid structured tool calls, while others repeatedly initiate or fail to terminate tracking tools. Although these behaviors represent practical deployment challenges, they do not necessarily indicate failures in visual event understanding or memory selection. W2. Limited validation of the proposed notion of proactiveness and the benchmark's generalization scope. Another important concern is whether the benchmark sufficiently evaluates proactive memory decisions rather than predefined event detection. The benchmark is constructed around annotated object-centric events, such as object movements, state changes, and transient visual information. However, detecting an event does not necessarily imply that the event is important enough to warrant memory storage or user notification. A human preference or event-importance evaluation would help establish the validity of the proposed operational definition of proactiveness.In addition, the benchmark's scope appears relatively limited compared with its broader motivation of long-horizon proactive memory. The dataset contains 400 clips and approximately 22.5 hours of video in total, with individual clips averaging only a few minutes. This setting does not adequately evaluate hour-scale continuous memory maintenance, long-term forgetting, or cross-session memory consistency. Moreover, Appendix B claims that the benchmark guarantees out-of-distribution evaluation because of newly constructed annotations and QA pairs. However, new annotations alone cannot guarantee that the underlying publicly available videos were absent from the evaluated models' pretraining data. These limitations weaken the generality of the conclusions regarding long-horizon proactive memory and out-of-distribution evaluation.
Questions for authors
Q1. Temporal evaluation validity (Sections 5.1–5.2 and Appendix E). Could the authors clarify whether the ground-truth event timestamps are quantized or aligned with the 1-FPS decision grid? If not, how do they justify using [email protected] as the primary timing metric when tool calls can only occur at one-second intervals? In addition to the existing threshold sensitivity analysis, reporting a discretization-aware timing metric or event-level timely recall would help determine whether the observed performance limitations arise from the models or from the evaluation protocol. Q2. Validity of proactive memory opportunities (Section 4). How do the authors determine whether an annotated event genuinely warrants proactive memory acquisition or notification, rather than merely representing a detectable visual event? Have the authors conducted any human evaluation of event importance, memory usefulness, or potential interruption costs? Such evidence would substantially strengthen the claim that ProactEgoMem evaluates proactive memory decisions rather than predefined event detection. Q3. Separating memory acquisition from tool execution failures (Sections 5.4–5.6).Can the authors provide a more fine-grained analysis separating visual event detection, memory selection, structured tool-call compliance, and lifecycle management? In particular, how much of the performance degradation is attributable to invalid or unparseable tool calls? Additionally, how would simple event-driven tracking or continuous memory acquisition perform under a comparable memory budget? Q4. Evaluation scope and out-of-distribution claims (Section 4 and Appendix B). Could the authors clarify the extent to which the current short-clip evaluation supports the paper's claims about long-horizon memory? Additionally, given that the underlying videos are drawn from publicly available datasets, what evidence supports the assertion that the benchmark guarantees out-of-distribution evaluation? Clarifying the distinction between novel annotations, unseen evaluation tasks, and potential overlap with pretraining video content would improve the credibility of this claim.
4 comments
Response to Reviewer Fg8j (2/4)
Table R8. Start@k. Intervention level, Daily dev, IC/ST/LT combined. Start@k in %. -: no match.
| Group | Model | [email protected] 1 fps | [email protected] 5 fps | [email protected] Event-driven | Start@1 1 fps | Start@1 5 fps | Start@1 Event-driven | Start@2 1 fps | Start@2 5 fps | Start@2 Event-driven | Start@5 1 fps | Start@5 5 fps | Start@5 Event-driven |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Open Source | Gemma4-31B | 11.3 | 15.9 | 13.8 | 43.4 | 47.6 | 44.6 | 58.5 | 63.5 | 64.6 | 88.7 | 88.9 | 86.2 |
| Qwen3.8-27B | 17.9 | 11.1 | 21.7 | 50.0 | 47.2 | 52.2 | 64.3 | 72.2 | 82.6 | 85.7 | 91.7 | 95.7 | |
| Qwen3.6-27B | 6.1 | 14.9 | 26.5 | 21.2 | 48.9 | 55.9 | 51.5 | 66.0 | 79.4 | 84.8 | 93.6 | 97.1 | |
| Qwen3.5-9B | 0.0 | 0.0 | 0.0 | 0.0 | 9.1 | 33.3 | 25.0 | 27.3 | 50.0 | 100.0 | 72.7 | 83.3 | |
| VidLLM | VideoLLaMA3-7B | 11.1 | 0.0 | 9.1 | 16.7 | 23.1 | 27.3 | 22.2 | 42.3 | 36.4 | 61.1 | 73.1 | 59.1 |
| LLaVA-OV-1.5-8B | 0.0 | 100.0 | 14.3 | 0.0 | 100.0 | 14.3 | 0.0 | 100.0 | 28.6 | 0.0 | 100.0 | 42.9 | |
| MiniCPM-o-4.5 | - | 0.0 | 0.0 | - | 0.0 | 0.0 | - | 100.0 | 16.7 | - | 100.0 | 16.7 | |
| Streaming | MMDuet | 0.0 | 0.0 | 0.0 | 25.0 | 0.0 | 50.0 | 75.0 | 100.0 | 75.0 | 100.0 | 100.0 | 100.0 |
| VideoLLM-Online | - | - | - | - | - | - | - | - | - | - | - | - | |
| JoyAI-VL | - | - | - | - | - | - | - | - | - | - | - | - | |
| All models | 10.6 | 11.8 | 15.6 | 33.8 | 41.9 | 42.5 | 52.1 | 61.3 | 62.9 | 83.1 | 87.6 | 82.0 |
W2-a (notion of proactiveness). We evaluate in a query-free setting: the model maintains ambient memory to offload the user's mental burden, which requires it to infer user intent, and an event counts only if a majority of three annotators judged it useful for later queries. The benchmark therefore scores the decision of what to remember, not detection alone. Finer-grained importance and interruption cost differ across user personas (e.g., quiet vs. chatty preferences drive opposite metric trends); we will discuss personalized proactivity as a limitation and next step.
W2-b (scope and out-of-distribution claims).
- We will narrow "long-horizon" and list hour-scale and cross-session memory as future work; minute-long, densely annotated clips suffice for our main topic of proactive memory.
- We will clarify in Appendix B that the benchmark provides novel labels and tasks. Blind QA (question only, Table R6) stays near the 25% chance level on ST (25.7%) and LT (29.5%), so these answers cannot come from priors or memorized video; IC (Information Capture, 45.3%) is due to text-rich multimodal pretraining.
Table R6. QA accuracy (%), Daily: Blind QA (question only) vs. Proact (the model's own proactive memory). Mean over 8 models. Chance 25.0.
| Group | Model | IC Blind QA | IC Proact | IC Δ | ST Blind QA | ST Proact | ST Δ | LT Blind QA | LT Proact | LT Δ |
|---|---|---|---|---|---|---|---|---|---|---|
| Open Source | Gemma4-31B | 57.8 | 67.9 | +10.1 | 32.0 | 32.8 | +0.8 | 35.3 | 32.1 | -3.2 |
| Qwen3.8-27B | 57.6 | 62.3 | +4.7 | 32.0 | 31.5 | -0.4 | 36.1 | 31.3 | -4.8 | |
| Qwen3.6-27B | 56.2 | 63.5 | +7.3 | 19.9 | 29.0 | +9.1 | 30.2 | 27.8 | -2.4 | |
| Qwen3.5-9B | 44.0 | 52.3 | +8.3 | 21.6 | 14.1 | -7.5 | 31.0 | 24.6 | -6.3 | |
| VidLLM | VideoLLaMA3-7B | 37.5 | 46.0 | +8.5 | 24.1 | 27.4 | +3.3 | 27.0 | 34.5 | +7.5 |
| LLaVA-OV-1.5-8B | 36.9 | 39.8 | +3.0 | 23.2 | 18.3 | -5.0 | 23.0 | 22.2 | -0.8 | |
| MiniCPM-o-4.5 | 34.9 | 34.7 | -0.2 | 27.4 | 17.0 | -10.4 | 24.2 | 21.4 | -2.8 | |
| Streaming | MMDuet | 37.3 | 41.8 | +4.5 | 25.7 | 23.2 | -2.5 | 29.0 | 30.2 | +1.2 |
| Mean | 45.3 | 51.0 | +5.8 | 25.7 | 24.2 | -1.6 | 29.5 | 28.0 | -1.4 |
Q1. Not quantized. At 5 FPS, [email protected] rises only from 10.6% to 11.8% and the mean start error stays at 2.33 s (Table R7 in W1), so the 1 FPS grid is not what limits timing; Table R8 in W1 reports Start@k for k = 0.2, 1, 2, and 5 s as the discretization-aware view.
Q2. Within fixed task rules, an event is kept only if a majority of three annotators judge it useful for the stated user need. Each annotated event is paired with questions that require remembering it, so usefulness is measured objectively by downstream QA, and acting on non-annotated events is a false alarm. Human judgments of importance and interruption cost depend on personal preference and are left to future work (see W2-a).
Q3. Within the rebuttal time constraints, we re-evaluated the models on a subset to test the following factors:
- (1) Tool-call compliance: unparseable output is confined to MMDuet and VideoLLM-Online; for the other 11 models, it is at most 0.1%, and the remaining tool errors are malformed memory (format) and tool invalid maintenance (lifecycle management; see Table R1 in W1, per-task breakdown below). Memory selection (event detection) errors are not tool errors; they appear as missed events and false alarms, i.e., recall and precision in the main tables.
Response to Reviewer Fg8j (3/4)
- (2) Event-driven tracking (Table R3): on a dev split, we designed an explicit prompt suite that instructs the model on the features of target events in detail instead of the query-free setting in the benchmark; the query-grounded description of the target triggering event raises open-source models by +7.2 to +27.6 points on IC and up to +12.8 on ST, while LT changes little (-1.6 to +4.9) and streaming models barely move, so selecting what to remember from the user's need is still the main bottleneck, even with an explicit query revealing user's intent.
- (3) Continuous acquisition: in Sec. 5.4, the No Memory setting samples 40 frames uniformly across the whole video clip, mimicking continuous acquisition, utilizing the same budget as proactive memory. Its QA accuracy still stays low.
All three points indicate that the current bottleneck is mainly proactive memory acquisition itself.
Table R1 (per task). Error shares (%) per task, Daily / Science, for the three error columns of Table R1 (definitions above). The remaining share of each task is silent or healthy.
| Model | IC Unparseable | IC Malformed Memory | IC Tool Invalid Maintenance | ST Unparseable | ST Malformed Memory | ST Tool Invalid Maintenance | LT Unparseable | LT Malformed Memory | LT Tool Invalid Maintenance |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.4-mini | 0.0 / 0.0 | 0.0 / 0.0 | 1.3 / 2.1 | 0.0 / 0.0 | 0.0 / 0.0 | 1.6 / 1.2 | 0.0 / 0.0 | 0.0 / 0.0 | 0.1 / 0.8 |
| Gemini 2.5 Flash | 0.1 / 0.0 | 0.0 / 0.0 | 1.8 / 1.5 | 0.2 / 0.0 | 0.0 / 0.0 | <0.1 / 1.1 | 0.1 / 0.0 | 52.5 / 0.0 | <0.1 / 0.1 |
| Gemma4-31B | 0.0 / 0.0 | 0.0 / 0.0 | <0.1 / 0.1 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / <0.1 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 |
| Qwen3.8-27B | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / <0.1 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / <0.1 | 0.2 / 0.0 |
| Qwen3.6-27B | 0.0 / 0.0 | 0.0 / 0.0 | 0.8 / 0.9 | <0.1 / <0.1 | 0.0 / 0.0 | <0.1 / <0.1 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 |
| Qwen3.5-9B | <0.1 / 0.0 | 0.0 / 0.0 | 0.1 / <0.1 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 |
| VideoLLaMA3-7B | 0.0 / 0.0 | 0.3 / 0.2 | 52.8 / 20.8 | <0.1 / 0.0 | 0.0 / 0.4 | 75.5 / 73.6 | <0.1 / 0.0 | <0.1 / 0.0 | 75.0 / 38.8 |
| LLaVA-OV-1.5-8B | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | <0.1 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 |
| MiniCPM-o-4.5 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.0 / 0.0 | 0.2 / 0.0 |
| MMDuet | 81.3 / 68.7 | 6.6 / 16.1 | 9.3 / 10.1 | 74.2 / 66.1 | 24.5 / 12.7 | 0.2 / 7.7 | 68.2 / 71.1 | 31.7 / 7.2 | 0.0 / 8.2 |
| VideoLLM-Online | 100.0 / 100.0 | 0.0 / 0.0 | 0.0 / 0.0 | 100.0 / 100.0 | 0.0 / 0.0 | 0.0 / 0.0 | 100.0 / 100.0 | 0.0 / 0.0 | 0.0 / 0.0 |
| Inkling-Small | 0.0 / 0.0 | 0.3 / 0.1 | 69.2 / 69.4 | 0.0 / 0.0 | 0.0 / 0.0 | 43.6 / 38.1 | 0.0 / 0.0 | <0.1 / 0.0 | 12.6 / 16.7 |
| JoyAI-VL | 0.1 / 0.1 | 0.1 / 0.1 | <0.1 / <0.1 | 0.1 / 0.1 | 0.2 / 0.1 | 0.0 / <0.1 | 0.1 / 0.1 | 0.1 / <0.1 | 0.0 / 0.0 |
| Mean over models | 14.0 / 13.0 | 0.6 / 1.3 | 10.4 / 8.1 | 13.4 / 12.8 | 1.9 / 1.0 | 9.3 / 9.4 | 13.0 / 13.2 | 6.5 / 0.6 | 6.8 / 5.0 |
Table R3. Paper prompt vs. event-driven prompt (the prompt explains the target triggering event), Intervention level, dev split.
| Group | Model | IC: paper → event-driven (Δ) | ST: paper → event-driven (Δ) | LT: paper → event-driven (Δ) |
|---|---|---|---|---|
| Open source | Gemma4-31B | 2.8 → 16.6 (+13.8) | 16.5 → 28.0 (+11.5) | 7.6 → 12.5 (+4.9) |
| Open source | Qwen3.8-27B | 3.5 → 31.1 (+27.6) | 28.0 → 27.9 (-0.1) | 10.2 → 10.8 (+0.6) |
| Open source | Qwen3.6-27B | 4.2 → 24.0 (+19.8) | 12.4 → 25.2 (+12.8) | 21.6 → 20.0 (-1.6) |
| Open source | Qwen3.5-9B | 6.8 → 14.0 (+7.2) | 0.0 → 6.1 (+6.1) | 0.0 → 3.4 (+3.4) |
| Video-LLM | VideoLLaMA3-7B | 2.5 → 0.7 (-1.9) | 1.2 → 5.6 (+4.4) | 4.1 → 3.2 (-0.9) |
| Video-LLM | LLaVA-OV-1.5-8B | 5.0 → 15.4 (+10.4) | 0.0 → 2.7 (+2.7) | 3.2 → 9.5 (+6.3) |
| Video-LLM | MiniCPM-o-4.5 | 6.1 → 20.0 (+13.9) | 0.0 → 6.2 (+6.2) | 3.4 → 0.0 (-3.4) |
| Streaming | MMDuet | 0.0 → 4.3 (+4.3) | 0.0 → 6.6 (+6.6) | 0.0 → 0.0 (0.0) |
| Streaming | VideoLLM-Online | 0.0 → 0.0 (0.0) | 0.0 → 0.0 (0.0) | 0.0 → 0.0 (0.0) |
| Streaming | JoyAI-VL | 0.0 → 0.0 (0.0) | 0.0 → 0.0 (0.0) | 0.0 → 0.0 (0.0) |
Response to Reviewer Fg8j (4/4)
Q4.
- (1) We agree that these videos are not extremely long, and we will replace "long-horizon" with this narrower claim and list hour-scale maintenance and cross-session memory as future work. Again, even with minute-long clips, the dense annotations do not weaken our goal of benchmarking proactive memories.
- (2) Out-of-distribution. We will soften "guarantees" in Appendix B and state three separate levels:
- Annotations are new: event intervals, tool lifecycles, and QA pairs were created for ProactEgoMem and were not public before this submission.
- The task is new: to our knowledge, no existing dataset supervises when to start, maintain, and stop memory tools during a stream.
- The videos are not guaranteed unseen: they come from public datasets and may appear in pretraining data. As the No Memory setting in Sec. 5.4 shows, the video input alone does not answer the questions, and blind QA (question only, Table R6 in W2-b) stays near the 25% chance level on ST (25.7%) and LT (29.5%). On IC, it reaches 45.3% via text priors, due to text-rich multimodal pretraining across all multimodal models, and we will add the discussion in the revision too.
Response to Reviewer Fg8j (1/4)
W1 (timing protocol and tool-call failures).
Table R1. Intervention-level step outcomes (%), Daily / Science, all three tasks pooled. Each reply is counted once under its worst outcome; each row sums to 100. Healthy tool call means the tool accepted the call, not that it matched an annotated event. Malformed Memory: valid JSON whose memory tool call does not follow the required schema (missing keys, unknown tool name, or wrong fields); it is not executed. Tool Invalid Maintenance: a well-formed call that is impossible in the current tool state, e.g. stopping a recording that is not running. JSON mode follows the released configs.
Table R7. 1 fps vs 5 fps. Intervention level, Daily dev, IC/ST/LT combined. |Δs|, σ in s; others %. -: no match.