acceptodds
Under review as a conference paper at ICLR 2027

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

Abstract

Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should proactively acquire and maintain useful memory as the user's surroundings evolve. Remembering the ambience on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user focuses on other activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The evaluation protocol spans two levels of proactiveness: the Notification Level asks for one timely notification that satisfies the user's intent; the deeper Intervention Level asks for memory maintenance through an active start-to-stop tool lifecycle on the user's behalf. Two parallel tracks, ProactEgoMem-Daily and ProactEgoMem-Sci, contain 300 and 100 clips for daily and scientific use cases, respectively, paired with 1000 and 500 questions in ProactEgoMem-QA, which measures the quality of proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory proactively with accurate timing and content. ProactEgoMem provides a rigorous evaluation foundation for multimodal proactive assistance and co-existing video agents.

Primary area:
datasets and benchmarks
Keywords:
Proactive Agents, Proactive Memory, Streaming Video Understanding, Egocentric Video

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fg8jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ProactEgoMem, a benchmark designed to evaluate proactive memory acquisition and maintenance in streaming egocentric videos. Unlike conventional video question answering benchmarks that focus on responding to user queries, ProactEgoMem aims to assess whether multimodal agents can autonomously identify relevant events, preserve useful information, and maintain memories without explicit user requests.The benchmark consists of three tasks: Information Capture (IC), State Tracking (ST), and Location Tracking (LT). It further distinguishes between two levels of proactiveness: the Notification Level, which evaluates timely event-triggered notifications, and the Intervention Level, which requires agents to manage memory tools through a start–maintain–stop lifecycle. The dataset includes 400 video clips and 1,500 QA pairs across daily-life and scientific laboratory scenarios.The authors evaluate 13 models from different model families and report substantial limitations in proactive event detection, temporal precision, tool lifecycle management, and memory-supported question answering. The results highlight several challenges in developing reliable proactive multimodal assistants.

Critical strengths

S1. A meaningful benchmark formulation for proactive memory in streaming egocentric videos. The paper addresses an important research problem by shifting the evaluation focus from reactive video understanding to proactive memory acquisition. In particular, the proposed Intervention Level extends conventional event-triggered notification by requiring agents to initiate, maintain, and terminate memory tools autonomously. This formulation introduces a meaningful temporal decision-making dimension that is insufficiently captured by conventional video QA benchmarks. I consider this task formulation the paper's strongest contribution, as it provides a potentially useful evaluation setting for future research on proactive multimodal agents. S2. A comprehensive evaluation framework across different tasks, scenarios, and model families. The benchmark covers three complementary memory tasks and includes both daily-life and scientific laboratory scenarios. The evaluation of 13 models, together with the analysis of temporal accuracy, event detection, tool lifecycle management, and downstream QA performance, provides useful empirical insights into current model limitations. The distinction between memory acquisition and subsequent memory utilization is particularly valuable. These results could help identify research directions for developing more reliable proactive memory systems.

Critical weaknesses

W1. Concerns about the temporal evaluation protocol and the interpretation of model failures. A major concern is the potential mismatch between the primary temporal metric and the streaming evaluation protocol. Section 5.1 adopts [email protected], which requires a predicted action to occur within 200 milliseconds of the ground-truth event. However, Section 5.2 evaluates models at 1 FPS, allowing tool calls only at discrete one-second intervals. Although Appendix E reports results under relaxed temporal thresholds, the impact of this discretization on the primary metric remains insufficiently clarified. Further, the evaluation does not fully disentangle failures in proactive event understanding from failures in tool-call formatting and execution. As discussed in Section 5.6, some models produce free-form descriptions rather than valid structured tool calls, while others repeatedly initiate or fail to terminate tracking tools. Although these behaviors represent practical deployment challenges, they do not necessarily indicate failures in visual event understanding or memory selection. W2. Limited validation of the proposed notion of proactiveness and the benchmark's generalization scope. Another important concern is whether the benchmark sufficiently evaluates proactive memory decisions rather than predefined event detection. The benchmark is constructed around annotated object-centric events, such as object movements, state changes, and transient visual information. However, detecting an event does not necessarily imply that the event is important enough to warrant memory storage or user notification. A human preference or event-importance evaluation would help establish the validity of the proposed operational definition of proactiveness.In addition, the benchmark's scope appears relatively limited compared with its broader motivation of long-horizon proactive memory. The dataset contains 400 clips and approximately 22.5 hours of video in total, with individual clips averaging only a few minutes. This setting does not adequately evaluate hour-scale continuous memory maintenance, long-term forgetting, or cross-session memory consistency. Moreover, Appendix B claims that the benchmark guarantees out-of-distribution evaluation because of newly constructed annotations and QA pairs. However, new annotations alone cannot guarantee that the underlying publicly available videos were absent from the evaluated models' pretraining data. These limitations weaken the generality of the conclusions regarding long-horizon proactive memory and out-of-distribution evaluation.

Questions for authors

Q1. Temporal evaluation validity (Sections 5.1–5.2 and Appendix E). Could the authors clarify whether the ground-truth event timestamps are quantized or aligned with the 1-FPS decision grid? If not, how do they justify using [email protected] as the primary timing metric when tool calls can only occur at one-second intervals? In addition to the existing threshold sensitivity analysis, reporting a discretization-aware timing metric or event-level timely recall would help determine whether the observed performance limitations arise from the models or from the evaluation protocol. Q2. Validity of proactive memory opportunities (Section 4). How do the authors determine whether an annotated event genuinely warrants proactive memory acquisition or notification, rather than merely representing a detectable visual event? Have the authors conducted any human evaluation of event importance, memory usefulness, or potential interruption costs? Such evidence would substantially strengthen the claim that ProactEgoMem evaluates proactive memory decisions rather than predefined event detection. Q3. Separating memory acquisition from tool execution failures (Sections 5.4–5.6).Can the authors provide a more fine-grained analysis separating visual event detection, memory selection, structured tool-call compliance, and lifecycle management? In particular, how much of the performance degradation is attributable to invalid or unparseable tool calls? Additionally, how would simple event-driven tracking or continuous memory acquisition perform under a comparable memory budget? Q4. Evaluation scope and out-of-distribution claims (Section 4 and Appendix B). Could the authors clarify the extent to which the current short-clip evaluation supports the paper's claims about long-horizon memory? Additionally, given that the underlying videos are drawn from publicly available datasets, what evidence supports the assertion that the benchmark guarantees out-of-distribution evaluation? Clarifying the distinction between novel annotations, unseen evaluation tasks, and potential overlap with pretraining video content would improve the credibility of this claim.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Fg8j (1/4)

W1 (timing protocol and tool-call failures).

  • (1) GT start times are annotator averages on direct timestamps, not on the 1 FPS grid. [email protected] matches the 0.22 s inter-annotator start error.
  • (2) We evaluate a denser 5 FPS for [email protected], and it barely changes the latency failure (Table R7): [email protected] moves from 10.6% to 11.8% and the mean start error from 2.70 s to 2.33 s, still longer than two 1 FPS steps; looser thresholds gain more (Table R8), **so the finer grid does not remove the lateness. We will use Start@1 in the main table (1 FPS) and will add a full 5 FPS run with [email protected] in the appendix later (5 FPS runs incur quadratic computational overhead, about 25 times as many input tokens). We report the results that could finish within the rebuttal period.
  • (3) Unparseable output is at most 0.1% for 11 of the 13 models, so formatting does not explain their failures. It is confined to MMDuet and VideoLLM-Online, and we design a lenient parser to remove the JSON parsing factor to repair their replies, but no matched events increases for memory acquisition performance

Table R1. Intervention-level step outcomes (%), Daily / Science, all three tasks pooled. Each reply is counted once under its worst outcome; each row sums to 100. Healthy tool call means the tool accepted the call, not that it matched an annotated event. Malformed Memory: valid JSON whose memory tool call does not follow the required schema (missing keys, unknown tool name, or wrong fields); it is not executed. Tool Invalid Maintenance: a well-formed call that is impossible in the current tool state, e.g. stopping a recording that is not running. JSON mode follows the released configs.

GroupModelJSON modeSilentHealthy tool callUnparseable replyMalformed MemoryTool Invalid Maintenance
ProprietaryGPT-5.4-mini✓96.6 / 92.32.4 / 6.40.0 / 0.00.0 / 0.01.0 / 1.3
ProprietaryGemini 2.5 Flash✓73.8 / 84.66.7 / 14.50.1 / 0.018.7 / 0.00.6 / 0.9
Open sourceGemma4-31B✓71.9 / 39.328.1 / 60.60.0 / 0.00.0 / 0.0<0.1 / <0.1
Open sourceQwen3.8-27B✓88.3 / 82.011.6 / 18.00.0 / 0.00.0 / <0.10.1 / <0.1
Open sourceQwen3.6-27B✓83.8 / 65.816.0 / 33.9<0.1 / <0.10.0 / 0.00.3 / 0.3
Open sourceQwen3.5-9B✓99.5 / 99.40.5 / 0.6<0.1 / 0.00.0 / 0.0<0.1 / <0.1
Video-LLMVideoLLaMA3-7B✗9.6 / 15.722.3 / 36.2<0.1 / 0.00.1 / 0.268.0 / 48.0
Video-LLMLLaVA-OV-1.5-8B✗99.8 / 99.80.2 / 0.20.0 / 0.00.0 / 0.0<0.1 / 0.0
Video-LLMMiniCPM-o-4.5✗99.9 / 100.0<0.1 / 0.00.0 / 0.00.0 / 0.00.1 / 0.0
StreamingMMDuet✗<0.1 / 0.71.4 / 9.574.9 / 68.820.1 / 12.23.6 / 8.8
StreamingVideoLLM-Online✗0.0 / 0.00.0 / 0.0100.0 / 100.00.0 / 0.00.0 / 0.0
StreamingInkling-Small✗3.4 / 5.255.7 / 54.80.0 / 0.00.1 / <0.140.8 / 40.0
StreamingJoyAI-VL✗99.7 / 99.8<0.1 / <0.10.1 / 0.10.1 / 0.1<0.1 / <0.1
AllMean over models63.6 / 60.311.2 / 18.113.5 / 13.03.0 / 1.08.8 / 7.6

Table R7. 1 fps vs 5 fps. Intervention level, Daily dev, IC/ST/LT combined. |Δs|, σ in s; others %. -: no match.

GroupModelmIoU 1 fpsmIoU 5 fps[email protected] 1 fps[email protected] 5 fps|Δs| 1 fps|Δs| 5 fpsσ 1 fpsσ 5 fpsPrec 1 fpsPrec 5 fpsRec 1 fpsRec 5 fpsF1 1 fpsF1 5 fps
Open SourceGemma4-31B8.66.411.315.92.312.132.242.344.82.037.544.58.03.7
Qwen3.8-27B6.88.017.911.12.142.002.282.3512.810.217.523.214.613.8
Qwen3.6-27B6.86.76.114.92.691.842.271.959.95.822.431.912.89.5
Qwen3.5-9B1.33.00.00.03.284.051.282.419.24.33.38.33.65.1
VidLLMVideoLLaMA3-7B2.93.011.10.04.413.583.012.781.00.713.418.71.91.4
LLaVA-OV-1.5-8B0.40.10.0100.05.880.040.070.005.62.21.70.62.50.9
MiniCPM-o-4.50.00.2-0.0-1.60-0.000.05.60.01.10.01.8
StreamingMMDuet0.90.30.00.01.931.600.690.004.30.92.20.52.90.7
VideoLLM-Online0.00.0------0.00.00.00.00.00.0
JoyAI-VL0.00.0------0.00.00.00.00.00.0
All models2.82.810.611.82.702.332.442.424.83.29.812.94.63.7
AuthorsOpenReview ↗

Response to Reviewer Fg8j (2/4)

Table R8. Start@k. Intervention level, Daily dev, IC/ST/LT combined. Start@k in %. -: no match.

GroupModel[email protected] 1 fps[email protected] 5 fps[email protected] Event-drivenStart@1 1 fpsStart@1 5 fpsStart@1 Event-drivenStart@2 1 fpsStart@2 5 fpsStart@2 Event-drivenStart@5 1 fpsStart@5 5 fpsStart@5 Event-driven
Open SourceGemma4-31B11.315.913.843.447.644.658.563.564.688.788.986.2
Qwen3.8-27B17.911.121.750.047.252.264.372.282.685.791.795.7
Qwen3.6-27B6.114.926.521.248.955.951.566.079.484.893.697.1
Qwen3.5-9B0.00.00.00.09.133.325.027.350.0100.072.783.3
VidLLMVideoLLaMA3-7B11.10.09.116.723.127.322.242.336.461.173.159.1
LLaVA-OV-1.5-8B0.0100.014.30.0100.014.30.0100.028.60.0100.042.9
MiniCPM-o-4.5-0.00.0-0.00.0-100.016.7-100.016.7
StreamingMMDuet0.00.00.025.00.050.075.0100.075.0100.0100.0100.0
VideoLLM-Online------------
JoyAI-VL------------
All models10.611.815.633.841.942.552.161.362.983.187.682.0

W2-a (notion of proactiveness). We evaluate in a query-free setting: the model maintains ambient memory to offload the user's mental burden, which requires it to infer user intent, and an event counts only if a majority of three annotators judged it useful for later queries. The benchmark therefore scores the decision of what to remember, not detection alone. Finer-grained importance and interruption cost differ across user personas (e.g., quiet vs. chatty preferences drive opposite metric trends); we will discuss personalized proactivity as a limitation and next step.

W2-b (scope and out-of-distribution claims).

  • We will narrow "long-horizon" and list hour-scale and cross-session memory as future work; minute-long, densely annotated clips suffice for our main topic of proactive memory.
  • We will clarify in Appendix B that the benchmark provides novel labels and tasks. Blind QA (question only, Table R6) stays near the 25% chance level on ST (25.7%) and LT (29.5%), so these answers cannot come from priors or memorized video; IC (Information Capture, 45.3%) is due to text-rich multimodal pretraining.

Table R6. QA accuracy (%), Daily: Blind QA (question only) vs. Proact (the model's own proactive memory). Mean over 8 models. Chance 25.0.

GroupModelIC Blind QAIC ProactIC ΔST Blind QAST ProactST ΔLT Blind QALT ProactLT Δ
Open SourceGemma4-31B57.867.9+10.132.032.8+0.835.332.1-3.2
Qwen3.8-27B57.662.3+4.732.031.5-0.436.131.3-4.8
Qwen3.6-27B56.263.5+7.319.929.0+9.130.227.8-2.4
Qwen3.5-9B44.052.3+8.321.614.1-7.531.024.6-6.3
VidLLMVideoLLaMA3-7B37.546.0+8.524.127.4+3.327.034.5+7.5
LLaVA-OV-1.5-8B36.939.8+3.023.218.3-5.023.022.2-0.8
MiniCPM-o-4.534.934.7-0.227.417.0-10.424.221.4-2.8
StreamingMMDuet37.341.8+4.525.723.2-2.529.030.2+1.2
Mean45.351.0+5.825.724.2-1.629.528.0-1.4

Q1. Not quantized. At 5 FPS, [email protected] rises only from 10.6% to 11.8% and the mean start error stays at 2.33 s (Table R7 in W1), so the 1 FPS grid is not what limits timing; Table R8 in W1 reports Start@k for k = 0.2, 1, 2, and 5 s as the discretization-aware view.

Q2. Within fixed task rules, an event is kept only if a majority of three annotators judge it useful for the stated user need. Each annotated event is paired with questions that require remembering it, so usefulness is measured objectively by downstream QA, and acting on non-annotated events is a false alarm. Human judgments of importance and interruption cost depend on personal preference and are left to future work (see W2-a).

Q3. Within the rebuttal time constraints, we re-evaluated the models on a subset to test the following factors:

  • (1) Tool-call compliance: unparseable output is confined to MMDuet and VideoLLM-Online; for the other 11 models, it is at most 0.1%, and the remaining tool errors are malformed memory (format) and tool invalid maintenance (lifecycle management; see Table R1 in W1, per-task breakdown below). Memory selection (event detection) errors are not tool errors; they appear as missed events and false alarms, i.e., recall and precision in the main tables.
AuthorsOpenReview ↗

Response to Reviewer Fg8j (3/4)

  • (2) Event-driven tracking (Table R3): on a dev split, we designed an explicit prompt suite that instructs the model on the features of target events in detail instead of the query-free setting in the benchmark; the query-grounded description of the target triggering event raises open-source models by +7.2 to +27.6 points on IC and up to +12.8 on ST, while LT changes little (-1.6 to +4.9) and streaming models barely move, so selecting what to remember from the user's need is still the main bottleneck, even with an explicit query revealing user's intent.
  • (3) Continuous acquisition: in Sec. 5.4, the No Memory setting samples 40 frames uniformly across the whole video clip, mimicking continuous acquisition, utilizing the same budget as proactive memory. Its QA accuracy still stays low.

All three points indicate that the current bottleneck is mainly proactive memory acquisition itself.

Table R1 (per task). Error shares (%) per task, Daily / Science, for the three error columns of Table R1 (definitions above). The remaining share of each task is silent or healthy.

ModelIC UnparseableIC Malformed MemoryIC Tool Invalid MaintenanceST UnparseableST Malformed MemoryST Tool Invalid MaintenanceLT UnparseableLT Malformed MemoryLT Tool Invalid Maintenance
GPT-5.4-mini0.0 / 0.00.0 / 0.01.3 / 2.10.0 / 0.00.0 / 0.01.6 / 1.20.0 / 0.00.0 / 0.00.1 / 0.8
Gemini 2.5 Flash0.1 / 0.00.0 / 0.01.8 / 1.50.2 / 0.00.0 / 0.0<0.1 / 1.10.1 / 0.052.5 / 0.0<0.1 / 0.1
Gemma4-31B0.0 / 0.00.0 / 0.0<0.1 / 0.10.0 / 0.00.0 / 0.00.0 / <0.10.0 / 0.00.0 / 0.00.0 / 0.0
Qwen3.8-27B0.0 / 0.00.0 / 0.00.0 / <0.10.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / <0.10.2 / 0.0
Qwen3.6-27B0.0 / 0.00.0 / 0.00.8 / 0.9<0.1 / <0.10.0 / 0.0<0.1 / <0.10.0 / 0.00.0 / 0.00.0 / 0.0
Qwen3.5-9B<0.1 / 0.00.0 / 0.00.1 / <0.10.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.0
VideoLLaMA3-7B0.0 / 0.00.3 / 0.252.8 / 20.8<0.1 / 0.00.0 / 0.475.5 / 73.6<0.1 / 0.0<0.1 / 0.075.0 / 38.8
LLaVA-OV-1.5-8B0.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.0<0.1 / 0.00.0 / 0.00.0 / 0.00.0 / 0.0
MiniCPM-o-4.50.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.2 / 0.0
MMDuet81.3 / 68.76.6 / 16.19.3 / 10.174.2 / 66.124.5 / 12.70.2 / 7.768.2 / 71.131.7 / 7.20.0 / 8.2
VideoLLM-Online100.0 / 100.00.0 / 0.00.0 / 0.0100.0 / 100.00.0 / 0.00.0 / 0.0100.0 / 100.00.0 / 0.00.0 / 0.0
Inkling-Small0.0 / 0.00.3 / 0.169.2 / 69.40.0 / 0.00.0 / 0.043.6 / 38.10.0 / 0.0<0.1 / 0.012.6 / 16.7
JoyAI-VL0.1 / 0.10.1 / 0.1<0.1 / <0.10.1 / 0.10.2 / 0.10.0 / <0.10.1 / 0.10.1 / <0.10.0 / 0.0
Mean over models14.0 / 13.00.6 / 1.310.4 / 8.113.4 / 12.81.9 / 1.09.3 / 9.413.0 / 13.26.5 / 0.66.8 / 5.0

Table R3. Paper prompt vs. event-driven prompt (the prompt explains the target triggering event), Intervention level, dev split.

GroupModelIC: paper → event-driven (Δ)ST: paper → event-driven (Δ)LT: paper → event-driven (Δ)
Open sourceGemma4-31B2.8 → 16.6 (+13.8)16.5 → 28.0 (+11.5)7.6 → 12.5 (+4.9)
Open sourceQwen3.8-27B3.5 → 31.1 (+27.6)28.0 → 27.9 (-0.1)10.2 → 10.8 (+0.6)
Open sourceQwen3.6-27B4.2 → 24.0 (+19.8)12.4 → 25.2 (+12.8)21.6 → 20.0 (-1.6)
Open sourceQwen3.5-9B6.8 → 14.0 (+7.2)0.0 → 6.1 (+6.1)0.0 → 3.4 (+3.4)
Video-LLMVideoLLaMA3-7B2.5 → 0.7 (-1.9)1.2 → 5.6 (+4.4)4.1 → 3.2 (-0.9)
Video-LLMLLaVA-OV-1.5-8B5.0 → 15.4 (+10.4)0.0 → 2.7 (+2.7)3.2 → 9.5 (+6.3)
Video-LLMMiniCPM-o-4.56.1 → 20.0 (+13.9)0.0 → 6.2 (+6.2)3.4 → 0.0 (-3.4)
StreamingMMDuet0.0 → 4.3 (+4.3)0.0 → 6.6 (+6.6)0.0 → 0.0 (0.0)
StreamingVideoLLM-Online0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)
StreamingJoyAI-VL0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)
AuthorsOpenReview ↗

Response to Reviewer Fg8j (4/4)

Q4.

  • (1) We agree that these videos are not extremely long, and we will replace "long-horizon" with this narrower claim and list hour-scale maintenance and cross-session memory as future work. Again, even with minute-long clips, the dense annotations do not weaken our goal of benchmarking proactive memories.
  • (2) Out-of-distribution. We will soften "guarantees" in Appendix B and state three separate levels:
    • Annotations are new: event intervals, tool lifecycles, and QA pairs were created for ProactEgoMem and were not public before this submission.
    • The task is new: to our knowledge, no existing dataset supervises when to start, maintain, and stop memory tools during a stream.
    • The videos are not guaranteed unseen: they come from public datasets and may appear in pretraining data. As the No Memory setting in Sec. 5.4 shows, the video input alone does not answer the questions, and blind QA (question only, Table R6 in W2-b) stays near the 25% chance level on ST (25.7%) and LT (29.5%). On IC, it reaches 45.3% via text priors, due to text-rich multimodal pretraining across all multimodal models, and we will add the discussion in the revision too.
Reviewer_3Q1JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark called ProactEgoMem to evaluate proactive memory in streaming egocentric video. It introduces three tasks grounded in ambient environmental needs: information capture, state tracking, and location tracking. The authors propose two levels of proactiveness: the Notification Level, which requires timely notifications, and the Intervention Level, which requires memory maintenance through an active start-to-stop tool lifecycle on the user's behalf. Their findings show that current state-of-the-art models still struggle with these capabilities.

Critical strengths
  1. The paper addresses an interesting and practical problem of proactive memory in streaming egocentric videos, which goes beyond traditional query-driven memory benchmarks.

  2. The two-level design is meaningful, especially the Intervention Level, which evaluates not only when to act but also how to maintain memory through a start-to-stop tool lifecycle. The evaluation across daily-life and scientific scenarios also makes the benchmark more comprehensive.

Critical weaknesses
  1. I think the evaluation of proactiveness is still somewhat limited. Although the paper introduces two levels, the models are given predefined tasks and memory goals. It would be interesting to see whether models can identify what needs to be remembered in a more open-ended setting, rather than just deciding when to start or stop tracking.

  2. Another concern is how the benchmark decides what is worth memorizing. This seems to be a key part of proactive memory, but the important events are mainly selected through human annotations, and the QA questions are generated from these events. How do we know which events are actually useful to remember? I think including events that do not need to be memorized, or testing different user needs, would make this evaluation more convincing.

  3. I would like to see more insights from the experiments. The paper shows that current models perform poorly and discusses several failure cases, such as false alarms, delayed actions, and incorrect tool calls. However, it is still unclear why models struggle with these tasks. Is the main issue video understanding, deciding when to act, or maintaining memory over time? More controlled experiments or ablations would help identify the main bottlenecks and give a clearer direction for future improvements.

Questions for authors

see my questions in weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3Q1J

We are grateful for the detailed reviews, constructive feedback, and your time and effort.

W1 (open-ended proactiveness). Our prompts provide only the available tools and are query-free evaluations; i.e., we assume that the user does not need to. prompt the model "help me track moving objects" so no target objects, status, times, or numbers of events are given, so the model already decides which object to remember, along with when to start and stop. This choice is where models fail: precision remains in single digits. Broader open-ended settings across diverse needs (e.g., mixing object status change and movement) are left to future work.

W2 (what is worth memorizing). Events follow fixed task rules (IC: legible dense information; ST: object states; LT: object moves away from the original location), and an event is kept only if a majority of three annotators judge it potentially useful for later user need. This keeps the first automatic evaluation of proactive memory objective. Personalized preferences (e.g., users who prefer silent or chatty models) require a large-scale human study and are beyond the scope of this benchmark.

W3 (where the bottleneck lies).

  • User-intent understanding: an explicit, event-driven prompt that names the triggering event (Table R3), instead of our query-free default, lifts open-source models by up to +27.6 points on IC and +12.8 on ST, so inferring what matters from the user's need is one main failure.
  • Memory maintenance: Oracle memory (ground-truth intervals) beats the model's Proact Memory by at least 20 points in 34 of 36 cells (Sec. 5.4), so maintaining memory over time is another failure factor. Even with Oracle memory, up to 50% headroom remains, which also indicates a problem with general video understanding.

Table R3. Paper prompt vs. event-driven prompt (the prompt names the triggering event), Intervention level, dev split.

GroupModelIC: paper → event-driven (Δ)ST: paper → event-driven (Δ)LT: paper → event-driven (Δ)
Open sourceGemma4-31B2.8 → 16.6 (+13.8)16.5 → 28.0 (+11.5)7.6 → 12.5 (+4.9)
Open sourceQwen3.8-27B3.5 → 31.1 (+27.6)28.0 → 27.9 (-0.1)10.2 → 10.8 (+0.6)
Open sourceQwen3.6-27B4.2 → 24.0 (+19.8)12.4 → 25.2 (+12.8)21.6 → 20.0 (-1.6)
Open sourceQwen3.5-9B6.8 → 14.0 (+7.2)0.0 → 6.1 (+6.1)0.0 → 3.4 (+3.4)
Video-LLMVideoLLaMA3-7B2.5 → 0.7 (-1.9)1.2 → 5.6 (+4.4)4.1 → 3.2 (-0.9)
Video-LLMLLaVA-OV-1.5-8B5.0 → 15.4 (+10.4)0.0 → 2.7 (+2.7)3.2 → 9.5 (+6.3)
Video-LLMMiniCPM-o-4.56.1 → 20.0 (+13.9)0.0 → 6.2 (+6.2)3.4 → 0.0 (-3.4)
StreamingMMDuet0.0 → 4.3 (+4.3)0.0 → 6.6 (+6.6)0.0 → 0.0 (0.0)
StreamingVideoLLM-Online0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)
StreamingJoyAI-VL0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)0.0 → 0.0 (0.0)
Reviewer_P7CCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ProactEgoMem, a benchmark for proactive memory in streaming egocentric video. It defines three tasks (information capture, state tracking, and location tracking) at two levels: sending timely notifications and maintaining memory through start–stop tool calls. The benchmark covers daily-life and laboratory videos. The authors evaluate a range of model families on event detection, timing, and interval overlap, and use downstream QA under no-, proactive-, and oracle-memory conditions to test whether the preserved information is useful. They report that current models struggle to acquire and maintain useful memory.

Critical strengths

This work provides a meaningful problem formulation for wearable assistants. It extends proactivity from a single respond-or-not decision to a start–maintain–stop memory lifecycle and evaluates the resulting memory through downstream QA. Under this setting, models must decide what to preserve before future questions are known. This targets a realistic need that remains underexplored in existing streaming and proactive benchmarks.

Critical weaknesses

W1. The criteria for user-relevant memory are unclear. The paper frames the benchmark around inferring user intent and remembering information relevant to user needs across long horizons; however, it does not report clear criteria for deciding which events should be included or excluded based on their usefulness to the user. A model could therefore lose precision for recording something useful simply because the annotators chose not to include it. Since evaluating user-relevant proactive memory is the benchmark's central contribution, this gap weighs heavily in my assessment.

W2. Protocol and scoring choices confound the main comparisons. As discussed in Section 5.6 and Appendix H (Fig. 19), the paper notes cases where perception succeeds but tool calling fails. However, the supplied code requests JSON-mode or JSON-constrained generation for general-purpose VLMs, while the provided video-LLM and streaming adapters lack equivalent format safeguards. The paper does not quantify the rates of unparseable outputs and invalid tool calls across models and tasks. Since unparseable outputs count as silence, performance differences in Table 4 cannot be attributed solely to perception or memory.

The timing comparison also needs rechecking. The supplied scorer code records a timing error of 0.00 when a model has no matched events, and averaging the Intervention-level cells in Table 4 with these zero entries reproduces the reported family means (1.3 s for streaming models vs. 2.5–3.1 s for the others). This suggests that undefined errors may have entered the family averages, which would directly affect the conclusion that streaming models act more promptly.

Questions for authors

Needs further clarification:

  1. What criteria did annotators use to include or exclude events based on their usefulness to the user? How do the 1,335 reference-accepted events in Table 3 relate to the 2,619 annotated events, and how were the three annotations merged?
  2. What are the per-model, per-task rates of unparseable outputs and invalid tool calls, and how do these failures affect comparisons given the different format safeguards?
  3. Were cells with no valid matches included in the family-level timing averages? Please report the averages after excluding them, together with valid-match counts. Relatedly, how can JoyAI-VL on IC have positive recall but zero mean absolute start error and zero Start @ 0.2?
  4. Did the reported Oracle experiments use the supplied implementation, which omits state labels for ST and uses different frame-sampling schemes for IC? If so, the oracle–proactive gaps should be re-reported after aligning these, since they underpin the headroom claim in Section 5.4.
  5. For each question type in Table 2, is it answerable in principle from the stored memory format, and what are its count and accuracy?
  6. The paper's central question concerns maintaining memory "over long time horizons" (Sec. 1), yet clips average about 192 s (Daily) and 233 s (Sci), shorter than most comparable benchmarks with reported durations in Table 5. In what sense does the benchmark evaluate long-horizon memory? If it does not, the framing should be revised accordingly.
  7. Which aspects of autonomous memory acquisition and lifecycle evaluation distinguish this benchmark from Wu et al. (2026)?
  8. Minor corrections: The Sci QA total is given as 500 (L25, L96, and the "Overall" row of Table 3), but the per-task Sci counts in Table 3 (299/104/157) sum to 560. Both 13 and 18 models are mentioned in the paper. Figure 3 needs a redo for clarity.

Suggestions, not hard requests:

  1. How much of the Daily–Sci performance gap remains after accounting for event duration and density? Adding representative Sci events and QA examples to Tables 1 and 2 would also help.
  2. Blind-QA and periodic-trigger/recording baselines under comparable memory and frame budgets would help establish the benefit of proactive selection. The paper should also specify that intervention matching evaluates object identity, while the correctness of recorded states and locations is assessed through downstream QA.
  3. Only lightweight proprietary models (Gemini 2.5 Flash, GPT-5.4-mini) are evaluated; including flagship models would show whether the reported difficulty holds for stronger systems, and the abstract's "even the strongest models" should otherwise be qualified as referring to the evaluated models.
6 comments
AuthorsOpenReview ↗

Response to Reviewer P7CC (1/4)

We are grateful for the detailed reviews, constructive feedback, and your time and effort.

W1 (criteria for user-relevant memory). ProactEgoMem deliberately sets proactivity to its maximum: the model receives no query or target and must decide on its own what to remember, making it the first proactive benchmark that also requires memory maintenance. What is worth remembering is decided in two steps:

  • First, every event must satisfy the objective rules of its task; the three task categories were distilled from about 4,000 hours of egocentric video reviewed for potentially useful ambient memory.
  • Second, an event is kept only if a majority of three annotators judge it potentially useful for later queries.

Personalized preferences are outside the scope of this benchmark (e.g., a preference for quiet or chatty models leads to opposite metric trends); we will discuss them as future work and state the selection criteria explicitly in the revision.

W2-a (format safeguards and unparseable outputs). Table R1 decomposes every Intervention-level step into five exclusive outcomes.

  • Unparseable output comes almost entirely from the two streaming models that emit free text by design (MMDuet 74.9 / 68.8%, VideoLLM-Online 100%); for the other 11 models it is at most 0.1%, even without JSON mode, so the format safeguard does not drive the comparisons among them.
  • JSON mode does not prevent schema errors (Gemini 2.5 Flash: 18.7% malformed on Daily); the remaining errors are lifecycle errors (tool invalid maintenance, e.g. VideoLLaMA3-7B 68.0%).
  • A lenient parser adds calls but no matches for these two models (Table R2), so their unparseable replies contain no correct actions.

We keep the as-is numbers, because a model that cannot call the tool acquires no memory.

Table R1. Intervention-level tool calling breakdown (%), Daily / Science, all three tasks pooled. Each reply is counted once under its worst outcome; each row sums to 100. Healthy tool call means the tool accepted the call, not that it matched an annotated event. Malformed Memory: valid JSON whose memory tool call does not follow the required schema (missing keys, unknown tool name, or wrong fields); it is not executed. Tool Invalid Maintenance: a well-formed call that is impossible in the current tool state, e.g. stopping a recording that is not running. JSON mode follows the released configs.

GroupModelJSON modeSilentHealthy tool callUnparseable replyMalformed MemoryTool Invalid Maintenance
ProprietaryGPT-5.4-mini✓96.6 / 92.32.4 / 6.40.0 / 0.00.0 / 0.01.0 / 1.3
ProprietaryGemini 2.5 Flash✓73.8 / 84.66.7 / 14.50.1 / 0.018.7 / 0.00.6 / 0.9
Open sourceGemma4-31B✓71.9 / 39.328.1 / 60.60.0 / 0.00.0 / 0.0<0.1 / <0.1
Open sourceQwen3.8-27B✓88.3 / 82.011.6 / 18.00.0 / 0.00.0 / <0.10.1 / <0.1
Open sourceQwen3.6-27B✓83.8 / 65.816.0 / 33.9<0.1 / <0.10.0 / 0.00.3 / 0.3
Open sourceQwen3.5-9B✓99.5 / 99.40.5 / 0.6<0.1 / 0.00.0 / 0.0<0.1 / <0.1
Video-LLMVideoLLaMA3-7B✗9.6 / 15.722.3 / 36.2<0.1 / 0.00.1 / 0.268.0 / 48.0
Video-LLMLLaVA-OV-1.5-8B✗99.8 / 99.80.2 / 0.20.0 / 0.00.0 / 0.0<0.1 / 0.0
Video-LLMMiniCPM-o-4.5✗99.9 / 100.0<0.1 / 0.00.0 / 0.00.0 / 0.00.1 / 0.0
StreamingMMDuet✗<0.1 / 0.71.4 / 9.574.9 / 68.820.1 / 12.23.6 / 8.8
StreamingVideoLLM-Online✗0.0 / 0.00.0 / 0.0100.0 / 100.00.0 / 0.00.0 / 0.0
StreamingInkling-Small✗3.4 / 5.255.7 / 54.80.0 / 0.00.1 / <0.140.8 / 40.0
StreamingJoyAI-VL✗99.7 / 99.8<0.1 / <0.10.1 / 0.10.1 / 0.1<0.1 / <0.1
AllMean over models63.6 / 60.311.2 / 18.113.5 / 13.03.0 / 1.08.8 / 7.6

Lenient parser. A call is accepted if (1) JSON has the wrong keys or tool names (e.g. tool_name, startReminder), (2) it is a bare command (e.g. "stop recording"), or (3) it is a free-text decision; a repaired stop closes the open tracker it names.

Table R2. Strict vs lenient parser. Paper setup (1 fps), intervention level, Daily dev, IC/ST/LT combined. |Δs|, σ in s; others %. -: no match.

ModelmIoU StrictmIoU Lenient[email protected] Strict[email protected] Lenient|Δs| Strict|Δs| Lenientσ Strictσ LenientPrec StrictPrec LenientRec StrictRec LenientF1 StrictF1 Lenient
MMDuet0.90.90.00.01.931.930.690.694.32.92.22.22.92.5
VideoLLM-Online0.00.0------0.00.00.00.00.00.0
AuthorsOpenReview ↗

Response to Reviewer P7CC (2/4)

Table R1 (per task). Error shares (%) per task, Daily / Science, for the three error columns of Table R1 (definitions above). The remaining share of each task is silent or healthy.

ModelIC UnparseableIC Malformed MemoryIC Tool Invalid MaintenanceST UnparseableST Malformed MemoryST Tool Invalid MaintenanceLT UnparseableLT Malformed MemoryLT Tool Invalid Maintenance
GPT-5.4-mini0.0 / 0.00.0 / 0.01.3 / 2.10.0 / 0.00.0 / 0.01.6 / 1.20.0 / 0.00.0 / 0.00.1 / 0.8
Gemini 2.5 Flash0.1 / 0.00.0 / 0.01.8 / 1.50.2 / 0.00.0 / 0.0<0.1 / 1.10.1 / 0.052.5 / 0.0<0.1 / 0.1
Gemma4-31B0.0 / 0.00.0 / 0.0<0.1 / 0.10.0 / 0.00.0 / 0.00.0 / <0.10.0 / 0.00.0 / 0.00.0 / 0.0
Qwen3.8-27B0.0 / 0.00.0 / 0.00.0 / <0.10.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / <0.10.2 / 0.0
Qwen3.6-27B0.0 / 0.00.0 / 0.00.8 / 0.9<0.1 / <0.10.0 / 0.0<0.1 / <0.10.0 / 0.00.0 / 0.00.0 / 0.0
Qwen3.5-9B<0.1 / 0.00.0 / 0.00.1 / <0.10.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.0
VideoLLaMA3-7B0.0 / 0.00.3 / 0.252.8 / 20.8<0.1 / 0.00.0 / 0.475.5 / 73.6<0.1 / 0.0<0.1 / 0.075.0 / 38.8
LLaVA-OV-1.5-8B0.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.0<0.1 / 0.00.0 / 0.00.0 / 0.00.0 / 0.0
MiniCPM-o-4.50.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.0 / 0.00.2 / 0.0
MMDuet81.3 / 68.76.6 / 16.19.3 / 10.174.2 / 66.124.5 / 12.70.2 / 7.768.2 / 71.131.7 / 7.20.0 / 8.2
VideoLLM-Online100.0 / 100.00.0 / 0.00.0 / 0.0100.0 / 100.00.0 / 0.00.0 / 0.0100.0 / 100.00.0 / 0.00.0 / 0.0
Inkling-Small0.0 / 0.00.3 / 0.169.2 / 69.40.0 / 0.00.0 / 0.043.6 / 38.10.0 / 0.0<0.1 / 0.012.6 / 16.7
JoyAI-VL0.1 / 0.10.1 / 0.1<0.1 / <0.10.1 / 0.10.2 / 0.10.0 / <0.10.1 / 0.10.1 / <0.10.0 / 0.0
Mean over models14.0 / 13.00.6 / 1.310.4 / 8.113.4 / 12.81.9 / 1.09.3 / 9.413.0 / 13.26.5 / 0.66.8 / 5.0

W2-b (timing averages). Thank you for pointing out; we have corrected it and relabeled zeroed [email protected] cells without matches as N/A. Excluding cells without valid matches (Table R4), start errors are 3.07 s (proprietary), 2.68 s (open source), 3.51 s (video-LLMs), and 2.51 s (streaming, partly from JoyAI-VL's 2 IC matches at 0.00 s). The claim for streaming advantage will be removed from Section 5.3. All other metrics are based on matched correspondence, so the other claims remain valid.

Table R4. Intervention-level mean absolute start error (s) per model family on Daily, before and after excluding model-task cells without a valid match. Mean over matches weighs every matched action equally. The streaming mean over cells includes JoyAI-VL on IC, whose 2 matches both have a 0.00 s start error.

FamilySubmission ver. (no-match cells as 0.00)Corrected, mean over cellsCorrected, mean over matchesValid cells portionValid matches
Proprietary3.073.073.026 / 6192
Open source2.462.682.4811 / 121,067
Video-LLM2.733.513.927 / 9130
Streaming1.252.513.286 / 12308

Q1. Criteria are twofold, mainly objective and complemented by subjective inter-annotator agreement. We deliberately sets proactivity to its maximum for general:

  • (1) Task-specific objective rules instead of human preference: The three task categories (information capture, state tracking, and location tracking) were distilled from about 4,000 hours of egocentric video reviewed for potentially useful ambient memory. This keeps the first automatic evaluation of proactive memory objective, with the goal that the stored memory must suffice to answer later queries.
  • (2) Majority voting among the three annotators to determine whether the events are useful for later queries. The gap the reviewer raises lies between memory that is generally useful for ambiance and memory tailored to individual users with different personas; e.g., a preference for quiet or chatty models leads to opposite metric trends. This is a natural next step and is outside the scope of this benchmark. We thank the reviewer for this suggestion and will discuss in the future.

We apologize that N = 1,335 is a typo coming from an earlier snapshot. The final 2,619 events are from the final release, comprising two tracks: 1,242 Daily and 1,377 Science; please see Table 3 for a detailed breakdown. Agreement recomputed on the final release: Daily N = 1,242, mIoU 0.983, mean |Δs| 0.22 s, Start@2 98.8%; the Sci track is built on the dense preliminary annotations of the original datasets (FineBio, BioVL-QR).

AuthorsOpenReview ↗

Response to Reviewer P7CC (3/4)

Table 3 (revised agreement columns). Inter-annotator agreement on the final Daily release, averaged over three annotators. The Sci track is built on the dense preliminary annotations of the original datasets (FineBio, BioVL-QR).

TaskNmIoU ↑σ(IoU) ↓Start@2 (%) ↑End@2 (%) ↑Mean abs. start error (s) ↓Mean abs. end error (s) ↓
IC2780.9380.19197.792.30.212.19
ST4830.9960.06099.699.60.140.23
LT4810.9950.05098.799.40.310.21
Overall1,2420.9830.10598.897.90.220.65

Q2. Table R1 reports, per model and per task, the share of steps that are silent, healthy tool call, unparseable, malformed memory, or tool invalid maintenance. Even without JSON mode, unparseable replies stay at or below 0.1% for 11 of the 13 models. See W2-a for the summary and the per-task part of Table R1: per task, unparseable output stays at or below 0.2% for every model except MMDuet and VideoLLM-Online, and tool invalid maintenance errors concentrate in VideoLLaMA3-7B (Daily ST 75.5%, LT 75.0%) and Inkling-Small (IC 69.2% Daily, 69.4% Sci). Lenient re-scoring of MMDuet and VideoLLM-Online adds no matches (Table R2; see W2-a).

Q3. Yes; corrected means and valid-cell counts are in W2-b. We will regenerate the Daily and Sci main tables and print "–" for every timing cell without a valid match. The JoyAI-VL IC cell relies solely on matched interventions, starting exactly at the ground-truth moment (0.00 s error). ** The family averages now report their valid-match counts (Table R4).**

Q4. Yes, and no information is omitted.

  • By default, every ST interval is defined as start = on/open and stop = off/closed, and the QA prompt states these tool semantics, so the interval itself carries the state.
  • For IC, both conditions sample 40 frames uniformly within their respective intervals for a fair comparison (within the ground-truth intervals for Oracle and within the model's own recordings for Proact Memory).

Since rerunning the entire benchmark is not feasible within the rebuttal period, we will update the prompt to reflect this minor difference in the final version.

Q5. Yes, all the questions are human-reviewed and answerable by humans watching the videos. From the stored memory format, every type is answerable except Unattended state, which is only partly answerable: the memory records when an object is on or open, not whether the user attended to it, and we do not ask models to predict attention. The Oracle Memory setting evaluates a model's performance even with ground-truth memory intervals, to show the gap to humans watching the videos. Table R5 gives the count and the No Memory, Proact, and Oracle accuracy of every type (Daily). The Oracle values for Final state (14.5%) and Unattended state (25.9%) fall below No Memory because of an evaluation bug that left intervals still open at the end of the clip out of the oracle memory; we will re-run these two types.

Table R5. QA accuracy (%) per question type, Daily.

TaskQuestion type#QNo MemoryProactOracle
ICText21552.153.481.8
ICSequence8936.944.276.6
ICAttribute6652.664.788.7
ICPrice4636.037.978.5
ICLegibility2761.364.083.2
ICDirection1934.944.094.3
ICOther4546.545.985.9
ICAll50747.451.482.4
STFrequency9223.420.872.8
STFinal state3322.921.514.5
STUnattended state2722.632.725.9
STDuration2331.733.451.0
STEvent order1420.124.060.4
STBefore/after1225.020.553.8
STOther4034.931.156.3
STAll24125.825.353.2
LTPlacement11626.227.268.4
LTSource4427.933.763.6
LTDestination1826.333.360.6
LTLast location2425.031.158.0
LTLocation count1725.19.634.2
LTFirst location1011.816.477.3
LTOther2322.521.354.2
LTAll25225.427.062.8

Q6. We will use this term instead of "long-horizon" and list hour-scale and cross-session memory as future work.

Q7. Wu et al. is a memory method for text-based agents, evaluated by task success on Terminal-Bench 2.0 and τ²-Bench. Our work evaluates on streaming egocentric videos.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful reading and constructive feedback. Reviewers agree that ProactEgoMem addresses a meaningful and underexplored problem: it extends proactivity from a single respond-or-not decision to a start, maintain, and stop memory lifecycle, and it evaluates the acquired memory through downstream QA (P7CC, 3Q1J, Fg8j). Below we summarize the concerns shared across reviews and how we address them. Detailed answers and the new results (Tables R1 to R8) are in the individual responses.

  1. Clarification on what is worth remembering (P7CC W1 + Q1, 3Q1J W2, Fg8j W2 + Q2)
    • We clarify that what is worth remembering is decided by objective task rules, summarized from 4k hours of video for offloading mental burden with ambient memory, and complemented by a subjective majority vote of three annotators; downstream QA measures its usefulness. ProactEgoMem is designed to evaluate maximal proactivity as the first automatic proactive memory evaluation. Personalized preference for quiet or chatty models drives opposite metric trends and is left as future work.
  2. Tool-call / format failures are confounded with perception and memory. (P7CC W2 + Q2, Fg8j W1 + Q3, 3Q1J W3)
    • Table R1 splits every step into silent, healthy tool call, unparseable, malformed memory, and tool invalid maintenance; unparseable replies stay at or below 0.1% for 11 of the 13 models, and a lenient parser does not increase the performance of the streaming models MMDuet and VideoLLM-Online (Table R2). Therefore, the benchmark results still mostly reflect perception and memory.
  3. "Long-horizon" framing vs. short clips. (P7CC Q6, Fg8j W2 + Q4)
    • We will narrow the claim of "long-horizon" and list hour-scale and cross-session memory as future work. The minute-long scope with densely annotated events still serves the target of this benchmark. The out-of-distribution guarantee is softened to new annotations and a new task. Blind QA stays near chance accuracy, so even videos possibly seen in web-scale pretraining do not significantly affect the evaluation.
  4. Additional controlled baselines and ablations to locate the bottleneck. (P7CC Q10, Fg8j Q3, 3Q1J W3)
    • Blind QA (Table R6) shows near-chance performance and bounds the prior knowledge of models; the No Memory setting already serves as the matched-budget uniform continuous recorder; the event-driven prompt (Table R3) shows the importance of intent inference and video understanding; and the gap between Oracle Memory and proactive memory (Sec. 5.4) isolates the memory maintenance requirement.
  5. Timing metric validity. (P7CC W2 timing paragraph + Q3, Fg8j W1 + Q1)
    • No-match cells are now excluded from the family averages (Table R4, with valid-match counts) and the streaming-advantage claim is removed, while the other claims stay unchanged. Ground-truth starts are actual timestamps rather than quantized to 1 FPS.
    • We further evaluate at 5 FPS, which barely moves [email protected] (10.6% to 11.8%, Table R7); the main table will report Start@1 alongside Start@k (Table R8), and a relaxed timing analysis is included in Appendix E, Figure 12.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.