acceptodds
Withdrawn from ICLR 2027

APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use. Yet existing streaming benchmarks and methods often focus on individual continuous videos or short clips, overlooking that real-world interactions are often intermittent and require memory to persist across interruptions. To fill this gap, we introduce APM-Bench, which reformulates real-world streaming interaction as multi-session life trajectories. It contains 549 sessions, 104 trajectories, and 2,719 candidates, spanning both objective and open-ended questions. Each session is a video with fine-grained annotations, and sessions within a trajectory revolve around related activities. Models then use persistent memory to answer questions about past sessions and provide proactive responses while maintaining real-time interaction. This raises challenges: persistent memory must be storable, selectively retain information, be injected at the right time, and remain efficient. Moreover, finite storage may leave required evidence unavailable, so assistants should recognize missing evidence. Therefore, we systematically evaluate general video models under different memory protocols and diverse specialized streaming memory systems, and test whether models acknowledge insufficient evidence. Our evaluation reveals a clear utility–latency–storage trade-off: existing methods still struggle to simultaneously achieve reliable long-term recall, low overhead, and effective proactive assistance across sessions. APM-Bench provides a comprehensive testbed for developing and comparing persistent memory systems under realistic streaming conditions. We hope it encourages future work that jointly considers utility, latency, and storage toward more practical persistent memory for real-world streaming assistants.

Primary area:
datasets and benchmarks
Keywords:
Streaming Video, Egocentric, Memory

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_b2S2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing streaming benchmarks evaluate memory within a single continuous video, but real use is intermittent: a user stops and resumes hours or days later. APM-Bench reorganizes egocentric video into multi-session trajectories of related activities, 104 trajectories and 549 sessions, with 2,719 candidates across 12 tasks in three capability families: cross-session understanding, real-time perception, and adaptive response. It jointly reports task utility, time to first token, and storage cost per hour, and adds a dedicated set testing whether models recognize when required evidence is unavailable instead of fabricating an answer.

Critical strengths
  1. The core observation that the visual stream is not replayed and anything the assistant will later need has to have been stored is good.
  2. The construction pipeline is reasonable.
Critical weaknesses
  1. Only one of eight specialized systems actually supports persistent memory.
  2. The video content is not new. APM-Bench builds on EgoLife and HD-EPIC. That is a legitimate and efficient choice, and the contribution is the session organization and the QA, not the footage.
  3. Multi-session organization plus joint cost reporting is a combination rather than a new capability.
  4. Open-ended scoring has no human or judge-agreement validation.
  5. A response passes a gate only if it correctly decides INTERVENE or SILENT, then DeepSeek-V4-Flash scores the rationale 1 to 5. For SILENT probes there is no response content to judge, so it is unclear what the 1 to 5 score is evaluating, and a degenerate always-silent model would still accumulate credit on negative probes.
Questions for authors
  1. Only OASIS supports persistent memory export. How do you interpret the other seven rows given that they retain state in-process, and can you evaluate at least one method with a genuine store and reload cycle?
  2. What is the domain breakdown of the 104 trajectories? HD-EPIC is kitchen-centric and Figure 1 suggests cooking is common, which would limit the generality of the real-world claim.
  3. Can you report human or second-judge agreement for the Gated LLM-Judge, and what score a SILENT probe's rationale receives given there is no response to judge?
Reviewer_vaReOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents APM-Bench, which evaluates persistent memory for streaming egocentric assistants when interaction is interrupted. It re-cuts EgoLife and HD-EPIC recordings into 104 trajectories. There are 2,719 items in 12 tasks over three groups. Cross-session understanding and real-time perception are multiple choice, and adaptive response asks the model when to speak or stay silent, scored by a gated DeepSeek-V4-Flash judge. There's also a 260-question set with an "insufficient evidence" option. The paper evaluates Gemini, Seed and several open models with raw video or text summaries as memory, plus eight streaming memory systems, and reports accuracy alongside storage per hour and time to first token.

Critical strengths
  1. I appreciate for the comprehensive metrics / eval studies on the streaming video understanding, which could be next potention super useful application over people's daliy life.

  2. the reported storage and latency next to accuracy is useful. Streaming memory papers rarely do this, and Fig. 2 and Table 3 make the trade-off easy to see.

  3. the pipeline is carefully designed.

Critical weaknesses
  1. The sessions are cut out of continuous recordings, and "raw video as memory" just plays all earlier sessions back. So for the general models it's basically one long video with the gaps removed. The paper doesn't show any system actually saving its memory, shutting down and loading it again, and for most memory systems the storage number is estimated from what they hold in memory while running (Sec. 4.1). Also, EgoLifeQA already asks cross-day memory questions on the same EgoLife footage. It should be in Table 1, with a clear explanation of what's new here. The variety is limited too, since EgoLife is six people living in one house for a week and HD-EPIC is all kitchens.

  2. Also raw video storage depends on the video format and resolution, and 1 fps frames could be stored much smaller, so please say how it was saved to make a fair comparison.

  3. The question generator might favor Gemini. The questions are built from captions written by Gemini and then Gemini 3.6 Flash wins by a lot. Please show this isn't a same-family advantage, for example with captions from another model or a human-written subset.

  4. Minor: No repeated runs or error bars, names are inconsistent (Video-Salmon-S vs video-SALMONN S, and what is "Memory Loss-Aware" in appendix vs "Evidence Availability-Aware"?)

  5. How do the always-quiet and always-speak baselines score, and how often does the judge agree with humans?

  6. What happens to perception if the latest frames are always included?

  7. Besides cutting the videos into sessions, how is this different from EgoLifeQA?

Questions for authors

see Weaknesses

Reviewer_SUhmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced APM-Bench to evaluate the persistent memory capabilities of video models on multi-session life trajectories in a streaming manner. They defined three capabilities: cross-session understanding, real-time perception, and adaptive response. To generate data for evaluating these capabilities, they built upon two egocentric datasets and organized them based on activity-related multi-session trajectories, and generated data using VLMs and human review. They evaluated several VLMs on the proposed benchmark and showed that they still struggle on many of the persistent memory capabilities.

Critical strengths

The paper proposed to address an interesting and important problem: persistent memory and proactive response across sessions of egocentric streaming video, which matters for real-world assistants such as smart glasses. The three capabilities defined to evaluate memory are also reasonable. Reporting latency and storage cost alongside accuracy is also a useful practical contribution.

Critical weaknesses

Weakness 1: Data quality and potential model-family bias. 

Most of the benchmark data were generated by LLMs/VLMs. Gemini-3.5-Flash wrote the captions that most candidates are built upon, and DeepSeek-V4-Flash proposed the cross-session and adaptive response candidates along with their reference responses. Although they used blind filtering and human review, the paper gives little evidence and analysis of the resulting data quality for the three benchmark capabilities.

There is also an overlap of LLMs between data generation and model evaluation. The captions from Gemini models were used for candidate generation, and Gemini 3.6 Flash is the top-ranked model in the evaluation. DeepSeek-V4-Flash both wrote the references and judged the responses. This may bias the evaluation results, and the paper did not show any LLM-judge and human agreement on the evaluation.

Weakness 2: Limited realism of cross-day memory and generality of conclusions. 

The benchmark data trajectory has about 69 minutes of video, which might not be able to reflect the real-world days-long streaming scenarios. For example, keeping only related sessions removes the unrelated history that a real assistant would have to search through in real-world use cases. The benchmark may therefore reduce to roughly one hour of long-context video QA for some data samples.

In addition, the paper concluded that “current methods still struggle to …”(L484-485) relies on the results of several lightweight proprietary models (Gemini 3.6 Flash, Seed-2.0-Lite). Other frontier models, such as the GPT series, are not evaluated. Finally, the suggested directions for improving the proposed capabilities are brief and untested.

Questions for authors

The human review stages are central to the data quality of the proposed benchmark, but some key details are missing. Could the authors clarify: (1) the criteria used in Stage 1 to select trajectories; (2) the number, background, and training of the annotators; (3) how many annotators reviewed each candidate and how disagreements were resolved; and (4) how much "human refinement" changed the candidates.

Reviewer_zT61OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This article introduces APM-Bench, a benchmark for persistent memory in multiple sessions over time for first-person streaming video assistants. The study organizes EgoLife and HD-EPIC recordings into 104 activity-related trajectories, including 549 sessions and 2,719 evaluation candidates. Twelve tasks cover cross-session understanding, current scene perception, and adaptive responses. A separate 260-question setup evaluates whether models can recognize when relevant historical evidence is unavailable. The evaluation compares general video models that use raw video or text summary memory with eight dedicated memory systems, reporting on task performance, storage, and latency. The results show significant differences between memory protocols and suggest that reliable cross-session recall and proactive assistance are still challenging.

Critical strengths

This is a useful evaluation supported by substantive annotation work. The benchmark study examines session-related activities across interruptions, highlighting the significant need for persistent personal assistants. It combines retrospective questions, current scenario questions, and auxiliary questions that require historical evidence, testing several different uses of memory. Its construction includes causal timestamp checks, video-based human verification, and independent annotation audits, reporting a consistency of 0.868 (Sections 3.1 and Appendices D.1–D.2). These steps give the benchmark a concrete contribution beyond existing video question reassembly.

The information provided by diagnostic assessments goes beyond the comprehensive rankings. This paper evaluates various memory representations and distinguishes between successful historical recall and the identification of unavailable evidence. The appendices also distinguish between intervention decisions and response quality, report failures in output formats, and compare curated trajectories with reference evidence and longer original historical records (Appendices B.1–B.7). These analyses help identify whether the system faces difficulties in evidence access, intervention decision-making, or response generation. This makes the benchmark useful for diagnosing the specific limitations of streaming assistants.

Critical weaknesses

Open-ended scoring requires more rigorous validation through human judgment. After passing a deterministic decision gate, all five adaptive response tasks rely on a single large language model (LLM) evaluator (Section 3.3 and Appendix A.3). The reported human consistency pertains only to dataset annotations; it does not demonstrate consistency between the evaluator and humans regarding actual model responses. This is particularly important for MPA and TPG, as in these tasks, multiple responses may provide useful assistance, and the evaluators are provided with reference annotations rather than the source video. Human review at the response level is required to determine the extent to which final scores and system differences reliably reflect the quality of the assistance. While decision/quality decomposition is helpful, it does not resolve this validation issue.

The explanation regarding the memory effect does not fully rule out the influence of differences in visible input. Section 4.1 used the Qwen baseline with the most recent four frames, rather than matching each model’s input to the current session while clearing historical memory. Input was also restricted to 1,024 or 128 sample frames, but it remains unclear whether the original video and text summary conditions provided the same frames from the current session. If historical video competes for the same frame budget, the improved perception resulting from the summary may stem in part from more intensive sampling of the current scene, rather than a reduction in interference from historical information. These comparisons remain useful as a complete system evaluation, but a more compelling explanation is that irrelevant memory impairs current perception.

Questions for authors

Can you provide a human evaluation of a representative subset of actual Adaptive Response outputs, including MPA and TPG, and report agreement with the LLM judge? Do the principal system comparisons remain consistent under human scoring?

How is the frame budget allocated between historical sessions and the current session? Can you provide a targeted same-backbone comparison with identical current-session frames under three conditions: no historical memory, raw-video memory, and summary memory? This would help distinguish memory benefits or interference from differences in current-scene evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.