- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Existing streaming benchmarks evaluate memory within a single continuous video, but real use is intermittent: a user stops and resumes hours or days later. APM-Bench reorganizes egocentric video into multi-session trajectories of related activities, 104 trajectories and 549 sessions, with 2,719 candidates across 12 tasks in three capability families: cross-session understanding, real-time perception, and adaptive response. It jointly reports task utility, time to first token, and storage cost per hour, and adds a dedicated set testing whether models recognize when required evidence is unavailable instead of fabricating an answer.
Critical strengths
- The core observation that the visual stream is not replayed and anything the assistant will later need has to have been stored is good.
- The construction pipeline is reasonable.
Critical weaknesses
- Only one of eight specialized systems actually supports persistent memory.
- The video content is not new. APM-Bench builds on EgoLife and HD-EPIC. That is a legitimate and efficient choice, and the contribution is the session organization and the QA, not the footage.
- Multi-session organization plus joint cost reporting is a combination rather than a new capability.
- Open-ended scoring has no human or judge-agreement validation.
- A response passes a gate only if it correctly decides INTERVENE or SILENT, then DeepSeek-V4-Flash scores the rationale 1 to 5. For SILENT probes there is no response content to judge, so it is unclear what the 1 to 5 score is evaluating, and a degenerate always-silent model would still accumulate credit on negative probes.
Questions for authors
- Only OASIS supports persistent memory export. How do you interpret the other seven rows given that they retain state in-process, and can you evaluate at least one method with a genuine store and reload cycle?
- What is the domain breakdown of the 104 trajectories? HD-EPIC is kitchen-centric and Figure 1 suggests cooking is common, which would limit the generality of the real-world claim.
- Can you report human or second-judge agreement for the Gated LLM-Judge, and what score a SILENT probe's rationale receives given there is no response to judge?