- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ScreenMem introduces a benchmark for proactive, query-agnostic memory maintenance over long-horizon computer-use streams, offering a valuable evaluation framework but leaving concerns about baseline strength, perception–memory disentanglement, and real-world validity.
Critical strengths
1. Novel and well-defined evaluation setting: ScreenMem introduces a meaningful benchmark for proactive, query-agnostic memory maintenance, separating causal memory formation from subsequent question answering over long-horizon computer-use streams.
2. Comprehensive empirical evaluation: The paper evaluates 15 memory systems across four paradigms, providing detailed analyses of memory capabilities, temporal horizons, and computational efficiency that reveal important limitations of existing approaches.
Critical weaknesses
1. Lack of strong and simple baselines: The paper does not evaluate straightforward approaches such as timestamped OCR or visual-description storage with retrieval, making it difficult to determine whether complex memory architectures provide meaningful advantages.
2. Insufficient disentanglement of perception and memory: Different systems use different visual encoders, sampling strategies, and input representations, making it unclear whether failures arise from visual information extraction or memory maintenance.
3. Limited real-world validity: Although the benchmark is grounded in real interaction traces, its reconstructed interfaces and scripted activities may not fully capture the complexity and unpredictability of real-world computer use. No direct validation of this gap is provided.
4. Limited statistical robustness and evaluation validity: The benchmark contains only 10 user-day streams, lacks uncertainty estimates for system rankings, and mixes short-answer and multiple-choice questions, potentially confounding comparisons across memory capabilities.
Questions for authors
Q1. How does the benchmark distinguish persistent memory from raw observation storage?
Appendix A indicates that some systems retain visual frames or cached representations. What criteria distinguish legitimate memory maintenance from effectively storing and retrieving the original observation stream? Could this distinction affect the interpretation of the benchmark results?
Q2. What explains the particularly poor performance on State Evolution questions?
Table 3 shows that the best system achieves only 15.8% accuracy on State Evolution. Can the authors provide a qualitative error analysis distinguishing failures to update outdated information, confusion between historical and current states, and failures to retrieve correctly maintained states?
3 comments
W3. Real-world validity of reconstructed videos
- Limited real-world validity: Although the benchmark is grounded in real interaction traces, its reconstructed interfaces and scripted activities may not fully capture the complexity and unpredictability of real-world computer use. No direct validation of this gap is provided.
Response. To our knowledge, ScreenMem is the first benchmark focused on proactive memory maintenance over day-long device-use video streams. Building such a benchmark is difficult: recording continuous device activity is costly, and removing sensitive information from hours of personal screen recordings without disrupting the interactions is particularly challenging.
We therefore reconstruct videos from real-world interaction traces—RLKWiC for PC (Bakhshizadeh et al., 2024) and FingerTip 20K for mobile (Yang et al., ICLR 2026). Every recorded replay follows its source trajectory, and we manually verify the reconstructed interactions and question evidence. Although replay is deterministic, the original traces capture real action sequences, task dependencies, and evolving device states, preserving important complexities of actual device use.
W4. Statistical robustness and mixed answer formats
- Limited statistical robustness and evaluation validity: The benchmark contains only 10 user-day streams, lacks uncertainty estimates for system rankings, and mixes short-answer and multiple-choice questions, potentially confounding comparisons across memory capabilities.
Response. ScreenMem offers 100.1 hours of long-horizon streams and 484 verified questions for comparing systems under an identical evaluation protocol. To quantify the variation associated with the user-day units, we added 100,000-replicate, platform-stratified day-clustered bootstrap intervals and paired comparisons. For example, MemOS: 31.4% [27.2%, 36.2%] and Mem0: 24.8% [20.8%, 28.5%], with a paired difference of +6.6 percentage points [2.9, 11.9]. The narrower Mem0 − EverMemOS difference is +1.0 point [−3.1, 4.7]. This analysis supports a more informative reading of the rankings: the leading separation is relatively consistent across resampled days, while smaller gaps are appropriately distinguished from it. The full analysis is reported in Appendix B.5 (User-Day-Clustered Accuracy Intervals).
Regarding mixed formats, the benchmark is designed to compare systems on exactly the same questions within each capability dimension. Different capabilities may naturally require different forms of questions, just as different examination subjects do. This does not invalidate within-capability system comparisons, although it means that differences between capability averages should not be interpreted as pure differences in intrinsic memory difficulty. The existing question-format breakdown in Appendix B.2 provides additional context.
Q1. Persistent memory versus raw observation storage
Q1. How does the benchmark distinguish persistent memory from raw observation storage?
Appendix A indicates that some systems retain visual frames or cached representations. What criteria distinguish legitimate memory maintenance from effectively storing and retrieving the original observation stream? Could this distinction affect the interpretation of the benchmark results?
Response. We appreciate the opportunity to clarify the benchmark's evaluation objective. ScreenMem evaluates whether a memory system can effectively support a user's future information needs, not whether it follows a particular internal implementation of memory maintenance. Consequently, the benchmark should not prescribe that memory must be textual, semantically summarized, or compressed in a specific way. Systems may retain text, visual frames, learned features, or caches; we evaluate their ability to support future questions end to end, regardless of the retained representation.
The key evaluation constraint is causal, query-agnostic ingestion: the system must decide what to persist before seeing the future questions, and subsequently answer using only what it has retained, without reopening the original video stream. Raw-observation retention is therefore not excluded by definition; rather, its usefulness must be judged by answer accuracy together with the ingestion and storage costs that we report. In other words, our benchmark measures how effectively and efficiently a system serves as persistent memory for the user, rather than judging whether its underlying storage mechanism is a sufficiently “memory-like” algorithm. This is the evaluation constraint already defined by the causal sealed-memory protocol in Sec. 3.4.
Q2. Why is State Evolution particularly difficult?
Q2. What explains the particularly poor performance on State Evolution questions?
Table 3 shows that the best system achieves only 15.8% accuracy on State Evolution. Can the authors provide a qualitative error analysis distinguishing failures to update outdated information, confusion between historical and current states, and failures to retrieve correctly maintained states?
Response. We have conducted a State Evolution error analysis on the 57 questions in this category, covering all 15 original memory systems. Following the three failure modes in your question, we distinguish (1) failure to update outdated information, (2) confusion between historical and current states, and (3) failure to retrieve a correctly maintained state. The table below summarizes two representative systems; the remaining cases are grouped as Other errors.
| System | Incorrect / 57 | Failure to update | Historical/current confusion | Failure to retrieve maintained state | Other errors |
|---|---|---|---|---|---|
| MemOS | 48 | 0 | 0 | 19 | 29 |
| M3-Agent | 54 | 3 | 0 | 20 | 31 |
For MemOS, 19 of 48 errors involve failure to retrieve a correctly retained state. For M3-Agent, 20 of 54 errors are retrieval failures and 3 involve outdated states not being updated. We found no cases of historical/current confusion under the strict criterion that both versions had reached the answerer; this does not rule out temporal information being lost earlier. Other errors aggregates remaining failures, including information missing from input captions, states not retained, and answer-stage mistakes. The full 15-system breakdown and category definitions are provided in Appendix B.6 (State Evolution Error Attribution).
We sincerely thank the reviewer for the detailed and constructive criticism. We have added two simple retrieval baselines, day-clustered uncertainty estimates, and a State Evolution error analysis to the manuscript, and conducted an evidence-window control reported in this response. We also clarify the goal of evaluating complete memory systems and the rationale for trace-grounded video reconstruction. We respond to each concern separately below.
W1. Strong and simple baselines
Response. To make this comparison explicit, we added two straightforward retrieval-based baselines: Caption-RAG, which stores timestamped visual descriptions, and OCR-RAG, which stores timestamped screen text. Both retrieve previously stored information to answer the same 484 questions under the same scoring rules as the existing systems.
A particularly informative finding is that Caption-RAG outperforms all 15 previously evaluated memory systems on PC (30.13% versus MemOS's 29.17%), although MemOS remains stronger overall (31.40% versus 27.48%). This demonstrates that more sophisticated memory architectures do not necessarily outperform a well-implemented simple retrieval baseline, and strengthens the practical significance of the comparison. We have updated the main results table and included the baseline results in the main results table, with implementation and cost details in Appendix A.4 and Appendix D.8, respectively.
W2. Disentangling perception and memory maintenance
Response. We agree that this distinction matters. Our evaluation is intended as an end-to-end comparison of memory systems, rather than an ablation in which all systems are required to use an identical visual encoder and memory implementation. The latter would change the native operation of several systems and no longer represent their released end-to-end capabilities.
To provide a more controlled comparison, we highlight four text-memory systems—A-MEM, Mem0, MemOS, and EverMemOS—that use the same Qwen3-VL-8B-Instruct backbone, the same Qwen3-VL-Embedding-2B retrieval model, and exactly the same 10-second caption inputs. Their performance is:
With the backbone, embedding model, and input observations held constant, overall accuracy still ranges from 18.4% to 31.4%. For example, MemOS and EverMemOS achieve the same PC accuracy (29.2%), yet differ considerably on mobile (35.5% vs. 14.0%). These findings suggest that beyond the choice of backbone and embedding models, differences in the end-to-end memory pipelines can also meaningfully affect performance. We list the visual-input systems separately because they use native frames rather than the shared captions. These comparisons are reported in Appendix B.4 (Shared-Backbone Comparisons), with system configurations in Appendix A.1.