acceptodds
Under review as a conference paper at ICLR 2027

ScreenMem: Benchmarking Proactive Memory Maintenance over Long-Horizon Computer-Use Streams

Abstract

Memory is a fundamental capability of computer-use agents, enabling them to retain and reuse information accumulated throughout everyday computer use. Yet existing evaluations do not fully characterize a key setting for persistent memory: continuously observing long-horizon device activity, deciding what to preserve before future information needs are known, and later answering from the resulting memory without revisiting the original stream. We introduce ScreenMem, a benchmark for proactive, query-agnostic memory maintenance over day-long computer-use streams. ScreenMem reconstructs application interfaces and replays interactions from open-source real-world traces, producing 10 user-day streams across PC and mobile devices totaling 100.1 hours, together with 484 human-verified questions covering facts and states, state changes, activities and events, and behavioral patterns. Under a unified causal-ingestion protocol, we evaluate 15 existing memory systems spanning recent-context, implicit visual, explicit multimodal, and generic text-memory paradigms on both PC and mobile streams. Across 15 systems, the strongest evaluated system reaches only 31.4% overall accuracy, while State Evolution accuracy peaks at 15.8%. Further analyses reveal substantial variation across temporal horizons, together with pronounced differences in ingestion and storage efficiency. ScreenMem can facilitate systematic analysis of current memory systems and further research on persistent memory maintenance for computer-use agents.

Primary area:
datasets and benchmarks
Keywords:
artificial intelligence, device-use agent, proactive memory maintenance, memory benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zvLoOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ScreenMem introduces a benchmark for proactive, query-agnostic memory maintenance over long-horizon computer-use streams, offering a valuable evaluation framework but leaving concerns about baseline strength, perception–memory disentanglement, and real-world validity.

Critical strengths

1. Novel and well-defined evaluation setting: ScreenMem introduces a meaningful benchmark for proactive, query-agnostic memory maintenance, separating causal memory formation from subsequent question answering over long-horizon computer-use streams.

2. Comprehensive empirical evaluation: The paper evaluates 15 memory systems across four paradigms, providing detailed analyses of memory capabilities, temporal horizons, and computational efficiency that reveal important limitations of existing approaches.

Critical weaknesses

1. Lack of strong and simple baselines: The paper does not evaluate straightforward approaches such as timestamped OCR or visual-description storage with retrieval, making it difficult to determine whether complex memory architectures provide meaningful advantages.

2. Insufficient disentanglement of perception and memory: Different systems use different visual encoders, sampling strategies, and input representations, making it unclear whether failures arise from visual information extraction or memory maintenance.

3. Limited real-world validity: Although the benchmark is grounded in real interaction traces, its reconstructed interfaces and scripted activities may not fully capture the complexity and unpredictability of real-world computer use. No direct validation of this gap is provided.

4. Limited statistical robustness and evaluation validity: The benchmark contains only 10 user-day streams, lacks uncertainty estimates for system rankings, and mixes short-answer and multiple-choice questions, potentially confounding comparisons across memory capabilities.

Questions for authors

Q1. How does the benchmark distinguish persistent memory from raw observation storage?

Appendix A indicates that some systems retain visual frames or cached representations. What criteria distinguish legitimate memory maintenance from effectively storing and retrieving the original observation stream? Could this distinction affect the interpretation of the benchmark results?

Q2. What explains the particularly poor performance on State Evolution questions?

Table 3 shows that the best system achieves only 15.8% accuracy on State Evolution. Can the authors provide a qualitative error analysis distinguishing failures to update outdated information, confusion between historical and current states, and failures to retrieve correctly maintained states?

3 comments

We sincerely thank the reviewer for the detailed and constructive criticism. We have added two simple retrieval baselines, day-clustered uncertainty estimates, and a State Evolution error analysis to the manuscript, and conducted an evidence-window control reported in this response. We also clarify the goal of evaluating complete memory systems and the rationale for trace-grounded video reconstruction. We respond to each concern separately below.

W1. Strong and simple baselines

  1. Lack of strong and simple baselines: The paper does not evaluate straightforward approaches such as timestamped OCR or visual-description storage with retrieval, making it difficult to determine whether complex memory architectures provide meaningful advantages.

Response. To make this comparison explicit, we added two straightforward retrieval-based baselines: Caption-RAG, which stores timestamped visual descriptions, and OCR-RAG, which stores timestamped screen text. Both retrieve previously stored information to answer the same 484 questions under the same scoring rules as the existing systems.

MethodPC (312 questions)Mobile (172 questions)Overall (484 questions)
Caption-RAG30.13% (94/312)22.67% (39/172)27.48% (133/484)
OCR-RAG24.68% (77/312)16.86% (29/172)21.90% (106/484)
MemOS (strongest existing system overall)29.17% (91/312)35.47% (61/172)31.40% (152/484)

A particularly informative finding is that Caption-RAG outperforms all 15 previously evaluated memory systems on PC (30.13% versus MemOS's 29.17%), although MemOS remains stronger overall (31.40% versus 27.48%). This demonstrates that more sophisticated memory architectures do not necessarily outperform a well-implemented simple retrieval baseline, and strengthens the practical significance of the comparison. We have updated the main results table and included the baseline results in the main results table, with implementation and cost details in Appendix A.4 and Appendix D.8, respectively.

W2. Disentangling perception and memory maintenance

  1. Insufficient disentanglement of perception and memory: Different systems use different visual encoders, sampling strategies, and input representations, making it unclear whether failures arise from visual information extraction or memory maintenance.

Response. We agree that this distinction matters. Our evaluation is intended as an end-to-end comparison of memory systems, rather than an ablation in which all systems are required to use an identical visual encoder and memory implementation. The latter would change the native operation of several systems and no longer represent their released end-to-end capabilities.

To provide a more controlled comparison, we highlight four text-memory systems—A-MEM, Mem0, MemOS, and EverMemOS—that use the same Qwen3-VL-8B-Instruct backbone, the same Qwen3-VL-Embedding-2B retrieval model, and exactly the same 10-second caption inputs. Their performance is:

SystemPC (%)Mobile (%)Overall (%)
A-MEM21.812.218.4
Mem027.619.824.8
MemOS29.235.531.4
EverMemOS29.214.023.8

With the backbone, embedding model, and input observations held constant, overall accuracy still ranges from 18.4% to 31.4%. For example, MemOS and EverMemOS achieve the same PC accuracy (29.2%), yet differ considerably on mobile (35.5% vs. 14.0%). These findings suggest that beyond the choice of backbone and embedding models, differences in the end-to-end memory pipelines can also meaningfully affect performance. We list the visual-input systems separately because they use native frames rather than the shared captions. These comparisons are reported in Appendix B.4 (Shared-Backbone Comparisons), with system configurations in Appendix A.1.

W3. Real-world validity of reconstructed videos

  1. Limited real-world validity: Although the benchmark is grounded in real interaction traces, its reconstructed interfaces and scripted activities may not fully capture the complexity and unpredictability of real-world computer use. No direct validation of this gap is provided.

Response. To our knowledge, ScreenMem is the first benchmark focused on proactive memory maintenance over day-long device-use video streams. Building such a benchmark is difficult: recording continuous device activity is costly, and removing sensitive information from hours of personal screen recordings without disrupting the interactions is particularly challenging.

We therefore reconstruct videos from real-world interaction traces—RLKWiC for PC (Bakhshizadeh et al., 2024) and FingerTip 20K for mobile (Yang et al., ICLR 2026). Every recorded replay follows its source trajectory, and we manually verify the reconstructed interactions and question evidence. Although replay is deterministic, the original traces capture real action sequences, task dependencies, and evolving device states, preserving important complexities of actual device use.

W4. Statistical robustness and mixed answer formats

  1. Limited statistical robustness and evaluation validity: The benchmark contains only 10 user-day streams, lacks uncertainty estimates for system rankings, and mixes short-answer and multiple-choice questions, potentially confounding comparisons across memory capabilities.

Response. ScreenMem offers 100.1 hours of long-horizon streams and 484 verified questions for comparing systems under an identical evaluation protocol. To quantify the variation associated with the user-day units, we added 100,000-replicate, platform-stratified day-clustered bootstrap intervals and paired comparisons. For example, MemOS: 31.4% [27.2%, 36.2%] and Mem0: 24.8% [20.8%, 28.5%], with a paired difference of +6.6 percentage points [2.9, 11.9]. The narrower Mem0 − EverMemOS difference is +1.0 point [−3.1, 4.7]. This analysis supports a more informative reading of the rankings: the leading separation is relatively consistent across resampled days, while smaller gaps are appropriately distinguished from it. The full analysis is reported in Appendix B.5 (User-Day-Clustered Accuracy Intervals).

Regarding mixed formats, the benchmark is designed to compare systems on exactly the same questions within each capability dimension. Different capabilities may naturally require different forms of questions, just as different examination subjects do. This does not invalidate within-capability system comparisons, although it means that differences between capability averages should not be interpreted as pure differences in intrinsic memory difficulty. The existing question-format breakdown in Appendix B.2 provides additional context.

Q1. Persistent memory versus raw observation storage

Q1. How does the benchmark distinguish persistent memory from raw observation storage?

Appendix A indicates that some systems retain visual frames or cached representations. What criteria distinguish legitimate memory maintenance from effectively storing and retrieving the original observation stream? Could this distinction affect the interpretation of the benchmark results?

Response. We appreciate the opportunity to clarify the benchmark's evaluation objective. ScreenMem evaluates whether a memory system can effectively support a user's future information needs, not whether it follows a particular internal implementation of memory maintenance. Consequently, the benchmark should not prescribe that memory must be textual, semantically summarized, or compressed in a specific way. Systems may retain text, visual frames, learned features, or caches; we evaluate their ability to support future questions end to end, regardless of the retained representation.

The key evaluation constraint is causal, query-agnostic ingestion: the system must decide what to persist before seeing the future questions, and subsequently answer using only what it has retained, without reopening the original video stream. Raw-observation retention is therefore not excluded by definition; rather, its usefulness must be judged by answer accuracy together with the ingestion and storage costs that we report. In other words, our benchmark measures how effectively and efficiently a system serves as persistent memory for the user, rather than judging whether its underlying storage mechanism is a sufficiently “memory-like” algorithm. This is the evaluation constraint already defined by the causal sealed-memory protocol in Sec. 3.4.

Q2. Why is State Evolution particularly difficult?

Q2. What explains the particularly poor performance on State Evolution questions?

Table 3 shows that the best system achieves only 15.8% accuracy on State Evolution. Can the authors provide a qualitative error analysis distinguishing failures to update outdated information, confusion between historical and current states, and failures to retrieve correctly maintained states?

Response. We have conducted a State Evolution error analysis on the 57 questions in this category, covering all 15 original memory systems. Following the three failure modes in your question, we distinguish (1) failure to update outdated information, (2) confusion between historical and current states, and (3) failure to retrieve a correctly maintained state. The table below summarizes two representative systems; the remaining cases are grouped as Other errors.

SystemIncorrect / 57Failure to updateHistorical/current confusionFailure to retrieve maintained stateOther errors
MemOS48001929
M3-Agent54302031

For MemOS, 19 of 48 errors involve failure to retrieve a correctly retained state. For M3-Agent, 20 of 54 errors are retrieval failures and 3 involve outdated states not being updated. We found no cases of historical/current confusion under the strict criterion that both versions had reached the answerer; this does not rule out temporal information being lost earlier. Other errors aggregates remaining failures, including information missing from input captions, states not retained, and answer-stage mistakes. The full 15-system breakdown and category definitions are provided in Appendix B.6 (State Evolution Error Attribution).

Reviewer_tEjJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work proposed benchmark ScreenMem, that studies what should be remember in long-time user interactions. The ScreenMem comprises 10 reconstructed real-world traces, together with 484 human-verified questions.

Critical strengths
  1. ScreenMem targets proactive, query-agnostic memory maintenance: a system must ingest a day-long stream causally, decide what to keep before any question is known, and then answer from a sealed memory without re-accessing the stream. This cleanly separates memory formation from query-conditioned retrieval and task execution, a gap that prior-history retrieval and task-conditioned benchmarks do not cover.
  2. It consists of real-world user traces on both PC and mobile applications. The dataset is well verified and annotated by human. The median memory distance is 7.9 h on PC and 5.0 h on mobile, and 41.3% of questions require retention of at least 8 hours — so it truly tests long-horizon retention rather than short context windows.
  3. Facts & States, State Evolution, Activities & Transient Events, and Behavioral Patterns break the benchmark away from a single aggregate score, and the breakdown proves informative: State Evolution is the weakest axis, with a peak of only 15.8%, while no implicit visual-memory system answers a single State Evolution question correctly.
Critical weaknesses
  1. Only 10 reconstructed user-days and 27 applications, with mobile represented by just 3 days versus 7 for PC. This structurally limits statistical strength and makes PC vs. mobile comparisons descriptive rather than controlled — the authors themselves note the two platforms differ in user days and question composition, so platform gaps "characterize benchmark behavior" rather than generalization ability.
  2. All Behavioral Patterns questions are four-option multiple choice with a 25% chance floor, and the paper explicitly notes several systems sit near chance on them; the PC set is heavily short-answer skewed (264 short-answer vs. 48 MCQ), so capability and format are confounded.
Questions for authors
  1. is it possible to extend more data?
  2. what about the human performance?
2 comments

We appreciate the reviewer's positive assessment of the long-horizon, query-agnostic evaluation setting and the four memory capability dimensions. Thank you also for raising important questions about benchmark scale, evaluation format, and human performance.

W1 / Q1. Dataset scale and extensibility

Only 10 reconstructed user-days and 27 applications, with mobile represented by just 3 days versus 7 for PC. This structurally limits statistical strength and makes PC vs. mobile comparisons descriptive rather than controlled — the authors themselves note the two platforms differ in user days and question composition, so platform gaps "characterize benchmark behavior" rather than generalization ability.

is it possible to extend more data?

Response. We would like to emphasize the amount of evaluation evidence represented by ten user-days. ScreenMem uses a complete user-day as its basic unit because the target is causal, query-agnostic memory maintenance over long-horizon device-use streams, not question answering over isolated short clips. The current benchmark contains 100.1 hours of video across 27 applications, with 484 human-verified questions spanning four memory capabilities. By duration, 100.1 hours corresponds to over 600 ten-minute segments; we intentionally retain the full day-long streams for evaluation, since splitting them into independent short clips would remove the long-horizon challenge.

This scale already provides a substantial, standardized basis for comparing memory systems: every system is evaluated on the same underlying user-day streams and answers the same 484 questions, while the day-clustered bootstrap analysis quantifies variation across user-days. For instance, the leading MemOS–Mem0 difference is +6.6 percentage points (95% CI [2.9, 11.9]) under paired day-level resampling. At the same time, 100.1 hours of ingestion per system represents a meaningful evaluation cost. A much larger collection of full-length streams would considerably increase the expense of reproducing results across many systems, potentially limiting broad benchmark adoption. The present scale balances long-horizon coverage, verified evaluation evidence, and practical reproducibility.

ScreenMem is also extensible by design. Our reusable rule-based templates and trace-to-replay verification pipeline support adding further applications and user-days under the same protocol. We intend to continue expanding coverage, including the mobile subset, and release the expanded data.

W2. Capability dimensions versus answer formats

All Behavioral Patterns questions are four-option multiple choice with a 25% chance floor, and the paper explicitly notes several systems sit near chance on them; the PC set is heavily short-answer skewed (264 short-answer vs. 48 MCQ), so capability and format are confounded.

Response. Thank you for pointing this out. We would like to clarify the primary purpose of the capability breakdown: it is to compare different memory systems on the same set of questions within each capability, not to isolate the independent effect of question format or to rank the inherent difficulty of different capabilities.

An analogy is an examination covering mathematics, physics, and philosophy: different subjects may naturally use different question formats, but students taking the same questions within each subject can still be compared directly. Likewise, each ScreenMem system is evaluated on the identical questions and scoring rules for Facts & States, State Evolution, Activities & Events, and Behavioral Patterns. The resulting within-capability comparisons across systems are therefore directly comparable, even if question formats vary between capabilities.

We agree that answer format can affect absolute accuracy. Accordingly, we do not use differences between capability averages to claim that one memory ability is intrinsically harder than another solely because of the score gap. The question-format breakdown is reported in Appendix B.2 (Results by Question Format).

Q2. Human performance

what about the human performance?

Response. ScreenMem targets an application where automated assistance is particularly valuable: remembering details and changes accumulated over hours of device use. Rather than treating unaided human recall as the intended target, we performed a human-assisted note-taking evaluation to provide a concrete point of reference under a similar observe first, answer later setting.

An evaluator watched each user-day video once, with the option to take notes and save screenshots during viewing. Afterward, the evaluator answered the questions using only those saved records, without revisiting the videos. The results are:

PlatformCorrect / TotalAccuracy
PC193 / 31261.9%
Mobile113 / 17265.7%
Overall306 / 48463.2%

This 63.2% human-assisted reference, compared with 31.4% for the strongest evaluated memory system (MemOS), illustrates the remaining opportunity for device-use memory agents.

Reviewer_CJaKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ScreenMem tests whether systems can preserve information from day-long computer-use streams before seeing future questions. It provides 10 reconstructed user days and 484 questions; the best of 15 evaluated systems reaches 31.4% accuracy.

Critical strengths
  1. The causal ingestion and sealed-memory protocol clearly separates memory formation from query-driven history search (§3.4).
  2. Human-verified questions with supporting time intervals cover facts, state changes, transient events, and recurring behavior. This makes the benchmark more informative than a single recall score (§3.3).
  3. Evaluation spans 15 systems and reports ingestion time, tokens, and storage. Including captioning costs avoids understating the expense of text-memory pipelines (§4; Appendix D).
Critical weaknesses
  1. Low accuracy is not clearly attributed to memory loss. Without an evidence-window QA control, failures in screen reading or answer matching remain mixed with failures to retain information (§4.1; Appendix E).
  2. The 484 questions cover only 10 reconstructed days, a limitation acknowledged in Appendix F. Table 8 shows substantial variation, but uncertainty in the smaller system differences is not reported.
Questions for authors
  1. How well do the answering models perform on frames from the annotated evidence intervals, or their corresponding captions for text-memory systems, under the same scoring rules?
  2. Can you report day-clustered confidence intervals for the leading systems, especially the smaller gaps below MemOS?
1 comment

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of our causal-ingestion protocol, human-verified annotations, and comprehensive system evaluation. We respond to both concerns below.

W1 / Q1. Distinguishing memory failure from perception and answering errors

Low accuracy is not clearly attributed to memory loss. Without an evidence-window QA control, failures in screen reading or answer matching remain mixed with failures to retain information (§4.1; Appendix E).

How well do the answering models perform on frames from the annotated evidence intervals, or their corresponding captions for text-memory systems, under the same scoring rules?

Response. Thank you for this suggestion. We conducted the requested evidence-window answering experiments using the same QA backbone (Qwen3-VL-8B-Instruct), the same 484 questions, and the same scoring rules:

Answering inputCorrect / TotalAccuracy
Frames from annotated evidence intervals340 / 48470.25%
Captions from annotated evidence intervals326 / 48467.36%
Best evaluated memory system (MemOS)152 / 48431.40%

When the relevant evidence is directly available, answer accuracy increases substantially. This indicates that locating and preserving the information needed for future questions is a major challenge for current memory pipelines. The evidence-window results also remain below 100%, indicating that perception and reasoning are not perfect even when the relevant evidence is provided. We therefore interpret the original results as end-to-end memory-system performance, rather than attributing every incorrect answer to forgetting.

W2 / Q2. Day-clustered confidence intervals

The 484 questions cover only 10 reconstructed days, a limitation acknowledged in Appendix F. Table 8 shows substantial variation, but uncertainty in the smaller system differences is not reported.

Can you report day-clustered confidence intervals for the leading systems, especially the smaller gaps below MemOS?

Response. We have added day-clustered bootstrap confidence intervals using the original question-level results. We perform 100,000 platform-stratified bootstrap resamples, treating each user-day as a cluster (7 PC days and 3 mobile days), with the same sampled days used for paired system comparisons. Representative results for leading systems are shown below; additional system intervals and paired comparisons are provided in Appendix B.5.

SystemOverall accuracy95% day-clustered CI
MemOS31.4%[27.2%, 36.2%]
Mem024.8%[20.8%, 28.5%]
EverMemOS23.8%[19.5%, 28.2%]
M3-Agent22.7%[19.5%, 25.7%]

Importantly, the paired MemOS − Mem0 gap is +6.6 percentage points, with a 95% CI of [+2.9, +11.9], whereas the Mem0 − EverMemOS gap is +1.0 point, with a CI of [−3.1, +4.7]. Thus, the leading gap is more consistent across resampled user-days, while the smaller differences among the next-ranked systems are not clearly resolved. We briefly summarize the uncertainty analysis in Sec. 4.1 and provide the bootstrap procedure, paired intervals, and sensitivity analysis in Appendix B.5 (User-Day-Clustered Accuracy Intervals).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions to the Manuscript

We sincerely thank all reviewers for their valuable feedback and constructive suggestions. We have carefully revised our manuscript, with all modifications highlighted in blue. The main updates are summarized below:

  • Section 4 (Table 3): Added two simple retrieval baselines, Caption-RAG and OCR-RAG, with implementation details in Appendix A.4 and efficiency measurements in Appendix D.8.
  • Section 4.1 and Appendix B.5 (Tables 10–11): Added day-clustered bootstrap confidence intervals and paired comparisons to assess the robustness of system rankings.
  • Appendix B.4 (Table 9): Added shared-backbone comparisons to analyze performance differences among memory systems under matched model and input configurations.
  • Appendix B.6 (Table 12): Added a detailed State Evolution error analysis, examining failures to update outdated states, confusion between historical and current states, and retrieval failures.
  • Conclusion: Refined the discussion of the competitive retrieval baseline and its implications for memory maintenance over long-horizon computer-use streams.

We thank the reviewers again for their suggestions, which have helped strengthen the empirical analysis and presentation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.