- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PMM-Bench and HyperOS-Mem for long-term personal memory over smartphone OS-level observation streams. PMM-Bench contains 3.83 million records collected from 20 Android users over 685 user-days, together with approximately 2,000 questions covering single-hop recall, cross-application aggregation, temporal reasoning, event tracking, and habit induction. HyperOS-Mem constructs a hierarchical graph over events, event-level facts, and long-term memories. Its fast path, DRED, performs identifier-first and evidence-deferred online event attribution, while the slow path, SAGE, uses an LLM to revise event structure, extract grounded facts, and consolidate memories. Retrieval combines dense and identifier-based seeds with bounded graph navigation.
Critical strengths
-
PMM-Bench is based on naturally occurring OS-level traces rather than only synthetic dialogues or scripted application trajectories. It includes cross-application and long-horizon activities and associates reference answers with supporting observations. The benchmark is therefore potentially useful independently of the proposed method.
-
The temporal-asymmetry motivation behind DRED is sensible and empirically supported. The paper correctly observes that an OS record must be processed when it arrives, while its event membership may become clear only after later evidence appears. Identifier-first linking, deferred assignment, and separate update rates for chat and non-chat observations form a coherent response to this problem. The ablations are meaningful: removing deferred attribution decreases event tracking and temporal reasoning by 8.2% and 6.2%, respectively.
Critical weaknesses
-
Beyond DRED, the system combines several familiar ingredients: structured event summaries, fact extraction, entity/topic anchors, long-term consolidation, provenance links, and iterative graph retrieval. The paper does not provide a sufficiently controlled comparison showing that SAGE or the proposed hierarchy is better than a generic event-centric summarization and Graph-RAG pipeline under matched representations, retrieval controller, model calls, and token budgets.
-
Although the raw record count is large, the benchmark contains only 20 users, with 90–120 questions written per user after annotators manually grouped that user’s records into events. The paper does not report whether question writers, event annotators, and final reviewers were independent, how many annotators covered each example, or any inter-annotator agreement statistics. Moreover, the reported event-segmentation precision and recall are computed on only one participant (Section 4.4.4). No per-user variance are provided for the main results. Treating approximately 2,000 questions as independent would overstate the effective sample size because questions from the same user share the same observation history.
-
Several components rely on manually specified or ecosystem-dependent preprocessing, including regular-expression key extraction, predefined strong-identifier types, merchant-name canonicalization through a dictionary, and a manually defined set of messaging applications with a separate prototype-update rate (Section 3.1-3.2). Since the evaluation is restricted to 20 Android users and does not include cross-language, cross-region, or held-out-application tests, it remains unclear how robust these components are outside the benchmark’s application ecosystem.
-
The external-benchmark results provide only limited evidence for broad generalization. On Mem-Gallery, HyperOS-Mem improves over the 128K long-context baseline by only 0.48 Judge and 1.25 F1, while obtaining a 2.16 lower EM score (Table 13). On MemBench, its overall improvement over the strongest baseline is 1.54 (Section 4.4.3), and it does not lead on three of the four reported subcategories (Table 14). Since no uncertainty estimates are reported, it is unclear whether these relatively small aggregate gains are robust.
Questions for authors
see the above "Critical Weaknesses" part.
4 comments
Response to Reviewer c5xc (2/2)
W3: Several components rely on manual, ecosystem-dependent preprocessing, and robustness outside the benchmark's ecosystem is untested.
Thank you for raising this point. These settings are configuration tables used by standardization and DRED, and the DRED ablation shows the behavior without them (Section 4.4.1, Table 14): removing dual-rate updating, which removes the distinction for messaging apps, lowers Judge by 2.0, and removing identifier-first linking lowers it by 3.9. We have collected data from 1,000 users, including 54 English-speaking users, and are curating it for release in batches, which will extend PMM-Bench across languages.
W4: Gains on the external benchmarks are small and come without uncertainty estimates.
Thank you for this insightful comment. Our main evaluation is on PMM-Bench, where HyperOS-Mem leads the strongest baseline by 8.5 Judge (Table 3) and DRED improves three other memory systems by 3.2–3.5 points (Table 4). Mem-Gallery and MemBench are dialogue-based; we use them to verify that the method generalizes beyond OS observation streams, and HyperOS-Mem achieves the best overall score on both. The size of the gain depends on how closely each benchmark's data resembles OS observation streams:
- Mem-Gallery (Table 15). Each dialogue fits within the 128K window, so long context reads all the evidence and serves as a full-access reference; on PMM-Bench, the same window holds under 2% of one user's stream, and long context scores 10.1. Dialogue turns are also well bounded and semantically complete, so event attribution, which drives the gains on OS streams, has less to correct. HyperOS-Mem still matches long context on Judge (58.11 vs. 57.63) and exceeds it on F1 (66.28 vs. 65.03) while using 22.6% of its context tokens, and it leads every memory system on all three metrics.
- MemBench (Table 16). MemBench separates participatory memory, from dialogue the agent takes part in, and observational memory, from information the agent passively observes. HyperOS-Mem is designed for passively observed streams and leads on observational factual questions (51.60 vs. 48.01). Each subcategory leader is strong on only one memory type: flat retrieval leads both participatory subcategories but scores 33.84 on observational factual questions, and long context leads observational reflective questions but scores 22.91 on observational factual questions. HyperOS-Mem ranks first or second on both observational subcategories and leads overall (56.60 vs. 55.06 for A-Mem).
The gain thus grows with how closely the data resembles OS observation streams: 8.5 Judge on PMM-Bench, 3.6 points on observational factual questions in MemBench, and parity with full-access long context on Mem-Gallery.
Thank you again for the constructive feedback, which has strengthened the paper. If the revision addresses your concerns, we would be grateful if you would consider raising the score.
Sincerely,
All Anonymous Authors
Official Comment by Reviewer c5xc
Thanks for your detailed explanation, which addresses most of my concerns. I maintain my original positive score and recommend a weak acceptance.
Follow-up to Reviewer c5xc
Dear Reviewer c5xc,
Thank you for reading our responses and for your continued support of the paper. We appreciate your constructive comments, which have helped us improve the work.
Best regards,
The Authors
Response to Reviewer c5xc (1/2)
We sincerely thank the reviewer for the positive assessment, for recognizing PMM-Bench as useful independently of our method and the temporal-asymmetry motivation of DRED as sensible and empirically supported, and for the detailed and constructive comments. We address each point below and have uploaded a revised PDF (main changes in blue).
W1: Beyond DRED, the components are familiar, and there is no controlled comparison of SAGE and the hierarchy against a generic event summarization and Graph-RAG pipeline.
Thank you for raising this great point. The paper isolates DRED through the plug-in study (Table 4) and the components of SAGE through ablations (Figure 5, Table 14). For the requested comparison, we run GraphRAG on the same DRED events (
Appendix I, Table 23): SAGE is replaced by GraphRAG, which builds an entity graph with community summaries, with the same response model and retrieval token budget.SAGE leads by 8.9 Judge, with gaps of 10.3 on cross-application aggregation and 10.0 on habit induction, both of which need evidence consolidated across events; SplitEvent fires on 7–10% of events, separating activities that online attribution had merged. GraphRAG falls behind because it compresses low-semantic OS observations into entity–relation triples, which drops the context that ties each record to its activity; even with 1.3× the LLM calls of SAGE, it cannot recover this context at query time. CompassMem, a hierarchical event-graph pipeline, points the same way: with DRED it reaches 38.2 Judge (Table 4), 5.0 below HyperOS-Mem. SAGE and the event–anchor hierarchy therefore contribute beyond DRED.
W2: The annotation protocol and agreement are not reported, event segmentation is evaluated on one participant, and treating 2,000 questions as independent overstates the sample size.
Thank you for these detailed questions.
Annotation protocol and agreement (
Appendix C.4, Appendix G). Event annotation, question writing and final review were carried out by different people: annotators grouped each user's records into events, question writers wrote 90–120 questions per user with reference answers and evidence sets, and internal reviewers verified every QA pair against its supporting observations on a dedicated review platform. To measure agreement, three annotators segmented the same observation stream, and three annotators independently reviewed all 1,985 questions, labeling each reference answer as fully, partially or not supported by its cited evidence; disagreements were resolved by re-examining the evidence.Event segmentation. The precision and recall in Section 4.4.4 are computed on 2,400 observations and 310 events from one participant's stream, and the 93% inter-annotator agreement above shows that this human reference is stable.
Effective sample size. Since questions from one user share the same observation history, we treat the user as the unit of analysis and compare HyperOS-Mem with the strongest baseline by a paired Wilcoxon test over the 20 users (Appendix G, Table 19; ± is the standard deviation over three runs):
(Continued in the next comment.)