acceptodds
Under review as a conference paper at ICLR 2027

HyperOS-Mem: OS-Centric Personal Memory via Fast–Slow Hierarchical Event Graph

Abstract

Existing personal memory systems and benchmarks predominantly focus on isolated interaction domains (e.g., dialogue or video), yielding memories that capture only partial slices of user activity trajectories. By contrast, users leave massive cross-app, cross-modal, and cross-temporal digital trajectories on Smartphone OS, such as the scattered conversations, orders, locations, and screens generated throughout a business trip. While building an OS-centric personal memory is highly promising, it faces three critical challenges: (1) fragment integration, as heterogeneous observations lack a unified structure; (2) event decoupling, given that parallel event logs intertwine chronologically; and (3) immediate attribution, because event membership and hierarchical context often require later evidence to resolve.To systematically study this problem, we construct PMM-Bench, comprising 3.83 million system-level records from 20 users, encompassing eight daily scenarios and featuring five representative evaluation settings. We further propose HyperOS-Mem, which organizes atomic observations into an evidence-grounded, temporally evolving, event-centric hierarchical memory graph through a coordinated fast–slow process.Experiments on PMM-Bench demonstrate that existing memory systems remain limited in this OS-level setting, whereas HyperOS-Mem achieves SOTA performance, validating the effectiveness of the event-centric memory paradigm.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Personal Agents, OS-Centric Memory, Hierarchical Event Graph, Personal Memory Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_c5xcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PMM-Bench and HyperOS-Mem for long-term personal memory over smartphone OS-level observation streams. PMM-Bench contains 3.83 million records collected from 20 Android users over 685 user-days, together with approximately 2,000 questions covering single-hop recall, cross-application aggregation, temporal reasoning, event tracking, and habit induction. HyperOS-Mem constructs a hierarchical graph over events, event-level facts, and long-term memories. Its fast path, DRED, performs identifier-first and evidence-deferred online event attribution, while the slow path, SAGE, uses an LLM to revise event structure, extract grounded facts, and consolidate memories. Retrieval combines dense and identifier-based seeds with bounded graph navigation.

Critical strengths
  1. PMM-Bench is based on naturally occurring OS-level traces rather than only synthetic dialogues or scripted application trajectories. It includes cross-application and long-horizon activities and associates reference answers with supporting observations. The benchmark is therefore potentially useful independently of the proposed method.

  2. The temporal-asymmetry motivation behind DRED is sensible and empirically supported. The paper correctly observes that an OS record must be processed when it arrives, while its event membership may become clear only after later evidence appears. Identifier-first linking, deferred assignment, and separate update rates for chat and non-chat observations form a coherent response to this problem. The ablations are meaningful: removing deferred attribution decreases event tracking and temporal reasoning by 8.2% and 6.2%, respectively.

Critical weaknesses
  1. Beyond DRED, the system combines several familiar ingredients: structured event summaries, fact extraction, entity/topic anchors, long-term consolidation, provenance links, and iterative graph retrieval. The paper does not provide a sufficiently controlled comparison showing that SAGE or the proposed hierarchy is better than a generic event-centric summarization and Graph-RAG pipeline under matched representations, retrieval controller, model calls, and token budgets.

  2. Although the raw record count is large, the benchmark contains only 20 users, with 90–120 questions written per user after annotators manually grouped that user’s records into events. The paper does not report whether question writers, event annotators, and final reviewers were independent, how many annotators covered each example, or any inter-annotator agreement statistics. Moreover, the reported event-segmentation precision and recall are computed on only one participant (Section 4.4.4). No per-user variance are provided for the main results. Treating approximately 2,000 questions as independent would overstate the effective sample size because questions from the same user share the same observation history.

  3. Several components rely on manually specified or ecosystem-dependent preprocessing, including regular-expression key extraction, predefined strong-identifier types, merchant-name canonicalization through a dictionary, and a manually defined set of messaging applications with a separate prototype-update rate (Section 3.1-3.2). Since the evaluation is restricted to 20 Android users and does not include cross-language, cross-region, or held-out-application tests, it remains unclear how robust these components are outside the benchmark’s application ecosystem.

  4. The external-benchmark results provide only limited evidence for broad generalization. On Mem-Gallery, HyperOS-Mem improves over the 128K long-context baseline by only 0.48 Judge and 1.25 F1, while obtaining a 2.16 lower EM score (Table 13). On MemBench, its overall improvement over the strongest baseline is 1.54 (Section 4.4.3), and it does not lead on three of the four reported subcategories (Table 14). Since no uncertainty estimates are reported, it is unclear whether these relatively small aggregate gains are robust.

Questions for authors

see the above "Critical Weaknesses" part.

4 comments
AuthorsOpenReview ↗

Response to Reviewer c5xc (1/2)

We sincerely thank the reviewer for the positive assessment, for recognizing PMM-Bench as useful independently of our method and the temporal-asymmetry motivation of DRED as sensible and empirically supported, and for the detailed and constructive comments. We address each point below and have uploaded a revised PDF (main changes in blue).

W1: Beyond DRED, the components are familiar, and there is no controlled comparison of SAGE and the hierarchy against a generic event summarization and Graph-RAG pipeline.

Thank you for raising this great point. The paper isolates DRED through the plug-in study (Table 4) and the components of SAGE through ablations (Figure 5, Table 14). For the requested comparison, we run GraphRAG on the same DRED events (Appendix I, Table 23): SAGE is replaced by GraphRAG, which builds an entity graph with community summaries, with the same response model and retrieval token budget.

Slow pathLLM calls (relative)JudgeCrossHabit
GraphRAG1.30×34.333.230.6
SAGE (HyperOS-Mem)1.0×43.243.540.6

SAGE leads by 8.9 Judge, with gaps of 10.3 on cross-application aggregation and 10.0 on habit induction, both of which need evidence consolidated across events; SplitEvent fires on 7–10% of events, separating activities that online attribution had merged. GraphRAG falls behind because it compresses low-semantic OS observations into entity–relation triples, which drops the context that ties each record to its activity; even with 1.3× the LLM calls of SAGE, it cannot recover this context at query time. CompassMem, a hierarchical event-graph pipeline, points the same way: with DRED it reaches 38.2 Judge (Table 4), 5.0 below HyperOS-Mem. SAGE and the event–anchor hierarchy therefore contribute beyond DRED.

W2: The annotation protocol and agreement are not reported, event segmentation is evaluated on one participant, and treating 2,000 questions as independent overstates the sample size.

Thank you for these detailed questions.

Annotation protocol and agreement (Appendix C.4, Appendix G). Event annotation, question writing and final review were carried out by different people: annotators grouped each user's records into events, question writers wrote 90–120 questions per user with reference answers and evidence sets, and internal reviewers verified every QA pair against its supporting observations on a dedicated review platform. To measure agreement, three annotators segmented the same observation stream, and three annotators independently reviewed all 1,985 questions, labeling each reference answer as fully, partially or not supported by its cited evidence; disagreements were resolved by re-examining the evidence.

StepCoverageAgreement
Event segmentation1 participant, 3 annotators93%
QA review1,985 questions, 3 annotators97%
LLM judge vs. human200 answers, 3 ratersκ = 0.94

Event segmentation. The precision and recall in Section 4.4.4 are computed on 2,400 observations and 310 events from one participant's stream, and the 93% inter-annotator agreement above shows that this human reference is stable.

Effective sample size. Since questions from one user share the same observation history, we treat the user as the unit of analysis and compare HyperOS-Mem with the strongest baseline by a paired Wilcoxon test over the 20 users (Appendix G, Table 19; ± is the standard deviation over three runs):

MethodJudgeF1HyperOS-Mem winsWilcoxon p
CompassMem34.7 ± 6.430.9 ± 5.218 / 20< 0.002
HyperOS-Mem43.2 ± 5.838.4 ± 4.6––

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer c5xc (2/2)

W3: Several components rely on manual, ecosystem-dependent preprocessing, and robustness outside the benchmark's ecosystem is untested.

Thank you for raising this point. These settings are configuration tables used by standardization and DRED, and the DRED ablation shows the behavior without them (Section 4.4.1, Table 14): removing dual-rate updating, which removes the distinction for messaging apps, lowers Judge by 2.0, and removing identifier-first linking lowers it by 3.9. We have collected data from 1,000 users, including 54 English-speaking users, and are curating it for release in batches, which will extend PMM-Bench across languages.

W4: Gains on the external benchmarks are small and come without uncertainty estimates.

Thank you for this insightful comment. Our main evaluation is on PMM-Bench, where HyperOS-Mem leads the strongest baseline by 8.5 Judge (Table 3) and DRED improves three other memory systems by 3.2–3.5 points (Table 4). Mem-Gallery and MemBench are dialogue-based; we use them to verify that the method generalizes beyond OS observation streams, and HyperOS-Mem achieves the best overall score on both. The size of the gain depends on how closely each benchmark's data resembles OS observation streams:

  • Mem-Gallery (Table 15). Each dialogue fits within the 128K window, so long context reads all the evidence and serves as a full-access reference; on PMM-Bench, the same window holds under 2% of one user's stream, and long context scores 10.1. Dialogue turns are also well bounded and semantically complete, so event attribution, which drives the gains on OS streams, has less to correct. HyperOS-Mem still matches long context on Judge (58.11 vs. 57.63) and exceeds it on F1 (66.28 vs. 65.03) while using 22.6% of its context tokens, and it leads every memory system on all three metrics.
  • MemBench (Table 16). MemBench separates participatory memory, from dialogue the agent takes part in, and observational memory, from information the agent passively observes. HyperOS-Mem is designed for passively observed streams and leads on observational factual questions (51.60 vs. 48.01). Each subcategory leader is strong on only one memory type: flat retrieval leads both participatory subcategories but scores 33.84 on observational factual questions, and long context leads observational reflective questions but scores 22.91 on observational factual questions. HyperOS-Mem ranks first or second on both observational subcategories and leads overall (56.60 vs. 55.06 for A-Mem).

The gain thus grows with how closely the data resembles OS observation streams: 8.5 Judge on PMM-Bench, 3.6 points on observational factual questions in MemBench, and parity with full-access long context on Mem-Gallery.

Thank you again for the constructive feedback, which has strengthened the paper. If the revision addresses your concerns, we would be grateful if you would consider raising the score.

Sincerely,
All Anonymous Authors

Reviewer_c5xcOpenReview ↗

Official Comment by Reviewer c5xc

Thanks for your detailed explanation, which addresses most of my concerns. I maintain my original positive score and recommend a weak acceptance.

AuthorsOpenReview ↗

Follow-up to Reviewer c5xc

Dear Reviewer c5xc,

Thank you for reading our responses and for your continued support of the paper. We appreciate your constructive comments, which have helped us improve the work.

Best regards,
The Authors

Reviewer_BK4zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds long-term personal memory from smartphone OS-level records: screen operations, app switches, notifications, and messages, logged passively across apps on the device. These records arrive at high frequency, carry little meaning on their own, and interleave across activities, which leads the authors to three challenges: fragment integration, event decoupling, and immediate attribution.

This work builds PMM-Bench, a benchmark of OS-level long-term personal memory records with about 2,000 QA pairs linked to supporting evidence, and proposes HyperOS-Mem. It standardizes heterogeneous device records into uniform atomic observations with canonical keys, giving fragments from different apps and modalities a common form. A fast path (DRED) then attributes these observations to parallel events online, and an LLM-based slow path (SAGE) later revises the events, extracts facts, and consolidates long-term memories into an event-centric hierarchical graph used for retrieval. This work shows better performance comparing with the baselines.

Critical strengths
  1. Each method component addresses the main challenges from the introduction section identified for continuous OS streams, and the design is solid, backed up by ablation studies: removing a component mainly hurts the question type it targets.
  2. PMM-Bench is a strong contribution. Uncurated OS logs from real users paired with human-written questions tied to supporting evidence are valuable.
Critical weaknesses
  1. The novelty of the setting is vague, and the value of storing low-level OS data is not justified. Multi-source personal memory benchmarks that address the experience coverage gap already exist as the authors also cite (e.g., MobileMem[1], ATM-Bench[2], SMMBench[3]). What really distinguishes PMM-Bench is its data source: a low-level interaction stream across all apps. However, neither the method nor the evaluation examines what this data adds over existing multi-source settings.

  2. HyperOS-Mem's goal is building a hierarchical event graph for accurate retrieval, but the paper does not describe the event graph clearly enough to see what is actually built. There is limited elaboration on the graph's nodes and edges: which node types exist, which relations connect them, and how events relate to each other beyond sharing an anchor.

[1] Deng, Xinle, et al. "MobileMem: Learning from a Year of Mobile Experiences." arXiv preprint arXiv:2608.13606 (2026). [2] Mei, Jingbiao, et al. "According to me: Long-term personalized referential memory qa." arXiv preprint arXiv:2603.01990 (2026). [3] Chai, Huacan, et al. "SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory." arXiv preprint arXiv:2605.15710 (2026).

Questions for authors

Thank you for submitting this work to ICLR! I have some questions about this work as below.

  • As mentioned in the weaknesses, it is unclear what new queries become possible by using OS-centric information. For example, the five question types and the slice analysis in Fig. 4 (number of apps, evidence time span) does not differentiate from what existing multi-source benchmarks already cover, and the stylus example in Fig. 2a also looks answerable with multiple per-app records such as shopping-app search histories and SMS, without OS-level traces. I suggest defining and evaluating query types that only this data can answer, such as behavior that leaves no record inside an app (browsing or comparing without buying), transitions between apps, temporal usage trajectories, or usage patterns. Two further analyses could help:

    • An ablation over record types (e.g., descriptive records only vs. adding interaction records), to show what the low-level OS data contributes.
    • Reporting evidence recall as a retrieval metric will show whether low-level records such as interface actions are actually retrieved and used. This is feasible considering the ground truth evidence for each QA is already recorded during benchmark construction.
  • Evaluation results tells that the method outperforms the baselines and transfers well to other benchmarks, and the ablations show that the gains come from each component. Still it remains unclear whether these gains come from the low-level OS data, so clarifying the actual novelty that produces the accuracy gap would strengthen the paper.

  • Could you provide a compact specification of the event graph, such as a figure or table listing node and edge types (including anchors) and how each is created? I found the event graph difficult to understand overall, and presenting it this way would make it much clearer. In addition, lots of terms such as "anchor" and "event" appear in Section 3 before they are clearly defined, which made the method hard to follow.

  • The main evaluation metric is the LLM-judge score. It will be helpful to have human score a sampled subset of the queries and report the correlation between the human scores and the LLM-judge scores to show that reported scores are reliable.

Minor comments that did not impact the score:

  • Table 1 lists location (L) and sensor (S) among the modalities of PMM-Bench, but it is unclear how these records are actually used in the method. Clarifying their role and how many questions rely on location or sensor evidence will be great.
  • A few works are missing from the references (e.g. MemBench).
  • Number mismatches between Figure 1 and introduction ("...sensor readings and screen operations make up 51.6%...")
9 comments
AuthorsOpenReview ↗

Response to Reviewer BK4z (1/3)

We sincerely thank the reviewer for the thoughtful review, for recognizing that each component addresses a challenge of continuous OS streams and that PMM-Bench is a strong contribution, and for the concrete suggestions. We have followed all of them and uploaded a revised PDF (main changes in blue).

W1: The novelty of the setting is vague, and the value of low-level OS data over existing multi-source benchmarks is not justified.

Thank you for raising this great point. PMM-Bench differs from existing multi-source benchmarks in its data source: as Table 1 shows, it is the only one built from the raw OS-level interaction records of real users. The value of this data rests on three points.

  • Access. Data inside third-party apps, such as a shopping app's search history, is not accessible to a system-level assistant; third-party apps are observed through screen operations, and the OS-level stream records all apps in the same form.
  • Method. As the reviewer notes among the strengths, DRED's identifier-first linking, deferred attribution and dual-rate updating address the properties of this stream identified in Section 1: records that carry little meaning on their own, event boundaries confirmed only by later evidence, and interleaved activities.
  • Benchmarks. Table 1 compares PMM-Bench with existing benchmarks by data source and naturalness, and Table 5 (Appendix B) extends this comparison to the three cited works:
BenchmarkData sourceReal usersRaw OS stream# QA
MobileMemLLM-synthesized app recordsPartlyNo7,415
ATM-BenchPhotos, videos, emailsYesNo1,038
SMMBenchPublic datasets, simulated dialogueNoNo1,877
PMM-BenchOS-level records (3.83M)20Yes~2,000

MobileMem seeds its LLM synthesis with real app-usage metadata and leaves out raw interaction content; PMM-Bench keeps these interface-level signals, which standardization and DRED are designed to process. Our response to Q1 below measures what this data adds.

Overall, we find that OS-level interaction records turn behavior that leaves no record inside any app into answerable memory: browsing and comparing without buying, transitions between apps and usage trajectories appear only in screen operations, and 60% of PMM-Bench questions can be answered only from these records.

W2: The event graph is not described clearly enough: which nodes and edges exist, and how events relate beyond shared anchors.

Thank you for pointing this out. Section 3 defines the node and edge types of the event graph and the component that creates each, and Table 2 summarizes them:

KindNameConnectsCreated by
NodeAtomic observation–Standardization
NodeEvent–DRED
NodeFact–ExtractFacts
NodeLong-term memory–Consolidation
NodeEntity / topic anchor–DRED / SummarizeEvent
EdgeMembershipobservation → eventDRED
EdgeAnchor linkevent – anchorDRED, SummarizeEvent
EdgeSplit linkevent → child eventSplitEvent
EdgeProvenancefact → observations; memory → eventsSAGE

Beyond shared anchors, events relate only through split links. Connecting events through anchors forms an event–anchor bipartite graph, so relations need not be extracted for every pair of events, and every abstraction stays traceable to its observations.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer BK4z (2/3)

Q1: Which queries can only OS-level data answer, and do record-type ablation and evidence recall show that the gains come from this data?

Thank you for this insightful question and the concrete suggestions. Section 4.3 already points this way: the gap between HyperOS-Mem and the baselines widens as evidence spreads across more applications (from 12.0 to 19.8 points over LightMem) and longer time spans (from 5.7 to 16.6 points over CompassMem; Figure 4). Following the reviewer's suggestions, we add three analyses (Appendix H, Tables 20–22); all methods receive the same OS data.

Which questions need interaction records (Table 20). 84% of questions have at least one evidence item from screen operations, and 60% (1,183 questions) can be answered only from them.

Queries only this data can answer (Table 21). We evaluate the 1,183 questions whose evidence lies only in screen operations, grouped by our five question types. Event tracking and habit induction correspond to the temporal usage trajectories and usage patterns the reviewer suggests, and these questions cover behavior that leaves no record inside any app, for example "After seeing a battery-swap discount in the UU app early on May 5, which plans did I compare next?", "How far did I get with Honor of Kings World, from downloading the game to claiming the gift pack?" and "When I see a rain or rainstorm warning, in what order do I usually check weather information?"

Question typeCompassMemHyperOS-Mem
Single-hop recall39.747.0
Aggregation across applications30.142.3
Temporal reasoning30.541.0
Event tracking28.042.5
Habit induction31.341.0
All OS-only questions31.942.8

On these questions the gap is 10.9, compared with 8.5 over all questions. Section 4.3 reports this subset.

Record-type ablation (Table 21). Descriptive records are SMS, notifications, gallery, location and sensor records:

SettingSingleCrossTempTrackHabitJudge
Descriptive records only39.827.830.227.428.930.8
+ Interaction records (full)48.243.541.142.840.643.2

Evidence recall (Table 22), within the retrieval budget (B = 20):

MethodScreen operationsOther recordsOverall
Flat dense retrieval31.8%46.2%37.5%
HyperOS-Mem67.3%76.1%71.6%

These results show that the gains come from OS-level interaction data: they are largest on questions only this data can answer, removing it lowers Judge from 43.2 to 30.8, and HyperOS-Mem recalls 67.3% of interaction-record evidence compared with 31.8% for flat retrieval. In the stylus example of Figure 2a, the purchase is visible in SMS, but the earlier comparison across apps exists only in screen operations, as the caption states.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer BK4z (3/3)

Q2: Provide a compact specification of node and edge types, and define terms such as anchor and event before use.

Thank you for this helpful suggestion. The specification is Table 2 (see our response to W2 above), and atomic observation, event, anchor, fact and long-term memory are defined at the start of Section 3, before they are used.

Q3: Report the correlation between human scores and LLM-judge scores on a sampled subset.

Thank you for this suggestion. Three human raters scored the system answers to 200 sampled questions with the same 0–5 rubric as GPT-4o (Appendix G, Table 18):

MeasureValue
Spearman ρ / Pearson r0.93 / 0.94
Quadratic weighted κ0.94
Identical scores84%
Correctness agreement (≥ 4)95%

The GPT-4o scores agree closely with the human ratings.

Minor comments: the role of location and sensor records, missing references, and a number mismatch in Figure 1.

Thank you for these careful comments.

  • Location and sensor. Location records contribute place keys and sensor records contribute temporal context; both help attribute observations to the right event. Their content serves as evidence for 8% and 12% of questions, respectively (Appendix H, Table 20).
  • References. We have added MemBench, ATM-Bench, DynamicMem and GAM (Section 4.4.3, Appendix B).
  • Figure 1. Thank you; we have unified the numbers in Figure 1 and the introduction.

Ethics: the paper does not mention IRB approval.

We appreciate the reviewer raising this. Data collection complied with the applicable institutional review requirements; the name of the review body is withheld to preserve anonymity and will be provided in the final version. Every participant signed an informed consent form covering the research purpose, collected data, intended use, privacy risks and controlled release. At the end of the study, each participant reviewed the raw records on their own device and confirmed the export (Appendix C.1). Direct identifiers were removed or masked, records that could not be reliably de-identified were excluded, and the benchmark is released only under a data-use agreement (Ethics statement; Appendix C.5).

Thank you again for the concrete suggestions, which have made the paper clearer. If these analyses address your concerns, we would be grateful if you would consider raising the score.

Sincerely,
All Anonymous Authors

Reviewer_5DxgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies personal memory at the OS level. The idea is to build long-term memory from the traces a smartphone leaves behind, such as screen operations, messages, notifications, photos, locations and sensor data. The authors make two contributions. The first is PMM-Bench, a dataset collected from 20 Android users over 685 user-days. It contains 3.83 million real records and about 2,000 QA pairs across five task types, and every answer is linked to its supporting evidence. The second is HyperOS-Mem, a framework with a fast path and a slow path. The fast path assigns incoming observations to parallel events in real time, and it does this without a generative LLM. The slow path is where the LLM comes in. Working in batches, it segments events, extracts grounded facts and consolidates long-term memories across events.

Critical strengths
  • This paper examine system-level traces that span apps, modalities and time. The data comes from real, everyday phone use, and every question is tied to specific observations. This makes the benchmark much closer to real life than benchmarks built on synthetic dialogues.

  • The authors compare against 8 baselines from three categories. The paper also includes detailed ablations, plug-in experiments, breakdowns by number of apps and by the time span of the evidence, precision and recall for event segmentation, and generalization tests on two public benchmarks.

  • Participants gave informed consent. The data was de-identified, and it is released with controlled access.

Critical weaknesses
  • I feel the dataset is still small in terms of size. Only 20 Android users and mainly in Chinese.

  • Fig. 8 and Fig. 9 seem generated by AI. Perhaps replot?

  • Apps are mostly in the Chinese market according to Figure 12.

  • Evaluation is limited. The evaluation only uses GPT-4o as a judge and was not evaluated by humans. There is no statistical information about the benchmark.

Questions for authors
  • What was the inter-annotator agreement for event segmentation and QA annotation? How closely did the LLM judge’s ratings agree with human ratings? Did the participants themselves review these questions?

  • How many questions require sensor, location, or image evidence to answer? Could you conduct ablation experiments by modality?

  • Can you provide standard deviation across multiple runs or a significance test.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 5Dxg (1/3)

We sincerely thank the reviewer for the careful reading, and for recognizing that PMM-Bench is grounded in real, everyday phone use with every question tied to specific observations, as well as the breadth of our evaluation. Below we clarify each point and describe the corresponding revisions to the paper; the revised PDF has been uploaded (main changes in blue).

W1: The dataset is small, with only 20 Android users and mainly Chinese content.

Thank you for raising this point. Compared with widely used long-term memory benchmarks such as LoCoMo, LongMemEval and PersonaMem, whose interaction histories are generated or simulated, the 20 real users of PMM-Bench span diverse demographics, occupations and app usage (Appendix C). PMM-Bench evaluates whether a memory system can integrate evidence across applications and across time from real, long-term, heterogeneous phone histories, with 685 user-days, 3.83 million records and about 2,000 questions. Among the benchmarks compared in Tables 1 and 5, it is the only one built from the raw OS-level records of real users.

Beyond the 20 users in the submission, we have collected data from 1,000 users, including 54 English-speaking users; the 20 users are those curated and de-identified by the submission deadline. We are continuing to curate and de-identify this data and will release it in batches, as now stated in the Conclusion, so that PMM-Bench grows into a large-scale resource for OS-level personal memory research covering both Chinese and English users.

W2: Figures 8 and 9 look AI-generated and should be replotted.

Thank you for pointing this out. Figure 8 is plotted with Python and ReportLab from the logged observations of one participant, showing their times, applications and event memberships. Figure 9 has been replaced with Table 6 (Appendix C.1), which lists each recorded trace family with its source, fields and modality.

W3: Most apps come from the Chinese market (Figure 12).

Thank you for this observation. The pipeline works on standardized atomic observations, and its strong identifiers (order, phone and tracking numbers) are common to other app ecosystems. Existing mobile memory benchmarks are likewise built within one ecosystem: MobileMem uses OPPO's native apps of seven types (bills, voice recorder, screen memo, documents, notes, calendar and to-do), with LLM-synthesized message content. PMM-Bench records the everyday use of third-party apps across eight scenarios from raw OS-level records (Appendix C.2, Table 7).

W4: The evaluation relies only on GPT-4o as judge, without human evaluation or statistics of the benchmark.

Thank you for this suggestion.

Evaluation setup. Scoring with a single LLM judge is common practice in memory evaluation: LongMemEval uses GPT-4o as its judge, and MobileMem uses a single Qwen model. We keep GPT-4o as the judge with type-specific rubrics and vary the response model; with GPT-5-mini in place of GPT-4.1-mini, HyperOS-Mem remains the best overall (Table 17). Token-overlap F1, which does not depend on an LLM, is reported alongside, and MemBench uses multiple-choice accuracy.

Dataset statistics. Appendix C.2 and Figures 9–11 report the composition of behavior and record types, activity levels, record volumes and app usage per user; statistics by evidence type are given in our response to Q2 below.

Human validation. Three raters scored the system answers to 200 sampled questions with the same 0–5 rubric as GPT-4o. Human and GPT-4o scores reach a quadratic weighted κ of 0.94 and agree on correctness for 95% of answers, so the GPT-4o judge closely tracks human ratings. Details are given in our response to Q1 below.

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer 5Dxg (2/3)

Q1: What is the inter-annotator agreement for event segmentation and QA, how well does the LLM judge agree with humans, and did participants review the questions?

Thank you for this insightful question. Event annotation, question writing and final review were carried out by different people. Reviewers work on a dedicated platform that shows each question with its reference answer, supporting events, original operation records and available screenshots; problematic QA pairs were returned for revision and verified again (Appendix C.4, Figure 13).

(1) Event segmentation and QA annotation.

  • Event segmentation. The three annotators segmented the same observation stream of one participant with a raw agreement of 93%. This is agreement among humans; the precision and recall in Figure 7(b) instead compare the system with the human annotation.
  • QA annotation. The three annotators independently reviewed all 1,985 questions, labeling each reference answer as fully, partially or not supported by its cited evidence, with a raw agreement of 97%. Disagreements were resolved by re-examining the evidence, and the QA pairs were revised accordingly.

(2) LLM judge vs. human ratings (Appendix G, Table 18). Three raters scored the system answers to 200 sampled questions with the same 0–5 rubric as GPT-4o:

MeasureValue
Identical scores84%
Difference ≤ 1 point98%
Quadratic weighted κ0.94
Pearson r / Spearman ρ0.94 / 0.93
Mean difference (LLM − human)+0.03
Correctness agreement (≥ 4)95%

(3) Participant review. At the end of the study, each participant reviewed the raw records on their own device and confirmed the export, which served to check the records and confirm the export scope (Appendix C.1). The questions and reference answers were then written by question writers and verified against the supporting observations by separate internal reviewers (Appendix C.4).

Overall, we find that linking every question to its supporting observations is the key to a reliable OS-level benchmark: it lets reviewers verify each answer against the raw records, keeps human agreement high (93% on event segmentation and 97% on QA support), and allows an LLM judge to match human ratings closely (κ = 0.94).

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer 5Dxg (3/3)

Q2: How many questions require sensor, location or image evidence, and can a modality ablation be provided?

Thank you for this helpful question.

Questions by evidence type (Appendix H, Table 20). We assign each evidence item to the source of its content. Gallery, location and sensor or device-status content serves as evidence for 9%, 8% and 12% of questions, and 27% of questions combine two or more sources.

Content sourceAny evidenceOnly this source
Screen operations84%60%
Notifications14%3%
SMS / messages2%1%
Gallery images9%3%
Location8%4%
Sensor / device status12%3%

Modality ablation (Appendix H, Table 21). Building memory without screen-interaction records:

SettingSingleCrossTempTrackHabitJudge
Without interaction records39.827.830.227.428.930.8
Full HyperOS-Mem48.243.541.142.840.643.2

Judge drops from 43.2 to 30.8, most on cross-application aggregation (−15.7) and event tracking (−15.4): screen-interaction records, which only OS-level collection provides, carry much of the evidence these tasks require.

Q3: Can you provide standard deviations across runs or a significance test?

Thank you for raising this question. Table 3 now reports the mean ± standard deviation over three independent runs, each rebuilding the memory of every user:

MethodJudge (mean ± std)F1 (mean ± std)
Long context (128K)10.1 ± 2.58.7 ± 3.3
Flat dense retrieval22.8 ± 6.219.9 ± 4.5
SimpleMem30.5 ± 5.427.1 ± 6.5
LightMem27.7 ± 7.524.4 ± 5.5
Mem025.2 ± 5.022.3 ± 5.7
A-Mem31.4 ± 6.627.9 ± 4.2
StructMem29.1 ± 5.525.8 ± 6.9
CompassMem34.7 ± 6.430.9 ± 5.2
↳ + identifier index36.6 ± 5.532.6 ± 5.9
HyperOS-Mem43.2 ± 5.838.4 ± 4.6

For significance, since questions from one user share the same observation history, we treat the user as the unit of analysis: a paired Wilcoxon test shows that HyperOS-Mem outperforms CompassMem, the strongest baseline, on 18 of 20 users (p < 0.002; Appendix G, Table 19), so the gain holds consistently across users.

Thank you again for the constructive review. If these clarifications and results address your concerns, we would be grateful if you would consider raising the score.

Sincerely,
All Anonymous Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank all reviewers for their constructive feedback. Reviewer c5xc\color{#188038}{\textbf{c5xc}} confirmed that our responses address most of the concerns and recommends acceptance. Reviewer BK4z\color{#1A56DB}{\textbf{BK4z}} engaged in two rounds of discussion, and we answered every follow-up question with new analyses and revisions. Reviewer 5Dxg\color{#D93025}{\textbf{5Dxg}} has not replied yet; the concerns were about dataset scale, human evaluation and benchmark statistics, all of which we addressed.

Core Contributions

  1. A new setting: OS-level personal memory. We introduce the problem of building long-term personal memory directly from the smartphone OS, which observes every app, including third-party apps, in one stream; PMM-Bench and HyperOS-Mem are designed for this setting.
  2. PMM-Bench. 3.83 million raw OS records from 20 real users over 685 user-days, with about 2,000 evidence-linked questions.
  3. HyperOS-Mem. DRED attributes observations to events online without a generative LLM, and SAGE refines them into grounded facts and long-term memory.

Strengths Acknowledged by Reviewers

  1. Valuable benchmark: "a strong contribution" (BK4z\color{#1A56DB}{\textbf{BK4z}}), "much closer to real life than benchmarks built on synthetic dialogues" (5Dxg\color{#D93025}{\textbf{5Dxg}}), and "potentially useful independently of the proposed method" (c5xc\color{#188038}{\textbf{c5xc}}).
  2. Sound method design: each component "addresses the main challenges" and is "backed up by ablation studies" (BK4z\color{#1A56DB}{\textbf{BK4z}}); the motivation of DRED is "sensible and empirically supported" (c5xc\color{#188038}{\textbf{c5xc}}).
  3. Thorough evaluation: eight baselines, detailed ablations, plug-in experiments and generalization tests (5Dxg\color{#D93025}{\textbf{5Dxg}}).

Major Concerns and Our Responses

  1. Value of OS-level data (BK4z\color{#1A56DB}{\textbf{BK4z}}). 60% of questions can be answered only from interaction records, and removing these records lowers HyperOS-Mem from 42.8 to 17.8 on them (Tables 20 and 21).
  2. Difference from prior benchmarks (BK4z\color{#1A56DB}{\textbf{BK4z}}). Unlike MobileMem and MemGUI-Bench, PMM-Bench records real users' everyday interactions at the system layer.
  3. Evaluation reliability (5Dxg\color{#D93025}{\textbf{5Dxg}}, BK4z\color{#1A56DB}{\textbf{BK4z}}, c5xc\color{#188038}{\textbf{c5xc}}). Human ratings agree closely with the LLM judge, annotators agree on segmentation and QA, and the gains hold across users and response models.
  4. Dataset scale and statistics (5Dxg\color{#D93025}{\textbf{5Dxg}}). Tables 6 and 20 report the benchmark statistics, and data collection now covers 1,000 users, including English-speaking users.

Sincerely,
All Anonymous Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.