- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PraxisMemBench, a benchmark for evaluating the joint maintenance of personal and procedural memories in long-term personal assistants and their impact on downstream tool use and task completion. The authors also propose PraxisMem, which combines location-aware GRPO to optimize memory content and organization with dynamic residual-routed OPSD to provide additional supervision for unresolved memory errors. Experiments show that PraxisMem achieves strong end-to-end task completion results on the proposed benchmark.
Critical strengths
- Meaningful research problem. The paper studies how personal context affects the applicability of procedural knowledge, going beyond conventional memory benchmarks that focus mainly on recalling historical facts.
- Systematic benchmark design. The benchmark covers multiple sessions, application domains, and task types.
- Clear presentation. The paper is well-written and easy to follow.
Critical weaknesses
- Limited external validity of the benchmark. The data are primarily generated with LLM assistance and validated through automated procedures, which may not fully capture the ambiguity, conflicting preferences, and complex state changes found in real-world user interactions.
- Insufficient algorithmic novelty. Location-aware GRPO and dynamic residual-routed OPSD build on existing reward-design methods for reinforcement learning and on-policy self-distillation, respectively. The proposed approach appears to combine existing techniques for a new task, but the paper does not sufficiently explain the underlying rationale for their applicability to the core problem.
- Limited statistical evidence for the ablation studies. The key ablations use only a small number of trajectories, with insufficient results across random seeds, confidence intervals, and comparisons under matched computational budgets.
- Some evaluation details require clarification. The paper should clarify how the overall Session SR is aggregated, the denominators used to calculate failure-type proportions, and the comparability of results across different evaluation subsets. For example, the proportion of relation-related failures increases from 59.8% in early sessions to 72.9% in later sessions. This shows that relation-related errors account for a larger share of the audited failures, but does not necessarily mean that their absolute frequency or per-task failure rate increases. The authors should report the number of tasks and failures, failure rates, and denominators for each period.
Questions for authors
See the weaknesses above.
6 comments
Limited external validity of the benchmark. The data are primarily generated with LLM assistance and validated through automated procedures, which may not fully capture the ambiguity, conflicting preferences, and complex state changes found in real-world user interactions.
We would like to point out that, in real interactions, changes in personal context can change how people carry out familiar tasks. Our benchmark trajectories capture this pattern naturally, and PraxisMem helps assistants adapt to such changes on tasks grounded in real conversations.
(1) Real interactions show the core pattern behind PraxisMemBench: changes in a user's personal context can change how a familiar task should be done. In a MemUse deployment log, a user explained across two sessions how their dog's loss of sight changed the way they reorganized their home: they moved items on shelves and in closets while keeping the furniture layout fixed [1]. PrISM shows a similar pattern: as users gained experience, some preferred reminders only when they missed a step [2, §7.4].
(2) PraxisMem improves performance on tasks grounded in real conversations. We evaluated 50 tasks: 42 based on real human–AI conversations from MemUse [1] and 8 based on real human–human conversations from RealTalk [3]. The logs show how users' preferences, circumstances, or constraints shaped what they did. We retained the dialogue history in chronological order, presented a follow-up request that required those memories, and defined action criteria based on the original dialogue. Each task averaged 28 sessions, 418 messages, about 20K history tokens, and 2.3 binary action criteria. Two raters scored the plans independently and resolved disagreements through discussion. A plan succeeds only if it passes all action criteria for its task. PraxisMem led on both metrics.
| Method | Action Criteria Passed | Plan Success |
|---|---|---|
| MemOS | 96/115 (83.5%) | 37/50 (74.0%) |
| Memory-R2 | 83/115 (72.2%) | 32/50 (64.0%) |
| PraxisMem | 104/115 (90.4%) | 41/50 (82.0%) |
(3) PraxisMemBench uses a human-led, LLM-assisted construction process. Engineers directly helped write, test, and refine the scripts so that the trajectories reflect the pattern seen in real interactions: changes in a user's preferences affect how later tasks should be handled.
(4) Our main benchmark is natural. Two experts independently rated the same 40 trajectory segments on a five-point scale. Mean ratings ranged from 4.31 to 4.64 across the three dimensions, with quadratic weighted kappa (QWK) of 0.695–0.751, indicating good agreement.
| Dimension | Segments | Expert 1 (/5) | Expert 2 (/5) | Mean (/5) | QWK |
|---|---|---|---|---|---|
| Naturalness of user requests | 40 | 4.43 | 4.20 | 4.31 | 0.751 |
| Plausibility of preferences and procedural constraints | 40 | 4.58 | 4.45 | 4.51 | 0.738 |
| Coherence of cross-session state changes | 40 | 4.75 | 4.53 | 4.64 | 0.695 |
(5) Lastly, we would also like to briefly defend our use of synthetic data. We note that several recent memory benchmarks published at leading conferences also use synthetic data in their construction, including LoCoMo (ACL) [4], LongMemEval (ICLR) [5], PersonaMem (COLM) [6], and HaluMem (EMNLP) [7]. This common choice comes from the nature of long-term memory itself. Real user histories are difficult to collect, and even when available, they are often long and messy, with no standard annotation of what should be remembered or which past details should guide a later action. Synthetic data lets us specify those dependencies and evaluate them consistently.
[1] Sumida et al. MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation. EMNLP 2026. Deployment log excerpt.
[2] Arakawa et al. Scaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initiative Dialogue. UIST 2025.
[3] Lee et al. RealTalk: A 21-Day Real-World Dataset for Long-Term Conversation. 2025.
[4] Maharana et al. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024.
[5] Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025.
[6] Jiang et al. Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. COLM 2025.
[7] Chen et al. HaluMem: Evaluating Hallucinations in Memory Systems of Agents. EMNLP 2026.
Limited statistical evidence for the ablation studies. The key ablations use only a small number of trajectories, with insufficient results across random seeds, confidence intervals, and comparisons under matched computational budgets.
Actually, the original ablation covered 200 samples: eight trajectories, each with 25 sequential sessions. These trajectories were designed to cover diverse user histories and task dependencies. Specifically, the eight trajectories covered all six joint-dependency subtypes, 17 of the 20 test application domains, and 61 of the 104 distinct SOPs used in the full test split. This already provides a useful estimate of performance on those trajectories. To address the reviewer’s request for broader coverage, we expanded the ablation to all 20 test trajectories, covering 500 sessions.
| Scope | Evaluated sessions | Test trajectories |
|---|---|---|
| Original (Table 3) | 200 | 8 |
| Expanded (full test split) | 500 | 20 |
| Variant | Combined Completion (%) ↑ | Interdependent Completion (%) ↑ | Avg SR (%) ↑ |
|---|---|---|---|
| Content-only GRPO | 64.1 | 55.8 | 24.4 |
| + Location-aware reward | 73.8 | 65.9 | 30.5 |
| + Criterion-routed OPSD | 77.4 | 71.9 | 33.4 |
| + Chunk-level routing (full PraxisMem) | 85.0 | 82.8 | 40.8 |
The ordering is unchanged on all three metrics. The gain in Interdependent Completion from content-only GRPO to full PraxisMem is nearly identical: +27.2 percentage points on the original eight trajectories and +27.0 points on the full test split.
Some evaluation details require clarification. The paper should clarify how the overall Session SR is aggregated, the denominators used to calculate failure-type proportions, and the comparability of results across different evaluation subsets. For example, the proportion of relation-related failures increases from 59.8% in early sessions to 72.9% in later sessions. This shows that relation-related errors account for a larger share of the audited failures, but does not necessarily mean that their absolute frequency or per-task failure rate increases. The authors should report the number of tasks and failures, failure rates, and denominators for each period.
Overall Session SR is defined in the appendix under “Session success and aggregation.” The right panel of Figure 6 shows the raw counts for all failures included in the analysis. Relation-related failures account for 101 of 169 failures (59.8%) in early sessions and 229 of 314 (72.9%) in late sessions. These percentages describe the composition of failures in each period.
We thank the reviewer for the detailed feedback and for recognizing the importance of studying how personal context shapes procedures, the systematic benchmark design, and the clear presentation. Below, we explain the rationale for our training method, present new evidence on trajectory naturalness and component gains, and clarify how the evaluation metrics are calculated.
Our main contribution, PraxisMemBench, introduces a novel setting for persistent assistance: changes in a user's personal context can change which procedures the assistant should follow across sessions. As the reviewer notes, this goes beyond recalling historical facts. As the user's needs change, the assistant must still be able to find the relevant personal context and procedures together.
First, consider what RL needs to teach the memory manager in this setting. There are four key challenges: remembering accurately, storing in the right place, assigning credit fairly, and knowing what to fix. DeltaMem, Memory-R2, and PraxisMem emphasize fundamentally different challenges.
(1) DeltaMem focuses on remembering accurately. Beyond this, PraxisMem also emphasizes storing information in the right place. Specifically, DeltaMem observes that different edits can produce semantically equivalent memory states, so it evaluates the resulting change in memory content. However, even correct information can be hard to use later if it is stored in the wrong place. For example, a memory manager may record a new travel restriction yet fail to retrieve it alongside the relevant booking procedure. Our location-aware reward therefore teaches the manager both what to remember and where to store it.
(2) Memory-R2 focuses on assigning credit fairly, while PraxisMem emphasizes knowing what to fix. Specifically, earlier edits change the starting conditions for later sessions, so Memory-R2's LoGo-GRPO resamples from a shared intermediate memory state to compare decisions under the same conditions. However, even under a fair comparison, every attempt can miss the same requirement, leaving no successful example to learn from. Our dynamic residual-routed OPSD provides specific guidance where the problem persists.
Second, we would like to clarify how our OPSD differs from prior work. A key question when applying OPSD is how privileged information (PI) is defined. In mathematical reasoning, Self-Distilled Reasoner gives the teacher a reference solution to guide the student's own solution. Our OPSD addresses memory maintenance across interactions. We found that what matters most is showing the teacher what needs fixing in the student's memory. We therefore define PI as the difference between the student's current memory state and the reference state. This difference shows what is missing, what needs revision, and where misplaced information belongs.
Finally, we would like to note that, to our knowledge, PraxisMem is the first to combine RL and OPSD to learn joint memory management. The two components complement each other: RL rewards content and placement across the full update trajectory, while OPSD provides local corrective guidance when all rollouts miss a requirement. The ablation results in Table 3 support this complementarity.
Overall, our algorithm advances three complementary capabilities of a memory manager: remembering accurately, storing information in the right place, and knowing what to fix.