acceptodds
Under review as a conference paper at ICLR 2027

Memorizing Who, Knowing How: Benchmarking Personal and Procedural Memory for Persistent Assistants

Abstract

Memory systems for persistent personal agents must move beyond isolated memories, maintaining both personal memory about whom they serve and procedural memory about how to act. Crucially, in harness-based personal assistance, procedures or skills are inherently user-conditioned: personal context determines which procedures apply, how they are instantiated, and when they should change. Existing benchmarks typically emphasize either personal or procedural memory, leaving their interaction underexplored. We introduce PraxisMemBench, a harness-based benchmark for persistent personal assistance across daily-life and work trajectories, where personal and procedural memory evolve together and later tasks depend on both. Experiments show that maintaining two memory types separately is insufficient, with completion dropping from single-type to interdependent dependencies (84.6%→77.9%). More broadly, across existing methods, relation-related failures increase from early to late sessions as interaction history grows (59.8%→72.9%), revealing a continual-maintenance challenge: writing correct memories is not enough if their evolving content and organization cannot be preserved for later retrieval. To address this, we propose PraxisMem, which uses privileged joint-state supervision to internalize the ability to maintain memory content and organization into the memory manager, with location-aware GRPO and dynamic residual-routed OPSD. PraxisMem reaches 40.78% overall Session Success Rate, compared with 20.60% for Claude Code’s native memory.

Primary area:
datasets and benchmarks
Keywords:
Persistent Personal Agents, Long-Term Memory, Memory Benchmarking, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tah4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PraxisMemBench, a benchmark for evaluating the joint maintenance of personal and procedural memories in long-term personal assistants and their impact on downstream tool use and task completion. The authors also propose PraxisMem, which combines location-aware GRPO to optimize memory content and organization with dynamic residual-routed OPSD to provide additional supervision for unresolved memory errors. Experiments show that PraxisMem achieves strong end-to-end task completion results on the proposed benchmark.

Critical strengths
  1. Meaningful research problem. The paper studies how personal context affects the applicability of procedural knowledge, going beyond conventional memory benchmarks that focus mainly on recalling historical facts.
  2. Systematic benchmark design. The benchmark covers multiple sessions, application domains, and task types.
  3. Clear presentation. The paper is well-written and easy to follow.
Critical weaknesses
  1. Limited external validity of the benchmark. The data are primarily generated with LLM assistance and validated through automated procedures, which may not fully capture the ambiguity, conflicting preferences, and complex state changes found in real-world user interactions.
  2. Insufficient algorithmic novelty. Location-aware GRPO and dynamic residual-routed OPSD build on existing reward-design methods for reinforcement learning and on-policy self-distillation, respectively. The proposed approach appears to combine existing techniques for a new task, but the paper does not sufficiently explain the underlying rationale for their applicability to the core problem.
  3. Limited statistical evidence for the ablation studies. The key ablations use only a small number of trajectories, with insufficient results across random seeds, confidence intervals, and comparisons under matched computational budgets.
  4. Some evaluation details require clarification. The paper should clarify how the overall Session SR is aggregated, the denominators used to calculate failure-type proportions, and the comparability of results across different evaluation subsets. For example, the proportion of relation-related failures increases from 59.8% in early sessions to 72.9% in later sessions. This shows that relation-related errors account for a larger share of the audited failures, but does not necessarily mean that their absolute frequency or per-task failure rate increases. The authors should report the number of tasks and failures, failure rates, and denominators for each period.
Questions for authors

See the weaknesses above.

6 comments

We thank the reviewer for the detailed feedback and for recognizing the importance of studying how personal context shapes procedures, the systematic benchmark design, and the clear presentation. Below, we explain the rationale for our training method, present new evidence on trajectory naturalness and component gains, and clarify how the evaluation metrics are calculated.

Insufficient algorithmic novelty. Location-aware GRPO and dynamic residual-routed OPSD build on existing reward-design methods for reinforcement learning and on-policy self-distillation, respectively. The proposed approach appears to combine existing techniques for a new task, but the paper does not sufficiently explain the underlying rationale for their applicability to the core problem.

Our main contribution, PraxisMemBench, introduces a novel setting for persistent assistance: changes in a user's personal context can change which procedures the assistant should follow across sessions. As the reviewer notes, this goes beyond recalling historical facts. As the user's needs change, the assistant must still be able to find the relevant personal context and procedures together.

First, consider what RL needs to teach the memory manager in this setting. There are four key challenges: remembering accurately, storing in the right place, assigning credit fairly, and knowing what to fix. DeltaMem, Memory-R2, and PraxisMem emphasize fundamentally different challenges.

(1) DeltaMem focuses on remembering accurately. Beyond this, PraxisMem also emphasizes storing information in the right place. Specifically, DeltaMem observes that different edits can produce semantically equivalent memory states, so it evaluates the resulting change in memory content. However, even correct information can be hard to use later if it is stored in the wrong place. For example, a memory manager may record a new travel restriction yet fail to retrieve it alongside the relevant booking procedure. Our location-aware reward therefore teaches the manager both what to remember and where to store it.

(2) Memory-R2 focuses on assigning credit fairly, while PraxisMem emphasizes knowing what to fix. Specifically, earlier edits change the starting conditions for later sessions, so Memory-R2's LoGo-GRPO resamples from a shared intermediate memory state to compare decisions under the same conditions. However, even under a fair comparison, every attempt can miss the same requirement, leaving no successful example to learn from. Our dynamic residual-routed OPSD provides specific guidance where the problem persists.

Second, we would like to clarify how our OPSD differs from prior work. A key question when applying OPSD is how privileged information (PI) is defined. In mathematical reasoning, Self-Distilled Reasoner gives the teacher a reference solution to guide the student's own solution. Our OPSD addresses memory maintenance across interactions. We found that what matters most is showing the teacher what needs fixing in the student's memory. We therefore define PI as the difference between the student's current memory state and the reference state. This difference shows what is missing, what needs revision, and where misplaced information belongs.

Finally, we would like to note that, to our knowledge, PraxisMem is the first to combine RL and OPSD to learn joint memory management. The two components complement each other: RL rewards content and placement across the full update trajectory, while OPSD provides local corrective guidance when all rollouts miss a requirement. The ablation results in Table 3 support this complementarity.

Overall, our algorithm advances three complementary capabilities of a memory manager: remembering accurately, storing information in the right place, and knowing what to fix.

Limited external validity of the benchmark. The data are primarily generated with LLM assistance and validated through automated procedures, which may not fully capture the ambiguity, conflicting preferences, and complex state changes found in real-world user interactions.

We would like to point out that, in real interactions, changes in personal context can change how people carry out familiar tasks. Our benchmark trajectories capture this pattern naturally, and PraxisMem helps assistants adapt to such changes on tasks grounded in real conversations.

(1) Real interactions show the core pattern behind PraxisMemBench: changes in a user's personal context can change how a familiar task should be done. In a MemUse deployment log, a user explained across two sessions how their dog's loss of sight changed the way they reorganized their home: they moved items on shelves and in closets while keeping the furniture layout fixed [1]. PrISM shows a similar pattern: as users gained experience, some preferred reminders only when they missed a step [2, §7.4].

(2) PraxisMem improves performance on tasks grounded in real conversations. We evaluated 50 tasks: 42 based on real human–AI conversations from MemUse [1] and 8 based on real human–human conversations from RealTalk [3]. The logs show how users' preferences, circumstances, or constraints shaped what they did. We retained the dialogue history in chronological order, presented a follow-up request that required those memories, and defined action criteria based on the original dialogue. Each task averaged 28 sessions, 418 messages, about 20K history tokens, and 2.3 binary action criteria. Two raters scored the plans independently and resolved disagreements through discussion. A plan succeeds only if it passes all action criteria for its task. PraxisMem led on both metrics.

MethodAction Criteria PassedPlan Success
MemOS96/115 (83.5%)37/50 (74.0%)
Memory-R283/115 (72.2%)32/50 (64.0%)
PraxisMem104/115 (90.4%)41/50 (82.0%)

(3) PraxisMemBench uses a human-led, LLM-assisted construction process. Engineers directly helped write, test, and refine the scripts so that the trajectories reflect the pattern seen in real interactions: changes in a user's preferences affect how later tasks should be handled.

(4) Our main benchmark is natural. Two experts independently rated the same 40 trajectory segments on a five-point scale. Mean ratings ranged from 4.31 to 4.64 across the three dimensions, with quadratic weighted kappa (QWK) of 0.695–0.751, indicating good agreement.

DimensionSegmentsExpert 1 (/5)Expert 2 (/5)Mean (/5)QWK
Naturalness of user requests404.434.204.310.751
Plausibility of preferences and procedural constraints404.584.454.510.738
Coherence of cross-session state changes404.754.534.640.695

(5) Lastly, we would also like to briefly defend our use of synthetic data. We note that several recent memory benchmarks published at leading conferences also use synthetic data in their construction, including LoCoMo (ACL) [4], LongMemEval (ICLR) [5], PersonaMem (COLM) [6], and HaluMem (EMNLP) [7]. This common choice comes from the nature of long-term memory itself. Real user histories are difficult to collect, and even when available, they are often long and messy, with no standard annotation of what should be remembered or which past details should guide a later action. Synthetic data lets us specify those dependencies and evaluate them consistently.

[1] Sumida et al. MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation. EMNLP 2026. Deployment log excerpt.

[2] Arakawa et al. Scaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initiative Dialogue. UIST 2025.

[3] Lee et al. RealTalk: A 21-Day Real-World Dataset for Long-Term Conversation. 2025.

[4] Maharana et al. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024.

[5] Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025.

[6] Jiang et al. Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. COLM 2025.

[7] Chen et al. HaluMem: Evaluating Hallucinations in Memory Systems of Agents. EMNLP 2026.

Limited statistical evidence for the ablation studies. The key ablations use only a small number of trajectories, with insufficient results across random seeds, confidence intervals, and comparisons under matched computational budgets.

Actually, the original ablation covered 200 samples: eight trajectories, each with 25 sequential sessions. These trajectories were designed to cover diverse user histories and task dependencies. Specifically, the eight trajectories covered all six joint-dependency subtypes, 17 of the 20 test application domains, and 61 of the 104 distinct SOPs used in the full test split. This already provides a useful estimate of performance on those trajectories. To address the reviewer’s request for broader coverage, we expanded the ablation to all 20 test trajectories, covering 500 sessions.

ScopeEvaluated sessionsTest trajectories
Original (Table 3)2008
Expanded (full test split)50020
VariantCombined Completion (%) ↑Interdependent Completion (%) ↑Avg SR (%) ↑
Content-only GRPO64.155.824.4
+ Location-aware reward73.865.930.5
+ Criterion-routed OPSD77.471.933.4
+ Chunk-level routing (full PraxisMem)85.082.840.8

The ordering is unchanged on all three metrics. The gain in Interdependent Completion from content-only GRPO to full PraxisMem is nearly identical: +27.2 percentage points on the original eight trajectories and +27.0 points on the full test split.

Some evaluation details require clarification. The paper should clarify how the overall Session SR is aggregated, the denominators used to calculate failure-type proportions, and the comparability of results across different evaluation subsets. For example, the proportion of relation-related failures increases from 59.8% in early sessions to 72.9% in later sessions. This shows that relation-related errors account for a larger share of the audited failures, but does not necessarily mean that their absolute frequency or per-task failure rate increases. The authors should report the number of tasks and failures, failure rates, and denominators for each period.

Overall Session SR is defined in the appendix under “Session success and aggregation.” The right panel of Figure 6 shows the raw counts for all failures included in the analysis. Relation-related failures account for 101 of 169 failures (59.8%) in early sessions and 229 of 314 (72.9%) in late sessions. These percentages describe the composition of failures in each period.

Reviewer_ySbtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PraxisMemBench, a benchmark for evaluating the joint maintenance of personal and procedural memory in persistent assistants, and proposes PraxisMem, a learned memory manager trained with location-aware GRPO and dynamic residual-routed self-distillation. On overall Session Success Rate, PraxisMem achieves 40.78%, compared with 20.60% for Claude Code’s native memory.

Critical strengths
  1. PraxisMemBench evaluates personal and procedural memory together across persistent, multi-domain trajectories. Its held-out domain split and evolving user-state design make it possible to examine failures that isolated memory-retrieval or single-task benchmarks may overlook.
  2. PraxisMem goes beyond rewarding memory content alone by explicitly accounting for where information is stored and which unresolved defects persist across updates. The ablation results provide initial evidence that these components contribute to improved memory organization and downstream task completion.
Critical weaknesses
  1. The paper distinguishes memory-conditioned Session SR from Strict E2E Session SR, but the headline result focuses on the former. Although this metric captures whether memory-dependent criteria are satisfied, it does not by itself establish successful end-to-end task execution. The practical significance of the reported gains is therefore not fully reflected by this result alone.
  2. Table 3 evaluates the training variants on only eight fixed test trajectories. The observed gains are encouraging, but such a small subset provides limited evidence that the improvements are robust across different user histories and task dependencies. The contribution of the individual components is therefore not yet fully established.
Questions for authors
  1. A clearer separation between memory-side metrics and downstream execution failures would help establish how much of the overall improvement comes directly from better memory management.
  2. The conclusions drawn from the eight-trajectory ablation would be more convincing with results across additional trajectories and multiple training seeds.
  3. A more explicit account of the methodological differences from DeltaMem, Memory-R2, and prior on-policy self-distillation work would clarify which part of the proposed training procedure provides the main advance [1–3]. [1] Zhang et al. DeltaMem: Towards Agentic Memory Management via Reinforcement Learning. 2026. [2] Yan et al. Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents. 2026. [3] Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.
4 comments

We thank the reviewer for the thoughtful feedback and for recognizing the value of evaluating personal and procedural memory together across held-out domains, as well as the evidence from our ablations. Below, we clarify how PraxisMem differs from prior methods and report expanded ablations and strict end-to-end results.

A more explicit account of the methodological differences from DeltaMem, Memory-R2, and prior on-policy self-distillation work would clarify which part of the proposed training procedure provides the main advance [1–3]. [1] Zhang et al. DeltaMem: Towards Agentic Memory Management via Reinforcement Learning. 2026. [2] Yan et al. Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agents. 2026. [3] Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.

First, we would like to note that PraxisMem takes a fundamentally different perspective on memory RL from DeltaMem and Memory-R2. Training a good memory manager involves four key challenges: remembering accurately, storing in the right place, assigning credit fairly, and knowing what to fix.

(1) DeltaMem focuses on remembering accurately. Beyond this, PraxisMem also emphasizes storing information in the right place.. Specifically, DeltaMem observes that different edits can produce semantically equivalent memory states, so it evaluates the resulting change in memory content. However, even correct information can be hard to use later if it is stored in the wrong place. For example, a memory manager may record a new travel restriction yet fail to retrieve it alongside the relevant booking procedure. Our location-aware reward therefore teaches the manager both what to remember and where to store it.

(2) Memory-R2 focuses on assigning credit fairly, while PraxisMem emphasizes knowing what to fix. Specifically, earlier edits change the starting conditions for later sessions, so Memory-R2's LoGo-GRPO resamples from a shared intermediate memory state to compare decisions under the same conditions. However, even under a fair comparison, every attempt can miss the same requirement, leaving no successful example to learn from. Our dynamic residual-routed OPSD provides specific guidance where the problem persists.

Second, we would like to clarify how our OPSD differs from prior work. A key question when applying OPSD is how privileged information (PI) is defined. In mathematical reasoning, Self-Distilled Reasoner gives the teacher a reference solution to guide the student's own solution. Our OPSD addresses memory maintenance across interactions. We found that what matters most is showing the teacher what needs fixing in the student's memory. We therefore define PI as the difference between the student's current memory state and the reference state. This difference shows what is missing, what needs revision, and where misplaced information belongs.

Finally, we would like to note that, to our knowledge, PraxisMem is the first to combine RL and OPSD to learn joint memory management. The two components complement each other: RL rewards content and placement across the full update trajectory, while OPSD provides local corrective guidance when all rollouts miss a requirement. The ablation results in Table 3 support this complementarity.

Overall, our algorithm advances three complementary capabilities of a memory manager: remembering accurately, storing information in the right place, and knowing what to fix.

Table 3 evaluates the training variants on only eight fixed test trajectories. The observed gains are encouraging, but such a small subset provides limited evidence that the improvements are robust across different user histories and task dependencies. The contribution of the individual components is therefore not yet fully established.

The conclusions drawn from the eight-trajectory ablation would be more convincing with results across additional trajectories and multiple training seeds.

Actually, the original ablation covered 200 samples: eight trajectories, each with 25 sequential sessions. These trajectories were designed to cover diverse user histories and task dependencies. Specifically, the eight trajectories covered all six joint-dependency subtypes, 17 of the 20 test application domains, and 61 of the 104 distinct SOPs used in the full test split. This already provides a useful estimate of performance on those trajectories. To address the reviewer’s request for broader coverage, we expanded the ablation to all 20 test trajectories, covering 500 sessions.

ScopeEvaluated sessionsTest trajectories
Original (Table 3)2008
Expanded (full test split)50020
VariantCombined Completion (%) ↑Interdependent Completion (%) ↑Avg SR (%) ↑
Content-only GRPO64.155.824.4
+ Location-aware reward73.865.930.5
+ Criterion-routed OPSD77.471.933.4
+ Chunk-level routing (full PraxisMem)85.082.840.8

The ordering is unchanged on all three metrics. The gain in Interdependent Completion from content-only GRPO to full PraxisMem is nearly identical: +27.2 percentage points on the original eight trajectories and +27.0 points on the full test split.

The paper distinguishes memory-conditioned Session SR from Strict E2E Session SR, but the headline result focuses on the former. Although this metric captures whether memory-dependent criteria are satisfied, it does not by itself establish successful end-to-end task execution. The practical significance of the reported gains is therefore not fully reflected by this result alone.

A clearer separation between memory-side metrics and downstream execution failures would help establish how much of the overall improvement comes directly from better memory management.

Crucially, Strict E2E Session SR depends on extra factors unrelated to memory, including task LLM’s ability to execute the task. Therefore, Memory-conditioned Session SR more directly measures whether the agent satisfies requirements that depend on memory. Two PraxisMem examples illustrate the difference:

  • Example 1 (Tool: reordering a past purchase). The agent followed the remembered procedure by comparing offers before adding the item to the cart. It placed the order with the correct item and expedited shipping, but skipped the required get_product call. It passed five of six tool steps and failed strict E2E.
  • Example 2 (Artifact: creating a follow-up task). The agent used the remembered title prefix, reflected the user’s preference, and created a high-priority task. The task instructions required status open, but the agent set it to pending. It passed five of six artifact checks and failed strict E2E.

In both cases, the agent had the correct memories, applied them correctly, and passed the memory checks. The strict failures came from additional execution requirements that the task LLM missed. This is why we use memory-conditioned SR as our primary metric. To address the reviewer’s question about complete task success, we also report strict E2E SR for three representative methods on the full test split. PraxisMem leads on both metrics for Tool and Artifact tasks.

MethodTool Memory SR (%)Tool Strict SR (%)Artifact Memory SR (%)Artifact Strict SR (%)
MemOS19.1215.6918.1816.88
Hermes26.3422.4423.2319.35
PraxisMem37.8830.3031.3328.00

Dear reviewer,

As we move closer to the end of the discussion panel, we haven't heard back from the reviewer. Please let us know if you have any further questions regarding our rebuttal.

Sincerely,

Authors

Dear reviewer,

As the discussion period is nearing its end, we would appreciate hearing whether our responses and the revised manuscript address your concerns. We would be happy to clarify any remaining questions.

Sincerely,

Authors

Reviewer_g4PgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PraxisMemBench, a benchmark where a persistent assistant serves the same user over many sessions, and later tool, QA and artifact tasks depend on both personal memory (who the user is) and procedural memory (how to do things for this user). The authors show that existing memory systems struggle as these two kinds of memory become coupled over time. They then propose PraxisMem, a memory manager trained with privileged reference memory states, location-aware GRPO and residual-routed self-distillation, and report clear gains over native and advanced memory systems.

Critical strengths

The coupling between user context and procedures is a reasonable and underexplored angle for agent memory. The benchmark is built with care, with executable mock applications, blind validation by independent testers, and a human audit of the LLM judge.

Critical weaknesses

PraxisMem is trained on its own benchmark. The training split comes from the same persona source, construction pipeline and annotation process as the test split, and the supervision is written in the authors' own schema. The training domains are disjoint from the test domains, but the task format is the same, so a stronger result is expected.

The baselines are not set up fairly. Most baselines are training-free, and it is not clear which data Memory-R2 is retrained on. I suggest training the learnable baselines on the same split with a comparable budget, adding a simple SFT baseline on the reference states, and testing PraxisMem on an existing memory benchmark.

The scenario is designed by the authors. The personas are fictional, the life events are synthesized to matter for the next task, and users state their procedural rules explicitly. The motivation is supported only by citations, and I could not find real user logs or a user study showing that this kind of coupled memory appears in real usage.

The user simulator is also built by the authors. It follows a script, and the same model serves as the judge. The human validation checks that tasks are executable and that the judge is reliable, not that the simulated users resemble real ones. I would like to see a comparison with real interactions, or at least a human study on how natural the trajectories are.

Questions for authors

Please see weaknesses.

4 comments

We thank the reviewer for the constructive feedback and for recognizing the underexplored link between user context and procedures, as well as the care taken in building and validating PraxisMemBench. Below, we address the concerns about training data, baseline fairness, and the realism of the benchmark trajectories.

PraxisMem is trained on its own benchmark. The training split comes from the same persona source, construction pipeline and annotation process as the test split, and the supervision is written in the authors' own schema. The training domains are disjoint from the test domains, but the task format is the same, so a stronger result is expected.

The baselines are not set up fairly. Most baselines are training-free, and it is not clear which data Memory-R2 is retrained on. I suggest training the learnable baselines on the same split with a comparable budget, adding a simple SFT baseline on the reference states, and testing PraxisMem on an existing memory benchmark.

Comparison is fair. Memory-R2 is retrained on the same PraxisMemBench training split used by PraxisMem. Specifically, we used all 20 training trajectories and converted their visible interactions and retrieval annotations into memory QA examples. We traced each answer to evidence available at query time. We used an 8→16→25-session curriculum and processed long sessions in 8K-token windows, carrying memory across windows while keeping local rerollouts and credit assignment at the session level. We retained Memory-R2’s shared extractor–manager, LoGo-GRPO, and downstream QA F1 reward with a compression penalty.

To test whether PraxisMem’s gains hold under shared training supervision, we trained a reference-state SFT baseline with the same base model, training split, and reference memory states. For Update, we generated and executed tool calls chunk by chunk, checked that each edit used only evidence available at that point, and verified the final store against the session-end reference state. For Retrieval, we executed list, grep, and read calls and verified that the answers were supported by the retrieved evidence and covered the required information.

MethodTool SR (%)Memory QA SR (%)Artifact SR (%)
Reference-state SFT17.6820.7114.00
PraxisMem37.8855.0031.33

PraxisMem improves across all three task types. SFT learns from memory states produced by demonstrations, but at deployment it must continue from its own edits. This mismatch leaves it less prepared to recover from omissions or misplaced entries, which helps explain its lower scores. In contrast, PraxisMem trains on the memory states it produces and learns how to revise them.

The scenario is designed by the authors. The personas are fictional, the life events are synthesized to matter for the next task, and users state their procedural rules explicitly. The motivation is supported only by citations, and I could not find real user logs or a user study showing that this kind of coupled memory appears in real usage.

Real user studies have documented that changes in a user’s personal context can change how an assistant should carry out a familiar task.

  • In the PrISM [1] user study (§7.4), three participants who initially wanted reminders before a step switched to wanting reminders only if they skipped it, as they became more familiar with the same task.
  • In a MemUse [2] deployment log, a user explained across two sessions how their dog’s loss of sight changed the way they reorganized their home: they moved items within shelves and closets while keeping the furniture layout fixed.

These examples illustrate the behavior PraxisMemBench targets: a change in personal circumstances can change which steps are appropriate in a recurring task.

[1] Arakawa et al. Scaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initiative Dialogue. UIST 2025.

[2] Sumida et al. MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation. EMNLP 2026. Deployment log excerpt.

The user simulator is also built by the authors. It follows a script, and the same model serves as the judge. The human validation checks that tasks are executable and that the judge is reliable, not that the simulated users resemble real ones. I would like to see a comparison with real interactions, or at least a human study on how natural the trajectories are.

We would like to point out that our benchmark trajectories naturally capture the behavior described above, and that PraxisMem improves performance on tasks grounded in real conversations.

(1) PraxisMem improves performance on tasks grounded in real conversations. We evaluated 50 tasks: 42 based on real human–AI conversations from MemUse [2] and 8 based on real human–human conversations from RealTalk [3]. The logs show how users' preferences, circumstances, or constraints shaped what they did. We retained the dialogue history in chronological order, presented a follow-up request that required those memories, and defined action criteria based on the original dialogue. Each task averaged 28 sessions, 418 messages, about 20K history tokens, and 2.3 binary action criteria. Two raters scored the plans independently and resolved disagreements through discussion. A plan succeeds only if it passes all action criteria for its task. PraxisMem led on both metrics.

MethodAction Criteria PassedPlan Success
MemOS96/115 (83.5%)37/50 (74.0%)
Memory-R283/115 (72.2%)32/50 (64.0%)
PraxisMem104/115 (90.4%)41/50 (82.0%)

(2) PraxisMemBench uses a human-led, LLM-assisted construction process. Engineers directly helped write, test, and refine the scripts to capture the pattern observed in the real interactions above: changes in a user's personal context affect how later tasks should be handled. The user simulator follows these scripts, sending fixed messages as written and generating replies when the wording is left open.

(3) Our main benchmark is natural. Two experts independently rated the same 40 trajectory segments on a five-point scale. Mean scores ranged from 4.31 to 4.64 across request naturalness, the plausibility of preferences and procedural constraints, and the coherence of cross-session state changes. Quadratic weighted kappa (QWK) ranged from 0.695 to 0.751, indicating good agreement.

DimensionSegmentsExpert 1 (/5)Expert 2 (/5)Mean (/5)QWK
Naturalness of user requests404.434.204.310.751
Plausibility of preferences and procedural constraints404.584.454.510.738
Coherence of cross-session state changes404.754.534.640.695

(4) Lastly, we would also like to briefly defend our use of synthetic data. We note that several recent memory benchmarks published at leading conferences also use synthetic data in their construction, including LoCoMo (ACL) [4], LongMemEval (ICLR) [5], PersonaMem (COLM) [6], and HaluMem (EMNLP) [7]. This common choice comes from the nature of long-term memory itself. Real user histories are difficult to collect, and even when available, they are often long and messy, with no standard annotation of what should be remembered or which past details should guide a later action. Synthetic data lets us specify those dependencies and evaluate them consistently.

[3] Lee et al. RealTalk: A 21-Day Real-World Dataset for Long-Term Conversation. 2025.

[4] Maharana et al. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024.

[5] Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025.

[6] Jiang et al. Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. COLM 2025.

[7] Chen et al. HaluMem: Evaluating Hallucinations in Memory Systems of Agents. EMNLP 2026.

Dear reviewer,

As we move closer to the end of the discussion panel, we haven't heard back from the reviewer. Please let us know if you have any further questions regarding our rebuttal.

Sincerely,

Authors

Dear reviewer,

As the discussion period is nearing its end, we would appreciate hearing whether our responses and the revised manuscript address your concerns. We would be happy to clarify any remaining questions.

Sincerely,

Authors

Comments on the paper

1 comment

We sincerely thank all reviewers for their constructive feedback. We are pleased that they recognize the underexplored link between personal context and procedures (R1, R3), the careful benchmark construction and validation (R1), the multi-domain evaluation with evolving user states (R2, R3), and the evidence from our ablations (R2).

PraxisMemBench studies how changes in personal context should change an assistant’s actions, a need supported by real user interactions. To our knowledge, PraxisMem is the first to combine RL and OPSD to learn joint memory management, helping the manager remember accurately, store information in the right place, and know what to fix. 

We addressed the concerns in our individual responses and uploaded a revised manuscript, with the main changes marked in blue:

  • [R2, R3] Expanded ablation (Table 3): Replaced the eight-trajectory results with results on all 20 test trajectories and 500 sessions. The ordering of the variants remains consistent across all three metrics.
  • [R1] Training baselines (Appendix E.3 and Table 13): Clarified how Memory-R2 was retrained on the same PraxisMemBench training split and added a reference-state SFT baseline using the same base model, training split, and reference states.
  • [R1, R3] Trajectory naturalness (Table 14): Added independent ratings from two experts on 40 trajectory segments, with mean scores of 4.31–4.64 out of 5.
  • [R2] Strict end-to-end success (Appendix D and Table 15): Reported both success metrics for three representative methods. PraxisMem leads on both metrics for Tool and Artifact tasks. Added examples separating successful memory use from downstream execution failures.
  • [R1, R3] Real-dialogue evaluation (Appendix F.4 and Table 16): Added real-world examples of personal–procedural coupling and evaluated 50 action-planning tasks grounded in MemUse and RealTalk conversations.

We hope these additions make the contributions, evaluation, and practical relevance of our work clearer.

(* As abbreviations, we refer to reviewers g4Pg as R1, ySbt as R2, and tah4 as R3, respectively.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.