acceptodds
Under review as a conference paper at ICLR 2027

From Memory to Action: Benchmarking Long-Term Agent Memory in Personal Assistant Environments

Abstract

Always-on personal assistants are an emerging application of agents powered by large language models (LLMs). Unlike conventional assistants, they operate over long-term, complex streams of user activities and agent interactions, accumulating information that is essential for tasks with ambiguous instructions, evolving user preferences, prior feedback, and reusable experience. This places substantial demands on agent memory, not only to form and maintain useful memories, but also to correctly apply them during downstream task execution. Existing benchmarks, however, either evaluate memory largely in isolation from end-task success or rely on simplified interaction histories that do not reflect the complexity of sustained daily use. As a result, there remains a gap in evaluating agent memory end to end, from memory formation to its eventual effect on task completion. To address this gap, we introduce a comprehensive memory benchmark comprising 100 tasks with fine-grained target-memory attribution, enabling separate diagnosis of Formation, Availability, Application, and Outcome. Evaluations across different LLMs show that the largest performance loss occurs after relevant memory becomes available but before it is translated into the intended action, indicating that memory utilization, rather than storage or retrieval alone, is a major bottleneck. Controlled memory-swap experiments further reveal that the utility of a generated memory depends strongly on the downstream executor that uses it. We hope these insights can guide future advances in agent memory for practical applications.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Agent Memory, Long-Term Memory, Personal Assistants, Tool Use, Agent Evaluation, Long-Horizon Task Execution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ct9zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a benchmark for evaluating whether personal assistants can retain information from long-term activity histories and use it in subsequent tasks across applications. The benchmark contains 100 tasks and 143 task–target-memory pairs, with annotations connecting historical evidence to specific decisions and scoring criteria. Evaluation separates Formation, Availability, Application, and final task outcomes. The experiments show that persistent memory improves task performance, while substantial losses remain after relevant information becomes available. Component-swap experiments and supervised fine-tuning further examine the contributions of memory generation and execution. The main contribution is the connection between memory evaluation and executable behavior. The main unresolved issue is whether the observed Application failures specifically reflect memory utilization difficulties or broader limitations in task execution.

Critical strengths
  1. The evaluation examines concrete actions and their parameters rather than relying only on factual recall. With 86 tasks requiring specified tool operations, the benchmark can expose cases where an assistant recalls the relevant information but fails to reflect it in the resulting action.
  2. The consolidation protocol withholds future task requests. Agents process history chronologically across separate sessions, with only persistent memory carried forward. This makes memory formation less dependent on knowing which information a later evaluation will require. (§3.2; Appendix A)
  3. The annotations connect historical evidence, target memories, task requirements, and scoring checks. These connections provide a useful basis for diagnosing where information is lost between historical observation and downstream behavior. (§3.3)
  4. The evaluation distinguishes satisfying a memory-dependent requirement from completing the entire task. Table 3 explicitly identifies cases where the relevant requirement is satisfied but other requirements fail, avoiding the assumption that every unsuccessful task reflects a memory failure.
  5. The component-swap experiments provide more insight than model rankings alone. Tables 4 and 5 show that the downstream value of a memory source depends on the executor, making the compatibility between memory representations and execution behavior a worthwhile subject for further investigation.
Critical weaknesses
  1. The Application category includes failures that may arise from general execution limitations. In Appendix D, an agent receives the relevant order quantity but stops at recommendations and confirmation instead of placing the order. This establishes incomplete execution, but does not establish a failure specific to memory utilization. Similar behavior could occur if the quantity were stated directly in the current request. The evidence supports the observation that information availability is insufficient for successful action, but the stronger attribution to memory utilization remains underdetermined.
  2. The operational details behind the lifecycle labels are insufficiently specified. Formation requires a semantic judgment about retained information, Availability requires determining what actually reached the execution context, and Application requires identifying satisfaction of a local requirement. The manuscript does not adequately explain how these judgments are made or how partial, outdated, and contradictory memories are handled. Task Score aggregation and the conditions for a complete task pass also remain insufficiently defined.
  3. The conclusion that utilization is a larger bottleneck than storage or retrieval depends on the memory interface used in these experiments. Relevant information can enter execution through system-prompt memory blocks, but the manuscript does not report memory lengths, injection coverage, truncation rules, or the proportion requiring active lookup. Without these details, high Availability may partly reflect the scaffold’s delivery mechanism. The limitation is acknowledged in Appendix F, but should also constrain the interpretation of the main findings.
  4. The construction procedure explicitly selects tasks that depend on memory. Consequently, the advantage of providing memory is partly expected from the benchmark design. A more informative reference would be the performance of each evaluated model when supplied with correct target memories. Construction-time feasibility checks establish that a solution exists, but do not establish how much execution difficulty remains for the evaluated models. The scope and findings of human review are also not sufficiently documented.
  5. The fine-tuning results show substantial interaction between memory and executor changes. In Table 5, replacing only the executor increases Pass Rate from 44% to 46%, replacing only the memory decreases it to 43%, and replacing both increases it to 55%. The Task Score difference-in-differences is 0.0506, compared with an overall improvement of 0.0973. Memory–executor compatibility is therefore central to the result and deserves a fuller explanation alongside the emphasis on executor improvement. The joint and execution-only training comparison also changes both supervision and optimization duration.
  6. The benchmark’s temporal span is clearer than its coverage of specific long-term memory difficulties. There are 143 target-memory observations across 100 tasks, averaging 1.43 per task, and some tasks share histories. Although the construction defines topology, trace, and cue dimensions, their distributions and associated results are not reported. It is therefore difficult to determine whether performance is limited by combining memories, resolving updates, inferring preferences across records, or executing actions based on relatively explicit information. (§4.1–4.3)
Questions for authors
  1. How does the Application definition distinguish memory utilization failures from general instruction-following or execution failures? For cases where an agent receives the relevant memory but stops at a recommendation or confirmation, would the same failure occur if the constraint were stated directly in the current request? What evidence supports attributing these omissions specifically to memory utilization?
  2. How are Formation, Availability, and Application labels assigned in practice? In particular, how are memories treated when they preserve the correct preference but omit its conditions, or retain both an outdated and a current version? Please also specify how task-level scoring checks are weighted and aggregated into Task Score, what determines a complete task pass, and whether any judgments depend on an LLM evaluator.
  3. To what extent does the Availability-to-Application gap depend on the current memory interface? What proportion of relevant information is injected directly into the system prompt, and what proportion requires active lookup? What are the memory-size limits and truncation rules? These details are needed to determine whether the reported bottleneck ordering is specific to this OpenHarness configuration.
  4. How should the interaction in Table 5 be incorporated into the interpretation of fine-tuning? Replacing the executor alone produces a relatively small Pass Rate improvement, whereas replacing both components produces a much larger gain. What changes in memory content, representation, or execution behavior explain this dependence? How do the authors distinguish improved executor capability from improved compatibility with the SFT-generated memories?
  5. What specific effects of consolidation supervision can be established from the joint versus execution-only comparison? Since the two settings differ in both supervision and optimization duration, the final Memory Gain difference does not isolate the contribution of consolidation training. Is there behavioral evidence that this supervision improves information selection, updating, or subsequent use beyond changes in general task execution?
  6. How well does each evaluated model perform when given the correct target memories directly? The construction-time feasibility probe demonstrates the existence of an executable solution, but does not establish the evaluated models’ performance under that condition. An oracle-memory reference would help determine how much of the remaining loss reflects memory representation and use, and how much reflects planning or tool-execution limitations.
  7. Why does the apparently corresponding Opus configuration have different results in Tables 1 and 4? Table 1 reports 58% Pass Rate and 0.7110 Task Score, whereas Opus-generated Memory with the Opus executor in Table 4 yields 59% and 0.7115. Were the same memory artifacts and evaluation conditions used? If these entries represent different runs or configurations, that distinction should be stated explicitly.
  8. How do the reported training sample counts, epochs, batch size, and optimization steps reconcile? Appendix E reports 492 joint-training samples and 213 execution-only samples, with 10 epochs and a global batch size of 16, but only 136 and 41 optimization steps. Assuming all listed samples are used without packing or filtering, the nominal totals would be approximately 308 and 133 steps, respectively. Since Table E1 specifies deletion of overlength samples, please report the actual post-filtering counts and any packing or step-limit settings that explain the difference.
  9. How were the 279 consolidation phases selected from the 98 complete consolidation-to-execution trajectories described in Appendix E.1? Appendix A specifies five consolidation phases per complete workflow, which would yield 490 phases if every phase were retained. Were phases excluded because of length, quality, duplication, or another criterion? Please clarify the relationship between the complete source trajectories and the final training samples.
3 comments
AuthorsOpenReview ↗

Response to ct9z (1/3): Application and Evaluation (Q1-Q2)

Q1. Application failures and current-request constraints

Stating the constraint directly in the current request can resolve some omissions, but failures to act still occur under explicit instructions. We conducted a paired comparison on 20 previously omitted target-memory observations from Qwen3.8-27B Base. We kept the model-generated memory unchanged and appended only the constraints relevant to the current task to the user request. We compared this condition against a contemporaneous replay of the original condition.

Review of the remaining cases distinguished three situations: three still did not send a message despite an explicit instruction to send it; five involved ambiguity between the request wording and the action required by the evaluation, so they do not establish unnecessary confirmation; and one fulfilled the target-memory requirement but failed a separate recipient requirement. These cases show that placing constraints in the request can help, while action-execution difficulties and task-specification issues remain.

Our Application diagnosis records whether the requirement associated with a target memory was fulfilled and separately identifies cases where that requirement was satisfied but other task requirements were not. The full-benchmark Task-Relevant Gold Memory condition appends the correct task-relevant memories directly to the current request. Appendix D.1–D.2, “Gold-Memory Reference Experiments,” reports its setup and results for all four models; Appendix D.3 reports the constraint comparison.

Q2. Lifecycle labels and task grading

Thank you for noting that the evaluation procedure needs more detail. We have clarified four aspects in Section 3.3 and the new Appendix sections A.3–A.4.

  1. Label assignment. Formation compares the target information with the model's saved memory, checks semantic retention, and records a supporting quotation. Availability checks memory injected into the execution prompt or returned by a memory lookup; reported Available counts include only targets that also satisfy Formation. Application checks whether answers or tool calls realize the corresponding requirement. For example, when a task requires ordering 120 units, mentioning the quantity is insufficient: the order must be placed with the correct quantity.
  2. Missing conditions and coexisting versions. Application assesses task-specific scope and exceptions. For example, a request to a contact for an introduction to a professor must use a dedicated introduction address rather than the ordinary correspondence address. Old and current rules may coexist if memory clearly identifies the current rule: an actual memory stating that a new folder supersedes the old folder was judged correctly retained. However, the target descriptions used by Formation and Availability do not enumerate every condition in the original history. These labels therefore do not guarantee detection of every case in which the value is retained but a condition is omitted; Appendix A.3 now states this limitation.
  3. LLM involvement. Lifecycle labels use Qwen3.8 for semantic judgments with recorded evidence quotations. Task grading combines deterministic checks with Sonnet 5: rules check tool calls and state effects, some answers are extracted by an LLM before matching, and subjective items are judged against reference answers and rubrics.
  4. Task Score and passing. Nonnegative item weights are normalized, and their weighted item scores yield completion. Task Score is completion multiplied by a safety gate, which sets the score to zero for a safety violation. Pass requires Task Score ≥ 0.70, so passing permits unmet requirements. Table 3 instead distinguishes whether the target and remaining requirements are satisfied under semantic assessment. We have renamed its first column “All requirements satisfied” and aligned the surrounding text and Appendix E to distinguish this category from formal Pass; all counts and percentages are unchanged.
AuthorsOpenReview ↗

Response to ct9z (2/3): Memory Interface and Training (Q3-Q5)

Q3. Memory injection and truncation

High Availability reflects automatic memory injection. Before execution, OpenHarness places MEMORY.md and automatically selected relevant files in the system prompt. The framework selects these files; the agent need not call a read tool.

All 485 Available observations across the four models have supporting evidence in the initial system prompt (100%). None requires a later lookup for the target information to first become available. This percentage is conditional on Availability, not all target memories or historical information. Active reads still occur, but these targets were visible at the start.

MEMORY.md is limited first to 200 lines, then to 25,000 UTF-8 bytes, backing up to a newline where possible. A truncation warning is appended outside the content byte limit. Up to five relevant files contribute their first 8,000 characters each, without an equivalent warning. These limits constrain injection, not persistent-memory storage; omitted content remains accessible through tools.

Of the 485 Available observations, 344 have correct Application and 141 do not. This shows a gap between availability and correct use under automatic injection. It does not establish that active retrieval is generally easier than Application, because the framework already delivers the relevant information. An interface relying on agent-initiated search and reading could have greater lookup losses and a different bottleneck ordering. Section 5.3.1 now states this qualification and the initial-injection count; Appendix A.2 gives the injection rules.

Q4. SFT interaction and compatibility

We agree that the interaction in Table 5 deserves a more explicit role in interpreting fine-tuning.

With Qwen-generated Memory fixed, replacing the Base executor with the Memory-SFT executor improves Pass Rate by 2 percentage points and Task Score by 0.0443. With SFT-generated Memory, the executor's Pass Rate gain rises to 12 percentage points. The interaction is 10 percentage points in Pass Rate and 0.0506 in Task Score, computed from the displayed values; the latter is substantial relative to the overall Task Score gain of 0.0973. These results show execution gains under fixed memory and larger gains when the trained memory and executor are used together.

Table 2 shows that Formed targets increase from 129 to 132 out of 143 observations. In Table 3, among Available targets under each model's own memory, omitted behavior decreases from 20/121 (16.5%) to 8/126 (6.3%), while incorrect behavior remains near 19.0%. Fewer omissions are the more pronounced change, consistent with more frequent fulfillment of requirements associated with available memories. These aggregates provide behavioral clues; how changes in memory content and representation produce the interaction requires further analysis.

We revised Section 5.4 to report the interaction alongside fixed-memory gains, and the Introduction and Conclusion to emphasize that fine-tuning gains depend on the combination of memory source and executor.

Q5. The role of consolidation supervision

Execution-only training improves execution, while joint training achieves a higher Task Score when the model forms and uses its own memory.

With Qwen3.8-27B Base memory fixed, execution-only training raises Task Score from 0.5529 to 0.6358, but this model scores 0.6062 with its own memory. With the Base executor fixed, replacing Base memory with execution-only memory also lowers the score, from 0.5529 to 0.5215. Better execution thus need not accompany improved downstream utility of self-generated memory.

Joint training scores 0.6502 with its own memory, versus 0.5972 with Base memory and 0.6062 for execution-only training with its own memory. Together with Table 5, this suggests that end-to-end performance depends on how memory formation and use work together.

A qualitative trace example concerns the standing requirement to disclose brand collaborations: the execution-only model recognized the rule but decided not to save it, whereas the jointly trained model continued to retain it. This is consistent with joint training helping preserve information useful for future tasks and provides a behavioral clue to the memory-utility differences.

A potential benefit of consolidation training is therefore preserving long-term information and supporting its later use alongside execution improvements. As noted in the paper, the settings are not data-matched; this is an interpretation supported by the results and behavioral observation, not an isolated causal effect of consolidation supervision.

AuthorsOpenReview ↗

Response to ct9z (3/3): Gold Memory and Experimental Details (Q6-Q9)

Q6. Gold-memory reference results

To provide the requested oracle-memory reference, we supplied each task's correct target memories directly in the current request, leaving subsequent reasoning and execution to the model. The table compares Task-Relevant Gold Memory with Model-Generated Memory, in which each model forms and uses its own memory, on the same 100 tasks.

ModelModel-Generated MemoryTask-Relevant Gold Memory
Qwen3.8-27B Base0.5530.613
Memory-SFT-Qwen3.8-27B (ours)0.6500.681
Claude Opus 50.7120.765
Qwen3.6-27B0.5330.624

Values are mean Task Scores. All four models improve, showing that supplying correct, task-relevant memories recovers part of the performance loss. This indicates room to improve both the memories models form and their subsequent use. The reference supplies correct content and makes its relevance to the task explicit, helping models use the required information more directly.

Even with correct target memories, scores remain 0.235–0.387 below the maximum of 1.0, leaving substantial difficulty in completing the task. Planning and tool-execution limitations manifest at this stage: beyond applying the supplied memories correctly, the model must interpret the current environment, organize action steps, and carry out operations through tools. The constraint comparison in Q1 provides a concrete example: even with task-relevant constraints in the request and an explicit instruction to send a message, three observations still lacked the required sending action. These results therefore show both the gains available from improving memory and its use, and the importance of planning and action execution after correct memories are supplied.

Appendix D.1–D.2 reports the gold-memory setup and results, and Appendix D.3 reports the constraint comparison; Section 5.3.1 summarizes the findings.

Q7. Opus results in Tables 1 and 4

Thank you for pointing out this inconsistency. It arose from inconsistent trial aggregation: Table 1 averaged two trials for some tasks, whereas Table 4 used trial 1 for each task. Both tables drew on the same pool of evaluation runs.

We have corrected Table 1 to use trial 1 consistently. The Opus 5 result under w/ Memory is now 59.0% Pass Rate and 0.7115 Task Score, matching Table 4. We have also updated the affected Opus 5 and Qwen3.6-27B results, Memory Gain, and the corresponding ranges in the abstract and main text.

Q8. Training samples and optimization steps

Thank you for pointing out the missing detail in the training counts. The 492 and 213 samples are counts before packing. We use binpack to combine multiple samples into training sequences of at most 120,000 tokens.

Joint and execution-only training produce 218 and 67 packed sequences, respectively. Both use 10 epochs and a global batch size of 16 packed sequences, with 136 and 41 completed optimization steps, respectively.

The “Delete” entry specifies how overlength samples are handled. All samples satisfy the length limit when checked with the training template, so none are removed for exceeding it.

We have added the packing configuration and separate counts for samples before packing, samples retained after length checks, packed sequences, and completed optimization steps in Appendix F.2. Appendix F.1 retains the qualification that the two settings differ in both training data size and optimization steps.

Q9. Consolidation phase selection

Thank you for pointing out the missing description of phase selection. Training samples are selected at the phase level, and 98 is the number of source trajectories that contribute at least one eligible phase.

For each phase, we inspect the complete persistent memory saved at its end to check whether every annotated information item introduced during that phase is retained. Qwen3.8 performs this check and cites supporting passages from memory for each positive judgment.

These 98 source trajectories contain 491 phases in total: 81 introduce no annotated information, and 131 fail to retain all newly introduced annotated information. All remaining 279 phases enter the training set, with no further length-based or deduplication filtering. Of the source trajectories, 97 contain five phases. The remaining trajectory contains 46 timeline files, divided into six phases of at most nine files each.

We have clarified in Section 5.4 that the consolidation phases are selected through a memory completeness check, added the selection procedure and counts in Appendix F.1, and limited the five-phase description in Appendix A.1 to evaluation tasks.

Reviewer_T1GXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper built a benchmark that tests whether agents can form memories from a mixed long-term personal activity history before downstream tasks appear and use them in cross-application action. The test contains 100 tasks. The analysis criteria include Formation, Availability, Application, and Outcome for diagnosis on why memory-related failures occur. Despite persistent memory improving task performance, there is still a visible gap between memory availability and successful application.

Critical strengths
  1. The motivation is strong, as agent memory utilization is still a critical open problem.

  2. The Formation–Availability–Application–Outcome decomposition makes sense and provides useful diagnosis for identifying where memory-dependent tasks fail

  3. Evaluation is comprehensive.

  4. The benchmark construction is dedicated, that includes carefully curated tasks.

Critical weaknesses

My main concern is the diagnostics. Poor performance may have various reasons, including weakness in reasoning abilities. I didn't find convincing justifications in the paper, that poor/good performances in many tasks are indeed or likely due to poor memory, rather than the underlying models' internal abilities. Across the models being tested, only Opus 5 could be classified as a frontier model with strong reasoning ability, while others are smaller models so diagnosis is less convincing on them.

Questions for authors

Please see Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to T1GX: Memory Diagnosis and Model Capabilities

We agree that general reasoning and execution abilities affect task performance.To examine performance when relevant information is explicitly supplied, we evaluated a Task-Relevant Gold Memory condition, in which the correct memories relevant to each task are appended directly to the current user request. This bypasses memory formation and provides task-specific memory selection, while leaving subsequent reasoning and execution to the model.

On the same 100 tasks, mean Task Scores are 0.613, 0.681, 0.765, and 0.624 for Qwen3.8-27B Base, Memory-SFT-Qwen3.8-27B (ours), Claude Opus 5, and Qwen3.6-27B, respectively. Even with the relevant memories explicitly supplied, all four models, including Opus 5, remain below full marks. Downstream difficulties therefore remain and may involve reasoning about rules, planning, and carrying out actions. Each model also scores higher in this condition than with its own generated memory, indicating that directly supplying correct task-relevant memories affects performance.

The gold-memory reference therefore complements our lifecycle diagnosis: it reports task performance with explicit memory assistance, while the lifecycle labels indicate whether information was formed, became available, and was correctly applied. Application records whether the requirement associated with a target memory was fulfilled; it does not by itself identify the cause of a failure. The full setup and comparisons with Model-Generated Memory and Full Gold Memory are reported in Appendix D, "Gold-Memory Reference Experiments," with a summary in Section 5.3.1.

Reviewer_CDdNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a benchmark for evaluating how long-term memory affects task execution in personal assistants. Built on Claw-Anything, it includes 100 tasks with synthetic user histories and memory annotations. The authors examine whether agents can form, access, and correctly use relevant memories. Results show that memory improves task success, but agents often fail to act correctly even when the relevant information is available. The paper also studies the effects of different memory generators, executors, and supervised fine-tuning.

Critical strengths

Relevant research problem. The paper goes beyond memory retrieval and examines whether agents can actually use long-term memories to complete tasks, which is an important problem for personal assistants.

Useful diagnostic analysis. The separation of memory formation, availability, and application helps identify where failures occur. The memory-swap experiments also provide useful insights into how memory quality interacts with executor capabilities.

Critical weaknesses
  1. Limited novelty. The benchmark is built on Claw-Anything, while existing benchmarks such as Mem2ActBench and MemoryArena already evaluate memory-guided agent actions. The paper does not clearly establish what new capabilities its benchmark measures.

  2. Insufficient human validation. Although the authors mention human review, they provide no details about the number of annotators, annotation guidelines, inter-annotator agreement, or rejection rates. This raises concerns about the reliability of the LLM-generated labels and graders.

  3. Limited dataset scale and diversity. The benchmark contains only 100 tasks across 13 synthetic personas, with multiple tasks sharing the same user histories. It is unclear whether the results generalize to more diverse users and real-world interactions.

  4. Lack of memory system baselines. The experiments mainly compare agents with and without memory across different LLMs. Comparisons with representative memory systems such as Mem0, A-MEM, and MemOS are missing, limiting the benchmark's demonstrated ability to distinguish memory approaches.

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to CDdN (1/2): Contribution and human validation

Thank you for recognizing the importance of evaluating memory use in personal assistants and the value of our diagnostic analysis. We address your four concerns in two parts.

Q1. Contribution relative to existing benchmarks

We have revised Section 2 to clarify the evaluation setting and the opening of Section 4 to separate our additions from the capabilities inherited from Claw-Anything.

Mem2ActBench evaluates offline memory-grounded tool-call generation. Its main evaluation supplies the correct tool to focus on parameter grounding, with tool selection studied separately. Our execution tasks require agents to select and call tools in service environments, use returned information to guide further actions, and satisfy task requirements.

MemoryArena also evaluates memory-guided environment interaction. It organizes evaluation into interdependent subtasks, where information or experience from earlier actions and feedback supports related later subtasks. For example, its bundled shopping tasks require agents to remember attributes of earlier purchases when choosing compatible products.

Our benchmark starts from a user's long-term personal activity history across applications, mixing topics, preferences, constraints, changing choices, and irrelevant content. Agents form persistent memory before seeing a subsequent request, then determine which user information applies to that request and the current environment and use it to guide judgments and actions.

We link historical evidence and target memories to specific decisions and task-level scoring criteria. This allows us to distinguish failures in Formation, Availability, and Application, and to separate correct memory Application from task failure due to other requirements.

Claw-Anything provides the long-term activity, cross-service state, and executable environments. We add the memory-formation and use protocol, target-memory annotations, checks for memory dependence, and formation-to-action diagnostics. Together, these elements evaluate whether agents retain user information from mixed personal histories and apply it to later personal-assistant decisions.

Q2. Human review and label reliability

We have added the review procedure and selection results to Section 4.2 and Appendix B.3, "Human Review and Task Selection."

One human reviewer checked four criteria: task feasibility, historical support for target memories and their scope, reference-solution validity, and whether graders distinguish correct from incorrect behavior on core memory-dependent requirements. The materials included task requests, historical and current environment records, target memories, reference solutions, and scoring specifications. Execution traces were also inspected when tasks were ambiguous or scores raised concerns.

Of 163 reviewed candidates, 24 were excluded because of substantive issues (14.7%): missing world facts, conflicts between the request and reference solution, incorrect memory scope, or grading rules that penalized valid alternative answers. We selected 100 test tasks from eligible candidates, considering train-test persona separation, task and memory coverage, and the difficulty distribution informed by existing model results. Thus, omission from the final set does not necessarily indicate a quality failure; the 14.7% exclusion rate applies to the reviewed candidate pool, not the full generation pipeline.

Because one person conducted the review, we cannot report inter-annotator agreement. Appendix B.3 explicitly states this limitation and provides the four review criteria.

AuthorsOpenReview ↗

Response to CDdN (2/2): Dataset coverage and memory-system baselines

Q3. Dataset scale, diversity, and generalization

The benchmark contains 100 tasks across 13 synthetic personas. Some tasks share histories and are therefore not independent history samples. Our conclusions apply to this synthetic, controlled personal-assistant setting; generalization to more diverse users and real-world interactions requires further validation. Section 4.3 and the appendix's Limitations subsection describe the shared histories, limited coverage, and synthetic setting.

We have added coverage statistics to Section 4.3, "Benchmark Composition." Of the 100 tasks, 66 depend on a single target memory and 34 on multiple target memories; 45 involve target memories with applicability conditions or exceptions. By historical evidence type, 88 tasks are classified as directly stated, while the remainder involve behavioral records or evidence across scenes. The benchmark mainly covers retention and subsequent application of historical information, including multiple-memory requirements and applicability conditions. Coverage of update resolution and implicit preference inference remains limited.

Q4. Representative memory-system baselines

We agree that our current experiments do not establish the benchmark's ability to distinguish memory systems such as Mem0, A-MEM, and MemOS. This remains a limitation of our experimental coverage.

Our experiments hold the agent scaffold and memory interface fixed. We compare w/o Memory and w/ Memory across models to examine whether formed memories enter the execution context and support correct action. Controlled memory swaps change the memory source under a fixed executor, allowing us to examine its effect on downstream performance and how memory utility varies across executors.

Related Work and Section 5.1 describe this scope. The appendix's Limitations subsection notes that alternative retrieval methods, context-management strategies, and tool planners may change the relative bottlenecks across stages. We therefore restrict our conclusions to the evaluated models and memory interface, without claiming comparative performance across memory systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.