- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark for evaluating whether personal assistants can retain information from long-term activity histories and use it in subsequent tasks across applications. The benchmark contains 100 tasks and 143 task–target-memory pairs, with annotations connecting historical evidence to specific decisions and scoring criteria. Evaluation separates Formation, Availability, Application, and final task outcomes. The experiments show that persistent memory improves task performance, while substantial losses remain after relevant information becomes available. Component-swap experiments and supervised fine-tuning further examine the contributions of memory generation and execution. The main contribution is the connection between memory evaluation and executable behavior. The main unresolved issue is whether the observed Application failures specifically reflect memory utilization difficulties or broader limitations in task execution.
Critical strengths
- The evaluation examines concrete actions and their parameters rather than relying only on factual recall. With 86 tasks requiring specified tool operations, the benchmark can expose cases where an assistant recalls the relevant information but fails to reflect it in the resulting action.
- The consolidation protocol withholds future task requests. Agents process history chronologically across separate sessions, with only persistent memory carried forward. This makes memory formation less dependent on knowing which information a later evaluation will require. (§3.2; Appendix A)
- The annotations connect historical evidence, target memories, task requirements, and scoring checks. These connections provide a useful basis for diagnosing where information is lost between historical observation and downstream behavior. (§3.3)
- The evaluation distinguishes satisfying a memory-dependent requirement from completing the entire task. Table 3 explicitly identifies cases where the relevant requirement is satisfied but other requirements fail, avoiding the assumption that every unsuccessful task reflects a memory failure.
- The component-swap experiments provide more insight than model rankings alone. Tables 4 and 5 show that the downstream value of a memory source depends on the executor, making the compatibility between memory representations and execution behavior a worthwhile subject for further investigation.
Critical weaknesses
- The Application category includes failures that may arise from general execution limitations. In Appendix D, an agent receives the relevant order quantity but stops at recommendations and confirmation instead of placing the order. This establishes incomplete execution, but does not establish a failure specific to memory utilization. Similar behavior could occur if the quantity were stated directly in the current request. The evidence supports the observation that information availability is insufficient for successful action, but the stronger attribution to memory utilization remains underdetermined.
- The operational details behind the lifecycle labels are insufficiently specified. Formation requires a semantic judgment about retained information, Availability requires determining what actually reached the execution context, and Application requires identifying satisfaction of a local requirement. The manuscript does not adequately explain how these judgments are made or how partial, outdated, and contradictory memories are handled. Task Score aggregation and the conditions for a complete task pass also remain insufficiently defined.
- The conclusion that utilization is a larger bottleneck than storage or retrieval depends on the memory interface used in these experiments. Relevant information can enter execution through system-prompt memory blocks, but the manuscript does not report memory lengths, injection coverage, truncation rules, or the proportion requiring active lookup. Without these details, high Availability may partly reflect the scaffold’s delivery mechanism. The limitation is acknowledged in Appendix F, but should also constrain the interpretation of the main findings.
- The construction procedure explicitly selects tasks that depend on memory. Consequently, the advantage of providing memory is partly expected from the benchmark design. A more informative reference would be the performance of each evaluated model when supplied with correct target memories. Construction-time feasibility checks establish that a solution exists, but do not establish how much execution difficulty remains for the evaluated models. The scope and findings of human review are also not sufficiently documented.
- The fine-tuning results show substantial interaction between memory and executor changes. In Table 5, replacing only the executor increases Pass Rate from 44% to 46%, replacing only the memory decreases it to 43%, and replacing both increases it to 55%. The Task Score difference-in-differences is 0.0506, compared with an overall improvement of 0.0973. Memory–executor compatibility is therefore central to the result and deserves a fuller explanation alongside the emphasis on executor improvement. The joint and execution-only training comparison also changes both supervision and optimization duration.
- The benchmark’s temporal span is clearer than its coverage of specific long-term memory difficulties. There are 143 target-memory observations across 100 tasks, averaging 1.43 per task, and some tasks share histories. Although the construction defines topology, trace, and cue dimensions, their distributions and associated results are not reported. It is therefore difficult to determine whether performance is limited by combining memories, resolving updates, inferring preferences across records, or executing actions based on relatively explicit information. (§4.1–4.3)
Questions for authors
- How does the Application definition distinguish memory utilization failures from general instruction-following or execution failures? For cases where an agent receives the relevant memory but stops at a recommendation or confirmation, would the same failure occur if the constraint were stated directly in the current request? What evidence supports attributing these omissions specifically to memory utilization?
- How are Formation, Availability, and Application labels assigned in practice? In particular, how are memories treated when they preserve the correct preference but omit its conditions, or retain both an outdated and a current version? Please also specify how task-level scoring checks are weighted and aggregated into Task Score, what determines a complete task pass, and whether any judgments depend on an LLM evaluator.
- To what extent does the Availability-to-Application gap depend on the current memory interface? What proportion of relevant information is injected directly into the system prompt, and what proportion requires active lookup? What are the memory-size limits and truncation rules? These details are needed to determine whether the reported bottleneck ordering is specific to this OpenHarness configuration.
- How should the interaction in Table 5 be incorporated into the interpretation of fine-tuning? Replacing the executor alone produces a relatively small Pass Rate improvement, whereas replacing both components produces a much larger gain. What changes in memory content, representation, or execution behavior explain this dependence? How do the authors distinguish improved executor capability from improved compatibility with the SFT-generated memories?
- What specific effects of consolidation supervision can be established from the joint versus execution-only comparison? Since the two settings differ in both supervision and optimization duration, the final Memory Gain difference does not isolate the contribution of consolidation training. Is there behavioral evidence that this supervision improves information selection, updating, or subsequent use beyond changes in general task execution?
- How well does each evaluated model perform when given the correct target memories directly? The construction-time feasibility probe demonstrates the existence of an executable solution, but does not establish the evaluated models’ performance under that condition. An oracle-memory reference would help determine how much of the remaining loss reflects memory representation and use, and how much reflects planning or tool-execution limitations.
- Why does the apparently corresponding Opus configuration have different results in Tables 1 and 4? Table 1 reports 58% Pass Rate and 0.7110 Task Score, whereas Opus-generated Memory with the Opus executor in Table 4 yields 59% and 0.7115. Were the same memory artifacts and evaluation conditions used? If these entries represent different runs or configurations, that distinction should be stated explicitly.
- How do the reported training sample counts, epochs, batch size, and optimization steps reconcile? Appendix E reports 492 joint-training samples and 213 execution-only samples, with 10 epochs and a global batch size of 16, but only 136 and 41 optimization steps. Assuming all listed samples are used without packing or filtering, the nominal totals would be approximately 308 and 133 steps, respectively. Since Table E1 specifies deletion of overlength samples, please report the actual post-filtering counts and any packing or step-limit settings that explain the difference.
- How were the 279 consolidation phases selected from the 98 complete consolidation-to-execution trajectories described in Appendix E.1? Appendix A specifies five consolidation phases per complete workflow, which would yield 490 phases if every phase were retained. Were phases excluded because of length, quality, duplication, or another criterion? Please clarify the relationship between the complete source trajectories and the final training samples.
3 comments
Response to ct9z (2/3): Memory Interface and Training (Q3-Q5)
Q3. Memory injection and truncation
High Availability reflects automatic memory injection. Before execution, OpenHarness places MEMORY.md and automatically selected relevant files in the system prompt. The framework selects these files; the agent need not call a read tool.
All 485 Available observations across the four models have supporting evidence in the initial system prompt (100%). None requires a later lookup for the target information to first become available. This percentage is conditional on Availability, not all target memories or historical information. Active reads still occur, but these targets were visible at the start.
MEMORY.md is limited first to 200 lines, then to 25,000 UTF-8 bytes, backing up to a newline where possible. A truncation warning is appended outside the content byte limit. Up to five relevant files contribute their first 8,000 characters each, without an equivalent warning. These limits constrain injection, not persistent-memory storage; omitted content remains accessible through tools.
Of the 485 Available observations, 344 have correct Application and 141 do not. This shows a gap between availability and correct use under automatic injection. It does not establish that active retrieval is generally easier than Application, because the framework already delivers the relevant information. An interface relying on agent-initiated search and reading could have greater lookup losses and a different bottleneck ordering. Section 5.3.1 now states this qualification and the initial-injection count; Appendix A.2 gives the injection rules.
Q4. SFT interaction and compatibility
We agree that the interaction in Table 5 deserves a more explicit role in interpreting fine-tuning.
With Qwen-generated Memory fixed, replacing the Base executor with the Memory-SFT executor improves Pass Rate by 2 percentage points and Task Score by 0.0443. With SFT-generated Memory, the executor's Pass Rate gain rises to 12 percentage points. The interaction is 10 percentage points in Pass Rate and 0.0506 in Task Score, computed from the displayed values; the latter is substantial relative to the overall Task Score gain of 0.0973. These results show execution gains under fixed memory and larger gains when the trained memory and executor are used together.
Table 2 shows that Formed targets increase from 129 to 132 out of 143 observations. In Table 3, among Available targets under each model's own memory, omitted behavior decreases from 20/121 (16.5%) to 8/126 (6.3%), while incorrect behavior remains near 19.0%. Fewer omissions are the more pronounced change, consistent with more frequent fulfillment of requirements associated with available memories. These aggregates provide behavioral clues; how changes in memory content and representation produce the interaction requires further analysis.
We revised Section 5.4 to report the interaction alongside fixed-memory gains, and the Introduction and Conclusion to emphasize that fine-tuning gains depend on the combination of memory source and executor.
Q5. The role of consolidation supervision
Execution-only training improves execution, while joint training achieves a higher Task Score when the model forms and uses its own memory.
With Qwen3.8-27B Base memory fixed, execution-only training raises Task Score from 0.5529 to 0.6358, but this model scores 0.6062 with its own memory. With the Base executor fixed, replacing Base memory with execution-only memory also lowers the score, from 0.5529 to 0.5215. Better execution thus need not accompany improved downstream utility of self-generated memory.
Joint training scores 0.6502 with its own memory, versus 0.5972 with Base memory and 0.6062 for execution-only training with its own memory. Together with Table 5, this suggests that end-to-end performance depends on how memory formation and use work together.
A qualitative trace example concerns the standing requirement to disclose brand collaborations: the execution-only model recognized the rule but decided not to save it, whereas the jointly trained model continued to retain it. This is consistent with joint training helping preserve information useful for future tasks and provides a behavioral clue to the memory-utility differences.
A potential benefit of consolidation training is therefore preserving long-term information and supporting its later use alongside execution improvements. As noted in the paper, the settings are not data-matched; this is an interpretation supported by the results and behavioral observation, not an isolated causal effect of consolidation supervision.
Response to ct9z (3/3): Gold Memory and Experimental Details (Q6-Q9)
Q6. Gold-memory reference results
To provide the requested oracle-memory reference, we supplied each task's correct target memories directly in the current request, leaving subsequent reasoning and execution to the model. The table compares Task-Relevant Gold Memory with Model-Generated Memory, in which each model forms and uses its own memory, on the same 100 tasks.
| Model | Model-Generated Memory | Task-Relevant Gold Memory |
|---|---|---|
| Qwen3.8-27B Base | 0.553 | 0.613 |
| Memory-SFT-Qwen3.8-27B (ours) | 0.650 | 0.681 |
| Claude Opus 5 | 0.712 | 0.765 |
| Qwen3.6-27B | 0.533 | 0.624 |
Values are mean Task Scores. All four models improve, showing that supplying correct, task-relevant memories recovers part of the performance loss. This indicates room to improve both the memories models form and their subsequent use. The reference supplies correct content and makes its relevance to the task explicit, helping models use the required information more directly.
Even with correct target memories, scores remain 0.235–0.387 below the maximum of 1.0, leaving substantial difficulty in completing the task. Planning and tool-execution limitations manifest at this stage: beyond applying the supplied memories correctly, the model must interpret the current environment, organize action steps, and carry out operations through tools. The constraint comparison in Q1 provides a concrete example: even with task-relevant constraints in the request and an explicit instruction to send a message, three observations still lacked the required sending action. These results therefore show both the gains available from improving memory and its use, and the importance of planning and action execution after correct memories are supplied.
Appendix D.1–D.2 reports the gold-memory setup and results, and Appendix D.3 reports the constraint comparison; Section 5.3.1 summarizes the findings.
Q7. Opus results in Tables 1 and 4
Thank you for pointing out this inconsistency. It arose from inconsistent trial aggregation: Table 1 averaged two trials for some tasks, whereas Table 4 used trial 1 for each task. Both tables drew on the same pool of evaluation runs.
We have corrected Table 1 to use trial 1 consistently. The Opus 5 result under w/ Memory is now 59.0% Pass Rate and 0.7115 Task Score, matching Table 4. We have also updated the affected Opus 5 and Qwen3.6-27B results, Memory Gain, and the corresponding ranges in the abstract and main text.
Q8. Training samples and optimization steps
Thank you for pointing out the missing detail in the training counts. The 492 and 213 samples are counts before packing. We use binpack to combine multiple samples into training sequences of at most 120,000 tokens.
Joint and execution-only training produce 218 and 67 packed sequences, respectively. Both use 10 epochs and a global batch size of 16 packed sequences, with 136 and 41 completed optimization steps, respectively.
The “Delete” entry specifies how overlength samples are handled. All samples satisfy the length limit when checked with the training template, so none are removed for exceeding it.
We have added the packing configuration and separate counts for samples before packing, samples retained after length checks, packed sequences, and completed optimization steps in Appendix F.2. Appendix F.1 retains the qualification that the two settings differ in both training data size and optimization steps.
Q9. Consolidation phase selection
Thank you for pointing out the missing description of phase selection. Training samples are selected at the phase level, and 98 is the number of source trajectories that contribute at least one eligible phase.
For each phase, we inspect the complete persistent memory saved at its end to check whether every annotated information item introduced during that phase is retained. Qwen3.8 performs this check and cites supporting passages from memory for each positive judgment.
These 98 source trajectories contain 491 phases in total: 81 introduce no annotated information, and 131 fail to retain all newly introduced annotated information. All remaining 279 phases enter the training set, with no further length-based or deduplication filtering. Of the source trajectories, 97 contain five phases. The remaining trajectory contains 46 timeline files, divided into six phases of at most nine files each.
We have clarified in Section 5.4 that the consolidation phases are selected through a memory completeness check, added the selection procedure and counts in Appendix F.1, and limited the five-phase description in Appendix A.1 to evaluation tasks.
Response to ct9z (1/3): Application and Evaluation (Q1-Q2)
Q1. Application failures and current-request constraints
Stating the constraint directly in the current request can resolve some omissions, but failures to act still occur under explicit instructions. We conducted a paired comparison on 20 previously omitted target-memory observations from Qwen3.8-27B Base. We kept the model-generated memory unchanged and appended only the constraints relevant to the current task to the user request. We compared this condition against a contemporaneous replay of the original condition.
Review of the remaining cases distinguished three situations: three still did not send a message despite an explicit instruction to send it; five involved ambiguity between the request wording and the action required by the evaluation, so they do not establish unnecessary confirmation; and one fulfilled the target-memory requirement but failed a separate recipient requirement. These cases show that placing constraints in the request can help, while action-execution difficulties and task-specification issues remain.
Our Application diagnosis records whether the requirement associated with a target memory was fulfilled and separately identifies cases where that requirement was satisfied but other task requirements were not. The full-benchmark Task-Relevant Gold Memory condition appends the correct task-relevant memories directly to the current request. Appendix D.1–D.2, “Gold-Memory Reference Experiments,” reports its setup and results for all four models; Appendix D.3 reports the constraint comparison.
Q2. Lifecycle labels and task grading
Thank you for noting that the evaluation procedure needs more detail. We have clarified four aspects in Section 3.3 and the new Appendix sections A.3–A.4.