acceptodds
Under review as a conference paper at ICLR 2027

PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

Abstract

Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for studying this capability because they retain preferences, task histories, tool routines, and learned skills across sessions. Yet whether retained experience actually improves them over time has not been systematically tested. We introduce PAST-Bench, a benchmark designed to isolate this question. Each agent runs through ordered sequences of fresh-session tasks under matched conditions that turn retained experience on and off. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update. We report both later-task gains and whether those gains follow the intended save, retrieve, and update pathway. Across seven base models and four agent frameworks, improvement is real but uneven across capabilities. Agents with the same headline gain can differ markedly in whether that gain is supported by evidence of the intended pathway. Guided by these findings, we develop Hermes+, which extends Hermes with five targeted interventions across stages of the agent loop. Hermes+ raises the average gain from retained experience and provides clearer pathway evidence, with its strongest improvement on tasks requiring outdated state to be replaced, although the effect remains capability- and model-dependent. Together, PAST-Bench and Hermes+ provide an evaluation and diagnostic foundation for studying how persistent agents can progress from retaining experience to systematically improving through it.

Primary area:
datasets and benchmarks
Keywords:
personal AI agents, persistent memory, cross-session evaluation, recursive self-improvement, agent benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EWusOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PAST-Bench, a benchmark for evaluating whether persistent AI agents benefit from retaining and reusing experience across sessions. The benchmark comprises 26 task families and 204 episodes covering four capabilities: memory, procedural reuse, information gathering, and updating outdated state. It evaluates later tasks under matched persistence-on and persistence-off conditions, with conversational context cleared between episodes, and supplements task-performance gains with a mechanism-evidence score derived from persistent artifacts and execution traces. The authors evaluate seven base models within Hermes and compare four baseline agent frameworks using a fixed model. They also introduce Hermes+, which incorporates five interventions targeting planning, memory representation, skill management, retrieval timing, and state consolidation. The results reveal substantial variation in persistence benefits across capabilities, models, and frameworks. Although Hermes+ increases the mean update-related persistence gain, it degrades procedural reuse and does not demonstrate a stable overall improvement. The study highlights the value of evaluating persistence capabilities separately and examining both task outcomes and the processes through which retained state is used.

Critical strengths

The paper addresses an important evaluation problem: whether persistent agents benefit from retained experience and how those benefits arise. Its matched persistence-on/off protocol, combined with fresh-session evaluation and a four-capability decomposition, provides a useful framework for separating persistence benefits from baseline task-solving ability. The experiments cover multiple models and frameworks and include individual intervention ablations, per-family results, and informative negative findings. In particular, the transparent reporting of procedural regressions, uneven cross-model transfer, and the absence of a stable overall Hermes+ improvement makes the study useful for understanding the practical limitations of persistent-agent systems.

Critical weaknesses
  1. The control-based interpretation of the reported persistence gains needs clarification. Section 3.2 states that a persistence gap is accepted as evidence of self-evolution only when it exceeds a bound established by control episodes. However, the manuscript does not make clear how this bound is computed, which configurations satisfy it, or how it relates to the published aggregates. This makes it difficult to assess exactly what the reported gaps establish beyond the matched persistence-on/off comparison. The authors appropriately describe Mech as a consistency indicator and provide sensitivity analyses in Appendix B.5; nevertheless, its relationship to the control-based acceptance rule and the interpretation of its aggregate score should be stated more explicitly.
  2. Some conclusions about the Hermes+ interventions should be limited to the effects directly measured. In Table 4, E3 increases the procedural persistence gap from +0.05 to +0.10 while reducing persistence-on performance from 0.55 to 0.47. The larger gap is a valid increase in the measured benefit of persistence within the modified configuration, but it does not establish improved procedural task performance. The statement that each intervention improves its target capability therefore needs qualification. Similarly, the focused analysis in Table 8 provides a plausible explanation involving E2 and E3, but does not clearly distinguish their interaction from a harmful effect of E2 itself, which already reduces procedural performance when applied alone. Given the explicitly limited scope of this diagnostic, the interaction should be presented as a tentative explanation unless stronger supporting evidence is available.
Questions for authors
  1. How is the control-based bound defined and used in interpreting the reported persistence gaps?
  2. Which state surfaces are removed under persistence-off, particularly the preseeded references in Information Gathering? How is this component intended to contribute to the interpretation of the overall score?
  3. Did the same task families inform both the development and evaluation of Hermes+? What task-specific information or benchmark metadata, if any, is accessible to the interventions?
6 comments
AuthorsOpenReview ↗

Response to Reviewer EWus (Part 1/2)

We sincerely thank you for the careful reading and for appreciating our transparent reporting of negative findings. We accept all three points and have revised the manuscript accordingly; all changes are highlighted in blue.

W1 / Q1: How is the control-based bound defined and used? How do Mech and its aggregate score relate to it?

A1: There is no separate numerical bound: the controls are comparisons built into every task family, and each rules out one alternative explanation of a gain. We agree that the phrase "clears the bound" suggested a threshold, and we have replaced it.

  • Shortcut controls show that the task is solvable once the information is given. They state the key information in the prompt, so a low persistence-off score reflects missing information rather than task difficulty.
  • No-retention, distractor, stale, and wrong-mechanism controls show that the gain comes from the retained item. They remove the earlier state, or present irrelevant, outdated, or misplaced information that should not be applied. An agent that reuses stored state indiscriminately loses points on them.

Every Δ is measured against the persistence-off run. Because the controls are part of each family's design rather than a filter applied afterwards, all 26 families are included in the reported results. We have replaced "clears the bound" in the introduction and Section 3.2 with this description.

Mech is a separate diagnostic and does not enter Δ. Δ asks whether retained experience improved the outcome. Mech, the mechanism score, asks whether the trace shows the intended write → retrieve → apply cycle: Mech = 1 means the full cycle is present, and Mech = 0 means it is absent. The aggregate Mech is the mean of its five components over families, so a value such as 0.64 means that, on average, most but not all of the expected evidence is present. We have stated this interpretation explicitly in Section 3.2.

W2: Claims about individual interventions should be limited to the measured effects. E3 raises the Procedural gap but lowers the persistence-on score, and the E2/E3 interaction in Table 8 is not separated from a direct effect of E2.

A2: We agree, and following your suggestion we have limited each claim to the measured effect.

Single interventions (Table 4): each raises Δ on its target ability, and three of them also raise the persistence-on score.

  • E2 on Memory: 0.77 → 0.80, Δ +0.26 → +0.30;
  • E4 on Information Gathering: 0.71 → 0.78, Δ +0.09 → +0.17;
  • E5 on Update: 0.62 → 0.70, Δ +0.12 → +0.16.

E3 doubles the Procedural Δ (+0.05 → +0.10) but lowers the persistence-on score (0.55 → 0.47). It therefore increases how much the agent relies on retained procedures, not procedural performance itself. We have rewritten the abstract, the introduction, and Section 4.4 accordingly.

The E2/E3 result (Table 8) is now reported as an observation only. We agree that the data do not separate an E2×E3 interaction from a direct effect of E2, since adding E2 alone already lowers the Procedural score. We therefore report the measured change, a Procedural Δ of +0.085 for the full system and +0.108 without E2, without attributing it to either explanation, and we have removed the interaction explanation from the introduction, Section 4.4, Appendix D.2, and the conclusion.

AuthorsOpenReview ↗

Response to Reviewer EWus (Part 2/2)

Q2: Which state is removed under persistence-off, particularly the preseeded references in Information Gathering? How does this component contribute to the interpretation of the overall score?

A3: For Hermes and Hermes+, which are used in all model comparisons and interventions, persistence-off disables the memory, user-profile, skill, and session-search tools, and stored memory is not injected into the prompt. The agent's shell and file tools run in a sandbox that is created fresh for each episode and holds no stored state, so the agent has no other way to reach it. The prompt, task tools, grader, and random seed are unchanged.

Information Gathering measures whether the agent consults an existing reference at the right moment. The reference is preseeded in both conditions, but it can be reached only through the persistence tools above, which are available only under persistence-on. This complements the other three abilities, where the agent must also decide what to save. The overall score weights the four abilities equally, and excluding Information Gathering does not change the conclusions: every model keeps a positive Overall Δ (+0.14 to +0.30, computed from Table 2), the model ordering is essentially unchanged, and among frameworks Hermes+ remains highest (+0.16). We have stated the role of this ability in Section 3.1.

Q3: Did the same task families inform both the development and evaluation of Hermes+? What task-specific information or benchmark metadata is accessible to the interventions?

A4: Hermes+ was designed from failures observed across all 26 families (Section 4.3), and no family was held out. We treat it as a diagnostic baseline that tests whether the failures PAST-Bench reveals can be addressed at specific stages of the agent loop.

  • The interventions receive no benchmark metadata: no family IDs, graders, expected artifacts, or episode roles. They see only the task prompt and the agent's own persistent state.
  • They are general loop-level rules applied identically to every task, with no family-specific settings.

We have stated what information the interventions can access in Section 4.3.

We thank you again for the constructive review. We hope these revisions address your concerns, and please feel free to let us know if you have any further questions.

Reviewer_EWusOpenReview ↗

Response to Submission9932

Thank you for the response. Your explanation that the controls are built into each task family, with no numerical acceptance threshold, resolves the main ambiguity in W1. The revisions you describe to the E3 claim and the withdrawal of the E2/E3 interaction explanation also address my concerns in W2. The details about how Hermes+ was developed are helpful.

I have three remaining questions:

  1. The second part of Q1 is still open. How do you handle Mech components that do not apply, and cases with zero denominators? This includes families without update episodes, zero near-evaluation scores, and cases where the agent writes no entries.

  2. In A3, does disabling persistence tools remove their schemas, or leave the same interfaces available with empty or inaccessible state? Please make this explicit in the description of the matched conditions. Could the existing traces also help explain whether tool unavailability or changes in retrieval behavior contribute to E3’s lower persistence-off score?

  3. I could not find a response to Q5. Could you report the paired run-level differences for the main Hermes/Hermes+ comparison from the existing runs? Please also explain what the human audit tells us about the reliability of persistence-gain comparisons. Agreement on individual scores does not by itself establish agreement on the differences between matched conditions.

Reviewer_fo8oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether personal agents can use experience retained across sessions to improve their performance on subsequent tasks. PAST-Bench contains 26 task families and 204 episodes covering memory, procedural reuse, information gathering, and state updates. The authors compare performance with and without access to retained state and examine whether execution traces show the expected saving, retrieval, and update behavior. They also use the benchmark to develop Hermes+. The results show that retained experience is often useful, but its benefits vary across capabilities. Combining several interventions improves update performance while hurting procedural reuse, leaving the overall persistence-on score unchanged.

Critical strengths

The paper’s most valuable contribution is its distinction between storing experience and actually benefiting from it. Fresh-session evaluation and matched controls over access to retained state make this question more tractable than a comparison of task success rates alone. This is a useful evaluation design with clear relevance to practical agent systems.

I also appreciate that the analysis goes beyond an aggregate score. The Hermes+ results expose trade-offs between capabilities and show that interventions that help individually may not combine well. These findings offer useful guidance for researchers building memory and skill systems.

Critical weaknesses

My first concern is how the paper determines whether an observed performance gap passes its controls. The text states that the gap must exceed a bound derived from control episodes, but I could not reconstruct that decision rule from the current description. Some components of Mech also depend on keyword matches, record counts, and outcome correctness. The sensitivity analysis establishes some stability of the score, but does not yet establish that it reliably distinguishes appropriate use of retained state from superficial trace matching. This matters directly to the benchmark’s central attribution claim.

Second, Hermes+ was developed from failures observed on the benchmark, but the boundary between development and evaluation tasks is unclear. Cross-model evaluation is useful, but it does not substitute for evaluation on new task families. Without knowing which families were used during development, I remain uncertain about how well the interventions and their proposed explanations transfer.

Questions for authors
  1. How exactly is the control-derived bound computed, and which task families pass it? Do the main conclusions hold when the analysis is restricted to families that pass these checks?
  2. Which task families were used to design and tune Hermes+? Were any families held out in advance? If so, please report their results separately, as this would materially affect my assessment of generalizability.
6 comments
AuthorsOpenReview ↗

Response to Reviewer fo8o (Part 1/2)

We sincerely thank you for the positive assessment and for the concrete questions. We address them point by point below; all changes in the revised manuscript are highlighted in blue.

Q1: How exactly is the control-derived bound computed, and which task families pass it? Do the main conclusions hold when the analysis is restricted to families that pass these checks?

A1: No separate bound is computed, so no family passes or fails one, and all 26 families are included in the reported results. The controls are comparisons built into every family, and each rules out one alternative explanation of a gain. We agree that the phrase "clears the bound" suggested a numerical threshold, and we have revised Section 3.2 to describe the controls as follows.

1. Each family contains two or three control episodes of the following types (Section 3.1).

ControlWhat it doesExplanation it rules out
No-retentionRemoves the state from earlier episodesThe gain does not need memory
ShortcutStates the key information in the promptThe evaluation prompt leaks the answer, or the task is simply too hard
DistractorInjects irrelevant or superficially similar informationThe agent reuses a surface pattern rather than the rule
StaleExposes obsolete memoriesThe agent applies an outdated version
Wrong-mechanismSurfaces an incorrect skill or the wrong evidence sourceThe agent uses whatever it retrieves, regardless of fit

2. The controls make Δ interpretable rather than filtering families.

  • Every family is run with persistence off, and every Δ is measured against this run.
  • The shortcut control shows that the task is solvable when the information is given, so a low persistence-off score reflects missing information, not task difficulty. Evaluation prompts have this information removed (Section 3.2), and during construction we checked that controls and distractors do not reveal it (Appendix A.2).
  • The other controls present information that should not be applied. An agent that reuses stored state indiscriminately loses points on them, so the evaluation gain is specific to the retained item.

Because these checks are part of each family's design rather than a filter applied afterwards, every family enters the results. We have replaced "clears the bound" in the introduction and Section 3.2 with this description and added the table above to the paper.

W1: Some Mech components depend on keyword matches, record counts, and outcome correctness. Does Mech distinguish appropriate use of retained state from superficial trace matching?

A2: Yes. Mech does not credit a trace merely for containing the right keywords or calls, because three of its five components depend on the outcome:

  • Recall accuracy counts only when the retrieval is followed by a correct result.
  • Update correctness requires that the stored artifact is actually updated and that the answer after the update is correct.
  • Retention horizon compares task scores on near and far evaluation episodes.

An agent that retrieves the right record but applies it wrongly therefore loses credit on these components. Beyond the sensitivity analysis in Appendix B.5, two further checks support this:

  • High gains come with mechanism evidence, and high Mech comes with gains. In the new separate-memory runs for this rebuttal (Hermes with two current base models, 52 model–family pairs; see our response to Reviewer eaYg), none of the 26 pairs with an above-median Δ has Mech below 0.5 (the lowest is 0.58). Conversely, all 26 pairs with an above-median Mech have a positive Δ, and only one has Δ below 0.1 (GLM-5.3-Flash on SM06), a family where the persistence-off score is already 0.90 and leaves little room for a gain. Across all pairs the correlation is weak (Spearman 0.11), consistent with Section 3.2: Mech and Δ answer different questions.
  • Where they disagree, the traces show identifiable behaviors. In such cases, the agent answers from the current task without any retrieval (where a low Mech is correct), retrieves a distractor record, or applies an outdated version of a rule.

We have added these checks to Appendix B.5.

AuthorsOpenReview ↗

Response to Reviewer fo8o (Part 2/2)

Q2: Which task families were used to design and tune Hermes+? Were any families held out in advance?

A3: Hermes+ was designed from failures observed in traces across all 26 families (Section 4.3), and no family was held out in advance. We designed it as a diagnostic baseline: its purpose is to test whether the failures that PAST-Bench reveals can be addressed at specific stages of the agent loop. Three properties limit the risk of fitting to the benchmark:

  1. The interventions receive no benchmark-specific information. They never see family IDs, graders, expected artifacts, or episode roles, only the task prompt and the agent's own persistent state.
  2. They are general rules, not per-family settings. The five interventions are loop-level rules applied identically to every task: consult saved state before planning, store typed memory, rank and patch skills, retrieve before answering, and write back corrections at the end of an episode. There are no family-specific parameters. The benchmark results revealed five recurring failures, and each intervention targets one of them (Section 4.3). "Tuned on MiniMax-M2.7" in Section 4.4 refers only to the single-mechanism ablations in Table 4, which were run with MiniMax-M2.7. To avoid this misunderstanding, we have revised the wording in Section 4.4.
  3. The Update gain of Hermes+ is positive on all five base models (Table 9).

We have stated what information the interventions can access in Section 4.3.

We thank you again for the constructive review. We hope these responses address your questions, and please feel free to let us know if you have any further questions.

Thanks for the clarifications, which resolve my question about the control bound. One remaining point: since no family was held out for Hermes+, could you report results on a few new families constructed after Hermes+ was finalized? If that is not feasible, please state this as a limitation.

Reviewer_eaYgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether personal agents improve on later tasks by retaining information from earlier interactions. The authors introduce PAST-Bench, which contains 26 synthetic task families and 204 episodes covering memory, procedural reuse, information gathering, and updating outdated information. Each evaluation episode starts in a fresh session and is compared under matched conditions with and without access to persistent state. The benchmark reports both the resulting task-score difference and a mechanism-evidence score measuring whether saved artifacts and execution traces follow the expected persistence pathway.

The evaluation varies seven base models within Hermes and compares four baseline agent frameworks with MiniMax-M2.7. All seven models benefit from persistence within Hermes, although the gains concentrate in different capabilities. Using the benchmark to diagnose failures, the authors introduce Hermes+, which adds five interventions to the agent loop.

Critical strengths

The matched persistence on/off comparisons are a neat way to separate the benefit of retained state from benefit of the underlying model or agent configuration.

Critical weaknesses

On the motivation for synthetic tasks. The paper motivates its benchmark through the premise that retaining experience should allow a personal agent to serve the same user better over time. However, it seems that all task families are synthetic and contain no real-user data. The prompts, graders, and expected artifacts were generated together from author-written capability definitions and design rules. Existing benchmarks informed the taxonomy, but no tasks were adapted from them.

I would expect the authors to first explain why existing user records or human-authored interactions cannot support the evaluation, or why collecting reasonably suitable real data would be infeasible or harmful. The construction section explains how the synthetic tasks were produced, but does not establish why this choice was necessary.

I found the following benchmarks that, while not exact counterparts to PAST-Bench, demonstrate alternatives for obtaining human-grounded data and evaluating the usefulness of personal information. At minimum, the authors should explain where these approaches are insufficient and/or whether their data could independently validate the corresponding parts of PAST-Bench.

  • LaMP (https://aclanthology.org/2024.acl-long.399/) constructs personalization tasks from sources including publication records, MovieLens tags, and product reviews. Its time-based setting evaluates future interactions using earlier user records. It is not an interactive tool-use benchmark, but demonstrates an existing route to grounding personalization evaluation in observed user behavior.
  • Multi-Session Chat, introduced in “Beyond Goldfish Memory” (https://arxiv.org/abs/2107.07567), collects human-authored conversations across sessions and evaluates methods that summarize and retrieve earlier conversations. These are crowdsourced role-playing interactions with simulated elapsed time, rather than natural deployment logs. Nevertheless, they demonstrate an alternative to having models generate the entire interaction sequence.
  • TREC iKAT (https://trec.nist.gov/pubs/trec33/papers/Overview_ikat.pdf) evaluates whether conversational assistants select relevant personal information and use it to produce useful, grounded responses. Its authored personas are not a substitute for real-user data, but it is a relevant precedent from information retrieval for the paper’s information-gathering objective.

On the realism of synthetic tasks. Now, it might be possible that existing datasets do not meet the specifications for every evidence. However, it's still necessary to establish that the synthetic tasks are preserving the requirements of the intended real-world application.

Right now, the paper assesses the quality of the dataset by checking for logically consistency, ambiguity, shortcuts, etc. These are useful checks. However, these specification don't help establish that current tasks represents real users' requirements.

For example, a correction can be internally consistent without establishing that its wording, importance, timing, and scope resemble the corrections an assistant would encounter in a realistic setting. Similarly, constructing a later task that requires an earlier fact does not establish how often and how real human conversations rely on previously referred facts.

Overall, I agree with the authors that retaining information does not guarantee using it well, and that controlled comparisons can help diagnose these failures. However, the proposed evaluation needs to distinguish between satisfying a constructed persistence contract and improving assistance under independently grounded user requirements. One way to address this is to (1) justify why existing human-grounded data cannot support the relevant experiments, (2) validate the generated dependencies and acceptable solutions against external evidence, and (3) test whether the conclusions survive shared histories and alternative valid persistence pathways.

Questions for authors

In the weaknesses.

11 comments
AuthorsOpenReview ↗

Response to Reviewer eaYg (Part 1/4)

We sincerely thank you for the careful review and for recognizing our matched persistence-on/off design. We address your concerns point by point below; all changes in the revised manuscript are highlighted in blue.

Q1: All task families are synthetic and contain no real-user data. Why not use real data, as LaMP, Multi-Session Chat, and TREC iKAT do, or explain why such data cannot be used?

A1: The real data this evaluation needs, records of real users working with a persistent agent over time, are not publicly available. We therefore built tasks that reproduce interaction types documented in real-user data, human-written data, and widely used benchmarks, including the works you cite.

1. No public data records real users working with a persistent agent. For PAST-Bench, real data would be records of real users working with a persistent personal agent over days or weeks: the agent saving memories and skills, acting on the user's mail and documents through tools, and being corrected. We searched for such data and found related resources that are valuable for personalization and dialogue research, but whose setting differs from ours:

  • Users' own records, such as reviews, ratings, and publications (LaMP; MemoryCD, built from Amazon reviews). These contain no agent, no tools, and no saving step.
  • Users' chats with assistants (WildChat, LMSYS-Chat-1M). The assistants in these logs do not keep state across chats or act through tools.
  • Public agent trajectories (e.g., WildClawBench). These come from benchmark runs, not from real users.

To our knowledge, no public data records real users working with a persistent agent. Such data would contain users' private mail, calendars, and documents; even model providers study real usage only in privacy-preserving aggregate form (Clio; How People Use ChatGPT). In addition, our evaluation runs the same later task twice, with and without the agent's saved memory, which a one-time real record cannot provide.

2. Our tasks are grounded in interaction types documented in prior work. Each of our four abilities corresponds to needs shown in real-user data, human-written data, or widely used benchmarks:

  • Memory. Users' past behavior reflects stable preferences (LaMP, MemoryCD: real user records). People refer back to facts from earlier sessions (Multi-Session Chat: human-written; LoCoMo). Deployed assistants now remember users across chats (ChatGPT Memory).
  • Information Gathering. Deciding which personal information is relevant to the current request (TREC iKAT: human relevance judgments) and finding it across many past sessions (LongMemEval: multi-session reasoning).
  • Update. New information must replace outdated information (LongMemEval: knowledge updates; MemoryAgentBench: conflict resolution).
  • Procedural Reuse. Reusing workflows and skills from earlier tasks improves later ones (Agent Workflow Memory, on Mind2Web and WebArena; Voyager; ExpeL).

The scenarios themselves are everyday assistant work, such as sharing a meeting note only with the right people, following an on-call handoff, or applying a corrected company rule. This matches large-scale studies of real usage, where practical guidance, seeking information, and writing make up nearly 80% of ChatGPT conversations (How People Use ChatGPT).

3. We have added these works to the paper. LaMP, Multi-Session Chat, and TREC iKAT ground personalization and long-term memory in real or human-written data, and they complement our controlled, executable setting. We have added them, together with the works above, to Related Work, and explained in Section 3.1 how our four abilities relate to them.

AuthorsOpenReview ↗

Response to Reviewer eaYg (Part 2/4)

Q2: The paper's quality checks (consistency, ambiguity, shortcuts) show that the tasks are well-formed, not that they reflect real users' needs. For example, a correction can be consistent but unlike real corrections in wording, importance, timing, and scope, and a task that needs an earlier fact does not show how often real conversations rely on earlier facts. Can the generated dependencies and expected answers be validated against external evidence?

A2: Yes, in two ways: the expected answers are concrete outcomes checked from tool logs, with the rubrics for open-ended outputs validated by two independent raters; and each dependency type we test appears in real-user data, human-written data, or established benchmarks. Below we also clarify what PAST-Bench measures and how the tasks vary the properties you mention.

1. PAST-Bench is a capability benchmark. It asks: when a later task depends on earlier information, does the agent save it, find it, and apply it correctly? How often such dependencies arise depends on how people use agents, which is outside the scope of a capability benchmark. We make this scope explicit in the introduction.

2. The tasks vary wording, timing, and scope rather than fixing one form.

  • Wording. The information the agent must keep is given in different ways: an explicit request to save it (7 families), a rule stated or given in a document without asking to save it (3), a preference that moves from an explicit request to a rule embedded in a note (1), or no statement at all, so that the agent must infer the rule from repeated cases or from its own failure (2). In the 7 Update families, it is a correction of earlier information, and the 6 Information Gathering families require the agent to learn from its own failed attempts.
  • Timing. In 25 of 26 families, reuse is tested both in a nearby later task and in a more distant one with different wording.
  • Scope of corrections. The Update families include a global rule change, a rule change limited to one scope, a temporary exception that later expires, a factual correction, and a weakly stated correction to a procedure.

The benchmark is therefore not tied to one wording or timing. We have added a table of these properties for each family to Appendix A.1.

3. The expected answers are concrete outcomes, and the rubrics agree across raters. For action tasks, the expected answer is a concrete outcome checked from tool logs. For example, the note must be shared with the two employees and not with the two external advisors, or the ticket must use the corrected value. For open-ended outputs, two raters independently scored 48 samples with our rubrics: they gave the same score on 83.3% of samples and scores within 0.25 on 97.9% (Appendix B.4).

4. External evidence supports the dependency types. As listed in A1, each type of dependency we test appears in real-user or human-written data (LaMP, Multi-Session Chat, TREC iKAT) or in established benchmarks (LongMemEval, MemoryAgentBench, Agent Workflow Memory). This supports that the dependencies we test are ones that real users and agents encounter.

AuthorsOpenReview ↗

Response to Reviewer eaYg (Part 3/4)

Q3: Do the conclusions still hold when all tasks share one memory over time (shared histories)?

A3: Yes. When all 26 families share one memory, retained experience helps almost as much as with separate memories (Overall Δ +0.47 vs. +0.49 for DeepSeek-V4.1-Flash and +0.44 vs. +0.44 for GLM-5.3-Flash), and families later in the order do not gain less.

1. The paper gives each family its own memory so that gains can be attributed. This lets us trace a later gain back to that family's own earlier episodes. If all families shared one memory, a change in score could also come from other families.

2. Interference within a family is already part of the benchmark. The task families include control tasks with distracting or outdated stored information. In addition, every Information Gathering family starts with 32–37 earlier sessions in memory, many of which are similar or outdated versions of the record the agent needs.

3. New experiment: all 26 families share one memory. We ran all 26 families one after another in one shared memory: each family starts from everything the families before it saved (memories, skills, and session history), with its own initial state added on top. We used two random orders, one run each, and report the average. The persistence-off condition does not use the memory, so it is identical in both settings. To compare the two settings under the same conditions, we re-ran the separate-memory setting at the same time with the same agent, models, and judge: Hermes with DeepSeek-V4.1-Flash and GLM-5.3-Flash, both judged by DeepSeek-V4.1-Flash. Because some of the original base models are no longer accessible through the API, we used current model versions. The numbers below are therefore compared with each other rather than with the paper's tables.

Agent / modelSettingMemory ΔProcedural ΔInfo. ΔUpdate ΔOverall Δ
Hermes / DeepSeek-V4.1-FlashSeparate+0.52+0.45+0.64+0.35+0.49
Shared+0.53+0.38+0.61+0.34+0.47
Hermes / GLM-5.3-FlashSeparate+0.54+0.39+0.57+0.28+0.44
Shared+0.51+0.40+0.58+0.29+0.44

4. Findings.

  • Retained experience helps almost as much with shared memory. Δ stays positive for both models and every ability, in each of the two orders. Overall Δ is +0.47 for DeepSeek-V4.1-Flash and +0.44 for GLM-5.3-Flash with shared memory, compared with +0.49 and +0.44 with separate memory; the largest change for any ability is 0.07. The order of the models is unchanged.
  • The gains do not shrink as experience from other families accumulates. The Spearman correlation between a family's position in the order and its Δ is +0.34 and −0.12 for DeepSeek-V4.1-Flash and +0.44 and +0.13 for GLM-5.3-Flash in the two orders, so families later in the order do not gain less.
  • The mechanism evidence is also nearly unchanged. Mech is 0.79 with shared memory vs. 0.81 with separate memory for DeepSeek-V4.1-Flash, and 0.77 vs. 0.80 for GLM-5.3-Flash.

We have added this experiment to Appendix G.1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their thorough reviews and constructive feedback. We are glad that the reviewers found the matched persistence-on/off design a useful way to separate the benefit of retained experience from that of the model and agent (all reviewers), valued the distinction between storing experience and actually benefiting from it (Reviewers fo8o, EWus), appreciated that the analysis goes beyond a single aggregate score and reports trade-offs and negative findings transparently (Reviewers fo8o, EWus), and found the paper clear and easy to follow (Reviewers eaYg, EWus).

We have revised the manuscript according to the reviewers' suggestions, with all changes highlighted in blue. We summarize and highlight the main updates below:

  • New experiment: all task families share one memory (Reviewer eaYg; Appendix G.1). All 26 families run one after another in a single shared memory, in two random orders, and are compared with separate-memory runs that use the same agent, models, and judge. Retained experience helps almost as much as with separate memories (Overall Δ +0.47 and +0.44 with shared memory vs. +0.49 and +0.44 with separate memory), and families later in the order do not gain less.
  • New analysis: re-scoring with graders that ignore the storage pathway (Reviewer eaYg; also relevant to Reviewers fo8o and EWus; Appendix G.2). We re-scored all test episodes of the new runs with a grader that accepts any storage pathway, and with one that scores outcomes only. Under both graders, every gain remains positive for every model and ability, and the order of models is unchanged; when any storage pathway is accepted, Overall Δ decreases by at most 0.05.
  • Grounding of the constructed tasks (Reviewer eaYg; Section 3.1, Related Work, Appendix A.1). We explain why the evaluation uses constructed tasks, relate our four abilities to real-user and human-written resources (e.g., LaMP, Multi-Session Chat, TREC iKAT), and describe how the tasks vary wording, timing, and scope.
  • Clarified controls, grader, and Mech (Reviewers fo8o, EWus, eaYg; Section 3.2). We replace "clears the bound" with a description of how the control episodes rule out alternative explanations of a gain, describe what the grader credits, and state how Mech and its aggregate score should be read, including why each type of experience has its own store.
  • Narrowed claims about Hermes+ (Reviewers fo8o, EWus; Sections 4.3–4.4). We state what information the interventions can access, limit the claims about single interventions to the measured effects, and report the E2/E3 result as an observation without a mechanistic claim.

We reply to each reviewer's concerns in detail below their individual reviews. Thank you again, and please feel free to reach out if you need any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.