- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PAST-Bench, a benchmark for evaluating whether persistent AI agents benefit from retaining and reusing experience across sessions. The benchmark comprises 26 task families and 204 episodes covering four capabilities: memory, procedural reuse, information gathering, and updating outdated state. It evaluates later tasks under matched persistence-on and persistence-off conditions, with conversational context cleared between episodes, and supplements task-performance gains with a mechanism-evidence score derived from persistent artifacts and execution traces. The authors evaluate seven base models within Hermes and compare four baseline agent frameworks using a fixed model. They also introduce Hermes+, which incorporates five interventions targeting planning, memory representation, skill management, retrieval timing, and state consolidation. The results reveal substantial variation in persistence benefits across capabilities, models, and frameworks. Although Hermes+ increases the mean update-related persistence gain, it degrades procedural reuse and does not demonstrate a stable overall improvement. The study highlights the value of evaluating persistence capabilities separately and examining both task outcomes and the processes through which retained state is used.
Critical strengths
The paper addresses an important evaluation problem: whether persistent agents benefit from retained experience and how those benefits arise. Its matched persistence-on/off protocol, combined with fresh-session evaluation and a four-capability decomposition, provides a useful framework for separating persistence benefits from baseline task-solving ability. The experiments cover multiple models and frameworks and include individual intervention ablations, per-family results, and informative negative findings. In particular, the transparent reporting of procedural regressions, uneven cross-model transfer, and the absence of a stable overall Hermes+ improvement makes the study useful for understanding the practical limitations of persistent-agent systems.
Critical weaknesses
- The control-based interpretation of the reported persistence gains needs clarification. Section 3.2 states that a persistence gap is accepted as evidence of self-evolution only when it exceeds a bound established by control episodes. However, the manuscript does not make clear how this bound is computed, which configurations satisfy it, or how it relates to the published aggregates. This makes it difficult to assess exactly what the reported gaps establish beyond the matched persistence-on/off comparison. The authors appropriately describe Mech as a consistency indicator and provide sensitivity analyses in Appendix B.5; nevertheless, its relationship to the control-based acceptance rule and the interpretation of its aggregate score should be stated more explicitly.
- Some conclusions about the Hermes+ interventions should be limited to the effects directly measured. In Table 4, E3 increases the procedural persistence gap from +0.05 to +0.10 while reducing persistence-on performance from 0.55 to 0.47. The larger gap is a valid increase in the measured benefit of persistence within the modified configuration, but it does not establish improved procedural task performance. The statement that each intervention improves its target capability therefore needs qualification. Similarly, the focused analysis in Table 8 provides a plausible explanation involving E2 and E3, but does not clearly distinguish their interaction from a harmful effect of E2 itself, which already reduces procedural performance when applied alone. Given the explicitly limited scope of this diagnostic, the interaction should be presented as a tentative explanation unless stronger supporting evidence is available.
Questions for authors
- How is the control-based bound defined and used in interpreting the reported persistence gaps?
- Which state surfaces are removed under persistence-off, particularly the preseeded references in Information Gathering? How is this component intended to contribute to the interpretation of the overall score?
- Did the same task families inform both the development and evaluation of Hermes+? What task-specific information or benchmark metadata, if any, is accessible to the interventions?
6 comments
Response to Reviewer EWus (Part 2/2)
Q2: Which state is removed under persistence-off, particularly the preseeded references in Information Gathering? How does this component contribute to the interpretation of the overall score?
A3: For Hermes and Hermes+, which are used in all model comparisons and interventions, persistence-off disables the memory, user-profile, skill, and session-search tools, and stored memory is not injected into the prompt. The agent's shell and file tools run in a sandbox that is created fresh for each episode and holds no stored state, so the agent has no other way to reach it. The prompt, task tools, grader, and random seed are unchanged.
Information Gathering measures whether the agent consults an existing reference at the right moment. The reference is preseeded in both conditions, but it can be reached only through the persistence tools above, which are available only under persistence-on. This complements the other three abilities, where the agent must also decide what to save. The overall score weights the four abilities equally, and excluding Information Gathering does not change the conclusions: every model keeps a positive Overall Δ (+0.14 to +0.30, computed from Table 2), the model ordering is essentially unchanged, and among frameworks Hermes+ remains highest (+0.16). We have stated the role of this ability in Section 3.1.
Q3: Did the same task families inform both the development and evaluation of Hermes+? What task-specific information or benchmark metadata is accessible to the interventions?
A4: Hermes+ was designed from failures observed across all 26 families (Section 4.3), and no family was held out. We treat it as a diagnostic baseline that tests whether the failures PAST-Bench reveals can be addressed at specific stages of the agent loop.
- The interventions receive no benchmark metadata: no family IDs, graders, expected artifacts, or episode roles. They see only the task prompt and the agent's own persistent state.
- They are general loop-level rules applied identically to every task, with no family-specific settings.
We have stated what information the interventions can access in Section 4.3.
We thank you again for the constructive review. We hope these revisions address your concerns, and please feel free to let us know if you have any further questions.
Response to Submission9932
Thank you for the response. Your explanation that the controls are built into each task family, with no numerical acceptance threshold, resolves the main ambiguity in W1. The revisions you describe to the E3 claim and the withdrawal of the E2/E3 interaction explanation also address my concerns in W2. The details about how Hermes+ was developed are helpful.
I have three remaining questions:
-
The second part of Q1 is still open. How do you handle Mech components that do not apply, and cases with zero denominators? This includes families without update episodes, zero near-evaluation scores, and cases where the agent writes no entries.
-
In A3, does disabling persistence tools remove their schemas, or leave the same interfaces available with empty or inaccessible state? Please make this explicit in the description of the matched conditions. Could the existing traces also help explain whether tool unavailability or changes in retrieval behavior contribute to E3’s lower persistence-off score?
-
I could not find a response to Q5. Could you report the paired run-level differences for the main Hermes/Hermes+ comparison from the existing runs? Please also explain what the human audit tells us about the reliability of persistence-gain comparisons. Agreement on individual scores does not by itself establish agreement on the differences between matched conditions.
Response to Reviewer EWus (Part 1/2)
We sincerely thank you for the careful reading and for appreciating our transparent reporting of negative findings. We accept all three points and have revised the manuscript accordingly; all changes are highlighted in blue.
A1: There is no separate numerical bound: the controls are comparisons built into every task family, and each rules out one alternative explanation of a gain. We agree that the phrase "clears the bound" suggested a threshold, and we have replaced it.
Every Δ is measured against the persistence-off run. Because the controls are part of each family's design rather than a filter applied afterwards, all 26 families are included in the reported results. We have replaced "clears the bound" in the introduction and Section 3.2 with this description.
Mech is a separate diagnostic and does not enter Δ. Δ asks whether retained experience improved the outcome. Mech, the mechanism score, asks whether the trace shows the intended write → retrieve → apply cycle: Mech = 1 means the full cycle is present, and Mech = 0 means it is absent. The aggregate Mech is the mean of its five components over families, so a value such as 0.64 means that, on average, most but not all of the expected evidence is present. We have stated this interpretation explicitly in Section 3.2.
A2: We agree, and following your suggestion we have limited each claim to the measured effect.
Single interventions (Table 4): each raises Δ on its target ability, and three of them also raise the persistence-on score.
E3 doubles the Procedural Δ (+0.05 → +0.10) but lowers the persistence-on score (0.55 → 0.47). It therefore increases how much the agent relies on retained procedures, not procedural performance itself. We have rewritten the abstract, the introduction, and Section 4.4 accordingly.
The E2/E3 result (Table 8) is now reported as an observation only. We agree that the data do not separate an E2×E3 interaction from a direct effect of E2, since adding E2 alone already lowers the Procedural score. We therefore report the measured change, a Procedural Δ of +0.085 for the full system and +0.108 without E2, without attributing it to either explanation, and we have removed the interaction explanation from the introduction, Section 4.4, Appendix D.2, and the conclusion.