acceptodds
Under review as a conference paper at ICLR 2027

Capability-Driven Self-Evolution of Agent Memory

Abstract

Memory self-evolution uses task feedback to iteratively improve executable memory programs that store and retrieve information from past interactions. Existing approaches typically adopt holistic evolution, deriving revision directions from mixed feedback and judging progress by overall performance. This can obscure optimization directions and hide capability-specific gains offset by regressions elsewhere, leaving promising directions underexplored. We introduce *capability-driven evolution*, which extends search guidance from overall performance to individual capability dimensions, preserving promising revisions and expanding exploration beyond the boundaries of holistic evolution. We propose PrisMem, which uses dependency-aware capability selection to prioritize targets with potential cross-capability benefits and history-guided diagnosis to refine capability specialists. Trace-guided integration compares evaluated programs on paired differential cases, using their behavioral differences to consolidate complementary gains into a unified memory program. Experiments show that PrisMem outperforms the strongest baselines by 10.54 and 7.83 percentage points on BEAM-1M and LongMemEval-M, respectively, demonstrating its effectiveness on million-token histories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent memory, Self evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fUa2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PRISMEM, a capability-driven framework for self-evolving agent memory programs. Rather than selecting revisions solely by aggregate task performance, PRISMEM organizes evolution around five predefined memory capabilities: factual retrieval, temporal tracking, preference extraction, multi-session synthesis, and adversarial handling. It first performs a cold-start stage to obtain a balanced initial program, then maintains capability-specific best-performing programs, selects target capabilities using a dependency-aware priority score, and uses history-guided failure cases to refine the corresponding specialist. Finally, it compares the execution traces of specialists and the overall-best program on differential cases to integrate complementary improvements into one unified memory program. The authors evaluate PRISMEM on BEAM and LongMemEval. Programs are evolved on shorter-context splits and evaluated on million-token test splits. PRISMEM outperforms the strongest self-evolving baselines by 10.54 percentage points on BEAM-1M and 7.83 points on LongMemEval-M.

Critical strengths

1. Strong empirical gains. PRISMEM reports substantial improvements over both static and self-evolving memory baselines on BEAM-1M and LongMemEval-M.

2. Broad empirical analysis. The paper includes component ablations, efficiency analysis, cross-model evaluation, transfer experiments, and detailed evolution traces, providing useful evidence for how the system evolves and where the gains come from.

Critical weaknesses

1. Unvalidated dependence on capability annotations. PRISMEM relies on a predefined five-capability taxonomy and GPT annotations of primary and secondary capabilities. These labels directly determine specialist selection, capability scheduling, and diagnostic case sampling. The paper does not assess annotation quality, agreement across annotators or models, sensitivity to label noise, or robustness to alternative capability definitions. It is therefore difficult to determine whether the reported gains arise from the proposed evolution framework or from a particular, potentially benchmark-specific labeling scheme.

2. Limited methodological novelty. Relative to existing self-evolving agent memory methods, PRISMEM proposes to organize program evolution around predefined memory capabilities. This is a coherent alternative to aggregate score driven evolution, but it primarily changes how the search process is organized rather than introducing a substantially new form of memory evolution. Moreover, recent work such as ModularRSI has already proposed the closely related high-level paradigm of decomposing a monolithic agent system, evolving the parts independently using execution trajectories, and integrating the resulting improvements. PRISMEM changes the decomposition axis from functional modules to memory capabilities, but this difference is not sufficiently developed or validated to establish a distinct conceptual contribution at the ICLR level.

Questions for authors

MemPro appears to be a closely related program-level self-evolving memory method, but it is not included in the empirical comparison. Is there a technical or protocol-related reason why a controlled comparison was not feasible? If feasible, could the authors compare PRISMEM and MemPro under matched backbone models, evolution budgets, and evaluation splits? This comparison would materially clarify whether capability-driven evolution improves over existing version-tree-based memory program evolution.

6 comments
AuthorsOpenReview ↗

Response to Reviewer fUa2 (1/3)

We appreciate your positive assessment of our results and broad empirical analysis. We are grateful for your valuable feedback, which we address in detail below.

W1: Dependence on capability annotations.

PrisMem's gains persist across annotation models, label perturbations, and alternative capability definitions; they are not tied to the original five-capability definitions.

The purpose of capability annotation in PrisMem is to group tasks with similar memory requirements and optimization directions, making diagnostic feedback more focused and allowing improvements to be recognized and further developed. This principle does not depend on a benchmark-specific labeling scheme. The five capabilities in our paper are one concrete instantiation. We have applied the same annotation scheme to BEAM and LongMemEval and achieved strong results on both (Table 2), with transfer experiments supporting its generalizability (Appendix C.2). We also evaluate PrisMem under the same scheme on AMA-Bench [1], a non-conversational memory benchmark based on agent action–observation trajectories. PrisMem achieves 40.41% accuracy, outperforming M* (26.66%) and EvolveMem (13.44%), further suggesting that PrisMem's gains stem from the evolution framework rather than benchmark-specific task labeling.

[1] AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications. Yujie Zhao, Boqin Yuan, et al. ICML 2026.

To further show that the benefits do not depend on fixed capability definitions, we conduct additional experiments on annotation models, label noise, and alternative capability definitions. All experiments use Qwen3.8-27B, the same BEAM splits, and a 20-round evolution budget.

1. Different annotation models

Keeping the capability definitions and annotation procedure unchanged, we independently annotate the tasks using Qwen3.8-27B. The resulting capability labels achieve 91.79% agreement with the GPT-5.5 annotations. Despite minor differences in task classifications, the two annotation models yield nearly identical evolution results, demonstrating robustness to annotator choice.

Annotation modelOverall score (%)
GPT-5.5 (default)63.39
Qwen3.8-27B63.33

2. Different levels of label noise

PrisMem is robust to moderate annotation errors: 20% label noise has little impact, while performance under 50% noise decreases but remains above M* (52.85%) and EvolveMem (40.01%). This tolerance stems from two design aspects. First, most tasks within each group retain similar memory requirements despite a few misclassifications, preserving sufficiently consistent responses to revisions for identifying capability-specific improvements. Second, every candidate is evaluated across all capability groups, allowing improvements triggered by mislabeled tasks to be recognized under their actual capabilities and further developed in the corresponding specialists. Together, these mechanisms mitigate the impact of individual misclassifications.

Label noiseOverall score (%)
0% (default)63.39
20%62.93
50%59.76

3. Alternative capability definitions

We construct two alternative definitions with different granularities:

  • Definition A (3 capabilities): Information retrieval and application, covering the retrieval and use of explicit facts, preferences, and constraints; Relational reasoning, covering temporal relations, state changes, and cross-session synthesis; and Evidence assessment, covering evidence sufficiency, conflicts, and abstention when necessary.
  • Definition B (7 capabilities): Factual retrieval, temporal reasoning, state updates, preference extraction, multi-session synthesis, conflict resolution, and abstention under insufficient evidence. Compared with the default five capabilities, this further splits temporal tracking and adversarial into more specific sub-capabilities.
Capability definitionOverall score (%)
Definition A (3 capabilities)61.66
Definition B (7 capabilities)63.50
Original 5 capabilities63.39

Both alternative definitions achieve strong results, indicating that PrisMem's benefits are not tied to the specific five capabilities. The coarser three-capability definition yields a slightly lower score, as grouping broader, more diverse requirements within one group may reintroduce ambiguous optimization directions and hidden capability gains. Nevertheless, it still substantially outperforms holistic evolution baselines, demonstrating PrisMem's effectiveness under different capability definitions.

Overall, these results support that capability classification provides the organizational basis for PrisMem's evolution paradigm, serving as a way to organize tasks with similar memory requirements and optimization directions, rather than a benchmark-specific labeling scheme on which the gains depend. The robustness experiments are included in Section 4.5 and Table 5 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer fUa2 (2/3)

W2: Methodological novelty and relationship to ModularRSI.

Contribution of PrisMem. PrisMem's contribution lies in identifying an overlooked cause of stagnation in memory evolution, where overall performance masks valuable capability-level progress, and introducing a systematic approach to identify such progress, guide subsequent optimization, and integrate complementary improvements into a unified program.

Self-evolving memory methods study how agents leverage task experience to improve memory construction, organization, retrieval, and use. A central challenge is how to identify valuable progress, determine subsequent optimization directions, and accumulate effective implementations. By analyzing the evolution trajectories of existing methods, we identify a structural source of stagnation, which motivates our capability-driven paradigm. Specifically, agent memory must satisfy diverse capability requirements, yet one modification can affect different capabilities differently (Fig. 1b). Consequently, mixed feedback across capabilities obscures revision directions, while overall judgment hides capability-level progress. In our analysis, 80.5% of revisions that do not improve the overall score still improve at least one capability (Fig. 1a). This reveals that stagnation in overall performance does not imply stagnation in capability development; valuable improvements may remain hidden without further refinement. Thus, the key question is how to turn such capability-level progress into systematic improvements.

Motivated by this insight, PrisMem introduces a new paradigm for self-evolving memory systems by expanding the evolution space along multiple capability dimensions and treating capability gains as valuable intermediate progress. It grounds diagnosis in specific capability requirements, continues refinement from the corresponding specialists, and leverages execution differences between specialists to integrate complementary improvements into a unified memory program. This provides a new approach to identify valuable capability-level progress and develop promising directions that holistic judgment would overlook. Our ablations support the importance of such designs: removing capability-driven evolution and trace-guided integration reduces performance by 10.15 and 3.16 pp, respectively (Table 4). Figure 3 further shows that PrisMem expands the exploration boundary beyond holistic evolution and achieves stronger overall performance. Therefore, our contribution is a new understanding of evolution stagnation and a complete method for identifying, developing, and integrating otherwise overlooked capability-level improvements.

Difference from ModularRSI. We acknowledge the conceptual similarity that both works decompose the evolution process, yet they target different scenarios and evolution bottlenecks, and are complementary. ModularRSI proposes functional modular decomposition for mechanism-level credit assignment in evolving programming and terminal-use agent harnesses. In contrast, PrisMem addresses the heterogeneous effects of revisions across capabilities in agent memory evolution, which remain important even after an issue is localized to a functional component.

For example, expanding the retrieval module may improve multi-session synthesis by retrieving more cross-session information, but also introduce irrelevant personal information that interferes with answering. Even when the issue and modification are localized to the retrieval module, this does not clarify whether or how the revision should be further refined: improvements in one capability may be offset by regressions in others, making the optimization direction ambiguous. Moreover, the overall score may fail to reflect capability-specific gains when these opposing effects cancel out, causing useful improvements to be overlooked before they can be further developed and integrated.

PrisMem addresses this challenge by identifying valuable revisions through capability-level evaluation and continuing refinement from the corresponding specialists, allowing capability-level progress to be developed further. Functional modularity provides controlled boundaries for implementation changes, whereas capability-driven evolution determines which improvements deserve continued development and how their benefits can be consolidated. These principles operate at different levels and can complement each other within the same system, which would be an interesting direction for future work.

Finally, we appreciate you bringing ModularRSI to our attention. First released on arXiv on September 14, 2026, it falls within the two-month concurrent-work window before the submission deadline under ICLR guidelines [2]. We cited this work and clarified the relationship and distinctions between the two methods in Appendix F of the revised paper.

[2] https://iclr.cc/Conferences/2027/ReviewerGuidelines#:~:text=A%3A%20We%20consider,available%20on%20arXiv.

AuthorsOpenReview ↗

Response to Reviewer fUa2 (3/3)

Q1: Controlled comparison with MemPro.

M*, one of the baselines in our paper, is a representative program-level memory evolution method, just like MemPro. Both organize evolution based on overall performance, fundamentally differing from PrisMem's capability-driven approach. We added the matched comparison with MemPro to better illustrate the advantages of capability-driven evolution. All methods use Qwen3.8-27B as the backbone model, with the same 20-round evolution budget and BEAM data split. The results are shown below.

MethodOverall Score (%)Avg. #Tokens (K)
M*52.8589.47
MemPro52.92313.78
PrisMem63.39100.59

MemPro's version-tree-based evolution performs similarly to M*, while PrisMem outperforms both. This comparison further supports the benefit of capability-driven evolution beyond holistic, version-tree-based evolution. The results are included in Appendix C.5 of the revised manuscript.

Reviewer_7YQ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Memory self-evolution uses task feedback to iteratively improve executable memory programs that store and retrieve information from past interactions. This paper studies an important problem for self-evolution of agent memory. Existing approaches typically adopt holistic evolution, deriving revision directions from mixed feedback and judging progress by overall performance. This can obscure optimization directions and hide capability-specific gains offset by regressions elsewhere, leaving promising directions underexplored. This study proposes PRISMEM which uses dependency-aware capability selection to prioritize targets with potential cross-capability benefits and history-guided diagnosis to refine capability specialists. It separates evolution into three stages: (1) cold start: which is to improve a seed memory program using feedback across multiple capabilities; (2) capability refinement which is to select capabilities according to their weaknesses and estimated dependencies, then refine capability-specific specialist programs using history-guided failure diagnosis; and (3) trace-guided integration, which is to compare specialists and the overall best program on paired examples, then integrate complementary improvements into a unified memory program. Trace-guided integration compares evaluated programs on paired differential cases, using their behavioral differences to consolidate complementary gains into a unified memory program. It considers five capabilities: factual retrieval, temporal tracking, preference extraction, multi-session synthesis, and adversarial handling. Experiments show that PRISMEM outperforms the strongest baselines by 10.54 and 7.83 percentage points on BEAM-1M and LongMemEval-M, respectively, demonstrating its effectiveness on million-token histories.

Critical strengths

S1: It is interesting to raise an important question on the current memory self-evolution methods: Existing approaches typically adopt holistic evolution, deriving revision directions from mixed feedback and judging progress by overall performance, which can obscure optimization directions and hide capability-specific gains, leaving promising directions underexplored. This study works on this direction and achieves good improvements in comparison with the current agent memory methods.

S2: The three-stage design (cold start, capability refinement and trace-guided integration) is interesting and clearly designed.

S3: The performance experiments show clear and very good results.

S4: The paper is clearly written with step-by-step reasoning.

Critical weaknesses

W1. The reported gains demonstrate the effectiveness of the complete system but may still not be convincing that it has really uncovered which problem comes from which part of the program. That is, the partition of the three stages (cold start, capability refinement and trace-guided integration) may help debug the program and enhance the overall performance. But it seems to me it may still not be able to locate the source of the problem. Maybe some good reasoning and isolated experiments may help convincing the approach reaches the initial goal in a convincing way.

W2: The evaluation using two benchmarks are interesting but it seems to testing on the evolution aspects are relatively small. Maybe larger evolution history datasets may lead to more convincing results.

Overall, I feel the study itself is innovative and interesting. But I feel without carefully structuring the agent memories and evolution history, the partitioning of the program/history into the current three stages could improve the overall performance but may still have a hard time to identify the fine-grained root cause, as initially proposed goal in this study. Some discussion may help convince me on this point.

Questions for authors

Please address the weakness pointed out in the last section in the rebuttal.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7YQ7

We are glad you find our approach innovative, its design clear, and its performance strong. We appreciate your suggestions on failure localization and larger-scale evaluation, and address each point below.

W1: How PrisMem localizes program failures.

Fine-grained problem localization comes from comparing execution traces with task requirements and the corresponding code, while the three stages organize what to improve and when. We have added further explanations in Section 3 and detailed case studies in Appendix E illustrating actual execution traces.

PrisMem aims to address two limitations of holistic evolution: mixed feedback makes optimization directions ambiguous, while overall evaluation can mask local capability improvements. The three evolution stages are therefore designed to organize capability-level exploration, expanding the evolution boundary. Within this process, PrisMem localizes failures by tracing how task-relevant information is processed across memory components, identifying where execution falls short of capability requirements, and examining the corresponding code to pinpoint the underlying implementation issues.

The memory interfaces make failures inspectable. Each memory program exposes five components: Extraction, Indexing, Planning, Retrieval, and Answer (Paragraph 2 of Section 3; Appendix A.5). Their intermediate outputs provide explicit boundaries for locating failures. In each round, the Reflection Agent examines diagnostic cases containing reference evidence, extracted memories, query plans, retrieved content, and reference/predicted answers (Appendix A.3), alongside the current implementation and recent revisions. Like a developer debugging a program, the Agent first analyzes case outcomes to identify where information processing breaks down, then inspects the corresponding code to pinpoint the implementation issue and propose targeted modifications. For example, missing information in extracted memory points to extraction issues; information preserved in memory but absent from retrieval points to indexing or retrieval problems; and sufficient retrieved evidence paired with an incorrect answer points to answer interpretation.

Capability-focused cases make diagnosis more specific. The same component can require different changes across capabilities. PrisMem addresses this by grounding diagnosis in specific capability requirements. First, history-guided diagnosis in stage II prioritizes cases that previous programs handled well but the current program fails, providing informative regression evidence while reducing redundant case selection (Section 3.1.2). Second, paired differential cases in stage III compare the base program and a capability specialist on the same tasks, examining differences in their query plans, retrieved content, answers, and code to identify which implementation changes explain their performance gap (Section 3.1.3). Replacing this selection with random cases reduces performance by 5.26 pp (Table 4), supporting the value of informative, targeted diagnostic cases.

Appendix E presents two case studies on temporal provenance loss and cross-cutting preference retrieval, illustrating the full process from identifying execution gaps and tracing them to implementation issues to making targeted code revisions and observing the resulting behavioral changes. In summary, the three stages organize what to improve and when, expanding the evolution boundary beyond holistic evolution, while fine-grained problem localization comes from comparing what each memory component produces, what the task requires, and how the current code implements it.

W2: Evaluation with a larger evolution dataset.

PrisMem also performs well on a larger dataset. We conducted additional experiments on AMA-Bench [1], which extends the history to agent action–observation trajectories across six domains: web interaction, software engineering, Text-to-SQL, open-domain QA, embodied tasks, and games. We use 1,752 questions for evolution, substantially more than the 280 and 120 questions used for BEAM and LongMemEval, and a separate 744 questions for testing, with an average history length of ~70.5K tokens. We retain the same capability definitions, annotation procedure, and Qwen3.8-27B backbone model. The results are shown below.

MethodOverall Score (%)Avg. #Tokens (K)
Mem011.0722.83
A-MEM23.2150.49
HippoRAG229.7034.98
SimpleMem12.6322.16
LightMem7.088.60
M*26.6614.19
EvolveMem13.4417.55
PrisMem40.4114.10

PrisMem also outperforms baselines on AMA-Bench, achieving a 10.71 pp gain over the strongest baseline. These results extend the evidence beyond conversational histories, demonstrate effective evolution on a larger dataset, and are included in Appendix C.4 of the revised manuscript.

[1] AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications. Yujie Zhao, Boqin Yuan, et al. ICML 2026.

Dear Reviewer 7YQ7,

Thank you again for your valuable comments. As the reviewer--author discussion phase is approaching its end, we would be grateful to know whether our response has addressed your concerns or whether there are any remaining questions we could clarify. We would be happy to use the remaining discussion time to address any further questions. Thank you for your time and feedback.

Sorry for my late reply. In general, the rebuttal answered some of my questions. However, I feel the weakness W1 is still not clearly answered. In particular, my comments: ** the partition of the three stages (cold start, capability refinement and trace-guided integration) may help debug the program and enhance the overall performance. But it seems to me it may still not be able to locate the source of the problem**. I feel the source of the problem should be analyzed from the source and semantics/structures of the program itself, instead of just based on the program response from the outside. I would still like to see some good reasoning to help me comprehend and convince the power of the approach.

For benchmark and performance testing, I saw there are some recent methods, such as Memora and RippleMem, which outperform the methods you tested as baselines, such as Mem0, A-MEM, HippoRAG2, and SimpleMem. I am wondering whether those methods can fit your application domain and whether you can outperform them. I mainly want to understand why instead of just checking the numbers.

Currently, I will still maintain my current score. But please submit some further rebuttal on my questions.

Reviewer_KP5QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PrisMem, a capability-driven self-evolution framework for Agent Memory. This approach retains and integrates effective improvements in different memory capabilities through dependency-aware capability selection, history-guided failure diagnosis, and trace-guided integration. On BEAM-1M and LongMemEval-M, PrisMem outperforms the strongest baseline by 10.54 and 7.83 percentage points respectively, and the effectiveness of the method is verified through ablation and transfer experiments.

Critical strengths
  1. The paper points out that overall performance evaluation may mask effective improvements in specific memory capabilities. By maintaining the capability-specific specialist programs and trace-guided integration, a more comprehensive self-evolution framework is proposed, with a clear research motivation.

  2. The experiments achieved substantial improvements on two long-term memory benchmarks, and provided multiple runs, component ablation, evolutionary trajectories, and cross-model and cross-dataset transfer results, which have good experimental support.

Critical weaknesses
  1. The robustness of the capability classification is insufficiently verified. Appendix A.1 defines five types of memory capabilities based on the existing Benchmarks, but lacks experiments with different classification granularities, random task grouping, and label perturbations. Since the capability classification directly affects the search direction, it is unclear whether the method relies on a specific capability definition.

  2. The independent contribution of the core search mechanism is not sufficiently isolated. Table 4 shows that removing capability-driven evolution leads to a substantial performance decline, but this ablation study does not sufficiently distinguish the benefits brought by semantic capability classification and multi-branch candidate retention. Although the authors compared Round-Robin, there is still a lack of direct comparison with simple error rate priority scheduling or general multi-branch search strategies.

Questions for authors
  1. Can the performance stability of PrisMem under different numbers of capabilities, classification methods, and label noise be evaluated?

  2. Can random task grouping or a general multi-branch program search baseline be added to verify the independent value of semantic capability division?

  3. Can the dependency-aware capability selection be compared with a simple error rate priority strategy to further prove its necessity?

3 comments
AuthorsOpenReview ↗

Response to Reviewer KP5Q (1/2)

We are pleased by your recognition of our clear motivation and strong empirical support. Thank you for your suggestions on classification robustness and controlled ablations. We address each point below. All experiments use Qwen3.8-27B, the same BEAM splits, and a 20-round evolution budget.

W1/Q1: Robustness of the capability classification.

PrisMem remains effective across different capability granularities, classification methods, and moderate label noise. The purpose of capability classification in PrisMem is to group tasks with similar memory requirements and optimization directions, making diagnostic feedback more focused and allowing improvements to be recognized and further developed. This principle does not depend on a unique or perfectly accurate classification. The five capabilities in our paper are one concrete instantiation. To evaluate capability classification robustness, we conduct additional experiments along three dimensions: granularity, classification method, and label noise.

1. Different numbers of capabilities

We construct two alternative classifications with different granularities:

  • 3 capabilities: Information retrieval and application, covering the retrieval and use of explicit facts, preferences, and constraints; Relational reasoning, covering temporal relations, state changes, and cross-session synthesis; and Evidence assessment, covering evidence sufficiency, conflicts, and abstention when necessary.
  • 7 capabilities: Factual retrieval, temporal reasoning, state updates, preference extraction, multi-session synthesis, conflict resolution, and abstention under insufficient evidence. Compared with the default five-capability classification, this further splits temporal tracking and adversarial into more specific sub-capabilities.
#CapabilitiesOverall score (%)
361.66
5 (default)63.39
763.50

All three settings achieve similar results and outperform the baselines, indicating that PrisMem does not rely on the specific five-capability classification. Finer classifications provide more focused feedback but spread the limited budget across more directions, whereas coarser ones concentrate the budget but risk mixing tasks with different optimization needs. Thus, granularity balances diagnostic focus against exploration coverage, and PrisMem remains effective across a relatively wide range of granularities.

2. Different classification methods

To assess robustness across classification methods, we relabel the tasks using Qwen3.8-27B while keeping other settings unchanged. We also introduce a random-grouping baseline that randomly assigns tasks to five groups.

Classification methodOverall score (%)
GPT-5.5 annotations (default)63.39
Qwen3.8-27B annotations63.33
Random grouping52.64

The two LLM annotators yield nearly identical results, demonstrating robustness to annotator choice. In contrast, random grouping substantially degrades performance as it mixes tasks with different memory requirements, reducing capability-driven evolution to multi-branch holistic evolution, which obscures optimization directions and masks revision effects within each group.

3. Different levels of label noise

We randomly perturb the capability labels of 20%, 50%, and 100% of the training tasks.

Label noiseOverall score (%)
0% (default)63.39
20%62.93
50%59.76
100% (random grouping)52.64

PrisMem is robust to moderate annotation errors: 20% label noise has little impact, while performance under 50% noise decreases but remains above M* (52.85%) and EvolveMem (40.01%). This tolerance stems from two design aspects. First, most tasks within each group retain similar memory requirements despite a few misclassifications, preserving sufficiently consistent responses for identifying capability-specific improvements. Second, every candidate is evaluated across all capability groups, allowing improvements triggered by mislabeled tasks to be recognized under their actual capabilities and further developed. Together, these mechanisms mitigate the impact of individual misclassifications. In contrast, 100% random reassignment mixes tasks with different requirements, obscuring revision effects and reducing capability-driven evolution to multi-branch holistic evolution, which substantially degrades performance.

Overall, these results show that PrisMem does not require a unique or perfectly accurate capability classification; it only requires tasks with similar requirements and optimization needs to be grouped approximately together. Its effectiveness persists across different classification granularities and moderate label noise, while semantically coherent groupings make feedback more focused and capability-level progress easier to identify under a limited evolution budget. The robustness experiments are included in Section 4.5 and Table 5 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer KP5Q (2/2)

W2/Q2: Separating semantic classification from multi-branch retention.

Semantic capability classification and multi-branch candidate retention play complementary roles: the former clarifies diagnostic feedback and exposes capability-level progress, while the latter maintains local gains for further development. We added two controlled variants:

  • Random grouping + multi-branch evolution: replace capability labels with five random groups while retaining group-wise evaluation, specialist maintenance and selection, and integration.
  • Capability grouping + single-branch evolution: retain original capability labels, dependency-aware capability selection, and capability-focused diagnosis, but always refine the overall-best program rather than maintaining capability specialists.
SettingsOverall score (%)
PrisMem63.39
Random grouping + multi-branch evolution52.64
Capability grouping + single-branch evolution54.23

Random grouping loses 10.75 pp despite preserving multi-branch search, as it still mixes tasks with different memory requirements within each group, obscuring optimization directions and masking capability-specific improvements. Single-branch evolution loses 9.16 pp despite retaining capability-focused guidance: valuable capability-level revisions may not receive further development when they do not improve the overall score. Thus, semantic capability classification provides focused optimization directions, while multi-branch specialist retention preserves local gains for continued refinement. These ablations demonstrate the value of semantic grouping beyond general multi-branch search and we include them in Section 4.4 and Table 4 of the revised manuscript.

W2/Q3: Dependency-aware selection versus error-first scheduling.

Capability selection aims to prioritize directions with the greatest potential for improvement under a limited evolution budget. This requires considering not only current performance gaps, but also cross-capability bottlenecks. The highest error rate alone is insufficient: a difficult capability may remain the weakest across rounds, repeatedly consuming the budget while leaving other capabilities underexplored. Moreover, tasks often involve multiple capabilities, so focusing exclusively on one capability's error rate can overlook cross-capability bottlenecks and limit further improvements.

We added an error-first baseline, which selects the capability with the highest average training error in each refinement round, while keeping other settings unchanged.

Capability selection strategyOverall score (%)
Dependency-aware (default)63.39
Round-robin60.84
Error-first55.23

Error-first performs substantially worse than PrisMem's dependency-aware strategy. Its evolution trace reveals the limitation: all 12 capability-refinement rounds select multi-session synthesis since its error rate remains the highest, leaving other capabilities without direct refinement. In contrast, dependency-aware selection considers both a capability's own error rate and its potential contribution to failures in tasks primarily targeting other capabilities. This helps identify latent cross-capability bottlenecks and prioritize refinements that may benefit multiple capabilities, making more effective use of the limited evolution budget. The comparison is included in Section 4.4 and Table 4 of the revised manuscript.

Dear Reviewer KP5Q,

Thank you again for your valuable comments. As the reviewer--author discussion phase is approaching its end, we would be grateful to know whether our response has addressed your concerns or whether there are any remaining questions we could clarify. We would be happy to use the remaining discussion time to address any further questions. Thank you for your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank the reviewers for their constructive feedback, highlighting the following strengths:

  1. Clear Motivation and Innovative Design: identifying an overlooked bottleneck in holistic memory evolution where aggregate evaluation masks capability-specific gains (KP5Q, 7YQ7), addressed by an innovative, clearly designed three-stage evolution framework (7YQ7, KP5Q).
  2. Strong Empirical Gains: substantial improvements over baselines across long-term memory benchmarks (KP5Q, 7YQ7, fUa2), with strong practical potential (fUa2).
  3. Thorough Analysis and Clarity: clear, step-by-step presentation (7YQ7) supported by comprehensive ablations, efficiency and transfer studies, and evolution trajectories (KP5Q, fUa2).

The major concerns and our responses are as follows:

  1. Capability classification robustness (KP5Q, fUa2): New experiments across granularities, the Qwen3.8-27B annotator, and 20%/50% label noise confirm that PrisMem's effectiveness does not rely on a fixed taxonomy or noise-free labels.
  2. Core search ablations (KP5Q) and MemPro comparison (fUa2): We added random grouping and error-first scheduling ablations, isolating the roles of semantic grouping and dependency-aware selection (KP5Q). Under matched settings, PrisMem also outperforms the program-level version-tree-based baseline, MemPro by +10.47 pp (fUa2).
  3. Fine-grained failure localization (7YQ7): We clarified how PrisMem pinpoints component-level issues across five inspectable memory components via history-guided regression cases and paired differential traces, illustrated with end-to-end execution-trace case studies.
  4. Methodological contribution and ModularRSI (fUa2): We clarified that PrisMem's contribution lies in identifying an overlooked cause of stagnation in memory evolution and introducing a systematic approach to address it. We also explained that ModularRSI and PrisMem address different challenges in evolution and can complement each other.

Follow-up discussion:

  • Reviewer 7YQ7 acknowledged our new results and, near the close of discussion, asked about failure localization and comparisons with static methods (Memora/RippleMem). While both points lie outside our core contribution (resolving holistic-evolution stagnation via capability-driven evolution rather than fine-grained bug localization or static memory design), we promptly and thoroughly addressed them: clarifying that the three stages serve to organize exploration rather than act as a bug-localization mechanism, detailing step-by-step module-and-code failure diagnosis, and highlighting that PrisMem operates in a fundamentally different scope (i.e., self-evolving memory) with clear adaptability advantages over manually designed static methods (Memora/RippleMem).
  • Reviewer fUa2 confirmed that all empirical and prior-work concerns (including ModularRSI and MemPro) were resolved, keeping their reservation solely on PrisMem still following the standard five-step evolution loop. We clarified in three points that operating within this loop (shared across many works such as GEPA, M*, MemPro and ModularRSI, and never claimed as our contribution) is not a valid basis to discount novelty, whereas our actual contribution of uncovering and resolving cross-capability stagnation within this loop has, to our knowledge, not been explored in prior work.

Paper Revision Summary:

  1. Added the new controlled search ablations to Section 4.4 (Table 4) and capability-classification robustness experiments to Section 4.5 (Table 5).
  2. Clarified program failure localization in Section 3 and added two execution-trace case studies in Appendix E.
  3. Added the larger-scale AMA-Bench evaluation (Appendix C.4), MemPro comparison (Appendix C.5), and ModularRSI discussion and contribution clarifications (Appendix F).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.