acceptodds
Under review as a conference paper at ICLR 2027

EvoDisco: Strategy Discovery for Self-Evolving Agent Memory

Abstract

Large language model agents accumulate experience in memory, and how that memory evolves during interaction is as important as what it stores. Recent methods update memory online, yet both the operators that transform memory and the rules that select them remain fixed by design. We propose EvoDisco, which formulates agent memory evolution as evolutionary strategy discovery: the agent must discover which operator to apply, when to apply it, and how to instantiate it, rather than rewriting memory under a static operator schedule. A learnable policy conditions on the current memory state and recent task history, then selects and instantiates one of four semantic operators—Abstract, Refine, Bridge, and Reframe—which respectively compress related memories, decompose overloaded entries, connect weakly linked experiences, and reorganize retrieved reasoning chains. The policy is updated with a bi-level fitness. An inner signal scores each transformation using a retrieval-quality improvement regularized by memory complexity. An outer signal scores strategic efficacy over a window of past decisions, combining historical operator-utility regret, utility-prediction error, and selection diversity. Under a one-pass online plus frozen-transfer protocol on ALFWorld, WebShop, and LifelongAgentBench (OS and DB), EvoDisco improves over strong memory baselines including MemP and MemRL. The largest gains are on ALFWorld ( online / frozen-transfer points over MemP; / over MemRL). Ablations attribute these gains to both fitness levels and the operator library.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
llm agents, agent memory, memory evolution, strategy discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bopWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoDisco learns a strategy for reorganizing graph-based agent memory while keeping the backbone LLM frozen. A lightweight policy selects an operator, an editing target, and instantiation parameters from a predefined library of Abstract, Refine, Bridge, and Reframe. Learning uses immediate retrieval-quality and complexity feedback, together with a window-level strategic-efficacy signal. Experiments on ALFWorld, WebShop, and LifelongAgentBench OS/DB report improvements over memory baselines, with the largest gains on ALFWorld.

Critical strengths
  1. Relevant problem and concrete formulation. Adapting how memory is organized is a worthwhile direction. The paper provides an explicit action space and learning procedure for state-dependent memory editing.

  2. Encouraging results and supporting analysis. The ALFWorld improvements are substantial. Component ablations, fixed-operator comparisons, qualitative edits, and policy-transfer experiments help characterize the system beyond aggregate performance.

Critical weaknesses
  1. The necessity of the learned controller is not established. A natural alternative is to provide an LLM with the current memory state, previous edits, and observed feedback, and ask it to select the next edit directly. Comparisons against fixed or random selection do not establish that training an additional MLP is preferable to this approach.

  2. The scope of strategy discovery is restricted. Operator categories, candidate-generation rules, parameter ranges, and evolution triggers are manually specified. The policy learns choices within this space rather than discovering new operations or learning when to edit. These restrictions may be reasonable, but their benefits and limitations are not sufficiently examined relative to more flexible LLM-directed editing.

  3. The fitness proxy requires stronger validation. Immediate utility uses changes in spreading-activation scores and graph complexity rather than downstream task improvement. Higher activation scores do not necessarily imply more useful retrieved information. The paper does not sufficiently establish that this proxy reliably identifies edits that improve task performance.

  4. The cost analysis does not justify the architectural choice. End-to-end runtime combines memory maintenance with task execution, and successful episodes can terminate earlier. Consequently, the reported speedups do not isolate controller efficiency. Module-level latency, token usage, and a matched MLP-versus-LLM selector comparison are needed to assess the claimed practical advantage.

  5. Limited evidence beyond the evaluated configuration. Experiments use only GPT-5-mini, and gains outside ALFWorld are comparatively small. There are also no comparisons with complete agents using native memory management, such as Claude Code. The results therefore leave uncertainty about the method’s benefits across backbone capabilities and deployed agent systems.

Questions for authors
  1. Evidence request: How does the MLP compare with an LLM selector given the same state information, operator library, and historical feedback? This would directly test the controller’s necessity.

  2. Clarification: Is there evidence that the retrieval-quality proxy correlates with downstream task improvements, rather than merely increasing graph activation scores?

  3. Evidence request: Can the reported cost be separated into task execution, memory writing, operator selection, and edit generation?

  4. Scope clarification: What motivates the four-operation library, and which useful memory transformations cannot be represented within it?

Reviewer_Qef4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EvoDisco trains a small policy to select semantic edits to a frozen LLM agent's memory graph. Selection covers four operators, graph candidates, and edit parameters, using retrieval-activation utility and an outer feedback signal. Experiments evaluate online learning, frozen-memory transfer, and policy transfer. The strongest gains occur on ALFWorld; margins on other benchmarks are smaller.

Critical strengths
  • The method offers a practical way to adapt memory organization while keeping the language model frozen.
  • The reported gains on ALFWorld are substantial.
  • The paper clearly separates online memory construction from evaluation with frozen memory.
Critical weaknesses
  1. Evaluation coverage is limited, and baseline adaptations need clarification. All experiments use GPT-5-mini on only three benchmark suites. (p.s. OS and DB are two domains of LifelongAgentBench). Episodes are capped at 15 steps, or 50 for ALFWorld. Without a dedicated evaluation on more complex, longer-horizon agent workflows, scalability beyond these bounded tasks remains unestablished. Appendix A.3 implements Self-RAG as a frozen-model retrieval decision and keep/drop filter, omitting its original trained reflection-token mechanism. This adaptation should be clearly labeled, with its prompts and departures from the original method documented.

  2. The ablations change several components at once. Removing outer fitness replaces all learned selection with uniform sampling; removing inner fitness also changes predictor targets and eliminates positive-utility replay. Fixed-operator variants disable target/parameter policy-gradient updates. To isolate the contributions, useful controls would retain the same memory system without evolution, train the selector directly on immediate utility, or fix the operator while retaining target/parameter learning.

  3. The smaller gains lack uncertainty estimates. Only seed 42 is specified, with no reported variation across runs or confidence intervals. In frozen-memory evaluation, Full exceeds Fixed Abstract by about four successes out of 140 ALFWorld tasks; gains over the strongest baselines on OS, WebShop, and DB amount to roughly one or two successes each.

  4. Wall-clock time alone is insufficient to assess computational cost. All methods traverse the online stream once. On ALFWorld, No Memory takes 36.8 hours versus EvoDisco's 21.6 despite having no memory-maintenance overhead. The authors' early-termination explanation needs support from actual interaction counts and a breakdown of execution and maintenance costs. A consistent comparison should report these metrics, input/output tokens, LLM/embedding calls, and GPU-hours where applicable for EvoDisco and every baseline, with hardware and concurrency specified.

  5. The feedback and Reframe semantics need validation. Equation 6 uses top-three activation scores without establishing their relationship to task success. Reframe increases edge weights, potentially improving this proxy without improving retrieval usefulness. Its reverse branch also writes a reversed causal relation despite the instruction to preserve facts. The paper should distinguish factual relations from retrieval directions and validate that these edits preserve useful knowledge.

  6. Reproducibility and independent verification remain concerns. Anonymized supplementary code is permitted and encouraged, but code and configurations are promised only after publication. Appendix A.5 leaves state-history windows, normalization, and variable-size feature encoding unspecified. The Self-RAG adaptation raises a related concern: without code and complete prompts, its fidelity to the original method and the fairness of the comparison are difficult to verify. Given the limited experimental coverage, these gaps and the absence of per-run records weaken confidence in the reproducibility and reliability of the findings.

  7. Minor comment. The abstract's claim to learn “when” to evolve memory suggests that the trigger itself is learned. In practice, evolution is triggered by a fixed episode interval or a score-drop threshold; the policy learns which operator, target, and parameters to select after triggering. Please clarify this distinction in the abstract, consistent with the conclusion's acknowledgment that the trigger is heuristic.

Questions for authors
  1. Do the benefits extend to another backbone and more complex, longer-horizon agent tasks? How were the baseline adaptations validated, particularly Self-RAG?
  2. Can the controls above isolate operator selection and the outer objective, with repetitions across seeds and task orders for the strongest comparison?
  3. In Figure 2(c–d), why do several cumulative success-rate curves reach their highest value at the first plotted point and then decline?
  4. Please compare computational costs for EvoDisco and all baselines under matched conditions, covering interaction counts, input/output tokens, LLM/embedding calls, and GPU-hours where applicable. Specify hardware and concurrency, separate execution from memory-maintenance costs, and explain No Memory's longer runtime.
  5. Does activation-based utility predict downstream improvement, and what do the reversed causal edges in Reframe represent?
  6. Can runnable code and full configurations be provided for independent verification?
Reviewer_LRtYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a learning method for how to organize an LLM agent memory graph. They employ an MLP to choose from a predefined set of memory operators while the backbone LLM remains frozen and uses the evolving memory to solve new tasks. The learning signals encourage retrieval similarity and penalizes graph complexity. On several agent benchmarks, their method achieves high performance exceeding other memory baselines.

Critical strengths
  1. Learning to produce better memory representations is important for developing self-evolving agent. The paper presents a sensible solution to this problem.

  2. The learned policy is lightweight and can be applied to any frozen model at test time.

  3. Strong results, especially on the held-out set, demonstrate the effectiveness of the method.

Critical weaknesses

I had concerns regarding insufficient ablations of its complex reward deigns.

  1. The reward design is quite complex, combining three terms: Regret, prediction error, and diversity, but they are not separately ablated. Additionally, they do not compare training the policy directly with immediate utility UU against training it with their outer reward EE.

  2. The paper uses the difference of retrieval activation before and after edits as immediate reward, but this does not necessarily mean more relevant or useful memories. The author could compare this with other proxies for retrieval relevance like an LLM-Judge. The importance of the graph complexity penalty term is also not ablated.

Questions for authors
  1. Can you compare your method with simple alternatives such as ReasoningBank and SYNAPSE?

  2. Can you show an example of how the memory graph evolves over time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.