- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The main idea of this paper is to reuse successful mutation strategies across related tasks and domains. They introduce EvoMem, a mechanism for transferring mutation knowledge across LLM-based evolutionary program-search runs. After each run, it extracts and stores promising ideas, and retrieves relevant instructions for next task/domain. They show improvement across several tasks including geometric optimization, multi-hop question answering, GPU kernel optimization, and related benchmarks.
Critical strengths
- I think this paper focus in an very important question.
- The experiments show good improvement.
- The paper show clear ablation studies.
Critical weaknesses
- For my understanding, memory retrieval depends on semantic relevance, but I think semantic relevance does not necessarily lead to a beneficial mutation. memory retrieval. I think this is pretty important for cross-task knowledge transfer.
- The paper uses CER to measure search efficiency, but for my understnding this metric doesn't include the additional computational overhead. such as task filtering, multi-view retrieval, LLM-based selection...I think those may cause token consumption and latency. I don't think average token usage for the evolution and memory-generation phases is clear enough. For example, latency and embedding cost is not clear to me.
- I didn't find a mechanism in the framework for detecting or resolving conflicts between historical optimization strategies and the current task. A memory that improves performance in one task may degrade performance in another.
- Although the paper evaluates cross-task memory transfer. But I think the improvements may primarily bacasue of near-tasks rather than generalizable strategies across substantially different problems. So how does it truly tsnafer between tasks is still unclear. In the setting, I feel like there always be some similar tasks. I'm curious how it performs when the target task is dissimilar to all other tasks. So most retrieved memories may be irrelevant or even harmful.
- I think the idea of extracting successful experiences into memory and retrieving them to guide future decisions has been explored a lot in agent memory research area. I feel like EvoMem largely combines existing techniques. What fundamentally new methodological contributions are introduced is a bit unclear to me.
Questions for authors
See weakness
2 comments
Thank you for the clarifications. However, my main concerns remain unresolved, especially W2 total computational overhead, W4 transfer without very related tasks. I think the novelty also remains unclear. I will maintain my original score.
Thank you for your detailed comments on how memory is selected, transferred, and evaluated. We share your concern that semantic relevance alone does not ensure a beneficial mutation. EvoMem therefore uses an agentic retrieval procedure that considers the current task, metric, parent program, and mutation context, as described in Section 3.4 and Appendix D.3. The agent first identifies potentially relevant source tasks and then uses keyword search, direct identifier lookup, and vector search over several card fields. Task summaries, mechanism descriptions, explanations, and metadata provide complementary information for this process. After combining the results, the agent assesses whether they are sufficient and may issue a focused follow-up query within a fixed iteration budget.
The selected advice then passes through a second level of assessment in the evolutionary loop. The mutation model can adapt or disregard a recommendation, and every resulting candidate undergoes the same correctness checks and objective evaluation as in the baseline. These checks and the subsequent population selection provide the practical safeguards when a historical strategy is unsuitable for the current task. Lineage and usage statistics preserve the evidence produced by this process: origin analysis considers parents, alternative children, and descendants, while usage tracking records per-task counts, fitness changes, and their medians. Post-run analysis retains this history alongside the idea, allowing its description to be interpreted in light of the conditions under which it was associated with improvement or deterioration.
This more involved selection process also brings the additional costs raised in your review. CER captures the number of evaluated candidates needed to reach a reference score, so assessing overall time or monetary savings requires considering memory construction and retrieval as well. In five completed geometric source runs, construction after evolution took a median of 5.49 minutes and a mean of 8.32 minutes. These measurements cover preparation of the bank from completed results and exclude the source evolution itself; the resulting bank can then be reused across later runs. Retrieval adds recurring costs for task filtering, query planning, embedding computation and search, and LLM-based selection and refinement. We use all-MiniLM-L6-v2 for embeddings, as specified in Appendix G. The phase-level token averages in Section 4 describe LLM usage, while latency also depends on model response times, retrieval, and execution scheduling. The practical trade-off is therefore whether the savings in candidate generation and evaluation outweigh these additional operations.
The Kissing Number case helps illustrate what the selection procedure retrieves when experience from different domains is combined. Our big bank audit covers 2,517 cards, with KernelBench accounting for the majority of cards. Banks from Circle Packing transfer experiments make up 9.9% of the corpus but supply 79.4% of card-selection events for Kissing Number 11D and 82.6% for Kissing Number 12D, including repeated retrievals. The selected cards describe multi-start SLSQP with perturbed initializations, vectorized pairwise-distance constraints, diverse initial configurations, and constraints expressed through squared distances. These are numerical optimization and constraint-handling strategies that can apply across distinct geometric formulations. The example thus illustrates how retrieval concentrates on relevant mechanisms within a heterogeneous bank, with shared geometric constraints providing a basis for transfer from circle packing to high-dimensional sphere configurations.
This connection between an idea, its evolutionary history, and its later use is also central to the methodological contribution we aim to make. EvoMem organizes memory around reusable mutation ideas linked to source programs, task context, introduction events, and subsequent outcomes. It consolidates those ideas across runs and selects advice in the context of the current task and parent program. By retaining the identities of selected cards, it also allows later analysis to connect their use with observed fitness changes. Our focus is on making this reuse of mutation knowledge traceable and informed by evolutionary evidence, while preserving the evaluator and selection process that determine which programs succeed on the target task.