- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CoDiMem couples memory-architecture search with updates to natural-language diagnostic guidance. An agent proposes hypotheses about memory failures, implements scoped code changes, and evaluates candidates against the incumbent through screening and confirmation stages. Trial records, including unsuccessful interventions, inform subsequent diagnosis. Experiments on AppWorld and MemoryArena with two backbones report improvements over the starting AWM architecture; on AppWorld with StepFun, accuracy increases from 59.03% to 69.44%.
Critical strengths
-
Well-structured search procedure. Linking hypotheses, executable modifications, observed memory changes, and paired task outcomes provides a disciplined approach to architecture evolution. Retaining unsuccessful trials for subsequent diagnosis is a useful design choice.
-
Useful empirical analysis. The paper combines held-out evaluation with search-cost accounting, a guidance-update ablation, and illustrative trial histories. These analyses provide insight beyond aggregate task accuracy.
Critical weaknesses
-
The central ablation does not isolate diagnostic-guidance learning. Section 4.4 acknowledges that the fixed-guidance variant’s access to previous trial records and round-specific recommendations is unspecified. Consequently, the reported improvement cannot be confidently attributed to guidance revision rather than differences in available historical information. This directly weakens evidence for the central contribution.
-
General-purpose reuse remains unestablished. The experiments demonstrate optimization within individual environments, without jointly developing one memory system across diverse benchmarks or transferring a fixed evolved architecture to an unseen environment. The results therefore leave unclear whether search discovers broadly reusable memory mechanisms or primarily environment-specific adaptations.
-
Practical value relative to integrated memory systems is unclear. The comparisons cover several research memory methods, but do not establish an advantage over representative agents using native memory mechanisms, such as Claude Code. Given the substantial offline search cost, evidence about deployment advantages and reuse across environments would materially strengthen the practical case.
Questions for authors
Clarification: Does the fixed-guidance variant receive exactly the same historical trial records and recommendations as full CoDiMem? If so, please specify how this is ensured.
Evidence request: Have you evaluated a fixed architecture evolved on one benchmark in another environment, rebuilding its memory bank but performing no further architecture search? Alternatively, is there evidence from joint development across benchmarks?
Evidence request: Are there comparisons with integrated native-memory agents under comparable execution and resource constraints? Such evidence would help assess practical benefits beyond the research baselines.