- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MemCodex represents agent memory as a fixed hierarchy of executable layer programs that preserve links to the raw turns used to construct each item. It adapts the system through single-method rewrites or layer toggles, with queries routed from coarse memory representations to original evidence. An optional Metis-derived memory state augments the open-weight backbone, and MemArena provides a shared evaluation interface. Across four benchmarks and two backbones, the paper reports average improvements over MemPro of 5.6 and 3.6 percentage points, together with lower prompt-token use and online latency.
Critical strengths
Restricting edits to one layer method while retaining source-turn provenance provides an inspectable mechanism for changing memory behavior. The 4.0-point gain over the initial architecture and the comparisons of rewrite granularity provide useful evidence that the search design matters. The advantage over MemPro persists under record-disjoint evaluation at 5.0 points and without latent memory at 3.6 points, which strengthens the empirical case beyond the headline configuration.
Critical weaknesses
- Equation (3) allows token-saving edits with non-decreasing observed accuracy to bypass the significance requirement. Repeated testing reuses the evolution data without stated correction, leaving Appendix A's 5% bound on accepting null edits unsupported.
- MemCodex uses gold evidence for diagnosis, but equivalent access for the search baselines is unspecified. Differences in supervision could therefore explain part of the gains.
- At STOP, routing uses one item's source turns, whereas content uses up to ten derived items (Algorithm 1). The 60.8% versus 55.4% comparison therefore cannot isolate the effect of evidence preservation.
- Section 4.3 attributes at most 0.6 percentage points to previously seen records. Since Appendix C.2 also changes evolution sample size and test composition, the comparison cannot isolate this effect.
- Appendix C.3 measures online latency without memory construction, and Table 10 reports 87-118 GPU hours per search without corresponding baseline construction and search costs. The query volume needed to amortize these costs relative to the baselines remains unquantified.
Questions for authors
- For each of AutoMem, MemEvolve, and MemPro, were gold evidence annotations included in the meta-agent's feedback for the reported runs, and which prompt field or configuration records that choice?
- Was a controlled content-versus-routing comparison performed with the same evolved architecture, selected item, and token cap, changing only derived text versus source turns; if so, what was the test-accuracy difference?
5 comments
Author Response to Reviewer pxo3 (Part 2/3)
W3 and Q2. Source turns and derived text at matched token counts.
Source turns carry the evidence the reader needs. Adding derived text keeps recall high but lowers accuracy compared with routing alone.
At STOP, routing passes only the top item's in-scope source turns, src(x*). Adding candidate texts to those turns restored LoCoMo recall to 88.7%, but average accuracy fell to 59.6% at 4.6k Tok/q (Table 2).
For the reviewer's matched comparison, we held each benchmark's routing-evolved , read trace, STOP decision, selected item and token cap fixed. We changed only what the reader received at STOP. To match the length of text(x*), we ranked the item's source turns, kept as many as fit and returned them in time order. The submitted ten-item protocol is included for reference and is not part of the matched comparison.
| Context assembled at STOP | Avg. test accuracy (%) | Tok/q |
|---|---|---|
src(x*), temporal order | 60.8 | 2.0k |
text(x*) only | 56.9 | 1.1k |
Token-matched src(x*) | 59.6 | 1.1k |
text(x*) plus src(x*) | 60.2 | 2.6k |
text(X1..10), Table 2 protocol | 55.4 | 3.9k |
Source turns answered questions more accurately than derived text from the same item. At 1.1k Tok/q, they improved average test accuracy by 2.7 points and led on all four benchmarks. Derived text can omit the detail a question asks for. In Figure 3, the summary leaves out the city Caroline is moving to. Adding the item's text to its full source turns lowered accuracy, consistent with Section 4.2's interpretation that extra derived text dilutes the evidence.
W4. Record exposure, evolve-set size and test composition.
MemCodex's lead does not depend on prior exposure to the evaluated records. It still leads MemPro when entire records are held out of evolution.
The record-disjoint comparison in Table 5 (Appendix C.2) changes both the records used for evolution and the questions used for testing. Compared with Table 1, it also reduces the evolve sets from 308/420/300 to 249/336/240 questions for LoCoMo, MAB and PersonaMind, respectively.
We kept the evolved on that split fixed and compared it with and MemPro on two sets of questions. One contained unused questions from records used in evolution, and the other contained questions from held-out records. The sets contained 995/296 questions for LoCoMo, 1,344/420 for MAB and 960/300 for PersonaMind.
| Benchmark | , seen / held out | MemPro, seen / held out | MemCodex , seen / held out |
|---|---|---|---|
| LoCoMo | 63.6 / 63.0 | 61.1 / 60.7 | 66.4 / 65.6 |
| MAB | 48.4 / 47.9 | 46.9 / 46.6 | 52.0 / 51.4 |
| PersonaMind | 55.9 / 55.1 | 53.8 / 53.5 | 59.7 / 59.3 |
MemCodex stayed ahead on held-out records. Against either MemPro or , the lead shifted by at most 0.4 points, and the direction depended on which reference we used. The smaller evolve sets account for part of the drop in absolute accuracy. Reducing the main evolve sets to the same sizes without changing the test questions lowered accuracy by 0.2/0.2/0.3 points. The test questions also changed, and and MemPro scored lower on the held-out questions too. The margin over MemPro is therefore more informative about record exposure than the change in MemCodex's absolute score.
In five LoCoMo folds, we evolved on eight records and tested every question from the other two. MemCodex averaged 65.8% against 60.9% for MemPro and led by 4.4 to 5.5 points in every fold. We will replace the "at most 0.6 points" statement in Section 4.3 with these controlled observations.
Author Response to Reviewer pxo3 (Part 3/3)
W5. Construction cost, search cost and break-even query volume.
We report per-query latency separately from memory construction and search. Each system pays its construction cost once per record. Search runs once per deployment, and users share the evolved architecture.
We measured latency from read(q) to the last answer token on one NVIDIA A100 80GB with Qwen3.5-27B and batch size one. We logged write cost separately, including writes to KV MEMORY.
For the requested cost comparison, we measured construction time per 1,000 LoCoMo turns and search GPU hours per benchmark at the shared 64-program budget. The Mem- entry reports training cost.
| System | Write s / 1k turns | Search or training GPU h | Meta-agent tokens (M) |
|---|---|---|---|
| Full context | 0 | n/a | n/a |
| Dense RAG | 1.2 | n/a | n/a |
| MemGPT | 38 | n/a | n/a |
| A-Mem | 52 | n/a | n/a |
| Mem0 | 61 | n/a | n/a |
| MemoryOS | 47 | n/a | n/a |
| HMO | 58 | n/a | n/a |
| Mem- | 44 | 21 to 30 | n/a |
| AutoMem | 49 | 71 to 88 | 1.9 to 2.3 |
| MemEvolve | 55 | 80 to 102 | 2.4 to 2.9 |
| MemPro | 63 | 95 to 126 | 3.4 to 4.1 |
| MemCodex | 57 | n/a | n/a |
| MemCodex | 54 + 9 KV | 87 to 118 | 2.8 to 3.3 |
MemCodex cost about as much to build and evolve as the search-based baselines, so the extra upfront cost matters mainly against systems that do not search. Writing its text layers and Metis-27B KV MEMORY took the same 63 seconds per 1,000 turns as MemPro. After a search of comparable cost, MemCodex saved about 3.0 seconds and 4.8k prompt tokens per query compared with MemPro (Table 1, Section 4.1).
Dividing a representative search cost of 100 GPU hours by the wall-clock time saved per query at batch size one gives search-only break-even volumes of about 160,000 queries against HMO and 51,000 against 32k full context. By reusing the architecture across users, a deployment can serve far more queries than a benchmark test split and recover the search cost once it reaches the break-even volume.
Author Response to Reviewer pxo3 (Part 1/3)
We thank the reviewer for recognizing the inspectable one-method edits with source-turn provenance and the evidence that the search design matters, and address below the acceptance protocol and the controlled comparisons that isolate each reported gain.
An edit must improve accuracy or lower token cost without worsening the other. Only the accuracy branch needs a significance test because token counts are exact and the pass rate is noisy.
A child that saves tokens must at least match its parent's observed accuracy. At equal token cost, it must improve accuracy and pass a one-sided exact McNemar test. The
SUMMARY.routeedit fixed 12 LoCoMo questions and broke 2 (p=0.006, Appendix B.6). Reusing the evolve split can at worst admit an edit that does not help on new questions, and it cannot inflate any reported number. As in MemEvolve and MemPro, search never reads the test split, and every reported test score comes from that held-out split (Section 4).The held-out questions show how accepted edits fared against their parents. Of the 67 children accepted across the four median-seed searches in Table 10, the 41 that passed McNemar improved mean test accuracy by 0.7 points. The 26 accepted on token savings maintained test accuracy (+0.1 points) while cutting evolve-split tokens per question by 3% to 8%. Over three seeds, Eq. 3 accepted 12 of 256 formatting-only null edits (4.7%), with test accuracy changes of -0.2 to +0.1 points.
We reran the Qwen3.5-27B searches with stricter acceptance rules, keeping the budget at 64 children per search and using the same validity checks K and three seeds. In the disjoint variant, we split the evolve questions into stratified halves and used one for diagnosis and the other for acceptance. The table sums acceptances across the four median-seed searches and reports retention as in Table 11.
The submitted rule admitted more edits and produced better final architectures on the test split. Every stricter rule kept fewer children and lowered test accuracy by 0.4 to 1.0 points. We will correct Appendix A's "one in twenty null edits" claim and state both acceptance branches explicitly.
No. The reported AutoMem, MemEvolve and MemPro searches received per-question verdicts and failed read traces, without gold turns. MemCodex used gold turns only to find where evidence was lost, never to accept an edit or answer a test question.
The
search.feedbackconfiguration field wasoutcome+tracefor the three baselines andoutcome+trace+goldfor MemCodex. The other option isoutcome. Gold turns identify the first method that lost evidence (Appendix B.1), while Eq. 3 uses only the frozen judge's verdicts and prompt-token counts. Part of the lead predates search. The initial g0 averages 56.8%, 1.6 points above MemPro (Table 1).To see how much gold turns mattered, we reran each system's search with Qwen3.5-27B, three seeds, the same evolve splits and 64 children per search. We added annotated gold turns to the baselines' existing feedback. Without gold turns, MemCodex used the failed trace and reference answer to find where evidence was lost. With outcomes alone, it used neither and sampled uniformly from the layer-method pairs that ran. All MemCodex variants kept the validity checks K.
search.feedbackoutcome+trace(Table 1)outcome+trace+goldoutcome+trace(Table 1)outcome+trace+goldoutcome+trace(Table 1)outcome+trace+goldoutcome+trace+gold(Table 1)outcome+traceoutcomeThe ranking held even when MemPro received gold turns and MemCodex did not. MemCodex led that comparison by 4.1 points and averaged 59.2% with outcomes alone. Adding gold turns helped MemPro, and removing them hurt MemCodex more, consistent with one-method edits benefiting from knowing which method failed. We will document
search.feedbackand the baseline prompts in Appendix C.4 and state the gold-based diagnosis in Appendix A.