acceptodds
Under review as a conference paper at ICLR 2027

MemCodex: Self-Programming Hierarchical Memory for Language Agents

Abstract

Agent memory faces heterogeneous access needs: a single-hop question may require one piece of evidence, whereas a multi-hop question must combine evidence from multiple sources. Predefined memory workflows cannot adapt to these varying needs. Recent adaptive methods search or learn over memory components and their compositions, but the design space itself remains predefined. We introduce MemCodex, a self-evolving hierarchical memory system that organizes experience into executable memory programs for summaries, relational knowledge, reusable skills, and latent memory. Open-ended program evolution searches the open design space of layer programs by rewriting how each layer is constructed, indexed, retrieved, and routed, thereby adapting both within-layer implementations and cross-layer composition. At query time, reads traverse the hierarchy from coarse to fine and stop once sufficient evidence is found, descending to the original history when needed. We further develop MemArena, a unified runtime that places heterogeneous data and memory systems behind a common interface. MemCodex improves average task success by 10.1% relative to the strongest adaptive-memory baseline, while using 3.4× fewer context tokens and achieving 2.1× faster inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, self-evolving memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pxo3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemCodex represents agent memory as a fixed hierarchy of executable layer programs that preserve links to the raw turns used to construct each item. It adapts the system through single-method rewrites or layer toggles, with queries routed from coarse memory representations to original evidence. An optional Metis-derived memory state augments the open-weight backbone, and MemArena provides a shared evaluation interface. Across four benchmarks and two backbones, the paper reports average improvements over MemPro of 5.6 and 3.6 percentage points, together with lower prompt-token use and online latency.

Critical strengths

Restricting edits to one layer method while retaining source-turn provenance provides an inspectable mechanism for changing memory behavior. The 4.0-point gain over the initial architecture and the comparisons of rewrite granularity provide useful evidence that the search design matters. The advantage over MemPro persists under record-disjoint evaluation at 5.0 points and without latent memory at 3.6 points, which strengthens the empirical case beyond the headline configuration.

Critical weaknesses
  1. Equation (3) allows token-saving edits with non-decreasing observed accuracy to bypass the significance requirement. Repeated testing reuses the evolution data without stated correction, leaving Appendix A's 5% bound on accepting null edits unsupported.
  2. MemCodex uses gold evidence for diagnosis, but equivalent access for the search baselines is unspecified. Differences in supervision could therefore explain part of the gains.
  3. At STOP, routing uses one item's source turns, whereas content uses up to ten derived items (Algorithm 1). The 60.8% versus 55.4% comparison therefore cannot isolate the effect of evidence preservation.
  4. Section 4.3 attributes at most 0.6 percentage points to previously seen records. Since Appendix C.2 also changes evolution sample size and test composition, the comparison cannot isolate this effect.
  5. Appendix C.3 measures online latency without memory construction, and Table 10 reports 87-118 GPU hours per search without corresponding baseline construction and search costs. The query volume needed to amortize these costs relative to the baselines remains unquantified.
Questions for authors
  1. For each of AutoMem, MemEvolve, and MemPro, were gold evidence annotations included in the meta-agent's feedback for the reported runs, and which prompt field or configuration records that choice?
  2. Was a controlled content-versus-routing comparison performed with the same evolved architecture, selected item, and token cap, changing only derived text versus source turns; if so, what was the test-accuracy difference?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer pxo3 (Part 1/3)

We thank the reviewer for recognizing the inspectable one-method edits with source-turn provenance and the evidence that the search design matters, and address below the acceptance protocol and the controlled comparisons that isolate each reported gain.

W1. Edit acceptance and reuse of the evolve split.

An edit must improve accuracy or lower token cost without worsening the other. Only the accuracy branch needs a significance test because token counts are exact and the pass rate is noisy.

A child that saves tokens must at least match its parent's observed accuracy. At equal token cost, it must improve accuracy and pass a one-sided exact McNemar test. The SUMMARY.route edit fixed 12 LoCoMo questions and broke 2 (p=0.006p=0.006, Appendix B.6). Reusing the evolve split can at worst admit an edit that does not help on new questions, and it cannot inflate any reported number. As in MemEvolve and MemPro, search never reads the test split, and every reported test score comes from that held-out split (Section 4).

The held-out questions show how accepted edits fared against their parents. Of the 67 children accepted across the four median-seed searches in Table 10, the 41 that passed McNemar improved mean test accuracy by 0.7 points. The 26 accepted on token savings maintained test accuracy (+0.1 points) while cutting evolve-split tokens per question by 3% to 8%. Over three seeds, Eq. 3 accepted 12 of 256 formatting-only null edits (4.7%), with test accuracy changes of -0.2 to +0.1 points.

We reran the Qwen3.5-27B searches with stricter acceptance rules, keeping the budget at 64 children per search and using the same validity checks KK and three seeds. In the disjoint variant, we split the evolve questions into stratified halves and used one for diagnosis and the other for acceptance. The table sums acceptances across the four median-seed searches and reports retention as in Table 11.

Acceptance protocolAvg. test accuracy (%)AcceptedRetention (%)
Submitted, α=0.05\alpha=0.0560.86753 to 78
Bonferroni within round, α=0.05/8\alpha=0.05/860.14466 to 85
Acceptance questions disjoint from diagnosis60.45065 to 86
Both variants59.83672 to 89

The submitted rule admitted more edits and produced better final architectures on the test split. Every stricter rule kept fewer children and lowered test accuracy by 0.4 to 1.0 points. We will correct Appendix A's "one in twenty null edits" claim and state both acceptance branches explicitly.

W2 and Q1. Gold turns in search feedback.

No. The reported AutoMem, MemEvolve and MemPro searches received per-question verdicts and failed read traces, without gold turns. MemCodex used gold turns only to find where evidence was lost, never to accept an edit or answer a test question.

The search.feedback configuration field was outcome+trace for the three baselines and outcome+trace+gold for MemCodex. The other option is outcome. Gold turns identify the first method that lost evidence (Appendix B.1), while Eq. 3 uses only the frozen judge's verdicts and prompt-token counts. Part of the lead predates search. The initial g0g_0 averages 56.8%, 1.6 points above MemPro (Table 1).

To see how much gold turns mattered, we reran each system's search with Qwen3.5-27B, three seeds, the same evolve splits and 64 children per search. We added annotated gold turns to the baselines' existing feedback. Without gold turns, MemCodex used the failed trace and reference answer to find where evidence was lost. With outcomes alone, it used neither and sampled uniformly from the layer-method pairs that ran. All MemCodex variants kept the validity checks KK.

Systemsearch.feedbackLoCoMoLMEMABPMindAvg.
AutoMemoutcome+trace (Table 1)60.157.346.052.954.1
AutoMemoutcome+trace+gold60.457.546.353.254.4
MemEvolveoutcome+trace (Table 1)60.658.046.653.554.7
MemEvolveoutcome+trace+gold61.058.347.054.155.1
MemProoutcome+trace (Table 1)61.258.447.154.055.2
MemProoutcome+trace+gold61.958.847.654.955.8
MemCodexoutcome+trace+gold (Table 1)67.162.952.360.960.8
MemCodexoutcome+trace66.162.251.360.059.9
MemCodexoutcome65.461.750.659.159.2

The ranking held even when MemPro received gold turns and MemCodex did not. MemCodex led that comparison by 4.1 points and averaged 59.2% with outcomes alone. Adding gold turns helped MemPro, and removing them hurt MemCodex more, consistent with one-method edits benefiting from knowing which method failed. We will document search.feedback and the baseline prompts in Appendix C.4 and state the gold-based diagnosis in Appendix A.

AuthorsOpenReview ↗

Author Response to Reviewer pxo3 (Part 2/3)

W3 and Q2. Source turns and derived text at matched token counts.

Source turns carry the evidence the reader needs. Adding derived text keeps recall high but lowers accuracy compared with routing alone.

At STOP, routing passes only the top item's in-scope source turns, src(x*). Adding candidate texts to those turns restored LoCoMo recall to 88.7%, but average accuracy fell to 59.6% at 4.6k Tok/q (Table 2).

For the reviewer's matched comparison, we held each benchmark's routing-evolved g∗g^*, read trace, STOP decision, selected item x∗x^* and token cap B=4,096B=4{,}096 fixed. We changed only what the reader received at STOP. To match the length of text(x*), we ranked the item's source turns, kept as many as fit and returned them in time order. The submitted ten-item protocol is included for reference and is not part of the matched comparison.

Context assembled at STOPAvg. test accuracy (%)Tok/q
src(x*), temporal order60.82.0k
text(x*) only56.91.1k
Token-matched src(x*)59.61.1k
text(x*) plus src(x*)60.22.6k
text(X1..10), Table 2 protocol55.43.9k

Source turns answered questions more accurately than derived text from the same item. At 1.1k Tok/q, they improved average test accuracy by 2.7 points and led on all four benchmarks. Derived text can omit the detail a question asks for. In Figure 3, the summary leaves out the city Caroline is moving to. Adding the item's text to its full source turns lowered accuracy, consistent with Section 4.2's interpretation that extra derived text dilutes the evidence.

W4. Record exposure, evolve-set size and test composition.

MemCodex's lead does not depend on prior exposure to the evaluated records. It still leads MemPro when entire records are held out of evolution.

The record-disjoint comparison in Table 5 (Appendix C.2) changes both the records used for evolution and the questions used for testing. Compared with Table 1, it also reduces the evolve sets from 308/420/300 to 249/336/240 questions for LoCoMo, MAB and PersonaMind, respectively.

We kept the g∗g^* evolved on that split fixed and compared it with g0g_0 and MemPro on two sets of questions. One contained unused questions from records used in evolution, and the other contained questions from held-out records. The sets contained 995/296 questions for LoCoMo, 1,344/420 for MAB and 960/300 for PersonaMind.

Benchmarkg0g_0, seen / held outMemPro, seen / held outMemCodex g∗g^*, seen / held out
LoCoMo63.6 / 63.061.1 / 60.766.4 / 65.6
MAB48.4 / 47.946.9 / 46.652.0 / 51.4
PersonaMind55.9 / 55.153.8 / 53.559.7 / 59.3

MemCodex stayed ahead on held-out records. Against either MemPro or g0g_0, the lead shifted by at most 0.4 points, and the direction depended on which reference we used. The smaller evolve sets account for part of the drop in absolute accuracy. Reducing the main evolve sets to the same sizes without changing the test questions lowered accuracy by 0.2/0.2/0.3 points. The test questions also changed, and g0g_0 and MemPro scored lower on the held-out questions too. The margin over MemPro is therefore more informative about record exposure than the change in MemCodex's absolute score.

In five LoCoMo folds, we evolved on eight records and tested every question from the other two. MemCodex averaged 65.8% against 60.9% for MemPro and led by 4.4 to 5.5 points in every fold. We will replace the "at most 0.6 points" statement in Section 4.3 with these controlled observations.

AuthorsOpenReview ↗

Author Response to Reviewer pxo3 (Part 3/3)

W5. Construction cost, search cost and break-even query volume.

We report per-query latency separately from memory construction and search. Each system pays its construction cost once per record. Search runs once per deployment, and users share the evolved architecture.

We measured latency from read(q) to the last answer token on one NVIDIA A100 80GB with Qwen3.5-27B and batch size one. We logged write cost separately, including writes to KV MEMORY.

For the requested cost comparison, we measured construction time per 1,000 LoCoMo turns and search GPU hours per benchmark at the shared 64-program budget. The Mem-α\alpha entry reports training cost.

SystemWrite s / 1k turnsSearch or training GPU hMeta-agent tokens (M)
Full context0n/an/a
Dense RAG1.2n/an/a
MemGPT38n/an/a
A-Mem52n/an/a
Mem061n/an/a
MemoryOS47n/an/a
HMO58n/an/a
Mem-α\alpha4421 to 30n/a
AutoMem4971 to 881.9 to 2.3
MemEvolve5580 to 1022.4 to 2.9
MemPro6395 to 1263.4 to 4.1
MemCodex g0g_057n/an/a
MemCodex g∗g^*54 + 9 KV87 to 1182.8 to 3.3

MemCodex cost about as much to build and evolve as the search-based baselines, so the extra upfront cost matters mainly against systems that do not search. Writing its text layers and Metis-27B KV MEMORY took the same 63 seconds per 1,000 turns as MemPro. After a search of comparable cost, MemCodex saved about 3.0 seconds and 4.8k prompt tokens per query compared with MemPro (Table 1, Section 4.1).

Dividing a representative search cost of 100 GPU hours by the wall-clock time saved per query at batch size one gives search-only break-even volumes of about 160,000 queries against HMO and 51,000 against 32k full context. By reusing the architecture across users, a deployment can serve far more queries than a benchmark test split and recover the search cost once it reaches the break-even volume.

Reviewer_hd64OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper presents MemCodex, a hierarchical memory for LLM agents containing 4 layers upon an append-only raw layer. A meta-agent rewrites one method of one layer or toggles a layer per child. A child is archived only if it passes the validity checks and an acceptance test on a held-out split of questions. The paper also presents MemArena, a unified evaluation harness over LoCoMo, LongMemEval, MemoryAgentBench and PersonaMind, with 11 baselines and two backbones. Ablations attribute the gains over MemPro to the layer hierarchy, evidence routing and per-method evolution.

Critical strengths
  1. The fixed five-method layer API and 4 memory constraints make each evolved edit clearly attributable to one method.
  2. The layer-hierarchy example and the accepted child make a strong mechanism.

MemArena could be a useful harness if released with wrappers/adapters.

Critical weaknesses

1.The paper combines MemPro's evolved memory programs, RAPTOR-style hierarchy, Mem0/Zep-style graph memory, MemoryOS heat scores, but the gain is only about 5 points over MemPro.

  1. The meta-agent is shown the failed evolve questions and rewrites a method to fix them. The same questions then decide acceptance. The paper's own Table 11 shows the test split retains only 53-78% of the evolve gain, and 10-24% without the acceptance test.

  2. The "margins exceed four standard deviations" claim uses standard deviations over search seeds only. It does not factor in test-question sampling noise. No paired significance test is run between MemCodex and the baselines on the test split.

  3. Evolve and test questions come from the same records. LoCoMo has only 10 records, and diagnoses expose their text to the meta-agent.

  4. Under the hierarchy, MemPro-style program rewrites average 56.7, below the unevolved g0 at 56.8. On LoCoMo it is 62.1 against 63.4. This suggests evolve-split overfitting rather than a controlled demonstration that per-method edits beat whole-program rewrites. The result rests on three seeds.

  5. Hand-designed baselines use "official code" with defaults and no per-benchmark tuning, while MemCodex is tuned to each benchmark. Absolute baseline numbers are low and depend on the GPT-4o judge and the Mem0 rubric.

Lack of Clarity in Writing: The introduction opens with about 30 citations in its first paragraph and a similar density in the second. They appear before the paper states what it is doing or why. Related work lists 40+ citations in single sentences, with no discussion of how they differ from this work.

Questions for authors
  1. Can you hold out a separate validation split, distinct from the diagnosis and acceptance data, to rerun the acceptance test? How would the results change?
  2. Why do MemPro-style program rewrites score below g0, and how does this square with the claim that per-method evolution is the key?
  3. Were the hand-designed baselines tuned under the same budget or protocol? Can you provide a full-context baseline that fits the long records?
6 comments
AuthorsOpenReview ↗

Author Response to Reviewer hd64 (Part 1/5)

We thank the reviewer for recognizing that the fixed layer interface makes each evolved edit attributable to one method and that the hierarchy example shows a clear mechanism, and address below the evaluation protocol and what MemCodex adds beyond its borrowed starting points.

W1. MemCodex's architectural contribution.

MemCodex is not a combination of borrowed parts. Its hierarchy keeps reads short while preserving access to the original evidence. Each layer has its own program, so evolution can adapt the memory to the task.

Three design choices make this work. First, each derived layer compresses the record at a different level of detail and is written and read at a different frequency. Many reads can stop at a coarse layer and pass a short context to the reader. Second, every derived item links back to its original turns. A read can retrieve those turns directly from the selected item or descend when it needs finer evidence. The original turns remain available even if the derived text omits a detail. Third, each layer runs its own program behind a fixed interface. Evolution can change how individual layers write memory and how queries move between them, opening up choices that a single program over flat memory does not expose. RAPTOR [1], MemoryOS [2], Zep [3], Mem0 [4] and MemPro [5] supply only the starting point, and evolution rewrites the text-layer programs.

Before evolution, MemCodex already outperforms MemPro using about a third of its prompt tokens. After evolution, it leads by 5.6 points, while MemPro uses 3.4 times as many prompt tokens and takes 2.1 times as long to answer (Table 1, Section 4.1).

We tested how much the starting components mattered by replacing two of them in g0g_0, one at a time. We evolved each variant under the main protocol with Qwen3.5-27B, three seeds, 64 children and the same validity checks. The last two rows report the submitted summary-tree and content-channel comparisons (Table 2, Appendix C.5). Accuracy is the four-benchmark test average, and Tok/q counts prompt tokens per question.

VariantInitial g0EvolvedTok/q
Submitted MemCodex56.860.82.0k
Recency-only eviction instead of MemoryOS heat56.560.62.0k
Untimed triples without graph supersession55.960.12.1k
RAPTOR-style summary treen/a56.92.7k
Content channel instead of routed source turnsn/a55.43.9k

The hierarchy and its use of source turns mattered more than the choice of starting components. Changing the eviction rule or the graph representation cost 0.2 and 0.7 points, while using a summary tree or passing derived text in place of source turns cost 3.9 and 5.4 points.

W2 and Q1. Separate questions for diagnosis and acceptance.

The acceptance test preserves most of the evolve-split gain on unseen questions. Little of the gain transfers when every valid edit is accepted.

In the submitted setup, the meta agent proposes edits from failed traces on the evolve split, and Eq. 3 compares each child with its parent on that same split. With the task agent and judge frozen, an edit must be no worse in observed accuracy or token use and must either improve accuracy significantly or use fewer tokens. Table 11 shows that the test split retains more of the evolve gain with this rule than when all valid edits enter the archive. Search never sees the test questions.

Following the reviewer's suggestion, we split each stratified evolve set into disjoint diagnosis and acceptance halves, QdQ_d and QvQ_v. The halves contained 154/154 questions for LoCoMo, 50/50 for LongMemEval, 210/210 for MAB and 150/150 for PersonaMind. Only QdQ_d supplied failure traces. We used QvQ_v to apply Eq. 3, weight parents in the archive and choose g∗g^*. The test splits, Qwen3.5-27B, three seeds, 64 children and validity checks stayed the same. Retention divides the test gain over g0g_0 by the gain over g0g_0 on the acceptance questions. Accepted counts are from the median seed.

Benchmarkg0 testSubmitted g*Disjoint g*Ret. submitted / disjoint (%)Accepted submitted / disjoint
LoCoMo63.467.166.664 / 7018 / 13
LongMemEval59.762.962.353 / 6514 / 9
MAB48.352.352.068 / 8416 / 13
PersonaMind55.760.960.578 / 8619 / 15
Average56.860.860.4

The test gain held when edits were judged on different questions from those that prompted them. Average test accuracy fell by 0.4 points, and retention rose on every benchmark. The smaller acceptance sets gave the exact McNemar test less power, and fewer children passed, especially on LongMemEval. The 17 children from the submitted median-seed searches that failed the disjoint re-test changed test accuracy by -0.3 to +0.2 points relative to their parents.

AuthorsOpenReview ↗

Author Response to Reviewer hd64 (Part 2/5)

W3. Paired test uncertainty against MemPro.

MemCodex's lead over MemPro holds when test-question sampling is taken into account. Both systems answer the same test questions, so their verdicts can be paired.

Table 9 holds the test questions fixed and varies only the search seed. For the paired comparison, we used each method's Qwen3.5-27B run with the median accuracy across the four benchmarks. The resulting margins differed from the Table 1 seed-mean margins by at most 0.5 points, on LongMemEval, where Table 1 reports +4.5 points.

We ran a one-sided exact McNemar test. Here, bb counts questions only MemCodex answered correctly, and cc counts questions only MemPro answered correctly. We computed 95% bootstrap intervals using 10,000 resamples of questions and, separately, of record clusters. LongMemEval has one record per question.

Benchmarknb / cMargin (points)p95% CI, questions95% CI, records
LoCoMo1,232178 / 105+5.9<0.001[3.3, 8.6][2.4, 9.4]
LongMemEval40057 / 37+5.00.025[0.3, 9.7][0.3, 9.7]
MAB1,680245 / 158+5.2<0.001[2.9, 7.5][2.7, 7.7]
PersonaMind1,200173 / 90+6.9<0.001[4.3, 9.5][4.1, 9.7]

The paired advantage over MemPro remained significant on all four benchmarks after Holm correction. LongMemEval had the smallest test set and the weakest evidence. The record-cluster intervals also excluded zero on every benchmark. The paired tests against HMO and full context gave p<0.001p<0.001 on every benchmark.

W4. Transfer to unseen records and diagnosis redaction.

MemCodex still leads MemPro when whole records are held out of search. The margin is close to what we report when we hold out only questions.

Appendix C.2 holds out whole records during search. Only the layer programs and layer mask transfer to the held-out records, each of which is written into a fresh store. In the main protocol, the meta agent sees failed questions, selected items and turns, and gold-turn information (Appendix B.6). The sandbox limits what an evolved method can read and change.

We repeated the LoCoMo comparison across five folds with Qwen3.5-27B and three seeds. Each search used the 20% evolve questions from eight records and tested every question from the other two. Each of the 1,540 questions belonged to one held-out fold, so the programs answering it had not seen its record during search.

FoldMemProg0MemCodexMargin
160.763.065.6+4.9
261.063.665.4+4.4
361.363.966.8+5.5
460.663.265.1+4.5
560.963.566.3+5.4
Mean60.963.465.8+4.9

MemCodex led MemPro in every fold, by 4.4 to 5.5 points. Under the main protocol, removing question and turn text, entity names, dates and the correctly answered example from every diagnosis changed average accuracy from 60.8 to 60.3. Across 201 accepted children from twelve searches, no audited method body contained a string literal found in a record. The fold results show that the programs can organize memory for new records, and the audit found no evidence of verbatim copying into the method bodies.

AuthorsOpenReview ↗

Author Response to Reviewer hd64 (Part 3/5)

W5 and Q2. Edit size and the selection rule.

The whole-program rewrites scored below the starting program for two reasons. Changing every layer at once makes validity failures more common. Selecting by evolve accuracy favors edits that fit those questions.

The validity checks rejected 41 of 64 layer rewrites, compared with 9 of 64 per-method edits (Section 4.2). One invalid change is enough to reject the whole child. The program-rewrites row in Table 2 changes more than the edit size. It rewrites every layer program in one child (Appendix C.5) and follows MemPro [5] in selecting by evolve-split accuracy, without Eq. 3's paired test or token condition. The caption's "one change per row" does not describe this comparison. Across three seeds, program rewrites scored 56.7 versus 56.8 for g0g_0, and 62.1 versus 63.4 on LoCoMo. That result reflects fitting the evolve split, so we separated the two factors directly.

We crossed one-method and whole-program edits with Eq. 3 and selection by evolve accuracy. All runs used the same four-layer hierarchy, Qwen3.5-27B, evolve and test splits, validity checks and 64 children per search. Test accuracy is the four-benchmark mean, with the standard deviation over five search seeds. Gains and retention use the same g0g_0, and counts are from the median seed.

Edit unitSelection ruleAvg. testEvolve / test gainRetentionAccepted / failed K
One methodEq. 360.8 ± 0.3+6.1 / +4.066%17 / 9
One methodEvolve accuracy58.6 ± 0.5+6.8 / +1.826%31 / 9
Whole programEq. 357.5 ± 0.4+1.9 / +0.737%4 / 44
Whole programEvolve accuracy56.6 ± 0.6+2.3 / -0.2-9%7 / 44

Both choices mattered, and edit size made the larger difference. With Eq. 3 fixed, one-method edits led whole-program edits by 3.3 test points. With one-method edits fixed, Eq. 3 added 2.2 points. Whole-program searches selected by evolve accuracy improved on the evolve split but finished below g0g_0 on test. With Eq. 3, they finished above g0g_0, although most whole-program children still failed validity checks and few were accepted.

Reviewer_VzY1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MemCodex organizes memory into raw history, summaries, graphs, and skills, and improves the system by rewriting individual memory methods. Derived memories guide retrieval back to their source turns, while query-dependent routing controls search depth. Experiments across four benchmarks report improved accuracy and lower query-time cost compared with the evaluated memory systems.

Critical strengths

Source preservation and method-level edits provide a concrete way to diagnose and correct retrieval failures. The stale-summary example illustrates how a targeted routing change can recover relevant evidence without redesigning the full memory pipeline.

The ablations separate the contribution of evolution from the initial architecture: average accuracy improves from 56.8 to 60.8. The record-disjoint evaluation retains a 5-point lead over MemPro, supporting generalization to unseen records.

Critical weaknesses

Evolution uses gold evidence turns to identify which method caused a failure. This provides stronger supervision than answer correctness alone, and whether competing search methods receive equivalent information is unclear. Performance without evidence annotations remains untested, limiting the conclusions about applicability to ordinary agent traces.

The retrieval baseline uses BGE over 512-token chunks without reranking. Since MemCodex ultimately supplies raw turns to the reader, comparison with a stronger retrieval pipeline is important for establishing the added value of the hierarchy and program search. The evolved raw-only ablation only partially addresses this concern.

Questions for authors
  1. Do the adaptive baselines receive equivalent gold-evidence information during search? How does MemCodex perform when diagnosis uses only answer outcomes and execution traces?
  2. How does MemCodex compare with raw retrieval using reranking and tuned retrieval depth under the same context budget?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer VzY1 (Part 1/2)

We thank the reviewer for recognizing the diagnosable method-level edits and the ablation that isolates evolution, and address below the gold-evidence supervision in the diagnosis and the strength of the raw-retrieval baseline.

W1 and Q1. Search feedback with and without gold turns.

No. The submitted AutoMem, MemEvolve and MemPro searches received no gold turns. MemCodex used them only to diagnose search failures, never to accept an edit or to answer a test question.

The baselines received answer outcomes and failed read traces. MemCodex also used G(q)G(q) to identify the first layer and method that lost the required evidence (Appendix B.1). Eq. (3) compares the child and parent using frozen-judge verdicts and prompt-token counts, with a one-sided exact McNemar test for the accuracy branch. Part of the lead predates search. The initial g0g_0 averaged 56.8% test accuracy, 1.6 points above MemPro (Table 1).

We tested the reviewer's suggestion in both directions. We added annotated gold turns to the baselines' usual feedback and ran MemCodex without them. The reruns followed Section 4 on all four benchmarks, with Qwen3.5-27B, three seeds, 64 children per search and the same evolve splits, task agent and judge. MemCodex kept its validity checks, and each baseline kept its selection rule. The table reports test accuracy (%).

Systemsearch.feedbackLoCoMoLMEMABPMindAvg.
AutoMemoutcome+trace (Table 1)60.157.346.052.954.1
AutoMemoutcome+trace+gold60.457.546.353.254.4
MemEvolveoutcome+trace (Table 1)60.658.046.653.554.7
MemEvolveoutcome+trace+gold61.058.347.054.155.1
MemProoutcome+trace (Table 1)61.258.447.154.055.2
MemProoutcome+trace+gold61.958.847.654.955.8
MemCodexoutcome+trace+gold (Table 1)67.162.952.360.960.8
MemCodexoutcome+trace66.162.251.360.059.9
MemCodexoutcome65.461.750.659.159.2

MemCodex kept its lead without gold turns. With outcomes and traces, it averaged 59.9% test accuracy and led MemPro by 4.7 points when neither search received gold turns and by 4.1 points when MemPro received them.

In the runs with traces but no gold turns, the meta agent used the reference answer and the failed read trace, including the methods executed and the items and turns selected, to infer which item supported the answer. It followed that item's source links to find the first method that lost the linked turns, as in Appendix B.1. It picked the same layer and method as the gold-based diagnosis on 78% of failed traces. On the test split, these searches retained 50% to 75% of their evolve-split gains, compared with 53% to 78% in Table 11.

AuthorsOpenReview ↗

Author Response to Reviewer VzY1 (Part 2/2)

W2 and Q2. Tuned raw retrieval under a shared context budget.

The hierarchy adds value even though the reader sees raw turns. The derived layers decide which raw turns reach the reader, a choice that a retriever ranking turns by query match cannot make.

GRAPH links entities across sessions, and an evolved SUMMARY route can keep a newer session in scope. Adding GRAPH gave the largest gain of any layer, 2.5 points (Section 4.2). On LoCoMo, 72% of multi-hop reads stopped above RAW. With all four text layers available, MemCodex averaged 60.8% test accuracy, compared with 54.1% with only RAW available under the same 64-child search budget (Table 2).

The submitted Dense RAG used BGE over 512-token chunks at default retrieval depth. For the comparison the reviewer asks for, we built an independent raw-text pipeline and tuned chunk size, retrieval depth and reranking with a 64-configuration random search. We added retrieval components in sequence, selected configurations on the evolve split and evaluated them on the test split over three seeds. All rows used Qwen3.5-27B, the same test questions and a context budget of B=4,096B=4{,}096. The MemCodex rows use the submitted setups, with KV MEMORY disabled in the marked variant. Rec. is LoCoMo gold-turn recall, and Tok/q counts task-agent prompt tokens per question.

Retrieval pipeline or memory systemAvg. (%)Rec. (%)Tok/q
Dense RAG, BGE, default (Table 1)47.674.63.6k
+ tuned chunk size and depth49.076.93.9k
+ bge-reranker-v2-m350.179.54.0k
+ turn-level retrieval51.481.03.9k
+ hybrid BM25 and dense, reciprocal rank fusion52.082.24.0k
+ two-round retrieval and query rewriting53.384.14.1k
MemCodex, one text layer (RAW), evolved (Table 2)54.171.21.8k
MemCodex, initial g0g_0 (Table 1)56.882.62.4k
MemCodex, no KV MEMORY (Appendix C.5)58.888.42.0k
MemCodex (Table 1)60.888.42.0k

Each retrieval upgrade improved accuracy, but MemCodex still led without KV MEMORY. The best retrieval pipeline averaged 53.3% test accuracy, 5.5 points behind the no-KV variant.

On LoCoMo, full MemCodex led that pipeline by 15.8 points on multi-hop questions and 14.9 on temporal questions, compared with 6.4 on single-hop questions (Table 8 for MemCodex). The larger gaps on multi-hop and temporal questions are consistent with routing through the cross-session links in GRAPH. With GPT-5.4-mini, both systems used only text, and the best retrieval pipeline averaged 55.6% test accuracy compared with 61.1% for MemCodex.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks from the Authors

Dear AC, SAC, and Reviewers,

We sincerely thank the AC and SAC for coordinating the review process and the reviewers for their constructive feedback, which helped us tighten the evaluation protocol and the controlled comparisons.

Summary of the Original Reviews

Major strengths include one-method edits that make each change attributable and inspectable (pxo3, hd64, VzY1), preserved source turns that help diagnose and correct retrieval failures (pxo3, VzY1), ablations that separate the gain of evolution from the starting architecture (pxo3, VzY1), a lead over MemPro on records held out of search (pxo3, VzY1), and clear worked examples (hd64, VzY1).

Key concerns include the use of gold evidence in diagnosis and whether the search baselines received it (pxo3, VzY1), an acceptance test that reuses the evolve questions behind each edit (pxo3, hd64), the meta agent's exposure to the evaluated records during search (pxo3, hd64), the tuning and strength of the baselines, including reranked retrieval and full context (hd64, VzY1), whether the ablations isolate source-turn routing and one-method edits (pxo3, hd64), and what MemCodex adds beyond the systems it builds on, including the size and significance of its lead over MemPro (hd64).

Summary of the Rebuttal and Discussion

We addressed all substantive concerns with targeted clarifications, new experiments, and supporting evidence:

  • Clarified that the reported AutoMem, MemEvolve and MemPro searches used no gold turns, and reran search both ways: MemCodex without gold turns still led MemPro with gold turns by 4.1 points (pxo3, VzY1).
  • Separated the diagnosis and acceptance questions: the test gain held, retention rose on every benchmark, and null edits were accepted at 4.7% against the nominal 5% level (pxo3, hd64).
  • Held out whole records from search in five LoCoMo folds: MemCodex led MemPro in every fold, by a margin close to the one with only questions held out (pxo3, hd64).
  • Tuned the hand-designed baselines on the evolve split and added reranked retrieval and 128k full context: MemCodex led all of them, and the best tuned baseline, HMO, trailed by 6.6 points (hd64, VzY1).
  • Isolated both design choices: at matched token counts, source turns beat derived text from the same item on all four benchmarks, and one-method edits beat whole-program edits under both selection rules (pxo3, hd64).
  • Showed that swapping two borrowed starting components mattered less than the hierarchy and its source turns, and that the lead over MemPro was significant on all four benchmarks under paired McNemar tests with Holm correction (hd64).

Final Resolution

Reviewer hd64 raised their rating from 1 (Clear rejection) to 2 (Weak rejection).

Reviewer pxo3 replied that our response addressed their main concerns, and we answered their two further questions in a follow-up.

Reviewer VzY1 did not reply during the author-reviewer discussion period and maintained a rating of 3 (Weak acceptance). We addressed all their points in detail, such as results without gold evidence turns, baseline searches rerun with them, and a comparison with tuned, reranked retrieval.

We thank the AC, SAC, and reviewers again for their thoughtful engagement and hope our responses are helpful in the final decision process.

Sincerely,

The Authors of Paper 21381

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.