acceptodds
Under review as a conference paper at ICLR 2027

GenMem: Generative Symbolic Memory for Self-Evolving Harness

24 views

Abstract

Long-term memory supports the self-evolution of LLM agents by retaining experience and skills across tasks and enabling their retrieval, reuse, and revision in subsequent long-horizon decision-making. Yet existing memory management approaches remain limited to discriminative retrieval and to address the sparse, hierarchical, and highly redundant structure of reusable experience: only a small, task-dependent subset of trajectories and memories warrants retention, retrieval, or revision. Learning these operations is further complicated by sparse, delayed, and indirect task-level feedback, with weak supervision across the memory lifecycle. Moreover, continual memory evolution introduces an architectural tension as addressing invariance: stored experience is perpetually revised, yet the addressing interface consumed by learned retrieval policies must remain stable. To address, we present GenMem, which reformulates memory management as generative symbolic addressing. Its core mechanism is the Symbolic Identifier (SID), a multi-level discrete token tuple drawn from a Cartesian-product address space that factorizes a million-scale sparse memory space using fewer than one hundred discrete symbols. Instead of generating ever-changing raw content, the memory agent learns to generate SIDs, while memory evolution rewrites the payload at a fixed address without shifting the address itself. Architecturally, GenMem couples a MemRetriever and a MemEvolver within a multi-agent harness, trained via GRPO with dense process and outcome rewards with two-channels optimization. Under offline memory evolution, experiments spanning ALFWorld, WebShop, multi-hop QA, medical reasoning, and deep research evaluate GenMem against strong memory-augmented baselines. Beyond accuracy, we find that SIDs exhibit emergent neuro-symbolic properties: they can be interleaved with natural-language tokens and are compositional—combining multiple SIDs composes their stored experiences to address novel situations not covered by any single entry.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6mQ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes GenMEM, a generative symbolic memory framework for self-evolving LLM agents. GenMem introduces a generative symbolic memory framework for self-evolving LLM agents to address the challenges of experience sparsity and content-dependent retrieval instability caused by memory updates. Its core innovation is the introduction of hierarchical Symbolic Identifiers (SIDs) constructed via residual quantization to decouple memory addressing from content. The system consists of a MemRetriever, a StackPlanner, and a MemEvolver, trained through SID alignment, supervised mid-training, and GRPO-based policy optimization. Experiments demonstrate that the method achieves good performance across interactive decision-making and QA tasks.

Critical strengths

1: The problem the paper is targeting is important. The paper correctly identifies a practical tension in self-evolving memory systems: retrieved memories can change over time, while the agent must continue accessing them reliably.

2: The experiment results is strong, achieving up to ten points improvement on WebShop Success rate.

Critical weaknesses

1: The paper is hard to read with a lot of theoretical frame. However, after reading the paper, I found the underlying intuition and implementation appear relatively straightforward. For example, the paper claims SID encodes mutually orthogonal categorical axes, however, in equation 3, it is basically a residual quantization.

2: My biggest concern after reading the paper is the over-claim problem. The central contribution of GenMem is presented as generative symbolic addressing for stable memory retrieval and evolution. The paper argues that conventional embedding-based retrieval suffers from address drift when memory content changes. While this is a reasonable motivation, address invariance is not unique to generative retrieval. For example, a memory system could maintain stable identifiers and fixed retrieval representations while allowing memory payloads to evolve. I'd like to see a direct comparison with such a baseline.

3: The data separation remains a concern. Table 2 labels Bamboogle, MuSiQue, NQ, and TriviaQA as out-of-domain benchmarks. However, Table 6 lists many of these rources in the initial memory bank. It also includes 1,664 GPQA-derived entries, while Section 5 evaluates symbolic reasoning on GPQA. I would appreciate more details on data preparation, split definitions, and overlap verification.

4: Unclear training objectives and evaluation configurations. Section 3 introduces separate advantage normalization for proxy and audit channels, but does not clearly define their respective data, rewards, sampling procedures, or combined update.

Questions for authors

1: Are the initial GenMem retrieval results in Table 4 obtained from the same evaluation as the PT-4L results in Table 18? Why does Table 18 show such a large degradation after mid-training, and what are the corresponding post-GRPO retrieval metrics? Please explain why Table 4's initial GenMem results exactly match PT-4L.

2: What are the exact data splits and overlap checks? How are the out-of-domain QA and GPQA evaluations separated from the corresponding source experiences used for memory construction and alignment?

3: Which training configurations and reward specifications were used to produce the reported results? In particular, could the authors clarify the differences between the main-text and appendix descriptions of the GRPO training settings and reward decomposition?

8 comments
AuthorsOpenReview ↗

Rebuttal for W1

Thank you for the constructive feedback. Below, we address your concerns by clarifying our terminology and mathematical framing, providing the requested fixed-key baseline, and detailing the exact data splits, retrieval metrics, and GRPO training specifications.

W1. Clarifying residual quantization vs. "orthogonal axes" and simplifying the presentation

We fully agree with your critique. Equation (3) implements standard multi-level residual quantization (RQ-KMeans), where each level ll quantizes the residual ril=ril−1−csil−1l−1r_i^{l} = r_i^{l-1} - c_{s_i^{l-1}}^{l-1} left by the preceding levels. While the (48,16,8,8)(48, 16, 8, 8) codebooks factorize 49,152 possible addresses using 80 level-specific symbols, Equation (3) produces a coarse-to-fine residual hierarchy, not "mutually orthogonal categorical axes."

Our intended point is simply that successive RQ levels capture coarse-to-fine task structure and play distinct functional roles in conditioning the model, which is supported by two diagnostics:

  1. Cross-level code statistics (Appendix C.1, p. 20): The total-correlation diagnostic (ICRentropy\text{ICR}_{\text{entropy}}) measures statistical dependence among code assignments across levels rather than semantic orthogonality.
  2. Level-perturbation diagnostic on GPQA (text-free): Holding all other inputs fixed, we perturbed either the fine-grained residual levels (C/DC/D, retaining prefix A/BA/B) or the coarse-grained prefix levels (A/BA/B, retaining suffix C/DC/D):
SID Conditioning SetupModified LevelsAccuracy (%)Δ\Delta vs. Gold SID
Original (Gold) SIDNone56.0–
Fine-grained suffix replacementC,DC, D (retain A,BA, B)55.0-1.0 pp
Coarse-grained prefix replacementA,BA, B (retain C,DC, D)51.0-5.0 pp

Modifying fine-grained residuals (C/DC/D) barely affects accuracy (-1.0 pp), whereas modifying coarse prefixes (A/BA/B) drops accuracy by 5.0 pp, confirming a coarse-to-fine conditioning hierarchy.

Manuscript revision: We will streamline the theoretical framing in Sections 1–2 and Appendix B, remove the "mutually orthogonal categorical axes" phrasing, and consistently describe SIDs as coarse-to-fine hierarchical residual codes.

AuthorsOpenReview ↗

Rebuttal for W2

W2. Direct comparison with a Frozen-Key + Evolving-Payload dense baseline

Following your suggestion, we implemented this exact Frozen-Key Embedding + Evolving-Payload baseline and compared it directly against GenMem.

(1) Controlled experimental setup. Both methods start from the exact same initial memory bank (30,577 entries), planner, updater (MemEvolver), and inference budget. Each entry in the baseline retains its initial embedding as a frozen retrieval key while its textual payload is updated in place during online evolution; at test time, queries match against the frozen keys (cosine Top-5) to fetch the latest payloads. We drive online evolution with a 1,500-query stream (NQ/Bamboogle) and evaluate before and after evolution on a disjoint 200-question probe set (HotpotQA/2WikiMultiHopQA). We also measure Content Relevance Hit Rate (scored 0–3 by a Qwen2.5-7B judge, with score ≥2\ge 2 counted as relevant):

Method (Identical Initial Bank, Updater & Probe Set)Initial Task Acc (%)Final Task Acc (%)Δ\Delta Acc (pp)Content Relevance Hit Rate (Before →\to After)
Frozen-Key Embedding + Evolving Payload47.042.0-5.059.0% →\to 52.0% (-7.0 pp)
GenMem (Learned SID Read/Write Routing)58.061.0+3.059.0% →\to 67.0% (+8.0 pp)

(2) Why freezing dense keys alone is insufficient. As payloads are revised over 1,500 evolution steps, the Frozen-Key baseline's content-relevance hit rate drops from 59.0% to 52.0% (-7.0 pp) and downstream accuracy drops from 47.0% to 42.0% (-5.0 pp), whereas GenMem improves from 58.0% to 61.0% (+3.0 pp) (consistent with Table 23, where enabling evolution in GenMem raises accuracy from 46.5% to 49.5% and F1 from 46.8% to 52.1%).

This contrast highlights two distinct failure modes in dense retrieval under evolution:

  • Re-indexing keys (Table 4 baseline) causes address drift, shifting which queries reach an entry;
  • Freezing initial instance keys (new baseline above) causes key–payload semantic decoupling: an initial embedding encodes surface-level wording of the seed instance rather than an abstract functional role, so uncoordinated updates cause the evolving payload to diverge from what the static vector key represents.

We center GenMem's contribution on learning coordinated read and write routing (MemRetriever and MemEvolver) over a shared, factorized symbolic address space, where each SID represents a coarse-to-fine strategy cluster and both policies are jointly trained via GRPO to read, rewrite, and revise slots coherently.

AuthorsOpenReview ↗

Rebuttal for W3&Q2

W3 & Q2. Data splits, ID/OOD definitions, and instance-level overlap audit

(1) Meaning of the ID/OOD labels in Table 2. Following the Search-R1 evaluation convention, the In-Domain (ID) and Out-of-Domain (OOD) labels in Table 2 refer strictly to the training data of the task-execution model (StackPlanner): only 2WikiMultiHopQA and HotpotQA were used to train the task-execution model, whereas Bamboogle, MuSiQue, NQ, and TriviaQA were held out from task-execution model training. However, source questions from these datasets contributed distilled experiences to the offline seed memory bank (Table 6; 4 rollout trajectories per source question). To avoid any ambiguity, we will relabel these columns in Table 2 as "Held-Out from Task-Model Training" and explicitly state their role in offline memory construction. (Note: True zero-shot domain transfer to benchmarks completely excluded from both memory construction and model training—BrowseComp-Plus, DeepResearch Bench, BIRD, and SWE-bench Verified—is evaluated separately in Table 3.)

(2) Exact split definitions and instance-level overlap verification. We performed an instance-level audit across all memory-construction sources (Table 6) and evaluation sets (Table 2 and Section 5). Below is the complete breakdown of source splits, distinct source questions, derived experience entries (4 rollouts per question), evaluation splits, and training-overlap counts:

DatasetMemory-Bank Source SplitSource Questions (Distinct IDs)Retained Experience Entries (Table 6)Evaluation Split / SelectionEvaluation QuestionsTask-Model Training Overlap
MuSiQuetrain2,3289,312Answerable dev2,4170
Natural Questions (NQ)train2,0208,080validation / dev3,6100
TriviaQAtrain2,4739,892Wiki & Web validation4,0000
Bamboogletest (125 Qs)125500test1250
GPQA (Sec. 5)Seed subset4161,664Main evaluation / Perturbation diagnostic4480

For MuSiQue, NQ, and TriviaQA, memory construction strictly uses the train split while evaluation uses the disjoint dev/validation splits (zero instance overlap). For Bamboogle (which only has a 125-question release) and the GPQA diagnostic in Section 5, we will explicitly document the in-bank seed construction protocol and split records in Appendix E.1 so that in-bank conditioning diagnostics are clearly distinguished from held-out generalization benchmarks.

Reviewer_LsYfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GenMem, a generative symbolic memory framework for self-evolving LLM agents. It uses hierarchical Symbolic Identifiers (SIDs) to enable stable memory addressing while allowing stored experiences to evolve. Experiments across multiple agent benchmarks demonstrate improved task performance and memory evolution.

Critical strengths

The paper identifies a meaningful challenge in evolving memory systems. Its proposed generative SID-based addressing decouples memory locations from their contents, offering a principled approach to maintaining stable addresses during memory evolution. It uses residual K-means quantization to construct a four-level discrete address space with codebook sizes (48, 16, 8, 8), supporting over 40,000 possible addresses using only 80 level-specific symbols. The design is technically meaningful because it provides a structured interface for both retrieval and update operations.

Critical weaknesses
  1. The paper claims that combining multiple SIDs enables compositional generalization to novel situations, but the experiments do not directly validate this property. Section 5 shows only that deeper SID prefixes raise GPQA accuracy, not that SIDs compose. Increasing the number of combined SIDs (k = 1, 5, 10) provides little additional benefit, while the “Symbolic Composition” experiments primarily evaluate retrieved or rewritten memory content.

  2. The paper compares against more than 20 methods, but almost none share GenMem's harness, planner training, or memory bank. Although the evaluation is broad, the paper does not analyze what fundamentally distinguishes GenMem from existing learned-memory methods, such as which specific modules of prior pipelines it improves (e.g., memory structure, retrieval, indexing, memory routing).

Questions for authors
  1. Figure 4 shows limited performance differences across beam sizes (k = {1,5,10}). If SID composition is beneficial, why does combining more SIDs not consistently improve accuracy? Could the authors provide a more direct quantitative evaluation of multi-SID composition?

  2. How does GenMem handle genuinely novel skills or task domains that are poorly represented in the initial experience distribution? Could the authors evaluate performance under substantial distribution shifts or when the number of required memory categories exceeds the effective capacity of the learned address space?

5 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1

Thank you for the constructive review and for recognizing the technical value of decoupling memory locations from evolving payloads via hierarchical SIDs. Below, we address your questions regarding multi-SID scaling in Figure 4, the isolated contribution of SID tokens versus experience text, and the scope of our compositionality claims.

W1 & Q1. Multi-SID scaling, protocol alignment, and isolating SID tokens from experience text

(1) Why beam size (k=1,5,10k = 1, 5, 10) showed limited gains in Figure 4, and re-evaluation under aligned reranking. In Figure 4, candidates were selected purely from raw autoregressive beam search (Beam@k) without the secondary dense reranking stage used in the main retrieval protocol (Section 4.2, Table 4). Because likelihood-based beam search tends to concentrate on high-probability candidates sharing the same prefix subtree, increasing kk under raw beam decoding mainly introduced near-duplicate or low-relevance SIDs, diluting the conditioning signal.

To evaluate multi-SID scaling without candidate redundancy, we aligned the protocol with our main experiments (Beam@50 followed by BGE reranking to Top-kk) and evaluated query + kk SIDs (text-free, without experience payloads) on 185 questions spanning nine QA and mathematical reasoning benchmarks:

Number of Combined SIDs (kk)Retrieval ProtocolText-Free Accuracy (%)Correct / TotalΔ\Delta vs. k=1k=1 (pp)
k=1k = 1Beam Search + BGE Reranking23.7844 / 185–
k=5k = 5Beam Search + BGE Reranking25.9548 / 185+2.17
k=10k = 10Beam Search + BGE Reranking30.2756 / 185+6.49

Once redundant beam candidates are filtered via reranking, accuracy increases monotonically with kk, yielding a +6.49 percentage-point gain from k=1k=1 to k=10k=10. This confirms that combining multiple distinct, relevant SIDs provides complementary conditioning cues beyond a single address.

(2) Isolating the contribution of SID tokens from retrieved and rewritten experience text. We agree that the right-hand side of Figure 4 (“Rewrite” and “Retrieved”) did not separate the effect of the symbolic SID tokens from the textual payloads. We therefore conducted a controlled ablation on GPQA that compares SID-absent vs. SID-present inputs while keeping the retrieved memory entries and exact experience text strictly identical within each pair:

Experience Text PayloadWithout SID Tokens (%)With SID Tokens (%)Incremental Gain from SIDs (pp)
Directly Retrieved Experience56.057.0+1.0
Task-Adaptive Rewritten Experience57.059.0+2.0

Appending SID tokens yields a consistent incremental gain (+1.0 pp with raw retrieved text and +2.0 pp with rewritten text) even when the exact textual experience is already present in the context. Furthermore, in an oracle-conditioned text-free perturbation test on 100 GPQA questions (Gold SID accuracy: 56.0%), perturbing fine-grained suffix tokens (L3/L4L_3/L_4) while retaining coarse prefix tokens (L1/L2L_1/L_2) reduces accuracy by only 1.0 pp (55.0%), whereas perturbing coarse prefix tokens (L1/L2L_1/L_2) drops accuracy by 5.0 pp (51.0%), confirming that SIDs provide structured, hierarchy-aware conditioning signals.

AuthorsOpenReview ↗

Rebuttal for W2

W2. Fundamental module-level distinctions and strictly controlled evaluations

We next (i) clearly delineate which specific pipeline modules GenMem improves over prior learned-memory systems, and (ii) isolate the contribution of each module under strictly identical harness, planner, and memory-bank conditions.


(1) Architectural taxonomy: Which specific pipeline modules does GenMem improve?

Existing agentic memory systems typically treat memory indexing, retrieval routing, and memory updating as disconnected, content-coupled heuristics. GenMem fundamentally redesigns three core modules by unifying read and write routing over a shared discrete symbolic address space (Z\mathcal{Z}):

Memory PipelineDense / Graph Memory (e.g., Mem0, SimpleMem)Skill-Library RL (e.g., SkillRL, SkillGraph, SkillRouter)Episodic / Lifecycle RL (e.g., MemRL, EvolveR)**GenMem **
1. Indexing & StructureFlat dense vectors or entity graphs; address is coupled to current text embedding fenc(x)f_{\text{enc}}(x).Hierarchical or graph-structured skill directories indexed by skill descriptions.Episodic trajectory buffers or distilled experience pools indexed by dense embeddings.Factorized Cartesian SID Space: 4-level residual quantization (48×16×8×848\times 16\times 8\times 8) maps experiences to fixed discrete slot coordinates z∈Zz \in \mathcal{Z} using only 80 tokens, decoupling where a skill lives from what its current text says.
2. Read Routing & UtilizationPassive top-kk cosine similarity lookup between query qq and payload xx; raw text concatenation.Embedding similarity or discriminative router over skill descriptions (SkillRouter).Discriminative scoring or value-weighted episodic lookup over stored trajectories.Generative Read Routing (MemRetriever): Autoregressively generates target SID zrz_r conditioned on query reasoning context, fetches the slot payload, and actively rewrites it into task-adaptive support x~r\tilde{x}_r.
3. Write Routing & EvolutionHeuristic append, FIFO eviction, or LLM-prompted merge triggered by high cosine similarity.Validation failures trigger offline/periodic skill regeneration during policy training.Runtime RL updates episodic utility weights or appends distilled reflections; requires re-indexing on text edits.Generative Write Routing (MemEvolver): Generates target update SID zez_e from trajectory τ\tau and performs in-place insert or revise (xi,zi)→(xi′,zi)(x_i, z_i) \to (x'_i, z_i) without altering the address ziz_i or retraining the model.

(2) Isolating module contributions under an identical harness, planner, and memory bank

To ensure that our performance gains are not confounded by differences in the execution harness, planner RL training, or seed data, we evaluate module contributions while holding the backbone, harness, and initial memory bank fixed:

Controlled Pipeline Configuration (Fixed Harness & Planner)Read Indexing / RoutingWrite Routing / EvolutionTask Accuracy (%)Token F1 (%)Δ\Delta Acc vs. Full
1. Full GenMemLearned SID (MemRetriever)Learned SID (MemEvolver)49.552.1–
2. w/o Memory (Query-Only)NoneNone41.045.2-8.5
3. w/o MemEvolver (Frozen Bank)Learned SID (MemRetriever)Disabled (Frozen payloads)46.546.8-3.0
4. w/o RL Post-Training (MT Only)Mid-trained SID (MemRetriever)Mid-trained SID (MemEvolver)38.542.6-11.0
5. w/o Beam Search (Greedy SID)Greedy SID decodingLearned SID (MemEvolver)42.045.7-7.5
6. Dense Embedding (Frozen Bank)Dense Cosine SimilarityDisabled (Frozen payloads)44.5–-5.0
7. Dense Embedding + EvolutionDense Cosine Similarity (Re-indexed)Learned MemEvolver payload edits37.041.2-12.5
8. Sparse TF-IDF (Frozen Bank)TF-IDF Lexical MatchingDisabled (Frozen payloads)43.0–-6.5
9. Sparse TF-IDF + EvolutionTF-IDF Lexical Matching (Re-indexed)Learned MemEvolver payload edits40.543.4-9.0
  • Module-level attribution (Table above): The complete memory pipeline gains 8.5 accuracy points over query-only execution. Enabling updates adds 3.0 accuracy points and 5.3 F1 points over the frozen bank. Memory-policy RL adds 11.0 accuracy points and 9.5 F1 points over the MT policies.
  • Matched-budget router comparison & cross-harness transfer (Tables 3–4): On the same memory bank, Top-50 budget, and BGE reranker (Table 4), GenMem outperforms SkillRouter, TF-IDF, and Qwen3-Emb both initially and across 9 evolution steps. Moreover, plugging GenMem skills into three harnesses (StackPlanner, TCRAG, OpenHands) with a frozen backbone and zero planner RL (Table 3) improves 10 of 12 task–harness settings.
AuthorsOpenReview ↗

Rebuttal for Q2

Q2. Handling novel domains under distribution shifts and address-space capacity limits

We appreciate this insightful question regarding the boundary conditions of GenMem. Below, we analyze how GenMem behaves under both moderate and extreme distribution shifts, quantify the utilization and collision dynamics of the fixed SID address space, and explicitly state the corresponding limitations.


(1) Novel skills and distribution shifts: Transferable meta-strategies vs. cold-start boundaries

Because each SID indexes a generalized problem-solving procedure rather than instance-specific facts (averaging 4.52 raw trajectories consolidated per active SID leaf; Table 8), GenMem's behavior under distribution shift depends on whether the target domain shares underlying operational primitives with the seed experience distribution:

  • Moderate distribution shift with shared procedural primitives (Table 3): When evaluated zero-shot on held-out benchmarks absent from seed memory construction—BrowseComp-Plus (Search), DeepResearch Bench (long-horizon research), BIRD (SQL generation), and SWE-bench Verified (repository-level code repair)—GenMem improves 10 out of 12 benchmark–harness settings (e.g., +2.04% to +2.12% RACE on DeepResearch Bench across all three harnesses, and +5.0% execution accuracy on BIRD SQL under StackPlanner). Here, transferable meta-strategies (e.g., multi-source verification, schema inspection, iterative fault localization) successfully bridge the domain gap.
  • Extreme distribution shift to isolated vertical domains (LegalBench diagnostic): Conversely, whenrelevant procedural priors are entirely absent from the initial distribution, cold-start retrieval provides limited utility and early trajectories can introduce noisy updates (noted in Section 6, Limitation 2). To test this boundary directly, we evaluated zero-shot transfer on LegalBench, a specialized legal reasoning benchmark with minimal procedural overlap with our seed bank, and observed limited transfer gains.
  • Online adaptation: While MemEvolver can progressively populate empty SID slots via insert or refine existing slots via revise as new trajectories arrive, rapid few-shot adaptation to entirely foreign domains without rebuilding the RQ-KMeans codebooks remains an open limitation.

(2) Address-space capacity, slot consolidation, and saturation bottlenecks

When constructing the 4-level RQ-KMeans hierarchy (48×16×8×848 \times 16 \times 8 \times 8), our design assumption is that 49,152 discrete leaf slots provide sufficient granularity to partition the manifold of reusable problem-solving strategies (Appendix B.1), whose intrinsic dimensionality is much smaller than the raw number of task instances:

  • Current occupancy and headroom for insert (Appendix D.2 & Table 15): Across 138,243 pre-summary seed experiences, offline clustering and leaf synthesis populate 16,416 active summaries (a 62.22% leaf utilization rate under our weighted metric, with normalized marginal entropy ≥0.97\ge 0.97 across all four levels; Figure 8). This leaves ~37.8% of the leaf address space unoccupied to absorb novel skill insertions during online evolution.
  • How novel synthesized experiences map into the fixed SID space (A+B→CA + B \to C diagnostic): To examine what happens when novel skills are synthesized from existing ones (Appendix H.1, Symbolic Case 1, where complementary verification failures AA and BB are fused into a consolidated procedure CC), we analyzed the target SID assignments of synthesized experience pairs ⟨A,B⟩→C\langle A, B \rangle \to C:
    • 20% of synthesized experiences land on previously unoccupied leaf slots, expanding coverage via insert;
    • 80% map to already occupied leaf slots within the same coarse prefix subtree, where MemEvolver merges them with the existing payload via in-place revise.
  • Effective capacity bottleneck (Slot collision before nominal exhaustion): While in-slot consolidation and multi-SID conditioning allow a fixed codebook to absorb incremental strategy variations without creating new addresses, effective capacity is bounded by routing resolution rather than the nominal 49,152 limit alone. Because each occupied SID holds a single consolidated payload, if the number of genuinely orthogonal skill categories exceeds thegranularity of the frozen RQ centroids, distinct procedures will be routed to the same leaf slot (slot collision), causing interference or over-generalization during revise well before all 49,152 leaves are filled.
Reviewer_1jKyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GenMem applies generative retrieval to an evolving bank of agent experiences. Residual quantization assigns each memory a four-token symbolic identifier (SID). A MemRetriever generates an address and rewrites the retrieved experience for a task-execution harness; a MemEvolver uses the resulting trajectory to insert or revise memory at an address. The address space and model parameters remain fixed during online evolution. The memory policies undergo alignment, supervised mid-training, and GRPO training. The paper reports improvements on interactive tasks and search-based QA, then studies online updates and the effect of providing SID tokens without memory text. The main contribution is the shared generative read/write interface for procedural memory, building on the semantic-ID retrieval methods discussed in Appendix A.

Critical strengths
  1. The memory interface addresses a practical problem. Generating the same structured identifier for retrieval and update targeting gives the two memory policies a common interface. Keeping the identifier fixed while editing its payload is a coherent design for reusable procedural experience. The paper also distinguishes successful, failed, and unverified source trajectories, which is preferable to treating every generated trace as a reliable lesson. The novelty lies mainly in adapting and training this interface for evolving agent memory.

  2. GenMem (RL) reports 93.6% ALFWorld success, 82.8% WebShop success, and 52.8 mean QA F1. The latter is 3.7 points above SDAR. Appendix G.4 includes meaningful component comparisons: freezing the memory bank reduces accuracy from 49.5 to 46.5 and F1 from 52.1 to 46.8 on the stated 200-example evaluation. Appendix E.3 also describes a sensible online protocol in which queries are scored before updates and probe evaluation cannot modify memory. The frozen-bank comparison supports the practical value of online memory updates within this system.

Critical weaknesses
  1. The experimental record does not establish which data and training configuration produced the central comparisons. Table 6 includes Bamboogle, MuSiQue, NQ, and TriviaQA among memory-construction sources, although Table 2 calls these out-of-domain; GPQA is also a source and an evaluation. This does not prove test-example leakage. However, Appendix E.1 leaves exact splits, sample counts, and overlap checks undocumented, while F.1 leaves comparison sources, checkpoints, budgets, and seeds unresolved. The main text gives 130 GRPO steps and Table 11 gives 200; C.3.3 leaves the implemented reward aggregation unconfirmed. Table 4's initial retrieval numbers also match the PT-4L row in Table 18, whereas the mid-trained SID-only model has HR@5 of 3.75 versus 13.15 on the training-sample evaluation described in G.2. These omissions prevent a reliable interpretation of the generalization and comparative-performance claims and are the main reason for my negative recommendation.

  2. The evidence for stable addressing and compositional generalization is narrower than the claims. Keeping an identifier fixed during a payload edit does not establish that the revised payload remains suitable for queries previously mapped there. Appendix B.2 acknowledges this, and Table 23 does not separate fixed addressing from learned retrieval, training, and rewriting. Similarly, the GPQA SID-prefix experiment shows that learned tokens can provide useful conditioning; it does not test composition on held-out combinations. Appendix G.5 acknowledges the missing shuffled/reassigned-token and held-out-combination controls. The paper should distinguish these demonstrated effects from the stronger mechanism claims. The current evidence is insufficient to attribute the gains specifically to those mechanisms.

Questions for authors
  1. Please identify the exact data splits, checkpoints, reward implementation, and training settings used for the main results. In particular, how were instance-level overlaps handled for the Table 6 sources that reappear in Table 2 or GPQA evaluation, how do Tables 4 and 18 relate, and which GRPO schedule was actually used? Which baseline results came from reruns with matched harnesses and inference budgets? These are requests for the records underlying the reported experiments.

  2. What existing evidence separates the effect of fixed addressing from learned retrieval and rewriting? Does any reported experiment evaluate previously unseen SID combinations? If these controls are unavailable, please clarify the intended scope of the addressing-stability and compositional-generalization claims. This affects how much of the methodological contribution can be attributed to the proposed mechanism.

4 comments
AuthorsOpenReview ↗

Rebuttal for Q1

Thank you for the careful review and for recognizing the value of the shared SID read/write interface. We respond to your questions about the experimental record and the attribution of the gains below.

Q1. Which data, training configurations, and comparison settings underlie the reported results?

  • (1) Data sources, ID/OOD labels, and overlap checks. The ID/OOD labels in Table 2 refer to training of the task-execution model. Bamboogle, MuSiQue, Natural Questions, and TriviaQA were excluded from that model's training following Search-R1 but contributed source experiences to the offline memory bank. We will relabel these benchmarks as held out from task-execution model training and clarify their use in memory construction.

    The memory-construction sources contain the following numbers of questions and derived experiences.

    DatasetSource splitSource questions (distinct IDs)Experience entries
    Bamboogletest125500
    MuSiQuetrain2,3289,312
    NQtrain2,0208,080
    TriviaQAtrain2,4739,892
    GPQAtrain4161,664
  • (2) Training schedules and the two-channel Proxy–Audit update. The 200-step setting in Table 11 refers to training StackPlanner, which uses four rollouts per prompt. The 130-step schedule in Section 4.1 refers to post-training MemRetriever and MemEvolver, which use eight rollouts per prompt. Appendix F.2 now distinguishes these two schedules.

    Memory-policy post-training uses a two-channel predictor–corrector update. The Proxy channel provides process-level feedback on memory operations, and the Audit channel uses the downstream task outcome from StackPlanner when given the supplied experience. For MemRetriever rollout ii, the two reward signals are

    = R_{\mathrm{SID},i} + R_{\mathrm{relevance},i} + R_{\mathrm{rewrite},i}, \qquad R_{\mathrm{Audit},i} = R_{\mathrm{outcome},i}.$$ We first take a temporary prediction step along the Proxy gradient. We then evaluate the Audit objective at the predicted parameters and use its gradient to correct the policy update. MemEvolver follows the same update procedure with its own process rewards for memory evolution. We will clarify the prediction and correction steps in Section 3 and specify the rewards for each agent in Appendix C.3. Further details on the two-channel Proxy–Audit update and the rewards for each memory agent are showned in Reviewer 6mQ7 (W4/Q3).
  • (3) How Tables 4 and 18 relate. Table 4 starts from the PT-4L MemRetriever analyzed in Table 18. Its parameters remain frozen while MemEvolver updates the memory bank, allowing us to evaluate memory evolution with a fixed retrieval policy. We will explicitly identify this initialization and the evaluation-sample correspondence in the experimental description.

  • (4) Why the MT retrieval metric decreases. Reference-SID HR@5 falls from 13.15% to 3.75% after MT with SID-only decoding. PT emphasizes query–SID alignment, whereas MT additionally trains reasoning, tool use, and experience rewriting. Interference between these objectives is a plausible explanation for the lower exact-reference hit rate. To assess the downstream consequences, we added evaluations of rewrite quality and task performance.

    MetricPTMT
    Rewrite quality, Qwen-32B judge (/8)5.6007.885
    Downstream exact match (%)19.5721.74
    Downstream token F1 (%)28.1031.17

    In 100 blinded paired judgments, MT wins 76, PT wins 1, and 23 are ties. Downstream EM and F1 increase by 2.17 and 3.07 points, respectively. These results show that MT improves downstream support in this evaluation while reducing reference-SID alignment.

    We additionally evaluated the post-GRPO MemRetriever on 200 questions across nine datasets using constrained SID beam@50 and BGE reranking to top five. It achieves R@50 of 50.50%, HR@1 of 14.50%, and HR@5 of 24.00%. We report this as an additional diagnostic under its own evaluation setup, separate from Table 18.

  • (5) Which baselines were rerun, and what was controlled. We reran FS-RAG, FL-RAG, TCRAG, and AgenticRAG-R1, and ran OpenHands for the cross-harness evaluation. The remaining baseline scores in Tables 1 and 2 were taken from the original publications. These comparisons retain the methods' native harnesses. We will mark each result's source and evaluation configuration in Appendix F.1.

    Table 3 provides paired comparisons with and without GenMem within StackPlanner, TCRAG, and OpenHands. Ten of the twelve harness–task settings improve, one is unchanged, and one decreases by one point. Table 4 uses the same budget of up to 50 candidates and BGE reranking to top five for all retrievers. The component comparisons in Table 23 further quantify the effects of memory updates and policy training within GenMem.

AuthorsOpenReview ↗

Rebuttal for Q2

Q2. Attribution of the gains and the scope of compositional generalization

We agree that demonstrated empirical effects should be strictly distinguished from broader mechanistic claims. Below, we provide additional control experiments to clarify what our current ablations isolate, report an oracle-conditioned SID perturbation diagnostic on GPQA, and explicitly narrow the scope of our claims in the revision.

(1) Isolating memory evolution, policy training, and retrieval setups. In GenMem, MemRetriever selects read addresses conditioned on queries, while MemEvolver selects update addresses conditioned on execution trajectories. Because the two modules observe different contexts, they may target different SIDs for the same task. Architecturally, address invariance guarantees that revising a payload preserves its slot identifier (xi,zi)→(xi′,zi)(x_i, z_i) \to (x'_i, z_i); whether the updated content remains useful for downstream queries is an empirical question evaluated below.

Within the fixed SID space and task-execution harness (Table 23; 200 examples across three evolution rounds):

  • Effect of online memory updates: Freezing the memory bank while retaining the trained MemRetriever and rewriting pipeline reduces accuracy from 49.5% to 46.5% (and Token F1 from 52.1% to 46.8%). Online evolution therefore contributes +3.0 accuracy points (+5.3 F1 points) under fixed SID addressing.
  • Effect of RL post-training: Replacing the GRPO-trained memory policies with mid-trained (MT) checkpoints reduces accuracy from 49.5% to 38.5% (and F1 from 52.1% to 42.6%), isolating a gain of +11.0 accuracy points (+9.5 F1 points) from joint policy optimization.

To further examine how evolution interacts with content-coupled retrieval, we added frozen-bank (no-evolution) controls for dense embedding and TF-IDF retrieval to complement Table 23:

Retrieval / Addressing SetupFrozen-Bank Accuracy (%)Evolving-Bank Accuracy (%)Δ\Delta from Evolution (pts)
GenMem (Learned SID addressing)46.549.5+3.0
Embedding retrieval (Content-coupled)44.537.0-7.5
TF-IDF retrieval (Content-coupled)43.040.5-2.5

Enabling online payload updates improves accuracy under GenMem (+3.0 points) but degrades performance when paired with content-coupled embedding (-7.5 points) or TF-IDF (-2.5 points) retrieval, where payload edits shift the re-indexed representations.

(2) Unseen query–SID pairings and hierarchical SID sensitivity. To test whether SID tokens act as structured conditioning signals rather than memorized query-specific strings, we conducted an oracle-conditioned SID perturbation diagnostic on 100 GPQA questions without providing any memory text payload. For each question, we take the reference 4-level SID (A,B,C,D)(A, B, C, D) associated with a successful seed trajectory and create unseen query–SID pairings by perturbing either the fine-grained suffix levels (C/DC/D, retaining prefix A/BA/B) or the coarse-grained prefix levels (A/BA/B, retaining suffix C/DC/D). For each setting, we evaluate three perturbation operators—level swapping, adjacent-index substitution, and random token substitution—and report the mean accuracy across the three operators:

SID Conditioning Setup (Text-Free)Modified LevelsAccuracy (%)Δ\Delta vs. Gold SID
Unperturbed reference (Gold) SIDNone56.0–
Fine-grained suffix perturbation (mean over swap / adjacent / random)C,DC, D (retain A,BA, B)55.0-1.0
Coarse-grained prefix perturbation (mean over swap / adjacent / random)A,BA, B (retain C,DC, D)51.0-5.0

Perturbing the fine-grained residual levels (C/DC/D) while preserving the coarse prefix (A/BA/B) reduces mean accuracy by only 1.0 point (56.0% →\to 55.0%), whereas perturbing the coarse prefix (A/BA/B) causes a 5.0-point drop (56.0% →\to 51.0%). Qualitative inspection confirms this pattern: on a GPQA organic chemistry question (Gold SID [16, 6, 7, 5]), all three C/DC/D perturbations ([16, 6, 5, 7], [16, 6, 6, 4], [16, 6, 6, 3]) preserve the domain prefix [16, 6] and guide the model to the correct option (A: 3), whereas A/BA/B perturbations ([15, 5, 7, 5], [39, 5, 7, 5]) shift the prefix toward unrelated domains (e.g., entertainment QA or household navigation) and revert the model to its unconditioned error (C: 2).

(3) Domain transfer boundary and summary of manuscript revisions. We also evaluated zero-shot transfer on LegalBench, a specialized domain entirely absent from our seed memory bank, and observed limited transfer gains. This indicates that GenMem's symbolic addressing relies on coverage within the foundational hierarchical codebook structure and does not automatically generalize to distant, unseen vertical domains.

AuthorsOpenReview ↗

Kindly Requesting for Review of Our Rebuttal

Dear Reviewer 1jKy,

We would like to express our gratitude for taking the time to review our paper and provide us with valuable comments. We understand that you have a busy schedule and apologize for any inconvenience caused by this reminder.

With only 2 days left in the discussion period, we would greatly appreciate your feedback on whether our responses have addressed your concerns and whether any further clarification would be helpful for your assessment.

Thank you for your consideration, and we look forward to hearing from you.

Best regards,

Authors of Paper 18220

AuthorsOpenReview ↗

Request for our rebuttal

Dear Reviewer 1jKy,

With only few hours left in the discussion period, we would greatly appreciate your feedback on whether our responses have addressed your concerns and whether any further clarification would be helpful for your assessment.

Thank you for your consideration, and we look forward to hearing from you.

Best regards,

Authors of Paper 18220

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We sincerely thank the Area Chair and all reviewers for their thoughtful feedback. We added controlled experiments, clarified the training and evaluation protocols, and refined the presentation. The main updates are summarized below.

Shared concerns — Attribution and stable addressing.

Under a fixed harness, planner, and initial bank, online evolution adds 3.0 accuracy points, and memory-policy RL adds 11.0 points. We also added the requested frozen-key dense baseline. Across 1,500 updates, GenMem’s accuracy improves from 58.0% to 61.0%, while the baseline declines from 47.0% to 42.0%, supporting coordinated read/write routing over evolving memory.

Reviewer 1jKy

  • Experimental records. We documented data splits, overlap checks, baseline provenance, and the 200-step StackPlanner versus 130-step memory-policy schedules.
  • Retrieval dynamics. We clarified the shared PT-4L initialization in Tables 4 and 18. Additional evaluations show improved downstream support after mid-training and post-GRPO retrieval of 50.50% R@50 and 24.00% HR@5.

Reviewer LsYf

  • Symbolic conditioning. With Beam@50 and BGE reranking, text-free accuracy increases from 23.78% with one SID to 30.27% with ten. Adding SIDs to identical experience text yields another 1–2 accuracy points on GPQA.
  • Contribution and scope. We clarified GenMem’s indexing and read/write modules, added controlled comparisons, and expanded the discussion of domain-transfer and address-space capacity limits.

Reviewer 6mQ7

  • Presentation and controls. We simplified the theoretical framing, described SIDs as coarse-to-fine residual codes, and added the frozen-key comparison above.
  • Data overlap. We explicitly disclose that the same 125 Bamboogle questions support seed construction and evaluation, and identify the GPQA perturbation experiment as an in-bank diagnostic.
  • Training clarification. Section 3, Appendix C.3, and the response to Reviewer 1jKy now consistently describe the two-channel Proxy–Audit predictor–corrector update, including gradient conflict projection and agent-specific rewards.

We appreciate Reviewer 6mQ7’s recognition of the additional experiments and the follow-up questions. We respectfully ask that the added evidence and revisions be considered in the final assessment, and thank the Area Chair and all reviewers again for their constructive guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.