- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets RL-trained memory managers for LLM agents, arguing that QA-correctness-only rewards (as in Memory-R1, MemBuilder) value a memory only through a finite set of sampled questions, leaving question-irrelevant but future-useful information unrewarded. UniMem-R1 adds an intrinsic, per-operation reward motivated by Conditional Mutual Information: the information a candidate memory fragment carries about the current dialogue session, conditioned on a locally retrieved subset of the existing memory bank. In practice this is operationalized as a residual-cosine score — context and candidate embeddings are projected off the span of retrieved memory embeddings via Tikhonov-regularized least squares, justified under a jointly Gaussian embedding model — shaped through a Gaussian "target band", mixed with session-level QA reward, and optimized with GRPO on a Qwen3-4B/8B policy over a four-partition structured memory. Auxiliary format, length, duplication, core-specificity, and attribution regularizers are applied.
Trained only on a 50-dialogue LongMemEval RL split, the method is evaluated on LongMemEval (held-out), LoCoMo (OOD), and MemoryAgentBench (OOD), against RAG, training-free prompting systems (Mem0, MemBuilder-P, an in-house Ours-P), and matched 4B RL baselines (retrained Memory-R1 and an SFT-free MemBuilder-RL). Ablations compare QA-only / CMI-only / CMI+QA and three intrinsic estimators, reward-weight and target-band ablations, training dynamics, and five qualitative case studies.
Critical strengths
- Cross-benchmark transfer evidence for the core claim. All RL methods are trained only on the LongMemEval split (Section 4.1, Table 6); LoCoMo and MemoryAgentBench are strictly out-of-distribution. UniMem-R1-4B beats the matched-backbone, same-training-data MemBuilder-RL-4B by +5.8 on LoCoMo, +12.3 on LME-S, and +20.0 on MemoryAgentBench overall (Tables 1–2), and even exceeds the training-free Qwen3.7-Max prompting baseline with identical memory structure (Ours-P) on LoCoMo and MAB (Table 1). This is a meaningful test of the "generalizable" claim, not just three in-distribution evaluations, and it is the right experimental design for the paper's motivating argument that QA-only rewards overfit to sampled questions.
- Complete implementation transparency. Full action spaces (Table 11), all prompts verbatim (Appendix I), GRPO and reward hyperparameters (Table 8), alternative estimator definitions (Appendix F), paired case studies with memory-bank statistics (Tables 12–16), and an anonymized code repository (footnote 1). This makes the work auditable and the ablations in Tables 3 and 10 interpretable, which matters because the method has many moving parts.
Critical weaknesses
-
What is trained is not a CMI reward, so the paper's information-theoretic framing oversells a hand-tuned heuristic. The stated reward is the conditional mutual information of Eq. 2, but the implementation optimizes something quite different. First, CMI is replaced by a residual-cosine similarity score that the paper itself admits "is not calibrated in nats" (Eq. 6, p. 5). Second, this score is passed through a Gaussian target band centered at μ=0.35 (Eq. 9, p. 5), which makes the reward non-monotone: a memory with a higher estimated information gain receives a lower reward once its score exceeds 0.35. An objective that penalizes high information gain is not "maximize information gain" in any sense. Third, redundancy is in practice controlled by separate hand-crafted penalties (duplication, core-specificity, attribution weighting; Appendix D, Eq. 17) that are never ablated individually — and removing the target band itself only changes accuracy by 0.7 points (Table 10). So the experiments validate "dense similarity-based reward shaping plus hand-tuned regularizers helps," not the CMI story the paper is framed around.
-
The headline "8.5%–57.0%" claim seems inflated, and the paper lacks variance estimates. The 57.0% endpoint (55.1 vs 35.1 on MemoryAgentBench, Tables 1–2) is measured against MemBuilder-RL — the authors' own re-implementation of MemBuilder with its SFT warm-start removed (Section 4.1); the original MemBuilder reports 84.23 on LoCoMo with the same 4B backbone, vs 68.1 for the weakened version here (Table 1). Against the stronger Memory-R1-4B baseline the gain is 30.3%, and the absolute gains over the best RL baseline are +5.8 / +8.0 / +12.8 points per dataset. More importantly, every number in the paper comes from a single training run — no seeds, standard deviations, or significance tests.
Questions for authors
- (Clarification) Eq. 9 makes the reward non-monotone in the estimated CMI: a fragment with residual cosine 0.9 is rewarded less than one at 0.35. What is the justification for a band-pass target on an information-gain estimate, and how were μ=0.35, σ=0.15 chosen other than by monitoring the training reward distribution? Please report sensitivity to μ and σ (Table 10 only ablates removing the band and varying α).
- (Clarification) Were the QA-only and CMI-only ablation runs in Table 3 trained with the same auxiliary regularizers (length, duplication, core-specificity, attribution weighting of Appendix D) as CMI+QA? If any regularizer is exclusive to the CMI+QA configuration, the Table 3 comparison is confounded; please state exactly which terms are active in each row.
- (Suggestion) Report mean ± std over at least 3 seeds for UniMem-R1-4B, MemBuilder-RL-4B, and the QA-only ablation on LongMemEval and MemoryAgentBench. With 50 training dialogues and 70 GRPO steps, several reported gaps (0.7–2.0 points in Tables 3 and 10) are plausibly within noise.
- (Clarification) The original MemBuilder uses SFT warm-start and reports 84.23 on LoCoMo with a Qwen3-4B backbone; your MemBuilder-RL reimplementation scores 68.1 (Table 1). Why is the SFT-free variant the appropriate matched baseline, and how does UniMem-R1 compare against MemBuilder with its SFT stage under your LongMemEval training split?
- (Clarification) Appendix G: the conditioning set is the 8 most-similar same-type entries plus 3 recent episodic entries. How sensitive is the estimator to this local-set size, given that redundancy with respect to entries outside the retrieved set is invisible to the score and handled only by the Appendix-D duplication penalties?
Rebuttal to Reviewer c2Ce
We appreciate your careful review! Changes in our paper are marked in blue, and our responses are as follows:
W1, Q1:
Thank you for your insightful suggestion and question!
CMI Estimator Trade-off. For RL training efficiency, our raw score uses a fast CMI-inspired proxy to jointly measure dialogue relevance and bank non-redundancy. Inspired by [1], we use estimators to model conditional dependence, and following [2], partial correlation can be computed from regression residuals. Motivated by these works, we remove directions related to existing memories from the dialogue and candidate embeddings, then compute the residual cosine.
Gaussian Shaping Rationale & Sensitivity. The Gaussian mapping favors scores near μ rather than always rewarding higher raw scores, avoiding reward hacking [3]. Since a concise fact is much shorter than a full dialogue, high scores (s>0.7) happen when the model copies raw dialogue instead of extracting concise facts, while s→0 indicates irrelevant text. During our experiments, we initially set μ=0.30, which caused large reward swings, so we set μ=0.35 to stabilize training. For σ, since raw scores lie in [0,1], choosing σ around 0.10–0.20 keeps the curve sharp enough to drop near 0 and 1, whereas a large σ (e.g., 0.8) makes it overly flat and fails to separate good and bad candidates [4]. Testing σ∈{0.10,0.15,0.20} below gives similar results on LongMemEval:
Alternative CMI Implementations. In Table 3 of our paper, logprob (67.3) and direct emb. (66.5) CMI proxies do not use residual projection, yet both clearly beat QA-only RL (56.5) on LongMemEval. Thus, gains come from the CMI objective itself rather than one specific heuristic, while residual projection keeps training fast and stable.
Experiments and clarifications about W1 and Q1 are added to Sec. 3.2–3.3, 4.3, and Appendix E (Table 13).
[1] InfoBridge: Balanced Multimodal Integration through Conditional Dependency Modeling, ICCV 25
[2] Conditional Independence Testing with Heteroskedastic Data and Applications to Causal Discovery, NeurIPS 22
[3] Scaling Laws for Reward Model Overoptimization, ICML 23
[4] Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning, ICLR 25
Q2:
Thank you for your question!
To ensure a fair comparison, all RL runs in Table 3 use the exact same Appendix D regularizers (format, length, duplication, core-specificity, and attribution weighting), except the Gaussian shaping (Only for CMI). Notably, even with all duplication and length penalties active, QA-only gets 56.5 vs. 68.8 for CMI+QA, showing hand-crafted penalties alone cannot replace CMI. We have further clarified this in Sec. 4.3.
W2,Q3,Q4:
Thank you for your suggestion and questions!
Matched Baseline & Headline Claim. To study RL reward design without mixing in teacher distillation gains, we train all RL methods without SFT on the exact same backbone and split. Also, original MemBuilder's Claude 4.5 Sonnet SFT trajectories are not released in its public repo, so SFT-free MemBuilder-RL is the cleanest matched baseline. We also removed "8.5%–57.0%" in the Abstract and Intro, and clarified this baseline in Sec. 4.1.
Multi-seed Runs. Due to time and resource limits, we could not finish 3-seed runs in time, and will add full multi-seed results (mean ± std) in the camera-ready version.
Q5:
Thank you for your question!
Following your question, we tested local-set sensitivity by replaying the LongMemEval training trajectory, keeping each session, candidate, and pre-operation bank fixed while varying top-k similar same-type entries and r recent episodic entries. Compared against the full bank (k=∞,r=∞), our default (k=8,r=3) stays within ~5% deviation, showing low sensitivity to local-set size.
In the first table, each cell (
left / right) shows raw-score deviation (%) / Gaussian-shaped reward deviation (%) (std of score differences vs. the full bank):In the second table, each cell (
left / right) shows scoring time per edit (ms) / relative time change (%) vs. the full bank:Although k=16,r=6 has <1% lower deviation, our setting (k=8,r=3) saves 10% more time per edit, which matters across numerous RL actions. Experiments and clarifications are added to Sec. 3.2 and Appendix G (Tables 14–15).