- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MemCredit, a reinforcement learning framework for search agents that compress their interaction history into a compact written memory each turn (the MemSearcher-style workflow). The key idea is that final-answer correctness alone does not tell the agent which memory updates were good: useful evidence can be retrieved but then lost or distorted when the memory is rewritten. MemCredit therefore adds two components on top of standard outcome-based RL training. First, a memory-state reward: after each turn, a frozen copy of the model checks how much more likely the gold answer has become given the updated memory, so the agent is rewarded for what it retains, not just what it retrieves. Second, conversation-level credit assignment: these per-turn gains are normalized across sampled trajectories and distributed over the full reasoning–querying–memory-writing conversation of each turn, in a way that provably cannot overturn the outcome signal. The extra scoring is used only during training; inference is unchanged.
Experiments use Qwen2.5-3B/7B on seven question-answering benchmarks. MemCredit improves average exact match by about 2 points (3B) and 1 point (7B) over MemSearcher, ablations support the two design choices (scoring the retained memory rather than the retrieved evidence, and crediting the whole conversation rather than only the memory-writing tokens), and a diagnostic shows retrieved likelihood gains are preserved in memory more often. An anonymous code release is consistent with the described method.
Critical strengths
- The method targets a real, clearly demonstrated failure mode and fixes it with a careful, verifiable construction. The paper shows concretely why outcome-only RL training is blind here: when all sampled trajectories get the same final reward, the training signal vanishes, and a memory update that preserves key evidence gets exactly the same credit as one that throws it away (Section 2.1, Figure 6). The proposed fix — a per-turn reward for retained evidence, distributed so that it provably can never overturn the final-outcome signal (Eq. 10, Lemmas 2–3) — is bounded and mathematically well-behaved, and the ablations in Table 2 directly validate the two key design choices: rewarding what the memory keeps rather than what was retrieved (−1.61 when changed), and crediting the turn's whole conversation rather than only the memory-writing step (−3.04 when changed). The released code generally matches the paper's description.
Critical weaknesses
-
The reported gains are too small to distinguish from single-run noise, and the one controlled comparison is ambiguous. Every result comes from a single training seed (123) with no variance or significance reporting; Appendix C.4 states "variation across independent training seeds is not measured." Yet the headline 7B gain over MemSearcher is only +1.03 average EM, with most per-dataset gains below one point (Table 1: NQ +0.36, TriviaQA +0.49, HotpotQA −0.07, MuSiQue +0.08); the average is mostly driven by Bamboogle (+4.00), a dataset of only ~125 questions — about 5 questions' difference. RL fine-tuning of 3B/7B models routinely varies by more than this across seeds. In addition, eight of the nine baseline rows are copied from Yue et al. (2026) under different training and retrieval settings (Appendix C.2), so the only apples-to-apples comparison is against MemSearcher — and Table 1's MemSearcher 3B average (43.44) is numerically identical to the "Outcome-only" ablation in Table 2, leaving unclear whether that baseline is the authors' own rerun or a copied number. Without multi-seed runs, the central empirical claim is not established.
-
The reward is circular, its robustness to reward hacking is never audited, and the key retention claim is measured with the training reward itself. The reward is the gold-answer likelihood under a frozen reader that is a copy of the very backbone being trained (Eq. 4; Appendix C.4). A policy could raise this reward by writing plausible answer-like guesses into memory without any supporting evidence — a risk the Ethics Statement itself acknowledges — but the paper provides no audit (e.g., how often memories contain the gold answer without it appearing in any retrieved document). The headline +10.9% retention improvement (Figure 4a) is likewise scored by the same frozen reader that defines the training reward, so it mainly shows that optimizing a metric improves that metric; no different-family reader or human check confirms the retained content is genuinely more useful. Finally, the closest prior work on RL-trained memory consolidation, MEM1 (arXiv:2506.15841, June 2025), is neither cited nor compared, weakening the case that MemSearcher is the only relevant baseline.
Questions for authors
- (Clarification) Is the MemSearcher row in Table 1 your own retrained reproduction under your retriever/training data, or the number reported by Yuan et al. (2026)? Why does its 3B average (43.44) exactly equal the "Outcome-only" ablation in Table 2? If they are the same run, please say so; if not, please report your reproduction.
- (Suggestion) Please run at least 3 training seeds for 3B MemCredit and the outcome-only variant and report mean/std of average EM, plus a paired significance test at the question level. Given Bamboogle's ~125 questions, please also report whether the 7B average gain survives excluding it.
- (Clarification) Reward-hacking audit: in trained-policy trajectories, what fraction of positive-gain memory updates introduce the gold answer string (or a near-alias) that does not appear verbatim in any retrieved observation up to that turn? Does Figure 4(a)'s positive-gain improvement persist when scored by a frozen reader from a different model family (e.g., Llama-3-8B)?
- (Clarification) The benchmark reader context cap is 2,048 tokens (Table 4) while training uses max 5 turns and evaluation allows 20 search turns; does memory ever exceed the scorer cap, and how are truncations handled in the diagnostic of Figure 4(a)?
- (Suggestion) Please cite and position against MEM1 (Zhou et al. 2025) and, if feasible, compare on at least one shared benchmark, since it also trains memory consolidation with RL.
6 comments
- Question-level net solved questions: Across the 51,713 test questions in Test-7, evaluating exact question-level outcomes shows consistent net gains in correctly answered questions across both factual and multi-hop benchmarks, even when excluding Bamboogle (125 questions):
| Benchmark Category | # Questions | 3B Macro EM | 3B Net Correct Qs | 7B 3-Seed EM | 7B 3-Seed Net Correct Qs |
|---|---|---|---|---|---|
| 2WikiMultiHopQA (Multi-hop) | 12,576 | +4.64% (43.50% 48.14%) | +584 Qs | +0.86% (up to +1.26%) | +108 Qs (up to +158 Qs) |
| MuSiQue & HotpotQA (Multi-hop) | 9,822 | +1.82% (30.90% 32.72%) | +147 Qs | +0.35% (up to +0.92%) | +30 Qs (up to +76 Qs) |
| PopQA, NQ & TriviaQA (Factual) | 29,190 | +0.76% (52.40% 53.17%) | +248 Qs | +0.90% (up to +1.00%) | +326 Qs (up to +372 Qs) |
| Bamboogle (Multi-hop) | 125 | +3.20% (41.60% 44.80%) | +4 Qs | +3.73% (up to +4.00%) | +5 Qs (up to +5 Qs) |
| All 4 Multi-hop Benchmarks | 22,523 | +2.87% (36.73% 39.60%) | +735 Qs | +1.32% (up to +1.77%) | +143 Qs (up to +238 Qs) |
| All Benchmarks Excl. Bamboogle (6 sets) | 51,588 | +1.76% (43.75% 45.51%) | +979 Qs | +0.71% (up to +1.01%) | +464 Qs (up to +606 Qs) |
| All 7 Benchmarks (Full Test-7) | 51,713 | +1.97% (3-seed: +2.03%) | +983 Qs | +1.14% (up to +1.33%) | +469 Qs (up to +610 Qs) |
As shown in the table, the improvements are driven by hundreds of additional correctly answered questions on large benchmarks rather than small-sample fluctuations on Bamboogle (+4 to +5 questions at 3B/7B, and +2 questions at 14B). Excluding Bamboogle entirely, MemCredit answers +979 more questions correctly at 3B (+1.76% Macro EM) and an average of +464 more questions (+606 on Seed 1) across the three seeds at 7B (+0.71% Macro EM). On the 22,523 multi-hop questions (HotpotQA, 2Wiki, MuSiQue, and Bamboogle), MemCredit improves Macro EM by +2.87% (+735 correct questions) at 3B and +1.32% (+143 correct questions on average; +238 on Seed 1) at 7B. While terminal EM margins naturally narrow from 3B (+2.03%) to 7B (+1.14%) and 14B (+0.78%) because stronger backbones are more resilient to minor memory omissions when inferring final answers, our diagnostic in W2 below confirms that 7B conditional memory retention still improves substantially (+7.23% under Qwen and +7.97% under Llama-3.1-8B). We will include these question-level breakdowns and multi-seed statistics in Appendix.
W2 & Q3 (Reward circularity, cross-family reader evaluation & training, and reward-hacking audit):
To rule out reward circularity and reward hacking, we (1) rescored all diagnostic trajectories with an independent cross-family Llama-3.1-8B-Instruct reader (+7.27% and +7.97% conditional retention at 3B/7B), (2) retrained the 3B actor from scratch using Llama-3.1-8B-Instruct as the frozen scorer (achieving 45.74% Macro EM, +2.30% over Outcome-only), and (3) audited all positive-gain updates against raw retrieved passages, confirming that 98.93%–98.94% are strictly grounded memory and genuinely ungrounded guesses occur in only 0.21%–0.42% of updates.
In addition to two structural safeguards in our framework, (i) the frozen reader is fixed at the initial checkpoint so the policy cannot alter the judge, and (ii) the local credit is zero-sum and bounded () so it cannot overturn the terminal outcome reward (Lemma 3), we performed three direct empirical audits:
- True conditional retention AND cross-family independent evaluation (Llama-3.1-8B-Instruct): First, we clarify that the 10.9% relative gain in Figure 4(a) (46.4% 51.4%, +5.03% absolute across all scored transitions) measured the unconditional positive-gain rate rather than the conditional retention rate (Appendix E.2). We therefore computed the exact conditional retention rate and post-write loss rate across the 1,152 diagnostic questions, and rescored all transitions using an independent, cross-family Llama-3.1-8B-Instruct reader (never used in training):
| Evaluator Family | Model | Method | Conditional Retention () | Post-Write Loss () | Llama Unconditional Gain () |
|---|---|---|---|---|---|
| Llama-3.1-8B (Independent) | 3B | MemSearcher | 63.91% | 36.09% | 47.40% (546/1,152) |
| Llama-3.1-8B (Independent) | 3B | MemCredit | 71.18% (+7.27% abs. / +11.4% rel.) | 28.82% (-7.27%) | 55.90% (+8.50%) |
| Llama-3.1-8B (Independent) | 7B | MemSearcher | 70.58% | 29.42% | 52.52% (605/1,152) |
| Llama-3.1-8B (Independent) | 7B | MemCredit | 78.55% (+7.97% abs. / +11.3% rel.) | 21.45% (-7.97%) | 56.42% (+3.90%) |
| Qwen2.5 (Frozen Scorer) | 3B | MemSearcher | 56.13% | 43.87% | — (Fig. 4a: 46.4%) |
| Qwen2.5 (Frozen Scorer) | 3B | MemCredit | 67.22% (+11.09% abs. / +19.8% rel.) | 32.78% (-11.09%) | — (Fig. 4a: 51.4%, +5.03% abs. / +10.9% rel.) |
| Qwen2.5 (Frozen Scorer) | 7B | MemSearcher | 61.16% | 38.84% | — |
| Qwen2.5 (Frozen Scorer) | 7B | MemCredit | 68.39% (+7.23% abs. / +11.8% rel.) | 31.61% (-7.23%) | — |
Under the external Llama-3.1-8B-Instruct evaluator, MemCredit improves conditional evidence retention by +7.27% absolute (+11.4% relative) at 3B and +7.97% absolute (+11.3% relative) at 7B (and unconditional positive gains by +8.50% and +3.90%), proving that the retained content is genuinely more informative to an independent model family. Second, to test whether training itself relies on self-family circularity, we retrained the Qwen2.5-3B actor from scratch using Llama-3.1-8B-Instruct as the frozen training scorer:
| Training Scorer (3B Actor) | Reader Family | Test Macro EM | Qwen Conditional Retention | Llama Conditional Retention |
|---|---|---|---|---|
| None (Outcome-only, Table 2) | — | 43.44% | 56.13% | 63.91% |
| Qwen2.5-3B-Instruct (Default) | Same-family | 45.41% (+1.97%) | 67.22% (+11.09%) | 71.18% (+7.27%) |
| Llama-3.1-8B-Instruct (Stronger) | Cross-family | 45.74% (+2.30%) | 68.57% (+12.44%) | 73.23% (+9.32%) |
Training with the cross-family Llama-3.1-8B-Instruct scorer achieves 45.74% Macro EM (+2.30% over Outcome-only's 43.44%, and +0.33% over the default Qwen2.5-3B reader's 45.41%), while further raising conditional retention to 68.57% (+12.44% under Qwen eval) and 73.23% (+9.32% under Llama eval). This confirms that MemCredit does not rely on same-family scorer circularity.
Thank you for your rigorous and constructive review. We have conducted extensive new experiments—including 3-seed training runs at both 3B and 7B, scaling to 14B, a full 7-benchmark evaluation of MEM1, cross-family reader training and evaluation, and a comprehensive reward-hacking audit—to directly address each of your concerns:
W1 & Q1 (Controlled baseline comparisons and the identity of the MemSearcher row in Table 1): Because setting λ=0 in MemCredit reduces mathematically and algorithmically to the exact MemSearcher outcome-only GRPO workflow, both tables report this identical controlled baseline run. We will state this explicitly in Section 4.1 and Appendix C.2, and visually separate our controlled reproductions from external reference numbers in Table 1.
W1 & Q2 (Multi-seed stability at 3B and 7B, question-level reports, and performance excluding Bamboogle): Across 3 independent training seeds at 3B and 7B plus a new 14B run, MemCredit outperforms MemSearcher in every single seed and scale (+2.03 ± 0.39% at 3B, +1.14 ± 0.16% at 7B winning on all benchmarks in the 3-seed average, and +0.78% at 14B). Excluding Bamboogle entirely, the 6-dataset gain remains +1.84 ± 0.37% at 3B, +0.71 ± 0.26% at 7B, and +0.64% at 14B.
Specifically, we trained 3 independent seeds (seeds 123, 42, and 1) under identical code, data, retriever, and training steps for both 3B and 7B models, scaled training to Qwen2.5-14B-Instruct, and evaluated question-level outcomes across all 51,713 test questions: