acceptodds
Under review as a conference paper at ICLR 2027

MemCredit: Reinforcement Learning for Memory-Based Search Agents via Retained-State Credit Assignment

Abstract

Memory-based search agents repeatedly retrieve external evidence, compress it into a compact memory, and condition subsequent search and reasoning on this retained state. However, existing reinforcement learning objectives are dominated by terminal answer rewards and provide little supervision on whether a search turn actually leaves the agent with a more useful memory. As a result, retrieving relevant evidence does not necessarily imply retaining the information needed by later decisions, creating an acquisition–retention gap. We introduce MemCredit, a reinforcement learning framework that jointly redesigns the intermediate reward and its credit assignment for memory-based search agents. First, MemCredit defines a retained-state reward that evaluates each search turn through the change in gold-answer answerability of the agent’s compact memory. Specifically, a frozen reference model scores consecutive memory states, and their potential difference provides a dense signal for whether the current search-memory transition improves the information actually available to future decisions. Second, we develop a memory-aware credit assignment mechanism that converts this retained-state reward into a bounded, group-relative local advantage and combines it additively with the standard trajectory-level outcome advantage. The resulting credit is assigned to the complete search-memory conversation including planning, querying, and memory updating rather than only to the query or memory-writing tokens, while final-answer learning remains governed by the terminal outcome reward. This design also supplies discriminative local supervision when sibling trajectories receive identical terminal rewards. Experiments show that MemCredit outperforms all the baselines over seven datasets. Additional analyses investigate the acquisition–retention gap and examine how retained-state rewards and conversation-level credit assignment contribute to the performance gains.

Primary area:
reinforcement learning
Keywords:
search agent, credit assign, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_pabPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MemCredit, a reinforcement learning framework for search agents that compress their interaction history into a compact written memory each turn (the MemSearcher-style workflow). The key idea is that final-answer correctness alone does not tell the agent which memory updates were good: useful evidence can be retrieved but then lost or distorted when the memory is rewritten. MemCredit therefore adds two components on top of standard outcome-based RL training. First, a memory-state reward: after each turn, a frozen copy of the model checks how much more likely the gold answer has become given the updated memory, so the agent is rewarded for what it retains, not just what it retrieves. Second, conversation-level credit assignment: these per-turn gains are normalized across sampled trajectories and distributed over the full reasoning–querying–memory-writing conversation of each turn, in a way that provably cannot overturn the outcome signal. The extra scoring is used only during training; inference is unchanged.

Experiments use Qwen2.5-3B/7B on seven question-answering benchmarks. MemCredit improves average exact match by about 2 points (3B) and 1 point (7B) over MemSearcher, ablations support the two design choices (scoring the retained memory rather than the retrieved evidence, and crediting the whole conversation rather than only the memory-writing tokens), and a diagnostic shows retrieved likelihood gains are preserved in memory more often. An anonymous code release is consistent with the described method.

Critical strengths
  1. The method targets a real, clearly demonstrated failure mode and fixes it with a careful, verifiable construction. The paper shows concretely why outcome-only RL training is blind here: when all sampled trajectories get the same final reward, the training signal vanishes, and a memory update that preserves key evidence gets exactly the same credit as one that throws it away (Section 2.1, Figure 6). The proposed fix — a per-turn reward for retained evidence, distributed so that it provably can never overturn the final-outcome signal (Eq. 10, Lemmas 2–3) — is bounded and mathematically well-behaved, and the ablations in Table 2 directly validate the two key design choices: rewarding what the memory keeps rather than what was retrieved (−1.61 when changed), and crediting the turn's whole conversation rather than only the memory-writing step (−3.04 when changed). The released code generally matches the paper's description.
Critical weaknesses
  1. The reported gains are too small to distinguish from single-run noise, and the one controlled comparison is ambiguous. Every result comes from a single training seed (123) with no variance or significance reporting; Appendix C.4 states "variation across independent training seeds is not measured." Yet the headline 7B gain over MemSearcher is only +1.03 average EM, with most per-dataset gains below one point (Table 1: NQ +0.36, TriviaQA +0.49, HotpotQA −0.07, MuSiQue +0.08); the average is mostly driven by Bamboogle (+4.00), a dataset of only ~125 questions — about 5 questions' difference. RL fine-tuning of 3B/7B models routinely varies by more than this across seeds. In addition, eight of the nine baseline rows are copied from Yue et al. (2026) under different training and retrieval settings (Appendix C.2), so the only apples-to-apples comparison is against MemSearcher — and Table 1's MemSearcher 3B average (43.44) is numerically identical to the "Outcome-only" ablation in Table 2, leaving unclear whether that baseline is the authors' own rerun or a copied number. Without multi-seed runs, the central empirical claim is not established.

  2. The reward is circular, its robustness to reward hacking is never audited, and the key retention claim is measured with the training reward itself. The reward is the gold-answer likelihood under a frozen reader that is a copy of the very backbone being trained (Eq. 4; Appendix C.4). A policy could raise this reward by writing plausible answer-like guesses into memory without any supporting evidence — a risk the Ethics Statement itself acknowledges — but the paper provides no audit (e.g., how often memories contain the gold answer without it appearing in any retrieved document). The headline +10.9% retention improvement (Figure 4a) is likewise scored by the same frozen reader that defines the training reward, so it mainly shows that optimizing a metric improves that metric; no different-family reader or human check confirms the retained content is genuinely more useful. Finally, the closest prior work on RL-trained memory consolidation, MEM1 (arXiv:2506.15841, June 2025), is neither cited nor compared, weakening the case that MemSearcher is the only relevant baseline.

Questions for authors
  1. (Clarification) Is the MemSearcher row in Table 1 your own retrained reproduction under your retriever/training data, or the number reported by Yuan et al. (2026)? Why does its 3B average (43.44) exactly equal the "Outcome-only" ablation in Table 2? If they are the same run, please say so; if not, please report your reproduction.
  2. (Suggestion) Please run at least 3 training seeds for 3B MemCredit and the outcome-only variant and report mean/std of average EM, plus a paired significance test at the question level. Given Bamboogle's ~125 questions, please also report whether the 7B average gain survives excluding it.
  3. (Clarification) Reward-hacking audit: in trained-policy trajectories, what fraction of positive-gain memory updates introduce the gold answer string (or a near-alias) that does not appear verbatim in any retrieved observation up to that turn? Does Figure 4(a)'s positive-gain improvement persist when scored by a frozen reader from a different model family (e.g., Llama-3-8B)?
  4. (Clarification) The benchmark reader context cap is 2,048 tokens (Table 4) while training uses max 5 turns and evaluation allows 20 search turns; does memory ever exceed the scorer cap, and how are truncations handled in the diagnostic of Figure 4(a)?
  5. (Suggestion) Please cite and position against MEM1 (Zhou et al. 2025) and, if feasible, compare on at least one shared benchmark, since it also trains memory consolidation with RL.
6 comments

Thank you for your rigorous and constructive review. We have conducted extensive new experiments—including 3-seed training runs at both 3B and 7B, scaling to 14B, a full 7-benchmark evaluation of MEM1, cross-family reader training and evaluation, and a comprehensive reward-hacking audit—to directly address each of your concerns:

W1 & Q1 (Controlled baseline comparisons and the identity of the MemSearcher row in Table 1): Because setting λ=0\lambda = 0 in MemCredit reduces mathematically and algorithmically to the exact MemSearcher outcome-only GRPO workflow, both tables report this identical controlled baseline run. We will state this explicitly in Section 4.1 and Appendix C.2, and visually separate our controlled reproductions from external reference numbers in Table 1.

W1 & Q2 (Multi-seed stability at 3B and 7B, question-level reports, and performance excluding Bamboogle): Across 3 independent training seeds at 3B and 7B plus a new 14B run, MemCredit outperforms MemSearcher in every single seed and scale (+2.03 ±\pm 0.39% at 3B, +1.14 ±\pm 0.16% at 7B winning on all benchmarks in the 3-seed average, and +0.78% at 14B). Excluding Bamboogle entirely, the 6-dataset gain remains +1.84 ±\pm 0.37% at 3B, +0.71 ±\pm 0.26% at 7B, and +0.64% at 14B.

Specifically, we trained 3 independent seeds (seeds 123, 42, and 1) under identical code, data, retriever, and training steps for both 3B and 7B models, scaled training to Qwen2.5-14B-Instruct, and evaluated question-level outcomes across all 51,713 test questions:

  1. Multi-seed results at 3B, 7B, and 14B:
Scale / SeedOutcome-only (MemSearcher)MemCreditΔ\Delta Macro EM (All 7)Δ\Delta Macro EM (Excl. Bam.)
3B Seed 123 (Table 1)43.44%45.41%+1.97%+1.76%
3B Seed 4243.19%44.87%+1.68%+1.52%
3B Seed 143.55%46.00%+2.45%+2.24%
3B Mean ±\pm Std (3 seeds)43.39 ±\pm 0.18%45.43 ±\pm 0.57%+2.03 ±\pm 0.39%+1.84 ±\pm 0.37%
7B Mean ±\pm Std (3 seeds) (detailed below)47.82 ±\pm 0.20%48.96 ±\pm 0.33%+1.14 ±\pm 0.16%+0.71 ±\pm 0.26%
14B (Qwen2.5-14B-Instruct)50.44%51.22%+0.78%+0.64% (50.32% →\rightarrow 50.96%)
Seed / Method (7B)NQTriviaQAPopQAHotpotQA2WikiMuSiQueBam.Avg. (All 7)Avg. (w/o Bam.)
Seed 123 / MemSearcher50.7566.7947.8049.5648.5224.4147.2047.8647.97
Seed 123 / MemCredit51.1167.2849.5049.4949.1424.4951.2048.89 (+1.03)48.50 (+0.53)
Seed 42 / MemSearcher50.9166.6847.8249.2848.4624.4545.6047.6047.93
Seed 42 / MemCredit51.1967.1949.6249.3349.1724.6249.6048.67 (+1.07)48.52 (+0.59)
Seed 1 / MemSearcher50.9466.9947.9049.7248.6724.5347.2047.9948.13
Seed 1 / MemCredit51.5867.5949.6250.3649.9325.7350.4049.32 (+1.33)49.14 (+1.01)
Mean ±\pm Std / MemSearcher50.87$\pm$0.1066.82$\pm$0.1647.84$\pm$0.0549.52$\pm$0.2248.55$\pm$0.1124.46$\pm$0.0646.67$\pm$0.9247.82$\pm$0.2048.01$\pm$0.10
Mean ±\pm Std / MemCredit51.29$\pm$0.2567.35$\pm$0.2149.58$\pm$0.0749.73$\pm$0.5549.41$\pm$0.4524.95$\pm$0.6850.40$\pm$0.8048.96$\pm$0.3348.72$\pm$0.36
3-Seed Mean Δ\Delta+0.43+0.53+1.74+0.21+0.86+0.48+3.73+1.14$\pm$0.16+0.71$\pm$0.26
  • Consistency across seeds, benchmarks, and 14B scale: MemCredit outperforms MemSearcher in every single seed at both 3B (+2.03 ±\pm 0.39%) and 7B (+1.14 ±\pm 0.16%), as well as at 14B (+0.78%, 51.22% vs. 50.44%), with the mean gain substantially exceeding the inter-seed standard deviation of either method. Crucially, while Seed 123 at 7B had a −0.07%-0.07\% difference on HotpotQA (49.56% vs. 49.49% in Table 1), in both Seed 42 and Seed 1—as well as in the 3-seed average 7B MemCredit outperforms MemSearcher across all benchmarks.
  • Surviving the exclusion of Bamboogle: Excluding Bamboogle (125 questions), the 6-dataset Macro EM gain remains +1.84 ±\pm 0.37% at 3B (+1.76% on Seed 123), +0.71 ±\pm 0.26% across the three seeds at 7B (+1.01% on Seed 1), and +0.64% at 14B (50.32% →\rightarrow 50.96%, as Bamboogle accounts for only 2 additional correct questions at 14B), confirming that the improvement is not an artifact of Bamboogle.
  1. Question-level net solved questions: Across the 51,713 test questions in Test-7, evaluating exact question-level outcomes shows consistent net gains in correctly answered questions across both factual and multi-hop benchmarks, even when excluding Bamboogle (125 questions):
Benchmark Category# Questions3B Macro EM Δ\Delta3B Net Δ\Delta Correct Qs7B 3-Seed EM Δ\Delta7B 3-Seed Net Δ\Delta Correct Qs
2WikiMultiHopQA (Multi-hop)12,576+4.64% (43.50% →\rightarrow 48.14%)+584 Qs+0.86% (up to +1.26%)+108 Qs (up to +158 Qs)
MuSiQue & HotpotQA (Multi-hop)9,822+1.82% (30.90% →\rightarrow 32.72%)+147 Qs+0.35% (up to +0.92%)+30 Qs (up to +76 Qs)
PopQA, NQ & TriviaQA (Factual)29,190+0.76% (52.40% →\rightarrow 53.17%)+248 Qs+0.90% (up to +1.00%)+326 Qs (up to +372 Qs)
Bamboogle (Multi-hop)125+3.20% (41.60% →\rightarrow 44.80%)+4 Qs+3.73% (up to +4.00%)+5 Qs (up to +5 Qs)
All 4 Multi-hop Benchmarks22,523+2.87% (36.73% →\rightarrow 39.60%)+735 Qs+1.32% (up to +1.77%)+143 Qs (up to +238 Qs)
All Benchmarks Excl. Bamboogle (6 sets)51,588+1.76% (43.75% →\rightarrow 45.51%)+979 Qs+0.71% (up to +1.01%)+464 Qs (up to +606 Qs)
All 7 Benchmarks (Full Test-7)51,713+1.97% (3-seed: +2.03%)+983 Qs+1.14% (up to +1.33%)+469 Qs (up to +610 Qs)

As shown in the table, the improvements are driven by hundreds of additional correctly answered questions on large benchmarks rather than small-sample fluctuations on Bamboogle (+4 to +5 questions at 3B/7B, and +2 questions at 14B). Excluding Bamboogle entirely, MemCredit answers +979 more questions correctly at 3B (+1.76% Macro EM) and an average of +464 more questions (+606 on Seed 1) across the three seeds at 7B (+0.71% Macro EM). On the 22,523 multi-hop questions (HotpotQA, 2Wiki, MuSiQue, and Bamboogle), MemCredit improves Macro EM by +2.87% (+735 correct questions) at 3B and +1.32% (+143 correct questions on average; +238 on Seed 1) at 7B. While terminal EM margins naturally narrow from 3B (+2.03%) to 7B (+1.14%) and 14B (+0.78%) because stronger backbones are more resilient to minor memory omissions when inferring final answers, our diagnostic in W2 below confirms that 7B conditional memory retention still improves substantially (+7.23% under Qwen and +7.97% under Llama-3.1-8B). We will include these question-level breakdowns and multi-seed statistics in Appendix.

W2 & Q3 (Reward circularity, cross-family reader evaluation & training, and reward-hacking audit):

To rule out reward circularity and reward hacking, we (1) rescored all diagnostic trajectories with an independent cross-family Llama-3.1-8B-Instruct reader (+7.27% and +7.97% conditional retention at 3B/7B), (2) retrained the 3B actor from scratch using Llama-3.1-8B-Instruct as the frozen scorer (achieving 45.74% Macro EM, +2.30% over Outcome-only), and (3) audited all positive-gain updates against raw retrieved passages, confirming that 98.93%–98.94% are strictly grounded memory and genuinely ungrounded guesses occur in only 0.21%–0.42% of updates.

In addition to two structural safeguards in our framework, (i) the frozen reader is fixed at the initial checkpoint ψ\psi so the policy cannot alter the judge, and (ii) the local credit is zero-sum and bounded (∣λzi,t∣≤0.2|\lambda z_{i,t}| \le 0.2) so it cannot overturn the terminal outcome reward AioutA_i^{\text{out}} (Lemma 3), we performed three direct empirical audits:

  1. True conditional retention AND cross-family independent evaluation (Llama-3.1-8B-Instruct): First, we clarify that the 10.9% relative gain in Figure 4(a) (46.4% →\rightarrow 51.4%, +5.03% absolute across all scored transitions) measured the unconditional positive-gain rate P(gM>0)P(g^M > 0) rather than the conditional retention rate P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) (Appendix E.2). We therefore computed the exact conditional retention rate P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) and post-write loss rate P(gM≤0∣gE>0)P(g^M \le 0 \mid g^E > 0) across the 1,152 diagnostic questions, and rescored all transitions using an independent, cross-family Llama-3.1-8B-Instruct reader (never used in training):
Evaluator FamilyModelMethodConditional Retention P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) (↑\uparrow)Post-Write Loss P(gM≤0∣gE>0)P(g^M \le 0 \mid g^E > 0) (↓\downarrow)Llama Unconditional Gain P(gM>0)P(g^M > 0) (↑\uparrow)
Llama-3.1-8B (Independent)3BMemSearcher63.91%36.09%47.40% (546/1,152)
Llama-3.1-8B (Independent)3BMemCredit71.18% (+7.27% abs. / +11.4% rel.)28.82% (-7.27%)55.90% (+8.50%)
Llama-3.1-8B (Independent)7BMemSearcher70.58%29.42%52.52% (605/1,152)
Llama-3.1-8B (Independent)7BMemCredit78.55% (+7.97% abs. / +11.3% rel.)21.45% (-7.97%)56.42% (+3.90%)
Qwen2.5 (Frozen Scorer)3BMemSearcher56.13%43.87%— (Fig. 4a: 46.4%)
Qwen2.5 (Frozen Scorer)3BMemCredit67.22% (+11.09% abs. / +19.8% rel.)32.78% (-11.09%)— (Fig. 4a: 51.4%, +5.03% abs. / +10.9% rel.)
Qwen2.5 (Frozen Scorer)7BMemSearcher61.16%38.84%—
Qwen2.5 (Frozen Scorer)7BMemCredit68.39% (+7.23% abs. / +11.8% rel.)31.61% (-7.23%)—

Under the external Llama-3.1-8B-Instruct evaluator, MemCredit improves conditional evidence retention by +7.27% absolute (+11.4% relative) at 3B and +7.97% absolute (+11.3% relative) at 7B (and unconditional positive gains by +8.50% and +3.90%), proving that the retained content is genuinely more informative to an independent model family. Second, to test whether training itself relies on self-family circularity, we retrained the Qwen2.5-3B actor from scratch using Llama-3.1-8B-Instruct as the frozen training scorer:

Training Scorer (3B Actor)Reader FamilyTest Macro EMQwen Conditional RetentionLlama Conditional Retention
None (Outcome-only, Table 2)—43.44%56.13%63.91%
Qwen2.5-3B-Instruct (Default)Same-family45.41% (+1.97%)67.22% (+11.09%)71.18% (+7.27%)
Llama-3.1-8B-Instruct (Stronger)Cross-family45.74% (+2.30%)68.57% (+12.44%)73.23% (+9.32%)

Training with the cross-family Llama-3.1-8B-Instruct scorer achieves 45.74% Macro EM (+2.30% over Outcome-only's 43.44%, and +0.33% over the default Qwen2.5-3B reader's 45.41%), while further raising conditional retention to 68.57% (+12.44% under Qwen eval) and 73.23% (+9.32% under Llama eval). This confirms that MemCredit does not rely on same-family scorer circularity.

Reviewer_pEkLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MemCredit trains memory-based search agents to preserve useful information after each search and memory update. A frozen reader scores the likelihood of an accepted final answer from the question and current memory. The method compares these changes among sibling trajectories, bounds the resulting group-relative credit, and applies it to the complete search–memory conversation, including reasoning, querying, and memory writing. Final-answer generation retains outcome-level supervision, and the frozen reader is not required at inference. On seven QA benchmarks, MemCredit reports macro-average exact match of 45.41% with Qwen2.5-3B and 48.89% with Qwen2.5-7B, compared with 43.44% and 47.86% for MemSearcher.

Critical strengths
  1. The reward target addresses a concrete bottleneck in memory-based search. Sections 2.3 and 3.1 distinguish acquiring useful evidence from preserving it in the state used by later turns. Scoring the updated memory directly is well aligned with that bottleneck.

  2. The component comparisons provide evidence beyond a single final-model score. Table 2 compares the full 3B method (45.41% macro EM) with evidence-state scoring (44.68%), memory-only local credit (44.03%), and outcome-only training (43.44%). These comparisons test the two central design decisions within the same workflow.

  3. The method adds intermediate learning signals without requiring annotated search or memory trajectories. MemCredit uses final-answer annotations and a frozen copy of the initial model, and removes the scorer at inference. Appendix F.2 illustrates a group in which all five siblings answer correctly and terminal supervision cannot distinguish their intermediate updates, while local credit still differentiates them. This is a useful training capability because it can provide additional feedback even after outcome rewards stop distinguishing sampled trajectories, without introducing an inference-time scoring dependency.

Critical weaknesses
  1. The evidence-retention claim is not established by the reported diagnostic. The paper claims a 10.9% relative improvement in retaining the gains from newly retrieved useful evidence. However, the reported measurement concerns the fraction of all scored memory transitions with positive retained-memory gain, rather than the retention rate conditional on useful evidence having been retrieved. These statistics answer different questions: more updates can improve reader-based answerability without demonstrating that a larger proportion of useful retrieval gains survives compression.

  2. The benefit under different levels of memory pressure is not established. The paper motivates MemCredit through information loss during memory compression, but does not evaluate how its advantage changes with the memory budget, the amount of retrieved evidence, or the number of memory updates. Consequently, it remains unclear whether the method is particularly effective when compression becomes more demanding.

  3. The improvement is modest. Compared with MemSearcher, MemCredit improves average exact match by 1.97 percentage points at 3B and 1.03 points at 7B. At 7B, several dataset-level gains are particularly small: 0.08 points on MuSiQue and 0.61 points on 2Wiki, with a slight decrease on HotpotQA. Although the average gains are positive, their magnitude limits the demonstrated practical impact of adding the proposed training machinery.

Questions for authors
  1. Does the reported 10.9% relative improvement measure retention conditional on useful evidence having been retrieved, or the unconditional fraction of memory updates with positive reader-based gain?
  2. How much memory compression do the evaluated tasks actually require? Do the existing results show whether MemCredit’s advantage changes with memory size or the number of successive memory updates?
  3. What additional training cost does frozen-reader scoring introduce compared with outcome-only training, in terms of model forward passes, GPU memory, and wall-clock time?
6 comments

Thank you for your constructive questions. We have conducted additional stratified evaluations, multi-seed and 14B scaling runs, and efficiency measurements to address each of your points:

W1 & Q1 (Clarification on the 10.9% statistic: unconditional positive gain in Figure 4a vs. conditional evidence retention): You are right that the 10.9% figure in Figure 4(a) (46.4% →\rightarrow 51.4%, +5.03% absolute) measured the unconditional positive-gain rate P(gM>0)P(g^M > 0); we apologize for the imprecise wording in the Abstract. Directly computing the conditional retention rate P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) shows an even larger gain for MemCredit: +11.09% absolute (+19.8% relative) at 3B and +7.23% (+11.8% relative) at 7B under Qwen2.5, and +7.27% (3B) / +7.97% (7B) under an independent Llama-3.1-8B reader.

As noted in Appendix E.2 (lines 985–986), measuring retention conditional on useful evidence requires conditioning on positive evidence-state gain gE>0g^E > 0. To directly establish conditional evidence retention P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0)—how often an update with gE>0g^E > 0 successfully preserves a positive memory gain gM>0g^M > 0 after compression—we evaluated the 1,152 diagnostic questions under both the frozen Qwen2.5 scorer and an independent cross-family Llama-3.1-8B-Instruct reader:

EvaluatorModelMethodConditional Retention P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) (↑\uparrow)Post-Write Loss P(gM≤0∣gE>0)P(g^M \le 0 \mid g^E > 0) (↓\downarrow)Llama Unconditional Gain P(gM>0)P(g^M > 0) (↑\uparrow)
Qwen2.5 (Frozen Scorer)3BMemSearcher56.13%43.87%— (Fig. 4a: 46.4%)
Qwen2.5 (Frozen Scorer)3BMemCredit67.22% (+11.09% abs. / +19.8% rel.)32.78% (-11.09%)— (Fig. 4a: 51.4%, +5.03% abs. / +10.9% rel.)
Qwen2.5 (Frozen Scorer)7BMemSearcher61.16%38.84%—
Qwen2.5 (Frozen Scorer)7BMemCredit68.39% (+7.23% abs. / +11.8% rel.)31.61% (-7.23%)—
Llama-3.1-8B (Independent)3BMemSearcher63.91%36.09%47.40%
Llama-3.1-8B (Independent)3BMemCredit71.18% (+7.27% abs. / +11.4% rel.)28.82% (-7.27%)55.90% (+8.50%)
Llama-3.1-8B (Independent)7BMemSearcher70.58%29.42%52.52%
Llama-3.1-8B (Independent)7BMemCredit78.55% (+7.97% abs. / +11.3% rel.)21.45% (-7.97%)56.42% (+3.90%)

When conditioned on useful evidence having been retrieved (gE>0g^E > 0), MemCredit increases the retention rate by +11.09% absolute (+19.8% relative) at 3B and +7.23% absolute (+11.8% relative) at 7B under Qwen2.5, and by +7.27% (3B) and +7.97% (7B) under the independent Llama-3.1-8B reader. Moreover, MemCredit preserves a larger median fraction of the evidence gain (gM/gEg^M / g^E: 0.76 vs. 0.58 at 3B; 0.90 vs. 0.89 at 7B), and when no new positive evidence is retrieved (gE≤0g^E \le 0), it more often improves answerability by synthesizing existing facts (P(gM>0∣gE≤0)P(g^M > 0 \mid g^E \le 0): 31.8% vs. 28.9% at 3B; 34.0% vs. 31.1% at 7B, consistent with Section 2.3). We will revise the Abstract, Section 1, and Section 4.3 to report both the unconditional positive-gain rate and the conditional retention rate.

W2 & Q2 (Degree of memory compression and performance under varying memory pressure): Analyses show that MemCredit improves conditional retention across all memory-length quartiles (+6.61% to +14.82%) and yields its largest downstream EM gains on high-compression questions requiring ≥3\ge 3 successive updates (+2.29% on Test-7 at 7B and +8.19% on the 3B diagnostic set).

Below are the three stratified analyses quantifying the compression bottleneck and MemCredit's performance under varying memory pressure:

  1. Per-turn compression ratio (20$\times$–50$\times$): Across the benchmarks, retrieved observations oi,to_{i,t} average 1,600–2,100 tokens per turn, whereas written memories Mi,tM_{i,t} average 49.8 vs. 84.7 tokens (3B) and 38.6 vs. 91.3 tokens (7B) for MemSearcher vs. MemCredit. This represents a 20$\times$–50$\times$ compression ratio at every turn. While memory length grows mildly over successive updates (e.g., 69 →\rightarrow 86 →\rightarrow 102 →\rightarrow 107 tokens at turns t=1,2,3,≥4t = 1, 2, 3, \ge 4 for 3B MemCredit), it remains under 110 tokens—more than 15$\times$ smaller than a single turn's raw retrieved passages. MemCredit writes ~35–50 more tokens than MemSearcher specifically to retain entity bindings and bridge facts that would otherwise be lost during compression.

  2. Conditional retention across memory-length quartiles (3B): To examine whether MemCredit consistently improves evidence retention across different memory sizes, we stratify the 3B diagnostic transitions (2,304 total updates across both methods, 1,152 per method) into four shared quartiles based on the token length of the written memory MtM_t. Memory length is measured using the Qwen2.5-3B-Instruct tokenizer on the written memory body alone (excluding the question, retrieved passages, and special tokens), with shared quartile boundaries computed over the pooled transitions (assigning boundary ties to the lower quartile so both methods are compared within identical token-length intervals): Q1 (0–33 tokens, N=597N=597), Q2 (34–58 tokens, N=567N=567), Q3 (59–80 tokens, N=575N=575), and Q4 (81–252 tokens, N=565N=565). Evaluating conditional retention P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) under the untruncated full-observation Figure 4(a) scoring protocol yields:

Memory-Length Quartile (Token Range)Pooled Updates (NN)MemSearcher (3B)MemCredit (3B)Δ\Delta Conditional Retention
Q1 (0–33 tokens, ultra-short)59761.92%70.00%+8.08%
Q2 (34–58 tokens, compact)56766.24%75.00%+8.76%
Q3 (59–80 tokens, moderate)57556.19%71.01%+14.82%
Q4 (81–252 tokens, detailed)56557.53%64.14%+6.61%
Overall (Full-observation protocol)2,30461.53% (379/616)69.27% (417/602)+7.74% (+11.09% under capped A1 protocol in W1)

Two key findings emerge from this stratified breakdown:

  • Consistent retention gains across all memory lengths: MemCredit achieves higher conditional evidence retention than MemSearcher in every single token-length quartile, with improvements ranging from +6.61% to +14.82% (and +7.74% overall under the full-observation protocol, complementing the +11.09% gain under the 2,048-capped A1 renderer in W1). Even in the longest memory stratum (Q4, 81–252 tokens), where the writer must synthesize and maintain multiple multi-hop facts simultaneously, MemCredit improves retention by +6.61% (64.14% vs. 57.53%), and achieves its largest margin in Q3 (59–80 tokens, +14.82%).
  • Shifting away from over-compressed, lossy summaries: While both methods retrieve useful evidence (gE>0g^E > 0) at nearly identical rates across the 1,152 transitions (616 vs. 602 transitions, ~53.5% vs. ~52.3%), MemSearcher places the vast majority of its updates into the ultra-short Q1–Q2 bins (438/616 updates ≤58\le 58 tokens), frequently dropping bridge entities. In contrast, MemCredit shifts updates into the more informative Q3–Q4 range (444/602 updates in 59–252 tokens) while simultaneously maintaining a much higher retention rate (71.01% in Q3 and 64.14% in Q4 vs. MemSearcher's 56.19% and 57.53%).
  1. Downstream EM on high-update-count questions (≥3\ge 3 searches): Grouping test questions by the number of searches required by the baseline confirms that MemCredit yields its largest accuracy gains on complex questions requiring ≥3\ge 3 successive search-and-compress updates: on the 4,714 such questions in Test-7 (7B), pooled EM rises from 38.74% to 41.03% (+2.29%), and on the 3B diagnostic subset, EM on ≥3\ge 3-turn questions rises from 11.48% to 19.67% (+8.19%). Likewise, across the four multi-hop benchmarks in Table 1 (HotpotQA, 2Wiki, MuSiQue, Bamboogle), MemCredit improves average EM by +2.87% (36.73% →\rightarrow 39.60%) at 3B and +1.16% (42.42% →\rightarrow 43.58%) at 7B.

W3 (Magnitude and statistical reliability of the improvements): Across 3 matched training seeds at 3B and 7B as well as a new 14B run, MemCredit improves over MemSearcher in every single seed and scale (+2.03 ±\pm 0.39% at 3B, +1.14 ±\pm 0.16% at 7B across all benchmarks in the 3-seed average, and +0.78% at 14B).

Specifically, we ran 3 matched training seeds (seeds 123, 42, and 1) under identical settings for both 3B and 7B, scaled training to Qwen2.5-14B-Instruct, and evaluated exact question-level outcomes across all 51,713 test questions in Test-7:

  1. Multi-seed training stability at 3B, 7B, and 14B:
Scale / SeedOutcome-only (MemSearcher)MemCreditΔ\Delta Macro EM (All 7)Δ\Delta Macro EM (Excl. Bam.)
3B Seed 123 (Table 1)43.44%45.41%+1.97%+1.76%
3B Seed 4243.19%44.87%+1.68%+1.52%
3B Seed 143.55%46.00%+2.45%+2.24%
3B Mean ±\pm Std (3 seeds)43.39 ±\pm 0.18%45.43 ±\pm 0.57%+2.03 ±\pm 0.39%+1.84 ±\pm 0.37%
7B Mean ±\pm Std (3 seeds) (detailed below)47.82 ±\pm 0.20%48.96 ±\pm 0.33%+1.14 ±\pm 0.16%+0.71 ±\pm 0.26%
14B (Qwen2.5-14B-Instruct)50.44%51.22%+0.78%+0.64% (50.32% →\rightarrow 50.96%)
Seed / Method (7B)NQTriviaQAPopQAHotpotQA2WikiMuSiQueBam.Avg. (All 7)Avg. (w/o Bam.)
Seed 123 / MemSearcher50.7566.7947.8049.5648.5224.4147.2047.8647.97
Seed 123 / MemCredit51.1167.2849.5049.4949.1424.4951.2048.89 (+1.03)48.50 (+0.53)
Seed 42 / MemSearcher50.9166.6847.8249.2848.4624.4545.6047.6047.93
Seed 42 / MemCredit51.1967.1949.6249.3349.1724.6249.6048.67 (+1.07)48.52 (+0.59)
Seed 1 / MemSearcher50.9466.9947.9049.7248.6724.5347.2047.9948.13
Seed 1 / MemCredit51.5867.5949.6250.3649.9325.7350.4049.32 (+1.33)49.14 (+1.01)
Mean ±\pm Std / MemSearcher50.87$\pm$0.1066.82$\pm$0.1647.84$\pm$0.0549.52$\pm$0.2248.55$\pm$0.1124.46$\pm$0.0646.67$\pm$0.9247.82$\pm$0.2048.01$\pm$0.10
Mean ±\pm Std / MemCredit51.29$\pm$0.2567.35$\pm$0.2149.58$\pm$0.0749.73$\pm$0.5549.41$\pm$0.4524.95$\pm$0.6850.40$\pm$0.8048.96$\pm$0.3348.72$\pm$0.36
3-Seed Mean Δ\Delta+0.43+0.53+1.74+0.21+0.86+0.48+3.73+1.14$\pm$0.16+0.71$\pm$0.26

MemCredit improves over MemSearcher in every seed at both 3B (+2.03 ±\pm 0.39%) and 7B (+1.14 ±\pm 0.16%), as well as at 14B (+0.78% overall, and +0.64% excluding Bamboogle). While Seed 123 at 7B showed a minor −0.07%-0.07\% difference on HotpotQA (49.56% vs. 49.49% in Table 1), in both Seed 42 and Seed 1—as well as in the 3-seed average 7B MemCredit outperforms MemSearcher across all benchmarks (including HotpotQA, with 2Wiki reaching +1.26% and MuSiQue +1.20% on Seed 1, and +0.71 ±\pm 0.26% average gain even when excluding Bamboogle).

Reviewer_2TEeOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MemCredit, a reinforcement-learning framework for search agents that compress their retrieval history into textual and compact memory. The primary motivation is that sometimes even though the correct memory is retrieved, the compact process can lost or distort the evidence. To solve this problem, MemCredit introduce the reward signal computed from "changes in answerability across consecutive memories" and the score of each memory-answer pair is computed from length-normalized ground-truth log-likelihood by a frozen judge. MemCredit further assign credits to local for search-memory conversation that produce the answer. Finally, experiments on 7 QA benchmarks report some improvement compared to MemSearcher on Qwen2.5-3B and Qwen2.5-7B.

Critical strengths
  1. The paper tries to tackle a well-defined and well-motivated problem in memory-based search. Specifically, the observation that evidence retrieval recall or accuracy does not mean retrieved evidence will keep intact before the model can reason on it. I think evaluation on the updated memory directly offers a reasonable training objective for this problem. The figure 1's entity confusion case example also illustrates a concrete failure mode that retrieval quality does not imply reason based on correct summary.

  2. The method and algorithm is generally practical and mathematically grounded. MemCredit requires final-answer annotation only with no inference-time scorer. Further, assigning local feedback to both retrieval and memory-writing actions are clearly reasonable and supported by the fixed-state interaction-gradient derivation.

  3. The experiments provide solid support for this design. Evaluations covers 7 benchmarks and two model sizes, and results show improvements on most or all datasets across all benchmarks and two model variants. The ablation study further demonstrates proposed combination of actions received rewards and the state being scored bring improvement.

Critical weaknesses
  1. Though the motivation and method are both sound, the improvement is still modest and not fully reliable. Specifically, the advantage margin of MemCredit over MemSearch can be viewed as modest at 3B, but mostly marginal at 7B (5/7 benchmarks have advantages of < 1% on benchmark). Thus, a natural concern for this method is that how much it really improve over MemSearch and is it generalizable to even larger model?

  2. It would also be good to distinguish this method from simpler ones. As mentioned in the related work, using intermediate states and turn based rewards are well known methods. The difference here is that MemCredit used these in conjunction with a memory, and particular normalization and credit span. However, during experiments there is no distinction between the gain and memory scores, or the zero sum projection and centering/clipping. There is also no sensitivity analysis on the local weight or clipping limit. The lemmas show useful properties of the coefficients but do not show that they are necessary to achieve the result. By comparing to simpler reward systems we can see which elements are necessary.

  3. Small issues: a. The figure 1's "Chinna" should be "China" in "MemCredit / written memory" block b. Despite the codebase implies the training data, it's better to note somewhere in the paper, since it's also an important part of your experiment.

Questions for authors
  1. Evidence retention: The abstract reports a 10.9% improvement in retaining gains when retrieved evidence increases answerability. However, Section 4.3 and Appendix E.2 describe an unconditional positive-memory-gain rate. Can you clarify this?

  2. Is the performance sensitive to the local-reward weight? And does the improvement depend on model size?

5 comments

Thank you for your positive assessment of our problem formulation and theoretical grounding, as well as your constructive questions regarding statistical reliability, model scaling, simpler reward designs, and hyperparameter sensitivity. We have conducted new multi-seed runs, a 14B scale extension, ablations, and diagnostics to address each point:

Q1 (Clarification on the 10.9% figure in Abstract vs. Section 4.3: unconditional positive gain vs. conditional evidence retention):

The 10.9% figure in Figure 4(a) (46.4% →\rightarrow 51.4%, +5.03% absolute) was indeed the unconditional positive-gain rate P(gM>0)P(g^M > 0) rather than the conditional retention rate; we apologize for the imprecise wording in the Abstract. Directly computing the true conditional retention rate P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) reveals an even stronger improvement for MemCredit: +11.09% absolute (+19.8% relative) at 3B and +7.23% (+11.8% relative) at 7B under Qwen2.5, and +7.27% (3B) / +7.97% (7B) under an independent Llama-3.1-8B reader.

Specifically, Section 4.3 and Figure 4(a) accurately reported the unconditional positive-gain rate P(gM>0)P(g^M > 0) across all scored memory transitions, which rises from 46.4% (MemSearcher) to 51.4% (MemCredit): an absolute gain of +5.03% (+4.17% on HotpotQA, +7.81% on 2Wiki, +3.12% on MuSiQue) and a relative gain of +10.9%. To directly measure the true conditional retention rate P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) (how often a positive evidence-state gain gE>0g^E > 0 from retrieval is preserved as a positive memory-state gain gM>0g^M > 0 after compression), we evaluated the diagnostic questions under both the frozen Qwen2.5 scorer and an independent cross-family Llama-3.1-8B-Instruct reader:

EvaluatorModelMethodConditional Retention P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) (↑\uparrow)Post-Write Loss P(gM≤0∣gE>0)P(g^M \le 0 \mid g^E > 0) (↓\downarrow)Llama Unconditional Gain P(gM>0)P(g^M > 0) (↑\uparrow)
Qwen2.5 (Frozen Scorer)3BMemSearcher56.13%43.87%— (Fig. 4a: 46.4%)
Qwen2.5 (Frozen Scorer)3BMemCredit67.22% (+11.09% abs. / +19.8% rel.)32.78% (-11.09%)— (Fig. 4a: 51.4%, +5.03% abs. / +10.9% rel.)
Qwen2.5 (Frozen Scorer)7BMemSearcher61.16%38.84%—
Qwen2.5 (Frozen Scorer)7BMemCredit68.39% (+7.23% abs. / +11.8% rel.)31.61% (-7.23%)—
Llama-3.1-8B (Independent)3BMemSearcher63.91%36.09%47.40%
Llama-3.1-8B (Independent)3BMemCredit71.18% (+7.27% abs. / +11.4% rel.)28.82% (-7.27%)55.90% (+8.50%)
Llama-3.1-8B (Independent)7BMemSearcher70.58%29.42%52.52%
Llama-3.1-8B (Independent)7BMemCredit78.55% (+7.97% abs. / +11.3% rel.)21.45% (-7.97%)56.42% (+3.90%)

We will revise the Abstract, Section 1, and Section 4.3 to clearly report both the unconditional positive-gain rate (Figure 4a) and this conditional evidence retention rate.

W2 & Q2 (part 1) (Comparison with simpler designs; sensitivity to λ\lambda and cc; role of Lemmas 2–3): Controlled ablations confirm that potential difference scoring gMg^M outperforms raw memory scoring Φ(Mt)\Phi(M_t) by +0.90% (45.41% vs. 44.51%), joint zero-sum projection outperforms coordinatewise clipping (which violates zero-sum balance in >42% of groups), and MemCredit is robust across hyperparameters λ∈{0.05,0.10,0.20}\lambda \in \{0.05, 0.10, 0.20\} and c∈{1,2,4}c \in \{1, 2, 4\}.

We evaluated the exact component ablations and hyperparameter variations you requested on the 3B model under the protocol of Table 2 (Appendix I):

Variant / AblationMacro EM (%)Δ\Delta vs. Full MemCredit (Absolute / Relative)
MemCredit (Full: λ=0.1,c=2\lambda = 0.1, c = 2, gain gMg^M, joint projection)45.41%—
Reward Target: Absolute potential Φ(Mt)\Phi(M_t) instead of gain gMg^M44.51%-0.90% (-1.98% relative)
Reward Target: Evidence-state scoring (A1, Table 2)44.68%-0.73% (-1.61% relative)
Credit Span: Query only44.77%-0.64% (-1.41% relative)
Credit Span: Memory-only credit (A2, Table 2)44.03%-1.38% (-3.04% relative)
Normalization: Coordinatewise clipping instead of joint projection45.23%-0.18% (-0.40% relative)
Sensitivity to local weight λ\lambda: λ=0.05\lambda = 0.05 / λ=0.20\lambda = 0.2044.93% / 45.24%-0.48% / -0.17%
Sensitivity to clipping bound cc: c=1.0c = 1.0 / c=4.0c = 4.045.13% / 45.17%-0.28% / -0.24%
Baseline: Outcome-only (λ=0\lambda = 0, MemSearcher, Table 2)43.44%-1.97% (-4.34% relative)

Three conclusions emerge from these comparisons:

  1. Memory gain (gMg^M) vs. absolute memory score (Φ(Mt)\Phi(M_t)): Scoring the potential difference gMg^M outperforms scoring the raw memory potential Φ(Mt)\Phi(M_t) by +0.90% (45.41% vs. 44.51%). An absolute score conflates the quality of the current update with information already accumulated in previous turns, whereas gMg^M isolates the marginal contribution of the current transition.
  2. Joint zero-sum box projection vs. coordinatewise centering/clipping: The two are not equivalent in practice. Across 33 logged training batches (11,927 eligible groups at 3B and 12,313 at 7B), at least one sibling coefficient saturates at ±c\pm c in 45.1% (3B) and 46.7% (7B) of groups. In 42.5% (3B) and 46.4% (7B) of groups, naive coordinatewise clipping violates the zero-sum group balance, whereas our joint projection preserves exact zero-sum balance (to within 4×10−74 \times 10^{-7}) and improves Macro EM from 45.23% to 45.41% (+0.18%).
  3. Hyperparameter robustness and role of Lemmas 2–3: MemCredit consistently outperforms the λ=0\lambda=0 baseline (43.44%) across all tested weights λ∈{0.05,0.10,0.20}\lambda \in \{0.05, 0.10, 0.20\} (44.93%–45.41%) and box bounds c∈{1,2,4}c \in \{1, 2, 4\} (45.13%–45.41%), demonstrating that performance does not rely on knife-edge tuning. We agree that Lemmas 2–3 establish desirable structural guarantees (order preservation, gain alignment, and bounded sign-preserving refinement) rather than empirical necessity alone; we will clarify this distinction in Section 3.2 and include the empirical ablation table in Appendix I.

W1 & Q2 (part 2) (Performance margin, multi-seed reliability, and generalizability across model scales): Multi-seed runs at 3B and 7B, together with a new Qwen2.5-14B-Instruct experiment, demonstrate that MemCredit reliably improves over MemSearcher across every seed and model scale: +2.03 ±\pm 0.39% at 3B, +1.14 ±\pm 0.16% at 7B, and +0.78% at 14B (51.22% vs. 50.44%).

Specifically, we trained 3 independent seeds (seeds 123, 42, and 1) under identical code, data, retriever, and training steps for both 3B and 7B models, and extended training to 14B:

Scale / SeedOutcome-only (MemSearcher)MemCreditΔ\Delta Macro EM (All 7)Δ\Delta Macro EM (Excl. Bam.)
3B Seed 123 (Table 1)43.44%45.41%+1.97%+1.76%
3B Seed 4243.19%44.87%+1.68%+1.52%
3B Seed 143.55%46.00%+2.45%+2.24%
3B Mean ±\pm Std (3 seeds)43.39 ±\pm 0.18%45.43 ±\pm 0.57%+2.03 ±\pm 0.39%+1.84 ±\pm 0.37%
7B Mean ±\pm Std (3 seeds) (detailed below)47.82 ±\pm 0.20%48.96 ±\pm 0.33%+1.14 ±\pm 0.16%+0.71 ±\pm 0.26%
14B (Qwen2.5-14B-Instruct)50.44%51.22%+0.78%+0.64% (50.32% →\rightarrow 50.96%)
Seed / Method (7B)NQTriviaQAPopQAHotpotQA2WikiMuSiQueBam.Avg. (All 7)Avg. (w/o Bam.)
Seed 123 / MemSearcher50.7566.7947.8049.5648.5224.4147.2047.8647.97
Seed 123 / MemCredit51.1167.2849.5049.4949.1424.4951.2048.89 (+1.03)48.50 (+0.53)
Seed 42 / MemSearcher50.9166.6847.8249.2848.4624.4545.6047.6047.93
Seed 42 / MemCredit51.1967.1949.6249.3349.1724.6249.6048.67 (+1.07)48.52 (+0.59)
Seed 1 / MemSearcher50.9466.9947.9049.7248.6724.5347.2047.9948.13
Seed 1 / MemCredit51.5867.5949.6250.3649.9325.7350.4049.32 (+1.33)49.14 (+1.01)
Mean ±\pm Std / MemSearcher50.87$\pm$0.1066.82$\pm$0.1647.84$\pm$0.0549.52$\pm$0.2248.55$\pm$0.1124.46$\pm$0.0646.67$\pm$0.9247.82$\pm$0.2048.01$\pm$0.10
Mean ±\pm Std / MemCredit51.29$\pm$0.2567.35$\pm$0.2149.58$\pm$0.0749.73$\pm$0.5549.41$\pm$0.4524.95$\pm$0.6850.40$\pm$0.8048.96$\pm$0.3348.72$\pm$0.36
3-Seed Mean Δ\Delta+0.43+0.53+1.74+0.21+0.86+0.48+3.73+1.14$\pm$0.16+0.71$\pm$0.26
  1. Statistical reliability: MemCredit achieves a positive improvement in every single seed at both 3B (+2.03 ±\pm 0.39%, +983 net solved questions) and 7B (+1.14 ±\pm 0.16%, +469 net solved questions on average) (and on the four multi-hop datasets in Section 4.2, improves from 36.73% to 39.60% (+2.87%) at 3B and from 42.42% to 43.58% (+1.16%) at 7B). Notably, in both Seed 42 and Seed 1 as well as the 3-seed mean, 7B MemCredit outperforms MemSearcher across all 7 out of 7 benchmarks (including HotpotQA), maintaining a +0.71 ±\pm 0.26% mean gain even when excluding Bamboogle.
  2. Generalizability to 14B and larger readers: Scaling the backbone to Qwen2.5-14B-Instruct shows that MemCredit continues to outperform MemSearcher across the seven benchmarks (51.22% vs. 50.44%, +0.78%). Importantly, at 14B Bamboogle accounts for only 2 additional correct questions (+1.60%), so the 6-dataset average excluding Bamboogle improves by +0.64% (50.32% →\rightarrow 50.96%, ~385 additional solved questions across the six large benchmarks). Furthermore, as shown in Q1, at 7B MemCredit improves conditional evidence retention by +7.23% (Qwen) and +7.97% (Llama-3.1-8B), and training the 3B actor with a larger cross-family Llama-3.1-8B-Instruct frozen reader boosts Macro EM to 45.74% (+2.30% over MemSearcher), confirming that our framework generalizes consistently from 3B to 7B and 14B.
Reviewer_bWdNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a reinforcement learning framework for improving memory-based LLM search agents through intermediate reward design and conversation-level credit assignment. The authors introduce a retained-state reward, which measures the change in answerability between consecutive memories using the gold-answer likelihood under a frozen language model. The resulting intermediate reward is normalized among sampled trajectories and incorporated into the advantages assigned to reasoning, search-query generation, and memory-writing actions, while retaining the original outcome-level supervision. Experiments using Qwen2.5-3B and Qwen2.5-7B show improvements over MemSearcher. Additional ablations and memory-quality analyses provide evidence that evaluating retained memories and applying intermediate credit to complete search interactions can improve performance.

Critical strengths
  1. The paper identifies a meaningful limitation of memory-based search agents. The authors distinguish between retrieving relevant information and successfully preserving that information in memory.

  2. The proposed intermediate reward is intuitive and requires no additional intermediate annotations. This makes MemCredit straightforward to incorporate into existing RL-based search-agent training.

  3. Rather than assigning intermediate rewards exclusively to memory-writing tokens, MemCredit also applies them to reasoning and search-query generation.

  4. The experiments cover multiple QA benchmarks and include relevant component ablations, using two different LLM sizes.

Critical weaknesses
  1. MemCredit uses the likelihood of the gold answer under a frozen reader to measure memory quality. However, a higher likelihood does not necessarily mean that the memory contains more accurate or better-supported information. Evaluating factual consistency and evidence preservation independently of the frozen reader would benefit the evaluation.

  2. The performance improvements in Table 1 are relatively modest, and statistical reliability is unclear. While the gains are positive, the 7B improvement is relatively small.

  3. The paper includes two relevant ablations, A1 and A2. The results provide evidence that both retained-state scoring and conversation-level supervision are useful. However, they do not fully demonstrate that the specific credit-assignment design is preferable to simpler alternatives.

Questions for authors
  1. The proposed method assigns the same intermediate advantage to reasoning, query generation, and memory-writing tokens. How about evaluating alternative credit spans, such as query generation plus memory writing, or query generation alone?

  2. Since group-relative credit compares memory gains at the same search turn, rather than from identical memory states, will this approach disadvantage trajectories that have already accumulated highly informative memories?

  3. How sensitive is MemCredit to the choice of the frozen reader? For example, would using a different model or a stronger reader produce similar improvements?

  4. Figure 4(c) suggests that MemCredit can produce longer responses during training. I'm also curious about the average number of search turns, generated tokens, and inference latency of the proposed method compared with MemSearcher.

7 comments

Thank you for your constructive feedback and supportive assessment. We have conducted additional multi-seed training runs, scale extensions to 14B, offline audits, and ablations. Below are our point-by-point responses:

W1 (Likelihood vs. factual support; evaluation independent of the frozen reader):

To evaluate memory quality independently of the training scorer, we conducted both a cross-family reader evaluation (using Llama-3.1-8B-Instruct) and a reader-free textual grounding audit against raw retrieved passages. Both evaluations confirm that MemCredit substantially improves conditional evidence retention (+7.27% to +11.09%) and preserves +21.9% more verbatim-grounded target entities while maintaining >99.5% factual consistency.

Specifically, we agree that reader likelihood measures answerability rather than certified factual support. In Section 4.3 (Figure 4a), we reported that MemCredit increases the unconditional positive-gain rate P(gM>0)P(g^M > 0) under the frozen Qwen2.5 scorer from 46.4% to 51.4% (+5.03% absolute, +10.9% relative). To directly measure conditional evidence retention P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) (how often useful retrieved evidence with gE>0g^E > 0 survives compression with gM>0g^M > 0) and verify factual support independently of the frozen reader, we conducted two rigorous audits across the 1,152 diagnostic questions:

  1. Conditional retention and cross-family reader evaluation (Llama-3.1-8B-Instruct): We rescored the diagnostic memory and evidence transitions using an independent reader from a different model family (Llama-3.1-8B-Instruct, never used during training) alongside the frozen Qwen2.5 scorer. We report (i) Conditional Retention Rate (P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0)), (ii) Post-Write Loss Rate (P(gM≤0∣gE>0)P(g^M \le 0 \mid g^E > 0), where useful retrieved evidence is lost during compression), and (iii) Cross-Family Positive-Gain Rate (P(gM>0)P(g^M > 0) under Llama-3.1-8B):
EvaluatorModelMethodConditional Retention P(gM>0∣gE>0)P(g^M > 0 \mid g^E > 0) (↑\uparrow)Post-Write Loss P(gM≤0∣gE>0)P(g^M \le 0 \mid g^E > 0) (↓\downarrow)Positive-Gain Rate P(gM>0)P(g^M > 0) (↑\uparrow)
Llama-3.1-8B (Independent)3BMemSearcher63.91%36.09%47.40%
Llama-3.1-8B (Independent)3BMemCredit71.18% (+7.27%)28.82% (-7.27%)55.90% (+8.50%)
Llama-3.1-8B (Independent)7BMemSearcher70.58%29.42%52.52%
Llama-3.1-8B (Independent)7BMemCredit78.55% (+7.97%)21.45% (-7.97%)56.42% (+3.90%)
Qwen2.5 (Frozen Scorer)3BMemSearcher56.13%43.87%— (Fig. 4a: 46.4%)
Qwen2.5 (Frozen Scorer)3BMemCredit67.22% (+11.09%)32.78% (-11.09%)— (Fig. 4a: 51.4%, +5.03% absolute, +10.9% relative)
Qwen2.5 (Frozen Scorer)7BMemSearcher61.16%38.84%—
Qwen2.5 (Frozen Scorer)7BMemCredit68.39% (+7.23%)31.61% (-7.23%)—

Under both evaluators, MemCredit substantially improves conditional evidence retention by +11.09% (3B) and +7.23% (7B) under Qwen2.5, and by +7.27% (3B) and +7.97% (7B) under the independent Llama-3.1-8B reader while also increasing unconditional positive gains under Llama-3.1-8B by +8.50% (3B) and +3.90% (7B).

  1. Reader-free evaluation of factual consistency and evidence preservation: To evaluate evidence preservation and factual consistency completely independently of any neural reader, we directly compared the written memories Mi,tM_{i,t} against the raw retrieved documents oi,≤to_{i,\le t} via exact normalized string matching and blind manual verification across the diagnostic updates:
Reader-Free Grounding & Consistency MetricMemSearcher (3B)MemCredit (3B)MemSearcher (7B)MemCredit (7B)
Strict Document Grounding Rate (verbatim supported by oi,≤to_{i,\le t})98.75% (396/401)98.93% (463/468)98.44% (442/449)98.94% (466/471)
Factual Consistency Rate (including valid date/format normalizations)99.75% (400/401)99.58% (466/468)99.78% (448/449)99.79% (470/471)
Unsupported / Hallucinated Content Rate (manual audit)0.25% (1/401)0.42% (2/468)0.22% (1/449)0.21% (1/471)
Updates Preserving Verbatim-Grounded Target Entities155 updates189 updates (+21.9%)163 updates165 updates (+1.2%)

This reader-free textual audit establishes two conclusions:

  • Direct evidence preservation without reader scoring: Evaluated purely by exact textual provenance against the retrieved passages oi,≤to_{i,\le t} (with zero involvement of any frozen reader), MemCredit successfully preserves verbatim-grounded target answer entities in 189 updates vs. 155 updates (+21.9% more preserved entities) at 3B and 165 vs. 163 at 7B.
  • High factual consistency with retrieved sources: In MemCredit, 98.93% (3B) and 98.94% (7B) of the audited memory updates are strictly grounded verbatim in the retrieved observations oi,≤to_{i,\le t} (exceeding MemSearcher's 98.75% and 98.44%). Blind manual inspection of the remaining cases confirms that nearly all are faithful date or surface-form normalizations (e.g., standardizing date formats), yielding an overall factual consistency rate of 99.58% (3B) and 99.79% (7B), with unsupported statements occurring in only 0.42% and 0.21% of updates. We will include both the cross-family reader evaluation and this reader-free textual grounding audit in Section 4.3 and Appendix.

W2 (Statistical reliability and multi-seed evaluation):

Three-seed training runs at 3B and 7B alongside a new 14B scaling experiment confirm that MemCredit outperforms MemSearcher in every single seed and across all model scales, solving hundreds of additional questions on large benchmarks.

Specifically, we conducted 3-seed training runs (seeds 123, 42, and 1) under identical code, data, retriever, and training steps for both 3B and 7B models, scaled training to Qwen2.5-14B-Instruct, and evaluated question-level net solved gains across all questions:

  1. Multi-seed training stability at 3B, 7B, and 14B:
Scale / SeedOutcome-only (MemSearcher)MemCreditΔ\Delta Macro EM (All 7)Δ\Delta Macro EM (Excl. Bam.)
3B Seed 123 (Table 1)43.44%45.41%+1.97%+1.76%
3B Seed 4243.19%44.87%+1.68%+1.52%
3B Seed 143.55%46.00%+2.45%+2.24%
3B Mean ±\pm Std (3 seeds)43.39 ±\pm 0.18%45.43 ±\pm 0.57%+2.03 ±\pm 0.39%+1.84 ±\pm 0.37%
7B Mean ±\pm Std (3 seeds) (detailed below)47.82 ±\pm 0.20%48.96 ±\pm 0.33%+1.14 ±\pm 0.16%+0.71 ±\pm 0.26%
14B (Seed 1, Qwen2.5-14B-Instruct)50.44%51.22%+0.78%+0.64% (50.32% →\rightarrow 50.96%)
Seed / Method (7B)NQTriviaQAPopQAHotpotQA2WikiMuSiQueBam.Avg. (All 7)Avg. (w/o Bam.)
Seed 123 / MemSearcher50.7566.7947.8049.5648.5224.4147.2047.8647.97
Seed 123 / MemCredit51.1167.2849.5049.4949.1424.4951.2048.89 (+1.03)48.50 (+0.53)
Seed 42 / MemSearcher50.9166.6847.8249.2848.4624.4545.6047.6047.93
Seed 42 / MemCredit51.1967.1949.6249.3349.1724.6249.6048.67 (+1.07)48.52 (+0.59)
Seed 1 / MemSearcher50.9466.9947.9049.7248.6724.5347.2047.9948.13
Seed 1 / MemCredit51.5867.5949.6250.3649.9325.7350.4049.32 (+1.33)49.14 (+1.01)
Mean ±\pm Std / MemSearcher50.87$\pm$0.1066.82$\pm$0.1647.84$\pm$0.0549.52$\pm$0.2248.55$\pm$0.1124.46$\pm$0.0646.67$\pm$0.9247.82$\pm$0.2048.01$\pm$0.10
Mean ±\pm Std / MemCredit51.29$\pm$0.2567.35$\pm$0.2149.58$\pm$0.0749.73$\pm$0.5549.41$\pm$0.4524.95$\pm$0.6850.40$\pm$0.8048.96$\pm$0.3348.72$\pm$0.36
3-Seed Mean Δ\Delta+0.43+0.53+1.74+0.21+0.86+0.48+3.73+1.14$\pm$0.16+0.71$\pm$0.26

Across all three seeds, MemCredit consistently outperforms the MemSearcher baseline by +2.03 ±\pm 0.39% at 3B and +1.14 ±\pm 0.16% at 7B (+0.71 ±\pm 0.26% at 7B even when excluding Bamboogle), and continues to improve performance at 14B (+0.78% overall, and +0.64% excluding Bamboogle).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.