acceptodds
Under review as a conference paper at ICLR 2027

UpliftMem: Learning Set-Level Uplift for Agent Memory Retrieval

Abstract

Large language model (LLM) agents reuse external memory to guide new tasks, but effective retrieval requires learning which memory sets improve execution. Such learning relies on costly outcome feedback: ordinary retrieval observes only executed sets, while evaluating alternatives requires additional rollouts. We introduce UpliftMem, which learns memory retrieval from set-level execution uplift relative to the same executor without memory. A theoretical analysis of how retrieval preferences restrict feedback coverage motivates targeted probing of alternative memory sets. Probe selection follows an expected value of sample information (EVSI) criterion, derived in closed form under a correlated Gaussian model, to allocate limited training rollouts according to their expected improvement in local retrieval decisions. The shared scorer is trained with a frozen executor and selects memory sets without test-time probes. Across ALFWorld, WebShop, and BigCodeBench, UpliftMem achieves the best success rates among evaluated baselines on the main evaluation sets. Controlled fixed-store and matched probe budget evaluations further demonstrate improved memory-use decisions and more effective use of execution feedback. Code is available at https://anonymous.4open.science/r/upliftmem-main-DB1B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Language Agents, Agent Memory, Memory Retrieval, Uplift Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_hW8cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UpliftMem, a framework for learning agent memory retrieval from execution feedback. A trainable scorer predicts the utility gain of a complete memory set relative to execution without memory, while the task executor remains frozen. An expected value of sample information (EVSI) criterion selects additional training executions under a limited rollout budget, and their outcomes supervise the scorer. The system also validates newly written memories. Experiments on ALFWorld, WebShop, BigCodeBench, and MemSyco-Bench include fixed-store comparisons, training ablations, and matched probe-budget evaluations.

Critical strengths

The paper addresses a retrieval problem in agent memory and provides useful evidence. Selecting memories according to their execution benefit aligns retrieval with the agent’s actual objective. The fixed-store experiments help isolate retrieval improvements, while the acquisition comparisons use the same warm-up checkpoint and match probe budgets. These controls support the practical value of the proposed approach in the evaluated settings and help distinguish retrieval and feedback-selection improvements from changes in memory content or additional executions alone.

Critical weaknesses

W1. Evidence with stronger agents is limited.

The evaluation covers only Qwen3-4B and Qwen3-8B, leaving unclear whether the benefits persist with stronger or different-family executors. This matters because the value of external guidance may change with the executor’s existing capabilities. For example, the ALFWorld unseen improvement over the no-memory baseline is 20.91 percentage points for 4B but 2.23 points for 8B (Table 1). This does not establish a general decline with model strength, but motivates checking the method’s incremental value with a stronger executor and competitive memory baselines.

W2. The portability and adaptation cost of the learned scorer are unclear.

The scorer is adapted separately for each executor, so the experiments demonstrate the applicability of the training procedure rather than the reuse of a trained selector. Since uplift depends on the executor, changing the backbone may change which memories are useful. The paper does not evaluate direct scorer transfer or the amount of target-executor feedback needed for adaptation. Successful zero-shot transfer should not be assumed, but characterizing the performance drop and recovery cost would clarify the practical value of maintaining a separate memory-selection module.

Questions for authors

Q1: Broader effectiveness.

Could the authors evaluate the method with a stronger executor or one from a different model family under matched evaluation settings? This would help assess whether the benefits persist as executor capabilities improve.

Q2: Scorer reuse and adaptation.

How well does a scorer trained for one executor transfer to another without additional training? If adaptation is needed, how much feedback from the target executor is required to approach the performance of a scorer adapted specifically to that executor?

6 comments
AuthorsOpenReview ↗

Response to Q2: Zero-Shot Scorer Reuse Across Executors and Task Types

We thank the reviewer for emphasizing scorer portability and target-side adaptation cost. We now report completed bidirectional zero-shot transfer between Qwen3-4B and Qwen3-8B on ALFWorld, WebShop, BigCodeBench, and MemSyco-Bench (Sec. 5.7, Table 4; App. F.8, Table 18), together with leave-one-task-type-out (LOTO) generalization (Table 5; App. E.3). These experiments test whether a trained memory-set scorer can be reused without collecting new target-side feedback or updating its parameters.

Controlled transfer protocols. In Scorer Transfer, the source-trained scorer replaces the target-specific scorer, while the target executor, memory store, and execution protocol remain fixed. This directly isolates reuse of the learned set-value function. Full Transfer additionally reuses the source memory store and protocol. The target executor remains frozen in both settings; neither protocol performs target-side scorer training or memory updates.

End-to-end zero-shot results. The table reports absolute results, combining the matched target-specific configuration (Table 1) with cross-executor changes (App. Table 18). ALFWorld and BigCodeBench entries are success rates (%); WebShop reports reward (0–100) / success rate (%).

TargetConfigurationALF seen / unseenWebShop reward / SRBCB overall / hard
4BTarget-specific76.86 / 70.9172.51 / 45.6039.77 / 20.45
4B8B→4B Scorer81.15 / 75.3972.57 / 45.4036.26 / 13.63
4B8B→4B Full81.86 / 78.3773.48 / 46.0037.43 / 13.63
8BTarget-specific81.43 / 79.1070.20 / 44.0042.69 / 22.73
8B4B→8B Scorer68.57 / 79.1069.22 / 42.4041.52 / 25.00
8B4B→8B Full80.00 / 79.1064.65 / 40.8042.40 / 22.73

With zero additional target-executor feedback, 8B→4B Scorer Transfer improves ALFWorld seen/unseen success by 4.29/4.48 pp over the target-specific scorer; Full Transfer increases these gains to 5.00/7.46 pp and improves WebShop success by 0.40 pp. For 4B→8B, Scorer Transfer matches 79.10% unseen ALFWorld success and improves BigCodeBench hard success by 2.27 pp. These settings already match or exceed executor-specific performance without adaptation.

Fixed-store transfer on MemSyco-Bench. We also test both directions on five identical post-extraction memory stores and five memory-use scenarios, averaged over three independent training runs. The reported metric is the mean absolute out-of-distribution (OOD) versus in-domain (ID) pass-rate gap from each target's own trained scorer (pp; main Table 4), not a gain over native retrieval:

Retrieval budget8B→4B gap (pp)4B→8B gap (pp)
K=3K=37.8 ± 0.77.3 ± 0.7
K=5K=54.1 ± 0.74.1 ± 0.4
K=10K=101.9 ± 0.42.8 ± 1.1

At the default K=5K=5, both transfer directions have a 4.1 pp average absolute gap, decreasing to 1.9/2.8 pp at K=10K=10. This controlled result complements end-to-end success rates by measuring portability of decisions about when and how to use memory.

Completed cross-task generalization. In LOTO, the scorer is trained on four MemSyco task types and evaluated on the excluded fifth, which contributes no warm-up, rollout, probing, or adaptation feedback. Across five stores and three independent training runs, the mean absolute gap from the all-task-trained reference is 13.3, 7.7, and 4.2 pp at K=3,5,10K=3,5,10, respectively (Table 5). All LOTO scorers are trained at K=5K=5; only the inference budget changes. At K=5K=5, held-out Objective Fact, Personalized Use, and Valid Selection differ by −0.4, −2.0, and −1.1 pp from the reference.

Adaptation-cost implication. The transfer evaluations establish a zero-target-feedback operating point, rather than merely evaluating a training procedure that must be rerun for every executor. In the ALFWorld and BigCodeBench settings highlighted above, no target-side adaptation is required to reach or exceed the executor-specific reference. Together with the fixed-store and LOTO results, these findings demonstrate reuse of learned set-level valuation across executors and previously unseen task types.

AuthorsOpenReview ↗

Response to Q1: Broader Effectiveness with a Different-Family 14B Executor

We thank the reviewer for suggesting a stronger, different-family executor evaluation. We have completed an additional end-to-end comparison with Ministral-14B, extending the main Qwen3-4B/8B experiments to a larger executor from another model family (App. F.9, Table 19). The evaluation covers ALFWorld seen/unseen and BigCodeBench overall/hard, with the same task evaluation protocols and a broad set of memory baselines, including ReasoningBank, MemP, MemRL, Mem0, A-MEM, and LightMem.

Results with Ministral-14B. UpliftMem achieves the highest success rate in all four comparisons. We report ReasoningBank as a directly relevant experiential-memory baseline and the strongest competing method for each metric:

Success rate (%)ReasoningBankStrongest other baselineUpliftMemGain over strongest
ALFWorld seen60.00Mem0 (80.00)82.14+2.14 pp
ALFWorld unseen64.18Mem0 (72.39)74.63+2.24 pp
BigCodeBench overall31.29Dense-RAG (38.89)40.94+2.05 pp
BigCodeBench hard15.91MemRL (27.27)29.54+2.27 pp

Compared directly with ReasoningBank, the improvements are +22.14/+10.45 pp on ALFWorld seen/unseen and +9.65/+13.63 pp on BigCodeBench overall/hard. UpliftMem also obtains the fewest ALFWorld unseen execution steps (16.21) among the evaluated methods. The complete results, including execution steps and all baselines, appear in Table 19.

What this establishes. The main Table 1 evaluates two Qwen executors, whereas the new Ministral-14B comparison tests a different model family with more parameters. UpliftMem's leading success rates in both interactive planning and code-generation tasks show that its end-to-end benefits extend beyond the original executor family. This is a separately adapted end-to-end evaluation; direct reuse of a trained scorer without target-side updates is tested independently in our response to Q2.

AuthorsOpenReview ↗

Updated Manuscript with Stronger-Executor and Zero-Shot Transfer Results

Following your suggestions, we have completed the additional Ministral-14B evaluation and incorporated bidirectional zero-shot scorer-transfer experiments across multiple benchmarks, together with fixed-store and held-out-task-type generalization results. These additions provide direct evidence of UpliftMem's effectiveness beyond the original executor family and the reuse of trained scorers without target-side adaptation. We sincerely appreciate your valuable feedback and would welcome your updated assessment of these new results.

Reviewer_rsN3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UpliftMem learns agent memory retrieval by predicting the utility gain of complete memory sets relative to execution without memory. It uses EVSI-based probing to allocate additional training rollouts. Experiments across three agent benchmarks and fixed-memory-store evaluations show promising performance, although the acquisition implementation and the independent contribution of set-level modeling require further validation.

Critical strengths
  1. The fixed-store evaluation provides compelling evidence of retrieval improvements. With memory content and the executor held fixed, UpliftMem improves average decision accuracy on MemSyco-Bench from 41.7%/44.4% to 66.5%/68.8%. This control substantially reduces confounding from memory construction quality and constitutes the strongest evidence supporting the method’s effectiveness.
  2. Coupling feedback acquisition with retrieval learning addresses a meaningful learning bottleneck. Retrieval preferences determine which memory sets receive execution feedback, potentially leaving underestimated alternatives without corrective evidence. The method explicitly allocates resources to exploring alternatives and outperforms random probing under matched probe counts, supporting the value of active feedback acquisition for memory retrieval.
Critical weaknesses
  1. The baseline selection is insufficiently aligned with the paper’s experiential-memory setting, weakening the performance comparison. Following the distinction between factual and experiential memory in Memory in the Age of AI Agents[1], UpliftMem’s reuse of strategies distilled from execution trajectories places its main agent experiments within experiential memory. However, several baselines in Table 1, including A-MEM and Mem0, primarily address retaining and retrieving conversational facts, with original evaluations centered on conversational QA benchmarks such as LoCoMo. Comparing them on ALFWorld and WebShop requires explaining how their memory construction, stored content, and retrieval interfaces were adapted for experience reuse. Appendix D.2 does not provide sufficient detail to assess this adaptation. Although MemP and MemRL are included, directly relevant experiential-memory approaches such as ReasoningBank[2]—already cited in the paper—are absent. Consequently, the results do not adequately distinguish methodological advantages from differences in task adaptation or establish superiority over experiential-memory alternatives.
  2. The experiments do not directly establish that modeling interactions between memories drives the improvements. The motivation for set-level modeling rests on complementarity, redundancy, and conflict between memories. However, Table 2 varies probing strategies and training losses without comparing the full set scorer against an additive singleton scorer. A direct control would replace �0��0� with �1��1�, while matching the memory store, execution feedback, candidate pool, search procedure, and empty-set option. Comparable performance from this control would undermine the necessity of interaction modeling. The fixed-store MemSyco-Bench evaluation demonstrates benefits from replacing the retriever, but does not exclude better individual-memory filtering or learning when to abstain as the main explanation. This omission directly limits the evidence for the paper’s central set-level contribution.

[1] Memory in the age of AI Agents: https://arxiv.org/html/2512.13564v2

[2] ReasoningBank: https://arxiv.org/abs/2509.25140

Questions for authors
  1. Baseline adaptation clarification: What do A-MEM and Mem0 store in the ALFWorld and WebShop experiments, and how are their conversational memory writing, updating, and retrieval mechanisms adapted for experience reuse? Are source trajectories, feedback access, and execution budgets matched across methods?
  2. Comparison material to the assessment: Why are directly relevant experiential-memory methods such as the cited ReasoningBank excluded? Can the authors provide comparisons under matched executors and execution budgets?
  3. Validation of the central contribution: With a fixed memory store, matched execution feedback, and identical search and empty-set options, how much does the full set scorer improve over an additive singleton scorer? Are the gains concentrated on identifiable complementary or conflicting memory combinations?
5 comments
AuthorsOpenReview ↗

Response to Q1–Q2: Experiential Memory Baselines and Feedback Efficiency

We thank the reviewer for recognizing the fixed-store and EVSI results. We clarified A-MEM/Mem0 adaptations, added ReasoningBank, extended evaluation to Ministral-14B, and expanded matched-budget and cost-aware probing experiments.

Q1. Experiential-memory adaptation and fair evaluation. A-MEM and Mem0 process execution trajectories through their native memory mechanisms. ALFWorld inputs contain the task objective and role-labeled interactions; WebShop inputs contain the shopping instruction and chronological action–observation history, with long observations truncated while preserving actions. Both write memories from successful training episodes, excluding test feedback. A-MEM constructs structured notes with experience, keywords, tags, contextual descriptions, and embeddings, retaining native linking, evolution, and search_agentic retrieval. Mem0 extracts reusable procedures via procedural_memory and uses vector retrieval. Retrieved memories are injected into the same executor at episode initialization.

End-to-end comparisons align task partitions, schedules, frozen executors, action limits, retrieval caps, and evaluation protocols; each method retains its native writing/update rules and learns from its own on-policy trajectories. Fixed-store MemSyco comparisons hold memory content and executors constant, whereas probing comparisons match the warm-up checkpoint, store, and additional-probe budget. See Apps. E.2 and E.4 for protocols and execution accounting, including reference reuse and writing-validation rollouts.

Q2. Experiential-memory baselines. We added ReasoningBank. UpliftMem improves all ten success-rate comparisons against ReasoningBank with Qwen3-4B/8B (Table 1):

SR (%)4B: ReasoningBank → Ours8B: ReasoningBank → Ours
ALFWorld seen65.00 → 76.8677.14 → 81.43
ALFWorld unseen66.42 → 70.9174.62 → 79.10
WebShop32.40 → 45.6031.00 → 44.00
BCB overall30.41 → 39.7737.72 → 42.69
BCB hard13.64 → 20.4511.36 → 22.73

WebShop gains reach +13.20/+13.00 pp. Independent-training standard deviations appear in App. F.1, Table 11.

Cross-family evaluation. With Ministral-14B (App. F.9, Table 19), UpliftMem leads all four success-rate metrics against ReasoningBank and the strongest other baseline:

SR (%)ReasoningBank → OursBest other baselineGain over best
ALFWorld seen60.00 → 82.14Mem0 80.00+2.14 pp
ALFWorld unseen64.18 → 74.63Mem0 72.39+2.24 pp
BCB overall31.29 → 40.94Dense-RAG 38.89+2.05 pp
BCB hard15.91 → 29.54MemRL 27.27+2.27 pp

This extends the experiential-memory comparison beyond the Qwen model family.

Matched-budget feedback selection. We compare EVSI, Random, MC-STD, and EIG from the same warm-up checkpoint and fixed store at 20–200 additional probes (Sec. 5.5; App. F.7, Figs. 9–10). Results below compare EVSI with Random:

MetricProbesRandomEVSIGain
WebShop cumulative SR (%)5042.048.0+6.0 pp
ALFWorld seen cumulative SR (%)5066.472.1+5.7 pp
ALFWorld unseen cumulative SR (%)5062.765.7+3.0 pp
BCB cumulative SR (%)5039.542.4+2.9 pp
WebShop mean reward2000.6700.694+0.024
BCB cumulative SR (%)20047.451.8+4.4 pp

On BCB, MC-STD yields 41.2%/50.3% at 50/200 probes versus EVSI's 42.4%/51.8%, supporting higher learning value per additional probe.

Cost-aware acquisition. EVSI compares each probe's marginal decision value with its acquisition cost and stops once the threshold or budget is reached (Sec. 4.3). Using two probing waves recorded during training (App. F.6, Fig. 8), at λprobe=0.1\lambda_{\mathrm{probe}}=0.1:

BenchmarkSelectedBudget usedUnused quota
ALFWorld43444.7%55.3%
WebShop1,34111.8%88.2%
BCB1,36374.2%25.8%

The second wave allocates fewer probes as remaining feedback value changes. Here unused quota concerns additional probes, not total training executions. Matched-budget tests identify which feedback to acquire; cost-aware acquisition determines how much to acquire

AuthorsOpenReview ↗

Response to Q3: Execution-Based Evidence for Set-Level Memory Interaction Benefits

We thank the reviewer for proposing an additive-singleton control. We added 480 task–memory-pair cases evaluated under four execution conditions, Independent Scoring ablations across three benchmarks, and fixed-store Evidence Conflict analysis. These experiments test whether modeling non-additive memory effects improves retrieval decisions and execution.

Matched four-condition interaction analysis. We sample 80 task–memory pairs per benchmark–executor configuration across ALFWorld, WebShop, and BigCodeBench with Qwen3-4B/8B (App. F.4, Table 16). For each case, the same frozen executor runs no memory, each singleton, and the pair. The realized interaction is Iab=Uab−Ua−Ub+U0I_{ab}=U_{ab}-U_a-U_b+U_0; positive and negative values indicate super- and sub-additivity.

Both approaches use the same trained scorer, singleton predictions, memory candidates, executor, realized outcomes, and abstention option. The additive control sums singleton uplifts to estimate pair value; UpliftMem predicts the joint uplift. Both select from {∅,{a},{b},{a,b}}\{\varnothing,\{a\},\{b\},\{a,b\}\} under the same rule. Only pair valuation changes.

Execution-based interaction results (Additive → Set; lower is better). Interaction MAE measures estimation error in non-additivity; local regret measures utility lost relative to the best of the four executed alternatives. All six configurations are shown (App. Table 16):

Benchmark / executorInteraction MAE ↓Local regret ↓
ALFWorld / 4B0.9154 → 0.36380.1801 → 0.1693
ALFWorld / 8B5.7737 → 0.43070.2588 → 0.2170
WebShop / 4B3.7798 → 0.29890.1499 → 0.1491
WebShop / 8B0.2208 → 0.18020.0885 → 0.0716
BigCodeBench / 4B0.3100 → 0.16650.1203 → 0.1079
BigCodeBench / 8B0.2423 → 0.24110.1525 → 0.1907

Set-level valuation reduces interaction MAE in 6/6 configurations and local regret in 5/6, linking more accurate pair valuation to improved local retrieval choices.

End-to-end independent-scoring ablation. We retain all other retrieval and learning components, replacing conditional set valuation with Independent Scoring (Sec. 5.4, Table 2). UpliftMem improves all ten success-rate comparisons:

Success rate (%)4B: Independent → Set8B: Independent → Set
ALFWorld seen63.57 → 76.8668.57 → 81.43
ALFWorld unseen68.66 → 70.9170.15 → 79.10
WebShop40.80 → 45.6043.40 → 44.00
BigCodeBench overall35.38 → 39.7737.43 → 42.69
BigCodeBench hard18.18 → 20.4518.18 → 22.73

Fixed-store retrieval and Evidence Conflict. With identical post-extraction memories and executors across RAG, A-MEM, Mem0, MemGPT, and LightMem, set-level scoring yields positive mean direction-aligned gains in all ten store–executor configurations (Table 3; App. F.3, Tables 14–15). Store-averaged comparisons are:

Memory-use metric (%)Qwen3-4B: Independent → SetQwen3-8B: Independent → Set
Decision accuracy ↑57.5 → 66.560.0 → 68.8
Sycophancy ↓40.3 → 29.529.3 → 16.5
Valid selection ↑57.3 → 61.661.8 → 70.7
Outdated-memory use ↓42.6 → 39.336.6 → 31.7
Evidence Conflict accuracy ↑59.12 → 71.1062.52 → 71.02
Evidence Conflict sycophancy ↓39.54 → 25.4231.62 → 11.76

On Evidence Conflict, joint valuation increases accuracy by 11.98/8.50 pp and reduces sycophancy by 14.12/19.86 pp (4B/8B). Together, the four-condition diagnostic, end-to-end ablation, and fixed-store evaluation show how memory interaction modeling improves retrieval decisions and downstream outcomes.

AuthorsOpenReview ↗

Updated Manuscript and Strengthened Set-Level Evidence

We have revised the manuscript to incorporate the additional evidence motivated by your suggestions, including ReasoningBank and Ministral-14B comparisons, controlled independent-scoring ablations, and execution-based memory-interaction analysis. We have also clarified experiential-memory baseline adaptations and expanded the feedback-efficiency evaluations. Thank you again for your constructive feedback. We would greatly appreciate your updated assessment of the revised manuscript and experimental evidence.

Reviewer_ZiUkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UpliftMem, an agent memory retrieval framework that learns to select memory sets based on their execution utility improvement relative to a no-memory baseline. It introduces a set-level uplift scorer to capture interactions among retrieved memories and an Expected Value of Sample Information (EVSI)-based probing strategy to efficiently collect informative execution feedback. The retriever is optimized through value and policy learning, alongside an iterative memory writing and validation mechanism. Experiments on ALFWorld, WebShop, BigCodeBench, and MemSyco-Bench demonstrate improvements over existing memory retrieval and management baselines.

Critical strengths
  1. The research problem is well-motivated. Many existing agent memory studies primarily focus on memory management while paying relatively little attention to memory retrieval. In addition, the paper highlights an important challenge: the large number of possible memory set combinations requires extensive rollouts to obtain execution labels, making it difficult to collect sufficient feedback under a limited budget.
  2. The EVSI-based probing strategy is reasonable. Compared to random probing, it prioritizes more valuable candidates for evaluation, helping achieve better performance under a limited budget.
Critical weaknesses
  1. The loss design seems somewhat over-engineered. For example, the necessity of each of the three sub-losses in the value loss has not been individually validated through ablation studies.
  2. The experiments appear to include only training and test sets, without a validation set. How were the hyperparameters selected?
  3. The proposed method essentially fine-tunes an existing reranker. I believe that OOD generalization is particularly important for works that train retrievers or rerankers. It seems impractical to require running the entire training pipeline on every new dataset before the retriever can be used.
  4. The quality of memory items in the memory pool is also influenced by the memory-writing validation process. Therefore, it is difficult to determine whether the performance improvements actually come from the trained reranker or simply from improvements in the quality of the memory pool.
  5. The reranker has only 0.6B parameters. Why do the authors still use LoRA to fine-tune such a small model? Is there any specific reason for this choice?
  6. I am also concerned about the training stability of the proposed method, as the memory pool continuously expands with new memory/experience items during retriever training. Under such dynamic conditions, how do the authors ensure training stability? Have they observed any instability during training?
  7. The font sizes in some figures are too small to read clearly, such as Figure 3.
Questions for authors

The same as weaknesses

4 comments
AuthorsOpenReview ↗

Response to Q1–Q3: Value-Loss Ablations, Validation Protocol, and OOD Generalization

We sincerely thank the reviewer for recognizing the motivation for set-level memory retrieval and EVSI-guided probing. In response, we added individual value-loss and independent-scoring ablations, a no-iterative-writing control, execution-based memory-interaction analysis, bidirectional zero-shot cross-executor transfer, completed leave-one-task-type-out (LOTO) generalization experiments, and memory-growth stability evaluations. We also clarified development-set selection, improved Figure 3, and extended end-to-end evaluation to Ministral-14B.

Q1. Why three value-loss terms? The set-value scorer supports three retrieval decisions: how much a set improves execution (regression), whether memory is beneficial relative to stopping (sign classification), and which candidate set offers greater uplift (pairwise ranking). We independently remove each loss while retaining the remaining training setup.

Removed value lossQwen3-4BQwen3-8BMean
Regression8.8012.7210.76
Sign classification (BCE)8.5013.5011.00
Pairwise ranking7.8212.7610.29

Entries are Full-minus-ablation gains (pp), averaged over five memory stores and ten direction-aligned MemSyco metrics. All three terms contribute under both executors. Main Tables 2–3 also ablate the overall value and policy objectives; Appendix Tables 14–15 give per-store results and standard deviations.

Q2. Validation and model selection. We use a fixed 50-task development set held out from the training partition to select hyperparameters and checkpoints. Development and test tasks are excluded from scorer training, memory writing, and probe acquisition. Final test evaluation uses the selected, fixed configuration (Sec. 5.1; App. E.1).

Q3. Out-of-distribution (OOD) generalization and scorer reuse. We now report two completed zero-shot generalization evaluations, covering changes of executor and held-out task types, alongside unseen-task agent results. These experiments directly examine whether learned set-level valuation can be reused without target-side scorer adaptation.

Zero-shot cross-executor transfer. We transfer scorers in both directions between Qwen3-4B and Qwen3-8B. Scorer transfer changes only the scorer, retaining the target executor's memory store and protocol; full transfer also reuses the source memory configuration. Neither requires target-executor scorer updates or memory updates. On ALFWorld, 8B→4B scorer transfer improves seen/unseen success by 4.29/4.48 pp, increasing to 5.00/7.46 pp under full transfer. In the reverse direction, unseen success is maintained and BigCodeBench hard-subset success improves by 2.27 pp with scorer transfer (App. F.8, Table 18).

Controlled zero-shot retrieval and held-out task types. MemSyco-Bench provides matched evaluations over five fixed memory stores. Executor-transfer gaps compare an out-of-distribution (OOD) scorer with the target-specific in-domain (ID) scorer. In LOTO, each scorer is trained on four of the five MemSyco task types and evaluated on the excluded fifth, with no held-out-type training feedback or adaptation; the reference is a scorer trained on all five types. Both experiments use three independent training runs. The table reports average absolute pass-rate gaps in percentage points, with the corresponding reference defined for each setting (main Tables 4–5; App. E.3).

Retrieval budget8B→4B (OOD–ID)4B→8B (OOD–ID)LOTO, 4B (LOTO–Full)
K=3K=37.8 ± 0.77.3 ± 0.713.3
K=5K=54.1 ± 0.74.1 ± 0.47.7
K=10K=101.9 ± 0.42.8 ± 1.14.2

At the default K=5K=5, the bidirectional executor-transfer gaps are 4.1 pp. Under LOTO, the held-out-type differences on Objective Fact, Personalized Use, and Valid Selection are −0.4, −2.0, and −1.1 pp, respectively. At K=10K=10, the average absolute LOTO gap is 4.2 pp; all LOTO scorers are trained at K=5K=5, with only the inference retrieval budget varied.

Unseen tasks and a different executor family. On ALFWorld's official unseen split, UpliftMem reaches 70.91%/79.10% success with Qwen3-4B/8B, exceeding Independent Scoring by 2.25/8.95 pp (Tables 1–2). We additionally evaluate the full UpliftMem method with Ministral-14B, beyond the Qwen model family. It achieves the highest reported success rates on ALFWorld seen/unseen (82.14%/74.63%) and BigCodeBench overall/hard (40.94%/29.54%), outperforming the strongest reported baseline in each comparison (App. F.9, Table 19). This cross-family end-to-end experiment complements the zero-shot scorer-transfer evaluations.

Together, the bidirectional executor-transfer, held-out-type LOTO, unseen-task, and cross-family end-to-end results provide complementary evidence for the portability and applicability of UpliftMem's learned memory-set valuation.

AuthorsOpenReview ↗

Response to Q4–Q7: Retrieval Attribution, Memory Writing, Interaction Analysis, Training Stability, and Presentation

We thank the reviewer for encouraging a closer analysis of retrieval attribution and training reliability. We expanded the controlled component ablations, added an execution-based interaction diagnostic, and evaluated adaptation as text-memory pools grow.

Q4. Retrieval gains versus memory quality. Writing validates a candidate individually on its source task (App. B.3); retrieval instead estimates the joint value of a memory set for the current task. We isolate these contributions with two complementary controls.

Retrieval control: fixed memory content. On MemSyco, we hold five post-extraction memory stores, the executor, and the evaluation protocol fixed, varying only the retrieval method (main Table 3; Fig. 3; App. Table 12).

RetrieverDecision 4B ↑Decision 8B ↑Syco. 4B ↓Syco. 8B ↓
Native41.744.449.040.0
Base reranker43.144.547.236.7
Independent Scoring57.560.040.329.3
UpliftMem66.568.829.516.5

With identical memory content, UpliftMem increases decision accuracy by 24.8/24.4 pp over Native, 23.4/24.3 pp over the pretrained Base Reranker, and 9.0/8.8 pp over Independent Scoring (4B/8B). These contrasts identify the retrieval contribution, execution-uplift adaptation, and joint-set valuation, respectively.

Writing control: end-to-end memory updates. The added w/o Iterative Writing ablation retains the remaining system while disabling iterative writing (Table 2). The full system gains:

Success-rate gain over w/o writing (pp)4B8B
ALFWorld seen+7.58+4.29
ALFWorld unseen+6.73+5.22
WebShop+3.60+0.40
BigCodeBench overall+1.76+1.46

Together, these controlled comparisons establish the distinct contributions of retrieval decisions and iterative memory writing.

Why joint valuation helps. In our new execution-based Memory Interaction Analysis (App. F.4, Table 16), we evaluate 80 task–memory pairs per benchmark–executor configuration across ALFWorld, WebShop, and BigCodeBench. For each task we execute no memory, either singleton, and the pair. The additive and set-level controls share the same frozen scorer, singleton predictions, memory candidates, executor, and realized outcomes; only pair valuation differs. Joint scoring reduces interaction MAE in all six configurations and local choice regret in five. On ALFWorld/8B, interaction MAE drops 5.7737→0.4307 and regret 0.2588→0.2170; on WebShop/8B, they drop 0.2208→0.1802 and 0.0885→0.0716. This links non-additive memory effects with actual selection quality.

Q5. Why LoRA for a 0.6B scorer? The scorer is adapted repeatedly as ordinary executions, EVSI probes, and writing validation generate new uplift observations. LoRA supports efficient iterative feedback learning by freezing the pretrained backbone while training low-rank updates and affine utility calibration, reducing trainable parameters and optimizer-state overhead. We use rank 8, scaling 16, dropout 0.1 under a consistent protocol across benchmarks, executors, and seeds (Algorithm 1; Apps. C.1, E.5).

Q6. Stability under memory growth. UpliftMem stores experience as external text, scored by a shared set-value model: admitting another memory does not introduce item-specific trainable parameters. Similarity recall provides 50 candidates per task, and fixed KK bounds downstream set construction (App. B.1). Historical execution observations continue to supervise value learning; policy paths are refreshed after scorer updates, as are EVSI belief features and covariance (Algorithm 1; App. A.4).

We added checkpoint-level memory-growth evaluations (App. Fig. 7):

BenchmarkMemory poolTask performance during growth
ALFWorld425→1,244 (2.93×)Unseen SR 57.46%→67.16%
BigCodeBench882→1,679 (1.90×)Overall SR 36.26%–38.30%
WebShop229→768 (3.35×)SR 43.2%–45.4%

Across three independent training seeds, SR standard deviations are 2.1/2.5 pp on WebShop and 2.6/2.4 pp on BigCodeBench (4B/8B; App. Table 11). The architecture and observed trajectories together demonstrate consistent adaptation as the external memory pool expands without increasing scorer parameter count.

Q7. Figure readability. Thank you for pointing this out. We revised Figure 3 with larger fonts and clearer labels for readability at manuscript scale.

AuthorsOpenReview ↗

Updated Manuscript and Additional Experimental Evidence

We have completed a comprehensive revision addressing your comments, including individual value-loss ablations, controlled retrieval and memory-writing comparisons, zero-shot generalization across executors and held-out task types, and training stability under expanding memory pools. We have also clarified the validation protocol and LoRA design, and improved figure readability. We sincerely appreciate your detailed feedback and would be grateful if you could revisit the revised manuscript and our responses.

AuthorsOpenReview ↗

Clarification on Unified Learning, Zero-Shot Reuse, and Retrieval Attribution

We sincerely appreciate the reviewer's thoughtful assessment. We would like to further clarify several design distinctions underlying the concerns about model complexity, generalization, and performance attribution.

Unified scorer and supervision. UpliftMem learns a single scalar set-uplift function Gθ(c,S)G_\theta(c,S). Regression, sign classification, and pairwise ranking provide complementary supervision for this same prediction, with sign and preference labels derived from collected execution-uplift observations. The retrieval policy is likewise defined through conditional gains of GθG_\theta, rather than a separate policy network. Thus, the objectives share both the scorer and execution feedback. Importantly, EVSI probing and feedback-driven updates operate during training; inference uses the learned scorer and deterministic beam search without additional probing (Secs. 4.2–4.4; App. C).

Distinct dimensions of zero-shot generalization. Our experiments examine the reuse of a trained scorer under two distribution shifts. Cross-executor Scorer Transfer applies the unchanged source-trained scorer to another executor while keeping the target memory store and execution protocol fixed. LOTO instead trains on four MemSyco task types and directly evaluates the scorer on the excluded fifth type, without feedback or adaptation from that type. These protocols separately test transfer across executors and previously unseen task types. Full Transfer additionally examines source memory-configuration reuse, while Ministral-14B evaluates cross-family end-to-end applicability (Sec. 5.7; Apps. E.3, F.8–F.9).

Retrieval versus memory writing. Writing validation assesses an individual memory on its source task for admission, whereas retrieval estimates the joint value of available memories for the current task, including the option to use no memory. A source-validated memory may interact differently with other memories in a new context. Accordingly, fixed-store comparisons isolate retrieval decisions, while the no-iterative-writing ablation separately evaluates memory-pool updates (Secs. 5.3–5.4; App. B.3).

We hope these clarifications make the shared learning architecture and the scope of our generalization and attribution experiments clearer.

Reviewer_yNXZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents UpliftMem, a method for learning which memory sets improve LLM agent performance. It measures the benefit of a memory set against execution without memory and uses an EVSI-based strategy to select useful training probes under a limited budget. Experiments on ALFWorld, WebShop, BigCodeBench, and MemSyco-Bench show improvements in task performance, memory selection, and feedback efficiency.

Critical strengths

The paper identifies an important problem: retrieval choices affect not only agent performance but also which feedback is available for learning. This provides a useful perspective on agent memory retrieval.

The combination of set-level uplift learning and decision-aware probing is well motivated. It addresses memory interactions and limited feedback in a unified framework.

The experiments are extensive. In particular, the fixed-memory-store evaluation and matched probe-budget comparisons provide strong evidence that the improvements come from better retrieval decisions and feedback use.

Critical weaknesses

The paper does not directly compare set-level scoring with independent memory scoring. This makes it difficult to isolate the benefit of modeling memory interactions.

The practical construction of the Gaussian uncertainty model is not fully clear, especially how its features and covariance are estimated. This limits reproducibility and makes the EVSI mechanism harder to assess.

Each configuration is trained only once. Repeated evaluations of the same checkpoint do not measure training variability, which weakens confidence in some performance differences.

Questions for authors
  1. Can you compare set-level uplift scoring with a simpler approach that scores each memory independently, while keeping the other components unchanged?

  2. How are the Gaussian features, covariance, and observation noise estimated in practice? How sensitive is EVSI to these choices?

  3. Can you provide results across multiple training seeds and report the total training rollout cost of each probe strategy?

4 comments
AuthorsOpenReview ↗

Response to Q1: Independent Scoring and Execution-Based Memory Interactions

We sincerely thank the reviewer for recognizing the importance of retrieval-dependent feedback coverage and for recommending a direct control for set-level scoring. In response, we added Independent Scoring to the main ablations (Sec. 5.4, Table 2), extended the comparison to five fixed memory stores (Table 3; App. F.3, Tables 14–15), and conducted a new execution-based Memory Interaction Analysis (App. F.4, Table 16). These experiments establish the contribution of joint-set valuation at the level of task success, memory-use decisions, and measured interactions.

Controlled end-to-end ablation. Independent Scoring evaluates each memory without conditioning on the current selected set, whereas UpliftMem scores the conditional contribution of a candidate to that set. We retain the remaining retrieval and learning components. Across three benchmarks and both executors, UpliftMem improves all ten success-rate comparisons, including ALFWorld seen/unseen and BigCodeBench overall/hard tasks (Table 2).

Success rate (%)4B Independent → Set8B Independent → Set
ALFWorld seen63.57 → 76.8668.57 → 81.43
ALFWorld unseen68.66 → 70.9170.15 → 79.10
WebShop40.80 → 45.6043.40 → 44.00
BigCodeBench overall35.38 → 39.7737.43 → 42.69
BigCodeBench hard18.18 → 20.4518.18 → 22.73

Fixed-store retrieval evidence. With identical post-extraction memories and executors, we compare the two scorers across RAG, A-MEM, Mem0, MemGPT, and LightMem. The mean direction-aligned advantage of set-level scoring is positive in all ten store–executor configurations, ranging from 3.0 to 12.5 pp across ten memory-use metrics. Decision accuracy improves from 57.5% to 66.5% (4B) and 60.0% to 68.8% (8B), while sycophancy decreases from 40.3% to 29.5% and 29.3% to 16.5%, respectively. Valid selection also improves, alongside reduced outdated-memory use (Table 3; App. Tables 14–15).

Execution-based interaction analysis. We additionally sample 80 task–memory pairs per benchmark–executor configuration, across six configurations. For each task we execute four alternatives: no memory, each memory alone, and the pair. Their realized utilities quantify non-additivity as Iab=Uab−Ua−Ub+U0I_{ab}=U_{ab}-U_a-U_b+U_0. The additive and set-level constructions reuse the same frozen scorer, singleton predictions, memory pool, and execution outcomes, differing only in how they value the pair. Interaction MAE measures the estimated non-additive effect; local regret measures the realized utility lost by choosing among these four alternatives.

Set-level valuation improves interaction MAE in all six configurations and local regret in five. On WebShop/8B, MAE falls 0.2208 → 0.1802 and regret 0.0885 → 0.0716; on BigCodeBench/4B, they fall 0.3100 → 0.1665 and 0.1203 → 0.1079 (App. Table 16). Thus, controlled downstream comparisons and execution-based diagnostics jointly establish why modeling complete memory sets improves retrieval decisions.

AuthorsOpenReview ↗

Response to Q2: Practical Gaussian Belief Construction and EVSI Sensitivity

We sincerely thank the reviewer for requesting implementation details and sensitivity evidence. We expanded the Gaussian belief construction in Apps. A.4 (pp. 16–17) and E.5 (pp. 31–32), and added a three-benchmark, six-variant sensitivity analysis (App. F.5, Table 17, p. 38). The belief models uncertainty around the neural scorer’s expected set uplift and lets EVSI value an additional rollout by its expected improvement in a retrieval decision.

Gaussian features. For each task and complete memory set, we extract the final-layer hidden state at the last non-padding token of Qwen3-Reranker-0.6B, using the same serialized input as set-value prediction. We L2-normalize the 1,024-dimensional state to form ϕ(c,S)\phi(c,S). Shared features correlate uncertain set values, so one execution can inform comparisons involving other sets. The Gaussian term is a linear uncertainty correction around the neural scorer, rather than a posterior over LoRA parameters.

Observation-noise estimation. We calibrate noise separately for each benchmark and frozen executor on 32 training task–set pairs. Each pair is executed three times with memory and three times without memory, using independent rollout randomness. We pool within-task sample variances for the two conditions, apply the documented variance floor, and sum them to estimate the scalar uplift-label noise variance σ^n2\widehat{\sigma}_n^2. This empirical estimate enters both historical covariance fitting and prospective EVSI observations. App. A.4 also models correlations from reused no-memory references; the implemented prospective calculation uses the estimated marginal-noise approximation.

Covariance construction. From up to 512 distinct recent executed task–set observations, we fit a ridge-regularized diagonal precision approximation (λ=1\lambda=1). Each direction’s precision accumulates squared normalized feature values weighted by the estimated label-noise variance. After scorer updates, historical inputs are re-encoded and the latent covariance is rebuilt. Although the latent covariance is diagonal, predicted values of different sets remain correlated through their shared representations, enabling one probe to reduce uncertainty in multiple retrieval comparisons.

Sensitivity under controlled acquisition. We hold the scorer checkpoint, execution history, candidate pool, and local comparisons fixed within each benchmark. Six variants test 256/512-dimensional feature projections with covariance refitting, 50% shrinkage toward trace-matched isotropic covariance, fully isotropic covariance, and 0.5×/2× perturbations of the empirically estimated noise variance. For noise perturbations, both historical covariance and prospective observation noise are recomputed. We measure initial EVSI ranking via Spearman ρ\rho and sequential Softmax Overlap@50, averaged across five paired selection seeds. The table shows ρ\rho / mean Overlap@50 (%); Table 17 also reports standard deviations and initial/greedy diagnostics.

Gaussian variantALFWorldBigCodeBenchWebShop
Noise ×0.50.989 / 90.40.998 / 73.20.983 / 98.8
Noise ×20.994 / 88.80.998 / 73.20.984 / 98.8
Feature-256−0.393 / 77.60.187 / 54.00.674 / 96.8
Feature-512−0.437 / 76.0−0.004 / 52.40.583 / 96.8
Shrinkage 50%−0.155 / 78.40.423 / 61.20.483 / 96.0
Isotropic−0.508 / 76.4−0.486 / 50.40.384 / 94.0

What the sensitivities reveal. Scaling estimated noise by 0.5× or 2× preserves initial EVSI priorities, with ρ=0.983\rho=0.983–0.9980.998 across benchmarks. Feature representation and covariance geometry exert a distinct effect on which feedback is valued. On BigCodeBench, shrinkage preserves 76% of the default initial Top-50, versus 2% under isotropic covariance. On ALFWorld, both have 0% initial Top-50 overlap, yet sequential greedy Overlap@50 reaches 74% and 64%, respectively. These paired diagnostics demonstrate why the uncertainty geometry affects initial probe priorities and why sequential marginal-value updates must be considered when assessing the resulting acquisition policy.

AuthorsOpenReview ↗

Response to Q3: Independent Training Runs and Cost-Aware Probe Allocation

We sincerely thank the reviewer for emphasizing independent training variability and execution cost. We clarified the repeated-run protocol, added end-to-end standard deviations across independently seeded training runs, expanded probe-efficiency evidence across three benchmarks, and introduced a cost-aware acquisition analysis.

Independent training variability. ALFWorld, WebShop, and BigCodeBench main results are averaged across three independently seeded training runs. Appendix F.1, Table 11, reports standard deviations for the methods, both executors, and their principal metrics, rather than repeated inference on one checkpoint. For UpliftMem, the SR standard deviations are 2.70/2.40 pp on ALFWorld seen, 3.00/3.30 pp on ALFWorld unseen, 2.10/2.50 pp on WebShop, and 2.60/2.40 pp on BigCodeBench overall (4B/8B). MemSyco main fixed-store evaluations use five runs and additional component ablations three runs, with per-store statistics in App. F.3 (Tables 12, 14–15). Probe-efficiency experiments independently repeat acquisition and retriever adaptation from a common warm-up checkpoint (App. E.6). We additionally report memory-growth checkpoint trajectories (App. F.2, Fig. 7): memory pools grow 2.93×, 1.90×, and 3.35× on ALFWorld, BigCodeBench, and WebShop while task performance is maintained.

Feedback efficiency at matched probe budgets. Sec. 5.5 and App. F.7 (Figs. 9–10) compare Random, MC-STD, EIG, and EVSI from the same warm-up checkpoint and fixed memory store, with the same additional-probe budgets of 20–200. At 50 probes, EVSI achieves 48.0% cumulative success on WebShop versus 42.0% for Random. ALFWorld seen/unseen coverage reaches 72.1%/65.7%, versus 66.4%/62.7% for Random. On BigCodeBench, EVSI reaches 42.4%, versus 39.5% for Random and 41.2% for MC-STD; at 200 probes, the corresponding figures are 51.8%, 47.4%, and 50.3%. These matched-budget experiments demonstrate higher downstream learning value per allocated additional probe across three benchmarks.

Cost-aware allocation and stopping. Fixed-quota strategies expend their prescribed additional-probe budgets, whereas EVSI compares each next probe’s marginal expected decision value against its acquisition cost, stopping when no candidate exceeds the cost or the budget is reached (Sec. 4.3). We added an analysis of acquisition states recorded during training, including two successive probing waves, under varying cost coefficients (App. F.6, Fig. 8). At the default λprobe=0.1\lambda_{\mathrm{probe}}=0.1, the cost-conditioned decisions are:

BenchmarkSelected probesBudget usedFewer than full quota
ALFWorld43444.7%55.3%
WebShop1,34111.8%88.2%
BigCodeBench1,36374.2%25.8%

The second acquisition wave reduces allocation more sharply as the value of remaining feedback changes. Thus, the matched-budget evaluations establish which probes yield more learning value, while cost-aware acquisition additionally controls how many probes are worth executing.

Full training rollout accounting. Appendix E.4 accounts for ordinary memory-assisted executions, no-memory reference executions, acquired probes, and all executed writing-validation attempts, counting executions reused across purposes only once. The matched-budget results measure feedback efficiency at equal additional-probe allocations, while Fig. 8 quantifies cost-conditioned probe selection over recorded acquisition states. Together, these analyses link feedback quality, adaptive acquisition volume, and the resource accounting of the complete training workflow.

AuthorsOpenReview ↗

Updated Manuscript and Additional Evidence

We have completed a comprehensive revision of the manuscript, incorporating the additional experiments, theoretical clarifications, and implementation details discussed in our responses. The revised paper also includes new cross-executor and cross-task generalization results, along with evaluations on Ministral-14B. We sincerely appreciate your constructive feedback and would be grateful if you could revisit the updated manuscript and our responses when convenient. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.