acceptodds
Under review as a conference paper at ICLR 2027

From Relevance to Decision Value: Low-Overhead Trajectory-Grounded Memory Valuation and Selection for LLM Agents

Abstract

Memory selection for large language model (LLM) agents hinges on a fundamental question: How should the value of a memory unit be defined? Existing methods often approximate memory value through relevance or novelty, while generative selectors and learned memory policies can require additional training and interaction. A practical observation is that many real-world agents operate repeatedly within bounded workflow domains, naturally accumulating execution histories over time. Motivated by this setting, we take a decision-centric view and define memory value by its query-conditioned contribution to downstream task utility. Based on this definition, we propose a practical trajectory-grounded Value-aware Memory (VAM) estimator, which directly induces a low-overhead Top- memory selection method without additional learned selectors and rollouts. We further analyze two key questions about the proposed estimator. First, when does the estimated value meaningfully reflect downstream memory contribution? Second, how much trajectory evidence is enough to make the valuation reliable? We answer the first through an average marginal-contribution interpretation, and the second by deriving a Valuation Reliability Assessment (VRA) method to quantify how strongly the available trajectory evidence supports the reliability of the resulting valuation. Experiments on ToolBench, HotpotQA, and ALFWorld demonstrate a favorable performance–cost trade-off across tool use, multi-hop question answering, and interactive decision making. The experiments further validate the practical utility of VRA and show that VAM can progressively improve downstream utility as execution history accumulates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, memory selection, memory valuation, LLM agents, trajectory-grounded learning, decision value

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TCszOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Value-aware Memory, a low-overhead memory selection method for large language model agents that estimates memory value from logged execution trajectories based on presence-absence utility differences. The authors also introduce Valuation Reliability Assessment via Fisher exact test to evaluate estimation reliability. Experiments on HotpotQA, ALFWorld, and ToolBench demonstrate a strong performance-cost trade-off.

Critical strengths
  1. Practical and low overhead. Value-aware Memory avoids expensive online generation or policy training during selection, achieving performance comparable to generative selectors with significantly fewer input tokens.

  2. Solid theoretical grounding. The formulation cleanly connects observational trajectory statistics to average marginal contributions, and the reliability assessment provides a principled statistical measure for finite sample evidence sufficiency.

Critical weaknesses

Strong reliance on structured and matched histories. Main experiments evaluate repeated executions on the exact same task instance. For cross-task transfer, the method relies heavily on ground-truth attributes to match task families and canonicalize actions, limiting generalization to open-ended tasks without structured metadata.

Questions for authors
  1. How can this method be applied to cross-task transfer in open-ended domains like ToolBench or HotpotQA where explicit structural attributes are unavailable for matching source and target tasks?

  2. Since the reliability assessment quantifies valuation reliability, why is it only used as a diagnostic metric rather than being integrated into the selection rule to penalize high-value memories with low statistical support during early executions?

3 comments
AuthorsOpenReview ↗

Response—Part 1/2

We thank the reviewer for recognizing the practical value and theoretical formulation. We address your two questions with additional experiments:

  1. Text-based cross-task matching (Q1): On 183 held-out ALFWorld games over 10 runs, task-text matching with VAM achieves 83.2% success, compared with 43.5% without memory and 95.3% with PDDL matching. It improves success by 39.7 pp and retains 76.6% of the PDDL-based improvement.
  2. VRA's role in selection (Q2): Inspection of 102 VAM–VRA ranking disagreements shows that 101 involve memories appearing only once or twice, exclusively in successful trajectories. VRA identifies this sparse-support mechanism. Experiments with reliability priority and hard filtering show that the former usually preserves the selected set, while the latter reduces Joint F1, supporting VRA's diagnostic role alongside VAM ranking.

Q1. How can VAM support cross-task transfer without explicit structural attributes for matching?

Task-text retrieval can identify useful source histories: VAM with text-based matching achieves 83.2% success on 183 held-out ALFWorld games, compared with 43.5% without memory.

We replaced PDDL-based source-family ranking with retrieval based on task descriptions. BM25 and dense embeddings (all-MiniLM-L6-v2) are combined using reciprocal rank fusion, followed by scoring-only cross-encoder reranking of the top 50 families. VAM then estimates memory values from the retrieved training histories and selects K=3 memories.

The evaluation covers six task types, draws source histories from 686 training families, and uses 10 runs per held-out game.

Source matchingSuccess rate
No Memory43.5%
PDDL attributes + VAM95.3%
Task-text retrieval + VAM83.2%

Text-based matching improves success by 39.7 percentage points over No Memory and retains 76.6% of the PDDL-based improvement, calculated as (83.2−43.5)/(95.3−43.5). This demonstrates that effective source-family ranking can use task text rather than exact structural-attribute matching. Text matching uses 18.30K input tokens per run, against 10.81K with PDDL matching.

To isolate the matching strategy, we kept retrieval quantity matched to the PDDL source-pool size (median four families) and retained the action-normalization and instantiation pipeline, which uses task descriptions, observable objects, and action text rather than hidden ground-truth structural attributes. All histories used for valuation come from the training split. Only the matching step changed, so the remaining 12.1 pp gap to PDDL matching is the cost of less precise source matching.

Application to ToolBench and HotpotQA. We have not yet run transfer on these benchmarks. The same approach would retrieve related histories using task descriptions and, for tool use, tool/API descriptions. VAM would then value reusable memory units represented consistently across those histories. ToolBench transitions can be aligned through tool identities and argument patterns. For HotpotQA, transfer requires shared evidence units or reusable reasoning memories, since the current paragraph candidates are question-specific. This separates two requirements: finding related source tasks through text, as demonstrated here, and establishing corresponding memory units across tasks, which remains open outside ALFWorld.

We have added the text-matching results and describe this route to transfer beyond explicit task-family metadata.

AuthorsOpenReview ↗

Response—Part 2/2

Q2. Why use VRA diagnostically rather than penalizing low-support memories during selection?

VRA identifies high scores supported by sparse evidence. Our selection experiments show why this diagnostic signal is useful without applying a blanket penalty: reliability-prioritized selection usually preserves the selected set, while hard filtering removes useful evidence and reduces performance.

What VRA detects. We inspected 102 VAM–VRA ranking-disagreement cases: 94 in HotpotQA (out of 25,000 history subsamples) and eight in ALFWorld. In each case, one memory has higher VAM but lower VRA than its comparator. The higher-value, lower-VRA memories have the following occurrence–outcome patterns:

Count pattern (n11, n10, n01, n00)Evidence patternCases
(1, 0, 1, …)One occurrence, successful; one success without the memory83
(2, 0, 1, …)Two occurrences, both successful; one success without the memory9
(1, 0, 2, …) or (1, 0, 3, …)One occurrence, successful; additional successes without the memory9
(5, 0, 0, …)Five occurrences, all successful; no success without the memory1

Here, n11/n10 count successful/failed trajectories containing the memory, and n01/n00 count those without it. The final count varies across cases in each pattern class.

101 of 102 cases involve only one or two occurrences, all successful. For example, counts (1,0,1,3) give VAM=1−1/4=0.75, although the memory appears only once. VRA distinguishes this high point estimate from strong statistical support.

Independent reference checks support this diagnosis. Among the 94 HotpotQA cases, 82 are distinguishable by at least one tested VRA threshold (0.5, 0.6, 0.7, or 0.8). The higher-value, lower-VRA memory is non-gold in 82/82 cases and outside the intervention-derived individual Top-2 in 80/82. In ALFWorld, the flagged action is outside the expert key path in 7/8 cases. These are subsample-level diagnostic cases, which may include repeated tasks.

Integrating VRA into selection. We tested two rules:

  1. Reliability priority with fallback: Prioritize candidates with VRA≥θ, rank them by VAM, and fill any remaining slots with other candidates ranked by VAM. Across θ∈{0.5,0.6,0.7,0.8}, HotpotQA selections are identical to ordinary VAM at n≤6; at n=8, at most 1.1% differ. The two orderings rarely disagree, and a disagreement in ordering between two already-selected memories does not itself change the Top-2 set.
  2. Hard filtering: Remove selected memories below θ without replacement. At θ=0.5, performance is:
History size nOrdinary VAM Joint F1VRA filtering Joint F1Difference
30.41460.3453−0.0693
40.49200.4808−0.0112
50.54380.5199−0.0239
60.56810.5616−0.0065
80.60050.5976−0.0029

These scores are looked up from the existing exhaustive intervention evaluations (answering model: DeepSeek-V4-Flash), so they are not on the scale of Table 1's HotpotQA column. Hard filtering leaves fewer than K hints and can discard useful paragraphs whose support is still limited. Its losses are consistent with the cost of withholding useful evidence outweighing the benefit of removing occasional poor selections.

The results support complementary roles: VAM ranks memories by estimated value; VRA identifies estimates whose supporting evidence is sparse. The tested integration rules provide no overall improvement, motivating our diagnostic-first design while leaving room for task-specific integration. We will add both the contingency-pattern analysis and the selection-rule comparisons to make this design choice explicit.

We hope these clarifications and additional results address your concerns. We have revised the manuscript accordingly and highlighted all changes in blue.

Thank you very much for your detailed reply. I will consider improving the rating.

Reviewer_M8F4OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VAM scores a memory by comparing the average reward of logged trajectories where it appears with those where it does not, then selects the top K. VRA uses a Fisher exact test to assess evidence for a positive association. The paper tests the method on HotpotQA, ToolBench, and ALFWorld, including transfer to held-out ALFWorld games.

Critical strengths

The held-out ALFWorld experiment is the most convincing result: VAM achieves 95.3% success, close to Reflection at 94.9%, with fewer input tokens. PMR uses the same histories but performs much worse, so the absence term appears to help. The method is also easy to implement.

The authors are careful about what the theory says. They acknowledge that more data cannot fix context mismatch and that VRA is not the probability that a value estimate is correct.

Critical weaknesses
  1. My main concern is what memory “presence” means in the agent experiments. Appendix C.1 counts a ToolBench or ALFWorld transition as present if the agent executed it. But the target in Eq. (2) concerns the benefit of giving the agent that transition as memory. A successful action can be a good predictor of success without being a useful hint. Table 7 also mixes models and search strategies, which could explain some of the association.

Proposition 3.1 makes the needed assumptions explicit, but I do not see evidence that they hold for these logs. The HotpotQA intervention experiment helps; the same check is needed in an agent environment. This is the main reason for my weak rejection recommendation.

  1. Two comparisons would make the experimental claims easier to judge. In the repeated-task setting of Table 1, an LLM selector could cache its choices when the query and candidate pool stay fixed. It is unclear how much of the cost advantage remains when both methods reuse previous work.

Also, Table 3 compares selections with the top two individual marginal values, not the best pair. These need not coincide: Table 6 finds interactions in 70% of gold-gold pairs. Since Appendix B.3 already evaluates all 45 pairs per question, the authors could directly report the reward lost by selecting their pair instead of the best one.

The results suggest that VAM is a useful selection rule. I am less convinced that its scores measure the memory contribution defined in the paper. A direct test of that claim could change my recommendation. I have not reproduced the experiments.

Questions for authors
  1. Can you clarify the link between executing a transition and receiving it as memory? A useful test would randomly provide or withhold a memory on ALFWorld, keep the others fixed, and compare the change in success with its VAM score.

  2. What are the costs for first use, cached reuse, and updates when LLM selectors can also cache their choices?

  3. On HotpotQA, how much reward is lost relative to the best pair in the existing exhaustive evaluation?

  4. With the number of trajectories held fixed, does VRA predict valuation error or selection regret? The current plots do not separate its usefulness from the effect of having more data.

4 comments
AuthorsOpenReview ↗

Response—Part 1/3

We thank the reviewer for the specific suggestions. Our additional experiments and analyses address all four questions:

  1. Hint utility (Q1): A controlled ALFWorld study comprising 6,000 executions finds that high-scoring non-terminal memories outperform low-scoring ones in 24 of 30 games, with five ties and one loss, yielding a 34.40 pp advantage under matched backgrounds. Terminal-action memories score highest but yield smaller gains (+8.1 pp); we report this as the boundary of the proxy.
  2. Caching (Q2): We provide first-use and cached-reuse cost accounting for 2, 5, and 10 executions. VAM retains the lowest projected input-token cost on HotpotQA and ToolBench at these reuse levels; caching can make generative selectors cheaper on ALFWorld.
  3. Best-pair comparison (Q3): Re-analysis of all 45 pairs per question across 100 HotpotQA questions gives VAM an 84% optimal-pair rate, 0.055 mean Joint F1 regret, and zero median regret, directly evaluating combination-level selection quality.
  4. VRA at fixed history size (Q4): Across 50 subsampling seeds and five history sizes, higher VRA is associated with better intervention-reference Top-2 alignment. At n=6, membership rises from 6.0% to 78.6%, while mean absolute deviation from full-history VAM falls from 0.280 to 0.097.

Q1. Does executing a transition predict its usefulness as a supplied hint?

The controlled ALFWorld experiment provides direct evidence: high-scoring non-terminal memories yield larger gains than low-scoring ones in 24 of 30 games under matched backgrounds. Terminal-action memories are the exception: they score highest but help less, and we report this in full.

In ALFWorld and ToolBench, logged presence records execution of a transition. To test its subsequent usefulness as a hint, we ran the test you suggest: we explicitly provided or withheld high-scoring non-terminal (H), low-scoring non-terminal (L), and terminal-action (T) memories. H is a random draw from the top-3 non-terminal memories by V^\hat V; L is the lowest-scoring non-terminal memory; T is the last action in ≥50% of a game's successful trajectories. These rules were fixed before the runs. Each game uses five two-memory backgrounds and five executions per background condition, with the execution agent and backgrounds fixed.

ConditionMean V^\hat V of added memorySuccess rateGain over background
Background only—39.9%—
Background + H0.74573.2%+33.3 pp
Background + L−0.71938.8%−1.1 pp
Background + T0.85948.0%+8.1 pp

H exceeds L in 24 games, ties in five, and falls below L in one, giving a mean gain advantage of 34.40 pp. A one-sided sign test over games, excluding ties, gives p<10⁻⁴. Standalone comparisons give 18 wins, six ties, and six losses, with a mean H–L advantage of 44.27 pp (p=0.011).

Terminal transitions are where this proxy fails. T scores higher than H on average and in 21 of 30 games, yet its gain is far smaller (+8.1 vs. +33.3 pp). In those 21 games, H's gain is larger in 15, T's in 3, and they tie in 3. This is the case you describe: a successful action can predict success without being a useful hint. A successful run executes its final step, but supplying that step helps much less than supplying an intermediate step that leads there.

Mid-score memories. On the six games where a mid-score non-terminal memory was also tested (mean V^\hat V 0.156; 300 executions), its gain is near zero: −2.0 pp standalone and +4.7 pp with backgrounds, against +54.0 pp for H with backgrounds on the same games.

Scope of the main ALFWorld results. The ALFWorld pools had no terminal filter (unlike ToolBench, where Finish actions are excluded; App. C.1). VAM's Top-3 contains a terminal memory in 89 of 100 games in Table 1 and in over 95% of held-out games in Table 2. In Table 1 each game has at most one such memory, so at least two of the three selected hints are non-terminal. We have not re-run these tables with terminal memories excluded and state this as a limitation.

Cohort. The study includes 6,000 H/L/T executions on 30 games sampled from 43 eligible games. Eligibility was specified before the interventions using existing Table 1 results: no-memory success ≤90%, VAM Top-3 success ≥10%, sufficient score separation, and enough background candidates. The new intervention outcomes were not used for selection.

AuthorsOpenReview ↗

Response—Part 2/3

Q1(continued). Does executing a transition predict its usefulness as a supplied hint?

Source heterogeneity in ToolBench. You are right that it matters. The source configurations in Table 7 have different logged success rates, and model-family stratification changes Top-2 selection in 21 of 43 tasks (one of the two memories in 19 of them). Negative-control API-error and repeated calls can also receive positive scores when they co-occur with successful trajectories; VAM's Top-2 contains such a call in 14 of 43 tasks. So on ToolBench the scores partly reflect which agent produced a trajectory, and we do not claim that they measure the contribution of an individual memory there. What Table 1 does establish on ToolBench is the benefit of the selected set: every method's selections are supplied to the same fixed agent, and VAM's selections raise success from 48.8% (no memory) to 69.8%, against 62.8% for PMR computed from the same logs. We will distinguish this intervention evidence from the assumptions underlying the observational interpretation.

Q2. What are the costs of first use, cached reuse, and updates?

Baseline selections can be cached when their inputs remain fixed. We now make this reuse explicit. Let T be Table 1's total input tokens and C its selection/synthesis input tokens. With caching, average input tokens over R executions, including the first, are

T‾(R)=(T−C)+C/R.\overline T(R)=(T-C)+C/R.

BenchmarkMethodOne-time selection cost CFirst executionAverage: 2 executionsAverage: 5 executionsAverage: 10 executions
HotpotQAVAM0493493493493
HotpotQALLM selector1,5282,0211,257799646
HotpotQAReflection6,4116,9013,6951,7721,131
HotpotQAProcedural4,0984,5892,5401,311901
ALFWorldVAM08,6678,6678,6678,667
ALFWorldLLM selector4168,7308,5228,3978,355
ALFWorldReflection1,0569,2408,7128,3958,289
ALFWorldProcedural7619,8159,4359,2069,130
ToolBenchVAM09,7979,7979,7979,797
ToolBenchLLM selector3,47014,59712,86211,82111,474
ToolBenchReflection11,13022,95817,39314,05412,941
ToolBenchProcedural4,75015,49813,12311,69811,223

All entries are input tokens; bold marks the lowest average within each benchmark and reuse count. Cached costs are accounting projections from Table 1's rounded selection shares, holding downstream execution costs fixed. Values are rounded after calculation.

At R=2, 5, and 10, VAM retains the lowest projected cost on HotpotQA and ToolBench. On ALFWorld, caching makes LLM selector or Reflection cheaper in the corresponding reuse regimes. Thus, the first-use advantage narrows with repeated reuse, and the end-to-end comparison depends on downstream execution as well as selection cost.

For updates, VAM incorporates new outcomes through count updates without LLM calls. An unchanged baseline input remains cacheable; incorporating new histories into generated reflections or procedures requires further synthesis. Caching also does not carry across tasks: in the cross-task setting of Table 2, where each held-out game is new, VAM uses 10.81K input tokens per run against 11.64K–12.56K for the LLM-based selectors. We will report these three cost regimes explicitly.

AuthorsOpenReview ↗

Response—Part 3/3

Q3. How much reward is lost relative to the best pair on HotpotQA?

VAM selects an empirically optimal pair on 84% of questions, with mean regret of 0.055 Joint F1 and zero median regret.

We re-analyzed the exhaustive evaluations of all 45 pairs for each of the 100 HotpotQA questions. For this pair-level analysis, reward is Joint F1. Optimal-pair rate is the proportion of questions where the selected pair attains the maximum observed reward, allowing ties; regret is the gap to that maximum.

MethodOptimal-pair rateMean pair regretMedian pair regret
VAM84.0%0.0550
Procedural83.0%0.0650
LLM selector80.0%0.0830
Reflection78.0%0.0920

PMR attains a 69% optimal-pair rate under the same reference. These results directly evaluate the selected combination and account for pairwise interactions, complementing Table 3's individual-value ranking comparison.

The individual-value Top-2 reference attains the best pair on 96% of questions, with mean pair regret 0.0064. We will report the direct pair-level results alongside the individual-value comparison.

A correction to Appendix B.3. This exhaustive evaluation, like Table 3, uses Joint F1 as reward and DeepSeek-V4-Flash for both the LLM-based selectors and the answering model. Appendix B.3 states the reward as supporting-fact EM and the model as that of the main HotpotQA experiments (MiniMax-M2.7); we will correct both. Scores from this evaluation are therefore not on the scale of Table 1's HotpotQA column.

Q4. Does VRA provide useful information when trajectory count is fixed?

Yes. At a fixed history size, higher VRA is associated with stronger agreement with the intervention-derived memory ranking and smaller deviations from full-history VAM.

Across 100 HotpotQA questions, we recomputed VAM and VRA with 50 history-subsampling seeds per question at each n∈{3,4,5,6,8}n\in\{3,4,5,6,8\}. Within each fixed n, we grouped memories selected by ordinary VAM into VRA bins. At n=6:

VRA binSelected memoriesIntervention-reference Top-2 membershipMean absolute deviation from full-history VAM
0–0.252336.0%0.280
0.25–0.5015547.7%0.240
0.50–0.805,06271.9%0.125
0.80–1.004,55078.6%0.097

The intervention reference ranks memories by their average marginal Joint F1 contribution over all other single-paragraph backgrounds:

Vemp(mi)=1N−1∑j≠i[R({mi,mj})−R({mj})],R=Joint F1.V_{\mathrm{emp}}(m_i)=\frac{1}{N-1}\sum_{j\ne i}\left[R(\{m_i,m_j\})-R(\{m_j\})\right],\qquad R=\text{Joint F1}.

The reference Top-2 uses these intervention scores, with ties broken by candidate index, independently of VAM and VRA scoring. Top-2 membership measures selection alignment. The final column measures estimation stability: the mean absolute difference between subsample VAM and full-history VAM for the memories in each bin.

With the trajectory count fixed at six, Top-2 membership rises from 6.0% to 78.6%, while deviation decreases from 0.280 to 0.097. Top-2 membership also increases across the reported VRA bins at each other tested history size. Thus, VRA provides diagnostic information beyond the total number of trajectories. These measurements assess selection alignment and stability relative to full history, complementing the pair-level reward regret in Q3.

A case analysis further illustrates the mechanism. Across 25,000 history subsamples, we identified 94 cases where one selected memory had higher VAM but lower VRA than the other. Of the 82 cases distinguishable by at least one tested threshold (0.5, 0.6, 0.7, or 0.8), the higher-value, lower-VRA memory was a non-gold paragraph in all 82 and outside the intervention-reference Top-2 in 80. These memories appeared only once or twice, always in successful trajectories. VRA therefore identifies high point estimates supported by sparse successful co-occurrences. Counts refer to subsample-level cases, which can include repeated questions.

We have incorporated the controlled intervention, cached-cost accounting, best-pair evaluation, and fixed-size VRA diagnostics into the manuscript revision.

We hope these clarifications and additional results address your concerns. We have revised the manuscript accordingly and highlighted all changes in blue.

Thank you for your rebuttal, I increase the score.

Reviewer_DrNiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select memories that improve an LLM agent’s downstream task performance. It proposes Value-aware Memory (VAM), which ranks candidate memories by the difference between the average utility of historical trajectories containing each memory and those not containing it, then selects the top-K memories. This approach reuses logged execution outcomes without training a selector or requiring additional generative selection at inference time. The paper establishes conditions under which this observational statistic corresponds to an average marginal contribution and introduces Valuation Reliability Assessment (VRA), based on Fisher’s exact test, to quantify statistical evidence for a positive memory–outcome association.

Experiments on HotpotQA, ALFWorld, and ToolBench report competitive performance with lower input-token consumption than generative selection baselines. The main experiments reuse histories from the same task instances, while a separate ALFWorld evaluation demonstrates transfer from training trajectories to structurally matched held-out tasks. Additional analyses examine memory budgets, redundancy, pairwise interactions, agreement with intervention-based memory rankings, and performance as execution history accumulates.

Critical strengths
  1. A practical method for valuing memory with minimal selection overhead. The method estimates memory utility from existing trajectory outcomes without training an additional model or making LLM calls during selection. In the reported HotpotQA setting, it achieves competitive Joint F1 score while using approximately 4× and 14× fewer total input tokens, respectively. It demonstrates a useful efficiency–performance tradeoff through a simple, implementable method, particularly when historical trajectories are already available.

  2. Evidence that the learned memory values transfer beyond previously attempted task instances. The held-out ALFWorld evaluation derives memory values from training-task histories and applies them to 183 held-out games. VAM achieves 95.3% success, compared with 94.9% for reflection, with approximately 7.1% fewer input tokens. Competitive performance under this separation provides evidence that the approach can support new tasks, extending its usefulness beyond same-instance memory reuse.

Critical weaknesses
  1. The evidence does not sufficiently connect the estimated value to the benefit of supplying a memory to an agent. In ALFWorld and ToolBench, memory presence is derived from transitions executed in historical trajectories. A transition can correlate with success because successful agents reach it, without its inclusion as a memory improving future decisions. The theoretical equivalence to marginal contribution requires assumptions about comparable contexts that are not empirically established. The HotpotQA intervention analysis provides useful support, but does not resolve this mismatch in the action-based environments. This is consequential because the paper’s central claim is that it measures decision value; the current evidence more firmly establishes an effective outcome-associated retrieval heuristic.

  2. The comparisons leave the contribution over closely related utility-based memory methods unresolved. The evaluation includes useful generic retrieval and prompting baselines, but lacks matched comparisons with methods such as MACLA and MemRL, which also use outcome-informed memory selection with frozen language models. MemRL is cited, but its methodological differences and performance relative to VAM are not adequately established; MACLA is absent. Because the core presence–absence estimator builds on existing data-valuation ideas, demonstrating what this adaptation contributes over closely related approaches is especially important. A comparison under the same agent, task split, available histories, and memory budget would help establish whether VAM offers a meaningful additional efficiency–performance advantage.

Questions for authors
  1. In ALFWorld and ToolBench, does “memory present” mean that a memory was supplied to the agent before execution, or that its corresponding transition occurred during execution? If it is the latter, what evidence supports using this association to estimate the benefit of supplying that transition as a future hint?

  2. Do you have evidence that memories with higher VAM scores produce larger improvements when included versus withheld under other matched conditions in ALFWorld or ToolBench? Existing results or a small controlled diagnostic would help determine whether VAM captures intervention benefit in these environments.

  3. What substantive differences distinguish VAM from MACLA’s Bayesian utility-based selection and MemRL’s reward-based memory ranking? Can existing evidence or a limited comparison using the same memory pool, agent, and evaluation budget establish the benefit of VAM’s estimator? Please also clarify whether the reported cost advantage assumes fresh selection for every attempt, and how it changes when baseline selections or synthesized memories can be reused.

4 comments
AuthorsOpenReview ↗

Response—Part 1/3

We thank the reviewer for recognizing the practical efficiency of VAM and the evidence for cross-task transfer. We address your concerns with the following clarifications and additional evidence:

  1. Memory presence (Q1): We distinguish transition execution in ALFWorld/ToolBench logs from paragraph provision in HotpotQA, clarify that all selected memories are supplied as hints at deployment, and test the execution-to-hint link through controlled interventions.
  2. Intervention benefit (Q2): Across 6,000 executions on 30 ALFWorld games selected using prespecified eligibility criteria, high-scoring non-terminal memories outperform low-scoring ones in 24 games, tie in five, and underperform in one under matched backgrounds, with a mean advantage of 34.40 percentage points. Terminal-action memories score highest but yield smaller gains (+8.1 pp); we report this as the boundary of the proxy.
  3. Related methods and cost (Q3): VAM performs on par with MemRL and MACLA at lower cost. From the same logged histories and with the same execution agent, VAM/MemRL-Q/MACLA-EU achieve 97.0%/96.5%/96.3% ALFWorld success and 84%/85%/64% HotpotQA optimal-pair rates. MemRL and MACLA first require generative memory construction, measured at 334.0% and 53.7% of the input-token budget for one reference VAM ALFWorld execution; VAM requires none. We also clarify how these costs amortize under cached reuse.

Q1. Does “presence” mean supplied as a hint, or executed during a logged trajectory?

  • Short answer: the latter, in ALFWorld and ToolBench. Presence means the transition from which the memory is extracted was executed in a logged trajectory. No memory was supplied when these logs were generated.
  • HotpotQA differs. Presence means the paragraph was supplied to the answering agent, so the logs directly record provision.
  • At deployment, all three are the same. The selected memories are supplied to the agent as hints.
  • We agree this creates a gap. In ALFWorld and ToolBench, V^\hat V measures the association between executing a transition and success, whereas the quantity of interest is the benefit of supplying it as a hint. Proposition 3.1 gives conditions under which the two coincide, but these are not guaranteed for our logs.
  • We tested the link directly (see Q2). With the agent and background memories fixed, supplying a high-V^\hat V non-terminal memory yields a larger gain than supplying a low-V^\hat V one in 24 of 30 ALFWorld games (5 ties, 1 loss). The link fails for terminal-action memories, as detailed in Q2.
  • Revision. Sec. 3 will define both logging regimes explicitly and state that, in execution-logged environments, V^\hat V is a proxy whose validity we test empirically. The intervention is added as Appendix B.6.
AuthorsOpenReview ↗

Response—Part 2/3

Q2. Do higher VAM scores correspond to larger intervention gains?

With matched backgrounds, high-scoring non-terminal memories outperform low-scoring ones in 24 of 30 games, tie in five, and underperform in one, with a mean advantage of 34.40 percentage points. Terminal-action memories are the exception: they score highest but yield smaller gains, and we report this in full.

We evaluated high-scoring non-terminal memories (H), low-scoring non-terminal memories (L), and terminal-action memories (T), both alone and alongside matched backgrounds. H is a random draw from the top-3 non-terminal memories by V^\hat V; L is the lowest-scoring non-terminal memory; T is the last action in ≥50% of a game's successful trajectories. These rules were fixed before the runs. For each game, we used five two-memory backgrounds and five executions per background condition. The intervention gain is

Δ(m;S0)=SuccessRate⁡(S0∪{m})−SuccessRate⁡(S0).\Delta(m;S_0)=\operatorname{SuccessRate}(S_0\cup\{m\})-\operatorname{SuccessRate}(S_0).

The same execution agent and background memories are used in each comparison, and gains are averaged across the five backgrounds.

MemoryMean V^\hat VStandalone successGain (pp)With-background successGain (pp)
Baseline—16.7%—39.9%—
H0.74565.7%+49.173.2%+33.3
L−0.71921.5%+4.838.8%−1.1
T0.85944.0%+27.348.0%+8.1

The standalone baseline supplies no memory; the with-background baseline supplies only the two background memories.

SettingH > LH = LH < LMean H–L difference
Standalone1866+44.27 pp
Matched background2451+34.40 pp

Using games as the paired comparison units, one-sided sign tests excluding ties give p=0.011 for standalone comparisons and p<10⁻⁴ for matched backgrounds. Because H and L share the same baseline within each comparison, their success-rate difference is also their intervention-gain difference.

Terminal transitions are where this proxy fails. T scores higher than H on average and in 21 of 30 games, yet its gain is far smaller (+8.1 vs. +33.3 pp with backgrounds). In those 21 games, H's gain is larger in 15, T's in 3, and they tie in 3. This is the dissociation you describe: a successful run executes its final step, so that step is strongly associated with success, but supplying it helps much less than supplying an intermediate step that leads there.

Mid-score memories. On the six games where a mid-score non-terminal memory was also tested (mean V^\hat V 0.156; 300 executions), its gain is near zero: −2.0 pp standalone and +4.7 pp with backgrounds, against +54.0 pp for H with backgrounds on the same games.

Scope of the main ALFWorld results. The ALFWorld pools had no terminal filter (unlike ToolBench, where Finish actions are excluded; App. C.1). VAM's Top-3 contains a terminal memory in 89 of 100 games in Table 1 and in over 95% of held-out games in Table 2. In Table 1 each game has at most one such memory, so at least two of the three selected hints are non-terminal. We have not re-run these tables with terminal memories excluded and state this as a limitation.

Cohort. The 30 games were sampled from 43 eligible games using criteria specified before the intervention runs: no-memory success ≤90%, VAM Top-3 success ≥10%, sufficient VAM-score separation, and enough candidate memories to construct the backgrounds. Selection used the existing Table 1 evaluations, rather than outcomes from the new interventions. The H/L/T analysis comprises 6,000 executions: 30 games × 4 conditions × 25 runs standalone, and 30 games × 5 backgrounds × 4 conditions × 5 runs with backgrounds.

Overall, among non-terminal transitions, high-scoring memories are useful hints while mid- and low-scoring ones give gains near zero; terminal transitions are the case where the score does not track hint benefit.

AuthorsOpenReview ↗

Response—Part 3/3

Q3. How does VAM compare with MemRL and MACLA, including construction costs and caching?

VAM performs on par with MemRL and MACLA at lower cost. Both methods must build their memories with LLM calls before they can select anything; VAM selects from the logged transitions directly. That construction step costs 334.0% (MemRL) and 53.7% (MACLA) of the input tokens of one reference VAM ALFWorld execution.

All three approaches use outcome information, but they differ in how memories are obtained and valued:

MethodMemory constructionEvidence used to value a memoryMemory-value signal
VAMNone: logged transitions are reused as they areOutcomes of histories containing and not containing the memoryDifference between the two mean outcomes
MemRLLLM callsRewards of episodes in which the memory was retrievedReward-based Q-value updates
MACLALLM calls (procedure abstraction and refinement)Successes and failures of episodes in which the procedure was usedBayesian expected utility

VAM is also the only one of the three that uses outcomes of histories in which the memory is absent. PMR in Tables 1–2 is VAM without this term; it trails VAM by 4.0 and 7.0 pp on same-instance ALFWorld and ToolBench, and by 29.7 pp in cross-task transfer. We have added MACLA and clarify the relationship to frozen-LLM utility-based methods in the related work.

Setup. All three methods start from the same logged histories and use the same execution agent, task set, and memory budget. For MemRL and MACLA we first ran their memory construction on these histories, then selected memories by Q-value (MemRL-Q) and by expected utility (MACLA-EU). Each constructed memory corresponds to one candidate transition or paragraph, so all methods select among the same candidates.

ALFWorld downstream performance. Each game is evaluated over 10 runs.

SelectorSuccess rateDifference vs VAM
VAM97.0%—
MemRL-Q96.5%−0.5 pp
MACLA-EU96.3%−0.7 pp

HotpotQA pair-level performance. Across 100 questions, we evaluated the selected pair against the best of all 45 candidate pairs using Joint F1, on the exhaustive intervention set (answering model: DeepSeek-V4-Flash).

SelectorOptimal-pair rateMean pair regret
VAM84.0%0.055
MemRL-Q85.0%0.052
MACLA-EU64.0%0.163

Optimal-pair rate counts questions where the selected pair attains the maximum observed Joint F1, allowing ties. Regret is the gap between that maximum and the selected pair's score. VAM and MemRL-Q perform similarly, while VAM has lower observed regret than MACLA-EU.

Memory-construction cost. The construction step that preceded the MemRL and MACLA results above:

MethodMeasured construction input tokens per taskConstruction / one reference execution
VAM00%
MemRL28,952334.0%
MACLA4,65353.7%

Ratios use 8,667 input tokens for one reference VAM ALFWorld execution, consistent with Table 1. VAM reaches comparable performance without this cost.

Caching and updates. Table 1 charges fresh generative selection on each attempt. With fixed inputs, baseline selections and constructed memories can be cached, so their one-time costs amortize across executions. Over 10 executions, the MemRL and MACLA construction costs correspond to 33.4% and 5.4% of the reference input-token budget per run. For the Table 1 baselines over 10 executions, the HotpotQA LLM selector averages 646 input tokens against 493 for VAM (2,021 on first use), and on ALFWorld cached LLM selector and Reflection become slightly cheaper than VAM (8,355 and 8,289 vs. 8,667); the full table is in our response to Reviewer M8F4 (Q2). Caching does not apply when new logged evidence arrives, where VAM updates counts without LLM calls while generated-memory workflows need further processing, or when each task is new, as in Table 2, where VAM uses 10.81K input tokens per run against 11.64K–12.56K for the LLM-based selectors. We have distinguish first-use, cached-reuse, and update costs in the revised manuscript.

We hope these clarifications and additional results address your concerns. We have revised the manuscript accordingly and highlighted all changes in blue.

Thank you for the detailed responses and additional experiments. These analysis address most of my questions. I will increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.