- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Value-aware Memory, a low-overhead memory selection method for large language model agents that estimates memory value from logged execution trajectories based on presence-absence utility differences. The authors also introduce Valuation Reliability Assessment via Fisher exact test to evaluate estimation reliability. Experiments on HotpotQA, ALFWorld, and ToolBench demonstrate a strong performance-cost trade-off.
Critical strengths
-
Practical and low overhead. Value-aware Memory avoids expensive online generation or policy training during selection, achieving performance comparable to generative selectors with significantly fewer input tokens.
-
Solid theoretical grounding. The formulation cleanly connects observational trajectory statistics to average marginal contributions, and the reliability assessment provides a principled statistical measure for finite sample evidence sufficiency.
Critical weaknesses
Strong reliance on structured and matched histories. Main experiments evaluate repeated executions on the exact same task instance. For cross-task transfer, the method relies heavily on ground-truth attributes to match task families and canonicalize actions, limiting generalization to open-ended tasks without structured metadata.
Questions for authors
-
How can this method be applied to cross-task transfer in open-ended domains like ToolBench or HotpotQA where explicit structural attributes are unavailable for matching source and target tasks?
-
Since the reliability assessment quantifies valuation reliability, why is it only used as a diagnostic metric rather than being integrated into the selection rule to penalize high-value memories with low statistical support during early executions?
3 comments
Response—Part 2/2
Q2. Why use VRA diagnostically rather than penalizing low-support memories during selection?
VRA identifies high scores supported by sparse evidence. Our selection experiments show why this diagnostic signal is useful without applying a blanket penalty: reliability-prioritized selection usually preserves the selected set, while hard filtering removes useful evidence and reduces performance.
What VRA detects. We inspected 102 VAM–VRA ranking-disagreement cases: 94 in HotpotQA (out of 25,000 history subsamples) and eight in ALFWorld. In each case, one memory has higher VAM but lower VRA than its comparator. The higher-value, lower-VRA memories have the following occurrence–outcome patterns:
| Count pattern (n11, n10, n01, n00) | Evidence pattern | Cases |
|---|---|---|
| (1, 0, 1, …) | One occurrence, successful; one success without the memory | 83 |
| (2, 0, 1, …) | Two occurrences, both successful; one success without the memory | 9 |
| (1, 0, 2, …) or (1, 0, 3, …) | One occurrence, successful; additional successes without the memory | 9 |
| (5, 0, 0, …) | Five occurrences, all successful; no success without the memory | 1 |
Here, n11/n10 count successful/failed trajectories containing the memory, and n01/n00 count those without it. The final count varies across cases in each pattern class.
101 of 102 cases involve only one or two occurrences, all successful. For example, counts (1,0,1,3) give VAM=1−1/4=0.75, although the memory appears only once. VRA distinguishes this high point estimate from strong statistical support.
Independent reference checks support this diagnosis. Among the 94 HotpotQA cases, 82 are distinguishable by at least one tested VRA threshold (0.5, 0.6, 0.7, or 0.8). The higher-value, lower-VRA memory is non-gold in 82/82 cases and outside the intervention-derived individual Top-2 in 80/82. In ALFWorld, the flagged action is outside the expert key path in 7/8 cases. These are subsample-level diagnostic cases, which may include repeated tasks.
Integrating VRA into selection. We tested two rules:
- Reliability priority with fallback: Prioritize candidates with VRA≥θ, rank them by VAM, and fill any remaining slots with other candidates ranked by VAM. Across θ∈{0.5,0.6,0.7,0.8}, HotpotQA selections are identical to ordinary VAM at n≤6; at n=8, at most 1.1% differ. The two orderings rarely disagree, and a disagreement in ordering between two already-selected memories does not itself change the Top-2 set.
- Hard filtering: Remove selected memories below θ without replacement. At θ=0.5, performance is:
| History size n | Ordinary VAM Joint F1 | VRA filtering Joint F1 | Difference |
|---|---|---|---|
| 3 | 0.4146 | 0.3453 | −0.0693 |
| 4 | 0.4920 | 0.4808 | −0.0112 |
| 5 | 0.5438 | 0.5199 | −0.0239 |
| 6 | 0.5681 | 0.5616 | −0.0065 |
| 8 | 0.6005 | 0.5976 | −0.0029 |
These scores are looked up from the existing exhaustive intervention evaluations (answering model: DeepSeek-V4-Flash), so they are not on the scale of Table 1's HotpotQA column. Hard filtering leaves fewer than K hints and can discard useful paragraphs whose support is still limited. Its losses are consistent with the cost of withholding useful evidence outweighing the benefit of removing occasional poor selections.
The results support complementary roles: VAM ranks memories by estimated value; VRA identifies estimates whose supporting evidence is sparse. The tested integration rules provide no overall improvement, motivating our diagnostic-first design while leaving room for task-specific integration. We will add both the contingency-pattern analysis and the selection-rule comparisons to make this design choice explicit.
We hope these clarifications and additional results address your concerns. We have revised the manuscript accordingly and highlighted all changes in blue.
Thank you very much for your detailed reply. I will consider improving the rating.
Response—Part 1/2
We thank the reviewer for recognizing the practical value and theoretical formulation. We address your two questions with additional experiments:
Q1. How can VAM support cross-task transfer without explicit structural attributes for matching?
Task-text retrieval can identify useful source histories: VAM with text-based matching achieves 83.2% success on 183 held-out ALFWorld games, compared with 43.5% without memory.
We replaced PDDL-based source-family ranking with retrieval based on task descriptions. BM25 and dense embeddings (all-MiniLM-L6-v2) are combined using reciprocal rank fusion, followed by scoring-only cross-encoder reranking of the top 50 families. VAM then estimates memory values from the retrieved training histories and selects K=3 memories.
The evaluation covers six task types, draws source histories from 686 training families, and uses 10 runs per held-out game.
Text-based matching improves success by 39.7 percentage points over No Memory and retains 76.6% of the PDDL-based improvement, calculated as (83.2−43.5)/(95.3−43.5). This demonstrates that effective source-family ranking can use task text rather than exact structural-attribute matching. Text matching uses 18.30K input tokens per run, against 10.81K with PDDL matching.
To isolate the matching strategy, we kept retrieval quantity matched to the PDDL source-pool size (median four families) and retained the action-normalization and instantiation pipeline, which uses task descriptions, observable objects, and action text rather than hidden ground-truth structural attributes. All histories used for valuation come from the training split. Only the matching step changed, so the remaining 12.1 pp gap to PDDL matching is the cost of less precise source matching.
Application to ToolBench and HotpotQA. We have not yet run transfer on these benchmarks. The same approach would retrieve related histories using task descriptions and, for tool use, tool/API descriptions. VAM would then value reusable memory units represented consistently across those histories. ToolBench transitions can be aligned through tool identities and argument patterns. For HotpotQA, transfer requires shared evidence units or reusable reasoning memories, since the current paragraph candidates are question-specific. This separates two requirements: finding related source tasks through text, as demonstrated here, and establishing corresponding memory units across tasks, which remains open outside ALFWorld.
We have added the text-matching results and describe this route to transfer beyond explicit task-family metadata.