acceptodds
Under review as a conference paper at ICLR 2027

Vantage: Learning Memory Advantage by Asymmetric Re-Attempt

Abstract

Agentic memory enables LLM agents to self-evolve without weight updates by accumulating experience in a memory bank and retrieving it for new queries. As the bank grows, the key is to learn each memory's utility score so that retrieval prioritizes the most useful memories. Existing methods learn utility from the outcome (success or failure) of each attempt, crediting a memory whenever it is retrieved in a successful attempt, even when the attempt would succeed without it. We therefore argue that utility should also account for memory advantage, i.e., the outcome change that a memory brings compared to a memory-free attempt on the same query. To this end, we propose Vantage, an efficient utility-learning framework that incorporates outcome changes without the cost of memory-free attempts: the change is approximated by immediately re-attempting the query with the updated memory bank, so the first attempt itself serves as the reference. The re-attempt is asymmetric: only failed queries are re-attempted, whereas successful queries, whose outcome rarely changes, are not; this limits repeated crediting and redundant memories. Under frozen-bank evaluation on diverse tasks spanning coding, computer use, and embodied planning, Vantage improves success rate (SR) over the strong baseline MemRL by an absolute 6.4% on training queries and 2.2% on held-out queries, with 40% fewer attempts and thus a 40% lighter memory bank. Incorporating outcome changes into MemRL also improves it, further validating the advantage view. Detailed ablations confirm that both the asymmetric re-attempt and the incorporation of outcome changes drive the gains, and the learned utilities align more closely with memory advantage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
memory-augmented agents, large language model agents, memory utility learning, experience reuse, self-improving agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CjALOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Vantage modifies outcome-based memory-utility learning by immediately retrying a task once after failure, using the updated memory bank, while skipping retries after success. The paper motivates this schedule through memory advantage relative to a memory-free attempt and also evaluates an explicit advantage-adjusted reward in MemRL-delta. The deployed Vantage algorithm itself retains the outcome reward and MemRL utility update. Across coding, OS, database and embodied-planning tasks, five-epoch Vantage reports average frozen-bank training-query SR of 78.9 versus 72.5 for ten-epoch MemRL, and held-out SR of 67.2 versus 65.0, with about 40% fewer total attempts/stored entries.

Critical strengths
  1. The scheduling change is simple and practically attractive. It allocates an immediate extra attempt to failures and avoids repeatedly storing almost identical successful trajectories. The benefit does not require updating the agent backbone or training an additional retriever.

  2. The paper carefully distinguishes runtime multi-attempt success from one-attempt frozen-bank evaluation. Section 4.1 explicitly warns that last-epoch SR is not directly comparable, and Table 3 includes genuinely held-out queries. Appendix C.1 also specifies restoration of the initial environment and a fresh interaction before retrying, avoiding direct carry-over of environment progress.

  3. The empirical evaluation includes four domains, a second backbone, explicit reward-weight comparisons, symmetric versus asymmetric retries, retry-depth controls and memory-count analyses. The observation that successful tasks succeed again in at least 98.3% of tested immediate retries supports the operational motivation for skipping them.

Critical weaknesses
  1. The central derivation does not establish that Vantage learns memory advantage. Section 3.3 correctly observes that subtracting a common memory-free baseline cancels when comparing two hypothetical rewards. However, Algorithm 1 never updates utilities using the reward difference; it applies separate outcome-based updates to potentially different retrieved sets. Equality of reward differences does not imply equality of utility estimates or rankings across queries. Even for one memory present in both attempts, starting from the same utility and applying the same two samples, the updates satisfy Q2(eta) - Q2(0) = -alpha * eta * b * (2 - alpha), where b is the common memory-free outcome. The immediate retry therefore implements a selective sampling/update schedule, not an identified estimator of the memory-free effect. This can still be a useful method, but the paper should either substantiate the approximation or frame its contribution more narrowly.

  2. The utility-versus-advantage analysis does not resolve this attribution issue. Section 4.4 labels a binary with-memory outcome y as an improvement when y exceeds the success fraction P from ten memory-free attempts. For every task with 0 < P < 1, this labels every success an improvement and every failure a degradation; the improvement share then reduces to success frequency on that subset. Appendix E.1 additionally removes unchanged outcomes and low-utility/no-memory cases, bins historical attempts using final utilities, and compares methods with different attempt schedules. The resulting plots are compatible with better success rates and sampling differences, but do not by themselves establish better estimation of memory advantage or individual-memory credit.

  3. The held-out benefit needs uncertainty estimates and a more targeted mechanism control. The primary held-out gains over MemRL are 1.2–2.8 points across domains, with evaluation sets of 139–342 queries. The additional-backbone results are useful corroboration, but there are no reported paired intervals or task-order repetitions for the main comparison. Existing ablations change reward weighting, retry symmetry or retry depth; they do not isolate immediate utility reranking from access to the newly written failure reflection. A matched selective-retry control that keeps utility ranking fixed would help determine what specifically drives the gain.

Questions for authors
  1. What assumptions connect the outcome-only updates in Algorithm 1 to the memory-free advantage defined in Section 3.2? Can the authors validate that connection on a common held-out query set, or explicitly characterize Vantage as a selective-retry schedule motivated by, rather than estimating, advantage?

  2. At a matched attempt budget, how much improvement remains when the immediate failure retry is retained but utility reranking is disabled or frozen? This would separate the schedule/new-reflection effect from utility learning.

  3. Can the authors report paired uncertainty for held-out SR and sensitivity to training-query order? For the advantage diagnostic, please use a common evaluation population and report expected success differences without conditioning away unchanged outcomes.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CjAL (1/3)

We thank the reviewer for the detailed analysis. This response addresses W1 and the first part of Q1; Responses 2 and 3 report the new experiments.

W1 / Q1 (first part): What assumptions connect the outcome-only updates of Algorithm 1 to the memory-free advantage?

Within a re-attempted query, one assumption suffices: both attempts are measured against the same memory-free outcome y∅y_\varnothing of the query (Section 3.2). Then (y(2)−y∅)−(y(1)−y∅)=y(2)−y(1)(y^{(2)} - y_\varnothing) - (y^{(1)} - y_\varnothing) = y^{(2)} - y^{(1)}. Algorithm 1 applies separate outcome-based updates to the two retrieved sets, but for a memory in the first set and not the second, and one in the second and not the first, the two updates move the gap between their utilities toward y(2)−y(1)y^{(2)} - y^{(1)}, the difference between the memory advantages of the two contexts, for every η\eta (Section 3.3, Appendix B.2).

Beyond it, we agree with the reviewer's derivation. For a memory present in both attempts, its utility depends on η\eta through bb, as Q2(η)−Q2(0)=−αηb(2−α)Q_2(\eta) - Q_2(0) = -\alpha \eta b (2 - \alpha) shows. Across queries, equal reward differences do not imply equal utilities or rankings, since memories are retrieved for queries with different memory-free outcomes. No further assumption we could justify closes this gap, so the immediate retry is a selective sampling and update schedule, not an identified estimator of the memory-free effect. Accordingly, the revision no longer says that the re-attempt approximates memory advantage. Section 3.3 now distinguishes learning from an explicit advantage signal, as MemRL$^\Delta$ and Vantage$^\Delta$ do, from learning utilities that reflect memory advantage, which Vantage aims at, and states what holds within a re-attempted query.

Validation on a common held-out query set. We ran it with the protocol the reviewer specifies in Q3 (Response 2, new Table 6). On the same 771 held-out queries, with unchanged outcomes kept, the high-minus-low utility contrast in mean d=y−y∅d = y - y_\varnothing is −0.104 for MemRL (its highest-utility memories bring the least gain), +0.056 for Vantage, and +0.189 for Vantage$^\Delta$, which uses memory-free attempts. Vantage's own contrast is not significant (95% CI [−0.02, +0.13]), but it exceeds MemRL's by 0.16 (95% CI [+0.05, +0.27]): without memory-free attempts, its utilities reflect memory advantage more closely than MemRL's.

AuthorsOpenReview ↗

Response to Reviewer CjAL (2/3)

Q1 (second part), W2 and Q3 (second part): Validation on a common held-out set

We agree that the original analysis (Section 4.4, now Appendix E.1) does not by itself establish better estimation of memory advantage: its results are also compatible with higher success rates and different sampling, since it is computed on each method's own training attempts and, with a fractional PP, counts every success as an improvement. It views the training process, asking how often attempts that retrieved memories of a given final utility ended above the memory-free expectation PP rather than below it; its bins leave out the lowest utilities (below 0.4), as the Q-value analysis of MemRL does (Zhang et al., 2026, Fig. 7). We therefore add the diagnostic the reviewer proposes.

Protocol. To have enough queries in each utility group, we pool the held-out queries of the four domains. Common evaluation population: every method is evaluated on the same 771 held-out queries (four domains), with its frozen bank and one greedy attempt per query. Expected success difference: the reference is the greedy memory-free outcome y∅y_\varnothing of the same query, deterministic under our greedy decoding, so a group's expected success difference is the mean of d=y−y∅∈−1,0,1d = y - y_\varnothing \in \\{-1, 0, 1\\}. No conditioning: the mean is taken over all queries in the group, so unchanged outcomes stay in it with d=0d = 0, and the three groups together cover all 771 queries. Queries are grouped by the learned utility of the top-ranked retrieved memory.

Table 6 (revised manuscript): Mean dd by the utility of the top-ranked memory (number of queries)

Utility of top-ranked memoryMemRLVantageVantage$^\Delta$
< 0.6+0.146 (185)+0.080 (262)+0.049 (447)
0.6–0.8+0.156 (135)+0.095 (201)+0.109 (156)
≥ 0.8+0.042 (451)+0.136 (308)+0.238 (168)
Difference, ≥ 0.8 minus < 0.6 (95% CI)−0.104 [−0.18, −0.03]+0.056 [−0.02, +0.13]+0.189 [+0.09, +0.29]

(CIs: normal approximation; the groups of a method are disjoint sets of queries.) Every group has a positive mean dd, so memory helps on average throughout; the question is whether higher utility marks a larger gain, which the last row tests. The pattern is the same with equal-size terciles of the 771 queries (257 each): the highest minus the lowest tercile is −0.129 for MemRL (95% CI [−0.20, −0.05]), +0.051 for Vantage and +0.150 for Vantage$^\Delta$, so Vantage's exceeds MemRL's by 0.18 (95% CI [+0.07, +0.29]); terciles formed within each domain give the same pattern.

  • MemRL over-credits. For 58% of held-out queries its top-ranked memory has utility ≥ 0.8, yet these queries gain the least from memory: the high − low contrast is significantly negative. In its top bin (≥ 0.9, 340 queries), memory changes the outcome for 15% of them; conditioning on changed outcomes would hide this behind a 60% improvement share.
  • The explicit reference removes this: Vantage$^\Delta$ shows a clearly increasing relation.
  • Vantage lies in between: it has the comparison after a failure that MemRL lacks, but not the memory-free outcome that Vantage$^\Delta$ uses. The inversion disappears and the relation increases, less steeply than with the explicit reference; its own contrast is not significant, but it differs from MemRL's by 0.16 (95% CI [+0.05, +0.27]).

Given this validation, we keep the paper's claim that Vantage's utilities reflect memory advantage more closely than MemRL's, now distinguished from learning with it (Response 1). This diagnostic is now the main analysis of utility tracking in Section 4.4 (Table 6), with the original analysis moved to Appendix E.1; Section 3.1 and Appendix G state that splitting credit among the memories of one context remains open.

AuthorsOpenReview ↗

Response to Reviewer CjAL (3/3)

W3 (mechanism control) / Q2: Selective-retry control with utility ranking fixed, at a comparable budget

We ran this control, and no improvement remains: it falls below MemRL on held-out queries in all four domains. With greedy decoding, a re-attempt with an unchanged context reproduces the failure, so a successful re-attempt comes from a changed context: (i) utility reranking or (ii) the newly written failure reflection. The control keeps the asymmetric re-attempt and the reflection, which the re-attempt retrieves first by similarity, and disables utility ranking (λ=0\lambda = 0). In our pipeline, disabled and frozen coincide: with utilities frozen at initialization, all candidates have equal utility, the within-pool z-score is zero (Appendix C.3), and ranking reduces to similarity. The control uses 31,557 attempts, against 30,185 for Vantage, so its lower SR is not due to a smaller budget. To isolate what the retry adds, we also set λ=0\lambda = 0 in MemRL, which does not re-attempt.

Table 17 (revised manuscript): Retry-only control, Frozen / Held-out SR (%)

MethodCodeOSDBPlanningAvg.
MemRL53.4 / 41.857.1 / 53.395.0 / 85.684.6 / 79.372.5 / 65.0
MemRL, λ=0\lambda = 053.1 / 40.955.1 / 46.093.6 / 83.585.9 / 76.471.9 / 61.7
Retry-only, λ=0\lambda = 053.8 / 40.455.4 / 45.393.4 / 84.286.6 / 75.772.3 / 61.4
Vantage62.3 / 43.064.3 / 56.095.3 / 87.893.7 / 82.178.9 / 67.2

Without utility ranking, the same schedule and reflections reach 61.4 held-out SR, against 65.0 for MemRL, and relative to MemRL with λ=0\lambda = 0 the retry changes average held-out SR by −0.3 (61.7 → 61.4) and frozen SR by +0.4, by at most 0.7 points on any domain. MemRL with λ=0\lambda = 0 runs 10 epochs (50,620 attempts), as MemRL does; stopped at a budget comparable to the control's, it reaches 71.7 frozen and 61.7 held-out SR, so there the retry changes them by +0.6 and −0.3. With utility ranking, the same retry adds +2.2 held-out and +6.4 frozen points (MemRL → Vantage), on every domain. At a comparable budget, the schedule and the new reflection thus add nothing on their own; the improvement comes from utility learning. With Table 5 (re-attempting successes as well: 64.3), the gain requires both utility learning and a schedule that does not re-credit successes.

W3 (uncertainty) / Q3 (first part): Paired uncertainty and order sensitivity

Paired uncertainty. All methods are evaluated on the same held-out queries, so we compare them per query, with exact McNemar tests on discordant pairs and paired bootstrap CIs (10,000 resamples, stratified by domain).

Table 18: Paired comparison on held-out queries, Vantage minus MemRL (Δ\Delta in points; bb / cc count the queries solved only by Vantage / only by MemRL)

CodeOSDBPlanningPooled (771)
Δ\Delta (95% CI)+1.2 [−0.9, +3.2]+2.7 [−0.7, +6.7]+2.2 [−0.7, +5.8]+2.9 [0.0, +6.4]+1.9 [+0.5, +3.4]
bb / cc8 / 46 / 24 / 15 / 123 / 8
McNemar pp0.3880.2890.3750.2190.011

Vantage is ahead in every domain, but the per-domain sets are small: a margin of 3–4 queries cannot reach p<0.05p < 0.05 even if every discordant pair favors one method. Pooled over the 771 queries, the difference is significant and its CI excludes zero. The pooled Δ\Delta weights domains by size; the domain average in Table 3 is +2.2.

Order sensitivity. Due to the short rebuttal period, we ran this check on OS and DB: we repeated runtime learning with two additional shuffled training orders; within each order, all methods see the same sequence.

Table 19: Three training orders on OS and DB, mean ± std over the orders (%) and the paired difference within each order

OS Held-outOS FrozenDB Held-outDB Frozen
MemRL53.5 ± 0.458.4 ± 1.286.3 ± 1.394.3 ± 1.3
Vantage56.9 ± 1.065.5 ± 1.488.0 ± 0.494.9 ± 0.3
Paired Δ\Delta (wins)+3.4 ± 0.7 (3/3)+7.1 ± 0.6 (3/3)+1.7 ± 1.5 (2/3, 1 tie)+0.6 ± 1.1 (2/3)

Absolute SR varies with the order, as expected for runtime learning, but the comparison is stable: on held-out queries, Vantage is ahead in 5 of the 6 domain × order comparisons and ties in the sixth. DB is close to saturation (Frozen SR 93–95%), so its margins amount to a few queries. We will extend these repetitions to all four domains in a subsequent revision.

The revision adds the λ=0\lambda = 0 controls (Section 4.3, Appendix D.6), the paired tests and the order repetitions (Appendix D.7). We thank the reviewer again for suggesting these analyses.

Dear Reviewer CjAL,

Thank you again for your detailed review. We hope our rebuttal has addressed your concerns, and we would be happy to clarify anything further.

Best regards, The Authors

Reviewer_sPGpOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces Vantage, a method to efficiently manage agent memory banks and update memory utility score based on memory advantage. The key mechanism is to re-attempt a failed query with an updated memory bank and update utility score based on the outcome again, but avoid doing so for successful queries to avoid repeated crediting. Evaluations and analyses show that this method improves over prior methods in performance during utility learning as well as when memory bank is frozen; the asymmetrical reattempt setup also brings more efficient memory usage.

Critical strengths
  • This benefits from the method is clearly demonstrated: more efficient memory construction, improvements even when memory is frozen.
  • The main evaluations are followed by proper ablations and further analyses including # of attempts, utility tracking, and control of memory redundancy, and benchmark comparisons.
  • The writing is quite clear and easy to follow.
Critical weaknesses
  • No substantial concerns from my perspective.
Questions for authors
  • Have you considered whether this method would work well if testing in a multi-task setting?
2 comments
AuthorsOpenReview ↗

Response to Reviewer sPGp

We sincerely thank the reviewer for the strong support and for recognizing that the benefits of Vantage are clearly demonstrated. We ran a new experiment to answer the question on multi-task settings.

Q: Would the method work in a multi-task setting? Yes, in both settings we tested. We read the question as one agent with one shared memory bank serving several kinds of tasks: across domains, one bank for OS and DB gives the same SR as two per-domain banks, with Vantage ahead of MemRL (new experiment), and within one domain, where one ALFWorld bank serves six task types, Vantage matches or exceeds MemRL on 5 of them.

Across domains: one bank for OS and DB (new). We ran runtime learning on the OS and DB training queries together (711 queries), with one shared bank and otherwise the settings of the paper, and evaluated it on these queries and on the 289 held-out queries of both domains.

Table 20 (revised manuscript): One bank for OS and DB, Frozen / Held-out SR (%), with the per-domain results of Table 3

MethodOS (Table 3)DB (Table 3)OS + DB, one bank (new)
No memory44.9 / 44.783.1 / 82.764.3 / 63.0
MemRL57.1 / 53.395.0 / 85.676.4 / 68.9
Vantage64.3 / 56.095.3 / 87.880.0 / 71.3

The shared bank matches the two per-domain banks pooled over the same queries, so sharing it changes neither method nor the gap between them. Retrieval keeps the domains apart: Phase A excludes memories below the similarity threshold δ\delta (Section 3.1), and OS and DB queries are far apart, so a query almost never retrieves a memory of the other domain, and the attempt rule applies unchanged.

Within a domain: six task types in one bank. One ALFWorld bank serves all six task types, so a memory written for one type can be retrieved for another.

Table 21: ALFWorld held-out queries solved, by task type (nn: held-out queries per type)

Task type (nn)No memoryMemRLVantage
Pick (35)303031
Pick Two (24)21415
Examine (13)81213
Clean (27)222525
Heat (16)01012
Cool (25)22019
Total (140)64111115

Vantage matches or exceeds MemRL on 5 of the 6 types. Memory matters most for the types the backbone rarely solves alone (Heat, Pick Two, Cool) and least for those it already solves (Pick, Clean). Such a mix is what the asymmetric re-attempt is designed for: in a shared bank, task types the backbone already solves produce many unchanged successes, which Vantage does not re-credit within an epoch, while failures on harder types trigger the re-attempt that contrasts memories on the same query.

The revision adds the shared-bank experiment and the per-type results in Appendix D.8.

Dear Reviewer sPGp,

Thank you again for your review and your support. We hope our rebuttal has addressed your question, and we would be happy to clarify anything further.

Best regards, The Authors

Reviewer_WYVtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Vantage, an asymmetric re-attempt strategy for improving memory utility learning in LLM agents. By retrying only failed queries with an updated memory bank, Vantage reduces redundant attempts and memory writes. Experiments across four task domains show improved success rates over MemRL with 40% fewer attempts.

Critical strengths
  1. The proposed method is simple and practical, improving memory learning without additional memory-free attempts.

  2. The results are consistent across different tasks, and the ablation studies provide useful evidence for the design choices.

Critical weaknesses
  1. The connection to memory advantage is not fully convincing. Vantage still updates utilities using task outcomes, and both attempts use retrieved memories. It is unclear whether the gains come from better utility learning or simply a more effective retry strategy.

  2. The method still assigns the same reward to all retrieved memories, leaving the issue of individual memory contribution unresolved.

Questions for authors

Can the authors better distinguish the benefits of selective retries from those of improved memory utility estimation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer WYVt (1/2)

We thank the reviewer for the careful review and for recognizing that the method is simple, needs no memory-free attempts, and gives consistent results. We have uploaded a revised manuscript with the changes in blue; the tables below are taken from it.

Q and W1: Selective retries vs. utility estimation. Yes. Crossing the retries with utility ranking separates their benefits: the retries alone bring no gain, but together with utility ranking they bring Vantage's gain over MemRL (point 1). Their benefit is thus realized through utility estimation. The retry decides which attempts are run and credited, and thus what the utilities learn; as point 3 shows, the resulting utilities track memory advantage better than MemRL's, without a memory-free attempt or a value of η\eta. Retries happen during runtime learning, not at evaluation: Frozen SR and Held-out SR use one attempt per query with the bank fixed, so the gains at evaluation come from what was stored and how it is ranked. Three tests support this:

1. Without utility ranking, retries do not help (new controls: Section 4.3; Appendix D.6, Table 17). We keep Vantage's re-attempt and memory writing but rank by similarity alone (λ=0\lambda = 0). The experience written after a failure has the query's own embedding, so the re-attempt still retrieves it first: the control keeps everything a retry offers except utility ranking. We also set λ=0\lambda = 0 in MemRL, which does not re-attempt; the four runs thus cross the re-attempt with utility ranking.

Table 17: Retry-only control, Frozen / Held-out SR (%) and attempts

MethodCodeOSDBPlanningAverageAttempts
MemRL (10 epochs)53.4 / 41.857.1 / 53.395.0 / 85.684.6 / 79.372.5 / 65.050,620
MemRL, λ=0\lambda = 0 (10 epochs)53.1 / 40.955.1 / 46.093.6 / 83.585.9 / 76.471.9 / 61.750,620
Retry-only, λ=0\lambda = 0 (5 epochs)53.8 / 40.455.4 / 45.393.4 / 84.286.6 / 75.772.3 / 61.431,557
Vantage (5 epochs)62.3 / 43.064.3 / 56.095.3 / 87.893.7 / 82.178.9 / 67.230,185

With a comparable budget, slightly larger than Vantage's (31,557 vs. 30,185 attempts), the retry-only control falls below MemRL on Held-out SR in all four domains (61.4 vs. 65.0 on average). Adding the re-attempt without utility ranking (MemRL, λ=0\lambda = 0 → retry-only) changes average Held-out SR by −0.3 (61.7 → 61.4) and Frozen SR by +0.4, by at most 0.7 points on any domain; at a budget comparable to the control's, MemRL with λ=0\lambda = 0 reaches 71.7 Frozen and 61.7 Held-out SR. With utility ranking, the same re-attempt adds +2.2 Held-out and +6.4 Frozen points (MemRL → Vantage), on every domain. These controls rule out a pure retry effect: the re-attempt helps through utility ranking, not on its own. Point 3 tests the utilities directly.

2. The schedule and the reward interact (Section 4.3, Table 5).

Table 5: Re-attempting successes and the reward weight η\eta, four-domain average Frozen / Held-out SR (%)

η=0\eta = 0η=0.5\eta = 0.5
Re-attempt after success too75.7 / 64.376.9 / 65.8
Re-attempt only after failure78.9 / 67.2 (Vantage)79.1 / 67.0 (Vantage$^\Delta$)

Not re-attempting successes adds +2.9 Held-out SR at η=0\eta = 0. With the attempts held fixed, changing the reward weight to η=0.5\eta = 0.5 recovers +1.5 of it; this lowers the reward on queries the backbone solves without memory, both for unchanged successes (1 → 0.5) and for degraded failures (0 → −0.5). At η=0.5\eta = 0.5, the selective schedule still adds +1.2 (65.8 → 67.0), consistent with the near-duplicate memories that re-attempts after success write: they lower the distinct sequences per write on OS from 73.1% to 59.9% (Section 4.4, Figure 5). Setting η=0.5\eta = 0.5 thus raises average Held-out SR when successes are re-attempted (+1.5) but not under the selective schedule (67.0 vs. 67.2): both reduce the credit of unchanged successes, so the schedule takes over much of the role of η\eta, an overlap that a pure retry effect would not produce. The overlap is not driven by one domain: on each of the four, not re-attempting successes raises Held-out SR at both values of η\eta, and by less at η=0.5\eta = 0.5 (Appendix D.1).

3. The utilities, tested directly (Section 4.4, Table 6). On the 771 held-out queries, MemRL's highest-utility memories bring the least gain from memory (high-minus-low contrast −0.104), while Vantage's contrast is +0.056; the difference, 0.16 (95% CI [+0.05, +0.27]; 0.18 [+0.07, +0.29] with equal-size terciles), shows that Vantage's utilities track memory advantage better. An advantage-adjusted reward would need a memory-free attempt per query and a value of η\eta, whose preferred value varies by metric on OS (Table 4); Vantage needs neither. Part 2/2 gives the full table.

AuthorsOpenReview ↗

Response to Reviewer WYVt (2/2)

W1: The connection to memory advantage. Indeed, Vantage updates utilities with task outcomes, and both attempts use retrieved memories; the connection to memory advantage comes from comparing the two attempts. Both are made on the same query, so they share its memory-free outcome, which cancels in their comparison: for any memory used only in the first attempt and any used only in the re-attempt, the outcome updates move the gap between their utilities toward the difference between the two contexts' memory advantages, for every η\eta (Section 3.3, Appendix B.2). The first attempt is thus a reference for this comparison, not a substitute for the memory-free baseline. This does not determine memories used in both attempts or utilities aggregated across queries, so we measure whether Vantage's utilities reflect memory advantage. Whether the gains come from better utility learning or from the retry itself is answered in Part 1/2: the retry helps through utility ranking, not on its own.

Measured effect (new Table 6 in Section 4.4; Appendix E.2). If utilities track memory advantage, queries whose top-ranked memory has a higher utility should gain more from memory. Each method runs the same 771 held-out queries once, greedily, with its frozen bank; the gain of a query is d=y−y∅d = y - y_\varnothing, with y∅y_\varnothing from a greedy memory-free attempt (unchanged outcomes count as 0).

Table 6: Mean dd by the utility QQ of the top-ranked memory (queries in parentheses)

Top-1 QQMemRLVantageVantage$^\Delta$
< 0.6+0.146 (185)+0.080 (262)+0.049 (447)
0.6–0.8+0.156 (135)+0.095 (201)+0.109 (156)
≥ 0.8+0.042 (451)+0.136 (308)+0.238 (168)
High − low [95% CI]−0.104 [−0.18, −0.03]+0.056 [−0.02, +0.13]+0.189 [+0.09, +0.29]
  • MemRL is inverted. 58% of its queries have a top-1 QQ of at least 0.8, yet these gain the least.
  • Vantage removes the inversion. Its mean dd rises with the utility. Its own contrast is not significant, but it exceeds MemRL's by 0.16 (95% CI [+0.05, +0.27], above zero for any correlation between the two). With equal-size terciles (257 queries each), the difference is 0.18 (95% CI [+0.07, +0.29]; Appendix E.2).
  • Vantage lies between MemRL and Vantage$^\Delta$. Vantage$^\Delta$ uses memory-free attempts to apply η=0.5\eta = 0.5 and tracks advantage most closely, but in our experiments it does not improve average Held-out SR (67.0 vs. 67.2); η=0.5\eta = 0.5 raises it in the setting that also re-attempts successes (+1.5, Part 1/2). Vantage reaches this accuracy without memory-free attempts and without a value of η\eta to choose.

W2: Individual memory contribution. We agree that Vantage does not isolate the individual contribution of each co-retrieved memory. MemRL lists the same limitation (MemRL, Sec. 6): both methods assign the same attempt-level reward to all retrieved memories. Section 3.1 now states it, in addition to Appendix G.

  • What a utility tracks. A utility is a recency-weighted average of the rewards of the attempts that retrieved the memory (Eq. (2)), so it mixes the memory's own contribution with the backbone's memory-free ability and with that of the co-retrieved memories. Different retrieval histories give different utilities, but a helpful and a non-contributing memory always retrieved together from the same initial utility receive identical updates.
  • What the re-attempt compares. Two contexts on the same query: the memory-free part cancels, and every memory swapped in moves by the same amount relative to every memory swapped out, but co-retrieved memories still share the reward. It separates contexts, not the memories within one.
  • Measuring or learning it. Leave-one-out attempts, removing one memory while the query and the other memories stay fixed, would measure it in a given context; splitting credit within an attempt, as RoMeRL does with marginal utility or as Shapley-style decompositions would, would learn it. Both are orthogonal to the attempt schedule and can be combined with Vantage.

Dear Reviewer WYVt,

Thank you again for your thoughtful review. We hope our rebuttal has addressed your concerns, and we would be happy to clarify anything further.

Best regards, The Authors

Thank you for the detailed response and additional experiments. The new retry-only control helps clarify where the performance gains come from, and the revised discussion provides a more precise connection to memory advantage. While individual memory credit assignment remains an open issue, my main concerns have been largely addressed. I will maintain my rating of 3 (Weak Acceptance).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive reviews. They found the method simple and practical (CjAL, WYVt) and its benefits clearly demonstrated (sPGp). Two questions recur: what drives the gain, and how outcome-only updates connect to memory advantage. We answer both with new analyses on the same held-out queries as the paper.

1. What drives the gain (WYVt, CjAL). Utility learning under the asymmetric schedule, not the retry on its own. A retry-only control keeps the asymmetric re-attempt and the new failure reflections but disables utility ranking (λ=0\lambda = 0). It falls below MemRL on held-out queries in all four domains (61.4 vs. 65.0), although it uses more attempts than Vantage (31,557 vs. 30,185). Without utility ranking, the re-attempt does not help: MemRL with λ=0\lambda = 0, which does not re-attempt, reaches 61.7, also at a comparable budget. Re-attempting successes as well gives 64.3. Vantage reaches 67.2.

2. Connection to memory advantage (CjAL, WYVt). Within a re-attempted query, both attempts share the query's memory-free outcome, which cancels in their comparison: for any memory used only in the first attempt and any used only in the re-attempt, the outcome updates move the gap between their utilities toward the difference between the two contexts' memory advantages, for every η\eta (Response 1 to CjAL). This does not determine memories used in both attempts or utilities aggregated across queries, so Vantage is motivated by memory advantage rather than estimating it, and we measure the learned utilities directly: on a common held-out set with every outcome kept, MemRL's highest-utility memories bring the least gain from memory, an inversion that Vantage removes, and Vantage$^\Delta$ tracks memory advantage most closely (Response 2 to CjAL). The revision makes the paper's logic explicit (Section 3.3), which frames the contribution more precisely. We aim to learn utilities that reflect memory advantage, and the paper builds toward this aim in three steps: (1) learning with memory advantage as the reward improves MemRL (MemRL$^\Delta$), but needs a memory-free attempt and a value of η\eta; (2) re-attempting failed queries immediately with the updated memory bank improves it further (Vantage$^\Delta$); (3) since the memory-free outcome cancels in this re-attempt, Vantage drops the memory-free attempt. Both Vantage$^\Delta$ and Vantage are thus contributions of the paper, and Vantage matches Vantage$^\Delta$ in average performance (Frozen / Held-out SR 78.9 / 67.2 vs. 79.1 / 67.0) without memory-free attempts or a value of η\eta to choose.

3. Robustness (CjAL). Pooled over the 771 held-out queries, Vantage solves 23 queries that MemRL fails and fails 8 that MemRL solves (+1.9 points, 95% CI [+0.5, +3.4], exact McNemar p=0.011p = 0.011), and it is ahead in every domain. Across three training orders on OS and DB, it is ahead in 5 of the 6 held-out comparisons and ties in the sixth (Response 3 to CjAL).

4. Multi-task (sPGp). One bank shared by OS and DB gives the same SR as two per-domain banks for every method, and in ALFWorld, where one bank serves six task types, Vantage matches or exceeds MemRL on 5 of them (Response to sPGp).

Revisions. The method and the main results are unchanged. The revision no longer says that the re-attempt approximates memory advantage (CjAL); it states what holds within a re-attempted query (Section 3.3) and measures the rest: on 771 common held-out queries, MemRL's highest-utility memories bring the least gain from memory, and Vantage's utilities track that gain more closely than MemRL's (difference 0.16, 95% CI [+0.05, +0.27]; Table 6). A few phrases in the abstract, the introduction and related work are adjusted accordingly, and the conclusion is shortened. The revised manuscript, with changes in blue, adds the logic of Response 1 to Section 3.3 (derivation in Appendix B.2), the retry-only control and MemRL with λ=0\lambda = 0 to Section 4.3 (Appendix D.6), and the held-out advantage diagnostic as the main analysis of utility tracking in Section 4.4 (Table 6), with the original analysis in Appendix E.1; the appendix also reports the paired tests and the order repetitions (D.7) and the shared-bank experiment and the per-type results (D.8). Section 3.1 states that credit is not split among co-retrieved memories.

AuthorsOpenReview ↗

Summary of Revisions

We thank the reviewers again. We have uploaded a revised manuscript in which all changes are marked in blue. The method and the main results are unchanged. The revisions add the analyses from the discussion and no longer describe the re-attempt as approximating memory advantage.

  • Section 3.1: states that every retrieved memory receives the same reward and that splitting it among co-retrieved memories is left open (WYVt, CjAL).
  • Section 3.3: separates the direct way, memory advantage as the reward (MemRL$^\Delta$, Vantage$^\Delta$), from Vantage's aim, utilities that reflect memory advantage without memory-free attempts; states what the re-attempt comparison gives within a query and where it stops, with the derivation in Appendix B.2, and leaves the rest to the test in Section 4.4 (CjAL W1/Q1, WYVt). A few phrases in the abstract, the introduction and related work are adjusted to match, and the conclusion is shortened.
  • Section 4.3: adds the retry-only control with utility ranking disabled, λ=0\lambda = 0, and compares it with MemRL at λ=0\lambda = 0 (WYVt, CjAL W3/Q2); details in Appendix D.6. The paragraph comparing η=0\eta = 0 and η=0.5\eta = 0.5 under the asymmetric re-attempt is retitled and no longer says that the first attempt suffices as the reference; it adds that η\eta and the asymmetric re-attempt both reduce the credit of unchanged successes (CjAL W1; WYVt Q/W1). The paragraph on the number of re-attempts after a failure now follows the comparison on re-attempting successes.
  • Section 4.4: the held-out advantage diagnostic on the 771 common held-out queries is now the main analysis of utility tracking (Table 6; CjAL Q1/W2/Q3). The training-process analysis, previously Figure 4, is now in Appendix E.1.
  • Appendix: derivation of the within-query comparison (B.2); retry-only control and MemRL at λ=0\lambda = 0 (D.6); paired tests and three training orders (D.7; CjAL W3/Q3); one bank shared by OS and DB and ALFWorld results by task type (D.8; sPGp); details and tercile checks of the held-out advantage diagnostic (E.2); credit splitting within an attempt as an open direction (G).

Figure and table numbers from Section 4.4 onward have changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.