- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Vantage modifies outcome-based memory-utility learning by immediately retrying a task once after failure, using the updated memory bank, while skipping retries after success. The paper motivates this schedule through memory advantage relative to a memory-free attempt and also evaluates an explicit advantage-adjusted reward in MemRL-delta. The deployed Vantage algorithm itself retains the outcome reward and MemRL utility update. Across coding, OS, database and embodied-planning tasks, five-epoch Vantage reports average frozen-bank training-query SR of 78.9 versus 72.5 for ten-epoch MemRL, and held-out SR of 67.2 versus 65.0, with about 40% fewer total attempts/stored entries.
Critical strengths
-
The scheduling change is simple and practically attractive. It allocates an immediate extra attempt to failures and avoids repeatedly storing almost identical successful trajectories. The benefit does not require updating the agent backbone or training an additional retriever.
-
The paper carefully distinguishes runtime multi-attempt success from one-attempt frozen-bank evaluation. Section 4.1 explicitly warns that last-epoch SR is not directly comparable, and Table 3 includes genuinely held-out queries. Appendix C.1 also specifies restoration of the initial environment and a fresh interaction before retrying, avoiding direct carry-over of environment progress.
-
The empirical evaluation includes four domains, a second backbone, explicit reward-weight comparisons, symmetric versus asymmetric retries, retry-depth controls and memory-count analyses. The observation that successful tasks succeed again in at least 98.3% of tested immediate retries supports the operational motivation for skipping them.
Critical weaknesses
-
The central derivation does not establish that Vantage learns memory advantage. Section 3.3 correctly observes that subtracting a common memory-free baseline cancels when comparing two hypothetical rewards. However, Algorithm 1 never updates utilities using the reward difference; it applies separate outcome-based updates to potentially different retrieved sets. Equality of reward differences does not imply equality of utility estimates or rankings across queries. Even for one memory present in both attempts, starting from the same utility and applying the same two samples, the updates satisfy Q2(eta) - Q2(0) = -alpha * eta * b * (2 - alpha), where b is the common memory-free outcome. The immediate retry therefore implements a selective sampling/update schedule, not an identified estimator of the memory-free effect. This can still be a useful method, but the paper should either substantiate the approximation or frame its contribution more narrowly.
-
The utility-versus-advantage analysis does not resolve this attribution issue. Section 4.4 labels a binary with-memory outcome y as an improvement when y exceeds the success fraction P from ten memory-free attempts. For every task with 0 < P < 1, this labels every success an improvement and every failure a degradation; the improvement share then reduces to success frequency on that subset. Appendix E.1 additionally removes unchanged outcomes and low-utility/no-memory cases, bins historical attempts using final utilities, and compares methods with different attempt schedules. The resulting plots are compatible with better success rates and sampling differences, but do not by themselves establish better estimation of memory advantage or individual-memory credit.
-
The held-out benefit needs uncertainty estimates and a more targeted mechanism control. The primary held-out gains over MemRL are 1.2–2.8 points across domains, with evaluation sets of 139–342 queries. The additional-backbone results are useful corroboration, but there are no reported paired intervals or task-order repetitions for the main comparison. Existing ablations change reward weighting, retry symmetry or retry depth; they do not isolate immediate utility reranking from access to the newly written failure reflection. A matched selective-retry control that keeps utility ranking fixed would help determine what specifically drives the gain.
Questions for authors
-
What assumptions connect the outcome-only updates in Algorithm 1 to the memory-free advantage defined in Section 3.2? Can the authors validate that connection on a common held-out query set, or explicitly characterize Vantage as a selective-retry schedule motivated by, rather than estimating, advantage?
-
At a matched attempt budget, how much improvement remains when the immediate failure retry is retained but utility reranking is disabled or frozen? This would separate the schedule/new-reflection effect from utility learning.
-
Can the authors report paired uncertainty for held-out SR and sensitivity to training-query order? For the advantage diagnostic, please use a common evaluation population and report expected success differences without conditioning away unchanged outcomes.
4 comments
Response to Reviewer CjAL (2/3)
Q1 (second part), W2 and Q3 (second part): Validation on a common held-out set
We agree that the original analysis (Section 4.4, now Appendix E.1) does not by itself establish better estimation of memory advantage: its results are also compatible with higher success rates and different sampling, since it is computed on each method's own training attempts and, with a fractional , counts every success as an improvement. It views the training process, asking how often attempts that retrieved memories of a given final utility ended above the memory-free expectation rather than below it; its bins leave out the lowest utilities (below 0.4), as the Q-value analysis of MemRL does (Zhang et al., 2026, Fig. 7). We therefore add the diagnostic the reviewer proposes.
Protocol. To have enough queries in each utility group, we pool the held-out queries of the four domains. Common evaluation population: every method is evaluated on the same 771 held-out queries (four domains), with its frozen bank and one greedy attempt per query. Expected success difference: the reference is the greedy memory-free outcome of the same query, deterministic under our greedy decoding, so a group's expected success difference is the mean of . No conditioning: the mean is taken over all queries in the group, so unchanged outcomes stay in it with , and the three groups together cover all 771 queries. Queries are grouped by the learned utility of the top-ranked retrieved memory.
Table 6 (revised manuscript): Mean by the utility of the top-ranked memory (number of queries)
Utility of top-ranked memory MemRL Vantage Vantage$^\Delta$ < 0.6 +0.146 (185) +0.080 (262) +0.049 (447) 0.6–0.8 +0.156 (135) +0.095 (201) +0.109 (156) ≥ 0.8 +0.042 (451) +0.136 (308) +0.238 (168) Difference, ≥ 0.8 minus < 0.6 (95% CI) −0.104 [−0.18, −0.03] +0.056 [−0.02, +0.13] +0.189 [+0.09, +0.29]
(CIs: normal approximation; the groups of a method are disjoint sets of queries.) Every group has a positive mean , so memory helps on average throughout; the question is whether higher utility marks a larger gain, which the last row tests. The pattern is the same with equal-size terciles of the 771 queries (257 each): the highest minus the lowest tercile is −0.129 for MemRL (95% CI [−0.20, −0.05]), +0.051 for Vantage and +0.150 for Vantage$^\Delta$, so Vantage's exceeds MemRL's by 0.18 (95% CI [+0.07, +0.29]); terciles formed within each domain give the same pattern.
- MemRL over-credits. For 58% of held-out queries its top-ranked memory has utility ≥ 0.8, yet these queries gain the least from memory: the high − low contrast is significantly negative. In its top bin (≥ 0.9, 340 queries), memory changes the outcome for 15% of them; conditioning on changed outcomes would hide this behind a 60% improvement share.
- The explicit reference removes this: Vantage$^\Delta$ shows a clearly increasing relation.
- Vantage lies in between: it has the comparison after a failure that MemRL lacks, but not the memory-free outcome that Vantage$^\Delta$ uses. The inversion disappears and the relation increases, less steeply than with the explicit reference; its own contrast is not significant, but it differs from MemRL's by 0.16 (95% CI [+0.05, +0.27]).
Given this validation, we keep the paper's claim that Vantage's utilities reflect memory advantage more closely than MemRL's, now distinguished from learning with it (Response 1). This diagnostic is now the main analysis of utility tracking in Section 4.4 (Table 6), with the original analysis moved to Appendix E.1; Section 3.1 and Appendix G state that splitting credit among the memories of one context remains open.
Response to Reviewer CjAL (3/3)
W3 (mechanism control) / Q2: Selective-retry control with utility ranking fixed, at a comparable budget
We ran this control, and no improvement remains: it falls below MemRL on held-out queries in all four domains. With greedy decoding, a re-attempt with an unchanged context reproduces the failure, so a successful re-attempt comes from a changed context: (i) utility reranking or (ii) the newly written failure reflection. The control keeps the asymmetric re-attempt and the reflection, which the re-attempt retrieves first by similarity, and disables utility ranking (). In our pipeline, disabled and frozen coincide: with utilities frozen at initialization, all candidates have equal utility, the within-pool z-score is zero (Appendix C.3), and ranking reduces to similarity. The control uses 31,557 attempts, against 30,185 for Vantage, so its lower SR is not due to a smaller budget. To isolate what the retry adds, we also set in MemRL, which does not re-attempt.
Table 17 (revised manuscript): Retry-only control, Frozen / Held-out SR (%)
Method Code OS DB Planning Avg. MemRL 53.4 / 41.8 57.1 / 53.3 95.0 / 85.6 84.6 / 79.3 72.5 / 65.0 MemRL, 53.1 / 40.9 55.1 / 46.0 93.6 / 83.5 85.9 / 76.4 71.9 / 61.7 Retry-only, 53.8 / 40.4 55.4 / 45.3 93.4 / 84.2 86.6 / 75.7 72.3 / 61.4 Vantage 62.3 / 43.0 64.3 / 56.0 95.3 / 87.8 93.7 / 82.1 78.9 / 67.2
Without utility ranking, the same schedule and reflections reach 61.4 held-out SR, against 65.0 for MemRL, and relative to MemRL with the retry changes average held-out SR by −0.3 (61.7 → 61.4) and frozen SR by +0.4, by at most 0.7 points on any domain. MemRL with runs 10 epochs (50,620 attempts), as MemRL does; stopped at a budget comparable to the control's, it reaches 71.7 frozen and 61.7 held-out SR, so there the retry changes them by +0.6 and −0.3. With utility ranking, the same retry adds +2.2 held-out and +6.4 frozen points (MemRL → Vantage), on every domain. At a comparable budget, the schedule and the new reflection thus add nothing on their own; the improvement comes from utility learning. With Table 5 (re-attempting successes as well: 64.3), the gain requires both utility learning and a schedule that does not re-credit successes.
W3 (uncertainty) / Q3 (first part): Paired uncertainty and order sensitivity
Paired uncertainty. All methods are evaluated on the same held-out queries, so we compare them per query, with exact McNemar tests on discordant pairs and paired bootstrap CIs (10,000 resamples, stratified by domain).
Table 18: Paired comparison on held-out queries, Vantage minus MemRL ( in points; / count the queries solved only by Vantage / only by MemRL)
Code OS DB Planning Pooled (771) (95% CI) +1.2 [−0.9, +3.2] +2.7 [−0.7, +6.7] +2.2 [−0.7, +5.8] +2.9 [0.0, +6.4] +1.9 [+0.5, +3.4] / 8 / 4 6 / 2 4 / 1 5 / 1 23 / 8 McNemar 0.388 0.289 0.375 0.219 0.011
Vantage is ahead in every domain, but the per-domain sets are small: a margin of 3–4 queries cannot reach even if every discordant pair favors one method. Pooled over the 771 queries, the difference is significant and its CI excludes zero. The pooled weights domains by size; the domain average in Table 3 is +2.2.
Order sensitivity. Due to the short rebuttal period, we ran this check on OS and DB: we repeated runtime learning with two additional shuffled training orders; within each order, all methods see the same sequence.
Table 19: Three training orders on OS and DB, mean ± std over the orders (%) and the paired difference within each order
OS Held-out OS Frozen DB Held-out DB Frozen MemRL 53.5 ± 0.4 58.4 ± 1.2 86.3 ± 1.3 94.3 ± 1.3 Vantage 56.9 ± 1.0 65.5 ± 1.4 88.0 ± 0.4 94.9 ± 0.3 Paired (wins) +3.4 ± 0.7 (3/3) +7.1 ± 0.6 (3/3) +1.7 ± 1.5 (2/3, 1 tie) +0.6 ± 1.1 (2/3)
Absolute SR varies with the order, as expected for runtime learning, but the comparison is stable: on held-out queries, Vantage is ahead in 5 of the 6 domain × order comparisons and ties in the sixth. DB is close to saturation (Frozen SR 93–95%), so its margins amount to a few queries. We will extend these repetitions to all four domains in a subsequent revision.
The revision adds the controls (Section 4.3, Appendix D.6), the paired tests and the order repetitions (Appendix D.7). We thank the reviewer again for suggesting these analyses.
Dear Reviewer CjAL,
Thank you again for your detailed review. We hope our rebuttal has addressed your concerns, and we would be happy to clarify anything further.
Best regards, The Authors
Response to Reviewer CjAL (1/3)
We thank the reviewer for the detailed analysis. This response addresses W1 and the first part of Q1; Responses 2 and 3 report the new experiments.
W1 / Q1 (first part): What assumptions connect the outcome-only updates of Algorithm 1 to the memory-free advantage?
Within a re-attempted query, one assumption suffices: both attempts are measured against the same memory-free outcome y∅ of the query (Section 3.2). Then (y(2)−y∅)−(y(1)−y∅)=y(2)−y(1). Algorithm 1 applies separate outcome-based updates to the two retrieved sets, but for a memory in the first set and not the second, and one in the second and not the first, the two updates move the gap between their utilities toward y(2)−y(1), the difference between the memory advantages of the two contexts, for every η (Section 3.3, Appendix B.2).
Beyond it, we agree with the reviewer's derivation. For a memory present in both attempts, its utility depends on η through b, as Q2(η)−Q2(0)=−αηb(2−α) shows. Across queries, equal reward differences do not imply equal utilities or rankings, since memories are retrieved for queries with different memory-free outcomes. No further assumption we could justify closes this gap, so the immediate retry is a selective sampling and update schedule, not an identified estimator of the memory-free effect. Accordingly, the revision no longer says that the re-attempt approximates memory advantage. Section 3.3 now distinguishes learning from an explicit advantage signal, as MemRL$^\Delta$ and Vantage$^\Delta$ do, from learning utilities that reflect memory advantage, which Vantage aims at, and states what holds within a re-attempted query.
Validation on a common held-out query set. We ran it with the protocol the reviewer specifies in Q3 (Response 2, new Table 6). On the same 771 held-out queries, with unchanged outcomes kept, the high-minus-low utility contrast in mean d=y−y∅ is −0.104 for MemRL (its highest-utility memories bring the least gain), +0.056 for Vantage, and +0.189 for Vantage$^\Delta$, which uses memory-free attempts. Vantage's own contrast is not significant (95% CI [−0.02, +0.13]), but it exceeds MemRL's by 0.16 (95% CI [+0.05, +0.27]): without memory-free attempts, its utilities reflect memory advantage more closely than MemRL's.