acceptodds
Under review as a conference paper at ICLR 2027

Experience Is Not Evidence: Coverage-Calibrated Memory Reuse for Tool-Using Agents

Abstract

Tool-using agents increasingly reuse their own past experience, yet a memory that looks relevant can quietly damage the trajectory it was meant to help. Having experience is not the same as having evidence that it transfers. We therefore replace the retrieval objective with an interventional one: the transfer gain of a request–memory pair, measured by running the same request with and without that memory, and generalized across pairs by tool-call structure rather than text similarity. What may be claimed about this gain is determined by coverage. Where practice contains no comparable structure, the sign of transfer is unidentifiable, making forced use-or-discard decisions provably lossy and motivating a genuinely resolving third action. We accordingly distinguish an average-value guarantee over candidate sets from a marginal per-pair certificate. The average-value guarantee opens on fresh confirmatory practice data and, with the policy frozen in advance, on a fresh fixed-design grid over held-out deployment domains. In contrast, independent validation with the representation frozen produces no pair-level openings and defers throughout. On a controlled benchmark designed so that the harmful candidate is the more textually similar one, only structurally incompatible memories are reliably harmful. Similarity and an LLM judge fail to identify them, whereas transfer-based gating incurs no observed harmful use on any tested model family and matches oracle utility on two of the three full-benchmark executors. Reliable harm appears only in our authored competing memories. A preregistered organic pilot produced isolated negative outcomes that failed its prespecified reliability gate and did not survive replication, bounding the scope of the result and reinforcing the need to measure transfer rather than infer it from a memory label.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
tool-using agents, agent memory, experience reuse, transfer estimation, coverage calibration, conformal prediction, abstention, safe policy improvement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B6zaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the issue of memory reuse in tool-calling agents. It argues that high textual similarity between a retrieved memory and the current request does not mean the memory is actually useful. Relying on it can lead to negative consequences, such as the agent blindly imitating incorrect actions. As for the method, the paper proposes a decision framework based on "coverage calibration." It introduces statistical guarantees (the average-value guarantee TRACE-AVG and the per-pair certificate TRACE-CERT) along with a crucial third action: "deferral". For the experiments, the authors constructed a controlled benchmark to evaluate their approach.

Critical strengths

The research problem is novel and well-targeted: The paper challenges the prevailing assumption in agent memory reuse that “textual relevance implies safety,” and insightfully points out that structurally incompatible memories can lead to negative transfer even when they are highly similar in text. This insight has important theoretical value and practical significance for building reliable and safe tool-using agents.

Critical weaknesses
  1. Writing and Presentation. The paper lacks an intuitive pipeline figure for the core method, which makes the complex mechanism harder for readers to follow.

  2. Limitations in Benchmark and Experiments. Although the authors acknowledge some limitations, the benchmark has a fundamental defect when evaluating agent memory harm, which limits the generality of its conclusions: reliable negative transfer comes only from authored adversarial memories and was not reproduced in organic settings. The benchmark is therefore a diagnostic stress test rather than a reflection of natural distributions.

Questions for authors
  1. Organic experiments. The paper honestly reports that no reliable negative transfer was found in organic retrieval, and that all reliable harmful effects come from authored memories. Does this mean the method’s practical value lies mainly in adversarial/malicious memory injection, rather than naturally occurring memory quality differences? If so, should the paper define its applicable scope more explicitly?  
  2. Practicality of TRACE-CERT. Experiments show TRACE-CERT still defers 100% of the time after 46,080 validation episodes. What data scale would be needed for this mechanism to actually “open” in a realistic setting? Does this mean per-pair certification is practically infeasible due to cost? Are there more economical paths worth exploring?  
  3. Domain sensitivity. The evaluation covers only 8 domains, and the deployment domains are near-replicates of the calibration domains. Can the authors quantify this heterogeneity, and clarify how it limits generalization?  
  4. Single-step limitation. The current decision unit is limited to single-step intervention, but memory in practice may span multi-step, long-horizon content. Can the authors clarify whether the framework and impossibility theorem extend to such settings?
5 comments

Thank you for the review, and for identifying the paper's core insight: textual relevance does not imply safe reuse, and a structurally incompatible memory can harm even when it is the most similar one. Manuscript changes are in red.

The new experiment directly addresses the concern that negative transfer is an artifact of authored memories (Appendix Y.1, Table 23).

We repeated the controlled mechanism test using the executor's own successful past executions as memory content. Each memory is an automatically serialized record of a successful execution, preserving the original request, tool call, arguments and success outcome without author-written procedural advice.

Design.

  • 96 target families across 16 tool domains, from 64 genuine source successes.
  • R=3R=3 per arm, 1,152 new episodes.
  • The design's SHA-256 was fixed before any target episode, and all results are reported.

Results.

  • Prespecified test. The same-object, conflicting-intent trace yields a negative effect: −0.021-0.021, domain-clustered 95% CI [−0.045,−0.003][-0.045,-0.003], with positive uptake of the recorded tool.
  • Larger effect. An opposite-action trace from another object produces −0.118-0.118, CI [−0.146,−0.087][-0.146,-0.087], and −0.354-0.354 on blind requests, negative in 13/16 domains.
  • Process evidence. Wrong-tool execution rises from 48/9648/96 to 82/9682/96 blind episodes, always on the target's own object.
  • Mirror. The same-action trace improves utility (+0.385+0.385).
  • Robustness. A fresh no-memory re-run in the same period (288288 episodes) leaves both effects negative with comparable magnitudes and intervals excluding zero (−0.024-0.024, [−0.049,−0.007][-0.049,-0.007]; −0.122-0.122, [−0.149,−0.090][-0.149,-0.090]), and both stay negative in 16/16 leave-one-domain-out refits.

The negative-transfer mechanism is therefore not restricted to instructions authored by the researchers. Crucially, each memory records a genuinely successful past action. Its harmfulness arises when that action is transferred to a request with a different intent.

This resolves the authored-content explanation within the controlled benchmark. The remaining limitation is the prevalence and impact of such conflicts under unconstrained organic retrieval, which we do not claim to establish. A token-overlap retriever, for instance, selects same-object traces and not the high-leverage other-object ones.

What remains is external validity, and it should be assessed separately from soundness. Two propositions are at issue:

  • (I) Internal validity. Are the measured intervention effects valid within the controlled design?
  • (E) External validity. Does their frequency represent naturally accumulated memory?

Every effect is a paired execution on independently reset replicas, scored against ground truth with no LLM judge. The authored effects (distractor −0.247-0.247, [−0.34,−0.17][-0.34,-0.17]; corrupted −0.420-0.420, [−0.50,−0.33][-0.50,-0.33]) are robust to the resampling unit. They keep their sign in 8/8 leave-one-domain-out refits and replicate on three further executors and on BFCL's public schemas. The experiment measures the effect of each specified memory; it does not isolate a causal effect of structural incompatibility from every other property of the memory.

The unresolved question is (E). It does not, by itself, invalidate the measured intervention effects (I) or the theoretical results under their stated assumptions. The review's concerns are concerns of generalization rather than of a specific theorem or estimator, and we respectfully ask that the soundness assessment distinguish these issues.

Presentation: pipeline figure.

The revised PDF opens Section 4 with a one-row pipeline figure (new Figure 2): paired rollouts → structural cell z=h(x,m,T)z=h(x,m,\mathcal T) → evidence-scaled interval τ^(z)±λs(z)\widehat\tau(z)\pm\lambda s(z) (a coverage set only at λ≥1\lambda\ge1) → use if the lower end clears 0, no memory if the upper end is below 0, abstain and flag otherwise. Next to it, Table 1 lists each variant's claim, assumption and experimental outcome.

Q1. Is the practical value mainly about adversarial memory injection?

No. The demonstrated scope is memory stores that contain competing procedures for the same object. That configuration has three sources, and two of them are benign:

  • Changed intent. In the authored benchmark, the lexical distractor is an authored memory describing a previously successful sibling action on the same object ("archive" before, "read" now). In Appendix Y.1, the same situation arises from an automatically serialized record of an actual successful execution.
  • Logging or summarization error. The corrupted memory states one action in its body and plays another (τ=−0.42\tau=-0.42).
  • Poisoning. An adversary inserts a same-object memory that points to a destructive tool (Ethics Statement).

The mechanism is therefore not specific to adversaries. What remains open is how often the benign sources produce harm in naturally accumulated memory. Organic retrieval settings produced isolated harm (one AppWorld task fell from 0.8000.800 to 0.1000.100) that did not replicate (Appendix X). The new trace-origin experiment shows that the benign changed-intent source suffices to produce reliable harm, from the executor's own logged successes (Appendix Y.1). The introduction's new Scope paragraph states this boundary.

Q2. TRACE-CERT defers 100%. What scale would open it, and are there cheaper paths?

The answer is a closed-form sample-size condition. With shortlist size KK, joint error α\alpha and q=α/(2K)q=\alpha/(2K):

  • a cell needs at least ⌈(1−q)/q⌉\lceil(1-q)/q\rceil calibration residuals (3939 at K=2K{=}2, 1919 at K=1K{=}1);
  • a positive lower bound additionally needs R>2log⁡(4K/α)(τ^−β^)2R>\frac{2\log(4K/\alpha)}{(\widehat\tau-\widehat\beta)^{2}} rollouts per arm per pair.

In the post-hoc K=1K{=}1 diagnostic (Appendix T), three deployment cells have margin τ^−β^≈0.25\widehat\tau-\widehat\beta\approx0.25 and need R≳118R\gtrsim118, against 9292 collected. The fourth has margin 0.0220.022 and needs about 15,00015{,}000. Part of the gap is budget and part is within-cell heterogeneity, which no rollout count removes. Because K=1K{=}1 was not frozen, this is a calculation, not a forecast.

Cheaper paths follow from the formula:

  • Fewer simultaneous candidates. A smaller KK halves the residual requirement.
  • Coarser cells. These raise ncaln_{\mathrm{cal}} per cell.
  • A variance-adaptive rollout radius. 154154 of 320320 validation pairs have zero within-pair variance. The two-sided empirical-Bernstein radius is 0.2950.295 against Hoeffding's 0.3090.309 at R=92R=92, and 0.190.19 against 0.240.24 at R=150R=150.

These calculations identify the rollout budget required to overcome the within-pair concentration term, conditional on a positive residual-envelope margin. They do not by themselves establish that a valid certificate will open. That also requires enough calibration residuals per cell, residual transport between practice and deployment, and a margin that survives re-estimation (coarser cells raise ncaln_{\mathrm{cal}} but can raise within-cell heterogeneity). The paper's finding is an empirical separation under the evaluated budgets: the average-value claim was attained and the pair-level claim was not. This is not a universal sample-complexity lower bound. That gap is the empirical content of the AVG/CERT distinction, not an incidental failure.

Q3. Only 8 domains, near-replicates. Quantify the heterogeneity.

  • Between-domain share of pair-level variance: 0.1%0.1\% to 9.4%9.4\% across supportive, structural-match, distractor and corrupted memories, in both domain sets.
  • Magnitude varies, sign does not. The distractor's domain means range from −0.14-0.14 to −0.47-0.47 (practice) and from −0.11-0.11 to −0.44-0.44 (held out). The sign is negative in 8/8 domains of each set and preserved in 8/8 leave-one-domain-out refits (Appendix V).
  • The held-out-domain bound is not carried by one domain. Its per-domain gain is positive in 8/8, from +0.104+0.104 to +0.139+0.139.

Scope: the domains share one read/write skeleton, so the evidence covers new objects and tool names within it, not new tool graphs. The +0.032+0.032 bound is for the specified fixed-design grid. The 0/1280/128 harmful uses there is an observation; its binomial upper bound (2.3%2.3\%) holds only under an i.i.d. reading of a fixed-design grid, and the revision labels it as such.

Q4. Single-step decision unit. Do the framework and the theorem extend?

  1. The estimand already covers whole trajectories. τ(x,m)\tau(x,m) is a difference in full-episode return; the memory is injected once at episode start. Theorem 1, Proposition 1 and the conformal certificate hold for any bounded episode return, at any horizon.
  2. Theorem 2 is horizon-free. Its two environments agree on every pre-deployment observation and differ only in the sign of the effect on an unpracticed region. Trajectory length does not enter the argument.
  3. What does not extend is mid-trajectory revision, i.e. choosing which memory to consult at step tt given the history. That is a sequential policy problem, which the revised Conclusion names as the main open direction.

The framework has also been run on multi-step environments: AppWorld (Appendix R; supportive workflow 0.462→0.5070.462\to0.507) and a non-ceiling τ\tau-bench-derived retail environment (Appendix W). The estimand was well defined in both. What did not appear was a reliable harmful class (organic distractors +0.251+0.251 on retail). This is why the harmful regime is stated for single-call competing memories, while the estimand and the identifiability result are not restricted to them.

Summary for the soundness assessment. The paired estimand is measured on reset replicas with ground-truth scoring. The theorems hold under their stated assumptions, and the review's concerns address generalization rather than any specific theorem or estimator. The confirmatory bounds were computed on folds frozen before collection. The limitation is the external reach of the harmful regime, and the paper now states it at the front.

Where each concern is addressed in the revised PDF (changes in red)

ConcernRevised PDF
Pipeline figureFigure 2 and Table 1, start of Section 4
Benchmark validityAbstract; Introduction, Scope paragraph; Section 6, organic paragraph; new Appendix Y.1 (trace-origin memories)
Q1 adversarial vs. naturalIntroduction, Scope paragraph; Ethics Statement
Q2 TRACE-CERT scaleSection 6 sample-size sentence; Appendix T (Table 17)
Q3 domain heterogeneityAppendix V, new heterogeneity paragraph
Q4 single-stepConclusion; Appendices R and W
Reviewer_yp89OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when a tool-using agent should reuse a retrieved memory. It defines memory utility as the change in full-trajectory outcome between executions with and without that memory, and estimates this effect using structural features of request-memory pairs. The paper distinguishes an average-value selector, TRACE-AVG, from a more conservative pair-level certificate, TRACE-CERT, and analyzes the limitations caused by missing coverage. Controlled experiments show that structurally incompatible memories can be harmful despite high textual similarity. The average-value guarantee becomes informative on fresh confirmatory data, while independent validation of the pair-level certificate leads to deferral throughout. Organic-memory pilots do not establish a reproducible harmful-transfer regime.

Critical strengths
  • The paper makes a useful distinction between average memory usefulness and pair-level certification. Defining utility through executions with and without a memory clarifies what safe reuse should establish and makes the limits of the guarantees informative.

  • The confirmatory experiments and transparent negative results are valuable. The paper explicitly reports that pair-level openings do not replicate and that organic-memory pilots do not establish reliable harm. This careful separation of positive average-value evidence from unsuccessful certification supports acceptance as a methodological and diagnostic contribution.

Critical weaknesses
  • The practical scope is narrow. Reliable harm is established with authored competing memories in single-tool-call episodes, while the organic AppWorld and BFCL pilots do not demonstrate a reproducible harmful-transfer regime. In addition, the deployed fallback does not implement Corollary 2’s assumption that deferral resolves the current request, and active repair does not open pair-level certificates. The evidence therefore supports average-value selection in the controlled setting more strongly than the broader reuse and repair mechanism.
  • The contribution of coverage calibration is difficult to separate from the hand-designed representation. Appendix C uses specific memory phrases and structural compatibility rules, whereas the adapted baselines in Appendix Q receive only four surface or metadata features. The point-estimate ablation in Appendix U is useful, but matched-feature rule-based and supervised controls are still needed to determine how much of the result follows from the encoder itself.
  • The paper is difficult to follow because many closely related guarantees, method variants, and experimental settings are introduced before their relationships and practical implications are clear. In particular, the distinction between TRACE-AVG and TRACE-CERT, and which theoretical claims are supported by the experiments, should be explained earlier. It is suggested to reorganize the presentation around the main research question and findings, with a concise comparison of each variant’s assumptions, guarantees, and empirical outcomes.
Questions for authors

Please refer to the weaknesses.

3 comments

Thank you for a precise review, and for recognizing the confirmatory experiments and negative results as a methodological and diagnostic contribution. Your second weakness asks for the control that decides how that contribution should be read. We ran it, and it sharpens rather than shrinks the paper's claim. The matched-feature and population-refit diagnostics reuse cached outcomes without additional model calls and are labelled exploratory. Separately, the preregistered trace-origin experiment adds 1,152 episodes, with 288 additional no-memory episodes for a post-hoc robustness check. All are in the updated supplementary. Manuscript changes are in red.

W2. How much of the result follows from the hand-designed encoder?

Design.

  • Every rule receives the same encoder h(x,m,T)h(x,m,\mathcal T), one-hot over its sub-features. The GBM additionally receives surface features, so it has strictly more information than TRACE.
  • All rules share the practice fold, the 8-candidate sets, the executor and the held-out tasks.
  • Default thresholds are selected on practice with domain-grouped cross-fitting, and ties are broken at random.
  • Candidates are then removed without reading any outcome: random availability, top-kk similarity retrieval, and similarity-biased sampling.
  • We report each rule at its default operating point and on its empirical zero-harm frontier, its own knob swept on the evaluation data.

The Appendix Q proxies, which see four surface features, are now labelled surface-feature controls. They make no claim about the original methods.

Results on 96 held-out-domain tasks (utility / expected harmful uses; never-memory 0.8260.826):

Rule, same hhauthored setp=0.5p{=}0.5 defaultp=0.5p{=}0.5 frontiertop-2 defaulttop-2 frontier
deterministic rule on hh0.833 / 8.90.853 / 6.9n/a0.830 / 6.1n/a
logistic P(help∣h)P(\text{help}\mid h)1.000 / 00.912 / 00.9130.826 / 00.826
GBM on hh + surface1.000 / 00.985 / 0.60.9090.984 / 1.70.827
point estimate1.000 / 00.984 / 1.00.9580.979 / 2.00.826
TRACE-AVG1.000 / 00.912 / 00.9580.826 / 00.826

What this establishes.

  1. The encoder alone is not the result; the encoder plus measured transfer is. A deterministic rule on the same hh takes 8.98.9 harmful uses on the authored set. On blind requests the supportive and distractor memories share one cell, and only the measured cell effect tells a rule to avoid it. Every rule fitted to paired practice reaches oracle utility. Measured transfer also absorbs encoder errors: on 96 fresh phrasings, where the intent lexicon misreads 47%47\% of non-blind requests, the deterministic rule takes 55 to 1313 harmful uses and every fitted rule takes 00.
  2. Default operating points differ. "Use if τ^>0\widehat\tau>0" takes harmful uses in all 24 shifted settings (18 distinct tasks); TRACE-AVG takes none. Three cells have E[τ]≥0\mathbb E[\tau]\ge0 but Pr⁡(τ<0)>0\Pr(\tau<0)>0, and the authored candidate set never tests whether such a cell is admissible on its own.
  3. Attainable envelopes coincide. Swept post hoc, a point estimate at t≈0.12t\approx0.12 traces TRACE-AVG's envelope in 21/21 settings. The logistic classifier is never higher. The GBM exceeds it only on fresh phrasings of its own training domains. This is a statement about what is attainable on observed data, not a validated selection.
  4. Evidence is population-specific, and the framework says so. On a new memory population (executor-logged traces, Appendix Y.1), two trace types share one blind cell. Refitting on that population with every rule sharing the same estimate, only the radius differs. Over 500 training splits, the point estimate uses the cell in 61% of splits; TRACE-AVG at λ=0.25\lambda=0.25 abstains in 97%. This is avoidance of harm rather than a utility gain, and the analysis is exploratory and cache-only.

The control therefore separates two contributions that the submission had blended.

  • The representation plus measured transfer produces the decision quality. This is the paper's estimand, τ(x,m)\tau(x,m), at work.
  • The evidence semantics decide which operating point may be fixed before evaluation and what may be claimed for it. The confirmatory selector fixed λ=0.25\lambda=0.25, its average-value bound held on an untouched held-out grid (+0.032+0.032), and the scale has a meaning: λ≥1\lambda\ge1 is a coverage set, λ<1\lambda<1 is an average-value policy only. A post-hoc envelope carries no such statement.

Proposition 1 applies to any frozen policy class, so the claim is not tied to our estimator. That is a design property of the framework, not a concession.

W3. Presentation

The revised PDF is organized around one question: when may an agent act on a memory's measured transfer, and how strongly?

  1. Pipeline figure at the start of Section 4 (new Figure 2): paired rollouts → structural cells → interval τ^±λs\widehat\tau\pm\lambda s → use / no memory / abstain-and-flag.
  2. Claim–assumption–evidence table opening Section 4 (new Table 1); full assumptions remain in Appendix B:
VariantClaimNeedsExperiments
point estimate τ^>t\widehat\tau>tnonenonesame post-hoc envelope as TRACE-AVG; at t=0t=0, harmful uses under every outcome-blind shift
TRACE-AVG (λ=0.25\lambda=0.25)average value JJfixed-design targetselection-adjusted confirmatory LCB +0.039+0.039; frozen-policy held-out grid LCB +0.032+0.032
TRACE-CERT (λ≥1\lambda\ge1)marginal pair-level coverage (over a random deployment pair)residual transportdoes not open: 0/320/32 under frozen validation
active acquisitionsurrogate greedy boundfrozen roundnarrows intervals; opens no certificate
  1. Experiment paragraphs name the result they test, matching the roadmap in Appendix D.
  2. Layout. Figure 1 no longer interrupts the contribution list.

W1. Scope; Corollary 2; active repair

  • Scope. Reliable harm is established in controlled single-call episodes: with authored competing memories, and, in a new preregistered follow-up, with the executor's own logged past successes (an opposite-action trace costs 0.1180.118, [0.087,0.146][0.087,0.146]; Appendix Y.1). The organic retrieval pilots do not establish a reproducible harmful regime. The introduction's new Scope paragraph and Table 1 state this.
  • Corollary 2 prices a defer action that resolves the current request. The deployed rule falls back to no memory and flags the cell. Section 4 names it "abstain and flag" and cites Corollary 2 as motivation for the action set only. Theorem 2 continues to characterize the identifiability limit of the current-request use/discard decision, including our no-memory fallback. It does not establish the optimality of abstain-and-flag; that would require a separate sequential acquisition analysis.
  • Active repair is a principle with a surrogate guarantee. Table 1 records that it narrowed intervals and opened no pair-level certificate.

What the paper establishes, after the control you asked for

  1. Decision object. The paired, full-trajectory transfer of one memory is the decision object. On a benchmark built so that the more similar candidate is the harmful one, similarity and an LLM judge rank harmful candidates near chance. A rule on the structure alone, without measured transfer, still takes harmful uses.
  2. Admissibility is a separate question from mean value. Cells with E[τ]≥0\mathbb E[\tau]\ge0 can hold harmful pairs. Candidate sets that always contain a better option never test admissibility, so the natural default is unsafe under shift.
  3. Claims have different prices. An average-value claim holds under a selection-adjusted confirmatory bound and again on an independent evaluation of the frozen policy. A pair-level claim does not open under a frozen 46,080-episode validation, and the sample-size condition R>2log⁡(4K/α)/(τ^−β^)2R>2\log(4K/\alpha)/(\widehat\tau-\widehat\beta)^2 explains why.

Each finding holds whichever estimator one prefers; that is what makes them methodological. The matched-feature control and the reorganized presentation address the fairness and clarity concerns as you posed them. Thank you for the review that prompted both.

Where each concern is addressed in the revised PDF (changes in red)

ConcernRevised PDF
W1 scope, Corollary 2, active repairIntroduction, Scope paragraph; Section 4, "abstain and flag"; Table 1
W2 encoder vs. decision ruleTable 4; Appendix U.1 (Tables 20 and 21); Appendix Q relabelled
W3 presentationFigure 2 and Table 1 at the start of Section 4; Figure 1 moved; Appendix D

Dear Reviewer yp89,

Thank you again for your constructive feedback. Following your suggestions, we have conducted matched-feature comparisons, added new empirical evidence, and reorganized the presentation to clarify the paper's methodological contribution.

We would appreciate hearing whether these revisions sufficiently address your concerns and whether any important issues remain.

Thank you for your time and consideration.

Reviewer_XTJCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how tool-using agents should decide whether a retrieved memory is actually useful for a new task. The authors argue that memory relevance and past success do not necessarily imply positive transfer, and instead formulate memory utility as the incremental effect of using a specific memory on the final task outcome. They propose a coverage-calibrated framework that uses structural information about tool calls to estimate transfer effects and determine when there is enough evidence to reuse a memory. The paper distinguishes between average-value guarantees and stronger per-pair certification, and provides theoretical results motivating abstention when structural coverage is insufficient. Experiments on a controlled benchmark show that structurally incompatible memories can cause negative transfer even when they are textually similar to the current task. The average-value method achieves strong results, while pair-level certification remains overly conservative and the harmful-transfer pattern does not reliably reproduce with organic memories.

Critical strengths
  1. The paper raises an interesting question about what makes a memory useful. I agree with the motivation that semantic similarity and historical success are not necessarily good indicators of whether a memory will help the current task. Formulating memory utility in terms of its incremental effect on task performance is a meaningful perspective.

  2. The theoretical formulation is relatively careful. I appreciate that the authors distinguish between average-value guarantees and per-pair certification, rather than treating all uncertainty estimates as equivalent. The discussion of coverage and identifiability also helps clarify when a decision can be supported by the available evidence.

  3. The controlled benchmark is thoughtfully designed to separate textual similarity from structural compatibility. The results clearly demonstrate that a more similar memory can be harmful when it suggests an incompatible tool-use procedure.

  4. I appreciate the transparency of the experimental analysis. The authors report both successful and unsuccessful validation results, including the failure of independently validated pair-level certification and the lack of reliable negative transfer in the organic-memory experiments. These findings are important for understanding the actual scope of the contribution.

Critical weaknesses
  1. My main concern is the practical relevance of the harmful-transfer phenomenon studied in the paper. The strongest results come from manually constructed memories, while the organic-memory experiments do not show reliably reproducible negative transfer. This raises the question of how often the proposed method would provide meaningful benefits in realistic agent memory systems.

  2. The pair-level certification procedure appears difficult to use in practice. Even after a large validation effort involving 46,080 episodes, it does not certify any memory use in the tested cases. While this is an interesting negative result, it limits the practical value of one of the paper's main proposed mechanisms.

  3. The benefit of uncertainty calibration over simpler transfer prediction is not entirely clear. In the main experiment, the point-estimate-only method achieves the same utility as TRACE-AVG, with no observed harmful memory uses. Although the authors provide additional experiments under candidate-set changes, I would like to better understand when the more complicated calibration procedure is necessary.

  4. The approach relies on structural representations of request-memory pairs and substantial offline paired evaluations. It is unclear how well this would scale to more complex agent tasks with diverse tools and less clearly defined action structures. Several theoretical ingredients also build on existing work in off-policy evaluation and selective prediction, so the methodological novelty could be better articulated.

Questions for authors

see weaknesses above

4 comments

Thank you for a careful review that engages with the paper's central distinction. We organize the response around three statements the paper establishes, then show how each of your four questions resolves against them. The matched-feature and population-refit diagnostics reuse cached outcomes without additional model calls and are labelled exploratory. Separately, the preregistered trace-origin experiment adds 1,152 episodes, with 288 additional no-memory episodes for a post-hoc robustness check. All are in the updated supplementary. Manuscript changes are in red.

Three statements the paper establishes

  • (S1) Average value and pair-level transfer are different estimands. E[τ(X,M)]>0\mathbb E[\tau(X,M)]>0 does not imply Pr⁡(τ(X,M)<0)=0\Pr(\tau(X,M)<0)=0. Our data contain exactly such cells: the self-contradicting "both playbooks" memory has mean +0.139+0.139, yet 4%4\% of its pairs are harmful. A guarantee of the form J(π^)≥0J(\widehat\pi)\ge0 and a pair-level coverage statement are therefore different statistical claims. The latter is marginal over a random deployment pair, Pr⁡(X,M)∼P(∣τ(X,M)−τ^(Z)∣≤r(Z))≥1−α\Pr_{(X,M)\sim P}(|\tau(X,M)-\widehat\tau(Z)|\le r(Z))\ge1-\alpha, not conditional on a fixed (x,m)(x,m).
  • (S2) Without structural coverage, the transfer sign is not identifiable. If QZ(A)=0Q_Z(A)=0 and PZ(A)≥αP_Z(A)\ge\alpha, then every mandatory binary gate has minimax regret at least αΔ/2\alpha\Delta/2 (Theorem 2).
  • (S3) At the evaluated evidence budget, average-value validation succeeds whereas pair-level certification does not open. The average-value claim is supported by a prespecified, selection-adjusted confirmatory result (+0.039+0.039), followed by an independent evaluation of the frozen policy on a held-out-domain fixed grid (+0.032+0.032). The pair-level certificate opens 0/320/32 under a frozen 46,080-episode validation.

W1. Practical relevance of the harmful-transfer phenomenon

Statement. The benchmark identifies a failure mechanism under controlled intervention: a memory that is more similar to the request can be harmful because its prescribed action is structurally incompatible with the task.

Evidence.

  • Effects are paired executions on reset replicas, scored against ground truth: distractor −0.247-0.247, corrupted −0.420-0.420.
  • Similarity cannot reject the distractor without also rejecting the supportive memory (equal mean overlap, 0.0610.061 each).
  • The configuration is a natural one for tool APIs: read/write pairs over one resource (get/delete, read/archive) are standard. On BFCL's public schemas, the effect reproduces under injection (distractor −0.087-0.087).
  • Scale amplifies rather than removes the effect: within Qwen3 the penalty grows from −0.35-0.35 to −0.67-0.67 between 8B and 32B.

New preregistered evidence: authored text is not necessary (Appendix Y.1, Table 23). Every memory in this follow-up is an automatically serialized record of the executor's own successful execution, preserving the original request, tool call, arguments and success outcome without author-written procedural advice. Design: 96 target families, 64 genuine source successes, R=3R=3, 1,152 new episodes; the SHA-256 of the design was fixed before any target episode, and all results are reported. A trace recording the opposite action on another object lowers utility by 0.1180.118 (domain-clustered 95% CI [−0.146,−0.087][-0.146,-0.087]), and by 0.3540.354 on blind requests ([−0.438,−0.260][-0.438,-0.260], negative in 13/16 domains). The wrong tool is called in 82/9682/96 blind episodes against 48/9648/96 without memory, always on the target's own object. Its mirror, the same action on another object, gains +0.385+0.385. The preregistered primary contrast, the same-object opposite-intent trace, also meets its criterion (−0.021-0.021, [−0.045,−0.003][-0.045,-0.003], positive uptake), but it is small. The sign follows the action the trace records, and a fresh no-memory re-run (288 episodes) leaves both effects negative with comparable magnitudes and intervals excluding zero (−0.024-0.024, −0.122-0.122), negative in 16/16 leave-one-domain-out refits.

Scope. How often the mechanism occurs in naturally accumulated memory is not estimated. Across five organic evaluations, no reproducible harmful-transfer regime emerged. Isolated negative outcomes, including one AppWorld task falling from 0.8000.800 to 0.1000.100, did not survive independent replication (Appendix X). The revised introduction states this boundary.

Implication. The controlled failure mechanism is scientifically meaningful regardless of its natural prevalence. Its deployment-level benefit depends on the frequency and severity of harmful transfer relative to the costs of abstention and evidence acquisition. On the cost side, the selector needs no request-time scorer, and it abstains or matches the baseline where harm is absent.

W3. When is calibration necessary, compared with a point estimate?

The question separates into three claims that must not be conflated: mathematical justification, necessity of a particular rule, and empirical superiority.

(a) Empirical superiority: none at matched risk, and we report this. Every rule reads the same encoder hh; candidates are removed without reading any outcome; ties are broken at random. Results on 96 held-out-domain tasks (utility / expected harmful uses; frontier = best utility with zero harmful uses, knob swept on the evaluation data):

Rule (same hh)p=0.5p{=}0.5 defaultp=0.5p{=}0.5 frontiertop-2 defaulttop-2 frontier
logistic P(help∣h)P(\text{help}\mid h)0.912 / 00.9130.826 / 00.826
GBM on hh + surface0.985 / 0.60.9090.984 / 1.70.827
point estimate0.984 / 1.00.9580.979 / 2.00.826
TRACE-AVG0.912 / 00.9580.826 / 00.826

A point estimate thresholded at t≈0.12t\approx0.12 traces the same envelope as TRACE-AVG in 21/21 settings (Table 4, Appendix U.1). That envelope, sup⁡θ:H^(πθ)=0J^(πθ)\sup_{\theta:\widehat H(\pi_\theta)=0}\widehat J(\pi_\theta), is chosen on observed outcomes, so it is a statement of expressiveness, not a validated selection.

(b) Mathematical justification: it comes from the protocol, not from the radius. Proposition 1 holds for any finite policy class frozen on data disjoint from the confirmatory fold. It applies equally to Πpoint={πt: πt(z)=1{τ^(z)>t}, t∈T}∪{π0},\Pi_{\mathrm{point}}=\{\pi_t:\ \pi_t(z)=\mathbf 1\{\widehat\tau(z)>t\},\ t\in\mathcal T\}\cup\{\pi_0\}, which would earn the same type of bound: with probability at least 1−δ1-\delta, JQC(π^)≥−ϵevalCJ_{Q_C}(\widehat\pi)\ge-\epsilon^C_{\mathrm{eval}} on the practice target, and JPC(π^)≥−ϵevalC−ϵrepCJ_{P_C}(\widehat\pi)\ge-\epsilon^C_{\mathrm{eval}}-\epsilon^C_{\mathrm{rep}} on a deployment target. A data-dependent policy selection must account for selection in its statistical bound. Proposition 1 does so uniformly over a prespecified finite policy class on the confirmatory fold, and the separate frozen-policy evaluation on held-out domains adds independent evidence. The rule's form is not what earns the bound; the selection-adjusted protocol is. TRACE-AVG is the instance we froze and validated. The bound was +0.039+0.039 on the confirmatory practice target and +0.032+0.032 on an untouched held-out-domain fixed-design grid, where the frozen single policy needs no transport term.

(c) What the radius contributes is a scale, not a guarantee. The radius λs(z)\lambda s(z) is measured in units of the pair-level spread inside each cell. At λ≥1\lambda\ge1 it is a coverage set, giving the marginal pair-level statement of Proposition 2; at λ<1\lambda<1 it is not. One parameter therefore indexes both claim strengths. A utility threshold tt has no such reading.

The default comparison also shows why an unvalidated rule is unsafe, independently of its form. "Use if τ^>0\widehat\tau>0" takes harmful uses in all 24 shifted settings (18 distinct tasks). Three cells have E[τ]≥0\mathbb E[\tau]\ge0 but Pr⁡(τ<0)>0\Pr(\tau<0)>0, and authored candidate sets, which always contain a better option, never test their admissibility.

Scope. TRACE-AVG at λ<1\lambda<1 is an average-value policy, not a per-decision safety certificate. Its zero observed harm under shift is an observation, and the +0.039+0.039 and +0.032+0.032 bounds do not extend to the shifted laws. Within its own family, λ=0.15\lambda=0.15 also has zero harm, with higher utility in 12/21 settings.

W2. The pair-level certificate opens nothing after 46,080 episodes.

Statement. This is the empirical content of S3, and it is quantitatively explained.

Evidence. With shortlist size KK, joint error α\alpha and q=α/(2K)q=\alpha/(2K), a cell needs at least ⌈(1−q)/q⌉\lceil(1-q)/q\rceil calibration residuals (3939 at K=2K{=}2), and a positive lower bound needs R>2log⁡(4K/α)(τ^−β^)2R>\frac{2\log(4K/\alpha)}{(\widehat\tau-\widehat\beta)^{2}} rollouts per arm per pair. In the post-hoc K=1K{=}1 diagnostic (Appendix T, Table 17 of the revised PDF), three deployment cells have margin ≈0.25\approx0.25 and need R≳118R\gtrsim118, against the 9292 collected. The fourth has margin 0.0220.022 and needs about 15,00015{,}000.

Scope. Some cells are therefore within a modest budget, while one is bound by within-cell heterogeneity. Because K=1K{=}1 was not frozen, this is a sample-size calculation, not a forecast. It also corrects our earlier phrase "near its floor" (revised Appendix T).

Implication. The negative result is not an implementation failure. It is the measured cost of a stronger claim, and it is why the paper places the deployable result at the average-value level. A frozen validation that refuses an attractive exploratory opening (13/32→0/3213/32\to0/32) is evidence of the protocol's discipline.

W4. Scalability, and novelty relative to OPE and selective prediction.

  • Cost. hh is computed at request time from the request, the memory and the static tool schemas. Paired practice is paid once. Against CMI, even when CMI is given an oracle scorer, the upfront cost breaks even after 5151 requests (empirical selector) or 170170 (confirmatory one). TRACE then issues one model call per request instead of 5252 (Appendix Q.1).
  • Richer tool spaces. The representation is the component that must be rebuilt. The analysis shows that measured transfer protects against encoder errors. On 96 fresh phrasings, the intent lexicon misreads 47%47\% of non-blind requests; a deterministic rule on hh then takes 55 to 1313 harmful uses, while every rule fitted to paired practice takes 00.
  • Novelty. The tools are classical, and the paper says so. The contribution is the decision-theoretic structure for agent memory built from them: the pair-level transfer estimand; the separation of identifiability (coverage, S2) from actionability (finite-sample spread, S1); and the measured gap between the two claim strengths at practical budgets (S3). The memory-gating methods we compare against neither separate these claims nor test them on frozen, independent folds.

Thank you again for the questions that sharpened these distinctions.

Where each concern is addressed in the revised PDF (changes in red)

ConcernRevised PDF
W1 practical relevanceIntroduction, Scope paragraph; Section 6, organic paragraph; Appendices X and Y.1
W2 TRACE-CERT opens nothingTable 1; Section 6 sample-size sentence; Appendix T (Table 17)
W3 calibration vs. point estimateTable 4; Appendix U.1 (Tables 20 and 21)
W4 cost, scalability, noveltyContribution list; Appendix Q.1 (Table 15); Appendix U.1, encoder errors

Dear Reviewer XTJC,

Thank you again for your thoughtful review. We hope that the new trace-origin experiments and the matched-feature analysis have addressed your concerns regarding practical relevance and the role of uncertainty calibration.

We would greatly appreciate your feedback on whether these additional results address your main concerns. We would also be happy to clarify any remaining questions.

Thank you for your time and consideration.

Comments on the paper

2 comments

We thank the AC and the three reviewers. We first summarize what the reviewers identified as the paper's strengths, because they agree on them and they define the contribution. We then answer each concern in turn, with the new evidence added in the revision (changes in red; main text within 10 pages).

What the reviewers identified as strengths

1. The problem and its framing are new and well targeted.

  • B6za: the paper "challenges the prevailing assumption in agent memory reuse that 'textual relevance implies safety,'" and the insight "has important theoretical value and practical significance for building reliable and safe tool-using agents."
  • XTJC: "semantic similarity and historical success are not necessarily good indicators of whether a memory will help the current task. Formulating memory utility in terms of its incremental effect on task performance is a meaningful perspective."
  • yp89: "Defining utility through executions with and without a memory clarifies what safe reuse should establish."

The paper replaces relevance with the paired, full-trajectory transfer τ(x,m)=E[U1−U0]\tau(x,m)=\mathbb E[U_1-U_0] of a specific memory, measured on reset replicas.

2. The theory is careful, and it distinguishes claims that are usually conflated.

  • XTJC: "The theoretical formulation is relatively careful. I appreciate that the authors distinguish between average-value guarantees and per-pair certification, rather than treating all uncertainty estimates as equivalent. The discussion of coverage and identifiability also helps clarify when a decision can be supported by the available evidence."
  • yp89: "a useful distinction between average memory usefulness and pair-level certification."

Concretely:

  • Without structural coverage, the transfer sign is not identifiable. Any mandatory binary gate has minimax regret at least αΔ/2\alpha\Delta/2 (Theorem 2).
  • With coverage, a frozen, selection-adjusted policy has a lower bound on average value (Proposition 1).
  • A pair-level claim is a marginal coverage statement (Proposition 2).

3. The controlled benchmark isolates the mechanism.

  • XTJC: it is "thoughtfully designed to separate textual similarity from structural compatibility. The results clearly demonstrate that a more similar memory can be harmful when it suggests an incompatible tool-use procedure."

The harmful memory is deliberately the more similar one. Similarity and an LLM judge rank harmful candidates near chance. The harm (distractor −0.247-0.247, corrupted −0.420-0.420) replicates across three further executors and on BFCL's public schemas, and it grows with scale within Qwen3.

4. The evidence is confirmatory, and negative results are reported.

  • yp89: "The confirmatory experiments and transparent negative results are valuable ... This careful separation of positive average-value evidence from unsuccessful certification supports acceptance as a methodological and diagnostic contribution."
  • XTJC: "I appreciate the transparency of the experimental analysis."

The average-value claim is supported twice: by a prespecified, selection-adjusted confirmatory bound (+0.039+0.039), and by an independent evaluation of the frozen policy on held-out domains (+0.032+0.032). Both are strictly positive, so memory reuse adds value over the no-memory policy on those targets. The stronger pair-level claim does not open under a frozen 46,080-episode validation. The paper reports this separation rather than hiding it.

Responses to the concerns, one by one

C1. Reliable harm was shown only with authored memories (B6za, XTJC, yp89). We tested this directly (Appendix Y.1). Each memory is an automatically serialized record of the executor's own successful execution, preserving the original request, tool call, arguments and success outcome without author-written advice. The design covers 96 families across 16 domains, its SHA-256 was fixed before any target episode, and all results are reported.

  • Prespecified contrast. A same-object opposite-intent trace gives −0.021-0.021, 95% CI [−0.045,−0.003][-0.045,-0.003].
  • Larger effect. An opposite-action trace on another object gives −0.118-0.118, CI [−0.146,−0.087][-0.146,-0.087], and −0.354-0.354 on blind requests. Wrong-tool calls rise from 48/9648/96 to 82/9682/96.
  • Robustness. A fresh no-memory re-run (288 episodes) and leave-one-domain-out refits (16/16 negative) keep both effects negative.

Author-written text is therefore not necessary for the mechanism. Prevalence under natural retrieval is a separate external-validity question that we do not claim to settle.

C2. Representation versus decision rule, and whether calibration is necessary (yp89, XTJC). Every rule now receives the same structural features under outcome-blind candidate shifts.

  • Defaults. At default thresholds, the point estimate takes harmful uses in all 24 shifts. TRACE-AVG and a conservative classifier, P(help∣h)>1/2P(\text{help}\mid h)>1/2, take none.
  • Frontiers. Swept post hoc, thresholded point estimates trace the same empirical envelope. That is a statement about attainable operating points, not a validated selection.
  • Source of the guarantee. The guarantee comes from the selection-adjusted protocol, not from the rule's form; Proposition 1 would certify a frozen threshold class equally.
  • New population. When evidence must be collected on a new population (trace-origin memories), sharing the same mean estimate across rules, the radius makes TRACE-AVG abstain on an unstable cell in 97% of 500 training splits, against 39% for the point estimate. This is exploratory.

C3. TRACE-CERT never opens (XTJC, B6za). The revision gives the sample-size condition. A cell needs ⌈(1−q)/q⌉\lceil(1-q)/q\rceil calibration residuals, and a positive bound needs R>2log⁡(4K/α)/(τ^−β^)2R>2\log(4K/\alpha)/(\widehat\tau-\widehat\beta)^2 rollouts per pair, conditional on a positive envelope margin. The gap to the average-value claim is an empirical separation at the evaluated budgets, not a universal lower bound.

C4. Presentation (yp89, B6za).

  • A pipeline figure (Figure 2) and a claim–assumption–evidence table (Table 1) now open Section 4.
  • The deployed action is named "abstain and flag"; Corollary 2 is cited as motivation only.
  • Active repair is presented as a principle that narrows intervals.

C5. Domains and horizon (B6za).

  • Domains. Between-domain variance is 0.10.1 to 9.4%9.4\% of pair-level variance, and the harm's sign holds in 8/8 practice and 8/8 held-out domains.
  • Horizon. τ\tau is a full-episode contrast, and Theorems 1 and 2 hold at any horizon. Mid-trajectory memory choice remains open.

Stated scope.

  • The average-value bounds hold for their specified targets, not arbitrary shift.
  • TRACE-AVG at λ<1\lambda<1 is not a per-decision certificate.
  • Evidence gathered on one memory population does not license decisions on another (Appendix Y.1).

Summary. The reviewers credited the problem, the theory and the transparency. The revision adds the evidence they asked for: harm without authored text, a fair matched-feature comparison, and an explicit price for pair-level claims. The result is an evidence hierarchy for agent memory reuse, supported by theory and independent evaluation. As the paper's title puts it, experience is not evidence: genuinely successful experience can mislead, and what may be claimed about reusing it depends on the evidence collected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.