- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the issue of memory reuse in tool-calling agents. It argues that high textual similarity between a retrieved memory and the current request does not mean the memory is actually useful. Relying on it can lead to negative consequences, such as the agent blindly imitating incorrect actions. As for the method, the paper proposes a decision framework based on "coverage calibration." It introduces statistical guarantees (the average-value guarantee TRACE-AVG and the per-pair certificate TRACE-CERT) along with a crucial third action: "deferral". For the experiments, the authors constructed a controlled benchmark to evaluate their approach.
Critical strengths
The research problem is novel and well-targeted: The paper challenges the prevailing assumption in agent memory reuse that “textual relevance implies safety,” and insightfully points out that structurally incompatible memories can lead to negative transfer even when they are highly similar in text. This insight has important theoretical value and practical significance for building reliable and safe tool-using agents.
Critical weaknesses
-
Writing and Presentation. The paper lacks an intuitive pipeline figure for the core method, which makes the complex mechanism harder for readers to follow.
-
Limitations in Benchmark and Experiments. Although the authors acknowledge some limitations, the benchmark has a fundamental defect when evaluating agent memory harm, which limits the generality of its conclusions: reliable negative transfer comes only from authored adversarial memories and was not reproduced in organic settings. The benchmark is therefore a diagnostic stress test rather than a reflection of natural distributions.
Questions for authors
- Organic experiments. The paper honestly reports that no reliable negative transfer was found in organic retrieval, and that all reliable harmful effects come from authored memories. Does this mean the method’s practical value lies mainly in adversarial/malicious memory injection, rather than naturally occurring memory quality differences? If so, should the paper define its applicable scope more explicitly?
- Practicality of TRACE-CERT. Experiments show TRACE-CERT still defers 100% of the time after 46,080 validation episodes. What data scale would be needed for this mechanism to actually “open” in a realistic setting? Does this mean per-pair certification is practically infeasible due to cost? Are there more economical paths worth exploring?
- Domain sensitivity. The evaluation covers only 8 domains, and the deployment domains are near-replicates of the calibration domains. Can the authors quantify this heterogeneity, and clarify how it limits generalization?
- Single-step limitation. The current decision unit is limited to single-step intervention, but memory in practice may span multi-step, long-horizon content. Can the authors clarify whether the framework and impossibility theorem extend to such settings?
5 comments
Presentation: pipeline figure.
The revised PDF opens Section 4 with a one-row pipeline figure (new Figure 2): paired rollouts → structural cell → evidence-scaled interval (a coverage set only at ) → use if the lower end clears 0, no memory if the upper end is below 0, abstain and flag otherwise. Next to it, Table 1 lists each variant's claim, assumption and experimental outcome.
Q1. Is the practical value mainly about adversarial memory injection?
No. The demonstrated scope is memory stores that contain competing procedures for the same object. That configuration has three sources, and two of them are benign:
- Changed intent. In the authored benchmark, the lexical distractor is an authored memory describing a previously successful sibling action on the same object ("archive" before, "read" now). In Appendix Y.1, the same situation arises from an automatically serialized record of an actual successful execution.
- Logging or summarization error. The corrupted memory states one action in its body and plays another ().
- Poisoning. An adversary inserts a same-object memory that points to a destructive tool (Ethics Statement).
The mechanism is therefore not specific to adversaries. What remains open is how often the benign sources produce harm in naturally accumulated memory. Organic retrieval settings produced isolated harm (one AppWorld task fell from to ) that did not replicate (Appendix X). The new trace-origin experiment shows that the benign changed-intent source suffices to produce reliable harm, from the executor's own logged successes (Appendix Y.1). The introduction's new Scope paragraph states this boundary.
Q2. TRACE-CERT defers 100%. What scale would open it, and are there cheaper paths?
The answer is a closed-form sample-size condition. With shortlist size , joint error and :
- a cell needs at least calibration residuals ( at , at );
- a positive lower bound additionally needs rollouts per arm per pair.
In the post-hoc diagnostic (Appendix T), three deployment cells have margin and need , against collected. The fourth has margin and needs about . Part of the gap is budget and part is within-cell heterogeneity, which no rollout count removes. Because was not frozen, this is a calculation, not a forecast.
Cheaper paths follow from the formula:
- Fewer simultaneous candidates. A smaller halves the residual requirement.
- Coarser cells. These raise per cell.
- A variance-adaptive rollout radius. of validation pairs have zero within-pair variance. The two-sided empirical-Bernstein radius is against Hoeffding's at , and against at .
These calculations identify the rollout budget required to overcome the within-pair concentration term, conditional on a positive residual-envelope margin. They do not by themselves establish that a valid certificate will open. That also requires enough calibration residuals per cell, residual transport between practice and deployment, and a margin that survives re-estimation (coarser cells raise but can raise within-cell heterogeneity). The paper's finding is an empirical separation under the evaluated budgets: the average-value claim was attained and the pair-level claim was not. This is not a universal sample-complexity lower bound. That gap is the empirical content of the AVG/CERT distinction, not an incidental failure.
Q3. Only 8 domains, near-replicates. Quantify the heterogeneity.
- Between-domain share of pair-level variance: to across supportive, structural-match, distractor and corrupted memories, in both domain sets.
- Magnitude varies, sign does not. The distractor's domain means range from to (practice) and from to (held out). The sign is negative in 8/8 domains of each set and preserved in 8/8 leave-one-domain-out refits (Appendix V).
- The held-out-domain bound is not carried by one domain. Its per-domain gain is positive in 8/8, from to .
Scope: the domains share one read/write skeleton, so the evidence covers new objects and tool names within it, not new tool graphs. The bound is for the specified fixed-design grid. The harmful uses there is an observation; its binomial upper bound () holds only under an i.i.d. reading of a fixed-design grid, and the revision labels it as such.
Q4. Single-step decision unit. Do the framework and the theorem extend?
- The estimand already covers whole trajectories. is a difference in full-episode return; the memory is injected once at episode start. Theorem 1, Proposition 1 and the conformal certificate hold for any bounded episode return, at any horizon.
- Theorem 2 is horizon-free. Its two environments agree on every pre-deployment observation and differ only in the sign of the effect on an unpracticed region. Trajectory length does not enter the argument.
- What does not extend is mid-trajectory revision, i.e. choosing which memory to consult at step given the history. That is a sequential policy problem, which the revised Conclusion names as the main open direction.
The framework has also been run on multi-step environments: AppWorld (Appendix R; supportive workflow ) and a non-ceiling -bench-derived retail environment (Appendix W). The estimand was well defined in both. What did not appear was a reliable harmful class (organic distractors on retail). This is why the harmful regime is stated for single-call competing memories, while the estimand and the identifiability result are not restricted to them.
Summary for the soundness assessment. The paired estimand is measured on reset replicas with ground-truth scoring. The theorems hold under their stated assumptions, and the review's concerns address generalization rather than any specific theorem or estimator. The confirmatory bounds were computed on folds frozen before collection. The limitation is the external reach of the harmful regime, and the paper now states it at the front.
Where each concern is addressed in the revised PDF (changes in red)
| Concern | Revised PDF |
|---|---|
| Pipeline figure | Figure 2 and Table 1, start of Section 4 |
| Benchmark validity | Abstract; Introduction, Scope paragraph; Section 6, organic paragraph; new Appendix Y.1 (trace-origin memories) |
| Q1 adversarial vs. natural | Introduction, Scope paragraph; Ethics Statement |
| Q2 TRACE-CERT scale | Section 6 sample-size sentence; Appendix T (Table 17) |
| Q3 domain heterogeneity | Appendix V, new heterogeneity paragraph |
| Q4 single-step | Conclusion; Appendices R and W |
Thank you for the review, and for identifying the paper's core insight: textual relevance does not imply safe reuse, and a structurally incompatible memory can harm even when it is the most similar one. Manuscript changes are in red.
The new experiment directly addresses the concern that negative transfer is an artifact of authored memories (Appendix Y.1, Table 23).
We repeated the controlled mechanism test using the executor's own successful past executions as memory content. Each memory is an automatically serialized record of a successful execution, preserving the original request, tool call, arguments and success outcome without author-written procedural advice.
Design.
Results.
The negative-transfer mechanism is therefore not restricted to instructions authored by the researchers. Crucially, each memory records a genuinely successful past action. Its harmfulness arises when that action is transferred to a request with a different intent.
This resolves the authored-content explanation within the controlled benchmark. The remaining limitation is the prevalence and impact of such conflicts under unconstrained organic retrieval, which we do not claim to establish. A token-overlap retriever, for instance, selects same-object traces and not the high-leverage other-object ones.
What remains is external validity, and it should be assessed separately from soundness. Two propositions are at issue:
Every effect is a paired execution on independently reset replicas, scored against ground truth with no LLM judge. The authored effects (distractor −0.247, [−0.34,−0.17]; corrupted −0.420, [−0.50,−0.33]) are robust to the resampling unit. They keep their sign in 8/8 leave-one-domain-out refits and replicate on three further executors and on BFCL's public schemas. The experiment measures the effect of each specified memory; it does not isolate a causal effect of structural incompatibility from every other property of the memory.
The unresolved question is (E). It does not, by itself, invalidate the measured intervention effects (I) or the theoretical results under their stated assumptions. The review's concerns are concerns of generalization rather than of a specific theorem or estimator, and we respectfully ask that the soundness assessment distinguish these issues.