acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Governing Memory Validity in Evolving Multi-Agent Systems

Abstract

Persistent memory lets language-model agents carry information across long- running collaborations, but leaves a lifecycle question open: what may a returning agent still act on once the shared state has changed? A memory can be correctly retrieved, relevant to the current task, and faithful to its source, and nonetheless be inadmissible for action. We formalize this as temporal memory admission and present TRACE, a training-free layer that treats re-entry as an eligibility decision rather than a storage or retrieval operation, reconciling a departure checkpoint against absence-period updates, resolving explicit and implicit invalidation, and releasing a bounded Return View only when it covers the returning role’s open obligations. We evaluate TRACE under a common five-agent return protocol on Memora, STALE Type II, and a derived ManBench-Return setting, under three ac- tor models. What separates methods is not overall accuracy but whether one can retain valid memory and reject stale memory at once, and no single-policy baseline can: restore admits stale state, reset discards valid state, each bottoming out at 0% on one of the two. TRACE is the only method high on both, reaching 92.6–98.3% valid-information availability with 98.4–99.5% invalid-information rejection on ManBench-Return, within 3.8 points of the best baseline’s overall accuracy. On STALE Type II it gains 18 to 28 points over the strongest memory-policy baseline at roughly 2.3× their tokens, while a write-time consolidation pipeline is more ac- curate still at 3.99× TRACE’s. On Memora, where updates are explicit, ordinary storage semantics suffice and TRACE offers no consistent gain.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent systems, LLM agents, long-term agent memory, temporal reasoning, temporal memory admission, memory validity, memory governance, selective forgetting, provenance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_do4LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of memory management in agents that go offline/do alternate work while the conversation continues to grow and new memories are collected. Specifically, the paper studies how the interaction of these agents with stale memory should be formalized and how to best update the memory to keep the agent aware of the "delta" between when it left and returned. They propose TRACE, a training free method built on top of their formalization and show that it brings improvement in model performance on certain existing memory benchmarks while also showing failures on others.

Critical strengths
  • The problem of studying drift in memory is crucial for several parallel fields such as personalization and continual learning. The problem is timely.
  • The need for formalizing memory update is growing and tracking state changes is subjective and is heavily task dependent. Even though I am not satisfied with the details the authors present, the authors have picked a good and useful problem to solve.
Critical weaknesses

Methodology problems:

  • There are several claims that the authors make that are not backed by citations and are never used as baselines in their experiments. For example, line 083 states that recency is not enough but they have not backing to this and do not provide this as a baseline either.
  • It is unclear what is the motivation for the structure of equation 1. This connects to the point above that not enough research context is discussed and motivated in the paper for the formalization that is proposed
  • Figure 2's example is never explained to the reader but it holds a lot of clarification information.
  • From 212-215, it is unclear what "high recall scan" and "grounded verifier" mean. This is core to the method and not enough explanation is provided for these when the reader is reading with the flow of the paper.
  • Even though this is discussed very slightly later, it is unclear what the individual size budget is in line 225.
  • Equation 3 is not explained properly. What is the reason why this is framed this specific way? It is not grounded properly into the set theory claim the paper makes.
  • It is unclear to me how this method's performance changes with different configs. In the main experiment, the authors use 4 acting agents and 1 returning agent and later perform ablations on the acting agents but keep this experiment restricted to one returning agent. It is unclear how the behavior changes if multiple agents are returning.
  • The choice of baseline models is not clear. Why these specific models?
  • The metrics are under-defined formally in the main paper. It is not clear what VIR, IIR, etc are - are these accuracies? F1 scores?
  • The authors make claims without any reported statistical significance. The method seems to be sensitive to non-determinism of LLMs and significance needs to be studied

Writing style:

  • The paper is not written coherently (potentially largely AI generated). The reader currently needs to keep a track of terminologies and sections frequently refer to details explained much later in the paper.
  • Citations in the document are broken and do not follow the standard ICLR template.
  • The abbreviations of metrics is not intuitive and frequently overwhelming to keep track of. For example, Why not use VSR for valid-state availability and ISR for invalid-state rejection instead of using VIR and IIR?
Questions for authors

Addressing the weaknesses above directly addresses my final score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Reviewer do4L

We thank the reviewer for the detailed feedback and for recognizing the importance of the problem. We apologize for the presentation difficulties and have grouped the concerns into five questions.

W1, W2 & W6. What motivates TRACE's formulation and Equations 1 and 3?

A1: Equation 1 separates relevance from admissibility; Equation 3 ensures that the admitted memories are jointly sufficient to resume work. Equation 1 is a problem formulation, not a theorem, and Equation 3 is implemented as a heuristic without a claimed guarantee.

Related work. The revised manuscript now discusses Generative Agents (Park et al., 2023), truth-maintenance systems (Doyle, 1979), and STALE (Chao et al., 2026). These works motivate checking a memory's premises beyond its recency; they do not substitute for a recency-only comparison.

Equation 1 (relevance vs. admissibility). A train recommendation can remain relevant to travel planning after the destination changes, yet no longer support the current trip. Admissibility therefore depends on the agent, the current state, the absence-period changes, and the obligations.

Equation 3 (joint sufficiency). Individually valid memories may still jointly provide insufficient information to resume work. Equation 2 defines the eligible records; Equation 3 selects a subset that covers obligations and critical dependencies within item-count and character limits while minimizing disclosure cost. Appendix C defines coverage through the union of supported dependencies: an obligation is covered only when all of its required dependencies are present. The implementation reserves mandatory records and then greedily adds uncovered support per unit cost. This is a constrained set-cover heuristic; we claim neither optimality nor a classical approximation guarantee. Insufficient coverage triggers reset or blocking.

W3, W4 & W5 . How do the pipeline, scanner, verifier, and context budgets work?

A2: Pipeline. The revised Section 3 explicitly links its subsections to Figure 2: checkpoint capture (block 1), invalidation and validation (block 2), obligation coverage (block 3), and signing/private-memory readmission (block 4). Appendix C details the scanner and verifier and provides a worked example.

Scanner and verifier. The scanner proposes event–memory pairs whose dependencies may have changed; a proposal is not itself a rejection decision. The verifier then examines the original observations and their temporal order to determine whether later evidence invalidates the earlier item's current use. Topic overlap alone is insufficient: discussing trains need not invalidate an itinerary, whereas changing the destination may.

Terminology. "Grounded" refers to source observations, not benchmark answers. "High-recall" describes the design intent of the proposal stage, not a measured recall guarantee.

Size budgets. We agree that "individual size budget" was ambiguous. A candidate does not have an independent budget; each candidate must fit within the Return View's remaining item-count and cumulative character capacity. As introduced earlier in the paper, work-state records are selected first, and private memories require admitted source state for readmission. We will clarify this budget at first use.

(W7, W8 & W10, W9 & WS3, WS1 & WS2 continue in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer Reviewer do4L

W7, W8 & W10. What do the experiments establish about generalization and reliability?

A3: Across models and policies. Qwen, Gemini, and DeepSeek test transfer across actor families, which supports generality within the tested models. The method baselines separately cover retaining, resetting, and updating memory.

Multiple returners. We agree that varying the active-team size with a single returner does not establish behavior with multiple returners. We have explicitly added simultaneous and staggered returns of multiple agents to Future Work.

Statistical reliability. Training-free operation does not eliminate LLM inference variability. Our current evaluation does not establish statistical significance, and cross-model consistency is not a substitute for uncertainty analysis. We will acknowledge this limitation and identify repeated runs and paired uncertainty analysis as follow-up work.

W9 & WS3 . How are Overall, VIA, and IIR defined?

A4: We apologize for the insufficient definitions.

VIA and IIR. VIA (Valid Information Availability) and IIR (Invalid Information Rejection) name the two complementary requirements of our return contract: preserving usable information and excluding inadmissible information. Both are success proportions, not F1 scores. They are organizing labels, not claimed standard metric acronyms.

Across benchmarks. We map each benchmark's native judgments to VIA/IIR, so shared labels do not imply identical scoring rules.

In the revision. The revised Metrics section explains Overall for Memora and ManBench-Return and gives explicit STALE VIA/IIR calculations; Appendix D.1 specifies the detailed definitions and aggregation. We appreciate the suggested alternatives and have clarified the metric names and definitions accordingly.

WS1 & WS2. How have we addressed writing clarity and citation formatting?

A5: We have revised the figure references and metric explanations and will clarify terminology at first use. We will also check the rendered citations against the ICLR template. If the reviewer could point us to the affected citations or locations, it would help us address the reported issues.


We sincerely thank the reviewer again for the careful reading and constructive feedback, which helped us improve the clarity of the formulation, method description, and evaluation. We hope these responses address the reviewer's concerns and are happy to provide further clarification.

Reviewer_do4LOpenReview ↗

Response to Rebuttal

  1. The original comment about lack of baselines still holds. The authors have not updated their results with any additional baselines (not even a random baseline)
  2. The terminology is still extremely hard for me to follow. For instance, "High-recall describes the design intent of the proposal stage, not a measured recall guarantee.", this is still unclear to me why this is called high-recall when there is a canonical usage policy for what recall means in prior literature.
  3. "A candidate does not have an independent budget" - how do we ensure fairness here if there is no fixed budget?
  4. The selection of parameters is still unclear. What drives and why have the authors not presented ablations for the number of returning agents?
  5. My concerns around the method seeming to be sensitive to non-determinism of LLMs is unaddressed. The authors have failed to provide significance numbers as well

In the current state of the paper, most of my concerns have simply been marked as future work or limitations instead of being properly rebutted and backed by strong empirical results. Due to this reason, I will keep my score unchanged.

Reviewer_5Tz2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies which memories a returning agent can safely use after its teammates have changed the shared state. TRACE combines update reconciliation, implicit invalidation checks, authorization and provenance checks, obligation coverage, and controlled readmission of private memories. The ablation results support the importance of implicit invalidation: removing it reduces Qwen’s overall score on STALE Type II from 52.33% to 26.50%. However, an arithmetic inconsistency in Table 6, the absence of compute-matched baselines, and the controlled construction of ManBench-Return limit confidence in the reported gains. I find the contribution promising, but recommend weak rejection pending clarification and stronger experimental controls.

Critical strengths
  1. A comprehensive return-time procedure. TRACE brings together state updates, implicit invalidation, authorization and provenance checks, coverage of outstanding obligations, and the readmission of private memories.
  2. Ablation results support a key design choice. Removing implicit invalidation reduces Qwen’s overall score on STALE Type II from 52.33% to 26.50%.
Critical weaknesses
  1. An internal arithmetic inconsistency in the results. In Table 6 on page 29, the pooled Static row reports an Overall score of 28.33%, but the mean of its three reported component scores is 30.78%. Some pooled component scores also cannot be reproduced by weighting the Early, Middle, and Late results.
  2. No compute-matched comparison. TRACE uses additional scanning and verification calls, making it difficult to separate the benefit of its admission design from the benefit of extra computation.
  3. Limited generalizability of ManBench-Return. This is a derived task that selects cases with a correct pre-departure answer and, in one condition, a correct majority among the active agents. It is more controlled than open-ended collaboration in practice.
Questions for authors
  1. Could you correct and recalculate Table 6? In particular, why can the pooled component scores for the comparison methods not be reproduced from the Early, Middle, and Late results, and would a correction affect the comparisons in Table 1?
  2. How much of TRACE’s advantage remains under an equal token or model-call budget? Could you add baselines that receive comparable opportunities for verification?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Reviewer 5Tz2

We thank the reviewer for the careful verification of our results and for the constructive suggestions. We have corrected the reported error and added the requested controls, as detailed below.

Q1&W1: Table 6 and its implications for Table 1

A1: The reviewer is correct, and we have fixed the error. The reported pooled components reproduced neither the stated Overall score nor the bucket-weighted aggregates. This was a table-assembly error: values from a preliminary run were inadvertently mixed with the final evaluation results. We apologize for the error and the resulting confusion.

What we corrected (revised manuscript submitted):

  • Table 6. All bucket-level and pooled entries are regenerated from the final per-episode evaluation records. Early, Middle, and Late contain 64, 64, and 72 episodes, respectively, and each pooled metric is computed as Mpooled=64ME+64MM+72ML200M_{\mathrm{pooled}}=\frac{64M_E+64M_M+72M_L}{200}.
  • Aggregation. Overall is the mean of SR, PR, and IPA, with rounding performed only after aggregation. IIR and LS are computed from within-episode joint judgments rather than from aggregate component scores.
  • Checkability. The revised table includes the complete bucket-level results for MemStrata and MemTX, so their pooled values can be verified directly.
  • Table 1. The corresponding Qwen STALE entries, and the numerical statements that depend on them, are synchronized.

Corrected pooled Overall. Static 30.00, Restore 6.33, Reset 9.67, MemStrata 32.17, MemTX 32.50, and TRACE 58.17. For example, Static's corrected Overall is (20.50 + 29.50 + 40.00)/3 = 30.00%.

Effect on conclusions. The corrections change the reported values but preserve the conclusion that TRACE achieves the highest Overall among the six methods in this evaluation. Its corrected advantage over the strongest baseline, MemTX, is 25.67 percentage points. The component-ablation study is separate from this main evaluation. We appreciate the reviewer's careful check, which helped us correct the presentation and make the aggregation procedure explicit.

W2. What do the additional experiments establish about the scanner and verifier?

A2: Both components contribute within TRACE: removing either one lowers Overall on both benchmarks. The full pipeline's advantage over Scanner-only is concentrated in retaining valid prior state.

Setup. We evaluated separate removal ablations, together with Scanner-only and Simple Verifier controls, using DeepSeek-v4.1-flash-1017. The additional subsets contain 84 STALE episodes and 104 ManBench-Return (MB) episodes and are separate from the original three-actor evaluation. Scores are percentages; tokens are average reported tokens per episode.

VariantSTALE OverallSTALE tokensMB Old-ValidMB Old-StaleMB OverallMB tokens
TRACE71.03411,26393.08100.0096.541,317
w/o scanner62.44609,06487.6492.3790.011,572
w/o verifier69.70571,30291.2195.7793.491,540
Scanner-only71.52511,30276.67100.0088.341,391
Simple Verifier68.21660,22273.3393.1883.261,372

Removal ablations. Removing the scanner reduces Overall by 8.59 points on STALE and 6.53 on ManBench-Return; removing the verifier reduces it by 1.33 and 3.05 points, respectively. These are downstream effects on task performance; they do not measure standalone detection precision, recall, or false-positive/negative rates.

Standalone controls. Scanner-only scores 0.49 points higher than TRACE on STALE but 16.41 points lower on ManBench Old-Valid; both reach 100% on Old-Stale. The full pipeline's advantage over Scanner-only on ManBench-Return therefore lies in retaining valid prior state. We do not claim that the full pipeline maximizes task accuracy in every setting.

(W3 and Q2 continue in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer Reviewer 5Tz2

W3: Scope of ManBench-Return

A3: We agree: ManBench-Return is a restricted diagnostic, not a comprehensive test of memory validity, and we will state this scope explicitly in the paper.

What it tests. Each eligible episode contrasts a departure answer with an absence-period challenger:

  • Old-Valid tests whether valid prior state is retained despite misleading updates.
  • Old-Stale tests whether prior state is rejected after a task change.

Together, they test whether admission follows current validity rather than information recency.

Why it is informative. It exposes cases in which answer correctness and state preservation diverge. In the main Qwen evaluation, Reset achieves 96.38% Overall but 0% VIA (Valid Information Availability), whereas TRACE achieves 94.32% Overall with 95.96% VIA and 98.78% IIR (Invalid Information Rejection). We therefore use ManBench-Return for a preservation–rejection analysis, not to claim uniformly better task accuracy.

Limitations. The two-candidate construction and the eligibility filtering limit generalization: every episode requires a correct departure answer, and Old-Stale additionally requires a correct majority among the active agents. These conditions do not capture the full difficulty of open-ended, long-horizon memory use. We will clarify the scope and the selection criteria in the paper. STALE and Memora provide complementary evidence on implicit invalidation and explicit updates, respectively, but do not eliminate this limitation.

Q2: Do TRACE's gains persist when baselines receive comparable verification opportunities?

A4: Thank you for this helpful suggestion. We added Static + verifier and MemStrata + verifier, alongside a Simple Verifier control.

Setup. The augmented baselines retain their own memory-processing procedures and add verification before final answering. The verifier receives the candidate memories, the absence-period updates, and the current task, and checks which memories are applicable to the answer. The token budget counts input and output tokens for memory processing, verification, and answering, including any retries. All methods share the same per-episode cap within each benchmark: 2,097,152 tokens for STALE, 65,536 for Memora, and 32,768 for ManBench-Return. Entries report Overall (%) / average tokens per episode.

BenchmarkEpisodesTRACEStatic + verifierSimple VerifierMemStrata + verifier
STALE8471.03 / 411,26345.56 / 401,28368.21 / 660,22240.00 / 406,284
Memora4065.38 / 33,88864.69 / 37,82959.20 / 44,10866.91 / 40,942
ManBench-Return10496.54 / 1,31789.81 / 89283.26 / 1,37279.21 / 952
  • STALE. TRACE exceeds Static + verifier and MemStrata + verifier by 25.47 and 31.03 percentage points, with slightly higher token consumption. Against Simple Verifier, TRACE improves Overall by 2.82 points while using 37.71% fewer tokens.
  • ManBench-Return. TRACE achieves the highest Overall, although it uses more tokens than the verifier-augmented Static and MemStrata baselines.
  • Memora. MemStrata + verifier exceeds TRACE by 1.53 points, consistent with TRACE's less consistent gains under explicit updates.

Adding verification to the evaluated baselines therefore does not eliminate TRACE's advantage on STALE and ManBench-Return. We nevertheless distinguish equal budget caps from equal realized computation, and we limit these conclusions to the evaluated subsets.


We sincerely thank the reviewer for the careful reading and constructive feedback, which helped us correct the reported results and strengthen the experimental controls. We hope these responses address the reviewer's concerns and are happy to provide further clarification.

Thank you for the detailed responses and additional experiments.

The corrected Table 6 and updated Table 1 resolve my concern about numerical consistency. The new comparisons with verifier-augmented baselines also substantially address my concern about computational fairness. I also appreciate the clarification of the scope and limitations of ManBench-Return.

I still have some reservations about the incremental benefits of the full TRACE pipeline over simpler alternatives. For example, Scanner-only slightly outperforms full TRACE on STALE, while the improvements on Memora remain inconsistent. These results leave some uncertainty about the necessity of the complete design and its benefits across different settings.

Nevertheless, the additional experiments and clarifications have addressed my main concerns. I will therefore raise my rating from 2 to 3.

Thank you again for your valuable time and effort throughout the rebuttal.

AuthorsOpenReview ↗

Response to Reviewer Reviewer 5Tz2

Dear Reviewer 5Tz2,

Thank you very much for your thoughtful follow-up, constructive engagement throughout the rebuttal, and for raising your score. We are glad that the updated tables, verifier-augmented baseline comparisons, and clarifications on ManBench-Return addressed your main concerns.

We also sincerely appreciate your remaining observation regarding the incremental benefits of the full TRACE pipeline over simpler alternatives (such as Scanner-only on STALE and the mixed results on Memora). In the revised version, we will include a more explicit and balanced discussion characterizing when lightweight components suffice and under which settings the full pipeline provides stronger benefits.

Thank you again for your valuable time and feedback—your suggestions have meaningfully strengthened the paper.

Reviewer_WFW6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the validity of persistent memory when an agent returns to a multi-agent system whose shared state has changed during its absence. The authors propose TRACE, a training-free framework that reconciles state updates, detects explicit and implicit memory invalidation, validates memory eligibility, and constructs an obligation-aware Return View. Experiments on Memora, STALE Type II, and ManBench-Return show improvements in handling implicit invalidation and distinguishing valid from stale information, alongside mixed task-level results and additional computational costs.

Critical strengths
  1. The paper addresses a practically important problem: previously valid memories may become unsuitable for action after a multi-agent system evolves. Treating agent return as a distinct memory-admission problem provides a meaningful perspective beyond conventional storage and retrieval.

  2. TRACE presents a clear and well-structured framework. Its decomposition into state reconciliation, invalidation detection, eligibility validation, obligation-aware selection, and controlled memory readmission makes the overall approach easy to understand and analyze.

  3. The experiments demonstrate substantial improvements on STALE Type II across three actor models. The component ablations also provide evidence that implicit invalidation detection contributes substantially to these gains.

Critical weaknesses
  1. The experiments do not establish a consistent improvement in end-to-end task performance. TRACE underperforms Reset on Memora Overall across all three models, while its strong admission metrics on ManBench-Return do not translate into the highest final-answer accuracy. The practical benefits of improved memory admission therefore remain insufficiently demonstrated.

  2. Although four component ablations are provided, the dependency scanner and verifier are not independently evaluated. The reported acceptance counts do not establish their detection accuracy, false-positive rates, or false-negative rates. The reliability of this central mechanism therefore remains unclear.

  3. The computational cost is not sufficiently justified. On the STALE subset, TRACE improves accuracy over cheaper baselines but requires approximately 2.3 times their tokens, while the more expensive CUPMem achieves substantially higher accuracy. Without matched-budget comparisons and broader cost evaluations, it remains unclear when TRACE offers a preferable accuracy–cost trade-off.

Questions for authors
  1. Can the authors evaluate the dependency scanner and verifier separately, reporting their detection accuracy? Would TRACE's improvements remain under a matched-token verification baseline?

  2. Under comparable computational budgets, when does TRACE provide meaningful improvements in final task performance beyond its memory-admission metrics? Evidence across all three benchmarks would substantially strengthen the practical justification.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Reviewer WFW6

We thank the reviewer for the thoughtful assessment and constructive suggestions. They helped us clarify when TRACE is beneficial, strengthen the experimental controls, and state the limits of our evidence more precisely.

W1&Q2. In which update scenarios does TRACE improve final task performance?

A1: TRACE helps most when prior information becomes invalid without an explicit replacement (STALE Type II) and when valid prior state must be retained against misleading updates (ManBench-Return Old-Valid). Its gains are least consistent under explicit updates (Memora).

  • Memora (explicit updates). Explicit replacement information already supports conventional memory updates. Our original results do not show a consistent Overall advantage over the strongest alternatives, although TRACE outperforms Simple Verifier in the additional budget-controlled comparison (Q3). Balancing retention and forgetting remains a limitation.
  • STALE (implicit invalidation). Prior information can become inapplicable without an explicit replacement statement. This is where TRACE shows its clearest task-level gains over the original memory-policy baselines: 25.67, 18.50, and 27.51 percentage points in Overall over the strongest listed baseline for Qwen, Gemini, and DeepSeek, respectively.
  • ManBench-Return (controlled retain/reject). The controlled conditions test both retaining valid prior state against misleading updates and rejecting stale state after a task change. In the additional DeepSeek-v4.1-flash-1017 evaluation, TRACE achieves 96.54% Overall and 93.08% on Old-Valid, compared with 76.67% for Scanner-only and 73.33% for Simple Verifier on Old-Valid. This supports TRACE's value in preserving valid prior state in this controlled setting; it does not establish generalization to open-ended collaboration.

Aggregate view. Across the original three benchmarks, TRACE has the highest equal-weight macro-average Overall for each actor, exceeding the strongest baseline under the same aggregation by 11.38, 9.62, and 10.29 points, respectively.

ActorBest baseline by macro-averageBaselineTRACEGain (pp)
Qwen3.5-122B-A10BMemTX61.1872.5611.38
Gemini-3-flashStatic75.3384.949.62
DeepSeek-v4-flash-0731Reset61.6671.9510.29

W2&Q1. What do the additional experiments establish about the scanner and verifier?

A2: Both components contribute within TRACE: removing either one lowers Overall on both benchmarks. The full pipeline's advantage over Scanner-only is concentrated in retaining valid prior state.

Setup. We evaluated separate removal ablations, together with Scanner-only and Simple Verifier controls, using DeepSeek-v4.1-flash-1017. The additional subsets contain 84 STALE Type II episodes and 104 ManBench-Return (MB) episodes and are separate from the original three-actor evaluation. Scores are percentages; tokens are average reported tokens per episode.

VariantSTALE OverallSTALE tokensMB Old-ValidMB Old-StaleMB OverallMB tokens
TRACE71.03411,26393.08100.0096.541,317
w/o scanner62.44609,06487.6492.3790.011,572
w/o verifier69.70571,30291.2195.7793.491,540
Scanner-only71.52511,30276.67100.0088.341,391
Simple Verifier68.21660,22273.3393.1883.261,372

Removal ablations. Removing the scanner reduces Overall by 8.59 points on STALE and 6.53 on ManBench-Return; removing the verifier reduces it by 1.33 and 3.05 points, respectively. These are downstream effects on task performance; they do not measure standalone detection precision, recall, or false-positive/negative rates.

Standalone controls. Scanner-only scores 0.49 points higher than TRACE on STALE but 16.41 points lower on ManBench Old-Valid; both reach 100% on Old-Stale. The full pipeline's advantage over Scanner-only on ManBench-Return therefore lies in retaining valid prior state. We do not claim that the full pipeline maximizes task accuracy in every setting.

(W3&Q1&Q2 continues in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer Reviewer WFW6

W3&Q1&Q2. Do the benefits persist under comparable budgets?

A3: Yes against Simple Verifier on all three benchmarks, against Scanner-only on ManBench-Return, and against the original baselines on STALE at 1M and 2M caps. Against Scanner-only on STALE, the result is an accuracy–cost trade-off, and under a 500K cap TRACE underperforms the original baselines.

We agree that computational cost must be considered. All compared methods receive the same per-episode token cap within each additional benchmark:

BenchmarkEpisodesToken cap
Memora4065,536
STALE Type II842,097,152
ManBench-Return10432,768

A shared cap controls the allowed budget, not realized expenditure or the number of model calls, so we report observed token consumption separately.

  • Memora. TRACE outperforms Simple Verifier by 6.18 points (65.38 vs. 59.20) while using 23.17% fewer reported tokens.
  • STALE Type II. TRACE outperforms Simple Verifier by 2.82 points while using 37.71% fewer reported tokens. Relative to Scanner-only, TRACE uses 19.57% fewer tokens for a 0.49-point decrease in Overall: an accuracy–cost trade-off, not uniform dominance.
  • ManBench-Return. TRACE improves Overall by 8.20 points over Scanner-only and 13.28 over Simple Verifier, while using 5.32% and 4.01% fewer reported tokens, respectively.

On these subsets, the observed improvements over the controls are thus not accompanied by greater aggregate token consumption. They do not establish latency or monetary-cost savings, nor do they fully isolate each design choice from differences in how computation is allocated. They complement the original comparison, in which TRACE incurred additional overhead relative to cheaper conventional baselines.

Budget sweep against the original baselines (STALE). To test whether this overhead explains TRACE's gains, we further compare the original memory policies on a separate 40-question STALE subset under three shared per-episode caps. Each cell reports Overall / VIA / IIR (%).

Method500K1M2M
Static50.00 / 64.29 / 42.8652.38 / 71.43 / 42.8640.48 / 50.00 / 35.71
MemStrata47.62 / 64.29 / 35.7150.00 / 57.14 / 42.8645.24 / 42.86 / 42.86
MemTX45.24 / 57.14 / 35.7147.62 / 64.29 / 35.7147.62 / 64.29 / 35.71
CUPMem0.00 / 0.00 / 0.000.00 / 0.00 / 0.0021.43 / 21.43 / 21.43
TRACE35.71 / 35.71 / 35.7173.81 / 64.29 / 78.5771.43 / 57.14 / 78.57

With 1M and 2M caps, TRACE achieves the highest Overall (73.81 and 71.43, vs. 52.38 and 47.62 for the next-best method). Under the tightest 500K cap, however, TRACE falls below all baselines except CUPMem (35.71 vs. 50.00 for Static): with the reduced budget, TRACE more often exhausts its token allowance before completing the task, and these episodes are counted as failures. TRACE therefore needs a sufficient budget to benefit, and we will report this limitation in the revision.


We sincerely thank the reviewer again for the careful and constructive feedback. We hope these responses address the reviewer's concerns and are happy to provide further clarification.

Reviewer_3A93OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE to study an agent that leaves a team and later returns after the shared state has changed. It checks which old memories can still be used, rather than simply restoring everything or starting with empty memory. The method combines a departure checkpoint with absence-period updates, checks explicit and implicit invalidation, and selects a bounded return context that covers the agent’s current obligations. Private memories are readmitted only when their associated work-state records survive. Experiments use three actor models on adapted Memora, STALE Type II, and ManBench settings. The largest accuracy gains are on implicit invalidation in STALE, while benefits are limited on Memora.

Critical strengths

The problem is well motivated. A memory can be relevant and correctly recorded but no longer suitable for the agent’s next action.

The STALE results are substantial across all three actor models. The ablation removing implicit invalidation causes the largest drop, which supports the importance of checking dependencies rather than just explicit replacement links.

Critical weaknesses

It is not clear how much of the STALE gain comes from the proposed admission design versus extra model computation.

ManBench-Return is a fairly restricted test of memory validity.

Questions for authors

How does TRACE compare with a simple verifier that receives the old memory, updates, and current task under the same total token budget?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Reviewer 3A93

We thank the reviewer for the insightful questions and constructive suggestions. Below we address the comparison with a simple verifier and the scope of ManBench-Return.

Q1&W1: How does TRACE compare with a simple verifier that receives the old memory, updates, and current task under the same total token budget?

A1: Under the same per-episode token cap, TRACE is more accurate than the Simple Verifier on all three benchmarks while consuming fewer tokens.

We ran exactly this comparison. We agree that TRACE also performs verification, so the relevant question is whether its structured return-time admission procedure adds value beyond a simpler verifier. Both methods use DeepSeek-v4.1-flash-1017 and share the same per-episode token cap within each benchmark: 65,536 on Memora (40 examples), 2,097,152 on STALE (84 examples), and 32,768 on ManBench-Return (104 examples).

BenchmarkSimple Verifier Overall (%)TRACE Overall (%)Overall gainToken reduction
Memora59.2065.38+6.18 pp23.17%
STALE68.2171.03+2.82 pp37.71%
ManBench-Return83.2696.54+13.28 pp4.01%

TRACE improves Overall by 6.18, 2.82, and 13.28 percentage points on Memora, STALE, and ManBench-Return, while reducing reported token consumption by 23.17%, 37.71%, and 4.01%, respectively. The gains therefore do not come from greater aggregate token consumption. Note that the comparison matches budget caps rather than realized expenditure, which is why we report observed consumption alongside accuracy.

Budget sensitivity on STALE. To check whether the conclusion depends on the chosen cap, we further compare the original memory policies on a separate 40 question STALE subset under three shared per-episode caps. Each cell reports Overall / VIA / IIR (%).

Method500K1M2M
Static50.00 / 64.29 / 42.8652.38 / 71.43 / 42.8640.48 / 50.00 / 35.71
MemStrata47.62 / 64.29 / 35.7150.00 / 57.14 / 42.8645.24 / 42.86 / 42.86
MemTX45.24 / 57.14 / 35.7147.62 / 64.29 / 35.7147.62 / 64.29 / 35.71
CUPMem0.00 / 0.00 / 0.000.00 / 0.00 / 0.0021.43 / 21.43 / 21.43
TRACE35.71 / 35.71 / 35.7173.81 / 64.29 / 78.5771.43 / 57.14 / 78.57

With 1M and 2M caps, TRACE achieves the highest Overall (73.81 and 71.43, vs. 52.38 and 47.62 for the next-best method). Under the tightest 500K cap, however, TRACE falls below all baselines except CUPMem (35.71 vs. 50.00 for Static): with the reduced budget, TRACE more often exhausts its token allowance before completing the task, and these episodes are counted as failures. TRACE therefore needs a sufficient budget to benefit, and we will report this limitation in the revision.

Why the structure matters: what verification controls, and when it is applied. A verify-and-update approach compares old and new information and revises stored memories. TRACE instead evaluates the departure checkpoint together with the relevant updates accumulated during the agent's absence, and then determines which records may enter the returning agent's context. Three elements go beyond a single verification step:

  • Dependency scan followed by source-grounded verification. The scan proposes potential implicit invalidations and can capture cross-attribute dependencies: a changed destination may invalidate a hotel recommendation even without an explicit update to the hotel attribute.
  • Separation of historical records from current eligibility. A historically correct memory remains stored while being excluded from the current Return View.
  • Admission checks. The procedure additionally checks update authority, source receipts, and epochs, and permits private-memory readmission only when the bound work state is admitted.

The contribution is this structured admission procedure around verification, not verification alone. We do not assume that a generic verifier is inherently incapable of detecting such dependencies; the matched-cap comparison above measures how the two behave in practice.

Ablation. TRACE without verification obtains 62.13%, 69.70%, and 93.49% Overall on Memora, STALE, and ManBench-Return, versus 65.38%, 71.03%, and 96.54% for full TRACE (drops of 3.25, 1.33, and 3.05 points). This supports the contribution of verification within TRACE, although the effect on STALE is modest. Because removing verification also changes the amount of computation, the ablation alone does not isolate the two; the Simple Verifier comparison above addresses the broader question of whether the complete procedure helps under a common token cap.

Scope. These conclusions are limited to the evaluated subsets and do not establish superiority over every possible verifier.

(W2 continues in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer Reviewer 3A93

W2: Scope of ManBench-Return

A2: We agree: ManBench-Return is a restricted diagnostic, not a comprehensive test of memory validity, and we will state this scope explicitly in the paper.

What it tests. Each eligible episode contrasts a departure answer with an absence-period challenger:

  • Old-Valid tests whether valid prior state is retained despite misleading updates.
  • Old-Stale tests whether prior state is rejected after a task change.

Together, they test whether admission follows current validity rather than information recency.

Why it is informative. It exposes cases in which answer correctness and state preservation diverge. In the main Qwen evaluation, Reset achieves 96.38% Overall but 0% VIA (Valid Information Availability), whereas TRACE achieves 94.32% Overall with 95.96% VIA and 98.78% IIR (Invalid Information Rejection). We therefore use ManBench-Return for a preservation–rejection analysis, not to claim uniformly better task accuracy.

Limitations. The two-candidate construction and the eligibility filtering limit generalization: every episode requires a correct departure answer, and Old-Stale additionally requires a correct majority among the active agents. These conditions do not capture the full difficulty of open-ended, long-horizon memory use. We will clarify the scope and the selection criteria in the paper. STALE and Memora provide complementary evidence on implicit invalidation and explicit updates, respectively, but do not eliminate this limitation.


We sincerely thank the reviewer for the careful reading and constructive comments, which have helped us clarify both the contribution of TRACE and the scope of our evaluation. We hope these responses address the reviewer's concerns and are happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.