- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the problem of memory management in agents that go offline/do alternate work while the conversation continues to grow and new memories are collected. Specifically, the paper studies how the interaction of these agents with stale memory should be formalized and how to best update the memory to keep the agent aware of the "delta" between when it left and returned. They propose TRACE, a training free method built on top of their formalization and show that it brings improvement in model performance on certain existing memory benchmarks while also showing failures on others.
Critical strengths
- The problem of studying drift in memory is crucial for several parallel fields such as personalization and continual learning. The problem is timely.
- The need for formalizing memory update is growing and tracking state changes is subjective and is heavily task dependent. Even though I am not satisfied with the details the authors present, the authors have picked a good and useful problem to solve.
Critical weaknesses
Methodology problems:
- There are several claims that the authors make that are not backed by citations and are never used as baselines in their experiments. For example, line 083 states that recency is not enough but they have not backing to this and do not provide this as a baseline either.
- It is unclear what is the motivation for the structure of equation 1. This connects to the point above that not enough research context is discussed and motivated in the paper for the formalization that is proposed
- Figure 2's example is never explained to the reader but it holds a lot of clarification information.
- From 212-215, it is unclear what "high recall scan" and "grounded verifier" mean. This is core to the method and not enough explanation is provided for these when the reader is reading with the flow of the paper.
- Even though this is discussed very slightly later, it is unclear what the individual size budget is in line 225.
- Equation 3 is not explained properly. What is the reason why this is framed this specific way? It is not grounded properly into the set theory claim the paper makes.
- It is unclear to me how this method's performance changes with different configs. In the main experiment, the authors use 4 acting agents and 1 returning agent and later perform ablations on the acting agents but keep this experiment restricted to one returning agent. It is unclear how the behavior changes if multiple agents are returning.
- The choice of baseline models is not clear. Why these specific models?
- The metrics are under-defined formally in the main paper. It is not clear what VIR, IIR, etc are - are these accuracies? F1 scores?
- The authors make claims without any reported statistical significance. The method seems to be sensitive to non-determinism of LLMs and significance needs to be studied
Writing style:
- The paper is not written coherently (potentially largely AI generated). The reader currently needs to keep a track of terminologies and sections frequently refer to details explained much later in the paper.
- Citations in the document are broken and do not follow the standard ICLR template.
- The abbreviations of metrics is not intuitive and frequently overwhelming to keep track of. For example, Why not use VSR for valid-state availability and ISR for invalid-state rejection instead of using VIR and IIR?
Questions for authors
Addressing the weaknesses above directly addresses my final score.
3 comments
Response to Reviewer Reviewer do4L
W7, W8 & W10. What do the experiments establish about generalization and reliability?
A3: Across models and policies. Qwen, Gemini, and DeepSeek test transfer across actor families, which supports generality within the tested models. The method baselines separately cover retaining, resetting, and updating memory.
Multiple returners. We agree that varying the active-team size with a single returner does not establish behavior with multiple returners. We have explicitly added simultaneous and staggered returns of multiple agents to Future Work.
Statistical reliability. Training-free operation does not eliminate LLM inference variability. Our current evaluation does not establish statistical significance, and cross-model consistency is not a substitute for uncertainty analysis. We will acknowledge this limitation and identify repeated runs and paired uncertainty analysis as follow-up work.
W9 & WS3 . How are Overall, VIA, and IIR defined?
A4: We apologize for the insufficient definitions.
VIA and IIR. VIA (Valid Information Availability) and IIR (Invalid Information Rejection) name the two complementary requirements of our return contract: preserving usable information and excluding inadmissible information. Both are success proportions, not F1 scores. They are organizing labels, not claimed standard metric acronyms.
Across benchmarks. We map each benchmark's native judgments to VIA/IIR, so shared labels do not imply identical scoring rules.
In the revision. The revised Metrics section explains Overall for Memora and ManBench-Return and gives explicit STALE VIA/IIR calculations; Appendix D.1 specifies the detailed definitions and aggregation. We appreciate the suggested alternatives and have clarified the metric names and definitions accordingly.
WS1 & WS2. How have we addressed writing clarity and citation formatting?
A5: We have revised the figure references and metric explanations and will clarify terminology at first use. We will also check the rendered citations against the ICLR template. If the reviewer could point us to the affected citations or locations, it would help us address the reported issues.
We sincerely thank the reviewer again for the careful reading and constructive feedback, which helped us improve the clarity of the formulation, method description, and evaluation. We hope these responses address the reviewer's concerns and are happy to provide further clarification.
Response to Rebuttal
- The original comment about lack of baselines still holds. The authors have not updated their results with any additional baselines (not even a random baseline)
- The terminology is still extremely hard for me to follow. For instance, "High-recall describes the design intent of the proposal stage, not a measured recall guarantee.", this is still unclear to me why this is called high-recall when there is a canonical usage policy for what recall means in prior literature.
- "A candidate does not have an independent budget" - how do we ensure fairness here if there is no fixed budget?
- The selection of parameters is still unclear. What drives and why have the authors not presented ablations for the number of returning agents?
- My concerns around the method seeming to be sensitive to non-determinism of LLMs is unaddressed. The authors have failed to provide significance numbers as well
In the current state of the paper, most of my concerns have simply been marked as future work or limitations instead of being properly rebutted and backed by strong empirical results. Due to this reason, I will keep my score unchanged.
Response to Reviewer Reviewer do4L
We thank the reviewer for the detailed feedback and for recognizing the importance of the problem. We apologize for the presentation difficulties and have grouped the concerns into five questions.
A1: Equation 1 separates relevance from admissibility; Equation 3 ensures that the admitted memories are jointly sufficient to resume work. Equation 1 is a problem formulation, not a theorem, and Equation 3 is implemented as a heuristic without a claimed guarantee.
Related work. The revised manuscript now discusses Generative Agents (Park et al., 2023), truth-maintenance systems (Doyle, 1979), and STALE (Chao et al., 2026). These works motivate checking a memory's premises beyond its recency; they do not substitute for a recency-only comparison.
Equation 1 (relevance vs. admissibility). A train recommendation can remain relevant to travel planning after the destination changes, yet no longer support the current trip. Admissibility therefore depends on the agent, the current state, the absence-period changes, and the obligations.
Equation 3 (joint sufficiency). Individually valid memories may still jointly provide insufficient information to resume work. Equation 2 defines the eligible records; Equation 3 selects a subset that covers obligations and critical dependencies within item-count and character limits while minimizing disclosure cost. Appendix C defines coverage through the union of supported dependencies: an obligation is covered only when all of its required dependencies are present. The implementation reserves mandatory records and then greedily adds uncovered support per unit cost. This is a constrained set-cover heuristic; we claim neither optimality nor a classical approximation guarantee. Insufficient coverage triggers reset or blocking.
A2: Pipeline. The revised Section 3 explicitly links its subsections to Figure 2: checkpoint capture (block 1), invalidation and validation (block 2), obligation coverage (block 3), and signing/private-memory readmission (block 4). Appendix C details the scanner and verifier and provides a worked example.
Scanner and verifier. The scanner proposes event–memory pairs whose dependencies may have changed; a proposal is not itself a rejection decision. The verifier then examines the original observations and their temporal order to determine whether later evidence invalidates the earlier item's current use. Topic overlap alone is insufficient: discussing trains need not invalidate an itinerary, whereas changing the destination may.
Terminology. "Grounded" refers to source observations, not benchmark answers. "High-recall" describes the design intent of the proposal stage, not a measured recall guarantee.
Size budgets. We agree that "individual size budget" was ambiguous. A candidate does not have an independent budget; each candidate must fit within the Return View's remaining item-count and cumulative character capacity. As introduced earlier in the paper, work-state records are selected first, and private memories require admitted source state for readmission. We will clarify this budget at first use.
(W7, W8 & W10, W9 & WS3, WS1 & WS2 continue in the next comment.)