- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the core challenges of long-term large-scale model agents in recognizing stale facts and failing to fully comply with deletion and access control in cross-session memory. However, key experiments revealed issues such as low task success rates and inconsistencies in some comparison protocols.
Critical strengths
- The paper points out the fundamental flaw of traditional RAGs in conflating relevance with admissibility and proposes a new paradigm of recall-as-execution.
- The paper proposes a relatively complete and unified memory execution framework, integrating memory writing, state updating, time querying into a single execution framework.
Critical weaknesses
- Section 4.2, Table 3, and Figure 3 show that TRACE performs exceptionally well in systems with an observable complete memory lifecycle, but the 4.0% task success rate indicates that its capabilities remain limited. Meanwhile, non-memory agent controls such as claude code and codex achieved higher absolute results in task success rate and StrictCore. Although the authors note that these systems are not direct controls with the same memory lifecycle, these results still limit TRACE's claims to practical applicability.
- The overall accuracy and compliance of the system described in the paper are highly vulnerable to the absolute precision of the front-end LLM compiler in parsing natural language semantics. In complex real-world dialogues, if the compiler misjudges operation types, misplaces entity relationships, or misses sensitive attributes, even if the execution engine runs perfectly, it will fall into the predicament of "garbage in, deterministic garbage out." The paper, however, merely simulates missing propositions by randomly discarding them, thus avoiding the fatal damage that such errors in understanding complex semantics can cause to the discrete state machine.
- The paper uses the “DELETE” instruction to perform logical deletion and designs the entire ledger as append-only. However, sensitive data after deletion remains in the original ledger, and other components can still potentially access this data as long as they can bypass the execution engine to access the underlying ledger.
- The paper points out that TRACE's main advantage over traditional RAG lies in its ability to handle memory updates, deletions, and temporal state changes. However, traditional similarity retrieval does not inherently perform these operations. Therefore, TRACE's superiority over Dense-RAG or Lexical-RAG only indicates that explicit state parsing is better suited for handling state update tasks than simple text retrieval; it does not sufficiently prove that TRACE's specific execution framework is superior to other structured state management schemes.
Questions for authors
- Despite TRACE's excellent lifecycle compliance, its absolute task success rate on AUTO MEMORY BENCH is only 4.0%. How do the authors explain the contradiction between this extremely low task completion rate and the actual effectiveness claimed by the proposed memory framework? Could the authors provide a detailed failure attribution analysis to clearly distinguish between memory compilation errors, recall failures, and downstream task execution failures?
- The current experiments primarily compare TRACE with retrieval methods or materialized state baselines that only store the latest state. Since the lack of historical versions itself leads to failed history queries, could the authors add versioned state management or event sourcing methods that also support time-based queries and access control? Compared to traditional state replay mechanisms, what are the unique advantages of TRACE?
- The paper mentions that TRACE uses logical deletion markers to achieve forgetting, but the original data remains in the immutable ledger. Furthermore, history replay could potentially recover subsequently deleted information. How do the authors prevent sensitive information from being re-exposed through historical or counterfactual queries? Can a deletion strategy that enforces deletion at all query times be evaluated, and the trade-off between this strategy and historical auditability analyzed?