- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a failure mode of lifelong personalized agents: retaining historically accurate memories can hinder adaptation when user preferences change. It introduces ITCHBENCH, a benchmark covering stationary and nonstationary preference trajectories, and proposes Finitude-Aware Memory (FAM), which separates archival retention from the influence of memories on current decisions. FAM combines learned validity estimates with relevance-based retrieval, protects commitments and constraints, and constructs bounded states for memory handoff. The paper also provides a theoretical analysis of the trade-off between estimation variance and stale-information bias. Experiments across six language-model backbones report improved adaptation and a favorable balance between plasticity and continuity.
Critical strengths
The paper addresses an important problem in long-term agent memory: historically accurate information can become misleading when user preferences change. Evaluating adaptation alongside continuity, including transient and context-dependent changes, provides a useful perspective on memory management.
Critical weaknesses
- Insufficient reproducibility of the benchmark and method. ITCHBENCH is a central contribution, yet the materials available for review provide no accessible release of the dataset, generator, evaluation code, or FAM implementation. Although Appendices B–C describe the framework, essential details remain insufficiently specified, including trajectory-level data partitioning, the controller’s training objective and optimization settings, feature extraction, and the construction and scoring of evaluation probes. The reference to results/drift_family_summary.csv does not resolve this issue because the file itself is unavailable. These omissions substantially limit independent verification of the benchmark and reported improvements. An accessible implementation or a sufficiently complete specification is needed.
- The interpretation of the theoretical calibration requires clarification. Theorem 2 provides a finite-window risk upper bound, and Appendix A derives the retention rule by minimizing that bound. However, Figure 3c and Table 5 compare the resulting scaling exponents with those of the empirically optimal lifetime. The upper bound alone does not establish this correspondence. The authors should clarify the process assumptions and interpretation of the calibration, and distinguish the proposed tuning rule from a claim about the actual optimal window across different drift processes.
Questions for authors
- Section 4 states that methods use a matched 4,096-token active-memory budget, whereas Table 13 reports 24,500 tokens for full history. How should these figures be reconciled, particularly given the matching accuracy results in Tables 2 and 13? Does the budget include protected information and handoff states?
- In Figure 3b, does introducing stale information also remove current evidence? Can current evidence and total prompt length be held fixed while comparing obsolete contradictory statements, neutral distractors, and explicitly time-qualified historical facts? This would help distinguish stale-memory interference from the effect of reduced current evidence.
- Which controller inputs, such as attribute identifiers, contextual scopes, confidence estimates, and protection flags—are inferred from dialogue, and which are supplied by the benchmark’s latent annotations? Are all three renderings of each trajectory assigned to the same split, with future-consistency supervision restricted to training trajectories?
- Since Equation 5 makes an episode’s validity non-increasing, how can retired memories become usable again during cyclic changes or preference reversals? What precisely constitutes “renewal” and “rollback” in Table 12, and how are baseline handoff states constructed under comparable capacity constraints?