- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a parametric memory management system, which aims to handle different requirements of retaining information in memory. For instance, some information is not valid after a short period of time and thus should be forgotten (e.g., “I had a bagel for breakfast should not remain valid even the next day”), but some should be retained much longer (e.g., “I work as a teacher should likely be relevant information for years”). The proposed system consists of two conceptual components, a controller to learn and enforce the memory retention policy and the LoRA-based memory substrate. Since real-user trajectories are not easily available, the authors essentially designed a simulator to create different types of memory tasks, as well as the dataset for evaluation. Experiments show that the proposed method balances the accuracy of the retrieved memory value (VAL) and when the system should abstain (ABS), compared with RAG and CL (continual learning) methods.
Critical strengths
- The motivation of this work is convincing -- addressing the needs of having different information retention policy with parametric memory
- The system design is novel -- while it borrows some existing components (e.g., FlyLoRA and LoRA-FA), they are somewhat repurposed and combined in a new way. Learning the policy for parametric memory also seems new.
Critical weaknesses
- The evaluation is done using only simulated scenarios and data. How effective the proposed design is to real-world problems still remains unknown. (The authors do state that the dataset is “a controlled stress test” and acknowledge the coverage is limited. In addition, Appendix D studies the effects of using natural language queries instead of templated ones, which shows a large performance drop.)
Questions for authors
- Should long-lived (STABLE) facts have expiration dates? Looking at Table 10, the “unmanaged d” baselines achieved much higher scores on VAL but low ABS scores (as they don’t easily forget).
- What are the results of trivial baselines (e.g., always abstain and always answer)?
- The ground-truth validity is generated by a fixed decay-horizon rule (App. C, Fig. 4). How would results change if validity were determined by a different process (e.g., invalidated by a different event)?
We wanted to follow up on our rebuttal to check if you have any remaining concerns. We would be happy to provide any further clarifications/details required. We would also appreciate if you could consider updating your recommendation if your concerns are resolved. Thank you for taking the time to review our work.