- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of memory management in LLM agents. The proposed method measures the leave-one-out performance of each entry in the memory and keeps only entries with positive contribution. Results on several datasets are promising.
Critical strengths
- The paper studies a timely problem. However, the problem becomes less and less relevant as LLM getting more and more powerful.
- The proposed method, LAMARCK, is simple to implement. However, the problem setting is unrealistic and self-contradictory. More about this in the weakness section.
Critical weaknesses
- The paper is AI generated with broken language and structure. It is painful to read this kind of paper.
- The proposed method lacks novelty, which is typical for AI generated papers. Deciding to keep or discard an entry based on performance change on a sealed test set is a very old idea.
- The problem setting is unrealistic and contradictory. More concretely, the adoption/rejection rule is based on performance on a sealed static test set. In production systems, the environment evolves over time, so a static test set quickly becomes outdated. The authors might argue that their approach is valuable in a static environment, but a static environment doesn't need a memory system in the first place. Furthermore, sometimes the agent has to memorize something that hurts the test set performance because that's what the user likes. The example in the paper's introduction is actually an example of this scenario, and the paper didn't handle it correctly. I doubt that the human authors of this paper even read the manuscript before submitting.
Questions for authors
- Have you read what the AI generated?