- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on whether the long-term memory baseline can distinguish the system that can maintain and compose persistent states from simply retrieve information from historical records. It contains low-cost auditing method, LLM judge and API calls with several metrics, such as the proportion of gold answers, the remaining answer-coverage gap, and the minimum budget needed to reduce the gap below a predefined threshold. Experiments show that more than half of the checkable answers in the knowledge-update dimension appear in individual records.
Critical strengths
This paper does not propose a new memory framework, but investigate whether the baseline have enough construct validity, which is very important for rapidly developed LLM memory research. This method can be easily adapted into existing benchmarks, because the metrics V and Rstr are very simple without any inference of LLM. This paper remains the unsuccessful hypothesis and unrecognizable results, which is highly recommended.
Critical weaknesses
The condition of proposition 1 is too general. The paper proposes that If the answer is the function of write record, the index-only system can reach to the upper bound, which contradicts the retrieval budget constraint established in the paper. The proposed metrics are adaptable to belief questions with clear and definite answers, which is too limited. Because many tasks do not have clear answers, this paper can only be deployed in few benchmarks. The structure of this paper is so complex, which contains many preregistered criteria and analysis that obscure the core contributions and conclusions. The metric Rstr uses absolute value instead of normalization, which may lead to unfair comparison between different benchmarks. The experiments are highly relied on BM25 and last-occurence reader, without sufficiently evaluating iterative retrieval methods.
Questions for authors
Please see the weakness above to explain some details in your paper clearly. Moreover, some questions are listed as follows: If the same answer appears in many events, how to determine which events provide correct evidence? Does the answer necessarily means that the event contains sufficient information to answer the question? Please mention the length threshold used to define “short gold” answer. The conclusion of this paper is stable if the threshold is changed? What text procedures are used to calculate V?