- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the interaction between generation instructions and scoring rules in long-term memory evaluation. The authors use four retrieval/memory backends and a fixed answer model, Qwen2.5-3B-Instruct, on LoCoMo. Within each backend, they hold questions, retrieved context, and decoding settings fixed while comparing different generation prompts. For unanswerable questions, human annotations distinguish changes in actual abstention behavior from changes in recognition by the keyword scorer. For answerable questions, token F1 increases while semantic correctness relative to the reference answers decreases, a finding examined using GPT-4o and a human-annotated sample. The paper also compares the benefits and errors of an expanded keyword rule and a semantic abstention judge. It recommends disclosing generation and scoring protocols together and reporting answer correctness and abstention separately.
Critical strengths
1. The research question has practical relevance. A memory system's final score depends on the retrieved information, how the model answers, and the scoring rules. The paper emphasizes specifying generation and scoring protocols when comparing systems and distinguishing answer correctness from abstention performance. This can help improve comparability across evaluations.
2. Controlled comparisons and human annotations help distinguish behavior from measurement. Within each backend, the authors fix the questions, answer model, retrieved evidence, and decoding settings, reducing the influence of retrieval changes on prompt comparisons. The unanswerable-question study reports both lexical scores and human-judged abstention rates. The answerable-question study also provides human checks on a matched sample, rather than relying solely on one automated metric.
3. The paper is transparent about the limits of its evidence and corrections to its analysis, and provides data for verification. It explicitly acknowledges the limitations of a single answer model, a specific benchmark, and the semantic judge. It discloses the correction to the statistical implementation, distinguishes descriptive rankings from statistical significance, and does not interpret the behavior–measurement gap directly as the proportion of score improvement that is spurious. The supplementary material provides raw responses, retrieved context, human labels, judge outputs, and analysis code, facilitating verification of the main results.
Critical weaknesses
1. Evaluation is limited to LoCoMo, with insufficient evidence across benchmarks. The controlled experiments cover only one long-term memory benchmark, and the central abstention finding is directly related to its specific two-phrase scoring rule. It remains unclear whether a similarly pronounced divergence between generation behavior and scoring would occur on other memory benchmarks with different question types, answer formats, and scoring protocols. I would like to see the key findings validated on at least one additional benchmark to distinguish defects in LoCoMo's particular implementation from broader evaluation problems.
2. The selection of answer models is insufficient. All controlled experiments use Qwen2.5-3B-Instruct. Although the four retrieval/memory backends provide different sources of context, they do not establish whether different answer models exhibit similar behavior. The effects of prompts on actual abstention and answer correctness may depend on model capability and instruction following. I would therefore like to see key comparisons using models from different families or at different capability levels. The authors acknowledge the single-model limitation, but experiments are still needed to assess its actual impact on the main findings.
3. The contribution beyond known metric limitations needs to be strengthened. For unanswerable questions, the prompt explicitly specifies the abstention phrases accepted by the keyword scorer, making a substantial increase in lexical credit largely predictable. Using human annotations to distinguish actual behavioral changes from changes in recognition is more valuable than simply demonstrating score increases. The supplementary material further shows that, on the repair test set, the P0′→P1 score difference decreases from approximately 0.693 under the two-phrase rule to 0.196 under the semantic judge, closer to the human estimate of 0.264. This supports the scoring repair and highlights the need to distinguish defects of the particular lexical rule from problems that remain after repair. I would still like to see whether these issues materially affect system comparisons under reasonable, standardized generation and scoring protocols. The divergence between token F1 and semantic correctness on answerable questions provides additional evidence, but its contribution beyond the known limitations of lexical metrics also needs clarification.
4. The decline in correctness needs a more concrete analysis. The supplementary material provides raw responses and aggregate analyses of answer length, abstention, and correctness, but a more direct explanation of how response quality deteriorates is still needed. Pairing the supplied human labels by question reveals that each annotator identifies 10 cases changing from CORRECT to INCORRECT and four changing from CORRECT to ABSTAINED. These changes cannot be inferred from net changes in label counts alone. I would like the paper to present representative cases and paired transitions, explaining the specific errors and how F1 changes for these responses.
Questions for authors
1. Validation across benchmarks. Could the authors repeat the key comparisons on another long-term memory benchmark, particularly to test whether the main findings hold under a scoring protocol that does not rely on the two-phrase abstention rule?
2. Validation across answer models. Could the authors include answer models from different families or at different capability levels to assess whether the changes in actual abstention, scorer recognition, and the divergence between F1 and correctness remain consistent?
3. Effects on system comparisons under standardized protocols. The supplementary material shows that the prompt difference on the abstention task becomes substantially smaller under the semantic judge. Could the authors further explain how much the remaining issues would change backend comparisons or other practical evaluation conclusions under reasonable, standardized generation and scoring protocols?
4. What specifically accounts for the decline in correctness? The raw responses and human labels in the supplementary material support further analysis. Could the paper present representative response pairs changing from CORRECT to INCORRECT and from CORRECT to ABSTAINED, explaining the specific error types and how F1 changes for these responses?
5. Disagreements over semantic correctness. In Table 8, annotator A labels 51 responses INCORRECT that annotator B labels CORRECT, while only two disagreements occur in the opposite direction. This indicates a pronounced directional difference in their correctness judgments. Although both annotators' paired effects have the same direction, which is also preserved in the consensus-subset analysis, representative disputed examples would still be useful. Are these disagreements concentrated in short answers, partially correct answers, or responses containing additional information? An independent review could help assess whether these judgment boundaries are related to the prompt condition.
6. Backend construction and context budgets. The supplementary material identifies all-MiniLM-L6-v2 as the dense embedding model and provides retrieval results and context. To facilitate regenerating these contexts, please further specify the chunking and retrieval configurations for BM25/dense retrieval, as well as memory construction, key configurations, and context budgets for Mem0/MemoryOS, or provide the corresponding construction scripts.