acceptodds
Under review as a conference paper at ICLR 2027

Generation–Evaluation Coupling in LoCoMo: A Construct-Validity Audit of Long-Term Memory Evaluation

Abstract

Long-term-memory benchmark scores can depend on both answer-generation instructions and how evaluators recognize the resulting behavior. We audit this interaction in LoCoMo with one fixed answer reader; within each of four retrieval/memory backends, questions, retrieved evidence, and decoding are held fixed across prompt conditions. Changing the generation prompt increases our archived all-item score by 0.214 on average. On unanswerable questions, a targeted change in the abstention instruction increases lexical credit by 66.7 percentage points while blinded human-judged abstention increases by 25.9 points, leaving a 40.8-point measurement gap (95% CI: 34.7–46.8). In the broader comparison between a neutral and a benchmark-aligned prompt on answerable questions, archived token F1 increases while GPT-4o reference-answer correctness decreases by 5.26 percentage points; a stratified human study on 240 paired units shows the same direction on the matched sample. A semantic abstention judge reduces lexical under-recognition but introduces false positives. These findings show that generation instructions can change both response behavior and its measurement. We recommend specifying generation and scoring protocols jointly and reporting answer correctness and abstention separately before interpreting aggregate gains as improvements in memory capability.

Primary area:
datasets and benchmarks
Keywords:
long-term memory, LLM agents, benchmark evaluation, construct validity, LoCoMo, semantic evaluation, abstention, generation-evaluation coupling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TDtFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the interaction between generation instructions and scoring rules in long-term memory evaluation. The authors use four retrieval/memory backends and a fixed answer model, Qwen2.5-3B-Instruct, on LoCoMo. Within each backend, they hold questions, retrieved context, and decoding settings fixed while comparing different generation prompts. For unanswerable questions, human annotations distinguish changes in actual abstention behavior from changes in recognition by the keyword scorer. For answerable questions, token F1 increases while semantic correctness relative to the reference answers decreases, a finding examined using GPT-4o and a human-annotated sample. The paper also compares the benefits and errors of an expanded keyword rule and a semantic abstention judge. It recommends disclosing generation and scoring protocols together and reporting answer correctness and abstention separately.

Critical strengths

1. The research question has practical relevance. A memory system's final score depends on the retrieved information, how the model answers, and the scoring rules. The paper emphasizes specifying generation and scoring protocols when comparing systems and distinguishing answer correctness from abstention performance. This can help improve comparability across evaluations.

2. Controlled comparisons and human annotations help distinguish behavior from measurement. Within each backend, the authors fix the questions, answer model, retrieved evidence, and decoding settings, reducing the influence of retrieval changes on prompt comparisons. The unanswerable-question study reports both lexical scores and human-judged abstention rates. The answerable-question study also provides human checks on a matched sample, rather than relying solely on one automated metric.

3. The paper is transparent about the limits of its evidence and corrections to its analysis, and provides data for verification. It explicitly acknowledges the limitations of a single answer model, a specific benchmark, and the semantic judge. It discloses the correction to the statistical implementation, distinguishes descriptive rankings from statistical significance, and does not interpret the behavior–measurement gap directly as the proportion of score improvement that is spurious. The supplementary material provides raw responses, retrieved context, human labels, judge outputs, and analysis code, facilitating verification of the main results.

Critical weaknesses

1. Evaluation is limited to LoCoMo, with insufficient evidence across benchmarks. The controlled experiments cover only one long-term memory benchmark, and the central abstention finding is directly related to its specific two-phrase scoring rule. It remains unclear whether a similarly pronounced divergence between generation behavior and scoring would occur on other memory benchmarks with different question types, answer formats, and scoring protocols. I would like to see the key findings validated on at least one additional benchmark to distinguish defects in LoCoMo's particular implementation from broader evaluation problems.

2. The selection of answer models is insufficient. All controlled experiments use Qwen2.5-3B-Instruct. Although the four retrieval/memory backends provide different sources of context, they do not establish whether different answer models exhibit similar behavior. The effects of prompts on actual abstention and answer correctness may depend on model capability and instruction following. I would therefore like to see key comparisons using models from different families or at different capability levels. The authors acknowledge the single-model limitation, but experiments are still needed to assess its actual impact on the main findings.

3. The contribution beyond known metric limitations needs to be strengthened. For unanswerable questions, the prompt explicitly specifies the abstention phrases accepted by the keyword scorer, making a substantial increase in lexical credit largely predictable. Using human annotations to distinguish actual behavioral changes from changes in recognition is more valuable than simply demonstrating score increases. The supplementary material further shows that, on the repair test set, the P0′→P1 score difference decreases from approximately 0.693 under the two-phrase rule to 0.196 under the semantic judge, closer to the human estimate of 0.264. This supports the scoring repair and highlights the need to distinguish defects of the particular lexical rule from problems that remain after repair. I would still like to see whether these issues materially affect system comparisons under reasonable, standardized generation and scoring protocols. The divergence between token F1 and semantic correctness on answerable questions provides additional evidence, but its contribution beyond the known limitations of lexical metrics also needs clarification.

4. The decline in correctness needs a more concrete analysis. The supplementary material provides raw responses and aggregate analyses of answer length, abstention, and correctness, but a more direct explanation of how response quality deteriorates is still needed. Pairing the supplied human labels by question reveals that each annotator identifies 10 cases changing from CORRECT to INCORRECT and four changing from CORRECT to ABSTAINED. These changes cannot be inferred from net changes in label counts alone. I would like the paper to present representative cases and paired transitions, explaining the specific errors and how F1 changes for these responses.

Questions for authors

1. Validation across benchmarks. Could the authors repeat the key comparisons on another long-term memory benchmark, particularly to test whether the main findings hold under a scoring protocol that does not rely on the two-phrase abstention rule?

2. Validation across answer models. Could the authors include answer models from different families or at different capability levels to assess whether the changes in actual abstention, scorer recognition, and the divergence between F1 and correctness remain consistent?

3. Effects on system comparisons under standardized protocols. The supplementary material shows that the prompt difference on the abstention task becomes substantially smaller under the semantic judge. Could the authors further explain how much the remaining issues would change backend comparisons or other practical evaluation conclusions under reasonable, standardized generation and scoring protocols?

4. What specifically accounts for the decline in correctness? The raw responses and human labels in the supplementary material support further analysis. Could the paper present representative response pairs changing from CORRECT to INCORRECT and from CORRECT to ABSTAINED, explaining the specific error types and how F1 changes for these responses?

5. Disagreements over semantic correctness. In Table 8, annotator A labels 51 responses INCORRECT that annotator B labels CORRECT, while only two disagreements occur in the opposite direction. This indicates a pronounced directional difference in their correctness judgments. Although both annotators' paired effects have the same direction, which is also preserved in the consensus-subset analysis, representative disputed examples would still be useful. Are these disagreements concentrated in short answers, partially correct answers, or responses containing additional information? An independent review could help assess whether these judgment boundaries are related to the prompt condition.

6. Backend construction and context budgets. The supplementary material identifies all-MiniLM-L6-v2 as the dense embedding model and provides retrieval results and context. To facilitate regenerating these contexts, please further specify the chunking and retrieval configurations for BM25/dense retrieval, as well as memory construction, key configurations, and context budgets for Mem0/MemoryOS, or provide the corresponding construction scripts.

Reviewer_uiqvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether score gains on LoCoMo, a long-term-memory benchmark, reflect better answers or only answers that better match the scoring rule, when only the generation prompt changes.

The authors use one answer reader, Qwen2.5-3B-Instruct, with four retrieval/memory backends: BM25, dense retrieval, Mem0, and MemoryOS. Within each backend, the questions, retrieved evidence, and decoding are fixed; only the system prompt changes. P0 is a neutral prompt. P1 is a benchmark-aligned prompt: it asks for short answers and tells the model to answer "not mentioned" when the answer is not in the context. P0′ keeps P1's other instructions but does not give the two phrases that the scorer accepts. On unanswerable questions (Category 5), P0′ and P1 are compared using the released lexical rule and blinded human labels of abstention. On answerable questions, P0 and P1 are compared using token F1 and GPT-4o reference-answer correctness, with a 240-unit human sample as a check.

On unanswerable questions, lexical abstention credit rises by about 67 percentage points from P0′ to P1, but human-judged abstention rises by only about 26 points. On answerable questions, token F1 rises while reference-answer correctness falls, and the human sample shows the same direction. A GPT-4o abstention judge recognizes more abstentions than the lexical rule, but it adds false positives. The authors conclude that generation prompts and scoring rules should be reported together, and that answer correctness and abstention should be reported separately.

Critical strengths
  1. The controlled design is clean. Within each backend, the reader, decoding, questions, and retrieved evidence are fixed, so the score change can be attributed to the prompt.

  2. Blinded human labels are used as a reference to separate abstention behavior from lexical recognition, instead of relying only on automatic metrics.

Critical weaknesses
  1. The paper is hard to read and follow. The main text spends much space on process details (e.g., hash freezes, package amendments, archived scorer ports) and uses many newly defined terms (e.g., generation contract, archived local token F1, construct-separated reporting contract). These make the paper difficult to understand. At the same time, important information is left to the appendix. For example, the main metric (Eq. 1) depends on F_ans, but how F_ans is computed is only given in Appendix A. The main text ends early on page 7, so there is room to explain this in the main text.

  2. The experiments are limited. There is only one reader, a small 3B model with low correctness (GPT-4o CORRECT rate of about 31% under P0), and only one benchmark, so it is unclear whether the findings hold for stronger readers or other benchmarks. The evaluator-repair part is also limited: it tests one fixed phrase list and one GPT-4o judge, and the judge's high false-positive rate (0.242 on natural outputs) is reported but not analyzed, e.g., which responses it wrongly labels as abstentions.

Questions for authors

See weakness.

Reviewer_XReHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper feels entirely AI generated and is very difficult to follow. Here's my best attempt at a summary: the paper looks to analyze a single benchmark LoCoMo trying to figure out how each component relates to final benchmark scores. The majority of the takeaway seems to involve abstention where these are not appropriately measured (e.g. not given enough credit when abstaining). Another claim they have is that generation instructions can change both the response behavior and measurement. This seems very obvious to me; why would language models be invariant to instructions?

Critical strengths

None; I found this paper difficult to parse.

Critical weaknesses
  1. Paper is very difficult to follow and reads entirely AI generated.
  2. I'm not sure any of the supposed claims are not things already widely understood.
  3. No discussion of robustness to input perturbations in related work or calibration literature is even considered.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.