- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Many existing works for long-term memory benchmarks organize evaluation around pre-defined case types and failure modes. This makes it difficult to identify new failure modes or capture novel failure requirements exhibited by current SOTA memory systems. In this work the authors propose MEMEVOBENCH, a dynamic evaluation fraamework that iteratively generates and validates test cases, evaluates multiple memory systems, and uses their failures to propose additional case types that guide subsequent test generation. For instance, during experiments, the authors start with a seeded pool of six user histories and four core case types and their framework across ten rounds produces ~1.5k questions and admits 31 additional type cards (grouped into 11 consolidated categories). The authors also evaluate multiple memory systems on their benchmark and find that their procedure distinguishes between the evaluated systems on more than half of its questions while remaining challenging overall.
Critical strengths
-
The paper identifies a very timely problem and the core motivation of using dynamic benchmarking for long term memory evaluation. I agree with the core motivation that standard benchmarks with pre-determined task types can miss many different failure modes that may happen in practice.
-
The construction pipeline is quite comprehensive with both factuality validation and question intent checks baked in. The process of evidence first generation, blind re-answering, and additional validation gates address provide safeguards at many levels.
Critical weaknesses
-
The core weakness in the current draft I find is that the human audit does not sufficiently validate the full automated pipeline. The majority of the framework relies on automated judges for validation, grading, type admission, etc. The human audit primarily reports aggregate case-quality and answer-grading results, alongside endorsements of the consolidated types. Although the authors report unanimous agreement on case validity and human-LLM agreement on answer correctness, they do not provide chance-corrected inter-rater agreement, human-human agreement on answer correctness, or quality breakdowns by user and case type.
-
I also have concerns on the scalability of the method given that the construction process may bias coverage toward the weaknesses of the systems supplying feedback towards coverage of different systems it interacts with during the feedback generation process. While the authors show that the evolved cases also expose failures in held-out systems such as MemU and A-Mem, this does not fully establish robustness to the choice of feedback systems as well as how one would scale if there were say >10 memory systems in consideration. A leave-one-system-out construction experiment, or repeated construction using different system pools, would help quantify this dependence.
Questions for authors
- Human validation: Could you provide more details on the reviewer qualifications, independence, binding alongside chance corrected inter-rater agreement, etc.
- Could you clarify how you distinguish genuinely new evaluation requirements from finer subdivisions of existing ones? Did human reviewers assess distinctness between consolidated types and novelty relative to the seed taxonomy?
- How sensitive are the discovered types and resulting comparisons to the choice of the four feedback systems?
3 comments
Response to Reviewer xpCq (PartⅡ)
Q2. Novelty of Consolidated Case Types
Thank you for raising this question. We compared each of the 11 consolidated case types in Appendix B with the four seed type specifications in Appendix C.2, focusing on evidence structure, reasoning operations, and answer verification rules. A refinement makes an existing evaluation requirement more specific. A combination tests multiple existing requirements in a single task. An extension brings evaluation scenarios outside the coverage of the seed types into the benchmark.
| Consolidated case type | Corresponding seed type(s) | Relationship to the seed specifications |
|---|---|---|
| CT01 Multi-constraint fact retrieval | Single-hop | Refinement. Retrieve a directly stated fact that satisfies all query constraints, such as entity, time, amount, and purpose, while excluding records that match only some constraints. |
| CT02 Temporal version reconstruction | Temporal | Refinement. Recover the attribute value that was valid at the specified time, such as the name of a project before it was renamed. |
| CT03 Evidence-grounded numerical reasoning | — | Extension. Select numerical records that satisfy the query constraints and compute a result that is not directly stated in the history. |
| CT04 Set-presupposition validation | Adversarial | Refinement. Check whether the evidence supports the membership or cardinality presupposed by the question. For example, if the history lists three options and the question asks for two, identify that the evidence provides no basis for selecting a particular pair. |
| CT05 Complementary evidence synthesis | Multi-hop | Refinement. Combine indispensable evidence that supports different parts of the answer while preserving the scope and qualifications of each piece of evidence. |
| CT06 Bounded trajectory reconstruction | Temporal + Multi-hop | Combination. Integrate evidence within a specified topic and time period to reconstruct intentions, actions, outcomes, and retrospective assessments in chronological order. |
| CT07 Bounded set reconstruction | Single-hop / Multi-hop | Refinement. Recover the complete membership of a specified set from a single record or multiple indispensable pieces of evidence, checking for omissions and extraneous members. |
| CT08 Entity–relation binding verification | Adversarial | Refinement. Make the existing attribution check more specific, for example by verifying whether a preference belongs to the user or to someone on whose behalf the user asked a question. |
| CT09 Interaction-occurrence verification | — | Extension. Use a subsequent response that uniquely entails a communicative act as evidence that the act occurred, and distinguish occurrence, non-occurrence, and insufficient evidence. |
| CT10 Conflicting-assertion resolution | — | Extension. Use explicit corrections or confirmations to resolve conflicting assertions about the same fact. When the evidence cannot resolve the conflict, preserve the disagreement and report the details that remain supported. |
| CT11 Value-evidence sufficiency | — | Extension. Evaluate whether the system gives the most specific answer supported by the evidence. For example, if the history supports "late June," the answer should preserve that range and indicate that the exact date is unknown. The seed adversarial specification explicitly excludes cases constructed solely by requesting an unrecorded degree of precision. |
In Appendix G, three human assessors independently evaluated within-type coherence by checking whether the Type Cards grouped under each consolidated type shared stable evidence requirements and reasoning operations. The distinctions between consolidated types and the requirements they add relative to the seed types were assessed through the comparison of the specifications in Appendices B and C.2 shown above.
Response to Reviewer xpCq (PartⅢ)
Q3. Sensitivity to Feedback-System Selection
Thank you for this suggestion. We carefully considered the proposed leave-one-system-out experiments and the idea of repeating benchmark construction with different pools of feedback systems. We agree that these experiments would help quantify how the choice of feedback systems affects the case types discovered and the resulting benchmark coverage. Given the limited time available during the rebuttal period, we plan to conduct these experiments in future work and examine how construction costs and benchmark coverage change as the number of feedback systems increases.
Response to Reviewer xpCq (PartⅠ)
Dear Reviewer xpCq,
Thank you for your valuable comments and suggestions. We appreciate your thoughtful feedback. Our point-by-point responses are provided below.
Q1. Human Validation and Agreement
We appreciate your valuable suggestions. All three annotators had experience in large language model evaluation or question-answering data annotation. They recorded their judgments independently, and final human labels were determined by majority vote. When evaluating system responses, the annotators had access to the question, supporting evidence, reference answer, and system response. During independent assessment, system identities, the LLM judge’s labels, and the other annotators’ judgments were hidden.
For the case validity audit, we drew a stratified sample of 220 cases from the 655 cases used for type analysis. The sample covered all six users and all 11 consolidated case types, with 20 cases per type. The annotators checked whether each question was answerable from the user history, whether the supporting evidence was sufficient, whether the reference answer was correct, whether the case matched its assigned type, and whether the question revealed the answer. Before majority voting, all three annotators agreed on the validity judgment for 198 cases (90.0%). After majority voting, 209 cases met all quality criteria (95.0%).
For the system answer correctness audit, the annotators evaluated 836 responses from four memory systems across the 209 valid cases, using the reference answers and supporting evidence. The majority-vote human labels agreed with the LLM judge’s labels on 784 responses (93.8%). Appendix G reports both agreement rates as raw percentages.