acceptodds
Under review as a conference paper at ICLR 2027

MemEvoBench: Feedback-Guided Case-Type Evolution for Long-Term Memory Evaluation

Abstract

Existing long-term memory benchmarks typically organize evaluation around predefined case types, making it difficult to incorporate requirements exposed by observed system failures. We introduce MemEvoBench, a feedback-guided benchmark evolution framework grounded in de-identified multi-session user dialogues. Each case type specifies a target memory behavior and executable rules for evidence discovery, case generation, and verification. The framework uses validated cross-system evaluation feedback to evolve these specifications and incorporates admitted types into subsequent case mining, turning instance-level failures into reusable evaluation requirements. In a ten-round study over six user histories, MemEvoBench constructed 1,553 evidence-grounded cases and expanded four seed types into 11 consolidated types. The evolved types exposed failures both in systems used during evolution and in additional systems that supplied no feedback. These findings support feedback-guided case-type evolution as a complement to predefined memory benchmarks.

Primary area:
datasets and benchmarks
Keywords:
memory, evolution, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_xpCqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Many existing works for long-term memory benchmarks organize evaluation around pre-defined case types and failure modes. This makes it difficult to identify new failure modes or capture novel failure requirements exhibited by current SOTA memory systems. In this work the authors propose MEMEVOBENCH, a dynamic evaluation fraamework that iteratively generates and validates test cases, evaluates multiple memory systems, and uses their failures to propose additional case types that guide subsequent test generation. For instance, during experiments, the authors start with a seeded pool of six user histories and four core case types and their framework across ten rounds produces ~1.5k questions and admits 31 additional type cards (grouped into 11 consolidated categories). The authors also evaluate multiple memory systems on their benchmark and find that their procedure distinguishes between the evaluated systems on more than half of its questions while remaining challenging overall.

Critical strengths
  • The paper identifies a very timely problem and the core motivation of using dynamic benchmarking for long term memory evaluation. I agree with the core motivation that standard benchmarks with pre-determined task types can miss many different failure modes that may happen in practice.

  • The construction pipeline is quite comprehensive with both factuality validation and question intent checks baked in. The process of evidence first generation, blind re-answering, and additional validation gates address provide safeguards at many levels.

Critical weaknesses
  • The core weakness in the current draft I find is that the human audit does not sufficiently validate the full automated pipeline. The majority of the framework relies on automated judges for validation, grading, type admission, etc. The human audit primarily reports aggregate case-quality and answer-grading results, alongside endorsements of the consolidated types. Although the authors report unanimous agreement on case validity and human-LLM agreement on answer correctness, they do not provide chance-corrected inter-rater agreement, human-human agreement on answer correctness, or quality breakdowns by user and case type.

  • I also have concerns on the scalability of the method given that the construction process may bias coverage toward the weaknesses of the systems supplying feedback towards coverage of different systems it interacts with during the feedback generation process. While the authors show that the evolved cases also expose failures in held-out systems such as MemU and A-Mem, this does not fully establish robustness to the choice of feedback systems as well as how one would scale if there were say >10 memory systems in consideration. A leave-one-system-out construction experiment, or repeated construction using different system pools, would help quantify this dependence.

Questions for authors
  • Human validation: Could you provide more details on the reviewer qualifications, independence, binding alongside chance corrected inter-rater agreement, etc.
  • Could you clarify how you distinguish genuinely new evaluation requirements from finer subdivisions of existing ones? Did human reviewers assess distinctness between consolidated types and novelty relative to the seed taxonomy?
  • How sensitive are the discovered types and resulting comparisons to the choice of the four feedback systems?
3 comments
AuthorsOpenReview ↗

Response to Reviewer xpCq (PartⅠ)

Dear Reviewer xpCq,

Thank you for your valuable comments and suggestions. We appreciate your thoughtful feedback. Our point-by-point responses are provided below.

Q1. Human Validation and Agreement

We appreciate your valuable suggestions. All three annotators had experience in large language model evaluation or question-answering data annotation. They recorded their judgments independently, and final human labels were determined by majority vote. When evaluating system responses, the annotators had access to the question, supporting evidence, reference answer, and system response. During independent assessment, system identities, the LLM judge’s labels, and the other annotators’ judgments were hidden.

For the case validity audit, we drew a stratified sample of 220 cases from the 655 cases used for type analysis. The sample covered all six users and all 11 consolidated case types, with 20 cases per type. The annotators checked whether each question was answerable from the user history, whether the supporting evidence was sufficient, whether the reference answer was correct, whether the case matched its assigned type, and whether the question revealed the answer. Before majority voting, all three annotators agreed on the validity judgment for 198 cases (90.0%). After majority voting, 209 cases met all quality criteria (95.0%).

For the system answer correctness audit, the annotators evaluated 836 responses from four memory systems across the 209 valid cases, using the reference answers and supporting evidence. The majority-vote human labels agreed with the LLM judge’s labels on 784 responses (93.8%). Appendix G reports both agreement rates as raw percentages.

AuthorsOpenReview ↗

Response to Reviewer xpCq (PartⅡ)

Q2. Novelty of Consolidated Case Types

Thank you for raising this question. We compared each of the 11 consolidated case types in Appendix B with the four seed type specifications in Appendix C.2, focusing on evidence structure, reasoning operations, and answer verification rules. A refinement makes an existing evaluation requirement more specific. A combination tests multiple existing requirements in a single task. An extension brings evaluation scenarios outside the coverage of the seed types into the benchmark.

Consolidated case typeCorresponding seed type(s)Relationship to the seed specifications
CT01 Multi-constraint fact retrievalSingle-hopRefinement. Retrieve a directly stated fact that satisfies all query constraints, such as entity, time, amount, and purpose, while excluding records that match only some constraints.
CT02 Temporal version reconstructionTemporalRefinement. Recover the attribute value that was valid at the specified time, such as the name of a project before it was renamed.
CT03 Evidence-grounded numerical reasoning—Extension. Select numerical records that satisfy the query constraints and compute a result that is not directly stated in the history.
CT04 Set-presupposition validationAdversarialRefinement. Check whether the evidence supports the membership or cardinality presupposed by the question. For example, if the history lists three options and the question asks for two, identify that the evidence provides no basis for selecting a particular pair.
CT05 Complementary evidence synthesisMulti-hopRefinement. Combine indispensable evidence that supports different parts of the answer while preserving the scope and qualifications of each piece of evidence.
CT06 Bounded trajectory reconstructionTemporal + Multi-hopCombination. Integrate evidence within a specified topic and time period to reconstruct intentions, actions, outcomes, and retrospective assessments in chronological order.
CT07 Bounded set reconstructionSingle-hop / Multi-hopRefinement. Recover the complete membership of a specified set from a single record or multiple indispensable pieces of evidence, checking for omissions and extraneous members.
CT08 Entity–relation binding verificationAdversarialRefinement. Make the existing attribution check more specific, for example by verifying whether a preference belongs to the user or to someone on whose behalf the user asked a question.
CT09 Interaction-occurrence verification—Extension. Use a subsequent response that uniquely entails a communicative act as evidence that the act occurred, and distinguish occurrence, non-occurrence, and insufficient evidence.
CT10 Conflicting-assertion resolution—Extension. Use explicit corrections or confirmations to resolve conflicting assertions about the same fact. When the evidence cannot resolve the conflict, preserve the disagreement and report the details that remain supported.
CT11 Value-evidence sufficiency—Extension. Evaluate whether the system gives the most specific answer supported by the evidence. For example, if the history supports "late June," the answer should preserve that range and indicate that the exact date is unknown. The seed adversarial specification explicitly excludes cases constructed solely by requesting an unrecorded degree of precision.

In Appendix G, three human assessors independently evaluated within-type coherence by checking whether the Type Cards grouped under each consolidated type shared stable evidence requirements and reasoning operations. The distinctions between consolidated types and the requirements they add relative to the seed types were assessed through the comparison of the specifications in Appendices B and C.2 shown above.

AuthorsOpenReview ↗

Response to Reviewer xpCq (PartⅢ)

Q3. Sensitivity to Feedback-System Selection

Thank you for this suggestion. We carefully considered the proposed leave-one-system-out experiments and the idea of repeating benchmark construction with different pools of feedback systems. We agree that these experiments would help quantify how the choice of feedback systems affects the case types discovered and the resulting benchmark coverage. Given the limited time available during the rebuttal period, we plan to conduct these experiments in future work and examine how construction costs and benchmark coverage change as the number of feedback systems increases.

Reviewer_8g57OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemEvoBench, a feedback-guided framework for evolving the types of evaluation cases used to assess long-term memory systems. Instead of only generating additional questions from a fixed taxonomy, the framework treats case types as executable specifications and uses validated system failures to propose, refine, and consolidate new evaluation requirements.

The authors conduct ten rounds of case-type evolution on six de-identified multi-session user histories, generating 1,553 evaluation cases and expanding four initial case types into 11 consolidated types. The framework is evaluated using multiple memory systems, including systems that did not provide feedback during evolution. The paper additionally reports cross-user transfer, held-out-user evaluation, and independent human auditing of case validity and type quality. These experiments investigate whether system feedback can reveal previously underrepresented long-term memory evaluation requirements.

Critical strengths
  1. MemEvoBench moves beyond generating harder instances within a fixed benchmark taxonomy and instead investigates whether failure feedback can lead to new, reusable evaluation requirements. Encoding case types as executable specifications is a valuable methodological choice because it turns a discovered failure pattern into a repeatable evaluation procedure. This offers a potentially general framework for expanding the coverage of agent-memory benchmarks.

  2. The evaluation includes discriminative cases across memory systems, transfer of newly discovered case types to other users, two held-out user histories, and independent human auditing. These are particularly important because a benchmark-evolution method should demonstrate more than the ability to generate difficult questions. The held-out-user experiment and human assessments provide meaningful evidence that the discovered types are not merely artifacts of the specific optimization histories or automatic validation pipeline.

Critical weaknesses
  1. Although the paper includes an important held-out-user experiment, the independent evaluation covers two additional user histories and produces 34 automatically validated cases spanning eight of the 11 consolidated types. This supports some degree of transfer, but leaves uncertainty about how consistently the full taxonomy applies to substantially different users, domains, and interaction patterns. The limitation concerns the breadth of the evidence, not the absence of transfer validation.

  2. The consolidated types are executable and receive human support, but semantic distinctness alone does not completely establish that each type captures a new capability requirement rather than a more specific operationalization of an existing one. This affects the interpretation of the reported growth from four to 11 types and the broader claim that the framework expands the space of memory abilities being evaluated.

Questions for authors
  1. In the held-out-user evaluation, only eight of the 11 consolidated types produced validated cases. Were the other three absent because the relevant evidence structures did not occur in those histories, or because the corresponding generation and validation rules failed? This distinction would help interpret the transfer results.

  2. Can the authors provide a clearer mapping from the 11 consolidated types to the four initial types, explaining which newly discovered types introduce genuinely different evidence requirements or memory behaviors rather than finer-grained variants?

  3. How should the system-discrimination results be interpreted given that the benchmark evolves partly from observed system failures? In particular, can the authors distinguish newly discovered evaluation coverage from the increased difficulty that follows from feedback-driven case selection?

3 comments

W1 & Q1: Transferability of the Three Uncovered Types

We sincerely thank the reviewer for raising this important question about the three uncovered types and their transferability. We agree that the original held-out-user evaluation—34 cases across two users, covering eight of the 11 types—does not by itself establish that every type can be instantiated in different user histories.

We revisited the original mining logs. For the three uncovered types, the two held-out histories yielded no candidate evidence satisfying the type requirements. Their absence was therefore not due to generated candidates being repeatedly rejected by the generation or verification rules. Whether a type produces cases depends on whether a user history contains the required evidence structure; we do not expect every type to yield cases for every user.

To further examine whether these types are operational beyond the discovery histories, we applied the frozen specifications of all 11 types to two additional users who had not participated in type discovery and whose histories contained a broader range of topics and evidence structures. New User 1’s history contains 3,467 conversational turns, and New User 2’s contains 3,084 conversational turns. We identified 183 finalized cases grounded in their original records: 94 for New User 1 and 89 for New User 2.

TypeNew User 1New User 2Total
CT01 Multi-constraint fact retrieval10515
CT02 Temporal version reconstruction9817
CT03 Evidence-grounded numerical reasoning7815
CT04 Set-presupposition validation8917
CT05 Complementary evidence synthesis10818
CT06 Bounded trajectory reconstruction91019
CT07 Bounded set reconstruction8715
CT08 Entity-relation binding validation9918
CT09 Interaction-act occurrence validation9918
CT10 Conflicting-statement resolution7916
CT11 Attribute-value evidence sufficiency8715
Total9489183

Each new user yielded cases for all 11 types. Every case retains its question, reference answer, and pointers to the supporting original user records. These results show that the three types absent from the original held-out evaluation are not limited to the histories used for type discovery: when the required evidence structures are present, their frozen specifications can produce cases for new users. We will add these per-user, per-type results to the revised paper and clarify that type transfer is conditional on the presence of suitable evidence, rather than requiring every type to yield cases in every user history.

W2 & Q2: Relationship Between the Final Types and the Four Seed Types

We sincerely thank the reviewer for raising the important question of whether the 11 consolidated types introduce substantive evaluation requirements. We agree that these 11 types do not represent 11 memory abilities entirely independent of the four seed types. To clarify their contribution, we compared the final type definitions in Appendix B with the four seed specifications in Appendix C.2, focusing on evidence structures, reasoning operations, and answer verification rules.

Here, refinement makes an existing evaluation requirement more concrete and executable. Combination tests requirements from multiple seed types within one task, yielding a structure that no single seed type fully describes. Extension adds evaluation requirements not explicitly covered by the seed specifications.

Consolidated case typeCorresponding seed type(s)Relationship to the seed specifications
CT01 Multi-constraint fact retrievalSingle-hopRefinement. Retrieve a directly stated fact that satisfies all query constraints, such as entity, time, amount, and purpose, while excluding records that match only some constraints.
CT02 Temporal version reconstructionTemporalRefinement. Recover the attribute value that was valid at the specified time, such as the name of a project before it was renamed.
CT03 Evidence-grounded numerical reasoningExtension. Select numerical records that satisfy the query constraints and compute a result that is not directly stated in the history.
CT04 Set-presupposition validationAdversarialRefinement. Check whether the evidence supports the membership or cardinality presupposed by the question. For example, if the history lists three options and the question asks for two, identify that the evidence provides no basis for selecting a particular pair.
CT05 Complementary evidence synthesisMulti-hopRefinement. Combine indispensable evidence that supports different parts of the answer while preserving the scope and qualifications of each piece of evidence.
CT06 Bounded trajectory reconstructionTemporal + Multi-hopCombination. Integrate evidence within a specified topic and time period to reconstruct intentions, actions, outcomes, and retrospective assessments in chronological order.
CT07 Bounded set reconstructionSingle-hop / Multi-hopRefinement. Recover the complete membership of a specified set from a single record or multiple indispensable pieces of evidence, checking for omissions and extraneous members.
CT08 Entity–relation binding verificationAdversarialRefinement. Make the existing attribution check more specific, for example by verifying whether a preference belongs to the user or to someone on whose behalf the user asked a question.
CT09 Interaction-occurrence verificationExtension. Use a subsequent response that uniquely entails a communicative act as evidence that the act occurred, and distinguish occurrence, non-occurrence, and insufficient evidence.
CT10 Conflicting-assertion resolutionExtension. Use explicit corrections or confirmations to resolve conflicting assertions about the same fact. When the evidence cannot resolve the conflict, preserve the disagreement and report the details that remain supported.
CT11 Value-evidence sufficiencyExtension. Evaluate whether the system gives the most specific answer supported by the evidence. For example, if the history supports “late June,” the answer should preserve that range and indicate that the exact date is unknown. The seed adversarial specification explicitly excludes cases constructed solely by requesting an unrecorded degree of precision.

This comparison shows that the contribution of the final types goes beyond assigning more specific names to the four seeds. Refinements specify evidence and verification boundaries more precisely; the combination tests requirements that must work together in one task; and extensions cover requirements absent from the seed specifications. We therefore interpret the 11 types as executable diagnostic evaluation requirements, rather than 11 foundational abilities unrelated to the seeds.

The human audit in Appendix G examined whether Type Cards grouped under the same consolidated type shared stable evidence requirements and reasoning operations. We will add the comparison above to the revised paper so that readers can assess both within-type coherence and what each type adds relative to the seed specifications.

Q3: How Should System Discrimination Be Distinguished from New Evaluation Coverage?

We sincerely thank the reviewer for raising this important question. To examine whether system feedback merely makes cases harder or also helps broaden the coverage of evaluation requirements, we added a feedback-free evolution baseline. We agree that the 804/1,553 discriminative cases reported in the paper, and system scores on that subset, characterize the diagnostic value of the final benchmark. They do not, by themselves, show that feedback discovered new evaluation requirements.

Both runs completed ten rounds and retained the case-mining, quality-checking, three-strategy type-evolution, and type-admission processes. In each round, the feedback-free run randomly sampled approximately the same number of quality-checked cases as the main run’s feedback set for Evolution. This kept the number of evolution rounds, case input volume, and computational scale in the same range. Evolution received each case’s question, reference answer, supporting evidence, and original type, but received no memory-system answers, correctness labels, or failure patterns. It therefore had comparable opportunities to propose types from cases without being guided by observed system failures.

After ten rounds, we compared the two sets of types in both directions, using their required evidence structures, core answer operations, and verification rules. We identified 25 feedback-free Type Cards with evaluation requirements identical or similar to those of the Type Cards reported in the paper. We then examined their coverage of the final 11 consolidated types and separately reviewed the remaining feedback-free candidates for independently supported requirements that the existing types could not absorb.

Ten-round resultFeedback-guided full runFeedback-free evolution
Finalized cases1,5531,676
Type Cards with identical or similar evaluation requirements3125
Consolidated types clearly covered11/116/11 (CT02, CT03, CT05, CT07, CT10, CT11)
Partially addressed, without all essential rules—3/11 (CT01, CT06, CT08)
Essential evaluation requirements missing—2/11 (CT04, CT09)
Confirmed independent requirements beyond the final 11 types—None confirmed

The feedback-free run produced 1,676 finalized cases, a similar and slightly larger case set than the full run’s 1,553. Thus, the observed type-coverage gaps cannot be explained simply by a lack of generated cases. Specifically, the feedback-free run did not develop CT04’s rule for validating a question’s presupposed set membership or cardinality. For CT09, it did not develop the key evidence rule that a subsequent response can uniquely establish that a communicative act occurred. The partially corresponding types also lack some required evidence or verification boundaries. Among the remaining feedback-free candidates, we have not confirmed an independently supported evaluation requirement that cannot be absorbed by the existing types.

These findings come from comparing executable type specifications and their evidence and verification rules, rather than inferring coverage from system scores on discriminative cases. The ten-round comparison therefore suggests that system feedback may contribute to discovering evaluation requirements beyond merely selecting harder cases.

Comments on the paper

2 comments

W1 & Q1: Feedback-Free Evolution Baseline

We sincerely thank the reviewer for recognizing the value of feedback-guided case-type evolution and for suggesting a direct comparison with a feedback-free approach. We agree that comparing evolution strategies alone cannot establish whether system feedback expands evaluation coverage.

To examine whether feedback merely helps select harder questions or also helps discover new evaluation requirements, we added a ten-round feedback-free evolution baseline. It retains case generation, quality checks, and type evolution. In each round, we randomly sample approximately the same number of quality-checked cases as in the main experiment’s feedback set and provide them to Evolution. The baseline receives each case’s question, reference answer, evidence, and original type, but no memory-system responses, correctness labels, or failure patterns. Both runs use ten evolution rounds and similar numbers of case inputs per round, keeping their evolution compute at a comparable scale.

After ten rounds, we compared the types in both runs in both directions, using their required evidence structures, core answering operations, and answer-verification rules. Relative to the main experiment’s Type Cards before consolidation, 25 feedback-free cards have identical or similar evaluation requirements. We then examined how well the feedback-free types cover the final 11 consolidated types and separately reviewed the remaining candidates for independently supported requirements that the existing types could not capture.

Ten-round resultFeedback-guided FullFeedback-free evolution
Finalized cases1,5531,676
Final consolidated types clearly covered11/116/11 (CT02, CT03, CT05, CT07, CT10, CT11)
Related requirements addressed, but key rules not fully covered—3/11 (CT01, CT06, CT08)
Essential evaluation requirements missing—2/11 (CT04, CT09)
Confirmed independent requirements beyond the final 11 types—None identified so far

The feedback-free run produced slightly more finalized cases, so the coverage gap cannot be explained simply by a lower overall case count. Specifically, it did not establish CT04’s requirement to validate the set membership or cardinality presupposed by a question. For CT09, it did not establish the key evidence rule that a subsequent response can confirm that a communicative act occurred. We also have not identified, among the remaining feedback-free candidates, an independently supported requirement that cannot be captured by the final types.

This comparison examines the types’ evidence structures and scoring rules directly, rather than inferring coverage from system accuracy on difficult cases. The results suggest that system feedback may do more than increase case difficulty: it may also help expand the evaluation requirements covered by the benchmark. We will add the baseline setup, type-level mappings, and coverage criteria to the revised paper.

W2: Are the Discovered Types Reproducible Across Different Settings?

We sincerely thank the reviewer for this valuable suggestion. We agree that a single ten-round evolution run over six user histories is insufficient to establish that the framework would consistently discover the same evaluation requirements with different discovery histories, seed types, or model configurations. Repeating the evolution under these settings and comparing the overlap and differences among the resulting types is a valuable validation step.

Given the limited rebuttal period, we cannot complete and adequately verify these additional runs at this stage. We will state this limitation explicitly in the revised paper. In subsequent experiments, we will vary the user histories, seed-type combinations, and model configurations, and report the overlap, unique requirements, and downstream case support across runs.

W3 & Q2: Did Case-Quality and Judging Noise Affect Type Discovery?

We sincerely thank the reviewer for raising the concern that feedback noise could influence subsequent type discovery. In the human audit, 11 of 220 sampled cases (5.0%) failed at least one quality criterion; for 52 of 836 system answers (6.2%), the automatic judgment disagreed with the human majority vote.

To trace the effect of this noise on the final types, we examined how the relevant records appeared in the Evolution feedback inputs and candidate-type source references. Some invalid cases and feedback with disputed judgments did contribute to type proposals. However, when we excluded the invalid cases and corrected the relevant judgments according to the human majority vote in this source check, no final type lost all of its valid sources. Other cases still supported the required evidence structures and evaluation requirements of the affected types.

Given the low observed rates of these issues in the audit and our tracing of how invalid cases and disputed judgments participated in type discovery, we have reason to believe that the evidential support for the reported final types has some robustness to this limited amount of noise. We will add the tracing procedure and results to the revised paper and report the effects of the two sources of noise separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.