acceptodds
Under review as a conference paper at ICLR 2027

EMEND: Evidence-guided Evolution for Memory Enhancement via Bottleneck Diagnosis

Abstract

Long-term memory failures are heterogeneous: evidence may be lost, unretrieved, insufficiently structured, or available while the answer model still fails; a uniform update can target the wrong bottleneck and introduce regressions. We present EMEND, a diagnosis-guided framework for bounded memory-strategy evolution: it estimates the bottleneck, mutates versioned strategy artifacts through a bounded LLM evolver under a locked runtime, and adopts candidates via paired evaluation, negative evidence, and rollback. It matches failure modes to bounded edits—extraction or provenance for ingestion gaps, support selection for retrieval gaps, typed state for state construction, and no memory mutation for answer-model limitations. On a fixed 1,540-question LoCoMo slice, a staged trajectory raises accuracy from 0.756 to 0.947 (final paired checkpoint: 16 repairs, 0 regressions); on MemBench, matched strategies yield a descriptive . We frame these as diagnostic and strategy claims on fixed slices, not public-task superiority.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-term memory, language agents, memory-augmented language models, retrieval-augmented generation, adaptive memory, self-evolving systems, bottleneck diagnosis, non-regression verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eJuFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces E2MEND, a diagnosis-guided framework for improving long-term memory systems through bounded strategy evolution. The framework distinguishes four potential sources of failure: ingestion, retrieval, state construction, and answer generation. Based on the diagnosed bottleneck, it proposes modifications to versioned memory strategies while keeping the underlying runtime fixed. Candidate changes are assessed through targeted screening, paired evaluation, negative evidence, and rollback mechanisms.

The authors report experiments on fixed evaluation slices from LoCoMo, MemBench, HotpotQA, and LongMemEval. The LoCoMo development trajectory increases accuracy from 0.756 to 0.947, but the paper explicitly identifies this as staged development rather than a fully controlled held-out comparison. Only the final 0.937-to-0.947 transition has an independently recorded paired checkpoint. The MemBench experiment reports a descriptive improvement from 0.770 to 0.863 across three paired runs. The paper limits its claims to fixed-slice diagnosis and strategy improvement rather than general benchmark superiority.

Critical strengths
  1. The paper highlights that an incorrect answer does not necessarily indicate a retrieval problem: relevant evidence may never have been stored, may have been retrieved but inadequately organized, or may be available to an answer model that still fails. Distinguishing these stages can guide more appropriate interventions and offers a meaningful perspective on memory-system debugging.

  2. The locked runtime, versioned strategy artifacts, targeted candidate screening, paired comparisons, and rollback procedures are sensible safeguards against uncontrolled strategy changes. The paper also distinguishes staged development, configuration sweeps, paired verification, and reference-only measurements. This transparency is important because it prevents the largest reported improvement from being automatically interpreted as an independently validated benchmark gain.

Critical weaknesses
  1. The 0.756-to-0.947 LoCoMo improvement is the endpoint difference of a staged development trajectory. Only the final 0.937-to-0.947 transition has an independently recorded paired checkpoint, with 16 repaired questions and no regressions on the fixed slice. The paper itself categorizes that transition as pending rather than strictly accepted. Therefore, the headline 19.1-percentage-point difference cannot establish the general effectiveness of diagnosis-guided evolution on unseen tasks. This is important because it is the most prominent empirical result.

  2. E2MEND combines diagnostic probes, constrained edits, candidate screening, historical feedback, and rollback. The results do not clearly establish whether correctly identifying the failure stage is necessary for the observed improvements, as opposed to searching a restricted strategy space with repeated verification. Since diagnosis-guided intervention is the paper's central contribution, stronger evidence connecting diagnostic correctness to repair effectiveness is needed to support its methodological significance.

Questions for authors
  1. Which reported improvements, if any, were confirmed on questions or user histories not used during strategy development or candidate selection? Please distinguish the evidence for the full development trajectory from the evidence for independently verified strategy improvements.

  2. Is there an existing diagnosis-free baseline that searches the same strategy space using the same candidate and evaluation budgets? Such a comparison would help determine whether bottleneck attribution, rather than bounded optimization alone, explains the gains.

  3. How is the correctness of the ingestion/retrieval/state-construction/answer-model diagnosis validated? Are there independently verified failure-stage labels or analyses showing that correctly diagnosed cases are more likely to benefit from the corresponding targeted intervention?

2 comments
AuthorsOpenReview ↗

Evidence-status table + is diagnosis necessary + diagnosis-correctness audit

We thank the reviewer. We agree the 19.1pp endpoint difference cannot establish general effectiveness; it is relabeled a development trajectory. Evidence status of every reported number:

Confirmed on held-out — the only transfer evidence:

ResultDev data used?Held-outEffectKey limitation
LoCoMo frozen artifactdev half only653 unseen+7.96pt, p=4.3e-7correction post-hoc (apparatus only); lever known from full-slice trajectory (S2.7)
MemBench frozen SlotTableno700 unseen+11.3pt, p=7.5e-13reads benchmark-provided annotations (appendix)

Development / pending — not transfer evidence:

ResultEffectTierKey limitation
LoCoMo 0.756→0.947+19.1ppdevelopment trajectoryno unseen data
LoCoMo 0.937→0.947+16/−0pending candidatebelow the 0.02 threshold
MemBench dev pair+10.5pt, p=4.9e-5fixed-slice pairedsame 200 questions used in development

Invalid / reference-only: two earlier LoCoMo runs (intervention inert; disclosed); MemBench paper rows 0.770→0.863 (fixed-slice descriptive); LongMemEval-S 0.950 (no matched comparator).

Q2 — is diagnosis necessary? We agree it is not shown necessary at the search level — no arm adopts anything on this slice, and LoCoMo's ceiling is reader-side (45% of stable errors carry their evidence in context, S3). We claim narrower: at the targeting level, diagnosis predicts where repairs are (12/12 repairs in the diagnosed group; 0/220 out-of-group triggers) and gates adoption safely (oracle-labeled scope analysis; illustrative). On MemBench the matched lever transfers (+11.3pt / 700). We do not claim diagnosis proposed the adopted candidate.

Q3 — how is diagnosis correctness validated? Operationally, without the LLM: gold evidence ids → stored facts → the failing item's retrieval snapshot decides extraction- vs retrieval-gap. An independent re-implementation reproduces the stored verdict on 68% of the decidable subset (S6; a lower bound). Disclosed limits: retrieval-rank instrumentation is unavailable at scale; Q_i is a routing predicate, not causal. New check: a second judge overturned 4 of 142 stably-wrong items (2.7%).

AuthorsOpenReview ↗

Title: Follow-up on Q2 (is diagnosis necessary?)

One correction and one clarification on Q2.

The targeting evidence does not hold, and we withdraw it. Reviewer wFov asked how many of the "220 out-of-group triggers" were wrong under the baseline. The answer is none: the count is 247, and all 247 were already correct, so none was repairable — just as the diagnosed group, being 78/78 baseline-wrong, could not regress. Both halves of that comparison are determined by correctness labels, so neither shows that diagnosis predicts where repairs are.

What we are not saying. This is not evidence that diagnosis can be removed. In the cumulative pipeline ablation, removing Layer-A and re-evolving costs 4.6 points (0.947 → 0.901), and removing all three mechanisms costs 9.3. That ablation is coupled rather than factorial, so it cannot book the accuracy to diagnosis alone — but the pipeline as built depends on it. Our claim narrows to this: diagnosis scopes the search and gates adoption. We do not claim a measured endpoint gain from it, and we already did not claim it proposed the adopted candidate.

Two smaller corrections to the same paragraph. "45% of stable errors carry their evidence in context" is the share with gold key-unit coverage ≥ 0.67 (64 of the 142 stably-wrong items); requiring full coverage gives 29.6%. And "gates adoption safely" should rest on the gate's own record rather than an oracle-labeled analysis: the live sampler adopted none of its 8 recorded candidates (best +1.01pt, p=0.22), and 22 of the 23 deterministic probes were rejected — the one that passed then held on 653 questions from conversations never used in development.

Reviewer_JMxQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a framework for bounded offline memory-strategy evolution. Layer-A and a fixed policy router map failures to eligible strategy edits under a locked runtime. Candidate adoption uses targeted replay and paired non-regression checks. Persistent history records candidate outcomes, while checkpoint rollback provides a safeguard against later degradation. Experiments report improvements on fixed benchmark slices.

Critical strengths
  1. Clear bottleneck decomposition. The framework distinguishes ingestion gaps, retrieval gaps, state-construction gaps, and answer-model gaps. This provides a clear basis for matching failures to appropriate strategy edits.
  2. Well-motivated controlled evolution. The locked-runtime, versioned-strategy architecture defines a clear modification boundary and makes changes inspectable and checkpoint-reversible. Separating candidate generation from adoption, while retaining negative evidence, is a valuable design choice.
  3. Positive preliminary results. The paired MemBench runs show improvements with SlotTable safe-combo on the evaluated slice. The paper distinguishes development trajectories, configuration sweeps, and reference-only comparisons. It also acknowledges limitations in evaluation and reproducibility.
Critical weaknesses
  1. Has generalization been further evaluated? Candidate generation, selection, and evaluation reuse fixed slices, with no untouched confirmation set. The evolved strategy should be frozen and evaluated on unseen questions using conversation-disjoint data splits.
  2. The LoCoMo improvement lacks a complete adoption record. The 0.756→0.947 result is a staged development trajectory. Only the final 0.937→0.947 update has an independently recorded paired checkpoint, with 16 repairs and no regressions. This update is classified as PENDING under the stated thresholds. However, incomplete intermediate records make it difficult to verify whether the earlier updates satisfied the adoption criteria. Intermediate strategy artifacts, paired outcomes, and decision statuses should be provided. The best accepted strategy should also be reported separately from the pending candidate.
  3. The state-construction predicate is underspecified. The predicate Q_i is defined semantically but not operationally.
  4. The benefit of diagnosis-guided search remains unclear. Improvements from GroupRank and SlotTable do not establish that diagnosis selects better edits than simpler search methods. As acknowledged in Appendix D.5, the cumulative ablation measures pipeline sensitivity rather than isolated component effects. Conduct equal-budget comparisons with random and greedy search using the same locked runtime, strategy space, and model allocation.
  5. Protected-group results are not fully reported. Could the protected-group partitions, per-group paired deltas, and corresponding tolerances be provided? False-acceptance and false-rejection rates based on independently assessed candidate outcomes would also help evaluate verification reliability.
Questions for authors
  1. Typos. Should “Appendix 4” in Section 6 refer to Table 4 in Appendix C? Appendix D.3 contains the duplicated citation text “Tan et al. Tan et al. (2025).”
4 comments
Reviewer_JMxQOpenReview ↗

Official Comment by JMxQ

Considering there is no author rebuttal available, I stick with rating 2.

AuthorsOpenReview ↗

Apologies for the Delay

I apologize—I’m still sorting through the latest experimental results and will get back to you by the end of today at the latest.

AuthorsOpenReview ↗

Generalization results + adoption ledger, Q_i rule, equal-budget baselines, protected groups

We thank the reviewer. The five critical weaknesses are answered in order — each with new experiments or released artifacts (ledgers, manifests, and audit tables in the supplement and evidence pack) — followed by the question under Questions For Authors.

  1. Generalization (W1). Agree — this was missing; both confirmations are now run: MemBench +11.3pt on 700 unseen events (p=7.5e-13); LoCoMo +7.96pt paired (p=4.3e-7), +6.58pt with gpt-4.1 — positive in all four categories and conversations (details in our reply to wFov). The 0.756→0.947 result is relabeled a development trajectory throughout.

  2. Adoption record (W2). Agree — corrected and released in full. Best accepted = 0.937; 0.947 pending (+16/−0 ≈ +0.010, below the 0.02 threshold), reported separately. The released ledger carries every candidate's exact edits, paired flips, and decision rationale (missing fields marked unavailable, never inferred). What each search produced:

    SearchVerified on full EOutcome
    live UCB sampler97best effect +1.1pt (within noise); its proposal filter excludes the global source-window path, so the winning lever was unreachable (S7)
    deterministic diagnosis-mapped frontier23one adoption — source-window grounding ×5 intents, +3.0–5.4pt across three baseline repeats: the frozen artifact tested on C
    — remaining probes22all within noise (|Δ| ≤ 1.5pt), declined with full records
    post-freeze reproduction4 equal-budget armsno further positive lever found

    Adoption is rare by design, and the gate's selectivity is what the held-out result validates — the one adopted candidate is the one that held at +7.96pt on unseen conversations. The frontier spanned five provenance classes (S2.7); adoption used E only.

  3. Q_i (W3). Agree — it was semantically but not operationally specified. Now operationalized and frozen before candidate generation (category rule 1/2→1, 3/4→0, overridden by frozen text-shape regexes); per-item manifests released (dev 403/887, test 293/653); a routing predicate, not a causal diagnosis.

  4. Diagnosis benefit (W4). We agree the cumulative ablation measured pipeline sensitivity rather than component effects, and that at the search level diagnosis shows no endpoint advantage — no arm, diagnosis-guided or not, reaches an adoptable edit here (uninformative, not negative). We disagree that the benefit of diagnosis-guided search is therefore unestablished: it is established at the targeting level. All 12 repairs of the unrestricted scope fall inside the audited group while its 220 out-of-group triggers repair nothing (scope-level oracle range analysis — label-restricted, illustrative, not deployable), and equal-budget random selection over the no-mask pool rarely matches (E=3.05 repairs / 2.29 regressions, P(zero regressions)=6.8%; oracle bound 12 vs masked 14).

  5. Protected groups & verification reliability (W5). Agree — underreported; now released in full: per-category paired deltas for all held-out comparisons with ε=0.02 and a noise reference (adoption on E used ε_k = 0). The corrected confirmation has no protected-category regression (single-hop is the largest gain, +41/−10). False acceptance did not occur at the search level; false-rejection is not estimable and not claimed.

  6. Typos (Questions For Authors) — agree; fixed in the revision source ("Appendix Table 4"; duplicated "Tan et al." citation removed).

Paper revisions: relabel the trajectory and separate accepted/pending (abstract included); add the MemBench and LoCoMo held-out evidence; add an evidence-status column to the main table; adopt routing-predicate terminology for Q_i; state the search-level limitation of the hard mask; release the audit artifacts.

Corrections to W2 (adoption record) and W4 (diagnosis benefit) Official Comment by Authors Comment: Two corrections to our reply. W1 and W5 are unaffected.

W4 (diagnosis benefit) — withdrawn. We wrote that the benefit "is established at the targeting level", citing 12/12 repairs inside the audited group against 0 repairs over 220 out-of-group triggers. Reviewer wFov asked how many of those out-of-group triggers were wrong under the baseline; the answer is none. The count is 247 (not 220) and all 247 were already correct, so none was repairable; the audited group is 78/78 baseline-wrong (the masked arm's 75 triggers are 75/75 baseline-wrong), so it could not regress. Both halves are determined by baseline-correctness labels, so neither speaks to targeting. We also re-ran the audit with no gold answers and no correctness labels over all 295 triggered items: the mask retains 281 (95% of the pool) and yields 11 repairs / 9 breaks (p=0.82) against a random equal-budget expectation of 11.4; the one label-free signal that does narrow the pool (risk=low, 262 kept vs 33 excluded) points the wrong way, net −2 inside versus +5 outside. The "oracle bound 12 vs masked 14" comparison is withdrawn as well: the two scopes are not nested (27 of the 75 masked triggers fall outside the unrestricted arm) and their repair sets overlap on only 10 items. W4 therefore stands as you wrote it. What remains is that diagnosis gates adoption — a safety role, not an accuracy gain.

W2 (adoption record) — two corrections. (i) "remaining probes: 22, all within noise (|Δ| ≤ 1.5pt)" is wrong. 20 of the 22 are inside that band; two were declined as clearly harmful, −2.6pt (query_rewriting) and −3.6pt (sw_core_only). The probe journals in the pack show this directly, so we correct it rather than leave it. (ii) "its proposal filter excludes the global source-window path, so the winning lever was unreachable" overstates the code: the filter drops retrieval.source_window unconditionally, but the adopted lever is the intent-level form of that switch, which the filter drops only once that family is marked blocked. We cannot show from the available logs that this applied, so we withdraw the inference. Relatedly, the per-candidate ledger we can release in full covers the 23 deterministic probes (exact edits, paired flips, per-category W/L, decision reason). We have no per-candidate ledger for the 97 sampler candidates, so we withdraw "best effect +1.1pt" and mark those fields unavailable rather than infer them, per the standard we stated.

Reviewer_wFovOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel framework named E2MEND that it improves usage of long-term memory in LLM agents by diagnosis-guided strategy evolution. Rather than applying conventional memory updates on agent, this method first diagnoses failures and selects corresponding strategy modifications. In particular, the framework follows an "Observe, Improve, Verify" loop such that it diagnoses the bottleneck and LLM-based evolved to suggest bounded modifications to versioned configurations and at last verification stage to evaluate improvenments. The main point is that the execution logic remains fixed and only schema-constrained strategy can be modified. To demonstrate this method, authors show extensive experiments within several benchmarks.

Critical strengths
  1. Clear formulation of proposed method.
  • Authors provide a structured formulation of memory failures by distinguishing four way bottleneck diagnosis. In Table 2, it shows an intuitive mapping between these failure categories and proper strategy modifications, providing a clear rationale for diagnosis-guided strategy selection
  1. Auditable strategy evolution framework
  • The design choice that it separates between locked execution runtime and mutable configurations gives a clear boundary for automated optimization. Furthermore, restricting the LLM evolved to schema-legal modifications induces updates interpretable and traceable. It also preserves a history of attempts and supports checkpoint-based rollback.
Critical weaknesses
  1. Limited evidence for generalization
  • One of the main experimental results in this paper is LoCoMo. Although the improvement from 0.756 to 0.947 is impressive, it is achieved with staged strategy evolution on the same fixed evaluation slice used for diagnosis. The authors explicitly limit their claims to improvement on fixed set. However, this paper does not evaluate the evolved strategies on unseen conversations. Therefore, it is still unclear whether the improvements can be extended beyond the evaluation slice for optimization.
  1. Insufficient validation on diagnosis guided strategy selection
  • In Figure 2 (b), it shows a cumulative ablation study on the proposed framework. However, it does not isolate the influence of diagnosis-based strategy restriction. In particular, it remains unclear whether constraints on eligible strategy based on diagnosed bottlenecks can induce an additional benefits over using the same diagnosis to guide strategy optimization without restrictions.
Questions for authors
  1. Generalization of evolved strategies
  • Could the authors evaluate the evolved strategies on held-out LoCoMo split to demonstrate whether the improvements can be transferred to unseen conversations?
  1. Controlled ablation study on the diagnosis based strategy restriction
  • Could the authors provide fair comparison with a variant that retains the same diagnosis information but removes the hard eligibility mask under the same candidate and budget?
1 comment
AuthorsOpenReview ↗

Held-out transfer confirmed on MemBench and LoCoMo + eligibility-mask ablation

We thank the reviewer; both questions were run as new experiments, every protocol amendment logged.

Q1 — held-out transfer. We answer on both benchmarks: MemBench is the preregistered clean case; LoCoMo — the split your question names — is the harder case, confirmed under a corrected protocol with everything disclosed below.

MemBench (event-disjoint) — the clean case. The frozen SlotTable strategy is a pure configuration (no route replay, no question/answer mapping); it reads the rel/attr/value annotations MemBench event records provide (appendix).

SlicebaselinefrozenΔW/Ltwo-sided p95% CI
dev, 200 questions0.8300.935+10.524/34.9e-5[+5.5, +15.5]
held-out, 700 events0.76290.8757+11.3103/247.5e-13[+8.3, +14.3]

Per category (held-out): conditional +29.3 … comparative −1.4 (all five in the pack).

LoCoMo (conversation-disjoint) — the harder case. Ten conversations split into an evolution half E (conv-30/42/43/44/47/49; 887 questions) and a confirmation half C (conv-26/41/48/50; 653 questions), balanced on category mix and accuracy. Evolution ran on E only. 23 candidates were verified on the full E set (exact McNemar; adoption required ε_k = 0), enumerated deterministically because the released UCB sampler's filter excludes the global source-window path (0/97; S7). Some candidates came from the paper's full-slice trajectory (which included C; source-window grounding gave +0.062 there); adoption used E only. Exactly one was adopted — source-window grounding across five intent types: +3.0 to +5.4pt on E across three baseline repeats (+80/−32 vs the sweep's baseline, two-sided p=6.6e-6). The frozen artifact differs in five source_window flags only; ingestion identical.

Comparison on C (653 items)baselinefrozenΔW/Ltwo-sided p
paired, gpt-4o0.77030.8499+7.9679/274.3e-7 (CI [+5.1, +11.0])
paired, gpt-4.10.79630.8622+6.5868/259.4e-6
vs strongest of 3 baseline runs0.79940.8499+5.05——

All four categories are positive (largest: single-hop +41/−10, p=1.5e-5; details S2.4) and all four held-out conversations improve (+3.2–+10.5pt).

Protocol status — what is confirmatory, what was corrected.

  • Confirmatory core. A single frozen artifact (five flags, frozen at adoption on E) and a single hypothesis (it beats its baseline on C); no configuration, threshold, or analysis choice changed after seeing C.
  • What was corrected. The measurement apparatus: two earlier evaluations ran with the intervention inert (window-map defect; one-call reproduction in S2.2) — invalid measurements, not negative evidence, and we report them. The repair is mechanical and symmetric (both arms on a freshly opened provider), forced by the demonstrated defect, not chosen among alternatives.
  • Corroboration. An independent reader replicates (gpt-4.1, p=9.4e-6); the gain is at least as large as the E-side gain; C has now been evaluated four times.

Noise and judging. Identical-configuration repeats flip 13.9–14.2% of the 653 labels (sub-two-point differences unresolvable; the gain is far outside that band); S3–S4 carry the failure anatomy and a non-replicating answer-layer extension, neither in any claim.

Q2 — eligibility-mask ablation. The mask's claimed function is targeting and safety — keeping bounded evolution inside the diagnosis-matched intervention class — not an endpoint lift by itself; the evidence matches that claim.

  • Search level. No arm, diagnosis-guided or not, adopts anything that passes the paired gate here — the slice admits no adoptable edit within budget, so this is uninformative for diagnosis.
  • Performance lives in the matched class. On MemBench the diagnosis-matched lever (typed state) transfers at +11.3pt on 700 unseen events. On LoCoMo the matched class is capped structurally (S3: 45% of stably-wrong items already carry their evidence in context — a reader limit no memory-side mechanism addresses).
  • The mask confines and protects. In a scope-level analysis (oracle range analysis: production mask signal ∩ gold-audited group; illustrative, not deployable), all 12 repairs of the unrestricted scope fall inside the group while its 220 out-of-group triggers yield none — the confound-free part — and the restricted scope does so at +13/+14/+15 repairs with zero regressions vs +12/−9 at 3.9× the triggers (zero regressions follows partly from the group being baseline-incorrect items; seeds differ only in sampling). Random equal-budget selection over that pool rarely matches it (P(zero regressions)=6.8%; oracle repair bound 12 < masked 14).

What we do not claim: search-level endpoint lift within budget, or that diagnosis proposed the adopted candidate (that enumeration was failure-volume ordered, no diagnosis-free control).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.