- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces E2MEND, a diagnosis-guided framework for improving long-term memory systems through bounded strategy evolution. The framework distinguishes four potential sources of failure: ingestion, retrieval, state construction, and answer generation. Based on the diagnosed bottleneck, it proposes modifications to versioned memory strategies while keeping the underlying runtime fixed. Candidate changes are assessed through targeted screening, paired evaluation, negative evidence, and rollback mechanisms.
The authors report experiments on fixed evaluation slices from LoCoMo, MemBench, HotpotQA, and LongMemEval. The LoCoMo development trajectory increases accuracy from 0.756 to 0.947, but the paper explicitly identifies this as staged development rather than a fully controlled held-out comparison. Only the final 0.937-to-0.947 transition has an independently recorded paired checkpoint. The MemBench experiment reports a descriptive improvement from 0.770 to 0.863 across three paired runs. The paper limits its claims to fixed-slice diagnosis and strategy improvement rather than general benchmark superiority.
Critical strengths
-
The paper highlights that an incorrect answer does not necessarily indicate a retrieval problem: relevant evidence may never have been stored, may have been retrieved but inadequately organized, or may be available to an answer model that still fails. Distinguishing these stages can guide more appropriate interventions and offers a meaningful perspective on memory-system debugging.
-
The locked runtime, versioned strategy artifacts, targeted candidate screening, paired comparisons, and rollback procedures are sensible safeguards against uncontrolled strategy changes. The paper also distinguishes staged development, configuration sweeps, paired verification, and reference-only measurements. This transparency is important because it prevents the largest reported improvement from being automatically interpreted as an independently validated benchmark gain.
Critical weaknesses
-
The 0.756-to-0.947 LoCoMo improvement is the endpoint difference of a staged development trajectory. Only the final 0.937-to-0.947 transition has an independently recorded paired checkpoint, with 16 repaired questions and no regressions on the fixed slice. The paper itself categorizes that transition as pending rather than strictly accepted. Therefore, the headline 19.1-percentage-point difference cannot establish the general effectiveness of diagnosis-guided evolution on unseen tasks. This is important because it is the most prominent empirical result.
-
E2MEND combines diagnostic probes, constrained edits, candidate screening, historical feedback, and rollback. The results do not clearly establish whether correctly identifying the failure stage is necessary for the observed improvements, as opposed to searching a restricted strategy space with repeated verification. Since diagnosis-guided intervention is the paper's central contribution, stronger evidence connecting diagnostic correctness to repair effectiveness is needed to support its methodological significance.
Questions for authors
-
Which reported improvements, if any, were confirmed on questions or user histories not used during strategy development or candidate selection? Please distinguish the evidence for the full development trajectory from the evidence for independently verified strategy improvements.
-
Is there an existing diagnosis-free baseline that searches the same strategy space using the same candidate and evaluation budgets? Such a comparison would help determine whether bottleneck attribution, rather than bounded optimization alone, explains the gains.
-
How is the correctness of the ingestion/retrieval/state-construction/answer-model diagnosis validated? Are there independently verified failure-stage labels or analyses showing that correctly diagnosed cases are more likely to benefit from the corresponding targeted intervention?
2 comments
Title: Follow-up on Q2 (is diagnosis necessary?)
One correction and one clarification on Q2.
The targeting evidence does not hold, and we withdraw it. Reviewer wFov asked how many of the "220 out-of-group triggers" were wrong under the baseline. The answer is none: the count is 247, and all 247 were already correct, so none was repairable — just as the diagnosed group, being 78/78 baseline-wrong, could not regress. Both halves of that comparison are determined by correctness labels, so neither shows that diagnosis predicts where repairs are.
What we are not saying. This is not evidence that diagnosis can be removed. In the cumulative pipeline ablation, removing Layer-A and re-evolving costs 4.6 points (0.947 → 0.901), and removing all three mechanisms costs 9.3. That ablation is coupled rather than factorial, so it cannot book the accuracy to diagnosis alone — but the pipeline as built depends on it. Our claim narrows to this: diagnosis scopes the search and gates adoption. We do not claim a measured endpoint gain from it, and we already did not claim it proposed the adopted candidate.
Two smaller corrections to the same paragraph. "45% of stable errors carry their evidence in context" is the share with gold key-unit coverage ≥ 0.67 (64 of the 142 stably-wrong items); requiring full coverage gives 29.6%. And "gates adoption safely" should rest on the gate's own record rather than an oracle-labeled analysis: the live sampler adopted none of its 8 recorded candidates (best +1.01pt, p=0.22), and 22 of the 23 deterministic probes were rejected — the one that passed then held on 653 questions from conversations never used in development.
Evidence-status table + is diagnosis necessary + diagnosis-correctness audit
We thank the reviewer. We agree the 19.1pp endpoint difference cannot establish general effectiveness; it is relabeled a development trajectory. Evidence status of every reported number:
Confirmed on held-out — the only transfer evidence:
Development / pending — not transfer evidence:
Invalid / reference-only: two earlier LoCoMo runs (intervention inert; disclosed); MemBench paper rows 0.770→0.863 (fixed-slice descriptive); LongMemEval-S 0.950 (no matched comparator).
Q2 — is diagnosis necessary? We agree it is not shown necessary at the search level — no arm adopts anything on this slice, and LoCoMo's ceiling is reader-side (45% of stable errors carry their evidence in context, S3). We claim narrower: at the targeting level, diagnosis predicts where repairs are (12/12 repairs in the diagnosed group; 0/220 out-of-group triggers) and gates adoption safely (oracle-labeled scope analysis; illustrative). On MemBench the matched lever transfers (+11.3pt / 700). We do not claim diagnosis proposed the adopted candidate.
Q3 — how is diagnosis correctness validated? Operationally, without the LLM: gold evidence ids → stored facts → the failing item's retrieval snapshot decides extraction- vs retrieval-gap. An independent re-implementation reproduces the stored verdict on 68% of the decidable subset (S6; a lower bound). Disclosed limits: retrieval-rank instrumentation is unavailable at scale; Q_i is a routing predicate, not causal. New check: a second judge overturned 4 of 142 stably-wrong items (2.7%).