Where Self-Evolving Memory Systems Lose Their Gains: A Full-Loop Audit

Self-evolving memory systems use large language model (LLM) agents to evaluate performance, diagnose failures, and revise shared memory components. These loops often improve early and then plateau. Their logic rests on three assumptions: evaluation identifies failures, edits repair them, and accepted repairs persist. We audit this chain in several self-evolution methods for memory systems on two conversational memory benchmarks, using question-level evidence independent of the agent. We find that evaluation provides unstable feedback for repair. Reevaluating an unchanged system changes which questions fail. At least half of the accepted single-step gains fall within this variation. Even edits that target the bottleneck stage do not prevent regressions. These edits raise scores much more often than other edits. Yet they break previously correct answers just as often, because the components they change are shared across queries. Across rounds, repairs do not accumulate reliably. Repairs decline toward the level of answer flips observed in an unchanged system. Regressions, however, persist. Addressing these weaknesses improves individual revisions but does not ensure sustained progress. We extend MemPro, one of the audited methods, with verified feedback, repair rounds, and paired acceptance. Development accuracy rises by up to 16.9 percentage points. Yet final held-out gains are at most 3.4 points, within the original loop's rerun variation. The remaining gains suggest that the bottleneck returns to feedback. They are below what a development set of the original size can resolve. Sustained self-evolution therefore requires repeated evaluation and question-level tracking of repairs and regressions. Selecting revisions also requires enough evidence to distinguish gains at the intended scale from evaluation variation.