COMET: A Framework for Diagnosis-Guided Retrieval Adaptation in Agent Memory

Agent memory systems increasingly refine how experience is stored, but memory retrieval often remains reliant on general-purpose dense embedders. We introduce COMET, a framework for connecting retrieval-failure diagnosis with targeted adaptation through five recurring failure patterns: **c**oncentration, **o**ver-fragmentation, **m**emory-query form mismatch, **e**ntity mismatch and **t**emporal mismatch. First, we **diagnose** which failure patterns are most prominent in long-term memory retrieval. Across three memory systems and two benchmarks (LoCoMo and PersonaMem), we operationalize the five failure patterns as measurable proxies, finding that they significantly predict held-out retrieval failures beyond standard controls. We find particularly substantial evidence for over-fragmentation, memory-query form mismatch and entity mismatch. Building on these results, we **adapt** three embedding models across families and scales by constructing 1250 queries from an independent dataset, with 250 targeting each of the five failure patterns. We find that O-250, M-250 and E-250, the three datasets corresponding to the most prominent failure patterns, generally produce the most effective LoRA adapters. OME-750, the adapter trained from their union, is our best-performing adapter overall, generally outperforming all 1250 queries combined; on Qwen3-Embedding-0.6B, it improves mean Recall@10 by **7.70 percentage points** above base and by **3.08 points** above matched-compute training on untargeted memory-retrieval datasets. Final answer generation accuracy also improves on LoCoMo by **8.57 points**. The adapter further improves nDCG@10 on eight benchmarks from the LMEB suite by **8.22 points**, matching EvoEmbedding-0.8B, a memory-specialized embedder trained with **245 times** as many independent queries. These findings suggest a general principle connecting memory-retrieval failures and retrieval adaptation: first diagnose, then adapt.