- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes COMET, a diagnosis-guided framework for improving retrieval in long-term agent memory systems. The authors identify five recurring retrieval failure patterns. They quantify these patterns using manually designed proxies and statistical regression analysis, finding that over-fragmentation, form mismatch, and entity mismatch are the most prominent factors associated with retrieval failures. Based on these findings, the authors construct targeted training queries and fine-tune existing embedding models using contrastive learning and LoRA.
Critical strengths
-
The paper provides a structured characterization of retrieval difficulties in long-term agent memory systems. The statistical analysis across multiple memory systems, benchmarks, and embedding models offers useful empirical evidence about factors associated with retrieval failures.
-
The experiments cover multiple embedding backbones and include several meaningful controls, such as generic fine-tuning, generic query augmentation, and hard-example selection. The proposed OME-750 adapter consistently improves aggregate retrieval performance over matched generic baselines across the three evaluated backbones.
Critical weaknesses
-
The main components of COMET are largely based on established techniques. The five failure patterns correspond to well-known retrieval challenges, including multi-hop retrieval, lexical or semantic mismatch, entity resolution, and temporal alignment. Their quantification relies on relatively simple heuristic proxies, followed by standard regression analysis. The subsequent adaptation uses conventional query augmentation, contrastive learning, and LoRA fine-tuning. The primary contribution therefore appears to be a strategy for selecting targeted training examples based on statistical associations with retrieval failures, rather than a fundamentally new retrieval or adaptation method.
-
The paper does not convincingly establish that the diagnosis stage is necessary for achieving the reported improvements. The three selected categories (over-fragmentation, form mismatch, and entity mismatch) represent common retrieval challenges that could plausibly have been chosen directly without conducting the proposed statistical diagnosis. Although the experiments demonstrate that training on these categories is effective, they do not establish whether the diagnosis procedure provides additional value beyond selecting known challenging retrieval scenarios. This substantially weakens the paper's central claim of introducing a general diagnosis-guided adaptation framework.
-
The paper assumes that failure patterns exhibiting stronger statistical associations with retrieval errors should receive greater emphasis during adaptation. However, statistical significance does not necessarily indicate that a particular failure pattern is more amenable to improvement through embedding fine-tuning. While the OME-750 adapter performs well, the experiments do not establish a clear relationship between the diagnosed failure patterns and the improvements obtained from their corresponding specialist adapters.
-
Several observations further raise concerns:
(1) The memory-query form mismatch proxy shows almost no change after targeted augmentation (0.788 to 0.779), despite this category being one of the most effective adaptation targets. This raises questions about whether the proposed diagnostic proxy adequately captures the difficulty being addressed.
(2) The same OME-750 training configuration is applied across different memory systems and embedding models, despite the paper identifying variations in their failure profiles. Consequently, the experiments do not demonstrate whether system-specific diagnosis actually leads to more effective adaptation strategies.
(3) Although OME-750 improves LoCoMo answering accuracy by 8.57 percentage points over the base model, its improvement over the matched generic augmentation baseline is only approximately 0.5 percentage points and is not statistically significant. The downstream benefit attributable specifically to diagnosis-guided adaptation therefore remains limited.
Questions for authors
-
Is the diagnosis stage actually necessary for the reported improvements? The three selected failure patterns correspond to well-established retrieval challenges. What additional benefit does the statistical diagnosis provide compared with directly constructing training examples for these categories? Could the authors provide an ablation or alternative selection strategy that isolates the contribution of diagnosis itself?
-
Does the effectiveness of diagnosis-guided adaptation depend on the failure profile of the target system? The paper identifies variations in failure profiles across memory systems and benchmarks, but ultimately uses the same OME-750 adapter. Would selecting or weighting training categories according to each system's diagnosed failure profile outperform the fixed OME-750 configuration? Such evidence would substantially strengthen the claim that diagnosis meaningfully guides adaptation.
-
How can the authors establish a stronger connection between individual failure patterns and their corresponding adaptation improvements? For instance, the form mismatch proxy barely changes after targeted augmentation, yet the M-250 adapter performs well. Does this indicate limitations in the diagnostic proxy, or that the improvements arise from other characteristics of the generated training data? A more direct analysis of pattern-specific improvements would help clarify the underlying mechanism.
7 comments
Response to Reviewer 4Wys (2/7): R2: Is the diagnosis stage necessary?
[R2: Is the diagnosis stage necessary?]
Weakness 2: "The paper does not convincingly establish that the diagnosis stage is necessary for achieving the reported improvements. The three selected categories (over-fragmentation, form mismatch, and entity mismatch) represent common retrieval challenges that could plausibly have been chosen directly without conducting the proposed statistical diagnosis. Although the experiments demonstrate that training on these categories is effective, they do not establish whether the diagnosis procedure provides additional value beyond selecting known challenging retrieval scenarios."
Question 1: "Is the diagnosis stage actually necessary for the reported improvements? The three selected failure patterns correspond to well-established retrieval challenges. What additional benefit does the statistical diagnosis provide compared with directly constructing training examples for these categories? Could the authors provide an ablation or alternative selection strategy that isolates the contribution of diagnosis itself?"
To isolate the contribution of diagnosis itself, we perform the diagnosis independently on two additional benchmarks, PersonaMem and REALTALK, and find that on every benchmark, the diagnosis significantly predicts adaptation performance: the more system/embedder/top- settings in which a mix's categories are significant on a benchmark, the better that mix's adapter performs on it. The diagnosis for PersonaMem selects C, M, E instead of O, M, E; the diagnosis for REALTALK selects O, M, E, as for LoCoMo.
Pearson correlation between each of the 10 three-category mixes' diagnosis score on one benchmark (rows) and its retrieval performance on another (columns):
| Diagnosis from | Performance on LoCoMo | Performance on PersonaMem | Performance on REALTALK |
|---|---|---|---|
| LoCoMo (selects O, M, E) | 0.77 (p = 0.009) | 0.52 (p = 0.12) | 0.78 (p = 0.008) |
| PersonaMem (selects C, M, E) | 0.35 (p = 0.32) | 0.83 (p = 0.003) | 0.42 (p = 0.23) |
| REALTALK (selects O, M, E) | 0.70 (p = 0.024) | 0.51 (p = 0.13) | 0.77 (p = 0.009) |
Here, we trained 10 adapters (one for each possible three-category mixture), calculated the average number of significant regressions for the adapters' three categories (using the paper's lexical M proxy), and found its correlation with adapter performance on the benchmarks. We note that all the diagonal correlations are significant under two-sided -values (standard test for Pearson's ; bold: p < 0.05), meaning each benchmark's diagnosis predicts adaptation performance on that benchmark. REALTALK performance uses LMEB's evaluation of REALTALK (nDCG@10), the benchmark's official protocol; the other two benchmarks use Recall@10.
Off the diagonal, correlations are significant only between LoCoMo and REALTALK (in both directions), because they are very similar in style (both conversational dialogue with targeted questions) and have very similar diagnoses ( and identical at 15, being 13 on LoCoMo vs. 11 on REALTALK, being 5 vs. 4, and being 6 vs. 2 for retrieval failure, with the same ranking and diagnosis scores correlating at , against 0.29 and 0.34 with PersonaMem's).
Here, we lay out the performance of the three-category mixtures on PersonaMem, LoCoMo, and REALTALK. As shown, all diagnosed adapters are statistically indistinguishable from the top-ranked adapter.
| Benchmark | Diagnosed mix | Gap to 1st | Spread, 1st to 10th | Bottom two |
|---|---|---|---|---|
| PersonaMem | CME (2nd) | 0.41 pp (95% CI [−0.84, +1.61], n.s.) | 3.66 pp | OET, COT |
| LoCoMo | OME (1st) | 0 | 2.42 pp | CMT, CMO |
| REALTALK | OME (2nd) | 0.06 pp (95% CI [−1.52, +1.78], n.s.) | 2.52 pp | CMT, COT |
Response to Reviewer 4Wys (3/7): R2 continued
In addition, the difference in diagnosis results between LoCoMo/REALTALK and PersonaMem are meaningful. The diagnoses differ in a single slot: LoCoMo/REALTALK selects O and PersonaMem selects C. Comparing each mix containing O with the matching mix containing C, with the other two categories held fixed:
| Other two categories | LoCoMo: O − C | PersonaMem: O − C | REALTALK: O − C |
|---|---|---|---|
| M, E (OME vs CME) | +0.43 | +0.38 | +0.80 |
| M, T (MOT vs CMT) | +1.07 (p < 0.001) | −0.53 | +0.76 |
| E, T (OET vs CET) | +1.79 (p < 0.001) | −0.43 | +1.01 (p = 0.044) |
| Mean | +1.10 (p < 0.001) | −0.19 | +0.85 (p = 0.024) |
(Recall@10 on LoCoMo and PersonaMem, three-seed means per question over the three memory systems; nDCG@10 on REALTALK, seed 42; Qwen3-Embedding-0.6B; paired cluster bootstrap over conversations/personas, one-sided p.) O outperforms C in all three swaps on LoCoMo and REALTALK, where diagnosis selects O, but not on PersonaMem, where diagnosis selects C; the difference between LoCoMo and PersonaMem is +1.29 points (significant with one-sided p = 0.018).
The above provides evidence that diagnosis separates strong adapters from weak adapters by identifying which general failure modes are most prevalent (we select three categories for consistency with the paper, and because there is a clear separation between the 3rd-ranked category and the bottom two on all three benchmarks).
This is related to the reviewer's point that "statistical significance does not necessarily indicate that a particular failure pattern is more amenable to improvement through embedding fine-tuning": across the 10 mixes, the categories more strongly associated with retrieval failure on a benchmark are also the ones whose training data produces the better adapters on that benchmark (Pearson = 0.77, 0.83 and 0.77 on LoCoMo, PersonaMem and REALTALK above), so the association measured by the diagnosis tracks how much fine-tuning on a category helps.
The reviewer suggests that O, M and E are "well-established retrieval challenges" and could have been selected without diagnosis. We acknowledge that, for instance, co-referring entity names affecting retrieval is well-known, but we argue that so are the other two categories (concentration, C, and temporal mismatch, T) which were not selected by diagnosis. Both LoCoMo and LongMemEval (Maharana et al., 2024; Wu et al., 2025) have categories specifically dedicated to temporal reasoning, and past research such as TEMPS has shown that dense retrievers match on topic but poorly on time, retrieving content that is on-topic yet temporally wrong (Hassani et al., 2026). Similarly, the "needle-in-a-haystack" effect where embedders fail to retrieve information buried beneath long distractor passages is well-documented in works such as LIMIT (Weller et al., 2025).
As such, if you wanted to fine-tune an embedder for memory retrieval and you were armed with results from past research suggesting that temporal mismatch and concentration were substantial issues, you may very plausibly believe that they would be effective adaptation targets; and yet, as we show with COMET-1250 which includes C and T along with OME-750, the marginal value of C and T on LoCoMo, where neither is diagnosed, is small, even negative. The value of the diagnostic step is that it bypasses the need for heuristic judgments and instead defers to empirical evidence to suggest which failure modes are most prevalent.
Response to Reviewer 4Wys (1/7): overview; R1: Methodological novelty
We thank the reviewer for their thoughtful comments, and for recognizing the value of the statistical analysis across memory systems, benchmarks and embedders, as well as the range of controls. The reviewer raises a number of reasonable concerns which we address below: 1) methodological novelty; 2) whether the diagnosis stage is necessary; 3) the connection between individual failure patterns and their adaptation gains, including the memory-query form mismatch proxy; 4) whether system-specific diagnosis leads to more effective adaptation; and 5) the answering gain over matched controls. We have also included further experiments in the general comment above.
[R1: Methodological novelty]
We agree that COMET does not propose a new model architecture or training algorithm; however, we believe its novelty originates from the following aspects: