acceptodds
Under review as a conference paper at ICLR 2027

COMET: A Framework for Diagnosis-Guided Retrieval Adaptation in Agent Memory

Abstract

Agent memory systems increasingly refine how experience is stored, but memory retrieval often remains reliant on general-purpose dense embedders. We introduce COMET, a framework for connecting retrieval-failure diagnosis with targeted adaptation through five recurring failure patterns: **c**oncentration, **o**ver-fragmentation, **m**emory-query form mismatch, **e**ntity mismatch and **t**emporal mismatch. First, we **diagnose** which failure patterns are most prominent in long-term memory retrieval. Across three memory systems and two benchmarks (LoCoMo and PersonaMem), we operationalize the five failure patterns as measurable proxies, finding that they significantly predict held-out retrieval failures beyond standard controls. We find particularly substantial evidence for over-fragmentation, memory-query form mismatch and entity mismatch. Building on these results, we **adapt** three embedding models across families and scales by constructing 1250 queries from an independent dataset, with 250 targeting each of the five failure patterns. We find that O-250, M-250 and E-250, the three datasets corresponding to the most prominent failure patterns, generally produce the most effective LoRA adapters. OME-750, the adapter trained from their union, is our best-performing adapter overall, generally outperforming all 1250 queries combined; on Qwen3-Embedding-0.6B, it improves mean Recall@10 by **7.70 percentage points** above base and by **3.08 points** above matched-compute training on untargeted memory-retrieval datasets. Final answer generation accuracy also improves on LoCoMo by **8.57 points**. The adapter further improves nDCG@10 on eight benchmarks from the LMEB suite by **8.22 points**, matching EvoEmbedding-0.8B, a memory-specialized embedder trained with **245 times** as many independent queries. These findings suggest a general principle connecting memory-retrieval failures and retrieval adaptation: first diagnose, then adapt.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, embedding models, memory retrieval, retrieval-augmented generation, long-term memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_4WysOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes COMET, a diagnosis-guided framework for improving retrieval in long-term agent memory systems. The authors identify five recurring retrieval failure patterns. They quantify these patterns using manually designed proxies and statistical regression analysis, finding that over-fragmentation, form mismatch, and entity mismatch are the most prominent factors associated with retrieval failures. Based on these findings, the authors construct targeted training queries and fine-tune existing embedding models using contrastive learning and LoRA.

Critical strengths
  1. The paper provides a structured characterization of retrieval difficulties in long-term agent memory systems. The statistical analysis across multiple memory systems, benchmarks, and embedding models offers useful empirical evidence about factors associated with retrieval failures.

  2. The experiments cover multiple embedding backbones and include several meaningful controls, such as generic fine-tuning, generic query augmentation, and hard-example selection. The proposed OME-750 adapter consistently improves aggregate retrieval performance over matched generic baselines across the three evaluated backbones.

Critical weaknesses
  1. The main components of COMET are largely based on established techniques. The five failure patterns correspond to well-known retrieval challenges, including multi-hop retrieval, lexical or semantic mismatch, entity resolution, and temporal alignment. Their quantification relies on relatively simple heuristic proxies, followed by standard regression analysis. The subsequent adaptation uses conventional query augmentation, contrastive learning, and LoRA fine-tuning. The primary contribution therefore appears to be a strategy for selecting targeted training examples based on statistical associations with retrieval failures, rather than a fundamentally new retrieval or adaptation method.

  2. The paper does not convincingly establish that the diagnosis stage is necessary for achieving the reported improvements. The three selected categories (over-fragmentation, form mismatch, and entity mismatch) represent common retrieval challenges that could plausibly have been chosen directly without conducting the proposed statistical diagnosis. Although the experiments demonstrate that training on these categories is effective, they do not establish whether the diagnosis procedure provides additional value beyond selecting known challenging retrieval scenarios. This substantially weakens the paper's central claim of introducing a general diagnosis-guided adaptation framework.

  3. The paper assumes that failure patterns exhibiting stronger statistical associations with retrieval errors should receive greater emphasis during adaptation. However, statistical significance does not necessarily indicate that a particular failure pattern is more amenable to improvement through embedding fine-tuning. While the OME-750 adapter performs well, the experiments do not establish a clear relationship between the diagnosed failure patterns and the improvements obtained from their corresponding specialist adapters.

  4. Several observations further raise concerns:

(1) The memory-query form mismatch proxy shows almost no change after targeted augmentation (0.788 to 0.779), despite this category being one of the most effective adaptation targets. This raises questions about whether the proposed diagnostic proxy adequately captures the difficulty being addressed.

(2) The same OME-750 training configuration is applied across different memory systems and embedding models, despite the paper identifying variations in their failure profiles. Consequently, the experiments do not demonstrate whether system-specific diagnosis actually leads to more effective adaptation strategies.

(3) Although OME-750 improves LoCoMo answering accuracy by 8.57 percentage points over the base model, its improvement over the matched generic augmentation baseline is only approximately 0.5 percentage points and is not statistically significant. The downstream benefit attributable specifically to diagnosis-guided adaptation therefore remains limited.

Questions for authors
  1. Is the diagnosis stage actually necessary for the reported improvements? The three selected failure patterns correspond to well-established retrieval challenges. What additional benefit does the statistical diagnosis provide compared with directly constructing training examples for these categories? Could the authors provide an ablation or alternative selection strategy that isolates the contribution of diagnosis itself?

  2. Does the effectiveness of diagnosis-guided adaptation depend on the failure profile of the target system? The paper identifies variations in failure profiles across memory systems and benchmarks, but ultimately uses the same OME-750 adapter. Would selecting or weighting training categories according to each system's diagnosed failure profile outperform the fixed OME-750 configuration? Such evidence would substantially strengthen the claim that diagnosis meaningfully guides adaptation.

  3. How can the authors establish a stronger connection between individual failure patterns and their corresponding adaptation improvements? For instance, the form mismatch proxy barely changes after targeted augmentation, yet the M-250 adapter performs well. Does this indicate limitations in the diagnostic proxy, or that the improvements arise from other characteristics of the generated training data? A more direct analysis of pattern-specific improvements would help clarify the underlying mechanism.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4Wys (1/7): overview; R1: Methodological novelty

We thank the reviewer for their thoughtful comments, and for recognizing the value of the statistical analysis across memory systems, benchmarks and embedders, as well as the range of controls. The reviewer raises a number of reasonable concerns which we address below: 1) methodological novelty; 2) whether the diagnosis stage is necessary; 3) the connection between individual failure patterns and their adaptation gains, including the memory-query form mismatch proxy; 4) whether system-specific diagnosis leads to more effective adaptation; and 5) the answering gain over matched controls. We have also included further experiments in the general comment above.

[R1: Methodological novelty]

Weakness 1: "The primary contribution therefore appears to be a strategy for selecting targeted training examples based on statistical associations with retrieval failures, rather than a fundamentally new retrieval or adaptation method."

We agree that COMET does not propose a new model architecture or training algorithm; however, we believe its novelty originates from the following aspects:

  1. The contribution of a memory-specialized embedder itself. To the best of our knowledge, besides concurrent work, the main papers exploring fine-tuned embedders for long-term memory retrieval are EvoEmbedding and HiNS; of the two, HiNS has no released weights, and EvoEmbedding introduces a sequential context architecture that incurs additional latency (context encoding 22.08s vs 3.80s for Qwen3-Embedding-4B, about 5.8x). We believe that focusing on the embedding model used, rather than the architecture used to store memories, is an approach that is itself underexplored and deserves more attention, particularly because a LoRA adapter can be plug-and-play across many memory systems and offers demonstrable retrieval gains at little additional cost.
  2. The diagnosis-to-adaptation loop. One of the primary strengths of our approach is its data efficiency; we close the gap to large-scale training through diagnosis, identifying which failure modes are most significant in memory tasks and using precisely these failure modes in constructing training data for adaptation. In contrast, EvoEmbedding uses EvoTrain-180K, a generalist dataset with more than 180,000 independent queries that does not select for specific failure modes suitable to memory tasks, and achieves comparable results to our 750- to 3000-query datasets at similar model scales. In the responses below, we provide further evidence for the link between diagnosis and adaptation.
AuthorsOpenReview ↗

Response to Reviewer 4Wys (2/7): R2: Is the diagnosis stage necessary?

[R2: Is the diagnosis stage necessary?]

Weakness 2: "The paper does not convincingly establish that the diagnosis stage is necessary for achieving the reported improvements. The three selected categories (over-fragmentation, form mismatch, and entity mismatch) represent common retrieval challenges that could plausibly have been chosen directly without conducting the proposed statistical diagnosis. Although the experiments demonstrate that training on these categories is effective, they do not establish whether the diagnosis procedure provides additional value beyond selecting known challenging retrieval scenarios."

Question 1: "Is the diagnosis stage actually necessary for the reported improvements? The three selected failure patterns correspond to well-established retrieval challenges. What additional benefit does the statistical diagnosis provide compared with directly constructing training examples for these categories? Could the authors provide an ablation or alternative selection strategy that isolates the contribution of diagnosis itself?"

To isolate the contribution of diagnosis itself, we perform the diagnosis independently on two additional benchmarks, PersonaMem and REALTALK, and find that on every benchmark, the diagnosis significantly predicts adaptation performance: the more system/embedder/top-kk settings in which a mix's categories are significant on a benchmark, the better that mix's adapter performs on it. The diagnosis for PersonaMem selects C, M, E instead of O, M, E; the diagnosis for REALTALK selects O, M, E, as for LoCoMo.

Pearson correlation between each of the 10 three-category mixes' diagnosis score on one benchmark (rows) and its retrieval performance on another (columns):

Diagnosis fromPerformance on LoCoMoPerformance on PersonaMemPerformance on REALTALK
LoCoMo (selects O, M, E)0.77 (p = 0.009)0.52 (p = 0.12)0.78 (p = 0.008)
PersonaMem (selects C, M, E)0.35 (p = 0.32)0.83 (p = 0.003)0.42 (p = 0.23)
REALTALK (selects O, M, E)0.70 (p = 0.024)0.51 (p = 0.13)0.77 (p = 0.009)

Here, we trained 10 adapters (one for each possible three-category mixture), calculated the average number of significant regressions for the adapters' three categories (using the paper's lexical M proxy), and found its correlation with adapter performance on the benchmarks. We note that all the diagonal correlations are significant under two-sided pp-values (standard test for Pearson's rr; bold: p < 0.05), meaning each benchmark's diagnosis predicts adaptation performance on that benchmark. REALTALK performance uses LMEB's evaluation of REALTALK (nDCG@10), the benchmark's official protocol; the other two benchmarks use Recall@10.

Off the diagonal, correlations are significant only between LoCoMo and REALTALK (in both directions), because they are very similar in style (both conversational dialogue with targeted questions) and have very similar diagnoses (OO and EE identical at 15, MM being 13 on LoCoMo vs. 11 on REALTALK, CC being 5 vs. 4, and TT being 6 vs. 2 for retrieval failure, with the same ranking O=E>M>C>TO = E > M > C > T and diagnosis scores correlating at r=0.98r = 0.98, against 0.29 and 0.34 with PersonaMem's).

Here, we lay out the performance of the three-category mixtures on PersonaMem, LoCoMo, and REALTALK. As shown, all diagnosed adapters are statistically indistinguishable from the top-ranked adapter.

BenchmarkDiagnosed mixGap to 1stSpread, 1st to 10thBottom two
PersonaMemCME (2nd)0.41 pp (95% CI [−0.84, +1.61], n.s.)3.66 ppOET, COT
LoCoMoOME (1st)02.42 ppCMT, CMO
REALTALKOME (2nd)0.06 pp (95% CI [−1.52, +1.78], n.s.)2.52 ppCMT, COT
AuthorsOpenReview ↗

Response to Reviewer 4Wys (3/7): R2 continued

In addition, the difference in diagnosis results between LoCoMo/REALTALK and PersonaMem are meaningful. The diagnoses differ in a single slot: LoCoMo/REALTALK selects O and PersonaMem selects C. Comparing each mix containing O with the matching mix containing C, with the other two categories held fixed:

Other two categoriesLoCoMo: O − CPersonaMem: O − CREALTALK: O − C
M, E (OME vs CME)+0.43+0.38+0.80
M, T (MOT vs CMT)+1.07 (p < 0.001)−0.53+0.76
E, T (OET vs CET)+1.79 (p < 0.001)−0.43+1.01 (p = 0.044)
Mean+1.10 (p < 0.001)−0.19+0.85 (p = 0.024)

(Recall@10 on LoCoMo and PersonaMem, three-seed means per question over the three memory systems; nDCG@10 on REALTALK, seed 42; Qwen3-Embedding-0.6B; paired cluster bootstrap over conversations/personas, one-sided p.) O outperforms C in all three swaps on LoCoMo and REALTALK, where diagnosis selects O, but not on PersonaMem, where diagnosis selects C; the difference between LoCoMo and PersonaMem is +1.29 points (significant with one-sided p = 0.018).

The above provides evidence that diagnosis separates strong adapters from weak adapters by identifying which general failure modes are most prevalent (we select three categories for consistency with the paper, and because there is a clear separation between the 3rd-ranked category and the bottom two on all three benchmarks).

This is related to the reviewer's point that "statistical significance does not necessarily indicate that a particular failure pattern is more amenable to improvement through embedding fine-tuning": across the 10 mixes, the categories more strongly associated with retrieval failure on a benchmark are also the ones whose training data produces the better adapters on that benchmark (Pearson rr = 0.77, 0.83 and 0.77 on LoCoMo, PersonaMem and REALTALK above), so the association measured by the diagnosis tracks how much fine-tuning on a category helps.

The reviewer suggests that O, M and E are "well-established retrieval challenges" and could have been selected without diagnosis. We acknowledge that, for instance, co-referring entity names affecting retrieval is well-known, but we argue that so are the other two categories (concentration, C, and temporal mismatch, T) which were not selected by diagnosis. Both LoCoMo and LongMemEval (Maharana et al., 2024; Wu et al., 2025) have categories specifically dedicated to temporal reasoning, and past research such as TEMPS has shown that dense retrievers match on topic but poorly on time, retrieving content that is on-topic yet temporally wrong (Hassani et al., 2026). Similarly, the "needle-in-a-haystack" effect where embedders fail to retrieve information buried beneath long distractor passages is well-documented in works such as LIMIT (Weller et al., 2025).

As such, if you wanted to fine-tune an embedder for memory retrieval and you were armed with results from past research suggesting that temporal mismatch and concentration were substantial issues, you may very plausibly believe that they would be effective adaptation targets; and yet, as we show with COMET-1250 which includes C and T along with OME-750, the marginal value of C and T on LoCoMo, where neither is diagnosed, is small, even negative. The value of the diagnostic step is that it bypasses the need for heuristic judgments and instead defers to empirical evidence to suggest which failure modes are most prevalent.

Reviewer_YJEaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes COMET, a framework for improving long-term memory retrieval by identifying five common retrieval failure patterns and constructing targeted training examples from the PerLTQA dataset. The authors identify three particularly important failure patterns and use LLM-generated query augmentations targeting them to fine-tune embedding models with LoRA. The authors report that training on 750 targeted examples improves retrieval performance more than fine-tuning on equally sized randomly sampled or generically augmented datasets, with gains transferring to downstream question answering and external retrieval benchmarks.

Critical strengths
  1. Principled and systematic approach. I appreciate the paper's core idea: diagnose retrieval failure modes, construct targeted training data to address them, and empirically evaluate whether this improves retrieval. This is a compelling direction compared with generic data augmentation and could provide a useful framework for improving agent memory systems.

  2. Promising data efficiency. The OME-750 adapter achieves improvements over generic augmentation using only 750 training examples, with gains across three embedding backbones and some transfer to external benchmarks. Although the evidence is not yet sufficient to validate the proposed diagnosis-guided mechanism, these results suggest that carefully curated training data can be effective.

Critical weaknesses

Overall Assessment

The paper presents many results but, unfortunately, is difficult to follow, with critical methodological details relegated to the appendices, particularly the construction of targeted training subsets (C-250, O-250, etc.). More importantly, although OME-750 achieves promising results, the paper provides insufficient evidence that these gains arise from diagnosing retrieval failure modes rather than differences in data construction and quality. COMET-1250 shows negligible or negative improvements over generic augmentation on LMEB, category selection based on LoCoMo introduces potential evaluation-selection bias, and all training data originates from PerLTQA. Overall, the evidence does not convincingly support the paper's central diagnosis-guided adaptation claim.

1. Insufficient evidence for diagnosis-guided adaptation

a. Insufficient evidence that diagnosing failure modes improves retrieval. The paper does not convincingly demonstrate that training on a specific failure category improves performance on examples exhibiting that failure mode more than other specialist adapters or generic training. Appendix D20 provides preliminary evidence for C, O, and E, but lacks direct comparisons between adapters. Moreover, the M-250 augmentation barely changes its targeted failure proxy (0.788 → 0.779, Table D13), raising concerns about whether the examples successfully instantiate the intended difficulty. The authors themselves acknowledge in Section 5 that a one-to-one causal mapping between failure patterns and specialist adapters is not established. Given that diagnosis-guided adaptation is the central contribution, this missing evidence substantially weakens the paper's conclusions.

b. Marginal improvements over generic augmentation. While OME-750 achieves meaningful improvements over Placebo-750 (e.g., +1.94 nDCG@10 on LMEB for Qwen3-0.6B), the full COMET-1250 dataset performs similarly to Placebo-1250 on LMEB:

  • Qwen3-0.6B: 57.67 vs. 57.23 (+0.44)
  • Qwen3-4B: 59.63 vs. 61.15 (−1.52)
  • BGE-large: 57.35 vs. 56.89 (+0.46)

LMEB is particularly important for evaluating transferability in contrast to LoCoMo+PersonaMem, as LoCoMo is used to select the failure categories and subsequently evaluate the models. The stronger OME-750 results are promising, but do not establish that the improvements arise from diagnosis-guided selection rather than differences in training-data quality or augmentation. Furthermore, the weaker performance of COMET-1250 raises questions about the contribution of C and T to the proposed five-category framework.

c. Unfair placebo comparison. Appendix B.1 acknowledges that COMET examples undergo stricter filtering and validation than placebo examples. Improvements may therefore result from better data quality rather than failure-targeted augmentation.

2. Insufficiently controlled methodology and limited generalizability

d. Insufficiently explained and validated data construction. Constructing the targeted subsets is central to the method, yet the procedure is largely relegated to the appendices. The correctness of generated examples and their assigned failure categories is not extensively validated.

e. Limited generalizability. All targeted training examples come exclusively from PerLTQA. Using additional source datasets is important to rule out dataset-specific biases and establish that the approach generalizes.

Questions for authors

Questions for authors — clarifications:

  1. Dataset construction: Appendix B.1 and Appendix F describe the prompts used to construct C-250, O-250, M-250, E-250, and T-250. However, this is the central methodological contribution and deserves a clearer explanation in the main text. How was the validity of the constructed examples assessed independently of the construction agent, particularly whether they correctly instantiate the intended failure pattern?

  2. Control datasets: Appendix B.1 states that Placebo-750 uses generic query augmentation but does not apply the same rejection and validation procedure as the targeted datasets. Could this difference in filtering rigor contribute to the observed improvements? How would the results change with equally validated placebo examples?

  3. Selection of OME-750: Was the decision to combine O, M, and E made exclusively based on the regression results, before examining any adapter performance? Were other three-category combinations evaluated during development?

Suggestions for further experiments:

  1. Failure-specific improvements: Please provide a category-by-adapter evaluation showing whether each specialist adapter improves retrieval on examples exhibiting its targeted failure mode more than other adapters and generic fine-tuning. This would provide stronger evidence for the proposed diagnosis-guided adaptation principle.

  2. Broader validation: The approach would benefit from constructing targeted training examples using at least one additional source dataset beyond PerLTQA [1] and independently repeating the failure diagnosis and category-selection procedure on additional benchmarks (e.g., Longmemeval [2] and Realtalk [3]), and evaluating whether the resulting category selection consistently predicts which targeted adapters perform best.

  3. Human validation: Appendix E audits only 20 cases per benchmark–memory-system combination with one human annotator. Given the dependence on LLM annotations and generated training examples, I recommend expanding the audit to approximately 100 cases per combination, involving multiple independent annotators, reporting inter-annotator agreement, and explicitly auditing the correctness of COMET-generated queries.

  4. Presentation: The paper is difficult to follow because essential methodological details, including the dataset construction procedure and regression specifications, are primarily explained in the appendices. These elements should be presented more clearly in the main methodology section.

[1] Du, Yiming, et al. "Perltqa: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering." Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10). 2024.

[2] Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." arXiv preprint arXiv:2410.10813 (2024).

[3] Lee, Dong-Ho, et al. "Realtalk: A 21-day real-world dataset for long-term conversation." arXiv preprint arXiv:2502.13270 (2025).

13 comments
AuthorsOpenReview ↗

Response to Reviewer YJEa (1/10): overview

We thank the reviewer for their detailed and careful review, and for recognizing the core idea of diagnosing retrieval failure modes and constructing targeted training data to address them, as well as its data efficiency. Below we address the reviewer's concerns: 1) category-specific improvements and the memory-query form mismatch proxy; 2) COMET-1250 and the contribution of C and T; 3) placebo fairness; 4) validation of the constructed data and human audits; 5) generalizability beyond PerLTQA and LoCoMo; 6) the selection of OME-750; and 7) presentation. Additional experimental results have been posted to a general comment above.

AuthorsOpenReview ↗

Response to Reviewer YJEa (2/10): R1: Category-specific improvements

[R1: Category-specific improvements]

Weakness 1a: "Moreover, the M-250 augmentation barely changes its targeted failure proxy (0.788 → 0.779, Table D13), raising concerns about whether the examples successfully instantiate the intended difficulty."

Suggestion 1: "Please provide a category-by-adapter evaluation showing whether each specialist adapter improves retrieval on examples exhibiting its targeted failure mode more than other adapters and generic fine-tuning."

The memory-query form mismatch proxy. We agree that the existing lexical proxy used for the M category is not fully sufficient for explaining the category's mechanism. As such, we provide a new proxy for memory-query form mismatch that the M-250 dataset significantly shifts toward greater mismatch and is evidenced to measure the intended effect: the degree to which the relationship between a query and its memories are implicit and non-obvious. The proxy is defined using a cross-encoder reranker: the cross-encoder (Qwen3-Reranker-8B) reads the query and memory together and judges whether the memory is relevant to the query, from which we subtract the average score that the reranker assigns to all candidate memories for the query. A lower score means the model had more difficulty relating the memory to the query directly, controlling for the difficulty of the query in general. The results are as follows:

  1. The M-250 rewrites make the query–memory link measurably less direct. Relative to their unmodified source questions, M-250 rewrites lower the score by 2.63 log-odds; a second reranker from a different family (bge-reranker-v2-m3) gives −2.92. Placebo rewrites of the same kind of source questions leave it unchanged (about 0). As a positive control, rewriting the same source questions to be more direct (quoting the evidence span) moves the score in the opposite direction by a similar amount (+3.34), suggesting that the proxy captures the desired effect. Rewrites targeting the other four categories move the score far less (C −0.28, T −0.21, O −0.15 n.s.) or in the opposite direction (E +0.93), so the proxy responds specifically to the implicitness introduced by M-250 rather than to rewriting or difficulty in general.
Change in score (rewrite − source, log-odds)Qwen3-Reranker-8Bbge-reranker-v2-m3
M-250 rewrites−2.63 (p < 0.001)−2.92 (p < 0.001)
Placebo rewrites−0.08 (n.s.)−0.09 (n.s.)
C-250 rewrites−0.28 (p = 0.001)−0.66 (p < 0.001)
T-250 rewrites−0.21 (p = 0.032)−0.80 (p < 0.001)
O-250 rewrites−0.15 (n.s.)+0.33 (n.s.)
E-250 rewrites+0.93 (p < 0.001)+1.47 (p < 0.001)
More-direct rewrites (positive control)+3.34 (p < 0.001)+2.86 (p < 0.001)
  1. Both the new proxy and the paper's lexical proxy track human judgements of implicitness. In two blinded pairwise audits on LoCoMo questions (single-gold questions; top vs. bottom 15% of each proxy; pairs from the same conversation, matched on question length; 40 pairs each, sides randomized, proxy values hidden), a blinded human rater chose which question's link to its memory is less direct:
Proxy used to select the pairsHigh-proxy side judged less directLow-proxy side judged less directToo close to callOne-sided binomial p
New reranker proxy25015< 10⁻⁷
Paper's lexical proxy226120.002

A second, independent rater from outside the project confirms both results: 35 / 0 / 5 for the reranker proxy (p < 10⁻¹⁰) and 24 / 1 / 15 for the lexical proxy (p < 10⁻⁶).

  1. Memory-query form mismatch remains a significant failure predictor under the new proxy. Substituting the new proxy into the paper's diagnosis regressions, it is significant in all 15 settings on both LoCoMo and PersonaMem (retrieval failure). The other diagnosed categories also remain significant alongside it: on LoCoMo, over-fragmentation remains significant in 15/15 settings and entity mismatch in 13/15, so the diagnosis still selects O, M and E.
AuthorsOpenReview ↗

Response to Reviewer YJEa (3/10): R1 continued

Together, these show that the M-250 data instantiates the intended difficulty and that the paper's proxy measures it, even though it does so coarsely: it counts shared form words, while M-250 rewrites change how implicit the link is more than which words are shared, which is why the lexical proxy barely changed.

Category-by-adapter evaluation. On LoCoMo, the O, M and E adapters significantly improve performance specifically on questions belonging to their own category. We compare the Recall@10 gain of the C, O, M, E, T-250 adapters for Qwen3-Embedding-0.6B on queries scoring highly on the categories' proxies (above the median) to queries scoring low (below the median), averaged over three seeds and three systems. We also compare whether the adapters gain more over the placebo on their corresponding categories on high-proxy queries compared to low-proxy queries. Both comparisons are supported by a cluster bootstrap over conversations. The results are as follows:

CategoryLoCoMo: own adapter vs. other adaptersLoCoMo: own adapter vs. placeboREALTALK: own adapter vs. other adaptersREALTALK: own adapter vs. placebo
C+0.59 (p = 0.097)+0.36 (p = 0.225)−0.41 (p = 0.657)−0.78 (p = 0.800)
O+3.09 (p = 0.001)+2.42 (p = 0.010)+3.97 (p < 0.001)+4.07 (p = 0.004)
M+1.94 (p < 0.001)+1.84 (p = 0.002)+6.61 (p < 0.001)+7.04 (p < 0.001)
E+1.06 (p = 0.009)+1.45 (p = 0.003)+1.35 (p = 0.040)+1.53 (p = 0.031)
T−0.57 (p = 0.630)−1.30 (p = 0.903)+4.25 (p = 0.043)+7.55 (p = 0.048)

(For M, we use the new reranker-based proxy described above. For O, we compare specifically on queries which exhibit incomplete retrieval with Recall@10 between 0 and 1, i.e. questions where the memories are fragmented and only partially retrieved. REALTALK uses the same test over the same three memory systems.)

We could not observe a similar effect for PersonaMem because the single-category adapters (C-250, O-250, M-250, E-250 and T-250) individually barely change retrieval for PersonaMem, exhibiting anywhere from a -2.09 to +1.92 change in Recall@10 relative to the base model versus +10.20 to +12.51 for LoCoMo. We believe that this is due to differences in question format between our training data and PersonaMem: for instance, while 100% of questions from our training data end in a "?", only 41% of questions do for PersonaMem. This is consistent with EvoEmbedding's results, which show no correlation between general performance on memory benchmarks and performance on PersonaMem (EvoEmbedding, Tables 1 and 10), and no correlation between LoCoMo and PersonaMem answer accuracy (Spearman ρ = 0.17, p = 0.59 across the 12 retrievers in their Table 10). On REALTALK, where the single-category adapters do change retrieval, the same test across all three memory systems shows the same pattern: O (+3.97, p < 0.001), M (+6.61, p < 0.001), E (+1.35, p = 0.040) and T (+4.25, p = 0.043) gain most on their own categories, relative to the other adapters and to the placebo.

Reviewer_VU8eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies retrieval failures in agent memory systems where relevant information is stored but not successfully retrieved. The authors introduce COMET, a framework that identifies five retrieval-failure patterns through continuous diagnostic proxies and uses the most prominent patterns to construct targeted training data for LoRA-based embedding adaptation. Experiments across LoCoMo, PersonaMem, three memory systems, and three embedding backbones demonstrate improvements in retrieval performance, with additional evaluations of downstream answering and external benchmark transfer.

Critical strengths
  1. The paper addresses a practically important and well-defined problem. By freezing memory storage and focusing on retrieval adaptation, it isolates a meaningful source of failure in long-term agent memory systems.

  2. COMET provides a clear and interpretable framework connecting retrieval-failure diagnosis with targeted data construction. The continuous proxies and held-out regression analyses offer useful empirical evidence for understanding retrieval difficulties.

  3. The retrieval improvements are consistent across three embedding backbones and supported by matched controls and statistical tests. Achieving these gains with only 750 training queries demonstrates promising data efficiency.

Critical weaknesses
  1. The methodological novelty is somewhat limited. COMET primarily combines predefined failure patterns, LLM-based query augmentation, and standard LoRA fine-tuning. The independent contribution of diagnosis-guided selection over a carefully designed fixed training-data mixture remains insufficiently established.

  2. The generality of the five failure patterns is not convincingly demonstrated. Section 4.1 reports substantial differences between LoCoMo and PersonaMem, yet the category-level analysis focuses primarily on LoCoMo. The evidence is insufficient to establish whether the proposed taxonomy consistently captures the dominant retrieval difficulties across different datasets.

  3. The effectiveness of diagnosis-guided adaptation across datasets remains unclear. OME-750 is selected based on the LoCoMo diagnosis and then applied unchanged to other benchmarks. Without independently diagnosing and adapting to different datasets, the experiments do not establish whether COMET provides a general adaptive framework or simply identifies an effective fixed training-data combination.

  4. The relationship between individual failure patterns and adaptation gains remains unclear. The experiments do not establish whether the targeted datasets improve their corresponding retrieval difficulties. Moreover, although OME-750 improves LoCoMo answering accuracy over the base model, its additional gain over generic augmentation is small, and PersonaMem answering accuracy barely changes.

Questions for authors
  1. If the failure patterns were independently diagnosed on PersonaMem, would a different training-data combination be selected? Would this dataset-specific adaptation outperform the fixed OME-750 configuration?

  2. Can the authors provide pattern-specific evaluations showing whether each targeted dataset improves retrieval on its corresponding failure mode? Such evidence would help establish the value of diagnosis-guided adaptation beyond generic data augmentation.

I would consider increasing it if the authors can provide convincing evidence addressing these two concerns.

5 comments
AuthorsOpenReview ↗

Response to Reviewer VU8e (1/5): overview; R1: Methodological novelty

We thank the reviewer for their thoughtful comments on our paper, and for their encouraging comments regarding the paper's strengths. We believe the reviewer raises a number of reasonable concerns which we will address below, including: 1) methodological novelty; 2) whether the diagnosis-to-adaptation framework still works in a different setting, e.g. PersonaMem; 3) whether the category-specific targeted datasets improve their corresponding retrieval difficulties; and 4) small answering margins over controls and on PersonaMem. In addition, we have also included further experiments surrounding our method in a general comment posted above.

[R1: Methodological novelty]

Weakness 1: "The methodological novelty is somewhat limited. COMET primarily combines predefined failure patterns, LLM-based query augmentation, and standard LoRA fine-tuning. The independent contribution of diagnosis-guided selection over a carefully designed fixed training-data mixture remains insufficiently established."

We agree that COMET does not propose a new model architecture or training algorithm; however, we believe its novelty originates from the following aspects:

  1. The contribution of a memory-specialized embedder itself. To the best of our knowledge, besides concurrent work, the main papers exploring fine-tuned embedders for long-term memory retrieval are EvoEmbedding and HiNS; of the two, HiNS has no released weights, and EvoEmbedding introduces a sequential context architecture that incurs additional latency (context encoding 22.08s vs 3.80s for Qwen3-Embedding-4B, about 5.8x). We believe that focusing on the embedding model used, rather than the architecture used to store memories, is an approach that is itself underexplored and deserves more attention, particularly because a LoRA adapter can be plug-and-play across many memory systems and offers demonstrable retrieval gains at little additional cost.
  2. The diagnosis-to-adaptation loop. As the reviewer points out, one of the primary strengths of our approach is its data efficiency; we close the gap to large-scale training through diagnosis, identifying which failure modes are most significant in memory tasks and using precisely these failure modes in constructing training data for adaptation. In contrast, EvoEmbedding uses EvoTrain-180K, a generalist dataset with more than 180,000 independent queries that does not select for specific failure modes suitable to memory tasks, and achieves comparable results to our 750- to 3000-query datasets at similar model scales. (See general comment above.) In the responses below, we provide further evidence for the link between diagnosis and adaptation.
AuthorsOpenReview ↗

Response to Reviewer VU8e (2/5): R2: Diagnosis to adaptation

[R2: Diagnosis to adaptation]

Weakness 3: "The effectiveness of diagnosis-guided adaptation across datasets remains unclear. OME-750 is selected based on the LoCoMo diagnosis and then applied unchanged to other benchmarks. Without independently diagnosing and adapting to different datasets, the experiments do not establish whether COMET provides a general adaptive framework or simply identifies an effective fixed training-data combination."

Here we provide additional experiments for the claim that diagnosis drives improved adaptation results. In particular, we independently diagnose two further benchmarks, PersonaMem and REALTALK, and show that the diagnosis-to-adaptation framework remains effective on each, rather than being a recipe for selecting a fixed combination on LoCoMo.

Weakness 2: "The generality of the five failure patterns is not convincingly demonstrated. Section 4.1 reports substantial differences between LoCoMo and PersonaMem, yet the category-level analysis focuses primarily on LoCoMo. The evidence is insufficient to establish whether the proposed taxonomy consistently captures the dominant retrieval difficulties across different datasets."

Question 1a: "If the failure patterns were independently diagnosed on PersonaMem, would a different training-data combination be selected?"

Yes, C/M/E (concentration, memory-query form mismatch, entity mismatch) would be selected instead of O/M/E. Using the paper's method of selecting based on the number of memory system/embedder/top-kk settings where the category is significant:

CategoryLoCoMo, failureLoCoMo, Recall@10PersonaMem, failurePersonaMem, Recall@10REALTALK, failure
O15152115
M1312151511
E1515151515
C5511144
T67012

(Number of settings, out of 15, in which the category significantly predicts the outcome after BH correction.)

PersonaMem has a low proportion of multi-memory questions (13.9% vs 26.8% for LoCoMo) and has 3.3x the average turn length versus LoCoMo, so selecting concentration (crucial information hidden between distracting information) and not overfragmentation for PersonaMem is reasonable, as the regression results reflect.

Question 1b: "Would this dataset-specific adaptation outperform the fixed OME-750 configuration?"

To test this, we trained all 10 possible three-category adapters (e.g. OME, CME, etc.) and evaluated them on PersonaMem, LoCoMo and REALTALK, using the same 2-epoch training protocol, and also repeated the full diagnosis independently on REALTALK, where it selects O, M, E, as for LoCoMo. CME-750 (the adapter trained on the C, M, E datasets) comes a close second on PersonaMem and is statistically indistinguishable from OME-750 (difference 0.41 pp, 95% CI [−0.84, +1.61]). More importantly, on every benchmark the diagnosis significantly predicts how well each of the 10 adapters performs, as seen below.

Pearson correlation between each of the 10 three-category mixes' diagnosis score on one benchmark (rows) and its retrieval performance on another (columns):

Diagnosis fromPerformance on LoCoMoPerformance on PersonaMemPerformance on REALTALK
LoCoMo (selects O, M, E)0.77 (p = 0.009)0.52 (p = 0.12)0.78 (p = 0.008)
PersonaMem (selects C, M, E)0.35 (p = 0.32)0.83 (p = 0.003)0.42 (p = 0.23)
REALTALK (selects O, M, E)0.70 (p = 0.024)0.51 (p = 0.13)0.77 (p = 0.009)
AuthorsOpenReview ↗

Response to Reviewer VU8e (3/5): R2 continued

For these 10 adapters, we calculated the average number of significant regressions for the adapters' three categories (using the paper's lexical M proxy), and found its correlation with adapter performance on the benchmarks (as above). We note that all the diagonal correlations are significant under two-sided pp-values (standard test for Pearson's rr; bold: p < 0.05), meaning each benchmark's diagnosis predicts adaptation performance on that benchmark. REALTALK performance uses LMEB's evaluation of REALTALK (nDCG@10), the benchmark's official protocol; the other two benchmarks use Recall@10.

Off the diagonal, correlations are significant only between LoCoMo and REALTALK (in both directions), because they are very similar in style (both conversational dialogue with targeted questions) and have very similar diagnoses (OO and EE identical at 15, MM being 13 on LoCoMo vs. 11 on REALTALK, CC being 5 vs. 4, and TT being 6 vs. 2 for retrieval failure, with the same ranking O=E>M>C>TO = E > M > C > T and diagnosis scores correlating at r=0.98r = 0.98, against 0.29 and 0.34 with PersonaMem's).

The following lays out the performance of the three-category mixtures on PersonaMem, LoCoMo, and REALTALK. As shown, all diagnosed adapters are statistically indistinguishable from the top-ranked adapter.

BenchmarkDiagnosed mixGap to 1stSpread, 1st to 10thBottom two
PersonaMemCME (2nd)0.41 pp (95% CI [−0.84, +1.61], n.s.)3.66 ppOET, COT
LoCoMoOME (1st)02.42 ppCMT, CMO
REALTALKOME (2nd)0.06 pp (95% CI [−1.52, +1.78], n.s.)2.52 ppCMT, COT

In addition, the difference in diagnosis results between LoCoMo/REALTALK and PersonaMem are meaningful. The diagnoses differ in a single slot: LoCoMo/REALTALK selects O and PersonaMem selects C. Comparing each mix containing O with the matching mix containing C, with the other two categories held fixed:

Other two categoriesLoCoMo: O − CPersonaMem: O − CREALTALK: O − C
M, E (OME vs CME)+0.43+0.38+0.80
M, T (MOT vs CMT)+1.07 (p < 0.001)−0.53+0.76
E, T (OET vs CET)+1.79 (p < 0.001)−0.43+1.01 (p = 0.044)
Mean+1.10 (p < 0.001)−0.19+0.85 (p = 0.024)

(Recall@10 on LoCoMo and PersonaMem, three-seed means per question over the three memory systems; nDCG@10 on REALTALK, seed 42; Qwen3-Embedding-0.6B; paired cluster bootstrap over conversations/personas, one-sided p.) O outperforms C in all three swaps on LoCoMo and REALTALK, where diagnosis selects O, but not on PersonaMem, where diagnosis selects C; the difference between LoCoMo and PersonaMem is +1.29 points (significant with one-sided p = 0.018).

We believe this addresses Weaknesses 2 and 3 quoted above. In regards to the concern of whether the paper's contribution was one good fixed mixture instead of a general framework, we primarily evaluated OME for the paper because we recognized that LoCoMo was more indicative of typical memory tasks such as single-hop, multi-hop and temporal retrieval versus PersonaMem, which mostly lacked explicit questions (instead using conversational turns to retrieve directly) and focused on user preferences; however, here we see that the diagnosis-to-adaptation link holds on PersonaMem and REALTALK as well. By design, the diagnosis uses LoCoMo as a representative memory benchmark: it identifies, with statistical evidence, the failure modes most prevalent in memory retrieval, on the hypothesis that training data targeting them yields adapters that generalize to other benchmarks with similar memory tasks. The experiments above test this hypothesis directly: OME transfers to datasets that played no part in its selection, and where a benchmark's diagnosis differs (PersonaMem), adaptation performance follows that benchmark's own diagnosis.

Reviewer_mhrJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

COMET connects memory-retrieval failure diagnosis with targeted embedding adaptation. It identifies five failure patterns and constructs training queries targeting the three most strongly supported by its analysis. LoRA adaptation on these 750 queries improves retrieval across three embedding backbones, with evaluation on LoCoMo, PersonaMem, and the external LMEB suite.

Critical strengths
  1. The paper connects failure analysis to a concrete intervention. Rather than treating retrieval misses as a single problem, it uses interpretable categories to guide training-data construction, making the diagnosis actionable.

  2. The targeted training achieves useful retrieval improvements with relatively little data and without changing the encoder architecture or memory store. Comparisons with unaugmented data, generic rewrites, and selection controls help establish the value of the constructed examples beyond ordinary fine-tuning.

Critical weaknesses
  1. The results do not yet establish whether the gains would persist with stronger, memory-specialized retrievers. Although multiple scales and model families are tested, adapting a stronger starting model under the same protocol would help distinguish a broadly useful training strategy from one that mainly corrects weaknesses of the selected backbones.

  2. The additional benefit for final answer quality is less convincing than the retrieval gains. On LoCoMo, OME-750 improves judged accuracy by only 0.51 percentage points over generic-rewrite training, without a statistically significant difference, while PersonaMem accuracy changes very little.

Questions for authors
  1. Could you evaluate diagnosis-guided adaptation on a stronger memory-specialized retriever, using matched training data and compute? This would help clarify whether the approach remains beneficial as baseline retrieval quality improves.

  2. What limits the translation of retrieval improvements into answer-quality gains over the matched controls? Additional analysis of which retrieved evidence changes the final answer would help clarify the practical benefit of targeted adaptation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mhrJ (1/2): overview; R1: Stronger base models

We thank the reviewer for their careful reading of our paper, and for recognizing that the diagnosis makes retrieval failures actionable and that the targeted data yields useful retrieval gains with little data, beyond ordinary fine-tuning. The reviewer raises two concerns, which we address below: 1) whether the gains persist when adapting a stronger starting retriever; and 2) why the gains in final answer quality over matched controls are smaller than the retrieval gains. We have also included further experiments in the general comment above.

[R1: Stronger base models]

Weakness 1: "The results do not yet establish whether the gains would persist with stronger, memory-specialized retrievers. Although multiple scales and model families are tested, adapting a stronger starting model under the same protocol would help distinguish a broadly useful training strategy from one that mainly corrects weaknesses of the selected backbones."

Question 1: "Could you evaluate diagnosis-guided adaptation on a stronger memory-specialized retriever, using matched training data and compute? This would help clarify whether the approach remains beneficial as baseline retrieval quality improves."

We did not fine-tune a memory-specialized baseline because no suitable one is publicly available: memory embedders are as a whole a fairly new research area, and the only adjacent literature we could find were EvoEmbedding and HiNS, of which HiNS does not release their weights and EvoEmbedding uses a completely novel architecture based on a contextual buffer that our fine-tuning data does not support. Instead, we ran additional experiments using the strongest model on the LMEB leaderboard, both overall (61.41, full LMEB leaderboard average) and in the dialogue (conversational memory) section: bge-multilingual-gemma2, using Google's Gemma 2 architecture:

BackboneBasePlacebo-750OME-750OME − Placebo
bge-multilingual-gemma2 (9B, 3 seeds)72.0774.1375.30+1.17
EvoEmbedding-0.8B (our evaluation on our benchmarks and memory systems)70.62———

(The reported numbers are mean Recall@10 over the six system/benchmark settings. All adapters and controls come from a re-run of our training pipeline; see the general comment.) bge-multilingual-gemma2's base (72.07) is stronger than every base model in the paper and than EvoEmbedding-0.8B run natively with its memory buffer (70.62). On it, OME-750 still improves over the base model by 3.23 points and outperforms a placebo trained under identical quality checks by 1.17 points (three-seed means).

Across all four backbones, ordered by their official LMEB score, OME-750 stays ahead of the quality-checked placebo, including on the strongest model on the leaderboard:

BackboneLMEB (official, nDCG@10)OME-750 − quality-checked Placebo-750
BGE-large42.17+2.28
Qwen3-Embedding-4B53.23+1.76
Qwen3-Embedding-0.6B54.71+2.91
bge-multilingual-gemma2 (9B)61.41+1.17

(LMEB scores are the leaderboard's mean over its 22 datasets, with instructions. Margins are Recall@10 over our six system/benchmark settings; full placebo results are in our response to Reviewer YJEa, R3.)

AuthorsOpenReview ↗

Response to Reviewer mhrJ (2/2): R2: Answer quality

[R2: Answer quality]

Weakness 2: "The additional benefit for final answer quality is less convincing than the retrieval gains. On LoCoMo, OME-750 improves judged accuracy by only 0.51 percentage points over generic-rewrite training, without a statistically significant difference, while PersonaMem accuracy changes very little."

Question 2: "What limits the translation of retrieval improvements into answer-quality gains over the matched controls? Additional analysis of which retrieved evidence changes the final answer would help clarify the practical benefit of targeted adaptation."

There are two reasons for this:

  1. On questions where there is room for improvement at all, there is a significant gain in answering. On LoCoMo, for questions where the base embedder leads to generating the wrong answer, the answering gain over the placebo control is +2.12pp (p=0.013); and on questions with incomplete retrieval, the answering gain is +2.80pp (p=0.002).
  2. In PersonaMem in particular, answering is far less sensitive to retrieval than on LoCoMo: the accuracy gap between full and zero recall is 20–25pp, against 62–64pp on LoCoMo (Appendix D). This is a property of the benchmark itself (PersonaMem; Jiang et al., 2025); indeed, EvoEmbedding's results show no correlation between general performance on memory benchmarks and performance on PersonaMem (EvoEmbedding, Tables 1 and 10), and no correlation between LoCoMo and PersonaMem answer accuracy (Spearman ρ = 0.17, p = 0.59 across the 12 retrievers in their Table 10). Due to its multiple-choice format (EvoEmbedding, §4.3), many answers to questions can be inferred even without the necessary memories.

Answer quality follows whether the complete evidence is retrieved. Grouping LoCoMo questions by which adapter retrieves all of their gold memories within the top 10 (OME-750 vs. Placebo-750, three systems and three seeds, 13,215 question-seeds):

Complete evidence retrieved byShare of questionsOME-750 accuracyPlacebo-750 accuracyDifference
OME-750 only5.1%75.828.0+47.8 (p < 0.001)
Placebo-750 only4.3%25.370.5−45.3
Both67.4%82.182.3−0.2
Neither23.2%25.624.9+0.8

Regarding which retrieved evidence changes the final answer, we provide a short case study. In RAG/LoCoMo, there are 63 questions where OME-750 answers correctly, the placebo answers incorrectly, and OME-750 retrieved a gold memory that the placebo missed. For example, for a question about Maria and U.S. states, only OME-750 retrieves the turn mentioning Oregon; for a question about Jon's studio, only OME-750 retrieves the turn about its Marley flooring; and for a question about Melanie's pets, the placebo misses the turn about Luna and answers "Oscar".

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/3): Is diagnosis necessary?

We would like to thank all our reviewers for their time and effort in evaluating our paper. There are several critiques which several reviewers raised simultaneously, which we found quite helpful in improving our results; below, we summarize these critiques and new experiments we performed to address them. Full tables and details for each point are in our responses to the individual reviewers. All new results come from a re-run of our training pipeline on a newer PyTorch build, with each backbone mining its own hard negatives and BGE-large using its standard query prefix, so numbers may differ slightly from those in our paper.

Is diagnosis really necessary? Several reviewers raised the point that although OME-750 was empirically successful and data-efficient, the link between the diagnosis and the success of OME-750 was not sufficiently established. In particular, reviewers noted that the diagnosis uses only LoCoMo and that "without independently diagnosing and adapting to different datasets, the experiments do not establish whether COMET provides a general adaptive framework or simply identifies an effective fixed training-data combination" (VU8e). Here, we perform the diagnosis independently on two additional benchmarks, PersonaMem and REALTALK, and find that on every benchmark, the diagnosis significantly predicts adaptation performance: the more system/embedder/top-kk settings in which a mix's categories are significant on a benchmark, the better that mix's adapter performs on it. The diagnosis for PersonaMem selects C, M, E instead of O, M, E; the diagnosis for REALTALK selects O, M, E, as for LoCoMo.

  • Across all 10 three-category adapters, the Pearson correlation between a benchmark's diagnosis score and adaptation performance on that benchmark is 0.77 (p = 0.009) on LoCoMo, 0.83 (p = 0.003) on PersonaMem and 0.77 (p = 0.009) on REALTALK. Across benchmarks, a diagnosis predicts another benchmark's results only where the two diagnoses agree: between LoCoMo and REALTALK, whose diagnoses are nearly identical (0.78 and 0.70, both p < 0.05), but not with PersonaMem, whose diagnosis differs (0.35 to 0.52, n.s.).
  • Each benchmark's diagnosed mix is first or statistically tied with first: OME ranks 1st on LoCoMo; CME trails the top mix by 0.41 pp on PersonaMem (95% CI [−0.84, +1.61]) and OME by 0.06 pp on REALTALK (95% CI [−1.52, +1.78]).
  • The diagnoses differ in one slot (O on LoCoMo and REALTALK, C on PersonaMem). Swapping O for C with the other two categories held fixed, O wins all three swaps on LoCoMo (mean +1.10, p < 0.001) and on REALTALK (mean +0.85, p = 0.024), but not on PersonaMem (−0.19); the LoCoMo–PersonaMem difference is +1.29 (p = 0.018).
  • OME-750 also ranks first on the paper's eight-dataset LMEB suite, where it outperforms all controls (59.09; Table D4) and all 9 other three-category combinations.
  • The two categories not selected by diagnosis, concentration (C) and temporal mismatch (T), are equally well-known challenges in the literature, yet as we show with COMET-1250 which includes C and T along with OME-750, the marginal value of C and T on LoCoMo, where neither is diagnosed, is small, even negative. The value of the diagnostic step is that it bypasses the need for heuristic judgments and instead defers to empirical evidence to suggest which failure modes are most prevalent.

(Details: VU8e R2, 4Wys R2, YJEa R5 and R6.)

AuthorsOpenReview ↗

General response (2/3): the M proxy; category-specific gains; answering gains

Why does the proxy for memory-query form mismatch not change? Two reviewers raised the concern that "the memory-query form mismatch proxy shows almost no change after targeted augmentation (0.788 to 0.779), despite this category being one of the most effective adaptation targets" (reviewers 4Wys, YJEa). Here, we provide a new proxy for memory-query form mismatch that the M-250 dataset significantly shifts toward greater mismatch and is evidenced to measure the intended effect: the degree to which the relationship between a query and its memories are implicit and non-obvious. The proxy is defined using a cross-encoder reranker (Qwen3-Reranker-8B), relative to all candidate memories for the query.

  • M-250 rewrites lower the score by 2.63 log-odds relative to their source questions (2.92 with a second reranker, bge-reranker-v2-m3; both p < 0.001); placebo rewrites leave it unchanged (−0.08, n.s.); rewrites for the other categories move it far less (C −0.28, T −0.21, O −0.15) or the other way (E +0.93); rewriting the same questions to be more direct moves it the other way (+3.34).
  • In blinded human audits, the high-proxy question was judged less direct in 25 of 25 decided pairs for the new proxy (p < 10−710^{-7}) and 22 of 28 for the paper's lexical proxy (p = 0.002); an independent second rater from outside the project gave 35 of 35 and 24 of 25.
  • Under the new proxy, memory-query form mismatch is significant in all 15 settings on both LoCoMo and PersonaMem, and the diagnosis still selects O, M and E.

Together, these show that the M-250 data instantiates the intended difficulty and that the paper's proxy measures it, even though it does so coarsely. (Details: 4Wys R3, YJEa R1.)

Do the category-specific datasets specifically improve performance on questions belonging to that category? Yes. On LoCoMo, the O, M and E adapters significantly improve performance specifically on questions belonging to their own category, both relative to the other category adapters (O +3.09, p = 0.001; M +1.94, p < 0.001; E +1.06, p = 0.009) and relative to the placebo (O +2.42, p = 0.010; M +1.84, p = 0.002; E +1.45, p = 0.003). We could not observe a similar effect for PersonaMem because the single-category adapters (C-250, O-250, M-250, E-250 and T-250) individually barely change retrieval for PersonaMem, exhibiting anywhere from a -2.09 to +1.92 change in Recall@10 relative to the base model versus +10.20 to +12.51 for LoCoMo. This is consistent with EvoEmbedding's results, which show no correlation between general performance on memory benchmarks and performance on PersonaMem (EvoEmbedding, Tables 1 and 10), and no correlation between LoCoMo and PersonaMem answer accuracy (Spearman ρ = 0.17, p = 0.59 across the 12 retrievers in their Table 10). On REALTALK, where the single-category adapters do change retrieval, the same test across all three memory systems shows the same pattern: O (+3.97, p < 0.001), M (+6.61, p < 0.001), E (+1.35, p = 0.040) and T (+4.25, p = 0.043) gain most on their own categories, relative to the other adapters and to the placebo. (Details: VU8e R3, 4Wys R3, YJEa R1.)

Why did answering barely change even if there was a retrieval gain? Several reviewers raised the concern that "the additional benefit for final answer quality is less convincing than the retrieval gains" (reviewer mhrJ). There are two reasons for this:

  1. On questions where there is room for improvement at all, there is a significant gain in answering. On LoCoMo, for questions where the base embedder leads to generating the wrong answer, the answering gain over the placebo control is +2.12pp (p=0.013); and on questions with incomplete retrieval, the answering gain is +2.80pp (p=0.002).
  2. In PersonaMem in particular, answering is far less sensitive to retrieval (a 20–25pp accuracy gap between full and zero recall, against 62–64pp on LoCoMo), a property of the benchmark itself: EvoEmbedding's results show no correlation between LoCoMo and PersonaMem answer accuracy (Spearman ρ = 0.17, p = 0.59 across the 12 retrievers in their Table 10), and due to its multiple-choice format, many answers to questions can be inferred even without the necessary memories.

(Details: mhrJ R2, VU8e R4, 4Wys R5.)

AuthorsOpenReview ↗

General response (3/3): stronger controls, base models and data sources

Does diagnosis-guided adaptation still outperform controls with stronger base models, stronger control construction and with different starting datasets? Yes to all three.

  • Stronger controls. We rebuilt the placebo with the exact quality checks used in the prompts for constructing the COMET datasets. The quality-checked placebo is no better than the original on Qwen3-Embedding-0.6B and BGE-large and only slightly better on Qwen3-Embedding-4B, and OME-750 outperforms it by +2.91 (Qwen3-Embedding-0.6B), +2.28 (BGE-large), +1.76 (Qwen3-Embedding-4B) and +1.17 (bge-multilingual-gemma2, 9B, three seeds) Recall@10. In addition, training on the rewritten OME-750 improves Recall@10 by +3.23 over training on the same selected, quality-checked source questions before rewriting (Section 4.2 and Appendix Table D9). (Details: YJEa R3.)
  • Stronger base models. We ran additional experiments using the strongest model on the LMEB leaderboard, both overall (61.41, full LMEB leaderboard average) and in the dialogue (conversational memory) section: bge-multilingual-gemma2 (9B). Its base (72.07) is stronger than every base model in the paper and than EvoEmbedding (70.62 on our benchmarks and memory systems), yet OME-750 still reaches 75.30, +1.17 over a placebo trained under identical quality checks (three-seed means). (Details: mhrJ R1.)
  • Alternative source datasets. We integrated another source dataset, REALTALK, into the methodology, using the exact same prompts for O, M and E. Built from REALTALK alone, the core conclusion holds: OME-750 outperforms a same-size placebo built from the same source by +4.67 points (70.43 vs 65.76). (Details: YJEa R5.)
  • Scaling. This was not explicitly requested by reviewers, but out of interest, we ran experiments to confirm whether the COMET datasets scale to 1500 and 3000 queries on our model backbones (BGE, Qwen3-0.6B, Qwen3-4B), which, for comparison, is still roughly 120x and 60x fewer queries than EvoTrain-180K used by EvoEmbedding. We scaled the datasets by combining the OME-750 datasets generated from PerLTQA and REALTALK above for 1500 queries, and by re-running the generation prompts a second time to get to 3000.

Retrieval (seed 42; mean Recall@10 over the six system × benchmark settings):

BackboneOME-750OME-1500OME-3000Placebo-1500OME-1500 − Placebo-1500
Qwen3-0.6B71.1172.5072.1970.96+1.54
BGE-large71.1171.3271.3469.85+1.47
Qwen3-4B72.5773.3972.2272.46+0.93

LMEB (seed 42; nDCG@10, seven tasks: QASPER, PeerQA, Covid-QA, MLDR, ESG-Reports, LoCoMo, LongMemEval. We exclude REALTALK because it was used to build the training data.):

BackboneOME-750OME-1500OME-3000Placebo-1500OME-1500 − Placebo-1500EvoEmbedding (similar scale)
Qwen3-0.6B60.6364.1463.0561.27+2.8760.97 (EvoEmbedding-0.8B)
BGE-large59.0760.3061.6859.56+0.74—
Qwen3-4B62.2066.6567.5563.83+2.8267.53 (EvoEmbedding-4B)

LMEB improves substantially under scaling: on the 0.6B backbone, it improves 3.51pp from 750 to 1500 queries and thus outperforms EvoEmbedding by 3.2pp at similar model scales while utilizing 120x fewer independent queries. Scaling for LMEB continues up to 3000 independent queries for BGE and Qwen3-4B, with the 4B model being competitive with EvoEmbedding-4B at 3000 queries.

Thank you for the detailed response and the additional experiments. The results on bge-multilingual-gemma2 are helpful, but they also show that a large part of the improvement over the base model is obtained by the placebo, and the margin of OME-750 over the quality-checked placebo tends to shrink as the base model becomes stronger (from about 2-3 points to 1.17). This suggests the benefit of diagnosis-guided data may diminish for stronger retrievers. The analysis of answer quality is informative, but it confirms that the net difference in questions with complete evidence retrieved is small (5.1% vs. 4.3%), consistent with the limited overall answer-quality gain. I appreciate the authors' effort, but my main concerns remain, so I keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.