acceptodds
Under review as a conference paper at ICLR 2027

Learning to Retrieve Long-term Memories with Unlabeled Conversations

Abstract

Long-term memory in LM agents relies on retrieving relevant information from external memory stores. Despite training on massive query-document pairs, general-purpose embedding models struggle with long-term memory retrieval, as query-memory relevance labels are rare and costly to obtain. In this work, we propose adding self-supervised retriever learning on unlabeled conversations to the training recipe to fill this gap. Specifically, we introduce MemRevela, which learns retrieval directly from the dependencies between conversational turns. An auxiliary language model predicts conversational fragments using retriever-weighted context, allowing the prediction loss to train the retriever. MemRevela then jointly optimizes this objective with standard MS MARCO contrastive learning. We evaluate five model sizes from 0.1B to 8B on LMEB's 22 datasets across four memory types. Without any task instructions, MemRevela-8B achieves the best performance in our comparison with dataset-averaged nDCG@10 63.24, exceeding NV-Embed-v2 by 3.43 absolute points. Our method also improves downstream question answering over long conversations compared with two proprietary embedding APIs. Without annotated labels, MemRevela injects long-term memory knowledge into the retriever: it spreads conversational turns in embedding space about as widely as general passages, whereas general-purpose models cluster them tightly. We show that conversational self-supervision and contrastive learning are complementary, each strengthening different memory types. Finally, the memory gains do not come at the expense of general retrieval: MemRevela stays comparable to contrastive-only baselines on BEIR, making conversational self-supervision a promising addition to standard retriever training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Memory retrieval, Long-term agent memory, Dense retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1pqsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies retrieval for long-term memory in LM agents. The authors argue that general-purpose embedding models do poorly here partly because they pack conversational turns into a narrow region of embedding space, which makes same-topic turns hard to tell apart. MemRevela applies Revela-style retriever-weighted language modeling to unlabeled conversations (WildChat, UltraChat): the retriever scores the other fragments in a group, those scores weight an auxiliary LM’s cross-fragment attention, and the LM’s next-token loss trains the retriever through the weights. This is followed by joint training with an MS MARCO contrastive loss, and the auxiliary LM is dropped at inference. On the 22 LMEB datasets, the method beats same-backbone RepLLaMA reproductions at every size from 0.6B to 8B, with the 8B model reaching 63.24 nDCG@10. It also gives small gains on LoCoMo QA over three external retrievers and roughly matches RepLLaMA on BEIR. The analysis shows sharper within-conversation evidence discrimination and a smaller dispersion gap between conversational and general text, and the ablations attribute most of the gain to conversational initialization.

Critical strengths
  1. The supervision source fits the problem well. Dependencies between conversation fragments are free and plentiful, and the method uses them without generated queries or relevance labels; since the auxiliary LM is discarded and memories are still encoded independently, the retriever can replace the embedding model in an existing memory pipeline with no other changes.

  2. The core comparison is properly controlled. Gains hold against same-backbone, same-label RepLLaMA reproductions at all four sizes from 0.6B to 8B, and the ElicitR comparison suggests the retriever-weighted LM mechanism alone does not account for them.

  3. The ablations are useful beyond this paper. Stage 1 alone already beats MS MARCO-only training on episodic memory, and conversational initialization accounts for most of the final gain, which tells follow-up work where the effort is worth spending.

Critical weaknesses
  1. The control needed for the central claim is missing. MemRevela gets about 7k extra steps of stage-1 training that RepLLaMA never sees, so the gain could come from additional self-supervised training rather than conversational structure. A compute-matched stage 1 on generic text (e.g., Wikipedia) would settle this.

  2. Gains over the controlled baseline are modest and partly confounded. The improvement over RepLLaMA is 0.95 to 2.04 nDCG@10 with no seed variance reported, and at 8B, where it is largest, the two models use different LoRA ranks (256 vs. 64). The headline margin over NV-Embed-v2 also leans on the instruction-free setting, given that RepLLaMA-3B alone already beats NV-Embed-v2.

  3. The representational-mismatch explanation is correlational. A smaller ΔD does not show that dispersion drives retrieval quality, and it could just as well come from MS MARCO passages tightening as from conversational turns spreading out. Centering or whitening RepLLaMA’s embeddings would be a cheap check; if that recovers most of the gain, the explanation should be revised.

Questions for authors

see weaknesses

3 comments

Thank you for your recognition on our work and the constructive and insightful review. We ran all three checks as you suggested.

W1. Stage 1 on generic text

We reran the same recipe with FineWeb in place of conversations (a broader sample of general text than Wiki).

RetrieverEpisodicDialogueSemanticProceduralDataset MeanType Mean
Stage 1 only
FineWeb, 1B60.4350.2547.6158.3852.4454.17
Conversations, 1B65.8755.3356.6655.1956.7358.26
FineWeb, 3B64.8249.9448.3958.9653.1955.53
Conversations, 3B68.1456.7159.8658.9259.5060.91
Full two-stage recipe
RepLLaMA-1B63.9857.6258.6658.9558.9459.80
FineWeb, 1B64.4357.9159.1458.6759.1660.04
MemRevela-1B (conversations)65.7958.8660.2559.4160.1561.08
RepLLaMA-3B66.9658.6260.6859.5160.3761.44
FineWeb, 3B66.1458.4360.6760.1560.4161.35
MemRevela-3B69.3758.6962.0560.4261.3662.63

The extra stage-1 steps on generic text give little gain over RepLLaMA (+0.22, +0.04), while conversations give +1.21 and +0.99. Before any labels, conversations are ahead of generic text by +4.29 and +6.31. The gain comes from the conversational data, not from additional self-supervised training.

W2. Size and reliability of the gains

Significance. Query-level paired randomization tests on nDCG@10 (by dataset, 10k resamples; baselines as in Table 1). Both gains are statistically significant (p < 0.001), and both 95% confidence intervals exclude zero.

ComparisonDataset Mean [95% CI]pDatasets won / lostSignificant after Holm, won / lost
MemRevela-8B vs RepLLaMA-8B+2.02 [+1.63, +2.42]< 0.00120 / 213 / 1
MemRevela-3B vs RepLLaMA-3B+0.96 [+0.57, +1.34]< 0.00115 / 710 / 2

LoRA rank at 8B. We retrained RepLLaMA-8B at rank 256 to match MemRevela-8B (nDCG@10, ×100):

ModelEpisodicDialogueSemanticProceduralDataset MeanType Mean
RepLLaMA-8B, rank 64 (Table 1)68.0759.9261.3559.9961.2062.33
RepLLaMA-8B, rank 256 (retrained)68.9059.5561.0659.8561.0362.34
MemRevela-8B, rank 25672.2661.1563.6961.7263.2464.70

The rank does not explain the gain. At matched rank the baseline is essentially unchanged (−0.17 on Dataset Mean), and MemRevela-8B is ahead by +2.21 (p < 0.001) and on all four memory types. We will report the rank-256 baseline in Table 1.

Instructions. With each baseline in its recommended instruction setting (LMEB [1] Table 3), MemRevela-8B (63.24, no instructions) is still ahead of NV-Embed-v2 (60.25, +2.99) and of the strongest baseline, bge-multilingual-gemma2 (61.41, +1.83). We agree that a contrastive-only LLM retriever is already competitive with NV-Embed-v2 without instructions on LMEB. This is why our main evidence is the controlled comparison with RepLLaMA, not the margin over external models.

W3. Is the dispersion explanation causal?

We agree that the analysis is correlational, and we ran the suggested check: centering and whitening on both 8B models (transform estimated on 20k held-out WildChat chunks), re-scored on the 8 dialogue and episodic LMEB datasets (nDCG@10):

TransformRepLLaMA-8BMemRevela-8BGap
None61.9563.93+1.98
Centering61.9163.97+2.06
Whitening (k=512)56.7858.87+2.09
Whitening (k=256)53.1554.58+1.43

Post-hoc correction does not recover the gain: centering leaves RepLLaMA unchanged, whitening hurts both models, and the gap stays between +1.4 and +2.1 throughout. The smaller ΔD comes from conversational turns spreading out (D_conv from 0.363 to 0.444), not from MS MARCO passages tightening (D_gen from 0.494 to 0.503).

The check also supports your caution: centering raises RepLLaMA's D_conv to 0.84 with no change in nDCG, so dispersion alone does not drive retrieval quality. We will present ΔD as a diagnostic of the representation change, not its cause.

Thank you again for the recognition. We hope this resolves your concerns and are happy to address any further questions.

[1] LMEB: Long-horizon Memory Embedding Benchmark, arxiv 2603.12572

Thank you for the detailed response and the additional experiments. They address my main concerns, and I will maintain my positive score.

Hi Reviewer1pqs,

We are glad that your concerns have been mostly resolved. And thank you again for your insightful and constructive suggestions and recognition on our work.

Regards, Authors

Reviewer_2b3cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MemRevela, a dense retriever training recipe for long-term conversational memory retrieval that uses unlabeled conversations as a self-supervision source. Following Revela, an auxiliary LM predicts each conversational fragment while attending to other fragments in its group, with cross-fragment attention weighted by retriever similarities, so the LM loss trains the retriever without labels. Training proceeds in two stages: conversational prediction only (WildChat and UltraChat), then a joint objective that adds MS MARCO contrastive learning. The auxiliary LM is discarded at inference. Retrievers from 0.1B to 8B are evaluated on LMEB (22 datasets), on downstream LoCoMo QA, and on BEIR. The paper also analyzes embedding geometry, arguing that general-purpose embedders compress conversational turns into a tight region and that MemRevela spreads them out.

Critical strengths

The problem setting and motivation are sound. Memory retrieval for LM agents is practically important, query–memory relevance labels are genuinely scarce, and unlabeled conversations are abundant. Adapting a label-free, LM-guided retriever objective (Revela) to conversational data is a reasonable and well-motivated direction. It also keeps the standard setup, so memories can still be indexed offline at no extra inference cost.

Critical weaknesses
  • Incremental contribution, and the key claim is not isolated. Seems training mechanism is taken directly from Revela. The main changes are the training corpus (conversations instead of general/code text) only and a second stage that adds MS MARCO contrastive learning. The paper's own ablation shows that keeping the conversational loss in the second stage adds almost nothing over simply fine-tuning the stage-1 retriever contrastively. The central claim is that unlabeled conversations specifically are what improves memory retrieval, but the experiments do not separate this from the generic benefit of Revela-style self-supervised pretraining. There is no Revela baseline trained on general raw text, no LM-free self-supervised baseline on the same conversational data, and no ablation of the retriever-weighting mechanism itself. The per-dataset results also show comparable gains on non-conversational long-document datasets, which points in the same direction.

  • Fairness of the comparison and the practical significance of the gains are unclear. All baselines are evaluated without task instructions, even though several strong baselines (e.g., Qwen3-Embedding, NV-Embed-v2, E5-Mistral) are instruction-tuned and are intended to be used with them. This may understate those baselines and inflate the headline margins. Against the only same-backbone controlled baseline, the improvements are modest. On the most representative conversational memory benchmarks the gains are small: about 1–2 points on downstream LoCoMo QA, and no improvement over the contrastive-only baseline on LongMemEval.

Questions for authors
  1. How does a Revela retriever trained on general raw text (e.g., Wikipedia, as in the original Revela paper), followed by the identical stage-2 MS MARCO training, perform on LMEB? This is necessary to attribute the gains to conversational data rather than to Revela-style pretraining in general.

  2. (Key control) How does an LM-free self-supervised baseline on the same conversational corpus compare, e.g., Contriever/ICT-style contrastive learning with adjacent fragments as positives, using the same backbone? The original Revela paper included such a comparison.

  3. Can you report the main baselines with their recommended task instructions on LMEB, alongside the instruction-free numbers? Does the ranking against NV-Embed-v2 / Qwen3-Embedding still hold?

  4. Can you break down the gains into conversational vs. non-conversational LMEB datasets, and explain why MemRevela does not improve over the contrastive-only baseline on LongMemEval? Variance across seeds for the same-backbone comparison would also help judge whether the 1–2 point gains are reliable.

=== Post Rebuttal Comment ===

apologize for late update and thank you for the thorough rebuttal and the additional experiments.

  • The new controls resolve my main concern. The FineWeb control (same recipe, generic text) stays at the RepLLaMA level, while conversational data gives a consistent gain. The pseudo-pair InfoNCE baseline on the same conversations shows that the LM-guided objective also matters. Together, these attribute the improvement to conversational data combined with the Revela-style objective, as claimed.

  • On instructions: the LMEB numbers with recommended instructions change the picture for some baselines substantially (e.g., bge-multilingual-gemma2 goes from 45.11 to 61.41), so the instruction-free Table 1 alone understates them. In the revision, please 1) report both settings for the main baselines in Table 1, 2) clearly state the evaluation setup used for each baseline (pooling, max length, prefixes/instructions), and 3) note that the with-instruction numbers come from the LMEB paper rather than your own pipeline, so the comparison is transparent and fair.

  • Given the above, I am raising my score to 3. However, I still view the contribution as incremental relative to Revela: the training mechanism is unchanged, and the main novelty is applying it to conversational data plus a second contrastive stage whose joint conversational loss adds little over sequential fine-tuning (Table 12).

3 comments

Thank you for the insightful questions and experimental suggestions. To address your concerns, we ran the requested experiments: a generic-text control (Q1), an LM-free baseline on the same conversations (Q2), a comparison with instruction-tuned baselines in their recommended setting (Q3), and significance tests with a conversational vs. non-conversational breakdown (Q4).

Q1. Is the gain from conversational data, or from Revela-style pretraining in general?

We ran the requested control: the same recipe with FineWeb in place of conversations (a broader sample of general text than Wik):

RetrieverEpisodicDialogueSemanticProceduralDataset MeanType Mean
Stage 1 only
FineWeb, 1B60.4350.2547.6158.3852.4454.17
Conversations, 1B65.8755.3356.6655.1956.7358.26
FineWeb, 3B64.8249.9448.3958.9653.1955.53
Conversations, 3B68.1456.7159.8658.9259.5060.91
Full two-stage recipe
RepLLaMA-1B63.9857.6258.6658.9558.9459.80
FineWeb, 1B64.4357.9159.1458.6759.1660.04
MemRevela-1B65.7958.8660.2559.4160.1561.08
RepLLaMA-3B66.9658.6260.6859.5160.3761.44
FineWeb, 3B66.1458.4360.6760.1560.4161.35
MemRevela-3B69.3758.6962.0560.4261.3662.63
  • Stage 1 only: conversations beat generic text by +4.29 (1B) and +6.31 (3B) on Dataset Mean. Procedural memory is the one exception.
  • Full recipe: generic-text pretraining gives little gain over RepLLaMA, while conversations give +1.21 and +0.99 and win on all four memory types.
  • Table 10 in Appendix shows the same: the mechanism applied to MS MARCO groups also stays at the RepLLaMA level (+0.01, −0.08).

This indicates that the gain comes from the conversational data, not from extra self-supervised training or the retriever-weighted objective by itself.

Q2. LM-free self-supervised baseline on the same conversational corpus

We trained a Contriever-style baseline on the same stage-1 data: two fragments from the same conversation are positives, fragments from other conversations are negatives (InfoNCE, RepLLaMA code and hyperparameters).

TrainingEpisodicDialogueSemanticProceduralDataset MeanType Mean
Llama-3.2-1B
Pseudo-pair InfoNCE on conversations55.4148.0643.8951.0848.0449.61
MemRevela stage 1 on conversations65.8755.3356.6655.1956.7358.26
Llama-3.2-3B
Pseudo-pair InfoNCE on conversations55.8248.6242.8351.0047.8249.56
MemRevela stage 1 on conversations68.1456.7159.8658.9259.5060.91

On the same unlabeled conversations, MemRevela stage 1 beats the Contriever baseline by +8.69 (1B) and +11.68 (3B) on Dataset Mean and on all four memory types. Together with the FineWeb control, this shows that both the conversational data and the LM-guided objective matter.

Q3. Instructions and input format

MemRevela and RepLLaMA retain their trained input format, using fixed query: / passage: prefixes (Appendix B), without task-specific instructions. To address the concern about instruction-tuned baselines, we additionally compare against LMEB's results with recommended instructions (Dataset Mean nDCG@10):

RetrieverNo instructions (our Table 1)With instructions(LMEB Table 3)
bge-multilingual-gemma245.1161.41
NV-Embed-v259.8160.25
KaLM-Embedding-Gemma353.9160.10
F2LLM-v2-4B57.8658.68
Qwen3-Embedding-8B54.6555.94
e5-mistral-7b-instruct49.6155.21
Qwen3-Embedding-4B51.4553.23
MemRevela-8B63.24-

MemRevela requires no task-specific instructions yet outperforms all 30 embedding models in LMEB's main table, including those evaluated with recommended instructions [1]. It outperforms the strongest baseline, bge-multilingual-gemma2, by 1.83 points and NV-Embed-v2 by 2.99 points. We will include these comparisons in Table 1.

Q4. Conversational vs. non-conversational gains, LongMemEval, and reliability

Conversational vs. non-conversational (8B).

MemRevela-8B improves significantly on both groups. The conversational gain looks smaller only because of one dataset, TMD; without it, the two groups gain about the same (2.14 v.s. 2.32).

GroupRepLLaMA-8BMemRevela-8BImprovement [95% CI]p
Conversational (6 dialogue datasets)59.9261.15+1.24 [+0.79, +1.69]< 0.001
Conversational without TMD (5)65.4167.54+2.14 [+1.65, +2.62]< 0.001
Non-conversational (16)61.7064.02+2.32 [+1.80, +2.83]< 0.001

Why TMD drops (from 32.47 to 29.21). In 11 of TMD's 12 subsets the query contains no content, only a time reference, and the gold set is every turn in that time window. These subsets test whether a date in the query matches the timestamp in the turn title, not whether the content is relevant.

Retrieval targetExample queryRepLLaMA-8BMemRevela-8BΔ [95% CI]
A specific turn, by content and date"What was in Melanie's painting ... on May 8th?"62.7968.74+5.95 [+2.80, +9.15]
All turns in a date range"What did we chat about between May 8th and June 9th?"67.1052.18−14.91 [−18.03, −11.75]
All turns in a month"What did we discuss in May?"45.6131.30−14.31 [−17.96, −10.58]
All turns on a date"What did we chat about on May 8th?"48.8938.76−10.13 [−11.92, −8.35]

On the only content-based subset, MemRevela-8B gains +5.95, in line with LoCoMo and ConvoMem. The loss comes from three subsets whose queries are a bare date, month, or date range. The other eight subsets show no significant difference: five need date arithmetic and are near zero for both models, and three have only 12 to 24 queries. We view timestamp alignment as a limitation of MemRevela (as we focus on the conversational turns) and will state it in the paper.

LongMemEval. The difference (from 80.65 to 78.91) is not significant after Holm correction, and the dataset is close to saturated: RepLLaMA-8B already reaches Recall@10 ≥ 89.9 on 4 of its 6 subsets. On the three harder conversational benchmarks, MemRevela-8B is the best model in Table 1:

DatasetRepLLaMA-8BMemRevela-8BΔ [95% CI]Subsets improved
LoCoMo55.6058.77+3.17 [+2.26, +4.07]5 / 5
ConvoMem68.8173.10+4.29 [+3.88, +4.70]6 / 6
REALTALK49.8652.01+2.15 [+0.71, +3.64]3 / 3

The largest gains are on ConvoMem subsets where topical similarity is not enough: implicit_connection (+6.4), where the query shares no wording with the evidence, and changing_evidence (+5.5), where both an original statement and its later update must be retrieved. LongMemEval is largely solvable by topical matching, which contrastive training already handles.

Reliability. We ran query-level paired randomization tests on nDCG@10 to show the reliability of the results (by dataset, 10k resamples, aggregated with the LMEB weighting used in Table 1). Both gains are statistically significant (p < 0.001), and both 95% confidence intervals exclude zero, so the improvement over the same-backbone baseline is reliable.

ComparisonDataset Mean [95% CI]Type Mean [95% CI]pDatasets won / lostSignificant after Holm, won / lost
MemRevela-8B vs RepLLaMA-8B+2.02 [+1.63, +2.42]+2.36 [+2.03, +2.69]< 0.00120 / 213 / 1
MemRevela-3B vs RepLLaMA-3B+0.96 [+0.57, +1.34]+1.17 [+0.83, +1.51]< 0.00115 / 710 / 2

Thank you again for the detailed and constructive review! The controls you asked for have made the paper stronger, and we hope they resolve your concerns. We are happy to address any further questions.

[1] LMEB: Long-horizon Memory Embedding Benchmark, arxiv 2603.12572

AuthorsOpenReview ↗

Kind Reminder on the Discussion Deadline

Dear Reviewer 2b3c,

As a gentle reminder, the author-reviewer discussion period closes on November 18, 2026 (AoE). We would greatly appreciate it if you could take a look at our rebuttal and let us know whether it addresses your concerns. If any questions remain, we would be glad to clarify further while there is still time.

Thank you very much for your time and effort, which helps improve the quality of our work!

Best regards,

The Authors

Reviewer_MZnXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the limitations of general purpose dense retrievers for conversational memory, arguing that conversation turns tend to cluster closely in embedding space, making it difficult to distinguish relevant from irrelevant information. Labeled relevance data for memory retrieval is scarce and expensive and the authors propose MemRevela, an adaptation of Revela that learns from unlabeled conversations. The key idea is to exploit crossfragment dependencies, meaning that if one conversation fragment helps predict another, the two are likely related. An auxiliary language model predicts fragments using their prefixes and neighboring fragments, while a retriever learns similarity-based weights through the prediction loss. MemRevela is trained on ~3.6M fragment instances from WildChat-1M and UltraChat-200k and evaluated across five model sizes using LMEB, LoCoMo, and BEIR, with RepLLaMA as a controlled baseline.

Critical strengths

The approach has the potential to be deployed at scale, making the paper intersting also from an application point of view. The overall idea is valuable and despite the method not being novel, the application to conversational retrieval is in my opinion a novelty. The experimental setup is also robust and the level of details provided on the results make the paper solid and reproducible.

Critical weaknesses

The improvements shown on the paper are small and should be confirmed by adding significance tests. The baselines are all models trained to use instructions or query prefixes. MemRevela and RepLLaMA does not. Baselines lose their trained input format, while the other two models do not. I think it is an important missing aspect that would need a better explanation. Similarly, the length limits of the models are very different and it is unclear the impact. I suggest to report baselines with their recommended instructions and length limits.

Questions for authors

I'd like the author to elaborate more on why BM25 is not applicable. Downstream analysis, from table 2 I cannot see RepLLaMA-8B and the LMEB baseline. Why is the case?

3 comments

We sincerely thank the reviewer for recognizing the practical value of our approach and the robustness of the experimental setup. The suggestions were very helpful: we have added significance tests, instruction-tuned baselines, a length-limit check, BM25, and an extended Table 2, and address each point below.

W1. Significance of the improvements

We ran query-level paired randomization tests on nDCG@10 (by dataset, 10k resamples, aggregated with the LMEB [1] weighting used in Table 1). In both size groups we tested, RepLLaMA is the runner-up, so this is a first-vs-second test. Both improvements over RepLLaMA are statistically significant (p < 0.001), with confidence intervals excluding zero. After Holm correction, MemRevela significantly outperforms RepLLaMA on 13 and 10 of the 22 datasets for the 8B and 3B models, respectively. We will include these tests in the revision.

Size groupComparisonDataset Mean [95% CI]Type Mean [95% CI]pDatasets won / lostSignificant after Holm, won / lost
7-12BMemRevela-8B vs RepLLaMA-8B+2.02 [+1.63, +2.42]+2.36 [+2.03, +2.69]< 0.00120 / 213 / 1
1-7BMemRevela-3B vs RepLLaMA-3B+0.96 [+0.57, +1.34]+1.17 [+0.83, +1.51]< 0.00115 / 710 / 2

W2. Instructions and input format

MemRevela and RepLLaMA retain their trained input format, using fixed query: / passage: prefixes (Appendix B), without task-specific instructions. To address the concern about instruction-tuned baselines, we additionally compare against LMEB's results with recommended instructions (Dataset Mean nDCG@10):

RetrieverNo instructions (our Table 1)w. instructions (LMEB Table 3)
bge-multilingual-gemma245.1161.41
NV-Embed-v259.8160.25
KaLM-Embedding-Gemma353.9160.10
F2LLM-v2-4B57.8658.68
Qwen3-Embedding-8B54.6555.94
e5-mistral-7b-instruct49.6155.21
Qwen3-Embedding-4B51.4553.23
MemRevela-8B63.24-

MemRevela requires no task-specific instructions yet outperforms all 30 embedding models in LMEB's main table [1], including those evaluated with recommended instructions. It outperforms the strongest baseline, bge-multilingual-gemma2, by 1.83 point, NV-Embed-v2 by 2.99 points, and Qwen3-Embedding-8B by 7.30 points, confirming that the ranking holds. We will include these comparisons in Table 1.

W3. Length limits

We re-evaluated MemRevela at 1,024 tokens (nDCG@10, ×100):

ModelMax lengthEpisodicDialogueSemanticProceduralDataset MeanType Mean
MemRevela-1B2,04865.7958.8660.2559.4160.1561.08
MemRevela-1B1,02465.7858.8960.2459.1560.08 (−0.07)61.02 (−0.06)
MemRevela-3B2,04869.3758.6962.0560.4261.3662.63
MemRevela-3B1,02469.3858.7162.0360.1961.29 (−0.07)62.58 (−0.05)

Dataset Mean drops by only 0.07 points at both sizes (Recall@10: −0.05 and −0.02), consistent with the ≤0.2 difference LMEB reports between 512 and 1,024 tokens [1]. Consequently, the performance gains in Table 1 cannot be attributed to length differences. We will add other scales in the revision.

Q1. BM25

We agree that BM25 is a valuable lexical baseline. We initially omitted it to focus our comparison on dense embedding models, and we will include it in Table 1 in the revised manuscript. Below are the results in the Table 1 format (×100):

RetrieverEpis N@10Dial N@10Sem N@10Proc N@10Dataset Mean N@10Type Mean N@10
BM2566.1148.6253.3151.2852.6454.83
MemRevela-8B72.2661.1563.6961.7263.2464.70

As shown above, MemRevela-8B substantially outperforms BM25 across all memory types, achieving gains of +10.60 in Dataset Mean N@10. While BM25 remains relatively competitive on Episodic retrieval where lexical overlap is common, it lags by 10.4–12.5 points in N@10 on Dialogue, Semantic, and Procedural tasks that require deeper semantic abstraction beyond surface-level keyword matching.

Q2. Other baselines in Table 2

Due to GPT API budget constraints, our initial submission only included 3 representative baselines. Following your suggestions, we have now added RepLLaMA-8B, Qwen3, NV-Embed-v2, and additional API-based models. MemRevela achieves the best average performance under both generation settings.

RetrieverLlama DialogObs.Summ.Avg.GPT-4o-mini DialogObs.Summ.Avg.
text-embedding-3-large50.1348.1834.6344.3152.5150.6636.4746.55
Voyage-3-large49.1747.2036.7944.3852.1749.6137.8446.54
E5-Mistral-7B-Instruct49.8247.2136.2744.4451.7849.4036.8546.01
Qwen3-Embedding-4B43.8948.6631.6841.4147.5449.9032.1243.19
Qwen3-Embedding-8B48.6147.9932.5343.0450.3350.0333.8444.73
NV-Embed-v248.5747.6437.1644.4650.3350.1238.9446.46
RepLLaMA-8B49.1547.4537.8544.8250.8250.3438.9246.69
MemRevela-8B51.1548.0937.9145.7253.4150.9538.6047.65

Thank you again for the detailed and constructive review, which is very helpful for us to further improve the paper quality! We are happy to address any further questions.

[1] LMEB: Long-horizon Memory Embedding Benchmark, arxiv 2603.12572

AuthorsOpenReview ↗

Kind Reminder on the Discussion Deadline

Dear Reviewer MZnX,

As a gentle reminder, the author-reviewer discussion period closes on November 18, 2026 (AoE). We would greatly appreciate it if you could take a look at our rebuttal and let us know whether it addresses your concerns. If any questions remain, we would be glad to clarify further while there is still time.

Thank you very much for your time and effort, which helps improve the quality of our work!

Best regards,

The Authors

Comments on the paper

1 comment

Dear reviewers, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.