- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies retrieval for long-term memory in LM agents. The authors argue that general-purpose embedding models do poorly here partly because they pack conversational turns into a narrow region of embedding space, which makes same-topic turns hard to tell apart. MemRevela applies Revela-style retriever-weighted language modeling to unlabeled conversations (WildChat, UltraChat): the retriever scores the other fragments in a group, those scores weight an auxiliary LM’s cross-fragment attention, and the LM’s next-token loss trains the retriever through the weights. This is followed by joint training with an MS MARCO contrastive loss, and the auxiliary LM is dropped at inference. On the 22 LMEB datasets, the method beats same-backbone RepLLaMA reproductions at every size from 0.6B to 8B, with the 8B model reaching 63.24 nDCG@10. It also gives small gains on LoCoMo QA over three external retrievers and roughly matches RepLLaMA on BEIR. The analysis shows sharper within-conversation evidence discrimination and a smaller dispersion gap between conversational and general text, and the ablations attribute most of the gain to conversational initialization.
Critical strengths
-
The supervision source fits the problem well. Dependencies between conversation fragments are free and plentiful, and the method uses them without generated queries or relevance labels; since the auxiliary LM is discarded and memories are still encoded independently, the retriever can replace the embedding model in an existing memory pipeline with no other changes.
-
The core comparison is properly controlled. Gains hold against same-backbone, same-label RepLLaMA reproductions at all four sizes from 0.6B to 8B, and the ElicitR comparison suggests the retriever-weighted LM mechanism alone does not account for them.
-
The ablations are useful beyond this paper. Stage 1 alone already beats MS MARCO-only training on episodic memory, and conversational initialization accounts for most of the final gain, which tells follow-up work where the effort is worth spending.
Critical weaknesses
-
The control needed for the central claim is missing. MemRevela gets about 7k extra steps of stage-1 training that RepLLaMA never sees, so the gain could come from additional self-supervised training rather than conversational structure. A compute-matched stage 1 on generic text (e.g., Wikipedia) would settle this.
-
Gains over the controlled baseline are modest and partly confounded. The improvement over RepLLaMA is 0.95 to 2.04 nDCG@10 with no seed variance reported, and at 8B, where it is largest, the two models use different LoRA ranks (256 vs. 64). The headline margin over NV-Embed-v2 also leans on the instruction-free setting, given that RepLLaMA-3B alone already beats NV-Embed-v2.
-
The representational-mismatch explanation is correlational. A smaller ΔD does not show that dispersion drives retrieval quality, and it could just as well come from MS MARCO passages tightening as from conversational turns spreading out. Centering or whitening RepLLaMA’s embeddings would be a cheap check; if that recovers most of the gain, the explanation should be revised.
Questions for authors
see weaknesses
3 comments
Thank you for the detailed response and the additional experiments. They address my main concerns, and I will maintain my positive score.
Hi Reviewer1pqs,
We are glad that your concerns have been mostly resolved. And thank you again for your insightful and constructive suggestions and recognition on our work.
Regards, Authors
Thank you for your recognition on our work and the constructive and insightful review. We ran all three checks as you suggested.
W1. Stage 1 on generic text
We reran the same recipe with FineWeb in place of conversations (a broader sample of general text than Wiki).
The extra stage-1 steps on generic text give little gain over RepLLaMA (+0.22, +0.04), while conversations give +1.21 and +0.99. Before any labels, conversations are ahead of generic text by +4.29 and +6.31. The gain comes from the conversational data, not from additional self-supervised training.
W2. Size and reliability of the gains
Significance. Query-level paired randomization tests on nDCG@10 (by dataset, 10k resamples; baselines as in Table 1). Both gains are statistically significant (p < 0.001), and both 95% confidence intervals exclude zero.
LoRA rank at 8B. We retrained RepLLaMA-8B at rank 256 to match MemRevela-8B (nDCG@10, ×100):
The rank does not explain the gain. At matched rank the baseline is essentially unchanged (−0.17 on Dataset Mean), and MemRevela-8B is ahead by +2.21 (p < 0.001) and on all four memory types. We will report the rank-256 baseline in Table 1.
Instructions. With each baseline in its recommended instruction setting (LMEB [1] Table 3),
MemRevela-8B(63.24, no instructions) is still ahead ofNV-Embed-v2(60.25, +2.99) and of the strongest baseline,bge-multilingual-gemma2(61.41, +1.83). We agree that a contrastive-only LLM retriever is already competitive withNV-Embed-v2without instructions on LMEB. This is why our main evidence is the controlled comparison with RepLLaMA, not the margin over external models.W3. Is the dispersion explanation causal?
We agree that the analysis is correlational, and we ran the suggested check: centering and whitening on both 8B models (transform estimated on 20k held-out WildChat chunks), re-scored on the 8 dialogue and episodic LMEB datasets (nDCG@10):
Post-hoc correction does not recover the gain: centering leaves RepLLaMA unchanged, whitening hurts both models, and the gap stays between +1.4 and +2.1 throughout. The smaller ΔD comes from conversational turns spreading out (
D_convfrom 0.363 to 0.444), not from MS MARCO passages tightening (D_genfrom 0.494 to 0.503).The check also supports your caution: centering raises RepLLaMA's
D_convto 0.84 with no change in nDCG, so dispersion alone does not drive retrieval quality. We will present ΔD as a diagnostic of the representation change, not its cause.Thank you again for the recognition. We hope this resolves your concerns and are happy to address any further questions.
[1] LMEB: Long-horizon Memory Embedding Benchmark, arxiv 2603.12572