acceptodds
Withdrawn from ICLR 2027

EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing embedding models are inherently static: they encode text segments in isolation, ignoring their surrounding context and temporal order. This paper introduces EvoEmbedding, a novel embedding model that generates evolvable representations for retrieval. It is tailored for long-context scenarios, where information is dynamic, sequential, and requires continuous state tracking. Our design is simple: EvoEmbedding maintains a continuously updated latent memory as it sequentially processes inputs, and uses it alongside the raw content to jointly generate evolvable embeddings. Consequently, for the same query, our model adapts its representation to retrieve distinct targets based on the evolving context, going beyond static semantic search. To equip the model with this capability, we construct EvoTrain-180K, a diverse dataset for the joint optimization of latent memory and retrieval. Furthermore, we introduce a memory queue to prevent representation collapse during recurrent encoding, alongside segment-batching techniques that tackle significant length variance and accelerate training by 3.8. Extensive experiments show that our model not only outperforms larger-scale specialists (e.g., Qwen3-Embedding-8B and KaLM-Embedding-Gemma3-12B) across a range of long-context retrieval benchmarks, but also generalizes well to downstream tasks (e.g., personalization) with contexts 10 longer than its training window. Notably, EvoEmbedding seamlessly integrates into agentic workflows to boost performance. For instance, a naive RAG pipeline equipped with our model surpasses dedicated agentic memory systems. Our code is available in the supplement.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Evolving Embedding, Long-Context Retrieval, Agentic Memory, Representation Learning, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HyJxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoEmbedding uses a recurrent latent memory to produce context-dependent retrieval embeddings. Separate LoRA adapters learn memory evolution and retrieval, with a FIFO queue holding 512 latent tokens and 16 new tokens generated per segment step. Training combines a memory answer-prediction loss through a frozen backbone and a length-weighted contrastive loss over segments. The authors construct 184,137 synthetic training examples and introduce dynamic segment batching. Evaluations cover eight retrieval datasets and downstream conversation and personalization tasks, including integration into existing memory systems. The 4B variant reports overall Recall@10 of 80.5 and NDCG@10 of 65.2, and naive RAG reaches 77.6% accuracy on LongMemEval with top-8 retrieval. Table 4 reports a reduction in training time from 101.4 to 26.6 hours with segment batching. The intended contribution is an embedding interface that tracks sequential context while remaining usable with an external generator.

Critical strengths
  1. The method addresses a consequential retrieval limitation through a reusable interface. Conditioning both segment and query representations on memory can help distinguish semantically similar facts at different historical states, while the final retriever still returns explicit text to an external generator. The dual-adapter design and frozen-backbone memory loss provide a concrete way to train that interface without requiring access to the generator's internal states. This is a useful design direction for memory-based applications.
  2. The empirical study covers several relevant uses rather than only one aggregate score. Table 1 evaluates both recall and ranking quality; Tables 2-3 examine downstream answering and integration into three memory systems; Table 10 varies retrieval budgets. The queue and memory-loss ablations show large performance changes, and Appendix C.4 reports the encoding-time trade-off rather than hiding it. The 77.6% LongMemEval result and the improvement from 43.57% to 62.73% for A-MEM in Table 3 support taking the approach seriously, even though additional controls are needed.
Critical weaknesses
  1. The experiments do not isolate the architectural benefit from task-specific training and context access. Most retrieval baselines are released static checkpoints, whereas EvoEmbedding receives targeted dynamic QA and evidence-segment supervision. Neither removing the memory loss nor removing the queue produces a static encoder trained on the same data. Contextual retrieval is also an established research direction: Contextual Document Embeddings (Morris and Rush, 2024), Late Chunking (Gunther et al., 2024), and InSeNT (Conti et al., 2025) are relevant missing comparisons or positioning references. The claim that existing embedding models inherently ignore surrounding context is therefore too broad. The paper's sequential bounded memory may offer a distinct advantage, but a matched-backbone, matched-data static or local-context encoder and a feasible contextual-embedding comparator are needed to show it. Appendix B describes LLM verification, but does not specify source-level holdouts or deduplication against evaluation contexts and QA pairs, or report a manual label audit. This is an unaddressed contamination and annotation-quality risk, not evidence that contamination occurred.
  2. Key execution details and the proposed collapse explanation are not established sufficiently. Equation (1) is sequential, conditioning segment t on M at t-1. Section 3.1 then batches consecutive segments using one pre-batch memory, but does not define their attention masks, segment output positions, per-segment memory states, or queue update order. This can change the algorithm, not just its speed. Algorithms 1-2 describe only one segment and do not resolve the ambiguity. The paper also does not specify the gradient horizon through recurrent memory. FIFO bounds the residence time of a particular stored token, but older information can be repeatedly propagated into newly produced tokens; it does not by itself bound the recursion of that information. Table 4 establishes accuracy degradation for particular removals, but does not directly measure representation collapse or separate it from a changed memory budget and optimization difficulty. These issues affect both reproducibility and the mechanism claimed for the central design.
Questions for authors
  1. Clarification essential to reproducing the method: Please give the batched forward pass, including attention masks, segment delimiters and output positions, K-token allocation, queue update order and the memory used for each segment embedding. Are representations for segments inside a batch computed from a common pre-batch memory or successive updated memories? Can they access later segments in that batch, and is the same procedure used during training and online evaluation?
  2. Evidence that could change the novelty and attribution assessment: What is the result of a static or local-context encoder initialized from the same 4B backbone and trained on EvoTrain with matched optimization and compute? Can you compare with a contextual embedding or late-chunking approach within a feasible context window, and distinguish the benefit of contextualization from the benefit of bounded recurrent memory?
  3. Clarification of data and evaluation integrity: How are evaluation contexts and QA pairs excluded from EvoTrain? Please state context sources, source-level train/development boundaries, and manual evidence-label audit results. Specify benchmark versions, segmentation, timestamp handling, and generator and judge prompts. Paired uncertainty over independent histories and repeated generation runs would help assess downstream margins.
  4. Clarification of the ablations: What replaces the queue, with what capacity and update cost? Are memory gradients truncated or detached? Can capacity-matched diagnostics distinguish representation collapse from poor optimization?
  5. Clarification of deployment cost: Table 8 reports encoding time of 22.08 seconds versus 5.52 for Qwen3-Embedding-8B. Please define the timing unit and report total encoding and query costs, index storage and incremental update behavior. Specify that zero token overhead concerns additional generator tokens. Also correct the bibliography: LongMemEval is ICLR 2025 and A-MEM is NeurIPS 2025, rather than the 2026 venues listed.
Reviewer_MtQfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoEmbedding addresses the limitations of static embeddings models by making embeddings depend on what came before them temporally. An LLM (Qwen) keeps a latent memory, a FIFO queue of 512 tokens. Each chunk is embedded together with the memory and the query is embedded with the final memory state. EvoEmbedding is trained on 180k synthetically generated examples using a contrastive retrieval loss and an additional memory generation loss.

The 4B EvoEmbedding model beats much bigger static embeddings overall across eight long context retrieval benchmarks. It outperforms LightMem and other agentic retrieval systems on LongMemEval and can be used as a plugin for existing memory systems, giving significant increases to downstream accuracy.

Critical strengths

This is a simple, well motivated idea that works well with any generator in addition to a smart memory generation loss function. It uses far less GPU memory than the baselines and performs well across multiple datasets. The data efficiency is impressive - very strong results from only 180k synthetic samples and a single epoch.

Critical weaknesses

The ablations in table 4 change how memory is managed or trained, but don’t remove it entirely. There is no version of EvoEmbedding trained on the 180k data samples without memory and with the same contrastive loss, so it is difficult to untangle where the contribution is coming from - the architecture or the dataset.

Only static baselines are considered. There is no comparison with other context aware chunk embedders (e.g. contextual document embedders).

Do EvoEmbeddings segments include session timestamps? I’m wondering where the temporal gap comes from between LighMem and EvoEmbedding-4B in table 9 (~35% worse on LoCoMo relative to LightMem) as this slightly undercuts your claim that naive RAG with EvoEmbedding “outperforms agentic memory systems” as it is worse on temporal and open-domains. Since full context performs 51.7 on temporal questions, this is likely not a retrieval error, but more likely something to do with resolved dates perhaps. A discussion of this limitation should be included in the main paper beyond being “slightly behind”.

All claims are made using a single seed, which can make it difficult to claim improvements where the gaps are often small - 3 seeds would be better.

Questions for authors

Have you tried training EvoEmbedding on the 180k data samples without memory? Can you add some other context-aware chunk embedders as baselines? Do EvoEmbeddings segments include session timestamps? This affects how the encodings can relate to temporal events as opposed to positions in the queue. Can you add additional seeds for training? Are the training and evaluation pipelines robust?

Reviewer_DiwSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EvoEmbedding, an LLM-based embedder for retrieval inside one long stream, such as a long document or a multi-session chat. The model reads the stream segment by segment. A memory LoRA writes K=16K=16 soft tokens per step into a FIFO queue of capacity C=512C=512, so the queue holds the last 32 steps. A second LoRA embeds each segment conditioned on the queue built before it, and the query is embedded with the final queue. Training uses a length-weighted multi-positive contrastive loss, whose negatives are the other segments of the same sample, plus a cross-entropy loss in which the frozen backbone must answer the query from the memory alone. Segment-batching is used for speed. The training set, EvoTrain-180K, is synthesized from FineWeb, LLM-written dialogues and extracted memories, with positives labeled by Gemini.

Models at 0.8B, 2B and 4B are evaluated on R@10/NDCG@10 over eight retrieval sets and on naive-RAG QA accuracy over LoCoMo, LongMemEval and three PersonaMem variants. The paper also uses the model as a reranker inside A-MEM, LightMem and MemoryOS, plots similarity curves for "firstly/lastly" queries, and ablates the queue, the memory loss, length weighting and batching.

Critical strengths
  1. The design is simple and easy to deploy. A fixed-size queue and two LoRAs on a frozen backbone keep GPU memory bounded (Table 8 shows lower peak VRAM than Qwen3-Embedding-4B), and the output is a plain 1024-d vector that works with any vector store. Routing the memory loss through the frozen backbone so the soft tokens stay readable is a neat idea, and Table 4 shows training falls apart without it. This is the part of the paper I would most like to see developed.

  2. Table 1 shows large retrieval gains at all three sizes. ESG-Reports goes from 63.6 to 84.0 R@10, QASPER from 73.8 to 85.1, and LoCoMo from 65.6 (best baseline) to 76.3. The 0.8B model beats KaLM-Embedding-Gemma3-12B overall (76.7 vs 72.7). If these numbers hold up against a fair control (W1), they would matter for long-document and chat retrieval.

  3. Table 3 is the one well-controlled comparison. All methods rerank the same 256 MiniLM candidates with the same top-20 budget and generator, and EvoEmbedding beats Qwen3-Reranker-4B, a cross-encoder, in all three frameworks at similar GPU memory. I checked the per-category numbers in this table and they are consistent.

Critical weaknesses
  1. Nothing in the paper ties the gains to the evolving memory. Appendix C.2 says that starting from Qwen3-4B makes the comparison with Qwen3-Embedding-4B fair, but the training data differ, and EvoTrain-180K (persona dialogues, user memories, coreference and temporal templates) looks close to the conversational test sets. The missing control is the same backbone, LoRA and contrastive loss trained on EvoTrain-180K with the memory removed.

Fig. 7 (left) suggests the memory does little. A queue of C=16C=16, which holds a single step, already reaches 79.1 overall R@10, against 80.5 at C=512C=512. KaLM-12B is at 72.7, so 6.4 of the 7.8-point margin is there without any long-range memory. The two ablations in Table 4 that remove memory components collapse to 15-17 on LoCoMo and 10-11 on LongMemEval, far below All-MiniLM-L6-v2 at the same k=4 (32.9 and 47.0 in Table 10). They show a failed training run and say nothing about what the memory adds. Contextual chunk embedding methods (prepending the preceding turns before a static embedder, late chunking, CDE, SitEmb) are the obvious competitors and are neither compared nor cited. Segment-batching also concatenates up to 2048 tokens of raw text, and the paper does not say whether segments in a batch attend to each other. If they do, every segment sees raw local context regardless of the memory.

  1. The temporal and long-context claims are not supported. In Fig. 4 the top row shows only "firstly" and "lastly", though the caption lists three settings. The t-SNE is colored by query keyword, so its clusters show that the three query vectors differ; the caption reads them as chronological positions. The query sees a queue holding the last 32 of 256 segments, so a "firstly" peak at index 0 is also what you would get if the model used the near-empty queue at the start of the stream as a position cue. The probe reports no retrieval accuracy, and I could not tell whether its template is among the 40+ training templates.

On real benchmarks, the case that motivates the paper (Fig. 1, an updated fact) barely moves. Against KaLM, LongMemEval knowledge-update is 66 vs 63 correct out of 78, temporal reasoning is 84 vs 81 out of 133, and LoCoMo temporal is 49.8 vs 47.7. Most of the LongMemEval margin comes from multi-session questions (95 vs 78). The claim of generalizing to 128K contexts, 10x the training window, follows from the design: the average training sample (21 segments) fits inside the 32-step queue, so the model works as a sliding window in training and at test time. On PersonaMME-128K the 4B model does not beat static baselines (72.81 vs 72.96 for Jina at k=4, 73.68 vs 74.07 for BGE-M3 at k=8).

  1. The agentic memory comparison does not add up, and the headline claim holds for static embedders too. In Table 2, weighting LightMem's per-category scores by the LongMemEval-S category sizes (133/133/78/70/56/30) gives 58.1%, while the table reports 70.2%. With the other categories fixed, getting to 70.2 would take about 78 correct answers among 56 single-session-assistant questions. Mem0's entries cannot come from these sizes at all (26.1% of 56 is 14.6 questions), while every other row reproduces its overall exactly. In Table 9, most rows reproduce their overall with the LoCoMo sizes (single-hop 841, multi-hop 282, temporal 321, open-domain 96). The MiniLM row only does so with the mapping used in Table 3 (282/321/96/841), and LightMem with a third one, so a column means different things in different rows. The text's statement that LightMem leads on Temporal and Open-domain rests on these columns.

Even taking Table 2 as given, naive RAG with KaLM (72.8) or Qwen3-Embedding-8B (71.4) already beats LightMem (70.2), so the abstract's claim that naive RAG with EvoEmbedding "surpasses dedicated agentic memory systems" applies to strong static embedders as well. Qwen3-Embedding-4B, the baseline that shares the backbone, is the weakest model in Table 1 (LongMemEval R@10 58.7 vs 84.7 for the 0.6B) yet beats the 0.6B downstream in Table 10 (60.6 vs 54.8 at k=4), which suggests it was misconfigured. The "zero token overhead" in Fig. 1 also leaves out that a 4B LLM reads the whole history twice; Table 8 reports 22.1 s of context encoding against 3.8 s for Qwen3-Embedding-4B.

Questions for authors
  1. Please train the same backbone, LoRA and loss on EvoTrain-180K with the memory input removed, plus a static variant that embeds each segment with the previous N segments prepended as raw text, and report Tables 1 and 2 for both. Do segments in the same batch attend to each other during the embedding pass?

  2. What is the effective memory horizon at test time? What is the typical batch size k on LoCoMo and LongMemEval, and does each batch push k·K tokens into the 512-slot queue, which would overwrite it entirely once k ≥ 32? How do you reconcile the C=16C=16 result with the claim that the queue tracks long-range state? A knowledge-update test where the old fact and its update are further apart than this horizon would answer this.

  3. For Fig. 4, please add the "middle" curve, report hit@1 and hit@5 for the first, middle and last mention, use query templates outside the training templates, and add a control with unrelated filler segments prepended. Does the "firstly" peak move with the content or stay at index 0?

  4. For each agentic memory row in Tables 2 and 9, was the number reproduced or copied, and with which generator, judge and LoCoMo category mapping? Please explain the LightMem and Mem0 numbers above and re-run all rows under one protocol.

  5. Which instruction prompt, pooling and maximum length did you use for Qwen3-Embedding-4B? Why does the LongMemEval ranking in Table 1 (KaLM above EvoEmbedding in R@10 and NDCG@10) disagree with Table 10, where EvoEmbedding leads KaLM by 4.8 points? Is the Table 1 version of LongMemEval the LMEB one with a different segmentation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.