acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence

Abstract

Memory-augmented assistants retrieve evidence from stored conversations and documents to answer queries. Most embedding-based retrievers compare single-vector representations of queries and memories, even when relevance depends on a small part of the content. Such local evidence can be matched to queries using the contextual token representations already computed by the same embedding models. Our method uses sparse token matching to improve retrieval without additional training. Each query token retrieves a small set of matching tokens across candidate memories. The matching scores are aggregated for each memory and combined with whole-memory similarity to rank memories. An optional low-rank query adapter can be trained with supervision while keeping the encoder and stored memory representations fixed. Our method improves retrieval across eight benchmarks and multiple model families without additional training. Query adaptation further increases the gains to 13.3 and 14.3 percentage points in average Recall@10 and NDCG@10 on Qwen3-Embedding-4B. These retrieval gains translate into higher question-answering accuracy on long-term memory benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
natural language processing, memory retrieval, dense retrieval, text embeddings, multi-vector retrieval

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_6FHdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a more effective way for AI assistants to retrieve relevant information from past conversations and documents. Instead of relying solely on a single embedding vector for each memory, the method leverages existing token-level representations to identify fine-grained relevant evidence, combining sparse token matching with whole-memory similarity. It works without additional training, while an optional lightweight query adapter further improves performance without modifying the original embedding model or stored memories.

Critical strengths
  1. The paper is well written and clearly organized. The motivation, methodology, and experimental setup are easy to follow, with helpful figures that illustrate the proposed approach.

  2. The proposed method is simple and practical. It effectively reuses existing token representations from frozen embedding models and achieves substantial retrieval improvements without requiring additional training.

  3. The paper provides extensive experiments across eight benchmarks and eight embedding backbones. The ablation studies, downstream question-answering evaluations, and efficiency analyses further demonstrate the effectiveness of the proposed method.

Critical weaknesses

1. Limited Technical Novelty.

Token-level retrieval has already been extensively studied in prior works such as ColBERT, XTR, and SMART. The proposed method mainly combines existing techniques, including sparse token matching, global-local score fusion, and lightweight query adaptation, without introducing a fundamentally new retrieval mechanism. Although the empirical improvements are clear, the paper does not sufficiently distinguish its technical contributions from closely related methods, particularly SMART.

2. Unclear Motivation for Memory-specific Retrieval.

Although the paper focuses on long-term memory retrieval, the underlying problem of information loss in single-vector embeddings is a general retrieval challenge rather than a memory-specific one. The proposed method does not explicitly consider unique properties of long-term memories, such as temporal relationships, memory consolidation, or evolving user information. Moreover, five of the eight benchmarks are conventional document retrieval tasks, and the main experiments largely resemble standard RAG over fixed text records. While Appendix F mentions memory construction following EvoEmbedding, the effectiveness on compressed or consolidated memories is not separately examined. It remains unclear why the method is specifically positioned as a memory retrieval approach rather than a general retrieval method.

3. Significant Overhead and Insufficient Scalability Evaluation.

The method introduces substantial storage and retrieval overhead. According to Table 13, even with SQ4 compression, index sizes remain approximately 10 to 12 times larger than the single-vector baseline, although stronger compression can reduce this overhead at some performance cost. More importantly, the latency evaluation uses only a 568-candidate memory bank and eight fixed queries. Although the reported latency includes query processing, the additional runtime cost of the query adapter is not separately analyzed. As the memory bank grows, the larger token index and the need to compute global scores for all candidates may significantly affect retrieval efficiency. Without evaluations on larger memory banks, index construction, and incremental updates, the practical scalability of the method remains unclear.

4. Incomplete Comparison with Lower-cost Fine-grained Retrieval.

The paper attributes its improvements to recovering fine-grained evidence through token-level matching. However, a simpler alternative would be to combine whole-memory embeddings with sentence-level or chunk-level embeddings. Although the authors evaluate Chunk-MaxP, smaller chunk sizes, and several pooling variants, they do not clearly establish a comparable baseline that fuses global embeddings with sentence-level or chunk-level retrieval scores. Such a comparison is important because the improvements may primarily come from combining global and local information rather than token-level matching itself. Without this baseline, it remains unclear whether token-level indexing is necessary, especially given its substantially higher storage and retrieval costs.

Questions for authors
  1. What are the key technical differences between the proposed method and existing token-level retrieval approaches, particularly SMART and XTR?
  2. Why is the method specifically positioned for memory retrieval rather than general document retrieval? Does the method remain effective when memories are explicitly summarized or consolidated?
  3. How does retrieval latency scale as the memory bank grows to tens or hundreds of thousands of records?
  4. Could the authors compare against a baseline that combines global embeddings with sentence-level or contextual chunk-level embeddings using the same fusion strategy?
5 comments

Dear Reviewer 6FHd,

Thank you for your constructive feedback. Retrieval scores below are Recall@10 / NDCG@10.


1. Relation to SMART/XTR and direct comparison (W1; Q1)

SMART is concurrent work using frozen token states and pooled-score fusion. It aggregates per-record MaxSim; we retrieve matches across memories, assign a retrieval-derived fallback per query token, and fuse standardized local and pooled scores. Whereas XTR specifically trains token-retrieval representations, we reuse a frozen single-vector encoder and retain its pooled scores.

We transferred SMART's public scoring and shared-head training recipe to text retrieval, retaining additive fusion and using the same source training data. Results are eight-benchmark means; trained results average three seeds.

MethodQwen-0.6BQwen-4BQwen-8B
SMART, training-free68.63/54.5465.74/51.5070.40/57.08
SMART, trained69.71/55.8969.12/54.6271.72/58.30
Ours, training-free74.22/61.1073.77/59.9472.90/59.08
Ours, full74.52/61.5975.73/62.3173.61/60.82

Our training-free method exceeds trained SMART on both mean metrics across all eight backbones (revised Table 12). Full Qwen-4B gains 6.62/7.69 points; training-free Gemma gains 3.93/3.45. With content tokens and fusion matched, training-free Qwen-4B still exceeds MaxSim by 3.63/2.32 points, supporting the additional value of our local scoring.


2. Sentence/chunk comparisons: gains beyond global–local fusion (W4; Q4)

We fixed the encoder, candidate bank, pooled scores, and fusion. Sentences were independently encoded; contextual chunks used the same full-memory token states. We tested 32/64/128-token windows with mean and last-token pooling. Results on 1,976 LoCoMo queries include the strongest tested chunk configuration:

Local scoring + pooled fusionQwen-0.6BQwen-4B
Sentence best match42.68/31.5141.17/28.90
Contextual chunks, 128-token mean52.95/38.8249.29/36.11
Ours, training-free67.98/53.4263.50/49.70

Against the strongest tested contextual-chunk configuration, our training-free method gains 15.03/14.60 points on Qwen-0.6B and 14.21/13.59 on Qwen-4B. This supports our fine-grained matching and scoring beyond simply combining global and local similarity (Appendix D, Table 13).


3. Memory-retrieval setting and answer quality (W2; Q2)

We study how assistants retrieve evidence from accumulated interaction histories to answer subsequent questions, following the conversational-memory setting of LoCoMo and LongMemEval. We rank original memory records in a fixed candidate bank using token-level evidence (Section 3.1), evaluating retrieval and answer correctness. Three conversational benchmarks test this use case; five document benchmarks assess transfer.

Across LoCoMo, LongMemEval, and REALTALK, full Qwen-4B improves average retrieval scores by 25.60/25.64 points. With the answering model and Top-10 budget fixed, QA accuracy improves by 4.42 points on LoCoMo and 4.80 on LongMemEval. These results concern the evaluated memory records; explicit summarization or consolidation was not separately tested.


4. Compression, scaling, and index updates (W3; Q3)

RP256+SQ4 reduces index and auxiliary storage by approximately 91%. LoCoMo results:

MethodMiBRecall/NDCG
Original pooled2.2346.87/34.24
ColBERTv2 FP3212.9964.71/52.05
Ours, full153.9368.69/54.51
Ours, full, RP256+SQ413.4867.28/53.51

At similar storage to ColBERTv2's FP32 configuration, our compressed method gains 2.57/1.46 points. Ours uses Qwen-0.6B with three-seed means. Quality covers 1,976 queries; storage uses the same 568-record bank, excluding encoder weights and raw text.

We tested 1k–100k distinct LongMemEval records using Qwen-0.6B with RP128+SQ4 on identical hardware. Median latencies include adaptation, token search, pooled scoring of all candidates, fusion, and ranking, but exclude query encoding:

RecordsPooledOurs + adapterAdded latency
1,0000.20 ms1.60 ms1.40 ms
10,0002.05 ms4.00 ms1.95 ms
100,00024.78 ms30.30 ms5.52 ms

At 100k records, storage totals 6.55 GiB, and the adapter itself takes 0.30 ms. These tests measure scaling costs; LoCoMo evaluates compression quality.

We tested incremental growth from 1k to 10k and 100k records. With the encoder, projection, and quantizer fixed, new records can be appended without re-encoding existing records or rebuilding the index. Updating the query adapter also leaves the memory index unchanged.


We thank you again for the constructive feedback, which has helped clarify our contribution and strengthen the empirical evidence.

We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.

Best regards, The Authors

Dear Reviewer 6FHd,

Thank you again for your constructive review and helpful suggestions.

We hope our clarifications on the contribution and memory-retrieval setting, together with the additional chunk comparisons and scalability experiments, have addressed your questions.

We are very grateful for your suggestions, which have strengthened the paper. We would greatly appreciate your feedback and would be happy to address any remaining questions.

Best regards,

The Authors

Area_Chair_Q3CkOpenReview ↗

Please respond to the authors' rebuttal

Hello,

The discussion period ends soon. Please respond to the authors' last comments and note whether they changed your evaluation.

Thanks,

Your AC

Reviewer_yLYVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses retrieval for memory-augmented assistants, where relevance often depends on a small local part of a stored memory but single-vector retrieval scores whole-memory similarity. The core idea is to reuse the contextual token representations that a frozen embedding model already computes: each query token performs sparse (HNSW-based) top-k matching against all candidate memory tokens, matches are aggregated by source-memory ID, and the resulting local "sparse" score is fused (via query-wise standardization and a weight α) with the standard pooled score. A missing-score "floor" estimates unreturned matches. An optional low-rank, norm-preserving query adapter (rank 32, LoRA-style) is trained from memory-level labels with a MaxSim cross-entropy surrogate while keeping the encoder and memory bank fixed. Evaluated on eight retrieval benchmarks and eight embedding backbones, the training-free method improves Recall@10/NDCG@10; query adaptation adds further gains (e.g., up to 13.3/14.3 points average on Qwen3-Embedding-4B). Downstream QA accuracy improves on LoCoMo (all 8 backbones) and LongMemEval (7/8). Compression (SQ4, RP128+SQ4) and latency/storage trade-offs are reported.

Critical strengths

Practical, training-free core method: The main contribution - reusing frozen token states for sparse matching fused with pooled scores- requires no retraining of the encoder and no changes to the stored memory index, which is a genuinely attractive deployment property. Broad empirical coverage: The evaluation spans eight benchmarks (conversational, financial, scientific, biomedical, long-document) and eight backbones across multiple families and sizes. The consistency of improvements over Original embedding across all eight backbones strengthens the generality claim. Downstream validation: The paper does not stop at retrieval metrics but shows that gains translate into higher QA accuracy/F1, with explicit per-backbone ranges reported (e.g., LoCoMo accuracy gains of 3.51–16.95 points). Reasonable ablation/component analysis. Table 4 isolates the contributions of sparse matching, fusion, and adaptation; Table 11 provides scoring controls (MaxSim vs. Sparse, with/without whole-memory); Table 10 isolates the missing-score floor vs. exact search. These help attribute the gains. Attention to efficiency: The authors report storage and query latency under compression and acknowledge the index-size cost (O(Nd + |M_q|d) vs. O(|M_q|d), Section 3.1), which is important given multi-vector methods' well-known storage burden.

Critical weaknesses

Limited Novelty: The paper itself states that SMART "reuses frozen hidden states with MaxSim and pooled-score fusion" (Section 2, Zhang et al. 2026a) and that XTR studies token retrieval and missing-score estimation (Lee et al. 2023). The proposed method combines these existing ingredients (sparse token matching + missing-score floor + pooled fusion + low-rank query adapter). The specific delta over SMART is not made crisp, and crucially SMART is not included as a baseline in Tables 1, 2, 5, or 13, despite being the closest competitor. This is a significant missing comparison. Several benchmarks appear near-saturated, undercutting the headline claim: On MLDR and CovidQA, Original embedding and baselines are already very high (e.g., MLDR Recall@10 of 95–100 across methods, Table 1), and the method sometimes does not lead (BM25 leads MLDR Recall@10; XTR leads MLDR NDCG@10). Averaging over such saturated tasks inflates the "overall" improvement narrative; a breakdown emphasizing where gains actually come from (conversational benchmarks) would be more honest. The abstract's "improves retrieval across eight benchmarks" is somewhat overstated given these cases. The Qwen3-Embedding-4B "Original embedding" numbers are anomalously low and drive the headline gain: On the 4B backbone, Original embedding LongMemEval Recall@10 is 59.2 while the 0.6B model gets 84.8 (Table 1a), and LoCoMo drops from 46.9 (0.6B) to 41.6 (4B). A larger model performing substantially worse than a smaller one on the same benchmark is suspicious and may indicate an encoding/prompting issue with the 4B baseline. Since the 13.3/14.3-point headline gain is computed against this weak 4B baseline, the result may be partly an artifact of a poorly configured baseline rather than the method's strength. This is not addressed or explained in the text.

Downstream QA improvements are small and uneven in places: LongMemEval F1 changes are tiny (e.g., 12.7→13.1, 13.1→12.4 for Qwen-4B which decreases; Table 3), and the paper notes LongMemEval F1 is only "a supplemental lexical metric" (Appendix F). On Qwen3-Embedding-8B LongMemEval, overall accuracy slightly decreases (74.0→73.8). The downstream claim is therefore weaker than the abstract implies, and the small F1 magnitudes are not discussed.

Only Qwen3-Embedding-4B is compared against most baselines in detail: Table 1 gives full baseline comparisons only for the two Qwen backbones; the other six backbones (Table 2, 8, 9) are compared only against Original embedding and the method's own variants, not against standalone systems or Search-Adaptor/MaxSim/Chunk-MaxP. This limits the strength of the cross-backbone claims. Fairness of storage/latency comparison is unclear. Table 13 reports that neural methods use GPU while BM25/XTR use CPU (Appendix E), so the latency numbers across methods are not directly comparable. The method's index (153.9 MiB Full for Qwen-0.6B LoCoMo) is ~69× larger than Original embedding (2.23 MiB); even SQ4 (26.17 MiB) is ~12×. This substantial cost is somewhat downplayed relative to the retrieval gains. Query adaptation gains are marginal on several backbones: The paper acknowledges Llama-3B shows a small decrease and EmbeddingGemma is unchanged after rounding (Section 5.3). On the 0.6B backbone, adaptation adds only 0.3/0.5 points (Table 4). The adapter's value thus appears backbone-dependent and sometimes negligible, which tempers the "Query adaptation further increases the gains" framing. No statistical significance testing: Trained results average three seeds and report SD in Tables 8/9, but no significance tests are reported for the headline comparisons, and the training-free results (which are deterministic) have no uncertainty estimate. Some reported differences (e.g., 0.3-point gains) fall within seed variability.

Questions for authors

Why is the Qwen3-Embedding-4B "Original embedding" baseline so much weaker than the 0.6B baseline on LoCoMo (41.6 vs 46.9) and LongMemEval (59.2 vs 84.8)? Could this reflect a prompting/instruction or encoding issue for the 4B model? How do the headline 13.3/14.3-point gains change if the 4B baseline is corrected? SMART (Zhang et al. 2026a) is described as reusing frozen hidden states with MaxSim and pooled-score fusion—very close to your method. Why is SMART not an empirical baseline, and what precisely distinguishes your sparse-matching-plus-floor formulation from SMART? Given that MLDR and CovidQA are near-saturated and the method does not always lead there, what is the "overall" gain if restricted to the conversational benchmarks where the method is designed to help?

For the full baseline comparison (standalone systems, Search-Adaptor, MaxSim, Chunk-MaxP), why are only the two Qwen backbones reported? Do the six other backbones show the same ordering against these baselines?

Are the differences in Tables 1/2 statistically significant? For the deterministic training-free results, how do you quantify robustness given that some gaps are under 1 point?

The latency comparison mixes CPU (BM25, XTR) and GPU (neural) execution. Can you provide a hardware-matched comparison, and clarify whether the 29.21 ms query latency for Ours (Full) includes HNSW search over all candidate tokens?

The missing-score floor appears critical (removing it costs up to 7.14 Recall@10 points, Section 5.3). How sensitive is the floor to the choice of k, and how does it behave when a memory has no retrieved tokens at all?

On LongMemEval, F1 changes are near zero or negative for some backbones (e.g., Qwen-4B 13.1→12.4). Does the method improve retrieval but not grounded answer quality on this benchmark, and how should these F1 numbers be interpreted?

4 comments

Dear Reviewer yLYV,

Thank you for your detailed feedback. Retrieval scores are Recall@10 / NDCG@10.


1. SMART/XTR and direct comparison (W1; Q2)

SMART is concurrent work using per-record MaxSim and pooled-score fusion. We retrieve tokens across memories, use a retrieval-derived fallback per query token, and fuse standardized local and pooled scores. Unlike XTR's specifically trained representations, we reuse a frozen single-vector encoder and retain its pooled scores.

We transferred SMART's public scoring and shared-head training recipe to text retrieval, retaining additive fusion and using the same source training data. Eight-benchmark means follow; trained results average three seeds.

MethodQwen-0.6BQwen-4BQwen-8B
SMART, training-free68.63/54.5465.74/51.5070.40/57.08
SMART, trained69.71/55.8969.12/54.6271.72/58.30
Ours, training-free74.22/61.1073.77/59.9472.90/59.08
Ours, full74.52/61.5975.73/62.3173.61/60.82

Our training-free method exceeds trained SMART on both mean metrics across all eight backbones (revised Table 12). Full Qwen-4B gains 6.62/7.69 points; training-free Gemma gains 3.93/3.45. With tokens and fusion matched, training-free Qwen-4B still exceeds MaxSim by 3.63/2.32, supporting our local scoring.


2. Baseline verification and gain breakdown (W2–W3; Q1, Q3)

We checked Qwen's official encoding settings. Re-encoding all 5,873 queries for each of Qwen-0.6B and Qwen-4B reproduced the pooled vectors and Top-10 rankings. LMEB v1, Table 22 also places 4B below 0.6B: its 4B Recall@10 is 41.28/58.74 on LoCoMo/LongMemEval, close to our 41.60/59.17. The audit leaves our gains unchanged.

Scope: full-method gainsQwen-0.6BQwen-4B
All eight benchmarks+8.96/+10.75+13.27/+14.35
Three conversational benchmarks+13.36/+14.51+25.60/+25.64
Excluding MLDR and CovidQA+11.13/+11.79+17.27/+16.80

Conversational tasks: LoCoMo, LongMemEval, REALTALK. Removing near-saturated tasks increases the gains.


3. Broader comparisons and statistics (W5, W7–W8; Q4–Q5)

Search-Adaptor comparisons cover all eight backbones; full-method gains are 1.50–8.85/3.99–10.03 points. Matched MaxSim covers five backbones across three families: full Qwen-8B gains 2.00/1.88, and Gemma 2.16/2.21. Chunk-MaxP covers the two original Qwen models.

We used 10,000 paired bootstrap resamples of memory banks within benchmarks, preserving shared-bank queries and averaging weights. All eight backbones' mean gains over original retrieval have positive 95% intervals for both metrics and variants:

Backbone/variantRecall gain [95% CI]NDCG gain [95% CI]
0.6B/free8.66 [6.61,10.89]10.25 [8.77,11.85]
0.6B/full8.96 [6.94,11.18]10.75 [9.31,12.29]
4B/free11.31 [9.40,13.17]11.97 [10.36,13.59]
4B/full13.27 [11.43,15.03]14.35 [12.59,16.10]

Across-seed SD is separate: full Qwen-4B scores 75.73±0.09/62.31±0.07. Adaptation improves both means on six backbones, adding 1.96/2.37 on 4B with positive paired intervals.


4. Answer accuracy and F1 (W4; Q8)

With the answering model, Top-10 budget, and evaluator fixed, full Qwen-4B improves LongMemEval accuracy by 4.80 points, a net 24 additional correct answers. F1 changes from 13.12 to 12.45, while reference-word coverage rises 69.35%→71.43%. Longer answers reduce lexical precision, explaining the differing accuracy and F1 trends.


5. Compression and matched-hardware costs (W6; Q6)

RP256+SQ4 reduces storage by 91%. LoCoMo results:

MethodMiBRecall/NDCG
Pooled2.2346.87/34.24
ColBERTv2 FP3212.9964.71/52.05
Ours full153.9368.69/54.51
Ours compressed13.4867.28/53.51

Compressed gains over ColBERTv2 FP32 are 2.57/1.46 at similar storage. Ours uses Qwen-0.6B; quality covers 1,976 queries, storage the same 568-record bank, excluding weights/text.

The original 29.21 ms includes encoding and HNSW search. On identical hardware, RP128+SQ4 adds 1.40/1.95/5.52 ms over pooled retrieval at 1k/10k/100k records, including adaptation but excluding encoding. At 100k, retrieval takes 30.30 ms and storage is 6.55 GiB. This measures scaling cost; LoCoMo tests compression quality.


6. Token budget and unmatched memories (Q7)

For each query token, k matches are retrieved across the bank. We tested k=16,32,64,128 on LoCoMo/LongMemEval with Qwen-0.6B/4B, Llama-8B, and Gemma (Figure 4; Appendix D, Table 14). Performance varies modestly in most configurations; every setting improves both metrics over pooled retrieval.

Missing matches receive the last retrieved similarity clipped at zero (Section 3.2). Unmatched memories remain eligible through pooled scoring (Section 3.3).


We thank you again for the detailed feedback. We would appreciate your reconsideration in light of this evidence.

Best regards,
The Authors

Dear Reviewer yLYV,

Thank you again for your detailed feedback and thoughtful suggestions.

We hope our additional comparisons, statistical analysis, and clarifications on answer quality and efficiency have addressed your questions.

We are very grateful for your suggestions, which have strengthened the paper. We would greatly appreciate your feedback and would be happy to address any remaining questions.

Best regards,

The Authors

Area_Chair_Q3CkOpenReview ↗

Please respond to the authors' rebuttal

Hello,

The discussion period ends soon. Please respond to the authors' last comments and note whether they changed your evaluation.

Thanks,

Your AC

Dear Reviewer yLYV,

Thank you again for your detailed review and constructive suggestions.

With the discussion period concluding today, we would like to briefly follow up on our responses. We hope the SMART comparisons, baseline verification, statistical analysis, and clarifications on QA and efficiency have addressed your questions.

We particularly appreciate your suggestion to examine gains on conversational benchmarks, which helped clarify where the method is most effective. We would greatly appreciate your feedback and would be happy to clarify any remaining points.

Best regards,

The Authors

Reviewer_pAyiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Paper reuses contextual token states from frozen embedding models for memory retrieval: performs global top-k search, aggregates matches by memory, fuzes normalized local scores with pool similarity.

The paper is well designed experiment that beats one-vector search; but what it adds beyond SMART is small. SMART also doesn't show up in the comparison tables.

Critical strengths
  • Authors performed reasonable sized tests: 8 models, 8 benchmarks, 5873 questions total. Improvements were seen across all 8 models
  • Design choice removal was methodological. Fallback scores is clearly important (1.6-7.1 points); top-64 shortlist beats checking every word.
  • Good results without extra training is a good result.
Critical weaknesses
  • What authors add over SMART: SMART already does frozen word vectors and summary-score blending. Authors cite it but don't add that to the benchmarks. This is the biggest concern with the paper.
  • Deployment and end-to-end evidence is narrower than broad retrieval claim: cost is measured on one representative small bank with limited query repetition; main QA comparison uses only pooled vs adapted retrieval.
  • Very large memory stores are untested as a part of this paper; which is more practical for commercial uses.
Questions for authors

Plese add comparisons with SMART

6 comments

Dear Reviewer pAyi,

Thank you for recognizing the experimental design and training-free results. Retrieval scores below are Recall@10 / NDCG@10.


1. Relation to SMART and direct comparison (W1; Q1)

SMART is concurrent work that also uses frozen token states and pooled-score fusion. Our method differs in local scoring: SMART aggregates per-record MaxSim; our method retrieves matches across memories, assigns a retrieval-derived fallback for each query token, and fuses separately standardized local and pooled scores.

We evaluated SMART’s public scoring implementation and shared-head training recipe in text retrieval, retaining its original additive fusion and using the same source training data as our adapter.

Results are eight-benchmark means; trained results average three seeds.

MethodQwen-0.6BQwen-4BQwen-8B
SMART public scoring, text adaptation68.63 / 54.5465.74 / 51.5070.40 / 57.08
SMART public training recipe, text adaptation69.71 / 55.8969.12 / 54.6271.72 / 58.30
Ours, training-free74.22 / 61.1073.77 / 59.9472.90 / 59.08
Ours, full74.52 / 61.5975.73 / 62.3173.61 / 60.82

Our training-free method exceeds the trained SMART adaptation on both average metrics across all eight evaluated backbones (revised Table 12). On Qwen-4B, the full method improves over that comparator by 6.62 / 7.69 points. On Gemma-300M, the training-free method alone improves by 3.93 / 3.45 points, extending the advantage beyond the Qwen family.

With content tokens and standardized fusion matched, our training-free method still exceeds MaxSim by 3.63 / 2.32 points on Qwen-4B, supporting the additional value of our local scoring.


2. QA gains and component breakdown (W2)

With the answering model, Top-10 retrieval budget, and evaluation procedure fixed, full Qwen-4B improves QA accuracy by 4.42 points on LoCoMo and 4.80 points on LongMemEval.

Retrieval methodLoCoMoLongMemEval
Original pooled63.44%67.80%
Ours, training-free61.30%72.80%
Ours, full67.86%72.60%

Query adaptation contributes differently across the two benchmarks. On LoCoMo, it adds 6.56 points, bringing the full method above original pooled retrieval. On LongMemEval, training-free retrieval already yields a 5.00-point gain, compared with 4.80 points for the full method. Evaluations cover 1,540 LoCoMo questions and 500 LongMemEval questions; the answering model and evaluation protocol are specified in Appendix F.


3. Compression and larger-scale costs (W2–W3)

RP256 + SQ4 reduces index and auxiliary storage from 153.93 MiB to 13.48 MiB—approximately 91%.

MethodIndex + auxiliary storageRecall@10 / NDCG@10
Original pooled2.23 MiB46.87 / 34.24
ColBERTv2 (FP32)12.99 MiB64.71 / 52.05
Ours, full, uncompressed153.93 MiB68.69 / 54.51
Ours, full, RP256 + SQ413.48 MiB67.28 / 53.51

Our full-method results are averaged over three seeds.

At a similar storage footprint to ColBERTv2’s FP32 configuration, our compressed method improves Recall@10 / NDCG@10 by 2.57 / 1.46 points. LoCoMo quality is evaluated on 1,976 queries; storage uses the same 568-record bank, excluding encoder weights and raw text. Our method uses Qwen-0.6B; ColBERTv2 uses its official encoder.

We extended cost measurements to 1,000–100,000 distinct LongMemEval records, using Qwen-0.6B with RP128 + SQ4. The table reports median retrieval latency, including query adaptation, token search, pooled scoring, fusion, and ranking, but excluding query encoding.

RecordsOriginal pooledOurs, compressed + adapterAdded latency
1,0000.20 ms1.60 ms1.40 ms
10,0002.05 ms4.00 ms1.95 ms
100,00024.78 ms30.30 ms5.52 ms

At 100,000 records, the added retrieval latency is 5.52 ms, with 6.55 GiB of index and auxiliary storage. These measurements evaluate scaling costs; the preceding LoCoMo experiment evaluates compression quality.


We thank you again for the constructive feedback, which has helped clarify our contribution and strengthen the empirical evidence.

We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.

Best regards,
The Authors

Thanks for the follow up, appreciate the time spent on this detailed answer and the experiments that were ran.

  1. SMART direct comparisons
  • Appreciate the detailed run updates. For SMART, did you tune the blend weights? Will you release
  • Unfortunately concurrent work will not check out, you've referenced it in your paper already and it came out in May.
  1. QA gains and component breakdown
  • Please add this to the updated paper.
  1. compression and larger-scale costs
  • Add details to the paper as necessary.

Overall, this increases my confidence. The score improvements over SMART on the other models are there (albeit smaller).

Dear Reviewer pAyi,

Thank you for the encouraging follow-up. We are glad that the additional comparisons and experiments have strengthened your confidence in the work.

For SMART, we retained its public additive fusion without tuning the blend weights. On Qwen-4B, our training-free method improves over training-free SMART by 8.03 / 8.44 Recall@10 / NDCG@10 points, and our full method improves over trained SMART by 6.62 / 7.69 points. Notably, our training-free method also exceeds trained SMART by 4.65 / 5.32 points. We plan to release the implementation and evaluation configurations.

The QA component breakdown has now been added to Appendix F, Table 21. The compression and larger-scale cost results, together with their measurement details, are included in Appendix E, Tables 19–20.

Thank you again for the constructive discussion, which has helped us strengthen the experimental evidence and improve the revised paper.

Best regards,
The Authors

Reviewer_KawTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves the retrieval stage of memory-based assistants that store conversation histories or documents to answer questions. The key motivation is that single-vector embeddings capture overall similarity between a query and a memory, making it difficult to retrieve relevant information when the answer depends on only a small portion of the memory.

The proposed method leverages token-level hidden states from a frozen, pretrained embedding model without modifying the encoder itself.

  • Sparse token matching: Each query token retrieves its top-kk nearest tokens (64 by default) across all candidate memories using HNSW. The retrieved token scores are then aggregated by their corresponding memories. For memories with no retrieved tokens, an XTR-style scoring mechanism assigns a floor value based on the similarity of the last retrieved token.
  • Local–global fusion: The token-level scores and conventional single-vector similarity scores are normalized per query and combined with equal weights (α=0.5\alpha = 0.5). No additional training is required for this stage.
  • Optional query adapter: A rank-32 low-rank residual adapter is trained exclusively on query tokens, leaving both the encoder and stored memory representations unchanged. The authors report consistent improvements across eight benchmarks, including three conversational memory benchmarks such as LoCoMo and LongMemEval, and five document retrieval benchmarks, using eight backbone models. With Qwen3-Embedding-4B, the method improves average Recall@10 and NDCG@10 by 13.2 and 14.3 points, respectively. The authors further claim that these retrieval improvements translate into better downstream question-answering accuracy.
Critical strengths
  1. Training-free applicability and broad evaluation. The method requires only storing additional token-level representations, without modifying existing embedding models or indexes, and consistently outperforms the original embeddings across all eight backbones and eight benchmarks. It is also encouraging that the retrieval improvements translate into downstream QA accuracy gains, ranging from 3.5 to 17 points across all eight backbones on LoCoMo. The ability to deploy the method without additional training could have a meaningful impact on real-world applications.

  2. Thorough ablation studies and efficiency analysis. The paper provides comprehensive controlled experiments, including a 2×2 comparison of MaxSim versus sparse matching and the use of global similarity (Table 11), exact search versus HNSW, the effect of the score floor (Table 10), sensitivity to the token budget kk and fusion weight α\alpha, and storage and latency measurements involving 4-bit quantization and random projection. Results involving training are reported as means and standard deviations across three random seeds.

Critical weaknesses
  1. Limited novelty over the closest prior work and missing direct comparisons.

As the paper acknowledges, SMART already applies MaxSim to hidden states from a frozen embedding model, combines the resulting scores with single-vector similarity, and explores lightweight adapter training. XTR introduces token-level retrieval and a floor-value estimation mechanism for missing scores. The proposed method therefore appears to combine XTR-style token retrieval with SMART's setting, making its novelty somewhat limited. However, a direct comparison with SMART is missing. Although the “MaxSim + Whole” variant in Table 11 appears similar to SMART, a comparison using SMART's official configuration, including its score fusion strategy and adapter, is needed to establish the contribution of the proposed method.

  1. The baselines appear unusually weak.
  • Anomalous performance of the original embeddings. The original Qwen3-Embedding-4B achieves lower performance (62.5/48.0) than the 0.6B model (65.6/50.8). The gap is particularly striking on LongMemEval Recall@10, where the 4B model scores 59.2 compared with 84.8 for the 0.6B model. The 8B model (69.1) also underperforms the 1B Llama model (74.8). Such results raise concerns that the original embeddings may not be functioning properly under the chosen settings, potentially due to input length, instruction formatting, or last-token pooling. Yet the headline improvements reported in the abstract (+13.2 Recall@10 and +14.3 NDCG@10 points) are based on this weakest 4B baseline. The gains are substantially smaller for stronger-performing backbones, with only +2.5 Recall@10 for Llama-8B and +2.8 for Llama-1B.
Questions for authors

The paper presents a practical method that can be applied to existing embedding models without additional training. The evaluation is broad, covering eight backbones, eight benchmarks, downstream QA, and efficiency analysis, with thorough ablation studies. However, the novelty appears limited, as the method closely resembles a combination of XTR and SMART. The lack of a direct comparison with SMART, the closest prior work, and the reliance on an unusually weak baseline for the headline results are significant concerns.

7 comments

Dear Reviewer KawT,

Thank you for your constructive feedback. We added direct SMART comparisons and an encoding audit to clarify our contribution and baseline reliability. Scores below are Recall@10 / NDCG@10.


1. Relation to SMART/XTR and direct comparison (W1)

SMART is concurrent work that also reuses frozen token states with pooled-score fusion. SMART aggregates per-record MaxSim; our method retrieves matches across memories, assigns a retrieval-derived fallback for each query token, and fuses separately standardized local and pooled scores. XTR studies token retrieval and missing-score estimation with specifically trained representations. We instead use the existing token states of a frozen single-vector encoder and retain its original pooled scores. Optional query adaptation updates only the local query branch, leaving the encoder, pooled scores, and memory index unchanged.

We evaluated SMART’s public scoring implementation and shared-head training recipe in text retrieval, retaining its original additive fusion and using the same source training data as our adapter.

Results are eight-benchmark means; trained results average three seeds.

MethodQwen-0.6BQwen-4BQwen-8B
SMART public scoring, text adaptation68.63 / 54.5465.74 / 51.5070.40 / 57.08
SMART public training recipe, text adaptation69.71 / 55.8969.12 / 54.6271.72 / 58.30
Ours, training-free74.22 / 61.1073.77 / 59.9472.90 / 59.08
Ours, full74.52 / 61.5975.73 / 62.3173.61 / 60.82

Our training-free method exceeds the trained SMART adaptation on both average metrics across all eight evaluated backbones, including the five additional encoders reported in revised Table 12. On the three Qwen backbones above, the full method improves over that comparator by 4.82 / 5.70, 6.62 / 7.69, and 1.89 / 2.52 percentage points, respectively. On Gemma-300M, the training-free method alone improves by 3.93 / 3.45 points, extending the advantage beyond the Qwen family.

With content tokens and standardized fusion matched, our training-free method still exceeds MaxSim by 3.63 / 2.32 points on Qwen-4B, supporting the additional value of our local scoring.


2. Baseline verification and stronger backbones (W2)

We verified the encoding settings against the official Qwen implementation. Re-encoding all 5,873 queries for each of Qwen-0.6B and Qwen-4B reproduced the original pooled query vectors and pooled-retrieval Top-10 rankings.

LMEB v1, Table 22 also reports lower Recall@10 for Qwen-4B than Qwen-0.6B on LoCoMo and LongMemEval, with values close to ours:

ModelLoCoMo: LMEB / oursLongMemEval: LMEB / ours
Qwen-0.6B46.82 / 46.8784.66 / 84.84
Qwen-4B41.28 / 41.6058.74 / 59.17

The improvement is not limited to Qwen-4B: on the stronger Llama-Embed-Nemotron-8B pooled-retrieval baseline, our full method gains 2.56 Recall@10 and 5.22 NDCG@10 points.


We thank you again for the constructive feedback, which has helped clarify our contribution and strengthen its empirical support.

We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.

Best regards,
The Authors

Also I have a additional question. On LongMemEval, QA accuracy improves by 4.80, while F1 decreases. The authors attribute this to reduced lexical precision caused by longer answers. Did the authors also conduct a qualitative evaluation of the generated answers? Could they provide examples demonstrating whether the proposed method produces more accurate and useful responses?

Dear Reviewer KawT,

Thank you for this thoughtful follow-up. Your suggestion prompted us to examine the generated answers more closely. We inspected saved LongMemEval outputs and checked the following examples against the annotated supporting conversations. Both examples use saved outputs from the original experiment.

1. Correcting an incorrect answer

Question: “How many days ago did I read the March 15th issue of The New Yorker?”

The question was asked on April 1. In the March 20 conversation, the user stated: “I finally got around to reading the March 15th issue of The New Yorker today.” The reference answer is 12 days; 13 days is also accepted under inclusive counting.

  • Original retrieval: “16 days ago.”
  • Our full method, excerpts: “You read the March 15th issue of The New Yorker on March 20, 2023 […] So, 12 days ago you read the March 15th issue of The New Yorker.”

Our answer correctly identifies the reading date and calculates the interval. Yet full-answer F1 decreases from 28.57% to 17.91%: the incorrect short answer shares “days ago” with the reference, while the correct answer also provides supporting dates and an explanation. The lower F1 accompanies a verifiable correction of the answer.

2. Resolving a question the baseline could not answer

Question: “How many days passed between the day I cancelled my FarmFresh subscription and the day I did my online grocery shopping from Instacart?”

The annotated supporting conversations record the cancellation discussion on January 5 and the Instacart order on February 28. The reference answer is 54 days, with 55 also accepted under inclusive counting.

  • Original retrieval, excerpt: “Since there is no confirmed date for your Instacart shopping trip, it’s not possible to calculate how many days passed between those two events.”
  • Our full method, excerpt: “Total days = 26 (Jan) + 28 (Feb) = 54 days.”

Our full answer identifies both dated events, quotes the relevant statements, and calculates the correct interval. It resolves the information need with a checkable explanation, although full-answer F1 decreases from 6.74% to 5.44%.

Excerpts are shortened for readability; F1 is calculated on the complete, unchanged outputs.

These examples illustrate concrete improvements in correctness and usefulness despite lower full-answer lexical F1: correcting an incorrect answer and resolving a previously unanswered question, with evidence-supported explanations.

Thank you again for encouraging us to substantiate answer quality with concrete examples. We hope these address your concern and would be grateful if you could take this additional evidence into account in your assessment.

Best regards,
The Authors

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and constructive feedback. We appreciate the thoughtful discussion and the opportunity to strengthen our paper.

We are writing to gently follow up on our rebuttal and revised manuscript. For ease of reference, the main updates are:

  • Reviewer KawT: Our training-free method exceeds trained SMART on both mean retrieval metrics across all eight backbones. On Qwen-4B, training-free and full-method gains over their respective SMART counterparts are 8.03 / 8.44 and 6.62 / 7.69 Recall@10 / NDCG@10 points. Training-free Gemma-300M also exceeds trained SMART by 3.93 / 3.45 points. We additionally verified the Qwen encoding pipeline.

  • Reviewer pAyi: Full Qwen-4B improves answer accuracy by 4.42 points on LoCoMo and 4.80 on LongMemEval. The QA component breakdown is now in Appendix F, Table 21. Compression reduces storage by approximately 91%, and the 100k-record test adds 5.52 ms over pooled retrieval, excluding query encoding (Appendix E, Tables 19–20).

  • Reviewer yLYV: We added paired confidence intervals, broader comparisons, and gain breakdowns. The gains are larger on conversational-memory benchmarks: +25.60 / +25.64 Recall@10 / NDCG@10 points for full Qwen-4B. The responses also address F1 interpretation and token-budget sensitivity.

  • Reviewer 6FHd: With the encoder and fusion strategy matched, our training-free method exceeds the strongest tested contextual-chunk configuration by 15.03 / 14.60 points on Qwen-0.6B and 14.21 / 13.59 on Qwen-4B. We also clarified the memory-retrieval setting and evaluated incremental index updates.

We remain available to answer any remaining questions and would greatly appreciate your feedback on the revised paper and responses.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.