- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a more effective way for AI assistants to retrieve relevant information from past conversations and documents. Instead of relying solely on a single embedding vector for each memory, the method leverages existing token-level representations to identify fine-grained relevant evidence, combining sparse token matching with whole-memory similarity. It works without additional training, while an optional lightweight query adapter further improves performance without modifying the original embedding model or stored memories.
Critical strengths
-
The paper is well written and clearly organized. The motivation, methodology, and experimental setup are easy to follow, with helpful figures that illustrate the proposed approach.
-
The proposed method is simple and practical. It effectively reuses existing token representations from frozen embedding models and achieves substantial retrieval improvements without requiring additional training.
-
The paper provides extensive experiments across eight benchmarks and eight embedding backbones. The ablation studies, downstream question-answering evaluations, and efficiency analyses further demonstrate the effectiveness of the proposed method.
Critical weaknesses
1. Limited Technical Novelty.
Token-level retrieval has already been extensively studied in prior works such as ColBERT, XTR, and SMART. The proposed method mainly combines existing techniques, including sparse token matching, global-local score fusion, and lightweight query adaptation, without introducing a fundamentally new retrieval mechanism. Although the empirical improvements are clear, the paper does not sufficiently distinguish its technical contributions from closely related methods, particularly SMART.
2. Unclear Motivation for Memory-specific Retrieval.
Although the paper focuses on long-term memory retrieval, the underlying problem of information loss in single-vector embeddings is a general retrieval challenge rather than a memory-specific one. The proposed method does not explicitly consider unique properties of long-term memories, such as temporal relationships, memory consolidation, or evolving user information. Moreover, five of the eight benchmarks are conventional document retrieval tasks, and the main experiments largely resemble standard RAG over fixed text records. While Appendix F mentions memory construction following EvoEmbedding, the effectiveness on compressed or consolidated memories is not separately examined. It remains unclear why the method is specifically positioned as a memory retrieval approach rather than a general retrieval method.
3. Significant Overhead and Insufficient Scalability Evaluation.
The method introduces substantial storage and retrieval overhead. According to Table 13, even with SQ4 compression, index sizes remain approximately 10 to 12 times larger than the single-vector baseline, although stronger compression can reduce this overhead at some performance cost. More importantly, the latency evaluation uses only a 568-candidate memory bank and eight fixed queries. Although the reported latency includes query processing, the additional runtime cost of the query adapter is not separately analyzed. As the memory bank grows, the larger token index and the need to compute global scores for all candidates may significantly affect retrieval efficiency. Without evaluations on larger memory banks, index construction, and incremental updates, the practical scalability of the method remains unclear.
4. Incomplete Comparison with Lower-cost Fine-grained Retrieval.
The paper attributes its improvements to recovering fine-grained evidence through token-level matching. However, a simpler alternative would be to combine whole-memory embeddings with sentence-level or chunk-level embeddings. Although the authors evaluate Chunk-MaxP, smaller chunk sizes, and several pooling variants, they do not clearly establish a comparable baseline that fuses global embeddings with sentence-level or chunk-level retrieval scores. Such a comparison is important because the improvements may primarily come from combining global and local information rather than token-level matching itself. Without this baseline, it remains unclear whether token-level indexing is necessary, especially given its substantially higher storage and retrieval costs.
Questions for authors
- What are the key technical differences between the proposed method and existing token-level retrieval approaches, particularly SMART and XTR?
- Why is the method specifically positioned for memory retrieval rather than general document retrieval? Does the method remain effective when memories are explicitly summarized or consolidated?
- How does retrieval latency scale as the memory bank grows to tens or hundreds of thousands of records?
- Could the authors compare against a baseline that combines global embeddings with sentence-level or contextual chunk-level embeddings using the same fusion strategy?
5 comments
Dear Reviewer 6FHd,
Thank you again for your constructive review and helpful suggestions.
We hope our clarifications on the contribution and memory-retrieval setting, together with the additional chunk comparisons and scalability experiments, have addressed your questions.
We are very grateful for your suggestions, which have strengthened the paper. We would greatly appreciate your feedback and would be happy to address any remaining questions.
Best regards,
The Authors
Please respond to the authors' rebuttal
Hello,
The discussion period ends soon. Please respond to the authors' last comments and note whether they changed your evaluation.
Thanks,
Your AC
Dear Reviewer 6FHd,
Thank you for your constructive feedback. Retrieval scores below are Recall@10 / NDCG@10.
1. Relation to SMART/XTR and direct comparison (W1; Q1)
SMART is concurrent work using frozen token states and pooled-score fusion. It aggregates per-record MaxSim; we retrieve matches across memories, assign a retrieval-derived fallback per query token, and fuse standardized local and pooled scores. Whereas XTR specifically trains token-retrieval representations, we reuse a frozen single-vector encoder and retain its pooled scores.
We transferred SMART's public scoring and shared-head training recipe to text retrieval, retaining additive fusion and using the same source training data. Results are eight-benchmark means; trained results average three seeds.
Our training-free method exceeds trained SMART on both mean metrics across all eight backbones (revised Table 12). Full Qwen-4B gains 6.62/7.69 points; training-free Gemma gains 3.93/3.45. With content tokens and fusion matched, training-free Qwen-4B still exceeds MaxSim by 3.63/2.32 points, supporting the additional value of our local scoring.
2. Sentence/chunk comparisons: gains beyond global–local fusion (W4; Q4)
We fixed the encoder, candidate bank, pooled scores, and fusion. Sentences were independently encoded; contextual chunks used the same full-memory token states. We tested 32/64/128-token windows with mean and last-token pooling. Results on 1,976 LoCoMo queries include the strongest tested chunk configuration:
Against the strongest tested contextual-chunk configuration, our training-free method gains 15.03/14.60 points on Qwen-0.6B and 14.21/13.59 on Qwen-4B. This supports our fine-grained matching and scoring beyond simply combining global and local similarity (Appendix D, Table 13).
3. Memory-retrieval setting and answer quality (W2; Q2)
We study how assistants retrieve evidence from accumulated interaction histories to answer subsequent questions, following the conversational-memory setting of LoCoMo and LongMemEval. We rank original memory records in a fixed candidate bank using token-level evidence (Section 3.1), evaluating retrieval and answer correctness. Three conversational benchmarks test this use case; five document benchmarks assess transfer.
Across LoCoMo, LongMemEval, and REALTALK, full Qwen-4B improves average retrieval scores by 25.60/25.64 points. With the answering model and Top-10 budget fixed, QA accuracy improves by 4.42 points on LoCoMo and 4.80 on LongMemEval. These results concern the evaluated memory records; explicit summarization or consolidation was not separately tested.
4. Compression, scaling, and index updates (W3; Q3)
RP256+SQ4 reduces index and auxiliary storage by approximately 91%. LoCoMo results:
At similar storage to ColBERTv2's FP32 configuration, our compressed method gains 2.57/1.46 points. Ours uses Qwen-0.6B with three-seed means. Quality covers 1,976 queries; storage uses the same 568-record bank, excluding encoder weights and raw text.
We tested 1k–100k distinct LongMemEval records using Qwen-0.6B with RP128+SQ4 on identical hardware. Median latencies include adaptation, token search, pooled scoring of all candidates, fusion, and ranking, but exclude query encoding:
At 100k records, storage totals 6.55 GiB, and the adapter itself takes 0.30 ms. These tests measure scaling costs; LoCoMo evaluates compression quality.
We tested incremental growth from 1k to 10k and 100k records. With the encoder, projection, and quantizer fixed, new records can be appended without re-encoding existing records or rebuilding the index. Updating the query adapter also leaves the memory index unchanged.
We thank you again for the constructive feedback, which has helped clarify our contribution and strengthen the empirical evidence.
We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.
Best regards, The Authors