- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes NEAR, a training-free scoring method for embedding-based memory retrieval. For each memory, NEAR computes a reference direction with its nearest memories, projects this direction out, and optionally repeats the process for up to layers. Because queries cannot access their neighborhoods without a preliminary search, NEAR obtains a query reference from a small table. The table has at most 16 spherical k-means anchors, each storing the averaged reference direction of its cluster. Original and residual similarities are fused with per-layer scales, and retrieval needs only one MIPS/HNSW search over a fixed index. Experiments cover eight LMEB tasks, nine frozen encoders, and downstream QA on LoCoMo and LongMemEval.
Critical strengths
- The method is training-free, and the paper does experiments with a lot of encoders.
- The evaluation is broad, testing nine encoders, a supervised adaptation baseline, multi-vector systems, and downstream QA. Development selection uses grouped CV to avoid leakage across banks, and the latency and storage overheads are reported honestly.
- The locality and correspondence ablations (Table 4) are interesting and informative.
Critical weaknesses
W1. Missing ablations and baselines needed to explain where the gains come from
NEAR combines several components: neighborhood-based references, score fusion with the original similarity, multiple residual layers, and per-layer scaling . The ablations in Table 4 vary only the type of reference (global, shuffled, one-sided), so it remains unclear how much of the gain comes from the local references themselves versus the fusion and depth.
For example, LC (Hara et al., 2015) is used in its original single-step form, without NEAR's multi-layer construction or its tuned fusion weight. Giving LC the same depth and fusion would show whether the local-reference construction is what matters.
Two groups of training-free baselines are also missing:
- Single-search-compatible neighborhood methods: NNN (Chowdhury et al., 2024), which the paper cites, and CSLS (Conneau et al., 2018) correct scores using nearest-neighbor statistics and can also be served with a single ANN search over precomputed values. Since single-index retrieval is a central claim, these are the most direct points of comparison.
- Global direction removal: The "Global reference" ablation in Table 4 removes a single global direction, whereas NEAR removes up to three. All-but-the-Top (Mu et al., 2018) with up to three removed directions, and whitening (Su et al., 2021), both combined with the same fusion, would provide a fairer test of whether locality is necessary.
W2. The banks are too small to test the method's main motivations
- From Appendix A, the average bank sizes are about 894 (REALTALK), 588 (LoCoMo), 475 (LongMemEval), 266 (PeerQA), 134 (ESG), 50 (QASPER), 30 (CovidQA), and 15 (MLDR). The efficiency study uses banks of 476 and 629 memories. At this scale, exhaustive search is trivial and HNSW with efSearch=128 is effectively exact, so I'm not sure the single-ANN-search design and the latency comparison in Section 4.6 tell us much here.
- Locality is degenerate on several benchmarks. Because with , the "local" neighborhood covers essentially the whole bank on QASPER, CovidQA, and MLDR, so the reference becomes a leave-one-out global mean. These are precisely the tasks where NEAR shows no gain. This fits the paper's locality hypothesis, but it means three of the eight benchmarks do not really test the method's premise.
- I would like to see NEAR evaluated in larger settings, such as LongMemEval_M (Wu et al., 2025), which has about 500 sessions per question, or standard large-corpus benchmarks such as BEIR (Thakur et al., 2021). This would show whether the gains hold when each query faces thousands of candidates or more, which is closer to realistic memory banks, and whether the single-ANN-search design actually pays off in latency at that scale.
W3. Gains are concentrated in a few tasks
- On QASPER, NEAR is below Base in Recall@10 for six of the nine encoders. CovidQA and MLDR are essentially flat (see W2 for a possible reason). On LongMemEval, one of the two core memory benchmarks, NEAR is not the best same-backbone method at any Qwen scale. NEAR is also slightly below Base on LongMemEval for EmbeddingGemma and Llama-Embed-Nemotron-8B.
References
- Conneau et. al. Word Translation Without Parallel Data.
- Mu et. al. All-but-the-Top: Simple and Effective Postprocessing for Word Representations.
- Su et. al. Whitening Sentence Representations for Better Semantics and Faster Retrieval.
- Thakur et. al. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
Questions for authors
- How does LC perform when it is given the same depth and fusion weight as NEAR? This would help separate the effect of the local-reference construction from that of the fusion and multi-layer design.
- How does NEAR compare with NNN or CSLS in both retrieval quality and latency, given that they are also training-free and single-search compatible?
- How does NEAR compare with All-but-the-Top (up to three removed directions) and whitening under the same score fusion?
- On larger banks such as LongMemEval_M or BEIR, do the gains hold? How does NEAR's latency compare with exhaustive (flat) search at that scale?
- For small banks (), the neighborhood covers almost the whole bank. Have you tried setting relative to (e.g., a fraction of the bank), and would that recover gains on QASPER, CovidQA, or MLDR?
- On LongMemEval, InvGC and Search-Adaptor outperform NEAR at all Qwen scales. Do you have an explanation, e.g., from a per-category breakdown?
4 comments
Dear Reviewer rtUN,
Thank you again for your thoughtful review and helpful suggestions.
We hope our enhanced baseline comparisons, component ablations, and additional analyses of memory-bank sizes and LongMemEval performance have addressed your questions.
We are grateful for your suggestions, which have strengthened the paper. We would greatly appreciate your feedback and would be happy to provide any further clarification.
Best regards,
The Authors
Dear Authors,
Thank you. My concerns are resolved. I have raised my scores.
Dear Reviewer rtUN,
Thank you for your positive reassessment and for raising your score. We are glad that our clarifications and additional results have addressed your questions.
We greatly appreciate your constructive suggestions and encouraging follow-up. We will make sure to include all these updates in the next version of the paper.
Best regards,
The Authors
Thank you for your constructive feedback. Your suggestions have helped us strengthen the comparisons, isolate NEAR's components, and evaluate retrieval at larger index sizes.
The revised manuscript cites the original papers for NNN, CSLS, ABTT, whitening, and the BEIR benchmark. The additional geometric comparisons are reported in Appendix H.
1. Strong comparisons with matched fusion and depth (W1)
We give LC optional depth and fusion, and evaluate ABTT with up to three removed directions and whitening using NEAR's original-score fusion and scaling. Tunable configurations are selected by development NDCG@10; NEAR retains its submitted configurations.
Below are Qwen-4B eight-task mean Recall@10/NDCG@10 (%) under exact search. Best values are bold and second-best values are underlined.
NEAR exceeds iterated LC with fusion by 6.41/6.71 percentage points. Across nine encoders, NEAR exceeds iterated LC, NNN, and CSLS on both task-averaged metrics. Including fused ABTT and whitening, NEAR is the only method ranked in the top two for both metrics on every encoder.
Holding each fused method's transform fixed, adding original matching improves Qwen-4B's eight-task means by 2.35/2.55 points for ABTT and 1.70/1.35 for whitening. These gains show that the complementary-matching design extends beyond a single geometric transformation.
The submitted Table 4 shows 2.16/2.15-point gains over global references. Across four encoders and eight tasks, retaining original matching and scaling improves both task-averaged metrics for every encoder. On Qwen-4B/LoCoMo, development-selected depth improves Recall@10/NDCG@10 from 55.63/40.76 to 56.74/42.53, gains of 1.11/1.77 points over one residual layer with the same total residual weight.
On the same 689-memory Qwen-4B/LoCoMo bank, median latency is 1.278 ms for NEAR, versus 0.212 ms for NNN and 0.226 ms for CSLS. NEAR obtains the above task-averaged quality advantages with approximately 1 ms of additional query cost. Timings use one CPU thread and include query construction and search, excluding encoding.
2. Larger-corpus evaluation and efficient ANN retrieval (W2)
On SciFact's 5,183 documents, Qwen-4B improves Recall@10/NDCG@10 from 91.83/77.52 to 92.00/78.46. The other BEIR tests were mixed: development selection retained Base on NFCorpus, while FiQA showed no consistent gain.
At 100K index keys, Qwen-0.6B with one-layer NEAR achieves 1.801 ms ANN latency, a 27.5× speedup over exhaustive search, with 99.22% Top-10 overlap.
The pooled index includes repeated session bodies and measures system scaling. These results demonstrate that the fused representations can use a fixed ANN index to reduce exhaustive scoring costs at larger index sizes. Construction, storage, and scaling results are reported in revised Appendix G.
3. Relative neighborhood sizes on small banks (W2)
We test neighborhoods containing 10%, 25%, and 50% of the remaining memories across four encoders, retaining the selected depth, residual form, and fusion weights. On CovidQA, the 50% neighborhood improves mean Recall@10/NDCG@10 by 0.25/0.24 points. QASPER remains below the original configuration and MLDR is mixed, so changing neighborhood size alone does not consistently improve these small-bank tasks.
4. Category-level analysis on LongMemEval (W3)
On Qwen-4B, NEAR improves Base NDCG@10 in all six official categories. Multi-session questions improve from 32.83 to 76.62, single-session user information from 27.23 to 74.16, and knowledge updates from 56.75 to 87.04: gains of 43.79, 46.93, and 30.29 percentage points, respectively.
The larger gaps relative to InvGC and Search-Adaptor are concentrated in multi-session retrieval and user information/preferences. Multi-session gaps also occur at the other two Qwen sizes.
We thank the reviewer again for the constructive feedback, which has helped us clarify our contribution and strengthen the empirical evidence.
We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.
Best regards,
The Authors