acceptodds
Under review as a conference paper at ICLR 2027

NEAR: Neighborhood-Aware Residuals for Memory Retrieval

Abstract

Memory retrieval aims to identify evidence from stored records to answer a query. Stored records with different factual details may share topics, entities, or events. Such shared contents would make records without the required evidence more likely to receive high similarity scores and outrank the correct evidence. In this paper, we use the information shared by these records to construct local references for query–memory matching. We propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these references to construct residual representations that complement the original embeddings. Specifically, each memory derives a reference direction from its neighbors, and each query obtains its reference from a precomputed table. Using these references, NEAR constructs projection residuals for both queries and memories. These residuals are combined with the original embeddings for a single approximate nearest-neighbor search over a fixed memory index. Experiments on eight long-context retrieval benchmarks show consistent improvements across nine embedding models. Downstream evaluations show that NEAR improves question-answering performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
memory retrieval, dense retrieval, frozen embeddings, neighborhood-aware representations, approximate nearest-neighbor search

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rtUNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NEAR, a training-free scoring method for embedding-based memory retrieval. For each memory, NEAR computes a reference direction with its k=50k=50 nearest memories, projects this direction out, and optionally repeats the process for up to L=3L=3 layers. Because queries cannot access their neighborhoods without a preliminary search, NEAR obtains a query reference from a small table. The table has at most 16 spherical k-means anchors, each storing the averaged reference direction of its cluster. Original and residual similarities are fused with per-layer scales, and retrieval needs only one MIPS/HNSW search over a fixed index. Experiments cover eight LMEB tasks, nine frozen encoders, and downstream QA on LoCoMo and LongMemEval.

Critical strengths
  • The method is training-free, and the paper does experiments with a lot of encoders.
  • The evaluation is broad, testing nine encoders, a supervised adaptation baseline, multi-vector systems, and downstream QA. Development selection uses grouped CV to avoid leakage across banks, and the latency and storage overheads are reported honestly.
  • The locality and correspondence ablations (Table 4) are interesting and informative.
Critical weaknesses

W1. Missing ablations and baselines needed to explain where the gains come from

NEAR combines several components: neighborhood-based references, score fusion with the original similarity, multiple residual layers, and per-layer scaling τℓ\tau_\ell. The ablations in Table 4 vary only the type of reference (global, shuffled, one-sided), so it remains unclear how much of the gain comes from the local references themselves versus the fusion and depth.

For example, LC (Hara et al., 2015) is used in its original single-step form, without NEAR's multi-layer construction or its tuned fusion weight. Giving LC the same depth and fusion would show whether the local-reference construction is what matters.

Two groups of training-free baselines are also missing:

  • Single-search-compatible neighborhood methods: NNN (Chowdhury et al., 2024), which the paper cites, and CSLS (Conneau et al., 2018) correct scores using nearest-neighbor statistics and can also be served with a single ANN search over precomputed values. Since single-index retrieval is a central claim, these are the most direct points of comparison.
  • Global direction removal: The "Global reference" ablation in Table 4 removes a single global direction, whereas NEAR removes up to three. All-but-the-Top (Mu et al., 2018) with up to three removed directions, and whitening (Su et al., 2021), both combined with the same fusion, would provide a fairer test of whether locality is necessary.

W2. The banks are too small to test the method's main motivations

  • From Appendix A, the average bank sizes are about 894 (REALTALK), 588 (LoCoMo), 475 (LongMemEval), 266 (PeerQA), 134 (ESG), 50 (QASPER), 30 (CovidQA), and 15 (MLDR). The efficiency study uses banks of 476 and 629 memories. At this scale, exhaustive search is trivial and HNSW with efSearch=128 is effectively exact, so I'm not sure the single-ANN-search design and the latency comparison in Section 4.6 tell us much here.
  • Locality is degenerate on several benchmarks. Because ∣Ni∣=min⁡(k,N−1)|\mathcal{N}_i| = \min(k, N-1) with k=50k=50, the "local" neighborhood covers essentially the whole bank on QASPER, CovidQA, and MLDR, so the reference becomes a leave-one-out global mean. These are precisely the tasks where NEAR shows no gain. This fits the paper's locality hypothesis, but it means three of the eight benchmarks do not really test the method's premise.
  • I would like to see NEAR evaluated in larger settings, such as LongMemEval_M (Wu et al., 2025), which has about 500 sessions per question, or standard large-corpus benchmarks such as BEIR (Thakur et al., 2021). This would show whether the gains hold when each query faces thousands of candidates or more, which is closer to realistic memory banks, and whether the single-ANN-search design actually pays off in latency at that scale.

W3. Gains are concentrated in a few tasks

  • On QASPER, NEAR is below Base in Recall@10 for six of the nine encoders. CovidQA and MLDR are essentially flat (see W2 for a possible reason). On LongMemEval, one of the two core memory benchmarks, NEAR is not the best same-backbone method at any Qwen scale. NEAR is also slightly below Base on LongMemEval for EmbeddingGemma and Llama-Embed-Nemotron-8B.

References

- Conneau et. al. Word Translation Without Parallel Data. 
- Mu et. al. All-but-the-Top: Simple and Effective Postprocessing for Word Representations.
- Su et. al. Whitening Sentence Representations for Better Semantics and Faster Retrieval.
- Thakur et. al. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.
Questions for authors
  1. How does LC perform when it is given the same depth and fusion weight as NEAR? This would help separate the effect of the local-reference construction from that of the fusion and multi-layer design.
  2. How does NEAR compare with NNN or CSLS in both retrieval quality and latency, given that they are also training-free and single-search compatible?
  3. How does NEAR compare with All-but-the-Top (up to three removed directions) and whitening under the same score fusion?
  4. On larger banks such as LongMemEval_M or BEIR, do the gains hold? How does NEAR's latency compare with exhaustive (flat) search at that scale?
  5. For small banks (N≤50N \le 50), the neighborhood covers almost the whole bank. Have you tried setting kk relative to NN (e.g., a fraction of the bank), and would that recover gains on QASPER, CovidQA, or MLDR?
  6. On LongMemEval, InvGC and Search-Adaptor outperform NEAR at all Qwen scales. Do you have an explanation, e.g., from a per-category breakdown?
4 comments

Thank you for your constructive feedback. Your suggestions have helped us strengthen the comparisons, isolate NEAR's components, and evaluate retrieval at larger index sizes.

The revised manuscript cites the original papers for NNN, CSLS, ABTT, whitening, and the BEIR benchmark. The additional geometric comparisons are reported in Appendix H.

1. Strong comparisons with matched fusion and depth (W1)

We give LC optional depth and fusion, and evaluate ABTT with up to three removed directions and whitening using NEAR's original-score fusion and scaling. Tunable configurations are selected by development NDCG@10; NEAR retains its submitted configurations.

Below are Qwen-4B eight-task mean Recall@10/NDCG@10 (%) under exact search. Best values are bold and second-best values are underlined.

MethodRecall@10NDCG@10
ABTT (unfused)67.8252.47
ABTT + fusion69.0554.30
Whitening (unfused)67.5054.04
Whitening + fusion69.81‾\underline{69.81}55.56
Iterated LC + fusion63.5548.79
NNN69.6955.23
CSLS67.2352.89
NEAR69.9655.49‾\underline{55.49}

NEAR exceeds iterated LC with fusion by 6.41/6.71 percentage points. Across nine encoders, NEAR exceeds iterated LC, NNN, and CSLS on both task-averaged metrics. Including fused ABTT and whitening, NEAR is the only method ranked in the top two for both metrics on every encoder.

Holding each fused method's transform fixed, adding original matching improves Qwen-4B's eight-task means by 2.35/2.55 points for ABTT and 1.70/1.35 for whitening. These gains show that the complementary-matching design extends beyond a single geometric transformation.

The submitted Table 4 shows 2.16/2.15-point gains over global references. Across four encoders and eight tasks, retaining original matching and scaling improves both task-averaged metrics for every encoder. On Qwen-4B/LoCoMo, development-selected depth improves Recall@10/NDCG@10 from 55.63/40.76 to 56.74/42.53, gains of 1.11/1.77 points over one residual layer with the same total residual weight.

On the same 689-memory Qwen-4B/LoCoMo bank, median latency is 1.278 ms for NEAR, versus 0.212 ms for NNN and 0.226 ms for CSLS. NEAR obtains the above task-averaged quality advantages with approximately 1 ms of additional query cost. Timings use one CPU thread and include query construction and search, excluding encoding.

2. Larger-corpus evaluation and efficient ANN retrieval (W2)

On SciFact's 5,183 documents, Qwen-4B improves Recall@10/NDCG@10 from 91.83/77.52 to 92.00/78.46. The other BEIR tests were mixed: development selection retained Base on NFCorpus, while FiQA showed no consistent gain.

At 100K index keys, Qwen-0.6B with one-layer NEAR achieves 1.801 ms ANN latency, a 27.5× speedup over exhaustive search, with 99.22% Top-10 overlap.

MethodExhaustive search (ms)ANN search (ms)Overlap with its own exact Top-10
Base21.5600.37595.31%
NEAR49.5371.80199.22%

The pooled index includes repeated session bodies and measures system scaling. These results demonstrate that the fused representations can use a fixed ANN index to reduce exhaustive scoring costs at larger index sizes. Construction, storage, and scaling results are reported in revised Appendix G.

3. Relative neighborhood sizes on small banks (W2)

We test neighborhoods containing 10%, 25%, and 50% of the remaining memories across four encoders, retaining the selected depth, residual form, and fusion weights. On CovidQA, the 50% neighborhood improves mean Recall@10/NDCG@10 by 0.25/0.24 points. QASPER remains below the original configuration and MLDR is mixed, so changing neighborhood size alone does not consistently improve these small-bank tasks.

4. Category-level analysis on LongMemEval (W3)

On Qwen-4B, NEAR improves Base NDCG@10 in all six official categories. Multi-session questions improve from 32.83 to 76.62, single-session user information from 27.23 to 74.16, and knowledge updates from 56.75 to 87.04: gains of 43.79, 46.93, and 30.29 percentage points, respectively.

The larger gaps relative to InvGC and Search-Adaptor are concentrated in multi-session retrieval and user information/preferences. Multi-session gaps also occur at the other two Qwen sizes.


We thank the reviewer again for the constructive feedback, which has helped us clarify our contribution and strengthen the empirical evidence.

We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.

Best regards,

The Authors

Dear Reviewer rtUN,

Thank you again for your thoughtful review and helpful suggestions.

We hope our enhanced baseline comparisons, component ablations, and additional analyses of memory-bank sizes and LongMemEval performance have addressed your questions.

We are grateful for your suggestions, which have strengthened the paper. We would greatly appreciate your feedback and would be happy to provide any further clarification.

Best regards,

The Authors

Dear Authors,

Thank you. My concerns are resolved. I have raised my scores.

Dear Reviewer rtUN,

Thank you for your positive reassessment and for raising your score. We are glad that our clarifications and additional results have addressed your questions.

We greatly appreciate your constructive suggestions and encouraging follow-up. We will make sure to include all these updates in the next version of the paper.

Best regards,

The Authors

Reviewer_59Q2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NEAR (Neighborhood-Aware Residuals), a training-free framework for memory retrieval. The method constructs local reference directions from neighboring memory embeddings and removes their projected components to obtain residual representations. A precomputed query-reference table enables query-side residual construction, while combining original and residual embeddings allows retrieval through a single ANN search. Experiments across eight benchmarks and nine embedding models demonstrate improvements in retrieval performance and downstream question answering.

Critical strengths

1.The paper addresses an important problem in memory retrieval, where semantically similar but factually different memories may interfere with retrieving relevant evidence.

2.The proposed method is training-free and integrates original and residual representations into a single ANN search, providing a practical retrieval framework.

3.Extensive experiments across eight benchmarks and nine embedding models demonstrate improvements in retrieval performance and downstream question-answering accuracy.

Critical weaknesses

1.The core hypothesis of this paper is that the mean direction of nearest neighbors can approximate shared semantic content, and that projection residuals help highlight discriminative factual information. Although the experiments demonstrate the usefulness of local references, they do not sufficiently establish that the improvements specifically result from removing irrelevant shared semantic content. The paper lacks controlled hard-negative retrieval experiments that account for entity overlap, event overlap, and factual differences, as well as analyses of how the scores of relevant evidence and distractors change before and after projection.

2.The authors construct anchors from memory embeddings and predict query reference directions based on the similarity between queries and these anchors. However, pretrained retrieval models often use different input formats for queries and documents, and their embedding spaces may not share identical local geometric distributions. The paper lacks an analysis of how closely the predicted query references approximate those constructed directly from the actual local neighborhoods. Furthermore, it remains unclear whether incorrect query-to-anchor assignments could cause the projection to remove important query information.

3.The novelty of the proposed method is not sufficiently distinguished from existing works, such as Representation Sharpening [a] and Local Residual Similarity [b]. The authors should further clarify the methodological differences and unique contributions of NEAR compared with these related approaches.

[a] A Representation Sharpening Framework for Zero Shot Dense Retrieval.

[b] Local Residual Similarity for Image Re-ranking.

4.The ablation studies are insufficient. For example, it remains unclear whether the original embedding similarity scores are indispensable, whether the proposed local references outperform simpler geometric calibration approaches, and whether the multi-layer recursive residual construction is truly necessary.

5.The results of Qwen3-Embedding-4B on LongMemEval are notably unusual. Its Base performance is substantially lower than that of the 0.6B and 8B models, while methods such as InvGC, DBSN, and Search-Adaptor improve Recall@10 to above 90%. The authors should investigate and explain this significant discrepancy.

Questions for authors

Please refer to the weaknesses.

3 comments

Thank you for your constructive feedback. Your suggestions have helped us clarify NEAR's contribution and strengthen the evidence for its design.

1. Neighborhood-derived complementary matching (W3)

We cite both recommended works, LRS and Representation Sharpening, and clarify their relationship to NEAR in the revised Related Work section.

NEAR turns neighborhood information into complementary matching signals that are combined with the original embedding score. We realize this design through simple and effective memory-specific projection residuals, constructed layer by layer and coupled with precomputed query-reference tables. The original and residual contributions are served through a single fixed index with one ANN search.

Relative to LRS, the reference construction and retrieval stage differ. LRS constructs anchors from an initially retrieved query neighborhood, forms residuals by subtraction, and reranks candidates. NEAR uses offline memory-specific projection residuals and a query-reference table to apply complementary matching before candidate generation. LRS

Relative to Representation Sharpening, the source of the additional representation differs. That method generates and encodes discriminative queries to augment documents; its IndexSharp variant also works offline. NEAR constructs query- and memory-side projection residuals directly from existing memory embeddings, without generating or encoding additional text. Representation Sharpening

This matching design has measurable retrieval value. In a fixed-configuration exact-ranking diagnostic on Qwen-4B/LoCoMo, Recall@10 rises from 41.70% to 56.70%, a gain of 15.00 percentage points. Among 1,976 queries, 136 have no relevant record in Base's entire Top-100, yet NEAR retrieves relevant evidence in its Top-10. A reranker restricted to those Base Top-100 lists cannot recover the missing records.

2. Improved candidate discrimination through complementary matching (W1)

Shared content motivates reference construction; the scoring role is to provide complementary matching. In the submitted Table 4, local rather than global-mean references improve eight-task mean Recall@10/NDCG@10 by 2.16/2.15 percentage points across four encoders.

Natural-candidate comparisons show improved discrimination. On LoCoMo, we pair labeled relevant evidence with a candidate not labeled relevant from Base's exact Top-10, requiring an absolute original cosine-score difference of at most 0.025. Pair selection does not use NEAR outcomes. The table reports the fraction of pairs in which relevant evidence ranks higher.

EncoderPairsBaseNEARGain (pp)
Qwen-0.6B1,17842.87%70.46%+27.59
Qwen-4B1,02142.90%72.77%+29.87
EmbeddingGemma1,30249.62%68.13%+18.51
ModernBERT1,29345.94%68.83%+22.89

In Figure 1, full NEAR moves the relevant evidence from Base's rank 75 to rank 1 among 680 memories. The actual fused score explains this change: margins below are relevant evidence minus Base's top-ranked distractor, including the scales and fusion weights used by NEAR. The residual contribution sums all selected layers.

Score componentRelevant evidence minus distractor
Original-matching contribution−1.5553
Residual-matching contribution+2.5627
Combined margin+1.0073

The residual contribution offsets the disadvantage in original matching, allowing the relevant evidence to overtake Base's top-ranked distractor.

Together, these analyses support local references as a source of effective complementary matching. A separate fixed-parameter, single-condition paired-rewrite diagnostic showed no aggregate gain.

3. Effective query references without preliminary retrieval (W2)

On LoCoMo, the mean cosine similarity between first-layer table-derived references and direct query-neighborhood references is 0.969–0.982 across four encoders. Full NEAR outperforms the mean of three controls with shuffled first-layer reference directions on both metrics across all four encoders, with Qwen-4B gains of 1.44/1.69 percentage points in Recall@10/NDCG@10. These results support obtaining useful neighborhood references without preliminary retrieval. The original query embedding remains an independent component of the final score.

4. Consistent top-two performance with enhanced geometric controls (W4)

The revised manuscript cites the original papers for LC, NNN, CSLS, ABTT, and whitening, with the additional comparisons reported in Appendix H.

We add the original-score fusion and scaling used in NEAR to ABTT (which removes dominant global directions) and whitening (which adjusts the global representation distribution), and give LC optional depth and fusion. Tunable configurations are selected by NDCG@10 on the original development sets, while NEAR retains its submitted configurations.

The table reports eight-task mean Recall@10/NDCG@10 (%) under exact search. For each encoder, the best value for each metric is bold and the second-best is underlined.

EncoderABTT + fusionWhitening + fusionIterated LC + fusionNNNCSLSNEAR
Qwen-0.6B67.81/53.3267.17/53.0166.29/51.8565.59/51.3765.66/51.6267.21‾\underline{67.21}/53.03‾\underline{53.03}
Qwen-4B69.05/54.3069.81‾\underline{69.81}/55.5663.55/48.7969.69/55.2367.23/52.8969.96/55.49‾\underline{55.49}
Qwen-8B71.16/56.1972.12/57.4269.46/54.4470.34/55.6669.95/55.2771.88‾\underline{71.88}/57.10‾\underline{57.10}
EmbeddingGemma76.14/61.10‾\underline{61.10}75.13/60.5773.62/59.5572.62/58.9472.54/58.5776.12‾\underline{76.12}/61.33
KaLM-12B74.87‾\underline{74.87}/59.1574.03/59.21‾\underline{59.21}73.37/57.4973.42/58.5573.12/57.9575.25/59.24
ModernBERT71.38/57.1770.17/57.0769.92/55.9069.71/55.6368.86/54.9770.95‾\underline{70.95}/57.14‾\underline{57.14}
Llama-1B75.47/61.62‾\underline{61.62}74.12/61.4774.62/61.0574.14/61.0873.92/60.3475.33‾\underline{75.33}/61.73
Llama-3B77.82‾\underline{77.82}/61.3077.09/61.89‾\underline{61.89}77.46/60.6376.30/60.2975.71/58.9478.81/62.42
Llama-8B77.80‾\underline{77.80}/62.9977.43/64.0476.94/62.5076.74/62.6376.05/62.2278.03/63.86‾\underline{63.86}

NEAR exceeds iterated LC with fusion, NNN, and CSLS on both task-averaged metrics for all nine encoders, and is the only method in this table ranked in the top two for both metrics on every encoder.

Fusion also benefits global geometric transforms. For Qwen-4B, the eight-task mean Recall@10/NDCG@10 (%) under exact search is:

MethodUnfusedOriginal + transformed matching
ABTT67.82/52.4769.05/54.30
Whitening67.50/54.0469.81/55.56

In a separate ablation with each fused method's selected transform fixed, restoring original matching improves Qwen-4B's eight-task mean Recall@10/NDCG@10 by 2.35/2.55 percentage points for ABTT and 1.70/1.35 for whitening. The gains from fusion in both ABTT and whitening show that the complementary-matching design extends beyond a single geometric transformation.

Across four encoders and all eight tasks:

  • Original matching complements residual matching. Retaining it improves both task-averaged metrics for every encoder, with average Recall@10/NDCG@10 gains of 0.63/0.90 percentage points over residual-only matching.
  • Additional depth provides an incremental benefit. With total residual weight fixed, development-selected depth improves mean Recall@10/NDCG@10 by 0.15/0.35 points over one layer, improving NDCG for all four encoders and Recall for three.

5. Benchmark-consistent baselines and improved evidence retrieval (W5)

Our LongMemEval Base scores closely match those in the original LMEB paper. The table directly compares LMEB v1's results with instructions against the Base results in our submitted appendix, reporting Recall@10/NDCG@10 (%). LMEB v1, Tables 20 and 22

EncoderLMEB v1Base in our submitted paper
Qwen3-Embedding-0.6B84.66/72.7784.84/72.82
Qwen3-Embedding-4B58.74/47.0658.96/47.11
Qwen3-Embedding-8B87.94/77.5987.93/77.40

For all three models, both metrics differ by at most 0.22 percentage points, and both reports place 4B below 0.6B and 8B.

In the fixed-configuration LongMemEval diagnostic, Qwen-4B Base Recall@10/@100 is 59.19%/91.57%, showing that many relevant records occur below the first ten positions. Among 155 queries with no relevant record in Base's Top-10, NEAR retrieves relevant evidence in its Top-10 for 123, demonstrating improved evidence ranking without changing the encoder.


We thank the reviewer again for the constructive feedback, which has helped us clarify our contribution and strengthen the empirical evidence.

We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.

Best regards,

The Authors

Dear Reviewer 59Q2,

Thank you again for your detailed review and constructive feedback.

We hope our clarifications of NEAR’s contribution and local-reference design, together with the additional comparisons, component analyses, and encoding verification, have addressed your questions.

We are grateful for your suggestions, which have helped strengthen the paper. We would greatly appreciate your feedback and welcome any remaining questions.

Best regards,

The Authors

Reviewer_er2hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NEAR, a training-free method for memory retrieval. It uses nearby memory records to estimate shared information and removes the corresponding component from each embedding. A small reference table allows queries to undergo a similar process. The method combines the original embeddings and residual representations in a fixed index, allowing retrieval through one ANN search.

Critical strengths
  1. The paper is well organized and easy to follow.
  2. The problem is demanded and important.
Critical weaknesses
  1. The evaluation is not solid enough: (1) The major retrieval quality experiment cannot explain whether the method generally works on different embedding families, as they only use Qwen embedding. As a method of embedding tuning, testing different embedding families is necessary. Although Table 2 shows different embeddings, the baseline is only Base, this cannot eliminate the possibility that other baselines outperform the method on other embeddings. (2) The latency-quality results cannot prove NEAR outperforms Base, as using 2x time to get 15% quality gain is unclear whether worthy or not, especially in many cases people care more about retrieval latency as long as the quality is acceptable. And without showing the other baselines, it's hard to assess whether such a performance is the best. (3) As a retrieval method, evaluation on diverse top-K is important. The paper currently only evaluates K from 4 to 16, which may not be significantly diverse. In usual retrieval evaluation K could be up to 100 or 1000.
    (4) Also there is no evaluation on different retrieval space sizes (memory bank sizes), which is critical to retrieval performance.
Questions for authors
  1. Could you include other baselines in Table 2? Or add other embedding families in table 1.
  2. Can you evaluate K until 100?
  3. Can you include other baselines in Table 5?
  4. How do index construction time, storage, query latency, and ANN search accuracy change as the bank grows? Measurements on larger banks would help assess whether the current cost trade-off remains practical.
4 comments

Thank you for your constructive feedback. Your suggestions have helped us strengthen the empirical evaluation and clarify NEAR’s practical value.

1. Consistent gains across embedding families (W1, Q1)

NEAR achieves the highest eight-task mean Recall@10 and NDCG@10 for every encoder among the six methods in the submitted appendix's Table A1. This comparison covers Base, LC, InvGC, DBSN, Search-Adaptor, and NEAR across nine encoders, including six non-Qwen encoders. Against the strongest competing method for each metric, NEAR gains 2.29/1.93 percentage points in Recall@10/NDCG@10 on EmbeddingGemma and 1.40 points in NDCG@10 on Llama-NV-Embed-Reasoning-3B.

For Qwen-4B, NEAR improves eight-task mean Recall@10 and NDCG@10 over Base by 7.10 and 7.45 percentage points, respectively. The revised main text now explicitly points to Appendix Table A1.

2. Retrieval gains and effective storage compression (W2, Q3)

On Qwen-4B/LoCoMo, NEAR exceeds DBSN, the strongest competing method on both quality metrics below, by 4.16/3.64 percentage points, with 1.466 ms additional retrieval latency.

MethodRecall@10 / NDCG@10 (%)Storage (MiB)Retrieval latency (ms)
Base41.39 / 29.696.3070.396
LC45.12 / 32.236.3090.281
InvGC49.34 / 35.886.3070.319
DBSN51.93 / 38.246.3090.351
Search-Adaptor50.67 / 37.8111.3190.679
NEAR56.09 / 41.8825.6761.817
Base (SQ4)41.38 / 29.620.9510.358
NEAR (SQ4)56.38 / 41.984.2581.569

SQ4 reduces NEAR's storage by 83.4% without reducing either quality metric in this evaluation. Comparisons for both LoCoMo and LongMemEval and measurement settings are provided in revised Appendix G (Table A10).

3. Sustained gains through Top-100 on LoCoMo (W3, Q2)

All four encoders improve both metrics at K=20, 50, and 100 on LoCoMo. These additional LMEB evaluations retain the original development-selected configurations and extend the K=4–16 analysis in submitted Figure 4. Cells below show NEAR minus Base in Recall@K / NDCG@K, in percentage points.

EncoderK=20K=50K=100
Qwen-0.6B+9.14 / +7.49+9.28 / +7.45+6.40 / +6.97
Qwen-4B+14.86 / +12.27+14.21 / +12.11+13.54 / +12.01
EmbeddingGemma+4.21 / +4.24+1.55 / +3.69+1.20 / +3.60
ModernBERT+5.75 / +5.17+4.12 / +4.81+3.90 / +4.80

These results are included in revised Appendix E, Table A6.

4. Efficient ANN retrieval at 100K index keys (W4, Q4)

At 100K index keys, NEAR achieves 1.801 ms ANN retrieval latency, a 27.5× speedup over exhaustive search, with 99.22% Top-10 overlap. Results are reported for Qwen-0.6B with one-layer NEAR. Pooled keys include repeated session bodies, so this evaluates system scaling.

Index keysBuild (s)Storage (MiB)Exhaustive search (ms)ANN search (ms)Speedup over exhaustiveOverlap with exact NEAR Top-10
1K0.608.200.4140.4650.89×100.00%
10K12.4180.842.9511.1902.48×99.84%
50K101.48403.7322.3961.61613.86×100.00%
100K246.85807.3249.5371.80127.5×99.22%

These results are included in revised Appendix G, Table A11.


We thank the reviewer again for the constructive feedback, which has helped us clarify our contribution and strengthen the empirical evidence.

We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment in light of the additional evidence.

Best regards,

The Authors

Dear Reviewer er2h,

Thank you again for your careful review and helpful suggestions.

We hope our broader baseline comparisons, quality–cost analysis, and evaluations at larger retrieval cutoffs and memory-bank sizes have addressed your questions.

We are grateful for your suggestions, which have strengthened the paper. We would greatly appreciate your feedback and would be happy to address any remaining questions.

Best regards,

The Authors

Thanks for the authors' reply. The added evaluation has same issues to some of my original concern: (1) Using 4x more time to get a 5% quality gain (using DBSN as example), making it unclear how practical it is. (2) The new experiments 3 and 4 in the reply are all comparing to Base instead of stronger baselines like DBSN.

Dear Reviewer er2h,

Thank you for the follow-up. Below, we directly address your further questions:

1. Improved answer accuracy with low additional retrieval cost

We agree that the relative ratio is a useful metric. But we would like to point out that the practical cost also depends on the absolute latency and its share of the total processing time.

Retrieval takes 0.35 ms for DBSN and 1.82 ms for NEAR , measured with a single CPU thread and including query preparation and search. NEAR requires no additional encoding or LLM pass. Across 1,540 questions per method, median answer-generation API time was 2.013 s for DBSN and 2.008 s for NEAR. So in a practical LLM memory-retrieval pipeline, where query encoding and answer generation are unchanged and dominate end-to-end latency, the additional 1.47 ms is a small fraction(0.07%) of the observed median answer-generation API times.

More importantly, the retrieval gain translates directly into better answers. Using the exact same answering model and Top-10 budget on Qwen-4B/LoCoMo, downstream QA accuracy rises from 65.1% (DBSN) to 72.3% (NEAR), i.e., +7.2 points or 111 additional correct answers out of 1,540 questions. We believe this 1.47 ms for +7.2 points of QA accuracy is a favorable trade-off in practice.

2. Direct Top-K comparison against DBSN (Exp. 3)

On LoCoMo, NEAR consistently outperforms DBSN (using its dev-selected configurations) across all four encoders through Top-100 (cells show NEAR minus DBSN in Recall@K / NDCG@K, percentage points):

EncoderK=20K=50K=100
Qwen-0.6B+9.35 / +7.13+8.84 / +6.96+5.65 / +6.41
Qwen-4B+3.85 / +3.55+3.40 / +3.38+3.84 / +3.44
EmbeddingGemma+4.21 / +4.24+1.55 / +3.69+1.20 / +3.60
ModernBERT+5.75 / +5.17+4.12 / +4.81+3.90 / +4.80

3. Direct 100K scaling comparison against DBSN (Exp. 4)

At 100K pooled keys (Qwen-0.6B, 1 CPU thread, including query preparation and ANN search), NEAR's ANN results stay closer to its exact ranking, and DBSN requires comparable latency to reach similar fidelity:

MethodSearch breadthRetrieval latency (ms)Overlap with own exact Top-10
DBSN1280.45792.34%
DBSN10242.47998.75%
NEAR1282.34599.22%

At the same search breadth (128), DBSN retains 92.34% of its exact Top-10, compared with 99.22% for NEAR. Reaching similar fidelity requires widening DBSN's search to 1024, where its latency (2.479 ms) is comparable to NEAR's (2.345 ms).

We hope these direct comparisons resolve your remaining concerns and would appreciate your reconsideration.

Best regards,

The Authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your helpful feedback, which has helped strengthen our evaluation and clarify NEAR’s contribution.

We would like to gently follow up on our rebuttal and revised manuscript. The key updates are:

  • Reviewer er2h: NEAR leads both eight-task mean retrieval metrics across all nine encoders in the submitted six-method comparison. On Qwen-4B/LoCoMo, it exceeds DBSN, the strongest quality baseline in the cost comparison, by 4.16 / 3.64 Recall@10 / NDCG@10 points. Gains persist through Top-100 for all four tested encoders on LoCoMo, with Qwen-4B gains of 13.54 / 12.01 points at Top-100. SQ4 reduces NEAR’s storage by 83.4% without reducing either quality metric in this evaluation.

  • Reviewer 59Q2: NEAR improves the ranking of relevant evidence among candidates with similar original scores. On Qwen-4B/LoCoMo, pairwise ordering accuracy increases from 42.90% to 72.77% across 1,021 natural candidate pairs. In the exact-ranking diagnostic, Recall@10 improves by 15.00 percentage points. NEAR also retrieves relevant evidence in its Top-10 for 136 queries where Base’s entire Top-100 contains none, demonstrating retrieval beyond the coverage of those candidate lists. We additionally verified the Qwen-4B encoding pipeline.

  • Reviewer rtUN: NEAR remains the only method ranked in the top two on both task-averaged metrics across all nine encoders in the enhanced-control comparison. On Qwen-4B, it exceeds iterated LC with fusion by 6.41 / 6.71 Recall@10 / NDCG@10 points. NEAR’s matching design extends beyond a single geometric transformation, improving ABTT by 2.35 / 2.55 and whitening by 1.70 / 1.35 Recall@10 / NDCG@10 points through score fusion (Qwen-4B, eight-task means, fixed transforms).

We remain available to answer any remaining questions and would greatly appreciate your feedback on whether the revised paper and responses address your concerns.

Best regards,

The Authors

Dear Reviewers and Area Chair,

Thank you again for your time and detailed feedback.

As the discussion period draws to a close, we hope our responses and revisions have addressed your questions on:

  1. NEAR’s contribution and reference design.
  2. Baseline comparisons and component ablations.
  3. Top-K evaluation, retrieval costs, and scalability.
  4. Encoding verification and task-level analysis.

These updates have strengthened the evaluation and clarified NEAR’s contribution. We would greatly appreciate your feedback and would be happy to provide any further clarification.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.