acceptodds
Withdrawn from ICLR 2027

REMEDI: Lightweight Editable Memory Adapters for Frozen Dual Encoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dual-encoder retrievers often fail on short queries that name concepts for which the query encoder lacks parametric knowledge. Prior work typically addresses this problem by fine-tuning on in-domain data, but doing so can degrade zero-shot retrieval on other tasks. We introduce REMEDI (Retrieval Enrichment via Memory-Editable Dictionary Interfaces), which keeps the retriever frozen and instead augments queries with an external, user-editable concept memory: a dictionary mapping each concept to a small set of textual descriptions. REMEDI encodes these descriptions once using the frozen document encoder and trains only a lightweight cross-attention adapter, leaving the pretrained encoder parameters, document embeddings, and retrieval index unchanged. At test time, REMEDI matches query spans to concepts, attends over the corresponding descriptions to construct a memory token, and inserts this token into the query between learned boundary tokens. Across eight benchmarks and five encoders, REMEDI achieves the best average nDCG@10 on every encoder, outperforming full fine-tuning and LoRA, with the largest gains on concept-heavy queries. It also generalizes to concepts unseen during training and stays within 0.64 points of the frozen retriever on out-of-domain BEIR tasks, where fine-tuning loses up to 5.2. Theoretically, we characterize when relevant memory lifts a query's positive above its hard negatives, and show when minimizing the contrastive loss guarantees correct ranking, first for linear encoders and then for Transformers.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dual-encoder, Representation Learning, Dense Retrieval, Memory based Adapters, Concept Dictionaries

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_E3J9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes REMEDI, a method for improving frozen dense retrievers on queries that mention concepts the encoder has little knowledge about. REMEDI keeps the encoder and document index fixed and introduces an external, editable concept dictionary that maps each concept to a few textual descriptions. A lightweight cross-attention adapter compresses the descriptions of matched concepts into a memory token, which is inserted into the query before re-encoding. The authors provide a theoretical analysis of when such memory can correct the ranking, first for linear encoders and then for Transformers. Experiments on eight datasets and five encoders show improvements over fine-tuning baselines, with small out-of-domain degradation on BEIR.

Critical strengths

Practical, deployment-friendly design. The method keeps the encoder and document index frozen, trains only a lightweight adapter, and stores knowledge in an external, editable concept dictionary. Knowledge can therefore be updated without retraining. Because queries without matched concepts fall back to the original retrieval path, the out-of-domain regression is much smaller than with fine-tuning (Table 3).

Critical weaknesses
  • Limited novelty relative to knowledge-based query expansion. At its core, the method links query spans to concepts in an external knowledge base and enriches the query with their descriptions. This is a long-standing idea in IR [1, 2], and recent work such as KPR [3] already injects entity knowledge into dense retrievers. The paper does not engage with the classical query expansion literature. Its only comparison against text-based expansion driven by the same memory, REMEDI (Text) in Appendix E.4, concatenates and truncates all memory items, which makes it a weak baseline. Stronger text-based alternatives driven by the same concept memory, such as Query2doc- or HyDE-style expansion [4, 5], are not evaluated (the HyDE baseline in Table 1 generates from the query alone, from my understanding). As a result, it remains unclear what is gained beyond injecting the expansion as a learned soft token rather than as text.

  • Empirical claims are not convincingly supported. The most striking gains come from MedQuAD, where the memory includes the test queries’ gold answers (Appendix D.2). The authors’ own controls show that, for RoBERTa, much of this gain disappears once these answers are removed, although Qwen3’s gain is largely retained (Table 14). For strong encoders, REMEDI underperforms the frozen model on all open-domain datasets in Table 1, while Table 2 shows mixed results. Relatedly, frozen baselines differ substantially across tables (e.g., 29.02, 56.5, and 60.04 for Qwen3 on NQ in Tables 1, 2, and 4(b)), suggesting unexplained differences in evaluation protocols. Moreover, the out-of-domain preservation on BEIR follows largely by construction, since most queries bypass the memory (Section 7.3). Taken together, these issues make it difficult to assess the method’s actual effectiveness.

[1] Entity Query Feature Expansion Using Knowledge Base Links. SIGIR 2014.

[2] Query Expansion with Freebase. ICTIR 2015.

[3] Dynamic Injection of Entity Knowledge into Dense Retrievers. Findings of EMNLP 2025.

[4] Query2doc: Query Expansion with Large Language Models. EMNLP 2023.

[5] Precise Zero-Shot Dense Retrieval without Relevance Labels. ACL 2023.

Questions for authors
  • Why do the frozen baselines for the same dataset and encoder differ substantially across tables? For example, Qwen3 on NQ scores 29.02 in Table 1, 56.5 in Table 2, and 60.04 in Table 4(b). Please clarify the evaluation protocol behind each table, and state which results should be taken as the main comparison.

  • How does REMEDI compare with stronger text-based expansion methods that use the same concept memory, such as appending only the most relevant description or using Query2doc/HyDE-style rewriting conditioned on the memory? Such a comparison would clarify whether the gains come from the learned soft-token interface or simply from access to the external knowledge.

Reviewer_u3LAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

REMEDI adds an editable concept memory to a frozen dense retriever. A small attention adapter turns concept descriptions into memory tokens that enrich the query, while keeping both encoders and the document index unchanged. The paper evaluates whether this improves retrieval while preserving performance outside the training domain.

Critical strengths
  1. Keeping the encoders and document index fixed makes the approach useful for deployed retrieval systems. Users can add concepts through memory without retraining, and queries without a concept match retain their original behavior.
  2. The experiments cover eight datasets and five encoders, with useful memory controls. REMEDI achieves good performance.
Critical weaknesses
  1. The largest MedQuAD gain depends heavily on memory containing test queries’ exact gold answers. The authors provide removal controls, but RoBERTa drops from 79.62 to 43.50 after retraining without the gold, below full fine-tuning at 54.04 (Tables 2 and 14).
  2. Tables 1 and 2 report substantially different scores for the same frozen models and datasets without explaining why. For example, frozen Qwen3 scores 29.02 on NQ in Table 1 and 56.5 in Table 2. Whether REMEDI improves over frozen also changes between these tables, making the overall evidence difficult to assess.
Questions for authors
  1. Could you use memory without gold answers as the main MedQuAD setting and repeat the concept-held-out evaluation under that setting?
  2. What differences in test queries, corpus, or relevance calculation explain the frozen-score discrepancies between Tables 1 and 2?
Reviewer_bCC3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes REMEDI, query augmentation technique for dual-encoder retrievers. The main motivation is to address new concepts in the query by using external data dictionary storing more context for the concepts. They show empirically some promising results for REMEDI against full-finetuning and data augmentation methods.

Critical strengths
  • The experiments cover 8 benchmarks with motivating six RQs with 5 bi-encoders.
  • The paper seems to provided extensive empirical and theoretical discussion. But paper in the current form is difficult read so better articulation of argument and narrative is encouraged.
Critical weaknesses
  • Figure 1 and FIgure 2 is unclear; pls make it clearer REMEDI works and why augmented query helps. Also, it diffcult to understand why and how REMEDI addressed retrieving new concepts.
  • If I get it right, GPT-4o (or a LLM) is used to get ranking evaluation. Then is does not seem lightweight.
  • There seems to lack of inconsistency in from and how contexts for concepts are retrieved. Sec. 4 mentions LLMs, documents, and external sources. Above sections mentions LLMs and BM25.
  • Furthermore, the motivating scenario for this work was query including new concept. It is not clear to me how query augmenting with external context helps if there is not relevant document (D) and the same is lacking in the parameter knowledge.
  • RQ5 needs more discussion and justification. Claim of generalisation is too strong in the current form.
Questions for authors
  • Are there experiments showing degradation on zero-shot retrieval on other tasks?
  • It also largely remain clear how external data dictionary is created and maintained. For it was mentioned this dictionary is user-editable but there is not experiments or discussion of updating this dictionary data.
  • Also quality of this data is important. Is there any discussion on this? As you have done for ablating different components of attention module learned in Sec 7.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.