REMEDI: Lightweight Editable Memory Adapters for Frozen Dual Encoders

Dual-encoder retrievers often fail on short queries that name concepts for which the query encoder lacks parametric knowledge. Prior work typically addresses this problem by fine-tuning on in-domain data, but doing so can degrade zero-shot retrieval on other tasks. We introduce REMEDI (Retrieval Enrichment via Memory-Editable Dictionary Interfaces), which keeps the retriever frozen and instead augments queries with an external, user-editable concept memory: a dictionary mapping each concept to a small set of textual descriptions. REMEDI encodes these descriptions once using the frozen document encoder and trains only a lightweight cross-attention adapter, leaving the pretrained encoder parameters, document embeddings, and retrieval index unchanged. At test time, REMEDI matches query spans to concepts, attends over the corresponding descriptions to construct a memory token, and inserts this token into the query between learned boundary tokens. Across eight benchmarks and five encoders, REMEDI achieves the best average nDCG@10 on every encoder, outperforming full fine-tuning and LoRA, with the largest gains on concept-heavy queries. It also generalizes to concepts unseen during training and stays within 0.64 points of the frozen retriever on out-of-domain BEIR tasks, where fine-tuning loses up to 5.2. Theoretically, we characterize when relevant memory lifts a query's positive above its hard negatives, and show when minimizing the contrastive loss guarantees correct ranking, first for linear encoders and then for Transformers.