- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes REMEDI, a method for improving frozen dense retrievers on queries that mention concepts the encoder has little knowledge about. REMEDI keeps the encoder and document index fixed and introduces an external, editable concept dictionary that maps each concept to a few textual descriptions. A lightweight cross-attention adapter compresses the descriptions of matched concepts into a memory token, which is inserted into the query before re-encoding. The authors provide a theoretical analysis of when such memory can correct the ranking, first for linear encoders and then for Transformers. Experiments on eight datasets and five encoders show improvements over fine-tuning baselines, with small out-of-domain degradation on BEIR.
Critical strengths
Practical, deployment-friendly design. The method keeps the encoder and document index frozen, trains only a lightweight adapter, and stores knowledge in an external, editable concept dictionary. Knowledge can therefore be updated without retraining. Because queries without matched concepts fall back to the original retrieval path, the out-of-domain regression is much smaller than with fine-tuning (Table 3).
Critical weaknesses
-
Limited novelty relative to knowledge-based query expansion. At its core, the method links query spans to concepts in an external knowledge base and enriches the query with their descriptions. This is a long-standing idea in IR [1, 2], and recent work such as KPR [3] already injects entity knowledge into dense retrievers. The paper does not engage with the classical query expansion literature. Its only comparison against text-based expansion driven by the same memory, REMEDI (Text) in Appendix E.4, concatenates and truncates all memory items, which makes it a weak baseline. Stronger text-based alternatives driven by the same concept memory, such as Query2doc- or HyDE-style expansion [4, 5], are not evaluated (the HyDE baseline in Table 1 generates from the query alone, from my understanding). As a result, it remains unclear what is gained beyond injecting the expansion as a learned soft token rather than as text.
-
Empirical claims are not convincingly supported. The most striking gains come from MedQuAD, where the memory includes the test queries’ gold answers (Appendix D.2). The authors’ own controls show that, for RoBERTa, much of this gain disappears once these answers are removed, although Qwen3’s gain is largely retained (Table 14). For strong encoders, REMEDI underperforms the frozen model on all open-domain datasets in Table 1, while Table 2 shows mixed results. Relatedly, frozen baselines differ substantially across tables (e.g., 29.02, 56.5, and 60.04 for Qwen3 on NQ in Tables 1, 2, and 4(b)), suggesting unexplained differences in evaluation protocols. Moreover, the out-of-domain preservation on BEIR follows largely by construction, since most queries bypass the memory (Section 7.3). Taken together, these issues make it difficult to assess the method’s actual effectiveness.
[1] Entity Query Feature Expansion Using Knowledge Base Links. SIGIR 2014.
[2] Query Expansion with Freebase. ICTIR 2015.
[3] Dynamic Injection of Entity Knowledge into Dense Retrievers. Findings of EMNLP 2025.
[4] Query2doc: Query Expansion with Large Language Models. EMNLP 2023.
[5] Precise Zero-Shot Dense Retrieval without Relevance Labels. ACL 2023.
Questions for authors
-
Why do the frozen baselines for the same dataset and encoder differ substantially across tables? For example, Qwen3 on NQ scores 29.02 in Table 1, 56.5 in Table 2, and 60.04 in Table 4(b). Please clarify the evaluation protocol behind each table, and state which results should be taken as the main comparison.
-
How does REMEDI compare with stronger text-based expansion methods that use the same concept memory, such as appending only the most relevant description or using Query2doc/HyDE-style rewriting conditioned on the memory? Such a comparison would clarify whether the gains come from the learned soft-token interface or simply from access to the external knowledge.