- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Retrieval-Aware Vocabulary Folding (RAVF) to reduce the memory footprint of asymmetric LLM retrievers while preserving retrieval accuracy. It reveals that rare tokens may carry critical lexical evidence, making naive vocabulary pruning unsafe. RAVF addresses this through utility-based anchor selection, token protection, and evidence-preserving folding, mapping removed tokens to compatible anchors while keeping the document encoder and retrieval index unchanged.
Critical strengths
- The paper is well-written and clearly organized.
- RAVF introduces a simple and intuitive vocabulary compression framework based on anchor selection, token protection, and vocabulary folding. It can be integrated into existing retrieval systems without modifying the document encoder or retrieval index.
- The paper provides extensive experiments across BEIR and CMTEB-R benchmarks, multiple model families and scales, and different compression ratios.
Critical weaknesses
1. Reported Latency Speedups Are Insufficiently Justified.
The paper reports substantial online query-encoding-plus-retrieval speedups over LightRetriever. Specifically, Table 1 reports the following results on FiQA2018:
| Method | LightRetriever | RAVF-0.9 | Speedup |
|---|---|---|---|
| Dense | 21.59s | 4.40s | 4.91× |
| Hybrid | 24.96s | 6.64s | 3.76× |
However, the source of these improvements remains unclear for several reasons.
First, EmbeddingBag already supports O(1) indexed access per token, while RAVF primarily reduces the number of stored embedding rows without changing the number of token lookups or embedding dimensionality. Although vocabulary compression may improve memory locality and cache efficiency, it does not reduce the asymptotic computational complexity of query encoding, making such substantial speedups difficult to explain.
Second, RAVF introduces an additional token-to-anchor mapping step before embedding lookup. This operation incurs extra computation and memory accesses, yet its runtime overhead is neither analyzed nor reported.
More importantly, Table 1 of the original LightRetriever paper [1] reports the following latency breakdown for processing 65,536 queries:
| Model | Tokenize (s) | Encode (s) | Search (s) | Total (s) | QPS |
|---|---|---|---|---|---|
| Full-Llama8B | 1.3746 | 109.4853 | 8.5133 | 119.3730 | 549 |
| LightRetriever-Llama8B | 0.8209 | 0.0412 | 8.5010 | 9.3630 | 6,999 |
These results show that query encoding accounts for only approximately 0.44% of the total online retrieval time in the original LightRetriever implementation, while search dominates the overall latency. This makes the reported 4–5× speedups particularly difficult to attribute to EmbeddingBag vocabulary compression alone, especially since RAVF leaves the retrieval index and search procedure unchanged.
Although the experimental settings differ between the two papers, including hardware, batch size, dataset, and search configuration, this discrepancy raises concerns about the efficiency evaluation. The authors should clarify how the LightRetriever baseline was implemented and measured, and provide a detailed latency breakdown covering tokenization, token-to-anchor mapping, embedding lookup, and retrieval under identical experimental settings. They should also explain whether the speedups arise from improved cache locality, reduced memory bandwidth requirements, or other implementation-level optimizations.
2. Limited Generalizability Across Model Backbones.
Although RAVF achieves relatively small accuracy drops on Llama3.1-8B, its performance degrades substantially on the Qwen2.5 series. For example, Appendix Table 15 shows that under RAVF-0.9, CMTEB-R Dense nDCG@10 drops from 65.0 to 54.8 for Qwen2.5-7B, compared with only 61.4 to 60.6 for Llama3.1-8B. Similar substantial degradation is observed across other Qwen2.5 model sizes.
These results raise concerns about the backbone generalizability of RAVF. It remains unclear why the method preserves retrieval effectiveness well on certain backbones but causes significant degradation on others, and whether its effectiveness depends on specific backbone characteristics. The authors should investigate the underlying causes of these differences and clarify under what conditions RAVF can be reliably applied across different model families.
3. Limited Practical Significance of Further Memory Compression and Acceleration
The practical significance of further optimizing LightRetriever is insufficiently justified.
First, regarding memory, LightRetriever has already substantially reduced the query-side model from an 8B LLM to a lightweight EmbeddingBag with only 525.3M parameters. It remains unclear whether this relatively small component constitutes a meaningful memory bottleneck in practical retrieval systems, especially compared with the memory required for document embeddings and retrieval indices.
Second, regarding latency, LightRetriever has already made query encoding nearly negligible. The original paper [1] reports the following latency breakdown for 65,536 queries:
| Model | Tokenize (s) | Encode (s) | Search (s) | Total (s) |
|---|---|---|---|---|
| Full-Llama8B | 1.3746 | 109.4853 | 8.5133 | 119.3730 |
| LightRetriever-Llama8B | 0.8209 | 0.0412 | 8.5010 | 9.3630 |
Query encoding accounts for only 0.44% of total latency, meaning that even eliminating it entirely would yield less than 0.5% end-to-end speedup. Since RAVF does not modify the retrieval index or search procedure, its reported several-fold speedups are difficult to reconcile with this breakdown.
Overall, the paper should better justify the practical necessity of further query-side memory compression and explain how it achieves substantial end-to-end acceleration when query encoding is already negligible.
[1] LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference.
Questions for authors
- Could the authors provide a detailed latency breakdown for tokenization, embedding lookup, token mapping, and retrieval to explain the reported 4–5× speedups?
- Why does RAVF exhibit substantially larger performance degradation on Qwen2.5 than on Llama3.1-8B? Does this indicate that the proposed method is effective only for certain model backbones?
- Given that query encoding accounts for only 0.44% of total retrieval time in the original LightRetriever evaluation, what is the practical benefit of further accelerating this component?