acceptodds
Withdrawn from ICLR 2027

Retrieval-Aware Vocabulary Folding for Query-Side Compression in Asymmetric LLM Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Asymmetric LLM retrievers reduce query latency by replacing online query-side LLM inference with a lightweight EmbeddingBag lookup. However, they retain a full tokenizer-aligned query vocabulary, introducing a large memory footprint despite the sparse usage of real retrieval queries. We reveal that this redundancy is not equivalent to token redundancy: rare tokens may carry critical lexical evidence for ranking, making naive vocabulary pruning unsafe. We propose Retrieval-Aware Vocabulary Folding (RAVF), a retrieval-aware vocabulary folding framework that compresses the query-side EmbeddingBag through utility-based anchor selection, token protection, and evidence-preserving folding. Instead of deleting removed tokens, RAVF maps them to compatible anchors while keeping the document encoder and retrieval index unchanged. Experiments on BEIR, CMTEB-R, and multiple LLM backbone scales show that RAVF significantly improves the accuracy–memory trade-off of asymmetric retrieval. Ablation studies further demonstrate that retrieval-aware folding and token protection are the main contributors, while lightweight InfoNCE calibration is sufficient without specialized distillation objectives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Retrieval, Vocabulary Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qtn8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Retrieval-Aware Vocabulary Folding (RAVF) to reduce the memory footprint of asymmetric LLM retrievers while preserving retrieval accuracy. It reveals that rare tokens may carry critical lexical evidence, making naive vocabulary pruning unsafe. RAVF addresses this through utility-based anchor selection, token protection, and evidence-preserving folding, mapping removed tokens to compatible anchors while keeping the document encoder and retrieval index unchanged.

Critical strengths
  1. The paper is well-written and clearly organized.
  2. RAVF introduces a simple and intuitive vocabulary compression framework based on anchor selection, token protection, and vocabulary folding. It can be integrated into existing retrieval systems without modifying the document encoder or retrieval index.
  3. The paper provides extensive experiments across BEIR and CMTEB-R benchmarks, multiple model families and scales, and different compression ratios.
Critical weaknesses

1. Reported Latency Speedups Are Insufficiently Justified.

The paper reports substantial online query-encoding-plus-retrieval speedups over LightRetriever. Specifically, Table 1 reports the following results on FiQA2018:

MethodLightRetrieverRAVF-0.9Speedup
Dense21.59s4.40s4.91×
Hybrid24.96s6.64s3.76×

However, the source of these improvements remains unclear for several reasons.

First, EmbeddingBag already supports O(1) indexed access per token, while RAVF primarily reduces the number of stored embedding rows without changing the number of token lookups or embedding dimensionality. Although vocabulary compression may improve memory locality and cache efficiency, it does not reduce the asymptotic computational complexity of query encoding, making such substantial speedups difficult to explain.

Second, RAVF introduces an additional token-to-anchor mapping step before embedding lookup. This operation incurs extra computation and memory accesses, yet its runtime overhead is neither analyzed nor reported.

More importantly, Table 1 of the original LightRetriever paper [1] reports the following latency breakdown for processing 65,536 queries:

ModelTokenize (s)Encode (s)Search (s)Total (s)QPS
Full-Llama8B1.3746109.48538.5133119.3730549
LightRetriever-Llama8B0.82090.04128.50109.36306,999

These results show that query encoding accounts for only approximately 0.44% of the total online retrieval time in the original LightRetriever implementation, while search dominates the overall latency. This makes the reported 4–5× speedups particularly difficult to attribute to EmbeddingBag vocabulary compression alone, especially since RAVF leaves the retrieval index and search procedure unchanged.

Although the experimental settings differ between the two papers, including hardware, batch size, dataset, and search configuration, this discrepancy raises concerns about the efficiency evaluation. The authors should clarify how the LightRetriever baseline was implemented and measured, and provide a detailed latency breakdown covering tokenization, token-to-anchor mapping, embedding lookup, and retrieval under identical experimental settings. They should also explain whether the speedups arise from improved cache locality, reduced memory bandwidth requirements, or other implementation-level optimizations.

2. Limited Generalizability Across Model Backbones.

Although RAVF achieves relatively small accuracy drops on Llama3.1-8B, its performance degrades substantially on the Qwen2.5 series. For example, Appendix Table 15 shows that under RAVF-0.9, CMTEB-R Dense nDCG@10 drops from 65.0 to 54.8 for Qwen2.5-7B, compared with only 61.4 to 60.6 for Llama3.1-8B. Similar substantial degradation is observed across other Qwen2.5 model sizes.

These results raise concerns about the backbone generalizability of RAVF. It remains unclear why the method preserves retrieval effectiveness well on certain backbones but causes significant degradation on others, and whether its effectiveness depends on specific backbone characteristics. The authors should investigate the underlying causes of these differences and clarify under what conditions RAVF can be reliably applied across different model families.

3. Limited Practical Significance of Further Memory Compression and Acceleration

The practical significance of further optimizing LightRetriever is insufficiently justified.

First, regarding memory, LightRetriever has already substantially reduced the query-side model from an 8B LLM to a lightweight EmbeddingBag with only 525.3M parameters. It remains unclear whether this relatively small component constitutes a meaningful memory bottleneck in practical retrieval systems, especially compared with the memory required for document embeddings and retrieval indices.

Second, regarding latency, LightRetriever has already made query encoding nearly negligible. The original paper [1] reports the following latency breakdown for 65,536 queries:

ModelTokenize (s)Encode (s)Search (s)Total (s)
Full-Llama8B1.3746109.48538.5133119.3730
LightRetriever-Llama8B0.82090.04128.50109.3630

Query encoding accounts for only 0.44% of total latency, meaning that even eliminating it entirely would yield less than 0.5% end-to-end speedup. Since RAVF does not modify the retrieval index or search procedure, its reported several-fold speedups are difficult to reconcile with this breakdown.

Overall, the paper should better justify the practical necessity of further query-side memory compression and explain how it achieves substantial end-to-end acceleration when query encoding is already negligible.

[1] LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference.

Questions for authors
  1. Could the authors provide a detailed latency breakdown for tokenization, embedding lookup, token mapping, and retrieval to explain the reported 4–5× speedups?
  2. Why does RAVF exhibit substantially larger performance degradation on Qwen2.5 than on Llama3.1-8B? Does this indicate that the proposed method is effective only for certain model backbones?
  3. Given that query encoding accounts for only 0.44% of total retrieval time in the original LightRetriever evaluation, what is the practical benefit of further accelerating this component?
Reviewer_vP7HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compresses the query-side EmbeddingBag of an asymmetric retriever while keeping the document encoder and index unchanged. It selects anchor tokens, protects selected lexical tokens, folds other observed tokens into nearby anchors, and calibrates the compact table. Experiments on BEIR and CMTEB-R suggest that folding preserves retrieval quality better than deleting tokens, particularly under aggressive compression.

Critical strengths
  • Keeping the document representation space unchanged is a useful deployment constraint. It allows query-side compression without rebuilding an existing document index.

  • The ablations identify folding as the main source of improvement and separately examine protection and calibration. This makes the contribution of each component reasonably clear.

Critical weaknesses
  • The provenance of Figure 5 is unresolved. Its caption says the examples were sampled from FiQA2018, NFCorpus, and SciFact, but none of the three queries could be found in the official query files checked for these datasets. The terms “drought,” “coral,” and “acidification” are also absent from those collections. Since the figure reports specific retrieval ranks, the examples need traceable query IDs and retrieval records. An undocumented rewrite or incorrect attribution could explain the discrepancy.

  • The roughly 5× query-plus-retrieval speedup needs a clearer explanation. Reducing table rows saves memory, but query length, embedding dimension, and the document index remain unchanged. A timing breakdown is needed to establish where the gain comes from. The training description is also inconsistent: Equations 14 and 16 differ in whether the denominator includes the positive example, and §5.1 specifies 12K steps for full-scale runs while Appendix A.5 says 2K steps were selected for the main paper.

Questions for authors
  • Please provide dataset versions, original query and positive-document IDs, and retrieval runs for Figure 5. Were any displayed queries rewritten or generated?

  • Which components account for the 21.59s-to-4.40s reduction under matched timing conditions?

  • Which loss and training-step count produced the main results?

Reviewer_W16xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To reduce the memory footprint of query EmbeddingBags, this paper proposes RAVF for asymmetric LLM retrieval. It selects anchor tokens using frequency and utility signals, protects lexical and rare tokens, maps removed tokens to nearby anchors, and optionally calibrates the compact table with InfoNCE. The paper evaluates RAVF’s retrieval accuracy and efficiency on BEIR and CMTEB-R and provides ablation studies on BEIR-6.

Critical strengths
  1. Compressing the query-side EmbeddingBag has practical value for deploying LLM-based retrievers. Building on LightRetriever, the paper identifies a concrete opportunity to further reduce storage requirements.
Critical weaknesses
  1. Limited novelty. Vocabulary folding is not a novel approach: [1] already investigates frequency-based vocabulary compression. The token protection is relatively straightforward, and InfoNCE tuning is an established technique.
  2. Limited empirical support and insufficient experimental validation. Table 1 shows low retrieval accuracy at ρ=0.1\rho=0.1, suggesting that this configuration may lack practical utility. At ρ=0.9\rho=0.9 and ρ=0.5\rho=0.5, the ablations in Table 3 show only marginal gains from protection and InfoNCE over variant A, providing limited empirical support for these two components. Table 10 indicates that frequency-based selection combined with folding already achieves strong performance, while incorporating the proposed utility score improves nDCG@10 by only 0.0016. Furthermore, Table 4 compares RAVF only against deletions, without evaluating these selection criteria in combination with folding.
  3. Insufficient implementation details and presentation issues. The paper does not explain how tokens outside the observed query vocabulary, i.e., V∖Vobs\mathcal{V}\setminus\mathcal{V}_{\mathrm{obs}}, are mapped. The measurement procedure for the Query+Retr. times in Table 1 lacks sufficient detail. In addition, the main text lacks pointers to the relevant appendix sections, making the paper difficult to navigate.

[1] Tan, S., Tam, W. L., Wang, Y., et al. “Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method.” Findings of ACL, 2023.

Questions for authors
  1. Do tokens that are unobserved in the training queries occur at test time? If so, how are these tokens mapped?
  2. Although RAVF compresses the EmbeddingBag, query tokens still require mapping and pooling. What accounts for the substantial speedup over LightRetriever?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.