- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Shared Signed Gate (SSG), a lightweight reranker that improves dense retrieval ranking quality while retaining the encoder, stored embeddings, and retrieval index. SSG uses explicit dimension-level supervision to learn candidate-conditioned residual weights that distinguish relevant documents from hard negatives. Across nine encoders and seven BEIR datasets, SSG improves average nDCG@10 for every encoder and outperforms evaluated baselines in detailed comparisons.
Critical strengths
- The paper proposes a simple yet effective approach to improving dense retrieval by learning dimension-wise weights. The relevance-derived supervision from positive documents and hard negatives is intuitive and provides a meaningful training signal.
- SSG improves retrieval quality without modifying the pretrained embedding model, recomputing document embeddings, or rebuilding the retrieval index, making it relatively easy to integrate into existing retrieval systems.
- The paper evaluates SSG across nine embedding models and seven BEIR datasets, achieving consistent improvements in average nDCG@10 across all nine models. The experiments also include comparisons with several baselines and detailed ablation studies.
Critical weaknesses
- Insufficient and unfair reranker comparisons. The paper lacks comparisons with stronger and more recent rerankers, such as BGE-reranker, Jina-reranker, and Qwen-reranker. Moreover, the MiniLM cross-encoder is evaluated in a zero-shot setting, while SSG is trained on each target dataset, making the comparison potentially unfair.
- Limited practicality and generalizability. SSG requires separately trained models for each dataset–encoder pair, along with domain-specific relevance annotations. This introduces additional annotation and training costs, limiting its practical applicability. Moreover, cross-domain transfer is not evaluated, leaving its ability to generalize to unseen domains unclear.
- Missing query-only fine-tuning baseline. Given the availability of supervised training data, a simpler alternative would be to fine-tune only the query encoder, e.g., using LoRA, while keeping document embeddings and the retrieval index fixed. This could improve retrieval effectiveness without introducing an additional reranking stage and its associated inference overhead. The paper does not compare against this alternative.
- Limited improvements relative to inference overhead. SSG introduces substantial additional latency for larger embedding models (e.g., 23.3% for Qwen3-8B), while the improvements on several datasets are marginal. For example, with Qwen3-8B, nDCG@10 increases only from 0.611 to 0.612 on FiQA, 0.915 to 0.916 on TREC-COVID, and 0.671 to 0.673 on NQ. These limited gains raise concerns about its practical cost-effectiveness, particularly for stronger retrievers.
Questions for authors
- Could the authors compare SSG with stronger rerankers, such as BGE-reranker, Jina-reranker, and Qwen-reranker, and provide a fairer comparison under similar supervision settings?
- How well does SSG generalize across different domains? Can a model trained on one dataset be directly applied to another without additional annotations and retraining?
- Why not directly fine-tune the query encoder using LoRA with the same training data, while keeping the document embeddings and index fixed? How would this compare with SSG in terms of retrieval performance and inference efficiency?
- Given the substantial inference overhead on larger embedding models and the marginal improvements on several datasets, in what scenarios would SSG offer a favorable accuracy–latency trade-off?
6 comments
Response to Reviewer CP46
W3 / Q3. Query-only fine-tuning with LoRA
We agree that query-only LoRA is a relevant alternative and have added it for both BGE-S and Qwen3-8B. The baseline adapts the query encoder using target-domain supervision while keeping document embeddings and the retrieval index fixed. We evaluate eight configurations per dataset, with up to 20 training epochs and validation-based hyperparameter and checkpoint selection.
Average test nDCG@10 and end-to-end latency (ms/query) are reported below.
| Method | BGE-S: nDCG@10 | BGE-S: Latency | Qwen3-8B: nDCG@10 | Qwen3-8B: Latency |
|---|---|---|---|---|
| Base Retrieval | 0.519 | 42.71 | 0.647 | 440.85 |
| Query-encoder LoRA | 0.510 | 48.18 | 0.637 | 464.87 |
| SSG (Ours) | 0.559 | 44.08 | 0.661 | 445.67 |
Under this protocol, SSG achieves higher average effectiveness and lower measured end-to-end latency than query-only LoRA for both encoders. With BGE-S, SSG achieves 0.559 versus 0.510 nDCG@10; with Qwen3-8B, it achieves 0.661 versus 0.637. These results show that directly fine-tuning the query encoder does not provide a better accuracy-latency trade-off than SSG under the evaluated settings, despite avoiding a separate reranking stage.
Implementation details and results appear in Tables 4, 7, 11 and 12.
W4 / Q4. Favorable accuracy-latency trade-off
We thank the reviewer for raising this concern. To better establish scenarios where SSG offers a favorable accuracy-latency trade-off, we additionally compare it against query-encoder LoRA and larger LoRA-fine-tuned rerankers under a common NVIDIA A100 setup. While our original L4 profiling reports 23.3% overhead for Qwen3-8B, the new evaluation allows direct comparison with alternative methods under matched hardware conditions. As detailed in Appendix A.5, this overhead is dominated by accessing the retrieved document vectors; candidate-statistics computation, gate prediction, and rescoring together generally require approximately 2-3 ms/query. The following table reports seven-dataset macro-average nDCG@10 and end-to-end latency (ms/query), including query encoding, retrieval, and applicable reranking costs.
| Method | BGE-S: nDCG@10 | BGE-S: Latency | Qwen3-8B: nDCG@10 | Qwen3-8B: Latency |
|---|---|---|---|---|
| Base Retrieval | 0.519 | 42.71 | 0.647 | 440.85 |
| Query-encoder LoRA | 0.510 | 48.18 | 0.637 | 464.87 |
| MiniLM (LoRA) | 0.499 | 301.70 | 0.523 | 693.71 |
| BGE-reranker-v2-m3 (LoRA) | 0.537 | 2039.67 | 0.589 | 2450.97 |
| Jina-reranker-v2 (LoRA) | 0.533 | 753.55 | 0.589 | 1161.00 |
| Qwen3-Reranker-0.6B (LoRA) | 0.562 | 3305.66 | 0.601 | 3714.31 |
| SSG (Ours) | 0.559 | 44.08 | 0.661 | 445.67 |
In this matched setup, SSG adds only 3.2% and 1.1% latency over Base Retrieval with BGE-S and Qwen3-8B, respectively. With BGE-S, SSG nearly matches the strongest fine-tuned reranker at approximately 75× lower end-to-end latency. More notably, with Qwen3-8B, SSG not only surpasses all evaluated fine-tuned rerankers in macro-average nDCG@10 (0.661 vs. at most 0.601), but also achieves approximately 8.3× lower end-to-end latency than Qwen3-Reranker-0.6B. Although gains over Base Retrieval are marginal on some datasets, these results highlight SSG’s ability to deliver competitive ranking effectiveness at substantially lower end-to-end inference latency than larger neural rerankers, even when operating with a strong embedding encoder. SSG is therefore particularly well suited to latency-sensitive systems that must preserve their existing encoder, stored document embeddings, and index. Detailed results and profiling are in Section 5.5 and Appendix A.6.
Thank You for Your Feedback!
We sincerely hope that these clarifications and additional experimental results have addressed your concerns and strengthened your assessment of our work. We would be grateful if you could kindly reconsider your rating in light of the new evidence.
Response to Reviewer CP46
We sincerely thank the reviewer for the careful reading and constructive suggestions. We appreciate your positive assessment of our methodology, experiments, and ablation studies. Your feedback has helped us improve the clarity of the paper and strengthen its empirical evaluation. Following your suggestions, we have added experiments addressing each concern. Revised passages in the manuscript are marked in blue.
W1 / Q1. Stronger rerankers and comparable supervision
We thank the reviewer for suggesting stronger reranker comparisons. We now evaluate BGE-reranker-v2-m3, Jina-reranker-v2-base-multilingual, and Qwen3-Reranker-0.6B, alongside MiniLM, both without target-domain adaptation and with supervised LoRA fine-tuning. For each encoder, SSG and all text rerankers operate on the same top-100 retrieved candidates. Each reranker's LoRA adapter is fine-tuned separately using the corresponding encoder's candidates and target-domain relevance labels. We search eight LoRA configurations per dataset (two ranks × four learning rates), selecting checkpoints by validation nDCG@10.
Average test nDCG@10 across seven datasets is reported below.
With BGE-S, SSG nearly matches the strongest fine-tuned reranker (0.559 vs. 0.562 nDCG@10) while achieving approximately 75× lower end-to-end latency (44.08 vs. 3305.66 ms/query). With Qwen3-8B, SSG achieves higher average effectiveness (0.661 vs. 0.601) at approximately 8.3× lower latency (445.67 vs. 3714.31 ms/query). These results demonstrate that SSG offers competitive or superior ranking effectiveness at a fraction of the inference latency of larger neural rerankers.
The complete comparisons appear in Section 5.5 and Table 4, with training settings in Table 7 and dataset-level results in Appendix A.6, Table 11.
W2 / Q2. Practicality and cross-domain generalization
We thank the reviewer for raising the question of cross-domain transfer without additional annotations or retraining. We have added experiments that train SSG separately on NFCorpus, FiQA, and SciFact using BGE-S, then directly apply each selected predictor to the other six datasets without target-domain fine-tuning or hyperparameter selection. We additionally evaluate Learning to Select (LTS) and CMC under the same source-to-target transfer setting, retaining each method's original inference procedure.
Unseen datasets improved over Base Retrieval:
Notably, SSG trained on either NFCorpus or FiQA improves retrieval across all six unseen datasets, demonstrating consistent transfer from these sources without target-domain supervision. For example, NFCorpus-trained SSG improves ArguAna from 0.592 to 0.612 and NQ from 0.492 to 0.513, while even slightly outperforming target-trained SSG on Touché-2020 (0.352 vs. 0.350).
Importantly, these models are trained on individual domain-specific datasets rather than large, heterogeneous collections, and none is explicitly optimized for cross-domain transfer. Despite this setting, SSG demonstrates stronger transfer consistency than the evaluated baselines, supporting its practicality when new target-domain annotations are unavailable.
The complete cross-domain results are provided in Appendix A.7, Table 13.