- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose FBNL reranking, where the document passages are encoded offline while during online inference the query and passages can still interact with each other through cross-attention in a decoder. Matryoshka Embedding Pooling (MEP) improves the efficiency by compressing the passage embeddings with mean pooling. The experimental results demonstrate good performance and efficiency gain compared to the baselines.
Critical strengths
- The ideas (FBNL and MEP) are simple but show competitive performance based on the experimental results and hold practical value in terms of efficiency.
Critical weaknesses
- I found the multi-stage training approach not well justified, which seems to be an independent improvement orthogonal to the main ideas (FBNL and MEP). It also weakens the experimental results as it masks the main sources of performance gain. Although there is an ablation study, a more principled approach would be either to remove multi-stage training from this contribution (leave it for a separate paper) or to apply the multi-stage training to the baselines as well.
- There is recent work in reranking that seems to be highly related but is neither discussed nor compared against, e.g. EBCAR [1], which uses passage and query embeddings directly and claims to have efficiency gains in addition to accuracy improvements.
[1] Yuan, Y., Shabani, A. and Liu, S., 2026, April. Embedding-based context-aware reranker. In International Conference on Learning Representations (Vol. 2026, pp. 97167-97186).
Questions for authors
N/A
11 comments
(2/2) Response to Reviewer E4oB
Weakness 2: There is recent work in reranking that seems to be highly related but is neither discussed nor compared against, e.g. EBCAR [1], which uses passage and query embeddings directly and claims to have efficiency gains in addition to accuracy improvements.
We thank the reviewer for pointing out this valuable work. We have cited and discussed EBCAR in Appendix J (Additional Related Work on Reranking) of the revised manuscript. Both methods reuse precomputed passage representations. EBCAR is a listwise reranker that jointly models the query and candidate passage embeddings, incorporating source document IDs and passage positions within each document. Lychee-Reranker performs pointwise reranking, using merged attention in the decoder to model fine-grained interactions between query context and passage representations compressed with MEP. Our main experiments focus on pointwise rerankers, so we discuss EBCAR as related work. We also could not find a publicly available EBCAR checkpoint for direct evaluation.
(1/2) Response to Reviewer E4oB: Training Stage and MEP Ablations (Follow-up to W1)
Weakness 1: It also weakens the experimental results as it masks the main sources of performance gain.
We thank the reviewer for raising this concern. Table 6 of the manuscript quantifies the gains from the later training stages at a fixed compression ratio of .
| Lychee-Reranker variant | Stage 1 | Stage 2 | Stage 3 | Gain from Stage 1 to Stage 3 |
|---|---|---|---|---|
| Nano | 57.44 | 58.28 | 58.54 | +1.10 |
| Small | 59.49 | 60.92 | 61.07 | +1.58 |
| Large | 62.04 | 63.37 | 63.53 | +1.49 |
Average BEIR nDCG@10 at .
Figure 9 of the manuscript compares two Nano variants trained through all three stages, one with MEP and one without. Both use the same initialization, training data, number of steps, and other hyperparameters. We evaluate both variants after each stage. On BEIR at , the variant trained with MEP scores 57.44 after Stage 1, compared with 38.57 for the variant trained without MEP. This gain of 18.87 points shows that MEP improves reranking under compression before knowledge distillation and model merging. Together, Figure 9 and Table 6 quantify the benefit of training with MEP under compression and the gains from multi-stage training.
(1/2) Response to Reviewer E4oB
We sincerely thank the reviewer
E4oBfor the constructive comments and suggestions, which are very helpful for improving our paper. We are also grateful that you recognised the strengths of our paper. We have updated the manuscript accordingly and kindly invite the reviewer to review the revised version for details. The main changes are marked in blue. Please find point-by-point responses below.We thank the reviewer for this comment. Foundation model development typically involves multi-stage training, and our work presents a reranker family together with its model training recipe and reproducible resources. Multi-stage pipelines are also used to train industrial rerankers such as Qwen3-Reranker, Jina Reranker v2, and mxbai-rerank-v2, as shown in the table below. We compare Lychee-Reranker with released industrial rerankers using the same candidate sets and evaluation metrics. Our training set contains approximately 3.9M instances, compared with approximately 19M SFT examples reported by Qwen3-Reranker. We therefore consider our comparison with these industrial rerankers fair. These industrial rerankers also use multi-stage training, and some are even trained on larger datasets.
Furthermore, FBNL and MEP optimize the architecture, while multi-stage training optimizes learning. These are complementary parts of the same foundation reranker. Table 6 and Figure 9 ablate the training stages and MEP, respectively, demonstrating their effectiveness. Specifically, Figure 9 compares Lychee-Reranker-Nano with and without MEP training under the same training settings, showing that MEP improves performance before knowledge distillation and model merging. Our time complexity analysis and practical throughput tests further demonstrate FBNL's inference efficiency. Together, these innovations yield a family of foundation rerankers combining efficiency, expressiveness, compactness, and test-time compute.
Finally, as shown in the table below, we provide more complete public resources than the other industrial rerankers, including model weights, training data, training and inference code, and intermediate checkpoints. We believe this supports reproducible research and helps advance the field.