acceptodds
Under review as a conference paper at ICLR 2027

Lychee-Reranker: Fast but Not Late Interaction for Compressed Document Reranking

Abstract

As retrieval systems scale, effective and efficient reranking becomes increasingly important. However, most existing encoder- and decoder-based rerankers jointly process every query–passage pair, tightly coupling their online computation and limiting deployment efficiency and flexibility. We present Lychee-Reranker, a fast but not late-interaction (FBNL) reranker that decouples query and passage computation while retaining expressive relevance modeling. Built on an encoder–decoder architecture, Lychee-Reranker pre-encodes passages using Matryoshka embedding pooling, while its decoder models system and user instructions together with query intent; cross-attention then captures fine-grained relevance between the resulting query context and passage representations. Together, these designs offer four key advantages: (i) efficiency from offline passage encoding, (ii) expressiveness from cross-attention, (iii) compactness from Matryoshka embedding pooling, and (iv) test-time compute through an adjustable compute budget. We instantiate Lychee-Reranker in three sizes, Nano, Small, and Large, with 0.27B, 1B, and 4B activated parameters, respectively. Extensive experiments on BEIR, MIRACL, and LMEB demonstrate strong reranking performance with superior efficiency. On BEIR and MIRACL, our models achieve competitive performance in multi-domain and multilingual reranking, on par with strong industrial rerankers such as the Qwen3/BGE-Reranker series. On LMEB-Dialogue, a compact embedding model paired with our Nano reranker, which has only 0.27B activated parameters, remains competitive with 7–12B embedding models. We will release our code, data, and models to facilitate academic research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Efficient Document Reranking, Information Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E4oBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose FBNL reranking, where the document passages are encoded offline while during online inference the query and passages can still interact with each other through cross-attention in a decoder. Matryoshka Embedding Pooling (MEP) improves the efficiency by compressing the passage embeddings with mean pooling. The experimental results demonstrate good performance and efficiency gain compared to the baselines.

Critical strengths
  • The ideas (FBNL and MEP) are simple but show competitive performance based on the experimental results and hold practical value in terms of efficiency.
Critical weaknesses
  • I found the multi-stage training approach not well justified, which seems to be an independent improvement orthogonal to the main ideas (FBNL and MEP). It also weakens the experimental results as it masks the main sources of performance gain. Although there is an ablation study, a more principled approach would be either to remove multi-stage training from this contribution (leave it for a separate paper) or to apply the multi-stage training to the baselines as well.
  • There is recent work in reranking that seems to be highly related but is neither discussed nor compared against, e.g. EBCAR [1], which uses passage and query embeddings directly and claims to have efficiency gains in addition to accuracy improvements.

[1] Yuan, Y., Shabani, A. and Liu, S., 2026, April. Embedding-based context-aware reranker. In International Conference on Learning Representations (Vol. 2026, pp. 97167-97186).

Questions for authors

N/A

11 comments
AuthorsOpenReview ↗

(1/2) Response to Reviewer E4oB

We sincerely thank the reviewer E4oB for the constructive comments and suggestions, which are very helpful for improving our paper. We are also grateful that you recognised the strengths of our paper. We have updated the manuscript accordingly and kindly invite the reviewer to review the revised version for details. The main changes are marked in blue. Please find point-by-point responses below.

Weakness 1: I found the multi-stage training approach not well justified, which seems to be an independent improvement orthogonal to the main ideas (FBNL and MEP). It also weakens the experimental results as it masks the main sources of performance gain. Although there is an ablation study, a more principled approach would be either to remove multi-stage training from this contribution (leave it for a separate paper) or to apply the multi-stage training to the baselines as well.

We thank the reviewer for this comment. Foundation model development typically involves multi-stage training, and our work presents a reranker family together with its model training recipe and reproducible resources. Multi-stage pipelines are also used to train industrial rerankers such as Qwen3-Reranker, Jina Reranker v2, and mxbai-rerank-v2, as shown in the table below. We compare Lychee-Reranker with released industrial rerankers using the same candidate sets and evaluation metrics. Our training set contains approximately 3.9M instances, compared with approximately 19M SFT examples reported by Qwen3-Reranker. We therefore consider our comparison with these industrial rerankers fair. These industrial rerankers also use multi-stage training, and some are even trained on larger datasets.

ModelTraining pipelineTraining data size
Lychee-RerankerSFT → KD → model soup~3.9M
Qwen3-RerankerSFT → model merging~19M
Jina Reranker v2English → cross-lingual → multilingual → hard-negative trainingNot reported
mxbai-rerank-v2GRPO → contrastive learning → preference learningNot reported

Furthermore, FBNL and MEP optimize the architecture, while multi-stage training optimizes learning. These are complementary parts of the same foundation reranker. Table 6 and Figure 9 ablate the training stages and MEP, respectively, demonstrating their effectiveness. Specifically, Figure 9 compares Lychee-Reranker-Nano with and without MEP training under the same training settings, showing that MEP improves performance before knowledge distillation and model merging. Our time complexity analysis and practical throughput tests further demonstrate FBNL's inference efficiency. Together, these innovations yield a family of foundation rerankers combining efficiency, expressiveness, compactness, and test-time compute.

Finally, as shown in the table below, we provide more complete public resources than the other industrial rerankers, including model weights, training data, training and inference code, and intermediate checkpoints. We believe this supports reproducible research and helps advance the field.

ModelWeightsTraining dataTraining codeInference codeIntermediate checkpoints
Lychee-Reranker✓✓✓✓✓
Qwen3-Reranker✓××✓×
Jina Reranker v2✓××✓×
mxbai-rerank-v2✓××✓×
AuthorsOpenReview ↗

(2/2) Response to Reviewer E4oB

Weakness 2: There is recent work in reranking that seems to be highly related but is neither discussed nor compared against, e.g. EBCAR [1], which uses passage and query embeddings directly and claims to have efficiency gains in addition to accuracy improvements.

We thank the reviewer for pointing out this valuable work. We have cited and discussed EBCAR in Appendix J (Additional Related Work on Reranking) of the revised manuscript. Both methods reuse precomputed passage representations. EBCAR is a listwise reranker that jointly models the query and candidate passage embeddings, incorporating source document IDs and passage positions within each document. Lychee-Reranker performs pointwise reranking, using merged attention in the decoder to model fine-grained interactions between query context and passage representations compressed with MEP. Our main experiments focus on pointwise rerankers, so we discuss EBCAR as related work. We also could not find a publicly available EBCAR checkpoint for direct evaluation.

AuthorsOpenReview ↗

(1/2) Response to Reviewer E4oB: Training Stage and MEP Ablations (Follow-up to W1)

Weakness 1: It also weakens the experimental results as it masks the main sources of performance gain.

We thank the reviewer for raising this concern. Table 6 of the manuscript quantifies the gains from the later training stages at a fixed compression ratio of r=4r=4.

Lychee-Reranker variantStage 1Stage 2Stage 3Gain from Stage 1 to Stage 3
Nano57.4458.2858.54+1.10
Small59.4960.9261.07+1.58
Large62.0463.3763.53+1.49

Average BEIR nDCG@10 at r=4r=4.

Figure 9 of the manuscript compares two Nano variants trained through all three stages, one with MEP and one without. Both use the same initialization, training data, number of steps, and other hyperparameters. We evaluate both variants after each stage. On BEIR at r=4r=4, the variant trained with MEP scores 57.44 after Stage 1, compared with 38.57 for the variant trained without MEP. This gain of 18.87 points shows that MEP improves reranking under compression before knowledge distillation and model merging. Together, Figure 9 and Table 6 quantify the benefit of training with MEP under compression and the gains from multi-stage training.

Reviewer_s8JxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces Lychee-Reranker, which uses an encoder to precompute document token states and a decoder with merged attention to score query-document relevance. The authors have conducted extensive work in this paper, including the development of models in three different sizes, as well as experiments and evaluations across a wide range of benchmarks, demonstrating that Lychee-Reranker achieves competitive ranking performance.

Critical strengths
  1. Practical architecture design inherited from T5Gemma2. Assigning document encoding to the offline stage and query-conditioned scoring to the online decoder is a useful deployment configuration of an encoder–decoder model.
  2. Promising effectiveness–efficiency trade-off. The models achieve competitive nDCG@10 against strong rerankers. The compression and two-pass results suggest that substantial reductions in estimated online computation may be possible with small effectiveness losses.
  3. Extensive experiments. The paper covers multiple benchmarks with lots of analyses and ablations across model sizes, compression ratios, and training stages.
Critical weaknesses
  1. The architectural contribution is poorly defined. The encoder–decoder structure and merged attention are inherited from T5Gemma2. The paper's architectural choice is primarily to encode documents separately and pass queries to the decoder. Moreover, calling this a new “Fast but Not Late Interaction” paradigm is confusing; perhaps the author intended to convey the idea of ​​"direct interaction", but the name alone provides no insight into what the architecture is actually like.
  2. The related-work discussion misses important context. The paper repeatedly invokes “Matryoshka”; however, there is no citation of the Matryoshka Embedding paper [1]. That work varies embedding dimensionality, whereas MEP pools along the token sequence; explaining this distinction is necessary to assess the contribution accurately. Moreover, E2Rank [2] also reranks with precomputed document embeddings and is relevant to the paper's central claim of reusable document representations, even though its scoring mechanism differs.
  3. Every reranker is evaluated on the top 100 documents retrieved by KaLM-Embedding-V2.5, which also supplies hard negatives during training. The choice of this retriever is not justified and is not common compared to previous reranking papers, more explanation is needed. Moreover, Lychee-Reranker is trained using negatives mined by the retrieval model itself; theoretically, it should therefore have an advantage over other models that were not trained on this specific data distribution. Evaluation with a conventional BM25 or another dense retriever would test whether the reported gains and compression trends generalize.
  4. The efficiency gains are not measured in practice. The paper's speed comparisons are all estimates, with no latency or throughput results. In this method, serving cost also depends on fetching cached document states, memory traffic, and batching. The paper therefore does not yet establish that its lower estimated computation translates into faster reranking in practice.

[1] Matryoshka representation learning [2] E2Rank: Your Text Embedding can Also be an Effective and Efficient Listwise Reranker

Questions for authors
  1. On identical hardware and with the same top-100 candidates, what are measured p50/p95 per-query latency and throughput for each Lychee model and the main baselines at practical batch sizes?
  2. How does the re-ranking performance of Lychee-Reranker compare to other baselines when used with different first-stage retrieval models?
11 comments
AuthorsOpenReview ↗

(1/4) Response to Reviewer s8Jx

We sincerely thank the reviewer s8Jx for the constructive comments and suggestions, which are very helpful for improving our paper. We are also grateful that you recognised the strengths of our paper. We have updated the manuscript accordingly and kindly invite the reviewer to review the revised version for details. The main changes are marked in blue. Please find point-by-point responses below.

Weakness 1: The architectural contribution is poorly defined. The encoder–decoder structure and merged attention are inherited from T5Gemma2. The paper's architectural choice is primarily to encode documents separately and pass queries to the decoder. Moreover, calling this a new “Fast but Not Late Interaction” paradigm is confusing; perhaps the author intended to convey the idea of ​​"direct interaction", but the name alone provides no insight into what the architecture is actually like.

We thank the reviewer for raising this point. As stated at the beginning of Section 3.1, Lychee-Reranker is built on the T5Gemma2 model family and inherits its encoder–decoder structure and merged attention. Our contribution lies in the FBNL reranking paradigm design built on this backbone.

Lychee-Reranker recognizes that queries and passages serve different roles: queries specify the information need, while passage representations can be reused across queries. Since passages are typically longer and contain redundant information, we encode them offline and compress them with MEP (Matryoshka Embedding Pooling). The decoder then performs fine-grained query-passage interactions online.

“Fast” refers to the reduction in online computation from offline passage encoding and MEP. “Not Late” refers to expressive query–passage interaction within the decoder, where merged attention uses compressed passage representations to update query states across layers. Late-interaction models compute relevance by simply aggregating token similarities after encoding the query and passage separately.

We will clarify the architectural contribution in the revised manuscript.

AuthorsOpenReview ↗

(2/4) Response to Reviewer s8Jx

Weakness 2: The related-work discussion misses important context. The paper repeatedly invokes “Matryoshka”; however, there is no citation of the Matryoshka Embedding paper [1]. That work varies embedding dimensionality, whereas MEP pools along the token sequence; explaining this distinction is necessary to assess the contribution accurately. Moreover, E2Rank [2] also reranks with precomputed document embeddings and is relevant to the paper's central claim of reusable document representations, even though its scoring mechanism differs.

We thank the reviewer for pointing out these important works. We cite and discuss MRL in Section 4.1 of the revised manuscript, clarifying that MRL learns nested representations of different dimensionalities within a single embedding, while MEP compresses passage representations along the sequence dimension.

We also discuss E2Rank in Appendix J (Additional Related Work on Reranking) of the revised manuscript. Both E2Rank and Lychee-Reranker reuse precomputed document embedding/representations. E2Rank uses document embeddings for efficient listwise reranking. Lychee-Reranker performs pointwise reranking, using merged attention in the decoder to model fine-grained interactions between query context and passage representations compressed with MEP. Since this work focuses on pointwise rerankers, we discuss E2Rank as related work.

AuthorsOpenReview ↗

(3/4) Response to Reviewer s8Jx

Weakness 3: Every reranker is evaluated on the top 100 documents retrieved by KaLM-Embedding-V2.5, which also supplies hard negatives during training. The choice of this retriever is not justified and is not common compared to previous reranking papers, more explanation is needed. Moreover, Lychee-Reranker is trained using negatives mined by the retrieval model itself; theoretically, it should therefore have an advantage over other models that were not trained on this specific data distribution. Evaluation with a conventional BM25 or another dense retriever would test whether the reported gains and compression trends generalize.

Question 2: How does the re-ranking performance of Lychee-Reranker compare to other baselines when used with different first-stage retrieval models?

We thank the reviewer for pointing this out. We chose KaLM-Embedding-V2.5 for its strong multilingual retrieval capabilities, allowing a consistent evaluation setup across BEIR and MIRACL. All rerankers are evaluated on the same top-100 candidates for each query.

We acknowledge that using the same retriever for negative mining and evaluation may favor Lychee-Reranker. To address this concern, we additionally evaluate all rerankers on the top-100 candidates retrieved by BGE-M3. As shown in the table below, Lychee-Reranker achieves the highest average nDCG@10 within each corresponding parameter group when using BGE-M3 as the first-stage retriever on BEIR. These results show that the reported gains generalize to a different first-stage retriever.

ModelSizeBEIR Avg.AACFCQADBFVFQAHQAMSNFNQQRSDSF
First-stage Retriever
BGE-M30.6B50.3054.0129.7039.1439.8480.9941.1169.4438.3231.6460.6388.5716.3564.15
Second-stage Reranker
Models with more than 4B parameters
Qwen3-Reranker-8B8B63.1485.3648.5750.8852.6394.3555.4283.0745.0238.3573.7188.9926.1878.34
Models with 1B–4B parameters
Qwen3-Reranker-4B4B61.5485.5448.0048.1447.1494.1151.9482.8543.9638.2871.0887.0224.6877.22
Lychee-Reranker-L4B62.3472.6446.9250.6951.4793.8956.6683.3946.9239.7675.3190.7423.4878.49
bge-reranker-v2-gemma2.5B58.4072.5738.1442.1147.2789.0446.9682.4247.3237.2570.8188.9520.2276.17
mxbai-rerank-large-v21.5B58.4267.2944.1143.7848.0292.7747.9581.8846.1634.5369.1188.7617.6577.50
Models with 0.5B–1B parameters
Lychee-Reranker-S1B60.2269.1744.6046.2748.9393.1752.2182.7446.4337.8872.8990.7321.4276.46
bge-reranker-large0.6B50.1328.7040.3437.8146.5692.7039.4282.1346.6332.8942.2672.9216.5272.79
bge-reranker-v2-m30.6B54.4543.7636.8538.0746.0189.5843.2081.6847.3233.4568.6589.1117.6072.59
Qwen3-Reranker-0.6B0.6B58.9978.5547.7046.2148.5693.2246.5280.5742.3037.0464.2384.4021.3176.30
mxbai-rerank-base-v20.5B56.1461.8940.5343.9247.6489.3043.4380.9145.3335.8364.8287.0713.8875.22
Models with fewer than 0.5B parameters
gte-reranker-base0.3B55.9158.2845.3638.4745.9893.7843.1980.6945.0335.4665.2781.9317.9275.49
jina-reranker-v20.3B55.7051.6433.7240.1048.1792.2144.9579.4547.4336.4467.2187.8419.4975.44
Lychee-Reranker-N0.27B57.8363.6842.8440.4747.2692.7746.0681.8045.3236.6169.0789.9220.4775.58

BEIR reranking results in nDCG@10 using BGE-M3 as the first-stage retriever. Bold marks the best score in each parameter group.

Reviewer_TbqzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FBNL (Fast but Not Late-interaction), a reranking paradigm, and builds the Lychee-Reranker model family on the T5Gemma2 encoder–decoder. The encoder encodes documents offline. Online, the decoder interacts with the document representations through cross-attention, and the relevance score is computed from the logits of the "yes" and "no" tokens. The authors also propose Matryoshka Embedding Pooling (MEP), which compresses document representations along the sequence dimension and is trained jointly over multiple compression ratios. Training has three stages: SFT, knowledge distillation, and model soup. On BEIR and MIRACL, the models perform on par with rerankers of similar size, with lower estimated online compute cost.

Critical strengths
  1. The method is sound and the idea is clear. Document encoding is moved offline while deep online interaction is kept. This combines the expressiveness of cross-encoders with the efficiency of precomputation-based methods. MEP is simple and adds no extra parameters, so a single model can support multiple compression ratios.
  2. It balances efficiency and effectiveness, which makes it practical. The models greatly reduce online compute while staying on par with rerankers of similar size. This gives the approach good potential for real-world use.
Critical weaknesses
  1. The relationship to prior work is not well discussed, and key comparisons are missing.

    • Precomputing document representations offline and doing deep interaction online is not a new idea. PreTTR [1] and MORES [2] have already explored similar designs for reranking. However, the paper does not cite them, does not explain how it differs from them, and does not compare against them at the same model size.
    • In terms of architecture choice, the authors should also compare with decoder-only approaches, especially those that reuse precomputed document KV caches (e.g., HyperRAG [3] and CompRank [4]), and explain what advantages the encoder–decoder architecture offers over them. Without these comparisons, it is hard to tell whether the gains come from the architecture itself or from MEP and the training pipeline.
  2. Some content is incomplete.

    • In the third stage of Section 4.2, the paper says the LoRA adapters are merged back into their backbones. But LoRA fine-tuning is never mentioned earlier in the main text, so this comes as a surprise. Please state in the main text which modules LoRA is applied to and give its main settings.
    • Results without compression (r=1r=1) are missing. Performance drops steadily from r=2r=2 to r=16r=16, but no uncompressed result is reported. This makes it impossible to see the performance upper bound or measure the actual loss caused by compression.
  3. The efficiency analysis is not thorough enough.

    • In Table 1, the columns Instruction Aware, Test-Time Compute, and MEP Support are identical across all models. This repeats the same information and could be stated in one sentence. I suggest using this space to break down where the efficiency gains come from: offline encoding, running only the decoder online, document tokens being read only as keys/values in attention without passing through the FFN, and MEP shortening the sequence length. It would help to show how much each part contributes.
    • It is unclear how the Cost in Tables 3 and 4 is computed. Is it a theoretical complexity estimate or a measured value? Are the baseline costs estimated under the same assumptions?

References

[1] MacAvaney et al. Efficient Document Re-Ranking for Transformers by Precomputing Term Representations. SIGIR 2020.

[2] Gao et al. Modularized Transformer-based Ranking Framework. EMNLP 2020.

[3] An et al. HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reuse. arXiv:2504.02921, 2025.

[4] Lu et al. CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring. arXiv:2606.11700, 2026.

Questions for authors

Clarification questions:

  1. What is the key difference between FBNL and prior work such as MORES and PreTTR? At the same model size, what advantages does your method offer?
  2. With the same model size and training data, how does your method compare with decoder-only approaches that reuse precomputed KV caches (e.g., HyperRAG and CompRank), in both effectiveness and efficiency?
  3. How does the model perform at r=1r=1? The text near Equation 9 says that r=1r=1 corresponds to the uncompressed representation. Is r=1r=1 included during training?
  4. Could you explain in more detail how the cost is computed and how much each component contributes to the overall efficiency gain?
5 comments
AuthorsOpenReview ↗

(1/4) Response to Reviewer Tbqz

We sincerely thank the reviewer Tbqz for the constructive comments and suggestions, which are very helpful for improving our paper. We are also grateful that you recognised the strengths of our paper. We have updated the manuscript accordingly and kindly invite the reviewer to review the revised version for details. The main changes are marked in blue. Please find point-by-point responses below.

Weakness 1: Precomputing document representations offline and doing deep interaction online is not a new idea. PreTTR [1] and MORES [2] have already explored similar designs for reranking. However, the paper does not cite them, does not explain how it differs from them, and does not compare against them at the same model size.

Question 1: What is the key difference between FBNL and prior work such as MORES and PreTTR? At the same model size, what advantages does your method offer?

We thank the reviewer for pointing out these important prior works. We agree that PreTTR and MORES have explored offline document representations with deep online interaction. We clarify the differences in architecture, compression, online interaction efficiency, and query processing below.

  • First, the encoding and interaction architectures differ. PreTTR caches lower-layer Transformer representations and uses the remaining upper layers for online joint query–document interaction. MORES separates document representation, query representation, and interaction into three modules. Lychee-Reranker builds on the pretrained T5Gemma2 encoder–decoder: the full encoder produces reusable passage representations offline, and the decoder performs relevance modeling through merged attention.
  • Second, MEP provides parameter-free compression. PreTTR uses learned projections to compress token embedding dimensions and decompresses the representations before online interaction. MORES caches either full-length document token representations or K/V projections for each interaction layer. Lychee-Reranker introduces Matryoshka Embedding Pooling (MEP), which compresses passage representations along the sequence dimension. It requires no additional compression or decompression network, and the same model supports multiple compression ratios.
  • Third, improved online interaction efficiency. PreTTR’s compression preserves the passage sequence length, and MORES’s cross-attention reads the full passage sequence. MEP reduces the passage sequence length from n to ⌈n/r⌉, and the decoder attends directly to this shorter sequence, reducing online attention computation. The compression ratio controls both storage and online computation, enabling flexible test-time compute.
  • Finally, Lychee-Reranker requires no separate query encoder. PreTTR first encodes the query through its shared lower Transformer layers, and MORES uses a dedicated query encoder. Lychee-Reranker feeds the query and instructions directly into the decoder, which models the query context and its fine-grained interaction with the passage.

We have cited and discussed both methods in Appendix J (Additional Related Work on Reranking) of the revised manuscript. We could not find publicly available checkpoints for the original rerankers. The original studies used BERT-based models. Applying these methods to modern LLM architectures would require further adaptation and retraining. At the same model size, we think Lychee-Reranker can offer more effective interaction, more efficient inference, reduced storage overhead, and flexible test-time compute.

AuthorsOpenReview ↗

(2/4) Response to Reviewer Tbqz

Weakness 1: In terms of architecture choice, the authors should also compare with decoder-only approaches, especially those that reuse precomputed document KV caches (e.g., HyperRAG [3] and CompRank [4]), and explain what advantages the encoder–decoder architecture offers over them. Without these comparisons, it is hard to tell whether the gains come from the architecture itself or from MEP and the training pipeline.

Question 2: With the same model size and training data, how does your method compare with decoder-only approaches that reuse precomputed KV caches (e.g., HyperRAG and CompRank), in both effectiveness and efficiency?

We thank the reviewer for this important suggestion. HyperRAG and the concurrent work CompRank explore the reuse of precomputed document KV caches and are closely related to our work. We have discussed both methods in Appendix J (Additional Related Work on Reranking) of the revised manuscript and clarify the following differences.

  • First, avoiding the storage overhead of per-layer document KV caches. HyperRAG stores document K/V across decoder layers. CompRank supports the reuse of precomputed document-side KV states. This can incur substantial storage overhead for large corpora. In contrast, Lychee-Reranker stores only the encoder outputs after MEP compression. The compression ratio further controls cache size.
  • Second, simpler implementation and deployment. HyperRAG uses KV-cache storage and loading together with a static KV layout, while CompRank uses block-structured attention, shared positional offsets, and attention-based scoring. These designs involve KV-cache management or support for structured attention masks, which can increase implementation and deployment complexity. Lychee-Reranker instead uses the encoder to produce passage representations offline and the decoder for online relevance modeling.
  • Third, bidirectional passage encoding. Both methods use causal document encoding in a decoder. Lychee-Reranker first encodes passages bidirectionally, compresses their representations through MEP, and then performs pointwise relevance modeling with the decoder. CompRank uses token selection for compression and attention-based scoring for listwise reranking, which differ from our compression and scoring mechanisms.

With the same model size and training data, we think Lychee-Reranker would be easier to deploy in industrial reranking systems because of its simple yet effective inference workflow. We draw this conclusion based on the above analysis. We could not find publicly released model checkpoints from the original authors for direct evaluation. Figure 9 of the original submission shows the performance gains from MEP. Section 5.3.4 and Table 7 of the revised manuscript additionally report measured online inference latency and throughput.

AuthorsOpenReview ↗

(3/4) Response to Reviewer Tbqz

Weakness2: In the third stage of Section 4.2, the paper says the LoRA adapters are merged back into their backbones. But LoRA fine-tuning is never mentioned earlier in the main text, so this comes as a surprise. Please state in the main text which modules LoRA is applied to and give its main settings.

Weakness 2: Results without compression (r=1) are missing. Performance drops steadily from r=2 to r=16, but no uncompressed result is reported. This makes it impossible to see the performance upper bound or measure the actual loss caused by compression.

Question 3: How does the model perform at r=1? The text near Equation 9 says that r=1 corresponds to the uncompressed representation. Is r=1 included during training?

We thank the reviewer for pointing this out. Due to space constraints, the LoRA training details were originally provided in Appendix D. We have now specified the LoRA target modules and main settings in Section 4.2 of the revised manuscript.

We did not include r = 1 during training, as our goal is to reduce redundancy in passage representations and improve efficiency with minimal loss in reranking effectiveness. We thank the reviewer for pointing out the missing uncompressed evaluation results at r = 1. We have added these results to Tables 19 and 20 of the revised manuscript for BEIR and MIRACL, respectively.

As shown in the table below, average performance at r = 2 is very close to that at r = 1 across all three model sizes and both benchmarks, with differences of at most 0.13 points. This supports our claim that passage representations contain redundancy and that using an appropriate compression ratio can improve efficiency with minimal loss in reranking performance.

ModelCRBEIR Avg.MIRACL Avg.
Lychee-Reranker-Nano158.7471.60
258.7571.63
458.5471.11
858.1770.38
1657.4769.38
3256.7468.02
6455.8866.85
12855.1166.01
Lychee-Reranker-Small161.2174.47
261.1774.34
461.0774.06
860.7473.61
1660.2572.60
3259.5471.81
6458.8470.82
12858.2070.16
Lychee-Reranker-Large163.8075.07
263.6975.12
463.5374.92
863.1574.50
1662.7073.85
3262.1873.26
6461.6072.46
12861.2371.97

CR denotes the compression ratio.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A Brief Follow-Up Before the Discussion Closes

Dear Reviewers,

Thank you for your time and feedback. With about three days left in the discussion period, please let us know if you have any remaining questions or concerns. We will do our best to address them before the discussion closes.

Best regards,
The Authors

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

We sincerely thank Reviewers Tbqz, s8Jx, and E4oB for taking the time to provide thoughtful comments, all of which have significantly improved the quality of our paper.

We introduce FBNL, a reranking paradigm that enables (i) efficient online serving, (ii) expressive relevance modeling, (iii) compact passage representations, and (iv) adjustable test-time compute. Lychee-Reranker achieves performance comparable to strong industrial rerankers. We make all research artifacts publicly available, including model weights, training data, training and inference code, and intermediate checkpoints.

We summarize our main responses to each reviewer below.


Reviewer Tbqz

(i) The reviewer asked us to discuss several closely related rerankers. We expanded the discussion in Appendix J to explain how Lychee-Reranker differs from them.  

(ii) The reviewer asked us to include the LoRA training settings in the main text and report results without compression. We summarized the settings already documented in Appendix D in Section 4.2 and added the r=1r=1 results to Tables 19 and 20.  

(iii) The reviewer asked where the efficiency gains come from. We quantified each component’s contribution to online inference efficiency.


Reviewer s8Jx

(i) The reviewer asked whether Lychee-Reranker generalizes across first-stage retrievers. Our experiments showed that Lychee-Reranker maintains its reranking gains across different first-stage retrievers and remains robust to compression.

(ii) The reviewer asked about practical inference efficiency. Our measurements on a single RTX 5090 show that Lychee-Reranker achieves low latency and high throughput (Table 7 of the revised manuscript).

(iii) The reviewer asked us to clarify our architectural contribution. We explained how FBNL combines offline passage encoding with fine-grained query–passage interaction through merged attention and how MEP enables test-time compute scaling by varying the compression ratio.

(iv) The reviewer asked how Lychee-Reranker compares with E2Rank. We added a detailed comparison on BEIR.


Reviewer E4oB

(i) The reviewer asked how much multi-stage training improves performance. We quantified the gains from the later stages in a stage-by-stage ablation (Table 6 of the original submission).

(ii) The reviewer asked how Lychee-Reranker compares with listwise rerankers. We compared Lychee-Reranker-Large with jina-reranker-v3 and jina-reranker-v3.5. It achieved higher nDCG@10 than both baselines and delivered 2.75× to 3.96× their online document throughput.


We sincerely thank you for coordinating the review. We have done our best to address the reviewers' concerns.

We would be grateful if you could consider our rebuttal and revised manuscript in your final assessment.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.