acceptodds
Desk rejected at ICLR 2027

CompRank: Efficient Long-List LLM Reranking via Token-Level Compression and Decoding-Free Scoring

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) rerankers have become an important component of modern retrieval and retrieval-augmented generation pipelines, yet their computational cost makes reranking long candidate lists expensive. We introduce CompRank, a token-efficient reranking framework that reduces redundant computation through contextualized KV compression and decoding-free attention scoring. CompRank first encodes each document with its full causal context and then exposes only a small subset of the resulting key–value (KV) states to the scoring stage, allowing the retained states to preserve information from the omitted context. We instantiate CompRank in two complementary variants: CompRank-QC, which conditions document representations on the query for stronger relevance modeling, and CompRank-Cache, which uses query-independent document representations to enable cross-query KV reuse. Experiments on seven BEIR datasets show that CompRank-QC achieves an average NDCG@10 of 46.7 with Mistral-7B, outperforming the evaluated efficient reranking baselines, with consistent effectiveness gains also observed on Qwen2.5-7B. Notably, with Mistral-7B, Step-10 compression retains only 10.2% of document KV states while reducing average NDCG@10 by only 0.5 points compared with full-token scoring. It also yields approximately end-to-end speedup at 500 candidates even when documents are re-encoded for each query, while document-side KV reuse further reduces online latency by amortizing document encoding across queries. On TREC-COVID, CompRank further generalizes from 30-document training lists to candidate sets of up to 500 documents. These results demonstrate that contextualized KV compression and reusable document representations provide an effective path toward scalable long-list LLM reranking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reranking, information retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1LadOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

CompRank reduces the document KV states used for attention-based reranking. It encodes documents independently, retains selected contextualized states, and ranks candidates using supervised attention scores. The QC variant conditions document encoding on the query, while the Cache variant supports reuse across queries. The main finding is that regularly spaced KV selection preserves much of the full-token effectiveness.

Critical strengths
  1. Step-10 retains 10.2% of document states while reducing average NDCG@10 from 47.2 to 46.7. This is a useful compression result.

  2. The matched-budget selection experiment provides evidence that distributing retained states across a document works better than concentrating them at its end.

Critical weaknesses
  1. Most of the architecture already exists in BlockRank. Block masking, shared document positions, attention-mass scoring, and temperature-scaled supervision are established components of that method. CompRank acknowledges BlockRank, but the contribution needs to be assessed primarily as a compression extension.

  2. The paper does not isolate that extension through a matched BlockRank-plus-compression experiment.

  3. Query-conditioned encoding is also not a new distinction. BlockRank’s Section 2.1 already examine putting the query before the documents and discuss the accuracy cost of query-independent representations.

  4. CompRank should explain what its QC/Cache formulation adds beyond combining this existing distinction with KV selection and measuring cached execution.

  5. The QC–Cache comparison does not isolate query conditioning. According to Sections 2.1 and 2.4, the comparison changes both query access and instruction placement. Documents in QC see the instruction and query; documents in Cache see neither. A cacheable model with a fixed instruction prefix would separate the effect of removing the query from the effect of removing the instruction.

  6. The document score contains a length-dependent preference. Section 2.3 sums attention over retained states. When token logits are equal, the score becomes (In page 4), so a document with more retained states receives a larger score. Training may compensate, but the paper provides no length-stratified analysis or comparison with length-normalized scoring.

  7. Adding candidates can change the relative ranking of existing documents. Averaging separately normalized attention heads means that a new candidate changes each head’s contribution by a different amount. The paper acknowledges candidate-set dependence but does not test stability under added distractors or duplicate candidates.

  8. Section 2.2 measures importance using attention from the final document token. That measures what this token attends to, which need not identify the evidence useful for ranking. Even with query-conditioned states, poor performance from this proxy does not establish that uniform selection is preferable to selection trained or measured specifically for relevance.

  9. Section 4.1 uses a full-token-trained checkpoint, then removes 90% of its input tokens for Select–Encode. The 2.9-point gap shows that this checkpoint benefits from full-context encoding. It does not establish the same advantage over a model trained to operate on compressed inputs, particularly when comparing computational cost.

  10. Appendix A trains the ranking loss at layer 20 using the [ token. Appendix C then reports that this layer and token perform best. If these measurements use the same trained checkpoint, the result partly reflects where supervision was applied. The authors should identify the checkpoint and selection data, and distinguish readout inspection from independently training alternative readouts.

  11. With only the stated loss at layer 20, later Transformer blocks cannot affect the ranking score and receive no gradient from that loss. Does training and inference stop at the scoring layer, or execute the full model? This affects the interpretation of “full-parameter fine-tuning” and the fairness of comparisons with methods that execute all layers.

  12. The experiments omit the comparisons most relevant to the contribution. BlockRank appears in the effectiveness table but not in the latency study. PE-Rank and Compress-then-Rank are cited without experimental comparison. TourRank and DeAR are absent from the discussion, despite their relevance to alternative ranking strategies and trained rerankers.

  13. The evidence for long-list generalization is narrow. Scaling from 30 to 500 candidates is tested only on TREC-COVID’s 50 queries. The main evaluation uses BM25 candidates, so it does not establish the same behavior with different candidate distributions or denser hard negatives. The omission of DL19/20 and several datasets used in RankGPT also leaves benchmark coverage less complete than the surrounding literature.

  14. The average Mistral gain over BlockRank is 1.2 points, with gains of only 0.1 on DBPedia and 0.3 on NFCorpus. No confidence intervals or variation across training runs are reported. The five sampling seeds in Section 4.1 concern randomized selectors with a fixed checkpoint; they do not establish the reliability of the main model improvements.

  15. Figure 2 reports 78.9 on TREC-COVID at 100 candidates, whereas Table 4 reports 80.5 for full-token QC under the main top-100 evaluation. Table 2 also uses a different full-token average from Table 4. Different checkpoints or settings may explain these values, but they should be identified explicitly so that the compression comparisons can be reconstructed.

Questions for authors

see Weaknesses above

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1Lad (1/3)

We thank the reviewer for the detailed comparison with BlockRank and the constructive suggestions regarding methodological contributions, controlled comparisons, and experimental reliability. We address these concerns with additional experiments and clarifications below.

1. Methodological Contributions and Matched Compression Comparisons (W1–W5)

Contributions beyond BlockRank (W1, W3, W4). CompRank builds upon BlockRank's block-decoupled encoding, but further reduces the KV states involved in relevance scoring through post-encoding contextualized KV compression. With Step-10, CompRank-QC retains only 10.2% of document states while achieving 46.7 average BEIR nDCG@10, compared with 47.2 under full-token scoring.

We further introduce two complementary configurations: CompRank-QC prioritizes relevance modeling through query-conditioned representations, while CompRank-Cache enables reusable query-independent document KV states. The latter reduces online latency to 1.231 s/query at 500 candidates, excluding offline cache construction.

Query-conditioned encoding itself is not claimed as a standalone novelty. We will clarify BlockRank's prior use of query prefixes and emphasize the contributions of contextualized KV compression, direct ranking supervision, and the effectiveness–reuse trade-off.

Matched BlockRank compression (W2). We additionally apply Step-10 to a BlockRank-style model trained with the joint next-token prediction and ranking objective. Under the same checkpoint, BEIR nDCG@10 changes from 45.8 to 45.3, showing that contextualized KV compression also benefits the existing architecture.

We further measure effectiveness and end-to-end latency on TREC-COVID using matched Mistral-7B settings, candidate lists, truncation policies, A100 80GB hardware, and batch size 1, without cross-query KV reuse.

Method100 docs: nDCG@10100 docs: s/query500 docs: nDCG@10500 docs: s/query
BlockRank, Full KV79.23.4577.016.80
BlockRank, Step-10 KV78.62.9876.712.60
CompRank-QC, Step-10 KV81.12.9279.512.00

At 500 candidates, Step-10 reduces BlockRank latency by 25%, with only a 0.3-point nDCG@10 decrease. CompRank-QC further improves nDCG@10 by 2.8 points over compressed BlockRank while being approximately 1.05× faster.

Isolating query and instruction placement (W5). We additionally train three matched Step-10 variants, keeping the scoring suffix and document positional settings fixed. All variants retain the same static instruction in the scoring suffix; only the information accessible during document encoding changes.

Information accessible during document encodingBEIR nDCG@10
Static instruction + query46.1
Static instruction only40.5
Neither instruction nor query39.4

The latter two configurations permit query-independent document caching. The 46.1 versus 40.5 comparison isolates the benefit of query conditioning, while 40.5 versus 39.4 isolates the additional contribution of the static instruction prefix. These are matched ablation variants, distinct from the submitted QC and Cache configurations.

AuthorsOpenReview ↗

Response to Reviewer 1Lad (2/3)

2. Scoring, Token Selection, and Ranking Stability (W6–W9)

Document-length bias (W6). Attention-mass aggregation can favor documents with more retained states. Under equal token-level logits, the score is proportional to the retained-state count.

We evaluate same-query, same-relevance-grade document pairs with different retained-state counts. Documents with more retained states receive higher scores in 57% of comparisons (ties counted as half). Normalizing scores by retained-state count, s~i=si/mi\tilde{s}_i=s_i/m_i, reduces this preference to 51%.

We also evaluate ranking effectiveness across query groups defined by the median truncated length of their candidate documents:

Scoring methodShortMediumLongOverall BEIR
Original attention mass45.846.747.646.7
Normalized by retained-state count45.946.547.446.6

Normalization substantially reduces the observed state-count preference but does not improve overall nDCG@10. We will report this sensitivity explicitly.

Ranking stability under candidate perturbations (W7). Although document encodings are candidate-independent, final scores depend on the candidate set. We therefore test ranking stability by augmenting the same 100 TREC-COVID candidates with either 50 sampled distractors or duplicate copies of 10 existing documents.

Ranking consistency is measured only over the original document IDs, without treating duplicates as additional relevant documents.

PerturbationKendall's τTop-10 overlap
Add 50 distractors0.9186%
Add 10 duplicates0.9489%

These results demonstrate substantial ranking stability under both perturbations, while confirming that candidate-set changes can affect relative scores.

Attention-based token selection (W8). The final document token's attention is used only as an intrinsic importance heuristic for KV selection, not as the final relevance score. Table 2 shows that, at 10% retention, stratified importance selection improves nDCG@10 from 44.3 (global importance) to 46.1, while Step-wise selection reaches 46.7. This supports the importance of distributed document coverage for the evaluated selectors.

Encode–Select versus Select–Encode (W9). The controlled experiment uses the same full-token-trained checkpoint, selected positions, and 10% retention budget. Encode–Select achieves 46.7, compared with 43.8 for Select–Encode. The 2.9-point difference isolates the benefit of preserving the omitted tokens' causal context under fixed model parameters. A separately trained Select–Encode model is outside the scope of this particular comparison.

3. Scoring Configuration and Training Computation (W10–W11)

Scoring-layer and decision-token selection (W10). As specified in Appendix A, we use zero-based layer index 20 (the 21st Transformer block) and the first output token ([) for scoring. The scoring layer was selected using 500 held-out MS MARCO validation examples, while the decision token and Step-10 stride were selected through training-side experiments rather than BEIR test-set tuning.

Appendix C reports the layer and decision-token attention-accuracy analyses. We will clarify the corresponding checkpoint and selection protocols, distinguishing the observed attention profiles from comparisons between independently trained configurations.

Gradient flow and execution depth (W11). The ranking loss is computed from attention-based scores at layer 20. Under the ranking-only objective, Transformer blocks after this layer receive no ranking-loss gradients. For joint training objectives, gradient flow also depends on the next-token prediction loss.

At inference, CompRank terminates the forward pass at the scoring layer, without executing subsequent Transformer blocks or autoregressive identifier decoding. We will clarify the distinction between trainable parameters, parameters receiving gradients, and inference execution depth.

AuthorsOpenReview ↗

Response to Reviewer 1Lad (3/3)

4. Additional Baselines and Benchmark Coverage (W12–W13)

Related methods (W12). Our matched BlockRank compression experiment provides a direct test of the benefit of post-encoding KV selection within an existing efficient reranking architecture.

We will also expand the discussion of related approaches, including PE-Rank's embedding-based passage representation, Compress-then-Rank's passage compression, and the alternative ranking strategies of TourRank and DeAR. These methods explore complementary design choices, whereas CompRank retains contextualized token-level KV states for decoding-free attention scoring.

Additional benchmarks and long-list evaluation (W13). We extend the evaluation to TREC Deep Learning 2019 and 2020 using the same BM25 top-100 candidates.

MethodTREC DL19TREC DL20
BlockRank, Full KV70.869.7
CompRank-QC, Step-1071.270.0

We further examine long-list generalization under different retrieval candidate sources:

Dataset / candidate source100 candidates500 candidates
TREC DL20 / BM2570.069.3
TREC-COVID / Dense retrieval77.676.9

CompRank-QC exhibits only a 0.7-point decrease when scaling from 100 to 500 candidates in both settings. These results complement the original TREC-COVID scaling experiment and broaden the evaluation beyond the seven BEIR datasets. We do not treat TREC DL19/20 as fully independent out-of-domain evaluations relative to MS MARCO-derived training.

5. Statistical Reliability and Result Consistency (W14–W15)

Statistical reliability (W14). We additionally examine uncertainty in the matched training-objective ablation. Ranking-only Step-10 achieves 46.1, compared with 45.8 for the joint-objective full-KV reference, a difference of +0.3 points.

Using paired query-level bootstrap within each dataset, followed by the original seven-dataset macro-averaging procedure, the estimated 95% confidence interval is [−0.2, +0.8]. Across three training seeds, the paired differences are +0.1, +0.3, and +0.5.

These results support the conclusion that Step-10 preserves competitive ranking effectiveness with substantially fewer KV states, rather than establishing a statistically significant effectiveness gain.

Full-token result consistency (W15). The full-token references of 46.8 in Table 2 and 47.2 in Table 4 correspond to different experimental settings:

  • Table 2: A fixed full-token-trained checkpoint used for matched inference-time compression ablations.
  • Table 4: The full-token reference from the main compression comparison, where configurations are trained separately.

The two values are therefore not repeated measurements of the same checkpoint.

For Figure 2, the correct Mistral-7B CompRank-QC Full-KV nDCG@10 at 100 candidates is 80.5, as reported in Table 4. The plotted 78.9 is an error and will be corrected.

We will also clarify that Figure 2 uses Mistral-7B Direct-List trained on 30-document lists, while Table 1 uses its default Qwen2.5-7B, 20-document training configuration. This explains the Direct-List scores of 74.2 and 77.4, respectively, without conflating the two experimental settings.

Reviewer_FFeTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces CompRank, which is a token-efficient reranking framework that reduces redundant computation through contextualized KV compression and decoding-free attention scoring. The compression (CompRank-Cache) is query independent and done offline., however CompRank-QC is query dependent. BEIR benchmark is used for evaluation and results show that caching can speedup the reranking by 1.4 times while maintaining the ranking performance.

Critical strengths

S1. The paper tries to tackle the one of the major limitations of LLM based rerankers, mainly in listwise setting. Since LLM ranking depends on the processing of the input tokens. This work proposes the (q,d) or (d only) encodings, which results in the latency gains.

S2. The compression cost a little in ranking performance. The Step-10 variant of proposed method that uses only 10% of the document states achieve almost same ranking performance. The experiments also shows the caching and ranking trade-off.

S3. The compression enables it to process upto 500500 documents in single LLM pass, where direct LLM de-grades as number of documents scales.

Critical weaknesses

W1. The setup is more like a pointwise reranker which follows PRP (Probability Ranking Principle), where relevance of one document is independent of other documents, however the listwise ranking goes beyond PRP. With block-diagonal masking and shared positional offsets, each document attends only to I(q) and its own context. Encoding is therefore equivalent to N independent passes over [I(q);Di][I(q); D_i] with a cached prefix, which is the prefill of a pointwise reranker. Cross-candidate interaction is confined to the short suffix. Therefore, the 4.9–9.5× speedup over Direct-List is a decoding vs. no-decoding gap, already established by ICR, FIRST and BlockRank. This also makes novelty limited. Given this cost structure, the informative baselines are a same-size pointwise reranker with prefix caching (e.g., RankLLaMA-7B) and BlockRank/FIRST, and no latency is reported for any of them. Further, document states are independent of N, so stability from 30 to 500 candidates largely follows from the architecture; any pointwise model extrapolates trivially. Direct-List is also run in a single pass over up to 500 candidates (~115–128k tokens, apparently on Qwen2.5-7B with 32k native context) rather than with the standard sliding window. Its degradation, and the error analysis in App. D, therefore reflect the baseline setup rather than a limit of identifier decoding.

W2. Experimental settting: Baselines differ across backbones, with BlockRank evaluated only on Mistral and Direct-List only on Qwen. The fine-tuned CompRank is compared mostly against zero-shot methods, and RankGPT-Mistral even falls below BM25, while established fine-tuned rerankers (RankZephyr, FIRST, RankLLaMA) are missing. In Fig. 2, Mistral-CompRank against Qwen-Direct-List, and its numbers disagree with the tables (full-token TREC-COVID 78.9 vs. 80.5; Direct-List 74.2 vs. 77.4). The scoring layer, decision token and stride are selected on unspecified data. Finally, CompRank-Cache underperforms its own BM25 input on three of seven datasets, which undercuts its value as an operating point.

W3 Evaluation is limited to BEIR benchmark. The paper should add standard benchmarks such as TREC DL19, 20. I was hoping to see to the length of the documents for each dataset, but paper does not provide such information. Also, experiments does not show any correlation between compression and document length.

Questions for authors
  1. Is Step-10 compression applied during training, inference, or both? Are the Table 4 rows separate checkpoints?
  2. Which backbone does each curve in Fig. 2 use, and why does full-token TREC-COVID at 100 documents differ between Fig. 2 (78.9) and Table 4 (80.5)?
  3. On which data were the layer, decision token, and stride selected?
  4. What accounts for the 1.7-point full-token gain over BlockRank?
  5. Does the forward pass stop at the scoring layer?
4 comments
AuthorsOpenReview ↗

Response to Reviewer FFeT (1/2)

We thank the reviewer for recognizing the effectiveness of contextualized KV compression and for the constructive suggestions. We provide additional controlled experiments and clarify the experimental configurations below.

W1. Pointwise-like Encoding and Efficiency Comparisons

CompRank adopts block-decoupled document encoding, where each document attends to the shared prefix and its own context. However, independent document encoding does not imply independent ranking: the trailing scoring block attends to all candidates, and their attention-derived scores are jointly normalized over the candidate set. Thus, CompRank combines efficient independent encoding with candidate-set-dependent scoring.

The efficiency gains arise from complementary mechanisms. At 500 candidates, Step-10 compression reduces CompRank-QC latency from 16.22 to 12.00 s/query (1.35×) without KV reuse. CompRank-Cache further reduces online latency to 1.231 s/query through cross-query reuse, excluding offline encoding. The originally reported 9.5× speedup compares against full-permutation Direct-List and includes differences in decoding cost.

To provide more informative comparisons, we additionally evaluate a prefix-cached 7B pointwise reranker and BlockRank under matched Mistral-7B settings. All methods use the same TREC-COVID candidates, document truncation policy, A100 80GB GPU, batch size 1, and synchronized end-to-end timing, without cross-query KV reuse.

Method100 docs: nDCG@10100 docs: s/query500 docs: nDCG@10500 docs: s/query
Pointwise-7B, prefix cache79.03.0877.513.40
BlockRank, Full KV79.23.4577.016.80
BlockRank, Step-10 KV78.62.9876.712.60
CompRank-QC, Step-10 KV81.12.9279.512.00

At 500 candidates, CompRank-QC achieves 2.0 and 2.8 points higher nDCG@10 than pointwise reranking and compressed BlockRank, respectively, while reducing latency by 10.4% and 4.8%.

We also evaluate two practical Direct-List variants, as detailed in our response to Reviewer R8gf:

Method500 docs: nDCG@10500 docs: s/query
Direct-List, sliding window79.020.80
Direct-List, top-10-only78.719.30
CompRank-QC, Step-10 KV79.512.00

Sliding-window Direct-List uses 30-document windows with stride 15, including all window calls and ranking updates. The top-10-only variant processes all candidates without truncation and generates only ten identifiers.

CompRank is 1.73× faster than sliding-window reranking and 1.61× faster than top-10-only generation, while maintaining higher nDCG@10. These results demonstrate that CompRank's advantages persist against both structurally similar rerankers and practical generative alternatives.

W2. Baseline Fairness and Experimental Consistency

The Direct-List results in Table 1 and Figure 2 correspond to different experimental configurations, which we will explicitly distinguish in the revision:

  • Table 1: Default Direct-List configuration, using Qwen2.5-7B trained on 20-document lists, achieving 77.4 nDCG@10 on TREC-COVID.
  • Figure 2: Mistral-7B Direct-List trained on 30-document lists to match CompRank, achieving 74.2 nDCG@10 at 100 candidates.

Thus, the difference reflects different backbones and training-list sizes, rather than inconsistent results from the same model.

Table 1 reports baselines within their respective backbone groups, while our additional Mistral-7B experiments provide training-matched pointwise and BlockRank comparisons. The latter directly address the reviewer's concern about comparisons with fine-tuned rerankers.

Regarding CompRank-Cache, its purpose is to provide a query-independent operating point with reusable document representations. Although its effectiveness varies across datasets, the measured 1.231 s/query online latency at 500 candidates demonstrates the practical benefit of KV reuse for repeated-query workloads.

AuthorsOpenReview ↗

Response to Reviewer FFeT (2/2)

W3. Benchmark Coverage and Document-Length Sensitivity

Additional benchmarks. We extend evaluation to TREC DL19 and DL20 using the same top-100 candidate protocol and Mistral-7B backbone.

MethodTREC DL19TREC DL20
BlockRank70.869.7
CompRank-QC, Step-1071.270.0

CompRank improves nDCG@10 by 0.4 and 0.3 points, respectively, providing additional evidence beyond the original seven BEIR datasets.

Document-length sensitivity. We group queries into short, medium, and long categories according to the median truncated length of their candidate documents, and macro-average results across the seven BEIR datasets.

Compression impactShortMediumLong
Step-10 minus Full KV (nDCG@10)−0.6−0.3−0.2

Compression preserves most ranking effectiveness across all three groups, with the observed degradation ranging from 0.2 to 0.6 points. We will additionally report per-dataset document-length and retained-KV distributions, including median and 90th-percentile statistics, to clarify the relationship between document length and compression.

Q1. Is Step-10 applied during training, inference, or both? Are Table 4 rows separate checkpoints?

Step-10 is applied during both training and inference in the main configuration. Each compression configuration in Table 4 is trained separately with its own checkpoint.

In contrast, Table 2 uses a single full-token-trained checkpoint and varies only inference-time selection. This controlled experiment isolates compression effects: full KV achieves 46.8, while retaining 10% of states achieves 46.7.

Q2. Which backbones are used in Figure 2, and why do the results differ from Tables 1 and 4?

All Figure 2 curves use Mistral-7B. Direct-List is trained on 30-document lists, matching CompRank, whereas Table 1 uses the default Qwen2.5-7B Direct-List trained on 20-document lists. This explains the 74.2 versus 77.4 difference.

For CompRank Full-KV, the correct TREC-COVID nDCG@10 at 100 candidates is 80.5, as reported in Table 4. The 78.9 shown in Figure 2 is a plotting error, which we will correct. We will also explicitly label the backbone and training configuration for each curve.

Q3. On which data were the scoring layer, decision token, and stride selected?

The scoring layer was selected using 500 held-out MS MARCO examples. Layer-wise evaluation identified zero-based layer index 20 (the 21st Transformer block), as analyzed in Appendix C.1.

The decision token and compression stride were determined through training-side experiments rather than BEIR test-set tuning. Appendix C.2 compares different readout positions, with the first output token ([) providing the strongest attention-based ranking signal. We adopt Step-10 as the default compression strategy and keep the scoring configuration fixed across evaluation datasets.

Q4. What accounts for the 1.7-point full-token gain over BlockRank?

A key difference is direct supervision of attention-derived relevance scores. To isolate this factor, we conducted matched Mistral-7B experiments with fixed training data, candidate lists, architecture, prompts, scoring layer, and attention readout.

Training objectiveFull KVStep-10 KVCompression impact
Next-token prediction only43.843.4−0.4
Next-token prediction + ranking loss45.845.3−0.5
Ranking loss only46.446.1−0.3

Adding ranking supervision improves full-KV nDCG@10 by 2.0 points, while ranking-only training provides a further 0.6-point gain over the joint objective. Furthermore, ranking-only training exhibits the smallest compression-induced reduction (−0.3).

These controlled results identify direct ranking supervision as an important contributor to CompRank's effectiveness. The original 47.2 versus 45.5 comparison additionally reflects differences between the complete method configurations, rather than isolating the training objective alone.

Q5. Does the forward pass stop at the scoring layer?

Yes. During inference, CompRank terminates the forward pass at the designated scoring layer (zero-based index 20 for Mistral-7B). Subsequent Transformer layers and autoregressive identifier decoding are not needed for ranking. This early-exit design contributes to the measured inference efficiency.

AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer FFeT

Dear Reviewer FFeT,

We sincerely thank you for your constructive feedback. In response to your concerns, we conducted additional experiments and provided clarifications:

  • (1) Methodological novelty and efficiency (W1): Under matched settings, CompRank outperforms pointwise and BlockRank baselines by 2.0/2.8 nDCG@10 points, while achieving 1.73×/1.61× speedups over practical Direct-List variants.
  • (2) Baseline fairness and consistency (W2): We clarified backbone differences, corrected a plotting error, and provided controlled experiments demonstrating the benefits of direct ranking supervision (+2.6 nDCG@10).
  • (3) Generalization and compression (W3): Additional TREC DL19/DL20 experiments confirm consistent improvements, while document-length analysis shows minimal compression degradation (0.2–0.6 nDCG@10 points).
  • (4) Implementation details (Q1–Q5): We clarified compression training, checkpoint configurations, hyperparameter selection, and early-exit inference.

We hope these additional results and clarifications address your concerns and would sincerely appreciate your reconsideration of the score in light of the new evidence.

Thank you again for your time and feedback.

Reviewer_FFeTOpenReview ↗

Response

I thank the authors for detailed rebuttal and experiments. I increase my score to 3.

Reviewer_R8gfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CompRank, an efficient LLM reranking framework combining contextualized KV-state compression and decoding-free attention-based scoring. It introduces query-conditioned and query-independent variants to balance ranking effectiveness and reusable document computation. Experiments across seven BEIR datasets demonstrate competitive effectiveness and reduced inference latency.

Critical strengths
  1. Effective contextualized KV compression. CompRank preserves most ranking effectiveness while retaining approximately 10% of document KV states. Controlled selection experiments support the benefits of distributed contextualized representations.

  2. Practical efficiency evaluation. The paper provides measured latency across different candidate-list sizes and explores document-state caching for cross-query reuse.

Critical weaknesses
  1. The independent methodological contribution is insufficiently isolated.

Several components build on established methods, including block-decoupled encoding, attention-based scoring, and token selection. The improvements over strong baselines are relatively modest, and the paper lacks sufficiently controlled comparisons isolating query conditioning, scoring supervision, and compression from the existing architecture.

  1. Limited fairness of the long-list comparison.

The Direct-List baseline is trained on 30-document lists but evaluated on lists containing up to 500 documents, requiring complete identifier permutations. The appendix confirms frequent invalid outputs under this extrapolation. Since evaluation focuses on nDCG@10, comparisons with practical window-based or top-k-only reranking approaches would provide stronger evidence of long-list superiority.

Furthermore, document scores aggregate attention mass over retained states, potentially introducing document-length bias. This effect is not directly evaluated.

Questions for authors
  1. Can the authors isolate the contribution of CompRank's scoring and compression mechanisms under matched training and architectural settings?

  2. Can the authors clarify how the long-list efficiency advantage compares with window-based or top-k-only reranking?

  3. How sensitive is attention-mass scoring to document length and the number of retained KV states?

3 comments
AuthorsOpenReview ↗

Response to Reviewer R8gf (1/2)

We thank the reviewer for recognizing the effectiveness of contextualized KV compression and our practical efficiency evaluation. We address these concerns with additional controlled experiments below.

1. Isolating the Contributions of Scoring, Compression, and Query Conditioning (Weakness 1; Question 1)

We agree that block-decoupled encoding and attention-based scoring are established techniques. Our contribution lies in combining full-context document encoding followed by selective KV exposure with direct supervision of decoding-free attention-based relevance scores, while exploring the trade-off between query-conditioned and query-independent document representations.

Compression and contextualization. Table 2 already provides a controlled comparison using a single full-token-trained Mistral-7B CompRank-QC checkpoint. Full-KV scoring achieves 46.8 average nDCG@10 across seven BEIR datasets, while retaining only 10% of document states through step-wise selection achieves 46.7. Under the same retention budget, unrestricted random selection obtains 44.8.

Furthermore, encoding the full document before retaining selected states achieves 46.7, compared with 43.8 when unselected tokens are removed before encoding. This 2.9-point difference supports the importance of preserving contextual information in retained KV states.

These controlled results are distinct from the configuration comparison in Table 4, where full-token scoring achieves 47.2 and Step-10 achieves 46.7 with 10.2% state retention.

Scoring supervision. We additionally conducted matched Mistral-7B experiments on the same seven BEIR datasets. We fixed the candidate lists, training data and budget, prompts, attention-based readout, and scoring layer. Within each row, full-KV and Step-10 evaluation use the same trained checkpoint, and the document encoder processes all tokens within the shared truncation limit.

Training objectiveFull KVStep-10 KVCompression impact
Next-token prediction only43.843.4−0.4
Next-token prediction + ranking loss (BlockRank-style joint objective)45.845.3−0.5
Ranking loss only (CompRank-style objective)46.446.1−0.3

Adding ranking supervision to next-token prediction improves full-KV nDCG@10 from 43.8 to 45.8 (+2.0 points). Using ranking loss alone further improves performance to 46.4 (+0.6 points over the joint objective and +2.6 points over next-token prediction alone).

Importantly, all three training objectives remain robust to inference-time KV compression, with performance reductions of only 0.3–0.5 points. The ranking-only objective achieves the smallest compression-induced decrease (−0.3 points), suggesting that directly optimizing the attention-derived relevance scores is compatible with aggressive KV compression.

These are additional matched-ablation runs, rather than the checkpoints used in Tables 2 and 4. Thus, we do not directly equate their absolute scores with those in the original compression experiments. The comparison isolates the effect of ranking supervision under matched architectural settings, although it does not independently establish the superiority of attention-mass scoring over every alternative readout function.

Query conditioning. We additionally examine query placement while keeping the instruction wording unchanged. Allowing the document encoder to access the query prefix improves average nDCG@10 from 40.5 to 46.1, supporting the benefit of query-conditioned document representations.

Together, these experiments clarify the respective contributions of contextualized KV compression, direct ranking supervision, and query conditioning, rather than attributing all improvements to a single architectural component.

AuthorsOpenReview ↗

Response to Reviewer R8gf (2/2)

2. Fair Comparison with Practical Long-List Reranking (Weakness 2; Question 2)

We agree that requiring a model trained on 30-document lists to generate 500 identifiers introduces extrapolation challenges. Invalid outputs in this setting do not imply an inherent limitation of identifier-based decoding.

The reported 9.5× speedup applies specifically to full-permutation Direct-List. For a more practical comparison, we evaluate:

  • Sliding-window Direct-List: Uses 30-document windows with stride 15, including all calls and ranking updates in latency.
  • Top-10-only Direct-List: Processes the full candidate list but generates only the top 10 identifiers, avoiding full-permutation decoding.

All methods use TREC-COVID, the same Mistral-7B model family, candidates and ordering, truncation policy, A100 80GB GPU, batch size 1, and synchronized end-to-end timing. None uses cross-query KV reuse. The 500-candidate top-10-only result is reported only when the full list fits the context and output budget without truncation.

Method100 candidates: nDCG@10100 candidates: s/query500 candidates: nDCG@10500 candidates: s/query
Direct-List, sliding window79.44.5479.020.80
Direct-List, top-10-only79.24.1878.719.30
CompRank-QC, Step-10 KV81.12.9279.512.00

At 500 candidates, CompRank-QC is 1.73× faster than sliding-window and 1.61× faster than top-10-only, with nDCG@10 gains of 0.5 and 0.8 points, respectively. At 100 candidates, it also achieves the highest nDCG@10 (81.1) and lowest latency (2.92 s/query).

Thus, CompRank retains effectiveness and efficiency advantages over practical generative reranking strategies. We will distinguish these results from the original 9.5× speedup, which involves full-permutation generation under substantial train–test list-length extrapolation.

3. Sensitivity to Document Length and Retained KV States (Weakness 2; Question 3)

We agree that attention-mass aggregation may favor documents with more retained states. Under equal token-level attention logits, a document retaining mim_i states receives a score proportional to its state count:

si=mi∑jmj.s_i = \frac{m_i}{\sum_j m_j}.

We conducted additional experiments to quantify this bias and assess count normalization.

Controlled pairwise analysis. For each query, we compare labeled document pairs with equal relevance grades but different retained-state counts. Under the submitted Mistral-7B CompRank-QC Step-10 configuration, the document with more states receives a higher score in 57% of comparisons (ties counted as half).

We normalize each score by its retained-state count:

s~i=simi,\widetilde{s}_i = \frac{s_i}{m_i},

where mim_i denotes the number of retained states. This reduces the preference from 57% to 51%, close to the unbiased 50% level.

Ranking effectiveness across document-length groups. Within each dataset, we divide queries into three equal-sized groups by median truncated candidate-document length, then macro-average results across seven BEIR datasets.

Scoring methodShortMediumLongOverall BEIR
Original attention-mass scoring45.846.747.646.7
Attention mass / retained-state count45.946.547.446.6

All results use the submitted Mistral-7B CompRank-QC Step-10 configuration.

Normalization substantially reduces state-count preference but changes overall BEIR nDCG@10 by only −0.1 points (46.7 to 46.6). The short-document group gains 0.1 points, while medium and long groups each lose 0.2 points.

Thus, retained-state count influences attention-mass scores, but count normalization does not consistently improve ranking effectiveness. This suggests that the relationship between state count and relevance information cannot be fully addressed by uniform normalization.

AuthorsOpenReview ↗

Summary of Our Rebuttal to Reviewer R8gf

Dear Reviewer R8gf,

We sincerely thank you for your constructive feedback. In response to your concerns, we conducted additional experiments and provided clarifications:

  • (1) Methodological contributions (W1/Q1): Controlled ablations confirm the benefits of ranking supervision (+2.6 nDCG@10), query conditioning (+5.6), and contextualized KV compression.
  • (2) Fair long-list comparisons (W2/Q2): CompRank achieves 1.73×/1.61× speedups over sliding-window/top-10-only baselines, with higher nDCG@10.
  • (3) Document-length bias (W2/Q3): Count normalization reduces state-count preference from 57% to 51%, with negligible impact (−0.1 nDCG@10) on ranking effectiveness.

We hope these additional results address your concerns and would sincerely appreciate your reconsideration of the score in light of the new evidence.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.