- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
CompRank reduces the document KV states used for attention-based reranking. It encodes documents independently, retains selected contextualized states, and ranks candidates using supervised attention scores. The QC variant conditions document encoding on the query, while the Cache variant supports reuse across queries. The main finding is that regularly spaced KV selection preserves much of the full-token effectiveness.
Critical strengths
-
Step-10 retains 10.2% of document states while reducing average NDCG@10 from 47.2 to 46.7. This is a useful compression result.
-
The matched-budget selection experiment provides evidence that distributing retained states across a document works better than concentrating them at its end.
Critical weaknesses
-
Most of the architecture already exists in BlockRank. Block masking, shared document positions, attention-mass scoring, and temperature-scaled supervision are established components of that method. CompRank acknowledges BlockRank, but the contribution needs to be assessed primarily as a compression extension.
-
The paper does not isolate that extension through a matched BlockRank-plus-compression experiment.
-
Query-conditioned encoding is also not a new distinction. BlockRank’s Section 2.1 already examine putting the query before the documents and discuss the accuracy cost of query-independent representations.
-
CompRank should explain what its QC/Cache formulation adds beyond combining this existing distinction with KV selection and measuring cached execution.
-
The QC–Cache comparison does not isolate query conditioning. According to Sections 2.1 and 2.4, the comparison changes both query access and instruction placement. Documents in QC see the instruction and query; documents in Cache see neither. A cacheable model with a fixed instruction prefix would separate the effect of removing the query from the effect of removing the instruction.
-
The document score contains a length-dependent preference. Section 2.3 sums attention over retained states. When token logits are equal, the score becomes (In page 4), so a document with more retained states receives a larger score. Training may compensate, but the paper provides no length-stratified analysis or comparison with length-normalized scoring.
-
Adding candidates can change the relative ranking of existing documents. Averaging separately normalized attention heads means that a new candidate changes each head’s contribution by a different amount. The paper acknowledges candidate-set dependence but does not test stability under added distractors or duplicate candidates.
-
Section 2.2 measures importance using attention from the final document token. That measures what this token attends to, which need not identify the evidence useful for ranking. Even with query-conditioned states, poor performance from this proxy does not establish that uniform selection is preferable to selection trained or measured specifically for relevance.
-
Section 4.1 uses a full-token-trained checkpoint, then removes 90% of its input tokens for Select–Encode. The 2.9-point gap shows that this checkpoint benefits from full-context encoding. It does not establish the same advantage over a model trained to operate on compressed inputs, particularly when comparing computational cost.
-
Appendix A trains the ranking loss at layer 20 using the [ token. Appendix C then reports that this layer and token perform best. If these measurements use the same trained checkpoint, the result partly reflects where supervision was applied. The authors should identify the checkpoint and selection data, and distinguish readout inspection from independently training alternative readouts.
-
With only the stated loss at layer 20, later Transformer blocks cannot affect the ranking score and receive no gradient from that loss. Does training and inference stop at the scoring layer, or execute the full model? This affects the interpretation of “full-parameter fine-tuning” and the fairness of comparisons with methods that execute all layers.
-
The experiments omit the comparisons most relevant to the contribution. BlockRank appears in the effectiveness table but not in the latency study. PE-Rank and Compress-then-Rank are cited without experimental comparison. TourRank and DeAR are absent from the discussion, despite their relevance to alternative ranking strategies and trained rerankers.
-
The evidence for long-list generalization is narrow. Scaling from 30 to 500 candidates is tested only on TREC-COVID’s 50 queries. The main evaluation uses BM25 candidates, so it does not establish the same behavior with different candidate distributions or denser hard negatives. The omission of DL19/20 and several datasets used in RankGPT also leaves benchmark coverage less complete than the surrounding literature.
-
The average Mistral gain over BlockRank is 1.2 points, with gains of only 0.1 on DBPedia and 0.3 on NFCorpus. No confidence intervals or variation across training runs are reported. The five sampling seeds in Section 4.1 concern randomized selectors with a fixed checkpoint; they do not establish the reliability of the main model improvements.
-
Figure 2 reports 78.9 on TREC-COVID at 100 candidates, whereas Table 4 reports 80.5 for full-token QC under the main top-100 evaluation. Table 2 also uses a different full-token average from Table 4. Different checkpoints or settings may explain these values, but they should be identified explicitly so that the compression comparisons can be reconstructed.
Questions for authors
see Weaknesses above
3 comments
Response to Reviewer 1Lad (2/3)
2. Scoring, Token Selection, and Ranking Stability (W6–W9)
Document-length bias (W6). Attention-mass aggregation can favor documents with more retained states. Under equal token-level logits, the score is proportional to the retained-state count.
We evaluate same-query, same-relevance-grade document pairs with different retained-state counts. Documents with more retained states receive higher scores in 57% of comparisons (ties counted as half). Normalizing scores by retained-state count, , reduces this preference to 51%.
We also evaluate ranking effectiveness across query groups defined by the median truncated length of their candidate documents:
| Scoring method | Short | Medium | Long | Overall BEIR |
|---|---|---|---|---|
| Original attention mass | 45.8 | 46.7 | 47.6 | 46.7 |
| Normalized by retained-state count | 45.9 | 46.5 | 47.4 | 46.6 |
Normalization substantially reduces the observed state-count preference but does not improve overall nDCG@10. We will report this sensitivity explicitly.
Ranking stability under candidate perturbations (W7). Although document encodings are candidate-independent, final scores depend on the candidate set. We therefore test ranking stability by augmenting the same 100 TREC-COVID candidates with either 50 sampled distractors or duplicate copies of 10 existing documents.
Ranking consistency is measured only over the original document IDs, without treating duplicates as additional relevant documents.
| Perturbation | Kendall's τ | Top-10 overlap |
|---|---|---|
| Add 50 distractors | 0.91 | 86% |
| Add 10 duplicates | 0.94 | 89% |
These results demonstrate substantial ranking stability under both perturbations, while confirming that candidate-set changes can affect relative scores.
Attention-based token selection (W8). The final document token's attention is used only as an intrinsic importance heuristic for KV selection, not as the final relevance score. Table 2 shows that, at 10% retention, stratified importance selection improves nDCG@10 from 44.3 (global importance) to 46.1, while Step-wise selection reaches 46.7. This supports the importance of distributed document coverage for the evaluated selectors.
Encode–Select versus Select–Encode (W9). The controlled experiment uses the same full-token-trained checkpoint, selected positions, and 10% retention budget. Encode–Select achieves 46.7, compared with 43.8 for Select–Encode. The 2.9-point difference isolates the benefit of preserving the omitted tokens' causal context under fixed model parameters. A separately trained Select–Encode model is outside the scope of this particular comparison.
3. Scoring Configuration and Training Computation (W10–W11)
Scoring-layer and decision-token selection (W10). As specified in Appendix A, we use zero-based layer index 20 (the 21st Transformer block) and the first output token ([) for scoring. The scoring layer was selected using 500 held-out MS MARCO validation examples, while the decision token and Step-10 stride were selected through training-side experiments rather than BEIR test-set tuning.
Appendix C reports the layer and decision-token attention-accuracy analyses. We will clarify the corresponding checkpoint and selection protocols, distinguishing the observed attention profiles from comparisons between independently trained configurations.
Gradient flow and execution depth (W11). The ranking loss is computed from attention-based scores at layer 20. Under the ranking-only objective, Transformer blocks after this layer receive no ranking-loss gradients. For joint training objectives, gradient flow also depends on the next-token prediction loss.
At inference, CompRank terminates the forward pass at the scoring layer, without executing subsequent Transformer blocks or autoregressive identifier decoding. We will clarify the distinction between trainable parameters, parameters receiving gradients, and inference execution depth.
Response to Reviewer 1Lad (3/3)
4. Additional Baselines and Benchmark Coverage (W12–W13)
Related methods (W12). Our matched BlockRank compression experiment provides a direct test of the benefit of post-encoding KV selection within an existing efficient reranking architecture.
We will also expand the discussion of related approaches, including PE-Rank's embedding-based passage representation, Compress-then-Rank's passage compression, and the alternative ranking strategies of TourRank and DeAR. These methods explore complementary design choices, whereas CompRank retains contextualized token-level KV states for decoding-free attention scoring.
Additional benchmarks and long-list evaluation (W13). We extend the evaluation to TREC Deep Learning 2019 and 2020 using the same BM25 top-100 candidates.
| Method | TREC DL19 | TREC DL20 |
|---|---|---|
| BlockRank, Full KV | 70.8 | 69.7 |
| CompRank-QC, Step-10 | 71.2 | 70.0 |
We further examine long-list generalization under different retrieval candidate sources:
| Dataset / candidate source | 100 candidates | 500 candidates |
|---|---|---|
| TREC DL20 / BM25 | 70.0 | 69.3 |
| TREC-COVID / Dense retrieval | 77.6 | 76.9 |
CompRank-QC exhibits only a 0.7-point decrease when scaling from 100 to 500 candidates in both settings. These results complement the original TREC-COVID scaling experiment and broaden the evaluation beyond the seven BEIR datasets. We do not treat TREC DL19/20 as fully independent out-of-domain evaluations relative to MS MARCO-derived training.
5. Statistical Reliability and Result Consistency (W14–W15)
Statistical reliability (W14). We additionally examine uncertainty in the matched training-objective ablation. Ranking-only Step-10 achieves 46.1, compared with 45.8 for the joint-objective full-KV reference, a difference of +0.3 points.
Using paired query-level bootstrap within each dataset, followed by the original seven-dataset macro-averaging procedure, the estimated 95% confidence interval is [−0.2, +0.8]. Across three training seeds, the paired differences are +0.1, +0.3, and +0.5.
These results support the conclusion that Step-10 preserves competitive ranking effectiveness with substantially fewer KV states, rather than establishing a statistically significant effectiveness gain.
Full-token result consistency (W15). The full-token references of 46.8 in Table 2 and 47.2 in Table 4 correspond to different experimental settings:
- Table 2: A fixed full-token-trained checkpoint used for matched inference-time compression ablations.
- Table 4: The full-token reference from the main compression comparison, where configurations are trained separately.
The two values are therefore not repeated measurements of the same checkpoint.
For Figure 2, the correct Mistral-7B CompRank-QC Full-KV nDCG@10 at 100 candidates is 80.5, as reported in Table 4. The plotted 78.9 is an error and will be corrected.
We will also clarify that Figure 2 uses Mistral-7B Direct-List trained on 30-document lists, while Table 1 uses its default Qwen2.5-7B, 20-document training configuration. This explains the Direct-List scores of 74.2 and 77.4, respectively, without conflating the two experimental settings.
Response to Reviewer 1Lad (1/3)
We thank the reviewer for the detailed comparison with BlockRank and the constructive suggestions regarding methodological contributions, controlled comparisons, and experimental reliability. We address these concerns with additional experiments and clarifications below.
1. Methodological Contributions and Matched Compression Comparisons (W1–W5)
Contributions beyond BlockRank (W1, W3, W4). CompRank builds upon BlockRank's block-decoupled encoding, but further reduces the KV states involved in relevance scoring through post-encoding contextualized KV compression. With Step-10, CompRank-QC retains only 10.2% of document states while achieving 46.7 average BEIR nDCG@10, compared with 47.2 under full-token scoring.
We further introduce two complementary configurations: CompRank-QC prioritizes relevance modeling through query-conditioned representations, while CompRank-Cache enables reusable query-independent document KV states. The latter reduces online latency to 1.231 s/query at 500 candidates, excluding offline cache construction.
Query-conditioned encoding itself is not claimed as a standalone novelty. We will clarify BlockRank's prior use of query prefixes and emphasize the contributions of contextualized KV compression, direct ranking supervision, and the effectiveness–reuse trade-off.
Matched BlockRank compression (W2). We additionally apply Step-10 to a BlockRank-style model trained with the joint next-token prediction and ranking objective. Under the same checkpoint, BEIR nDCG@10 changes from 45.8 to 45.3, showing that contextualized KV compression also benefits the existing architecture.
We further measure effectiveness and end-to-end latency on TREC-COVID using matched Mistral-7B settings, candidate lists, truncation policies, A100 80GB hardware, and batch size 1, without cross-query KV reuse.
At 500 candidates, Step-10 reduces BlockRank latency by 25%, with only a 0.3-point nDCG@10 decrease. CompRank-QC further improves nDCG@10 by 2.8 points over compressed BlockRank while being approximately 1.05× faster.
Isolating query and instruction placement (W5). We additionally train three matched Step-10 variants, keeping the scoring suffix and document positional settings fixed. All variants retain the same static instruction in the scoring suffix; only the information accessible during document encoding changes.
The latter two configurations permit query-independent document caching. The 46.1 versus 40.5 comparison isolates the benefit of query conditioning, while 40.5 versus 39.4 isolates the additional contribution of the static instruction prefix. These are matched ablation variants, distinct from the submitted QC and Cache configurations.