- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates attention-based listwise reranking using internal attention weights from large language models, focusing on how to select ranking-relevant attention heads and how to improve their ranking performance through training.
The authors propose a framework consisting of three components. First, a frequency-based head selection method identifies attention heads according to how frequently they appear in instance-wise Top-k sets. Second, att-sft introduces InfoNCE and RankNet losses into supervised fine-tuning to directly optimize attention-based document scores. Finally, att-dpo constructs preference pairs by perturbing attention distributions and further optimizes ranking performance using an approxNDCG-based objective.
The method is evaluated on five BEIR datasets across multiple model backbones, and the experimental results demonstrate improvements over existing attention-based reranking methods.
Overall, I believe the problem studied in this paper is meaningful, and the empirical results demonstrate the effectiveness of the proposed approach. However, the necessity and practical benefits of the complete training framework are not sufficiently justified. In particular, the head selection strategy does not adequately consider actual ranking quality or complementarity among selected heads. Therefore, I believe the current results are not yet sufficient to fully support the paper's main contributions.
Critical strengths
-
The research problem is meaningful. Attention-based ranking methods can directly leverage internal attention weights of LLMs to rank documents without autoregressively generating ranking outputs. This paper further investigates how to select and train relevant attention heads, which is valuable for improving this ranking paradigm.
-
The paper provides useful comparisons and ablation studies within the attention-based reranking setting. The authors conduct experiments across multiple BEIR datasets and model backbones, with comparisons against existing attention-based reranking methods. The paper also includes ablation studies on head selection and supervised fine-tuning, and the results demonstrate performance improvements from the proposed components.
Critical weaknesses
-
The trade-off between methodological complexity and performance gains is insufficiently justified. The paper introduces a complete training pipeline consisting of head selection, att-sft, and att-dpo, which increases the overall complexity of the approach. However, the experiments mainly compare against existing attention-based reranking methods, without sufficient comparisons with simpler dedicated rerankers, such as cross-encoders, in terms of ranking effectiveness, computational cost, and inference efficiency. Furthermore, the efficiency analysis primarily emphasizes the advantages of attention-based ranking over generative ranking, which are not unique to this work. Therefore, the current results do not sufficiently establish whether the proposed complex framework provides a favorable effectiveness–efficiency trade-off in practical applications.
-
The head selection strategy does not sufficiently consider actual ranking quality or complementarity among heads. The proposed frequency-based criterion primarily selects heads according to how frequently they appear in instance-wise Top-k sets, while the default scoring function mainly considers the attention assigned to positive documents. However, this does not directly measure actual ranking quality, as a head may assign high attention to a relevant document while assigning even higher attention to irrelevant documents. Moreover, the method evaluates heads independently without explicitly considering correlations or redundancy among them. Consequently, multiple highly similar heads may be selected, while less frequent but complementary heads may be excluded. Although the paper provides ablation studies on head selection, comparisons with ranking-quality-aware or diversity-aware selection strategies are still missing.
Questions for authors
Does the proposed training framework still offer advantages over simpler dedicated rerankers, such as cross-encoders, under comparable computational budgets? Could the authors provide more direct comparisons of ranking effectiveness and computational cost to justify the necessity of the complete training framework?
The current head selection method primarily relies on individual head statistics. Have the authors considered correlations or redundancy among different heads? Would selecting heads based directly on ranking quality (e.g., nDCG), or introducing diversity-aware selection, lead to better results?
3 comments
Part II
Q2. Have you considered correlations or redundancy among heads? Would ranking-quality-aware (e.g., nDCG) or diversity-aware selection lead to better results?
We thank the reviewer for this constructive suggestion. We would first like to clarify the scope of our design, and then provide new experiments with the nDCG-based scoring the reviewer suggests.
(1) Scoring and selection are orthogonal components in our framework
Our study focuses on the selection method: given a score for each attention head, how to select a subset of heads. Our selection method is agnostic to the scoring function and can be combined with arbitrary scoring criteria. The ranking-quality-aware (e.g., nDCG) and diversity-aware criteria mentioned by the reviewer operate at the scoring level rather than the selection level, and are therefore orthogonal to our selection method: they can be directly plugged into our framework as alternative scoring functions.
(2) New experiment: nDCG-based head scoring
Following the reviewer's suggestion, we implemented the nDCG scoring, where each attention head is scored by its nDCG on the head-selection (detection) examples. We compare it with the QR scoring (the default in our paper) and the Core scoring, each combined with both the previous TopK-avg selection and our frequency-based (freq) selection:
| Scoring | Selection | nq-4b | hotpotqa-4b | fiqa-4b | avg |
|---|---|---|---|---|---|
| QR | TopK-avg | 62.9 | 76.1 | 43.2 | 60.7 |
| QR | freq (ours) | 63.3 | 76.0 | 43.6 | 61.0 |
| Core | TopK-avg | 62.6 | 76.3 | 43.7 | 60.9 |
| Core | freq (ours) | 62.8 | 76.5 | 43.8 | 61.0 |
| nDCG | TopK-avg | 61.8 | 76.0 | 42.8 | 60.2 |
| nDCG | freq (ours) | 62.4 | 75.8 | 43.7 | 60.6 |
Two observations emerge. First, for every scoring function, our frequency-based selection outperforms the previous TopK-avg selection in average nDCG@10 (QR: 61.0 vs. 60.7; Core: 61.0 vs. 60.9; nDCG: 60.6 vs. 60.2), showing that the proposed selection method is robust to and complementary with different scoring choices. Second, the nDCG scoring underperforms the QR scoring under both selection methods. We attribute this to an inconsistency between head selection and inference: nDCG is computed with gold ranking labels during head selection, but such labels are unavailable at inference time, where ranking can only be derived from the raw document scores. In contrast, the QR statistics are computed directly on the document scores, which is exactly the quantity used at inference, so this scoring is better aligned with the inference-time behavior and transfers better.
In summary, scoring criteria and selection strategies address different aspects of head selection and are orthogonal in our framework. Our frequency-based selection consistently improves over the previous selection under all tested scoring functions, and the direct ranking-quality scoring does not help due to the selection and inference inconsistency. We will add this analysis and the table to the revised paper.
Thank you for the author's response. I am maintaining my original rating.
Part I
We thank the reviewer for the constructive comments. Below we respond to the two questions in order. All new experimental results will be added to the revised manuscript.
Q1. Does the proposed training framework still offer advantages over simpler dedicated rerankers, such as cross-encoders, under comparable computational budgets?
We thank the reviewer for this question. We first clarify the cost structure of our framework, and then provide new experiments comparing against a cross-encoder reranker under matched conditions.
(1) Our method does not introduce additional training cost relative to existing training pipelines
Our contribution lies in the training method and strategy, not in the construction of training data. We use the same model scale and exactly the same training data as previous training-based approaches, so the overall training cost is identical. In particular, att-sft only changes the training objective, and its cost is the same as standard SFT. While att-dpo requires additional training, it is an optional component: even without att-dpo, our att-sft already surpasses previous baselines (average nDCG@10 of 64.1 vs. 63.2 for listwise SFT). Therefore, under the same training cost, our method achieves the best performance, and att-dpo brings further gains with a modest amount of additional training.
(2) New comparison with a cross-encoder reranker under matched conditions
Since our method performs listwise ranking, our main comparisons have focused on listwise baselines. To directly answer this question, we additionally conduct experiments with Qwen3-Reranker-4B, a cross-encoder that has the same scale and belongs to the same model family as Qwen3-4B, which isolates the difference of ranking paradigms from model capacity. We consider two settings: (a) ranking without training; (b) training on the same data, where the listwise training data is converted into pointwise form by assigning scores according to the document order. The results are as follows:
The cross-encoder performs substantially worse in both settings, and the trained pointwise model does not exceed the listwise baselines. We attribute this to two factors. First, the evaluation datasets are listwise datasets, so the pointwise paradigm, although usable, is not well matched to the evaluation setting. Second, the listwise training data lacks pointwise annotations; the scores we assign according to document order deviate considerably from the true relevance labels, which further limits the pointwise training. In addition, the cross-encoder paradigm requires scoring each query and document pair separately, whereas our approach performs ranking within the backbone's listwise forward pass, which is more favorable for inference efficiency.
In summary, under the same model scale and the same training data, the complete framework achieves the best effectiveness at a training cost equal to prior listwise training pipelines, and every component of the pipeline contributes to the final result. We will add this cross-encoder comparison and the cost discussion to the revised paper.