acceptodds
Under review as a conference paper at ICLR 2027

Better Heads, Better Rankers: Selecting and Training Attention Heads for Listwise Reranking

Abstract

Attention-based ranking has become a key paradigm for listwise ranking in large language models (LLMs). It reads the attention weights between internal query tokens and document tokens as the basis for ranking, thereby dispensing with autoregressive decoding and offering markedly better efficiency and interpretability than generative rankers. However, its effectiveness depends on two interrelated design choices: which attention heads supply the ranking signal, and how to perform training augmentation on these heads. Previous studies have not yet established a unified paradigm for these designs. In this paper, we conduct a systematic study of head selection and training for attention-based ranking. On the selection side, we show that prevailing practice of averaging per-head scores over a development set is brittle and the resulting set lacks overall representativeness. We instead propose an frequency-based criterion that can select adaptive, representative head sets. On the training side, we introduce a unified two-stage recipe that optimizes the selected heads. During supervised fine-tuning, att-sft augments the training objective with loss terms computed from the head-aggregated attention scores. During reinforcement learning, we first identify the discrepancies between training and inference, and then propose att-dpo, which can directly align preferences on the attention distribution. Across two backbones and several ranking benchmarks, the recipe consistently surpasses untrained and training-related baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
attention weights, listwise ranking, attention heads

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ugP1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates attention-based listwise reranking using internal attention weights from large language models, focusing on how to select ranking-relevant attention heads and how to improve their ranking performance through training.

The authors propose a framework consisting of three components. First, a frequency-based head selection method identifies attention heads according to how frequently they appear in instance-wise Top-k sets. Second, att-sft introduces InfoNCE and RankNet losses into supervised fine-tuning to directly optimize attention-based document scores. Finally, att-dpo constructs preference pairs by perturbing attention distributions and further optimizes ranking performance using an approxNDCG-based objective.

The method is evaluated on five BEIR datasets across multiple model backbones, and the experimental results demonstrate improvements over existing attention-based reranking methods.

Overall, I believe the problem studied in this paper is meaningful, and the empirical results demonstrate the effectiveness of the proposed approach. However, the necessity and practical benefits of the complete training framework are not sufficiently justified. In particular, the head selection strategy does not adequately consider actual ranking quality or complementarity among selected heads. Therefore, I believe the current results are not yet sufficient to fully support the paper's main contributions.

Critical strengths
  1. The research problem is meaningful. Attention-based ranking methods can directly leverage internal attention weights of LLMs to rank documents without autoregressively generating ranking outputs. This paper further investigates how to select and train relevant attention heads, which is valuable for improving this ranking paradigm.

  2. The paper provides useful comparisons and ablation studies within the attention-based reranking setting. The authors conduct experiments across multiple BEIR datasets and model backbones, with comparisons against existing attention-based reranking methods. The paper also includes ablation studies on head selection and supervised fine-tuning, and the results demonstrate performance improvements from the proposed components.

Critical weaknesses
  1. The trade-off between methodological complexity and performance gains is insufficiently justified. The paper introduces a complete training pipeline consisting of head selection, att-sft, and att-dpo, which increases the overall complexity of the approach. However, the experiments mainly compare against existing attention-based reranking methods, without sufficient comparisons with simpler dedicated rerankers, such as cross-encoders, in terms of ranking effectiveness, computational cost, and inference efficiency. Furthermore, the efficiency analysis primarily emphasizes the advantages of attention-based ranking over generative ranking, which are not unique to this work. Therefore, the current results do not sufficiently establish whether the proposed complex framework provides a favorable effectiveness–efficiency trade-off in practical applications.

  2. The head selection strategy does not sufficiently consider actual ranking quality or complementarity among heads. The proposed frequency-based criterion primarily selects heads according to how frequently they appear in instance-wise Top-k sets, while the default scoring function mainly considers the attention assigned to positive documents. However, this does not directly measure actual ranking quality, as a head may assign high attention to a relevant document while assigning even higher attention to irrelevant documents. Moreover, the method evaluates heads independently without explicitly considering correlations or redundancy among them. Consequently, multiple highly similar heads may be selected, while less frequent but complementary heads may be excluded. Although the paper provides ablation studies on head selection, comparisons with ranking-quality-aware or diversity-aware selection strategies are still missing.

Questions for authors

Does the proposed training framework still offer advantages over simpler dedicated rerankers, such as cross-encoders, under comparable computational budgets? Could the authors provide more direct comparisons of ranking effectiveness and computational cost to justify the necessity of the complete training framework?

The current head selection method primarily relies on individual head statistics. Have the authors considered correlations or redundancy among different heads? Would selecting heads based directly on ranking quality (e.g., nDCG), or introducing diversity-aware selection, lead to better results?

3 comments

We thank the reviewer for the constructive comments. Below we respond to the two questions in order. All new experimental results will be added to the revised manuscript.


Q1. Does the proposed training framework still offer advantages over simpler dedicated rerankers, such as cross-encoders, under comparable computational budgets?

We thank the reviewer for this question. We first clarify the cost structure of our framework, and then provide new experiments comparing against a cross-encoder reranker under matched conditions.

(1) Our method does not introduce additional training cost relative to existing training pipelines

Our contribution lies in the training method and strategy, not in the construction of training data. We use the same model scale and exactly the same training data as previous training-based approaches, so the overall training cost is identical. In particular, att-sft only changes the training objective, and its cost is the same as standard SFT. While att-dpo requires additional training, it is an optional component: even without att-dpo, our att-sft already surpasses previous baselines (average nDCG@10 of 64.1 vs. 63.2 for listwise SFT). Therefore, under the same training cost, our method achieves the best performance, and att-dpo brings further gains with a modest amount of additional training.

(2) New comparison with a cross-encoder reranker under matched conditions

Since our method performs listwise ranking, our main comparisons have focused on listwise baselines. To directly answer this question, we additionally conduct experiments with Qwen3-Reranker-4B, a cross-encoder that has the same scale and belongs to the same model family as Qwen3-4B, which isolates the difference of ranking paradigms from model capacity. We consider two settings: (a) ranking without training; (b) training on the same data, where the listwise training data is converted into pointwise form by assigning scores according to the document order. The results are as follows:

nDCG@10ModelParadigmnq-4bhotpotqa-4bfiqa-4bavg
Pre-training (zero-shot ranking)Qwen3-Reranker-4Bpointwise (cross-encoder)21.331.411.021.2
Pre-training (listwise ranking)Qwen3-4B (ours)listwise62.976.143.260.7
SFT, pointwiseQwen3-Reranker-4Bpointwise (cross-encoder)38.744.730.638.0
SFT, listwiseQwen3-4Blistwise69.074.546.263.2
att-sft, listwise (ours)Qwen3-4Blistwise69.176.746.664.1
att-dpo + att-sft (ours)Qwen3-4Blistwise69.577.246.464.4

The cross-encoder performs substantially worse in both settings, and the trained pointwise model does not exceed the listwise baselines. We attribute this to two factors. First, the evaluation datasets are listwise datasets, so the pointwise paradigm, although usable, is not well matched to the evaluation setting. Second, the listwise training data lacks pointwise annotations; the scores we assign according to document order deviate considerably from the true relevance labels, which further limits the pointwise training. In addition, the cross-encoder paradigm requires scoring each query and document pair separately, whereas our approach performs ranking within the backbone's listwise forward pass, which is more favorable for inference efficiency.

In summary, under the same model scale and the same training data, the complete framework achieves the best effectiveness at a training cost equal to prior listwise training pipelines, and every component of the pipeline contributes to the final result. We will add this cross-encoder comparison and the cost discussion to the revised paper.

Q2. Have you considered correlations or redundancy among heads? Would ranking-quality-aware (e.g., nDCG) or diversity-aware selection lead to better results?

We thank the reviewer for this constructive suggestion. We would first like to clarify the scope of our design, and then provide new experiments with the nDCG-based scoring the reviewer suggests.

(1) Scoring and selection are orthogonal components in our framework

Our study focuses on the selection method: given a score for each attention head, how to select a subset of heads. Our selection method is agnostic to the scoring function and can be combined with arbitrary scoring criteria. The ranking-quality-aware (e.g., nDCG) and diversity-aware criteria mentioned by the reviewer operate at the scoring level rather than the selection level, and are therefore orthogonal to our selection method: they can be directly plugged into our framework as alternative scoring functions.

(2) New experiment: nDCG-based head scoring

Following the reviewer's suggestion, we implemented the nDCG scoring, where each attention head is scored by its nDCG on the head-selection (detection) examples. We compare it with the QR scoring (the default in our paper) and the Core scoring, each combined with both the previous TopK-avg selection and our frequency-based (freq) selection:

ScoringSelectionnq-4bhotpotqa-4bfiqa-4bavg
QRTopK-avg62.976.143.260.7
QRfreq (ours)63.376.043.661.0
CoreTopK-avg62.676.343.760.9
Corefreq (ours)62.876.543.861.0
nDCGTopK-avg61.876.042.860.2
nDCGfreq (ours)62.475.843.760.6

Two observations emerge. First, for every scoring function, our frequency-based selection outperforms the previous TopK-avg selection in average nDCG@10 (QR: 61.0 vs. 60.7; Core: 61.0 vs. 60.9; nDCG: 60.6 vs. 60.2), showing that the proposed selection method is robust to and complementary with different scoring choices. Second, the nDCG scoring underperforms the QR scoring under both selection methods. We attribute this to an inconsistency between head selection and inference: nDCG is computed with gold ranking labels during head selection, but such labels are unavailable at inference time, where ranking can only be derived from the raw document scores. In contrast, the QR statistics are computed directly on the document scores, which is exactly the quantity used at inference, so this scoring is better aligned with the inference-time behavior and transfers better.

In summary, scoring criteria and selection strategies address different aspects of head selection and are orthogonal in our framework. Our frequency-based selection consistently improves over the previous selection under all tested scoring functions, and the direct ranking-quality scoring does not help due to the selection and inference inconsistency. We will add this analysis and the table to the revised paper.

Thank you for the author's response. I am maintaining my original rating.

Reviewer_42qLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies attention-based listwise ranking, where an LLM scores candidate documents from query–document attention weights in a single prefill pass, and addresses two design questions: which attention heads should supply the ranking signal, and how those heads should be trained. For selection, the authors replace the prevailing dev-set mean top-k heuristic with a frequency-based criterion that counts how often each head appears in per-example top-k sets, yielding an outlier-robust, size-adaptive head set at no additional cost. For training, they propose a two-stage recipe: att-sft augments SFT with InfoNCE and weighted-RankNet terms computed on head-aggregated attention scores, while att-dpo addresses a train–inference mismatch of standard DPO by redefining the policy as the attention-derived document distribution, constructing preference pairs via perturbation, and optimizing an approxNDCG-based SimPO objective with a KL anchor. Experiments on Qwen3-4B/8B (plus Llama-3.1-8B) across five BEIR datasets show consistent, additive gains over both training-free and trained baselines.

Critical strengths
  • This paper is well-structured and easy to follow.
  • The paper cleanly separates the two neglected design dimensions (head selection, head training) and unifies them into one pipeline.
Critical weaknesses
  • Limited contribution and marginal gains. SFT- and preference-based training are already well studied for LLM reranking (RankVicuna/RankZephyr, FIRST-style SFT, QRRanker, HeadRank). att-sft applies standard ranking losses (InfoNCE, RankNet) to attention scores, and att-dpo recombines existing components (SimPO, approxNDCG, a KL anchor); the core novelty therefore reduces to the frequency-based selection rule. Yet its improvement over simply taking top-k average heads is very marginal.
  • The selection criterion is a hand-tuned heuristic, and the robustness claim is not well supported. The paper motivates Freq by robustness and adaptivity, but Table 2 shows non-trivial sensitivity to its own hyperparameters. This falls short of the claimed task-agnostic robustness and suggests practitioners would need per-task hyperparameter search, weakening the practical advantage over TopKavg.
  • Numerical inconsistencies between the text and the tables, which undermine confidence in the reported results. For example:
    • The text claims Freq-QR improves over QR by "+1.3 on Qwen3-8B", but Table 1 shows 69.6 vs. 69.0 (+0.6).
    • The text says TopKavg(21/22) "trails by 0.7/0.4", while Table 2 implies 0.8/0.5
    • The text claims att-sft beats Havg(8) "by 1.4–1.7 on average", but on 8B the gap to Havg(8) is only ~0.4 in Table 3

Please check and correct all numbers carefully.

Suggestions:

  • Figure 1 is dense and hard to read. I suggest increasing the font size/weight and color contrast of the labels, and simplifying or splitting the three stages so the pipeline can be parsed at a glance.
  • There is insufficient spacing between the Figure 2 caption and the surrounding main text; please add vertical space to separate them.
  • The literature review is insufficient. Beyond the listwise-LLM line, I encourage the authors to discuss broader document reranking work, e.g., pointwise cross-encoder rerankers.
Questions for authors
  • Why is the untrained Generative baseline on Llama3-8B identical to the retrieval order (Table 6)? Is instruction-following failing for this backbone, and if so, does this undermine the generative-ranker comparison in Table 6?
Reviewer_x84MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies attention-based listwise reranking, where document scores are obtained by aggregating query-to-document attention weights without autoregressive decoding. It proposes a frequency-based criterion for selecting ranking-relevant attention heads, followed by two training stages: att-sft, which combines next-token prediction with attention-based InfoNCE and weighted RankNet losses, and att-dpo, which optimizes an approxNDCG-based objective using perturbed attention distributions and KL regularization. Experiments on five BEIR datasets and multiple backbones report improvements over training-free and trained baselines, together with lower inference latency than generative reranking.

Critical strengths

The research direction and training–inference alignment are well motivated. Obtaining ranking scores from internal attention avoids autoregressive decoding, and directly supervising those same scores is a logical way to improve their usefulness. The combination of positive-document discrimination and pairwise ordering supervision provides a coherent training recipe.

The empirical study includes meaningful controls and ablations. The paper evaluates multiple backbones and datasets, compares different head sets and loss combinations, and includes an RL-stage comparison starting from a shared SFT checkpoint with the same training data. These experiments provide encouraging evidence that the attention-aware training components can improve reranking within the studied setting.

Critical weaknesses

The independent methodological contribution of att-dpo is insufficiently established. According to Appendix A.1, the perturbed negative branch is stop-gradient. Consequently, the gradient of the preference term follows the direction of increasing the gold-label-based approxNDCG of the original distribution, with the negative branch primarily determining a sample-dependent scaling factor. This makes the method closely related to supervised metric optimization with adaptive weighting and KL regularization. The paper does not compare against the most direct alternative: optimizing negative approxNDCG plus the same KL regularizer under a matched training budget. Without this comparison, it is unclear whether the perturbation-based preference construction contributes beyond direct optimization of the inference-time ranking metric. Furthermore, random perturbation does not guarantee an inferior ranking, despite the corresponding claim in the method section. The direction is sensible, but the current evidence does not substantiate the stronger interpretation of att-dpo as a distinct preference-optimization mechanism.

The performance evidence leaves uncertainty about the relevance and robustness of the reported gains. The training supervision is distilled from legacy GPT-4 and GPT-3.5 ranking outputs. Using fixed supervision permits controlled comparisons within this setting, but it leaves open whether the gains persist with stronger teacher rankings or against stronger reranking systems. In particular, the study does not establish whether the proposed losses provide a general advantage or mainly improve learning from the characteristics and noise of these older pseudo-labels. This limits my confidence in the broader practical significance of the final performance improvements. This concern is amplified by the relatively modest incremental gains from att-dpo over att-sft—approximately 0.5–0.6 points in average nDCG@10—and the absence of multi-seed uncertainty estimates. The candidate-pool discrepancy between the main text and appendix, together with insufficient detail about checkpoint selection and the separation of the NQ head-selection examples from evaluation queries, further complicates interpretation of these gains.

Questions for authors

What does the perturbation-based objective add beyond direct metric optimization? Please compare att-dpo with direct approxNDCG optimization using the same KL regularizer, initialization, data, and training budget. Please also clarify the preference interpretation when the perturbed distribution can outperform the original one.

How robust are the reported improvements to supervision quality and evaluation choices? Is there evidence that the gains persist with stronger teacher-generated rankings or against stronger reranking baselines? Please provide uncertainty estimates for the incremental att-dpo gains and clarify the actual candidate count, checkpoint-selection criterion, and separation between head-selection and evaluation examples.

2 comments

We thank the reviewer for the careful and constructive comments. Below we respond to the two questions in order. All new experimental results will be added to the revised manuscript.


Q1. What does the perturbation-based objective add beyond direct metric optimization?

We thank the reviewer for this insightful question. We would first like to clarify a misunderstanding about how the negative branch is constructed: it is not simply a sample-adaptive scaling factor. The negative branch in att-dpo is built upon two explicit hyperparameters: a scaling factor (set to 0.02 in our experiments) and Gaussian noise (set to 0.5). The negative branch is therefore constructed with an additional, controlled perturbation on top of the original distribution, rather than acting as a mere sample-dependent reweighting of the supervised metric objective.

To directly address the reviewer's concern, we conducted hyperparameter ablations that disentangle the contribution of each component under exactly the setup the reviewer suggests (same initialization, data, KL regularizer, and training budget). We consider two informative settings:

  1. (scaling factor = 0, Gaussian noise = 0): both perturbation components are removed, which reduces att-dpo to directly optimizing the negative approxNDCG. As shown in the table below, this setting yields negative gains: performance drops below the pre-training baseline on all six configurations (average nDCG@10: 59.0 vs. 60.7 at 4B; 60.0 vs. 61.6 at 8B). This demonstrates that direct metric optimization without the contrastive positive and negative construction actually degrades the attention's ranking ability, confirming that the contrastive design is necessary.
  2. (scaling factor = 0.6, Gaussian noise = 0.5): with a large scaling factor, the perturbation becomes less that random perturbation can no longer guarantee the perturbed ranking is inferior to the original one, which is precisely the scenario the reviewer raises. In this setting, training still brings some gains, but they are substantially smaller than att-dpo (average nDCG@10: 61.1 vs. 62.1 at 4B; 61.8 vs. 62.7 at 8B), with most of the improvement over the pre-training baseline lost. This indicates that a strong and reliable contrast between the positive and negative branches is critical, and in att-dpo this contrast is precisely controlled by the scaling factor.
avg nDCG@10nq-4bhotpotqa-4bfiqa-4bavg-4bnq-8bhotpotqa-8bfiqa-8bavg-8b
Pre-training62.976.143.260.763.776.844.461.6
att-dpo (ours), (0.02, 0.5)66.076.144.162.166.476.844.962.7
(0.6, 0.5)63.776.043.761.164.876.544.161.8
(0, 0)61.872.343.059.063.072.944.260.0

In summary, both parameters in the negative-branch construction are meaningful and necessary, and together they answer the reviewer's two sub-concerns: (i) the scaling factor guarantees that the perturbed negative is always inferior to the positive one, ensuring a well-defined preference signal (and thereby resolving the concern about cases where the perturbed distribution outperforms the original); (ii) the Gaussian noise ensures the negative branch carries genuine perturbation information, so it does not degenerate into a mere sample-adaptive scaling factor. The ablation shows that removing either property substantially hurts or even reverses the gains. We will add this ablation table and the corresponding discussion to the revised version of the paper.

Q2. How robust are the improvements to supervision quality and evaluation choices?

We thank the reviewer for these careful questions and address each concern below.

(1) The gains are not from fitting the training distribution, since the evaluation is OOD by design

We would like to clarify that our evaluation is strictly out-of-distribution: all training data is drawn from MS MARCO, while the evaluation datasets (NQ, HotpotQA, FiQA) are completely orthogonal to it, with no overlap in data distribution or domain. As shown in Appendices B.1 and B.2, the training and test data differ substantially in statistics such as the number of documents and token lengths. Therefore, the improvements cannot be attributed to better fitting of the characteristics or noise of the training pseudo-labels.

To further eliminate this concern, we additionally evaluated on a Chinese search dataset from our production system. Training only on English MS MARCO data and evaluating directly on the Chinese dataset, without any Chinese training data, yields a clear improvement: nDCG@10 improves from 0.8298 (pre-training) to 0.8791 (after training). This cross-lingual transfer, from English training data to a completely different language and business scenario, demonstrates that our method improves the model's general ranking capability rather than its ability to fit a particular training distribution.

(2) Multi-seed uncertainty estimates

Following the reviewer's suggestion, we report att-dpo results with three random seeds below. Regardless of which seed is used, the resulting model outperforms both the pre-training baseline and the headrank baseline. The average nDCG@10 across the three seeds is 61.97±0.07 at 4B and 62.74±0.14 at 8B, compared with 61.63 and 61.33 for headrank and 60.73 and 61.63 for the pre-training baseline. Notably, even the worst-performing seed still exceeds the best baseline (61.9 vs. 61.6 at 4B; 62.6 vs. 61.6 at 8B), so the incremental gains are stable and clearly not an artifact of seed variance or checkpoint cherry-picking. The same trend holds on 5 of the 6 individual dataset/scale configurations, and on the remaining one (HotpotQA-4B) the pre-training baseline is already saturated (76.1).

avg nDCG@10nq-4bhotpotqa-4bfiqa-4bavg-4bnq-8bhotpotqa-8bfiqa-8bavg-8b
Pre-training62.976.143.260.763.776.844.461.6
headrank65.975.243.861.666.074.044.061.3
att-dpo (ours), seed 166.076.144.162.166.476.844.962.7
att-dpo, seed 266.175.544.261.966.976.345.562.9
att-dpo, seed 365.876.143.862.066.476.744.862.6

(3) Experimental details: head-selection and evaluation split, checkpoint selection

  • Head-selection vs. evaluation split: both the attention-head selection and the evaluation datasets are taken directly from prior work (QRhead[1] and Corehead[2]); we did not construct or split the data ourselves. Head selection uses a separate set of 100 NQ examples, and evaluation is conducted on the remaining NQ examples. This protocol is fully consistent with previous work, ensuring comparability.
  • Checkpoint selection: we evaluate all checkpoints and report the best result, and this protocol is applied identically to all baselines and all methods, so the comparison remains fair.

We will add the multi-seed results, the Chinese-dataset evaluation, and these clarification details to the revised paper.

References:

[1] Query-focused retrieval heads improve long-context reasoning and re-ranking

[2] Contrastive Retrieval Heads Improve Attention-Based Re-Ranking

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.