acceptodds
Under review as a conference paper at ICLR 2027

SPD: Single Pass Decoding for Generative Reranking

Abstract

Large language models (LLMs) achieve state-of-the-art generative ranking quality, but the ranking they produce must be decoded, and autoregressive decoding spends one sequential forward pass per emitted token. We observe that the only tokens a ranker must emit are the N ordinal values, corresponding to the number of positions available to rank, naming the items in ranked order, and that this narrow, permutation-structured output format admits decoding strategies which are much more efficient than left-to-right generation. We introduce SPD (Single Pass Decoding), a format-specialized decoding strategy that decodes all N ordinals in O(1) forward passes. SPD reads an N × K item-position score matrix off the LLM’s prefill hidden states with a lightweight self-attention head, then decodes the ordinals as the optimal bipartite assignment of that matrix via the Hungarian algorithm, yielding a valid permutation by construction rather than by repair, without enumerating candidate prefixes or re-querying the LLM. Through a systematic study of training signals and backbone adaptation, we show that LoRA-based finetuning combined with autoregressive LLM ranking distillation reaches 28 ms endto-end inference, a speed-up of 64× end-to-end over the reasoning teacher while maintaining ranking quality on par. We provide a complete ablation decomposing the contributions of architecture, training signal, and backbone adaptation. Our framework connects generative ranking to combinatorial optimization, opening a path toward other O(1)-decode mechanisms for real-time ranking.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficient LLM inference, generative reranking, non-autoregressive decoding, learning to rank, linear assignment problem, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oHj6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPD, a single-pass decoding method for generative reranking. It predicts an item-position score matrix from LLM hidden states and uses Hungarian matching to produce a valid permutation. Experiments on an internal dataset and Amazon Beauty report substantial latency reductions with relatively small ranking quality degradation.

Critical strengths

1.The paper addresses a practical latency bottleneck in generative reranking. On the internal dataset, SPD reduces inference latency from 88 ms to 28 ms compared with the non-reasoning teacher, while achieving similar ranking quality.

2.The method is simple and guarantees valid permutations through Hungarian matching, with negligible solver overhead at the evaluated slate size.

Critical weaknesses

1.Insufficient and unfair baseline comparisons. The paper only evaluates two datasets and does not compare against existing efficient rerankers or a simple single-pass scoring + sorting baseline. Moreover, the reported speedups are confounded by removing reasoning tokens (64×) or comparing a 32B teacher with a 0.6B student (45×), making it unclear how much improvement comes from the proposed decoding method itself. And lack of comparisons with existing efficient baselines.

2.Incomplete ablation studies. Table 3 lacks the click-label + LoRA setting, Table 4 only compares three scoring heads, and Table 5 reports latency for a single slate size (N=50). There is no ablation showing that Sinkhorn training or Hungarian decoding is necessary, nor any scalability analysis across different slate sizes.

Questions for authors

1.The paper discusses existing efficient reranking methods such as FIRST, ONE-PASS, PE-Rank and DiffuRank but does not compare against them experimentally. Can the authors provide quality–latency comparisons under comparable settings?

2.Can the authors compare SPD against a simple single-pass scoring + sorting baseline and ablate the Sinkhorn training objective? The current ablations do not establish the benefit of the proposed assignment-based formulation over simpler alternatives.

3.Table 2 reports only 26 decode passes for Qwen3-0.6B on Amazon Beauty, despite each slate containing 50 candidates. Could the authors clarify how complete rankings are generated and how incomplete or invalid outputs are handled during evaluation?

Reviewer_VYTLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Single Pass Decoding (SPD) for LLM-based reranking. The method extracts candidate representations from a single prefill pass, uses an additional self-attention scoring head to construct a candidate–position score matrix, and applies the Hungarian algorithm to obtain a valid permutation. This avoids the sequential decoding overhead of generating ranked item identifiers token by token.

During training, the method uses complete rankings generated by a teacher model as supervision, employs a Sinkhorn relaxation for differentiable training, and adapts the backbone through LoRA. The experiments demonstrate substantial reductions in inference latency while retaining strong ranking performance. On Amazon Beauty, the 0.6B SPD model achieves approximately an 8.2× speedup over the reported autoregressive baseline of the same parameter scale. However, some ranking metrics decrease relative to the 32B teacher, so the results are better characterized as a favorable trade-off between ranking quality and efficiency.

Critical strengths
  1. The method effectively exploits the structure of ranking to reduce sequential computation. It directly uses candidate representations obtained from a single forward pass to predict a complete permutation, avoiding autoregressive generation of item identifiers. The design combines candidate representation learning with permutation optimization and achieves substantial latency reductions for small candidate sets. This approach is also potentially informative for other tasks with fixed output formats and explicit combinatorial constraints.

  2. The connection between continuous relaxation during training and discrete assignment during inference is clear. The method uses the Sinkhorn operator to provide differentiable soft assignments during training and the Hungarian algorithm to enforce one-to-one assignments between items and positions during inference. This design combines end-to-end learning with explicit output constraints and provides a useful engineering approach to parallel prediction of discrete structures.

Critical weaknesses
  1. The evaluation of robustness to input order and applicability across candidate-set sizes is insufficient.The model uses a causal Transformer and extracts representations from the final token of each candidate description, so the order of candidates may affect the contextual information available to each representation. The paper does not report results obtained by randomly permuting the same candidate set, making it difficult to assess ranking stability. Furthermore, the experiments cover only one internal dataset and one public dataset, without a systematic analysis of changes in candidate count, input length, or task domain. The applicability boundaries of the method therefore remain unclear.

  2. The control experiments are insufficient to distinguish the contributions of the proposed architecture from those of teacher distillation. Although the paper includes a Qwen3-0.6B baseline of the same parameter scale, it does not adequately compare SPD against an autoregressive student trained on the same data with the same teacher-ranking supervision. Consequently, the observed quality advantages may jointly arise from teacher distillation, backbone adaptation, and changes to the ranking architecture. This missing comparison weakens the evidence for the specific added value of SPD.

  3. The lack of direct comparisons with related fast ranking methods leaves its practical advantages insufficiently established. The main experiments primarily compare SPD against autoregressive models, without comparisons to methods such as FIRST and ONE-PASS under consistent data, hardware, and evaluation conditions. Although the results demonstrate efficiency gains over autoregressive generation, they do not establish whether SPD offers a better trade-off among ranking quality, inference latency, and deployment cost than existing fast ranking approaches.

Questions for authors
  1. For the N*K score matrix, where N denotes the number of candidates and K the number of ranking positions, could the authors clarify the task definition, training constraints, and inference procedure when N>K? For example, when N>K, does the method perform Top K selection, or use additional positions to handle a complete permutation? When N<K, how are unfilled positions handled? Are there corresponding experiments on ranking quality and efficiency?

  2. Has SPD been evaluated on open-source backbones beyond the Qwen family?

Reviewer_6w8iOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPD (Single Pass Decoding), an efficient approach for LLM-based generative reranking that eliminates sequential autoregressive decoding. SPD extracts item representations from a single backbone forward pass, constructs an item-position score matrix using a lightweight self-attention head, and applies the Hungarian algorithm to produce a valid ranking permutation. The method is trained through offline teacher distillation with Sinkhorn-based optimization and LoRA adaptation. Experiments on a proprietary dataset and Amazon Beauty demonstrate substantial inference speedups while largely preserving ranking quality.

Critical strengths
  1. Efficient and practical decoding design. SPD replaces sequential autoregressive decoding with a single forward pass and combinatorial assignment, substantially reducing inference latency while guaranteeing valid ranking permutations.

  2. Strong empirical efficiency gains. SPD achieves 3.1× speedup over the non-reasoning autoregressive teacher and 64× over the reasoning teacher, with nearly unchanged ranking performance on the internal dataset.

  3. Useful ablation studies. The paper investigates different scoring head architectures, training signals, and backbone adaptation strategies, helping identify the contributions of self-attention, teacher distillation, and LoRA.

Critical weaknesses
  1. Limited novelty and insufficient comparison with related work. The idea of leveraging LLM hidden states with lightweight ranking heads has already been explored in Language Ranker [1]. Moreover, the paper lacks direct comparisons with recent efficient reranking methods, such as FIRST [2], ONE-PASS [3], DiffuRank [4], and PE-Rank [5]. Although SPD introduces an item-position assignment formulation with Hungarian decoding, its advantages over these existing approaches remain insufficiently demonstrated, making it difficult to assess its methodological novelty and practical contributions.

    References

    [1] Language Ranker: A Lightweight Ranking Framework for LLM Decoding. NeurIPS 2025.

    [2] FIRST: Faster Improved Listwise Reranking with Single Token Decoding. EMNLP 2024.

    [3] ONE-PASS: Single Forward Pass Decoding for Listwise Reranking. WWW 2026.

    [4] DiffuRank: Effective Document Reranking with Diffusion Language Models. arXiv 2026.

    [5] Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models (PE-Rank). WWW 2025.

  2. Limited evidence for the necessity of Hungarian assignment. The paper does not compare SPD against a simpler alternative that predicts a scalar relevance score for each item and obtains the ranking through sorting. Such an approach would also guarantee a valid permutation without autoregressive decoding. Can the authors demonstrate that item-position assignment provides meaningful advantages over direct relevance scoring?

  3. Potentially misleading attribution of inference speedups. The highlighted 64× speedup is measured against a reasoning teacher, whereas removing reasoning tokens alone reduces latency from 1,807 ms to 88 ms. SPD achieves a more modest 3.1× speedup over the non-reasoning autoregressive teacher. The paper should more clearly distinguish improvements attributable to single-pass decoding from those obtained by eliminating reasoning tokens.

  4. Limited benchmark diversity and reproducibility. The experiments rely on one proprietary dataset and the Amazon Beauty dataset, with no evaluation on established public reranking benchmarks such as MS MARCO or BEIR. The proprietary dataset also limits independent reproducibility.

  5. Insufficient isolation of decoding improvements from model distillation. SPD requires additional teacher distillation, a trained scoring head, and LoRA adaptation of the backbone. Therefore, the reported improvements reflect changes in both model training and decoding, rather than decoding alone.

  6. Limited scalability analysis across candidate set sizes. Although SPD requires a constant number of backbone forward passes, its total computational cost still depends on the number of candidates through prefill, self-attention, and the assignment solver. The paper primarily reports latency for 50 candidates without a systematic latency–quality analysis across different slate sizes. How does SPD perform as the number of candidates increases, particularly toward the maximum of 150 items in the proprietary dataset?

  7. Insufficient evaluation of ranking consistency with the teacher. The paper claims to preserve autoregressive teacher ranking quality but mainly reports AUC, Recall, and NDCG metrics. These metrics do not directly measure whether SPD reproduces the teacher's complete permutations. Could the authors report Kendall's Tau, Spearman correlation, or pairwise ranking agreement to quantify the fidelity of the distilled rankings?

  8. Limited evidence of generalization across backbone architectures and model scales. The proposed approach is primarily evaluated using a 0.6B student backbone, while the Amazon Beauty experiment additionally involves a 32B teacher. It remains unclear whether the effectiveness of extracting rankings from prefill hidden states generalizes to different LLM architectures and student sizes. Experiments across multiple backbones would help establish the robustness of the approach.

Questions for authors

see weakness

Comments on the paper

2 comments
AuthorsOpenReview ↗

Authors’ Response (1/2)

We thank the reviewers for their thoughtful comments and questions. As several of the points are shared between reviewers, we will address them by topic. We have submitted an updated draft of the paper to address all the topics brought up by the reviewers. Changes are marked in blue. We used the extra page for the new results in the main text, and put the supporting tables in a new appendix. Due to comment length we are dividing this response in 2 parts.

Extra Baselines and Datasets (Reviewers 6w8i, VYTL, oHj6)

All reviewers noted the possibility of improving the work with extra baselines, both dataset and methods. We have done just that: adding the baselines FIRST, PE-Rank, ONE-PASS, DiffuRank, and Language Ranker. On the dataset side, we have added the requested datasets. Concretely, the paper now evaluates on four public benchmarks: Amazon Beauty, MS MARCO (Contriever top-50), BEIR Quora and BEIR MS MARCO, each with 1,118 evaluation slates of 50 candidates. Between them they cover product recommendation, web search and duplicate-question retrieval (VYTL). We also report TREC DL19 and DL20. We ran every baseline ourselves on the same slates, prompts and metric code, and trained each one on our 32B teacher where the method allows it, so that the comparison is like for like. The results are in the new Table 2 (Section 5.3), and Figure 2 plots quality against latency, with SPD on the Pareto frontier on every dataset. SPD is the only 0.6B method that nears parity to its 32B teacher: its AUC is within 4% of the teacher's on all four datasets, and 1% above it on Beauty, at 44–88× lower latency. ONE-PASS and Language Ranker at 0.6B stay at chance, since a head on a frozen small backbone cannot recover the teacher's ranking. The larger rerankers are better on quality. FIRST (7B) beats SPD even when we retrain it on our teacher, and fine-tuned DiffuRank (8B) beats it on three of the four datasets, at 12–13× the parameters and 7–9× the latency. PE-Rank, fully fine-tuned at 0.6B, is the strongest small baseline. We now say this directly in the paper: SPD's contribution is the quality it reaches per millisecond and per parameter, not the best quality overall.

Auto-regressive Baseline and Single-pass Scoring Baseline (Reviewers 6w8i and oHj6)

We added both baselines as controls. Each shares SPD's 0.6B backbone, LoRA, training data, step budget and teacher, and changes only the decoder. The AR student decodes the ranking token by token. The score+sort baseline keeps the single pass, but predicts one score per item, is trained with a Plackett–Luce loss, and ranks by sorting. Both are rows in Table 2 and are discussed in Section 5.4. We note that the submission already ablated Hungarian decoding: the linear-probe head in our head ablation (now Table 6) ranks items by argsorting its scores instead of solving the assignment, and it is below the self-attention head with Hungarian (AUC .5841 vs .5907). We now say this explicitly in Section 5.6, and the score+sort control repeats the comparison on the four public benchmarks. Against score+sort, SPD has higher AUC and higher Recall@10 on all four datasets (for example, Recall@10 of .602 vs .401 on MS MARCO C-50), while score+sort has higher Recall@1 on the two MS MARCO sets. The assignment formulation therefore buys whole-list quality rather than top-1 accuracy. We also decoded the trained SPD score matrix without Hungarian, by sorting items on their expected position. This raises AUC, since AUC only rewards where the relevant item lands, but Hungarian reproduces the teacher's permutation more faithfully. Our reading is that the item–position training objective is what drives quality; Hungarian is the right decoder when the whole permutation matters, and expected-position sorting is a free alternative when only the top does. On the Sinkhorn question (oHj6), removing it lowers AUC on every dataset, most on MS MARCO C-50 (Table 4).

Distinguishing the contributions of the proposed architecture from those of teacher distillation (Reviewers VYTL and 6w8i)

The AR student answers this question directly. Trained on the same teacher, it reaches about the same quality as SPD: slightly higher AUC on three datasets, lower on Beauty, and the same agreement with the teacher's order. It is however 14–41× slower, and it finishes a complete ranking on only 3–7% of slates. Our conclusion, which we now state in the abstract and in Section 5.4, is that distillation supplies the quality, and single-pass decoding supplies the speed and a valid permutation every time. This is a narrower claim than in the submission, and we believe it is the correct one. The ablations in Table 4 point the same way. Training SPD on click labels instead of the teacher, with LoRA (the cell missing from Table 3, oHj6), is at chance on MS MARCO C-50 and well below SPD elsewhere. Freezing the backbone is near chance everywhere except Quora. Both the teacher signal and backbone adaptation are needed.

AuthorsOpenReview ↗

Authors' Response (2/2)

[Author's response continued from prior comment]

Attribution of the Speed-ups (Reviewers 6w8i and oHj6)

The reviewers are right that the 64× headline mixes removing reasoning with the decoding itself, and we no longer lead with it. On the internal dataset the teacher is itself a 0.6B model, so the 3.1× over the teacher without reasoning (88 ms to 28 ms) is already a same-size, no-reasoning comparison. Table 1 now reports speed-ups relative to that teacher. On the public benchmarks, Table 7 (Appendix C) breaks the speed-up over the 32B teacher with reasoning into three parts: removing reasoning gives 6–10×, distilling to 0.6B while keeping autoregressive decoding gives 2–4×, and single-pass decoding gives a further 14–41×. The last factor is the one we attribute to SPD. The 45× on Beauty that oHj6 points out compared the 32B teacher with the 0.6B student, so it mixed distillation and decoding; the decomposition separates the two.

Scaling with slate size (Reviewers 6w8i, VYTL and oHj6)

Figure 3 now covers N = 10 to 150 for every ranker. SPD's latency comes from the prefill alone and grows roughly linearly, from 29 ms to 281 ms, and the Hungarian step stays under 3 ms even at N = 150. At N = 150, SPD is 46× faster than the 32B teacher and 45× faster than the AR student. On quality, SPD at N = 20 is close to its teacher (AUC .732 vs .745). At N = 100 the full head falls to chance, and we traced this to the teacher rather than to SPD: the 32B teacher names a median of only 16 of the 100 items, so most of each training target is input-order filler, which SPD learns to reproduce. Training a top-10 head on the items the teacher actually ranks fixes this, reaching AUC .814 against the teacher's .712. Both results are in Section 5.9, and the full-head failure is listed as a limitation. On VYTL's question about N > K: the score matrix need not be square. With K < N positions, the rectangular Hungarian assigns K distinct items to the K positions, and the remaining items follow in order of their best position score from the model. The loss uses only the teacher's top-K positions. With K > N, the solver fills N of the K positions and the ranking is read in position order. We describe this in Section 4.1. With K = 10 of N = 50, the top-K head beats the full head on AUC on every dataset (Table 2), because it never trains on the teacher's input-order tail.

Robustness to Permutations (Reviewer VYTL)

We re-scored every ranker on three random orderings and on the reversed order of each slate. SPD's AUC moves by at most 3% on every dataset, in line with the other trained rankers (FIRST and DiffuRank up to 3%, the 32B teacher at most 1%); Language Ranker-32B is the most sensitive, losing 10.5% on MS MARCO C-50. SPD's ordering of the tail does change with the input order, since that tail is learned from the teacher's input-order filler, but where the relevant item lands does not. On input length, no ranker moves by more than .05 AUC between the shortest and the longest third of slates. On domain, the datasets now span e-commerce, web passages and duplicate questions. Details are in Section 5.9 and Appendix F.

Fidelity to the Teacher (Reviewer 6w8i)

We now report Kendall τ against the teacher in Appendix E. Doing so surfaced a caveat: the 32B teacher names only 21–27 of the 50 items, so a τ over the full list mostly measures agreement on the input-order tail. We therefore compute τ on the items the teacher emits. SPD reaches .36–.45 against an input-order floor of .05–.27, the same as the AR student, while the 32B rankers, which share the teacher's backbone, agree more. Since τ is a linear function of pairwise agreement (τ = 2p − 1 when there are no ties), the same numbers answer the question about pairwise agreement.

Other Backbones and Scales (Reviewers 6w8i and VYTL)

SPD carries over to Llama-3.2-1B and to the hybrid-attention Qwen3.5 models with the same recipe; Appendix G (Table 12) now covers six backbones besides Qwen3-0.6B. Agreement with the teacher grows with size up to 4B, and at 4B SPD matches or exceeds its teacher's AUC on every dataset except Beauty. We also added three scoring heads (oHj6); none beats the default head on both datasets we tried (Table 11).

The 26 Decode Passes (Reviewer oHj6)

Thank you for catching this. The 26 is the mean number of tokens the zero-shot 0.6B model generates on Beauty before it stops, not the number of passes needed for a full ranking. In fact it almost never produces one: on the public sets it names about 3 items. Our parser keeps the items a model emits, in order, and appends the rest in input order, so every output is scored as a full permutation and no slate is dropped. The same parser is used for every autoregressive row. Even the 32B teacher completes a ranking on only about 1% of slates. Appendix D has the numbers. SPD, by contrast, outputs a valid permutation by construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.