- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SPD, a single-pass decoding method for generative reranking. It predicts an item-position score matrix from LLM hidden states and uses Hungarian matching to produce a valid permutation. Experiments on an internal dataset and Amazon Beauty report substantial latency reductions with relatively small ranking quality degradation.
Critical strengths
1.The paper addresses a practical latency bottleneck in generative reranking. On the internal dataset, SPD reduces inference latency from 88 ms to 28 ms compared with the non-reasoning teacher, while achieving similar ranking quality.
2.The method is simple and guarantees valid permutations through Hungarian matching, with negligible solver overhead at the evaluated slate size.
Critical weaknesses
1.Insufficient and unfair baseline comparisons. The paper only evaluates two datasets and does not compare against existing efficient rerankers or a simple single-pass scoring + sorting baseline. Moreover, the reported speedups are confounded by removing reasoning tokens (64×) or comparing a 32B teacher with a 0.6B student (45×), making it unclear how much improvement comes from the proposed decoding method itself. And lack of comparisons with existing efficient baselines.
2.Incomplete ablation studies. Table 3 lacks the click-label + LoRA setting, Table 4 only compares three scoring heads, and Table 5 reports latency for a single slate size (N=50). There is no ablation showing that Sinkhorn training or Hungarian decoding is necessary, nor any scalability analysis across different slate sizes.
Questions for authors
1.The paper discusses existing efficient reranking methods such as FIRST, ONE-PASS, PE-Rank and DiffuRank but does not compare against them experimentally. Can the authors provide quality–latency comparisons under comparable settings?
2.Can the authors compare SPD against a simple single-pass scoring + sorting baseline and ablate the Sinkhorn training objective? The current ablations do not establish the benefit of the proposed assignment-based formulation over simpler alternatives.
3.Table 2 reports only 26 decode passes for Qwen3-0.6B on Amazon Beauty, despite each slate containing 50 candidates. Could the authors clarify how complete rankings are generated and how incomplete or invalid outputs are handled during evaluation?
Authors’ Response (1/2)
We thank the reviewers for their thoughtful comments and questions. As several of the points are shared between reviewers, we will address them by topic. We have submitted an updated draft of the paper to address all the topics brought up by the reviewers. Changes are marked in blue. We used the extra page for the new results in the main text, and put the supporting tables in a new appendix. Due to comment length we are dividing this response in 2 parts.
Extra Baselines and Datasets (Reviewers 6w8i, VYTL, oHj6)
All reviewers noted the possibility of improving the work with extra baselines, both dataset and methods. We have done just that: adding the baselines FIRST, PE-Rank, ONE-PASS, DiffuRank, and Language Ranker. On the dataset side, we have added the requested datasets. Concretely, the paper now evaluates on four public benchmarks: Amazon Beauty, MS MARCO (Contriever top-50), BEIR Quora and BEIR MS MARCO, each with 1,118 evaluation slates of 50 candidates. Between them they cover product recommendation, web search and duplicate-question retrieval (VYTL). We also report TREC DL19 and DL20. We ran every baseline ourselves on the same slates, prompts and metric code, and trained each one on our 32B teacher where the method allows it, so that the comparison is like for like. The results are in the new Table 2 (Section 5.3), and Figure 2 plots quality against latency, with SPD on the Pareto frontier on every dataset. SPD is the only 0.6B method that nears parity to its 32B teacher: its AUC is within 4% of the teacher's on all four datasets, and 1% above it on Beauty, at 44–88× lower latency. ONE-PASS and Language Ranker at 0.6B stay at chance, since a head on a frozen small backbone cannot recover the teacher's ranking. The larger rerankers are better on quality. FIRST (7B) beats SPD even when we retrain it on our teacher, and fine-tuned DiffuRank (8B) beats it on three of the four datasets, at 12–13× the parameters and 7–9× the latency. PE-Rank, fully fine-tuned at 0.6B, is the strongest small baseline. We now say this directly in the paper: SPD's contribution is the quality it reaches per millisecond and per parameter, not the best quality overall.
Auto-regressive Baseline and Single-pass Scoring Baseline (Reviewers 6w8i and oHj6)
We added both baselines as controls. Each shares SPD's 0.6B backbone, LoRA, training data, step budget and teacher, and changes only the decoder. The AR student decodes the ranking token by token. The score+sort baseline keeps the single pass, but predicts one score per item, is trained with a Plackett–Luce loss, and ranks by sorting. Both are rows in Table 2 and are discussed in Section 5.4. We note that the submission already ablated Hungarian decoding: the linear-probe head in our head ablation (now Table 6) ranks items by argsorting its scores instead of solving the assignment, and it is below the self-attention head with Hungarian (AUC .5841 vs .5907). We now say this explicitly in Section 5.6, and the score+sort control repeats the comparison on the four public benchmarks. Against score+sort, SPD has higher AUC and higher Recall@10 on all four datasets (for example, Recall@10 of .602 vs .401 on MS MARCO C-50), while score+sort has higher Recall@1 on the two MS MARCO sets. The assignment formulation therefore buys whole-list quality rather than top-1 accuracy. We also decoded the trained SPD score matrix without Hungarian, by sorting items on their expected position. This raises AUC, since AUC only rewards where the relevant item lands, but Hungarian reproduces the teacher's permutation more faithfully. Our reading is that the item–position training objective is what drives quality; Hungarian is the right decoder when the whole permutation matters, and expected-position sorting is a free alternative when only the top does. On the Sinkhorn question (oHj6), removing it lowers AUC on every dataset, most on MS MARCO C-50 (Table 4).
Distinguishing the contributions of the proposed architecture from those of teacher distillation (Reviewers VYTL and 6w8i)
The AR student answers this question directly. Trained on the same teacher, it reaches about the same quality as SPD: slightly higher AUC on three datasets, lower on Beauty, and the same agreement with the teacher's order. It is however 14–41× slower, and it finishes a complete ranking on only 3–7% of slates. Our conclusion, which we now state in the abstract and in Section 5.4, is that distillation supplies the quality, and single-pass decoding supplies the speed and a valid permutation every time. This is a narrower claim than in the submission, and we believe it is the correct one. The ablations in Table 4 point the same way. Training SPD on click labels instead of the teacher, with LoRA (the cell missing from Table 3, oHj6), is at chance on MS MARCO C-50 and well below SPD elsewhere. Freezing the backbone is near chance everywhere except Quora. Both the teacher signal and backbone adaptation are needed.