- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies batched pointwise LLM scorers, which score multiple candidate documents in a single prompt and return per-candidate scores. The central observation is that reordering candidates within the prompt changes their scores, and while this often leaves ranking metrics like nDCG@10 essentially unchanged, it can meaningfully change the downstream decisions those scores feed into: what a threshold retains, what a reader answers from the top results, and which chosen/rejected pair is selected for preference training. The authors formalize this as "presentation dependence," decompose it into order, partition, marker, and skeleton channels, and show that prompt-time fixes (round-robin partitioning, logit calibration) do not improve decision stability. They then introduce order-consistency SFT (OC-SFT), a training objective that adds a penalty term pulling a candidate's scores under different permutations toward their mean, on top of a standard distillation loss. They evaluate extensively across passage reranking (18 collections), multi-document QA (3 collections), and response ranking (5 collections), using 12 base models from three families. OC-SFT consistently leads on decision-stability measures while maintaining comparable ranking quality, and a single OC-SFT permutation produces more reproducible decisions than ten averaged off-the-shelf permutations on two of three downstream consumers.
Critical strengths
The most consequential strength is the conceptual reframing itself. The paper identifies a real blind spot in how batched scorers are evaluated: ranking metrics do not capture decision stability, and two scorers that are indistinguishable on nDCG@10 can differ enormously in what a threshold actually retains. This is not a toy observation. It directly affects retrieval-augmented generation pipelines, preference data construction for RLHF, and any system where a score cutoff determines what gets surfaced. The fact that the paper backs this up with measurements on three separate downstream consumers (thresholding, reading, pair selection) across 26 evaluation collections makes it hard to dismiss as a niche issue. The practical takeaway, that comparisons of such scorers should report decision stability alongside ranking quality, is well earned by the evidence.
The second strength is the thoroughness and rigor of the experimental setup. Twelve base models across three families, three tasks, 18 reranking collections, multiple first stages, ablations on every design choice (number of views, teacher permutations, consistency weight, serving width, partition rule, answer skeleton), and a clear separation of what each experiment controls for. The controls are unusually careful: the same-evidence verdict-flip analysis, the pool-replacement transfer test, the engine-call batching vs. candidate-order decomposition, and the marker-renaming non-transfer result all anticipate questions a skeptical reader would ask. This is one of the more thorough empirical studies I have seen at this venue in recent cycles.
Critical weaknesses
The main weakness is that the practical impact on end-to-end system performance is never measured. The paper measures decision stability (retained-set overlap, verdict flip, pair flip) but never closes the loop: does a more stable scorer actually produce a better downstream model when used for preference training, or a more accurate RAG system when used for retrieval? The paper measures what changes, not whether those changes matter for the final user-facing output. The frozen-consumer setup isolates the scorer's contribution cleanly, which is good experimental methodology, but it leaves open the question of whether the instability that OC-SFT removes was actually hurting anything in practice. It is plausible that stochastic variation in preference pairs acts as a form of regularization, or that a retrained reader simply learns to be robust to the variation. Without at least one end-to-end experiment, the practical significance of the stability gains remains an assumption rather than a demonstrated fact.
A second, more contained weakness is that OC-SFT specifically addresses order dependence but does not generalize to other presentation channels the paper itself identifies. The marker-renaming experiment in Table 6b shows this concretely: order-consistency training does not transfer to marker sensitivity, and a separate marker-consistency penalty is needed. Order is clearly the dominant channel for the tasks studied, so this does not undercut the practical value of the method, but it does mean the solution is narrower than the problem as motivated. The paper frames "presentation dependence" broadly and then addresses one facet of it, without always making this boundary sharp.
Questions for authors
I thank the authors for the efforts they invested in this work. I would be happy to get answers to the following questions:
-
Have you run or considered running an end-to-end experiment where the scorer's output is used for preference training (e.g., DPO) and the downstream model's quality is compared across scorers with different stability levels? Even a small-scale version would substantially strengthen the practical motivation.
-
The within-window penalty transfers to cross-window perturbations measured by τ-PSI, but why? Is there an argument beyond "it works empirically" for why reducing within-window order variance should reduce sensitivity to changed window companions? The companion-coupling decomposition in Figure 5 is suggestive but does not directly explain the transfer.
-
ArguAna is a consistent outlier throughout the paper (lowest retained-set overlap for OC-SFT in Table 22, unusual width-effect decomposition in Table 27, one of the few collections where improvement is smallest). What is different about counterargument retrieval that makes it resistant to the method?
3 comments
Repsonse to authors' comments
I thank the authors for the thorough and substantive rebuttal. The new end-to-end experiments (§6.5, App. R) directly address my primary concern, and the additional analyses for Q2 and Q3 are appreciated. I update my assessment below. On W1/Q1 (end-to-end preference training). The new DPO experiment is the most important addition. The pair-quality numbers are convincing: OC-SFT's 69–75% gold-correct rate versus 49–57% for single-order distillation is a meaningful gap, and the fact that a single OC-SFT pass outperforms three averaged single-order passes (75.1% vs. 65.2%) is a clean demonstration. I appreciate that the authors also tested my two alternative hypotheses directly rather than dismissing them. That said, the downstream numbers themselves tell a more nuanced story than the pair-quality numbers suggest. The DPO policy gap between OC-SFT and single-order distillation is 0.17 points (40.78 vs. 40.61), well within seed variance, and the reward-model gap is 0.5 points (77.6 vs. 77.1). This is consistent with the authors' own observation that "the scoring order moves policy accuracy about as much as the training seed." So the new experiment simultaneously (a) validates that scorer instability produces measurably worse preference pairs, and (b) shows that DPO training is fairly robust to that difference. The practical case for OC-SFT is strongest in the direct-scoring regime (reranking, thresholding), where instability cannot be absorbed by a downstream learner, and the paper already makes that case well. I would encourage the authors to be forthright about this nuance in the final version rather than leading with pair quality alone. The honest reading is: OC-SFT matters most where scores are consumed directly; when scores feed into further training, the gap narrows considerably. On W2 (scope beyond order). The response is satisfactory. Targeting the dominant channel deliberately, rather than claiming full generality, is the right framing. The note that the same penalty generalizes to markers when applied over marker renamings is useful. On Q2 (transfer to changed companions). The causal-LM argument, that shuffles force the scorer to produce stable scores regardless of which (and how many) companions appear in the prefix, is intuitive and now backed by the 97% companion-coupling variance reduction (Table R3) and the Pool-PSI drop (0.102 to 0.031). This adequately explains the empirical transfer. On Q3 (ArguAna). The explanation that counterargument relevance induces roughly 2x higher per-document score variance, placing more documents within one SD of the cutoff, is plausible and supported by Table R4. This is a satisfying answer. Updated assessment. The rebuttal meaningfully strengthens the paper. The new end-to-end experiment closes the most important open question, even if the answer ("stability matters for direct scoring; DPO absorbs most of the noise") is more qualified than the framing suggests. The conceptual contribution (decision stability as a first-class evaluation axis) and the experimental thoroughness remain the paper's strongest assets. I maintain my score of weak accept.
Response to Reviewer nKSp
Thank you for looking into the revision carefully and for the updated assessment. We are glad that our new experiments and analyses resolve W1, W2 and Q1-Q3.
We agree with your takeaway on the two regimes. Where a system acts on each query's scores directly, nothing averages across queries, so OC-SFT's lower order noise improves both decision reproducibility (Table 1) and single-pass accuracy over the other trained scorers (Table R1). Where scores curate preference data (Table R7), an untrained scorer's errors carry into the reward model and policy, whereas between trained scorers the extra errors are mostly ties rather than inversions. A learner over thousands of pools therefore absorbs them, while OC-SFT still yields substantially more gold-correct pairs from the same scoring budget.
Our reply above opened with preference training because W1/Q1 asked about it. In the revised PDF, §6.5 already leads with direct single-pass accuracy before preference training, and we will state this distinction explicitly in §6.5 and the §7 conclusion in the final version. Thank you again!
Response to Reviewer nKSp
Thank you for the review, and for describing the problem as a blind spot in how batched scorers are evaluated. Your main question led to new end-to-end experiments in the revised PDF (in blue; §6.5, and App. R at the end of the PDF). OC-SFT selects markedly more gold-correct preference pairs, and trained scorers' pairs train a DPO policy almost as well as gold labels.
W1/Q1 (end-to-end preference training):
Pairs from either trained scorer bring the downstream models close to gold labels. Off-the-shelf pairs do hurt them, costing the reward model 3.6 points and the policy 1.4 points relative to OC-SFT's pairs. As LLM scorers increasingly label preference data, the scorer's quality thus carries into the models trained on its pairs, and OC-SFT yields the most correct pairs from the same candidate pools.
W1, your two alternative explanations (thanks for raising both)
W2 (scope beyond the order channel): We agree that OC-SFT targets one channel, and we target it deliberately, since it is the one a deployed system cannot fix in the prompt. Moreover:
Q2 (transfer to changed companions): we added App. R.4 for this.
Q3 (what sets ArguAna apart): Even on ArguAna, OC-SFT keeps the highest overlap of the trained variants (0.615 against 0.484-0.544, Table 22).
Your review noted that even a small end-to-end experiment would strengthen the practical case. We hope the new results do that, and if so, we would be grateful if you would take them into account in your assessment.