acceptodds
Under review as a conference paper at ICLR 2027

Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

Abstract

In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or which chosen/rejected pair enters preference training. Because the candidates share that prompt, reordering them changes their scores. The same query over the same candidates should still yield the same decision. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66–0.84 when reordered. No prompt-time change we test resolves that dependence: the only one that improves ranking quality does not measurably improve decision stability. We introduce order-consistency SFT (OC-SFT), which attenuates it in the weights by penalizing disagreement between a candidate's scores across orderings. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks. It is also more stable on 12 base models than order-averaged distillation, which trains on labels averaged across permutations. One OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison of such scorers should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://anonymous.4open.science/r/order-dependence-llm-scorers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM evaluation, reliability, position bias, permutation invariance, consistency regularization, reranking, reward models, preference learning, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nKSpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies batched pointwise LLM scorers, which score multiple candidate documents in a single prompt and return per-candidate scores. The central observation is that reordering candidates within the prompt changes their scores, and while this often leaves ranking metrics like nDCG@10 essentially unchanged, it can meaningfully change the downstream decisions those scores feed into: what a threshold retains, what a reader answers from the top results, and which chosen/rejected pair is selected for preference training. The authors formalize this as "presentation dependence," decompose it into order, partition, marker, and skeleton channels, and show that prompt-time fixes (round-robin partitioning, logit calibration) do not improve decision stability. They then introduce order-consistency SFT (OC-SFT), a training objective that adds a penalty term pulling a candidate's scores under different permutations toward their mean, on top of a standard distillation loss. They evaluate extensively across passage reranking (18 collections), multi-document QA (3 collections), and response ranking (5 collections), using 12 base models from three families. OC-SFT consistently leads on decision-stability measures while maintaining comparable ranking quality, and a single OC-SFT permutation produces more reproducible decisions than ten averaged off-the-shelf permutations on two of three downstream consumers.

Critical strengths

The most consequential strength is the conceptual reframing itself. The paper identifies a real blind spot in how batched scorers are evaluated: ranking metrics do not capture decision stability, and two scorers that are indistinguishable on nDCG@10 can differ enormously in what a threshold actually retains. This is not a toy observation. It directly affects retrieval-augmented generation pipelines, preference data construction for RLHF, and any system where a score cutoff determines what gets surfaced. The fact that the paper backs this up with measurements on three separate downstream consumers (thresholding, reading, pair selection) across 26 evaluation collections makes it hard to dismiss as a niche issue. The practical takeaway, that comparisons of such scorers should report decision stability alongside ranking quality, is well earned by the evidence.

The second strength is the thoroughness and rigor of the experimental setup. Twelve base models across three families, three tasks, 18 reranking collections, multiple first stages, ablations on every design choice (number of views, teacher permutations, consistency weight, serving width, partition rule, answer skeleton), and a clear separation of what each experiment controls for. The controls are unusually careful: the same-evidence verdict-flip analysis, the pool-replacement transfer test, the engine-call batching vs. candidate-order decomposition, and the marker-renaming non-transfer result all anticipate questions a skeptical reader would ask. This is one of the more thorough empirical studies I have seen at this venue in recent cycles.

Critical weaknesses

The main weakness is that the practical impact on end-to-end system performance is never measured. The paper measures decision stability (retained-set overlap, verdict flip, pair flip) but never closes the loop: does a more stable scorer actually produce a better downstream model when used for preference training, or a more accurate RAG system when used for retrieval? The paper measures what changes, not whether those changes matter for the final user-facing output. The frozen-consumer setup isolates the scorer's contribution cleanly, which is good experimental methodology, but it leaves open the question of whether the instability that OC-SFT removes was actually hurting anything in practice. It is plausible that stochastic variation in preference pairs acts as a form of regularization, or that a retrained reader simply learns to be robust to the variation. Without at least one end-to-end experiment, the practical significance of the stability gains remains an assumption rather than a demonstrated fact.

A second, more contained weakness is that OC-SFT specifically addresses order dependence but does not generalize to other presentation channels the paper itself identifies. The marker-renaming experiment in Table 6b shows this concretely: order-consistency training does not transfer to marker sensitivity, and a separate marker-consistency penalty is needed. Order is clearly the dominant channel for the tasks studied, so this does not undercut the practical value of the method, but it does mean the solution is narrower than the problem as motivated. The paper frames "presentation dependence" broadly and then addresses one facet of it, without always making this boundary sharp.

Questions for authors

I thank the authors for the efforts they invested in this work. I would be happy to get answers to the following questions:

  1. Have you run or considered running an end-to-end experiment where the scorer's output is used for preference training (e.g., DPO) and the downstream model's quality is compared across scorers with different stability levels? Even a small-scale version would substantially strengthen the practical motivation.

  2. The within-window penalty transfers to cross-window perturbations measured by τ-PSI, but why? Is there an argument beyond "it works empirically" for why reducing within-window order variance should reduce sensitivity to changed window companions? The companion-coupling decomposition in Figure 5 is suggestive but does not directly explain the transfer.

  3. ArguAna is a consistent outlier throughout the paper (lowest retained-set overlap for OC-SFT in Table 22, unusual width-effect decomposition in Table 27, one of the few collections where improvement is smallest). What is different about counterargument retrieval that makes it resistant to the method?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nKSp

Thank you for the review, and for describing the problem as a blind spot in how batched scorers are evaluated. Your main question led to new end-to-end experiments in the revised PDF (in blue; §6.5, and App. R at the end of the PDF). OC-SFT selects markedly more gold-correct preference pairs, and trained scorers' pairs train a DPO policy almost as well as gold labels.


W1/Q1 (end-to-end preference training):

  • Setup. We close the loop on math, where gold labels check every solution, over three seeds and three scoring orders.
    • Each scorer ranks 8 sampled solutions per problem from 3 policy models, and we pair its top- and bottom-scored solution, then train reward models on these pairs.
    • We train an OLMo-2 1B DPO policy on its own solutions, with the rejected one drawn from the lower half of the ranking.
  • OC-SFT selects the most gold-correct pairs. Its top-versus-bottom pairs put a correct solution above an incorrect one in 69-75% of cases across the 3 models, against 49-57% for single-order distillation and 39-46% off the shelf. One OC-SFT pass gives better pairs than 3 averaged single-order passes (75.1% vs 65.2% on Qwen3-4B).
Training pairsReward model, held-out acc. (Qwen3-1.7B)DPO policy, GSM8K/MATH-500 mean
Gold81.340.93
Random53.233.04
Off the shelf74.039.40
Single-order77.140.61
OC-SFT77.640.78

Pairs from either trained scorer bring the downstream models close to gold labels. Off-the-shelf pairs do hurt them, costing the reward model 3.6 points and the policy 1.4 points relative to OC-SFT's pairs. As LLM scorers increasingly label preference data, the scorer's quality thus carries into the models trained on its pairs, and OC-SFT yields the most correct pairs from the same candidate pools.

W1, your two alternative explanations (thanks for raising both)

  1. We found no evidence that order variation acts as a regularizer: the off-the-shelf scorer, whose pairs vary most across orders, trains the weakest models.
  2. A learner absorbs the noise in DPO, but not where scores decide directly.
    • With DPO, single-order and OC-SFT pairs end within 0.2 points, and the scoring order moves policy accuracy about as much as the training seed.
    • In reranking, the scores themselves decide, so OC-SFT's stability pays off directly: one OC-SFT pass is more accurate than 10 averaged off-the-shelf passes, and it loses the least to a single random order (0.007 nDCG@10, against 0.019 for single-order distillation and 0.049 off the shelf; Table R1).

W2 (scope beyond the order channel): We agree that OC-SFT targets one channel, and we target it deliberately, since it is the one a deployed system cannot fix in the prompt. Moreover:

  • Order is the dominant channel for the tasks we study, as you note, and the only one that changes with every query (§3).
  • Holding the order fixed only selects one of the order-dependent outputs (§1).
  • The same penalty extends to other channels. As you note from Table 6b, the order penalty does not carry over to markers, but applied over marker renamings it does reduce marker sensitivity.

Q2 (transfer to changed companions): we added App. R.4 for this.

  • Shuffles vary which companions precede a candidate. OC-SFT's training shuffles keep a candidate's companions but change its slot and their order (§4, footnote 2). Since the scorer is a causal language model, a candidate's encoding depends on the companions before it, which across shuffles can be none or all 19 of them. Keeping a score stable across shuffles therefore pushes the score to depend less on which companions precede a candidate, so we expect the effect to carry over when a τ-PSI shuffle changes the companions.
  • Measurements showing the transfer. OC-SFT removes 97% of the score variance consistent with companion coupling (App. C.1, Table R3). When the companions themselves change under pool replacement, Pool-PSI falls from 0.102 off the shelf to 0.031, against 0.099 for single-order distillation (Table 5).

Q3 (what sets ArguAna apart): Even on ArguAna, OC-SFT keeps the highest overlap of the trained variants (0.615 against 0.484-0.544, Table 22).

  • We rule out small set size (three collections retain fewer documents and all keep higher overlap) and score bunching near the cutoff (Table R4).
  • Instead, a document's score varies across orders ~2x as much as elsewhere, placing more documents within one score SD of the cutoff than on any other collection (Table R4).
  • The gap opens only during training on MS MARCO, and we attribute it to ArguAna's notion of relevance, where the relevant passage argues against the query.

Your review noted that even a small end-to-end experiment would strengthen the practical case. We hope the new results do that, and if so, we would be grateful if you would take them into account in your assessment.

Reviewer_nKSpOpenReview ↗

Repsonse to authors' comments

I thank the authors for the thorough and substantive rebuttal. The new end-to-end experiments (§6.5, App. R) directly address my primary concern, and the additional analyses for Q2 and Q3 are appreciated. I update my assessment below. On W1/Q1 (end-to-end preference training). The new DPO experiment is the most important addition. The pair-quality numbers are convincing: OC-SFT's 69–75% gold-correct rate versus 49–57% for single-order distillation is a meaningful gap, and the fact that a single OC-SFT pass outperforms three averaged single-order passes (75.1% vs. 65.2%) is a clean demonstration. I appreciate that the authors also tested my two alternative hypotheses directly rather than dismissing them. That said, the downstream numbers themselves tell a more nuanced story than the pair-quality numbers suggest. The DPO policy gap between OC-SFT and single-order distillation is 0.17 points (40.78 vs. 40.61), well within seed variance, and the reward-model gap is 0.5 points (77.6 vs. 77.1). This is consistent with the authors' own observation that "the scoring order moves policy accuracy about as much as the training seed." So the new experiment simultaneously (a) validates that scorer instability produces measurably worse preference pairs, and (b) shows that DPO training is fairly robust to that difference. The practical case for OC-SFT is strongest in the direct-scoring regime (reranking, thresholding), where instability cannot be absorbed by a downstream learner, and the paper already makes that case well. I would encourage the authors to be forthright about this nuance in the final version rather than leading with pair quality alone. The honest reading is: OC-SFT matters most where scores are consumed directly; when scores feed into further training, the gap narrows considerably. On W2 (scope beyond order). The response is satisfactory. Targeting the dominant channel deliberately, rather than claiming full generality, is the right framing. The note that the same penalty generalizes to markers when applied over marker renamings is useful. On Q2 (transfer to changed companions). The causal-LM argument, that shuffles force the scorer to produce stable scores regardless of which (and how many) companions appear in the prefix, is intuitive and now backed by the 97% companion-coupling variance reduction (Table R3) and the Pool-PSI drop (0.102 to 0.031). This adequately explains the empirical transfer. On Q3 (ArguAna). The explanation that counterargument relevance induces roughly 2x higher per-document score variance, placing more documents within one SD of the cutoff, is plausible and supported by Table R4. This is a satisfying answer. Updated assessment. The rebuttal meaningfully strengthens the paper. The new end-to-end experiment closes the most important open question, even if the answer ("stability matters for direct scoring; DPO absorbs most of the noise") is more qualified than the framing suggests. The conceptual contribution (decision stability as a first-class evaluation axis) and the experimental thoroughness remain the paper's strongest assets. I maintain my score of weak accept.

AuthorsOpenReview ↗

Response to Reviewer nKSp

Thank you for looking into the revision carefully and for the updated assessment. We are glad that our new experiments and analyses resolve W1, W2 and Q1-Q3.

We agree with your takeaway on the two regimes. Where a system acts on each query's scores directly, nothing averages across queries, so OC-SFT's lower order noise improves both decision reproducibility (Table 1) and single-pass accuracy over the other trained scorers (Table R1). Where scores curate preference data (Table R7), an untrained scorer's errors carry into the reward model and policy, whereas between trained scorers the extra errors are mostly ties rather than inversions. A learner over thousands of pools therefore absorbs them, while OC-SFT still yields substantially more gold-correct pairs from the same scoring budget.

Our reply above opened with preference training because W1/Q1 asked about it. In the revised PDF, §6.5 already leads with direct single-pass accuracy before preference training, and we will state this distinction explicitly in §6.5 and the §7 conclusion in the final version. Thank you again!

Reviewer_BTHEOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Use of LLMs as scorers in tasks such as passage reranking, response ranking, and multi-document question answering traditionally entails various prompt-level dependencies that can result in variation in rankings, given the same candidates. In particular, for scorers that ingest lists of candidates for scoring within a single context window, the order of presentation of those candidates within the context window can alter their score assigned to each candidate, affecting both the overall score magnitudes and the relative rankings of the candidates. The authors present OC-SFT, a novel, training-based method that reduces ranking variance in LLM scorers of batched candidates. OC-SFT trains a LoRA adapter to a frozen base model to generate output scores across batched candidate inputs with minimized variance in scoring. Unlike order-averaged distillation methods that use averaged candidate scores from a teacher model over multiple permutations of the input candidates as labels, OC-SFT only requires a single iteration of the teacher model, which is the same base model as the student using a random permutation of the candidates, to generate a distillation label. To ensure order invariance in the student’s output, the system minimizes the student’s score variance (per candidate) by computing a consistency regularization penalty (i.e., with respect to the student score per permutation per candidate relative to the average student score per candidate across permutations). Compared with various untrained, trained, and purpose-built external reranking systems, OC-SFT consistently achieves best order stability and on-par ranking quality across varying batch sizes.

Critical strengths
  1. The author’s proposed method achieves new SOTA LLM scoring consistency across various scoring tasks (e.g., passage reranking, multi-document QA, and response ranking), as well as across varying candidate batch sizes, maintaining near-SOTA scoring quality while achieving significant improvement in scoring consistency across candidate order permutations.
  2. The author’s uniquely and appropriately focus on the consistency of LLM scores (rather than quality). Competing methods (e.g., batched self-consistency or order-averaged distillation) achieve competitive or superior score quality but fall far behind in score consistency. The author’s proposed method maintains competitive score quality while significantly improving score consistent, ensuring that downstream task decisions remain constant across candidate order permutations.
  3. Although other trained (e.g., order-averaged distillation) and untrained (e.g., batched self-consistency) methods achieve comparable or superior score quality, as noted in strength (2), they lack the score consistency of OC-SFT and, importantly, entail significant additional cost during training (i.e., order-averaged distillation) and inference (i.e., batched self-consistency), typically requiring inference over 10x as many permutations as OC-SFT.
Critical weaknesses
  1. In Section 6.1, the authors make a potentially misleading or inaccurate claim regarding the relative performance of OC-SFT and GPT-5.4.
  2. In certain areas, the paper lacks clarity by providing insufficient detail regarding statements that justify key architectural or metric reporting decisions. We have included a question requesting clarification of each such instance.
  3. The paper could include more detailed justification for the limited scope of its quality measures. These decisions appear plausible but require more specific discussion / support.
Questions for authors
  1. Can you clarify the results on the relative performance of OC-SFT with respect to GPT-5.4? You state that “GPT-5.4 leads every variant we trained on nDCG@10 while reproducing less of its retained set than OC-SFT”, it appears from Table 1 that GPT-5.4 outperforms OC-SFT on nDCG@10 AND retained set overlap on 2 of the 3 tasks (i.e., passage reranking and multi-doc QA).

  2. There are various instances in the paper of imprecise or unclear statements that result in confusion about the point in discussion. Please provide clarification of each such statement below:

  • In Section 3 (subsection “Measuring what the consumers produce”) , you state “For each consumer, we report output quality and how often the output changes, since a scorer could hold it fixed by making it worse.” What does it mean that “the scorer could hold it fixed by making it worse”?
  • In Section 3 (subsection “Width and scope”) , you state “The decomposition runs over permutations of a fixed candidate set, so replacing a candidate changes x rather than pi and falls outside it.” What is the significance of this statement?
  1. Can you provide more detail on the typical use cases of LLM scorers to validate the use of only nDCG@1 / nDCG@10 scoring quality metrics, as opposed to using a larger head size and/or incorporating metrics ranking tail performance?
2 comments
AuthorsOpenReview ↗

Response to Reviewer BTHE

Thank you for the review, and for highlighting OC-SFT's consistency gains across tasks and batch sizes without the 10x inference cost of averaging over permutations. Each of your three points led to a change in the revised PDF (in blue, with new appendix material in App. R at the end of the PDF). We clarified the GPT-5.4 comparison in §6.1 and both §3 sentences, and added deeper and full-pool quality metrics in App. R.3 (Table R2).


W1/Q1 (the GPT-5.4 sentence in §6.1): we agree the sentence was too broad, and §6.1 now states both cases:

  • Passage reranking. The sentence concerned this task only - GPT-5.4 has higher nDCG@10 (0.468 vs 0.459) but lower retained-set overlap (0.707 vs 0.835).
  • QA and response ranking.
    • GPT-5.4 has higher quality and changes fewer answers and preferred pairs, but it gives nearly half of all candidate pairs the same grade.
    • When tied candidates keep their input order, it changes more than OC-SFT (0.200 vs 0.125 answers, 0.851 vs 0.661 pairs; Table 1, note b).

W2/Q2 (the two §3 sentences on quality and scope): thank you; we rewrote both in §3.

  1. Quality and output change. The first sentence now explains that we report both together because a scorer that ranks candidates by ID would never change its output, and so would look perfectly stable while ignoring relevance.
  2. Scope. The second sentence now states that the decomposition varies only the order of a fixed candidate set, since order carries no information about relevance while the candidate set may legitimately inform a score.
    • We measure changes in which candidates share a window with τ-PSI, which repartitions the same pool, and changes to the pool itself with pool replacement (App. C, Table 5).

W3/Q3 (ranking quality beyond nDCG@10):

  • Why nDCG@10? It is the standard reranking metric (Craswell et al., 2020; Thakur et al., 2021). On QA it covers all 10 candidates, including the 5 our reader sees, and on response ranking nDCG@1 fits consumers that use the top response.
  • Why deeper metrics? You are right that our passage threshold looks beyond the top 10, since it keeps about 19 passages per query. We therefore added deeper and full-pool metrics for both tasks (Table R2).
  • Every deeper metric ranks the variants as nDCG@10 does. Single-order is lowest and OC-SFT highest on each.
Single-pass quality, mean over 10 random ordersPassage rerankingResponse ranking
nDCG@10nDCG@20nDCG@100MAP@100full-pool nDCGpairwise acc.
Single-order0.4600.4460.4160.2420.8500.701
Order-avg., DebiasFirst, perm. aug.0.466-0.4670.4530.4210.2490.863-0.8640.737-0.740
OC-SFT0.4720.4570.4240.2520.8650.744


Do these changes resolve your concerns? If so, we would be grateful if you would consider updating your score.

AuthorsOpenReview ↗

Follow-up for Reviewer BTHE on W1-W3

Thank you again for your review. Before the discussion period closes on Wednesday, Nov 18 AoE, we wanted to check whether the revised PDF addresses your three points:

  • W1/Q1 (GPT-5.4): we narrowed the sentence in §6.1, which now states the comparison for each task separately.
  • W2/Q2 (§3 clarity): we rewrote both sentences you quoted, including an example of a scorer that ranks candidates by ID and so never changes its output.
  • W3/Q3 (deeper metrics): we added nDCG@20, nDCG@100, MAP@100, full-pool nDCG as well as pairwise accuracy; on each, single-order is lowest and OC-SFT highest (App. R.3, Table R2).

Thank you again, and we would be glad to clarify anything that is still unclear.

Reviewer_ZsWoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies batched pointwise LLM scoring, where candidates share one prompt, so reordering them changes their scores. Rather than ranking metrics alone, the authors measure how often downstream decisions change: a frozen threshold's retained set, a frozen reader's answer, and a selected chosen/rejected pair. They propose OC-SFT, self-distillation plus a penalty on score variance across two shuffled orders. On Qwen3-4B, OC-SFT matches the other trained variants on nDCG@10 (0.459 vs. 0.449–0.455) while lowering τ-PSI from 0.209 to 0.083 and raising retained-set overlap from 0.656 to 0.835 (Table 1). The stability gain holds across 12 base models, three first stages, and pool replacement.

Critical strengths
  1. Thorough experiments. The main result holds across model families and sizes, all three tasks, and different first-stage retrievers. The validity controls rule out the obvious alternative explanations, such as tied scores or readers simply getting different evidence. The gain also transfers to pool replacement, which the method was not trained on. I find the main empirical claim credible.

  2. Useful framing at low cost. The paper shows that scorers with nearly the same nDCG@10 can produce quite different threshold outputs (Table 22), so ranking metrics alone are not enough to compare them. The method itself is simple: it needs one teacher pass for labels and adds no inference cost at serving time.

Critical weaknesses
  1. OC-SFT may be largely a re-weighted form of permutation augmentation. Augmentation fits every shuffled view to the teacher target, and mean_i (s_i − ỹ)² = (s̄ − ỹ)² + mean_i (s_i − s̄)², so it already penalizes cross-view variance with weight 1. OC-SFT’s effective weight is about 1 + λ = 6, since its anchor term contributes one more. The gain attributed to the explicit penalty (§6.3) may therefore come from a larger weight rather than a new objective. Tuning also favors OC-SFT. Although λ is selected on held-out ranking quality rather than stability, it lands at the top of a six-value grid, giving OC-SFT the largest variance weight considered, while augmentation has no such weight to tune.
  2. The practical case is narrower than the headline suggests.
  • An order-free alternative is left out of the headline comparison. On passage reranking, the primary task, one batched pass is no better in quality than scoring candidates alone (Table 16). A pointwise B = 1 student, which is order-free by construction, matches OC-SFT in quality (Table 28b), at roughly 1.8× the latency of B = 20 (Table 31). This alternative is discussed in §6.4 but does not appear in Table 1 or the abstract.
  • The benefit is mixed where batching helps, and stability is not shown to bring accuracy. OC-SFT beats ten-pass averaging on QA but not on response-ranking pair flip (Table 1). Retained-set F1 is level with the other trained variants (Table 22), and reader accuracy is level with single-order distillation (Table 23d).
  • Retention is not matched. Overlap rises with retained-set size, yet per-variant set sizes are not reported, and the only matched-retention check i could find (App. D.6) compares against the off-the-shelf scorer rather than the trained baselines. Since the paper’s main recommendation is to report decision stability, it should show overlap at matched retention and compare against the order-free alternative.
Questions for authors
  1. How do τ-PSI, retained-set overlap and nDCG@10 change with λ, including values above 5? How does permutation augmentation perform with an added variance term at weight λ = 5? If the gap closes, the contribution is the weighting rather than a new objective.
  2. What is the retained-set size of each variant in Table 1, and how do the trained variants compare when thresholded to a common retention fraction?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZsWo

Thank you for the thorough review, for finding the main empirical claim credible, and for connecting OC-SFT to permutation augmentation in W1. You are right that augmentation penalizes cross-view variance at a fixed weight. In the revised PDF (in blue; App. R at the end), we state the identity in §4 and run both tests you proposed. Your augmentation variant matches OC-SFT, as you anticipated (App. R.1), and at matched retention OC-SFT keeps the highest overlap of the trained variants (App. R.2). We also added end-to-end accuracy results (§6.5, App. R.6).


W1/Q1 (relation to permutation augmentation)

  • Identity (§4). Augmentation's loss is an anchor on the view mean plus a variance penalty. In expectation, OC-SFT raises that penalty's weight from 1 to 1 + 4λ, 21 times at λ = 5.
  • Weight curve (App. R.1, Figure R1). We retrained Qwen3-4B at λ ∈ {0, 0.25, 1, 2, 5, 10, 20} as well as augmentation alone and your proposed variant, augmentation + 5 (s1 − s2)², which equals λ = 5 in expectation.
    • From λ = 0 to 5, mean τ-PSI falls from 0.124 to 0.082 and mean retained-set overlap rises from 0.77 to 0.85; both improve on each of the 16 public collections, and overlap also at matched retention,
    • nDCG@10 stays within 0.469-0.480 across the range,
    • Augmentation matches λ = 0 and your variant matches λ = 5.
  • A larger weight changes the decisions at level ranking quality. Augmentation's fixed weight closes about two thirds of the τ-PSI gap between single-order distillation and OC-SFT, and the larger weight closes the rest at the same training cost (§6.3). We therefore present both in §4 as one family with a tunable variance weight, where augmentation fixes the weight and OC-SFT sets it.
  • The default λ = 5 is conservative for stability. We select λ on held-out ranking quality alone, by the same rule that picks every variant's checkpoint (App. B.3), so the tuning never targets stability. On the extended curve, overlap keeps rising up to λ = 10 before levelling off at similar nDCG@10.

W2a (the order-free B = 1 alternative): As you note, a pointwise B = 1 student is order-free and matches OC-SFT on passage reranking (0.466 vs 0.459). On the other two tasks, the batched scorer is ahead, and we revised §6.4 to set out when each design fits:

  • QA: batching adds 0.043 nDCG@10 over serving the same checkpoint one candidate at a time, since co-present candidates inform the score (Table 16).
  • Response ranking: OC-SFT at B = 4 beats both B = 1 students by 0.033-0.086 nDCG@1 on 3 out of 4 collections in Table 19b.
  • Passage reranking: at equal quality, batching cuts per-query latency by 45%, and one batched checkpoint serves any width from 1-30 candidates, whereas the pointwise student degrades when served 20 at once
  • Labels: even the B = 1 student trains better on labels from a batched teacher, by 0.029 nDCG@10 (Table 11).

W2b (benefit where batching helps, and accuracy):

  1. 10 passes vs. 1: On response ranking, 10 averaged off-the-shelf passes do change fewer pairs than one OC-SFT pass (0.636 vs 0.661, Table 1), at 10x the inference. At an equal number of passes, OC-SFT leads on all three consumers (Figure 4, App. D.6).
  2. On set consumers, OC-SFT cuts flips most at full accuracy: even 10 averaged passes add at most 0.013 F1 and no reader EM, so order noise costs reproducibility here (Table R6).
  3. Where the scores themselves rank, stability does bring accuracy:
    • Scoring the pool in one random order instead of averaging 10 loses 0.049 nDCG@10 off the shelf, 0.019 for single-order distillation and 0.007 for OC-SFT, and the other trained scorers, permutation augmentation included, need 1.4-1.9 averaged passes to match one OC-SFT pass (Table R1).
    • On verifiable math solutions (§6.5, App. R.6), OC-SFT also selects better preference pairs. Its top-versus-bottom pairs put a correct solution above an incorrect one in 69-75% of cases, against 49-57% for single-order distillation.

W2c/Q2 (overlap at matched retention):

  • At their own F1-tuned cutoffs, Table 1 systems retain 16.2-20.4 passages per query; OC-SFT (19.1) retains fewer than GPT-5.4 (20.1), jina-v3 (19.3), DebiasFirst and perm. aug.
  • Set size does not explain the overlap gap: at a common retention fraction per collection, each row below moves by at most 0.003 and OC-SFT leads on all 18 collections, as at five fixed fractions from 0.05 to 0.50 (App. R.2).
Passage reranking, 18 collectionsDocs/query, own cutoffOverlap, own cutoffOverlap, common retention
Single-order18.40.6620.661
Order-averaged19.10.7430.745
DebiasFirst20.40.7590.756
Perm. aug.20.10.7600.759
OC-SFT19.10.8350.838

With both proposed tests run, we hope the revision resolves your concerns, and we would be grateful if you would consider updating your score.

AuthorsOpenReview ↗

Follow-up for Reviewer ZsWo on proposed tests

Dear Reviewer ZsWo, thank you again for proposing two concrete tests in your review. As the discussion closes on Wednesday AoE, we wanted to check whether our new results resolve your questions:

  • W1/Q1 (consistency weight): raising λ from 0 to the default 5 makes the retained set more reproducible and lowers τ-PSI on each of the 16 public collections, at similar nDCG@10. Augmentation alone lands at λ = 0, your proposed variant at λ = 5 (§4, App. R.1).
  • W2c/Q2 (matched retention): at a common retention fraction, OC-SFT keeps the highest overlap of the trained variants on all 18 collections (App. R.2)
  • W2b (accuracy): in a single pass, OC-SFT is the most accurate trained scorer in Table R1, and on math it selects more gold-correct preference pairs than single-order distillation (§6.5, App. R.6).

Thank you again for your feedback! We would be glad to answer anything that remains open.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers for their thoughtful feedback and are encouraged that all three accept the paper's central finding, that LLM scorers of equal ranking quality can still make different decisions when the candidate order changes. nKSp calls it "a real blind spot in how batched scorers are evaluated" and the study "one of the more thorough empirical studies I have seen at this venue in recent cycles". ZsWo finds "the main empirical claim credible", and BTHE notes that OC-SFT reaches this consistency without the 10x inference of averaging over orders.

We uploaded a revised PDF with changes in blue and all new appendix analyses grouped in App. R at the end of the PDF. Its main addition, a new §6.5 (details in App. R.6), follows the scorer's decisions to accuracy and to the models trained on them (nKSp W1, ZsWo W2b).


Closing the loop, from scorer to trained model (§6.5):

The submission showed that the order changes what 3 frozen consumers decide, and that OC-SFT makes all 3 decisions more reproducible than any other trained scorer at equal ranking quality (Table 1, Figure 4). We now follow one of these decisions, the choice of a preference pair, into training. On math, where gold labels check every solution, each scorer selects pairs from a policy's own sampled solutions, and we train reward models on its top-versus-bottom pairs and an OLMo-2 1B DPO policy on pairs with a rejected solution from its lower half (App. R.6).

  • An order-stable scorer builds more useful pairs. OC-SFT's top- and bottom-scored solutions form a correct-over-incorrect pair in 69-75% of cases across 3 policy models, against 49-57% for single-order distillation and 39-46% off the shelf. One OC-SFT pass beats even 3 averaged single-order passes (75.1% vs 65.2%). Most other pairs are ties, so from the same labelling budget OC-SFT yields more pairs that carry a correctness signal.
  • Pairs from a trained scorer train models close to gold pairs. The DPO policy nearly matches gold pairs. The reward model recovers most of the gain that gold pairs give over random pairs, while off-the-shelf pairs leave both models measurably behind.
Training pairsReward model (Qwen3-1.7B), held-out acc.DPO policy (OLMo-2 1B), GSM8K/MATH-500 mean
Gold81.340.9
Random53.233.0
Off the shelf74.039.4
Single-order77.140.6
OC-SFT77.640.8

As LLM scorers increasingly label preference data, the scorer's quality carries into the models trained on its pairs. Because the DPO learner absorbs the remaining order noise, both trained scorers come close to gold labels, and OC-SFT's advantage shows where nothing absorbs it, in the most correct pairs from the same candidate pools and in single-pass accuracy.

Where scores are used directly, a single order costs accuracy, and OC-SFT loses the least. One OC-SFT pass is more accurate than 10 averaged off-the-shelf passes on passage reranking, and single-order distillation needs about 3-5 averaged passes to match it on reranking and response ranking (Table R1).


Further additions and clarifications from the reviews:

  • For ZsWo: the consistency-weight curve, including ZsWo's proposed variant, shows that a larger weight makes the decisions more reproducible at level ranking quality (§4, §6.3, App. R.1), and OC-SFT keeps the highest overlap of the trained variants at matched retention on all 18 collections (§6.1, App. R.2).
  • For nKSp: we show why the within-window penalty reaches changed companions (App. R.4), trace ArguAna's lower overlap to larger score movement across orders (App. R.5), and state the method's scope in §3.
  • For BTHE: we clarified the GPT-5.4 comparison in §6.1 and the two §3 sentences, and added deeper and full-pool ranking metrics, which rank the variants as nDCG@10 does (App. R.3).

We give the detailed answers, with the numbers each reviewer asked for, in the replies under each review.

AuthorsOpenReview ↗

Authors' summary of the discussion

Dear Area Chair,

All three reviewers accept the paper's central finding, that LLM scorers of equal ranking quality can still make different decisions when the candidate order changes. The reviews then asked whether the instability costs anything in practice and what OC-SFT adds over permutation augmentation; our evision tests both.

  • nKSp asked whether the scorer's decisions affect the models trained on them. In the new end-to-end experiment, OC-SFT selects more gold-correct preference pairs than single-order distillation (§6.5). nKSp has since confirmed that the revision resolves W1, W2 and Q1-Q3.
  • ZsWo proposed two tests, both of which we ran. Raising the consistency weight from augmentation's level to our default makes the retained set more reproducible on each of the 16 public collections at similar nDCG@10 (W1, App. R.1). At matched retention, OC-SFT keeps the highest overlap of the trained variants on all 18 collections (W2c, App. R.2). One OC-SFT pass also matches 1.4-1.9 averaged passes of all other trained scorers, permutation augmentation included (W2b, Table R1).
  • BTHE's review led us to clarify the GPT-5.4 comparison in §6.1 and two sentences in §3, and to add ranking metrics beyond nDCG@10 (Table R2). OC-SFT is highest among the trained variants on every added metric.

We hope the reviewer discussion can take up whether these results resolve ZsWo's and BTHE's concerns. We thank the reviewers for their feedback, and you for coordinating the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.