acceptodds
Under review as a conference paper at ICLR 2027

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

Abstract

Small leaderboard gaps are often interpreted as evidence that one language model is better than another, but their sign may depend on which benchmark items are included. We test this using item-level responses from five benchmarks and a family-label-free spectral approximation to multidimensional item-response theory (MIRT). In owner-disjoint folds, one owner half identifies items with low residual differential item functioning across model families (low-DIF). These items are used to score models in the other half with frozen weights that preserve the benchmark's composition across metadata-defined item groups and easiness strata. Equally short matched-random subtests provide a baseline for variation due to item subsampling. Full-benchmark and low-DIF rankings remain strongly correlated (–). Yet in four of five benchmarks, 30.9–47.1% of cross-family pairs initially within one percentage point reverse order, exceeding their matched-random medians by 16.9–28.6 percentage points (all ). The fifth benchmark shows no reliable excess ( points, ). The pattern survives all pre-specified population perturbations, and residual item–family signatures replicate across owner halves; however, no family shows a consistent advantage across benchmarks. Thus, globally stable rankings can still leave individual near-tie orderings sensitive to benchmark composition, and sub-one-point leaderboard gaps should be accompanied by evidence that the implied ordering is composition-robust.

Primary area:
datasets and benchmarks
Keywords:
language model evaluation, benchmark robustness, leaderboard reliability, item response theory, differential item functioning, ranking stability

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KLRzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper looks at whether model rankings in leaderboards are robust to the specific composition of each benchmark (ie which specific samples are including in the benchmark). Their method builds subsets of the same benchmarks which consist of low differential functioning samples and compares scores on those benchmarks to scores on a random subsample of the same benchmarks. They find that while scores are very correlated in both settings, there is still a noticeable amount of ranking reversals, particularly within pairs of models that originally have close scores.

Critical strengths

This paper is very well presented, motivated, and easy to follow. For instance I found figure 1 very helpful and visually pleasing. From my understanding (not very familiar with this field), their method appears sound and results robust.

Critical weaknesses
  1. I am not very familiar with this field, but the authors' main finding, that rankings are not always stable, especially when initial score differences are small, does not seem very surprising. I can also therefore not properly assess novelty and impact of the work.
  2. The authors should further elaborate on takeaways from their method. Can this be used to inform either the construction of benchmarks or the interpretation of benchmark results?
  3. The authors' analysis does not include top frontier models like claude gemini or chatgpt. Even though from my understanding open models are not required for this analysis?
Questions for authors
  1. Can you please elaborate on what figure 2 shows? How correlated are ranking reversals to the initial difference in performance between both models?
  2. How do you explain that reversal rates are different for different benchmarks? For instance why is WinoGrande so different?
  3. Are their other methods that assess confidence in benchmark results that this work should compare to? Or even simple statistical methods like bootstrapping or confidence intervals?
Reviewer_gwUiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether near-tied model rankings change when benchmarks are recomposed using family-related differential item functioning (DIF). It estimates a family-blind multidimensional capability representation, identifies residual family-related item effects, and constructs low-DIF subsets while preserving benchmark blueprint constraints. Owner-disjoint cross-fitting separates subset construction from ranking evaluation. Across five benchmarks, four show substantially more near-tie reversals than matched random recompositions, while WinoGrande shows no reliable excess under the primary specification. Extensive controls examine discrimination, reliability, model-population choices, alternative capability estimation, and capability-profile matching.

Critical strengths
  1. The result goes beyond the unsurprising observation that near-ties can reverse. The relevant comparison is excess reversal relative to length- and blueprint-matched random subsets. The primary excesses are approximately 16.9-28.6 percentage points on the four positive benchmarks, despite relatively high overall rank correlations. This identifies a practically important distinction between global leaderboard stability and reliability of small claimed improvements.

  2. The validation is comprehensive and appropriately bounded. Owner-disjoint construction, discrimination and reliability controls, population perturbations, and the direct-logistic-MIRT sensitivity analysis address several plausible artefacts. The paper retains the negative WinoGrande result and explicitly does not cliam that low-DIF subsets are intrinsically better benchmarks. These features strengthen the credibility of the central stress-test finding.

Critical weaknesses

I did not identify an acceptance-blocking weakness. The principal limitations concern the magnitude and scope of the findings, and the paper largely acknowledges them.

  1. Effect sizes remian estimator-dependent. Direct logistic MIRT preserves the four-benchmark qualitative pattern but materially reduces some excess reversal rates, including MMLU from approximately 24.4 to 9.2 percentage points. Capability-profile matching also exposes limited common support. Thus, the headline rates should not be interpreted as estimator-invariant measures of an intrinsic family effect. Since Appendix B.1 approriately distinguishes the replicated primary result from the additional diagnostics that remain specific to the spectral analysis, this just limits interpretation instead of invalidating the main finding.

  2. Generalisation remains bounded by the selected checkpoint population and benchmark collection. Owner separation does not necessarily imply independent training lineage, and a conservative lineage filter cannot completely recover unavailable training histories. The demonstrated robustness supports conclusions about the analysed collection, but transfer to independently trained frontier systems or substantially different evaluation formats is not established. This is just a scope limitation.

Questions for authors

None requiring clarification for my current recommendation. Appendix B.1 already clarifies that the direct-MIRT analysis reproduces the qualitative primaty finding, while effect sizes change and several additional diagnostics remain specific to the spectral estimator.

Reviewer_K92BOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

As I understand it, the authors aim to explore ways of characterizing types of ranking instability that occur when selecting different subsets of items from benchmarks. This is of interest because comparisons between LMs often come down to <1% gaps. Beyond simply bootstrap sampling random uniform subsets, the authors use psychometrics techniques to stratify a subsample of items over feature like item easiness or topic-category, whose distribution in the population we already expect to influence rankings, and then filter the stratified subset by a feature that may or may not influence rankings. High excess ranking shifts between the filtered stratified subset vs the control unfiltered subset shows the feature in question does relate to rankings. The authors specifically explore model family (e.g. llama, qwen, etc) as that feature as quantified by the differential item performance (DIF) within an IRT model, essentially how much item responses are biased for a family away from what the ability and ease estimations of the IRT model would expect. Out of 5 benchmarks they find that 4 have substantial excess rank instability for near tied model pairs, even while the treatment subsample and original full benchmark are highly correlated. They conduct many further analyses over different IRT models, robustness to model population and experimental configurations, excess instability within model families, and an prompted-LM qualitative analysis of items that are more or less biased by family. These all show that the family bias effect remains, and that it defies simple characterization such as one family always doing better or items with more family bias having some kind of consistent qualitative trait. The conclusion remains that the field should be careful when making judgements based on these small 1% gaps, and that analysis of the ranking robustness to item composition choices such as the one studied here can show when gaps are real.

Critical strengths
  1. The paper takes a very creative and subtle approach to analyzing the problem ranking instability in evaluation. While most work, including other IRT work simply takes an engineering approach of how to make evals have lower noise, this work takes a much more scientific approach of asking WHAT is different between different sets of items that rank models differently.
  2. The paper extensively ablates its analyses over different IRT models, robustness to model population and experimental configurations, not just showing that their basic approach generalizes beyond their specific setup and perhaps could be adapted by future work to test biases over other differential item functioning features.
Critical weaknesses
  1. I was extraordinarily difficult for me to understand the paper. In the abstract and introduction the paper characterizes its research question with descriptions such as "whether a near-tied ordering is robust to which benchmark items are included" (line 37). This seems like it can be trivially answered as "no" by just resampling the item population and showing that hypothesis testing on near ties often fails. So it was very difficult to understand the motivation behind their rather elaborate method which is not very lucidly detailed in section 3 (e.g., basic terms like "owner" are left completely undefined, and the "rank-K reconstruction" isn't explained). Over all the paper has an intensely telegraphic style, which I would encourage the authors not to think of as a mater of taste, but instead over compression of information to the point of ambiguity. For example abundant "noun plague" such as "residual item–family signatures replicate across owner halves" (line 024).
  2. While I applaud the author's rigorous humility about interpreting their results when they say things like "Low-DIF anchors do not define a superior measurement construct or justify item removal" (line 462) I come away from the paper unsure of what use can really be made of this phenomenon. But I do think there likely is something more concrete to be said here, perhaps with a little more experimentation. My own conjecture would be that high-DIF items are perhaps finding test items that have leaked into the training of some models and not others, or subtler forms of benchmark contamination such as training on synthetic examples based on benchmarks. That hypothesis is just an example of what a more concrete interpretation of these results might look like and of course it's up to the authors what interpretations to support. I'd just like to see it more clearly stated what they think the significance of this all is beyond that we maybe shouldn't trust 1% performance gaps (which I think anyone working on evaluation already agrees with).

To be clear I think both of these weaknesses can be solved with revision of the paper text and that the underlying research is very good.

Questions for authors

Did you try just filtering on family DIF without stratifying over the other features?

Also for the content audit, did you see if models can predict low or high DIF on held out items with supervision from low and high DIF examples?

Do you have any ideas about what might make winogrande different from the other datasets such that it doesn't show this phenomenon?

Are there DIF methods already established in previous work which could show the effect that you discover? I ask because then the reader might not need to slog through understanding the custom IRT model you have and just trust a bit more that it works similarly to established methods. Maybe this is the purpose of your Direct logistic MIRT ablation but if so can you describe that approach as essentially coming from settled IRT literature?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript revisions to improve clarity and readability

Dear reviewers and Area Chair,

Thank you for the detailed reviews and follow-up comments. Over the past few days, we have worked through the manuscript section by section, with particular attention to the concerns about terminology, dense explanations, and readability.

To make the revisions easier to review, the marked manuscript uses:

  • Red for improvements to the organization and flow of the paper, including clearer explanations and reduced repetition.
  • Blue for terminology that has been simplified or standardized.
  • Green for additional experimental analyses and the corresponding references in the main text.

The main changes are as follows:

  1. Clarified the research question and contribution. We revised the abstract and introduction to explain more directly what the audit tests and why the comparison with matched-random subtests matters.

  2. Simplified terminology and reduced repetition. We removed overlapping names for the same concepts, introduced key definitions earlier, and shortened repeated explanations.

  3. Reworked the method explanations. We revised dense passages, made the connections between capability adjustment, DIF estimation, and ranking evaluation more explicit.

  4. Incorporated additional analyses and clarified their interpretation. Appendix H now presents further analyses discussed in our responses, including bootstrap intervals for model-pair score differences, WinoGrande diagnostics, and prediction of DIF from held-out item text.

We recognize that our earlier wording sometimes made it hard for readers to follow the argument. Improving this has been a central priority of the revision. We appreciate the reviewers’ careful engagement and the specific examples that helped us identify where we needed clearer explanations.

Thanks again to all the reviewers, your opinions really helped us make the paper better!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.