acceptodds
Withdrawn from ICLR 2027

Not Even Ordered: What a Published Leaderboard Entitles a Reader to Quote

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A leaderboard reports one total order over its methods, computed under evaluation choices its paper does not state. Those choices are a rule deciding when a prediction counts as the reference, crossed with an output budget, a post-processing step or a normalisation. We fix the items and the released predictions, re-score them in every cell of a declared grid, and ask what part of the published order survives. We do this for 24 leaderboards in four domains, retraining nothing and reimplementing no method. Together those tables print 298 places. Read permissively, a pair stands unless some cell contradicts it, and 170 ranks stand. Read strictly, a pair stands only where every cell separates it, and 99 survive. The distance between the two readings is mostly not the choices: 159 ranks go to resolving power a benchmark never had, before any convention is varied, and 40 to the grid. The surviving share runs from none to nine tenths, and does not replicate between two editions of one shared task. Survival is a property of a table and its grid, not of a benchmark. Of the orderings a table did establish, 6 are reversed by a declared choice and survive their own board's correction, and 5 of those survive a single correction over all 24 grids. None of the six is the matching rule's, the choice this literature discusses most. Deciding which orderings are at risk needs no pairwise statistic. We release the protocol, the audited split, the frozen predictions, the checker and the analysis code.

Primary area:
datasets and benchmarks
Keywords:
benchmarking, evaluation methodology, leaderboards, reproducibility, multiple comparisons, multiverse analysis, machine translation evaluation, molecular machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_sjzgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a method to extract ordered information from leaderboards. Since the concept is not explicitly defined in the text, I assume a leaderboard refers to a ranking table evaluating ML models. The items extracted from these leaderboards are organized into a contradiction-free partial order.

However, the paper is difficult to read. The problem must be formally defined, ideally by using a running example within the 9 pages of the main paper. The core problem needs to be fully understandable just by reading the main text (especially given that the main paper plus appendices spans 83 pages!). Furthermore, it is difficult to discern how this approach differs from existing research in the domains of rank aggregation, consensus ranking, or Kemeny consensus. Overall, there is a strong need to clearly define the problem. As it currently stands, it is nearly impossible for a researcher outside this specific field to understand the exact problem being solved and to appreciate any potentially original results.

Critical strengths

It is difficult to identify the main strengths of the paper when the core problem is not clearly defined. One potential strength, however, is the rich dataset provided for evaluation, which is thoroughly documented in the extensive appendices.

Critical weaknesses

The paper is difficult to read. The problem must be formally defined, ideally by using a running example within the 9 pages of the main paper. The core problem needs to be fully understandable just by reading the main text (especially given that the main paper plus appendices spans 83 pages!). Furthermore, it is difficult to discern how this approach differs from existing research in the domains of rank aggregation, consensus ranking, or Kemeny consensus.

Questions for authors

Overall, there is a strong need to clearly define the problem. As it currently stands, it is nearly impossible for a researcher outside this specific field to understand the exact problem being solved and to appreciate any potentially original results.

Reviewer_xW9ZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors study the stability of leaderboard orderings when evaluation conventions vary while predictions and test items remain fixed. In this paper, they intersect the resulting orders across declared grids for 24 leaderboards, distinguish rank reversals from unresolved comparisons, and report longest-chain summaries alongside a metabolite-prediction case study.

Critical strengths

The analysis distinguishes unresolved comparisons from descriptive reversals. These distinctions make the survey more informative than a collection of rank-change examples.

Critical weaknesses

The conclusions about which rankings readers can trust go further than the analysis supports. The paper’s rule for identifying a unique winner fails on its own example, which directly undermines what it presents as its main contribution.

Questions for authors

See above.

Reviewer_QknvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyses the sensitivity of published model rankings to evaluation choices by re-scoring fixed predictions without retraining the models. It evaluates 24 leaderboards across retrosynthesis, metabolite prediction, docking and machine translation under grids of matching criteria (rules used to determine correctness) and other settings. It constructs a partial order containing comparisons whose direction is consistent across configurations, summarises this order using its longest-chain length, and distinguishes numerical changes from statistically supported reversals.

Across the 24 leaderboards, the longest sequences of models that remain ordered the same way under every tested evaluation setting contain 170 model entries in total out of 298 model entries. When statistically significant differences are required in every configuration, the summed longest-chain length drops to 99. Six model pairs exhibit statistically significant reversals after correction for multiple comparisons within each leaderboard, with statistically significant differences also present in the original direction. All six reversals result from changes in prediction cutoff.

Critical strengths

The paper conducts a broad empirical analysis of evaluation choices that are not always reported or considered and analyses the sensitivity of leaderboards to these choices. By using existing predictions on fixed test items, the analysis avoids differences due to retraining. Most comparisons vary only scoring choices, although the docking analysis also varies post-processing of the predicted poses.

The paper separates numerical rank differences from statistically significant reversals and also considers whether the original difference was statistically significant. It shows that changing the matching-rule does not produce any statistically significant reversals that survives multiple-testing correction in any of the cases where the original pair was statistically separated. This negative result appropriately limits the claim that evaluation choices overturn established model comparisons.

Critical weaknesses

The paper conflates the length of a consistently ordered chain with supported leaderboard positions. A chain establishes an ordering among its members, but does not establish their positions relative to models outside that chain. This problem can be seen in Figure 3: LocalRetro occupies the sole “tier 1” position, although GraphRetro, RetroFormer and Graph2SMILES are also undominated (no other model outperforms these models across all configurations) in the displayed relation. The caption acknowledges that the tiers are not ordered blocks, but the surrounding text emphasises that the originally first-ranked system is outside the top tier and the originally second-ranked system uniquely occupies it. This presentation risks attributing ranking information to the partial order that it does not contain. Because the paper’s central contribution concerns which ranking claims readers can make, this is a substantive issue with the interpretation of its headline results.

All six certified reversals of originally separated pairs are a result of changing the prediction cutoff. A system outperforming another at top-1 but underperforming it at top-5 demonstrates different relative performance at different candidate budgets and does not invalidate an explicitly qualified top-1 claim. The main text needs to justify why a ranking claim should hold across the configurations included in each grid, or identify which broader claims in the original publications are contradicted by the observed reversals. This concern is compounded by the presentation, as the text repeatedly shifts between pair counts, chain lengths, and statistical separation, making it difficult to determine which evidence supports which conclusion. In particular, §3.1 attributes most of the reduction from 298 to 170 to the benchmark’s statistical resolving power, whereas the decomposition in §3.3 concerns a different sequence, 298 → 139 → 99. Since the 170 figure does not require statistical significance, this attribution needs clarification.

Questions for authors

What precisely does “supported places” establish about model ranks? Can you justify the interpretation of longest-chain length beyond describing the largest subset whose members retain the same relative ordering across all tested configurations? How should a reader interpret the unique top tier in Figure 3 when four models are undominated?

Which claims in the source papers are contradicted by the six cutoff-driven reversals? Were these claims explicitly about top-1 performance, or did they assert superiority independently of the prediction budget? Evidence that the audited claims were underspecified or broader than a particular cutoff would materially strengthen the motivation.

Please clarify the attribution of lost ranking information to statistical uncertainty versus evaluation choices. The 170 figure requires consistent signs without statistical separation, while the 139 and 99 figures consider statistical significance. Which conclusions apply to each quantity, and what supports attributing most of the 298-to-170 reduction to the benchmark’s statistical resolving power?

Reviewer_HjDROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose what they call an "instrument" for reading leaderboards. The authors claim that many leaderboards publish the results without actual criteria or evaluation choices, and that the results can become inconsistent depending on what prediction is considered "correct". The authors conduct a "survey" over 24 leaderboards in 4 domains and conclude that the leaderboards examined are not robust to evaluation choices. It is not clear what they are trying to prove and there seems to be no takeaway message.

Critical strengths

None

Critical weaknesses

Content

  • The authors make claims that are not substantiated. Take first sentence of abstract: "A leaderboard reports one total order over its methods, computed under evaluation choices its paper does not state." Clearly the "leaderboards" (the benchmarks) the authors use in this paper do state their evaluations which is what allows the authors to apply their "instrument". So which leaderboards are the authors referring to?
  • The authors are extrapolating quite a bit from a small set of "leaderboards" (19 of 24 of them are from the same WMT tasks). This following paragraph at bottom of introduction:

    C1: most of what a leaderboard prints it never established: the pairs its own cell cannot separate from zero. C2: of the pairs it did establish, six orderings are reversed by a declared choice and survive their own board’s correction. ..

    • is strictly limited to the "survey" the authors conduct. This is not a generalizable statement that the authors can prove yet it is presented as such.
  • Further, this notion of a "robust" ordering is not motivated enough. The authors seem to assume that an ordering should remain invariant across a grid of evaluation "choices". But such "choices" can correspond to meaningfully different evaluation objectives. For example, recall@1 and recall@10 evaluate different capabilities (exact match vs broad ranking), and a reversal between the two does not imply that either ranking is incorrect or unsupported.
  • The authors repeatedly interprets statistically unresolved differences as evidence that a leaderboard did not establish its reported ranking. I find this unconvincing. A leaderboard can report an ordering of observed performance on a fixed test set without claiming that every pairwise difference is statistically significant.
  • This paper does not seem to have a clear takeaway message. The authors conduct a "survey" in which they apply different scoring criterion to different benchmarks and show that the results can be inconsistent, without much further analysis. This is not very surprising nor useful.

Presentation

  • The use of English is highly irregular and makes it difficult to follow what the paper is about.

    • breaking down the first paragraph of the introduction. bold are my comments as I try to parse the text.

    This (this paper) is a survey (turns out it wasn't much of a survey, more a set of experiments that ran over) of published leaderboards and an instrument for reading them (the leaderboards?). The instrument (this paper?) takes a leaderboard’s released predictions and a declared set of evaluation choices and returns the part of its (leaderboard's) ordering that no choice contradicts. The survey (i wouldn't call this a survey. more of a set of experiments) applies it to twenty-four leaderboards in four domains. One predictor (never defined. I think the authors are trying to say the model being evaluated?) in one domain is used to find the choices worth declaring (what does this mean?). That predictor (now i'm confused. what is a predictor?) is instrumentation (of what? the act of measurement itself?) rather than a contribution (by whom? the leaderboard authors? the author of this paper? the model?): it is not offered as a state of the art (in what?), and nothing in the survey (the experiments, right?) depends on how it performs.

    • The entire paper is written with such hyperbole and obscure references. This writing is not ready for human consumption.
  • Unclear use of terminology and missing references.

    • Page 2. Verdicts are corrected by Holm over the cell-level tests each grid creates. who is Holm?
      • I see that the authors are referring to the Holm-Bonferroni method. The first time this full name (dash joined) is referenced is in page 17. The actual method is never cited nor described as far as I can tell in the 83 pages.
    • "cell", "grid" which the authors repeatedly point to starting from the abstract/introduction are never properly defined. The formal definition appears in the bottom of page 4, after 43 previous references of such "cell" (28) and "grid"(15).
  • Overall reading and trying to follow the cryptic writing of this paper required me to jump back and forth to try to understand what the opaque references could be referring to. Presentation needs work.

Questions for authors
  • What is the takeaway message here? What is the scientific contribution of this paper? This was not made very clear to me.
  • What are some leaderboards that you have found that do not state their evaluation choices?
  • Why is it that only the "longest chain" of orderings are considered valid? Are the authors making an assumption that there exists a single ground truth metric of "correctness" that all "choices" of a leaderboard is trying to approximate, and thus they should be consistent? Why is it not the case that pairwise comparisons can be valid? This is not clear to me.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.