acceptodds
Under review as a conference paper at ICLR 2027

HOW STABLE ARE BENCHMARK-DERIVED DIFFICULTY RANKINGS?

Abstract

Analyses of benchmark data often identify which distinctions appear hardest, guiding model diagnosis, data curation, and subsequent evaluation. Yet a difficulty ranking is produced by a measurement procedure: fixed examples are represented, a supervised readout (probe) is fitted, and its scores are ordered. We construct pairwise discrimination tasks from benchmark labels and assess the stability of their ordering under representation and readout choices. On 400 AIOps2025 failure events spanning 55 mechanism pairs, we compare three representations and four fixed probes using grouped out-of-fold evaluation. Across all six probe contrasts under a fixed temporal representation, mean absolute rank shifts range from 3.69 to 14.55 positions; in the motivating logistic-regression–random-forest contrast, only three of the hardest ten pairs overlap. Numerical-feature, fit-seed and alternative grouped-fold controls place this displacement in context. RCAEval RE1 shows heterogeneous probe sensitivity across three distinct systems. Across six OpenML tabular datasets meeting prespecified eligibility criteria, median span-scaled rank displacement ranges from 0.065 to 0.233. Benchmark-derived pair difficulty is a measurable, procedure-relative output whose stability varies across settings and domains.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, benchmark difficulty, measurement stability, ranking stability, readout sensitivity

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Qw4POpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how stable benchmark-derived “pair difficulty” rankings are when the measurement procedure changes. For each pair of class labels, a binary probe is fitted and its out-of-fold balanced accuracy is used to rank pairs from hardest to easiest. Examples and folds are fixed while the representation or the probe is varied. Stability is quantified with rank correlation, mean absolute rank shift, and overlap of the hardest sets. Experiments cover a wide range of benchmarks. The authors report that rankings can either change substantially under probe and representation variation or sometimes agree closely. The authors also show that stability controls (removal of an extreme feature, fit seeds, alternative grouped folds) highlight that the effect is not explained by numerical feature perturbation, stochastic fitting or valid grouped-fold reassignment. Finally, the authors show that sensitivity is heterogeneous across bencjmark systems.

Critical strengths
  • Thorough, well-executed experimental design. The paper changes one measurement component at a time, keeps examples and folds fixed, uses grouped folds to avoid leakage, and adds several controls (an extreme feature, fit seeds, alternative folds).
  • Wide range of datasets. The experiments cover a microservice failure benchmark, three RCAEval systems and six OpenML datasets, and the paper reports both where the effect is strong and where it is weak.
Critical weaknesses
  • The biggest weakness of the paper is that is it mostly exploratory work. The paper documents that rankings move but does not really explain why. The findings seem expected, since probes with different inductive biases (linear vs kernel vs tree) will order class pairs differently. This way, the paper lacks novelty.
  • The paper does not provide analysis of which pairs move and why. Such analysis would turn a descriptive finding into something actionable.
  • The experimental analysis does not provide random-ranking baselines to be compared to MARS and NMARS.
  • The paper only provides rank statistics, not the accuracy of each probe or the spread of scores across pairs. Large rank shifts may therefore reflect poor probe performance (close to chance?), rather than unstable measurement.
Questions for authors
  • Could the authors report mean balanced accuracy and score spread per probe and dataset, and show how each pair’s rank shift relates to this?
  • The discussion recommends auditing hard subsets across readouts. Which readouts should be included, and what level of rank agreement would indicate that a subset is reliably hard?
3 comments
AuthorsOpenReview ↗

Follow-up: Explaining Pair-Level Changes and Their Selection Consequences

Thank you for raising the concern that different probes may be expected to produce different difficulty rankings. Our previous response addressed the BA distributions, score gaps, random references, and readout-audit criteria. We have since examined the saved out-of-fold predictions and score constraints to clarify what the observed disagreements reveal beyond differences in model inductive bias.

Which pairs change, and what drives their score differences? In the primary AIOps2025 Temporal LR–RF comparison, the BA increase for application_code–network_loss (.524 to .810) comes entirely from application_code recall improving from 5/21 to 17/21; network_loss recall remains 17/21. Conversely, for dns_connectivity–resource_cpu, DNS recall declines from 14/21 to 9/21 while CPU recall improves from 55/58 to 58/58, producing a net BA decrease of .093. The directions of the principal class-recall differences persist across all ten grouped assignments of the same 400 events. This identifies which class-level prediction errors drive the two task-score changes, although the underlying feature-level decision mechanisms remain untested.

Why do these differences matter for task selection? At BA tolerance ϵ=.02\epsilon = .02 and budget q=10q = 10, RF's hardest-ten selection is unique under its score constraints. Four of its tasks cannot enter any legal LR selection. The remaining six are individually eligible under LR, but retaining all six requires five additional predecessor tasks, exceeding the ten-task budget. A verified legal selection achieves five shared tasks, establishing that exactly five replacements are necessary. Thus, individual task eligibility does not determine whether a collection of tasks can be selected jointly.

Is this incompatibility inevitable whenever rankings differ? No. Among 36 OpenML probe comparisons at the same BA tolerance, 21 retain effective task-eligibility constraints on both sides. Eleven of these have different original hardest-task lists yet admit a common legal selection. Moreover, each of the six OpenML datasets admits at least one selection of its prescribed size satisfying all four probes' constraints simultaneously.

What these findings establish. The results distinguish three questions: whether rankings disagree, whether individual tasks remain eligible, and whether a common fixed-budget selection is feasible. The last cannot be inferred from rank displacement or original set overlap alone. For a specified set of justified readouts, selection budget, and score resolution, the minimum required replacements provide a direct compatibility test. Zero required replacements establishes feasibility of a common selection, not a universal guarantee of ranking reliability or downstream diagnostic benefit.

AuthorsOpenReview ↗

Final Clarification on the Novelty and Scope of the Contribution

Thank you for your detailed assessment. We agree that different inductive biases can naturally produce different difficulty rankings. This observation motivates our study but is not its central contribution.

The revised manuscript addresses a distinct question: when does ranking disagreement necessarily require different fixed-budget task selections after accounting for score resolution?

In AIOps2025, the original LR and RF hardest-ten lists share only three tasks. At BA tolerance 0.02, however, the maximum feasible overlap is five. Four RF-selected tasks are individually ineligible under LR, while retaining the other six together would require five additional predecessor tasks, exceeding the budget. Thus, exactly five replacements are unavoidable. Conversely, all 36 tested OpenML probe contrasts admit fully overlapping legal selections at the same resolution, even when their original hard sets differ.

We have also addressed the requested measurement checks: Appendix B reports BA distributions for all 60 primary configurations, Appendix D provides score-gap analyses and random-ranking references, and our previous follow-up identifies the class-level prediction changes behind two representative task shifts.

These analyses establish which selection disagreements are unavoidable under the stated constraints, without claiming a general feature-level explanation or downstream performance improvement.

We hope this distinction clarifies the contribution and its scope in the revised manuscript.

Thank you for your detailed rebuttal and for the additional clarifications. I have read them carefully. I appreciate the effort to address my comments, and I am raising my score to 2. My main concern remains: the contribution is largely exploratory. The rebuttal does not change this assessment.

Reviewer_96z8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines whether lists of the hardest tasks derived from benchmark data are stable or depend on how difficulty is measured. The authors build pairwise discrimination tasks from benchmark labels, hold the examples and fold assignments fixed, and vary one component at a time. Experiments on several benchmarks show that benchmark derived pair difficulty is procedure-relative.

Critical strengths
  1. The task studied is interesting and useful for benchmark creation works.

  2. Most experiments are well designed. With controlled factors, the experiments clear show different contributors to the instability of task difficulty rankings.

Critical weaknesses
  1. Rankings can turn tiny score differences into large rank changes, especially when many pairs have similar values. It is unclear whether large rank changes reflect meaningful score changes.

  2. All probes are fixed in experiments. While this is a reasonable design choice, probes fitting one panel well may not be suitable for another. So part of the observed instability could reflect probe misfit. An ablation study on same probe with different settings can be helpful to further understand the problem studied.

  3. Readbility: (1) I suggest the authors to add references and extra explanations about the task background in introduction section. There is no reference now. There should be at least citations for the benchmarks used, such as AIOps. (2) When introducing the designs in Section 3, this paper supposes that the readers are familiar with these datasets. This may not be the case, so I suggest the authors to add extra explanations or examples about what those data look like.

Questions for authors
  1. Please revising the writing of the paper (See point 3 in weaknesses).

  2. Following weaknesses 1, How large are the score differences behind the rank changes? Would a practically tied analysis (e.g., treating closed BA as tied) change MARS or hardest-set overlap substantially?

3 comments
AuthorsOpenReview ↗

Follow-up: What Near-Tie Handling Reveals About Task Selection

Thank you for emphasizing the importance of distinguishing rank changes caused by small score differences from changes that remain meaningful at a stated score resolution. Our previous response reported the BA-gap analysis, tolerance sensitivity, and same-probe parameter checks. We have since examined the selection constraints more closely.

Why near-tie handling does not eliminate all selection differences. In the primary AIOps2025 LR–RF comparison, at BA tolerance ϵ=.02\epsilon=.02 and budget q=10q=10, four tasks in RF's hardest ten are ineligible under LR. The remaining six are individually eligible, but selecting all six requires five additional predecessor tasks, exceeding the budget. A verified legal selection achieves five shared tasks, establishing that exactly five replacements are necessary. Thus, allowing near-tied scores to be reordered does not resolve all joint selection conflicts.

Why ranking changes can nevertheless be reconciled. Among the 36 OpenML probe comparisons, 21 retain effective task-eligibility constraints on both sides. In eleven of these, the original hardest-task lists differ, yet a common legal selection exists. Furthermore, each of the six datasets admits a selection of its prescribed size satisfying all four probes' constraints simultaneously at ϵ=.02\epsilon=.02.

What this adds to the practical-tie analysis. These contrasting results show that rank displacement and original hard-set overlap do not determine whether a common fixed-budget selection is feasible. Score-resolution-aware selection analysis distinguishes disagreements that necessarily change the selected tasks from those that can be reconciled under the tested budget and tolerance.

AuthorsOpenReview ↗

Follow-up: Cross-probe comparison under alternative parameter settings

Thank you for raising the concern that fixed probe settings may contribute to the observed ranking differences.

Our previous response examined parameter sensitivity within each probe. We have now crossed the existing LR regularization settings (C = 0.1, 1, 10) with the RF depth settings (4, 8, None), using saved out-of-fold predictions for the same 55 AIOps2025 Temporal tasks and primary grouped split. No additional models were fitted.

For a selection budget of ten tasks, the minimum required replacements after allowing score-consistent reorderings are:

LR CBA tolerance 0.02BA tolerance 0.05
0.142
1 (original)52
1063

These counts hold for all three tested RF depths. The RF rankings are identical across depths, although a few task-level BA scores differ. Across the nine parameter pairings, mean absolute rank displacement ranges from 12.40 to 13.18.

Thus, the fixed-budget LR–RF selection incompatibility is not specific to the original parameter pairing. This result establishes robustness to the tested parameter variations on the fixed cohort and split; it does not establish optimal probe suitability or generalization to independently sampled data.

Thank you for your explanations. I updated my score accordingly.

Reviewer_FT8GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how robust the difficulty rankings of existing classification datasets are when the input representation or classifier changes. The paper first constructs held-out datasets from previous classification tasks and compares those rankings across feature representations and four standard classifiers. The paper notes a large difference in rankings between logistic regression models and random forest models, illustrating the lack of ranking robustness. The paper argues that difficulty rankings depend on the procedure used to produce them, and practitioners interested in reproducing them should account for that.

Critical strengths
  • Controlled comparisons: they hold examples and folds fixed while changing the classifier or representation, making the comparisons interpretable.
  • Alternative explanations checked: they examine random seeds, alternative grouped splits, and an extreme-valued feature to assess whether these explain the observed instability.
Critical weaknesses
  • The practical significance of the study remains insufficiently established. Although the authors motivate difficulty rankings through data collection, benchmark curation, and diagnostic prioritization, they do not demonstrate that the observed instability adversely affects these activities or that accounting for it improves outcomes. The relevance to the broader Deep Learning/LLM community is also unclear (although out of scope of the paper). Robustness of evaluation rankings is important to these areas, but this paper studies rankings of task difficulty with relatively simple ML techniques rather than deep learning models leaderboards. It is hard to evaluate how important the problem revealed by the work is.
  • Limited insight beyond an expected observation. Different representations and classifiers have different inductive biases, so disagreement about which distinctions are difficult is unsurprising. The study quantifies this disagreement but offers limited explanation of when it should occur or which dataset properties predict it.
  • Rank displacement is not sufficient evidence of meaningful differences. Small score differences can produce substantial rank changes when tasks have similar performance. Although the paper reports some score-level differences and uncertainty analyses, it does not clearly connect the highlighted rank shifts to practically meaningful thresholds.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Follow-up: When Ranking Disagreement Changes Task Selection

Thank you for highlighting the practical significance of ranking instability. Our previous response showed that some ranking differences require changes to a fixed-budget hard-task list even after accounting for score resolution. We have since examined the frozen OpenML scores more closely to understand when such selection changes are necessary and when they can be avoided.

Selection compatibility despite nontrivial constraints. At BA tolerance ϵ=.02\epsilon=.02, all 36 OpenML probe comparisons permit complete overlap under their prescribed task budgets. However, this is not simply because ordering constraints disappear. Nine comparisons have no strict precedence constraints on either side, six retain constraints on one side, and 21 retain effective task-eligibility constraints on both sides. Among these 21, eleven have different original hardest-task lists yet admit a common legal selection.

A concrete example. In Satimage, the LR–RBF comparison has an original hardest-set overlap of only 4/5. The two probes impose 66 and 47 strict precedence relations, respectively. Nevertheless, LR's original five-task selection also satisfies the RBF constraints. Thus, different original selections and substantial ordering constraints do not necessarily imply incompatible task lists.

Beyond pairwise agreement. We additionally verified joint compatibility across all four probes. At ϵ=.02\epsilon=.02, each of the six OpenML datasets admits at least one common selection of its prescribed size that satisfies all four probes' constraints simultaneously.

What this means for benchmark use. AIOps2025 and OpenML illustrate two distinct outcomes: ranking disagreements can force different follow-up task selections, or they can be reconciled despite nontrivial score constraints. This distinction cannot be established from rank displacement or original set overlap alone. Our analysis provides a concrete way to determine whether changing the measurement readout necessarily changes the selected tasks, conditional on the stated budget and BA resolution. It evaluates selection compatibility rather than the downstream benefit of any particular selection.

AuthorsOpenReview ↗

Clarification on Practical Significance and the Contribution

The practical significance of this study concerns an inference made when benchmark difficulty measurements are used to prioritize tasks for further investigation: does disagreement between measurement procedures necessarily imply different task priorities?

Different classifiers may be expected to produce different rankings. However, this observation alone does not establish whether the disagreement forces a change in a fixed-budget task selection. Rank displacement and original top-k overlap cannot determine this, particularly when task scores are close.

Our revised analysis makes this distinction testable under a stated budget and BA tolerance. On AIOps2025, LR and RF require at least five different selections out of ten at tolerance 0.02, and two remain unavoidable at 0.05. In contrast, all 36 tested OpenML probe comparisons permit fully overlapping feasible selections at tolerance 0.02. Notably, eleven comparisons retain constraints on both readouts and have different original hardest-task lists, yet still admit a common selection.

These results establish what can and cannot be inferred from ranking disagreement. In the first case, no common hard-task list satisfies both measurement constraints, so the reported priorities are necessarily readout-dependent. In the second, different rankings do not necessarily imply incompatible priorities. This distinction provides a concrete check before treating benchmark-derived rankings as a basis for selecting tasks to investigate.

The contribution is therefore not the expected observation that classifiers disagree, but a method and empirical evidence identifying when such disagreement entails an unavoidable change in task selection. Demonstrating whether a particular selection subsequently improves data collection or diagnosis is a distinct downstream evaluation question, rather than the outcome measured here.

Reviewer_JQoZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper ranks class pairs by classification difficulty using balanced accuracy, then examines how these rankings change across representations and probes while keeping samples and fold assignments fixed. The main experiment uses 400 AIOps2025 events and 55 class pairs. Control experiments and additional tests on RCAEval and OpenML show that ranking stability varies across measurement settings and datasets.

Critical strengths

The paper asks a useful question: how much do difficulty rankings depend on the procedure used to measure them? This cautions against treating such rankings as fixed properties of a dataset. The experiments keep samples, fold assignments, and hyperparameters fixed. Feature mappings are fitted only on training folds, which helps limit confounding factors.

Critical weaknesses
  1. The results do not clearly separate probe effects from estimation noise. The paper does not report the distribution of balanced accuracy scores or the gaps between scores. If many scores are close, sampling noise alone could cause large ranking changes. This leaves the claim of "consequential reordering" insufficiently supported.

  2. The overlap results need a random baseline. Two random sets of ten pairs drawn from 55 pairs would share about 1.8 pairs on average. The observed overlap of three needs to be interpreted against this distribution. For RE1, each system has only ten pairs, so Jaccard overlap for the three hardest pairs can take very few values.

  3. The robustness evidence is limited. Two alternative fold assignments are not enough to estimate variation across folds. The claim that the effect "persists through controls" is therefore stronger than the evidence supports.

4 The paper does not explain why stability varies across datasets. Differences in score gaps, sample sizes, or other dataset properties could help explain these results. Without this analysis, the external experiments offer limited insight into when rankings are stable.

Questions for authors

Q1. If you group class pairs by their BA gaps to neighboring pairs in the ranking, are large rank changes concentrated among pairs with small gaps?

Q2. What is the distribution of the overlap count between two random subsets of size 10 drawn from 55 pairs?

Q3. Could you explain the choice of only two alternative fold assignments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.