- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how stable benchmark-derived “pair difficulty” rankings are when the measurement procedure changes. For each pair of class labels, a binary probe is fitted and its out-of-fold balanced accuracy is used to rank pairs from hardest to easiest. Examples and folds are fixed while the representation or the probe is varied. Stability is quantified with rank correlation, mean absolute rank shift, and overlap of the hardest sets. Experiments cover a wide range of benchmarks. The authors report that rankings can either change substantially under probe and representation variation or sometimes agree closely. The authors also show that stability controls (removal of an extreme feature, fit seeds, alternative grouped folds) highlight that the effect is not explained by numerical feature perturbation, stochastic fitting or valid grouped-fold reassignment. Finally, the authors show that sensitivity is heterogeneous across bencjmark systems.
Critical strengths
- Thorough, well-executed experimental design. The paper changes one measurement component at a time, keeps examples and folds fixed, uses grouped folds to avoid leakage, and adds several controls (an extreme feature, fit seeds, alternative folds).
- Wide range of datasets. The experiments cover a microservice failure benchmark, three RCAEval systems and six OpenML datasets, and the paper reports both where the effect is strong and where it is weak.
Critical weaknesses
- The biggest weakness of the paper is that is it mostly exploratory work. The paper documents that rankings move but does not really explain why. The findings seem expected, since probes with different inductive biases (linear vs kernel vs tree) will order class pairs differently. This way, the paper lacks novelty.
- The paper does not provide analysis of which pairs move and why. Such analysis would turn a descriptive finding into something actionable.
- The experimental analysis does not provide random-ranking baselines to be compared to MARS and NMARS.
- The paper only provides rank statistics, not the accuracy of each probe or the spread of scores across pairs. Large rank shifts may therefore reflect poor probe performance (close to chance?), rather than unstable measurement.
Questions for authors
- Could the authors report mean balanced accuracy and score spread per probe and dataset, and show how each pair’s rank shift relates to this?
- The discussion recommends auditing hard subsets across readouts. Which readouts should be included, and what level of rank agreement would indicate that a subset is reliably hard?
3 comments
Final Clarification on the Novelty and Scope of the Contribution
Thank you for your detailed assessment. We agree that different inductive biases can naturally produce different difficulty rankings. This observation motivates our study but is not its central contribution.
The revised manuscript addresses a distinct question: when does ranking disagreement necessarily require different fixed-budget task selections after accounting for score resolution?
In AIOps2025, the original LR and RF hardest-ten lists share only three tasks. At BA tolerance 0.02, however, the maximum feasible overlap is five. Four RF-selected tasks are individually ineligible under LR, while retaining the other six together would require five additional predecessor tasks, exceeding the budget. Thus, exactly five replacements are unavoidable. Conversely, all 36 tested OpenML probe contrasts admit fully overlapping legal selections at the same resolution, even when their original hard sets differ.
We have also addressed the requested measurement checks: Appendix B reports BA distributions for all 60 primary configurations, Appendix D provides score-gap analyses and random-ranking references, and our previous follow-up identifies the class-level prediction changes behind two representative task shifts.
These analyses establish which selection disagreements are unavoidable under the stated constraints, without claiming a general feature-level explanation or downstream performance improvement.
We hope this distinction clarifies the contribution and its scope in the revised manuscript.
Thank you for your detailed rebuttal and for the additional clarifications. I have read them carefully. I appreciate the effort to address my comments, and I am raising my score to 2. My main concern remains: the contribution is largely exploratory. The rebuttal does not change this assessment.
Follow-up: Explaining Pair-Level Changes and Their Selection Consequences
Thank you for raising the concern that different probes may be expected to produce different difficulty rankings. Our previous response addressed the BA distributions, score gaps, random references, and readout-audit criteria. We have since examined the saved out-of-fold predictions and score constraints to clarify what the observed disagreements reveal beyond differences in model inductive bias.
Which pairs change, and what drives their score differences? In the primary AIOps2025 Temporal LR–RF comparison, the BA increase for
application_code–network_loss(.524 to .810) comes entirely fromapplication_coderecall improving from 5/21 to 17/21;network_lossrecall remains 17/21. Conversely, fordns_connectivity–resource_cpu, DNS recall declines from 14/21 to 9/21 while CPU recall improves from 55/58 to 58/58, producing a net BA decrease of .093. The directions of the principal class-recall differences persist across all ten grouped assignments of the same 400 events. This identifies which class-level prediction errors drive the two task-score changes, although the underlying feature-level decision mechanisms remain untested.Why do these differences matter for task selection? At BA tolerance ϵ=.02 and budget q=10, RF's hardest-ten selection is unique under its score constraints. Four of its tasks cannot enter any legal LR selection. The remaining six are individually eligible under LR, but retaining all six requires five additional predecessor tasks, exceeding the ten-task budget. A verified legal selection achieves five shared tasks, establishing that exactly five replacements are necessary. Thus, individual task eligibility does not determine whether a collection of tasks can be selected jointly.
Is this incompatibility inevitable whenever rankings differ? No. Among 36 OpenML probe comparisons at the same BA tolerance, 21 retain effective task-eligibility constraints on both sides. Eleven of these have different original hardest-task lists yet admit a common legal selection. Moreover, each of the six OpenML datasets admits at least one selection of its prescribed size satisfying all four probes' constraints simultaneously.
What these findings establish. The results distinguish three questions: whether rankings disagree, whether individual tasks remain eligible, and whether a common fixed-budget selection is feasible. The last cannot be inferred from rank displacement or original set overlap alone. For a specified set of justified readouts, selection budget, and score resolution, the minimum required replacements provide a direct compatibility test. Zero required replacements establishes feasibility of a common selection, not a universal guarantee of ranking reliability or downstream diagnostic benefit.