- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper suggests the MESA framework to evaluating machine learning methods. Each method is run for multiple seeds for each configuration in a predetermined set of plausible settings. Then, models are scored by both the volume of the configurations in which they significantly beat the empirically k-th best competitor, as well as the stability of that region (for numerical configurations where proximity of two configurations makes sense). The paper also proposes a dual view to evaluate benchmark datasets by how consistently they separate models from the kth-best model (again across configurations). The method is illustrated using two cases studies on conformal prediction and anomaly detection
Critical strengths
- The stability of ML evaluations is an interesting topic
- Some form of perturbation analysis as done in the paper seems like the right direction
- The paper is written in an easily understandable way.
Critical weaknesses
- The paper does not really convince me that its relatively complicated approach is necessary to achieve its goals (lots of alternative perturbation methods are mentioned in the related work, but there do not seem to be any meaningful comparisons to them). Similarly, the value add of separately reporting volume and stability is not really demonstrated.
- k is a relatively hard to interpret hyperparameter, and having hyperparameters in evaluation methods is usually not great (quite related to the paper's main thrust)
- MESA is much more expensive to run than standard benchmarking (by a factor of n_configurations*n_seeds)
- The existence of some configurations that perturb model SOTA does not seem to establish that MESA yields improved model selection for real world settings compared to standard benchmarking.
Questions for authors
What is the exact difference to performance profiles?
8 comments
Reply to Reviewer ineq (2/2)
Weakness #3: Computational cost.
We agree with the reviewer that MESA has a higher computational cost than standard benchmarking. The factor of R (number of runs/seeds) is not unique to MESA, as repeated runs are already recommended practice for reliable comparisons (Bouthillier et al., 2021), but the reviewer is correct to point out that the number of configurations can cause further strain on computational cost/time. In future work, we plan on reducing this cost by applying supervised learning or imputation methods, as in Zeng & Papailiopoulos [1], to lessen the amount of computation needed. We would also like to point out that MESA’s dataset scores can focus future evaluations on benchmarks that separate methods best.
[1] Zeng, Yuchen, and Dimitris Papailiopoulos. "You Don't Need to Run Every Eval." arXiv preprint arXiv:2606.24020 (2026).
Weakness #4: Unclear improvements in model selection.
We respectfully disagree. A standard benchmark answers which method leads at one configuration, but deployment conditions may change over time. If the best method changes across configurations, the benchmark’s answer holds on only part of the space, and the practitioner cannot tell if the model is still the best under their current deployment conditions. In our case studies, we find this heterogeneity in performance across feasible experimental parameters. MESA tells a practitioner whether a method clearly leads across plausible configurations, and these declared leads hold up on held-out runs (Weakness #1).
We are happy to answer any follow-up questions or concerns and would be grateful if the reviewer would consider raising their score.
- Could you please describe the experiment regarding weakness 1 in more detail? Is the left column using mean method performance over configurations? The baselines I had in mind were closer to simply using performance profiles. If I understand correctly, this experiment also does not show anything about the usefulness of stability over volume.
Reply to Reviewer ineq (1/2)
We thank the reviewer for their helpful comments and address the questions and weaknesses below. In short, we claim that leads declared by MESA hold on held-out runs far more often than the leader a standard benchmark reports, and MESA captures geometry and dataset quality, whereas standard benchmarking and performance profiles do not.
Question #1: Difference from performance profiles.
Performance profiles (Dolan and Moré, 2002) report ρs(τ), the fraction of problems where method s is within a factor of τ of the best performing method. MESA has three distinct differences:
Weakness #1: Potentially unnecessary complexity.
Standard benchmark leaderboards compare point estimates on a single (often unreported) configuration. MESA goes further to quantify if a method clearly leads at each plausible configuration and gives practitioners a higher degree of granularity. Most perturbation-based approaches in our related work (e.g., multiverse analysis) analyze each method in isolation and do not seek to evaluate method performance in comparison to others, so we compare MESA against the rule a standard benchmark issues. In this way, MESA can be thought of as bringing multiverse-esque perturbation to model evaluation.
To test whether a statistical test yields better conclusions than mean performance, we ran a new analysis on the paper’s existing meta-experiments. At each configuration, we use half of the runs to declare a winner and check whether it still leads on the other half (Table 1). Code for this analysis is available in the anonymous repository within the
rebuttalsubdirectory.MESA’s stability score also describes the shape of the space where a method leads, which does not have a clear analogue from traditional benchmarking. The resulting information can be useful in determining if certain methods should be deployed: a highly unstable method may not suit a high-stakes scenario because small perturbations could cause its performance to collapse.
Weakness #2: Interpretation of k.
We would like to clarify that k is not a parameter to tune, but rather specifies which claim is being evaluated. Practitioners often want to identify the best method (k=1), but sometimes want a ranking. Full rankings are rarely recoverable in a meta-experiment, since each adjacent pair of methods would need to be separated from inter-run noise at each configuration. Top-k thus acts as a hybrid. The claim carries more information than a single winner while keeping enough power to recover signal. For example, a practitioner with the budget to inspect the top three methods in their specific setting could set k=3.