acceptodds
Under review as a conference paper at ICLR 2027

MESA: Scoring the Volume and Stability of Comparative Claims

Abstract

The rapid pace of AI/ML research produces a steady stream of methods claiming state-of-the-art performance, each backed by a leading benchmark score. However, reported scores offer practitioners limited guidance on adoption, as they are obtained under one configuration of the many ad-hoc choices an experiment requires. This leaves two questions unresolved: (1) the amount of the plausible configuration space in which a method outperforms others, and (2) whether that lead survives a small change to the configuration. Existing evaluation tactics fall short of answering these questions, as they either evaluate methods under fixed configurations or analyze each method in isolation. We introduce Meta-Experiment Stability Analysis (MESA), which answers both. MESA evaluates a set of candidate methods across plausible experimental configurations to determine the regions where each method is in the top- after accounting for the noise between repeated runs. MESA scores each method's top- region by (1) its volume, the plausibility-weighted fraction of the space it covers, and (2) its stability, the typical distance from a point in the region to its boundary. The same meta-experiment can then be reused to evaluate how well benchmark datasets separate the methods. We demonstrate MESA through two case studies from different areas of AI/ML. In both, no method leads on a majority of the configuration space on any dataset. These results suggest that a leading benchmark score is weak evidence that a method will lead under a practitioner's own choices, and that MESA's volume and stability scores give evaluation a more informative target.

Primary area:
datasets and benchmarks
Keywords:
method evaluation, benchmark quality, stability, robustness, method comparison, sensitivity analysis, meta-experiments

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ineqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper suggests the MESA framework to evaluating machine learning methods. Each method is run for multiple seeds for each configuration in a predetermined set of plausible settings. Then, models are scored by both the volume of the configurations in which they significantly beat the empirically k-th best competitor, as well as the stability of that region (for numerical configurations where proximity of two configurations makes sense). The paper also proposes a dual view to evaluate benchmark datasets by how consistently they separate models from the kth-best model (again across configurations). The method is illustrated using two cases studies on conformal prediction and anomaly detection

Critical strengths
  • The stability of ML evaluations is an interesting topic
  • Some form of perturbation analysis as done in the paper seems like the right direction
  • The paper is written in an easily understandable way.
Critical weaknesses
  • The paper does not really convince me that its relatively complicated approach is necessary to achieve its goals (lots of alternative perturbation methods are mentioned in the related work, but there do not seem to be any meaningful comparisons to them). Similarly, the value add of separately reporting volume and stability is not really demonstrated.
  • k is a relatively hard to interpret hyperparameter, and having hyperparameters in evaluation methods is usually not great (quite related to the paper's main thrust)
  • MESA is much more expensive to run than standard benchmarking (by a factor of n_configurations*n_seeds)
  • The existence of some configurations that perturb model SOTA does not seem to establish that MESA yields improved model selection for real world settings compared to standard benchmarking.
Questions for authors

What is the exact difference to performance profiles?

8 comments
AuthorsOpenReview ↗

Reply to Reviewer ineq (1/2)

We thank the reviewer for their helpful comments and address the questions and weaknesses below. In short, we claim that leads declared by MESA hold on held-out runs far more often than the leader a standard benchmark reports, and MESA captures geometry and dataset quality, whereas standard benchmarking and performance profiles do not.

Question #1: Difference from performance profiles.

Performance profiles (Dolan and Moré, 2002) report ρs(τ)\rho_s(\tau), the fraction of problems where method ss is within a factor of τ\tau of the best performing method. MESA has three distinct differences:

  1. Noise. MESA grew from the premise that a new method should be adopted only when it clearly outperforms existing methods. Performance profiles count every point-estimate win, whereas MESA ensures that the winner at each configuration is separable from run-to-run noise.
  2. Geometry. Performance profiles ignore the ordering of configurations, so it is unable to quantify if a method’s wins form one cohesive region or many small fragments. The stability score provided by MESA adds this important context.
  3. Benchmark scores. MESA is able to use its meta-experiment information to evaluate the discriminatory abilities of the benchmark datasets, whereas performance profiles are solely for methods.

Weakness #1: Potentially unnecessary complexity.

Standard benchmark leaderboards compare point estimates on a single (often unreported) configuration. MESA goes further to quantify if a method clearly leads at each plausible configuration and gives practitioners a higher degree of granularity. Most perturbation-based approaches in our related work (e.g., multiverse analysis) analyze each method in isolation and do not seek to evaluate method performance in comparison to others, so we compare MESA against the rule a standard benchmark issues. In this way, MESA can be thought of as bringing multiverse-esque perturbation to model evaluation.

To test whether a statistical test yields better conclusions than mean performance, we ran a new analysis on the paper’s existing meta-experiments. At each configuration, we use half of the runs to declare a winner and check whether it still leads on the other half (Table 1). Code for this analysis is available in the anonymous repository within the rebuttal subdirectory.

Case StudyDataset% of Mean Performance Leads Holding% of MESA Leads Holding
Conformal PredictionCIFAR-10058%94%
Places36561%94%
Pl@ntNet80%98%
iNaturalist86%98%
Average71%96%
Anomaly DetectionCardio79%94%
Mammography84%96%
Thyroid59%85%
Vowels64%87%
Average71%90%
Table 1: MESA’s leads hold more consistently than mean performance across each benchmark dataset.

MESA’s stability score also describes the shape of the space where a method leads, which does not have a clear analogue from traditional benchmarking. The resulting information can be useful in determining if certain methods should be deployed: a highly unstable method may not suit a high-stakes scenario because small perturbations could cause its performance to collapse.

Weakness #2: Interpretation of k.

We would like to clarify that k is not a parameter to tune, but rather specifies which claim is being evaluated. Practitioners often want to identify the best method (k=1k=1), but sometimes want a ranking. Full rankings are rarely recoverable in a meta-experiment, since each adjacent pair of methods would need to be separated from inter-run noise at each configuration. Top-k thus acts as a hybrid. The claim carries more information than a single winner while keeping enough power to recover signal. For example, a practitioner with the budget to inspect the top three methods in their specific setting could set k=3k=3.

AuthorsOpenReview ↗

Reply to Reviewer ineq (2/2)

Weakness #3: Computational cost.

We agree with the reviewer that MESA has a higher computational cost than standard benchmarking. The factor of R (number of runs/seeds) is not unique to MESA, as repeated runs are already recommended practice for reliable comparisons (Bouthillier et al., 2021), but the reviewer is correct to point out that the number of configurations can cause further strain on computational cost/time. In future work, we plan on reducing this cost by applying supervised learning or imputation methods, as in Zeng & Papailiopoulos [1], to lessen the amount of computation needed. We would also like to point out that MESA’s dataset scores can focus future evaluations on benchmarks that separate methods best.

[1] Zeng, Yuchen, and Dimitris Papailiopoulos. "You Don't Need to Run Every Eval." arXiv preprint arXiv:2606.24020 (2026).

Weakness #4: Unclear improvements in model selection.

We respectfully disagree. A standard benchmark answers which method leads at one configuration, but deployment conditions may change over time. If the best method changes across configurations, the benchmark’s answer holds on only part of the space, and the practitioner cannot tell if the model is still the best under their current deployment conditions. In our case studies, we find this heterogeneity in performance across feasible experimental parameters. MESA tells a practitioner whether a method clearly leads across plausible configurations, and these declared leads hold up on held-out runs (Weakness #1).

We are happy to answer any follow-up questions or concerns and would be grateful if the reviewer would consider raising their score.

  • Could you please describe the experiment regarding weakness 1 in more detail? Is the left column using mean method performance over configurations? The baselines I had in mind were closer to simply using performance profiles. If I understand correctly, this experiment also does not show anything about the usefulness of stability over volume.
Reviewer_QYwsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MESA: a rigorous protocol for evaluating and comparing AI/ML empirical methods. It evaluate candidate methods across different experimental configurations and scores the method by both performance and stability (variance across different experimental settings). The authors shows that on selected existing benchmarks and areas (e.g. conformal prediction, anomaly detection), not method is overwhelming good than other methods, and calls into a more rigorous evaluation of AI/ML empirical research.

Critical strengths
  • The paper is timely, as it provides a nice protocol for us to evaluate empirical AI/ML research. Using it allows humans to evaluate their method more rigorously but also allow auto-research agents to check their method survives changing the experimental setup.

  • The paper presents nice case studies on two fields, showing the effectiveness of their proposed framework / protocol.

Critical weaknesses
  • in empirical research, there can be a lot of variables. For example, in LLM pre-training, there can be many hparams --- batch size (and its schedule), learning rate (and its schedule), optimizers, training data volume, model size ..., and that even if a method is indeed good, it might not manifest itself when the hparams is tuned badly. In other words, it might be the case that a method has a high ceiling only if you have tuned the hparams correct. However, stress testing it might result in the method having bad performance due to suboptimal configs. It is unclear if MESA takes this nature of methods into account.

  • It would be good if the authors can try on areas with clear good methods (e.g. transformers over RNNs / skip connections / Adam optimizers) and analyze that MESA indeed can show strength of such methods. This would show that MESA would not prematurely eliminate a good method.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer QYws

We thank the reviewer for the encouraging assessment and comments regarding the timeliness of this work and potential use in auto-research. We would like to address the weaknesses pointed out by the reviewer.

Weakness #1: Methods with hyperparameters that require tuning.

We agree with the reviewer that MESA should not perturb hyperparameters that a practitioner would usually tune or optimize over. We attempt to clearly state that MESA is meant for ad-hoc choices not made via tuning/optimization (Sections 3.1 and 5), but understand that the current writing may not be sufficiently clear. We will revise the writing to emphasize this more. In our case studies, the perturbed parameters are the ones the source code sets ad-hoc (Appendix A.2). If a reader would tune such a parameter, the meta-experiment can be reconfigured to tune it within each configuration, with no change to MESA itself.

Weakness #2: Lack of positive control.

We thank the reviewer for this excellent suggestion - we agree that including an example of a positive control would considerably improve the work. To start down this path, we have run a simple meta-experiment comparing MLPs with skip connections (“residual”) and without (“plain”) on Fashion-MNIST, using a fixed optimizer (SGD with momentum), no normalization layers, five paired runs, and test accuracy as the score. We use the example of skip connections because it is the item on the reviewer’s list with the least compute required. We perturb four experimental parameters:

Stable GridWide Grid
Depth (Linear Layers)4, 8, 164, 8, 16, 32
Width64, 128, 25664, 128, 256
Learning Rate1e-4, 1e-3, 1e-21e-4, 1e-3, 1e-2, 1e-1
Batch Size64, 128, 25664, 128, 256
Residual Top-1 Volume1.000.72
Plain Top-1 Volume0.000.05
No Method Declared Top-10%23% (19%: neither network trains)
Table 1: MESA recognizes the consistent strong performance of MLPs with skip connections.

On the stable grid, residual networks have top-1 volume and stability of 1. If we expand the grid to include a learning rate of 1e-1 and depth of 32, the residual networks without normalization diverge/fail to train as expected, and the residual volume is then reduced to 0.72 (plain volume is 0.05). MESA declares no winner on 23% of the configuration space in this expanded grid, including 19% of the space on which neither network trains.

The code for this example is in the anonymous repository under the rebuttal subdirectory. We hope this example of a positive control is helpful in demonstrating MESA’s ability to recognize methods that are clear winners in a certain application.

Clarification.

We just wanted to quickly clarify that MESA’s stability score is not the variance across settings, but rather the typical distance from a winning configuration to the nearest non-winning configuration.

We are happy to answer any follow-up questions or concerns that the reviewer may have and would be grateful if they would consider raising their score.

Thanks for the clarifications. I am willing to increase my score (2 -> 3). Could the authors answer typically how much datapoints do we need to have clear separation between two models? and how does that change if the gap between two models are small v.s. large. Do you need linearly more datapoints as we scale down the gap or exponentially more datapoints?

We would like to thank the reviewer for the increase in their score and are happy to answer the follow-up question. Two methods can be separated at two different levels. We will clarify both: at a particular configuration ψ\boldsymbol{\psi} and across the configuration range (i.e., the volume score). At both levels, the number of datapoints needed scales with the inverse square of the gap (e.g., halving the gap requires about four times as many datapoints). Specifically:

  • Separating two methods at a specific configuration ψ\boldsymbol{\psi}. Two methods have a significant separation when their mean difference over RR runs is more than tγ,R−1⋅ε^k/Rt_{\gamma,R-1}\cdot\hat{\varepsilon}_k/\sqrt{R}, where ε^k\hat{\varepsilon}_k is the standard deviation of their difference across runs. Thus, the smallest gap MESA can detect shrinks at a square-root rate in RR.
  • Separating two methods’ volumes. Volume is the π\pi-probability that a configuration is contained in a method’s top-kk region. Thus, we can think of volume estimated across NN configurations sampled from π\pi as the average of NN independent Bernoulli variables. The standard binomial bound gives us that its RMSE against the full-grid volume is at most RMSE≤1/(2N)\mathrm{RMSE}\leq1/(2\sqrt{N}). It follows that volumes that differ by Δ\Delta need N=O(1/Δ2)N=O(1/\Delta^2).
Reviewer_1UXGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MESA, a framework for evaluating the comparative advantages of machine learning methods. MESA evaluates candidate methods across different experimental configurations and uses statistical tests to identify their top-k SOTA regions. It then uses Volume and Stability to measure the coverage and stability of these regions. The authors also introduce two metrics to evaluate the ability of benchmarks to distinguish different methods. Through two case studies on conformal prediction and anomaly detection, the authors show that the advantages of different methods often depend on experimental configurations, and a single benchmark score may not fully reflect their performance.

Critical strengths
  • The paper addresses the limitations of traditional SOTA evaluation under fixed experimental configurations. Studying the stability of method advantages across different configurations is a meaningful problem.

  • MESA combines statistical significance tests, Volume, and Stability to evaluate the comparative advantages of different methods.

  • The authors conduct experiments in two different areas, conformal prediction and anomaly detection, covering multiple methods and datasets.

Critical weaknesses
  • “Modeling the Machine Learning Multiverse” has already studied the winning regions of different methods across experimental configurations. The authors should more clearly demonstrate what additional practical value MESA provides.

  • Volume heavily depends on the manually selected parameter space, which may lead to unstable conclusions. The paper may simply replace the bias of selecting a single experimental configuration with the bias of designing the configuration space.

  • The mathematical definition of Stability may not accurately reflect robustness to parameter changes. For example, in Equation 4(line 192), learning rates of 1e-5, 1e-4, and 1e-3 have very different relative distances in the original scale and the log scale. Therefore, the same experimental configurations may produce different Stability scores.

  • The experiments only show that SOTA rankings change across configurations and that Volume and Stability provide different information. However, they do not demonstrate that MESA selects better methods than traditional benchmarks, average rankings, or simple win rates.

Questions for authors

The authors should more clearly demonstrate what additional practical value MESA provides.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer 1UXG

We thank the reviewer for their thoughtful feedback. We would like to address the identified weaknesses and questions.

Question #1/Weakness #4: Does MESA select better methods?

A standard benchmark answers which method leads at one (often unreported) configuration, but deployment conditions may differ from that configuration or change over time. If the best method changes across configurations, the benchmark’s answer holds on only part of the space, and the practitioner cannot tell if the model is still the best under their current deployment conditions. In our case studies, we find this heterogeneity in performance across feasible experimental parameters. MESA tells a practitioner whether a method clearly leads across plausible configurations.

To test whether MESA’s statistical test yields better conclusions than mean performance at each configuration, we ran a new analysis on the paper’s existing meta-experiments. At each configuration, we use half of the runs to declare a winner (using mean performance and MESA) and then check whether the leader holds on the other half (Table 1). Code for this analysis is available in the anonymous repository within the rebuttal subdirectory.

Case StudyDataset% of Mean Performance Leads Holding% of MESA Leads Holding
Conformal PredictionCIFAR-10058%94%
Places36561%94%
Pl@ntNet80%98%
iNaturalist86%98%
Average71%96%
Anomaly DetectionCardio79%94%
Mammography84%96%
Thyroid59%85%
Vowels64%87%
Average71%90%
Table 1: MESA’s leads hold more consistently than mean performance across each benchmark dataset.

Weakness #1: Comparison to Bell et al. (2022).

Bell et al. determine winning regions using a Gaussian process surrogate model, and explicitly leaves run-to-run noise to future work. MESA adds:

  1. A noise-aware decision at each configuration, which is important, as much of the configuration mass in our case studies has no best method separable from run-to-run noise.
  2. Scalar volume scores comparable across methods and datasets, as well as stability scores to quantify a method’s fragility to small changes in experimental parameters.
  3. Scores to measure the discriminatory ability of benchmark datasets. Furthermore, MESA also scores the evaluated configurations directly, while a surrogate model fit to the multiverse often adds approximation error and fits poorly [1]. We view Bell et al.’s approach as complementary: in spaces too large to enumerate, a design like Bell et al.’s could choose which configurations to run, and MESA could then be used to create scores over the resulting configurations.

[1] Rewolinski, Zachary T., et al. "Pinpointing Pivot Points: The DECIDE Framework for Data Analysis Decision Importances." arXiv preprint arXiv:2610.38797 (2026).

Weakness #2: Results are sensitive to choice of configuration space.

We agree that MESA’s volume score moves the conversation from choosing a single configuration to defining a larger space of plausible configurations. We see this as a feature rather than a bug: a single benchmark is the special case of a point-mass π\pi on an individual configuration, but MESA makes this otherwise hidden choice explicit and opens up a healthy debate. Practitioners can also rescore the released runs using their own π\pi. The levels in our case studies are inspired by values from the papers which propose each candidate method (Appendix A.2).

Weakness #3: Results are sensitive to choice of parameter scale.

Yes, the reviewer is correct to point out that the stability score depends on the parameter scale. Similar to π\pi, this scale is fixed a priori, and most parameters have a natural scale (the scale on which their effect is roughly linear). For the reviewer’s learning-rate example, this would be addressed using the log-scale for this specific parameter, giving equal spacing to the possible levels. Parameters with unusual distance scales can also plug in a different distance function without changing the rest of MESA.

We are happy to answer any follow-up questions or concerns and would be grateful if the reviewer would consider raising their score.

Thank you for the response and additional experiments.

The new results address some of my concerns. However, the sensitivity of Volume and Stability to different parameter choices remains unclear. Also, the new experiments mainly validate the statistical tests, rather than the effectiveness of Volume and Stability for method selection.

I will keep my original rating.

We would like to thank the reviewer for engaging with our response. We address both remaining concerns below.

Sensitivity of volume and stability scores to parameter choices. We tested the sensitivity of volume and stability to parameter choices, specifically:

  • The weights π\pi on each parameter’s levels.
    • We drew each parameter’s level weights from a Dirichlet distribution, varying the departures from uniform weights from moderate (Dirichlet(20)) to extreme (Dirichlet(1)).
  • Which levels are included in the configuration grid.
    • We removed each level of each parameter one at a time from the configuration space (251 cases across the eight datasets between both case studies). Table 2 below shows how often the leading method for both volume and stability remains the same.
Perturbation% with Volume Leader Unchanged% with Stability Leader Unchanged
Level Weights - Moderate; Dirichlet(20)90%90%
Level Weights - Intermediate; Dirichlet(5)79%84%
Level Weights - Extreme; Dirichlet(1)62%70%
Single Level Removed from Configuration Space93%89%
Table 2: Percentage of volume and stability leaders which remain the same after applying different parameter choices.

We observe that the leader changes (for both measures) primarily where the scores already show a close race. Specifically, when we remove a level, 13 of the 18 leader changes in volume occur on the CIFAR-100 and Thyroid datasets, whose top two volumes differ by less than 0.02, and 26 of the 28 stability leader changes occur on datasets whose top two stability scores also differ by less than 0.02. When a method has a clear lead in volume or stability on a dataset, we observe that removing a single level or moderately reweighting π\pi rarely changes that score’s leader. In the cases of extreme reweighting, we see that some reweightings can concentrate nearly all the π\pi-weight on a single level of ψ\boldsymbol{\psi}, thus making changes in score leaders somewhat expected.

Effectiveness of volume and stability. We agree that the experiment from our previous response validates the per-configuration decisions that volume is computed on, rather than validating the scores themselves. It may also be worth seeing the positive control example in our reply to Reviewer QYws, in which we validate the scores against a commonly-known answer (MLPs with skip connections). Specifically, the experiment is concerned with MLPs with skip connections (residual networks), which are known to train more reliably at depth. We compare residual networks to MLPs without skip connections (plain networks) on the Fashion-MNIST dataset, finding that residual networks have volume 1.00 and stability 1.00, showing/recovering this known advantage across the configuration space. To test the limits of residual networks, we expand the grid to include depth 32 and learning rate 0.1, where residual networks with no normalization fail to train. We find that the volume of residual networks falls to 0.72, while plain networks have volume 0.05. Of the 23% of the space left without a declared winner, most (19% of the space) are ones where both network types fail to train.

We would be happy to answer any follow-up questions and would appreciate if the reviewer would reconsider their score.

Reviewer_Lg42OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MESA asks how often a method wins across different experimental settings and how easily it loses that lead. The authors evaluate methods across a grid of settings and use repeated runs to decide where each method has a significant lead. They call the share of the configuration distribution covered by those settings “volume.” They then measure the average distance to a setting where the method no longer qualifies, which they call “stability.” They also use the same results to examine how well each dataset separates the methods. The authors test this on class-conditional conformal prediction and isolation-based anomaly detection, with four datasets per task and ten runs per configuration. The results show that winning in more settings does not necessarily mean keeping a lead under small changes. The datasets also differ quite a bit in how often they identify a winner.

Critical strengths
  1. I like the distinction between volume and stability. For method selection, both the breadth of a method’s wins and its sensitivity to small changes matter. The Classwise CP and RC3P results on CIFAR-100 show why both numbers are useful (Table 1). This is the strongest part of the paper.
  2. The dataset comparison is useful too. Mammography identifies a top-1 winner on 74.7% of configurations, compared with 43.3% for Thyroid (Table 3). Treating these as equally informative benchmarks would overlook a substantial difference for this group of methods.
Critical weaknesses
  1. I am not convinced that the test establishes a statistically significant top-k set. In Section 3.1, Equation 2, the authors choose the competitor using the same runs that they then use to test the gap. Section 4.1, Equation 9, does this for both methods at the frontier. The effect of that selection on the test is not explained. For k greater than one, testing only the kth versus the (k+1)th method also seems insufficient: other comparisons across that boundary can have different variances. A justification or a simulation checking false winner and top-k declarations would help resolve this.
  2. The paper does not check how much its conclusions depend on its own choices of configuration weights and distances. Section 5 gives equal weight to each listed parameter level, and Section 3.3 measures stability using a standardized distance on the ordered parameters. Changing the listed levels changes the distribution being averaged over; using a different parameter scale can also change the distances. Since these choices determine the volume and stability scores, it is hard to tell whether the method rankings would hold under other reasonable choices. Section 6 acknowledges this dependence, but the experiments do not test it. I would want to see that check before treating the scores as reliable guidance for choosing a method.
Questions for authors
  1. How is selection of the compared methods accounted for in Equations 2 and 9? What error guarantee is intended, and has it been checked in simulations with unequal pairwise variances?
  2. Section 5 says ten runs are sufficient, referring to the diagnostic in Appendix A.1. Have the authors checked a subset of configurations with substantially more runs? Do the volumes and stability rankings stay similar?
  3. How much do the rankings depend on the selected levels and parameter scales? Do they stay similar with other reasonable configuration weights or log scales for some ordered parameters?
  4. Appendix A.7 describes the conformal-prediction score as a winner indicator. Could the authors define it explicitly in Section 5.1.1, including ties and cases where no method is feasible? Has the t-test been checked for these discrete differences with only ten runs?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer Lg42 (1/2)

We thank the reviewer for their thoughtful feedback. We would like to address the identified weaknesses and questions.

Question #1 / Weakness #1: Concerns regarding statistical significance.

The reviewer is correct that the original rules only compare the kth best method to the (k+1)-th best method, and that this is not necessarily sufficient to claim a top-k guarantee. We appreciate this feedback that strengthens our work and adapt the definition of top-k to say:

  • A method is top-k at a configuration if it beats at least (n-k) competitors, each one-sided at α/k\alpha/k (Eq. 2).
  • A top-k set is resolved if every member beats every non-member, each pair one-sided at α/(k(n−k)+1)\alpha/(k(n-k)+1), to ensure every comparison across the boundary is tested (Eq. 9).

Impacts on conclusions. Using these new definitions, we then re-ran our case studies to observe if it caused changes in conclusions. Specifically, since we already had the results of each run, this consisted of re-computing the scores with the new rules mentioned above (and described more in depth below). We found that:

  • The method-level conclusions hold, as the volume leaders are unchanged on all eight datasets and the old and new volumes have rank correlations from 0.90 to 1.00.
  • Dataset resolution volume drops due to the loss of power (Table 1).
Case StudyDatasetOriginal Rule Resolution VolumeRevised Rule Resolution Volume
Conformal PredictionCIFAR-1000.3390.184
Places3650.3610.188
Pl@ntNet0.3980.296
iNaturalist0.4020.319
Anomaly DetectionCardio0.6740.428
Mammography0.7470.565
Thyroid0.4330.179
Vowels0.4890.177
Table 1: Resolution volume across the original and revised Eq. 9 rules.

Error guarantee and interpretation. The statistical test is applied to each configuration, so thus the guarantee is per configuration. Specifically, if the top-k claim at ψ\boldsymbol{\psi} is false, it is declared as top-k with probability at most α\alpha. It follows that E[spurious volume]≤α∗π(ψ:claim is false)\mathbb{E}[\text{spurious volume}]\leq\alpha*\pi(\psi:\text{claim is false}).

Eq. 2. Unequal variances would break the original rule: the method with the kth highest mean may be significantly better than the method with the k+1th highest mean, but not than the k+2th method due to its larger variance. However, this is unlikely to be driving our results on methods, as the SD ratios in the anomaly detection case study are about even (median SD ratio of top two methods is 1.2), and in conformal prediction the score is 0/1, so tied methods have equal variance.

Eq. 9. Here, the new rule includes a Bonferroni correction to test at level α/(k(n−k)+1)\alpha/(k(n-k)+1), restoring a level α\alpha test at the cost of power.

Question #3 / Weakness #2: Concerns regarding π\pi and dπd_\pi.

With respect to the dependence on choice of π\pi, this is no different from standard benchmarking, which can be thought of as a special case where π\pi is a point mass. We believe that one of the strengths of MESA is that it makes the choice of π\pi explicit.

Weights. We will revise the paper to add empirical examples of conclusions changing as the weights change. In a preliminary analysis, we randomly redrew the weights of each parameter’s levels (1,000 draws per dataset) to observe:

Amount of Random ReweightingPercent of Unchanged Volume Leaders
Moderate: Dirichlet(20)90%
Intermediate: Dirichlet(5)79%
Extreme: Dirichlet(1)62%
Table 2: The volume leaders (averaged over the eight datasets) change more frequently as the randomly assigned weights become more extreme/uneven, as expected.

Scale. Yes, stability depends on the parameter scale. Like π\pi, this scale is fixed a priori, and there is usually a natural one, specifically the scale in which a parameter’s effect is roughly linear. Parameters with unusual distance scales can use a different distance function without changing the rest of MESA. We will make this choice of scale a more clear decision in the work.

AuthorsOpenReview ↗

Reply to Reviewer Lg42 (2/2)

Question #2: Results under increased number of runs.

For a preliminary analysis into this question, we ran 40 new seeds (for a total of 50 seeds) on a 5% sample of the conformal configurations, stratified by dataset and top-1 status. None of the leads declared from the paper’s 10 runs were overturned by the new runs. The main change is an increase in resolution volume, as a higher number of runs allows for more confidence in differences between method performance:

Dataset# of Configurations in Sample% of Configs with a Top-1 Lead Declared on 10 Runs% of Configs with a Different Top-1 Leader on 50 RunsResolution Volume, 10 → 50 Runs
CIFAR-10070432%0%0.15 → 0.38
Places3651,05627%0%0.17 → 0.41
Pl@ntNet1,08835%0%0.28 → 0.44
iNaturalist1,21636%0%0.28 → 0.41
Table 3: No method’s volume lead is overturned by more runs, but dataset resolution volume is increased by more samples providing more power. The rightmost column is estimated from the 5% sample of the conformal configurations.

Question #4: Conformal score.

Section 5.1.1 identifies the SOTA method in each run, and ff is its indicator. For configuration ψ\boldsymbol{\psi} and run rr, let FF be the set of feasible methods (with UnderCovGap≤δα\mathrm{UnderCovGap}\leq\delta\alpha). We then have that

\begin{cases} \mathbf{1}\\{h\in F, \\ \mathrm{AvgSize}_h=\min_{h'\in F}\mathrm{AvgSize}_{h'}\\}, & F\neq\emptyset,\\\\ \mathbf{1}\\{h \text{ has the smallest UnderCovGap, then the smallest AvgSize}\\}, & F=\emptyset. \end{cases}$$ Thus, every exactly tied method scores 1. For these discrete differences, we will revise the work to use the exact sign-flip test: the paired t-test rejected in 4-10% of tests where it did not, and volume leaders are unchanged. We are happy to answer any follow-up questions or concerns that the reviewer may have and would be grateful if they would consider raising their score.

We wanted to follow up with the reviewer to see if we adequately answered their question. We also wanted to point out that for the reviewer’s Question #3, our recent reply to Reviewer 1UXG extends Table 2 to the stability score and removing levels from the configuration space. We would welcome any additional comments and/or questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Paper Uploaded: Summary of Changes

We thank all reviewers for their feedback, which we believe has improved the strength of the work. We have uploaded a revised version of the paper (changes shown in blue).

We hope the revised paper adequately answers the questions and comments made by the reviewers. The main changes are:

  1. Revised decision rules (Lg42). We now declare a method to be top-kk at a configuration ψ\boldsymbol{\psi} if it significantly outperforms at least n−kn-k other candidates, each at the Bonferroni-corrected level α/k\alpha/k (Section 3.1). Similarly, a top-kk set is resolved only if each member outperforms every non-member, each at the Bonferroni-corrected level α/(k(n−k)+1)\alpha/(k(n-k)+1) (Section 4.1). We have propositions for both rules to show that the probabilities of false claims are bounded by α\alpha at every configuration.
    • The results were recomputed under these new rules, hence the updates to the numbers in the tables. The volume leader is unchanged on all eight datasets, while resolution volumes are lower due to the stricter rules/correction terms.
  2. Conformal score and test (Lg42). We define the conformal score (including ties and the case where no method is feasible) in Section 5.1.1, and use the exact sign-flip test for the conformal case study (Appendix A.3).
  3. New Section 5.3 - “Validating MESA”. This section contains many of the examples we have discussed here, such as the positive control of MLPs with skip connections (QYws), a comparison of declared leads against mean performance (1UXG, ineq), and the sensitivity of volume/stability scores to the configuration space (Lg42, 1UXG).
  4. Miscellaneous Clarifications. These include:
    • The scope of MESA’s experimental parameters does not include tuned hyperparameters selected via optimization (Section 3.1, QYws).
    • The distance scale for experimental parameters is fixed a priori (Section 3.3; Lg42, 1UXG).
    • Resolution volume grows with the number of runs, hence datasets should be compared at equal RR (Section 4.3).

There were other changes as well, but we wanted to highlight these. In particular, if we mentioned in a thread that we would revise the work to account for a question or comment, it should be addressed in the revised manuscript.

Lastly, we want to note that these changes have caused the numbering of appendices and tables to shift.

We thank the reviewers again for the helpful feedback and discussion. For convenience, we summarize some main concerns raised in the reviews and point to where each is addressed. Details are in the individual reviewer threads and the revised manuscript (changes in blue).

  1. Q (Reviewer Lg42): Are the decision rules statistically valid? A: The revised rules bound the probability of a false top-kk claim and false resolution claim by α\alpha at every configuration.
    • A method is now top-kk if it significantly outperforms at least n−kn-k other candidates at level α/k\alpha/k (Section 3.1). Similarly, a top-kk set is resolved if each member outperforms every non-member at level α/(k(n−k)+1)\alpha/(k(n-k)+1) (Section 4.1). Each rule makes an incorrect declaration at a configuration ψ\boldsymbol{\psi} with probability at most α\alpha (Propositions 1 & 2). Under these revised rules, the volume leader is unchanged on all eight case study datasets. Old and new volumes are similar (rank correlations of 0.9-1.0), and resolution volumes drop, as expected under the stricter revised rules.
  2. Q (Reviewers Lg42 & 1UXG): How sensitive are the scores to the configuration space? A: The volume/stability leaders are unchanged under 90%/90% of moderate reweightings and 93%/89% of single-level removals (Section 5.3).
    • We ran a new experiment which redrew each parameter’s level weights from Dirichlet distributions, and separately removed each level of each parameter one-at-a-time. Leaders change more often under stronger reweighting, e.g., 62%/70% unchanged under Dirichlet(1), which can put nearly all of a parameter’s weight on one level.
  3. Q (Reviewers 1UXG & QYws): Does MESA identify good methods? A: We include a positive control example suggested by Reviewer QYws, in which we show MESA recovers the known advantage of skip connections by giving residual MLPs volume and stability of 1.00 (Section 5.3).
    • We compare MLPs with and without skip connections on Fashion-MNIST, including depth, width, learning rate, and batch size as experimental parameters. We then stress-test the experiment by widening the grid to include settings where residual networks without normalization can fail to train, observing that the volume of residual networks falls to 0.72 (MLPs without skip connections have volume 0.05), with the majority (19% of the 23%) of the space with no declared leader being a configuration where neither network trains.
  4. Q (Reviewers 1UXG & ineq): How does MESA compare to picking the mean winner? A: MESA’s declared leads hold on held-out runs 96% of the time in the conformal prediction case study and 90% of the time in the anomaly detection case study, versus 71% on each from the mean-performance winner (Section 5.3).
    • At each configuration, we use half the runs to declare a leader and check if it has the highest mean on the held-out half. Since the performance profile value ρs(1)\rho_s(1) is exactly the mean winner’s share of configurations, the comparison also applies to performance profiles.

We are happy to answer any further questions the reviewers may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.