acceptodds
Under review as a conference paper at ICLR 2027

Beyond Arrow: From Impossibility to Possibilities in Multi-Criteria Benchmarking

Abstract

Modern benchmarks such as HELM MMLU account for multiple metrics like accuracy, robustness and efficiency. When trying to turn these metrics into a single ranking, natural aggregation procedures can produce cyclic pairwise comparisons or rankings that are unstable to changes in the model set. We formalize this aggregation as a social choice problem where each metric induces a preference ranking over models on each dataset, and a benchmark operator aggregates these votes across metrics. While prior work has focused on Arrow's impossibility result, we identify sufficient conditions under which the impossibility disappears and meaningful multi-criteria benchmarking becomes possible. In particular, we deal with three restrictions on the combinations of rankings and show that on single-peaked, group-separable and distance-restricted preferences, the benchmark operator allows for the construction of well-behaved rankings of the involved models. Empirically, we investigate several modern benchmark suites like HELM MMLU and verify which structural conditions are fulfilled on which benchmark problems.

Primary area:
datasets and benchmarks
Keywords:
Multi-criteria benchmarking, model evaluation, social choice theory, preference aggregation, restricted preference domains, leaderboard rankings

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vM9TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper theorizes the aggregation procedure of multiple metrics in modern benchmark following a "social choice problem" point of view, in order to provide unified coherent rankings. Such rankings should require to avoid cyclic pairwise comparisons (thus require transitivity) and pairwise changes due to new instances to rank (thus require stability). The paper recalls Arrow's impossibility theorem, stating that an aggregation that fully satisfies a given set of desirable axioms is, in general, impossible. Hence, in line with other works, the paper relaxes a hypothesis on the domain on which to operate, and thus provides three cases expressed as sufficient conditions (called single-peakedness, group-separability, distance-restrictedness) on the rankings to aggregate such that the simple yet useful pairwise majority voting aggregation procedure produces well-behaved rankings. Some experiments quantify how much these conditions are satisfied with the HELM MMLU benchmark.

Critical strengths

The most notable strength is the solid theoretical formalization of the problem of aggregating multiple criteria in a coherent ranking. The Motivation is well-documented, the existing pathologies are convincing, and the overall approach is usually explained textually in addition to the math, which facilitates understanding of the technical parts.

The benchmark chosen for the experimental part of the paper is perfectly aligned with the theory developed in the paper and is timely, given the overwhelming presence of large AI models and the need to evaluate them rigorously.

Critical weaknesses

As the paper is mostly theoretical, the experimental part occupies only a very minor proportion of the main text (without considering supplementary material), and does not provide deep insights, figures, tables, discussions on the implications of the results for the practitioners and model developers. As such, the interested reader is left with a "so what?" feeling, since mostly statistical results are provided.

It is not crystal clear to me where the actual theoretical contributions start and end, and what is mostly re-used from other works, in sections 4 and 5. In that vein, a claimed contribution is the proofs that the three sufficient conditions allow for a well-behaved ranking, but the proofs are buried in supplementary material, which is a troubling choice. I believe the 9-page limit was a hard constraint for this work.

Minor typo: point missing at the end of line 295. Minor comment: "models" seem to have turned to "algorithms" around line 327.

Questions for authors

Beyond a nice theory, what does the paper give to the practitioners? What are the deep implications of the findings of the experimental section? If the paper is accepted, one extra page will be allowed; what would be shown and clearly exposed on that page?

Can you please clarify what is truly "yours" and "new" from what is "re-used" and "known" in sections 4 and 5? I believe most is "yours", but reinforcing the distinction throughout the paper might help better understand the contributions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vM9T (Part 1/2)

Dear Reviewer vM9T, we thank you for your positive and encouraging review. We are very happy that you find our theoretical formalization solid and the benchmark perfectly aligned with the theory. We agree with you that the distinction between existing results and our contributions as well as the implications of the tests for practitioners should be more emphasized in the main text.

What is new in Sections 4 and 5?

Our contributions are the formalization of multi-criteria, multi-task benchmarking as a preference aggregation problem in two steps, the application of the restricted preference domains to this setting, the diagnostic protocol and the empirical tests across choices of models, metrics and datasets. As far as we know, prior work on benchmarking has not investigated these restricted domains.

  1. We formalize the aggregation within each dataset, where metrics act as voters and models as alternatives, and the subsequent aggregation of the resulting rankings across datasets. Arrow’s theorem and the commonality sharing rule described in Section 4 are existing results. Our contribution is their application within this framework.

  2. We apply the three restricted preference domains to benchmarking and give them clear meaning in this setting. The results on single-peakedness and group separability are existing results from the social choice literature: Theorem 5.2 applies Black’s result [1], while Theorem 5.4 uses Inada’s results [2,3]. For Theorem 5.6, we provide a direct proof for metric rankings that disagree on at most one pairwise comparison.

  3. We turn these conditions into a diagnostic protocol that can be tested directly on benchmark data and empirically investigate when the conditions hold.

The central question of our work is: do multi-criteria benchmarks naturally exhibit structure that makes meaningful aggregation possible? Our framework offers testable sufficient conditions, while our experiments investigate where this structure occurs in practice. We will reinforce this distinction more thoroughly in the revision.

What does the paper give to practitioners? What are the implications of the experiments?

Our framework gives a procedure for constructing an overall ranking of models across several metrics and datasets, together with a diagnostic for checking whether the conditions for meaningful aggregation are satisfied.

The main practical takeaway is the diagnostic: benchmark designers can test whether the chosen models and metrics satisfy sufficient conditions for a consistent majority ranking. Reviewer Merm also highlights the protocol among the paper’s “several practical takeaways”.

In our response to Reviewer 3AgR, we cover the benefits for model selection and stability to changes in the model set and report additional transitivity and sensitivity experiments.

The implications of our experiments are particularly clear when we summarize the results by the kinds of metrics being aggregated. Holding the number of metrics fixed at three, the following table lists the percentage of profiles on individual subjects satisfying each property, across the 83 model sets and 57 subjects:

Metric setSingle-peakednessGroup separabilityTransitive pairwise majority
Three accuracy metrics99.98%99.77%100.00%
Two accuracy metrics and one efficiency metric82.39%88.33%99.94%
One accuracy metric, two efficiency metrics39.79%53.61%80.84%

We see that the choice of metrics matters substantially for determining whether the overall ranking is well-behaved. Combining one accuracy metric with efficiency metrics such as runtime and output length results in trade-offs, yet pairwise majority still induces a consistent ranking in 80.84% of the profiles on individual subjects.

The sensitivity experiments reported in our response to Reviewer 3AgR give a further implication. When all three metrics are recalculated on paired bootstrap samples of questions, the original winner model beats every other model in 87.1% of samples on average, while the entire ranking remains unchanged in 75.6%, averaged across the 3,825 profiles where the original majority ranking is transitive. Hence, practitioners can examine both whether the criteria induce a consistent ranking and how sensitive the selected model and the complete ranking are to changes in the evaluation questions.

Continued in Part 2/2.

[1] D. Black. On the rationale of group decision-making. Journal of Political Economy, 56(1):23–34, 1948.

[2] K. Inada. A note on the simple majority decision rule. Econometrica, 32(4):525–531, 1964.

[3] K. Inada. The simple majority decision rule. Econometrica, 37(3):490–506, 1969.

AuthorsOpenReview ↗

Response to Reviewer vM9T (Part 2/2)

What will we add to the main text?

You are right that the limit of nine pages was a hard constraint for us. We will use the additional page to include short proof sketches for the theorems, a concrete example showing how the diagnostic leads to a model comparison as well as the table above with its implications for benchmark design. We will also summarize the new sensitivity results and explain how practitioners can examine both the selected model and the complete ranking.

We will upload the revised manuscript during the discussion period.

We hope these clarifications answer your questions. We sincerely thank you for your support and appreciation of our work!

Reviewer_vM9TOpenReview ↗

Revisions

I thank the authors for their exhaustive answer, to all reviewers comments. I am willing to read the revised version to check the changes, although I would not encourage using the extra page for proof sketches for the theorems. It seems that most reviewers would be happy to see the practical applications and implications of this work, hence I would suggest developing those aspects in high priority.

Reviewer_MermOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a growing problem in ML evaluation: how to turn many benchmark measurements into a single meaningful model ranking. Modern benchmarks rarely measure only one thing. A model may be evaluated on accuracy, robustness, inference time, output length, cost, fairness, or other criteria. In practice, benchmark designers often collapse these measurements into a single leaderboard by averaging scores or ranks. The paper argues that this common approach can be misleading because different metrics may encode genuinely different preferences over models.

The authors frame the problem using an analogy from social choice theory. Metrics are treated like “voters,” models are treated like “candidates,” and the benchmark’s job is to combine the metric-level rankings into one overall ranking. This framing makes clear why aggregation can fail. For example, one metric may prefer model A over B, another may prefer B over C, and a third may prefer C over A, producing a cycle with no coherent best-to-worst ordering. Similarly, rank-based rules can be unstable: adding a new model to the benchmark can change the relative order of two models that were already being compared. These are not just theoretical concerns. The paper shows that such cycles and ranking flips occur in real HELM MMLU benchmark settings, especially when accuracy and efficiency metrics are mixed.

Rather than treating these failures as proof that single leaderboards are always impossible, the paper asks a more constructive question: when is aggregation actually safe? The key idea is that real benchmark metrics are not arbitrary. They often have structure. For example, several accuracy metrics may mostly agree with one another, or models may lie along a rough spectrum from smaller/faster to larger/more capable, with different metrics preferring different points on that spectrum. The paper identifies three kinds of structures: 1. single-peakedness, 2. group separability, and 3. distance-restrictedness, that are enough to make pairwise-majority aggregation behave well. Intuitively, if metric disagreements are orderly rather than chaotic, then a single ranking can be defended.

The empirical part of the paper tests whether these structures actually appear in modern benchmarks. On HELM MMLU, the authors find that the proposed conditions often hold when the metrics are closely related, especially for sets of accuracy metrics. This makes sense, as exact match, prefix exact match, and similar accuracy measures tend to rank models in broadly similar ways. However, the structure becomes much less common when accuracy metrics are combined with efficiency metrics like inference time or output length. This is also intuitive: a model that is highly accurate may be slower or more verbose, while a faster model may be less accurate. In such cases, forcing everything into one leaderboard is more questionable.

The practical takeaway is that benchmark designers should not automatically publish a single leaderboard just because multiple metrics are available. Instead, they should first ask whether the metric-induced rankings have enough structure to justify aggregation. If the diagnostic passes, then simple aggregation methods like pairwise majority can produce coherent rankings. If it fails, that is a warning sign: the benchmark may need to report separate dimensions, revise the metric set, or be more explicit about the trade-offs being made.

For researchers, the paper’s broader implication is that leaderboard validity should be treated as an object of study, not as a default assumption. The contribution is less about proposing a new benchmark or model, and more about giving the field a principled vocabulary for deciding when a benchmark ranking is meaningful. It shifts the question from “which model is best?” to “is this benchmark structured in a way that makes the notion of ‘best’ well-defined?”

Critical strengths

I enjoyed reading this paper, and I think there are several practical takeaways from it, especially for practitioners. Specifically,

  • Addresses a real and increasingly important evaluation problem. The paper targets a genuine weakness in modern ML benchmarking: the tendency to collapse multiple, potentially conflicting metrics into a single leaderboard without checking whether such aggregation is meaningful.

  • Provides a clear conceptual bridge between ML benchmarking and social choice theory. Treating metrics as voters and models as candidates is a useful framing that makes benchmark pathologies easier to understand and reason about. It also connects current ML evaluation issues to a mature body of theory.

  • Moves beyond impossibility results toward constructive conditions. Rather than simply arguing that aggregation can fail, the paper identifies concrete structural conditions under which coherent aggregation is possible. This makes the work more actionable than a purely negative critique of leaderboards.

  • Offers an intuitive diagnostic for leaderboard validity. The paper’s main practical contribution is the idea that benchmark designers should test whether metric-induced rankings have enough structure before reporting a single aggregate ranking. This is a valuable shift from assuming that aggregation is always legitimate.

  • Demonstrates that the issue is empirically relevant, not merely theoretical. The paper shows that cycles and ranking instability occur in real HELM MMLU settings, especially when combining accuracy and efficiency metrics. This strengthens the motivation and shows that the proposed concerns matter in practice.

  • Empirical findings align well with intuition while still being informative. The results show that aggregation is more defensible when metrics measure similar constructs, such as related accuracy metrics, and less defensible when metrics encode trade-offs, such as accuracy versus efficiency. This supports a practical interpretation of when single leaderboards should or should not be trusted.

  • Encourages more transparent benchmark design. By making the assumptions behind aggregation explicit, the paper can help practitioners decide when to report one ranking, when to report multiple dimensions, and when to revisit the metric or model set.

Critical weaknesses

Despite the strengths, I find myself reluctant to recommend an accept for this paper, and it has to do with venue fit more than the contents of this work. Here are some of my major complaints -

  • The technical novelty is somewhat limited. The main theoretical ingredients—single-peakedness, group separability, distance-restrictedness, and the use of majority aggregation under restricted domains—come from established social choice theory. The paper’s contribution is mainly in applying and empirically testing these ideas in ML benchmarking, rather than developing substantially new theory.

  • The practical prescription remains somewhat incomplete. The paper gives useful diagnostics for when aggregation is justified, but offers less concrete guidance for what benchmark designers should do when the diagnostics fail. In many realistic settings, especially when combining accuracy, efficiency, cost, and safety metrics, these conditions may not hold.

  • The sufficient conditions may be too strong or brittle. The proposed structural restrictions are clean theoretically, but real benchmark data often contains near-ties, noisy measurements, and small score perturbations. The paper acknowledges tie-breaking issues, but approximate or robust versions of the conditions would likely be more useful in practice.

  • The empirical scope is somewhat narrow. Most of the analysis centers on HELM MMLU, where the metric structure may be especially favorable for the proposed diagnostics. The additional PMLB and OpenML results suggest that the same structural conditions are much less prevalent in other settings, raising questions about how broadly the framework applies.

  • The main empirical conclusions are partly intuitive. It is unsurprising that closely related accuracy metrics produce more structured rankings, while mixing accuracy and efficiency creates stronger trade-offs and more instability. The paper validates this carefully, but the empirical findings may not feel especially surprising to practitioners.

  • The approach diagnoses leaderboard validity but does not necessarily improve model selection. The paper helps determine when a single ranking is coherent, but it does not show that using the proposed aggregation procedure leads to better downstream decisions, better benchmark design choices, or improved model selection in practice.

  • The ICLR fit may be somewhat borderline. The work is relevant to ML evaluation, but its center of gravity is closer to social choice theory and benchmark methodology than to core ML modeling, learning algorithms, or empirical ML performance. Some reviewers may view it as a useful evaluation-methodology paper but not a strong main-track ICLR contribution. It may be a better fit for a benchmarking/evaluation-focused venue or track, such as the NeurIPS Datasets and Benchmarks track, or a targeted workshop on ML evaluation, benchmarks, reliable leaderboards, or social choice and ML. It could also fit workshops adjacent to responsible AI evaluation, model evaluation methodology, or decision-making / preference aggregation in ML, where the social-choice framing would likely be more central rather than peripheral.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer Merm (Part 1/2)

Dear Reviewer Merm, we sincerely thank you for your very thoughtful and constructive feedback. We are particularly grateful for your recognition of the many practical takeaways of our work. Below, we clarify the relevance of our contribution to ICLR and discuss additional tests that address your concerns.

The novelty of our contribution and its fit to ICLR

Our main contributions comprise the formalization of multi-criteria, multi-task benchmarking as a preference aggregation problem in two steps, the application of the restricted preference domains to this setting, the diagnostic protocol and the empirical tests across choices of models, metrics and datasets. To our knowledge, existing literature on benchmarking has not examined the restricted preference domains. It is true that the social choice results on single-peakedness and group separability are not new; we give these conditions clear meaning in benchmarking and examine where they occur in practice and what this structure means for benchmark design.

We are firmly convinced that this question is directly relevant to ICLR. The ICLR 2027 Call for Papers [1] explicitly invites submissions on the topic of benchmarking. Moritz Hardt’s ICLR 2024 keynote talk titled The Emerging Science of Benchmarks [2] addressed the validity of model rankings and multi-task benchmarks with a connection to social choice. Furthermore, recent ICLR conference papers deal with ranking aggregation using social choice such as Designing Rules to Pick a Rule: Aggregation by Consistency [3], while MMTEB [4] applies Borda aggregation to rank models. Our work contributes to this research by identifying and empirically analyzing when the metric rankings allow for well-behaved aggregation.

The empirical scope and agreement between metrics

We agree that stronger agreement between closely related accuracy metrics is an intuitive result. Nonetheless, our paper also examines whether well-behaved aggregation is possible when the metrics represent competing objectives.

As an additional test, using the three metrics accuracy, runtime and output length, we investigate what happens if each of the three metrics prefers a different model. Across the 83 model sets and 57 MMLU subjects, 760 profiles have a uniquely best model for each of the three metrics. Among those, 256 (33.7%) profiles satisfy at least one of the sufficient conditions. Therefore, the conditions can be fulfilled even when accuracy and efficiency metrics favour different models, and in these cases, majority voting still yields a complete and transitive ranking of models.

Our interpretation of the lower prevalence of the conditions on PMLB and OpenML is that this structure differs across benchmarks and is highly dependent on the choice of models and metrics as well as the heterogoneity between them. We have therefore extended our analysis on these suites to understand what practitioners can still obtain when the diagnostic fails.

Continued in Part 2/2.

[1] The 15th International Conference on Learning Representations, Call for Papers.

[2] M. Hardt. The Emerging Science of Benchmarks. Invited talk at the 12th International Conference on Learning Representations, 2024.

[3] R. E. Berker, B. Armstrong, V. Conitzer and N. B. Shah. Designing Rules to Pick a Rule: Aggregation by Consistency. Proceedings of the 14th International Conference on Learning Representations, 2026.

[4] K. Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. Proceedings of the 13th International Conference on Learning Representations, 2025.

AuthorsOpenReview ↗

Response to Reviewer Merm (Part 2/2)

What should benchmark designers do when the conditions fail?

Since the conditions are sufficient, but not necessary, their failure does not by itself rule out the possibility of a well-behaved ranking. We will include the following procedure in the revised version of the paper:

  1. If none of the conditions is satisfied, test the pairwise majority relation directly. If it is complete and transitive, report the resulting ranking.
  2. If the full ranking is not transitive, check whether a model nevertheless beats every other model. Such a model can still be found when the objective is merely to choose one model.
  3. If no such model exists, take the smallest group of models in which every model beats every model outside the group together with the conflicting comparisons within it. This yields a shortlist of models when the group is smaller than the original model set. Further selection can then depend on the application’s priorities. For example, if the main objective of the evaluation is accuracy, then the model with the highest accuracy in this group should be selected.

The third step above is based on the concept of a Smith set, a tournament solution from the computational social choice literature, as described in [5, Section 3.3.3].

We evaluated these possibilities on all three benchmarks under the alphabetical tie-breaking:

BenchmarkAt least one condition holdsComplete and transitive majority rankingModel beats every other model
HELM MMLU3,081 / 4,7313,825 / 4,7313,965 / 4,731
PMLB17 / 6351 / 6360 / 63
OpenML4 / 8040 / 8061 / 80

On HELM MMLU, this test considers accuracy, runtime and output length. PMLB uses accuracy and two robustness metrics, while OpenML includes accuracy and training and test runtime. The last column includes the cases with a transitive ranking. For instance, on OpenML, although only four datasets satisfy a sufficient condition, 40 induce a complete and transitive majority ranking. On another 21 datasets, a model beats every other model despite the full ranking not being transitive. Of the remaining 19 datasets, 18 admit a smaller shortlist under the third step of the procedure above.

These results provide concrete guidance for model comparison and selection even when the structural conditions from our paper fail. They also distinguish the requirements for choosing one model from those for reporting a complete benchmark ranking.

Near-ties, noisy measurements and small score perturbations

Moreover, in our response to Reviewer 3AgR we discuss additional sensitivity experiments, which we will include in the revised version of the paper. New experiments on changes in runtime allow for reversals of pairwise comparisons when models have similar values of runtime. In the experiments with questions sampled with replacement, the original winner model beats every other model in 98.2% of samples when only runtime is computed from these questions and in 87.1% when all three metrics are computed. In the latter case, pairwise majority is transitive in 94.4% of samples. These percentages are averaged across the original 3,825 profiles that induce a transitive majority ranking.

We hope these clarifications and experiments resolve your concerns. We would be very grateful if you would consider revisiting your assessment and are happy to answer any questions. Thank you very much!

[5] F. Brandt, M. Brill and P. Harrenstein. Tournament Solutions. In F. Brandt, V. Conitzer, U. Endriss, J. Lang and A. D. Procaccia (eds.), Handbook of Computational Social Choice, Chapter 3, pages 57–84. Cambridge University Press, 2016.

Reviewer_3AgROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

ML models are judged by their performance on benchmarks, with leaderboards that rank the performance of models. However, these rankings compress information across multiple criteria, where different models with different strengths perform differently. Arrow's theorem establishes the impossibility of reconciling all of these rankings consistently. However, the authors of this paper propose a protocol for identifying cases where a consistent aggregation is possible. They test on HELM MMLU, PMLB, and OpenML.

Critical strengths
  • The authors validate their protocol with a wide variety of models and real benchmarks. On HELM MMLU, they use 83 model sets and 42 metric subsets. While the benchmarks and models are older, I do not find any reason to think that significantly diminishes their analysis.
Critical weaknesses
  • I am not sure if formalizing benchmarking as preference aggregation is well-suited to the way model benchmarking is actually used. For instance, the authors use accuracy, time, and length. But a marginal increase in time could lead to a large change in the aggregated ranking, even if the measure of most importance was accuracy, or if most models were closely clustered in time.

  • I'm not convinced of the usefulness of this protocol. It isn't clear to me what being able to create a consistent order in this situation gives a practitioner. There are tradeoffs to different metrics, the interplay between metrics can be complicated, and the paper doesn't make clear what significant benefit majority voting provides, even when these conditions hold.

Questions for authors
  • What does a practitioner gain from a consistent majority ranking compared with reporting each metric separately? Where might this protocol provide a significant benefit?

  • Is stability meant to hold only within each dataset? Can combining datasets with commonality sharing still flip two models if a third model is removed?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3AgR (Part 1/2)

Dear Reviewer 3AgR, we sincerely thank you for your thoughtful and constructive feedback. We appreciate you appraising the number of model and metric sets used in our experiments. Below are our responses to your questions and concerns.

What does a practitioner gain from a consistent majority ranking as compared with reporting each metric separately? Where might this protocol provide a significant benefit?

We agree that reporting each metric separately can be valuable. However, an aggregated ranking is crucial whenever a decision needs to be made and there is a disagreement between metrics. For instance, imagine a team that chooses a model for deployment or shortlists models for further testing. In this case, one would want to examine accuracy, runtime and output length simultaneously. Our protocol helps practitioners to identify under which conditions one consistent ordering of models that enables such comparison is possible. Furthermore, if we have a complete and transitive strict majority relation, the resulting ranking minimizes the total number of pairwise disagreements with the individual metric rankings. Hence, it can be intuitively interpreted as an overall agreement with the criteria.

There is an additional stability benefit: within a dataset, removing a model does not change the comparison of two remaining models when their metric values are unchanged. This matters for example when a shortlist changes because a model becomes unavailable. Our contribution is to identify and empirically investigate when these pairwise comparisons induce a consistent overall ranking.

We have run additional tests on 83 model sets and 42 metric subsets (3,486 combinations of model and metric sets and 198,702 profiles on individual subjects) that illustrate the transitivity of the induced pairwise majority relation on HELM MMLU. We use alphabetical tie-breaking and ten repetitions of random tie-breaking independently for each metric in each profile and compare these results with uniformly random profiles with the same numbers of models and metrics.

Weak pairwise majority is transitiveHELM MMLUHELM MMLU, random tie-breaking (mean of ten)Uniformly random profiles
3 models, 3 metrics98.343%98.426%94.444%
5 models, 3 metrics94.025%94.762%67.569%
9 models, 3 metrics80.000%80.825%14.198%
9 models, 4 metrics58.480%57.708%0.033%
All 198,702 profiles on individual subjects93.651%93.542%66.638%
Configurations transitive on all 57 subjects78.371%75.175%0.872%

The results imply that transitivity occurs more often on HELM MMLU as compared to the uniformly random case.

Formalizing benchmarking as preference aggregation and sensitivity to a marginal increase in time

We think that our framework - which is more general than a mere multi-criteria benchmarking problem and can be used to formalize any problem that produces evaluations over a finite set of alternatives - is suitable for benchmarking. Our work is also based on a comprehensive body of literature studying benchmarking as a social choice problem [1,2,3,4].

It is true that accuracy and runtime have different units. So combining them requires a choice about their corresponding weights in the decision process. In the case of Borda score and mean win rate one does not need to specify a numerical conversion rate between, e.g., one percentage point of accuracy and one second of runtime. These rules are, however, not stable to changes in model set, as we explain in the paper.

The procedure of majority voting can also be adapted to include different metric weights: we can repeat a metric’s ranking and thus incorporate integer weights. This preserves single-peakedness and group separability, so their transitivity guarantees apply under the corresponding theorem assumptions. An application that is more focused on accuracy can thus assign greater importance to this criterion.

Continued in Part 2/2.

[1] G. Zhang and M. Hardt. Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:58984–59002, 2024.

[2] M. Hardt. The Emerging Science of Machine Learning Benchmarks. Princeton University Press, 2026.

[3] M. Hardt. The Emerging Science of Benchmarks. Invited talk at the 12th International Conference on Learning Representations, 2024.

[4] M. Rofin, V. Mikhailov, M. Florinskiy, A. Kravchenko, E. Tutubalina, T. Shavrina, D. Karabekyan and E. Artemova. Vote’n’Rank: Revision of Benchmarking with Social Choice Theory. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 670–686, 2023.

AuthorsOpenReview ↗

Response to Reviewer 3AgR (Part 2/2)

Furthermore, we have examined your concern about marginal runtime changes in two new experiments. Using the three metrics accuracy, runtime and output length, we evaluated all 83 model sets on all 57 subjects. Pairwise majority yields a transitive ranking in 3,825 of these 4,731 configurations.

First, for those 3,825 cases, we checked whether the outcome is unchanged under every independent runtime change within the given range (±1%, ±5% or ±10% of the model’s original runtime on the given subject), while holding the values of accuracy and output length fixed:

Change in runtimeProfiles where the original winner model beats every other modelProfiles where the entire ranking is unchanged
±1%98.20%95.74%
±5%92.39%80.86%
±10%85.54%67.27%

Second, we draw 10,000 paired bootstrap samples of MMLU questions for each subject. Each sample consists of the original questions that are sampled with replacement. We recalculate either runtime alone, keeping accuracy and output length fixed, or all three metrics together. The table reports the percentage of samples in which the original winner or ranking remains unchanged, averaged across the 3,825 profiles.

MetricsOriginal winner model beats every other modelEntire ranking is unchanged
Runtime only98.2%95.9%
Accuracy, runtime and output length87.1%75.6%

Marginal changes in runtime thus rarely overturn the selected winner model in our experiments.

Is stability meant to hold only within each dataset? Can combining datasets with commonality sharing still flip two models if a third model is removed?

Yes, the stability guarantee concerns pairwise majority within each dataset. Commonality sharing in Section 4.3 selects a representative ranking across datasets. Removing a model can still change the comparison of two other models in this ranking.

We hope these clarifications answer your questions, and we would be very grateful if you would consider revisiting your assessment. Thank you very much!

Reviewer_3AgROpenReview ↗

Response to Authors

I thank the authors for their detailed response and the revised manuscript.

The sensitivity experiments in B.6, the procedure in section 7, and the example in 5.1 each address several of my concerns, and I am satisfied with their inclusion.

I would suggest including the stability guarantee note provided on commonality sharing in some form within the paper.

Overall, I am satisfied with the new results and revisions to the paper. I also found the responses and comments to/from reviewer vM9T compelling. As a result, I plan to raise my score.

Dear Reviewer 3AgR, we sincerely thank you for raising your score and for your positive feedback on the revised manuscript.

We will include the clarification on stability under commonality sharing in the paper. Thank you very much!

Comments on the paper

1 comment

We sincerely thank all reviewers for their thoughtful feedback. We have just uploaded a revised version of our paper, which includes the following changes:

  • A concrete model comparison example and results for competing objectives (Section 5.1; Appendix D.1).
  • Implications for benchmark design and sensitivity to changes in runtime and evaluation questions (Section 6.2; Appendix B.6).
  • A procedure for model selection with tests on all three benchmarks (Section 7; Appendix D.2) and additional results on transitivity of pairwise majority (Appendix E.3).

We have marked the additions in blue. The code has also been updated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.