- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper theorizes the aggregation procedure of multiple metrics in modern benchmark following a "social choice problem" point of view, in order to provide unified coherent rankings. Such rankings should require to avoid cyclic pairwise comparisons (thus require transitivity) and pairwise changes due to new instances to rank (thus require stability). The paper recalls Arrow's impossibility theorem, stating that an aggregation that fully satisfies a given set of desirable axioms is, in general, impossible. Hence, in line with other works, the paper relaxes a hypothesis on the domain on which to operate, and thus provides three cases expressed as sufficient conditions (called single-peakedness, group-separability, distance-restrictedness) on the rankings to aggregate such that the simple yet useful pairwise majority voting aggregation procedure produces well-behaved rankings. Some experiments quantify how much these conditions are satisfied with the HELM MMLU benchmark.
Critical strengths
The most notable strength is the solid theoretical formalization of the problem of aggregating multiple criteria in a coherent ranking. The Motivation is well-documented, the existing pathologies are convincing, and the overall approach is usually explained textually in addition to the math, which facilitates understanding of the technical parts.
The benchmark chosen for the experimental part of the paper is perfectly aligned with the theory developed in the paper and is timely, given the overwhelming presence of large AI models and the need to evaluate them rigorously.
Critical weaknesses
As the paper is mostly theoretical, the experimental part occupies only a very minor proportion of the main text (without considering supplementary material), and does not provide deep insights, figures, tables, discussions on the implications of the results for the practitioners and model developers. As such, the interested reader is left with a "so what?" feeling, since mostly statistical results are provided.
It is not crystal clear to me where the actual theoretical contributions start and end, and what is mostly re-used from other works, in sections 4 and 5. In that vein, a claimed contribution is the proofs that the three sufficient conditions allow for a well-behaved ranking, but the proofs are buried in supplementary material, which is a troubling choice. I believe the 9-page limit was a hard constraint for this work.
Minor typo: point missing at the end of line 295. Minor comment: "models" seem to have turned to "algorithms" around line 327.
Questions for authors
Beyond a nice theory, what does the paper give to the practitioners? What are the deep implications of the findings of the experimental section? If the paper is accepted, one extra page will be allowed; what would be shown and clearly exposed on that page?
Can you please clarify what is truly "yours" and "new" from what is "re-used" and "known" in sections 4 and 5? I believe most is "yours", but reinforcing the distinction throughout the paper might help better understand the contributions.
5 comments
Response to Reviewer vM9T (Part 2/2)
What will we add to the main text?
You are right that the limit of nine pages was a hard constraint for us. We will use the additional page to include short proof sketches for the theorems, a concrete example showing how the diagnostic leads to a model comparison as well as the table above with its implications for benchmark design. We will also summarize the new sensitivity results and explain how practitioners can examine both the selected model and the complete ranking.
We will upload the revised manuscript during the discussion period.
We hope these clarifications answer your questions. We sincerely thank you for your support and appreciation of our work!
Revisions
I thank the authors for their exhaustive answer, to all reviewers comments. I am willing to read the revised version to check the changes, although I would not encourage using the extra page for proof sketches for the theorems. It seems that most reviewers would be happy to see the practical applications and implications of this work, hence I would suggest developing those aspects in high priority.
Response to Reviewer vM9T (Part 1/2)
Dear Reviewer vM9T, we thank you for your positive and encouraging review. We are very happy that you find our theoretical formalization solid and the benchmark perfectly aligned with the theory. We agree with you that the distinction between existing results and our contributions as well as the implications of the tests for practitioners should be more emphasized in the main text.
What is new in Sections 4 and 5?
Our contributions are the formalization of multi-criteria, multi-task benchmarking as a preference aggregation problem in two steps, the application of the restricted preference domains to this setting, the diagnostic protocol and the empirical tests across choices of models, metrics and datasets. As far as we know, prior work on benchmarking has not investigated these restricted domains.
We formalize the aggregation within each dataset, where metrics act as voters and models as alternatives, and the subsequent aggregation of the resulting rankings across datasets. Arrow’s theorem and the commonality sharing rule described in Section 4 are existing results. Our contribution is their application within this framework.
We apply the three restricted preference domains to benchmarking and give them clear meaning in this setting. The results on single-peakedness and group separability are existing results from the social choice literature: Theorem 5.2 applies Black’s result [1], while Theorem 5.4 uses Inada’s results [2,3]. For Theorem 5.6, we provide a direct proof for metric rankings that disagree on at most one pairwise comparison.
We turn these conditions into a diagnostic protocol that can be tested directly on benchmark data and empirically investigate when the conditions hold.
The central question of our work is: do multi-criteria benchmarks naturally exhibit structure that makes meaningful aggregation possible? Our framework offers testable sufficient conditions, while our experiments investigate where this structure occurs in practice. We will reinforce this distinction more thoroughly in the revision.
What does the paper give to practitioners? What are the implications of the experiments?
Our framework gives a procedure for constructing an overall ranking of models across several metrics and datasets, together with a diagnostic for checking whether the conditions for meaningful aggregation are satisfied.
The main practical takeaway is the diagnostic: benchmark designers can test whether the chosen models and metrics satisfy sufficient conditions for a consistent majority ranking. Reviewer Merm also highlights the protocol among the paper’s “several practical takeaways”.
In our response to Reviewer 3AgR, we cover the benefits for model selection and stability to changes in the model set and report additional transitivity and sensitivity experiments.
The implications of our experiments are particularly clear when we summarize the results by the kinds of metrics being aggregated. Holding the number of metrics fixed at three, the following table lists the percentage of profiles on individual subjects satisfying each property, across the 83 model sets and 57 subjects:
We see that the choice of metrics matters substantially for determining whether the overall ranking is well-behaved. Combining one accuracy metric with efficiency metrics such as runtime and output length results in trade-offs, yet pairwise majority still induces a consistent ranking in 80.84% of the profiles on individual subjects.
The sensitivity experiments reported in our response to Reviewer 3AgR give a further implication. When all three metrics are recalculated on paired bootstrap samples of questions, the original winner model beats every other model in 87.1% of samples on average, while the entire ranking remains unchanged in 75.6%, averaged across the 3,825 profiles where the original majority ranking is transitive. Hence, practitioners can examine both whether the criteria induce a consistent ranking and how sensitive the selected model and the complete ranking are to changes in the evaluation questions.
Continued in Part 2/2.
[1] D. Black. On the rationale of group decision-making. Journal of Political Economy, 56(1):23–34, 1948.
[2] K. Inada. A note on the simple majority decision rule. Econometrica, 32(4):525–531, 1964.
[3] K. Inada. The simple majority decision rule. Econometrica, 37(3):490–506, 1969.