- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to identify Pareto-optimal large language models using pairwise human preference comparisons across multiple objectives. The authors propose Vector-Borda Empirical-Gap Elimination (VB-EGE), an adaptive sampling method that estimates a Borda score for each model and objective and gradually removes models whose Pareto status can be determined. The paper shows that the Borda scores preserve the Pareto frontier under the Bradley–Terry model. It also provides theoretical guarantees on the correctness and sample complexity of the proposed method. Experiments on synthetic problems and three Arena datasets show that VB-EGE requires fewer comparisons than the uniform sampling baselines.
Critical strengths
- The paper addresses a useful problem in LLM evaluation. Instead of ranking models using a single score, it identifies the Pareto frontier across different abilities, allowing models with different strengths to be considered.
- The paper provides theoretical guarantees for the proposed method. The authors prove that vector-Borda scores preserve the Pareto frontier and establish bounds on the sample complexity of the algorithm.
Critical weaknesses
- The paper aims to reduce the cost of LLM evaluation, but it does not compare the proposed approach with direct benchmark evaluation. For tasks such as mathematics and coding, model performance can be measured directly without repeated pairwise comparisons. It is unclear whether the proposed method is more cost-effective in these cases, or when pairwise comparisons would be preferable.
- The adaptive sampling strategy is limited. The method samples every active model on every objective and selects opponents uniformly at random. It does not adaptively select the most informative model pairs or objectives, which may require unnecessary comparisons.
- The method relies on uniform opponent sampling, even for models that have already been eliminated. Although this keeps the Borda scores consistent, it is unclear how much this affects sampling efficiency.
- The paper uses strict Pareto dominance rather than standard Pareto dominance. It is unclear how this choice affects the identified Pareto frontier, particularly when models have similar performance on some objectives.
- All three main baselines use uniform sampling. The paper would benefit from comparisons with other adaptive sampling methods to better demonstrate the advantages of VB-EGE.
- The real-world experiments sample from existing Arena comparison records rather than collect new human preferences. It is unclear whether the reported improvements would also hold in an online evaluation setting.
Questions for authors
- For tasks such as mathematics and coding, model performance can be measured directly using existing benchmarks. How does the evaluation cost of the proposed method compare with direct benchmark evaluation? In what situations would repeated pairwise comparisons be more cost-effective?
- The method samples every active model on every objective and selects opponents randomly. Why not adaptively select more informative model pairs and objectives? Could this reduce the number of comparisons?
- The paper uses strict Pareto dominance rather than standard Pareto dominance. Have the authors considered epsilon-dominance to handle models with similar performance? How would different dominance definitions affect the identified Pareto frontier?
- What is new in VB-EGE compared with existing Empirical-Gap Elimination methods, apart from using Borda scores?
- Why are the main experiments limited to uniform sampling baselines? How does VB-EGE compare with other adaptive sampling methods?
- The real-world experiments only retain model subsets where every pair has comparison records for every objective. How would the method perform with sparse or incomplete comparison data, which are common in real Arena settings?
3 comments
Q5: Comparisons beyond uniform sampling.
- We would like to clarify that Appendix E already includes an adaptive BT comparison: our heuristic extension of the Track-and-Stop method in Goldberger and Rudi (2026), Optimal Top- Identification from Pairwise Comparisons (https://arxiv.org/abs/2607.08979), to the Pareto frontier identification case. Our extension fits BT scores across models for each objective and adaptively selects both model pairs and objectives. VB-EGE achieves a lower capped mean comparison count in 27 of the 28 tested settings; please see Appendix E for details. We have not found other directly applicable adaptive baselines in the literature and would welcome references to relevant methods.
Q6: Sparse or incomplete comparison logs.
- Thank you for the question. Our current method requires a complete query graph for each objective, and the replay experiments accordingly retain model subsets with complete pair coverage. We note that full pair-query access is also assumed by prior works such as Jamieson et al. (2015) (https://arxiv.org/abs/1502.00133) and Goldberger and Rudi (2026) (https://arxiv.org/abs/2607.08979). Exploiting sparse historical graphs would require graph-aware estimation and sampling. Our present contribution focuses on establishing fixed-confidence Pareto identification under full pair-query access, but this is a natural and interesting direction for future work.
Weaknesses
W1: Missing comparison with direct benchmark evaluation.
- Answer to Q1 explains the complementary roles of direct grading and preference-based evaluation. Our efficiency claim concerns identifying a frontier defined by human preferences; a benchmark cost comparison requires a common evaluation objective.
W2: Limited adaptivity over pairs and objectives.
- VB-EGE already concentrates focal sampling on unresolved models. Finer allocation may offer further gains; answer to Q2 explains the design and its worst-case guarantee, and answer to Q5 points to the existing adaptive BT comparison.
W3: Sampling opponents that have already been eliminated.
- Eliminated models stop receiving focal samples but remain possible opponents. This keeps each active model's Borda mean fixed, allowing it to reuse observations from earlier phases. Within the same phase, sampling only active focal models requires fewer comparisons than continuing to sample all models. Focal Borda uses the same opponent distribution and provides a controlled comparison of the benefit from elimination.
W4: Strict rather than standard Pareto dominance.
- Answer to Q3 explains why strict dominance matches our intended screening criterion. Standard dominance and -approximation require separate treatment and are left as future extensions.
W5: Uniform-sampling baselines.
- Appendix E already compares against an adaptive BT heuristic, as detailed in answer to Q5. The Focal Borda baseline separately measures the benefit of our adaptive elimination procedure under the same Borda representation and confidence schedule.
W6: Historical logs rather than fresh human feedback.
- Table 11 shows substantial savings under matched replay conditions: Focal Borda uses 3.96–16.54 times as many comparisons as VB-EGE. Practical evaluation can build on existing feedback, including the over 82 million human votes accumulated by Arena (https://arena.ai/blog/arena-100m-revenue). Online experiments require fresh human judgments, additional funding, and time; we leave this validation to future work.
We hope our responses address your concerns and encourage a more favorable assessment of our paper.
Thank you for the detailed responses and clarifications. I appreciate the authors' efforts. However, some concerns remain:
Sampling efficiency (W2/W3): The method uses uniform opponent sampling, including previously eliminated models. Although this preserves the Borda score definition, its impact on sampling efficiency remains unclear.
Adaptive baselines (W5): I appreciate the comparison in Appendix E. However, the Pareto-specific adaptive BT heuristic uses an approximate stopping rule without a proven correctness guarantee, making the comparison less conclusive. The authors do not sufficiently explain its poor performance, making the comparison less conclusive.
Real-world evaluation (W6): The experiments rely on repeated sampling from existing Arena records. It remains unclear whether the reported efficiency gains would hold when collecting fresh human preferences in an online setting.
Response to Reviewer 254S
We thank the reviewer for the questions on evaluation scope, adaptivity, and dominance definitions, and address each point below.
Questions
Q1: Direct benchmarks and pairwise evaluation.
Q2: Adaptive selection of pairs and objectives.
Q3: Strict dominance, standard dominance, and ϵ-approximation.
Thank you for the suggestion. In our setting, strict dominance matches our conservative screening objective: a model is excluded only when another improves every evaluated capability. If a user values a capability on which models are tied, price or speed may guide the choice; standard dominance can discard such an option. Also, choosing ϵ is difficult in practice because a difference in Borda scores does not directly tell users how much model performance they would be willing to sacrifice. We therefore retain exact strict identification as the primary target.
Handling standard dominance at coordinate ties would require additional assumptions, while ϵ-approximation requires a specified tolerance and revised stopping and correctness proofs. We leave these extensions to future work, as they are not necessary for the Pareto frontier identification problem studied here. Related approximate formulations are considered by Auer et al. (2016) (https://proceedings.mlr.press/v51/auer16.html) and Koné et al. (2023) (https://arxiv.org/abs/2307.00424).
Q4: Novelty beyond existing EGE methods.