acceptodds
Under review as a conference paper at ICLR 2027

Sample Efficient Best-LLM Identification via Active Pairwise Preference Elicitation

Abstract

Practitioners deploying Large Language Models (LLMs) routinely face the following problem: among the many candidate LLMs available, which one performs best on my task? Public leaderboards rank models on generic benchmarks that may not reflect a particular use case, while building a custom evaluation benchmark in-house is often prohibitively expensive. In this paper, we study the problem of best-LLM identification when the practitioner has access to a small unlabeled pool of prompts drawn from their use case. We elicit from an annotator pairwise preferences between two candidate responses to the same prompt, and we ask how to select the next triple to label so that the best model is identified with as few labels as possible. We introduce TOPS (Top-focused Online Preference Sampling), an active acquisition strategy that maintains a Bradley-Terry posterior over model abilities and samples comparisons using acquisition scores that combine response dissimilarity with estimated reductions in uncertainty about the margins between the leader and its challengers. Across 17 settings, TOPS achieves the best average rank for cumulative regret, final regret, and the rank of the selected model. Using 500 queries, TOPS is able to identify the single best model from 20 candidates in 62% of runs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM selection, Pairwise preference elicitation, Sample-efficient evaluation, Active evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PXLLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to efficiently identify the best LLM among a pool of candidates. It uses strategic pairwise comparisons, focusing on models that could be the best and prioritizing prompts on which their responses differ.

Critical strengths

The paper is clearly written and has a well-defined objective. The proposed method is intuitive and easy to follow.

Critical weaknesses

My main concern is the novelty of the method. It builds largely on existing ideas, such as those in GURO (https://proceedings.neurips.cc/paper_files/paper/2024/hash/8443219a991f068c34d9491ad68ffa94-Abstract-Conference.html) and FewerBattles (https://openreview.net/pdf?id=XUVqFRp9oi). The changes make sense for this task, but they seem fairly incremental.

Questions for authors

I have no more questions for Authors. I do not think minor revisions would substantially change my assessment.

3 comments

Thank you for the thoughtful and constructive feedback!

W1: My main concern is the novelty of the method. It builds largely on existing ideas, such as those in GURO (https://proceedings.neurips.cc/paper_files/paper/2024/hash/8443219a991f068c34d9491ad68ffa94-Abstract-Conference.html) and FewerBattles (https://openreview.net/pdf?id=XUVqFRp9oi). The changes make sense for this task, but they seem fairly incremental.

Thank you for raising this question! First, we would like to note the novelty in the problem setup which TOPS tackles. TOPS focuses on identifying the single model to deploy, and jointly selects the two models and the prompt. GURO and FewerBattles both aim to learn an accurate ordering of all the items, and does not have a prompt axis. This distinction changes which comparisons are useful, since resolving the ordering of weaker models may contribute little to identifying the best model.

Second, we would like to note the novel components in the design of TOPS, specifically the active challenger set and scoring comparisons by their predicted reduction in uncertainty about leader-challenger margins. Removing the challenger set produces the largest degradation in our ablations, and these components are not present in GURO or FewerBattles. We also provide a theoretical characterization of TOPS, providing regret and stopping-time guarantees specifically for the top-focused pruning.

Finally, TOPS empirically outperforms both of these baselines in our experiments, identifying the winner with a budget of 500 queries 62.4% of the time, compared to 44.1% and 33.5% for GURO and FewerBattles respectively. We hope this clarifies the contributions of TOPS and its differences from GURO and FewerBattles.

Thank you for your response. I will maintain my rating at 3.

Thank you again for your time and effort in reviewing our paper and for your constructive feedback!

Reviewer_LkvgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes TOPS, which maintains a Laplace-approximated Bradley–Terry posterior, retains plausible challengers to the current leader, and samples model-pair–prompt comparisons using response dissimilarity weighted by predicted variance reduction in leader–challenger margins. Across 17 evaluation settings, TOPS achieves the best average rank on final regret, cumulative regret, and selected-model rank, identifying the best model among 20 candidates in 62% of runs with 500 queries.

Critical strengths

This work directly targets uncertainty relevant to best-model identification, providing a principled way to concentrate annotations on comparisons that can change the deployment decision.

Critical weaknesses
  1. The experiments use 20 candidate models, but the candidate set may span a substantial performance range, as illustrated by the inclusion of both gpt-5.4-mini and Qwen/Qwen2.5-7B-Instruct. If many candidates are clearly weaker than the leading models, identifying the best model may already require relatively few comparisons, potentially overstating the method’s effectiveness when selecting among closely matched candidates. Consequently, the current results do not fully substantiate the paper’s motivation that model selection is prohibitively expensive, and their relevance to choosing among near-optimal models in practical deployment remains limited.
Questions for authors

N/A

2 comments

Thank you for the thoughtful and constructive feedback!

W1: The experiments use 20 candidate models, but the candidate set may span a substantial performance range, as illustrated by the inclusion of both gpt-5.4-mini and Qwen/Qwen2.5-7B-Instruct. If many candidates are clearly weaker than the leading models, identifying the best model may already require relatively few comparisons, potentially overstating the method’s effectiveness when selecting among closely matched candidates. Consequently, the current results do not fully substantiate the paper’s motivation that model selection is prohibitively expensive, and their relevance to choosing among near-optimal models in practical deployment remains limited.

Thank you for raising this question! First, we would like to note that in Appendix F.5, we did conduct an empirical analysis of varying the number of models, where we retain only the top 5 or top 10 models under each setting's full-pool Bradley-Terry ranking. This more closely mimics the challenging scenario where there are many near-optimal models. We find that TOPS has the best performance on all three metrics with K=10K=10 finalists. Its advantage does degrade at K=5K=5, but here, the problem is small enough that all methods perform roughly the same, and even random selection is competitive.

Second, while we agree that there may be weaker models among the 20 candidates like Qwen-2.5-7B, we note that the leading models do contain several closely competitive candidates. For example, examining the gap between the top 2 models in the 17 settings in the main paper, we find that the median BT gap is 0.168 logits, corresponding to a fitted BT win probability of 54.2% for the winner against the runner-up. Across the 17 settings, there are 6 unique model winners. Thus, the result that TOPS identifies the top model in 62.4% of runs (compared with at most 45.9% for any baseline) is non-trivial.

Dear Reviewer Lkvg,

Thank you again for your time and valuable feedback. Since there is roughly one day left in the rebuttal period, we were wondering if our response has adequately addressed your concerns. If so, we would be grateful if you could update your evaluation accordingly. If there are any remaining questions or comments, we would be happy to discuss.

Thank you!

Reviewer_wSWHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to identify the best LLM for a specific use case with a limited budget of pairwise preference annotations. It proposes TOPS, which selects model pairs and prompts by combining uncertainty about the current leader and its challengers with response dissimilarity. The authors provide conditional theoretical guarantees and evaluate the method across 17 settings, mostly using AI judges. With 500 queries, TOPS identifies the best model in 62% of runs, compared with at most 46% for the baselines.

Critical strengths

The paper addresses a practical model-selection problem. Focusing on finding the best model for a specific use case, rather than recovering a complete ranking, makes efficient use of a limited annotation budget.

Moreover, TOPS achieves the best average rank across 17 settings and identifies the winner in 62% of runs with 500 queries, compared with at most 46% for the baselines.

Critical weaknesses

The prompt-selection heuristic may bias evaluation. TOPS favors prompts where responses have high embedding dissimilarity. However, different wording does not necessarily mean different quality, and similar answers can differ in a crucial factual detail.

Moreover, emphasizing certain prompts could change the apparent winner relative to the full prompt pool.

Also, the theoretical guarantee does not directly match the experimental target. The theory concerns the winner under latent model strengths, whereas experiments evaluate the winner obtained by fitting BT to all pool judgments. The authors acknowledge this and provide a sufficient condition for agreement. Still, the guarantees are conditional on that connection, which limits how directly they support the practical identification claim

Questions for authors

Regarding the cost accounting, the main setup assumes cached responses from every candidate model on every prompt. Annotation savings therefore do not automatically imply lower total evaluation cost, particularly for expensive generators. Could you provide numerical evidence from your experiments showing whether the statement above holds?

3 comments

Thank you for the thoughtful and constructive feedback!

W1: The prompt-selection heuristic may bias evaluation. TOPS favors prompts where responses have high embedding dissimilarity. However, different wording does not necessarily mean different quality, and similar answers can differ in a crucial factual detail.

We use embedding dissimilarity as a heuristic for semantic difference between two model outputs, in order to prioritize potentially informative comparisons (i.e. comparisons where the semantic meaning is very different). However, we do not use it solely as a measure of response quality differences as the reviewer suggests. Weighting comparisons by quality differences alone would favor clearly separated responses, but the goal of best-model identification is to distinguish between closely matched candidates, so larger quality differences may not correspond to more informative comparisons.

While we agree that embeddings may miss subtle but consequential differences in meaning, they are commonly used to measure semantic similarity in prior work [1, 2]. Further, we note that our acquisition score is weighted by the dissimilarity and we sample proportionally to that score. We also use uniform exploration with probability 0.05. Thus, comparisons with low dissimilarity can still be queried, and no sample is fully excluded only by dissimilarity (unless the two texts are identical and the dissimilarity is zero).

W2: Moreover, emphasizing certain prompts could change the apparent winner relative to the full prompt pool.

Thank you for raising this question! First, we would like to note that the ground truth for our experiments is the BT oracle fitted to the entire pool, which is the intended target, and so any bias in the estimation method would only harm the results for that particular method.

Second, we actually experimented with an Inverse Probability Weighted (IPW) estimator for TOPS initially, but found its performance to be much worse than the regular BT estimator. Specifically, each label receives the weight ωt=1Mtπt(qt)\omega_t = \frac{1}{M_t\pi_t(q_t)}

where πt(qt)\pi_t(q_t) is the probability of the triple being drawn, and MtM_t is the number of unused triples before the draw. We then fit a weighted BT MAP estimate. Note that this will change both the estimator and the subsequent queries (which seek to minimize variance of this estimator).

The following table reports results over the 17 main settings with 10 seeds and 500 queries. For final regret (FR), cumulative regret (CR), and selected model rank (Rank), we report each method's mean rank against the seven baselines, with one standard error of the mean across settings. Lower is better. The last column gives the percentage of runs where the winner is identified.

MethodFR Rank Avg.CR Rank Avg.Rank Rank Avg.Runs selecting the winner
TOPS1.88 ± 0.301.18 ± 0.102.03 ± 0.3862.4%
TOPS + IPW3.94 ± 0.514.29 ± 0.424.06 ± 0.4942.9%

We can observe that IPW results in significantly worse performance, likely due to it having a higher variance than base TOPS, despite being unbiased. Further, we note that as the acquisition process is likely to only be run once, minimizing overall error is likely more important than having an unbiased estimator. We have added this analysis to Appendix F.9 of the revision.

W3: Also, the theoretical guarantee does not directly match the experimental target. The theory concerns the winner under latent model strengths, whereas experiments evaluate the winner obtained by fitting BT to all pool judgments. The authors acknowledge this and provide a sufficient condition for agreement. Still, the guarantees are conditional on that connection, which limits how directly they support the practical identification claim

We agree that the theoretical guarantees are for a slightly different target than the experimental setup, but we believe the sufficient condition provided in Corollary D.0.1 is sufficient to connect the two. Our main purpose with the theoretical analysis was to provide intuition for how challenger-set pruning and proportional sampling support identification under the stated assumption, but it does not directly guarantee performance under the exact experimental protocol.

Q1: Regarding the cost accounting, the main setup assumes cached responses from every candidate model on every prompt. Annotation savings therefore do not automatically imply lower total evaluation cost, particularly for expensive generators. Could you provide numerical evidence from your experiments showing whether the statement above holds?

Thank you for the suggestion! For inference, 4 of our 20 models were API-based: gpt-4.1-mini, gpt-5.4-mini, gpt-5.4-nano, and Kimi-K2.5. Generating responses from these models across six datasets, with 5,329 prompts per model, cost a total of \77.41.Thiscorrespondstoanaverageofapproximately77.41. This corresponds to an average of approximately \\0.0036 per model per prompt. We ran the remaining 16 models on our local server, and so cannot directly estimate their inference cost. Using the average API cost as a rough proxy gives an estimated cost of \$36 to generate responses from all 20 models on a pool of N=500 prompts, though this extrapolation assumes that the API average is representative of the locally hosted models, which is not the case since the locally hosted models are much smaller and would thus be expected to be cheaper.

For human annotations, AlpacaFarm [3] reports approximately \0.30perhumanpreference,whichwouldyieldatotalof0.30 per human preference, which would yield a total of \\150 for a budget of T=500T=500 annotations. Thus, we see that for a typical study, human annotations do account for the majority of the cost.

Further, we would like to note that in Appendix F.8 of our initial submission, we did include an analysis of TOPS when the prompt cache is partially complete (and the dissimilarity tensor is completed via matrix completion). We find that only querying 4 models (of 20) per prompt does not degrade performance in 3/5 AlpacaEval subsets tested, suggesting that a complete response cache may not be necessary in some instances.

[1] Label-Efficient Model Selection for Text Generation. ACL 2024.

[2] SEMSCORE: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity. arXiv:2401.17072.

[3] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. NeurIPS 2023.

Dear Reviewer wSWH,

Thank you again for your time and valuable feedback. Since there is roughly one day left in the rebuttal period, we were wondering if our response has adequately addressed your concerns. If so, we would be grateful if you could update your evaluation accordingly. If there are any remaining questions or comments, we would be happy to discuss.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.