- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper aims to efficiently identify the best LLM among a pool of candidates. It uses strategic pairwise comparisons, focusing on models that could be the best and prioritizing prompts on which their responses differ.
Critical strengths
The paper is clearly written and has a well-defined objective. The proposed method is intuitive and easy to follow.
Critical weaknesses
My main concern is the novelty of the method. It builds largely on existing ideas, such as those in GURO (https://proceedings.neurips.cc/paper_files/paper/2024/hash/8443219a991f068c34d9491ad68ffa94-Abstract-Conference.html) and FewerBattles (https://openreview.net/pdf?id=XUVqFRp9oi). The changes make sense for this task, but they seem fairly incremental.
Questions for authors
I have no more questions for Authors. I do not think minor revisions would substantially change my assessment.
3 comments
Thank you for your response. I will maintain my rating at 3.
Thank you again for your time and effort in reviewing our paper and for your constructive feedback!
Thank you for the thoughtful and constructive feedback!
Thank you for raising this question! First, we would like to note the novelty in the problem setup which TOPS tackles. TOPS focuses on identifying the single model to deploy, and jointly selects the two models and the prompt. GURO and FewerBattles both aim to learn an accurate ordering of all the items, and does not have a prompt axis. This distinction changes which comparisons are useful, since resolving the ordering of weaker models may contribute little to identifying the best model.
Second, we would like to note the novel components in the design of TOPS, specifically the active challenger set and scoring comparisons by their predicted reduction in uncertainty about leader-challenger margins. Removing the challenger set produces the largest degradation in our ablations, and these components are not present in GURO or FewerBattles. We also provide a theoretical characterization of TOPS, providing regret and stopping-time guarantees specifically for the top-focused pruning.
Finally, TOPS empirically outperforms both of these baselines in our experiments, identifying the winner with a budget of 500 queries 62.4% of the time, compared to 44.1% and 33.5% for GURO and FewerBattles respectively. We hope this clarifies the contributions of TOPS and its differences from GURO and FewerBattles.