- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper ranks LLMs from pairwise human preference labels while using fewer labels. Each model pair is formulated as a hypothesis testing problem with two e-processes. The evaluation stops when one e-process reaches 1/, so the probability of returning the worse model is at most alpha. Prompts are sampled from a distribution that favors prompts with clear annotator preference, and an LLM predicts the preference probability of each prompt. The pairwise evaluator is used as a black-box comparison function within quicksort or merge sort, and the error budget alpha is allocated across pairs by a convex program. The paper provides bounds on the expected number of labels for pairwise evaluation and full ranking. Experiments on LMArena, PPE, and UltraFeedback show that ranking with adaptive alpha uses about 7-22% of the labels, while achieving ranking quality close to Bradley-Terry ranking fitted on all labels.
Critical strengths
-
The error guarantee does not depend on the predictor. The LLM predictor affects only the rate of evidence accumulation, while the guarantee on returning the worse model still holds (Lemma 3.1). Since the predictor is the least reliable component of the method, this property is important.
-
The proposed error allocation reduces the number of labels. On all three datasets, adaptive alpha uses up to about 40% fewer labels than constant alpha, with similar ranking quality. Active ranking and active testing use more labels under the same setting.
Critical weaknesses
-
My main concern is the application scenario. The paper gives no concrete deployment scenario and no online experiment. Chatbot Arena and the paper's experiments evaluated on a selected dataset are different. In practice, when would people run only a small part of a benchmark instead of the full benchmark? Lower cost is not a convincing reason. Take code testing as an example. We do not skip tests to save cost, because a missed test case may reveal a failure in some function of the code. Some methods reduce the amount of test data. However, they are proved in the code space, and the proofs do not carry over to LLM applications.
-
Suppose the assumption above holds, that is, pruning a benchmark is valid. Then modeling the LLM evaluation process with e-processes is insightful. However, as the authors state in lines 137-143, e-process modeling is not the main contribution of the paper. Related work such as SERPANT [1] modeled the process first. But the experimental baselines do not include SERPANT.
Questions for authors
See Weakness
4 comments
Thank you for taking the time to review our responses. Since the discussion period ends in a few days, we wanted to double-check whether your questions have been addressed. Please let us know if you need any further clarification.
Thank you for the clarification. I tend to keep my current rating.
Thank you for acknowledging our response. We hope the additional clarification about the problem statement and SERPANT experiments addressed your concerns.
Since we are entering the final day of the rebuttal, we wished to double check if you had any remaining concerns. If your concerns have been addressed, please consider reassessing the overall evaluation accordingly. If not, we are open to any suggestions to help improve the paper in the future.
We thank the reviewer for their thoughtful comments. Please find our responses below.
Our method is tailored towards pairwise human evaluation. In this setting, cheap verification such as unit tests are either unavailable or cannot tell responses apart: two code snippets may pass the same tests even though one is cleaner and handles more edge cases. Each time a new model is released, it must be compared against existing models, and every comparison requires fresh human judgments. Examples include internal model-release evaluations on fixed datasets and online arena-style leaderboards. In our work, we simulate the online setup by sequentially accessing the labels in the LMArena and PPE datasets, which are subsets from ChatBot Arena.
Why fewer human labels is acceptable unlike skipping unit tests? Software testing aims to find any failure, so a skipped test can hide a bug in a single trajectory and cannot be extended to quantify a model’s quality. Pairwise ranking instead estimates a population preference (the probability that humans prefer one model over another). Our e-process test stops once enough evidence has been collected (irrespective of the prompts used), and the probability of declaring the wrong winner is bounded by α.
SERPANT is not directly comparable to our method: it returns a partial ordering of models, not a full ranking. More importantly it has no stopping criterion. It keeps a set of unresolved model pairs and, in each round, labels randomly selected examples across all unresolved pairs, updating each pair's e-process.
For a fair comparison, we ran SERPANT with the same sample budget our method (QSA) needed to produce a full ranking, and report the fraction of model pairs it resolves:
Note that using the same budget that we used for our method, SERPANT resolves almost no pairs on LMArena and PPE, and only about a third on UltraFeedback. It spreads its labels uniformly over all O(n2) model pairs, so evidence builds up slowly on each one. Our method instead resolves the O(nlogn) comparisons the sorting algorithm requests, concentrates labels on one pair at a time, and selects informative prompts. So while our method can return a full ranking with this budget, Serpent cannot even produce a partial ranking to compare it with.