- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies how to cost-effectively evaluate the pairwise win rate between two LLM generators with provable accuracy guarantees.
Critical strengths
This work leverages the agreement from coupled generation to construct cross-comparisons, enabling unbiased estimation of the win rate for independent outputs while simultaneously cutting both generation and judgment costs.
Critical weaknesses
The zero-judgment shortcut in paper requires the two fully decoded responses to be identical, including whitespace. For long texts, high temperatures, and open-ended generation, the probability that two complete responses are exactly the same can be very low.
Questions for authors
reference Weaknesses
2 comments
Thank you for your response. After considering it, I have decided to maintain my original score.
Dear Reviewer hZve,
Thank you for your careful reading and for raising the role of exact matches in longer and more open-ended responses. We have clarified the mechanisms and organized the supporting evidence as follows.
Next, we respond to your comment.
Exact matches enable the double-match cancellation shortcut. The crossed estimator also reduces the sampling requirement through score concentration. Section 2.3 now explains these effects using Eq. (6), which separates how often a block contains a mismatch from how closely its score concentrates around half credit. Theorem 2 connects this variance to the expected stopping cost. Crossing independent coupled pairs preserves the target win probability whether or not their responses match (pp. 3–4).
The narrative task illustrates the distinction. Under the Qwen2.5-7B-Instruct judge, double matches determine 1.94% of block scores without judging, while crossed-score variance is 0.0504 compared with 0.0726 for Independent 2. Both methods use four responses per block. Holding scores and stopping times fixed, cancellation saves 0.6% of distinct judgments. This separates the judgments skipped within a block from the savings due to collecting fewer blocks. Section 4.4 and Table 4 report this analysis (pp. 8–9).
For CommonGen prompts requesting 80–100-word narratives, Crossed 2 uses 29.5% fewer generated responses and 31.2% fewer distinct judgments. For E2E at temperatures 0.70 and 1.00, the reductions are 12.5% and 19.2%. These reductions compare with the lowest independent mean for each resource at error tolerance 0.02 and 95% confidence. Section 4.3 and Table 3 report the results (pp. 7–8).
On WildBench with Mistral Small 3 24B, Crossed 2 saves both resources under Qwen, Haiku, Sonnet, and GPT, with reductions of 1.9–12.6% in responses and 14.8–20.3% in distinct judgments at the same precision. Double matches determine 2.08% of block scores. With cancellation disabled, Crossed 2 still requests 12.0–18.9% fewer comparisons than the lowest independent count. Table 5 reports distinct judgments after caching, and Table 9 reports these requests before caching. The results appear in Section 4.5 and Appendix F.2 (pp. 9 and 21–22).