Ranking Large Language Models via Comparison Graph Truncation

Large language model (LLM) evaluation is a central problem in modern machine learning. While benchmark scores provide a standard way to compare models on many tasks, evaluating open-ended responses remains challenging because of the lack of a single correct answer. Indeed, pairwise comparison offers a practical approach, but noisy judgments can make the resulting rankings unstable. To address this issue, we repeat each comparison and apply truncation to retain only comparisons with sufficient agreement. Specifically, our method removes comparisons with weak agreement while retaining enough evidence to rank the models. Theoretically, we prove that moderate truncation can reduce ranking error, while removing too many comparisons can increase it. Additionally, we conduct simulations and experiments with real LLMs, showing that our method improves ranking accuracy.