- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Convergent Elo (CElo), an online pairwise evaluation method that replaces Elo's fixed learning rate with participant-specific diminishing learning rates based on individual comparison counts. The authors establish almost-sure convergence of both relative ratings and the full rating vector under stationary outcomes and fixed, potentially nonuniform matchmaking.
Critical strengths
A major strength of this paper is its theoretical analysis of the convergence properties of Elo with participant-specific diminishing learning rates. The authors establish almost-sure convergence of both relative ratings and the full rating vector under nonuniform matchmaking, without requiring Bradley–Terry realizability or global recentering.
Critical weaknesses
-
The experiments on real LLM evaluation rely entirely on historical Arena Text (2025) and Arena Vision (2024) comparison records, which raises concerns about their relevance to more recent LLMs and evaluation settings. More importantly, CElo achieves only marginal improvements over standard Elo, with BCE reductions of just 0.000616 and 0.003378 on the two datasets, respectively. Furthermore, the paper does not provide a direct comparison with a properly tuned global diminishing-gain Elo baseline on real Arena data, despite emphasizing participant-local learning rates as a key design choice. These limitations make it difficult to assess the practical significance of the proposed method.
-
The paper presents participant-local diminishing K-factors as a central contribution, but a closely related approach has already been explored in Elo-based Learner Modeling for the Adaptive Practice of Facts (Pelánek et al., 2017, User Modeling and User-Adapted Interaction), which is not cited or discussed. In particular, the prior work uses a participant-specific learning rate of the form , which is mathematically equivalent to CElo's proposed schedule when (\alpha=1), up to reparameterization. This substantial overlap raises concerns about the algorithmic novelty of CElo.
Questions for authors
Could the authors clarify the algorithmic differences between CElo and Pelánek et al. (2017)? In particular, the participant-specific diminishing learning rate proposed in the prior work appears mathematically equivalent to CElo's update schedule when (\alpha=1). What are the key algorithmic contributions of CElo beyond this existing approach?
Thank you for the review, especially for pointing us to Pelánek et al. (2017) and identifying the missing comparison with a tuned global-clock baseline on Arena. We have clarified the overlap with prior work and added the direct comparison.
What is new relative to Pelánek et al. (2017)? We agree that their gain U(n)=A/(1+Bn) is equivalent, up to reparameterisation, to the CElo gain at α=1. We now credit this overlap in Related Work (Section 1.3) and alongside the update rule (Section 2). We do not claim novelty for inverse-count decay or participant-local counters. Pelánek et al. use the schedule to estimate student ability and item difficulty in an online system for learning geography. They show empirically that the estimates settle and agree closely with joint maximum likelihood estimates.
Their work does not prove almost-sure convergence of the unprojected rating vector, including its common offset, or characterise the limiting relative ratings under Bradley–Terry misspecification. Theorem 1 and Proposition 1 establish these results under fixed i.i.d. matchmaking with connected support and stationary conditional outcome means, without assuming bounded iterates or requiring projection or recentring. Under misspecification, the limiting relative ratings minimise the matchmaking-weighted KL discrepancy. The revision also derives asymptotic confidence intervals for rating differences (Theorem 3) and a fixed-sample confidence set for the population target (Proposition 2, Appendix C.6). These results, rather than the inverse-count update itself, are the paper's primary contributions.
The algorithmic extension is to allow a tunable exponent 1/2<α≤1 rather than fix inverse-count decay. This specification has meaningful theoretical implications. For instance, this range gives the Robbins-Monro-type nonsummable but square-summable gains used in our convergence proof. Under Theorem 3's conditions, the centred error is OP(n−α/2), and only the α=1 case requires an additional spectral condition. For 1/2<α<1 under Bernoulli outcomes and Bradley–Terry realisability, the pairwise interval uses only the two participants' current ratings and gains. The choice of exponent also matters empirically: among unaveraged CElo configurations in Fig. 1, α=0.90 has the lowest median error at the validation horizon, whereas α=1 has the lowest at 108 comparisons.
Direct comparison with tuned global-clock Elo and practical gains. We now tune global-clock Elo independently on each practical chronological dataset, expanding the search domains where validation approached a boundary (Appendix E.6). The paired moving-block bootstrap compares per-comparison loss differences on the same held-out observations while resampling contiguous blocks. The 95% intervals favour CElo over both tuned global-clock Elo and fixed-gain Elo on Arena Text, Arena Vision, and BASIL across all tested block lengths, with no clear difference on TCEC (Tables 6–7 and Appendix E.6). Though CElo's absolute raw BCE gains over fixed-gain Elo are modest, they are comparable in magnitude to TrueSkill's on both Arena streams. The results show that the convergent update remains competitive for prediction on the evaluated streams; they do not establish definitive superiority of one method over another.
Choice of Arena releases.
arena-human-preference-140kandVisionArena-Battleare the most recent comparable large-scale public Arena releases we identified that provide timestamped individual human votes. Newer public leaderboard snapshots report aggregates and cannot be used to replay the rating updates. Our question is how rating rules learn from the same sequence of preferences, not how the latest models rank. Model identities serve only to index the ratings, so the age of the models does not invalidate the comparison. What may affect the result is a change in the comparison process itself, such as more uneven participation or a different degree of Bradley–Terry misspecification. We therefore do not assume the observed gains carry over to newer streams.Controlled estimation experiments. In revised Fig. 2, we independently tune the initial gain, clock offset, and decay exponent for both clocks with equal validation budgets. CElo reduces mean final target error by 6.3-39.9% across the four heterogeneous regimes, while the methods are nearly tied under homogeneous participation. In Fig. 1, we tune configurations independently on shared validation streams and add time-averaged Elo and time-averaged CElo. All five unaveraged CElo configurations finish with lower median target error than both fixed-gain and time-averaged Elo, and time-averaged CElo finishes with the lowest median error. These simulations measure estimation error against a known stationary target, complementing the held-out predictive comparisons.