acceptodds
Under review as a conference paper at ICLR 2027

A Dynamic Bradley-Terry Approach to Learning Time-Varying Leaderboards from Sparse Comparisons in Sports and LLM Evaluation

Abstract

State-of-the-art rankings of large language models, such as those of Arena.AI, are summarized in leaderboards that are continually updated over time based on a steady stream of head-to-head comparisons between models. Such time-varying leaderboards must learn from sparse and uneven data while accommodating temporary model absences. This challenge also arises in international sports, where, unlike in league competitions, teams play sporadically and with highly imbalanced schedules in mostly bilateral series. We develop a dynamic Bradley–Terry (BT) estimator for jointly estimating smooth, item-specific ability trajectories and time-varying contextual effects using a temporal basis representation and penalized logistic likelihood. By pooling information across time, the method can recover latent abilities during temporary gaps in an item’s comparison history when the observations on either side provide sufficient information to identify the chosen temporal representation. We establish recovery guarantees and estimation rates that depend on the ability-curves’ smoothness and the conditioning of the joint time–comparison design, thereby accounting for uneven comparison information across items and time. Extensive simulations examine performance across diverse temporal patterns, imbalanced comparison schedules, and temporary item absences, and demonstrate the benefits of temporal information pooling. Applications to LLM evaluations from Arena.AI (previously Chatbot Arena) and international T20 cricket further illustrate the methodology, yielding lower held-out errors than static and kernel BT methods under constructed interpolation holdouts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Bradley–Terry model, Dynamic paired comparisons, Basis representation, Penalized likelihood, Varying-coefficient model, Sparse comparison data, Dynamic ranking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ub7VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers the BT model with time-varying coefficients from sparse pairwise comparisons. The parameters are estimated by basis expansions and a penalized Bradley–Terry likelihood. Theoretical results on identification and convergence rates are presented. The numerical studies illustrate the effectiveness of the proposed method.

Critical strengths
  1. Ranking from sparse, time-varying comparisons is an important and practical problem.

  2. The paper provides a systematic theoretical analysis, including model identifiability, parameter estimation error, and prediction error.

Critical weaknesses
  1. The methodological novelty is limited. The proposed estimator combines existing basis expansion and penalized likelihood methods. Its advantages over existing dynamic ranking methods need further justification.

  2. The numerical comparisons include only static and kernel-based baselines, omitting existing dynamic BT models. Furthermore, the real-data experiments show that the method can reconstruct past rankings, but do not consider estimating current rankings using only historical data.

Questions for authors
  1. What is the main technical contribution beyond existing dynamic Bradley–Terry models and penalized spline estimation theory?

  2. How does the method compare with an existing dynamic BT model under the same evaluation protocol?

  3. How does performance change when only past observations are available?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ub7V (part 1 of 4)

We thank the reviewer for a careful review; addressing these comments has sharpened the paper's claims and strengthened its evidence. The revised manuscript marks every change in blue, and each response below ends with the location of the corresponding revision.

Comments ub7V.4 and ub7V.6: What is new beyond existing dynamic BT models and spline theory.

Our main contributions beyond existing dynamic BT models and penalized spline theory are the following.

  • Identification from the joint design. We show that the ability trajectories are identified if and only if the joint time–comparison Gram matrix is positive definite. This allows identification when the comparison graph is sparse or disconnected at individual times, a setting in which local methods (Bong et al., 2020; Karlé and Tyagi, 2023; Tian et al., 2024) have few or no comparisons within a smoothing window.
  • Recovery versus prediction. We introduce a design constant κn\kappa_n that separates trajectory recovery from prediction on the observed schedule. This explains why prediction can be consistent while recovery during a gap requires additional design conditions.
  • Time-varying contextual effects. We extend the model to item-specific covariate effects that vary over time. This setting is not covered by kernel BT, and in our simulations it gives some of the largest gains over KickScore (Table 1, Comment ub7V.7).

What the theory establishes.

We build on penalized likelihood and spline smoothing (O'Sullivan et al., 1986; Wood, 2017; Huang and Su, 2021). We study how comparison times and item pairings jointly determine identification and estimation error. For centered ability curves, we write A(h)\mathcal A(h) for integrated squared ability error and Pn(h)\mathcal P_n(h) for the corresponding error in pairwise contrasts, weighted by the comparison design. We define

\frac{\mathcal P_n(h)}{\mathcal A(h)}$$ to relate these two errors on the fitted trajectory space $\mathcal B_n$. We show that identification is equivalent to positive definiteness of the corresponding joint-design Gram matrix. This explains when we can combine information across time to estimate trajectories despite gaps in comparisons at individual times. Under the manuscript's i.i.d. random-design assumptions, fixed item count, deterministic basis and penalty sequences, bounded approximant contrasts, finite leverage, and localization condition $\ell_n^2w_n\to0$, we obtain $$\begin{aligned}\mathcal P_n(\widehat\beta-\beta^0)&=O_p(w_n),\\ \mathcal A(\widehat\beta-\beta^0)&=O_p(a_n^2+w_n/\kappa_n),\\ \mathcal E_n&=O_p(w_n),\end{aligned}$$ where $\mathcal E_n$ is same-design excess log loss, $a_n^2$ is trajectory approximation error, and $w_n=\widetilde a_n^2+d_n/n+\rho_nJ_{M_n}(\beta_n^*)$, with $d_n=(J-1)M_n$. These bounds show that the trajectory-error bound increases as $\kappa_n$ decreases, while the prediction bound depends on $w_n$. For example, for twice-differentiable abilities with $\kappa_n\gtrsim M_n^{-1}$, the manuscript's polynomial-design corollary gives a trajectory squared-error bound of $O_p(n^{-3/5})$ and an excess log-loss bound of $O_p(n^{-4/5})$. In Appendix B.7, we give examples of different trajectories inside a gap that produce the same probabilities for the observed comparisons. **Manuscript revisions.** The manuscript's Introduction now states these contributions, namely identification from the joint design, the design constant separating recovery from prediction, and time-varying covariate effects, and contrasts dBT with local-window methods and with penalized spline regression. **Comments ub7V.5 and ub7V.7: dBT is more accurate than KickScore, an existing dynamic BT model, on the same data.** We compared static BT, dBT, kernel BT and KickScore (Maystre et al., 2019) on the same 500 saved datasets in each of the 17 original simulation scenarios and on the same 15 Arena and 75 T20 holdouts, with identical observations, holdouts and recovery targets. We selected KickScore's covariance structure and parameters among 128 configurations by maximizing the approximate marginal likelihood on each training set, following the random-search design of Maystre et al. (2019): constant or affine components combined with Wiener, exponential or Matérn processes, with parameter ranges set for the $[0,1]$ time scale and fixed before any fitting. In contextual scenarios we fitted independent Gaussian processes for each item's baseline and covariate coefficients using the same covariates as dBT.
AuthorsOpenReview ↗

Response to Reviewer ub7V (part 2 of 4)

Simulations.

In Table 1, we report probability MSEs with MCSEs. These results show that dBT has lower mean probability MSE than KickScore in 16 of 17 scenarios, with a median reduction of 32%, and lower error than KickScore in most individual datasets in 16 scenarios. The gains are largest for crossing curves and for contextual effects that vary over time; in the null-effect scenario, KickScore and kernel BT, which omits the covariate, have lower error.

Table 1. Simulation scenarios: probability MSE, mean (MCSE), R=500R=500. Entries are in units of 10−310^{-3}.

ScenarioStatic BTdBTKernel BTKickScore
Complete exposure16.126 (0.032)0.860 (0.013)1.014 (0.014)1.226 (0.015)
10% absences19.172 (0.050)1.119 (0.022)1.959 (0.030)1.624 (0.027)
25% absences21.971 (0.050)2.032 (0.043)5.686 (0.061)3.616 (0.049)
40% absences26.538 (0.063)6.790 (0.396)12.014 (0.097)9.173 (0.082)
90% weight reduction20.732 (0.049)1.522 (0.032)3.122 (0.046)2.272 (0.040)
98% weight reduction21.696 (0.050)1.920 (0.042)5.054 (0.075)3.122 (0.050)
20 comparisons/time22.529 (0.118)7.769 (0.151)10.031 (0.133)8.792 (0.134)
50 comparisons/time22.084 (0.075)3.579 (0.069)7.011 (0.078)5.130 (0.069)
Unequal exposure: 25% gaps23.768 (0.079)3.625 (0.083)6.912 (0.088)5.853 (0.081)
Unequal exposure: 40% gaps28.804 (0.093)9.839 (0.197)13.792 (0.137)12.079 (0.112)
Missing bridging edges9.088 (0.033)1.101 (0.029)1.410 (0.032)1.390 (0.028)
Crossing curves: 25% gaps46.338 (0.089)2.300 (0.050)10.538 (0.110)6.386 (0.081)
Informative, balanced42.458 (0.372)10.908 (0.234)43.406 (0.382)17.263 (0.296)
Informative, context gap70.054 (0.558)29.916 (0.601)116.041 (0.778)75.884 (0.841)
Null effect, context gap16.590 (0.152)15.133 (0.347)5.603 (0.096)6.946 (0.097)
Three informative features28.998 (0.034)8.794 (0.051)26.648 (0.028)13.233 (0.105)
Three informative + three irrelevant31.110 (0.047)14.003 (0.087)26.648 (0.028)18.179 (0.071)

Arena and T20 holdouts, and fitting time.

In Table 2, we report log loss and win-fraction MSE for the Arena and T20 holdouts, averaged over the original 100 collections. These results show that dBT has the smallest mean error for both criteria in both applications. dBT and kernel BT also require less fitting time than KickScore in every scenario, including parameter selection.

Table 2. Application holdouts: mean (conditional MCSE) over 100 collections.

Application and criteriondBTKernel BTKickScore
Arena: log loss0.63661 (0.00029)0.63765 (0.00032)0.63829 (0.00032)
Arena: win-fraction MSE0.00198 (0.00003)0.00259 (0.00005)0.00306 (0.00005)
T20: log loss0.65796 (0.00540)0.66625 (0.00514)0.66128 (0.00435)
T20: win-fraction MSE0.08229 (0.00269)0.08563 (0.00287)0.08398 (0.00270)

Manuscript revisions. We include KickScore in manuscript Sections 4 and 5 (Figures 2–5 and the text) and in the appendix simulation figures; Appendix C.9 reports the implementation, Tables 1 and 2 above, and the dataset-by-dataset and fitting-time comparisons (manuscript Tables 2–5).

Comments ub7V.5 and ub7V.8: Current rankings from past observations only.

Evaluation design.

We predicted current rankings at checkpoints near 40%, 60% and 80% of the observation period, fitting only observations at times s<ts<t to predict at tt. We held out the entire batch at tt, retained the original sparse observation schedules, and evaluated accuracy across all items and pairs. We required at least 20 earlier comparisons per item, 100 overall and a connected comparison graph.

For the original dBT fit and kernel BT, we used up to five chronological validation dates within the available history. In each fold, we fitted strictly earlier observations and predicted the next complete batch, constructing spline bases from that fold's training times. We then refitted the selected model on all observations before tt. For KickScore, we selected covariance parameters using the approximate marginal likelihood on the training data. We also selected the smoothing strength by approximate restricted maximum likelihood (REML) under the binomial model, using the original spline basis and penalty matrix (Wood, 2017); from each REML fit, we evaluated linear and flat continuation beyond the last training time.

We used 500 saved datasets per scenario and averaged each dataset's three checkpoint scores before calculating means and MCSEs across datasets. We evaluated contextual ranks at x=1x=1 for scalar covariates and x=0x=0 for vector covariates.

AuthorsOpenReview ↗

Response to Reviewer ub7V (part 3 of 4)

dBT remains the most accurate method.

In Table 3, we report mean absolute rank errors with MCSEs. These results show that dBT remains the most accurate method in this setting: with the penalty tuned by chronological cross-validation or by REML, it has the smallest mean rank error in 11 of 17 scenarios, and the REML variant with flat continuation has smaller error than KickScore in 12 scenarios and smaller error than the cross-validated dBT fit in 13. KickScore or kernel BT is best in the remaining scenarios, including both 40% absence designs. Unlike a gap, the period after the last observation is constrained from one side only, so the extrapolated value depends on how the curve is continued.

Table 3. Past-only current ranking: mean absolute rank error, mean (MCSE), R=500R=500. REML variants use linear or flat continuation.

ScenarioStatic BTdBTKernel BTKickScoredBT REML LineardBT REML Flat
Complete exposure2.1325 (0.003)0.7118 (0.009)0.7375 (0.009)0.7143 (0.008)0.6490 (0.008)0.6258 (0.008)
10% absences2.1322 (0.003)0.7528 (0.010)0.8158 (0.009)0.7975 (0.008)0.6875 (0.009)0.6688 (0.009)
25% absences2.0072 (0.005)0.8383 (0.010)0.9609 (0.010)0.8623 (0.008)0.7543 (0.009)0.7405 (0.008)
40% absences1.8340 (0.004)1.2560 (0.010)1.1676 (0.009)0.9760 (0.006)1.1867 (0.007)1.1783 (0.007)
90% weight reduction2.0992 (0.005)0.7915 (0.010)0.8545 (0.010)0.7925 (0.009)0.7130 (0.009)0.6928 (0.008)
98% weight reduction2.0260 (0.005)0.8318 (0.010)0.9402 (0.011)0.8352 (0.008)0.7433 (0.009)0.7313 (0.009)
20 comparisons/time2.0270 (0.007)1.3233 (0.013)1.3483 (0.012)1.2432 (0.011)1.2145 (0.011)1.1990 (0.011)
50 comparisons/time2.0067 (0.005)1.0475 (0.011)1.1034 (0.011)0.9870 (0.008)0.9298 (0.009)0.9228 (0.009)
Unequal exposure: 25% gaps1.9595 (0.004)0.9898 (0.011)1.1019 (0.010)0.9598 (0.008)0.8633 (0.009)0.8582 (0.009)
Unequal exposure: 40% gaps1.9098 (0.004)1.3232 (0.010)1.2639 (0.010)1.0723 (0.007)1.2447 (0.008)1.2337 (0.008)
Missing bridging edges2.1323 (0.003)1.0852 (0.012)0.7695 (0.010)0.8310 (0.010)1.0078 (0.012)0.9865 (0.012)
Crossing curves: 25% gaps1.7383 (0.004)0.7658 (0.007)0.9742 (0.009)0.8595 (0.007)0.6993 (0.006)0.7298 (0.006)
Informative, balanced2.8770 (0.013)1.2373 (0.013)2.3860 (0.015)1.5678 (0.013)1.2770 (0.012)1.2675 (0.012)
Informative, context gap3.5078 (0.008)1.7280 (0.017)2.7970 (0.010)2.5745 (0.013)1.7252 (0.017)1.7570 (0.017)
Null effect, context gap2.8583 (0.007)2.0243 (0.010)1.5712 (0.016)1.7070 (0.016)2.0140 (0.009)2.0085 (0.009)
Three informative features2.2408 (0.007)1.5163 (0.014)1.4515 (0.015)1.5272 (0.016)1.5285 (0.013)1.5208 (0.013)
Three informative + three irrelevant2.2407 (0.007)1.5702 (0.013)1.4515 (0.015)1.4463 (0.016)1.6035 (0.013)1.5918 (0.014)

Manuscript revisions. Manuscript Section 4 adds a paragraph on current rankings from past observations; Appendix C.10 reports the evaluation design and Table 3 as manuscript Table 6; Appendix A.5 describes REML selection and the continuation of each curve beyond the last observation.

Thank you again for your time and for the detailed review. Your questions prompted the comparison with KickScore and the evaluation of current rankings from past observations, and we believe the paper is better for them. If any point remains unclear, please let us know and we will reply quickly. If our response addresses your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Response to Reviewer ub7V (part 4 of 4)

References

  • H. Bong, W. Li, S. Shrotriya, and A. Rinaldo (2020). Nonparametric estimation in the dynamic Bradley–Terry model. Proceedings of AISTATS 2020, PMLR 108, 3317–3326.
  • E. Karlé and H. Tyagi (2023). Dynamic ranking with the BTL model: a nearest neighbor based rank centrality method. Journal of Machine Learning Research, 24(269), 1–57.
  • X. Tian, J. Shi, X. Shen, and K. Song (2024). A spectral approach for the dynamic Bradley–Terry model. Stat, 13(3), e722.
  • F. O'Sullivan, B. S. Yandell, and W. J. Raynor, Jr. (1986). Automatic smoothing of regression functions in generalized linear models. Journal of the American Statistical Association, 81(393), 96–103. doi:10.1080/01621459.1986.10478243.
  • S. N. Wood (2017). Generalized Additive Models: An Introduction with R, second edition. Chapman and Hall/CRC. Author's book page.
  • J. Z. Huang and Y. Su (2021). Asymptotic properties of penalized spline estimators in concave extended linear models: Rates of convergence. The Annals of Statistics, 49(6), 3383–3407. doi:10.1214/21-AOS2088.
  • L. Maystre, V. Kristof, and M. Grossglauser (2019). Pairwise Comparisons with Flexible Time-Dynamics. Proceedings of KDD 2019. doi:10.1145/3292500.3330831.
Reviewer_TLqjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem that "leaderboards change over time." The authors propose a dynamic Bradley-Terry method applicable both to LLM evaluation and to international sports. Overall, the motivation is clear and the applications are relevant, but the writing and presentation remain inaccessible to non-specialists.

Critical strengths

Traditional leaderboards assume fixed ability, yet AI models iterate rapidly and team form fluctuates; static methods are clearly inadequate. The paper accurately targets the pain points of sparse data, imbalanced schedules, and temporary absences of a team/model .

By pooling information across time—using comparison data from before and after an item's absence to infer its performance during the gap—the method addresses the break in the data problem common in real-world settings. The paper also provides theory showing under what conditions this works.

There are extensive simulation studies (covering diverse temporal patterns, imbalanced schedules, and temporary absences) as well as two real-data applications (T20 cricket and Chatbot Arena), reporting lower held-out errors than static and kernel BT methods .

Critical weaknesses

The methodology section is dense with notation and matrix derivations that a non-statistics reader will struggle to follow. I suggest adding a non-technical summary or an illustrative figure in the main text showing intuitively how temporal information pooling works .

The paper acknowledges the method fails when an absence is too long or surrounding information is insufficient, but a layperson cannot tell how long is too long. I recommend providing empirical rules or failure-case demonstrations so readers know when not to trust the interpolated results .

Held-out errors are evaluated on constructed interpolation holdouts, which is precisely the scenario this method is designed to excel at. How the method performs on unseen extrapolation is not sufficiently discussed in the main text; please add such an evaluation or at least state this limitation .

The method treats comparison times and model pairings as given, and excludes ties or both bad outcomes. In real crowdsourced evaluation, ties may be a non-negligible fraction; the impact of this exclusion on leaderboard conclusions should be reported more transparently .

Questions for authors

See above weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer TLqj (part 1 of 4)

We thank the reviewer for a thoughtful review from the reader's perspective; addressing these comments has made the paper clearer and more transparent. The revised manuscript marks every change in blue, and each response below ends with the location of the corresponding revision.

Comment TLqj.5: A non-technical summary and an illustrative figure.

We thank the reviewer for suggesting a more accessible presentation of the methodology. In the revised manuscript, Section 2 opens with a non-technical summary: we represent each item's ability as a smooth curve over time and estimate all curves jointly, so that comparisons before and after an item's absence, together with comparisons among the other items, determine its estimated ability during the absence. A new Figure 1 illustrates this in a simulated example with four items, one of which is absent for part of the period: panel (a) shows when each item is compared; panel (b) shows that the absent item is disconnected from the others within the absence but connected to all of them over the whole period; and panel (c) shows the dBT estimate bridging the absence close to the true curve.

Manuscript revisions. Manuscript Section 2 (opening paragraph and Figure 1).

Comment TLqj.6: When an interpolated estimate can be trusted.

A practical rule.

As a practical rule, an estimate inside an absence can be trusted when (i) the item has comparisons on both sides of the absence, (ii) the comparison graph is connected on each side, and (iii) the estimate is stable under stronger smoothing. Appendix B.7 gives the reason: comparisons on either side identify a polynomial contrast across the gap, whereas a sufficiently flexible spline can vary inside the gap without changing any observed comparison probability. Conditions (i) and (ii) supply the information that bridges the gap, and (iii) checks that the fit is not using the remaining freedom. The revised Appendix A.5 states these checks.

How error grows with the length of the absence.

In Table 1, we report the error for absences covering 10%, 25% and 40% of the observation period, with comparisons on both sides in each case. The error of every method increases with the length of the absence. dBT has the smallest mean error at each length, and lower error than KickScore (Maystre et al., 2019), a Gaussian-process trajectory model fitted to the same datasets, in most individual datasets. In our simulations, absences up to 40% of the period therefore remain interpretable under conditions (i) and (ii), with error that grows with their length.

Table 1. Absences covering 10%, 25% and 40% of the observation period, with comparisons on both sides: probability MSE, mean (MCSE), R=500R=500, in units of 10−310^{-3}, and the fraction of the 500 datasets in which dBT has lower probability MSE and lower ordering error than KickScore.

AbsenceStatic BTdBTKernel BTKickScoredBT lower: prob. MSEdBT lower: ordering
10% absences19.172 (0.050)1.119 (0.022)1.959 (0.030)1.624 (0.027)0.940.75
25% absences21.971 (0.050)2.032 (0.043)5.686 (0.061)3.616 (0.049)0.990.91
40% absences26.538 (0.063)6.790 (0.396)12.014 (0.097)9.173 (0.082)0.970.97

Failure cases and the remedy.

Condition (iii) matters in the worst cases. In Table 2, we report the 5 datasets with the largest dBT errors among the 500 of the 40% absence scenario. Their designs are full rank, so support checks alone do not flag them, but increasing the selected penalty by five grid steps brings their error below kernel BT's. A fixed increase of the penalty is not a general remedy, because the same change raises the error in other datasets. We therefore added selection by restricted maximum likelihood (REML), which chooses the smoothing strength from the data of each fit. In the past-only evaluation of Comment TLqj.7 below, REML selection with flat continuation has lower rank error than the original dBT fit in 13 of 17 scenarios, by up to 13%.

Table 2. Selected extreme datasets: probability MSE under stronger smoothing.

DatasetdBTdBT: penalty +5 stepsKernel BT
Dataset 3430.1150010.0087290.013025
Dataset 4770.0937990.0054050.013573
Dataset 700.0879690.0039380.015923
Dataset 1300.0842100.0009850.010161
Dataset 2150.0649580.0050580.011875

Manuscript revisions. Appendix A.5 states the support checks, the REML option and the continuation rules, and includes Table 2 as manuscript Table 1; Section 2 points to it.

AuthorsOpenReview ↗

Response to Reviewer TLqj (part 2 of 4)

Comment TLqj.7: Extrapolation, with current rankings estimated from past observations only.

Interpolation versus extrapolation.

To investigate extrapolation, we estimated current rankings from past observations alone on the same saved datasets: at each checkpoint tt, we fit only observations at times s<ts<t and predict the next batch, so each item's curve is extended beyond its last comparison.

Evaluation design.

We predicted current rankings at checkpoints near 40%, 60% and 80% of the observation period, fitting only observations at times s<ts<t to predict at tt. We held out the entire batch at tt, retained the original sparse observation schedules, and evaluated accuracy across all items and pairs. We required at least 20 earlier comparisons per item, 100 overall and a connected comparison graph.

For the original dBT fit and kernel BT, we used up to five chronological validation dates within the available history. In each fold, we fitted strictly earlier observations and predicted the next complete batch, constructing spline bases from that fold's training times. We then refitted the selected model on all observations before tt. For KickScore, we selected covariance parameters using the approximate marginal likelihood on the training data. We also selected the smoothing strength by approximate REML under the binomial model, using the original spline basis and penalty matrix (Wood, 2017); from each REML fit, we evaluated linear and flat continuation beyond the last training time.

We used 500 saved datasets per scenario and averaged each dataset's three checkpoint scores before calculating means and MCSEs across datasets. We evaluated contextual ranks at x=1x=1 for scalar covariates and x=0x=0 for vector covariates.

AuthorsOpenReview ↗

Response to Reviewer TLqj (part 3 of 4)

dBT remains the most accurate method.

In Table 3, we report mean absolute rank errors with MCSEs. These results show that dBT remains the most accurate method in this setting: with the penalty tuned by chronological cross-validation or by REML, it has the smallest mean rank error in 11 of 17 scenarios, and the REML variant with flat continuation has smaller error than KickScore in 12 scenarios and smaller error than the cross-validated dBT fit in 13. KickScore or kernel BT is best in the remaining scenarios, including both 40% absence designs. Unlike a gap, the period after the last observation is constrained from one side only, so the extrapolated value depends on how the curve is continued.

Table 3. Past-only current ranking: mean absolute rank error, mean (MCSE), R=500R=500. REML variants use linear or flat continuation.

ScenarioStatic BTdBTKernel BTKickScoredBT REML LineardBT REML Flat
Complete exposure2.1325 (0.003)0.7118 (0.009)0.7375 (0.009)0.7143 (0.008)0.6490 (0.008)0.6258 (0.008)
10% absences2.1322 (0.003)0.7528 (0.010)0.8158 (0.009)0.7975 (0.008)0.6875 (0.009)0.6688 (0.009)
25% absences2.0072 (0.005)0.8383 (0.010)0.9609 (0.010)0.8623 (0.008)0.7543 (0.009)0.7405 (0.008)
40% absences1.8340 (0.004)1.2560 (0.010)1.1676 (0.009)0.9760 (0.006)1.1867 (0.007)1.1783 (0.007)
90% weight reduction2.0992 (0.005)0.7915 (0.010)0.8545 (0.010)0.7925 (0.009)0.7130 (0.009)0.6928 (0.008)
98% weight reduction2.0260 (0.005)0.8318 (0.010)0.9402 (0.011)0.8352 (0.008)0.7433 (0.009)0.7313 (0.009)
20 comparisons/time2.0270 (0.007)1.3233 (0.013)1.3483 (0.012)1.2432 (0.011)1.2145 (0.011)1.1990 (0.011)
50 comparisons/time2.0067 (0.005)1.0475 (0.011)1.1034 (0.011)0.9870 (0.008)0.9298 (0.009)0.9228 (0.009)
Unequal exposure: 25% gaps1.9595 (0.004)0.9898 (0.011)1.1019 (0.010)0.9598 (0.008)0.8633 (0.009)0.8582 (0.009)
Unequal exposure: 40% gaps1.9098 (0.004)1.3232 (0.010)1.2639 (0.010)1.0723 (0.007)1.2447 (0.008)1.2337 (0.008)
Missing bridging edges2.1323 (0.003)1.0852 (0.012)0.7695 (0.010)0.8310 (0.010)1.0078 (0.012)0.9865 (0.012)
Crossing curves: 25% gaps1.7383 (0.004)0.7658 (0.007)0.9742 (0.009)0.8595 (0.007)0.6993 (0.006)0.7298 (0.006)
Informative, balanced2.8770 (0.013)1.2373 (0.013)2.3860 (0.015)1.5678 (0.013)1.2770 (0.012)1.2675 (0.012)
Informative, context gap3.5078 (0.008)1.7280 (0.017)2.7970 (0.010)2.5745 (0.013)1.7252 (0.017)1.7570 (0.017)
Null effect, context gap2.8583 (0.007)2.0243 (0.010)1.5712 (0.016)1.7070 (0.016)2.0140 (0.009)2.0085 (0.009)
Three informative features2.2408 (0.007)1.5163 (0.014)1.4515 (0.015)1.5272 (0.016)1.5285 (0.013)1.5208 (0.013)
Three informative + three irrelevant2.2407 (0.007)1.5702 (0.013)1.4515 (0.015)1.4463 (0.016)1.6035 (0.013)1.5918 (0.014)

Manuscript revisions. Manuscript Section 4 adds a paragraph on current rankings from past observations; Appendix C.10 reports the evaluation design and Table 3 as manuscript Table 6; the Discussion states the distinction between interpolation and extrapolation.

Comment TLqj.8: Ties and “both bad” outcomes do not change the conclusions.

What we did.

In the Arena cohort, ties are 17.4% of the decisive outcomes and ties combined (2,997 ties, 14,232 decisive outcomes), and there are 2,328 “both bad” outcomes. We repeated the analysis under two alternative rules, in both the training and the test sets and with the same parameter selection: ties counted as half a win for each model, and ties and “both bad” outcomes both counted as half wins, as the Arena leaderboard does.

AuthorsOpenReview ↗

Response to Reviewer TLqj (part 4 of 4)

Conclusions are unchanged.

The weekly leaderboards are nearly unchanged: the top three models are the same in every week, and no model moves by more than 1 position. In the holdouts (Table 4), dBT has the smallest mean error for both criteria under every rule and lower error than static and kernel BT in every collection; log loss rises for all methods once ties are included, because a tie scored as 1/21/2 contributes at least log⁡2\log 2. In T20, ties are rare (17 ties against 763 decisive matches, 2.2%); counting them as half wins leaves the annual leaderboards and the holdout conclusions unchanged.

Table 4. Arena holdouts under three outcome rules: mean (conditional MCSE) over the 100 collections. Win-fraction MSE is in units of 10−310^{-3}.

Outcome ruleStatic BTKernel BTdBT
Win-fraction MSE
Ties excluded (paper)3.85 (0.06)2.59 (0.05)1.98 (0.03)
Ties as half wins2.82 (0.04)2.64 (0.04)1.69 (0.03)
Ties and both bad as half wins2.43 (0.03)2.13 (0.03)1.63 (0.02)
Log loss
Ties excluded (paper)0.6396 (0.0003)0.6376 (0.0003)0.6366 (0.0003)
Ties as half wins0.6552 (0.0002)0.6560 (0.0002)0.6540 (0.0002)
Ties and both bad as half wins0.6625 (0.0002)0.6627 (0.0002)0.6616 (0.0002)

Manuscript revisions. Manuscript Section 2 states the tie rule, and the data descriptions in Section 5 point to Appendix D.4, which reports the outcome counts, the leaderboard agreement for both applications (manuscript Table 9), the Arena holdouts under three outcome rules (Table 4 above, manuscript Table 8) and the T20 holdouts under both rules (manuscript Table 10).

Thank you again for your time and for the detailed review. Your comments led to new experiments on extrapolation and on ties, and to a clearer presentation, and we think the paper is better for it. If anything is still unclear, please tell us and we will respond quickly. If our response has addressed your concerns, we would be grateful if you could consider raising your score.

References

  • S. N. Wood (2017). Generalized Additive Models: An Introduction with R, second edition. Chapman and Hall/CRC. Author's book page.
  • L. Maystre, V. Kristof, and M. Grossglauser (2019). Pairwise Comparisons with Flexible Time-Dynamics. Proceedings of KDD 2019. doi:10.1145/3292500.3330831.
Reviewer_rWsNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper estimates how relative abilities change over time when some items are compared rarely or disappear for a period. It represents each item’s ability as a smooth curve, fits all curves jointly with a penalized Bradley–Terry likelihood, and extends the model to time-varying context effects. Its main theoretical contribution is to distinguish what is needed to predict outcomes on the observed schedule from what is needed to recover an item’s ability during a gap. Simulations test both probability and ranking recovery. On constructed holdouts in Chatbot Arena and international T20 cricket, the method has lower average error than static and kernel Bradley–Terry. These are interpolation results: observations after each holdout are available for fitting.

Critical strengths
  1. Appendix B.7 gives a useful contrast: data on either side can identify a fixed-degree polynomial, but a sufficiently flexible spline can vary entirely inside an unobserved interval without changing any observed matchup probability. This explains both why temporal pooling can work and why smoothness alone does not guarantee recovery.
  2. Figure 1 evaluates the pairs involving absent items during their absences, when the true probabilities and rankings are known. With a fixed comparison budget and 500 datasets per configuration, the proposed method has lower mean probability and ordering error than both baselines across the tested gap lengths.
Critical weaknesses
  1. The closest competing approach is missing from the experiments. Static BT cannot track change, and kernel BT weights observations by their distance from the target time. A method such as KickScore (Maystre et al., 2019) can also use observations on both sides of a gap through a model of the full trajectory. Without such a comparison, the results show the value of pooling across time but do not establish a practical advantage of the proposed spline estimator over an existing way to do so.
  2. The authors correctly keep test battles out of model selection. However, they select eight models for high variation in weekly win fractions, which can also change when the opponent mix changes. The 100 draws reuse 15 overlapping windows containing 1,044 distinct battles. Winning all 100 draws is useful evidence for these selected windows, but it leaves open how often the gain occurs across the broader leaderboard.
Questions for authors
  1. On the same absence simulations and real-data holdouts, how does dBT compare with a tuned Gaussian-process ranking model that also uses observations before and after the gap?
  2. If targets are selected by exposure alone, without screening on weekly win fractions, how does the paired per-battle log-loss difference versus kernel BT vary across targets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer rWsN (part 1 of 3)

We thank the reviewer for a careful and constructive review; addressing these comments has improved the paper. The revised manuscript marks every change in blue, and each response below ends with the location of the corresponding revision.

Comments rWsN.4 and rWsN.6: dBT remains more accurate and faster than KickScore on the identical data.

We compared static BT, dBT, kernel BT and KickScore (Maystre et al., 2019) on the same 500 saved datasets in each of the 17 original simulation scenarios, using the original observations, holdouts and recovery regions.

How we fitted and tuned KickScore.

We used KickScore 0.2.0 with the logistic likelihood, recursive Gaussian-process fitting and expectation propagation (EP; tolerance 10−410^{-4}, at most 300 iterations). We selected among 128 covariance configurations by maximizing the approximate marginal likelihood on the training data. The covariance families and the random-search selection follow Maystre et al. (2019): constant or affine components, alone or combined with Wiener, exponential or Matérn processes (ν=3/2,5/2\nu=3/2,5/2). Because the experiments in Maystre et al. (2019) take user-supplied parameter ranges and measure time in years, we set the ranges for the [0,1][0,1] time scale and the magnitude of the simulated curves, drew the configurations log-uniformly within them with a recorded seed, and used the same 128 configurations for every dataset: [10−3,16][10^{-3},16] for constant variance, [10−3,64][10^{-3},64] for dynamic and slope variances, and [0.01,4][0.01,4] for length scales. We used EP damping 0.50.5, retrying with 0.10.1 and 0.050.05 when needed for convergence.

Covariates.

For contextual scenarios, we used KickScore's weighted-feature interface to fit independent Gaussian processes for each item's baseline and covariate coefficients. We searched [10−3,4][10^{-3},4] for the context-to-baseline variance ratio. We used the same covariates for static BT, dBT and KickScore, and fitted kernel BT without covariates, as in the paper. For KickScore predictions, we integrated over the fitted Gaussian uncertainty.

Recovery during absences and under sparse schedules.

In Table 1, we report probability MSEs and MCSEs for the 17 original simulation scenarios. These results show that dBT has lower mean probability MSE than KickScore in 16 scenarios, with a median reduction of 32%.

Table 1. Simulation scenarios: probability MSE, mean (MCSE), R=500R=500. Entries are in units of 10−310^{-3}.

ScenarioStatic BTdBTKernel BTKickScore
Complete exposure16.126 (0.032)0.860 (0.013)1.014 (0.014)1.226 (0.015)
10% absences19.172 (0.050)1.119 (0.022)1.959 (0.030)1.624 (0.027)
25% absences21.971 (0.050)2.032 (0.043)5.686 (0.061)3.616 (0.049)
40% absences26.538 (0.063)6.790 (0.396)12.014 (0.097)9.173 (0.082)
90% weight reduction20.732 (0.049)1.522 (0.032)3.122 (0.046)2.272 (0.040)
98% weight reduction21.696 (0.050)1.920 (0.042)5.054 (0.075)3.122 (0.050)
20 comparisons/time22.529 (0.118)7.769 (0.151)10.031 (0.133)8.792 (0.134)
50 comparisons/time22.084 (0.075)3.579 (0.069)7.011 (0.078)5.130 (0.069)
Unequal exposure: 25% gaps23.768 (0.079)3.625 (0.083)6.912 (0.088)5.853 (0.081)
Unequal exposure: 40% gaps28.804 (0.093)9.839 (0.197)13.792 (0.137)12.079 (0.112)
Missing bridging edges9.088 (0.033)1.101 (0.029)1.410 (0.032)1.390 (0.028)
Crossing curves: 25% gaps46.338 (0.089)2.300 (0.050)10.538 (0.110)6.386 (0.081)
Informative, balanced42.458 (0.372)10.908 (0.234)43.406 (0.382)17.263 (0.296)
Informative, context gap70.054 (0.558)29.916 (0.601)116.041 (0.778)75.884 (0.841)
Null effect, context gap16.590 (0.152)15.133 (0.347)5.603 (0.096)6.946 (0.097)
Three informative features28.998 (0.034)8.794 (0.051)26.648 (0.028)13.233 (0.105)
Three informative + three irrelevant31.110 (0.047)14.003 (0.087)26.648 (0.028)18.179 (0.071)
AuthorsOpenReview ↗

Response to Reviewer rWsN (part 2 of 3)

Dataset by dataset.

Because we fit all methods to the same datasets, we also compare dBT and KickScore dataset by dataset. In Table 2, we report the fraction of datasets in which dBT has lower error than KickScore, for probability MSE and ordering error. These results show that dBT has lower error in most datasets for both criteria in 16 of 17 scenarios. The gains are largest for crossing curves and for contextual effects that vary over time. In the null-effect scenario, where the true context effect is zero, KickScore and kernel BT, which omits the covariate, have lower error.

Table 2. Simulation scenarios: fraction of the 500 datasets in which dBT has lower error than KickScore.

ScenarioProbability MSEOrdering error
Complete exposure0.990.82
10% absences0.940.75
25% absences0.990.91
40% absences0.970.97
90% weight reduction0.950.88
98% weight reduction0.960.87
20 comparisons/time0.670.63
50 comparisons/time0.960.87
Unequal exposure: 25% gaps0.960.92
Unequal exposure: 40% gaps0.740.78
Missing bridging edges0.860.66
Crossing curves: 25% gaps0.990.99
Informative, balanced0.910.86
Informative, context gap0.980.96
Null effect, context gap0.060.07
Three informative features0.980.98
Three informative + three irrelevant0.980.98

Arena and T20 holdouts.

In Table 3, we report log loss and win-fraction MSE for all 15 Arena and 75 T20 holdouts, averaged over the original 100 collections. We calculate win-fraction MSE by comparing predicted and empirical win fractions against each target's held-out opponents; the conditional MCSE measures uncertainty in this average, with the observed data fixed. These results show that dBT has the smallest mean error for both criteria in both applications, with larger gains in Arena than in T20.

Table 3. Application holdouts: mean (conditional MCSE) over 100 collections.

Application and criteriondBTKernel BTKickScore
Arena: log loss0.63661 (0.00029)0.63765 (0.00032)0.63829 (0.00032)
Arena: win-fraction MSE0.00198 (0.00003)0.00259 (0.00005)0.00306 (0.00005)
T20: log loss0.65796 (0.00540)0.66625 (0.00514)0.66128 (0.00435)
T20: win-fraction MSE0.08229 (0.00269)0.08563 (0.00287)0.08398 (0.00270)

Fitting time.

In Table 4, we report mean fitting times in seconds and MCSEs across the 500 datasets. These results show that dBT and kernel BT require less fitting time than KickScore in every scenario.

Table 4. Simulation scenarios: fitting and tuning time in seconds, mean (MCSE), R=500R=500.

ScenariodBTKernel BTKickScore
Complete exposure2.86 (0.011)2.53 (0.0089)332 (0.9)
10% absences2.54 (0.0097)2.6 (0.0086)383 (1.2)
25% absences2.58 (0.0091)2.68 (0.0091)431 (1.4)
40% absences4.56 (0.019)2.91 (0.01)468 (1.6)
90% weight reduction2.6 (0.011)2.52 (0.0093)650 (3.2)
98% weight reduction2.48 (0.0099)4.25 (0.022)1.15e+3 (7.7)
20 comparisons/time2.34 (0.013)2.8 (0.012)89.2 (0.37)
50 comparisons/time2.52 (0.0092)2.55 (0.0084)223 (0.86)
Unequal exposure: 25% gaps2.51 (0.012)2.72 (0.01)735 (3.2)
Unequal exposure: 40% gaps3.69 (0.017)2.94 (0.011)830 (3.2)
Missing bridging edges2.79 (0.011)2.56 (0.0088)568 (3.1)
Crossing curves: 25% gaps2.42 (0.0094)2.8 (0.0099)690 (2.5)
Informative, balanced4.27 (0.11)0.538 (0.002)81.3 (0.33)
Informative, context gap4.72 (0.12)0.534 (0.002)102 (0.47)
Null effect, context gap3.49 (0.062)0.534 (0.0019)101 (0.45)
Three informative features21.8 (0.53)0.959 (0.0031)519 (2.2)
Three informative + three irrelevant60.6 (1.2)0.996 (0.0034)966 (2.4)

Comments rWsN.5 and rWsN.7: The gain concentrates where abilities change over time.

In the paper, we selected the holdout windows from the comparison schedule alone, as windows with low local training exposure. We selected the eight target models by noise-adjusted variation in weekly win fractions, computed on a discovery sample whose battles are excluded from every test set; this screen identifies models whose abilities change over time, which is the setting the method is designed for. The 100 collections resample these 15 windows, so they do not add evidence beyond them; the reviewer's question about the broader leaderboard calls for an evaluation that does not depend on the screen.

AuthorsOpenReview ↗

Response to Reviewer rWsN (part 3 of 3)

All 16 models on windows selected by exposure alone.

To answer the reviewer's question directly, we also evaluated all 16 models with targets selected by exposure alone: one seven-day window with few comparisons per model, chosen from comparison dates and counts, with observations required on either side of the window and support checked for the basis coefficients. In Table 5, we report each model's per-battle log loss and the paired difference between dBT and kernel BT, separately for the eight original targets and the other eight models.

Table 5. Exposure-selected Arena windows: per-battle log loss and difference (dBT −- kernel BT). Negative differences favor dBT.

ModelBattlesdBTKernel BTDifference
Original targets
ChatGPT-4o latest1250.56280.5655−0.00265
Claude 3.7 Sonnet1190.59510.5937+0.00143
Grok 3 Mini1200.65100.6579−0.00697
Grok 3 Preview970.63390.6339+0.00005
Llama 4 Maverick1190.70170.7051−0.00338
o31530.67040.6741−0.00373
o3-mini1090.63970.6369+0.00279
o4-mini1120.66180.6619−0.00006
Mean−0.00156
Other models
Claude 3.5 Haiku1180.61080.6125−0.00170
Claude 3.5 Sonnet1220.65280.6515+0.00137
Gemini 2.0 Flash700.64700.6478−0.00081
Gemma 3 27B1190.69030.6908−0.00048
GPT-4.11040.67100.6700+0.00107
Llama 3.3 70B720.56980.5690+0.00074
Qwen Max640.67280.6739−0.00107
QwQ-32B660.68520.6820+0.00323
Mean+0.00029

The advantage of dBT is more prominent for the models whose abilities change over time, as we expect.

Manuscript revisions. We include KickScore in manuscript Sections 4 and 5 (Figures 2–5 and the text), in the appendix simulation figures and in Appendix C.9, where we report the implementation and Tables 1–4 above as manuscript Tables 2–5.

Thank you again for your time and for the detailed review. Your questions led us to run the KickScore comparison and the exposure-only evaluation, and we believe the paper is stronger as a result. If anything remains unclear, please let us know and we will reply promptly. If our response addresses your concerns, we would be grateful if you would consider raising your score.

References

  • L. Maystre, V. Kristof, and M. Grossglauser (2019). Pairwise Comparisons with Flexible Time-Dynamics. Proceedings of KDD 2019. doi:10.1145/3292500.3330831.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair and all reviewers

We thank the reviewers for careful and constructive reviews. They raised four substantive points---the absence of a comparison with the closest dynamic competitor, KickScore (Reviewers rWsN and ub7V); the evaluation on interpolation holdouts only, without an assessment of current rankings estimated from past observations (Reviewers TLqj and ub7V); the exclusion of ties and “both bad” outcomes without a sensitivity analysis (Reviewer TLqj); and the density of the methodology section and the need for a sharper statement of the contributions (Reviewers TLqj and ub7V). Each point is addressed below with new evidence computed on the identical saved datasets and holdouts of the submission, and a revised manuscript has been uploaded in which all changes are marked in blue.

What we added.

  • Comparison with KickScore (Maystre et al., 2019), with its covariance structure selected by marginal likelihood among 128 configurations, on all 500 saved datasets of each of the 17 simulation scenarios and on all 15 Arena and 75 T20 holdouts. dBT has lower mean probability MSE in 16 of 17 scenarios, with a median reduction of 32%, lower error in most individual datasets in 16 scenarios, the smallest mean error for both criteria in both applications, and shorter fitting times in every scenario. The gains are largest in the two scenarios that depend most on pooling information across time: crossing ability curves with gaps, where dBT's error is 64% lower than KickScore's, and time-varying contextual effects with a context gap, where it is 61% lower.
  • Current rankings from past observations. At checkpoints that use only earlier observations, dBT, with the penalty tuned by chronological cross-validation or by restricted maximum likelihood (REML), has the smallest mean rank error in 11 of 17 scenarios, and the REML variant with flat continuation has smaller error than KickScore in 12.
  • Ties and “both bad” outcomes. Counting ties, and additionally “both bad” outcomes, as half wins leaves the Arena and T20 leaderboards nearly unchanged, and dBT keeps the smallest holdout error under every rule.
  • Exposition and contributions. Section 2 now opens with a non-technical summary and an illustrative figure, and the Introduction states the contributions (identification from the joint time–comparison design, a design constant that separates recovery from prediction, and time-varying covariate effects) and contrasts dBT with local-window methods and with penalized spline regression.

Our main claim holds under the new evidence. The paper's claim is that estimating all ability curves jointly, through a global basis and penalty, recovers abilities during absences and under sparse schedules better than local smoothing. The identical-data comparison shows that this advantage also holds against the strongest existing trajectory model, and the past-only evaluation shows that it extends to current rankings.

Manuscript revision.

  • Introduction: The contributions are stated explicitly, and dBT is contrasted with local-window methods and with penalized spline regression (Reviewer ub7V).
  • Section 2: Opens with a non-technical summary and a new Figure 1, which shows how comparisons before and after an absence, together with those among the other items, determine the estimate during the absence (Reviewer TLqj). It also points to Appendix A.5 for REML selection, continuation beyond the last observation and support checks.
  • Section 4: KickScore is added to Figures 2 and 3 and to the text, and a new paragraph reports current rankings estimated from past observations only (Reviewers rWsN, TLqj and ub7V).
  • Section 5: KickScore is added to Figures 4 and 5 and to the text (Reviewers rWsN and ub7V), and the data descriptions point to the tie sensitivity analysis in Appendix D.4 (Reviewer TLqj).
  • Discussion: Distinguishes interpolation within an absence from extrapolation beyond the last observation and summarizes the past-only results (Reviewer TLqj).
  • Appendix: A.5 describes REML selection, the continuation rules and the support checks, with the penalty diagnostic in Table 1; C.9 reports the comparison with KickScore (Tables 2–5), and KickScore is added to all simulation figures; C.10 reports current rankings from past observations (Table 6); D.4 reports ties and non-decisive outcomes (Tables 8–10).

References

  • L. Maystre, V. Kristof, and M. Grossglauser (2019). Pairwise Comparisons with Flexible Time-Dynamics. Proceedings of KDD 2019. doi:10.1145/3292500.3330831.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.