- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper considers the BT model with time-varying coefficients from sparse pairwise comparisons. The parameters are estimated by basis expansions and a penalized Bradley–Terry likelihood. Theoretical results on identification and convergence rates are presented. The numerical studies illustrate the effectiveness of the proposed method.
Critical strengths
-
Ranking from sparse, time-varying comparisons is an important and practical problem.
-
The paper provides a systematic theoretical analysis, including model identifiability, parameter estimation error, and prediction error.
Critical weaknesses
-
The methodological novelty is limited. The proposed estimator combines existing basis expansion and penalized likelihood methods. Its advantages over existing dynamic ranking methods need further justification.
-
The numerical comparisons include only static and kernel-based baselines, omitting existing dynamic BT models. Furthermore, the real-data experiments show that the method can reconstruct past rankings, but do not consider estimating current rankings using only historical data.
Questions for authors
-
What is the main technical contribution beyond existing dynamic Bradley–Terry models and penalized spline estimation theory?
-
How does the method compare with an existing dynamic BT model under the same evaluation protocol?
-
How does performance change when only past observations are available?
4 comments
Response to Reviewer ub7V (part 2 of 4)
Simulations.
In Table 1, we report probability MSEs with MCSEs. These results show that dBT has lower mean probability MSE than KickScore in 16 of 17 scenarios, with a median reduction of 32%, and lower error than KickScore in most individual datasets in 16 scenarios. The gains are largest for crossing curves and for contextual effects that vary over time; in the null-effect scenario, KickScore and kernel BT, which omits the covariate, have lower error.
Table 1. Simulation scenarios: probability MSE, mean (MCSE), . Entries are in units of .
| Scenario | Static BT | dBT | Kernel BT | KickScore |
|---|---|---|---|---|
| Complete exposure | 16.126 (0.032) | 0.860 (0.013) | 1.014 (0.014) | 1.226 (0.015) |
| 10% absences | 19.172 (0.050) | 1.119 (0.022) | 1.959 (0.030) | 1.624 (0.027) |
| 25% absences | 21.971 (0.050) | 2.032 (0.043) | 5.686 (0.061) | 3.616 (0.049) |
| 40% absences | 26.538 (0.063) | 6.790 (0.396) | 12.014 (0.097) | 9.173 (0.082) |
| 90% weight reduction | 20.732 (0.049) | 1.522 (0.032) | 3.122 (0.046) | 2.272 (0.040) |
| 98% weight reduction | 21.696 (0.050) | 1.920 (0.042) | 5.054 (0.075) | 3.122 (0.050) |
| 20 comparisons/time | 22.529 (0.118) | 7.769 (0.151) | 10.031 (0.133) | 8.792 (0.134) |
| 50 comparisons/time | 22.084 (0.075) | 3.579 (0.069) | 7.011 (0.078) | 5.130 (0.069) |
| Unequal exposure: 25% gaps | 23.768 (0.079) | 3.625 (0.083) | 6.912 (0.088) | 5.853 (0.081) |
| Unequal exposure: 40% gaps | 28.804 (0.093) | 9.839 (0.197) | 13.792 (0.137) | 12.079 (0.112) |
| Missing bridging edges | 9.088 (0.033) | 1.101 (0.029) | 1.410 (0.032) | 1.390 (0.028) |
| Crossing curves: 25% gaps | 46.338 (0.089) | 2.300 (0.050) | 10.538 (0.110) | 6.386 (0.081) |
| Informative, balanced | 42.458 (0.372) | 10.908 (0.234) | 43.406 (0.382) | 17.263 (0.296) |
| Informative, context gap | 70.054 (0.558) | 29.916 (0.601) | 116.041 (0.778) | 75.884 (0.841) |
| Null effect, context gap | 16.590 (0.152) | 15.133 (0.347) | 5.603 (0.096) | 6.946 (0.097) |
| Three informative features | 28.998 (0.034) | 8.794 (0.051) | 26.648 (0.028) | 13.233 (0.105) |
| Three informative + three irrelevant | 31.110 (0.047) | 14.003 (0.087) | 26.648 (0.028) | 18.179 (0.071) |
Arena and T20 holdouts, and fitting time.
In Table 2, we report log loss and win-fraction MSE for the Arena and T20 holdouts, averaged over the original 100 collections. These results show that dBT has the smallest mean error for both criteria in both applications. dBT and kernel BT also require less fitting time than KickScore in every scenario, including parameter selection.
Table 2. Application holdouts: mean (conditional MCSE) over 100 collections.
| Application and criterion | dBT | Kernel BT | KickScore |
|---|---|---|---|
| Arena: log loss | 0.63661 (0.00029) | 0.63765 (0.00032) | 0.63829 (0.00032) |
| Arena: win-fraction MSE | 0.00198 (0.00003) | 0.00259 (0.00005) | 0.00306 (0.00005) |
| T20: log loss | 0.65796 (0.00540) | 0.66625 (0.00514) | 0.66128 (0.00435) |
| T20: win-fraction MSE | 0.08229 (0.00269) | 0.08563 (0.00287) | 0.08398 (0.00270) |
Manuscript revisions. We include KickScore in manuscript Sections 4 and 5 (Figures 2–5 and the text) and in the appendix simulation figures; Appendix C.9 reports the implementation, Tables 1 and 2 above, and the dataset-by-dataset and fitting-time comparisons (manuscript Tables 2–5).
Comments ub7V.5 and ub7V.8: Current rankings from past observations only.
Evaluation design.
We predicted current rankings at checkpoints near 40%, 60% and 80% of the observation period, fitting only observations at times to predict at . We held out the entire batch at , retained the original sparse observation schedules, and evaluated accuracy across all items and pairs. We required at least 20 earlier comparisons per item, 100 overall and a connected comparison graph.
For the original dBT fit and kernel BT, we used up to five chronological validation dates within the available history. In each fold, we fitted strictly earlier observations and predicted the next complete batch, constructing spline bases from that fold's training times. We then refitted the selected model on all observations before . For KickScore, we selected covariance parameters using the approximate marginal likelihood on the training data. We also selected the smoothing strength by approximate restricted maximum likelihood (REML) under the binomial model, using the original spline basis and penalty matrix (Wood, 2017); from each REML fit, we evaluated linear and flat continuation beyond the last training time.
We used 500 saved datasets per scenario and averaged each dataset's three checkpoint scores before calculating means and MCSEs across datasets. We evaluated contextual ranks at for scalar covariates and for vector covariates.
Response to Reviewer ub7V (part 3 of 4)
dBT remains the most accurate method.
In Table 3, we report mean absolute rank errors with MCSEs. These results show that dBT remains the most accurate method in this setting: with the penalty tuned by chronological cross-validation or by REML, it has the smallest mean rank error in 11 of 17 scenarios, and the REML variant with flat continuation has smaller error than KickScore in 12 scenarios and smaller error than the cross-validated dBT fit in 13. KickScore or kernel BT is best in the remaining scenarios, including both 40% absence designs. Unlike a gap, the period after the last observation is constrained from one side only, so the extrapolated value depends on how the curve is continued.
Table 3. Past-only current ranking: mean absolute rank error, mean (MCSE), . REML variants use linear or flat continuation.
| Scenario | Static BT | dBT | Kernel BT | KickScore | dBT REML Linear | dBT REML Flat |
|---|---|---|---|---|---|---|
| Complete exposure | 2.1325 (0.003) | 0.7118 (0.009) | 0.7375 (0.009) | 0.7143 (0.008) | 0.6490 (0.008) | 0.6258 (0.008) |
| 10% absences | 2.1322 (0.003) | 0.7528 (0.010) | 0.8158 (0.009) | 0.7975 (0.008) | 0.6875 (0.009) | 0.6688 (0.009) |
| 25% absences | 2.0072 (0.005) | 0.8383 (0.010) | 0.9609 (0.010) | 0.8623 (0.008) | 0.7543 (0.009) | 0.7405 (0.008) |
| 40% absences | 1.8340 (0.004) | 1.2560 (0.010) | 1.1676 (0.009) | 0.9760 (0.006) | 1.1867 (0.007) | 1.1783 (0.007) |
| 90% weight reduction | 2.0992 (0.005) | 0.7915 (0.010) | 0.8545 (0.010) | 0.7925 (0.009) | 0.7130 (0.009) | 0.6928 (0.008) |
| 98% weight reduction | 2.0260 (0.005) | 0.8318 (0.010) | 0.9402 (0.011) | 0.8352 (0.008) | 0.7433 (0.009) | 0.7313 (0.009) |
| 20 comparisons/time | 2.0270 (0.007) | 1.3233 (0.013) | 1.3483 (0.012) | 1.2432 (0.011) | 1.2145 (0.011) | 1.1990 (0.011) |
| 50 comparisons/time | 2.0067 (0.005) | 1.0475 (0.011) | 1.1034 (0.011) | 0.9870 (0.008) | 0.9298 (0.009) | 0.9228 (0.009) |
| Unequal exposure: 25% gaps | 1.9595 (0.004) | 0.9898 (0.011) | 1.1019 (0.010) | 0.9598 (0.008) | 0.8633 (0.009) | 0.8582 (0.009) |
| Unequal exposure: 40% gaps | 1.9098 (0.004) | 1.3232 (0.010) | 1.2639 (0.010) | 1.0723 (0.007) | 1.2447 (0.008) | 1.2337 (0.008) |
| Missing bridging edges | 2.1323 (0.003) | 1.0852 (0.012) | 0.7695 (0.010) | 0.8310 (0.010) | 1.0078 (0.012) | 0.9865 (0.012) |
| Crossing curves: 25% gaps | 1.7383 (0.004) | 0.7658 (0.007) | 0.9742 (0.009) | 0.8595 (0.007) | 0.6993 (0.006) | 0.7298 (0.006) |
| Informative, balanced | 2.8770 (0.013) | 1.2373 (0.013) | 2.3860 (0.015) | 1.5678 (0.013) | 1.2770 (0.012) | 1.2675 (0.012) |
| Informative, context gap | 3.5078 (0.008) | 1.7280 (0.017) | 2.7970 (0.010) | 2.5745 (0.013) | 1.7252 (0.017) | 1.7570 (0.017) |
| Null effect, context gap | 2.8583 (0.007) | 2.0243 (0.010) | 1.5712 (0.016) | 1.7070 (0.016) | 2.0140 (0.009) | 2.0085 (0.009) |
| Three informative features | 2.2408 (0.007) | 1.5163 (0.014) | 1.4515 (0.015) | 1.5272 (0.016) | 1.5285 (0.013) | 1.5208 (0.013) |
| Three informative + three irrelevant | 2.2407 (0.007) | 1.5702 (0.013) | 1.4515 (0.015) | 1.4463 (0.016) | 1.6035 (0.013) | 1.5918 (0.014) |
Manuscript revisions. Manuscript Section 4 adds a paragraph on current rankings from past observations; Appendix C.10 reports the evaluation design and Table 3 as manuscript Table 6; Appendix A.5 describes REML selection and the continuation of each curve beyond the last observation.
Thank you again for your time and for the detailed review. Your questions prompted the comparison with KickScore and the evaluation of current rankings from past observations, and we believe the paper is better for them. If any point remains unclear, please let us know and we will reply quickly. If our response addresses your concerns, we would be grateful if you would consider raising your score.
Response to Reviewer ub7V (part 4 of 4)
References
- H. Bong, W. Li, S. Shrotriya, and A. Rinaldo (2020). Nonparametric estimation in the dynamic Bradley–Terry model. Proceedings of AISTATS 2020, PMLR 108, 3317–3326.
- E. Karlé and H. Tyagi (2023). Dynamic ranking with the BTL model: a nearest neighbor based rank centrality method. Journal of Machine Learning Research, 24(269), 1–57.
- X. Tian, J. Shi, X. Shen, and K. Song (2024). A spectral approach for the dynamic Bradley–Terry model. Stat, 13(3), e722.
- F. O'Sullivan, B. S. Yandell, and W. J. Raynor, Jr. (1986). Automatic smoothing of regression functions in generalized linear models. Journal of the American Statistical Association, 81(393), 96–103. doi:10.1080/01621459.1986.10478243.
- S. N. Wood (2017). Generalized Additive Models: An Introduction with R, second edition. Chapman and Hall/CRC. Author's book page.
- J. Z. Huang and Y. Su (2021). Asymptotic properties of penalized spline estimators in concave extended linear models: Rates of convergence. The Annals of Statistics, 49(6), 3383–3407. doi:10.1214/21-AOS2088.
- L. Maystre, V. Kristof, and M. Grossglauser (2019). Pairwise Comparisons with Flexible Time-Dynamics. Proceedings of KDD 2019. doi:10.1145/3292500.3330831.
Response to Reviewer ub7V (part 1 of 4)
We thank the reviewer for a careful review; addressing these comments has sharpened the paper's claims and strengthened its evidence. The revised manuscript marks every change in blue, and each response below ends with the location of the corresponding revision.
Comments ub7V.4 and ub7V.6: What is new beyond existing dynamic BT models and spline theory.
Our main contributions beyond existing dynamic BT models and penalized spline theory are the following.
What the theory establishes.
We build on penalized likelihood and spline smoothing (O'Sullivan et al., 1986; Wood, 2017; Huang and Su, 2021). We study how comparison times and item pairings jointly determine identification and estimation error. For centered ability curves, we write A(h) for integrated squared ability error and Pn(h) for the corresponding error in pairwise contrasts, weighted by the comparison design. We define
\frac{\mathcal P_n(h)}{\mathcal A(h)}$$ to relate these two errors on the fitted trajectory space $\mathcal B_n$. We show that identification is equivalent to positive definiteness of the corresponding joint-design Gram matrix. This explains when we can combine information across time to estimate trajectories despite gaps in comparisons at individual times. Under the manuscript's i.i.d. random-design assumptions, fixed item count, deterministic basis and penalty sequences, bounded approximant contrasts, finite leverage, and localization condition $\ell_n^2w_n\to0$, we obtain $$\begin{aligned}\mathcal P_n(\widehat\beta-\beta^0)&=O_p(w_n),\\ \mathcal A(\widehat\beta-\beta^0)&=O_p(a_n^2+w_n/\kappa_n),\\ \mathcal E_n&=O_p(w_n),\end{aligned}$$ where $\mathcal E_n$ is same-design excess log loss, $a_n^2$ is trajectory approximation error, and $w_n=\widetilde a_n^2+d_n/n+\rho_nJ_{M_n}(\beta_n^*)$, with $d_n=(J-1)M_n$. These bounds show that the trajectory-error bound increases as $\kappa_n$ decreases, while the prediction bound depends on $w_n$. For example, for twice-differentiable abilities with $\kappa_n\gtrsim M_n^{-1}$, the manuscript's polynomial-design corollary gives a trajectory squared-error bound of $O_p(n^{-3/5})$ and an excess log-loss bound of $O_p(n^{-4/5})$. In Appendix B.7, we give examples of different trajectories inside a gap that produce the same probabilities for the observed comparisons. **Manuscript revisions.** The manuscript's Introduction now states these contributions, namely identification from the joint design, the design constant separating recovery from prediction, and time-varying covariate effects, and contrasts dBT with local-window methods and with penalized spline regression. **Comments ub7V.5 and ub7V.7: dBT is more accurate than KickScore, an existing dynamic BT model, on the same data.** We compared static BT, dBT, kernel BT and KickScore (Maystre et al., 2019) on the same 500 saved datasets in each of the 17 original simulation scenarios and on the same 15 Arena and 75 T20 holdouts, with identical observations, holdouts and recovery targets. We selected KickScore's covariance structure and parameters among 128 configurations by maximizing the approximate marginal likelihood on each training set, following the random-search design of Maystre et al. (2019): constant or affine components combined with Wiener, exponential or Matérn processes, with parameter ranges set for the $[0,1]$ time scale and fixed before any fitting. In contextual scenarios we fitted independent Gaussian processes for each item's baseline and covariate coefficients using the same covariates as dBT.