- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a framework for operating Game Arena continuously. It combines uncertainty-based match scheduling, an aggregate ranking that gives equal total weight to each game, and capability profiles based on expert annotations of game demands. The deployment evaluates 23 models across 10 games using 64,251 matches. The main practical claim is that this design can support ongoing evaluation while reducing the number of required matches.
Critical strengths
-
The paper addresses an important operational problem. Model additions, evaluation costs, and comparability across games are closely connected in a live benchmark. Bringing them into a single system is useful, and the deployment is substantial enough to expose practical issues that a small simulation would miss.
-
The ranking design remains inspectable. Per-game results and uncertainty estimates accompany the aggregate ranking, while the capability profiles are described as relative summaries. This gives readers more information than a single leaderboard score and makes the intended interpretation of the outputs reasonably clear.
Critical weaknesses
-
The evidence does not yet establish reliable evaluation at a lower budget or across refreshes. The 60.3% saving compares 64,251 completed matches with a hypothetical 161,920 matches at the stated caps. This is a valid workload comparison, but there is no corresponding estimate of ranking error, top-k stability, or coverage against a reference evaluation or uniform sampling at the same budget. The experiments also show one aggregate snapshot rather than an actual sequence of model additions with incremental costs and ranking changes. These omissions matter because efficient continuous evaluation is the central contribution. A bounded comparison using existing logs, with explicit limits from unobserved matches, and a documented refresh example would be more informative than additional aggregate match counts.
-
The uncertainty calculations need to respect the dependence in the data. Appendix A.4 expands a homogeneous k-versus-k episode into k² identical cross-team records for the scheduler's confidence fit. Those records share the same episode outcome. The paper acknowledges that the resulting q is a scheduling signal, but its scale affects when sampling is reduced, so calibration against ranking error is important. Separately, the ULB bootstrap resamples within ordered-model-pair buckets, placing the two mirrored directions in different buckets. This does not preserve their shared environment seed. The ULB input may collapse team episodes, so the k² expansion should not automatically be attributed to that bootstrap. The actual units need to be stated for each stage, followed by a sensitivity check using whole episodes and mirrored seed groups where appropriate. The direction or magnitude of any interval error cannot be inferred from dependence alone.
Questions for authors
- What ranking accuracy or stability is retained at the reported budget saving, relative to a defined reference and uniform sampling? Can existing records support a limited comparison without assuming outcomes for unobserved matches?
- What enters the scheduler fit and ULB bootstrap: episodes, expanded player-pair records, or collapsed team outcomes? How sensitive are stopping decisions and intervals to preserving episode and mirrored-seed groups?
- Can you document one actual refresh, including the added models, incremental matches, ranking changes, and rules for reusing earlier records?
4 comments
Response to Reviewer zrB8 (2/2)
W2 & Q2: Exact Statistical Units Across Scheduler Fit, ULB Fit, and Bootstrap
Auditing both codebases clarified the exact unit at each stage, now stated explicitly in §4.1, Appendix B.1, and Appendix J (Table J.1, Figure J.1):
- Within-game scheduler (): Expanded each homogeneous team episode into cross-team records, inflating -scores in Word Association and Coin Game. Recomputing on the live logs with team outcome per episode moves 38/624 and 55/708 scheduled pairs below 0.95 (Figure J.1); in 25 simulated scheduler runs per game on the fitted ratings, the rule lowers savings from 79.6% to 69.2% and from 75.6% to 62.8% and lowers the false-sign rate among pairs with from 1.68% to 0.26% and from 0.43% to 0.03% (mirrored seeds are paired operationally across and with unpaired orientations kept eligible for mirror repair regardless of , Appendix A.4, while seed-clustered was not separately simulated).
- ULB point fit (): Already collapsed each episode to whole-episode team outcome (and makes point estimates invariant to constant per-game multiplicity).
- ULB bootstrap intervals (, Table J.1): Resampling whole episodes gives a mean marginal 95% CI width of 13.82 Elo (691/741 separated pairs, versus 12.65 Elo under unclustered records). Preserving shared mirror seeds as clusters keeps the 13.82 Elo mean marginal width (13.32 median; marginal medians vary across runs by Monte Carlo error) and narrows mean pairwise width to 19.69 vs. 19.86 Elo (693/741 separated pairs) due to variance reduction across paired Bargaining ( width) and Poker ( width) seeds.
W1 & Q3: Documentation of One Live Refresh () and Record-Reuse Rules
In §6.2 and Appendix I (Tables I.2–I.5, Figures I.1–I.3), we document the refresh from the submitted snapshot (23 models; 64,251 episodes at the initial release cut, 64,257 in the archived CSV export) to (39 active models, 147,015 completed episodes):
- Reuse and versioning rules: Unchanged submissions keep their historical records (28,573 reused episodes, USD 367.2k; 162 Word Association episodes among these models are not in the active set, 148 of which involve Claude Opus 5, which is not active in that game in ); existing pairs receive new matches only when (e.g., if a refit lowers or the pair remained below target at the cut) or to complete an unpaired mirror seed (§3.1, Appendix A.4; 2,081 re-check episodes = 1.76% of post- play, USD 11.7k). Eight models received unified-harness upgrades and entered as new submission IDs (34,725 episodes), replacing their 35,522 pre-upgrade episodes ( active episodes; Table I.2).
- Incremental matches and ranking changes: Onboarding the 16 new models required 81,636 episodes (USD 639.1k) versus 223,160 (USD 1.806M) under uniform incremental caps (63.4% episode and 64.6% USD savings), while cohort-mean-anchored ratings across the 15 unchanged models preserve Kendall's (, 2/105 pair flips, maximum rank shift 1; Table I.3, Table I.5).
Dear reviewer zrB8, thank you again for your careful and constructive reviews. We have posted responses to each of your questions and uploaded a revised manuscript that addresses the points you raised. Please let us know if you have any remaining concerns.
Thank you for the detailed responses. The Chess replay answers my main question about the reported savings. Since the full tournament had already been played, there is a reference ranking to compare against. It is helpful to see that the scheduler recovered the same five models at the top with about a quarter of the games. The example of expanding from 23 to 39 models also shows how earlier matches were reused in practice.
These results address the two concerns that mattered most to my original assessment. I therefore raise my score to 3.
Response to Reviewer zrB8 (1/2)
We thank Reviewer zrB8 for the careful review and questions on unobserved matches, statistical dependence units, and refresh documentation.
All three points are addressed in §4.1, §6.1–§6.3, Appendix B.1, and Appendices H–J and L.
W1 & Q1: Retained Ranking Accuracy, Calibration, and Limits of Existing Logs
In the live S0/S1 logs, lopsided pairs stop after one mirrored round (2 episodes) and unplayed matches are unobserved, so subsampling those logs uniformly across all pairs without assuming unobserved outcomes cannot extend past the initial warm-up round. Comparing round-robin and adaptive scheduling on real records without outcome assumptions therefore requires a tournament logged to the full cap. Because completing full round-robins across all ten games would cost an additional USD 2.369M (Table I.1), we use a complete recorded Chess round-robin (Appendix L), together with controlled ground-truth simulations for N∈{10,20,30,40,50} models (§6.1, Table 3, Appendix H [Tables H.1–H.3, Figures H.1–H.5]) and allocation checks on the 147,015 live episodes: