acceptodds
Under review as a conference paper at ICLR 2027

Continuous Evaluation of LLM Agents in Competitive Environments

Abstract

Games are a robust test for large language model agents: they require sequential decisions, adversarial or cooperative interaction, state tracking, reasoning under uncertainty, and long-horizon planning. However, a one-time game benchmark can go stale as models, prompts, environment implementations, and evaluation policies change, and refreshing it is costly: full round-robin evaluation scales poorly in the number of model pairs, and the resulting outcomes live on incompatible per-game scales. We present a framework for continuous evaluation of LLM agents in game settings, organized around three coupled components. First, an adaptive scheduler that maps pairwise uncertainty to reduced per-pair match targets, concentrating budget on comparisons that remain unresolved. Second, a Unified Leaderboard (ULB) that reduces heterogeneous outcomes to pairwise evidence and fits a game-balanced model by minorization- maximization with bootstrap intervals, so that high-throughput games cannot dominate the aggregate ranking. Third, a game skill-profile methodology that scores each game along diverse skill dimensions, making the capability coverage of the suite explicit and turning game selection into a design decision rather than an artifact of availability. In this paper, we specify the research design and outcome reduction of the current implementation and evaluate the framework on a ten-game suite spanning perfect-information board games, imperfect information games, and social/language settings, reporting LLM capability profiles, evaluation-cost savings across model-refresh cycles, and an empirical Pareto frontier relating model performance to inference cost.

Primary area:
datasets and benchmarks
Keywords:
game evaluation, LLM agents, Bradley–Terry, adaptive scheduling, leaderboard

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_zrB8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a framework for operating Game Arena continuously. It combines uncertainty-based match scheduling, an aggregate ranking that gives equal total weight to each game, and capability profiles based on expert annotations of game demands. The deployment evaluates 23 models across 10 games using 64,251 matches. The main practical claim is that this design can support ongoing evaluation while reducing the number of required matches.

Critical strengths
  1. The paper addresses an important operational problem. Model additions, evaluation costs, and comparability across games are closely connected in a live benchmark. Bringing them into a single system is useful, and the deployment is substantial enough to expose practical issues that a small simulation would miss.

  2. The ranking design remains inspectable. Per-game results and uncertainty estimates accompany the aggregate ranking, while the capability profiles are described as relative summaries. This gives readers more information than a single leaderboard score and makes the intended interpretation of the outputs reasonably clear.

Critical weaknesses
  1. The evidence does not yet establish reliable evaluation at a lower budget or across refreshes. The 60.3% saving compares 64,251 completed matches with a hypothetical 161,920 matches at the stated caps. This is a valid workload comparison, but there is no corresponding estimate of ranking error, top-k stability, or coverage against a reference evaluation or uniform sampling at the same budget. The experiments also show one aggregate snapshot rather than an actual sequence of model additions with incremental costs and ranking changes. These omissions matter because efficient continuous evaluation is the central contribution. A bounded comparison using existing logs, with explicit limits from unobserved matches, and a documented refresh example would be more informative than additional aggregate match counts.

  2. The uncertainty calculations need to respect the dependence in the data. Appendix A.4 expands a homogeneous k-versus-k episode into k² identical cross-team records for the scheduler's confidence fit. Those records share the same episode outcome. The paper acknowledges that the resulting q is a scheduling signal, but its scale affects when sampling is reduced, so calibration against ranking error is important. Separately, the ULB bootstrap resamples within ordered-model-pair buckets, placing the two mirrored directions in different buckets. This does not preserve their shared environment seed. The ULB input may collapse team episodes, so the k² expansion should not automatically be attributed to that bootstrap. The actual units need to be stated for each stage, followed by a sensitivity check using whole episodes and mirrored seed groups where appropriate. The direction or magnitude of any interval error cannot be inferred from dependence alone.

Questions for authors
  1. What ranking accuracy or stability is retained at the reported budget saving, relative to a defined reference and uniform sampling? Can existing records support a limited comparison without assuming outcomes for unobserved matches?
  2. What enters the scheduler fit and ULB bootstrap: episodes, expanded player-pair records, or collapsed team outcomes? How sensitive are stopping decisions and intervals to preserving episode and mirrored-seed groups?
  3. Can you document one actual refresh, including the added models, incremental matches, ranking changes, and rules for reusing earlier records?
4 comments
AuthorsOpenReview ↗

Response to Reviewer zrB8 (1/2)

We thank Reviewer zrB8 for the careful review and questions on unobserved matches, statistical dependence units, and refresh documentation.

All three points are addressed in §4.1, §6.1–§6.3, Appendix B.1, and Appendices H–J and L.

W1 & Q1: Retained Ranking Accuracy, Calibration, and Limits of Existing Logs

In the live S0/S1S_0/S_1 logs, lopsided pairs stop after one mirrored round (2 episodes) and unplayed matches are unobserved, so subsampling those logs uniformly across all pairs without assuming unobserved outcomes cannot extend past the initial warm-up round. Comparing round-robin and adaptive scheduling on real records without outcome assumptions therefore requires a tournament logged to the full cap. Because completing full round-robins across all ten games would cost an additional USD 2.369M (Table I.1), we use a complete recorded Chess round-robin (Appendix L), together with controlled ground-truth simulations for N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace models (§6.1, Table 3, Appendix H [Tables H.1–H.3, Figures H.1–H.5]) and allocation checks on the 147,015 live episodes:

  1. Real Chess round-robin vs. scheduler (Appendix L, Table L.1, Figure L.1): In the complete 18-model, 6,120-episode Chess round-robin, all skipped episodes are observed: replaying the scheduler saves 76.1% of the episodes (1,462 vs. 6,120) while remaining highly correlated with the complete round-robin ranking (ρ=0.9965\rho = 0.9965, τK=0.9780±0.0114\tau_K = 0.9780 \pm 0.0114, and 100.0% Top-5 recovery, versus τK=0.9585±0.0188\tau_K = 0.9585 \pm 0.0188, ρ=0.9926\rho = 0.9926, and 54.5% Top-5 recovery for uniform allocation of the same episodes). Uniform allocation only knows to stop at 23.9% in hindsight from the scheduler's stopping point: without adaptive confidence (qij>0.95q_{ij}>0.95), a fixed round-robin offers no stopping criterion, whereas the scheduler stops automatically, nearly halves remaining pairwise ordering error (1.10% vs. 2.08%; 75.2% vs. 61.1% accuracy on close pairs ≤40\le 40 Elo), has zero sign flips on any pair stopped before the cap, and reaches a τK\tau_K that uniform sampling matches only at 2.5×2.5\times the budget (60% of the complete tournament).
  2. Simulations across N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace models (Tables H.1–H.2, Figures H.1–H.3): Across N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace, episode savings rise from 61.6% to 76.3% while maintaining ranking accuracy: Spearman correlation against true skill stays ρ=0.9744\rho = 0.9744–0.9946 (96.19%–97.71% pairwise accuracy) and direct agreement with Full Round-Robin reaches ρ=0.9928\rho = 0.9928–0.9935 (97.26%–97.30% pairwise agreement) at N=40N = 40–50. Beyond the initial warm-up rounds (≥15\ge 15% of cap) through convergence (Figure H.2), the adaptive scheduler consistently exceeds uniform smaller-rr Round-Robin (r=1,…,20r = 1, \dots, 20) at matched budgets, and retains 69.6%–77.2% savings across five skill, draw, seat-bias, and Poker noise regimes (Table H.3).
  3. Stopping calibration (Figure H.5): Pairs entering the qij>0.95q_{ij} > 0.95 early-stop region achieve 99.87% empirical true-sign accuracy (0.13% false-sign rate), while in the 39-model live logs, closely contested pairs (∣ΔElo∣≤40|\Delta\text{Elo}| \le 40) retain 95.6% of cap 2Tg2T_g versus 15.5% for ∣ΔElo∣≥500|\Delta\text{Elo}| \ge 500 (§6.1, Figure A.1).
AuthorsOpenReview ↗

Response to Reviewer zrB8 (2/2)

W2 & Q2: Exact Statistical Units Across Scheduler Fit, ULB Fit, and Bootstrap

Auditing both codebases clarified the exact unit at each stage, now stated explicitly in §4.1, Appendix B.1, and Appendix J (Table J.1, Figure J.1):

  • Within-game scheduler (qijq_{ij}): Expanded each 2×22\times 2 homogeneous team episode into k2=4k^2 = 4 cross-team records, inflating zz-scores in Word Association and Coin Game. Recomputing qijq_{ij} on the live logs with 1×1\times team outcome per episode moves 38/624 and 55/708 scheduled pairs below 0.95 (Figure J.1); in 25 simulated scheduler runs per game on the fitted ratings, the 1×1\times rule lowers savings from 79.6% to 69.2% and from 75.6% to 62.8% and lowers the false-sign rate among pairs with qij≥0.95q_{ij} \ge 0.95 from 1.68% to 0.26% and from 0.43% to 0.03% (mirrored seeds are paired operationally across (i,j)(i,j) and (j,i)(j,i) with unpaired orientations kept eligible for mirror repair regardless of TijeffT_{ij}^{\mathrm{eff}}, Appendix A.4, while seed-clustered qijq_{ij} was not separately simulated).
  • ULB point fit (R^i\hat R_i): Already collapsed each 2×22\times 2 episode to 1×1\times whole-episode team outcome (and wg=1/Ngw_g = 1/N_g makes point estimates invariant to constant per-game multiplicity).
  • ULB bootstrap intervals (B=500B = 500, Table J.1): Resampling 1×1\times whole episodes gives a mean marginal 95% CI width of 13.82 Elo (691/741 separated pairs, versus 12.65 Elo under unclustered 4×4\times records). Preserving shared mirror seeds as clusters keeps the 13.82 Elo mean marginal width (13.32 median; marginal medians vary across B=500B = 500 runs by Monte Carlo error) and narrows mean pairwise ΔElo\Delta\text{Elo} width to 19.69 vs. 19.86 Elo (693/741 separated pairs) due to variance reduction across paired Bargaining (0.883×0.883\times width) and Poker (0.900×0.900\times width) seeds.

W1 & Q3: Documentation of One Live Refresh (S0→S1S_0 \to S_1) and Record-Reuse Rules

In §6.2 and Appendix I (Tables I.2–I.5, Figures I.1–I.3), we document the refresh from the submitted snapshot S0S_0 (23 models; 64,251 episodes at the initial release cut, 64,257 in the archived CSV export) to S1S_1 (39 active models, 147,015 completed episodes):

  • Reuse and versioning rules: Unchanged submissions keep their historical records (28,573 reused S0S_0 episodes, USD 367.2k; 162 Word Association episodes among these models are not in the active S1S_1 set, 148 of which involve Claude Opus 5, which is not active in that game in S1S_1); existing pairs receive new matches only when Tijeff>cijT_{ij}^{\mathrm{eff}} > c_{ij} (e.g., if a refit lowers qijq_{ij} or the pair remained below target at the S0S_0 cut) or to complete an unpaired mirror seed (§3.1, Appendix A.4; 2,081 re-check episodes = 1.76% of post-S0S_0 play, USD 11.7k). Eight S0S_0 models received unified-harness upgrades and entered as new submission IDs (34,725 episodes), replacing their 35,522 pre-upgrade episodes (28,573+2,081+34,725+81,636=147,01528{,}573 + 2{,}081 + 34{,}725 + 81{,}636 = 147{,}015 active S1S_1 episodes; Table I.2).
  • Incremental matches and ranking changes: Onboarding the 16 new models required 81,636 episodes (USD 639.1k) versus 223,160 (USD 1.806M) under uniform incremental caps (63.4% episode and 64.6% USD savings), while cohort-mean-anchored ratings across the 15 unchanged S0S_0 models preserve Kendall's τK=0.9619\tau_K = 0.9619 (ρ=0.9929\rho = 0.9929, 2/105 pair flips, maximum rank shift 1; Table I.3, Table I.5).

Dear reviewer zrB8, thank you again for your careful and constructive reviews. We have posted responses to each of your questions and uploaded a revised manuscript that addresses the points you raised. Please let us know if you have any remaining concerns.

Thank you for the detailed responses. The Chess replay answers my main question about the reported savings. Since the full tournament had already been played, there is a reference ranking to compare against. It is helpful to see that the scheduler recovered the same five models at the top with about a quarter of the games. The example of expanding from 23 to 39 models also shows how earlier matches were reused in practice.

These results address the two concerns that mattered most to my original assessment. I therefore raise my score to 3.

Reviewer_pUN1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a framework for continuously evaluating and ranking LLMs in game playing. This framework combines adaptive match scheduling, game-balanced Bradley-Terry rankings, and expert-informed skill profiles. It examines 23 models across 10 games and 64251 episodes, reporting 60.3% fewer episodes than uniform allocation at the configured match limits. It also presents overall and per-game rankings, capability profiles, and cost-performance comparisons for those models.

Critical strengths

This is a practical framework for sustainable and maintainable evaluation. This framework provides an efficient, reusable design to keep arena-style leaderboards up to date. New models can be added and recomputed from past match results at a lower cost without rerunning the full comparison. This keeps evaluation costs manageable as more models join the arena. If maintained well, this platform could provide an efficient way to add new models to the leaderboard, which is especially useful as models are updated frequently nowadays.

Critical weaknesses

Insufficient empirical validation of ranking reliability and incremental evaluation. I think the framework's design is quite reasonable, but the experiments don't fully establish that the reduced evaluation budget preserves reliable rankings. This paper does not compare its rankings with a reference leaderboard or a higher-budget evaluation to show the consistency. Similarly, the paper describes how a new model is incorporated, but it does not demonstrate this process or report the resulting trade-off between additional evaluation cost and ranking accuracy. I will consider raising the score if the author can fully address my concerns.

Questions for authors
  1. Could you compare some of your game rankings with existing reference leaderboards? Please report ranking alignment and evaluation cost for some shared models to show whether your method achieves similar rankings at lower cost or with fewer matches.
  2. Please demonstrate how a "new model" is added through a controlled experiment. For example, select one or two models (such as GPT-5.6-Luna, there is no need to select expensive models), refit the initial state using the remaining models and demonstrate how this model is added using the proposed procedure. Then, compare the results (scores and rank) with the original full-model evaluation.
  3. How sensitive is the overall ranking to replacing equal-per-game weighting with equal-per-category weighting?
  4. How are the 15 game experts selected, and what expertise do they have in the evaluated games?
5 comments
AuthorsOpenReview ↗

Response to Reviewer pUN1 (1/2)

We thank Reviewer pUN1 for the constructive feedback. Each concern and question is addressed in §5, §6.1–§6.3, and Appendices G, H, I, J, and L.

W1 & Q1: Reference Ranking Alignment and Evaluation Cost (Episodes and Dollars)

  1. Reference ranking alignment (§6.1, Table 3, Appendices H and L): External game leaderboards use different prompting harnesses, move parsers, retry limits, and game subsets, which confound harness differences with scheduler accuracy. To test ranking alignment cleanly without harness confounding, we compare the adaptive scheduler directly against Full Round-Robin. On real gameplay data, due to API budget constraints (completing full round-robins across the ten-game suite would cost an additional USD 2.369M), we use a complete round-robin from an earlier Chess tournament (18 models, 6,120 episodes; Appendix L, Table L.1): replaying the scheduler saves 76.1% of episodes while producing rankings highly correlated with the complete round-robin (ρ=0.9965\rho = 0.9965, τK=0.9780±0.0114\tau_K = 0.9780 \pm 0.0114, 100.0% Top-5 recovery, versus τK=0.9585±0.0188\tau_K = 0.9585 \pm 0.0188 and 54.5% Top-5 recovery for uniform allocation of the same episodes, which only knows to stop at 23.9% in hindsight from the scheduler's stopping point, has nearly twice the pairwise ordering error [2.08% vs. 1.10%], and requires 2.5×2.5\times the budget [60% of the tournament] to match the scheduler's τK\tau_K). We then run controlled simulations across N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace models (Table 3, Table H.2, Figures H.1–H.3) to verify across pool sizes that the scheduler saves episodes while maintaining ranking accuracy: episode savings grow from 61.6% to 76.3%, direct Spearman correlation with Full Round-Robin reaches ρ=0.9928\rho = 0.9928 (97.26% pairwise agreement) at N=40N = 40 and ρ=0.9935\rho = 0.9935 (97.30%) at N=50N = 50, and the scheduler outperforms smaller fixed-target Round-Robin beyond the initial warm-up rounds (≥15\ge 15% of cap) across every NN (Figure H.2).
  2. Evaluation cost in episodes and API dollars (§6.1–§6.2, Table 2, Tables I.1–I.2): Across the 39-model suite, adaptive scheduling used 147,015 episodes (USD 1.438M) versus 384,200 (USD 3.807M) under uniform caps over the pairs that were scheduled, saving 61.7% of episodes and 62.2% of API dollars (USD 2.369M saved; 100-hand Poker accounts for 72.5% of actual spend, USD 34.82/played episode and USD 35.51/episode at cap, where adaptive stopping saves USD 1.642M). Across the live 23→3923 \to 39 refresh (Table I.2), onboarding the 16 new models cost USD 639.1k versus USD 1.806M under uniform incremental caps (64.6% USD saved, USD 1.167M saved). These baselines exclude the 293 pair–game combinations that were never scheduled because one of the two models had been retired (Appendix I); counting them at their caps would raise the episode saving to 63.1%.

W1 & Q2: Controlled Experiment Adding a Single Model (Including GPT-5.6 Luna)

We ran the requested hold-out insertion experiment as a simulation across all 39 models (§6.2, Appendix I, Figure I.3), alongside our live 16-model refresh (Table I.2). Because matches that the scheduler never requested have no logged outcomes, each model is removed and re-inserted against the remaining pool with outcomes drawn from the per-game ratings fitted on all data (12 runs per model):

  • GPT-5.6 Luna hold-out insertion: Inserting GPT-5.6 Luna (#33, 104.1 Elo) into the remaining 38-model pool via the adaptive scheduler requires 6,006 episodes versus 22,040 at full cap (72.7% episode savings) and ends 3.67 ranks from its full-data rank on average (vs. 6.00 for equal-budget uniform allocation; adjacent neighbors are within 5 Elo) and +22+22 Elo above its full-data unified score of 104.1 Elo (Appendix I).
  • Scores and ranks across all 39 models: Single-model insertions require a mean of 5,264 episodes (74.4% mean savings; range 68.2%–90.1%, e.g., 1,641 episodes / 90.1% saved for #1 Gemini 4 Carbon and 3,136 episodes / 81.2% saved for #3 Claude Fable 5.2) and end a mean of 1.31 ranks from the full-data rank, versus 2.74 for matched-budget uniform allocation; the adaptive rule is closer for 26 of the 39 models and uniform allocation is closer for 11, including the top-ranked and bottom-ranked models. When sampled outcomes are replaced by expected scores under the same allocations, the rank distances are 1.29 and 2.72, so the difference comes from episode allocation, not sampling noise. Concentrating play on close opponents (stopping after 2 episodes on 98% of ≥400\ge 400-Elo gaps vs. 6.9 in live play) also pulls inserted unified Elo scores toward the pool center (roughly 0.76×0.76\times full-data Elo +44+ 44 Elo; 61–71 Elo below full-data scores for the top three models and 14 Elo below for GPT-6.1 Sol; Figure I.3, Appendix I).
AuthorsOpenReview ↗

Response to Reviewer pUN1 (2/2)

Q3: Sensitivity to Equal-Per-Category Weighting

In §6.3 and Appendix J (Table J.2, Figure J.2), reweighting the ten games so the three Table 1 categories (5 perfect-information, 3 imperfect-information, and 2 team games) each contribute equal 1/3 weight yields Kendall's τK=0.9190\tau_K = 0.9190 and Spearman's ρ=0.9883\rho = 0.9883 against equal-per-game ULB across all 39 models (τK=0.9368\tau_K = 0.9368 on S0S_0), with 5/5 Top-5 overlap (Gemini 4 Carbon, Gemini 4 Argon, Claude Fable 5.2, GPT-5.5, and GPT-6 Astra).

Q4: Selection and Expertise of the 15 Game Specialists

We added full rater provenance and per-game coverage details to §5 and Appendix G (Table G.3):

  • Selection and independence: All N=15N = 15 raters were AI research scientists (either specializing in game research or game enthusiasts), recruited as volunteers from dedicated game-research and game-enthusiasts groups, and none of the 15 raters are authors of this paper (their individual identities and institutional affiliation remain anonymous to comply with the double-blind review requirement).
  • Coverage and reliability: For each game, a rater could mark the game as unknown and skip it; otherwise the rater rated the six constructs and self-reported familiarity (0–5; one of the 14 Four in a Row raters left familiarity blank). This left ng∈[3,15]n_g \in [3, 15] raters per game (median 8.5: 15 for Chess, Checkers, and Poker; 14 for Four in a Row; 9 for Word Association; 8 for Dots and Boxes; 7 for Bargaining; 4 for Dark Hex; and 3 for Lines of Action and Coin Game), with mean familiarity 2.67–4.33 among them and overall ordinal Krippendorff's αord=0.745\alpha_{\mathrm{ord}} = 0.745 (95% rater-bootstrap CI [0.725,0.824][0.725, 0.824]; squared-difference α=0.766\alpha = 0.766).

Dear reviewer pUN1, thank you again for your careful and constructive reviews! We have posted responses to each of your questions and uploaded a revised manuscript that addresses the points you raised. Please let us know if you have any remaining concerns.

Reviewer_PF8DOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper proposes a Bradley-Terry based benchmarking framework for continuously evaluating LLM agents on a collection of games where new models are added and updated over time. The authors argue that one-off game benchmarks go stale and that fixes such as rerunning a full round-robin tournament whenever a model is added is prohibitively expensive. The framework has three components, the scheduler, the unified leaderboard, and skill profiles. The scheduler is responsible for selecting which games to play and which model pairs to evaluate, the unified leaderboard aggregates results across games, and the skill profiles provide a multidimensional view of each model's performance. The framework is evaluated on 23 models and 10 games, reporting 60.3% fewer episodes than a uniform scheduler baseline.

Critical strengths

S1: The overall idea seems promising and could be useful for evaluating LLMs in a more dynamic way than existing approaches. Reading between the lines, optimizing the selection of matches to play to reduce evaluation cost is a clear strength but this is not clearly argued and is not directly stated as a contribution.

Critical weaknesses

W1: Many concepts are introduced without sufficient explanation, motivation or definition. The overall concept of what is being proposed is vague. The writing is not clear in places. In some places I get the impression that a methodology is being proposed but in others it seems like a particular implementation is being described (e.g. the four-minute update interval) and this is not clearly separated. Overall the paper is held back by not presenting a clear consistent narrative.

W2: The proposed benchmark methodology is not directly compared to existing approaches, and the issues with existing approaches and how the proposed approach addresses them are not clearly demonstrated with experiments. For example, the savings are only measured against filling every pair to a fixed cap, with no check that the ranking is preserved. The key research questions should be more clearly defined and the contributions should target those questions rather than being a list of features of the proposed system. Also the dynamics of the new system are not fully explored, e.g. how the system behaves when new models are added, or when models are updated.

Questions for authors

Q1: What are the key differences between LLM and human rating systems? Humans also change skill over time, different players may join at different times, and there are even more complex dynamic interactions where players may selectively participate in games. These have been studied in the literature (e.g. "Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength") or in the games industry and many have been applied to LLMs. What do the authors see as the key differences that make existing approaches that seek to address Elo/BT issues for human players unsuitable for LLMs? Is it the combination of multiple games?

Q2: Why is the update invoked every 4 minutes? Is this a hyperparameter that was tuned or is there a theoretical reason for this choice? Is this arbitrary? Was this choice evaluated?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PF8D (1/2)

We thank Reviewer PF8D for the comments and questions on dynamic rating systems, update intervals, and refresh behavior.

Clarifications on Scope, Cost Motivation, and Methodology vs. Deployment Settings (S1, W1)

  1. Within-game pair scheduling vs. game selection: The scheduler does not choose which games to play; each of the ten games is scheduled separately, and Algorithm 1 allocates episodes across the model pairs {i,j}\lbrace i, j \rbrace active in that game (§3, first sentence).
  2. Cost reduction and Contribution 2: Reducing evaluation cost across refreshes is stated in the Abstract, the scheduling rule that achieves it is Contribution 2 in §1, and the reduction is now quantified in both episodes and token-level API dollars across the 39-model suite (USD 2.369M saved; §6.1, Table I.1) and the live 23→3923 \to 39 refresh (USD 1.167M saved on onboarding; Table I.2).
  3. Methodology and deployment settings (W1): We revised §3 and Appendix A.1 to separate the methodology from deployment-specific operational settings. §3–§5 define the methodology: the scheduling rule (pairwise confidence, cubic target reduction, and periodic batch updates with request limit BB in Algorithm 1), the Unified Leaderboard, and the skill-profile construction. Operational settings of the current deployment are isolated in Appendix A.1 (the four-minute invocation period, how throughput limit BB is bounded from submission counts, daily run rate, and concurrent capacity, submission/episode eligibility, request ordering, and capacity/health filters), which notes that the invocation interval is an operational feature, not a model parameter, and does not alter the confidence estimate or target-reduction rule (see Q2).

W2: System Behavior Across Model Additions and Ranking Preservation

Section 1 poses the core operational and statistical questions (match allocation, cross-game aggregation, refresh absorption, and capability coverage) and introduces Contributions 1–4 (§2–§5) to address them; §6.1–§6.4 then report the corresponding results (scheduler, refresh, Unified Leaderboard, skill profiles). In §6.2–§6.3 (Table 4, Figure 2) and Appendix I (Tables I.2–I.5, Figures I.1–I.3), we document the live expansion from 23 to 39 models across 6 post-S0S_0 arrival waves (Waves 3–8 of the 8 chronological waves; 63.4% onboarding episode savings and τK=0.9619\tau_K = 0.9619 rank stability on unchanged models; Table I.4), alongside multi-generation model updates across provider families (Figure I.2) and 8 models re-evaluated as new submissions under an updated unified harness (34,725 episodes; Table I.2, Table I.5). To directly check that adaptive stopping preserves round-robin rankings (§6.1, Appendices H and L): due to API budget constraints (completing full round-robins across all ten games would cost an additional USD 2.369M), we use a complete real-game round-robin from an earlier Chess tournament (18 models, 6,120 episodes; Appendix L, Table L.1), where the scheduler saves 76.1% of episodes while producing rankings highly correlated with the complete round-robin (ρ=0.9965\rho = 0.9965, τK=0.9780±0.0114\tau_K = 0.9780 \pm 0.0114, 100.0% Top-5 recovery, vs. τK=0.9585±0.0188\tau_K = 0.9585 \pm 0.0188 and 54.5% Top-5 recovery for matched-budget uniform allocation, which knows to stop at 23.9% only in hindsight and needs 2.5×2.5\times the budget, 60% of the tournament, to match the scheduler's τK\tau_K). Controlled simulations across N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace models (Table 3, Appendix H) likewise confirm 61.6%–76.3% episode savings while maintaining high rank correlation with Full Round-Robin (ρ=0.9681\rho = 0.9681–0.9935).

Q1: Key Differences Between LLM Benchmarks and Human Rating Systems (WHR, Glicko, TrueSkill)

Combining ten games on a balanced scale is one difference (human platforms such as Lichess, Board Game Arena, and Yucata maintain separate per-game ratings; §1). Even within a single game, two differences make human dynamic systems such as Whole-History Rating, Glicko-2, or TrueSkill a poor fit:

  1. Stationary checkpoints vs. drifting human skill: Time-series priors in WHR and rating-deviation inflation in Glicko track human learning or rust over time. Pinned LLM checkpoints have fixed weights (model or harness updates enter as new submissions), and the 15 unchanged models keep nearly the same order from S0S_0 to S1S_1 (τK=0.9619\tau_K = 0.9619, maximum rank shift 1; Table I.3); inflating uncertainty over time would trigger redundant paid API matches on already-resolved pairs.
  2. Active paid scheduling and mirrored seeds: Human ladders passively observe self-selected matches at zero marginal API cost, whereas an LLM benchmark pays per episode (USD 0.56–USD 34.82 per played episode across games, USD 35.51 at cap in Poker; Table I.1), selects informative pairs from C=H−1C = H^{-1}, and pairs Seat-0 and Seat-1 on identical seeds.
AuthorsOpenReview ↗

Response to Reviewer PF8D (2/2)

Response to Reviewer PF8D (part 2 of 2)

Q2: Why a 4-Minute Update Interval?

The 4-minute interval is an operational feature of the platform's job scheduler (how often the handler polls completed matches and refills the request queue; Appendix A.1) and does not impact ranking performance. It is not a tuned hyperparameter and has no theoretical derivation; we did not tune or ablate the live timer because it does not enter the Bradley–Terry objective, the confidence estimate qijq_{ij} (Eq. 2), the cubic target-reduction rule TijeffT_{ij}^{\mathrm{eff}}, or the Unified Leaderboard fit. It is also decoupled from episode duration (§3.1): in-flight Created episodes count immediately toward cijc_{ij} so long games (such as 100-hand Poker) are not double-scheduled, while only Completed episodes enter the fit, and the per-invocation cap BB scales with the configured daily run rate and this period to pace worker throughput (Appendix A.1). In the recorded Chess round-robin replay (Appendix L), refreshing confidence much less frequently (once every three full passes over all pairs) leaves ranking accuracy unchanged (τK=0.9777\tau_K = 0.9777 vs. 0.9780, and vs. 0.9637 for matched-budget uniform allocation) and only modestly lowers episode savings from 76.1% to 68.4% because more episodes are queued between refits.

Thank you for the detailed response. I have updated my score.

Reviewer_nvqdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a framework for continuously evaluating LLM agents in competitive and cooperative games. A per-game Bradley–Terry estimator guides a heuristic reduction in pairwise evaluation targets; a separate game-balanced Bradley–Terry fit produces a unified leaderboard with bootstrap intervals. Expert ratings of game demands are combined with per-game performance to construct six-dimensional model profiles, and normalized inference costs support a performance–cost comparison. Across 23 models and ten games, the paper reports 64,251 episodes, 60.3% fewer than filling every configured pairwise budget, together with aggregate and per-game rankings, skill profiles, and an empirical Pareto frontier.

Critical strengths
  1. The evaluation spans 23 models and ten games with different information structures, roles, and cooperation requirements, and retains detailed per-game results alongside the unified ranking (Tables 1, 3, and E.1–E.10). This breadth makes the work a useful empirical resource and exposes substantial specialization that a single aggregate score would conceal.

  2. The paper specifies important operational and statistical choices, including pending-episode accounting, mirror repair, outcome conversion, equal total game weights, and separate scheduling and reporting estimators (Sections 3–4 and Appendices A–B). Together with expert-rated game-demand profiles, this provides a concrete framework for operating and interpreting an evolving benchmark, while explicitly acknowledging that the scheduling rule is heuristic and the profiles are relative summaries.

Critical weaknesses
  1. The reported 60.3% reduction establishes a smaller episode count relative to configured caps, but does not establish evaluation efficiency at comparable ranking quality: Table 2 defines the uniform baseline as the number of pairs times the cap, without reporting the ranking obtained from that baseline or an independent reference. There is no ranking-error versus budget comparison against uniform/random allocation, or an ablation showing that the confidence-based cubic rule outperforms simply using smaller fixed targets; Figure A.1 shows allocation patterns rather than retained accuracy. Because reliable rankings under a reduced budget are central to the contribution, the current evidence cannot distinguish useful savings from insufficient evaluation, despite the substantial scale of the collected data.

  2. The uncertainty procedures do not preserve the independent sampling units described in the paper: Appendix A.4 states that a homogeneous two-versus-two episode generates four identical records, while Appendix B.5 resamples individual records within ordered-pair buckets and, where mirrors are used, separates the two orientations. This treats duplicated team outcomes as independent bootstrap observations and fails to preserve dependence between episodes sharing a mirror seed; equal total game weights do not fix that resampling problem. The same duplication also increases the likelihood curvature used by the scheduler, so episode/mirror-cluster inference and empirical calibration are needed to establish the reliability of both the reported 95% intervals and the resulting stopping behavior, even though the paper appropriately describes scheduler confidence as a heuristic signal.

Questions for authors
  1. Can you quantify ranking quality at matched budgets against uniform/random allocation and smaller fixed targets, using a sufficiently evaluated reference subset or a controlled simulation with known strengths? Pairwise ordering error or rank agreement across repeated runs would clarify how much of the reported saving survives an accuracy constraint; this is validation of the current efficiency claim rather than a request for additional games.

  2. Can you recompute uncertainty using episodes as clusters, preserving both orientations together when a mirror seed is shared, and compare scheduler behavior when each team episode contributes one outcome? Please report changes in interval widths, relevant pairwise rank differences, and episode savings, distinguishing conditional bootstrap variability from empirically validated coverage under adaptive sampling.

  3. How sensitive is the unified ranking to the within-game comparison distribution induced by the scheduler? Equal total game weights leave frequently sampled pairs with greater influence inside each game, while Tables E.1–E.10 demonstrate strong variation in model performance across games; a comparison with equal-pair weighting or an equal-game average of per-game expected win probabilities would help assess whether the aggregate is robust to scheduling choices.

  4. What empirical evidence supports the claimed behavior across model-refresh cycles? The abstract promises savings across such cycles, but the reported results give one aggregate evaluation; an actual release sequence or a clearly labeled replay showing the incremental cost of adding a model and the stability of existing rankings would substantiate the continuous-evaluation contribution.

6 comments
AuthorsOpenReview ↗

Response to Reviewer nvqd (1/2)

Thank you for the detailed and technically thorough review. We have incorporated the requested simulations, cluster-bootstrap checks, team-scheduler recomputation, aggregation comparisons, and refresh data into §4.1, §6.1–§6.3, Appendix B.1, Appendix D, and Appendices H–J and L.

W1 & Q1: Matched-Budget Ranking Quality vs. Smaller Fixed Targets and Multi-Regime Validation

To compare round-robin and adaptive scheduler efficiency and ranking quality on real gameplay data, all episodes up to the cap must be observed; due to API budget constraints (running full round-robins across all ten games would require an additional USD 2.369M), we use a complete real-game round-robin from an earlier Chess tournament (Appendix L), and we then run controlled ground-truth simulations for N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace models and five skill/noise regimes (§6.1, Table 3; Appendix H, Tables H.1–H.3, Figures H.1–H.5) comparing directly against Round-Robin at every fixed per-pair round target r=1,…,Tr = 1, \dots, T (2T=402T = 40 baseline cap; 2T=1002T = 100 in the Poker regime):

  • Smaller fixed target on real Chess round-robin data (Appendix L, Table L.1, Figure L.1): On the complete 18-model, 6,120-episode Chess round-robin, replaying the scheduler saves 76.1% of episodes (using 23.9% of the tournament) while producing rankings highly correlated with the complete round-robin (ρ=0.9965\rho = 0.9965, τK=0.9780±0.0114\tau_K = 0.9780 \pm 0.0114, 100.0% Top-5 recovery), versus τK=0.9585±0.0188\tau_K = 0.9585 \pm 0.0188 (ρ=0.9926\rho = 0.9926, 54.5% Top-5 recovery) for Round-Robin with a smaller fixed target and the same number of episodes. A smaller fixed-target Round-Robin only knows to stop at 23.9% in hindsight from the scheduler's stopping point: without adaptive confidence (qij>0.95q_{ij}>0.95), there is no stopping criterion across games of differing noise or entrants of differing strength, whereas the scheduler stops automatically, nearly halves remaining pairwise ordering error (1.10% vs. 2.08%; 75.2% vs. 61.1% accuracy on close pairs ≤40\le 40 Elo), and reaches a τK\tau_K that uniform allocation matches only at 2.5×2.5\times the budget (60% of the complete tournament).
  • Matched-budget curves vs. smaller fixed targets (Figure H.2): Plotting Spearman's ρ\rho and Kendall's τK\tau_K against episodes played (% of full cap) directly tests the smaller-fixed-target baseline: beyond the initial warm-up rounds (≥15\ge 15% of cap) through convergence, the adaptive scheduler sits above uniform smaller-rr Round-Robin at matched budgets across all pool sizes N=10…50N = 10\dots 50 (e.g., at N=30N = 30 [28.6% cap], τK=0.9470,ρ=0.9917\tau_K = 0.9470, \rho = 0.9917 vs. equal-budget Round-Robin's τK=0.9330,ρ=0.9880\tau_K = 0.9330, \rho = 0.9880; at N=50N = 50 [23.7% cap], τK=0.9531\tau_K = 0.9531 vs. 0.9441).
  • Scaling and direct Round-Robin agreement (Tables H.1–H.2, Figures H.1, H.3): Episode savings grow from 61.6% (N=10N = 10) to 68.5% (N=20N = 20), 71.4% (N=30N = 30), 74.9% (N=40N = 40), and 76.3% (N=50N = 50). Spearman correlation against true skill stays ρ=0.9744\rho = 0.9744–0.9946 (96.19%–97.71% pairwise ordering accuracy), while direct correlation between the adaptive and Full Round-Robin leaderboards rises from ρ=0.9681\rho = 0.9681 (95.20% pairwise agreement) at N=10N = 10 to ρ=0.9928\rho = 0.9928 (97.26%) at N=40N = 40 and ρ=0.9935\rho = 0.9935 (97.30%) at N=50N = 50.
  • Regime sensitivity (Table H.3, Figure H.4): Repeating the simulation at N=30N = 30 across Gaussian skills (SD=220\mathrm{SD} = 220), wide skill spread (SD=350\mathrm{SD} = 350), a uniform ladder ([−400,+400][-400, +400]), first-mover advantage plus draws (+60+60 Elo Seat-0, ν=0.35\nu = 0.35), and compressed high-noise Poker (SD=110,2T=100\mathrm{SD} = 110, 2T = 100) yields 69.6%–77.2% episode savings and ρ=0.9873\rho = 0.9873–0.9953 direct agreement with Full Round-Robin in all five settings.
AuthorsOpenReview ↗

Response to Reviewer nvqd (2/2)

W2 & Q2: Episode / Mirror-Seed Cluster Bootstrap, 1×1\times Team Scheduler, and Calibration

We audited all three pipeline stages across the 39-model dataset (§4.1, §6.3, Appendix B.1, Appendix D, and Appendix J [Table J.1, Figure J.1]):

  1. ULB point and bootstrap units (Table J.1): Our ULB estimation pipeline already collapsed each 2×22\times 2 team episode to one team outcome (and wg=1/Ngw_g = 1/N_g makes ULB point estimates invariant to per-game multiplicity; we updated §4.1, Appendix B.1, and the sample-size interval discussion in Appendix D accordingly). Comparing five bootstrap resampling schemes (B=500B = 500), resampling 4×4\times cross-product records artificially narrows mean marginal 95% CI width to 12.65 Elo, whereas resampling 1×1\times whole episodes gives 13.82 Elo (691/741 separated pairs). Preserving shared mirror seeds as clusters keeps the 13.82 Elo mean marginal width (13.32 median; marginal medians vary across B=500B = 500 runs by Monte Carlo error) and narrows pairwise ΔElo\Delta\text{Elo} intervals (19.69 vs. 19.86 Elo; 693/741 separated pairs), because playing the same deal in both seat orientations reduces variance in Bargaining (0.883×0.883\times width) and Poker (0.900×0.900\times width).
  2. Scheduler behavior with one outcome per team episode (Appendix B.1, Figure J.1): The deployed within-game scheduler expanded 2×22\times 2 team episodes into four cross-team records, inflating zz-scores by roughly 2×2\times in Word Association and Coin Game. Recomputing qijq_{ij} on the live logs with one outcome per episode moves 38 of the 624 scheduled Word Association pairs and 55 of the 708 scheduled Coin Game pairs below the 0.95 threshold. In 25 simulated scheduler runs per game on the fitted ratings, the one-outcome rule lowers episode savings from 79.6% to 69.2% and from 75.6% to 62.8%, and lowers the false-sign rate among pairs with qij≥0.95q_{ij} \ge 0.95 from 1.68% to 0.26% and from 0.43% to 0.03%.
  3. Stopping calibration vs. conditional bootstrap variability (§7, Appendix B.5, Figure H.5): Across sequential batch updates in the ground-truth simulation, pairs entering the qij>0.95q_{ij} > 0.95 early-stop region have 99.87% empirical true-sign accuracy (0.13% false-sign rate), which checks the scheduler's directional stopping threshold across repeated runs. The 95% ULB bootstrap intervals hold the realized pair counts cijc_{ij} fixed and measure conditional resampling variability given the collected data (§7, Appendix B.5); unconditional coverage under adaptive stopping is not claimed.

Q3: Sensitivity of the Unified Ranking to Within-Game Pair Weighting and Aggregation Rules

In §6.3 and Appendix J (Table J.2, Figure J.2), we compare Game-Balanced ULB across all 39 models (741 pairs) and the 23-model S0S_0 snapshot against alternative weighting rules:

  • Equal-pair weighted BT: Kendall's τK=0.9406\tau_K = 0.9406, Spearman's ρ=0.9909\rho = 0.9909, 4/5 Top-5 overlap (S0S_0: τK=0.9526\tau_K = 0.9526, 5/5).
  • Equal-game average of fitted BT win probabilities: τK=0.9406\tau_K = 0.9406, ρ=0.9925\rho = 0.9925, 4/5 Top-5 overlap (S0S_0: τK=0.9684\tau_K = 0.9684, 5/5).
  • Equal-category BT, equal-game empirical win rate, mean per-game rank, unweighted pooled BT, and leave-one-game-out: τK=0.9190\tau_K = 0.9190, 0.9163, 0.9379, and 0.9460 (τK∈[0.9298,0.9730]\tau_K \in [0.9298, 0.9730] across all 10 single-game drops). The same four models (Gemini 4 Carbon, Gemini 4 Argon, Claude Fable 5.2, and GPT-6 Astra) hold the top four places under every rule except equal-category weighting (GPT-5.5 fourth, GPT-6 Astra fifth) and under all ten single-game drops, while fifth- and sixth-ranked GPT-5.5 (283.6 Elo) and GPT-6.1 Sol (278.4 Elo) sit within 5.2 Elo.

Q4: Empirical Evidence Across Model-Refresh Cycles

In §6.2 and Appendix I (Tables I.2–I.5, Figures I.1–I.3), the expansion from 23 to 39 models across the 6 post-S0S_0 waves (Waves 3–8) reused 28,573 historical episodes among unchanged S0S_0 submissions (USD 367.2k) with 2,081 confidence re-checks (USD 11.7k), onboarded 16 new models in 81,636 episodes (USD 639.1k) vs. 223,160 (USD 1.806M) under uniform incremental caps (63.4% episode and 64.6% USD savings), preserved τK=0.9619\tau_K = 0.9619 (ρ=0.9929\rho = 0.9929, maximum shift 1) among unchanged S0S_0 models, and saved 53.1%–65.6% of the episodes on each wave's new pairs across all 8 chronological waves (Waves 1–2 forming S0S_0 and Waves 3–8 post-S0S_0; Table I.4).

Dear reviewer nvqd, thank you again for your careful and constructive reviews. We have posted responses to each of your questions and uploaded a revised manuscript that addresses the points you raised. Please let us know if you have any remaining concerns.

Reviewer_catoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework for continuously evaluating LLM agents across competitive and cooperative games. It combines adaptive match scheduling, a game-balanced BT leaderboard, and skill profile based on expert ratings of game demands. Experiments cover 23 models across ten games. The scheduler uses fewer episodes than uniform allocation at predefined match budgets. The paper also report per-game results and a cost–performance Pareto frontier, aiming to make repeated evaluations more affordable and model comparisons easier to interpret.

Critical strengths

The paper addresses a practical problem: keeping game-based evaluation useful as models change. It considers operational details such as pending matches and role swaps. Giving each game equal total weight prevents high-volume games from dominating the leaderboard, while retaining separate per-game results. Evaluating 23 models across 10 games provides broad empirical coverage.

Critical weaknesses

The reported 60.3% reduction is relative to filling predefined match budgets. The paper does not show whether the reduced evaluation preserves the ranking, or whether uniform sampling with the same smaller budget would work equally well. Continuous evaluation is central to the paper, but the experiments mainly present one aggregate snapshot. They do not demonstrate the cost and ranking stability across successive model updates. The skill profiles are potentially useful, but Section 6.3 mostly describes the plots. It offers little analysis of what they reveal beyond the per-game leaderboards.

Questions for authors

How does scheduler compare with uniform sampling at the same budget, using a larger evaluation as the ranking reference? Could the authors show a sequence of model additions and report the additional matches needed for each update? Could the authors give a more concrete example where the skill profiles reveal meaningful differences between similarly ranked models, supported by their gameplay?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cato (part 1/2)

Thank you for the constructive assessment and questions. All three requested analyses have been added to §6.1, §6.2, §6.4, and Appendices H, I, K, and L.

Q1: Comparison with Uniform Sampling at Matched Budgets Against a Larger Reference

Ideally, we would compare round-robin and adaptive scheduler efficiency and ranking performance on real gameplay data across every game; due to API budget constraints (running full round-robins across the ten-game suite would cost an additional USD 2.369M), we use a complete real-game round-robin from an earlier Chess tournament (Appendix L), and we then run controlled ground-truth simulations for N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace models (§6.1, Table 3; Appendix H, Tables H.1–H.3, Figures H.1–H.5) to confirm across pool sizes that the scheduler saves episodes while maintaining ranking accuracy:

  • Real Chess round-robin vs. scheduler (Appendix L, Table L.1, Figure L.1): On the complete 18-model, 6,120-episode Chess round-robin, where skipped episodes are observed, replaying the scheduler saves 76.1% of episodes (1,462 vs. 6,120) while remaining highly correlated with the complete round-robin ranking (ρ=0.9965\rho = 0.9965, τK=0.9780±0.0114\tau_K = 0.9780 \pm 0.0114, 100.0% Top-5 recovery), versus τK=0.9585±0.0188\tau_K = 0.9585 \pm 0.0188 (ρ=0.9926\rho = 0.9926, 54.5% Top-5 recovery) for uniform sampling of the same number of episodes. Uniform sampling only knows to stop at 23.9% of the cap in hindsight from the scheduler's stopping point: unlike uniform allocation, which provides no stopping criterion across games or entrants, the scheduler halts automatically once pairwise orderings reach confidence (qij>0.95q_{ij}>0.95), nearly halves remaining pairwise ordering error (1.10% vs. 2.08%; 75.2% vs. 61.1% accuracy on close pairs ≤40\le 40 Elo), and reaches a τK\tau_K that uniform sampling matches only at 2.5×2.5\times the budget (60% of the complete tournament).
  • Matched-budget curves across N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace simulations: Simulating tournaments from independent true skills (2T=402T = 40 full cap, totaling 1,800 to 49,000 episodes per game across N∈{10,20,30,40,50}N \in \lbrace 10, 20, 30, 40, 50 \rbrace models), Figure H.2 plots Spearman's ρ\rho and Kendall's τK\tau_K across the full budget trajectory. Beyond the initial warm-up rounds (≥15\ge 15% of cap) through convergence, the adaptive scheduler sits above uniform Round-Robin with smaller fixed per-pair targets (r=1,…,20r = 1, \dots, 20) at matched budgets across all pool sizes (e.g., at N=30N = 30 [28.6% cap], τK=0.9470\tau_K = 0.9470 vs. 0.9330; at N=50N = 50 [23.7% cap], τK=0.9531\tau_K = 0.9531 vs. 0.9441) because resolved mismatches stop after 1–2 mirrored rounds while closely matched pairs retain near-full budget (Figure H.5).
  • Scaling and agreement with Full Round-Robin: Episode savings grow from 61.6% (N=10N = 10) to 74.9% (N=40N = 40) and 76.3% (N=50N = 50), while Spearman correlation reaches ρ=0.9744\rho = 0.9744–0.9946 against true skill (Table H.1) and rises from ρ=0.9681\rho = 0.9681 (N=10N = 10) to ρ=0.9928\rho = 0.9928–0.9935 (N=40N = 40–50, 97.26%–97.30% pairwise agreement) directly against Full Round-Robin (Table H.2, Figure H.3). Across the live 39-model suite (147,015 episodes), closely contested pairs (∣ΔElo∣≤40|\Delta\text{Elo}| \le 40) use 95.6% of their configured cap compared with 15.5% for ∣ΔElo∣≥500|\Delta\text{Elo}| \ge 500, reducing total API spend by 62.2% (USD 1.438M vs. USD 3.807M; §6.1, Table I.1).
AuthorsOpenReview ↗

Response to Reviewer cato (2/2)

Q2: Sequence of Model Additions and Incremental Match Cost

In §6.2 and Appendix I (Tables I.2–I.5, Figures I.1–I.3), we report both the live expansion after the S0S_0 snapshot (23→3923 \to 39 models across 6 post-S0S_0 waves, Waves 3–8) and all 8 chronological arrival waves across May–November 2026 (Waves 1–2 forming S0S_0 and Waves 3–8 post-S0S_0):

  • Live S0S_0 (23) →S1\to S_1 (39) refresh (Table I.2): The 15 unchanged S0S_0 submissions reused 28,573 historical episodes (USD 367.2k) and required 2,081 existing-pair confidence re-checks (USD 11.7k). Onboarding all 16 new models took 81,636 episodes (USD 639.1k) versus 223,160 (USD 1.806M) under uniform incremental caps, saving 63.4% of onboarding episodes and 64.6% of API cost (USD 1.167M saved) while preserving Kendall's τK=0.9619\tau_K = 0.9619 (ρ=0.9929\rho = 0.9929, maximum rank shift 1) on unchanged S0S_0 models (Table I.3).
  • Eight chronological arrival waves (Waves 1–2 S0S_0, Waves 3–8 post-S0S_0) and simulated single-model hold-out insertion (Table I.4, Figure I.3): Counting every episode played on the pairs that a wave introduces against the uniform cap for those pairs, per-wave savings span 53.1%–65.6% (for example, 64.9% saved with τK=0.9810\tau_K = 0.9810 existing-pool stability in Wave 8 when adding Gemini 4 Carbon, Claude Fable 5.2, and GPT-6.1 Terra). Simulated single-model hold-out insertions across all 39 models (outcomes drawn from the fitted per-game ratings) require 5,264 episodes on average (74.4% saved vs. cap) and end 1.31 ranks from the model's full-data rank on average, versus 2.74 for matched-budget uniform allocation.

Q3: Concrete Example Where Skill Profiles Reveal Differences Between Similarly Ranked Models

In §6.4 and Appendix K (Table K.1, Figure K.1), we examine Claude Opus 5 and GPT-5.6 Sol, which had nearly identical unified ratings in S0S_0 (247.0 vs. 243.3 Elo, overlapping 95% CIs; 248.2 vs. 221.8 in S1S_1), across 378 direct head-to-head matches and 210 episode logs (including 2,000 Poker hands):

  • Opposite domain performance: Consistent with its higher scores on the planning, tactical, and spatial profile axes, Claude Opus 5 wins every combinatorial board matchup against GPT-5.6 Sol except Chess, 54–1–13 in total (4–0 in Checkers, 8–2 in Four in a Row, 28–8 in Dots and Boxes, and 14–1–3 in Lines of Action; GPT-5.6 Sol wins Chess 8–0). Conversely, consistent with its higher scores on the belief, social, and language axes, GPT-5.6 Sol wins Poker 118–82 (ranking #2 in Poker) and Bargaining 8–12–2.
  • Gameplay logs: Inspection of the 210 episode transcripts shows a corresponding reversal in reasoning allocation: Claude Opus 5 expends 1.7×1.7\times–2.9×2.9\times more reasoning tokens per turn in these four board games (e.g., 11,752 vs. 4,195 tokens/turn in Dots and Boxes; 11,968 vs. 5,731 in Four in a Row), whereas GPT-5.6 Sol expends 2.15×2.15\times more reasoning tokens per turn in Poker (1,819 vs. 844 across 11,661 actions, checking or calling on 56.2% vs. 38.6% of actions and betting or raising on 29.4% vs. 49.6%) and 2.11×2.11\times more in Bargaining (2,043 vs. 968).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.