acceptodds
Under review as a conference paper at ICLR 2027

GeneralsArena: Evaluating End-to-End Agent Systems through Persistent Real-Time Competition

Abstract

When users delegate work to an LLM agent, they care about both the quality of the result and how quickly it is delivered. Longer reasoning or additional agents may improve the result, but also introduce delays and coordination overhead. Therefore, an end-to-end evaluation should capture the consequences of these choices together while allowing systems to organize their own work. We introduce GeneralsArena, which uses real-time strategy competition as a controlled proxy for this outcome-oriented evaluation. Two systems explore a partially observed map and command forces to capture the opposing Capital while defending their own. Each system decides how to plan, act, and organize its agents, while the game continues during reasoning and communication. Further deliberation or delegation can help, but can also allow an opponent to seize an opportunity. Match outcomes jointly reflect decision quality, execution efficiency, and coordination. A persistent ladder compares complete systems, with subscription-service availability measured under the evaluation workload. Across 16 local runtimes, 96 runtime–harness–prompt combinations, and 24 subscription-service configurations, we find that systems sharing model weights can perform very differently, gains depend on the full system configuration, and service availability can change which service ranks first. GeneralsArena provides an evaluation framework for studying how system-level choices translate into end-to-end outcomes. Results, replays, and an introductory video are available at generalsarena.github.io.

Primary area:
datasets and benchmarks
Keywords:
Agent System, Benchmark, Arena, End-to-end evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HdqWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose GeneralsArena, an evaluation framework for agent systems based on persistent real-time strategy competition. In GeneralsArena, each entrant is a complete runtime–harness–prompt configuration competing in real time on partially observable maps, and each system decides how to plan, act, and organize its agents, while the game continues during reasoning and communication. In this way, a persistent ladder compares complete systems and several findings are found.

Critical strengths

GeneralsArena designs an evaluation framework which treats runtime, harness, and prompt as a complete system configuration, rather then comparing models alone. And the metrics capture abilities and constraints of different agents under real-world settings.

Critical weaknesses
  1. Although GeneralsArena assess different agents in a specific settings, this paper doesn't provide information and analysis of correlation between GeneralsArena with other agent benchmarks(e.g., SWE-bench, WebArena). For an useful benchmark, it is important to provide in-depth analysis.
  2. For an single game, it is hard to support the broad claim of "end-to-end agent system evaluation", therefore I think it is better to further refine the claim and contribution of this paper.
  3. The framework should provide better tools for attribute the score/advantages of different agents to different abilities;
  4. Latency sensitivity may systematically undervalue "slow but accurate" systems.
Questions for authors

See weaknesses in above.

4 comments

Thank you for recognizing the value of evaluating complete runtime–harness–prompt systems under realistic deployment constraints. We agree that validating against established agent benchmarks, clarifying our platform scope, and attributing scores to concrete behaviors are essential, and we address each point below.


W1 — Correlation with other agent benchmarks

To evaluate how GeneralsArena aligns with established agent benchmarks, we considered SWE-bench and WebArena alongside newer interactive tool-use benchmarks. SWE-bench focuses on repository-level software engineering patch generation, which carries high per-instance container execution cost across 16 serving configurations and is overly difficult for ~30B-scale open-weight models—compressing solve rates near the floor and making it hard to separate runtimes—while WebArena is an earlier web-navigation benchmark (2023) that is less discriminative for current tool-use agent workflows. We therefore selected two newer benchmarks that better match interactive agent scenarios—Terminal-Bench (core 0.1.1, 77 multi-step command-line tasks in a live shell with Terminus-1) and tau2-bench (60 multi-turn customer-service tool-use tasks across airline and retail, using a local Qwen3.8-27B at temperature 0 as user simulator and judge across all runtimes)—and evaluated all 16 arena runtimes under the exact same weights, serving engines, and sampling settings:

Spearman correlation16 runtimes8 models
Turn-based ladder vs Terminal-Bench0.600.74
Turn-based ladder vs tau2-bench0.630.99
Turn-based ladder vs mean of both0.700.93
Real-time ladder vs Terminal-Bench0.550.62
Real-time ladder vs tau2-bench0.440.79
Real-time ladder vs mean of both0.530.83
Reference: Terminal-Bench vs tau2-bench0.680.79
Runtime (turn-based order)Turn-based Rating (rank)Real-time Rating (rank)Latency (s)Terminal-Bench solvedtau2 pass@1
Qwen3.8 FP8 / SGLang low + DSpark1812 (1)1548 (5)13.537/770.80
Qwen3.8 FP8 / vLLM low1723 (2)1423 (7)24.338/770.77
Gemma4 BF16 / SGLang on1683 (3)1766 (4)9.026/770.85
Qwen3.8 BF16 / SGLang low1592 (4)1413 (8)23.736/770.84
Qwen3.8 FP8 / SGLang low1588 (5)1410 (9)33.234/770.82
Ornith-1.5 FP8 / SGLang off1535 (6)1772 (3)2.530/770.76
Qwen3.6 FP8 / SGLang off1514 (7)1880 (1)1.726/770.74
Gemma4 BF16 / SGLang off1461 (8)1777 (2)2.619/770.66
Qwen3.8 FP8 / Transformers low1433 (9)1337 (15)98.42/770.29
Qwen3.8 FP8 / SGLang xhigh1424 (10)1343 (14)56.125/770.84
Laguna-XS FP8 / SGLang on1414 (11)1377 (11)8.020/770.54
Granite-4.2 BF16 / SGLang off1400 (12)1349 (13)10.315/770.50
Muse BF16 / SGLang low1381 (13)1392 (10)14.028/770.50
Gemma4 BF16 / vLLM off1361 (14)1371 (12)1.114/770.29
Qwen3.8 FP8 / SGLang off1352 (15)1513 (6)8.536/770.79
Nemotron-3.5 BF16 / SGLang off1330 (16)1311 (16)1.313/770.49
  • Turn-based ratings correlate strongly with both benchmarks. Across the 16 runtimes, the turn-based ladder correlates with Terminal-Bench at Spearman 0.60, with tau2-bench at 0.63, and with their mean at 0.70—matching how well the two external benchmarks agree with each other (0.68). Merging multiple deployments of the same model weights by their median (8 models) yields 0.99 on tau2 and 0.93 on the two-benchmark mean (inter-benchmark reference: 0.79). Scoring tau2 with only deterministic database-state checks (no LLM judge) gives virtually the same correlation (0.60).
  • Why real-time ratings differ from static benchmarks, and how deployment differences carry over. Static benchmarks score task completion without penalizing per-step latency, whereas our real-time ladder accounts for response time while the game clock advances, so fast runtimes (such as Qwen3.6 at 1.7 s) rank higher in real time and slow runtimes (such as Qwen3.8 xhigh at 56 s) rank lower. The external benchmarks also reproduce the serving differences seen in our arena: Gemma4 solves 14 Terminal-Bench tasks on vLLM versus 19 on SGLang, and Qwen3.8 at xhigh reasoning hits the per-task timeout on 37 of the 77 tasks.

W2 — Scope of the end-to-end claim

GeneralsArena is a configurable match platform that provides diverse match environments as evaluation proxies for different real-world agent deployment scenarios. In our framework, "end-to-end" refers to evaluating the complete deployed system—model weights, serving engine, harness, and prompt working together—under controllable operational conditions:

  • Configurable environments for different evaluation scenarios. By adjusting the timing protocol, observation modality, agent organization, and access limits (Section 3.1), the platform supports end-to-end evaluation across distinct settings: the 16-runtime and 96-configuration ladders test serving stacks and harness–prompt combinations (Sections 4.1–4.2), the subscription ladder incorporates service access limits via ESR (Section 4.3), and the protocol configurations test single-actor execution, visual observations, and turn-based timing (Section 4.4).
  • Consistency with external benchmarks across settings (W1) and paused-clock extension (W4). In W1, we evaluated our runtimes on real-world agent benchmarks (Terminal-Bench and tau2-bench) and analyzed how different arena timing settings relate to benchmark performance. In W4, we further configured a paused-clock match environment to show how the platform can evaluate slow, high-accuracy systems without latency pressure.
  • Exposing system-level differences with high reproducibility. Evaluating the complete system captures performance gaps that model-only evaluations miss: identical Gemma4 weights differ by 406 rating points across two official serving engines, while rankings across three independent 600-game ladders remain highly reproducible (Spearman 0.947, 0.891, and 0.950, Appendix B.2). We will clarify this platform scope in the introduction and contribution list.

W3 — Attributing scores to abilities

Our request and match logs map rating differences to concrete behaviors when we hold two of serving engine, harness and model fixed:

  • Same model and harness across serving engines (Gemma4-26B on vLLM vs. SGLang). The 406-point gap for Gemma4-26B under Reference, vLLM (Rating 1371.1, 6W/46D/13L) vs. SGLang (Rating 1777.2, 43W/16D/37L), comes from two output differences. First, vLLM's official parser omits move.send on 5,868 of 7,505 rejected requests (30.9% rejection rate vs. 0.0% on SGLang). Second, vLLM has occasional ~180 s runaway generations that hit the token limit (36% of match time vs. 0% on SGLang), which slows opening expansion at tick 200 from 28.7 cells on SGLang to 12.5 cells on native vLLM and 12.9 cells on constrained vLLM.
  • Same model and serving engine across harnesses (Qwen3.6-35B under Reference vs. Pi). Comparing Qwen3.6-35B on SGLang under Reference (Rating 1718.7–1772.2, 46.2%–56.4% win rate) and Pi (Rating 1522.4–1537.3, Neutral 0W/41D/0L and Detailed 1W/39D/0L) shows how harness design shapes execution. Reference passes an explicit enum of legal move IDs, allowing Qwen3.6 to execute 268.7 valid decisions per 1,000 ticks with only a 3.5% rejection rate. Pi only constrains coordinate format, so once Qwen3.6 outputs a single out-of-view move, the rejection left in the history triggers an error cascade (jumping from 3.8% to 44.8% illegal moves) that stalls expansion into a draw.
  • Different models under the same serving engine and harness (SGLang + Reference + Neutral). The logs also explain why models rank differently:
    • Action validity at high speed (Qwen3.6-35B vs. Nemotron3.5-Lightning and Muse-Glimmer-30B): Qwen3.6 (1.67 s latency) and Nemotron3.5 (1.32 s) both act rapidly, but Qwen3.6 keeps rejections at 3.5% (75W/8D/30L, Rating 1880.2, Rank 1) whereas Nemotron3.5 wastes 15.8% of requests on invalid moves (5W/39D/20L, Rating 1310.9, Rank 16) and Muse-Glimmer-30B has 63.4% of requests rejected (3W/46D/8L, Rating 1391.7).
    • Multi-agent delegation overhead (Qwen3.6-35B vs. Ornith-1.5-35B): Both are fast 35B-A3B MoE models on SGLang/FP8/off, but Qwen3.6 acts as a single root session (0% child games, Rating 1880.2), whereas Ornith-1.5 spawns child sessions in 92.3% of games, increasing contention and rejections (7.4%, Rating 1771.7). When delegation is disabled in our single-actor configuration, Ornith's latency drops from 2.53 s to 1.62 s and its win rate rises from 47.0% to 59.8%.

To make these analyses reusable, we designed a three-module diagnostic toolkit that we will release with the arena (and expose as structured post-game feedback for agents and developers): (1) an interface & error-cascade analyzer that breaks down tool-call rejections by error code and prior rejection streak, (2) a tempo, concurrency & stall profiler that tracks per-tick territory expansion, child-session contention, and ≥30 s generation stalls, and (3) an automated draw & outcome classifier that labels each draw as invalid actions, no progress, unconverted lead or contested.

W4 — Are slow but accurate systems undervalued?

In a real-time match environment, response latency does affect performance and can put slow but accurate systems at a disadvantage, although latency alone does not determine the overall ranking (as shown in Table 1, the fastest system is not the strongest). We chose the real-time setting as our primary configuration because in many real-world interactive deployments, response latency has a direct impact on user experience and operational effectiveness.

At the same time, GeneralsArena is a general, configurable evaluation platform that supports different match configurations for different evaluation criteria. In addition to the turn-based configuration in the paper, we configured a paused-clock match environment (halting the game clock while models think and advancing 10 ticks per round) and ran a 504-game controlled tournament across eight runtimes (median latencies from 1.7 to 56 s, playing all 28 pairs × 3 map seeds × both seats = 168 games per protocol on dedicated GPUs):

Runtime (by median latency)Median latency (s)Real-time score (rank)Turn-based score (rank)Paused score (rank)Paused − real-time
Qwen3.6 FP8 / SGLang off1.70.702 (1)0.452 (5)0.369 (5)−0.333
Ornith-1.5 FP8 / SGLang off2.50.702 (1)0.393 (6)0.262 (7)−0.440
Gemma4 BF16 / SGLang off2.60.512 (4)0.310 (8)0.310 (6)−0.202
Qwen3.8 FP8 / SGLang off8.50.357 (6)0.357 (7)0.190 (8)−0.167
Gemma4 BF16 / SGLang on9.00.702 (1)0.714 (1)0.738 (2)+0.036
Qwen3.8 FP8 / SGLang low + DSpark13.50.464 (5)0.655 (2)0.667 (3)+0.202
Qwen3.8 FP8 / vLLM low24.30.345 (7)0.619 (3)0.619 (4)+0.274
Qwen3.8 FP8 / SGLang xhigh56.10.214 (8)0.500 (4)0.845 (1)+0.631
Protocol pair (8 runtimes)Spearman
Real-time vs turn-based−0.02
Real-time vs paused−0.27
Turn-based vs paused0.79
  • Configuring turn-based or paused-clock play lets slow, high-accuracy systems take the lead. When we switch from real-time play to the paused-clock configuration, the slowest runtime—Qwen3.8 at xhigh reasoning (56 s median latency)—rises from last place in real time (rank 8, score 0.214) to first place (rank 1, score 0.845), a paired score gain of +0.631. Slower reasoning configurations such as Qwen3.8 + DSpark and vLLM low also rise into the top tier in both turn-based and paused-clock play.
  • Latency-tolerant configurations align more closely with static benchmarks. Turn-based and paused-clock rankings agree closely with each other (Spearman 0.79), and as shown in W1, the turn-based configuration achieves a higher correlation with static agent benchmarks (0.70 on 16 runtimes and 0.93 on 8 models) than the real-time configuration (0.53). GeneralsArena can thus be configured either in real-time mode to account for deployment latency or in turn-based / paused-clock mode to evaluate deliberate reasoning without time pressure.

Thank you again for your thoughtful comments on the scope and practical relevance of our evaluation. In our responses above, we report correlations with Terminal-Bench and tau2-bench across all 16 runtimes, clarify the scope of “end-to-end” evaluation, and connect performance differences to concrete execution behaviors. We also provide a controlled comparison of timing protocols, showing how slower reasoning configurations can take the lead when the game clock is paused during inference. We would greatly appreciate your assessment of whether these additions address your concerns about benchmark relevance, diagnostic value, and latency sensitivity. Please let us know if any points warrant further clarification or discussion.

Reviewer_ZQNyOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GeneralsArena, a benchmark where complete LLM agent systems (runtime service, harness and strategy prompt) play a real-time strategy game. The game keeps running while agents reason, so response time affects the result. The main contributions are this real-time arena, a persistent Glicko-2 ladder, and an Effective Service Rating (ESR) that combines Rating with API availability for subscription services. The paper tests 16 local runtimes, 96 runtime–harness–prompt combinations, 24 subscription services and three protocol changes. It finds that systems with the same model weights can perform very differently, and that availability can change which service ranks first.

Critical strengths

1 Response time counts toward the result without a hand-made penalty. The game does not pause during inference, unlike VideoGameBench Lite and OmniGameArena (Sec. 2.2). The data shows a real effect: Qwen3.8-27B falls from 1512.9 to 1343.0 as reasoning goes from off to xhigh, while latency rises from 8.51 s to 56.12 s (Table 1). This is the paper's main new idea, and the evidence supports it. 2 Careful ladder checks. Two extra independent ladders keep the same leader and the same top five. Their rank correlations with the main ladder range from 0.891 to 0.950 (Appx. B.2). A late entrant reaches first place after 20 of its own games (Appx. B.3). This makes the runtime results in Sec. 4.1 credible.

Critical weaknesses

-1 Draws dominate, so the ladder often can't separate systems. The factorial ladder has 75.5% draws (Appx. E.3), and 29 of the 96 entrants win no games (Tables 8–9). Each entrant plays only 39–47 games. The paper's own bootstrap puts nine entrants in the top tier (Appx. E.2). The prompt gaps in Sec. 4.2 (5.6, 13.5 and 33.4 points) are all smaller than one RD, so the claim that "prompt effects vary across runtime–harness combinations" is not supported. 2 The link to real agent use is not shown. The paper never checks arena rankings against any outside agent benchmark. Some large gaps seem to come from tool-call problems rather than reasoning. For example, Gemma4 on vLLM rejects 30.9% of its decisions, partly because of missing 'move.send' fields (Appx. E.1). This setup is the low end of the 406-point Gemma spread that the paper uses as a headline result.

Questions for authors

1- (Clarification) Why does Qwen3.6, the strongest runtime, go 0/41/0 under Pi/Neutral and 1/39/0 under Pi/Detailed? Is this how the model behaves under Pi, or an adapter issue? 2- (Clarification) Can you give uncertainty intervals for the Detailed–Neutral gaps in Sec. 4.2? If they are not reliable, will you soften the prompt claims? 3- (Clarification) If you fix the vLLM move.send issue, how much of the 406-point Gemma gap remains? Do the arena rankings track any outside agent benchmark for the same runtimes?

4 comments

Thank you for highlighting the role of latency in real-time evaluation and for closely examining our stability analyses. We agree that the draws and prompt differences in the 96-configuration ladder should be analyzed carefully and that tool-call formatting errors should be separated from model capability, and we address each point below.


W1 — Draws and the resolution of the factorial ladder

In the 96-configuration ladder (1,450 draws in 1,920 games), a draw is itself an informative competitive outcome rather than an uninformative tie: under Glicko-2, drawing against a higher-rated opponent increases an entrant's rating while drawing against a lower-rated opponent decreases it (for instance, the 29 configurations without a win record 1,050 draws and only 102 losses across 1,152 games, and the ladder still orders them by how effectively they hold off stronger opponents). To see what behaviors produce these draws, we categorized all 1,450 draws from the match logs using four priority-ordered rules:

Draw classRule (in priority order)Draws (N = 1,450)
Invalid actionsOne side has ≥25% of its requests rejected402 (27.7%)
No progressStopped by the stagnation rule326 (22.5%)
Unconverted leadTime limit, leader holds ≥2× the territory373 (25.7%)
ContestedTime limit, no 2× territory lead349 (24.1%)
  • Draws arise from genuine model bottlenecks and closely matched capabilities. As the breakdown shows, draws happen when a model repeatedly outputs invalid actions (27.7%), when neither side can break through and the match hits the 300 s stagnation rule (22.5%), when the leading side fails to convert a ≥2× territory advantage before the time limit (25.7%), or when two sides trade territory evenly (24.1%). All four patterns reflect real agent behaviors during play, and indicate that many configurations share similar capability levels or similar execution bottlenecks.
  • Stronger configurations establish a clear lead, while similarly capable ones naturally cluster. Under our setup, genuinely stronger systems still separate at the top, whereas configurations with similar capabilities frequently draw against one another and therefore receive close ratings—which is expected when many tested configurations have comparable strength.
  • The ladder still captures a reliable overall relative ordering. Even with frequent draws among near-peers, the relative ordering of the 16 runtimes in the 96-configuration ladder correlates strongly with the 600-game 16-runtime main ladder (Spearman 0.87 on marginal means and 0.83 on the Reference harness averaged over both prompts), where three independent 600-game runs recover the exact same leader and top-five set (Spearman 0.947, 0.891, and 0.950, Appendix B.2). Capturing this consistent relative ordering across systems is the primary goal of the arena, and we will clarify this interpretation in Section 4.2.

W1 / Q2 — Uncertainty of the Detailed–Neutral differences

We computed 95% bootstrap confidence intervals for the average Detailed-minus-Neutral rating difference within each harness across the 96-configuration ladder:

HarnessDetailed − Neutral95% interval
Reference+5.63[−18.09, 29.69]
Pi−13.49[−19.12, 17.37]
Codex−33.40[−34.55, 3.49]
  • The performance changes between Detailed and Neutral have a wide uncertainty range. As the table shows, the estimated differences span a wide range and do not establish a consistent gain or drop across harnesses. We also noted this in the paper (where the prompt main effect accounts for only 0.5% of the rating variation in Appendix E.2): under our current Neutral and Detailed prompt configurations, the impact on overall rankings is limited.
  • Why we still organize the prompt as a standalone module. In real-world agent tasks, prompt design plays a major role in shaping agent behavior, so we still treat the prompt as an explicit, standalone module in our evaluation framework. We will revise Section 4.2 to report these uncertainty ranges instead of claiming that prompt effects vary across runtime–harness pairs, and we will explore more diverse prompt designs in follow-up work to study their impact on system performance.

W2 / Q3a — How much of the 406-point Gemma gap remains after fixing move.send?

In the paper, our vLLM deployment of Gemma4 directly used vLLM's officially recommended Gemma 4 recipe (gemma4 tool and reasoning parsers, the official vLLM v0.28.0 chat template, and the same sampling parameters as SGLang). Because vLLM's official gemma4 parser does not enforce JSON-schema constraints on tool arguments, 5,868 of the 7,505 rejections on vLLM came from a missing move.send field. To test how much of the gap comes from this official configuration issue, we ran a 120-game matched tournament on the same Gemma4 weights across three conditions—official vLLM, vLLM optimized with JSON-schema constrained decoding, and SGLang—over 40 matched slots (4 mid-tier opponents × 5 map seeds × both seats) with opponent ratings anchored to the main ladder:

Gemma4 condition (40 games each)W/D/LMean scoreRejection rateMissing move.sendAnchored rating
vLLM, native tool calls (paper)13/0/270.32567.4%26.7%1344
vLLM, JSON-schema constrained20/0/200.5000.7%0.0%1474
SGLang, native tool calls33/0/70.8250.0%0.0%1750
Per-game averagevLLM, constrained decodingSGLang
Accepted decisions per game303304
Game time inside ≥30 s requests36%0%
Tick at which 10 cells are held23659
Territory at tick 20012.928.7
  • The format errors come from the official vLLM recipe, and fixing them reduces rejections from 67.4% to 0.7%. Adding JSON-schema constrained decoding eliminates missing-move.send errors and raises vLLM's anchored rating from 1344 to 1474, showing that a provider's default official recipe is not always optimal for every runtime.
  • A 276-point gap between SGLang (1750) and vLLM (1474) still remains after the fix. Once format errors are removed, both stacks complete the same number of valid decisions per game (303 vs. 304), but constrained vLLM suffers from occasional ~180 s runaway generations that hit the token limit in 35 of the 40 games (36% of game time vs. 0% on SGLang), slowing early expansion (12.9 vs. 28.7 cells at tick 200). This confirms our paper's conclusion that the serving stack substantially affects system performance, and we will add this experiment and note to the paper.

W2 / Q3b — Agreement with external agent benchmarks

We evaluated all 16 arena runtimes on Terminal-Bench core 0.1.1 (77 buildable shell tasks, Terminus-1 agent, single attempt) and tau2-bench (60 airline and retail customer-service tool-use tasks, single trial, using a local Qwen3.8-27B at temperature 0 as user simulator and judge across all runtimes), keeping the exact same model weights, serving engines, and sampling settings:

Spearman correlation16 runtimes8 models
Turn-based ladder vs Terminal-Bench0.600.74
Turn-based ladder vs tau2-bench0.630.99
Turn-based ladder vs mean of both0.700.93
Real-time ladder vs Terminal-Bench0.550.62
Real-time ladder vs tau2-bench0.440.79
Real-time ladder vs mean of both0.530.83
Reference: Terminal-Bench vs tau2-bench0.680.79
  • Turn-based ratings align with both external benchmarks. Across the 16 runtimes, the turn-based ladder correlates with Terminal-Bench at 0.60, with tau2-bench at 0.63, and with their mean at 0.70—matching the agreement between the two external benchmarks themselves (0.68). Merging deployments of the same model weights by their median (8 models) yields 0.99 on tau2 and 0.93 on the two-benchmark mean (inter-benchmark reference: 0.79).
  • Why real-time ratings differ from static benchmarks. Static benchmarks score task completion without penalizing per-step latency, whereas our real-time ladder penalizes slow responses while the game clock runs. Consequently, the fastest runtimes overperform their static benchmark rank in real time (Qwen3.6 at 1.7 s and Gemma4 on SGLang at 2.6 s), while the slowest runtimes drop (Qwen3.8 xhigh at 56 s and Qwen3.8 BF16 low at 24 s).
  • External benchmarks surface the same deployment differences. Gemma4 solves 14 Terminal-Bench tasks on vLLM versus 19 on SGLang, Qwen3.8 xhigh hits the per-task timeout on 37 of 77 tasks, the Transformers backend suffers from low throughput and format errors, and Muse repeatedly passes zip codes as integers instead of strings on tau2 retail.

Q1 — Why does Qwen3.6 draw almost every game under Pi?

After verifying that the Pi adapter code has no bug (Pi refreshes the board observation every turn, and unlike Reference's explicit legal-move enum, Pi's tool schema only constrains coordinate format), we ran two 2×2 request-replay experiments on the frozen Qwen3.6 service. We crossed the move schema (Pi's coordinate format vs. an explicit legal-move enum) with the conversation context (full session history vs. current turn only) across 200 decision states following an accepted move (Experiment C, 181 move calls) and all 173 archived decision states following a rejected move (Experiment C2):

Request variantAfter an accepted action (C)After a rejection (C2, N = 173)After 1 rejectionAfter 2–4After ≥5
Coordinate pattern, with history (Pi as deployed)3.8%44.8%37.5%45.8%54.8%
Legal-move enum, with history3.2%41.3%42.1%35.6%47.2%
Coordinate pattern, current turn only12.2%22.1%20.4%15.8%30.8%
Legal-move enum, current turn only4.8%17.6%15.6%17.9%20.7%
  • A single rejected action triggers a persistent error loop in real matches. Under Pi's deployed setup (coordinate format + session history), Qwen3.6's illegal-move rate is only 3.8% after an accepted move, but jumps to 44.8% immediately after a rejection (888 of 891 rejections in the archived games are out-of-view coordinates, and 667 immediately follow a prior rejection).
  • Prior rejections in the conversation history drive this loop. Removing the session history on the exact same post-rejection states cuts the illegal-move rate by 22.1 percentage points (down to 22.1%). When history is kept, the illegal-move rate climbs from 37.5% after one rejection to 54.8% after five or more consecutive rejections.
  • Why this produces draws. Once Qwen3.6 makes one out-of-view move under Pi's loose coordinate schema, the rejection record in context causes it to keep outputting invalid moves turn after turn (even adding a legal-move enum only lowers the post-rejection rate with history by 3.1 points, to 41.3%), stalling its expansion until the match ends in a draw.

Thank you again for your constructive feedback. Our responses above include a controlled experiment correcting Gemma4’s tool-call formatting, an analysis of Qwen3.6’s repeated errors under Pi, and evaluations on Terminal-Bench and tau2-bench using the same runtimes. We also provide a breakdown of draws and uncertainty intervals for the prompt comparisons, and acknowledge that the current evidence does not establish a consistent prompt effect. We would appreciate it if you could take a look at these results and let us know whether they address your concerns about ranking reliability and the sources of the observed performance gaps. We welcome any remaining questions and would be happy to discuss them further.

Reviewer_hid6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a persistent real-time strategy arena for evaluating complete agent systems, including the runtime, harness, prompts, and response latency. Systems compete in a partially observed environment that continues advancing during reasoning and communication. Experiments cover 16 local runtimes, 96 runtime–harness–prompt configurations, and 24 subscription-service configurations. Glicko-2 ratings summarize competitive outcomes, while an Effective Service Rating (ESR) additionally incorporates availability under the evaluation workload. The results demonstrate substantial configuration effects and illustrate how latency and access constraints affect outcomes in this environment.

Critical strengths
  1. A useful system-level evaluation unit. Evaluating the complete system captures interactions that a model-only ranking misses. Configurations sharing Gemma4 weights span 406 rating points, and the fastest configuration is not the strongest (Table 1, pp. 6–7). This makes the platform informative for studying configuration choices under time pressure, with objective competitive outcomes rather than an independently chosen latency penalty.

  2. Substantial empirical coverage and transparent qualification. The runtime, harness/prompt, service, and protocol comparisons provide several informative perspectives. Three independently initialized runtime ladders retain the same leading five systems, with rank correlations of 0.947, 0.891, and 0.950 (Appendix B.2). Replays and explicit qualifications about ESR and cross-protocol comparisons help readers distinguish robust group-level findings from uncertain individual ranks.

Critical weaknesses
  1. ESR is sensitive to reporting and observation conventions whose practical effect is not sufficiently quantified. The reference rating of 1500 is a convention, and the transformed score is not a calibrated win probability against an actual reference agent; changing the reference can reverse rankings (Appendix A.2). Availability is also conditional on counted opportunities: a successful probe can be excluded when the opponent fails, and cooldown periods create no new denominator events (Appendix C.4). Consequently, 100% availability is not continuous availability, and the displayed rating deviation does not quantify uncertainty in availability or ESR. These acknowledged limitations materially constrain the interpretation of the service ranking.

  2. Fine-grained ranking and protocol interpretations need stronger controls. Protocol ladders differ in adaptive pairing, GPU allocation, and, for the visual setting, participant set; they do not isolate a single causal factor (Appendix F.1). The main runs do not enforce matched seat swaps, while the 96-configuration bootstrap leaves nine systems in the highest nondominated confidence tier (Appendices B.4, C.3, E.2). High draw rates further combine potentially different behaviors. These issues do not invalidate the arena, but weaken claims about precise winners or the magnitude of individual design effects.

Questions for authors
  1. How stable are subscription rankings over a reasonable range of ESR reference ratings and alternative cooldown/cancelled-pair counting conventions? Please report uncertainty for availability and ESR, rather than rating deviation alone.
  2. Can the recorded data support an availability estimate independent of opponent failures and a breakdown of excluded/cooldown periods? A new fixed wall-clock request-stream experiment would be a useful extension; please distinguish it from what existing logs establish.
  3. Which protocol conclusions survive matched opponents, map seeds, seat swaps, and hardware allocation? If such controls are unavailable, please identify the conclusions that should remain descriptive rather than causal.
  4. Can draws be separated into meaningful defensive play, lack of progress, invalid actions, and failure to convert an advantage? This would clarify what the very high visual-protocol draw rate measures. Transfer to real time-sensitive tool tasks remains future validation, rather than something established by the present game results.
4 comments

Thank you for the detailed review and for recognizing the value of evaluating complete agent systems. We agree that the ESR conventions and uncertainty should be quantified and that causal protocol conclusions require matched controls, and we address each point below with new analyses and controlled experiments.


W1 / Q1a — How much does the 1500 reference rating matter?

We recomputed ESR across all 24 subscription services by holding each service's Rating and availability fixed while sweeping the reference rating from 1200 to 1800 (grouped below by identical top-three orderings), and by testing alternative cancellation and cooldown counting conventions:

Reference rangeSpearman by reference ratingLeaderTop three (identical within each range)
1200–15001200: 0.833 / 1300: 0.937 / 1400: 0.988 / 1500: 1.000Volcengine Doubao Seed 2.0 LiteVolcengine Doubao Seed 2.0 Lite, Volcengine DeepSeek V4 Flash, Volcengine Ark Code Latest
1600–18001600: 0.995 / 1700: 0.985 / 1800: 0.978Volcengine Doubao Seed 2.0 LiteVolcengine Doubao Seed 2.0 Lite, Volcengine DeepSeek V4 Flash, Bailian DeepSeek V4 Flash
  • Most pairwise orderings never change. Because ESR increases monotonically with both Rating and availability, whenever one service is at least as strong on both metrics, their relative order holds at every reference value. This applies to 179 of the 276 service pairs (65%).
  • 1500 matches the empirical mean of the pool. The mean Rating across the 24 services is 1481, and using 1481 as the reference rating yields a Spearman correlation of 0.999 with the reported order.
  • Rankings remain stable across reference values and counting rules. Across reference ratings from 1400 to 1800, Spearman correlation with the reported order stays at or above 0.978, the leader never changes, and the top three only swap the third-place service between 1200–1500 and 1600–1800. Likewise, counting opponent-canceled probes as successes yields a Spearman correlation of 0.999 with the reported ESR ranking (only swapping ranks 7 and 8, which differ by 0.1 ESR points). We will add this sensitivity analysis to Appendix A.2.

W1 / Q1b — Uncertainty for availability and ESR

We computed the 95% Wilson confidence interval for availability and the corresponding ESR interval (from Rating ± 2 RD and the Wilson bounds) across all 24 subscription services, and show representative services from the top, middle, and tail groups below:

TierService (ESR rank)Rating ± RDAvailability [Wilson 95%]ESR [propagated range]
Top tierVolcengine Doubao Seed 2.0 Lite (1)1675 ± 62100.0% [98.5, 100.0]1675 [1544, 1800]
Top tierVolcengine DeepSeek V4 Flash (2)1641 ± 62100.0% [98.6, 100.0]1641 [1512, 1766]
Top tierVolcengine Ark Code Latest (3)1604 ± 63100.0% [98.4, 100.0]1604 [1473, 1730]
Middle groupBailian Qwen3.8 Flash (6)1607 ± 6389.7% [82.8, 94.0]1558 [1424, 1685]
Middle groupMiMo V2.5 Pro (7)1598 ± 6485.1% [76.5, 90.9]1529 [1393, 1659]
Middle groupFirst-party Kimi K3 (8)1610 ± 6683.1% [71.5, 90.5]1529 [1380, 1669]
Middle groupBailian Qwen3.7 Max (9)1714 ± 7566.7% [53.0, 78.0]1511 [1364, 1642]
Tail groupBailian Qwen3.8 Max (23)1282 ± 8460.0% [44.6, 73.7]1174 [963, 1365]
Tail groupBailian GLM-5.2 (24)1238 ± 12535.5% [21.1, 53.1]1035 [711, 1315]

Adjacent ESR intervals overlap: the top three services (led by Volcengine Doubao Seed 2.0 Lite at 1544–1800) are clearly separated from the lower group (ranks 19–24, all below 1485), whereas in the middle of the table (such as ranks 6–11, all falling within 1364–1685) the intervals interleave. This indicates that our arena reliably separates stronger systems from weaker ones, while configurations with very close performance exhibit natural rank fluctuations. Our repeated fixed-scenario diagnostic experiment in Appendix B.4 (Table 5) supports the same conclusion: pairs with larger rating gaps produce clear win/loss separation (33 wins, 17 draws, and 0 losses for the higher-rated side), whereas near-peer pairs show dynamic fluctuations and frequent draws (11 wins, 38 draws, and 1 loss). We will add the full 24-service interval table and this discussion to the main text and Appendix C.

W1 / Q2 — Opponent-canceled probes and cooldown periods

We recalculated availability without opponent failures and tallied cooldown time over the 104-hour evaluation window. The 11 fully available services had no own failures and no cooldown, and stay at 100% when opponent-canceled probes are included. The other 13 services:

Service (ESR rank)AcceptedOwn failuresCanceled by opponentAvailabilityCanceled counted as successCooldown (h)
Bailian DeepSeek V4 Flash (4)5915279.7%80.3%75.0
Bailian Qwen3.8 Flash (6)10412289.7%89.8%59.8
MiMo V2.5 Pro (7)8014285.1%85.4%66.3
First-party Kimi K3 (8)4910283.1%83.6%50.0
Bailian Qwen3.7 Max (9)3417066.7%66.7%84.0
MiniMax M2.5 (11)21014693.8%93.9%11.0
MiniMax M2.7 (14)21411399.5%99.6%0.1
MiMo V2.5 (16)1608495.2%95.3%26.5
First-party GLM-5.3 (20)557188.7%88.9%35.0
Bailian DeepSeek V4 Pro (21)5514079.7%79.7%70.0
Kimi K2.7 Code (22)4312178.2%78.6%60.0
Bailian Qwen3.8 Max (23)2416660.0%65.2%75.8
Bailian GLM-5.2 (24)1120035.5%35.5%95.5
  • Opponent-canceled probes barely change the results. Of 3,906 scheduled opportunities, 3,600 started normally, 160 failed on the service's own side, and 146 passed the service's readiness probe but were canceled because the opponent failed (Appendix C.4). Counting those 146 probes as successes changes availability by at most 5.2 percentage points (Bailian Qwen3.8 Max, from 60.0% to 65.2%) and only swaps ranks 7 and 8 (1529.5 vs. 1529.4), leaving the leader and top five unchanged.
  • The cooldown rule follows real quota plans and is the same for every service. Commercial plans combine monthly allowances with 5-hour rolling windows, so we cool down 5 minutes after a transient failure and 5 hours after a quota rejection. The recorded hours (50–95.5 h for plans that hit 5-hour blocks, 11 h for MiniMax M2.5 under brief rate limits) reflect each plan's actual quota behavior.
  • Fixed wall-clock request stream. Providers have changed their coding-plan rules since our evaluation window, so a new request stream run now would not be on an equal footing with the recorded tournament. We will update both experiments together under matched plan terms.

W2 / Q3 — Which protocol conclusions survive matched controls?

Hardware, map seeds and seat order do not drive the results. Every active side runs on one dedicated NVIDIA H20 GPU (TP=1) that is never shared with another match, and model loading and warm-up finish before the match clock starts (Appendix C.2), so dynamic allocation only reassigns idle GPUs. Three independent 600-game ladders with different map seeds and seat assignments (Appendix B.2) give the same top five, with rank correlations of 0.947, 0.891 and 0.950. The visual and single-actor ladders cover different runtime sets, so we keep them as descriptive comparisons.

To test timing protocols with opponents, map seeds, seats and hardware all fixed, we ran a new 504-game tournament. Eight runtimes (median latency 1.7 to 56 s) played all 28 pairings on three map seeds in both seat orders (ab and ba, 168 games per protocol) under real-time, turn-based and paused-clock play, where the paused clock stops during model inference and advances 10 ticks per round:

Runtime (by median latency)Latency (s)Real-time score (rank)Turn-based score (rank)Paused score (rank)Turn-based − real-timePaused − real-time
Qwen3.6 FP8 / SGLang off1.70.702 (1)0.452 (5)0.369 (5)−0.250−0.333
Ornith-1.5 FP8 / SGLang off2.50.702 (1)0.393 (6)0.262 (7)−0.310−0.440
Gemma4 BF16 / SGLang off2.60.512 (4)0.310 (8)0.310 (6)−0.202−0.202
Qwen3.8 FP8 / SGLang off8.50.357 (6)0.357 (7)0.190 (8)+0.000−0.167
Gemma4 BF16 / SGLang on9.00.702 (1)0.714 (1)0.738 (2)+0.012+0.036
Qwen3.8 FP8 / SGLang low + DSpark13.50.464 (5)0.655 (2)0.667 (3)+0.190+0.202
Qwen3.8 FP8 / vLLM low24.30.345 (7)0.619 (3)0.619 (4)+0.274+0.274
Qwen3.8 FP8 / SGLang xhigh56.10.214 (8)0.500 (4)0.845 (1)+0.286+0.631
  • Seat order and map seeds do not change the ranking. Within each protocol, scores from the two seat orders correlate at Spearman 0.83 (real-time), 0.74 (turn-based) and 0.80 (paused), and balanced ab/ba pairing cancels map-side asymmetry.
  • Protocol choice reorders the ranking. Real-time rankings correlate with turn-based ones at only −0.02 and with paused ones at −0.27, with score shifts in the direction reported in Section 4.4. Slower reasoning runtimes gain in turn-based play and fast non-thinking runtimes drop, as the last two columns show.
  • Pausing the clock confirms the role of latency. The slowest runtime, Qwen3.8 at xhigh reasoning (56 s), goes from last in real time to first, a gain of +0.631, and turn-based and paused rankings agree at Spearman 0.79.

W2 — Precise winners in the 96-configuration ladder

With 96 configurations and a finite budget (1,920 games, or 39–47 per configuration), adaptive matchmaking concentrates games among the leading contenders. The top group separates clearly, with nine configurations in the top bootstrap tier, while tail configurations play fewer decisive games and their individual positions are less sharply resolved. The overall ordering still holds. Across the 16 runtimes, the 96-configuration ladder correlates with the 600-game main ladder at Spearman 0.87 on marginal means and 0.83 on the Reference harness averaged over both prompts, and the new balanced round-robin in W2 / Q3 agrees with the main ladder at 0.88 for its eight runtimes. Resolving close configurations needs more games per entrant, as in the 16-runtime ladder (57–117 games per entrant), where three independent runs recover the same leader and top five. We will add this clarification to Section 4.2.


W2 / Q4a — What do the draws consist of?

We categorized every draw in the 96-configuration ladder and the visual-observation ladder from the match logs by applying four deterministic rules in priority order (where our stagnation rule declares a draw once a match has run for 12 minutes and neither side has attacked, advanced toward the enemy Capital, captured a cell, gained new vision, or assigned a task in the last 300 s, and we will add this rule to Appendix C.3):

Draw classRule (in priority order)96 configurations (1,450)Image observations (512)
Invalid actionsOne side has ≥25% of its requests rejected402 (27.7%)159 (31.1%)
No progressStopped by the stagnation rule326 (22.5%)81 (15.8%)
Unconverted leadTime limit, leader holds ≥2× the territory373 (25.7%)132 (25.8%)
ContestedTime limit, no 2× territory lead349 (24.1%)140 (27.3%)
  • Draws split evenly across the four behaviors. Invalid actions account for 27.7% of the factorial draws and 31.1% of the visual-ladder draws, while the remaining draws split across mid-game stalls, ≥2× territory leads that were not converted before timeout, and evenly contested matches (which also include defensive holds where neither side reaches a 2× territory lead).
  • Timeout draws are closely tied to inference latency. In the controlled tournament from W2 / Q3, the exact same 168 matchups produced a 23.2% draw rate in real-time play, but only 3.6% when the game clock was paused during model inference. We will add this classification and representative replays to Appendix E.

Q4b — Transfer to real tool-use tasks

There is currently no widely adopted, standardized benchmark for real-time, time-sensitive tool-use tasks—existing tool-use benchmarks hold the environment static while waiting for model responses, which is one of the main motivations for introducing GeneralsArena. To verify that our evaluation framework itself captures genuine tool-use capability, we evaluated all 16 arena runtimes on two established, latency-insensitive interactive tool-use benchmarks—Terminal-Bench (core 0.1.1, 77 buildable shell tasks with the Terminus-1 agent, single attempt) and tau2-bench (60 airline and retail customer-service tool-use tasks, single trial, using a local Qwen3.8-27B at temperature 0 as user simulator and judge across all runtimes)—using the exact same weights, serving stacks, and sampling settings:

Spearman correlation16 runtimes8 models
Turn-based ladder vs Terminal-Bench0.600.74
Turn-based ladder vs tau2-bench0.630.99
Turn-based ladder vs mean of both0.700.93
Real-time ladder vs Terminal-Bench0.550.62
Real-time ladder vs tau2-bench0.440.79
Real-time ladder vs mean of both0.530.83
Reference: Terminal-Bench vs tau2-bench0.680.79
  • Turn-based ratings correlate strongly with both external benchmarks. Across the 16 runtimes, the turn-based ladder correlates with the two-benchmark mean at Spearman 0.70, matching the agreement between the two external benchmarks themselves (0.68). Merging deployments of the same model weights by their median (8 models), the turn-based ladder correlates with tau2 at 0.99 and with the two-benchmark mean at 0.93 (inter-benchmark reference: 0.79).
  • Real-time ratings extend this tool-use signal to time-sensitive execution. Because Terminal-Bench and tau2 do not penalize per-step response time, our turn-based ladder aligns closely with them, confirming the validity of our evaluation method. Our real-time ladder then incorporates the real-time latency cost that static benchmarks omit: fast runtimes such as Qwen3.6 (1.7 s median latency) rank higher in real time than on the static benchmarks, while slow runtimes such as Qwen3.8 at xhigh reasoning (56 s) rank lower.

Thank you again for your careful feedback on the robustness and interpretation of our evaluation. Our responses above include sensitivity analyses for the ESR reference and availability-counting conventions, uncertainty estimates for availability and ESR, matched comparisons across timing protocols, and a behavioral breakdown of draws. We would greatly appreciate it if you could review these additions and share whether they address your concerns about ranking reliability and the interpretation of protocol effects. Please let us know if any concerns remain or if further clarification would be helpful. We would be happy to continue the discussion.

Comments on the paper

3 comments

We thank all reviewers for their thoughtful feedback and constructive suggestions. We have posted detailed responses with new experiments and analyses, including evaluations on Terminal-Bench and tau2-bench using the same runtime configurations, matched comparisons of real-time, turn-based, and paused-clock protocols, and further analyses of execution failures, ranking uncertainty, and ESR sensitivity. These additions aim to clarify the connection to established agent benchmarks, the effects of latency and deployment choices, and the interpretation of our rankings. We would greatly appreciate it if you could review our responses and let us know whether they address your concerns or whether any points require further evidence or clarification. We welcome continued discussion and would be happy to follow up on any remaining questions.

Dear Reviewers,

Thank you again for your thoughtful feedback. We have uploaded a revised manuscript that incorporates the experiments, analyses, and clarifications from our general and reviewer-specific responses. As the discussion period draws to a close, we would appreciate a brief reply letting us know whether the revisions address your concerns. If any questions remain, we would welcome the opportunity to clarify them or provide additional evidence before the discussion closes.

AuthorsOpenReview ↗

Final Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their careful evaluations and constructive suggestions. During the rebuttal process, we have conducted extensive additional experiments and analyses to strengthen the empirical foundation of GeneralsArena and clarify the scope and interpretation of our evaluation framework.

We would like to highlight the key improvements:

  1. External benchmark validation. We evaluated all 16 runtime configurations on Terminal-Bench and tau2-bench using consistent model weights and serving configurations. Our turn-based ratings show Spearman correlations of 0.60 and 0.63 with these benchmarks, respectively, and 0.70 with their combined scores, providing evidence that GeneralsArena captures capabilities relevant to established agent evaluations.

  2. Controlled analysis of latency sensitivity. We conducted 504 additional games with matched opponents, map seeds, seat assignments, and hardware across real-time, turn-based, and paused-clock protocols. The slowest reasoning configuration moves from last place in real-time evaluation to first place when inference latency is excluded, demonstrating that GeneralsArena can distinguish reasoning capability from latency-sensitive deployment performance through configurable protocols.

  3. Attribution of performance differences. We investigated the underlying causes of several major performance gaps through controlled experiments. Correcting Gemma4's tool-call formatting substantially reduces invalid actions, while a remaining performance gap reveals the impact of generation stalls. We also analyzed Qwen3.6's failures under the Pi harness and developed diagnostic analyses of invalid actions, execution stalls, and different categories of draws. These results demonstrate the importance of evaluating the complete agent system rather than model weights alone.

  4. Ranking reliability and uncertainty. We further analyzed the high draw rates and ranking uncertainty in the 96-configuration ladder, clarified the limits of fine-grained rankings, and reported uncertainty intervals for prompt comparisons. In particular, we revised our claims to acknowledge that the current evidence does not establish a consistent advantage for either prompt configuration.

  5. Service availability and ESR robustness. We provided sensitivity analyses of the Effective Service Rating across all 24 services, including uncertainty estimates, alternative availability-counting conventions, and cooldown statistics. The leading service remains unchanged across reference ratings from 1200 to 1800, supporting the robustness of the principal ranking while clarifying the interpretation and limitations of availability estimates.

Importantly, we have also clarified that end-to-end evaluation refers to the complete deployed agent system, including model weights, serving infrastructure, harness, and prompting strategy. GeneralsArena is intended as a configurable and reproducible evaluation platform, rather than a claim that a single game can capture every aspect of real-world agent performance.

All these experiments, analyses, and clarifications have been incorporated into the revised manuscript. We believe these additions substantially strengthen the reliability, practical relevance, and diagnostic value of GeneralsArena.

We sincerely appreciate the reviewers' valuable feedback, which has helped us improve both the experimental rigor and the presentation of our contributions. We hope the revised manuscript provides sufficient evidence to support a positive assessment of our work.

Thank you again for your time and consideration.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.