- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose GeneralsArena, an evaluation framework for agent systems based on persistent real-time strategy competition. In GeneralsArena, each entrant is a complete runtime–harness–prompt configuration competing in real time on partially observable maps, and each system decides how to plan, act, and organize its agents, while the game continues during reasoning and communication. In this way, a persistent ladder compares complete systems and several findings are found.
Critical strengths
GeneralsArena designs an evaluation framework which treats runtime, harness, and prompt as a complete system configuration, rather then comparing models alone. And the metrics capture abilities and constraints of different agents under real-world settings.
Critical weaknesses
- Although GeneralsArena assess different agents in a specific settings, this paper doesn't provide information and analysis of correlation between GeneralsArena with other agent benchmarks(e.g., SWE-bench, WebArena). For an useful benchmark, it is important to provide in-depth analysis.
- For an single game, it is hard to support the broad claim of "end-to-end agent system evaluation", therefore I think it is better to further refine the claim and contribution of this paper.
- The framework should provide better tools for attribute the score/advantages of different agents to different abilities;
- Latency sensitivity may systematically undervalue "slow but accurate" systems.
Questions for authors
See weaknesses in above.
4 comments
W2 — Scope of the end-to-end claim
GeneralsArena is a configurable match platform that provides diverse match environments as evaluation proxies for different real-world agent deployment scenarios. In our framework, "end-to-end" refers to evaluating the complete deployed system—model weights, serving engine, harness, and prompt working together—under controllable operational conditions:
- Configurable environments for different evaluation scenarios. By adjusting the timing protocol, observation modality, agent organization, and access limits (Section 3.1), the platform supports end-to-end evaluation across distinct settings: the 16-runtime and 96-configuration ladders test serving stacks and harness–prompt combinations (Sections 4.1–4.2), the subscription ladder incorporates service access limits via ESR (Section 4.3), and the protocol configurations test single-actor execution, visual observations, and turn-based timing (Section 4.4).
- Consistency with external benchmarks across settings (W1) and paused-clock extension (W4). In W1, we evaluated our runtimes on real-world agent benchmarks (Terminal-Bench and tau2-bench) and analyzed how different arena timing settings relate to benchmark performance. In W4, we further configured a paused-clock match environment to show how the platform can evaluate slow, high-accuracy systems without latency pressure.
- Exposing system-level differences with high reproducibility. Evaluating the complete system captures performance gaps that model-only evaluations miss: identical Gemma4 weights differ by 406 rating points across two official serving engines, while rankings across three independent 600-game ladders remain highly reproducible (Spearman 0.947, 0.891, and 0.950, Appendix B.2). We will clarify this platform scope in the introduction and contribution list.
W3 — Attributing scores to abilities
Our request and match logs map rating differences to concrete behaviors when we hold two of serving engine, harness and model fixed:
- Same model and harness across serving engines (Gemma4-26B on vLLM vs. SGLang). The 406-point gap for Gemma4-26B under Reference, vLLM (Rating 1371.1, 6W/46D/13L) vs. SGLang (Rating 1777.2, 43W/16D/37L), comes from two output differences. First, vLLM's official parser omits
move.sendon 5,868 of 7,505 rejected requests (30.9% rejection rate vs. 0.0% on SGLang). Second, vLLM has occasional ~180 s runaway generations that hit the token limit (36% of match time vs. 0% on SGLang), which slows opening expansion at tick 200 from 28.7 cells on SGLang to 12.5 cells on native vLLM and 12.9 cells on constrained vLLM. - Same model and serving engine across harnesses (Qwen3.6-35B under Reference vs. Pi). Comparing Qwen3.6-35B on SGLang under Reference (Rating 1718.7–1772.2, 46.2%–56.4% win rate) and Pi (Rating 1522.4–1537.3, Neutral 0W/41D/0L and Detailed 1W/39D/0L) shows how harness design shapes execution. Reference passes an explicit enum of legal move IDs, allowing Qwen3.6 to execute 268.7 valid decisions per 1,000 ticks with only a 3.5% rejection rate. Pi only constrains coordinate format, so once Qwen3.6 outputs a single out-of-view move, the rejection left in the history triggers an error cascade (jumping from 3.8% to 44.8% illegal moves) that stalls expansion into a draw.
- Different models under the same serving engine and harness (SGLang + Reference + Neutral). The logs also explain why models rank differently:
- Action validity at high speed (Qwen3.6-35B vs. Nemotron3.5-Lightning and Muse-Glimmer-30B): Qwen3.6 (1.67 s latency) and Nemotron3.5 (1.32 s) both act rapidly, but Qwen3.6 keeps rejections at 3.5% (75W/8D/30L, Rating 1880.2, Rank 1) whereas Nemotron3.5 wastes 15.8% of requests on invalid moves (5W/39D/20L, Rating 1310.9, Rank 16) and Muse-Glimmer-30B has 63.4% of requests rejected (3W/46D/8L, Rating 1391.7).
- Multi-agent delegation overhead (Qwen3.6-35B vs. Ornith-1.5-35B): Both are fast 35B-A3B MoE models on SGLang/FP8/off, but Qwen3.6 acts as a single root session (0% child games, Rating 1880.2), whereas Ornith-1.5 spawns child sessions in 92.3% of games, increasing contention and rejections (7.4%, Rating 1771.7). When delegation is disabled in our single-actor configuration, Ornith's latency drops from 2.53 s to 1.62 s and its win rate rises from 47.0% to 59.8%.
To make these analyses reusable, we designed a three-module diagnostic toolkit that we will release with the arena (and expose as structured post-game feedback for agents and developers): (1) an interface & error-cascade analyzer that breaks down tool-call rejections by error code and prior rejection streak, (2) a tempo, concurrency & stall profiler that tracks per-tick territory expansion, child-session contention, and ≥30 s generation stalls, and (3) an automated draw & outcome classifier that labels each draw as invalid actions, no progress, unconverted lead or contested.
W4 — Are slow but accurate systems undervalued?
In a real-time match environment, response latency does affect performance and can put slow but accurate systems at a disadvantage, although latency alone does not determine the overall ranking (as shown in Table 1, the fastest system is not the strongest). We chose the real-time setting as our primary configuration because in many real-world interactive deployments, response latency has a direct impact on user experience and operational effectiveness.
At the same time, GeneralsArena is a general, configurable evaluation platform that supports different match configurations for different evaluation criteria. In addition to the turn-based configuration in the paper, we configured a paused-clock match environment (halting the game clock while models think and advancing 10 ticks per round) and ran a 504-game controlled tournament across eight runtimes (median latencies from 1.7 to 56 s, playing all 28 pairs × 3 map seeds × both seats = 168 games per protocol on dedicated GPUs):
| Runtime (by median latency) | Median latency (s) | Real-time score (rank) | Turn-based score (rank) | Paused score (rank) | Paused − real-time |
|---|---|---|---|---|---|
| Qwen3.6 FP8 / SGLang off | 1.7 | 0.702 (1) | 0.452 (5) | 0.369 (5) | −0.333 |
| Ornith-1.5 FP8 / SGLang off | 2.5 | 0.702 (1) | 0.393 (6) | 0.262 (7) | −0.440 |
| Gemma4 BF16 / SGLang off | 2.6 | 0.512 (4) | 0.310 (8) | 0.310 (6) | −0.202 |
| Qwen3.8 FP8 / SGLang off | 8.5 | 0.357 (6) | 0.357 (7) | 0.190 (8) | −0.167 |
| Gemma4 BF16 / SGLang on | 9.0 | 0.702 (1) | 0.714 (1) | 0.738 (2) | +0.036 |
| Qwen3.8 FP8 / SGLang low + DSpark | 13.5 | 0.464 (5) | 0.655 (2) | 0.667 (3) | +0.202 |
| Qwen3.8 FP8 / vLLM low | 24.3 | 0.345 (7) | 0.619 (3) | 0.619 (4) | +0.274 |
| Qwen3.8 FP8 / SGLang xhigh | 56.1 | 0.214 (8) | 0.500 (4) | 0.845 (1) | +0.631 |
| Protocol pair (8 runtimes) | Spearman |
|---|---|
| Real-time vs turn-based | −0.02 |
| Real-time vs paused | −0.27 |
| Turn-based vs paused | 0.79 |
- Configuring turn-based or paused-clock play lets slow, high-accuracy systems take the lead. When we switch from real-time play to the paused-clock configuration, the slowest runtime—Qwen3.8 at xhigh reasoning (56 s median latency)—rises from last place in real time (rank 8, score 0.214) to first place (rank 1, score 0.845), a paired score gain of +0.631. Slower reasoning configurations such as Qwen3.8 + DSpark and vLLM low also rise into the top tier in both turn-based and paused-clock play.
- Latency-tolerant configurations align more closely with static benchmarks. Turn-based and paused-clock rankings agree closely with each other (Spearman 0.79), and as shown in W1, the turn-based configuration achieves a higher correlation with static agent benchmarks (0.70 on 16 runtimes and 0.93 on 8 models) than the real-time configuration (0.53). GeneralsArena can thus be configured either in real-time mode to account for deployment latency or in turn-based / paused-clock mode to evaluate deliberate reasoning without time pressure.
Thank you again for your thoughtful comments on the scope and practical relevance of our evaluation. In our responses above, we report correlations with Terminal-Bench and tau2-bench across all 16 runtimes, clarify the scope of “end-to-end” evaluation, and connect performance differences to concrete execution behaviors. We also provide a controlled comparison of timing protocols, showing how slower reasoning configurations can take the lead when the game clock is paused during inference. We would greatly appreciate your assessment of whether these additions address your concerns about benchmark relevance, diagnostic value, and latency sensitivity. Please let us know if any points warrant further clarification or discussion.
Thank you for recognizing the value of evaluating complete runtime–harness–prompt systems under realistic deployment constraints. We agree that validating against established agent benchmarks, clarifying our platform scope, and attributing scores to concrete behaviors are essential, and we address each point below.
W1 — Correlation with other agent benchmarks
To evaluate how GeneralsArena aligns with established agent benchmarks, we considered SWE-bench and WebArena alongside newer interactive tool-use benchmarks. SWE-bench focuses on repository-level software engineering patch generation, which carries high per-instance container execution cost across 16 serving configurations and is overly difficult for ~30B-scale open-weight models—compressing solve rates near the floor and making it hard to separate runtimes—while WebArena is an earlier web-navigation benchmark (2023) that is less discriminative for current tool-use agent workflows. We therefore selected two newer benchmarks that better match interactive agent scenarios—Terminal-Bench (core 0.1.1, 77 multi-step command-line tasks in a live shell with Terminus-1) and tau2-bench (60 multi-turn customer-service tool-use tasks across airline and retail, using a local Qwen3.8-27B at temperature 0 as user simulator and judge across all runtimes)—and evaluated all 16 arena runtimes under the exact same weights, serving engines, and sampling settings: