acceptodds
Under review as a conference paper at ICLR 2027

ALAMEINBENCH: EVALUATING LONG-HORIZON STRATEGIC PLANNING OF LLM AGENTS IN A HISTORICAL WARGAME

Abstract

Evaluating LLM agents in strategic environments requires tasks that combine long-horizon planning, coordinated resource allocation, and adaptation to an opponent under explicit rules. We introduce AlameinBench, a benchmark built on a turn-based hex-grid wargame for evaluating these capabilities over complete campaigns. It comprises three scenarios centered on supplied advance, mine clearance, and timed withdrawal, each evaluated from two playable roles. The resulting six scenario–role conditions couple multi-unit coordination, logistics, phase-dependent actions, and delayed scoring within a shared game environment. AlameinBench combines a common observation and action interface, rule-validated execution, and an evaluation protocol covering terminal outcomes, scenario-specific progress, execution reliability, and computational cost. We also introduce Strategy–Allocation–Execution (SAE), a modular neuro-symbolic reference agent that connects strategic objectives to unit assignments and executable actions. Using this framework, we evaluate six LLM profiles alongside four LLM-free heuristic policies and a rule-based self-play reference. In the reported fixed-seed evaluation, performance varies across scenarios and roles, and a deterministic domain-informed heuristic matches or exceeds the best SAE profile in four of six conditions. Campaign traces further reveal differences in objective maintenance, task execution, and score sources. AlameinBench provides a structured testbed and reference baselines for studying long-horizon planning and strategic decision-making in LLM-based agents.

Primary area:
datasets and benchmarks
Keywords:
large language model agents, strategic reasoning, long-horizon planning, rule-grounded environments, wargame benchmarks, hierarchical planning, tool use, agent evaluation, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YngAOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce AlameinBench, a turn-based historical wargame benchmark for evaluating long-horizon strategic planning in LLM agents. The benchmark consists of three scenarios involving supply-constrained advances, minefield clearance, and timed withdrawals, each evaluated from both Axis and Allied perspectives. The authors also propose Strategy–Allocation–Execution (SAE), a modular neuro-symbolic reference agent, and evaluate six LLM profiles within this architecture alongside LLM-free heuristic policies and a rule-based self-play baseline.

Critical strengths

None

Critical weaknesses
  • Insufficient experimental rigor: The evaluation relies on a single fixed seed per condition, despite the stochastic nature of the game. Without multiple runs or statistical analysis, it is difficult to assess the reliability of the reported performance differences and model rankings.
  • Limited novelty: The proposed benchmark shares similarities with existing wargame benchmarks (e.g., WGSR-Bench [1] and TextWargame [2]), while the SAE architecture builds upon established neuro-symbolic planning approaches (e.g., LLM-Modulo [3]). Although the benchmark combines logistics, phase constraints, and multi-unit coordination, the paper does not clearly establish what novel evaluation capabilities or methodological insights it offers beyond prior work.
  • Missing architectural ablations: The paper lacks ablation studies to assess the contributions of individual SAE components, such as strategic decomposition, persistent task management, and symbolic checking. Comparisons against simpler LLM-based agents without SAE are also missing, making it difficult to establish the effectiveness of the proposed architecture.
  • Limited benchmark diversity and discriminative power: The benchmark consists of only three fixed scenarios, two playable roles, and a single rule-based opponent, limiting its ability to assess generalizable strategic planning capabilities. Moreover, several conditions yield nearly identical scores across models. For example, in Table 4, five of the six SAE profiles score 35 in September Axis, and five score 55 in October Allies. This raises concerns about whether these conditions meaningfully distinguish agents' strategic capabilities.
  • Limited insights into strategic planning capabilities: Although the paper provides score decompositions and selected trajectory analyses, it does not systematically connect performance differences to specific capabilities such as long-horizon planning, resource allocation, and adaptation to opponents. It also remains unclear whether the evaluation distinguishes strategic planning from action frequency and local tactical execution. Consequently, the results characterize task performance more clearly than they reveal models' underlying strategic strengths and weaknesses.
  • Insufficient validation of LLM-Judge metrics: The supplementary LLM-Judge metrics, including missed opportunities and plan coherence, lack independent validation against human expert assessments or other objective measures. This raises concerns about their reliability as indicators of strategic planning quality.

References

[1] Qiyue Yin et al., "WGSR-Bench: Wargame-based Game-theoretic Strategic Reasoning Benchmark for Large Language Models," arXiv, 2025. https://arxiv.org/abs/2506.10264
[2] Zilie Chen et al., "TextWargame: A Benchmark Evaluating LLMs on Strategic Reasoning," ICCCS, 2026. https://doi.org/10.1109/ICCCS69761.2026.11613233
[3] Subbarao Kambhampati et al., "Position: LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks," ICML, 2024. https://arxiv.org/abs/2402.01817

Questions for authors
  • Role-dependent model behavior: What underlying model characteristics or strategic behaviors explain the Axis–Allies performance differences in Table 5? Could the authors provide trajectory-level evidence to substantiate their claims of model specialization?
  • Robustness across opponent policies: All agents are evaluated against the same rule-based opponent. Could the authors additionally evaluate SAE agents against SR0–SR3 to investigate whether the reported performance differences persist across different opponent policies?
4 comments

We thank the reviewer. Our contribution is AlameinBench, a benchmark simulating First El Alamein, Alam el Halfa, and Second El Alamein in 1942 North Africa. SAE is one baseline.

1. Experimental rigor. We evaluate six models and four architectures across six conditions with a common seed and opponent. The seed controls combat and non-engineer clearance rolls; we consider its overall impact small. Maps, deployments, phases, objectives, and scoring rules are fixed. Engineer clearance is deterministic when eligible; engine events record task outcomes.

2. Novelty. WGSR-Bench reports reasoning subtasks and an interactive example agent; TextWargame emphasizes text interfaces and action hierarchies. AlameinBench jointly links plans, actions, engine effects, score components, and costs over complete campaigns, a protocol absent from the examined WGSR report. This exposes preservation versus interdiction behind an improved defensive score. SAE implements principles related to LLM-Modulo; our contribution is the executable task family and joint diagnostics.

3. Architecture comparisons and simpler baselines. Direct selects actions; Hybrid combines model phase goals, program-generated candidates, and model selection; OAH adds risk-benefit filtering. These three omit SAE's persistent tasks and independent checker. All agents undergo legality checks. These provide explicit system comparisons of planning, candidate filtering, and task management under each architecture's tools and budgets.

Jul/Sep/Oct = July/September/October; A = Axis, L = Allies. Each cell lists Direct/Hybrid/OAH/SAE in final Axis VP.

ModelJul-AJul-LSep-ASep-LOct-AOct-L
GPT-6 Astra34/32/31/3129/28/29/3771/62/47/3836/50/38/5366/60/60/6556/55/42/55
Claude Opus 534/28/31/3128/28/28/3051/44/47/3537/35/35/4859/64/60/4354/56/58/55
Claude Sonnet 525/25/25/4328/28/28/2854/44/46/3536/38/35/4165/61/63/3654/62/57/53
Kimi K355/34/31/2528/28/28/3851/56/50/3537/38/35/5061/60/64/1744/43/59/55
DeepSeek V4 Pro25/39/34/3129/28/28/2856/62/58/3545/35/35/4762/64/64/6156/53/54/55
GLM-5.331/25/31/2529/28/28/3453/46/41/3536/37/35/5060/63/64/5651/52/55/55

4. Diversity and discrimination. Supply-qualified advance, coordinated clearance, and timed withdrawal test distinct dependencies from both roles. In the two highlighted conditions, Direct/Hybrid/OAH yield 12 distinct scores spanning 41-71 in September-Axis and 13 spanning 42-62 in October-Allies, across 18 model-architecture results each. These differences demonstrate discrimination within this task family despite SAE's clustering.

5. Strategic capabilities. We connect preparation, unit coordination, resource eligibility, and replanning to resulting task effects. DeepSeek's September-Axis run has 162 moves and nine replans but no clearance points. In July-Axis, DeepSeek remains at scoring column 36, while Sonnet reaches 40 with qualifying supply. These traces distinguish repeated activity from sustained objective progress.

6. Judge validity. Rule-computed outcomes anchor task-success claims; Judge assessments supplement them. Conversion measures confirmed opportunity use, severity grades misses, and coherence assesses within-turn action connections. Rubrics, step citations, duplicate/reference checks, and abstention make outputs inspectable. These checks establish protocol conformance and traceability to recorded steps. SR3's 46/50 conversion (92%) but improvement over SR0 in only one of six conditions shows that opportunity use and campaign success capture distinct outcomes.

Q1. Role-dependent behavior. September scoring records show GPT-6 Astra clears one mine as Axis (3 VP) but allows six to be cleared as Allies (18 VP); Sonnet 5 clears none but allows two (6 VP). In October-Allies, DeepSeek makes no moves or attacks in turns 11-15 while relaxing its withdrawal ceiling; Sonnet makes 16 late moves, and enemy withdrawal rewards are 75 versus 73. All SAE Allied runs avoid isolation penalties. These action and scoring records explain the observed role differences in objective pursuit and denial.

Q2. Different opponents. SR0-SR3 are evaluated baselines facing the common rule-based opponent. They share candidate generation/scoring and vary deterministic or randomized selection, providing heuristic references for model decisions. Keeping the opponent fixed makes task comparisons interpretable across systems. Their zero in-game model calls also anchor cost comparisons. SR0 matches or exceeds the best SAE score in four conditions, providing a direct reference for LLM-based planning.

I thank the authors for mostly addressing my concerns on paper's positioning (novelty), architecture comparison & simpler baselines and diversity & discrimination & judge validity and remaining questions. But, I think claim that seed effects are "small" is unsupported without justification because all results conducted with single runs. So, I recommend the author conduct the multi-seed experiments. In addition, I am not convinced that the authors provided meaningful insights about connection between specific strategic capabilities and the model's capabilities beyond simply reporting the models' scores. Could you provide further analysis linking the scores to concrete strategic capabilities?

Considering the current rebuttal status, I raised the score from clear rejection to weak rejection

Thank you. We added 12 direct GLM-5.3 SAE-versus-SR0 games across all six conditions and two seeds.

1. Central claims (W2-W4). AlameinBench contributes a historically grounded benchmark and auditable protocol; SAE is a reference baseline. SAE is strictly worst in 22/36 cells. Against the best LLM architectures, SR0 ties twice and loses four times. We will revise the Abstract, Sections 6-7, and Conclusion accordingly. Kimi's 17 under SAE versus 60-64 under other architectures makes its weakness configuration-dependent. Model specialization and action-count associations will be scoped to the tested configuration, without claiming SAE superiority or isolated component effects.

2. GLM's -13. We identified and fixed a bug affecting the original run; the corrected score is 56. Both used seed 1942, so this is not a seed comparison. We will correct Table 4, retain the original record, and keep original Judge correlations separate from corrected scores.

3. New experiments (W1, Q2). GLM-SAE directly faced SR0 under seeds 1942/1943 with identical code, prompts, model settings, budgets, and failure handling. Both batches used the same reasoning effort and streaming. Scores below are Axis VP; roles identify GLM's side.

ScenarioRole19421943
JulyAxis2831
JulyAllies2525
SeptemberAxis4845
SeptemberAllies4450
OctoberAxis5047
OctoberAllies5757

The mean absolute score difference is 2.5 VP, maximum 6 VP. These measurements replace our blanket small-seed-effect claim. As A.3 explains, shared seeds do not align random draws across different action histories; model generation also varies. These results cover GLM-SAE versus SR0, not all-architecture rankings or SR1-SR3 robustness.

4. Reliability and planning (W5). Logs separate transport/protocol errors, rejected proposals, fallback actions, execution, and objective progress. All 12 games reached a final outcome, including eight fallback actions. The 162-move/no-clearance case shows why this separation matters: activity can coexist with no progress on the clearance objective. We will report reliability alongside task progress.

5. Judge (W6). Judge links structured annotations to trajectory evidence. Schema/citation checks establish conformance and traceability. Primary conclusions rely on engine scores and recorded execution; Judge provides auxiliary diagnostics. The six-profile correlation will be presented as exploratory association, not validation of judgment accuracy.

Thank you for the clarification. Based on the recent response, I decided to raise the score from weak reject to weak accept.

Reviewer_s9hkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a benchmark meant to study long-horizon planning in a multi-agent environment with different tasks (e.g., resource allocation, adaptation to the opponent). The paper then compares several methods (including LLM-based methods) on this benchmark.

Critical strengths

A useful benchmark can encourage research in a specific field/task. The tasks discussed in the paper are broadly relevant. LLM-based agents are currently subject of much research, including that on AI-based game agents.

Critical weaknesses

Proposing a benchmark is an important contribution but the problem being studied needs to be formally defined first. Otherwise we can end up with a specific piece of software which allows us to measure something within it but it is unclear whether such measurements are very specific to this benchmark or applicable to the broader problem. I will use pathfinding as an example. The publication of game maps and pathfinding problems on them by Sturtevant on his MovingAI website was helpful not only because the results in the many subsequent papers that used MovingAI maps/problems became comparable but also because the measures defined by the benchmark were scientifically interesting beyond the benchmark itself. Indeed path suboptimality is relevant to shortest-path pathfinding, real-time search, multi-agent pathfinding and many other problems/tasks.

What I did not find in this paper is the formal problem being studied by the benchmark (i.e., something analogous to the shortest-path problem in graph search). Without understanding the underlying mathematical problem is difficult to appreciate the design decisions in the benchmark. For instance, is it important that there are "six scenario-role conditions"? And why six and not 60? And why are the sides called Axis or Allies? Why not side A and side B?

The authors claim that the benchmark supports the evaluation of multi-unit coordination, planning, action timing and adaptation "over complete campaigns". But none of those four tasks listed appears to be defined mathematically. Their names are broad (e.g., multi-unit coordination) and the objective functions involved in the evaluation can substantially change based on the specifics (e.g., multi-agent pathfinding uses objective functions different from multi-unit attacks). The language in which the benchmark is described is very game-specific (e.g., "cleared mines"). I am thus unclear how portable the findings reported in the paper are.

Questions for authors

None

1 comment

We thank the reviewer. AlameinBench is a benchmark for coordination under resource, dependency, and timing constraints. Its historical campaigns instantiate this decision problem; SAE is one baseline. We study how feasible decisions produce delayed objective gains.

1. What is the underlying mathematical problem?

AlameinBench is a family of finite-horizon, two-player stochastic games with centralized control of each side's units. A task is indexed by scenario cc and role rr. State sts_t includes unit positions and attributes, supply, active side, turn, phase, and scoring state; both sides' state is available. Legal actions satisfy:

st+1=Fc(st,at,ξt),at∈Ac(st)s_{t+1}=F_c(s_t,a_t,\xi_t),\quad a_t\in\mathcal{A}_c(s_t)

where ξt\xi_t represents rule-governed randomness. A policy maps observations and interaction history to actions. Let TT be termination and Vc(sT)V_c(s_T) the terminal Axis score. Utilities are:

UAxis,c=Vc,UAllies,c=−VcU_{\mathrm{Axis},c}=V_c,\quad U_{\mathrm{Allies},c}=-V_c

Against opponent policy μ\mu, the objective is:

max⁡πEπ,μ[Ur,c(sT)]\max_{\pi}\mathbb{E}_{\pi,\mu}[U_{r,c}(s_T)]

The engine computes VcV_c from the initial score and scoring events, with explicit eligibility conditions and contributions. Each scenario thus specifies which state transitions earn value.

2. How are coordination, planning, timing, and adaptation defined within this problem?

These are coupled requirements for maximizing the same utility. Coordination concerns unit actions whose feasibility or effects depend on other units' states and actions. Resource allocation selects which units and supply resources support an objective under those constraints. Planning selects action sequences that establish prerequisites for later rewards. Timing restricts eligibility through turn and phase. Adaptation updates the policy's decisions as observations and execution results change.

Write P(s)P(s) for an action's prerequisites and G(s′)G(s') for its intended effect, defined by scenario rules. Establishing one predicate can enable subsequent actions. July checks supply eligibility for frontier gains; September links qualifying clearance events to rewards; October requires eligible western exits after turn 10. Traces show whether PP held, aa was executed, GG followed, and the corresponding score event occurred. This makes the capability discussion operational within the stated decision problem.

3. Why six conditions rather than sixty?

The six conditions follow AlameinBench's historical scope: three battles in the 1942 North African campaign—First El Alamein (July), Alam el Halfa (September), and Second El Alamein (October)—each evaluated from both sides. These stages motivate simulated objectives: supplied advance, minefield clearance, and withdrawal within a time window. Terrain, deployments, and resource asymmetries give these objectives an operational setting.

Thus, 3 scenarios × 2 roles yields six conditions, each a complete multi-turn campaign. Both roles matter because achieving an objective and preventing it impose different decisions. Expanding to sixty would add coverage through meaningful changes in constraints, objectives, or historical settings. The present scope supports a focused study of strategic decisions across three historical battles and both sides' perspectives.

4. Why Axis and Allies rather than A and B?

Axis and Allies preserve the historical roles in simulations of First El Alamein, Alam el Halfa, and Second El Alamein (July, September, and October 1942). This context connects terrain, deployments, resource asymmetries, and objectives. Campaign simulation makes force coordination, supply, and timing executable decisions with rule-verifiable consequences. The names make each role's constraints and goals interpretable, preserving the historical basis of the task design and its strategic requirements.

5. What is scientifically meaningful beyond this game?

Travel assistants must coordinate flights, hotels, and transfers within a budget and recover from cancellations. Embodied household agents must complete dependent steps in unfamiliar homes. Figure's Helix 2.5 reports 56% full-task success across three chores in 30 unseen homes, illustrating the challenge of reliable completion.

AlameinBench makes this decision layer inspectable through rules, traces, and objective effects. DeepSeek's 162 movements and nine replans produce no clearance gains: activity fails to deliver progress. October score decomposition separates preserving friendly units from denying withdrawal. Analogously, a travel agent spending less after abandoning a required booking, or a robot folding a towel but failing to put it away, has not completed the intended task.

Our contribution is an evaluation method linking action dependencies, execution traces, and objective completion, diagnosing where a plan fails to produce the intended outcome.

Reviewer_Qht3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AlameinBench, a benchmark for evaluating long-horizon planning in LLM agents through a turn-based historical wargame. It includes three scenarios involving supplied advance, mine clearance, and timed withdrawal, each evaluated from both sides. The benchmark uses final game scores, action traces, rule compliance, and supplementary LLM-judge metrics. The authors also introduce Strategy–Allocation–Execution (SAE), a reference agent combining LLM-based planning with explicit task tracking and rule-checked execution. Experiments evaluate six LLM profiles using SAE, alongside four LLM-free heuristic policies and a rule-based reference. The results show that a rule-based heuristic matches or outperforms all SAE models in four of the six settings. The authors also analyze game logs to identify planning failures and explain how final scores were obtained.

Critical strengths

Useful Trace-Based Analysis. The paper links agent actions to game states and final scores, allowing a closer look at planning failures. For example, one agent makes 162 moves without achieving its mine-clearance objective. Such analyses help explain agent behavior beyond final scores.

Critical weaknesses
  1. Unclear Benchmark Novelty. Existing game benchmarks already evaluate strategic decision-making and long-horizon planning [1–3]. AlameinBench introduces different game rules and trace-based analysis, but it is unclear what additional insights these features provide compared with existing benchmarks.
  2. Insufficient Validation of Strategic Capabilities. The paper aims to evaluate long-horizon planning, multi-unit coordination, resource allocation, and strategic adaptation. However, it is unclear how well the proposed metrics distinguish these abilities. The case studies are useful, but the paper lacks controlled experiments to validate the claimed evaluation capabilities.

[1] 2024 – ICLR – CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents

[2] 2024 – NeurIPS – Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach

[3] 2025 – ICLR – BALROG: Benchmarking Agentic LLM and VLM Reasoning on Games

Questions for authors
  1. What specific evaluation capabilities or insights does AlameinBench provide that existing game benchmarks [1–3] do not? In particular, what additional value does its trace-based analysis offer?
  2. How do the proposed metrics distinguish long-horizon planning, multi-unit coordination, resource allocation, and strategic adaptation? What evidence supports their effectiveness in evaluating these individual capabilities?
  3. How reliable are the reported model comparisons given that each scenario–role condition is evaluated with a single random seed? Is there any evidence that the observed differences remain consistent across different seeds or game conditions?
2 comments

We thank the reviewer. AlameinBench is our benchmark for verifiable strategic execution in three battles of the 1942 North African campaign. SAE is one baseline.

1. What capabilities and insights does AlameinBench add?

Compared with CivRealm, TextStarCraft II, and BALROG, AlameinBench adds joint, rule-verifiable assessment of four execution capabilities: maintaining goals across turns; coordinating dependent unit actions; allocating units and supply to meet prerequisites; and adapting to changed states while restoring goal progress. Traces link plans and actions to rule-verified effects, scores, and costs.

Historical tasks expose these dependencies: supplied advance in July, clearance preparation in September, and positioning for October's withdrawal window. Both roles assess objective achievement and denial.

Score decomposition reveals the source of improvement. In October-Allies, rule self-play yields -20 + 73 withdrawal + 7 isolation = 60 Axis VP; most SAE runs yield -20 + 75 + 0 = 55. Gains reflect avoiding friendly isolation while enemy withdrawal rewards rise. Traces distinguish force preservation from withdrawal interdiction.

We compare six models and four architectures across six conditions with a common opponent policy, scenario setup, and scoring rules. Direct selects actions; Hybrid combines model-defined phase goals, program-generated candidates, and model selection; OAH adds risk-benefit filtering. SAE uses persistent tasks and independent checking. All agents undergo legality checks. Tools and budgets vary by architecture; these are whole-system comparisons.

Jul/Sep/Oct = July/September/October; A = Axis, L = Allies. Each cell lists Direct/Hybrid/OAH/SAE in final Axis VP.

ModelJul-AJul-LSep-ASep-LOct-AOct-L
GPT-6 Astra34/32/31/3129/28/29/3771/62/47/3836/50/38/5366/60/60/6556/55/42/55
Claude Opus 534/28/31/3128/28/28/3051/44/47/3537/35/35/4859/64/60/4354/56/58/55
Claude Sonnet 525/25/25/4328/28/28/2854/44/46/3536/38/35/4165/61/63/3654/62/57/53
Kimi K355/34/31/2528/28/28/3851/56/50/3537/38/35/5061/60/64/1744/43/59/55
DeepSeek V4 Pro25/39/34/3129/28/28/2856/62/58/3545/35/35/4762/64/64/6156/53/54/55
GLM-5.331/25/31/2529/28/28/3453/46/41/3536/37/35/5060/63/64/5651/52/55/55

Direct/Hybrid/OAH span 41-71 VP in September-Axis and 42-62 in October-Allies, where SAE scores cluster. The tasks distinguish systems beyond SAE.

2. How do metrics distinguish the four capabilities?

Diagnostics connect prerequisites, actions, state changes, and objective gains:

  • Planning: Do earlier preparations enable later objectives? Cross-turn traces connect positioning to supply-qualified advance, clearance, or eligible withdrawal.
  • Coordination: Do unit actions form a working sequence? Unit-level traces connect staging and follow-up; coherence (1-5) assesses connections across phases within a turn.
  • Allocation: Are units and supply placed where task prerequisites can be met? Assignment and state records identify resource availability; scoring events establish resulting progress.
  • Adaptation: Do responses to changed states or stalled tasks restore progress? Replanning triggers are compared with subsequent actions and objective effects.

DeepSeek's September-Axis run records nine replans (six for stalled tasks, three for enemy changes) and 162 moves, yet zero clearance points. The four demands interact; the matrix evaluates how whole systems turn them into objective gains, with traces locating execution breakdowns.

Judge conversion measures the fraction of confirmed opportunities used; severity (0-3) grades misses. Citations, reference checks, and abstention support inspection. Campaign outcomes and cross-turn traces assess long-term effectiveness; coherence describes within-turn execution. SR3 uses 46/50 opportunities (92%) yet improves over SR0 in only one of six conditions, showing that opportunity use and campaign effectiveness capture distinct outcomes.

3. How reliable are single-seed comparisons?

The seed controls combat and non-engineer clearance rolls; we consider its impact on overall performance small. Maps, deployments, unit attributes, phases, objectives, and scoring rules are fixed. Agents determine coordination, supply allocation, and withdrawal timing; eligible engineer clearance is deterministic.

SAE GPT-6 Astra scores 38 versus Sonnet 5's 35 in September-Axis, but allows 53 versus 41 Axis VP as Allies, showing role-dependent outcomes. Actions and engine scoring events confirm clearance failures and distinguish force preservation from withdrawal interdiction.

Thank you for the detailed rebuttal and additional experimental results. While the clarifications are helpful, my main concerns regarding the benchmark's distinct value over existing work and the validation of its claimed strategic capabilities remain insufficiently addressed, so I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.