acceptodds
Withdrawn from ICLR 2027

SC2Forge: A Calibrated Benchmark and a Program-Writing Agent for Full-Game StarCraft II

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A language model can play a real-time strategy game from inside the control loop or write the program that plays it. The first needs a model call per decision and, on our benchmark, at least 5.5 wall-clock seconds per game second; the second delivers a compiled package that plays a typical game faster than real time, with no model call. We contribute SC2FullGame-Bench, which scores a StarCraft II bot package on 93 full-game cells, 90 against the built-in opponents (three maps, ten levels and three races) and 3 against a community bot. It also rates each package with four probe games (economy, tech, combat and defense). Before any submission is scored, the benchmark is calibrated: two ladders of known order and a package's agreement with itself on a repeated seed show that it separates agents of different strength. We also contribute SC2Forge, a framework that turns a minimal demonstration package, the runtime's documentation and a playbook of named openings into a compiled package through a write-test-revise loop with a verifier ladder, a tool-using rewrite arm and a capability-keyed archive. SC2Forge's package scores 0.943, its best of three runs, at the level of VibeCraft's expert-system Protoss package (0.932), the bot its playbook was written from. The package is separated above an iterated prompt (0.767) and an off-the-shelf coding agent (0.803); a coding agent given the playbook under the framework's own model budget reaches 0.900, within 0.05 per cell of the framework's package on sealed suite A's six seeds and separated below it on the released suite's six seeds. The framework also generates Terran and Zerg packages, each scoring above VibeCraft's package for that race; on connect-four the loop, its arms adapted to the game, delivers a policy scoring 0.977 against a scripted 0.611.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, Benchmark, StarCraft II

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mns5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper makes two contributions. First, it introduces a game bench SC2FullGame-Bench on StarCraft II with various map, opponent level and race settings. It has 93 game cells. [3 races * 10 levels * 3 maps + 3 community bots]. The strength of various-level opponents are calibrated. It also reports reference scores on capability subarea of the game, which do not directly impact the main loop.

Second, it also creates SC2Forge, a write-test-edit framework driven by LLM. Starting from a seed package, an API index and an expert playbook, it iteratively rewrites the agent with six arms. After 6-stage verifier filter succeed, candidates are sent to play development panels and probe games to determine if the edit improves, regresses or has no gain against the parent iteration.

The main results of the experiment show that SC2Forge reaches 0.943 with Protoss race in 279 games (best one if 3 runs, average 0.914). It outperforms iterated prompt method and coding agent off the shelf, but it is statistically indistinguishable from baseline expert system VibeCraft, which is 0.932. Various ablation experiments are conducted to quantize the effect of each module. The system also generates solutions for zerg and terran races, and generalization results for connect-four and microrts environments.

Critical strengths

S1. Solid benchmarks and evaluation experiment design. It crosschecks the relative strength and reproduces the ladder. The builtin bots create levels with playing strength parity, along with scripted bot with army/worker cap.

S2. The details of the agent loop are clearly described, such as its available six arms, the six-stage verifier, acceptance criteria, and the final selection methods.

S3. The transparency of the experiment details, such as number of games and run sample size, estimates, and statistical significance. Detailed produced strategy, package, code are also present in the appendix.

Critical weaknesses

W1. The produced package is largely reproducing the playbook's source, so matching the source performance is weak evidence that the agent loop actually works. The playbook is already fed to the agent as an input; the package only differs from the seed package in one opening file, and the authors acknowledge "The package implements a build the playbook describes, not one the loop discovered" (§5.1).

W2. The necessity of the framework components are not established. “The delivered package is that generation-one revision, which the judge scored no gain”. (§5.1), so it is the result of a one-time agentic edit. The proposing arms, feedback and revision loops are not ablated.

W3. The reported best number is a best of 3 result 0.943, while the average result is 0.914, which is 0.018below the baseline VibeCraft 0.932.

W4. The bench's opponent strength range are limited. Vibecraft and SC2Forge reach 0.9+ scores, which means builtin AIs are no longer a challenging opponents. The opponent pool and playing style is also narrow. For example, the authors state that against early attacking bot Paul the system (0/9) performs much worse than the baseline Vibecraft (2/9). (Appendix A)

Questions for authors
  1. Is there evidence that the framework produces strategy beyond the playbook?

  2. With the weakness addressed in W2, can a minimal system with one agentic edit, veifier and no further iterations achieve similar performance? If so what is the necessity of other modules?

  3. Would the authors report the three-run mean (0.914) as the primary result? Does it change the conclusion of main comparisons against VibeCraft?

Reviewer_hYPPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SC2FullGame-Bench, which scores a StarCraft II bot on 93 full-game cells: three maps, ten built-in AI levels and three races, plus three games against one community bot. The benchmark is calibrated by checking that it recovers the order of two ladders of known strength, and by measuring same-seed outcome agreement to size its paired sign and cell-permutation tests. It also proposes SC2Forge, an LLM write-test-revise loop (six proposal arms, a static and smoke-test “verifier ladder”, a 12-cell development panel, a judge and a MAP-Elites archive) that turns a minimal seed bot, an API index and a playbook of openings taken from the VibeCraft Protoss bot into a bot that plays without model calls. The best of three Protoss runs scores 0.943 (mean 0.914), not separated from VibeCraft (0.932) and above an iterated-prompt baseline (0.767), an off-the-shelf coding agent (0.803) and an in-loop LLM agent (0.441); ablations, Terran and Zerg runs, two sealed map suites, and transfers to connect-four and microRTS are also reported.

Critical strengths

The evaluation is more careful than is typical for LLM game agents: paired comparisons on identical cells and seeds, a measured noise floor used to plan sample sizes, separation claimed only when two tests agree, held-out map suites, and preregistered predictions with the failed ones reported. The paper is candid about its limits, for instance that the delivered bot implements a build from the playbook rather than one the loop discovered. The cost argument for writing a program instead of putting the model in the control loop is concrete. The generated bot needs no model calls and plays faster than real time, while the in-loop agent needs 581 calls per game and at least 5.5 wall-clock seconds per game second.

Critical weaknesses

First, the loop adds little beyond its expert input. The delivered bot differs from the seed only in its opening plan, which now follows the playbook’s blink-stalker build. It came from generation one and was scored “no gain” by the loop’s own judge, and generations 2 to 5 accepted nothing. It does not exceed the bot the playbook was taken from, and it is not separated from a coding agent given the same playbook and token budget (0.900) on three seeds or on sealed suite A. The headline 0.943 is also the best of three runs, compared against single-run baselines.

Second, the benchmark is near its ceiling for the agents it compares. The calibration only establishes ordering up to built-in lv10 (0.900), below where the main comparisons sit. Only 3 of 93 cells use a non-built-in opponent, and in a supplementary study the bot won none of nine games against an early-attacking AI Arena bot (Paul), against six for VibeCraft. The paper is also very hard to read: the main text is dense with numbers and coined terms, and much of what is needed to check the claims is in 25 pages of appendices.

Questions for authors
Reviewer_CAK8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SC2FullGame-Bench, a benchmark for video game player agents that uses StarCraft II, and SC2Forge, an agent that evolves a symbolic SC2-player agent using repeated calls to an LLM (DeepSeed-V4-Flash) using a MAP-Elites archive. This evolutionary approach produces a code-based player agent that beats several baseline agents and ablation conditions, but does not significantly outperform an existing code-based player agent which is itself provided as the initial parent in the evolutionary loop. If I understand correctly, SC2Forge is run for 3 seeds, taking the best agent from the 3. Also, evolution does not appear to improve the scripted agent beyond the first generation. The main selling point of the SC2FullGame-Bench framework seems to be that it reproduces two “ladders” of ability among prior agents, namely reproducing the hierarchy among in-game bots and among variants of a scripted player agent that is given various caps on the number of units it may produce.

Critical strengths

With the caveat that I have not done a literature review, this would appear to be the first work that uses LLM-driven evolution to generate coding agents for player full SC2 games. But the present work does not do this idea justice.

Critical weaknesses

For what it’s worth, I find the paper completely unreadable due to the apparent ubiquity of unconstrained AI-generated text.

As is stated in the conclusion, the proposed approach involving evolution through large models is unsuccessful: it does not result in a substantial improvement over the base agent that was used to seed the evolutionary process. It is also unclear what exactly the MAP-Elites archive’s feature dimensions are, or even how the overall evolutionary loop works, due to the prevalence of muddled AI-generated jargon, and the inclusion of disparate and inconsequential facts all over the place.

The contribution of the benchmark is unclear. It would appear to just be StarCraft-II itself, which as I understand it already exists as a benchmark in various forms. Much ado is made about “ladders” being reproduced by the benchmark, but it’s unsurprising that having in-game bots of various difficulty levels play against each other should reproduce this difficulty ranking. This is by definition and by design. The same goes for enforcing variously strong handicaps to a scripted agent. This preliminary ladder-reproduction phase is then apparently used to determine the number of samples that should be run when evaluating additional agents, but this methodology is not clearly presented.

Maybe the key idea of the benchmark is to have newly-proposed agents play against a bunch of predefined agents in order to determine the agent’s score? Fine, but really, if we want to rank a pair of SC2 agents, wouldn’t the first order of business be to run these agents against each other? (It looks like the present work never does this, by the way, when comparing agents.) Certainly, something like an Elo score from among a pool of agents will also be of interest, but this is a ubiquitous idea. Indeed, why not use Elo score instead of the costly 93-cell evaluation here?

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.