- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper makes two contributions. First, it introduces a game bench SC2FullGame-Bench on StarCraft II with various map, opponent level and race settings. It has 93 game cells. [3 races * 10 levels * 3 maps + 3 community bots]. The strength of various-level opponents are calibrated. It also reports reference scores on capability subarea of the game, which do not directly impact the main loop.
Second, it also creates SC2Forge, a write-test-edit framework driven by LLM. Starting from a seed package, an API index and an expert playbook, it iteratively rewrites the agent with six arms. After 6-stage verifier filter succeed, candidates are sent to play development panels and probe games to determine if the edit improves, regresses or has no gain against the parent iteration.
The main results of the experiment show that SC2Forge reaches 0.943 with Protoss race in 279 games (best one if 3 runs, average 0.914). It outperforms iterated prompt method and coding agent off the shelf, but it is statistically indistinguishable from baseline expert system VibeCraft, which is 0.932. Various ablation experiments are conducted to quantize the effect of each module. The system also generates solutions for zerg and terran races, and generalization results for connect-four and microrts environments.
Critical strengths
S1. Solid benchmarks and evaluation experiment design. It crosschecks the relative strength and reproduces the ladder. The builtin bots create levels with playing strength parity, along with scripted bot with army/worker cap.
S2. The details of the agent loop are clearly described, such as its available six arms, the six-stage verifier, acceptance criteria, and the final selection methods.
S3. The transparency of the experiment details, such as number of games and run sample size, estimates, and statistical significance. Detailed produced strategy, package, code are also present in the appendix.
Critical weaknesses
W1. The produced package is largely reproducing the playbook's source, so matching the source performance is weak evidence that the agent loop actually works. The playbook is already fed to the agent as an input; the package only differs from the seed package in one opening file, and the authors acknowledge "The package implements a build the playbook describes, not one the loop discovered" (§5.1).
W2. The necessity of the framework components are not established. “The delivered package is that generation-one revision, which the judge scored no gain”. (§5.1), so it is the result of a one-time agentic edit. The proposing arms, feedback and revision loops are not ablated.
W3. The reported best number is a best of 3 result 0.943, while the average result is 0.914, which is 0.018below the baseline VibeCraft 0.932.
W4. The bench's opponent strength range are limited. Vibecraft and SC2Forge reach 0.9+ scores, which means builtin AIs are no longer a challenging opponents. The opponent pool and playing style is also narrow. For example, the authors state that against early attacking bot Paul the system (0/9) performs much worse than the baseline Vibecraft (2/9). (Appendix A)
Questions for authors
-
Is there evidence that the framework produces strategy beyond the playbook?
-
With the weakness addressed in W2, can a minimal system with one agentic edit, veifier and no further iterations achieve similar performance? If so what is the necessity of other modules?
-
Would the authors report the three-run mean (0.914) as the primary result? Does it change the conclusion of main comparisons against VibeCraft?