- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies coding agents for AFSIM. Its scripting language is rare in training data, and a task counts as solved only if the required events appear in the simulation trace, not merely if the project compiles and runs. The paper proposes AFSIM-Agent with a specific workflow. It also introduces AFSIM-Bench, with 80 objective tasks and 34 executable tasks (13 bug fixes, 21 full generations) scored at the structure, runtime, and behavior levels. On the executable tasks, 41.2–47.1% for Claude Code, Codex CLI, and OpenClaw, and 58.8% for AFSIM-Agent; on the objective tasks, all harnesses score similarly. Ablations on the executable tasks show that removing all executable feedback causes the largest drop, while removing any other mechanism changes results by one to three tasks.
Critical strengths
-
AFSIM-Bench separates "the project runs" from "the project does what was asked." Executable tasks are scored at the structure, runtime, and behavior levels using hidden predicates over engine-written event traces, and the same backbone is run with no harness, with general harnesses, and with the domain agent. This yields a useful observation: general harnesses get nearly every project to run, yet more than half still fail to produce the required behavior. This matters because it documents a concrete failure mode of current coding agents in domains where correctness is defined by runtime behavior.
-
The reporting is transparent. The paper gives per-task results and failure-type breakdowns, validates its oracles with reference and sabotaged solutions, and openly reports null results and significance caveats, including a module whose removal does not hurt. This matter.
Critical weaknesses
-
Novelty is limited, and the evaluation is not convincing. The agent combines standard components and the authors themselves claim no novelty for any individual mechanism. Both the benchmark and the agent are built by the authors for a single DSL, which makes it hard to rule out that the agent is tailored to the benchmark. The comparison is also unfair: the general harnesses receive none of the curated documentation, skill cards, or behavior checks given to AFSIM-Agent, and the remaining lead is only a few tasks, which the authors acknowledge is not significant. It is unknown how the agent performs on other DSLs or tasks. This matters because the paper shows neither a new method nor evidence that the approach works beyond its own benchmark.
-
The writing is hard to follow. The paper relies on coined terms that are never defined (e.g., "LID context discipline," "rungs," "dose checker") and leaves most technical details and the related work to the appendix.
Questions for authors
-
Can you report a general harness (e.g., Claude Code) given the same documentation, skill cards, and simulation-checking tool as AFSIM-Agent? Without this comparison, the gain may come from the materials rather than from the agent design.
-
Apart from the single specification in Figure 1, the paper shows no example tasks. Could you provide representative examples of each task type (objective questions, bug-fix tasks, and full-generation tasks), together with examples of their scoring predicates, so that readers can judge the difficulty and quality of the benchmark?