acceptodds
Under review as a conference paper at ICLR 2027

AFSIM-Agent: Hierarchical Self-Reflective Coding Agent for Cross-Language Adaptation to Unseen Scripting Languages

Abstract

Code agents solve many tasks, but their competence rests on pretraining that supplies the language's syntax and abundant examples. Domain-specific languages (DSLs) break both: models lack the domain's grammar, cannot interpret compiler diagnostics, and have few examples to learn from. We study AFSIM, an open-source multi-domain modeling and simulation framework from the US Air Force Research Laboratory (AFRL), whose scripting language is rare in training data. General coding agents fail often when writing it; we aim for an agent that masters an unfamiliar DSL without retraining. We propose AFSIM-Agent, with five modules. (1) A curated domain knowledge surface turns the DSL into retrievable priors: a read-only document tree of grammar, documentation, and worked examples, plus skill cards linking engineering tasks to source files. (2) An executable feedback ladder supplies four levels of signal—static constraints, minimal-harness compilation, full simulation, and independent fidelity audit—constructing the trace-level signals general agents never observe. (3) A verification-gated repair loop admits only edits verified at the current file revision, so each round builds on verified state. (4) A locally in-distribution (LID) context discipline keeps every LLM call in-distribution and the main context limited to decomposition strategy, control flow, and normalized state, so long-horizon tasks do not dilute critical signals. (5) A deterministic plan-audit gate and anti-reward-hacking guardrails constrain the plan before code is written and the result against the specification at submission. We also build AFSIM-Bench: 114 tasks with compilable reference code and a reproducible measurement instrument for this domain. On the 34 executable tasks, the bare model passes 17.6%, the three general harnesses reach 41.2%–47.1%, and AFSIM-Agent reaches 58.8%; the harness buys the ability to make a project run, and domain grounding buys the ability to make it satisfy intent.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code agents, domain-specific languages (DSLs), agent benchmarks, cross-language adaptation, modeling and simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SY15OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies coding agents for AFSIM. Its scripting language is rare in training data, and a task counts as solved only if the required events appear in the simulation trace, not merely if the project compiles and runs. The paper proposes AFSIM-Agent with a specific workflow. It also introduces AFSIM-Bench, with 80 objective tasks and 34 executable tasks (13 bug fixes, 21 full generations) scored at the structure, runtime, and behavior levels. On the executable tasks, 41.2–47.1% for Claude Code, Codex CLI, and OpenClaw, and 58.8% for AFSIM-Agent; on the objective tasks, all harnesses score similarly. Ablations on the executable tasks show that removing all executable feedback causes the largest drop, while removing any other mechanism changes results by one to three tasks.

Critical strengths
  1. AFSIM-Bench separates "the project runs" from "the project does what was asked." Executable tasks are scored at the structure, runtime, and behavior levels using hidden predicates over engine-written event traces, and the same backbone is run with no harness, with general harnesses, and with the domain agent. This yields a useful observation: general harnesses get nearly every project to run, yet more than half still fail to produce the required behavior. This matters because it documents a concrete failure mode of current coding agents in domains where correctness is defined by runtime behavior.

  2. The reporting is transparent. The paper gives per-task results and failure-type breakdowns, validates its oracles with reference and sabotaged solutions, and openly reports null results and significance caveats, including a module whose removal does not hurt. This matter.

Critical weaknesses
  1. Novelty is limited, and the evaluation is not convincing. The agent combines standard components and the authors themselves claim no novelty for any individual mechanism. Both the benchmark and the agent are built by the authors for a single DSL, which makes it hard to rule out that the agent is tailored to the benchmark. The comparison is also unfair: the general harnesses receive none of the curated documentation, skill cards, or behavior checks given to AFSIM-Agent, and the remaining lead is only a few tasks, which the authors acknowledge is not significant. It is unknown how the agent performs on other DSLs or tasks. This matters because the paper shows neither a new method nor evidence that the approach works beyond its own benchmark.

  2. The writing is hard to follow. The paper relies on coined terms that are never defined (e.g., "LID context discipline," "rungs," "dose checker") and leaves most technical details and the related work to the appendix.

Questions for authors
  1. Can you report a general harness (e.g., Claude Code) given the same documentation, skill cards, and simulation-checking tool as AFSIM-Agent? Without this comparison, the gain may come from the materials rather than from the agent design.

  2. Apart from the single specification in Figure 1, the paper shows no example tasks. Could you provide representative examples of each task type (objective questions, bug-fix tasks, and full-generation tasks), together with examples of their scoring predicates, so that readers can judge the difficulty and quality of the benchmark?

Reviewer_NsPeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets low-resource domain-specific languages: the model has not seen the syntax, cannot interpret compiler diagnostics, and has few examples. Using AFSIM (a simulation framework used by AFRL) as the case study, the authors propose an agent that combines a document-grounded domain knowledge base with skill cards, a four-level executable feedback ladder (static check, minimal compile harness, full simulation, independent fidelity audit), a verification gate that invalidates stale evidence whenever a file changes, context discipline that keeps each LLM call in-distribution, and a deterministic plan-audit gate. They also build AFSIM-Bench: 114 tasks, of which 34 are executable (13 bug fix, 21 full generation) and judged by a frozen behavioral oracle. With a single frozen backbone and no fine-tuning, the agent passes 20/34 executable tasks versus 6/34 for the bare model and 15/16/14 for three general harnesses.

Critical strengths
  1. A genuine and under-served problem: long-tail DSLs are a real obstacle for agent deployment, and this setting is scarcer than another Python benchmark.
  2. Behavioral judging instead of compilation or exit codes, with the agent's internal verifier and the final frozen evaluator kept strictly separate.
  3. The stale-evidence rule is a transferable design idea: any edit invalidates prior verification, so only evidence produced on the current file version is accepted.
  4. Unusual honesty: the null result for the plan gate, the non-independence of internal audits, and the single-model/single-DSL scope are all disclosed.
  5. Good engineering hygiene: per-task tables, failure categories, and a version/commit/oracle-hash binding.
Critical weaknesses
  1. The main cross-system claim is not statistically supported. §5.1 (F3) states that the paired differences against the three general harnesses are not significant and are of the same magnitude as batch re-test fluctuation. The abstract nonetheless uses a "58.8% vs 41.2–47.1%" comparative framing, which overstates the evidence.
  2. The gain is not attributed. Domain knowledge base, behavioral feedback ladder, repair scheduling, and context discipline are introduced as a bundle, so the improvement cannot be assigned to domain grounding rather than orchestration. The informative control is a general harness given the same documentation and the same executable feedback.
  3. Limited external validity: 34 executable tasks across 25 case families, one DSL, one backbone. The paper's own failure analysis shows 10 of 21 full-generation tasks fail for all five systems, so the ceiling — not the sample size — limits the gain.
  4. Construct validity of the hidden oracle is not established. Whether each threshold is derivable from the public requirements is not mapped task by task, and semantic equivalence is not checked with multiple independent valid solutions (Appendix I).
  5. Reproducibility is restricted: the runtime and the full scenario corpus are not released, and the mock provider validates only the scoring flow, not real success rates. The title's "hierarchical self-reflective" framing is also in tension with the plan gate's null result.
Questions for authors
  1. Can you inject the same domain documentation and the same behavioral feedback into one general harness? Does the specialized orchestration retain a significant residual gain?
  2. Can you report per-task paired tests or confidence intervals for the 34 executable tasks rather than aggregate counts, given the acknowledged batch fluctuation?
  3. Can you provide a mapping from each hidden oracle threshold to the public requirement it enforces, and check at least a few tasks against multiple independently written valid solutions?
  4. Which component drives the full-generation gain (9/21 vs 4–5/21) — the knowledge base, the executable feedback, or the repair loop? The current ablation covers feedback strongly but leaves the others descriptive.
  5. What is the release plan for the runtime and scenario corpus, and which claims should readers treat as unreproducible until then?
Reviewer_XfqKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets code agents writing AFSIM, a low-resource simulation DSL, without retraining. AFSIM-Agent combines a curated knowledge surface, a four-rung executable feedback ladder, revision-gated repair, an LID context discipline, and plan-audit/anti-bypass guards. AFSIM-Bench has 114 tasks, of which 34 executable tasks are scored at the behavior level over simulation event traces. With one backbone DeepSeek v4.1 Flash, executable pass rates are 17.6% for the bare model, 41–47% for three general harnesses, and 58.8% for AFSIM-Agent. The ablation identifies executable feedback as the load-bearing mechanism.

Critical strengths
  • The paper makes the gap between runs and satisfies intent measurable through a layered oracle covering structure, runtime and behavior, together with an L0 to L6 level score. This is methodologically useful for domains where correctness is defined over runtime traces, such as hardware description languages and PLC logic.
  • The reporting is candid. The authors disclose the null result for the plan gate, the 10 tasks that every system fails, and the lack of statistical significance. The A1 ablation, which removes all executable feedback, converges to the same failure layer as the bare model, which is convincing mechanistic evidence.
Critical weaknesses
  • The headline claim lacks statistical support. The lead over general harnesses is only 4 to 6 tasks, 20/34 versus 14–16/34, and it is concentrated in the 21 full-generation tasks. The authors themselves state in F3 that the lead is of the order of batch retest noise and that none of the paired comparisons is significant. The evaluation also uses a single run and a single backbone, with no variance reported. Apart from A1, ablation deltas are 1 to 3 cells, and removing the plan gate even improves results. The abstract’s claim of mastering an unseen DSL is therefore overstated.
  • The comparison is confounded, and leakage is possible. The general harnesses appear to lack the AFSIM doc tree, the knowledge base, and full-simulation behavior feedback. If so, the gain reflects access to domain resources rather than agent design. The paper is missing a control that gives a general harness the same documentation and simulator access. In addition, tasks are derived from the official demo suite, while the knowledge surface contains the official demo projects, so reference solutions may be nearly retrievable. A leakage analysis is needed.
Questions for authors
  • Do Tier-2 harnesses have access to the AFSIM runtime, event logs, and doc tree? Do they share AFSIM-Agent’s prompt?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.