acceptodds
Under review as a conference paper at ICLR 2027

CodeWorldBench: How Well Can Coding Agents Build Executable Agent Worlds?

Abstract

Modern coding agents increasingly demonstrate the capability to scale and generate complex interactive agent environments. However, existing benchmarks evaluate the ability to generate repositories from scratch or repair localized bugs, leaving its capacity to construct fully functional, stateful environments largely unmeasured. Therefore, we introduce CodeWorldBench, a benchmark designed to assess the quality of code-driven world construction. We construct 118 environments spanning common workflows and professional software domains, requiring agents to generate or repair repositories that realize a stateful world, including its data, tool interfaces, and transition logic. Rather than relying on code similarity, we test behavioral equivalence: submitted and reference worlds start from the same seed, receive hidden tool calls, and are compared on their outputs and resulting states. Our evaluation reveals vulnerabilities in current systems. While models preserve already-correct behavior, they struggle to construct environments from scratch and fail at repairing broken behavior. Agents fail on state transitions and validation order rather than syntax. Because agent-built environments increasingly serve as the grounding for training subsequent models, an environment that runs but transitions incorrectly supplies a damaging false signal. Uncovering these flaws provides insights and directions for robust Recursive Self-Improvement and the self-evolution of future agents.

Primary area:
datasets and benchmarks
Keywords:
code-driven world generation, executable environments, coding agents, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YRuGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CodeWorldBench to evaluate whether coding agents can implement or repair tool backends from specifications. The benchmark contains 118 backends, each with a task to build it from scratch and a task to repair it. The results show that coding agents generally preserve existing correct behavior, but struggle to build backends from scratch or repair faulty behavior. The downstream evaluation further suggests that agent-generated backends are not yet reliable enough for direct use by tool-using agents.

Critical strengths

The evaluation is broad. It covers multiple model and agent configurations across 118 backends. The downstream experiment also examines how backend correctness affects task success.

Critical weaknesses
  1. The term “executable agent world” is not clearly defined. The paper does not explain what distinguishes it from a tool backend.
  2. The distinction from related work is insufficiently clear. BackendForge (Guo et al., 2026) studies backend generation, and EnvFactory (Xu et al., 2026) studies stateful tool environments. The paper should compare their task inputs, required outputs, and evaluation criteria with those of CodeWorldBench to identify its specific contribution.
  3. The released data raise quality concerns. Our inspection of the released repository found that FactSet’s run_screen is specified as a read operation with no state effects, yet its tests require changes to session state. Moreover, 229 of 795 professional write tools lack a direct successful test case.
  4. The experimental setup is hard to follow. The paper should distinguish the coding agent that builds or repairs a backend from the downstream agent that uses it, and provide one concrete example showing how each is evaluated.
Questions for authors
  1. What properties distinguish an “executable agent world” from a stateful tool backend in this benchmark?
  2. What does CodeWorldBench evaluate that BackendForge and EnvFactory do not?
  3. How do the authors resolve the specification–test conflict for FactSet’s run_screen? How does the lack of direct successful tests for 229 of 795 professional write tools affect the reported results?
Reviewer_MrZoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CodeWorldBench evaluates whether coding agents can construct or repair executable, stateful worlds rather than merely produce runnable repositories. The benchmark contains 118 environments across 20 categories, exposes 4,417 tools, and evaluates 25,015 hidden cases. Candidate and reference worlds begin from the same seed and receive identical tool calls; the evaluator compares returned values, persistent state, and event effects. The GEN setting requires construction from specifications and initial data, while SWE tasks require repairing semantic defects that arise during repository generation. The paper evaluates several coding agents and scaffolds, analyzes common semantic failures, and studies how imperfect constructed worlds affect a downstream agent on a runnable subset.

Critical strengths

S1 An important benchmark target. State-transition fidelity is a distinct capability from syntax generation, repository completion, or acting inside a fixed environment. Measuring outputs together with persistent state directly addresses failure modes that ordinary unit tests or output-only checks can miss. S2 Substantial breadth and useful task separation. The benchmark covers 118 environments, 4,417 tool interfaces, and 25,015 hidden cases across common workflows and professional software domains. Separating build-from-scratch GEN tasks from semantic-repair SWE tasks makes the results easier to interpret and exposes a large gap between preserving correct behavior and repairing broken dynamics. S3 Multiple evaluation views. The paper compares model and scaffold combinations, categorizes semantic errors, examines multistep and cross-tool behavior, and includes a downstream execution study. These analyses make the benchmark more informative than a single aggregate pass rate and support the core conclusion that executable code need not implement the intended world.

Critical weaknesses

W1 Reference-world validity is not independently established. The reference implementations and tests are themselves produced and repaired with substantial GPT-5.5 assistance. Automated consistency checks are valuable, but agreement among an LLM-generated specification, implementation, and test suite is not equivalent to independent domain validation. The paper does not report a blinded human audit rate, disagreement rate, or defect estimate for the reference worlds and hidden cases. Benchmark conclusions depend on these artifacts being correct. W2 Behavioral equivalence is sampled unevenly. Prepared multistep histories and persistent-state comparisons improve on isolated unit tests, but the hidden suite remains a finite sample over selected histories, seeds, and state fields. Coverage varies across environments and tools. A solution can pass the observed traces while violating longer-horizon behavior, rare validation orderings, or cross-tool invariants. The current paper needs stronger mutation testing or resampling evidence to show that scores are stable under plausible expansions of the hidden suite. W3 Model comparisons mix capability with scaffold choices and single-run estimates. The appendix documents context, output, sampling, and reasoning settings, which is helpful. However, OpenAI models run under Codex while most other models run under Claude Code, and the Qwen scaffold matrix shows that repair scores can move substantially across harnesses. These are legitimate system comparisons, but they do not cleanly identify model capability. The paper also reports point estimates without rerun or environment-level bootstrap uncertainty, so close rankings may reflect scaffold engineering or stochastic variation. W4 The downstream claim should be narrowed. The downstream study shows lower task success when an execution agent operates in imperfect generated worlds, which is useful evidence of test-time impact. However, it uses 36 runnable worlds common to the compared conditions and does not study training on those worlds or recursive self-improvement. Construction failures are excluded from the runnable subset. The result supports degradation during downstream execution, not claims about model-training damage or self-evolution. W5 Aggregate GEN and SWE numbers are not directly comparable. GEN results macro-average across environments, while SWE results micro-average cases. Both choices can be defensible, but the different weighting means that headline percentages across the two settings are on different statistical scales. Reporting both macro and micro views for each setting would make comparisons and error attribution clearer.

Questions for authors

Q1 What fraction of reference implementations and hidden tests fail an independent, blinded audit by domain-knowledgeable reviewers, and how are discovered benchmark defects adjudicated and versioned? Q2 How sensitive are environment scores and model rankings to new seeds, newly generated multistep histories, mutation tests, and additional cross-tool invariants that were not used during reference-world repair? Q3 Can you report environment-clustered confidence intervals, rerun variance, and the subset of rankings that can be estimated under a fixed scaffold and closely matched inference settings? Q4 For the downstream study, what is the result when all construction attempts are included, including non-runnable worlds, with uncertainty clustered by environment and at least one additional execution agent?

Reviewer_ef48OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CodeWorldBench construct 118 synthetic environments using GPT-5.5 based on human seeds (this is done in a two step process: first a human "catalog entry" is taken and turned into specifications, then the specifications are turned into code).

This is turned into a benchmark in two variants.

  1. GEN: Models recreate these synthetic environments based on seeds based on similar context as the original

  2. SWE: Taking syntehtic environments produced in the GEN setting by Qwen 3.6, other models have to repair them based on detailed specs

Critical strengths

Lots of training data is already synthetically generated to serve as training environments, so this certainly is a timely area of research.

Critical weaknesses

Novelty

While framed around stateful environment creation, overall most of the GEN setting seems to be a pretty normal specification -> repository generation task, somewhat similar to commit0, etc., with the main difference being that the created environments take the form of MCP services.

The question of statefulness that seemed to be one of the motivation is not explored in detail. Similarly, one of the motivations seemed to have been the use of synthetic data (synthetically generated training environments), but after the reading the paper it is not clear to me how this was formalized into a clear research question with clear experiments.

GEN benchmark

  1. low separation between models: On GEN task (all 118 envs): Qwen 3.8 27B is only separated by 6 %pts from Astra 6. The spread between Astra 6 and Luna is also only around 7 %pts.

  2. The low separation between models of very different sizes (and performances on other benchmarks) could point to other confounding factors. For example, a) some of the tests could be overly specific to an implementations b) some requirements might not be included in the GEN prompt c) the synthetic construction of the environments with GPT 5.5 could introduce non-trivial biases

  3. If the claim of the paper is that models aren't great with statefulness & creating "worlds", then how can we take synthetic environments (generated by GPT 5.5) as the basis of the benchmark? Wouldn't the flaws of GPT 5.5 on this task be very conflating?

SWE benchmark

  1. Possible biases because all models operate on a codebase that's generated by Qwen 3.6 27B (and the sibling model Qwen 3.8 27B also happens to come out on top of the professional SWE style setting)

  2. Issues around solvability etc. carry over from the GEN task

Analysis

  • Paper claims "Tool mix is a property of the harness not the model" (but Fig 4a that serves as evidence only compares 3 harnesses with Qwen 3.8/3.6, so it's not establish if this effect generalizes across models, or what the inherent effect of models is)
  • Paper later then only compares models "Closed-source models act on the world; open-source models read about it", which seems to be contradictory because models/harnesses are mixed

Readability

  • The paper would have profited from formulating clear research questions and answer them with clear experiments.
  • Figures 2-3 have unreadably small labels
  • The results of the SWE setting are not visualized, but results have to be compared across 2 extremely large tables
Questions for authors
  1. Could you provide more details about what was being used to ground the environments? THe paper mentions "categlogue entries" as the main soruce of human input. The contracts also are generated by GPT 5.5. How close does a task like "AutoCAD" match a realistic workload/task?

  2. How did we confirm that all tasks are actually solvable and none of the specifications for the tasks lacking?

  3. How do you rule out that the synthetic construction biases the benchmark results?

  4. What score does GPT-5.5 itself get on GEN?

  5. Can you provide quantitative details about the role of statefulness in generation role of the benchmark (this was one of the motivations that separated it from e.g., commit0 etc.) E.g., which failures are attributed to state etc.

  6. Some tasks like MA are solved to 33-34% by almost all models. What explains this?

  7. Could you provide details for the filtering of the generated environments? How was vagueness etc. defined? Who reviewed? How many environment were filtered out?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.