- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CodeWorldBench to evaluate whether coding agents can implement or repair tool backends from specifications. The benchmark contains 118 backends, each with a task to build it from scratch and a task to repair it. The results show that coding agents generally preserve existing correct behavior, but struggle to build backends from scratch or repair faulty behavior. The downstream evaluation further suggests that agent-generated backends are not yet reliable enough for direct use by tool-using agents.
Critical strengths
The evaluation is broad. It covers multiple model and agent configurations across 118 backends. The downstream experiment also examines how backend correctness affects task success.
Critical weaknesses
- The term “executable agent world” is not clearly defined. The paper does not explain what distinguishes it from a tool backend.
- The distinction from related work is insufficiently clear. BackendForge (Guo et al., 2026) studies backend generation, and EnvFactory (Xu et al., 2026) studies stateful tool environments. The paper should compare their task inputs, required outputs, and evaluation criteria with those of CodeWorldBench to identify its specific contribution.
- The released data raise quality concerns. Our inspection of the released repository found that FactSet’s run_screen is specified as a read operation with no state effects, yet its tests require changes to session state. Moreover, 229 of 795 professional write tools lack a direct successful test case.
- The experimental setup is hard to follow. The paper should distinguish the coding agent that builds or repairs a backend from the downstream agent that uses it, and provide one concrete example showing how each is evaluated.
Questions for authors
- What properties distinguish an “executable agent world” from a stateful tool backend in this benchmark?
- What does CodeWorldBench evaluate that BackendForge and EnvFactory do not?
- How do the authors resolve the specification–test conflict for FactSet’s run_screen? How does the lack of direct successful tests for 229 of 795 professional write tools affect the reported results?