- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EvoPolicyGym, a benchmark for studying how coding agents improve executable policies through environment feedback. It combines standardized interfaces, controlled episode budgets, documented policy revisions, and comprehensive evaluation. The framework integrates 357 environment variants across 18 ecosystems. The authors evaluate five coding agents on eight tasks (mainly relating to control tasks), with three runs per agent–task pair and 128 training episodes per run. Astra achieves the highest mean score on seven tasks. A feedback ablation shows the value of richer diagnostics, while development trajectories and case studies illustrate both successful policy revisions and regressions. The main contribution is reusable infrastructure for evaluating autonomous policy development across different diverse environments.
Critical strengths
-
The framework brings diverse environments under a centralizd shared policy and evaluation interfaces; support for adding new environments. This could possibly be useful resource for games, robotics, control, and research on self-improving agents.
-
Eval protocol is quite well-structued. Fixed episode budgets, saved policy revisions, and separate validation and held-out assessment provide clear basis on comparing agents / policies.
-
There is some intruging insights. I find it interesting that progress is often non-monotonic. later revisions can undo earlier gains, the benefits of richer feedback also vary substantially across tasks. The Crafter case is particularly intreesting, as the policy incorrectly treated full energy as evidence that sleeping had ended. I see that a seemingly minor error of interpreting environment state can disrupt a policy, and diagnostic feedback can help identify and correct it.
Critical weaknesses
First, I actually think this paper has potential to be good paper (refer to the strength part I wrote) Yet as of now, I think it slightly miss the bar of ICLR. But I'm happy to raise my score if the authors address my concerns.
-
Limited conceptual novelty and insufficient positioning. The basic idea of improving executable policies through LLM-generated revisions and environment feedback is not new, and the paper does not sufficiently position itself relative to the closest related work. I would need to see this positioning before supporting acceptance. I don't mean this is a fatal weakness. But instead of a brand-new contribution, the paper’s main value should be assessed through the scale and diversity of its environment suite, the soundness of its experiments and conclusions, and the new insights demonstrated by those experiments. The authors should make clearer what the benchmark enables and what the experimental findings establish beyond existing work. That said, paper would benefit from a more thorough discussion of related work, including direct comparisons with ALE-Bench, CodeClash, and Frontier-Eng. Two particularly relevant works are Learning Game-Playing Agents with Generative Code Optimization (Kuang et al., 2025), which already studies executable game-playing policies refined by LLMs using execution feedback, and Understanding the Challenges in Iterative Generative Optimization with LLMs (Nie et al., 2026), which investigates how init, execution traces length, and experience batching affect lm driven optimization. These connections are relevant both to the novelty and intepretation of findings.
-
The rich-versus-minimal feedback ablation provides useful evidence for the value of diagnostics, but both conditions retain reward feedback and iterative revision. Tracking initial and later policy versions are somehow informative, but does not establish how much feedback-guided refinement contributes relative to independent code generation.
-
The experiments cover eight tasks from the much larger environment catalog, with three independent development runs per condition. Also the feedback ablation and detailed case studies focus on Astra. This leaves uncertainty about smaller performance differences and about whether the observed feedback effects generalize across agents and environments.
Questions for authors
-
I'm rly curious about how are the interaction budget and the privacy of evaluation inputs enforced in the reported experiments. What prevents agents from running additional simulator rollouts outside the rule or accessing private evaluation data?
-
What components must a developer implement to integrate a new environment, and which capabilities are provided automatically by the framework? Could the authors clarify the boundary between adding an environment adapter and changing the core protocol? This is VERY important for the actual usefulness of the work.
Author Response
Thank you for your positive assessment of the benchmark’s clarity, breadth, and usefulness for future research on self-evolving agents. We particularly appreciate your recognition that EvoPolicyGym provides a foundation for studying how coding agents develop and improve executable policies. We are also using this infrastructure to improve models’ capabilities for autonomous policy development. And we hope that our design, which gives agents the freedom to organize their own trials within a fixed environment interaction budget, will inspire future evaluation protocols for self-improving agents.