acceptodds
Under review as a conference paper at ICLR 2027

EvoPolicyGym: Benchmarking Coding Agents for Autonomous Policy Evolution

Abstract

Self-evolving agents that learn from interaction can improve their behavior by iteratively developing policies from environmental feedback. However, evaluating this capability remains challenging because agents may differ not only in the policies they produce, but also in how they explore the environment, allocate interaction budgets, and revise their policies over time. We introduce EvoPolicyGym, a benchmark framework for evaluating autonomous policy evolution under a standardized interaction budget. EvoPolicyGym assesses policy development trajectories and applies a validation protocol to select a policy from each run for independent evaluation on held-out episodes. The framework integrates 357 environment variants across 18 ecosystems. We evaluate four coding agents across all eight tasks spanning games and continuous-control domains. The results show that Astra achieves the highest mean final performance among the evaluated agents. Analysis of policy development trajectories further reveals that Astra iteratively identifies policy failures and incorporates the resulting evidence into subsequent policy revisions. These results establish EvoPolicyGym as a framework for studying how coding agents transform interaction experience into reusable executable policies, and provide evidence of emerging capabilities in autonomous policy development.

Primary area:
datasets and benchmarks
Keywords:
Policy Evolution,Self Evolving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TG7AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoPolicyGym, a benchmark for studying how coding agents improve executable policies through environment feedback. It combines standardized interfaces, controlled episode budgets, documented policy revisions, and comprehensive evaluation. The framework integrates 357 environment variants across 18 ecosystems. The authors evaluate five coding agents on eight tasks (mainly relating to control tasks), with three runs per agent–task pair and 128 training episodes per run. Astra achieves the highest mean score on seven tasks. A feedback ablation shows the value of richer diagnostics, while development trajectories and case studies illustrate both successful policy revisions and regressions. The main contribution is reusable infrastructure for evaluating autonomous policy development across different diverse environments.

Critical strengths
  1. The framework brings diverse environments under a centralizd shared policy and evaluation interfaces; support for adding new environments. This could possibly be useful resource for games, robotics, control, and research on self-improving agents.

  2. Eval protocol is quite well-structued. Fixed episode budgets, saved policy revisions, and separate validation and held-out assessment provide clear basis on comparing agents / policies.

  3. There is some intruging insights. I find it interesting that progress is often non-monotonic. later revisions can undo earlier gains, the benefits of richer feedback also vary substantially across tasks. The Crafter case is particularly intreesting, as the policy incorrectly treated full energy as evidence that sleeping had ended. I see that a seemingly minor error of interpreting environment state can disrupt a policy, and diagnostic feedback can help identify and correct it.

Critical weaknesses

First, I actually think this paper has potential to be good paper (refer to the strength part I wrote) Yet as of now, I think it slightly miss the bar of ICLR. But I'm happy to raise my score if the authors address my concerns.

  1. Limited conceptual novelty and insufficient positioning. The basic idea of improving executable policies through LLM-generated revisions and environment feedback is not new, and the paper does not sufficiently position itself relative to the closest related work. I would need to see this positioning before supporting acceptance. I don't mean this is a fatal weakness. But instead of a brand-new contribution, the paper’s main value should be assessed through the scale and diversity of its environment suite, the soundness of its experiments and conclusions, and the new insights demonstrated by those experiments. The authors should make clearer what the benchmark enables and what the experimental findings establish beyond existing work. That said, paper would benefit from a more thorough discussion of related work, including direct comparisons with ALE-Bench, CodeClash, and Frontier-Eng. Two particularly relevant works are Learning Game-Playing Agents with Generative Code Optimization (Kuang et al., 2025), which already studies executable game-playing policies refined by LLMs using execution feedback, and Understanding the Challenges in Iterative Generative Optimization with LLMs (Nie et al., 2026), which investigates how init, execution traces length, and experience batching affect lm driven optimization. These connections are relevant both to the novelty and intepretation of findings.

  2. The rich-versus-minimal feedback ablation provides useful evidence for the value of diagnostics, but both conditions retain reward feedback and iterative revision. Tracking initial and later policy versions are somehow informative, but does not establish how much feedback-guided refinement contributes relative to independent code generation.

  3. The experiments cover eight tasks from the much larger environment catalog, with three independent development runs per condition. Also the feedback ablation and detailed case studies focus on Astra. This leaves uncertainty about smaller performance differences and about whether the observed feedback effects generalize across agents and environments.

Questions for authors
  1. I'm rly curious about how are the interaction budget and the privacy of evaluation inputs enforced in the reported experiments. What prevents agents from running additional simulator rollouts outside the rule or accessing private evaluation data?

  2. What components must a developer implement to integrate a new environment, and which capabilities are provided automatically by the framework? Could the authors clarify the boundary between adding an environment adapter and changing the core protocol? This is VERY important for the actual usefulness of the work.

Reviewer_PmnQOpenReview ↗
Rating
2(−2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark for coding agents in policy evolution. The framework encompasses 18 ecosystems, including 357 environment variants. Across models of increasing strength, the results show how coding agents can evolve executable policies, with Astra performing best.

Critical strengths

Ultimately, I think the main strength is that other researchers can build on top of this benchmark to study how coding agents can evolve their own policies. The paper is clearly written and easy to follow. The benchmark includes a significant number of ecosystems and environments, and the authors show the value of the benchmark with several frontier models. Although no major findings come out of this benchmark, in my opinion, it is still useful for the future of self-evolving agents.

Critical weaknesses

I haven’t found critical weaknesses, considering its primary area is datasets and benchmarks. This benchmark is an engineering project with no major findings, but that is expected in this area.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Author Response

Thank you for your positive assessment of the benchmark’s clarity, breadth, and usefulness for future research on self-evolving agents. We particularly appreciate your recognition that EvoPolicyGym provides a foundation for studying how coding agents develop and improve executable policies. We are also using this infrastructure to improve models’ capabilities for autonomous policy development. And we hope that our design, which gives agents the freedom to organize their own trials within a fixed environment interaction budget, will inspire future evaluation protocols for self-improving agents.

The rebuttal has been extremely underwhelming. Since you haven't addressed the reviews from @TG7A (at all) and @QjHq (successfully), I've decided to decrease my score.

Reviewer_QjHqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for coding agents that develop executable policies through repeated environment trials. The protocol charges a fixed number of training episodes, saves immutable policy submissions, selects among committed candidates on separate validation episodes, and evaluates the selected policy on held-out episodes. The framework lists 357 named environment variants across 18 ecosystems; the empirical study compares five agents on eight selected tasks, analyzes their saved development trajectories, and ablates rich versus minimal feedback for Astra. Astra has the highest observed mean final score on seven of the eight tasks, and the case studies connect particular diagnostics to subsequent code revisions.

The paper's real contribution is an evaluation protocol and empirical study: versioned, held-out trajectories reveal when policies improve or regress, and the Astra ablation shows that richer diagnostics help on several tested tasks.

Yet these are policy-level observations, chiefly from Astra, rather than a systematic account of which capabilities frontier models lack or which model changes would help. Overall gains also mix initial code construction, search, feedback use, and candidate selection, while the empirical evidence covers only eight selected tasks from the 357-variant catalog. Prior work also already studies feedback-driven code revision and bounded trajectories. Crucially, I do not find explicit in-depth insights into capability gaps of frontier models, and guidance/vision toward how to improve the frontier models.

Based on my assessment that this work mostly introduce new benchmark and do some empirical study, I remain hesitant in leaning toward acceptance. Considering quality rank of other papers in the same direction in my batch, I made the decision of score-1.

Critical strengths

The framework lists 357 named environment variants across 18 ecosystems

Critical weaknesses

1. [Contribution / prior work] Say what the combined protocol adds

The introduction asks how to evaluate coding agents that repeatedly revise executable policies after environmental feedback. That is a worthwhile object of study. Yet Voyager already revised executable Minecraft skills from environmental feedback, and Frontier-Eng already combined bounded feedback with trajectory analysis, including a closed-loop data-center control policy. EvoPolicyGym's clearer increment is the combination of episode accounting, frozen policy versions, separate candidate selection, and held-out testing across environments. Please compare that combination directly with the closest settings and explain what distinct, reliable information it produces.

2. [Construct / cost scope] Separate episode-matched development from a pure feedback-use claim

Section 3 and the setup fix 128 training episodes per run . Those are clear bounds, and immutable submissions make the process auditable. Equal limits do not show equal actual model calls, tokens, elapsed time, or computation inside agent-written searches. Appendix D.4 illustrates the distinction: in FrankaKitchen, 291 of 308 submissions evaluate just one episode while the agent runs its own parameter search. Final performance thus measures the complete coding-agent development system under these limits; it cannot by itself identify how much came from feedback diagnosis rather than initial code construction or search.

3. [Benchmark validity / scope] Show what has been checked beyond the eight study tasks

Figure 2 and Section 4 clearly distinguish the 357 named catalog variants from the eight tasks used in the experiments, and the text says variants are not independent task families. The remaining question is what the wider catalog can measure: Appendix A.4 says its inventory script extracts identifiers without importing simulators, while the model and feedback studies use only eight tasks. The paper therefore demonstrates adapter coverage more directly than execution quality, model discrimination, or score interpretability across the catalog. Either give the task-selection criteria, representative quality audit; or narrow the empirical benchmark claim explicitly tied to the eight tasks.

4. [Readability / metric meaning] Explain the main scores where readers first use them

Table 1 names each task's score before Table 2 compares agents on those scores. Some operational meanings arrive only in Appendix A.1. A short explanation next to Table 1, plus a behavioral milestone for tasks where return can be misleading, would let readers interpret the headline numbers on first reading.

5. [Feedback / causal scope] Describe rich-versus-minimal as the value of added diagnostics

Section 5.4 compares only Astra. Both conditions still provide aggregate and per-episode rewards, execution status, and budget information; rich feedback adds diagnostics, trajectories, and replay artifacts. The ablation therefore tests the marginal value of those additions, not feedback versus no feedback.

Overall, the paper propose a benchmark and some empirical analysis centered on Astra. But only results are far from enough for ICLR papers. Current version show no deeper insights in gaps of frontier models or guidance/vision toward how to improve them.

Questions for authors

see the weaknesses

2 comments
AuthorsOpenReview ↗

Author Response (1/2)

Thank you for your feedback. However, we disagree with several aspects of your assessment and explain our reasoning below.

1. Our evaluation includes the ability to organize the optimization process autonomously

Voyager demonstrates feedback-driven skill learning in Minecraft, but does not establish a unified protocol across environments that separates development, candidate selection, and final testing as ours does.

Frontier-Eng targets a broad range of engineering optimization problems, including interactive control tasks, but its budget design is not specifically tailored to autonomous policy development. For example, its main experiments use 100 OpenEvolve iterations. In SustainDC, each normally completed candidate evaluation executes 4 scenarios, each with 192 control steps. The external framework organizes candidate generation, while the evaluator prescribes the testing workload for each candidate.

Consider developing a robotic grasping policy with a budget of 128 environment trials. Allocating exactly 4 trials to each candidate allows 32 candidates to be evaluated. An autonomous agent could instead allocate one trial each to 64 candidates, eliminate clearly unsuccessful approaches, and then allocate 16 additional trials each to 4 promising candidates to assess their reliability under different initial conditions. Both allocations consume 128 trials, but the latter leaves the agent to decide how many approaches to explore and when to investigate them more thoroughly.

We fix the total development episode budget and leave these allocation decisions to the agent. The agent can also adapt its development approach to the failures it observes: inspecting existing trajectories, redesigning the control logic, or writing its own search program. In Kitchen, agents actually generated parameter-search programs and made extensive use of single-episode trials for local comparisons.

The distinction therefore goes beyond the budget unit: the former primarily compares optimization performance within a prescribed search and evaluation process; we also evaluate how agents design and adapt that process. This more closely reflects how users delegate autonomous development: they provide an objective, tools, and an overall resource budget, and let the agent decide how to proceed. Held-out testing then assesses whether the resulting policies generalize.

2. An environment-trial budget measures how efficiently agents use limited interaction opportunities

Consider policy development for a real robotic arm. After a failed grasp, agent A makes ten model calls to analyze existing trajectories and inspect its code, then conducts one new robot trial. Agent B makes one model call to generate a parameter-search script, then conducts ten robot trials. A uses more model calls, but B consumes ten times as many robot trials. Each robot trial also requires physical execution, object resets, and equipment time. Using more analysis to reduce the number of expensive trials is precisely an ability that autonomous policy development should reward.

We therefore use environment episodes as the primary resource constraint, giving every agent the same budget of 128 development trials while allowing it to choose its own reasoning, programming, and search methods. The many single-episode submissions in Kitchen illustrate this autonomous experimental design; environment trials initiated by search programs are charged to the same budget. The final comparison measures which complete development system produces better held-out policy performance under the same environment interaction budget. The contribution of additional diagnostic information is examined separately through the rich-versus-minimal ablation.

AuthorsOpenReview ↗

Author Response (2/2)

3. Environment coverage and the experimental task subset serve different purposes

There are clear precedents for selecting task subsets from broad environment suites for in-depth evaluation. DeepMind's RL Unplugged selects 9 tasks from the Control Suite, covering a range of difficulties. The Melting Pot Contest, co-organized by Google DeepMind and others, selects 4 environments based on diversity, solvability, and the cooperative behaviors they require.

Our 357 named variants describe the framework's adapter coverage, while the 8 experimental tasks support systematic comparisons of agents' policy-development capabilities. These tasks come from 8 different ecosystems and cover exploration and resource management, planning under incomplete information, visual continuous control, robotic manipulation, and locomotion, with diverse observation and action formats. Task selection thus has an explicit rationale grounded in capability coverage. We will make this rationale more prominent in the main text and clarify that our empirical conclusions concern these 8 tasks; inclusion in the catalog does not imply that every variant has undergone the same degree of empirical validation.

4. Explain scores and their behavioral meaning alongside the main results

We will add operational definitions of each task's score near Table 1 and supplement potentially ambiguous scores with behavioral metrics. For example, TwoArmTransport's score of 97.29 is a cumulative return, not a 97.29% success rate; a nonzero Humanoid return does not imply sustained walking. In the revision, we will also report task success rates for robotic transport, full-lap completion for racing, and whether sustained walking is achieved in Humanoid, helping readers connect numerical scores to actual policy capabilities.

5. The ablation specifically studies feedback richness

A score is itself a form of feedback. Section 5.4 explicitly compares rich and minimal feedback to study how feedback richness affects policy optimization, rather than comparing the presence and absence of feedback. Both conditions retain rewards, execution status, and budget information; the rich condition additionally provides diagnostics, trajectories, and replay artifacts. Thus, the “value of added diagnostics” identified in the review is precisely the question this experiment is designed to address.


Finally, RSI is attracting considerable attention, and related work shares common themes in feedback-driven iterative improvement. Different research perspectives lead to different understandings of the problem and different design choices. We believe that our design and analysis reflect a deep understanding of autonomous policy optimization in interactive environments and constitute a substantive contribution beyond prior work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.