acceptodds
Under review as a conference paper at ICLR 2027

LLM-Designed Training Environment for RL with Multi-Agent Reasoning

Abstract

Reinforcement learning pipelines for Large Language Model (LLM) training often rely on manually redesigned environments between stages, requiring practitioners to heuristically infer which configuration will best improve the current policy. To automate this process, we propose the LLM-as-Environment-Engineer framework in which the current policy model analyzes failure trajectories together with contextual information and proposes modifications to the next-stage training environment configuration. We also introduce MAPF-FrozenLake, a controllable testbed whose generator exposes multi-dimensional environment configurations, making it suitable for studying and benchmarking environment redesign. On this testbed, we condition the environment engineer on structured summaries of policy behavior, failure cases, and environment statistics, from which it produces the configuration for the next training stage. With Qwen3-4B as the learner, our framework outperforms fixed-environment training, curricula designed by a human RL expert, and stronger external models (e.g., GPT-5.4) acting as environment designers, and the resulting 4B policy surpasses much larger frontier LLMs (e.g., GPT, Gemini) on our benchmarks; the gains further transfer to other backbones and to ALFWorld. We further analyze which forms of context are most effective, finding that successful environment updates rely on failure evidence and preserve configurations that already work. Interestingly, the current RL checkpoint serves as a better environment engineer than the original base model, suggesting that policy learning improves the model's ability to diagnose its remaining weaknesses.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Environment, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FQtMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses the current RL learner as an environment engineer that changes the configuration of its future training-data generator after inspecting validation failures. MAPF-FrozenLake provides controllable grid-size, obstacle, and conflict settings. Experiments compare fixed, human, external-model, and learner-designed curricula, with transfer tests on other backbones and ALFWorld.

Critical strengths
  1. The controlled generator and separation of two-agent training from higher-agent-count evaluation strengthen the experimental design. The setup makes it possible to examine distribution design rather than merely adding generated examples.
  2. The designer-swap experiments and fixed-learner checkpoint comparison are particularly useful. They provide evidence that task training improves environment-design utility, beyond a comparison with the original untrained policy alone.
Critical weaknesses
  1. The non-LLM controls do not establish an advantage over a competent automatic curriculum using the same failure statistics, such as a success-rate or learning-progress allocator. Since the generator exposes a small structured configuration space, this comparison is important for attributing the benefit specifically to LLM-based design.
  2. The qualitative reasoning labels are derived from pooled traces and partly judged by other LLMs. They characterize behaviors but do not establish causal self-diagnosis. The main result tables also lack independent-training uncertainty for the smaller gains, especially the five-agent and weaker-backbone cases.
Questions for authors
  1. How does the loop compare with an automatic non-LLM curriculum using the same failure breakdown, configuration bounds, and total training budget?
  2. Are the designer ordering and higher-agent-count gains consistent across independent training and initial-configuration seeds?
3 comments

W1 & Q1: Comparison with a competent non-LLM curriculum under the same failure breakdown, configuration bounds and training budget.

We implemented the two allocators the reviewer names, together with a random configuration per round and four unsupervised environment design (UED) methods ported to our configuration space:

(1) a success-rate allocator, which assigns more data to map sizes with higher validation failure rates;

(2) a learning-progress allocator, which assigns more data to map sizes whose validation valid rate changes most;

(3) a random configuration per round;

(4) Prioritized Level Replay (PLR) [1];

(5) ACCEL [2];

(6) PAIRED [3];

(7) the Paired Open-Ended Trailblazer (POET) [4].

Every designer reads the same failure breakdown of the current checkpoint, its output passes through the same configuration bounds, and it trains with the same total budget as our method; only the designer is replaced. The implementation is described in §3.1 (Implementation and Baselines) and Appendix K. Benchmark Sum (valid / optimal, %), with the designers of Table 2 included for reference:

Designer3-agent4-agent5-agent
Qwen3-4B (random), no designer, 3 seeds39.83±0.55 / 27.72±1.4225.05±1.41 / 16.92±0.8113.78±1.15 / 8.93±0.32
Random configuration per round39.25 / 29.0024.57 / 18.5713.33 / 9.33
Success-rate allocator44.08 / 30.3327.05 / 20.4816.89 / 10.67
Learning-progress allocator46.17 / 30.7529.71 / 20.4816.00 / 10.22
PLR45.25 / 28.1729.33 / 19.7117.00 / 10.44
ACCEL41.92 / 27.2527.24 / 19.6214.89 / 10.11
PAIRED36.17 / 27.0020.57 / 16.8613.44 / 9.33
POET43.67 / 30.1725.90 / 18.7615.89 / 9.11
Human-adaptive (Table 2)41.92 / 27.3328.38 / 18.1915.78 / 10.44
GPT-5.4 as designer (Table 2)46.58 / 28.9229.52 / 19.0517.56 / 10.67
Qwen3-8B as designer (Table 2)44.75 / 27.5828.19 / 20.1017.00 / 9.89
Untrained base model as designer (Table 2)45.21 / 30.0027.62 / 17.8115.22 / 10.89
Learner as designer (ours, seed 0)51.67 / 31.6733.14 / 21.3318.67 / 11.00

The per-size results are reported as the red rows of Table 1 and discussed in §3.2 (Main Results), and Appendix K gives the full results (Table 13). The success-rate and learning-progress allocators are above the fixed configuration at every agent count on both valid and optimal rate, which confirms that a competent automatic curriculum helps, whereas a new random configuration every round does not. Yet the learner-as-designer outperforms both allocators, and all other designers, at every agent count on both valid and optimal rate. This attributes the remaining benefit to the LLM-based design itself: with the same failure breakdown, bounds and budget, the policy being trained produces better training distributions than a competent automatic curriculum.

W2 (i): The qualitative reasoning labels do not establish causal self-diagnosis.

Table 3 is presented as a qualitative characterization of behaviors (§3.4), and we do not use it as causal evidence. The evidence that RL training improves the designer comes from comparisons that change only the designer while keeping the learner and its training identical:

(1) in Table 2, the RL-trained checkpoint as designer outperforms the untrained base model as designer at every agent count (valid 51.67 / 33.14 / 18.67 vs. 45.21 / 27.62 / 15.22 at 3 / 4 / 5 agents);

(2) in the fourth group of Table 5, designers taken from successive checkpoints of the loop yield increasingly better results for the same learner.

To establish the causal link, we alter only the failure report the engineer reads, with the setting otherwise identical to our method:

(1) shuffle, which permutes the per-size statistics across map sizes, keeping the format and the amount of information but breaking the correspondence between failures and sizes;

(2) aggregate, which keeps only the overall rates.

We briefly report this in §3.3 ("Does the engineer use the failure report?") of the revised PDF, with the setup and full results in Appendix N. Benchmark Sum (valid / optimal, %):

Report read by the engineer3-agent4-agent5-agent
Unmodified (ours, seed 0)51.67 / 31.6733.14 / 21.3318.67 / 11.00
Shuffle40.25 / 26.4223.71 / 18.2914.89 / 10.67
Aggregate37.82 / 21.8319.43 / 15.8111.56 / 9.67

Both variants fall below the unmodified loop at every agent count on both metrics, and below the success-rate and learning-progress allocators of our response to W1 on valid rate. The engineer's allocation also follows the report it reads; for example, when the shuffled report shows 5×5 as the strongest size, the engineer assigns 40% of the budget to it. Since only the report is altered, the drop in performance and the shift in allocation give causal evidence that the engineer's gain comes from the content of its failure report rather than from a fixed prior curriculum.

To avoid relying on LLM judges, we also check the written diagnosis with fixed rules. Checkpoints after 0 (base model), 1, 2 and 3 rounds of training act as the engineer on the true and the shuffled reports of rounds 1–3 (8 samples per report), and every number in the analysis is checked against the report (Appendix N). Hallucinated: fraction of cited numbers absent from the report; wrong denominator: fraction of responses that attach a count to the wrong total; ρ: Spearman correlation with training steps over individual responses:

MetricReportBase1 round2 rounds3 roundsρ (p)
Hallucinated (%)true9.61.32.33.1−0.26 (0.014)
shuffled6.72.51.00.6−0.45 (<0.001)
Wrong denominator (%)true29.216.74.20.0−0.31 (0.002)
shuffled75.020.84.28.3−0.35 (<0.001)
Length (characters)true15,1995,1935,0504,344−0.71 (<0.001)
shuffled14,7444,3904,4254,552−0.54 (<0.001)

On both the true and the shuffled reports, all three measures improve as RL training proceeds, each significantly correlated with the number of training steps: hallucination drops sharply, as the engineer fabricates far fewer numbers; wrong denominators decrease until they almost disappear; and the analysis shrinks to about a third of its length, indicating more precise reasoning. This is consistent with §3.6 ("Does more training make a better engineer?", Table 5, fourth group), where later checkpoints also design better curricula. Together with the intervention above, this shows with quantitative, rule-based measures rather than LLM-judged labels that RL training makes the engineer's diagnosis more faithful to the evidence and that its decisions depend on that diagnosis.

W2 (ii): Independent-training uncertainty for the smaller gains (five agents, weaker backbone).

We trained two additional seeds for both our method and the random-configuration baseline on Qwen3-4B and on Gemma-3-4B, so each has three runs including the original one (seed 0). A new seed changes the generated data and the data order in every round, while the round-0 configuration stays the same. Benchmark Sum (valid / optimal, %; mean ± std over 3 seeds):

BackboneMethod3-agent4-agent5-agent
Qwen3-4Brandom39.83±0.55 / 27.72±1.4225.05±1.41 / 16.92±0.8113.78±1.15 / 8.93±0.32
Qwen3-4BOurs51.31±1.41 / 34.44±2.6132.89±0.24 / 22.95±1.4219.22±0.56 / 11.93±0.94
Gemma-3-4Brandom7.92±0.66 / 7.16±1.015.36±0.24 / 4.54±0.381.41±0.28 / 1.22±0.19
Gemma-3-4BOurs10.19±0.59 / 8.83±1.116.41±0.56 / 6.19±0.781.85±0.13 / 1.71±0.17

On Qwen3-4B, the 5-agent gain is +5.4 valid and +3.0 optimal points, and at every agent count and on both metrics the worst seed of our method is above the best seed of the baseline. On Gemma-3-4B, our method is above the baseline for every seed on every metric, and its worst seed is above the baseline's best seed on five of the six metrics, including both 5-agent metrics (the exception is 3-agent optimal rate, 7.58 vs. 8.08). Gemma-3-4B is a weak MAPF solver (below 11% valid rate after training), so its absolute gains are small (+2.3 / +1.1 / +0.4 valid points), but they are consistent across seeds, and the ordering stated in §3.5 holds. The smaller gains are therefore not an artifact of a single run, on either the five-agent benchmark or the weaker backbone. In the revised PDF, Table 1 and §3.2 report the Qwen3-4B 3-seed means and Table 4 the Gemma-3-4B 3-seed means, with the per-seed results in Appendix J (Table 12) and Appendix E (Table 7).

Q2: Consistency of the designer ordering and of the higher-agent-count gains across seeds.

The higher-agent-count gains are consistent across independent training runs, as shown in our response to W2 (ii): with three seeds, the gain over Qwen3-4B (random) is +11.5 / +7.8 / +5.4 valid points at 3 / 4 / 5 agents. They are also consistent across initial configurations: in the second group of Table 5, starting the loop from a GPT-5.4-designed, an expert-designed or a random round-0 configuration gives 53.08 / 50.58 / 51.67 (3-agent), 37.23 / 33.71 / 33.14 (4-agent) and 20.74 / 17.22 / 18.67 (5-agent) valid, all well above Qwen3-4B (random).

For the designer ordering, the lead of the learner-as-designer over every other designer is robust: its 3-seed mean (51.31 / 32.89 / 19.22 valid) remains above all 11 other designers in the table of our response to W1 at every agent count, and even its worst seed is above the best of them. The top position of the learner-as-designer is therefore not an artifact of a single run.

Summary of revisions.

All results and changes in this response can be found in the revised PDF, marked in red, and we hope these revisions address the reviewer's concerns. We thank the reviewer for the positive assessment and the precise suggestions, which have helped us strengthen the paper. Following the reviewer's suggestions, we:

(1) compared our loop with the success-rate and learning-progress allocators the reviewer suggested, together with five other non-LLM adaptive designers, under the identical setting;

(2) repeated our method and the random-configuration baseline over three seeds on both Qwen3-4B and Gemma-3-4B;

(3) intervened on the failure report the engineer reads and measured the quality of its written diagnosis with rule-based checks.

References

[1] Prioritized Level Replay, Jiang et al., 2021.

[2] Evolving Curricula with Regret-Based Environment Design, Parker-Holder et al., 2022.

[3] Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, Dennis et al., 2020.

[4] Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions, Wang et al., 2019.

Reviewer_Gcv5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an LLM-as-Environment-Engineer framework, in which an RL-trained policy iteratively adjusts its future training data distribution based on validation failures and training history. The authors introduce MAPF-FrozenLake, a synthetic multi-agent pathfinding environment with configurable map sizes, obstacle densities, and conflict ratios. Experiments demonstrate improvements over fixed-distribution RL training, human-designed curricula, and external LLM-based environment designers. Additional experiments examine transfer across model backbones and to ALFWorld.

Critical strengths
  1. Interesting research direction. Using the current policy to adapt its own future training distribution is a potentially useful approach to automating curriculum design in RL. 2. Controlled evaluation and informative comparisons. The configurable MAPF-FrozenLake environment enables controlled comparisons of different design strategies. The same-backbone experiments and the comparison between different engineer checkpoints provide useful preliminary evidence.
Critical weaknesses
  1. Overly simplistic experimental setting and limited generalizability.

The primary benchmark consists entirely of procedurally generated textual grid-world problems. More importantly, the proposed "environment engineering" is restricted to adjusting a few predefined numerical parameters, rather than creating or modifying environments in a meaningful way. The ALFWorld experiment similarly only adjusts sampling proportions across six existing task categories. These highly constrained settings are insufficient to substantiate the paper's broad claims about automated environment design.

  1. Missing essential non-LLM baselines and unclear necessity of LLM-based design.

The environments and ground-truth solutions are already generated automatically using classical algorithms. The LLM only determines the sampling distribution and difficulty parameters, which could potentially be optimized using simple rule-based, performance-driven, or conventional automated curriculum-learning methods. Without comparisons against these non-LLM adaptive approaches, it remains unclear whether an LLM is necessary or offers any meaningful advantage over standard synthetic data generation and curriculum optimization.

  1. Insufficient experimental rigor and limited evidence for the claimed mechanism.

The main results lack statistical variation across independent training runs. Comparisons against zero-shot frontier models do not isolate the benefits of environment design from task-specific RL training. Moreover, the claim that RL improves the model's ability to diagnose its own weaknesses is primarily supported by downstream performance and qualitative trajectory analysis, without directly validating diagnostic accuracy or establishing the proposed causal mechanism.

Questions for authors
  1. How does the proposed approach compare against rule-based, performance-driven, or conventional automated curriculum-learning methods?

  2. Can the authors evaluate the framework on more established and practically relevant benchmarks, such as coding or mathematical reasoning tasks, to demonstrate its generalizability beyond synthetic grid-world environments?

  3. The RL reward combines accuracy and response-length penalties using several hand-designed thresholds and adaptive weights. What motivates these choices, and how sensitive are the results to the reward design? Could a simpler correctness-based reward achieve comparable performance?

9 comments

W1: The setting is too constrained to support claims about environment design.

We respectfully disagree that the controlled setting undermines our conclusions; it is a deliberate choice, made for three reasons:

(1) The CBS solver provides the optimal solution for every instance, so the reward is exactly verifiable and cannot be hacked.

(2) Because the generator code, the reward and the evaluation are all fixed, any difference between runs can be attributed to the design decisions alone, which is what makes our "swap only the designer" comparisons possible; Reviewer FQtM highlights this as a strength.

(3) In our framework the designer is the policy being trained, which under our compute budget is an open model of 4B–8B parameters. Open models of this size cannot yet reliably write complete, executable environments with verifiable rewards (OMNI-EPIC [1] relies on frontier proprietary models for this), whereas editing a structured configuration is within their reach.

The design space is small, but the design problem is not trivial. The space has 24 coupled parameters (data ratio on the simplex, hole ratio and wait ratio for each of 8 map sizes), and the same per-size failure rate can call for opposite edits: obstacle collisions call for a lower hole ratio, whereas agent conflicts call for more WAIT-requiring instances. With the same starting checkpoint, the same feedback and exactly this configuration space, the valid rate of the 12 designers in our response to W2 ranges from 36.17 to 51.67, 20.57 to 33.14 and 13.33 to 18.67 at 3, 4 and 5 agents, with the learner as designer at the top in each case. How these parameters are set therefore changes the outcome substantially.

Regarding the scope of our claims, in the unsupervised environment design (UED) literature "environment design" refers to choosing the free parameters of an underspecified environment (PAIRED [2]), which is exactly what our engineer does. The ALFWorld experiment is not meant as a second showcase of environment design; its purpose is to show that the loop generalizes to a different agentic task, where the engineer sets the task distribution over the six existing categories.

Q2: Evaluation on coding or mathematical reasoning.

We applied the full loop to two new task families with an otherwise identical setting, the engineer again being the current learner checkpoint:

(1) math, with procedurally generated problems from eight Reasoning Gym [3] categories and exact answer checking, where the engineer sets each category's data ratio and difficulty;

(2) code, with program-synthesis problems graded by hidden unit tests.

Qwen3-4B (random) trains for the same number of steps on its round-0 data. We evaluate on MATH-500 [4, 5] and AMC23 for math and on HumanEval [6] and MBPP [7] for code, none of which is used for training or design. We briefly report this in "Across task families" in §3.5 of the revised PDF (bottom block of Table 4), with the full setup in Appendix M. Accuracy / pass@1 (%, single seed):

MethodMATH-500AMC23HumanEvalMBPP
Base Qwen3-4B65.2535.1679.6569.20
Qwen3-4B (random)83.0064.2285.3775.90
Learner as designer (ours)86.7071.5691.8480.00

Our loop outperforms Qwen3-4B (random) on all four benchmarks, by +3.70 (MATH-500), +7.34 (AMC23), +6.47 (HumanEval) and +4.10 (MBPP).

W2 & Q1: Comparison with rule-based, performance-driven and automated curriculum methods.

To answer this question, we compare against seven non-LLM adaptive designers, including four unsupervised environment design (UED) methods ported to our configuration space:

(1) a random configuration per round;

(2) a success-rate allocator, which assigns more data to map sizes with higher validation failure rates;

(3) a learning-progress allocator, which assigns more data to map sizes whose validation valid rate changes most;

(4) Prioritized Level Replay (PLR) [8], which replays high-regret configurations;

(5) ACCEL [9], which edits replayed configurations;

(6) PAIRED [2], which searches for regret-maximizing configurations;

(7) the Paired Open-Ended Trailblazer (POET) [10], which evolves configurations with the learner.

The experimental setting is identical to that of our method, with only the designer replaced; the implementation is described in §3.1 (Implementation and Baselines) and Appendix K. Benchmark Sum (valid / optimal, %):

Designer3-agent4-agent5-agent
Qwen3-4B (random), no designer, 3 seeds39.83±0.55 / 27.72±1.4225.05±1.41 / 16.92±0.8113.78±1.15 / 8.93±0.32
Random configuration per round39.25 / 29.0024.57 / 18.5713.33 / 9.33
Success-rate allocator44.08 / 30.3327.05 / 20.4816.89 / 10.67
Learning-progress allocator46.17 / 30.7529.71 / 20.4816.00 / 10.22
PLR45.25 / 28.1729.33 / 19.7117.00 / 10.44
ACCEL41.92 / 27.2527.24 / 19.6214.89 / 10.11
PAIRED36.17 / 27.0020.57 / 16.8613.44 / 9.33
POET43.67 / 30.1725.90 / 18.7615.89 / 9.11
Learner as designer (ours, seed 0)51.67 / 31.6733.14 / 21.3318.67 / 11.00

The per-size results are reported as the red rows of Table 1 and discussed in §3.2 (Main Results), and Appendix K gives the full results (Table 13). A new random configuration every round stays within 1.7 points of the fixed configuration on every metric and is slightly lower on all three valid rates, so new training data alone does not help. The performance-driven allocators and PLR are above the fixed configuration at every agent count on both valid and optimal rate, which confirms that adaptivity helps. Yet the learner-as-designer outperforms all seven designers at every agent count on both valid and optimal rate, which shows that, with the same feedback and design space, an LLM designer adds a clear benefit beyond simple adaptive rules.

W3 (i): Variation across independent training runs.

To measure the variation across independent training runs, we trained two additional seeds for both our method and Qwen3-4B (random), so each has three runs including the original one (seed 0). A new seed changes the generated data and the data order in every round, while the round-0 configuration stays the same. Benchmark Sum (valid / optimal, %):

MethodSeed3-agent4-agent5-agent
Qwen3-4B (random)0 (paper)40.42 / 26.0826.67 / 16.1015.11 / 9.11
139.33 / 28.4224.38 / 17.7113.11 / 9.11
239.75 / 28.6724.10 / 16.9513.11 / 8.56
mean ± std39.83±0.55 / 27.72±1.4225.05±1.41 / 16.92±0.8113.78±1.15 / 8.93±0.32
Ours0 (paper)51.67 / 31.6733.14 / 21.3318.67 / 11.00
152.50 / 36.8332.67 / 23.5219.78 / 12.89
249.75 / 34.8332.86 / 24.0019.22 / 11.89
mean ± std51.31±1.41 / 34.44±2.6132.89±0.24 / 22.95±1.4219.22±0.56 / 11.93±0.94

The improvement is stable across seeds: +11.5 / +7.8 / +5.4 valid and +6.7 / +6.0 / +3.0 optimal points at 3 / 4 / 5 agents, and for every agent count and both metrics the worst seed of our method is above the best seed of the baseline. In the revised PDF, §3.1 (Implementation and Baselines) states the additional seeds, Table 1 reports the 3-seed mean for both trained Qwen3-4B rows (in red), and §3.2 (Main Results) gives the resulting gains; Appendix J gives the seed protocol, the mean ± std (Table 11) and the per-seed, per-map-size results (Table 12).

W3 (ii): Separating environment design from task-specific RL training.

We respectfully disagree that our comparisons do not isolate the benefit of environment design. The zero-shot frontier models only situate the absolute performance level and are never used to attribute the gains; attribution rests on two controlled comparisons already in the submission:

(1) Qwen3-4B (random) receives exactly the same GRPO training as our method (same base model, hyperparameters, reward and number of steps) on a fixed configuration; §3.1 introduces it to isolate the contribution of our framework from that of GRPO itself, and §3.2 concludes that our gain over it is attributable to environment redesign rather than to task specialization.

(2) Table 2 fixes the learner and the training and swaps only the designer (human-fixed, human-adaptive, GPT-5.4, Qwen3-8B, the untrained base model and the learner itself), so any difference in that table is due to the designer alone.

The revision strengthens both comparisons:

(1) Qwen3-4B (random) and our method now have three seeds each, with no overlap between seeds at any agent count (our response to W3 (i));

(2) seven non-LLM adaptive designers, including four UED methods, are compared under the same protocol and all fall below our method (our response to W2).

Both additions are reported in Table 1 and §3.2 of the revised PDF.

Reviewer_9dALOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for LLM-designed training environment for RL. They test a 4B model as a learner.

Critical strengths

The question on whether the learner carries useful information for designing its own curriculum is interesting and potentially useful.

Critical weaknesses

The novelty of the paper is very limited, as the paper completely misses on a vast amount of well known related work on unsupervised environment design (UED) and open-ended learning: see POET (Wang et al., 2019), Enhanced POET (Wang et al., 2020), PAIRED (Dennis et al., 2020), PLR (Jiang et al., 2021), ACCEL (Parker-Holder et al., 2022), as well as very close to this method, LLM-based environment design such as EnvGen (Zala et al., 2024) and OMNI-EPIC (Faldor et al., 2025). Showing that an adaptive curriculum beats fixed random configs is nothing new and a well-established results from said literature. The paper needs to compare and contrast with that literature, and use some of those methods as baselines.

No error bars are computed anywhere, seemingly only 1 seed used, which is not enough.

Questions for authors

Can the authors thoroughly compare and contrast with the UED literature and use methods from that literature as baselines? See references in weaknesses.

6 comments

W1 & Q1: Relation to UED / open-ended learning and LLM-based environment design; these methods as baselines.

We thank the reviewer for pointing us to the UED and open-ended learning literature. That an adaptive curriculum beats fixed random configurations is indeed well established by this literature, and it is not what our paper claims. The questions we study do not arise in UED, where the designer and the learner are separate entities:

(1) whether the policy being trained can design its own training distribution;

(2) whether RL training on the task improves this ability.

The table below contrasts our loop with the methods the reviewer lists:

MethodWho designsDesigner is the learnerDesigner improves during trainingFeedback used for designDesign outputLearnerIn our comparison
POET [1]Evolutionary mutation with a minimal criterion✗✗Agent return on each environmentMutated environment parametersPopulation of small networks (ES)Ported (single learner)
Enhanced POET [2]As POET, plus a population-based novelty measure (PATA-EC)✗✗Returns of all agents on each environmentMutated environmentsPopulation of small networks (ES)Reduces to the POET port
PAIRED [3]RL adversary✗✓ (trained on a separate regret objective)Regret between two agentsLevel built element by elementSmall RL agents (PPO)Ported (simplified)
PLR [4]Score-based replay of sampled levels✗✗Value loss (regret estimate), stalenessReplay distribution over levelsSmall RL agent (PPO)Ported
ACCEL [5]PLR plus random edits of high-regret levels✗✗Value loss (regret estimate)Edited levelsSmall RL agent (PPO)Ported
EnvGen [6]External frozen LLM (GPT-4)✗✗Per-skill success ratesEnvironment configurationsSmall RL agent (PPO)In the submission (Table 2)
OMNI [7]Learning progress, filtered by an LLM judging interestingness✗✗Per-task learning progressSampling weights over a task setSmall RL agent (PPO)Learning-progress allocator
OMNI-EPIC [8]Frontier proprietary models (Claude 3 Opus, GPT-4o)✗✗Archive of learned and failed tasksExecutable environment and reward codeRL agent (DreamerV3)Not applicable
OursThe policy being trained (Qwen3-4B)✓✓ (through RL on the task itself)Per-size failure types, valid and optimal rates, training stageGenerator configuration (data ratio, hole ratio, wait ratio per map size)The same LLM (GRPO)—

Our loop is the only one in which the designer is the learner itself, and the only one whose design ability improves through RL on the task, with no separate design objective.

Already in the submission, Table 2 compares the learner-as-designer against adaptive designers that receive the same per-round feedback:

(1) a senior RL researcher who redesigns the configuration every round;

(2) external LLM designers (GPT-5.4 and Qwen3-8B), which are the EnvGen paradigm in our setting, i.e., an external LLM that reads per-skill feedback and writes environment configurations for a fixed learner;

(3) the untrained base model.

For the revision, we ported PLR, ACCEL, PAIRED and POET to our configuration space and added three non-LLM adaptive designers:

(1) a random configuration per round;

(2) a success-rate allocator, which assigns more data to map sizes with higher validation failure rates;

(3) a learning-progress allocator, which assigns more data to map sizes whose validation valid rate changes most, and which is also the task-sampling rule of OMNI.

Two methods have no separate run. Enhanced POET differs from POET in PATA-EC and its transfer criterion, which are defined over a population of agents, so with a single learner it reduces to our POET port. OMNI-EPIC writes executable environment code and success detectors with frontier proprietary models; our setting deliberately fixes the generator and the verifier-checked reward so that any difference between runs comes from the design decisions alone, so OMNI-EPIC has no counterpart here. For the other methods, the experimental setting is identical to that of our method, with only the designer replaced; the implementation is described in §3.1 (Implementation and Baselines) and Appendix K. Benchmark Sum (valid / optimal, %):

Designer3-agent4-agent5-agent
Qwen3-4B (random), no designer, 3 seeds39.83±0.55 / 27.72±1.4225.05±1.41 / 16.92±0.8113.78±1.15 / 8.93±0.32
Random configuration per round39.25 / 29.0024.57 / 18.5713.33 / 9.33
Success-rate allocator44.08 / 30.3327.05 / 20.4816.89 / 10.67
Learning-progress allocator (OMNI's sampling rule)46.17 / 30.7529.71 / 20.4816.00 / 10.22
PLR45.25 / 28.1729.33 / 19.7117.00 / 10.44
ACCEL41.92 / 27.2527.24 / 19.6214.89 / 10.11
PAIRED (simplified)36.17 / 27.0020.57 / 16.8613.44 / 9.33
POET (single learner)43.67 / 30.1725.90 / 18.7615.89 / 9.11
Human-adaptive (Table 2)41.92 / 27.3328.38 / 18.1915.78 / 10.44
GPT-5.4 as designer (EnvGen paradigm, Table 2)46.58 / 28.9229.52 / 19.0517.56 / 10.67
Qwen3-8B as designer (EnvGen paradigm, Table 2)44.75 / 27.5828.19 / 20.1017.00 / 9.89
Untrained base model as designer (Table 2)45.21 / 30.0027.62 / 17.8115.22 / 10.89
Learner as designer (ours, seed 0)51.67 / 31.6733.14 / 21.3318.67 / 11.00

The per-size results are reported as the red rows of Table 1 and discussed in §3.2 (Main Results), and Appendix K gives the full results (Table 13). The learner-as-designer outperforms all 11 other designers at every agent count on both valid and optimal rate. This shows that, beyond adaptivity, it matters who designs: with the same feedback and design space, the policy being trained produces better training distributions than UED, non-LLM and external-LLM designers.

The table above also shows that RL training improves the learner as a designer: the RL-trained checkpoint outperforms the untrained base model as designer at every agent count on both metrics. The submission tests this further in the fourth group of Table 5, where checkpoints trained for 1, 2 and 3 rounds each design the next configuration for the same one-round learner, with the setting otherwise identical. This is discussed in §3.6 ("Does more training make a better engineer?"), with the full results in Appendix F (Table 8). Benchmark Sum (valid / optimal, %):

Designer checkpoint3-agent4-agent5-agent
1-round checkpoint43.42 / 27.7525.33 / 18.4113.33 / 9.56
2-round checkpoint44.08 / 28.8326.00 / 18.7515.00 / 9.44
3-round checkpoint47.60 / 28.7529.71 / 20.3817.44 / 10.22

Later checkpoints yield higher valid rates at every agent count. The design ability of our engineer therefore keeps improving with RL training on the task, which none of the separate designers above can offer.

The revised PDF also adds a related-work paragraph on environment design (Appendix A, referenced from §4) covering POET, Enhanced POET, PAIRED, PLR / Robust PLR [9], ACCEL, EnvGen, OMNI and OMNI-EPIC.

W2: No error bars; a single seed.

Following the reviewer's suggestion, we have now trained two additional independent seeds for both our method and Qwen3-4B (random), so each has three runs including the original one (seed 0). A new seed changes the generated data and the data order in every round, while the round-0 configuration stays the same. Benchmark Sum (valid / optimal, %):

MethodSeed3-agent4-agent5-agent
Qwen3-4B (random)0 (paper)40.42 / 26.0826.67 / 16.1015.11 / 9.11
139.33 / 28.4224.38 / 17.7113.11 / 9.11
239.75 / 28.6724.10 / 16.9513.11 / 8.56
mean ± std39.83±0.55 / 27.72±1.4225.05±1.41 / 16.92±0.8113.78±1.15 / 8.93±0.32
Ours0 (paper)51.67 / 31.6733.14 / 21.3318.67 / 11.00
152.50 / 36.8332.67 / 23.5219.78 / 12.89
249.75 / 34.8332.86 / 24.0019.22 / 11.89
mean ± std51.31±1.41 / 34.44±2.6132.89±0.24 / 22.95±1.4219.22±0.56 / 11.93±0.94

The improvement is stable across seeds: +11.5 / +7.8 / +5.4 valid points at 3 / 4 / 5 agents, and for every agent count and both metrics the worst seed of our method is above the best seed of the baseline (e.g., 18.67–19.78 vs. 13.11–15.11 valid at 5 agents). The gains reported in the submission are therefore not an artifact of a single run. In the revised PDF, §3.1 (Implementation and Baselines) states the additional seeds, Table 1 reports the 3-seed mean for both trained Qwen3-4B rows (in red), and §3.2 (Main Results) gives the resulting gains; Appendix J gives the seed protocol, the mean ± std (Table 11) and the per-seed, per-map-size results (Table 12).

Additional experiment: math and code reasoning.

To show that the loop is not limited to grid-world planning, we also applied it to math (procedurally generated problems from eight Reasoning Gym [10] categories) and code (program synthesis graded by hidden unit tests), with the engineer setting each category's data ratio and difficulty and the setting otherwise identical to our method. Qwen3-4B (random) trains for the same number of steps on its round-0 data. We briefly report this in "Across task families" in §3.5 of the revised PDF (bottom block of Table 4), with the full setup in Appendix M. We evaluate on MATH-500 [11, 12] and AMC23 for math and on HumanEval [13] and MBPP [14] for code. Accuracy / pass@1 (%):

MethodMATH-500AMC23HumanEvalMBPP
Base Qwen3-4B65.2535.1679.6569.20
Qwen3-4B (random)83.0064.2285.3775.90
Learner as designer (ours)86.7071.5691.8480.00

Our loop outperforms Qwen3-4B (random) on all four benchmarks, none of which is used for training or design, so its gains carry over to established math and code benchmarks.

Summary of revisions.

All results and changes in this response can be found in the revised PDF, marked in red, and we hope these revisions address the reviewer's concerns. We thank the reviewer again for the constructive comments, which have helped us strengthen the paper. Following the reviewer's suggestions, we:

(1) compared our loop with seven adaptive designers, including ports of PLR, ACCEL, PAIRED and POET, under the identical setting;

(2) repeated our method and Qwen3-4B (random) over three seeds;

(3) applied the loop to math and code reasoning;

(4) added a related-work paragraph on environment design (Appendix A).

References

[1] Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions, Wang et al., 2019.

[2] Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions, Wang et al., 2020.

[3] Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, Dennis et al., 2020.

[4] Prioritized Level Replay, Jiang et al., 2021.

[5] Evolving Curricula with Regret-Based Environment Design, Parker-Holder et al., 2022.

[6] EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents, Zala et al., 2024.

[7] OMNI: Open-endedness via Models of human Notions of Interestingness, Zhang et al., 2024.

[8] OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code, Faldor et al., 2025.

[9] Replay-Guided Adversarial Environment Design, Jiang et al., 2021.

[10] Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards, Stojanovski et al., 2025.

[11] Measuring Mathematical Problem Solving With the MATH Dataset, Hendrycks et al., 2021.

[12] Let's Verify Step by Step, Lightman et al., 2024.

[13] Evaluating Large Language Models Trained on Code, Chen et al., 2021.

[14] Program Synthesis with Large Language Models, Austin et al., 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.