- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses the current RL learner as an environment engineer that changes the configuration of its future training-data generator after inspecting validation failures. MAPF-FrozenLake provides controllable grid-size, obstacle, and conflict settings. Experiments compare fixed, human, external-model, and learner-designed curricula, with transfer tests on other backbones and ALFWorld.
Critical strengths
- The controlled generator and separation of two-agent training from higher-agent-count evaluation strengthen the experimental design. The setup makes it possible to examine distribution design rather than merely adding generated examples.
- The designer-swap experiments and fixed-learner checkpoint comparison are particularly useful. They provide evidence that task training improves environment-design utility, beyond a comparison with the original untrained policy alone.
Critical weaknesses
- The non-LLM controls do not establish an advantage over a competent automatic curriculum using the same failure statistics, such as a success-rate or learning-progress allocator. Since the generator exposes a small structured configuration space, this comparison is important for attributing the benefit specifically to LLM-based design.
- The qualitative reasoning labels are derived from pooled traces and partly judged by other LLMs. They characterize behaviors but do not establish causal self-diagnosis. The main result tables also lack independent-training uncertainty for the smaller gains, especially the five-agent and weaker-backbone cases.
Questions for authors
- How does the loop compare with an automatic non-LLM curriculum using the same failure breakdown, configuration bounds, and total training budget?
- Are the designer ordering and higher-agent-count gains consistent across independent training and initial-configuration seeds?
3 comments
Reply-2
W2 (i): The qualitative reasoning labels do not establish causal self-diagnosis.
Table 3 is presented as a qualitative characterization of behaviors (§3.4), and we do not use it as causal evidence. The evidence that RL training improves the designer comes from comparisons that change only the designer while keeping the learner and its training identical:
(1) in Table 2, the RL-trained checkpoint as designer outperforms the untrained base model as designer at every agent count (valid 51.67 / 33.14 / 18.67 vs. 45.21 / 27.62 / 15.22 at 3 / 4 / 5 agents);
(2) in the fourth group of Table 5, designers taken from successive checkpoints of the loop yield increasingly better results for the same learner.
To establish the causal link, we alter only the failure report the engineer reads, with the setting otherwise identical to our method:
(1) shuffle, which permutes the per-size statistics across map sizes, keeping the format and the amount of information but breaking the correspondence between failures and sizes;
(2) aggregate, which keeps only the overall rates.
We briefly report this in §3.3 ("Does the engineer use the failure report?") of the revised PDF, with the setup and full results in Appendix N. Benchmark Sum (valid / optimal, %):
| Report read by the engineer | 3-agent | 4-agent | 5-agent |
|---|---|---|---|
| Unmodified (ours, seed 0) | 51.67 / 31.67 | 33.14 / 21.33 | 18.67 / 11.00 |
| Shuffle | 40.25 / 26.42 | 23.71 / 18.29 | 14.89 / 10.67 |
| Aggregate | 37.82 / 21.83 | 19.43 / 15.81 | 11.56 / 9.67 |
Both variants fall below the unmodified loop at every agent count on both metrics, and below the success-rate and learning-progress allocators of our response to W1 on valid rate. The engineer's allocation also follows the report it reads; for example, when the shuffled report shows 5×5 as the strongest size, the engineer assigns 40% of the budget to it. Since only the report is altered, the drop in performance and the shift in allocation give causal evidence that the engineer's gain comes from the content of its failure report rather than from a fixed prior curriculum.
To avoid relying on LLM judges, we also check the written diagnosis with fixed rules. Checkpoints after 0 (base model), 1, 2 and 3 rounds of training act as the engineer on the true and the shuffled reports of rounds 1–3 (8 samples per report), and every number in the analysis is checked against the report (Appendix N). Hallucinated: fraction of cited numbers absent from the report; wrong denominator: fraction of responses that attach a count to the wrong total; ρ: Spearman correlation with training steps over individual responses:
| Metric | Report | Base | 1 round | 2 rounds | 3 rounds | ρ (p) |
|---|---|---|---|---|---|---|
| Hallucinated (%) | true | 9.6 | 1.3 | 2.3 | 3.1 | −0.26 (0.014) |
| shuffled | 6.7 | 2.5 | 1.0 | 0.6 | −0.45 (<0.001) | |
| Wrong denominator (%) | true | 29.2 | 16.7 | 4.2 | 0.0 | −0.31 (0.002) |
| shuffled | 75.0 | 20.8 | 4.2 | 8.3 | −0.35 (<0.001) | |
| Length (characters) | true | 15,199 | 5,193 | 5,050 | 4,344 | −0.71 (<0.001) |
| shuffled | 14,744 | 4,390 | 4,425 | 4,552 | −0.54 (<0.001) |
On both the true and the shuffled reports, all three measures improve as RL training proceeds, each significantly correlated with the number of training steps: hallucination drops sharply, as the engineer fabricates far fewer numbers; wrong denominators decrease until they almost disappear; and the analysis shrinks to about a third of its length, indicating more precise reasoning. This is consistent with §3.6 ("Does more training make a better engineer?", Table 5, fourth group), where later checkpoints also design better curricula. Together with the intervention above, this shows with quantitative, rule-based measures rather than LLM-judged labels that RL training makes the engineer's diagnosis more faithful to the evidence and that its decisions depend on that diagnosis.
Reply-3
W2 (ii): Independent-training uncertainty for the smaller gains (five agents, weaker backbone).
We trained two additional seeds for both our method and the random-configuration baseline on Qwen3-4B and on Gemma-3-4B, so each has three runs including the original one (seed 0). A new seed changes the generated data and the data order in every round, while the round-0 configuration stays the same. Benchmark Sum (valid / optimal, %; mean ± std over 3 seeds):
| Backbone | Method | 3-agent | 4-agent | 5-agent |
|---|---|---|---|---|
| Qwen3-4B | random | 39.83±0.55 / 27.72±1.42 | 25.05±1.41 / 16.92±0.81 | 13.78±1.15 / 8.93±0.32 |
| Qwen3-4B | Ours | 51.31±1.41 / 34.44±2.61 | 32.89±0.24 / 22.95±1.42 | 19.22±0.56 / 11.93±0.94 |
| Gemma-3-4B | random | 7.92±0.66 / 7.16±1.01 | 5.36±0.24 / 4.54±0.38 | 1.41±0.28 / 1.22±0.19 |
| Gemma-3-4B | Ours | 10.19±0.59 / 8.83±1.11 | 6.41±0.56 / 6.19±0.78 | 1.85±0.13 / 1.71±0.17 |
On Qwen3-4B, the 5-agent gain is +5.4 valid and +3.0 optimal points, and at every agent count and on both metrics the worst seed of our method is above the best seed of the baseline. On Gemma-3-4B, our method is above the baseline for every seed on every metric, and its worst seed is above the baseline's best seed on five of the six metrics, including both 5-agent metrics (the exception is 3-agent optimal rate, 7.58 vs. 8.08). Gemma-3-4B is a weak MAPF solver (below 11% valid rate after training), so its absolute gains are small (+2.3 / +1.1 / +0.4 valid points), but they are consistent across seeds, and the ordering stated in §3.5 holds. The smaller gains are therefore not an artifact of a single run, on either the five-agent benchmark or the weaker backbone. In the revised PDF, Table 1 and §3.2 report the Qwen3-4B 3-seed means and Table 4 the Gemma-3-4B 3-seed means, with the per-seed results in Appendix J (Table 12) and Appendix E (Table 7).
Q2: Consistency of the designer ordering and of the higher-agent-count gains across seeds.
The higher-agent-count gains are consistent across independent training runs, as shown in our response to W2 (ii): with three seeds, the gain over Qwen3-4B (random) is +11.5 / +7.8 / +5.4 valid points at 3 / 4 / 5 agents. They are also consistent across initial configurations: in the second group of Table 5, starting the loop from a GPT-5.4-designed, an expert-designed or a random round-0 configuration gives 53.08 / 50.58 / 51.67 (3-agent), 37.23 / 33.71 / 33.14 (4-agent) and 20.74 / 17.22 / 18.67 (5-agent) valid, all well above Qwen3-4B (random).
For the designer ordering, the lead of the learner-as-designer over every other designer is robust: its 3-seed mean (51.31 / 32.89 / 19.22 valid) remains above all 11 other designers in the table of our response to W1 at every agent count, and even its worst seed is above the best of them. The top position of the learner-as-designer is therefore not an artifact of a single run.
Summary of revisions.
All results and changes in this response can be found in the revised PDF, marked in red, and we hope these revisions address the reviewer's concerns. We thank the reviewer for the positive assessment and the precise suggestions, which have helped us strengthen the paper. Following the reviewer's suggestions, we:
(1) compared our loop with the success-rate and learning-progress allocators the reviewer suggested, together with five other non-LLM adaptive designers, under the identical setting;
(2) repeated our method and the random-configuration baseline over three seeds on both Qwen3-4B and Gemma-3-4B;
(3) intervened on the failure report the engineer reads and measured the quality of its written diagnosis with rule-based checks.
References
[1] Prioritized Level Replay, Jiang et al., 2021.
[2] Evolving Curricula with Regret-Based Environment Design, Parker-Holder et al., 2022.
[3] Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, Dennis et al., 2020.
[4] Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions, Wang et al., 2019.
Reply-1
W1 & Q1: Comparison with a competent non-LLM curriculum under the same failure breakdown, configuration bounds and training budget.
We implemented the two allocators the reviewer names, together with a random configuration per round and four unsupervised environment design (UED) methods ported to our configuration space:
(1) a success-rate allocator, which assigns more data to map sizes with higher validation failure rates;
(2) a learning-progress allocator, which assigns more data to map sizes whose validation valid rate changes most;
(3) a random configuration per round;
(4) Prioritized Level Replay (PLR) [1];
(5) ACCEL [2];
(6) PAIRED [3];
(7) the Paired Open-Ended Trailblazer (POET) [4].
Every designer reads the same failure breakdown of the current checkpoint, its output passes through the same configuration bounds, and it trains with the same total budget as our method; only the designer is replaced. The implementation is described in §3.1 (Implementation and Baselines) and Appendix K. Benchmark Sum (valid / optimal, %), with the designers of Table 2 included for reference:
The per-size results are reported as the red rows of Table 1 and discussed in §3.2 (Main Results), and Appendix K gives the full results (Table 13). The success-rate and learning-progress allocators are above the fixed configuration at every agent count on both valid and optimal rate, which confirms that a competent automatic curriculum helps, whereas a new random configuration every round does not. Yet the learner-as-designer outperforms both allocators, and all other designers, at every agent count on both valid and optimal rate. This attributes the remaining benefit to the LLM-based design itself: with the same failure breakdown, bounds and budget, the policy being trained produces better training distributions than a competent automatic curriculum.