- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DyMGen, a controllable world model for unsupervised environment design. It learns latent generation actions from offline trajectories, allowing an adversarial teacher to dynamically control environment generation as the student explores. The authors introduce mechanisms to improve generation consistency and evaluate the approach on Key Minigrid, Minigrid, and Sokoban. DyMGen outperforms random world-model generation and PLR-based baselines, but remains below DEGen trained in the true environment.
Critical strengths
S1. Interesting approach to controllable world models.
The paper presents an interesting way to learn environment-generation controls directly from offline trajectories, enabling online adversarial curriculum learning without a handcrafted environment-generation interface.
S2. Promising empirical results.
DyMGen consistently outperforms the world-model baselines across three environments. The improvements are particularly clear in Key Minigrid and Sokoban, demonstrating the potential of adversarially controlled world models for curriculum learning.
Critical weaknesses
W1. Limited practical justification compared to DEGen.
The main motivation is to avoid requiring a handcrafted, dynamically controllable simulator. However, all experiments are conducted in environments where such simulators already exist. DyMGen is computationally more expensive and performs worse than DEGen. The paper does not demonstrate its advantages in the intended scenarios where access to a simulator is limited or unavailable. This weakens the practical significance of the contribution beyond DEGen.
W2. Potentially unfair comparison due to the Sokoban training data.
According to Appendix B.1, 25% of the Sokoban world-model training data comes from levels generated using true-environment UED, rather than domain randomization. This gives the world-model methods access to potentially stronger training data than Env-DR. Therefore, the improvement over Env-DR may be partially attributed to the training data rather than the proposed method. Although the comparison among world-model baselines remains valid, the paper should include a data-source ablation or a more carefully matched comparison to support its broader claims.
W3. Insufficient presentation of the main experimental results.
The main experimental results are not presented clearly enough. Figure 3 provides an overview of the training curves, but the small plots and overlapping curves make it difficult to compare the methods quantitatively. Important results, including the final solve rates reported in Tables 4–6, are only provided in the appendix. Since these results are central to evaluating the effectiveness of DyMGen, a summary table in the main paper would make the empirical contributions much easier to assess.
Questions for authors
Q1. Practical advantages over DEGen.
According to Figure 3 and Tables 4–6, DyMGen consistently underperforms DEGen across all three environments (74% vs. 83% on Key Minigrid, 86% vs. 96% on Minigrid, and 34% vs. 49% on Sokoban). Furthermore, Table 8 shows that DyMGen requires 15h14m per 10,000 PPO updates, compared to 8h18m for DEGen, even without accounting for the additional world-model training cost reported in Table 9. Given these results, what are the practical advantages of DyMGen over DEGen? Can the authors provide evidence of its benefits in settings where access to a simulator is limited or unavailable?
Q2. Fairness of the Sokoban comparison.
Appendix B.1 states that 25% of the Sokoban world-model training data comes from levels generated using true-environment UED, rather than domain randomization. Moreover, Appendix C.8 (Figure 17) shows that increasing the dataset size by 30× substantially improves DyMGen's performance. Given that Table 6 reports a 34% solve rate for DyMGen versus 0% for Env-DR, to what extent can this improvement be attributed to the proposed method rather than the stronger training data? Could the authors provide (1) an ablation varying the proportion of UED-generated trajectories while keeping the dataset size fixed, and (2) a comparison where the baselines have access to the same curated training data? In particular, how does DyMGen perform when its world model is trained exclusively on domain-randomized trajectories?
Q3. Validity of adversarially generated environments.
Section 7 acknowledges that the adversarial teacher sometimes generates trajectories inconsistent with the true environment, such as those containing multiple goals, keys, and doors. How frequently do such violations occur? Could the authors quantitatively evaluate the generated environments in terms of (1) validity under the environment constraints, (2) solvability, (3) consistency with previously observed states, and (4) correctness of predicted rewards and termination signals? It would also be helpful to compare these metrics between random and adversarial generation and examine how such inconsistencies affect student training.
8 comments
Rebuttal (2/3)
W2 and Q2
Thank you for identifying this point. We agree that the comparison with standard Env-DR alone cannot isolate the contribution of DyMGen from the influence of dataset composition.
To address this, we ran two additional sets of experiments. The first was to ablate the percentage of UED generated trajectories, and the second was examine the performance of DR, if the levels were randomly sampled from a set of UED-generated levels rather than randomly generated ones.
Dataset Ablation
Below we include a table that shows the performance of Sokoban DyMGen when trained with a dataset containing 25% (what is shown in the paper), 10% and 0% UED-generated levels. It is clear that these levels are necessary, as solvable randomly generated sokoban levels are so exceedingly uncommon, and so learning environment dynamics, specifically learning when levels are complete, becomes increasing more difficult with less examples of known solvable levels.
| Level | 25% UED | 10% UED | 0 UED |
|---|---|---|---|
| Sokoban_Jr_1_01 | 0.88 ± 0.12 | 0.50 ± 0.19 | 0.00 ± 0.00 |
| Sokoban_Jr_1_02 | 0.86 ± 0.12 | 0.30 ± 0.14 | 0.00 ± 0.00 |
| Sokoban_Jr_1_03 | 0.30 ± 0.13 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_04 | 0.39 ± 0.12 | 0.10 ± 0.08 | 0.00 ± 0.00 |
| Sokoban_Jr_1_05 | 0.33 ± 0.11 | 0.09 ± 0.09 | 0.00 ± 0.00 |
| Sokoban_Jr_1_06 | 0.54 ± 0.14 | 0.01 ± 0.01 | 0.00 ± 0.00 |
| Sokoban_Jr_1_07 | 0.65 ± 0.13 | 0.14 ± 0.08 | 0.00 ± 0.00 |
| Sokoban_Jr_1_08 | 0.18 ± 0.12 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_09 | 0.61 ± 0.13 | 0.23 ± 0.12 | 0.00 ± 0.00 |
| Sokoban_Jr_1_12 | 0.03 ± 0.02 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_15 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_16 | 0.81 ± 0.12 | 0.26 ± 0.13 | 0.00 ± 0.00 |
| Sokoban_Jr_1_17 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_18 | 0.19 ± 0.11 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_19 | 0.04 ± 0.03 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_20 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_21 | 0.88 ± 0.12 | 0.34 ± 0.16 | 0.00 ± 0.00 |
| Sokoban_Jr_1_22 | 0.16 ± 0.09 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_23 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Sokoban_Jr_1_24 | 0.04 ± 0.02 | 0.00 ± 0.00 | 0.00 ± 0.00 |
| Mean | 0.34 ± 0.06 | 0.10 ± 0.04 | 0.00 ± 0.00 |
Env-DR Ablation
To examine the effect of the dataset on DR in the true environment, we ran two additional experiments, where we sampled the DR levels from the set of levels used to generate the world model training data. We tested both when 100% of the levels were sampled from the UED levels and 50% were UED levels and 50% were random levels. Neither experiment improved over the 0% solve rate that the standard Env-DR run achieved. The very low training return is consistent with these levels being too difficult to provide an effective curriculum from initialisation. The mean return on the training levels of the agents (8 seeds), trained only on the UED levels, only reached 0.0042. To contextualise, for DEGen, the training level mean return was 0.45 at the end of training, and 0.55 for DyMGen.
These additional comparisons suggest that access to the UED level set alone is insufficient for effective learning from scratch, and highlight the value of an adaptive curriculum, as in methods like DyMGen.
W3
Thank you for this suggestion. We will add a summary table of mean evaluation solve rates to the main results section to make the quantitative comparison easier to assess.
Rebuttal (3/3)
Q3
Thank you for raising this point of measuring consistency, we have put together a quantitative analysis of performance with respect to consistency. To do this, we have mapped the world model outputs back to a simulator that predicts the expected behaviour of the underlying environment.
For each environment, we compare the full architecture with an ablation without noise mixing or the selector, under both random and trained-adversary generation actions.
We report four metrics:
- Disagreement cells: percentage of generated cells inconsistent with previous observations.
- Object-count violations: percentage of trajectories containing multiple goals, keys, or doors in Minigrid and Key Minigrid. This check imposes no box-count restriction in Sokoban.
- Done accuracy: accuracy of full trajectory done predictions (percentage of trajectories where every done prediction was correct)
- Reward error: mean of the sum of the reward absolute error, per trajectory
Note: we have updated this comment with more precise definitions of what the done accuracy and reward error are measuring. We have also provided additional per environment step results in our response below
We have not included solvability, as directly computing solvability from trajectories that may only have partial coverage of the full level, is somewhat infeasible, given it would require the contents of the unseen portion of the level to be inferred, and the method of how these contents were inferred would skew the results. Additionally, computing solvability of Sokoban levels is very much non-trivial.
Key Minigrid
| World model | Adversary | Disagreement cells | Object-count violations | Done accuracy | Reward error (MAE) |
|---|---|---|---|---|---|
| Full Architecture | Random | 0.1337% | 26.37% | 100.0000% | 0.001500 |
| Full Architecture | Trained | 1.9728% | 77.66% | 100.0000% | 0.004219 |
| Ablation | Random | 0.8974% | 93.14% | 100.0000% | 0.000912 |
| Ablation | Trained | 5.1305% | 99.98% | 99.9935% | 0.003565 |
Minigrid
| World model | Adversary | Disagreement cells | Object-count violations | Done accuracy | Reward error (MAE) |
|---|---|---|---|---|---|
| Full Architecture | Random | 0.0369% | 2.42% | 100.0000% | 0.001034 |
| Full Architecture | Trained | 2.9119% | 52.44% | 100.0000% | 0.002020 |
| Ablation | Random | 0.8368% | 69.29% | 100.0000% | 0.000359 |
| Ablation | Trained | 6.3650% | 99.32% | 100.0000% | 0.004415 |
Sokoban
| World model | Adversary | Disagreement cells | Object-count violations | Done accuracy | Reward error (MAE) |
|---|---|---|---|---|---|
| Full Architecture | Random | 0.0208% | 0.00% | 99.4899% | 0.004989 |
| Full Architecture | Trained | 0.0246% | 0.00% | 89.7953% | 0.094268 |
| Ablation | Random | 0.0307% | 0.00% | 100.0000% | 0.009314 |
| Ablation | Trained | 2.8784% | 0.00% | 1.0377% | 0.959258 |
From these results, we clearly see the benefit of the noise mixing and the selector, as the full architecture consistently outperforms the ablation. Under random latent generation actions, the percentage disagreement is extremely low, and whilst adversarial control does increase this, the full architecture reduces the effect of this. The likely reason for the Sokoban disagreement percentage under adversary control not showing such a significant increase compared to Minigrid and Key Minigrid is that the training levels for Sokoban are smaller than the Minigrid and Key Minigrid levels (based on [2]), and so maintaining consistency is easier for the world model.
Whilst these results do show there is room to improve the architecture, as the results show that the the adversary is able to reduce the consistency of the world model. However it is illustrated that our architecture design does clearly improve performance compared to the noise and selector free ablation. These results, whilst demonstrating there is need for future work, also highlight the importance of the contributions we have introduced in this work.
[2] Mead, Harry, et al. "Improving Regret Approximation for Unsupervised Dynamic Environment Generation." Advances in Neural Information Processing Systems 38 (2026): 168734-168771.
Rebuttal (1/3)
Thank you for your review, we appreciate you acknowledging the strengths of the work, and below we will address the issues and questions raised.
W1 and Q1
Thank you for raising this point, and allowing us to highlight the distinction between the use cases of DyMGen and DEGen. We agree that, when a suitable dynamically controllable simulator is available, DEGen is preferable in our experiments: it achieves higher solve rates and has lower computational cost, even before accounting for world-model pretraining. We explicitly acknowledge this in Section 7. The practical advantage of DyMGen is that it enables dynamic adversarial curriculum learning from a fixed offline dataset, without requiring access to such a simulator during curriculum and policy training, or a handcrafted interface for dynamically modifying the environment.
Our experiments evaluate this capability in domains with known simulators, allowing us to measure transfer to the true environment and compare against simulator-based training. However, we agree that this is a controlled evaluation of the offline-access setting, rather than a demonstration in a domain for which no simulator exists.
In terms of comparisons to other work that also assumes no simulator access, we are specifically interested in examining on-policy training (using PPO), entirely in synthetic world model trajectories. The vast majority of offline RL is instead focused on off-policy learning. The primary difference here, and the reason we have not included these offline RL baselines, is because unlike these methods, we are not directly extracting the policy from the offline dataset, instead only using the data to learn the dynamics of the world, with the policies we train being entirely tabula rasa. We have focused on this setting so we can explicitly show that teacher and student with no prior domain knowledge can generate a sufficient training curriculum, even in a world model.
However, these methods are not mutually exclusive. To show this, we compare our method with two of the most common offline RL, Behaviour Cloning (BC), and Implicit Q-Learning, whilst also showing the performance of a policy pretrained using behaviour cloning, and then fine-tuned in our world model.
Here we see relatively comparable performance between the offline RL methods and DyMGen, but significant improvements gained through world model fine-tuning. We note that in minigrid, random fine tuning outperforms DyMGen, this is likely due to the lack of level complexity possible in standard minigrid, and so the greater generation diversity of random fine tuning is more beneficial than the capacity for the DyMGen to generate more complex and challenging level configurations. However, we clearly see significant performance improvements with DyMGen fine tuning in both Sokoban and Key Minigrid, clearly showing the benefit of DyMGen when only having access to an offline dataset.