acceptodds
Under review as a conference paper at ICLR 2027

Controlling World Model Generation for Imagined Curriculum Learning

Abstract

Recent work has begun to explore performing Unsupervised Environment Design (UED) within learned world models, aiming to enable adaptive training curricula without requiring continued access to the true environment. However, existing approaches provide limited control over the trajectories generated by the world model. In this work, we propose an improved method of controllable world-model-based UED by introducing Dynamically-controlled Model Generation (DyMGen), a world-model architecture that learns latent generation actions through which an adversarial teacher can control world generation dynamically as a student explores the environment. This enables online adversarial environment generation without requiring a handcrafted simulator that explicitly supports this dynamic environment modification. We show that agents trained in adversarially controlled DyMGen worlds significantly outperform those trained with random generation, and approach the performance of UED agents trained with full access to the true environment. Our results demonstrate the potential for learning environment-design interfaces directly from offline experience, illustrating the potential for UED within learned, controllable world models.

Primary area:
reinforcement learning
Keywords:
Generalisation, Reinforcement Learning, Open Ended Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Koq1OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DyMGen, a controllable world model for unsupervised environment design. It learns latent generation actions from offline trajectories, allowing an adversarial teacher to dynamically control environment generation as the student explores. The authors introduce mechanisms to improve generation consistency and evaluate the approach on Key Minigrid, Minigrid, and Sokoban. DyMGen outperforms random world-model generation and PLR-based baselines, but remains below DEGen trained in the true environment.

Critical strengths

S1. Interesting approach to controllable world models.

The paper presents an interesting way to learn environment-generation controls directly from offline trajectories, enabling online adversarial curriculum learning without a handcrafted environment-generation interface.

S2. Promising empirical results.

DyMGen consistently outperforms the world-model baselines across three environments. The improvements are particularly clear in Key Minigrid and Sokoban, demonstrating the potential of adversarially controlled world models for curriculum learning.

Critical weaknesses

W1. Limited practical justification compared to DEGen.

The main motivation is to avoid requiring a handcrafted, dynamically controllable simulator. However, all experiments are conducted in environments where such simulators already exist. DyMGen is computationally more expensive and performs worse than DEGen. The paper does not demonstrate its advantages in the intended scenarios where access to a simulator is limited or unavailable. This weakens the practical significance of the contribution beyond DEGen.

W2. Potentially unfair comparison due to the Sokoban training data.

According to Appendix B.1, 25% of the Sokoban world-model training data comes from levels generated using true-environment UED, rather than domain randomization. This gives the world-model methods access to potentially stronger training data than Env-DR. Therefore, the improvement over Env-DR may be partially attributed to the training data rather than the proposed method. Although the comparison among world-model baselines remains valid, the paper should include a data-source ablation or a more carefully matched comparison to support its broader claims.

W3. Insufficient presentation of the main experimental results.

The main experimental results are not presented clearly enough. Figure 3 provides an overview of the training curves, but the small plots and overlapping curves make it difficult to compare the methods quantitatively. Important results, including the final solve rates reported in Tables 4–6, are only provided in the appendix. Since these results are central to evaluating the effectiveness of DyMGen, a summary table in the main paper would make the empirical contributions much easier to assess.

Questions for authors

Q1. Practical advantages over DEGen.

According to Figure 3 and Tables 4–6, DyMGen consistently underperforms DEGen across all three environments (74% vs. 83% on Key Minigrid, 86% vs. 96% on Minigrid, and 34% vs. 49% on Sokoban). Furthermore, Table 8 shows that DyMGen requires 15h14m per 10,000 PPO updates, compared to 8h18m for DEGen, even without accounting for the additional world-model training cost reported in Table 9. Given these results, what are the practical advantages of DyMGen over DEGen? Can the authors provide evidence of its benefits in settings where access to a simulator is limited or unavailable?

Q2. Fairness of the Sokoban comparison.

Appendix B.1 states that 25% of the Sokoban world-model training data comes from levels generated using true-environment UED, rather than domain randomization. Moreover, Appendix C.8 (Figure 17) shows that increasing the dataset size by 30× substantially improves DyMGen's performance. Given that Table 6 reports a 34% solve rate for DyMGen versus 0% for Env-DR, to what extent can this improvement be attributed to the proposed method rather than the stronger training data? Could the authors provide (1) an ablation varying the proportion of UED-generated trajectories while keeping the dataset size fixed, and (2) a comparison where the baselines have access to the same curated training data? In particular, how does DyMGen perform when its world model is trained exclusively on domain-randomized trajectories?

Q3. Validity of adversarially generated environments.

Section 7 acknowledges that the adversarial teacher sometimes generates trajectories inconsistent with the true environment, such as those containing multiple goals, keys, and doors. How frequently do such violations occur? Could the authors quantitatively evaluate the generated environments in terms of (1) validity under the environment constraints, (2) solvability, (3) consistency with previously observed states, and (4) correctness of predicted rewards and termination signals? It would also be helpful to compare these metrics between random and adversarial generation and examine how such inconsistencies affect student training.

8 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for your review, we appreciate you acknowledging the strengths of the work, and below we will address the issues and questions raised.

W1 and Q1

Thank you for raising this point, and allowing us to highlight the distinction between the use cases of DyMGen and DEGen. We agree that, when a suitable dynamically controllable simulator is available, DEGen is preferable in our experiments: it achieves higher solve rates and has lower computational cost, even before accounting for world-model pretraining. We explicitly acknowledge this in Section 7. The practical advantage of DyMGen is that it enables dynamic adversarial curriculum learning from a fixed offline dataset, without requiring access to such a simulator during curriculum and policy training, or a handcrafted interface for dynamically modifying the environment.

Our experiments evaluate this capability in domains with known simulators, allowing us to measure transfer to the true environment and compare against simulator-based training. However, we agree that this is a controlled evaluation of the offline-access setting, rather than a demonstration in a domain for which no simulator exists.

In terms of comparisons to other work that also assumes no simulator access, we are specifically interested in examining on-policy training (using PPO), entirely in synthetic world model trajectories. The vast majority of offline RL is instead focused on off-policy learning. The primary difference here, and the reason we have not included these offline RL baselines, is because unlike these methods, we are not directly extracting the policy from the offline dataset, instead only using the data to learn the dynamics of the world, with the policies we train being entirely tabula rasa. We have focused on this setting so we can explicitly show that teacher and student with no prior domain knowledge can generate a sufficient training curriculum, even in a world model.

However, these methods are not mutually exclusive. To show this, we compare our method with two of the most common offline RL, Behaviour Cloning (BC), and Implicit Q-Learning, whilst also showing the performance of a policy pretrained using behaviour cloning, and then fine-tuned in our world model.

Environment (Mean Solve Rate)WM - DyMGenWM - RandomWM - DyMGen - FTWM - Random - FTBCIQL
Sokoban0.34 ± 0.060.00 ± 0.000.49 ± 0.010.40 ± 0.010.33 ± 0.010.35 ± 0.01
MiniGrid0.86 ± 0.050.72 ± 0.030.85 ± 0.040.95 ± 0.020.76 ± 0.020.79 ± 0.01
Key MiniGrid0.74 ± 0.040.21 ± 0.050.88 ± 0.030.73 ± 0.030.73 ± 0.010.78 ± 0.02

Here we see relatively comparable performance between the offline RL methods and DyMGen, but significant improvements gained through world model fine-tuning. We note that in minigrid, random fine tuning outperforms DyMGen, this is likely due to the lack of level complexity possible in standard minigrid, and so the greater generation diversity of random fine tuning is more beneficial than the capacity for the DyMGen to generate more complex and challenging level configurations. However, we clearly see significant performance improvements with DyMGen fine tuning in both Sokoban and Key Minigrid, clearly showing the benefit of DyMGen when only having access to an offline dataset.

AuthorsOpenReview ↗

Rebuttal (2/3)

W2 and Q2

Thank you for identifying this point. We agree that the comparison with standard Env-DR alone cannot isolate the contribution of DyMGen from the influence of dataset composition.

To address this, we ran two additional sets of experiments. The first was to ablate the percentage of UED generated trajectories, and the second was examine the performance of DR, if the levels were randomly sampled from a set of UED-generated levels rather than randomly generated ones.

Dataset Ablation

Below we include a table that shows the performance of Sokoban DyMGen when trained with a dataset containing 25% (what is shown in the paper), 10% and 0% UED-generated levels. It is clear that these levels are necessary, as solvable randomly generated sokoban levels are so exceedingly uncommon, and so learning environment dynamics, specifically learning when levels are complete, becomes increasing more difficult with less examples of known solvable levels.

Level25% UED10% UED0 UED
Sokoban_Jr_1_010.88 ± 0.120.50 ± 0.190.00 ± 0.00
Sokoban_Jr_1_020.86 ± 0.120.30 ± 0.140.00 ± 0.00
Sokoban_Jr_1_030.30 ± 0.130.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_040.39 ± 0.120.10 ± 0.080.00 ± 0.00
Sokoban_Jr_1_050.33 ± 0.110.09 ± 0.090.00 ± 0.00
Sokoban_Jr_1_060.54 ± 0.140.01 ± 0.010.00 ± 0.00
Sokoban_Jr_1_070.65 ± 0.130.14 ± 0.080.00 ± 0.00
Sokoban_Jr_1_080.18 ± 0.120.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_090.61 ± 0.130.23 ± 0.120.00 ± 0.00
Sokoban_Jr_1_120.03 ± 0.020.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_150.00 ± 0.000.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_160.81 ± 0.120.26 ± 0.130.00 ± 0.00
Sokoban_Jr_1_170.00 ± 0.000.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_180.19 ± 0.110.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_190.04 ± 0.030.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_200.00 ± 0.000.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_210.88 ± 0.120.34 ± 0.160.00 ± 0.00
Sokoban_Jr_1_220.16 ± 0.090.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_230.00 ± 0.000.00 ± 0.000.00 ± 0.00
Sokoban_Jr_1_240.04 ± 0.020.00 ± 0.000.00 ± 0.00
Mean0.34 ± 0.060.10 ± 0.040.00 ± 0.00

Env-DR Ablation

To examine the effect of the dataset on DR in the true environment, we ran two additional experiments, where we sampled the DR levels from the set of levels used to generate the world model training data. We tested both when 100% of the levels were sampled from the UED levels and 50% were UED levels and 50% were random levels. Neither experiment improved over the 0% solve rate that the standard Env-DR run achieved. The very low training return is consistent with these levels being too difficult to provide an effective curriculum from initialisation. The mean return on the training levels of the agents (8 seeds), trained only on the UED levels, only reached 0.0042. To contextualise, for DEGen, the training level mean return was 0.45 at the end of training, and 0.55 for DyMGen.

These additional comparisons suggest that access to the UED level set alone is insufficient for effective learning from scratch, and highlight the value of an adaptive curriculum, as in methods like DyMGen.

W3

Thank you for this suggestion. We will add a summary table of mean evaluation solve rates to the main results section to make the quantitative comparison easier to assess.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q3

Thank you for raising this point of measuring consistency, we have put together a quantitative analysis of performance with respect to consistency. To do this, we have mapped the world model outputs back to a simulator that predicts the expected behaviour of the underlying environment.

For each environment, we compare the full architecture with an ablation without noise mixing or the selector, under both random and trained-adversary generation actions.

We report four metrics:

  1. Disagreement cells: percentage of generated cells inconsistent with previous observations.
  2. Object-count violations: percentage of trajectories containing multiple goals, keys, or doors in Minigrid and Key Minigrid. This check imposes no box-count restriction in Sokoban.
  3. Done accuracy: accuracy of full trajectory done predictions (percentage of trajectories where every done prediction was correct)
  4. Reward error: mean of the sum of the reward absolute error, per trajectory

Note: we have updated this comment with more precise definitions of what the done accuracy and reward error are measuring. We have also provided additional per environment step results in our response below

We have not included solvability, as directly computing solvability from trajectories that may only have partial coverage of the full level, is somewhat infeasible, given it would require the contents of the unseen portion of the level to be inferred, and the method of how these contents were inferred would skew the results. Additionally, computing solvability of Sokoban levels is very much non-trivial.

Key Minigrid

World modelAdversaryDisagreement cellsObject-count violationsDone accuracyReward error (MAE)
Full ArchitectureRandom0.1337%26.37%100.0000%0.001500
Full ArchitectureTrained1.9728%77.66%100.0000%0.004219
AblationRandom0.8974%93.14%100.0000%0.000912
AblationTrained5.1305%99.98%99.9935%0.003565

Minigrid

World modelAdversaryDisagreement cellsObject-count violationsDone accuracyReward error (MAE)
Full ArchitectureRandom0.0369%2.42%100.0000%0.001034
Full ArchitectureTrained2.9119%52.44%100.0000%0.002020
AblationRandom0.8368%69.29%100.0000%0.000359
AblationTrained6.3650%99.32%100.0000%0.004415

Sokoban

World modelAdversaryDisagreement cellsObject-count violationsDone accuracyReward error (MAE)
Full ArchitectureRandom0.0208%0.00%99.4899%0.004989
Full ArchitectureTrained0.0246%0.00%89.7953%0.094268
AblationRandom0.0307%0.00%100.0000%0.009314
AblationTrained2.8784%0.00%1.0377%0.959258

From these results, we clearly see the benefit of the noise mixing and the selector, as the full architecture consistently outperforms the ablation. Under random latent generation actions, the percentage disagreement is extremely low, and whilst adversarial control does increase this, the full architecture reduces the effect of this. The likely reason for the Sokoban disagreement percentage under adversary control not showing such a significant increase compared to Minigrid and Key Minigrid is that the training levels for Sokoban are smaller than the Minigrid and Key Minigrid levels (based on [2]), and so maintaining consistency is easier for the world model.

Whilst these results do show there is room to improve the architecture, as the results show that the the adversary is able to reduce the consistency of the world model. However it is illustrated that our architecture design does clearly improve performance compared to the noise and selector free ablation. These results, whilst demonstrating there is need for future work, also highlight the importance of the contributions we have introduced in this work.

[2] Mead, Harry, et al. "Improving Regret Approximation for Unsupervised Dynamic Environment Generation." Advances in Neural Information Processing Systems 38 (2026): 168734-168771.

Reviewer_hRpfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work aims to develop a world model for training RL agents by automatically generating curricula. An adversarial teacher controls the generation with learned latent generation actions that maximize regret of the student (RL agent). Thus, training samples are generated where the agent is maximally challenged while the task is still solvable. In training the teacher and students are trained together and in inference the teacher influences the generation live. The method also includes a mechanism aiming to preserve consistency of already generated/provided regions. The resulting pipeline is then used to generate training curricula with the potential benefit of not requiring a handcrafted environment simulator.

Critical strengths
  • The goal of this work to produce an adversarial world model generating curricula for training RL agents is meaningful and important.
  • The method to this end is novel and involves influencing the generation with actions which I find particularly interesting
  • The method provides a mechanism for preserving consistency by deciding when to keep or edit an observation.
Critical weaknesses
  • Contribution - The work does not sufficiently experimentally demonstrate its benefits and contribution practical significance. Compared to DEGen, DyMGen underperforms and incurs extra costs. A lower performance in some respect compared to a baseline is not a problem by itself and limitations are clearly established, which is commendable. However, a benefit in some aspect has to be established to demonstrate the significance of the contribution. For example, an experiment demonstrating DyMGen in a setting where a handcrafted simulator is not available would make the method more compelling. Another potential advantage for the policy could be the fact that gradient propagation through the entire system is possible. Currently the experiment section is rather limited which makes it hard to understand if, when and why the models works.
  • Positioning to existing work - "RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning" - It seems like this work presents a very close idea - they improve offline RL with an adversarial learned environment model. "We formulate the problem as a two-player zero sum game against an adversarial environment model." The difference I see seems to be that RAMBO is performing turns - train the world model, then the policy, then train and generate adversarial data from the world model, improve the policy - and form a loop. In this work this is done jointly per step, which is a major difference. It is unclear however, if this is a positive significant change compared to the RAMBO method. As this is a closely related work, this authors may consider to use it as a baseline and position themselves according to it.
  • Evaluation soundness - I have concerns over the metric used in Table 1 - As it measures categorical error in latent space the effect of this error or the significance of the improvement on actual observation space is unclear. Different latent errors may translate differently in scale and quality after decoding - and retraining may change the latent space altogether, changing the latent metrics. Could the authors supplement those metrics with evaluation in decoded observations? For example, cell content accuracy for evaluating consistency.
  • Effect of editing - The effect of editing the observations is not clear. While the available qualitative samples show editing with random sampling, it has to be shown what the original unedited generation would look like to establish the value of the edit itself.
  • Consistency not preserved - The authors develop a method for preserving consistency. The only visual examples shown of generation action rollout are in Ap.. C.4. The authors claim "We can also clearly see it is only the unobserved parts of the trajectory are controlled" However Fig.8 clearly shows inconsistencies with the history throughout the rollout (e.g. t=4, t=5 on the first row but also others).
  • Limitations - Apart from the mentioned limitations DyMGen seems to feed the entire history rather than having an autoregressive design. This ensures the generation consistency with previous time steps, wthich seems to be crucial for their world model output selector training. However, it also introduces a new limitation of requiring the full history, which does not seem to be the case with previous work.
Questions for authors
  • How do you position yourself among the existing work? As DEGen outperforms this method in the evaluation, is there an aspect where the proposed method brings any benefit? How does your work compare with RAMBO-RL and related methods and what is the benefit in comparison?
  • What is the practical significance of the evaluation results? How do you relate the latent space results to a more meaningful anchored representation (e.g. image space)?
  • What is the effect of editing? Can you demonstrate visually strategies that the teacher develops to generate curricula?
  • How well is consistency preserved? What are the factors for the quality of consistency preservation?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for taking the time to review our work, and we appreciate the feedback that has been provided. Below we provide a response to each of the raised issues.

W1 and Q1

The core contribution of this work is it represents the first, as far as we are aware, demonstration that dynamic control of world model generation can enable significant improvement of policies trained within these world models. When we refer to dynamic generation, we are referring to our model's capacity to allow the adversary to generate the environment as it is explored by the agent. The most directly comparable work to ours [1], shown with the WM - PLR baseline, only allows control of the initial state, and we clearly show this is insufficient in partially observable settings. In Sokoban, we show DyMGen is necessary to get any performance out of a policy trained in a world model.

In regard to positioning this work, we are specifically interested in examining on-policy training (using PPO), entirely in synthetic world model trajectories. The vast majority of offline RL is instead focused on off-policy learning. The primary difference here, and the reason we have not included offline RL baselines, is because unlike these methods, we are not directly extracting the policy from the offline dataset, instead only using the data to learn the dynamics of the world, with the policies we train being entirely tabula rasa. We have focused on this setting so we can explicitly show that teacher and student with no prior domain knowledge can generate a sufficient training curriculum, even in a world model.

However, these methods are not mutually exclusive. To show this, we compare our method with two of the most common offline RL, Behaviour Cloning (BC), and Implicit Q-Learning, whilst also showing the performance of a policy pretrained using behaviour cloning, and then fine-tuned in our world model.

Below we include a table showing the mean solve rate of these methods in each of the three environments.

Environment (Mean Solve Rate)WM - DyMGenWM - RandomWM - DyMGen - FTWM - Random - FTBCIQL
Sokoban0.34 ± 0.060.00 ± 0.000.49 ± 0.010.40 ± 0.010.33 ± 0.010.35 ± 0.01
MiniGrid0.86 ± 0.050.72 ± 0.030.85 ± 0.040.95 ± 0.020.76 ± 0.020.79 ± 0.01
Key MiniGrid0.74 ± 0.040.21 ± 0.050.88 ± 0.030.73 ± 0.030.73 ± 0.010.78 ± 0.02

Here we see relatively comparable performance between the offline RL methods and DyMGen, but significant improvements gained through world model fine-tuning. We note that in minigrid, random fine tuning outperforms DyMGen, this is likely due to the lack of level complexity possible in standard minigrid, and so the greater generation diversity of random fine tuning is more beneficial than the capacity for the DyMGen to generate more complex and challenging level configurations. However, we clearly see significant performance improvements with DyMGen fine tuning in both Sokoban and Key Minigrid, clearly showing the benefit of DyMGen when only having access to an offline dataset.

In regards to direct comparisons between DyMGen and DEGen, we note these methods are not directly comparable. We would be comparing a method that has access to the environment (DEGen), to DyMGen, which only has access to an offline dataset. We clearly demonstrate the benefit of DyMGen in scenarios where there is no access to a simulator. However, we strongly feel that the inclusion of the DEGen baseline enhances the paper, as we feel it's incredibly important to contextualise the performance gains from DyMGen. The results show that the relative performance gain of DyMGen is generally comparable to DEGen when comparing random and curriculum-based methods in both the true environment and the world models respectively.

We agree that, when a suitable dynamically controllable simulator is available, DEGen is preferable in our experiments because it achieves higher solve rates at lower computational cost. DyMGen instead enables dynamic curriculum learning from a fixed offline dataset without simulator access during curriculum and policy training. Our benchmarks provide a controlled evaluation of this access constraint.

[1] Güzel, A.H., Jackson, M., Liesen, J., Rocktäschel, T., Foerster, J., Bogunovic, I. and Parker-Holder, J., 2026. Imagined autocurricula. Advances in Neural Information Processing Systems, 38, pp.114787-114808.

AuthorsOpenReview ↗

Rebuttal (2/3)

W2 and Q1

Thank you for highlighting RAMBO as closely related work. Both approaches use adversarial learned models, but differ in what the adversary controls and its objective. RAMBO updates the world-model parameters to minimise policy return. DyMGen keeps the pretrained world model fixed and trains an adversary to select latent generation actions that maximise approximate regret, using MNA. This lets the teacher adapt the curriculum as the student explores, while retaining the pretrained dynamics model.

Our focus is on learning from complete synthetic trajectories with dynamic environment generation. Updating a lightweight adversary avoids repeatedly optimising the larger world model during student training. RAMBO instead uses its world model only to rollout short (approximately 3 to 5 step) synthetic trajectories.

There are two problems with directly updating the world model parameters in our setting. The first is practical - during student training, DyMGen only is updating a relatively lightweight adversary, directly updating the much larger underlying world model would significantly slow training. The second problem is that we strongly hypothesise that a RAMBO-like implementation when rolling out full trajectories would maintain poor consistency over the rollout, as you are relying on the soft constraint of also retraining on the training data to maintain consistency. We show with Equations 11 through 13 that, at optimality, a world model trained using our method will not produce infeasible observations. However, even at optimality, there are no guarantees that optimising the loss function described by Equation 9 in the RAMBO paper will ensure feasible observations.

W3 and Q2

Thank you for pointing this out, it could definitely have been possible latent accuracy not to align with actual image accuracy. However, below we include same table, but showing output cell accuracy, rather than latent accuracy, and we see that the same patterns are observed.

MetricUnconditionedConditioned (ε = 0.0)Conditioned (ε = 0.2)Conditioned (ε = 0.5)Selector (ε = 0.2)
True-Action Error Rate (%)0.670.060.090.240.10
Post-Exploration Error Rate (%)0.090.810.170.120.11

W4 and Q3

We have added example rollouts for each world model in Appendix C.5. The Key Minigrid example illustrates a goal behind a locked door, while the Sokoban example illustrates box-pushing structures generated during exploration. These examples help show the structures generated by the teacher that facilitate the improved solve-rate of the final policies on the evaluation levels.

With regard to the question on the effect of editing - ultimately the unconditioned world model rollouts would just visually appear as a different configuration of cells generated. The purpose of Figures 8, 9 and 10 is to illustrate that, under identical conditions, but with different randomly generated latent generation actions, the world model will output two different observations if there are parts of the observation that had not previously been observed. We hope the rollouts shown in Appendix C.5 sufficiently show that the adversary is able to generate more useful training trajectories than random generation, but if we can provide further clarity here, please follow up with additional questions.

AuthorsOpenReview ↗

Rebuttal (3/3)

W5 and Q4

Thank you for raising this point of measuring consistency, we have put together a quantitative analysis of performance with respect to consistency. To do this, we have mapped the world model outputs back to a simulator that predicts the expected behaviour of the underlying environment.

We would also briefly note that, as you can see from the tables below, the world models are not perfectly consistent - however, the examples provided by the reviewer (e.g. t=4, t=5 on the first row of Figure 8 Ap. C.4) are not observations inconsistent with the history. If you would like to outline exactly the why you think these specific examples are inconsistent, we can provide clarification - but in this case they are not.

For each environment, we compare the full architecture with an ablation without noise mixing or the selector, under both random and trained-adversary generation actions.

We report four metrics:

  1. Disagreement cells: percentage of generated cells inconsistent with previous observations.
  2. Object-count violations: percentage of trajectories containing multiple goals, keys, or doors in Minigrid and Key Minigrid. This check imposes no box-count restriction in Sokoban.
  3. Done accuracy: percentage of trajectories where every done prediction was correct
  4. Reward error: mean of the sum of the reward absolute error, per trajectory

Key Minigrid

World modelAdversaryDisagreement cellsObject-count violationsDone accuracyReward error (MAE)
Full ArchitectureRandom0.1337%26.37%100.0000%0.001500
Full ArchitectureTrained1.9728%77.66%100.0000%0.004219
AblationRandom0.8974%93.14%100.0000%0.000912
AblationTrained5.1305%99.98%99.9935%0.003565

Minigrid

World modelAdversaryDisagreement cellsObject-count violationsDone accuracyReward error (MAE)
Full ArchitectureRandom0.0369%2.42%100.0000%0.001034
Full ArchitectureTrained2.9119%52.44%100.0000%0.002020
AblationRandom0.8368%69.29%100.0000%0.000359
AblationTrained6.3650%99.32%100.0000%0.004415

Sokoban

World modelAdversaryDisagreement cellsObject-count violationsDone accuracyReward error (MAE)
Full ArchitectureRandom0.0208%0.00%99.4899%0.004989
Full ArchitectureTrained0.0246%0.00%89.7953%0.094268
AblationRandom0.0307%0.00%100.0000%0.009314
AblationTrained2.8784%0.00%1.0377%0.959258

From these results, we clearly see the benefit of the noise mixing and the selector, as the full architecture consistently outperforms the ablation. Under random latent generation actions, the percentage disagreement is extremely low, and whilst adversarial control does increase this, the full architecture reduces the effect of this. The likely reason for the Sokoban disagreement percentage under adversary control not showing such a significant increase compared to Minigrid and Key Minigrid is that the training levels for Sokoban are smaller than the Minigrid and Key Minigrid levels (based on [2]), and so maintaining consistency is easier for the world model.

Whilst these results do show there is room to improve the architecture, as the results show that the the adversary is able to reduce the consistency of the world model. However it is illustrated that our architecture design does clearly improve performance compared to the noise and selector free ablation. These results, whilst demonstrating there is need for future work, also highlight the importance of the contributions we have introduced in this work.

[2] Mead, Harry, et al. "Improving Regret Approximation for Unsupervised Dynamic Environment Generation." Advances in Neural Information Processing Systems 38 (2026): 168734-168771.

W6

To provide further clarity on the architectural differences between our work and previous work, we can clarify for both the world model, as well as the student/adversary agents.

For the world model, for history conditioning, we use a standard transformer design that is widely used in existing work [3]. With KV-caching, transformer inference does represent an autoregressive architecture, as only the most recent token is passed into the model, and cached K and V values can be used for attention. Whilst we do notate this as ht=fψ(z0:t,a0:t)h_t = f_\psi(z_{0:t}, a_{0:t}), this is only to represent that hth_t is conditioned on the full history, rather than practically using the full history as an input.

With regard to the adversary and student, the RNN architecture we are using is identical to previous work, with an LSTM used to condition on history.

[3] Robine, Jan, et al. "Transformer-based world models are happy with 100k interactions." arXiv preprint arXiv:2303.07109 (2023).

Reviewer_oykrOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Dynamically-controlled Model Generation (DyMGen). Specifically, a world model generates latent actions learned from offline trajectories while maintaining consistency with trajectory history. Moreover, the online adversarial environment generation facilitates task learning for students during exploration. The experiments show that DyMGen outperforms baselines trained with random generation or full access to the true environment.

Critical strengths

Beyond merely predicting the scenes following a student action, this paper provides a learned control interface for adaptive curriculum generation, where an adversarial teacher can generate future dynamically as a student explores the environment.

Critical weaknesses

The impact of generation consistency on student learning. The paper designs generation action noise and output selection to improve consistency. Could the authors provide ablations on student solve rate with and without these designs?

Questions for authors

See Weaknesses

1 comment

Thank you for suggesting this ablation. We evaluated the contribution of noise mixing and the selector to student solve rate in Key Minigrid.

Removing the selector or noise mixing reduces mean solve rate from 0.74 to 0.63 and 0.61, respectively, while removing both reduces it to 0.14. These results highlight the clear benefit of both noise mixing and the selector in policy training.

LevelDyMGenNo Selector0.0 Noise Mixing + No Selector0.0 Noise Mixing
FourRooms2_Key0.89 ± 0.100.91 ± 0.050.15 ± 0.070.89 ± 0.06
FourRooms3_Key0.75 ± 0.120.90 ± 0.050.03 ± 0.030.78 ± 0.06
FourRooms_Key0.83 ± 0.100.96 ± 0.020.26 ± 0.120.75 ± 0.10
SixteenRooms2_Key0.69 ± 0.110.48 ± 0.120.23 ± 0.080.64 ± 0.11
SixteenRooms_Key0.99 ± 0.010.91 ± 0.050.25 ± 0.060.70 ± 0.11
StandardMaze2_Key0.54 ± 0.110.26 ± 0.120.08 ± 0.040.20 ± 0.11
StandardMaze3_Key0.84 ± 0.100.44 ± 0.110.10 ± 0.040.65 ± 0.14
StandardMaze_Key0.41 ± 0.100.19 ± 0.130.03 ± 0.030.25 ± 0.11
Mean0.74 ± 0.040.63 ± 0.040.14 ± 0.030.61 ± 0.05

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.