acceptodds
Under review as a conference paper at ICLR 2027

Recursively Self-Improving World Models for Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) achieves strong data efficiency by leveraging synthetic experience for policy optimization; however, training a world model still requires substantial real interaction data, not only for accurate predictions but also for adequate coverage of real transitions. In this work, we investigate whether a world model can improve itself through its own generated experience to better support policy learning. We show that self-training amplifies concentration tendencies in generated experience, arising from mode-seeking sampling in unconditional generators and coupled model-policy interaction in conditional dynamics models. Reversing the resulting parameter displacement through negative extrapolation steers the world model toward generating more diverse and useful synthetic experience for downstream policy learning. Across both unconditional and conditional world-model formulations, our approach improves policy performance and data efficiency over the respective backbones on MuJoCo, DeepMind Control Suite, and Meta-World tasks with modest additional computation.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, RL, world models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eeUHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a simple RSI principle for world models. The authors show that self-training worsens a trend toward reduced diversity in the generated samples. This effect comes from mode-seeking sampling in unconditional generators and from model–policy coupling in conditional dynamics models. To reduce the resulting parameter shift, they apply negative extrapolation, which pushes the world model to produce synthetic experience that are diverse and more useful for training downstream policies.

Critical strengths
  1. Exporing RSI for world model sounds like an interesting idea, the author provide algorithms for both unconditional generative replay and for conditional latent dynamics.

  2. The writing is easy to follow and the ideas are explained very clearly.

Critical weaknesses
  1. In S^2ER, the author used tFID as a selection metric. However, it is widely known that FID can be easily hacked [1] [2]. First and second order moment can not guarantee robust representation alignment for the two distribution compared. Besides, compute FID is costly. It is a bit confued to me why S2ER increases total GPU-hours by only 1.7% over SYNTHER. I suggest the author to provide some reasoning for why they used tFID as selection metric, and why this aligns with the downstream control performance.

  2. The authors argued that the method is data efficient, but did not compare with some other data efficient world models like [3] and [4]. Any reason for this?

[1] Jayasumana, Sadeep, et al. "Rethinking fid: Towards a better evaluation metric for image generation." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Yang, Jiawei, et al. "Representation Fr'echet Loss for Visual Generation." arXiv preprint arXiv:2604.28190 (2026).

[3] Sonigra, Romil V., and P. R. Kumar. "Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamics." arXiv preprint arXiv:2605.23089 (2026).

[4] Aghabozorgi, Mehran, et al. "WIMLE: Uncertainty-aware world models with IMLE for sample-efficient continuous control." arXiv preprint arXiv:2602.14351 (2026).

Questions for authors

As authors mentioned, 'Recursive self-training further amplifies the concentration'. Will use methods like TRPO together with the proposed algorithms help with this issue? I wonder if this can be a replacement for degradation threshold. The authors can either provide some theoretical or experical analysis.

Reviewer_iXxbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether a world model used for model-based reinforcement learning can iteratively improve itself using its own generated experience, rather than relying solely on additional real interaction data. The authors observe that model-generated experience tends to be concentrated along the distribution modes, and that recursively training on this synthetic data amplifies this concentration. Building on recent work in image generation, they propose to treat the parameter displacement induced by self-training as an informative direction, and to reverse it through negative extrapolation, steering the world model toward more diverse synthetic experience. The idea is instantiated in two complementary settings for unconditional generative models (S^2ER) and conditional latent dynamics models (ReDream), both with modest computational overhead. The stated claims are supported by experiments on MuJoCo, the DeepMind Control Suite, and Meta-World, where both methods improve policy performance over baselines.

Critical strengths
  1. This paper asks a very interesting and timely question, which connects recent literature on self-improvement for generative models to model-based reinforcement learning. Treating unconditional generative replay and conditional latent dynamics within a single unified framework and applying the same principle to both is a conceptually appealing and practical contribution. This also generalizes the contribution beyond a single method or architecture.
  2. Both S2ERS^2ER and ReDream are simple methods which are easy to add to existing pipelines. They introduce a small number of new hyperparameters to be tuned, and add little computation (the GPU hour increase is only 1.7% for S2ERS^2ER and 5.6% for ReDream). The diagnostic experiments given in the paper are well-designed and informative. The precision/recall/tFID sweep along the self-training and extrapolation directions (Figure 3) clearly illustrates the coverage tradeoff. Comparing ReDream’s negative extrapolation against positive extrapolation and random perturbation (Figure 6) is a good way to test whether the specific direction matters.
Critical weaknesses

. It is unclear whether the anti-alignment argument proposed in NEON should hold in this setting. In NEON, the anti-alignment argument relies on a distributional mismatch between the generative model and the chosen sampler. In ReDream, according to Eq. 5 and the description of Ddream\mathcal{D}_{\text{dream}}, the dynamics term of Eq. 6 performs self-training on samples from the model’s own conditional distribution. This should result in zero expected gradient at the reference parameters for any input distribution, since Ey∼pϕ[∇ϕlog⁡pϕ(y∣x)]=0E_{y \sim p_\phi}[\nabla_\phi \log p_\phi(y|x)] = 0. The policy-induced state distribution therefore cannot produce a consistent parameter displacement through the expected gradient alone, which is required for NEON’s anti-alignment argument.

a. The results given in Figure 6 suggest that the proposed method actually does yield a useful parameter displacement. If the displacement were truly zero-mean noise, then positive and negative extrapolation would be statistically equivalent, but they have different behavior. The source of this displacement is therefore unexplained. This may be because the targets are point predictions rather than samples (Appendix D).

  1. The empirical gains of S2ERS^2ER are task-specific and cannot concretely be attributed to negative extrapolation. At each refresh step, S2ERS^2ER evaluates candidates and selects the one with the lowest tFID against held-out real transitions. This selection step is an optimization against real data. It is therefore difficult to distinguish whether the cause of the observed gains is truly the negative extrapolation or simply the grid search over many candidates compared with real data (or a combination of the two).

    a. S2ERS^2ER’s improvements over SynthER are clear on Walker2d and HalfCheetah, but small on Hopper and Cheetah-Run, and PGR performs better on Reacher-Hard. Since the method is only tested on six tasks, the claim that S2ERS^2ER consistently improves performance would be strengthened by adding significance tests or testing on additional tasks.

    b. The novelty in the unconditional generation setting is also limited. S2ERS^2ER applies NEON’s update (Eq. 2) to the SynthER generator, using the same argument given in the original NEON paper about mode-seeking samplers causing concentration. It seems that the main additions are the real-loss gate and the tFID-based grid search. These are reasonable engineering choices, but the paper should state clearly that the unconditional contribution is mainly an application of NEON to generative replay.

Questions for authors
  1. The ReDream schedule was tuned on Sweep, which is one of the evaluation tasks, and the paper reports the median and mean success rates across 12 tasks to show improvement. Could the authors evaluate on held-out tasks that were not used for hyperparameter selection?
  2. Could the authors perform additional ablation experiments on S2ERS^2ER to clearly determine the cause of the observed gains in the unconditional generation experiments? An example would be to replace negative extrapolation with random perturbations of the same size. With the same number of candidates, choose the one with the lowest tFID against the held-out true data. This is similar to Figure 6, but for S2ERS^2ER with the selection step.
  3. Can the authors explain why TD-MPC2 achieves a near zero success rate on several tasks in Figure 15? How were the TD-MPC2 experiments configured for Meta-World?
Reviewer_td4HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper applies negative extrapolation to world models: briefly train a model on its own samples, then move its parameters in the opposite direction. S^2ER uses this idea to improve synthetic replay, with a real-data loss constraint and a distributional score to select the update. REDREAM applies it to latent dynamics. Experiments cover six MuJoCo/DMC tasks and 12 Meta-World tasks, with further analyses of coverage, update direction, and computational cost.

Critical strengths
  • The results suggest a useful way to improve data efficiency without collecting more real experience. S^2ER improves over SynthER on all six tasks in Table 1. REDREAM improves the reported aggregate Meta-World performance, including median success from 0.53 to 0.80. Gains in both settings, with the reported small computational overhead, make the approach practically interesting.

  • The algorithms make the update easier to follow, and the analyses help explain what it changes. Figure 3 shows the trade-off between coverage and precision, and Figure 6 compares negative extrapolation with positive updates and random perturbations of similar size. These controls give useful evidence that the update direction matters.

Critical weaknesses
  • I couldn't reconcile the stated DreamerV3 backbone with the supplementary code. The authors say that the other training settings are identical. The supplied code/redream/dreamer.py uses a projector and a Barlow-style representation loss, with no observation decoder -- and the observation decoder was identified as being critical in dreamer. Disabling REDREAM retains that backbone, so a matched comparison is possible. Which implementation produced the reported main comparison and the decoder-based ablation in Appendix C? Without this mapping, I can't see precisely what was compared.

Lines 289–293 give a reason for omitting reconstruction during self-training and point to Appendix C. I would like that the authors explain there which losses remain in ordinary model training and summarize the scope of the Box-Close ablation, so readers can understand why this extra update can omit a component used to learn the representation.

  • I am not yet convinced by the explanation for concentration in the conditional setting. For a fixed input, fitting targets sampled exactly from the current model gives zero expected likelihood gradient at the starting parameters. Policy conditioning alone does not explain a preferred direction. Finite samples and target construction could matter: in the supplementary material, redream.py uses the predicted mean reward, whereas Eq. (5) describes sampling. Which of these choices produces the useful direction? Figure 5's extended self-training experiment does not by itself identify the cause of the single-step online correction.

The opening also needs a concrete explanation before the mechanism claim at lines 43–47. What exactly is the generated “experience,” what is fitted to it, and why should reversing that fit help? Lines 125–138 distinguish transition sampling from conditional rollouts and introduce GϕG_\phi; Section 4.3 later introduces latent dynamics. A short example and an explicit connection between these models would help at first introduction. In Section 4.3, please distinguish what carries over from the earlier analysis from what is an empirical hypothesis about latent models. The later algorithms help, but do not remove this difficulty on a first reading.

Questions for authors
  1. Do REDREAM and its baseline use exactly the same representation loss and architecture in Figures 2 and 15? Please identify the backbone and configurations used for these results and the decoder-bearing implementation used for Appendix C.

  2. Table 1 does not specify the observations. The supplied S^2ER code uses state vectors; did every DMC baseline receive the same inputs?

  3. In Eqs. (5)–(6), what produces a systematic self-training direction when targets are drawn from the current model? Were rewards sampled or replaced by their predicted means? Please specify the self-training optimizer and step count for the main results and Figure 5, and explain what supports attributing the drift to policy coupling.

Comments on the paper

1 comment
Area_Chair_ZKHyOpenReview ↗

Reviewer-Author Discussion Phase is Ending

Dear Reviewers,

Thank you for your service in the review process. Now that we are approaching the deadline of the author-reviewer discussion period. Please take time to read authors' responses and let them know if it addresses your concerns. Thank you.

Best,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.