acceptodds
Under review as a conference paper at ICLR 2027

Sample Efficient World Models using Gradient Penalized Latent Dynamics

Abstract

World models are trained on one-step transitions and then rolled out for dozens of steps to train a policy. Where those transitions are sparse, the learned transition function can change rapidly between neighboring states, and long imagined rollouts compound the resulting errors. In continuous control, neighboring states evolve under similar transition laws, so a transition observed at one state is informative about its neighbors. To exploit all such information, we propose GPLD (Gradient Penalized Latent Dynamics), a method that makes possible the sharing of this information between neighboring states. We derive it as the small-neighborhood limit of a finite-difference penalty on transition laws in a finite MDP, which yields the squared Frobenius norm of the dynamics Jacobian. We apply it to DreamerV3’s posterior latent distribution, the map through which observations shape the learned dynamics, and estimate it with Hutchinson-style stochastic probes. With a single configuration held fixed across every task, GPLD raises final return over DreamerV3 on 15 of 18 proprioceptive DeepMind Control tasks (sign test p = 0.0075) and mean final return by 9.8% on the 12 tasks where DreamerV3 has not already saturated (18.1% in normalized return), and by 20.7% on the six humanoid and dog tasks. On quadruped it reaches high return earlier, and its multi-step prediction error is lower than DreamerV3’s on most measured quantities and significantly higher on none. Sharing information between neighboring states, as proposed here, improves the sample efficiency of world models.

Primary area:
reinforcement learning
Keywords:
Latent World Models, Sample Efficient Reinforcement Learning, Continuous Control Reinforcement Learning, Gradient Penalties, regularization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_WJnrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that to increase sample efficiency in world models in continuous control tasks, a state’s transition distribution can be enhanced by borrowing strength from neighboring states' transition distributions. The authors provide a mathematical justification by starting with a shrinkage estimator and showing that minimizing the negative log-likelihood of the observed transitions with a finite-difference penalty produces the same shrinkage estimator. They then use a penalty based on the squared Frobenius norm of the transition function's Jacobian in continuous space. In DreamerV3, this is the Jacobian of the posterior probability function. Afterwards, empirical results show improvement over DreamerV3 primarily in proprioceptive rather than pixel based observations.

Critical strengths

The first primary strength is a strong theoretical justification for the introduced regularization penalty. As mentioned in the summary, the paper establishes the relationship between the shrinkage estimator and the penalty but the explanation was slightly esoteric given the style of mathematics. The second strength is that on proprioceptive tasks the new method improved upon the baseline.

Critical weaknesses

The first critical weakness is that on pixel based observations the improvement seemed to diminish according to table 9. Second, the method acts as an improvement on DreamerV3 but this model is slightly outdated and there have been substantial improvements on world models since Danijar Hafner first released it. How well would this method generalize to other world model architectures?

Questions for authors

Has any work been done on extending this method to other world model architectures? Please explain the reason for the gap in performance between the proprioceptive observation and the pixel observations.

2 comments
AuthorsOpenReview ↗

Response to the queries regarding other World Models and pixel results gap

Thank you for the positive assessment and for a summary that captures the argument accurately.

Why the gain is smaller on pixels (Table 11). The gap comes from the input the penalty acts on. GPLD shares information between states whose encoder outputs are close. Proprioceptive observations are low-dimensional and smooth in the physical state, so nearby states have nearby encoder outputs from the start. Pixel observations are not, so this holds only after the encoder has been trained. Until then, the penalty smooths a geometry that does not yet reflect the dynamics. Consistent with this, a warm-started encoder restores the gain on pixel quadruped-walk (general comment, point 1).

Other world-model architectures. DreamerV3 (Hafner et al., 2025) is a widely used baseline for world-model agents, which is why we built on it. The method itself is not tied to DreamerV3: the derivation needs only a differentiable transition distribution. As a first extension, we added GPLD to MBPO, a model-based method with a Gaussian-ensemble world model. There it raises final return on two of five MuJoCo environments and the area under the learning curve on all five (general comment, point 3; App. P).

Accessibility of the theory. Thank you for raising this. Briefly, the argument has three steps, which Figure 1 (p. 2, ll. 72-81) also illustrates:

  • (a) A penalty on differences between neighboring states' transition distributions lets each state's estimate share information with its neighbors, most where it has few visits.
  • (b) As the neighborhood shrinks, the penalty becomes the squared Frobenius norm of the Jacobian of the transition map.
  • (c) In DreamerV3 it is applied to the posterior, and the prior is smoothed by being trained toward it.

In the revision we have added a short overview of this argument at the end of the introduction (p. 2, ll. 86-93). We would value the reviewer's view on which step would benefit most from a fuller explanation, for example the equivalence between the shrinkage estimator and the penalized likelihood, or the passage from neighbor differences to the Jacobian, and we will expand the steps the reviewer identifies.

Thank you again for your review. As the discussion period ends soon, we wanted to check whether our response and the revised PDF (changes marked in blue) address your concerns. We would be glad to answer any further questions. If they do address your concerns, we would be grateful if you would consider updating your score.

Reviewer_U7JHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an action-conditioned state-affine latent transition model trained with recursive rollout supervision for visual planning. The analysis separates per-step prediction errors from their subsequent propagation and characterizes state-affine transitions as having state-independent Jacobians.

Critical strengths
  • The paper provides a clear connection between transition structure and recursive error propagation.
  • The exact decomposition of state-affine dynamics offers a useful perspective on model design.
  • The combination of higher average task success, fewer predictor parameters, and faster planning is practically appealing.
Critical weaknesses
  • The incremental contribution relative to existing structured latent dynamics needs clearer substantiation. A more direct comparison with relevant structured predictors would help clarify what SALT contributes.
  • The evidence connecting the proposed propagation structure to improved planning remains incomplete. The explanation of the large planning gap on the OGBench-Cube is insufficient.
  • The empirical comparison has limited coverage and replication. LeWM is the only controlled learned-model baseline.
Questions for authors
  1. How stable are the main results and Table 3 for independent training runs?
  2. Does SALT retain its advantage even when using a nonlinear predictor?
  3. Can the analysis include all the history-dependent propagation pathways in LeWM and directly assess whether SALT better ranks candidate action sequences?
1 comment
AuthorsOpenReview ↗

Potential review mismatch

Thank you for your time. We believe this review may refer to a different submission: it discusses "SALT", an action-conditioned state-affine latent transition model for visual planning, its comparison with LeWM, and results on OGBench-Cube, none of which appear in our paper. We have informed the Area Chair.

For reference, our submission proposes GPLD, a decaying penalty on the Jacobian of DreamerV3's posterior, derived from a transition-estimation argument, and evaluates it on proprioceptive and pixel DMC tasks against DreamerV3 and other regularizers. We would be grateful for a review of this submission, and we are happy to answer any questions about it.

Reviewer_ZSCXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper targets inaccurate world-model estimation where transition data is sparse, and proposes GPLD, which penalizes the Jacobian energy of DreamerV3’s posterior. The motivation is that neighboring states evolve under similar transition laws: a finite-difference penalty in a finite MDP induces a shrinkage estimator that shares information across neighboring states, and its small-neighborhood limit is the squared Frobenius norm of the dynamics Jacobian. On proprioceptive DMC, GPLD raises final return on 15 of 18 tasks (Table 1, sign test p = 0.0075) and on five of the six humanoid and dog tasks (Table 2), while the gain largely disappears under pixel observations (Table 8).

Critical strengths

The penalty and its decay schedule are derived from the shrinkage argument, and the tabular experiment confirms the predicted effective-sample-size gain. The smoothness diagnostics show that GPLD makes the posterior less sensitive to its input rather than sharper.

Critical weaknesses
  1. Weak baseline. In Table 5 the 17-task means at 500k steps are 654.5 for the rerun DreamerV3 and 689.7 for GPLD against 730.5 for the scores released with [1] and the gap is larger on walker-run (202.6 and 380.3 vs. 632.7).
  2. Gap between theory and method. Theorem 1 in Appendix D holds only while the estimate stays on the observed-support face which the authors note fails in the sparse regime and Section 6 states that the penalized posterior map is coordinate-dependent.
  3. Insufficient comparison. [2] already adds a Jacobian regularizer to world-model dynamics but is not included as a baseline and GPLD’s lead over VAT [3] in Appendix N.2 is modest (1.204 vs. 1.118 normalized return).

I would raise my score to Weak Accept if the authors (1) reproduce the gain on a DreamerV3 baseline that matches the scores released with [1], and (2) include [2] as a baseline.

Questions for authors
  1. No gain under pixel observations. In Table 8 the normalized final return over 20 pixel tasks is 1.033 and the warm-start diagnostic in Table 20 uses a single task (quadruped-walk) with λ₀ = 0.5. Please claim why this happens.
  2. Incomplete cost and schedule analysis. Table 6 reports a 1.50× wall-clock overhead while the equal-compute control in Table 7 covers only three tuning tasks and Section 4.3 states that the 1/n schedule from Appendix C.1 is not compared.
4 comments
AuthorsOpenReview ↗

Results for both requested experiments: DreamerV3 at its released-score configuration, and Fang et al. (2024) as a baseline

Thank you for the detailed review and for stating what would change your assessment. We address both conditions first.

(1) A baseline that matches the released scores. The released scores come from the April 2024 code (App. H, p. 34, l. 1783); on the 2025 code that both our arms use, that setting (size12m, train ratio 512, action repeat 2) scores lower. The 2025 code's own default (size1m, train ratio 1024, action repeat 1) reproduces the released 500k scores (walker-run 602 vs 632.7, cheetah-run 579 vs 576, hopper-stand 584 vs 573) and the published 1M scores of this configuration (median ratio 0.97, 16 tasks). We reran GPLD there with the paper's settings (λ0 = 0.25, ρ = 0.5, square-root decay), setting decay coefficient c = 4000 instead of 1000, because this setting makes four times as many updates per environment step. At 1M steps, five seeds, on the nine tasks DreamerV3 does not saturate here:

TasksMetricDreamerV3GPLDRatioAhead
Tuning (3)Final770.5 ± 9.9812.2 ± 8.71.0542/3
Tuning (3)nAUC528.5 ± 28.3576.8 ± 17.51.0912/3
All nineFinal546.7 ± 17.6598.2 ± 18.71.0947/9
All ninenAUC393.6 ± 10.5429.9 ± 14.01.0926/9

Ratio is GPLD's mean over tasks divided by DreamerV3's. GPLD remains ahead of this stronger baseline, the clearest gains are cheetah-run (811.3 ± 22.0 vs 712.1 ± 24.0) and pendulum-swingup nAUC (741.9 ± 5.8 vs 703.0 ± 12.7, Welch's t-test p < 0.05). This baseline saturates three of the paper's twelve unsaturated tasks. We have added these runs to App. H (p. 34, ll. 1815-1829; Tables 6-7); the paper's main tables are unchanged.

(2) Fang et al. as a baseline. Our App. A imprecisely said Fang et al. regularize "world-model dynamics". They penalize the Jacobian of the deterministic recurrent update, ∂h_t/∂(h_{t-1}, z_{t-1}), i.e. the RNN state of the RSSM. GPLD penalizes the posterior, a distribution over the next latent state, which is what our transition-estimation argument concerns. The penalties act on different parts of the model, so Fang et al. is complementary rather than a direct alternative, and the two could in principle be used together. App. A is corrected (p. 16, ll. 837-838).

We implement it as published: the unsquared Frobenius norm, estimated with one random projection (Hoffman et al., 2019), with their λ = 0.01. All other settings are as in the paper. Over the 12 unsaturated tasks, with Table 32's seeds (Fang et al. has five on the tuning tasks):

DreamerV3Fang et al.VATGPLD
Mean final return582.7 ± 11.7586.4 ± 17.4622.9 ± 9.6650.6 ± 9.4
Mean nAUC431.7 ± 10.3449.4 ± 13.9456.2 ± 9.3473.5 ± 9.8
Normalized final (mean of task ratios)1.0000.9831.1181.204
Tasks above DreamerV37/129/1211/12

Fang et al. is level with DreamerV3 overall and below it on the sparse-reward tasks (acrobot-swingup-sparse 41.0 vs 91.9; cartpole-swingup-sparse 503.1 vs 789.6). Fang et al. and GPLD have the same wall-clock cost to within 5%. Details are in App. N.4 (p. 58, ll. 3080-3091; Figure 14).

(3) Margin over VAT. Over the 12 unsaturated tasks, GPLD's gain over DreamerV3 is 20.4% against 11.8% for VAT (normalized 1.204 vs 1.118; p. 53, ll. 2853-2854; Table 32). VAT lowers the posterior's Jacobian energy partly by sharpening it (entropy falls, Table 31); GPLD lowers it while entropy rises (Tables 25-26).

(4) Theory. Theorem 1 only bounds the remainder to a smoother surrogate, with its scope stated (App. D, p. 24, ll. 1285-1293). The gains rest on the bias-variance results (Theorems 2-4, p. 25, ll. 1299-1345), whose variance term is largest when data is scarce. Coordinate dependence is stated (Sec. 6, pp. 9-10, ll. 485-487), so every smoothness diagnostic is measured relative to the scale of the posterior's inputs (p. 8, ll. 424-426) and cannot be lowered by rescaling.

(5) Pixels and compute. Please see the general comment, points 1-2. The warm-start run's λ0 = 0.5 is stated on p. 33, l. 1734 and p. 46, l. 2475.

(6) 1/n schedule. We now compare λ0/(1 + n/c) with the square-root schedule (Sec. 4.3, p. 6, ll. 285-288), choosing c on walker-run from {1000, 8000, 16850, 64000}. Final return at 1M steps, five seeds:

TaskDreamerV3Square-root1/n
walker-run456.3 ± 61.5625.4 ± 48.5678.3 ± 10.9
cheetah-run638.8 ± 40.2741.8 ± 30.6717.9 ± 45.0
hopper-stand681.5 ± 85.2702.4 ± 62.4586.4 ± 90.4
Normalized11.1881.157

With its best constant, 1/n is ahead only on walker-run, where c was chosen, which is consistent with the paper's choice of the square-root schedule. We have added these runs to App. K (pp. 44-45, ll. 2374-2393; Tables 22-23).

Thank you again for your review and for stating what would change your assessment. We believe the revision addresses both conditions: GPLD remains ahead of DreamerV3 in the 2025 default configuration that reproduces the released scores (App. H, Tables 6-7), and Fang et al. is now included as a baseline (App. N.4, Tables 32-33). We would be glad to answer any remaining questions before the discussion period ends, and if these address your concerns, we would be grateful if you would reconsider your score.

Thank the author for the rebuttal. Most of my concerns have been resolved. I have raised my score accordingly.

Thank you for considering our responses and for raising your score.

Reviewer_rb1ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Gradient Penalized Latent Dynamics (GPLD), a Jacobian-based regularization method intended to improve the sample efficiency of learned world models. The approach is motivated by theoretical arguments about sharing transition information between neighboring states and is evaluated extensively using DreamerV3.

Critical strengths

The work is clearly motivated and offers a principled connection between smoothness regularization and transition estimation. It contains an extensive experimental evaluation, demonstrating improvements on 15 of 18 proprioceptive control tasks, including more complex humanoid and quadruped environments. The methodological contribution is strong, since it includes comparisons against alternative regularizers, ablations, and analyses of prediction accuracy and computational overhead. The approach is simple and could be integrated into existing world models.

Critical weaknesses

Currently, the algorithmic contribution is primarily an application of an existing regularization principle to DreamerV3. This is fine and I think the method can be transferred to other algorithms as well but could be discussed a little further. There is currently a gap between theory and implementation, where the theoretical analysis concerns smoothness of transition distributions, whereas the proposed method regularizes the observation-conditioned posterior. I think the connection here needs to be better explained. Currently, the method seems to provide no improvement on pixel-based tasks, which substantially limits the applicability to general-purpose world models.The training requires a lot more wall-clock time. This needs to be discussed.

Questions for authors

Can the authors further justify why regularizing the posterior should induce the kind of transition smoothing described in the theoretical analysis? What explains the absence of improvements on pixel-based tasks? This doesn't need to be tested but discussed. How robust are the improvements to alternative world-model architectures?

2 comments
AuthorsOpenReview ↗

Explaining the contribution connections to posterior distribution choice and other queries

Thank you for the positive assessment and the constructive questions.

Contribution beyond applying an existing regularizer. GPLD is not a generic smoothness penalty added to DreamerV3. Its ingredients follow from the transition-estimation argument rather than being chosen by hand (Contribution 1, p. 2, ll. 101-105). The argument fixes the form of the penalty and its neighbor weights. It also determines which map inside the world model to penalize (Sec. 4.2, p. 5, ll. 245-249; Table 3, p. 18, ll. 923-937). Finally, it explains why the coefficient must decay as data accumulates (Sec. 4.3, p. 6, ll. 281-283). Prior Jacobian penalties are general-purpose smoothness regularizers, and they typically derive neither the map nor the schedule (App. A, p. 16). Because our derivation needs only a differentiable transition distribution, the same penalty can be defined in other world models. As a first check, we added it to MBPO. There it raises final return on two of five MuJoCo environments and the area under the learning curve on all five (general comment, point 3; App. P).

Why penalizing the posterior smooths the transition model. The posterior is itself a transition estimate. It outputs q(z_t | h_t, e_t), a distribution over the next latent state. Its input h_t = f(h_{t-1}, z_{t-1}, a_{t-1}) encodes the previous latent state and action, and e_t encodes the current observation. The Jacobian of the posterior therefore has one block for h_t and one for e_t (App. B, p. 17, ll. 914-917), and the penalty covers both. The h_t block limits how much the predicted next-state distribution changes between neighboring previous states and actions. This is the neighbor sharing that the theory describes. The e_t input keeps this estimate tied to the observed data, and the posterior is the only distribution over latent states that has this input (Sec. 4.2, p. 5, ll. 245-249; Table 3, p. 18, ll. 923-937). The prior then inherits this smoothing. Imagination rolls the prior, and DreamerV3 trains the prior toward stop-gradient posterior targets. A smoother posterior therefore gives the prior smoother targets (Sec. 4.2, p. 5, ll. 249-251). We measure this directly:

  • The unpenalized prior. Its input-scaled Jacobian energy falls to 0.57, 0.73 and 0.75 of DreamerV3's, significantly on two of three tasks, and its entropy rises. It becomes smoother, not sharper (Sec. 5.3, p. 8, ll. 427-429; Table 27; p. 48, ll. 2587-2591).
  • Penalizing the prior directly is worse than penalizing the posterior: normalized return 1.008 against 1.106 over eight tasks (p. 5, ll. 251-253; Table 17).
  • Open-loop error. 25-step error from real states is lower than DreamerV3's on 10 of 14 measured quantities (p. 8, ll. 430-431 and p. 9, ll. 451-452; Table 28).

The pieces of this argument appeared in Sec. 4.2 and App. B; in the revision we have brought them together in Sec. 4.2 (p. 5, ll. 240-245). We would welcome the reviewer's suggestions on any part of this connection that remains unclear.

Pixels, wall clock, other architectures. See points 1-3 of the general comment.

Presentation. We have added a short overview of the argument in Figure 1 at the end of the introduction (p. 2, ll. 86-93), so that the intuition comes before the derivation. If there are particular sections the reviewer found hardest to follow, we would be grateful to know, so that we can prioritize them in the revision.

Thank you again for your review. As the discussion period ends soon, we wanted to check whether our response and the revised PDF (changes marked in blue) address your concerns. We would be glad to answer any further questions. If they do address your concerns, we would be grateful if you would consider updating your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.