- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper observes that to increase sample efficiency in world models in continuous control tasks, a state’s transition distribution can be enhanced by borrowing strength from neighboring states' transition distributions. The authors provide a mathematical justification by starting with a shrinkage estimator and showing that minimizing the negative log-likelihood of the observed transitions with a finite-difference penalty produces the same shrinkage estimator. They then use a penalty based on the squared Frobenius norm of the transition function's Jacobian in continuous space. In DreamerV3, this is the Jacobian of the posterior probability function. Afterwards, empirical results show improvement over DreamerV3 primarily in proprioceptive rather than pixel based observations.
Critical strengths
The first primary strength is a strong theoretical justification for the introduced regularization penalty. As mentioned in the summary, the paper establishes the relationship between the shrinkage estimator and the penalty but the explanation was slightly esoteric given the style of mathematics. The second strength is that on proprioceptive tasks the new method improved upon the baseline.
Critical weaknesses
The first critical weakness is that on pixel based observations the improvement seemed to diminish according to table 9. Second, the method acts as an improvement on DreamerV3 but this model is slightly outdated and there have been substantial improvements on world models since Danijar Hafner first released it. How well would this method generalize to other world model architectures?
Questions for authors
Has any work been done on extending this method to other world model architectures? Please explain the reason for the gap in performance between the proprioceptive observation and the pixel observations.
2 comments
Thank you again for your review. As the discussion period ends soon, we wanted to check whether our response and the revised PDF (changes marked in blue) address your concerns. We would be glad to answer any further questions. If they do address your concerns, we would be grateful if you would consider updating your score.
Response to the queries regarding other World Models and pixel results gap
Thank you for the positive assessment and for a summary that captures the argument accurately.
Why the gain is smaller on pixels (Table 11). The gap comes from the input the penalty acts on. GPLD shares information between states whose encoder outputs are close. Proprioceptive observations are low-dimensional and smooth in the physical state, so nearby states have nearby encoder outputs from the start. Pixel observations are not, so this holds only after the encoder has been trained. Until then, the penalty smooths a geometry that does not yet reflect the dynamics. Consistent with this, a warm-started encoder restores the gain on pixel quadruped-walk (general comment, point 1).
Other world-model architectures. DreamerV3 (Hafner et al., 2025) is a widely used baseline for world-model agents, which is why we built on it. The method itself is not tied to DreamerV3: the derivation needs only a differentiable transition distribution. As a first extension, we added GPLD to MBPO, a model-based method with a Gaussian-ensemble world model. There it raises final return on two of five MuJoCo environments and the area under the learning curve on all five (general comment, point 3; App. P).
Accessibility of the theory. Thank you for raising this. Briefly, the argument has three steps, which Figure 1 (p. 2, ll. 72-81) also illustrates:
In the revision we have added a short overview of this argument at the end of the introduction (p. 2, ll. 86-93). We would value the reviewer's view on which step would benefit most from a fuller explanation, for example the equivalence between the shrinkage estimator and the penalized likelihood, or the passage from neighbor differences to the Jacobian, and we will expand the steps the reviewer identifies.