- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper identifies a shared-prior conflict in world models that condition the prior on the action. To address this issue, the paper proposes a drop-in module named MixturePrior. The authors analyze why action-dependent weighting outperforms any fixed mixture in transition prediction. The authors demonstrate the compatibility of MixturePrior with both DreamerV3 and TWISTER. They evaluate MixturePrior on Atari 100K and DeepMind Control 500K. They tune hyperparameters introduced by MixturePrior separately for each task and report improvements over methods that use fixed hyperparameters.
Critical strengths
- The paper presents a well-motivated approach to improving the prior-fitting objective by using the action as an explicit routing signal.
- The paper provides a clear and logically consistent theoretical account of both the prior-fitting difficulty and the role of action-dependent routing in resolving it.
Critical weaknesses
- The reliance on task-specific hyperparameter selection raises concerns about practical utility and evaluation fairness. The main results in Table 1 rely on selecting the number of components and the gating temperature separately for individual tasks. In contrast, the standard DreamerV3 and TWISTER baselines do not rely on per-task hyperparameter tuning, making the comparison less compelling from both a practical and an evaluation-fairness perspective. In current world-model evaluation protocols, demonstrating strong performance under a fixed hyperparameter configuration is important for establishing robustness and transferability across tasks. Moreover, the related-work discussion does not sufficiently cover the latest developments[1-4] in world models.
- The advantage over TWISTER largely disappears when MixturePrior is evaluated with fixed hyperparameters. According to Table 9, even the best fixed MixturePrior configuration for TWISTER does not outperform the published TWISTER baseline. On Atari-100k, MixturePrior () achieves a mean HNS of 122.76, whereas the TWISTER baseline achieves 162.09. On the DeepMind Control Suite, MixturePrior () achieves a mean return of 795.80, which is still below TWISTER's 801.80. The current experiments do not yet convincingly demonstrate that MixturePrior provides a robust and practically useful improvement, though the underlying idea is appealing.
References
[1] Morihira, N., Nahar, A., Bharadwaj, K., Kato, Y., Hayashi, A. and Harada, T., 2026. R2-Dreamer: Redundancy-reduced world models without decoders or augmentation. In International Conference on Learning Representations.
[2] Peng, Z.H., Li, S., Li, Z., Ruan, S., Liu, Y. and He, Y., 2026. From Observations to Events: Event-Aware World Model for Reinforcement Learning. In International Conference on Learning Representations.
[3] Zhang, B., Zhang, W., Feng, Z., Xiao, W., Sun, J., Chen, J. and Wang, G., 2026. Mixture-of-World Models: Scaling Multi-Task Reinforcement Learning with Modular Latent Dynamics. In International Conference on Learning Representations.
[4] Cohen, L., Wang, K., Kang, B., Gadot, U. and Mannor, S., 2026. Simulus: Combining Improvements in Sample-Efficient World Model Agents. arXiv preprint arXiv:2502.11537.
Questions for authors
- Applicability to other world-model architectures. Could MixturePrior be integrated into other latent world models, such as STORM, which predict future frames directly?
- Could you provide an automatic procedure for selecting and , or a reliable default configuration that can be used across tasks without manual tuning?
- Could the authors further clarify the role and architecture of the Component MLP in Figure 2? In particular, what are its exact inputs?
- What was the complete hyperparameter search space and selection procedure? Section 5.1 lists six configurations, but Table 13 includes additional settings such as for Gopher.
- Please report the training cost and computational overhead. What does "milliseconds per update" in Appendix D mean? Does it account for all training components of the world model?
- Could the authors provide more intuition for why the required correction and the supplied correction should interact multiplicatively, rather than using a ratio-based quantity such as where ?
- In Figure 1, "wight" should be corrected to "weight".
- Please further discuss the limitations of this work. In particular, what is the relationship between improvements in the prior-fitting risk and downstream task performance? Does a lower prior-fitting risk necessarily translate into better control performance?
7 comments
Thank you for providing the additional hyperparameter selection procedure based on held-out trajectories.
However, my concern regarding the need to select and on a per-task basis has not yet been fully resolved. In particular, I would appreciate additional evidence addressing the following two points:
-
Evaluation of the automated selection protocol at full benchmark scale.
Please apply the proposed automated hyperparameter selection procedure to the full set of benchmark tasks, and report the selected for each task together with the corresponding final control performance. More importantly, the results obtained using the automatically selected hyperparameters should be compared against the relevant baselines. This would help demonstrate whether the proposed validation criterion can reliably identify suitable hyperparameters without relying on manual task-specific tuning or retrospective selection based on evaluation returns. -
Computational and data cost of hyperparameter selection.
Please report the additional budget required by the proposed selection procedure, including the cost of training candidate models, collecting validation trajectories, and computing the NLL-based selection metric. This is important for understanding the practical cost-benefit trade-off of the proposed procedure relative to simply using a fixed default configuration.
The remaining concern is whether the proposed method can deliver consistent performance improvements without manual task-specific hyperparameter selection or an expensive hyperparameter search procedure. At present, it is still unclear whether the validation-based selection strategy can robustly translate into improved downstream control performance across a broad range of tasks.
Alternatively, if the authors can identify a single fixed configuration applicable across all tasks and demonstrate consistent improvements over the baselines, this would also substantially address my concern and positively affect my overall assessment.
We thank you for your constructive feedback. We have supplemented full-benchmark results on NLL-based automatic selection, the fixed configuration, and computational cost, together with a preliminary STORM transfer experiment.
1. Automatic selection vs. fixed (4,1)
We extend the Appendix E.2 one-standard-error rule to 26 Atari-100k and 20 DMC tasks, with candidates (1,1), (2,1), (4,1), (8,1), (4,0.5), (4,2). NLL uses held-out trajectories and 64 MC latent samples, with control returns excluded from selection.
Table 1. Automatic selection vs. fixed (4,1)
| Metric | Automatic | Fixed (4,1) | Δ (Auto − Fixed) |
|---|---|---|---|
| Atari HNS (n=26) | 93.82 | 125.62 | −31.79 |
| DMC Return (n=20) | 797.75 | 815.13 | −17.38 |
The automatic rule is worse in aggregate on both suites, with per-task wins/ties/losses of 10/4/12 (Atari) and 6/3/11 (DMC).
2. Why the automatic rule underperforms the fixed configuration
The NLL rankings have moderate split-half reproducibility (Spearman ρ = 0.44 on Atari and 0.61 on DMC). The problem is that a reproducible NLL ranking is not a ranking by control return. Inside the one-SE band the NLL differences between configurations are small, so the tie-break systematically favors smaller K: 19 of 46 selections are K=1 (14 Atari, 5 DMC). These low-K choices produce the largest losses — Up N Down (−327.78 HNS), Krull (−251.99 HNS), and Crazy Climber (−231.55 HNS) on Atari, and Cartpole Swingup Sparse (−656.00 return) and Pendulum Swingup (−465.00 return) on DMC — while other tasks move in the opposite direction (e.g., Walker Stand +913.55 return on DMC). The aggregate deficit is therefore driven by a handful of mis-selected tasks rather than a uniform degradation: most tasks are near the fixed reference, but the few collapsed ones dominate the mean.
3. The fixed configuration is a strong reference
Against the published backbone scores, fixed (4,1) raises Atari HNS from 112.40 to 125.62 (+13.22) and DMC return from 786.05 to 815.13 (+29.08).
Table 2. Matched component count (K=1 vs. K=4)
| Metric | K=1 | K=4 | Δ | W/T/L |
|---|---|---|---|---|
| Atari HNS (n=26) | 80.46 | 125.62 | +45.16 (+56.1%) | 15/1/10 |
| DMC Return (n=20) | 803.89 | 815.13 | +11.24 (+1.4%) | 8/1/11 |
With backbone, pipeline, and τ = 1 held fixed, four components improve the mean on both benchmarks and win on the majority of tasks (15/26 on Atari, 8/20 on DMC). The automatic rule's deficit is a tail effect from a few low-K mis-selections, whereas the fixed (4,1) configuration is both close to the reference on most tasks and better in aggregate — so we adopt it as the default.
4. Cost
Table 3. Selection cost
| Item | Logged GPU-hours |
|---|---|
| Candidate training (measured subset) | 2,098 |
| Trajectory collection | 1.503 |
| NLL computation | 6.18 |
| Loading / JIT / startup / writing (residual) | 1.20 |
| End-to-end (measured, 46 tasks) | 8.88 (0.42%) |
| Full 46-task extrapolation | 11.04 (0.43%) |
Selection adds 0.42% relative to logged candidate-training time on the measured subset (0.43% under full-benchmark extrapolation). The extrapolated training denominator is approximately 2,579 GPU-hours; candidate-training times include queueing.
5. STORM transfer
We also evaluate STORM (7 tasks, 6 configs, one seed, 20 episodes).
Table 4. STORM transfer
| Task | Native | (2,1) | (4,1) | (8,1) | (4,0.5) | (4,2) |
|---|---|---|---|---|---|---|
| Pong | 12.85 | −3.95 | 14.05 | −6.05 | 9.60 | 4.80 |
| Breakout | 8.00 | 11.70 | 12.95 | 12.70 | 9.50 | 11.15 |
| Seaquest | 492.00 | 531.00 | 431.00 | 550.00 | 435.00 | 356.00 |
| MsPacman | 2462.50 | 1778.00 | 2415.50 | 3454.50 | 2068.50 | 1845.50 |
| BattleZone | 6750.00 | 10250.00 | 7100.00 | 8900.00 | 9050.00 | 10650.00 |
| Boxing | 79.90 | 71.60 | 53.50 | 57.50 | 66.25 | 76.80 |
| Cartpole Swingup | 823.50 | 727.28 | 749.14 | 823.47 | 721.09 | 807.64 |
Fixed (4,1) beats native STORM on Pong, Breakout, and BattleZone; 5/7 tasks beat native when taking the best candidate per task (descriptive, not automatic selection). This is preliminary evidence of cross-architecture transfer.
Full per-task results (46 tasks) are provided in a separate supplementary response.
CW1–2, Q2, Q4 — Fairness, Fixed Configurations, Selection
Fixed configurations do not beat the published TWISTER baseline — our native reproduction shows the same gap — yet Table 2 (p. 9) shows consistent gains from more components at τ=1.
We select K and τ by one-step prior-predictive NLL on held-out trajectories, keep configurations within one standard error of the minimum, and fix them before any control evaluation.
Q1 — Compatibility with STORM
Yes—MixturePrior can be integrated into STORM. STORM uses categorical stochastic latents and an observation decoder, so the factorized prior interface is present: its Transformer representation feeds the Component MLP and the action the gate.
Q3 — Component MLP Inputs and Architecture
The direct input to the Component MLP is the action-updated backbone feature, with the network structure in Appendix B.1 :
ht+1=fb(ht,zt,at),
The Component MLP is one shared linear–ELU–linear network producing KFC logits, normalized within each factor; the action enters a separate gating MLP whose weights are shared across factors.
Q5 — Training Cost and Overhead
Timings cover one full world-model optimizer update. ms/update in Tables 8–9 (p. 29) is log-derived (TWISTER: TensorBoard-event wall-clock; DreamerV3: last valid
fps/trainwindow), with CPC retained in both TWISTER conditions; interaction, evaluation and I/O are excluded.Q6 — Interpreting the Correction Value
The multiplicative interaction is the first-order reduction in KL prior-fitting risk a candidate component achieves. With prior p, candidate r, target qˉ, and correction di(j)=ri(j)−pi(j), the action-conditional KL risk falls by
−dϵdRa(p+ϵd)ϵ=0=Ex∣ai,j∑pi(j)qˉi(j)di(j)=Ex∣ai,j∑pi(j)(qˉi(j)−pi(j))di(j)=Ar(a).
Since ∑jdi(j)=0, the correction value is an inverse-probability-weighted inner product of required and supplied correction: aligned changes add, opposing ones subtract.
Q7 — Figure 1 Typo
It now reads “weight.”
Q8 — Prior Fitting vs. Downstream Control
Lower one-step prior-fitting risk does not necessarily yield higher control returns.
Manuscript changes. E.2, Table 12 the validation-based selection and transfer, Section 5.1 with E.6, Table 17 the search space, and C.2, Table 7 the routing ablation with the Section 5.4 dissociation. Appendix D and Table 10 the timing conventions; Section 3.2 with A.3 the correction-value interpretation. Section 2 adds the citations and Appendix D adds the limitations subsection.