acceptodds
Under review as a conference paper at ICLR 2027

MIXTUREPRIOR: ACTION-ROUTED TRANSITION PRIORS FOR LATENT WORLD MODELS

Abstract

Latent world models train policies on imagined rollouts, in which every future state is sampled from a learned stochastic transition prior. Although this prior is conditioned on the action, a single set of parameters must fit the next-state distribution for every action. A correction that helps one action can therefore hurt another, and such opposing corrections cancel in the shared parameters, so the prior can stall while individual actions remain under-fitted. Our key idea is that the prior should offer several candidate corrections and that the action should decide where each one is used. We propose MIXTUREPRIOR, which predicts multiple categorical transition components from the backbone's action-updated feature and combines them with an action-only gate. Because it replaces only the stochastic prior, MIXTUREPRIOR plugs into recurrent and Transformer-based world models without further changes. We analyze this design through the correction value of a component and show that, when a component helps some actions but hurts others, action-dependent weighting achieves a strictly larger first-order improvement than any fixed mixture. We integrate MIXTUREPRIOR into DreamerV3 and TWISTER and evaluate it on 26 Atari-100k games and 20 DeepMind Control tasks. With everything else fixed, four components improve the aggregate mean over a single-component prior in all four backbone–benchmark combinations. In a trained model, we further find components whose correction value changes sign across actions, which is the condition under which our analysis predicts action routing to help.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, latent dynamics, mixture priors, action-dependent routing, stochastic transition priors

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MMQ9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper identifies a shared-prior conflict in world models that condition the prior on the action. To address this issue, the paper proposes a drop-in module named MixturePrior. The authors analyze why action-dependent weighting outperforms any fixed mixture in transition prediction. The authors demonstrate the compatibility of MixturePrior with both DreamerV3 and TWISTER. They evaluate MixturePrior on Atari 100K and DeepMind Control 500K. They tune hyperparameters introduced by MixturePrior separately for each task and report improvements over methods that use fixed hyperparameters.

Critical strengths
  1. The paper presents a well-motivated approach to improving the prior-fitting objective by using the action as an explicit routing signal.
  2. The paper provides a clear and logically consistent theoretical account of both the prior-fitting difficulty and the role of action-dependent routing in resolving it.
Critical weaknesses
  1. The reliance on task-specific hyperparameter selection raises concerns about practical utility and evaluation fairness. The main results in Table 1 rely on selecting the number of components KK and the gating temperature τ\tau separately for individual tasks. In contrast, the standard DreamerV3 and TWISTER baselines do not rely on per-task hyperparameter tuning, making the comparison less compelling from both a practical and an evaluation-fairness perspective. In current world-model evaluation protocols, demonstrating strong performance under a fixed hyperparameter configuration is important for establishing robustness and transferability across tasks. Moreover, the related-work discussion does not sufficiently cover the latest developments[1-4] in world models.
  2. The advantage over TWISTER largely disappears when MixturePrior is evaluated with fixed hyperparameters. According to Table 9, even the best fixed MixturePrior configuration for TWISTER does not outperform the published TWISTER baseline. On Atari-100k, MixturePrior (K=4,τ=1K=4,\tau=1) achieves a mean HNS of 122.76, whereas the TWISTER baseline achieves 162.09. On the DeepMind Control Suite, MixturePrior (K=4,τ=2K=4,\tau=2) achieves a mean return of 795.80, which is still below TWISTER's 801.80. The current experiments do not yet convincingly demonstrate that MixturePrior provides a robust and practically useful improvement, though the underlying idea is appealing.

References

[1] Morihira, N., Nahar, A., Bharadwaj, K., Kato, Y., Hayashi, A. and Harada, T., 2026. R2-Dreamer: Redundancy-reduced world models without decoders or augmentation. In International Conference on Learning Representations.

[2] Peng, Z.H., Li, S., Li, Z., Ruan, S., Liu, Y. and He, Y., 2026. From Observations to Events: Event-Aware World Model for Reinforcement Learning. In International Conference on Learning Representations.

[3] Zhang, B., Zhang, W., Feng, Z., Xiao, W., Sun, J., Chen, J. and Wang, G., 2026. Mixture-of-World Models: Scaling Multi-Task Reinforcement Learning with Modular Latent Dynamics. In International Conference on Learning Representations.

[4] Cohen, L., Wang, K., Kang, B., Gadot, U. and Mannor, S., 2026. Simulus: Combining Improvements in Sample-Efficient World Model Agents. arXiv preprint arXiv:2502.11537.

Questions for authors
  1. Applicability to other world-model architectures. Could MixturePrior be integrated into other latent world models, such as STORM, which predict future frames directly?
  2. Could you provide an automatic procedure for selecting KK and τ\tau, or a reliable default configuration that can be used across tasks without manual tuning?
  3. Could the authors further clarify the role and architecture of the Component MLP in Figure 2? In particular, what are its exact inputs?
  4. What was the complete hyperparameter search space and selection procedure? Section 5.1 lists six configurations, but Table 13 includes additional settings such as K=4,τ=0.95K=4,\tau=0.95 for Gopher.
  5. Please report the training cost and computational overhead. What does "milliseconds per update" in Appendix D mean? Does it account for all training components of the world model?
  6. Could the authors provide more intuition for why the required correction and the supplied correction should interact multiplicatively, rather than using a ratio-based quantity such as qˉi(j)−pi(j)ri(j)−pi(j),\frac{\bar q_i(j)-p_i(j)}{r_i(j)-p_i(j)}, where ri(j)−pi(j)≠0r_i(j)-p_i(j)\neq 0 ?
  7. In Figure 1, "wight" should be corrected to "weight".
  8. Please further discuss the limitations of this work. In particular, what is the relationship between improvements in the prior-fitting risk and downstream task performance? Does a lower prior-fitting risk necessarily translate into better control performance?
7 comments

CW1–2, Q2, Q4 — Fairness, Fixed Configurations, Selection

Fixed configurations do not beat the published TWISTER baseline — our native reproduction shows the same gap — yet Table 2 (p. 9) shows consistent gains from more components at τ=1\tau=1.

We select KK and τ\tau by one-step prior-predictive NLL on held-out trajectories, keep configurations within one standard error of the minimum, and fix them before any control evaluation.

Source taskSelected (K,τ)(K,\tau)Confirmation Δ\DeltaNLL vs. (4,1)(4,1)Transfer targetReturn difference vs. target-local selection (95% CI)
Battle Zone(4,0.5)(4,0.5)+0.000034 [−0.000156, 0.000234]Acrobot Swingup+86.09 [44.63, 131.07]
Acrobot Swingup(4,1)(4,1)0 (same configuration)Battle Zone−1,266.67 [−3,533.33, 1,100.83]

Q1 — Compatibility with STORM

Yes—MixturePrior can be integrated into STORM. STORM uses categorical stochastic latents and an observation decoder, so the factorized prior interface is present: its Transformer representation feeds the Component MLP and the action the gate.

Q3 — Component MLP Inputs and Architecture

The direct input to the Component MLP is the action-updated backbone feature, with the network structure in Appendix B.1 :

h~t+1=fb(ht,zt,at),\widetilde{\mathbf h}_{t+1} =f_b(\mathbf h_t,\mathbf z_t,\mathbf a_t),

The Component MLP is one shared linear–ELU–linear network producing KFCKFC logits, normalized within each factor; the action enters a separate gating MLP whose weights are shared across factors.

Q5 — Training Cost and Overhead

Timings cover one full world-model optimizer update. ms/update in Tables 8–9 (p. 29)​ is log-derived (TWISTER: TensorBoard-event wall-clock; DreamerV3: last valid fps/train window), with CPC retained in both TWISTER conditions; interaction, evaluation and I/O are excluded.

BackboneTaskNative: Mean ± SD (ms)MixturePrior: Mean ± SD (ms)Relative change
TWISTERAlien267.995 ± 14.410276.406 ± 26.622+3.14%
TWISTERCartpole Swingup252.864 ± 16.191258.389 ± 17.914+2.18%
DreamerV3Alien926.254 ± 14.086922.965 ± 15.479−0.36%
DreamerV3Cartpole Swingup1038.421 ± 24.7501051.095 ± 18.755+1.22%

Q6 — Interpreting the Correction Value

The multiplicative interaction is the first-order reduction in KL prior-fitting risk a candidate component achieves. With prior pp, candidate rr, target qˉ\bar q, and correction di(j)=ri(j)−pi(j)d_i(j)=r_i(j)-p_i(j), the action-conditional KL risk falls by

−ddϵRa(p+ϵd)∣ϵ=0=Ex∣a∑i,jqˉi(j)di(j)pi(j)=Ex∣a∑i,j(qˉi(j)−pi(j))di(j)pi(j)=Ar(a).\begin{aligned}-\frac{d}{d\epsilon}R_a(p+\epsilon d)\big|_{\epsilon=0}&=\mathbb{E}_{x\mid a}\sum_{i,j}\frac{\bar q_i(j)d_i(j)}{p_i(j)}\\\\&=\mathbb{E}_{x\mid a}\sum_{i,j}\frac{\bigl(\bar q_i(j)-p_i(j)\bigr)d_i(j)}{p_i(j)}=\mathcal{A}_r(a).\end{aligned}

Since ∑jdi(j)=0\sum_j d_i(j)=0, the correction value is an inverse-probability-weighted inner product of required and supplied correction: aligned changes add, opposing ones subtract.

Q7 — Figure 1 Typo

It now reads “weight.”

Q8 — Prior Fitting vs. Downstream Control

Lower one-step prior-fitting risk does not necessarily yield higher control returns.

TaskRoutingDynamics KL ↓Joint NLL ↓Control score ↑
KangarooAction-conditioned15.1922.237,540
Uniform11.6017.449,525
Constant6.5210.645,800
Train-shuffled8.9914.10989
Up N DownAction-conditioned14.7825.99103,560
Uniform21.4148.258,442
Constant16.5535.504,865
Train-shuffled15.0326.0252,620

Manuscript changes. E.2, Table 12 the validation-based selection and transfer, Section 5.1 with E.6, Table 17 the search space, and C.2, Table 7 the routing ablation with the Section 5.4 dissociation. Appendix D and Table 10 the timing conventions; Section 3.2 with A.3 the correction-value interpretation. Section 2 adds the citations and Appendix D adds the limitations subsection.

Thank you for providing the additional hyperparameter selection procedure based on held-out trajectories.

However, my concern regarding the need to select KK and τ\tau on a per-task basis has not yet been fully resolved. In particular, I would appreciate additional evidence addressing the following two points:

  1. Evaluation of the automated selection protocol at full benchmark scale.
    Please apply the proposed automated hyperparameter selection procedure to the full set of benchmark tasks, and report the selected (K,τ)(K,\tau) for each task together with the corresponding final control performance. More importantly, the results obtained using the automatically selected hyperparameters should be compared against the relevant baselines. This would help demonstrate whether the proposed validation criterion can reliably identify suitable hyperparameters without relying on manual task-specific tuning or retrospective selection based on evaluation returns.

  2. Computational and data cost of hyperparameter selection.
    Please report the additional budget required by the proposed selection procedure, including the cost of training candidate models, collecting validation trajectories, and computing the NLL-based selection metric. This is important for understanding the practical cost-benefit trade-off of the proposed procedure relative to simply using a fixed default configuration.

The remaining concern is whether the proposed method can deliver consistent performance improvements without manual task-specific hyperparameter selection or an expensive hyperparameter search procedure. At present, it is still unclear whether the validation-based selection strategy can robustly translate into improved downstream control performance across a broad range of tasks.

Alternatively, if the authors can identify a single fixed (K,τ)(K,\tau) configuration applicable across all tasks and demonstrate consistent improvements over the baselines, this would also substantially address my concern and positively affect my overall assessment.

We thank you for your constructive feedback. We have supplemented full-benchmark results on NLL-based automatic selection, the fixed configuration, and computational cost, together with a preliminary STORM transfer experiment.

1. Automatic selection vs. fixed (4,1)

We extend the Appendix E.2 one-standard-error rule to 26 Atari-100k and 20 DMC tasks, with candidates (1,1), (2,1), (4,1), (8,1), (4,0.5), (4,2). NLL uses held-out trajectories and 64 MC latent samples, with control returns excluded from selection.

Table 1. Automatic selection vs. fixed (4,1)

MetricAutomaticFixed (4,1)Δ (Auto − Fixed)
Atari HNS (n=26)93.82125.62−31.79
DMC Return (n=20)797.75815.13−17.38

The automatic rule is worse in aggregate on both suites, with per-task wins/ties/losses of 10/4/12 (Atari) and 6/3/11 (DMC).

2. Why the automatic rule underperforms the fixed configuration

The NLL rankings have moderate split-half reproducibility (Spearman ρ = 0.44 on Atari and 0.61 on DMC). The problem is that a reproducible NLL ranking is not a ranking by control return. Inside the one-SE band the NLL differences between configurations are small, so the tie-break systematically favors smaller K: 19 of 46 selections are K=1 (14 Atari, 5 DMC). These low-K choices produce the largest losses — Up N Down (−327.78 HNS), Krull (−251.99 HNS), and Crazy Climber (−231.55 HNS) on Atari, and Cartpole Swingup Sparse (−656.00 return) and Pendulum Swingup (−465.00 return) on DMC — while other tasks move in the opposite direction (e.g., Walker Stand +913.55 return on DMC). The aggregate deficit is therefore driven by a handful of mis-selected tasks rather than a uniform degradation: most tasks are near the fixed reference, but the few collapsed ones dominate the mean.

3. The fixed configuration is a strong reference

Against the published backbone scores, fixed (4,1) raises Atari HNS from 112.40 to 125.62 (+13.22) and DMC return from 786.05 to 815.13 (+29.08).

Table 2. Matched component count (K=1 vs. K=4)

MetricK=1K=4ΔW/T/L
Atari HNS (n=26)80.46125.62+45.16 (+56.1%)15/1/10
DMC Return (n=20)803.89815.13+11.24 (+1.4%)8/1/11

With backbone, pipeline, and τ = 1 held fixed, four components improve the mean on both benchmarks and win on the majority of tasks (15/26 on Atari, 8/20 on DMC). The automatic rule's deficit is a tail effect from a few low-K mis-selections, whereas the fixed (4,1) configuration is both close to the reference on most tasks and better in aggregate — so we adopt it as the default.

4. Cost

Table 3. Selection cost

ItemLogged GPU-hours
Candidate training (measured subset)2,098
Trajectory collection1.503
NLL computation6.18
Loading / JIT / startup / writing (residual)1.20
End-to-end (measured, 46 tasks)8.88 (0.42%)
Full 46-task extrapolation11.04 (0.43%)

Selection adds 0.42% relative to logged candidate-training time on the measured subset (0.43% under full-benchmark extrapolation). The extrapolated training denominator is approximately 2,579 GPU-hours; candidate-training times include queueing.

5. STORM transfer

We also evaluate STORM (7 tasks, 6 configs, one seed, 20 episodes).

Table 4. STORM transfer

TaskNative(2,1)(4,1)(8,1)(4,0.5)(4,2)
Pong12.85−3.9514.05−6.059.604.80
Breakout8.0011.7012.9512.709.5011.15
Seaquest492.00531.00431.00550.00435.00356.00
MsPacman2462.501778.002415.503454.502068.501845.50
BattleZone6750.0010250.007100.008900.009050.0010650.00
Boxing79.9071.6053.5057.5066.2576.80
Cartpole Swingup823.50727.28749.14823.47721.09807.64

Fixed (4,1) beats native STORM on Pong, Breakout, and BattleZone; 5/7 tasks beat native when taking the best candidate per task (descriptive, not automatic selection). This is preliminary evidence of cross-architecture transfer.

Full per-task results (46 tasks) are provided in a separate supplementary response.

Reviewer_6bJ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that in latent world models, a single set of parameters for the stochastic transition prior must fit the next-state distribution for every action, so the corrections required by different actions can cancel in the shared parameters — leaving the prior at a stationary point overall while individual actions remain under-fitted. To address this, the authors propose MixturePrior: from the backbone's action-updated feature, it predicts K categorical transition components and mixes them in probability space with an action-only gate. Because it replaces only the stochastic prior, MixturePrior plugs into DreamerV3 and TWISTER without further changes. The theory analyzes the prior-fitting subproblem alone, defines the correction value of a component, and proves two results: action-dependent weighting achieves a strictly larger first-order improvement than any fixed weighting, and at a stationary single-component prior, splitting the head and routing the copies by action opens a strict descent direction. Experiments span two backbones, 26 Atari-100k games, and 20 DMC tasks, with one seed per configuration. At τ = 1, four components raise the aggregate mean over a single-component prior in all four backbone–benchmark combinations, by 9.7% and 56.1% on Atari-100k and by 7.2% and 1.4% on DMC for TWISTER and DreamerV3, respectively. Checkpoint diagnostics further observe components whose correction value changes sign across actions, and a gain from the interaction between splitting and routing. The same diagnostics, however, show that end-to-end trained gates are nearly action-independent.

Critical strengths
  • The theoretical analysis is relatively clear and comes with targeted mechanistic validation.
  • The method is simple and plug-and-play, and it works with both RNN and Transformer backbones. The reporting is transparent. The authors release per-task, per-configuration scores and openly report negative results: the end-to-end gate barely depends on the action, the parameter-matched controls are inconclusive, and increasing K does not always help.
Critical weaknesses
  1. The statistical evidence for control performance is weak. Every result comes from a single seed, and the paper reports no robust statistics such as IQM or confidence intervals. Table 1 takes the per-task maximum over several single-seed runs and compares it with official multi-seed averages, which inherently favors the proposed method. The matched K=1 baseline is also clearly weaker than the official results of the native backbones in three of the four combinations, most pronounced on Atari-100k. So it is unclear whether the gains over a weak baseline would carry over to a backbone running at its normal strength. On DreamerV3 Atari, the +56.1% gain comes mainly from a few high-variance games, and the DMC gains are largely within noise. Overall, the claim in RQ1 is not yet supported.
  2. The theory is disconnected from the observed gains, and the motivation is insufficient. The paper's own diagnostics show that the end-to-end-trained gate barely varies with the action. Shuffling the gate's action input leaves the NLL almost unchanged, and on most checkpoints the learned gate does worse than the best fixed mixture. In other words, the action routing that the theory explains is not what produces the gains. More likely explanations are the added capacity or the extra expressiveness of the mixture-of-categoricals output, but the paper lacks the key ablations needed to tell these explanations apart. In addition, the native prior already takes as input a hidden state that contains action information. The paper's justification for routing on the action — that it is observed at every imagined step (Sec 3.1) — applies equally to the latent state. It therefore does not establish why the action, rather than the state or state–action pair, is the right routing variable.
Questions for authors
  1. Could you run at least 3 seeds for K=1 and K=4 (τ=1) on all tasks and report IQM, bootstrap confidence intervals, and probability of improvement? On DreamerV3 Atari, the mean gap between K=4 and K=1 comes mainly from a few games such as Up N Down, Crazy Climber, and Krull. If this gain shrinks substantially across multiple seeds, would the main conclusions need to be revised?
  2. Why is the K=1 baseline clearly below the official results? Could this point to a problem in the reproduction pipeline or in the K=1 interface?
  3. Could you add two controls: (i) an end-to-end-trained, action-independent mixture with learnable weights, and (ii) a parameter-matched single-head prior on all tasks? Since the learned gate is nearly action-independent, these experiments are key to separating "action routing" from "capacity / output expressiveness."
  4. How does the paper's motivation relate to works such as [1][2], which use MoE backbones to mitigate gradient conflicts in single/multi-task learning? Please discuss more about how MixturePrior relates to Mixture-of-Experts. If time permits, could you compare against an MoE prior head under matched parameter counts?
  5. Could you compare against stronger action-conditioned priors, such as per-action output heads, FiLM, or multiplicative action modulation?

[1] Obando-Ceron, Johan, et al. "Mixtures of experts unlock parameter scaling for deep rl." arXiv preprint arXiv:2402.08609 (2024).

[2] Pu, Yuan, et al. "One model for all tasks: Leveraging efficient world models in multi-task planning." International Conference on Learning Representations. Vol. 2026. 2026.

4 comments

Thank you for the feedback on statistical evidence, baselines, and the link between theory and observed performance.

CW1, Q1 — Statistical Evidence and Robustness

Three seeds per configuration on TWISTER Boxing and Cartpole Swingup:

TaskK=1K=1: Mean ± SD95% CIK=4K=4: Mean ± SD95% CI
Boxing81.44 ± 18.22[36.19, 126.69]75.61 ± 5.94[60.85, 90.36]
Cartpole Swingup833.21 ± 5.84[818.71, 847.72]861.94 ± 9.74[837.73, 886.14]

The K=1K=1 baseline is an internal ablation, not the native prior . At fixed τ=1\tau=1 , K=4K=4 beats K=1K=1 in all four backbone–benchmark combinations, exceeds DreamerV3 on both benchmarks and stays below TWISTER, so the gains come from more components within the interface.

A 10,000-draw paired game-bootstrap on human-normalized scores (HNS) tests whether the gain rests on a few high-variance games:

StatisticEstimate95% CI
Mean HNS difference+45.16[10.17, 85.50]
Median paired per-game HNS difference+3.64[−2.20, 26.16]
IQM HNS difference+15.17[−5.57, 77.12]
Probability of improvement (PI)0.577[0.385, 0.769]
Mean HNS difference excluding Up N Down, Crazy Climber, and Krull+15.77[−5.46, 40.23]

CW2, Q3–5 — Routing Mechanism, Capacity, Prior-Head Design

Four routing variants with identical backbone, data, and budget . Action leads on both axes in Up N Down; Kangaroo reverses the order.

TaskRoutingDynamics KL ↓Joint NLL ↓Control score ↑
KangarooAction15.1922.237,540
Uniform11.6017.449,525
Constant6.5210.645,800
Train-shuffled8.9914.10989
Up N DownAction14.7825.99103,560
Uniform21.4148.258,442
Constant16.5535.504,865
Train-shuffled15.0326.0252,620

Against the parameter-matched PM-Single prior, MixturePrior wins three of six Atari games with a higher mean HNS, so matched capacity alone does not explain the gains.

Unlike MoE work on parameter scaling in value-based RL or gradient conflicts in multitask planning , MixturePrior mixes the transition prior itself: components predict alternative categorical distributions for the same latent, combined per factor by an action-only gate.

Two further parameter-matched heads on Boxing: MoE-logit and multiplicative single head,.

Prior headPrior parametersFrozen-feature Joint NLL ↓End-to-end return ↑
MixturePrior1,189,89237.440071.94
MoE-logit1,187,76040.376569.10
Multiplicative single head1,189,80841.924393.04

Q2 — The K=1 Baseline and Reproduction

The K=1K=1 configuration is an internal single-component ablation: the gate weight is identically one, so routing is inactive while the prior stays action-conditioned through the action-updated feature. On TWISTER:

PriorHead architectureParameters
MixturePrior K=1K=1Linear–ELU–linear component network394,496
Native TWISTERAffine projection525,312

The gap is, however, not specific to the K=1K=1 interface: our native-TWISTER reproduction also falls below the published value and lands close to the K=1K=1 result.

Manuscript changes. Appendix C.2, Table 7 and Section 5.4 report the routing ablation; Appendix E.8, Table 23 and E.7, Tables 21–22 add the three-seed and bootstrap analyses. Section 5.1 and Appendix B.1 recast K=1K=1 as an internal single-component ablation and give the head parameter counts; Appendix E.5, Tables 15–16 add the parameter-matched comparisons, Appendix B.4 the split–gate check, and Section 3.1 the motivation for action-based routing.

  • Thank you for the additional experiments and analyses. However, the response does not yet fully address the following concerns:
    • The authors note that “Unlike MoE work on parameter scaling in value-based RL or gradient conflicts in multitask planning, MixturePrior mixes the transition prior itself.” However, prior work such as [2] also introduces MoE into a learned dynamics model, partly for transition prediction. The current discussion therefore does not yet sufficiently clarify the fundamental distinction and advantages of MixturePrior relative to these methods. In particular, direct comparisons with closely related prior-head designs, such as MoE-logit and the multiplicative single head, remain limited to Boxing, making it difficult to draw more general conclusions. I encourage the authors to provide a deeper analysis of how MixturePrior differs from related MoE-based world models in terms of where the mixture is applied, the routing variables, and the underlying optimization motivation, and to include direct comparisons on a broader set of tasks.
    • Regarding CW2 and Q3–5, the authors state that “Action leads on both axes in Up N Down; Kangaroo reverses the order.” Does this task-dependent reversal instead suggest that the benefit of action routing is not yet consistent? The contrasting results on these two tasks are currently insufficient to establish action routing as a generally important source of MixturePrior’s performance gains. I encourage the authors to further investigate the source of this cross-task variation and to avoid drawing overly general conclusions from these results.

1. On the distinction from related MoE world models

We thank you for pointing this out and agree that our previous wording was too general. The implementation makes the distinction concrete along three axes.

(i) Where the mixture is applied. In our RSSM implementation, the K components share a single trunk on the action-updated deterministic feature and branch only in the final categorical output blocks. The mixture is taken over normalized next-state categorical distributions,

p(z∣h,a)=∑kgatek(a) pk(z∣h),p(z \mid h, a) = \sum_k \mathrm{gate}_k(a)\, p_k(z \mid h),

via log-sum-exp. It is neither a mixture of logits nor a mixture placed on the reward, termination, or observation heads. The KL losses (dyn, rep) act on this prior alone; the backbone, encoder, and policy heads remain unchanged. By contrast, M3W assigns experts across tasks or agents, and MoE-World routes state, reward, and termination predictions to task-specific experts.

(ii) Routing variable. The gate depends on the action alone,

πk=softmax(gψ(a)/τ),\pi_k = \mathrm{softmax}\big(g_\psi(a)/\tau\big),

shared across all latent factors, with the default input being the normalized action a/max⁡(1,∣a∣)a/\max(1, |a|). It is not a task id, an observation, or a state selector. Actions are conditioning variables within one transition objective, not separate tasks — precisely the setting analyzed in our Propositions 2 and 3.

(iii) Optimization motivation. The gate output layer is zero-initialized, so training starts from an exact uniform mixture, with K=1 nested in K=4. We target the cancellation of per-action corrections in a shared prior head, and Theorem 3 yields a strict descent direction for splitting plus action-dependent gating (χ≠0\chi \neq 0) even when splitting alone has zero curvature. This is a statement about the prior-fitting subproblem, not about parameter scaling or task-gradient conflicts.

Our code also implements the closely related controls you ask for: gate_input ∈ {action, uniform, constant, shuffled, state, state_action}. We agree that the MoE-logit and multiplicative-head comparisons are currently limited to Boxing and are exploratory; we will extend them to a broader task set and refrain from drawing general conclusions from them.


2. On the cross-task reversal between Up N Down and Kangaroo

We agree, and the architecture explains the reversal. The gate sees only the action (_gate_input returns the action; no deterministic state is provided). Action-based routing can therefore help only when the action carries state information: on Up N Down the decisive actions are state-specific along both axes, whereas on Kangaroo frequent actions (NOOP, FIRE) occur across many states and are a poor proxy for state. The observed task-dependence is thus consistent with the design rather than an unexplained anomaly.

More importantly, our controlled experiments show that the gain does not come from action routing. With the component bank frozen, a refitted action gate lowers held-out NLL on 5 of 10 checkpoints, is significantly worse on Kangaroo, and is indistinguishable on 4; end-to-end gates change NLL by at most 0.014 when their action inputs are shuffled. In an end-to-end four-arm ablation (action / uniform / constant / shuffled), we further found that uniform and constant are functionally the same mechanism (gate entropy ≈ ln4; max weight 0.250 vs 0.264) yet differ by 61% in held-out dynamics KL. This gap can only arise from run-to-run variance at one seed, so the ordering among the four arms is not reportable as a conclusion. All four K=4 arms, however, greatly exceed K=1.

We therefore do not claim action routing as a general source of the gain. The paper's claim rests on the matched K=1 → K=4 comparison (Table 2), where four components raise the mean in all four backbone–benchmark settings; the primary source is multiple components, with action-dependent routing being a task-dependent additional effect. We have revised the text to report the ablation as task-dependent and to avoid general conclusions from these two tasks.

Thank you for the clarifications and additional experiments. I acknowledge your revised position that the gains primarily arise from using multiple prior components, with action routing providing at most a task-dependent additional effect. While this clarification improves the presentation, it also weakens the connection between the paper’s original action-routing motivation and the empirically supported source of improvement.

Moreover, the main evidence still rests on internal K=1→K=4K=1\rightarrow K=4 comparisons. Most control results remain single-seed, the new three-seed experiments cover only two tasks, and direct comparisons with the MoE-logit and multiplicative-head designs remain limited to Boxing. Consequently, the statistical robustness and cross-task generality of the gains, as well as their advantages over closely related designs, remain insufficiently established. I therefore maintain my original rating and overall assessment.

Reviewer_9ocnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

What: The paper proposes MixturePrior, an action-router mixture of transition priors for latent world models. Why: A shared transition prior can suffer from conflicting updates across actions (and hence impacting prediction accuracy) How: The authors use multiple categorical priors to predict candidate next state distributions, combined using an action conditioned gate. The method is supported by some theoretical analysis and experiments on DreamerV3 and TWISTER across 46 tasks

Critical strengths
  • Simple approach applicable to different world-model architectures.
  • Clear theoretical motivation and analysis of conflicting action-dependent corrections.
  • Improved performance over single-component priors across all four benchmark settings.
Critical weaknesses
  • Action routing is not convincingly validated. Shuffling actions barely affects the learned gate, and a fixed mixture outperforms the learned gate on 9/10 checkpoints. An end-to-end comparison against uniform, learned action-independent, and shuffled routing may further help the paper.

  • Experimental robustness is limited. Each configuration uses only one seed, and task-wise hyperparameter selection uses evaluation results. The authors should report multiple seeds, confidence intervals, and results using a fixed or independently selected configuration.

Questions for authors

See critical Weaknesses section.

2 comments

Thank you for recognizing the theoretical motivation and the applicability of MixturePrior across world-model architectures.

CW1 — Evidence for Action Routing

We train four routing variants on Kangaroo and Up N Down:

TaskRoutingDynamics KL ↓Held-out Joint NLL ↓Control score ↑
KangarooAction15.1922.237,540
Uniform11.6017.449,525
Constant6.5210.645,800
Train-shuffled8.9914.10989
Up N DownAction14.7825.99103,560
Uniform21.4148.258,442
Constant16.5535.504,865
Train-shuffled15.0326.0252,620

On Up N Down, action-conditioned routing attains the lowest dynamics KL and held-out Joint NLL together with the highest control score. On Kangaroo the predictive ordering reverses, Uniform obtains the higher control score, and the refitted action gate of Table 5 is worse than the fitted global mixture. We therefore report the ablation as task-dependent rather than as a uniform advantage. Uniform and Constant retain the multi-component predictor while removing action dependence from the mixing weights, so Action's advantage over both on Up N Down supports the contribution of action-conditioned weighting within the common component architecture.

CW2 — Robustness and Hyperparameter Selection

The results show positive aggregate gains under common configurations, with substantial variation across tasks and evaluation statistics.

The fixed (K,τ)=(1,1)(K,\tau)=(1,1) versus (4,1)(4,1) comparison in Table 2 (p. 9) uses the same configurations across tasks and yields positive aggregate differences in all four backbone–benchmark combinations, so the aggregate improvement is not solely attributable to task-wise configuration selection.

To examine training-seed variability, we run three seeds across six configurations on TWISTER Boxing and Cartpole Swingup:

TaskK=1K=1: Mean ± SD95% CIK=4K=4: Mean ± SD95% CI
Boxing81.44 ± 18.22[36.19, 126.69]75.61 ± 5.94[60.85, 90.36]
Cartpole Swingup833.21 ± 5.84[818.71, 847.72]861.94 ± 9.74[837.73, 886.14]

The direction of the mean difference varies between tasks and the confidence intervals overlap, consistent with the task dependence of Section 5.3: neither the benefit of additional components nor the useful component count is uniform across tasks.

For the fixed DreamerV3 Atari comparison, we run 10,000 paired game-bootstrap resamples:

StatisticEstimate95% CI
Mean HNS difference+45.16[10.17, 85.50]
Median paired per-game HNS difference+3.64[−2.20, 26.16]
IQM HNS difference+15.17[−5.57, 77.12]
Task-level win fraction (PI)0.577[0.385, 0.769]
Mean HNS difference excluding Up N Down, Crazy Climber, and Krull+15.77[−5.46, 40.23]

The mean difference is supported by its confidence interval; the median, IQM, and probability-of-improvement intervals are not. Removing Up N Down, Crazy Climber, and Krull together leaves an interval spanning zero, although removing any one of them individually preserves a positive mean.

We also examine validation-based selection on DreamerV3 Battle Zone and Acrobot Swingup, selecting configurations on held-out observation NLL and fixing them before independent confirmation and control evaluation. The procedure selects (4,0.5)(4,0.5) and (4,1)(4,1), and Battle Zone's confirmation difference from the default is indistinguishable from zero, so configuration selection is separated from control-test evaluation.

Manuscript changes. Appendix C.2, Table 7 reports the four-way end-to-end routing ablation, with Section 5.4; Appendix E.8, Table 23 and E.7, Tables 21–22 add the three-seed and bootstrap analyses. Table 1 is relabelled as retrospective (p. 8), with the selection bias and the one-standard-error rule in Appendix E.2–E.3 (p. 31); Section 5.2 (pp. 7–8) qualifies cross-task generality and Section 6 (p. 10) separates our gains from the published backbones.

Reviewer_9ocnOpenReview ↗

Thanks

Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns.

The new routing experiments are helpful, although the results are still mixed. Action-conditioned routing performs well on Up N Down, but uniform routing does better on Kangaroo. This provides some evidence that action routing can be useful, but I think its advantage over simpler routing strategies is still not fully established.

The additional robustness analysis is also helpful. The fixed-configuration results help address my concern about task-wise hyperparameter selection. However, the main experiments still rely on a single seed, and the confidence intervals for several aggregate metrics overlap with zero. So I think the robustness of the improvements remains somewhat uncertain.

Overall, the response addresses some of my concerns, and I appreciate that the authors have made the limitations clearer in the revised paper. I still find the approach simple and interesting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.