acceptodds
Under review as a conference paper at ICLR 2027

Split the World: Are Regional Dynamics Better Than Global Dynamics for Latent World Models?

Abstract

Latent world models, due to their efficiency, have become one of the main paradigms for world modeling. In latent space, they learn from data the dynamics governing how an agent’s actions affect the environment and use these dynamics to predict the environment state after executing actions. Traditionally, most existing latent world models learn a single global dynamics model from the full task dataset, despite possible differences in dynamics across different regions of the task state space. Existing approaches have shown that regional dynamics can be useful. However, either global or regional dynamics are used by default, without a clear and general theory for selecting between them. Motivated by this, we study when regional dynamics are better than global dynamics and formulate this question as a dynamics-selection problem. To answer this question, we propose the Worst-Case Partition Gain criterion, which is based on Wasserstein distances between regional response distributions. Using this criterion, we establish a necessary-and-sufficient condition for selecting the better dynamics under reasonable assumptions. Furthermore, we develop a dynamics-selection algorithm based on this condition, with a calibrated threshold held fixed across different modeling configurations. We evaluate our method using, to our knowledge, the first systematic experimental framework for dynamics selection, comparing 22 methods on transfer across different numbers of regions and world-model backbones. Experimental results demonstrate state-of-the-art selection performance: our method correctly selects the better dynamics in all 8 region-count transfer configurations and in 18 of 20 experimental configurations overall.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Latent World Models, Dynamics Selection, Regional Dynamics, Model Specialization, Long-Horizon Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6PUZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether a latent world model should use a globally shared predictor or separate regional predictors for a given latent-space partition. The main experiments use spectral clustering of frozen representations and regional affine regression to estimate centered action responses. Worst-Case Partition Gain (WCPG) uses the minimum normalized squared 2-Wasserstein distance between regional response summaries and a frozen threshold to select a branch. A conditional theoretical connection to planning success is established under deviation budgets and a sufficient margin. Both branches are post-trained and evaluated across four tasks and two backbones. WCPG achieves 8/8 correct region-count-transfer selections and 10/12 correct backbone-transfer selections, yielding 90% overall and 93.75% balanced accuracy.

Critical strengths
  1. A meaningful problem with an interpretable comparison. Sharing frozen representations and planning protocols reduces representation-related confounding and helps isolate the trade-off between parameter sharing and data splitting.

  2. A simple criterion with promising cross-region-count results. WCPG can be computed without fully training both target branches. Its frozen threshold achieves 8/8 correct selections on a cross-K set containing four Regional and four Global winners, with supporting matched aggregation ablations.

Critical weaknesses
  1. Regional modeling partitions samples through latent-state clustering and fits separate predictors. However, multimodality of the pooled data does not necessarily require regional models: a sufficiently expressive state-conditioned global model may capture the same local dynamics. The experiments do not sufficiently distinguish benefits of dynamical decomposition from capacity or optimization limitations of the chosen global predictor, and matching component architectures does not equalize total capacity. The evidence supports an empirical criterion for particular model families and partitions more directly than a broadly applicable selection principle.

  2. Appendix A.1’s 21 alternatives are author-constructed statistical selectors. This is reasonable, but a cost-explicit comparison against direct validation-prediction-error-based selection is missing. Calibration uses only three retained configurations, and testing reuses the same four tasks. All 12 backbone-transfer cases favor Global, where Always Global achieves 100%, exceeding WCPG’s 83.33%; recognition of Regional winners under backbone transfer is therefore untested.

Questions for authors
  1. Do Regional gains persist after controlling total capacity and training budget? Can existing evidence distinguish benefits of dynamical decomposition from expressiveness or optimization limitations of the global predictor, and delimit the threshold’s model-family dependence?

  2. Is there a direct baseline using limited trial training followed by validation-prediction-error-based selection? What is WCPG’s accuracy–compute trade-off after accounting for partitioning, response estimation, and candidate-training costs?

  3. How stable are the success gaps and selector ranking under the paired, nested evaluation? Do existing results support transfer to uncalibrated tasks or recognition of Regional winners under another backbone?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 6PUZ (Q1a)

We were impressed by the expertise you brought to your review. We feel fortunate that our paper received such a careful and thorough reading. Thank you for the time and effort you devoted to the review and for your expert advice. To address your questions, we conducted a series of follow-up experiments. Their results have substantially strengthened our work. We will include all these follow-up experiments in the revised manuscript. Your questions cover the concerns raised in the weaknesses section of your review, so we address those concerns in our answers below.

Q1a. Do Regional gains persist after controlling total capacity and training budget?

A1a. To answer this question, we ran an additional experiment matching total trainable capacity and training budget. We find that regional gains persist after controlling total trainable capacity and training budget measured by sample exposure. On both tested LeWM tasks, spectral Regional retained a higher mean success rate than Global-wide: +2.71 pp on TwoRoom (61.38% vs. 58.67%) and +0.76 pp on Cube (49.69% vs. 48.93%). Global-wide matched the three experts’ combined trainable parameter count within 0.003%. It trained for 50 epochs on the full dataset. Each expert trained for 50 epochs on its own subset, giving both branches the same nominal sample exposure.

AuthorsOpenReview ↗

Response to Reviewer 6PUZ (Q1b)

Q1b. Can existing evidence distinguish benefits of dynamical decomposition from expressiveness or optimization limitations of the global predictor?

A1b. We ran an additional random-partition control. Yes. The random-partition control distinguishes two gains in the original comparison: the additional gain from choosing a spectral split with the same experts, and the gain from using regional predictors even under a random split. TwoRoom showed both gains, Cube only the second, Reacher only the first, and PushT neither.

The two benefits are differences in mean success rates.

Regional−Random=Benefit 1.\mathrm{Regional}-\mathrm{Random}=\mathrm{Benefit}\ 1.

Random−Global=Benefit 2.\mathrm{Random}-\mathrm{Global}=\mathrm{Benefit}\ 2.

TaskBenefit 1Benefit 2WCPG correct
TwoRoom+2.05 pp+1.20 ppYes
Cube−1.11 pp+2.67 ppYes
Reacher+0.71 pp−1.47 ppYes
PushT−1.02 pp−0.80 ppYes

We evaluated all three branches on the same 250 paired instances per task at a 50-step horizon. Both Regional variants used three experts and the same training protocol. The first contrast isolates partition choice at fixed capacity. The second measures the gain over Global under a random split, without separately attributing that gain to Global’s expressiveness or optimization.

AuthorsOpenReview ↗

Response to Reviewer 6PUZ (Q1c)

Q1c. Can existing evidence delimit the threshold’s model-family dependence?

A1c. Theorem 3 delimits when a shared threshold applies.

Within the stated comparison class, the same theoretical threshold gives the correct selection whenever the following margin condition holds.

τ⋆=2κ,∣b(Π)−τ⋆∣>ϵ(c).\tau_\star=2\kappa,\qquad |b(\Pi)-\tau_\star|>\epsilon(c).

When all configurations satisfy these conditions, the same threshold can be reused across them. Changing the predictor family can change the cost of specialization and hence the required margin (Appendix G.3, Eq. 93 and G.6, Eq. 130). On Sub-JEPA, WCPG correctly selected Global in 10/12 configurations where Global won (Table D.1) and Regional in 2/2 configurations where Regional won (A3d), for a combined accuracy of 12/14 (85.71%).

Reviewer_JiowOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies under what condition a regional dynamics model outperform global dynamics model, given an encoder and a latent state partition. It formulates this into the dynamics-selection problem, and propose Worst-Case Partition Gain criterion which measures the difference in dynamics between the most similar pair of regions. Empirical results demonstrate that the proposed criterion can effectively select the better dynamics-modeling approach.

Critical strengths

The question is worth studying, as the world often contains relatively independent components that may benefit from separate modeling.

Critical weaknesses
  • The discussion omits relevant work on factored models, particularly factored Markov decision processes (MDPs). Clarifying the relationship to these approaches would help situate the contribution.
  • The proposed idea appears applicable to state-space models more broadly. What makes latent world model special, and which aspects of the method specifically depend on latent representations are not explained.
  • The method assumes a given partition, although partition quality appears central to its effectiveness. Because learned latent spaces are often difficult to interpret, constructing a meaningful partition may itself be challenging. The paper should clarify how this dependence affects the method’s applicability.
Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Jiow(W1)

Thank you for taking the time to review our paper. We appreciate your recognition that the question of when regional dynamics models improve on a global model is worth studying.

W1. The discussion omits relevant work on factored models, particularly factored Markov decision processes (MDPs). Clarifying the relationship to these approaches would help situate the contribution.

A1. We already discuss VCD (Lei et al., 2023) in Section 2 of the original submission. VCD explicitly factorizes the latent transition model into causal mechanisms for reuse across environments. Our paper asks whether, given a partition of the latent space, regional dynamics models outperform a global dynamics model. These questions are orthogonal. Factorization concerns how a predictor represents the dynamics internally. Our selection criterion asks whether to use one predictor across all regions or a separate predictor for each region. Either choice can accommodate a factored model. We would welcome a reference to any work on factored MDPs that addresses this selection problem directly.

AuthorsOpenReview ↗

Response to Reviewer Jiow(W2)

W2. The proposed idea appears applicable to state-space models more broadly. What makes latent world model special, and which aspects of the method specifically depend on latent representations are not explained.

A2. There may be a misunderstanding about how our method uses the latent representation. The original submission already explains why the representation matters. Section 3.1 states,

Because planning takes place in latent space, the dynamics differences used for selection need to remain distinguishable there.

Algorithm 1 shows exactly where the encoder enters the computation: within each region, we regress the latent displacement Δz=h(s′)−h(s)\Delta z = h(s') - h(s) on the centered action. Theorem 1 gives a sufficient condition under which states with sufficiently different action responses remain separated in the latent space.

More importantly, our paper studies action-conditioned dynamics for planning. State-space models in general need not have an action input, whereas WCPG defines each region’s response through the variation in latent displacement induced by actions.

AuthorsOpenReview ↗

Response to Reviewer Jiow(W3)

W3. The method assumes a given partition, although partition quality appears central to its effectiveness. Because learned latent spaces are often difficult to interpret, constructing a meaningful partition may itself be challenging. The paper should clarify how this dependence affects the method’s applicability.

A3. There may be a misunderstanding about what we assume. We ask users to supply a candidate partition. Our method then assesses whether it is useful. We agree that it can be difficult to tell whether a partition is useful just by looking at the latent space. This is exactly the question our paper asks: given a candidate partition, is it better to train a predictor for each region or to train one predictor on all the data? The user does not need to know the answer in advance. If the proposed split hurts planning performance, WCPG should select Global.

Our main experiments use spectral clustering to generate candidate partitions automatically (Section 4.1 and Appendix E.6). When we varied the number of regions in Section 4.2, Regional performed better in four configurations and Global in the other four. WCPG selected the branch with the higher observed mean success rate in all eight (Table C.1). We also compared five partitioning methods for stability (Appendix E.1, Table E.2) and tested WCPG on K-means++ and random Voronoi partitions, where it selected the observed winner in six of eight configurations (Appendix E.2).

A user can also start with regions that are easier to define in the original environment. In TwoRoom, for example, the left room, doorway, and right room have clear physical meanings. These labels can be used in latent space if the encoder preserves enough information to distinguish them. We tested this directly in Appendix F: a linear probe recovered the three-region labels from frozen LeWM representations with 99.67% test macro-F1, and the five-region labels with 98.01% (Table F.1). This gives users a practical way to specify candidate regions without having to interpret individual latent coordinates.

Reviewer_DhtZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the problem of choosing (without training) global dynamic predictor vs separate regional predictors for a latent world model. They propose a new Worst-case partition gain score that measures the smallest normalized wasserstein distance between regional distributions. Using this score, the authors propose a calibrated threshold for selection. This method was evaluated on four tasks on two world models and found the method selects the higher success branch in most cases.

Critical strengths
  • Paper addresses a core practical problem! Also, paper does comparisons with shared encoder and matched planning protocol to understand the effect of dynamics specialization.
  • The proposed method can be computed without training and provides better branch in most of the configs they report.
Critical weaknesses
  • Limited evidence of generalization. The evaulation covered only 4 tasks, needs more paired uncertainty estimates.
  • The cutoff value is chosen from limited examples, so its not clear if the calibration step is better than other alternatives.
Questions for authors
  • Sensitivity of the results to cutoff value?
  • Please add confidence intervals, paired intervals to understsand if the results are statistically reliable
4 comments
AuthorsOpenReview ↗

Response to Reviewer DhtZ(W1a)

Thank you for your thoughtful review and positive assessment of our work. We especially appreciate your suggestion to report paired confidence intervals. It helped us show how precisely we could estimate the success gap in each configuration. We address the overlapping questions on cutoff sensitivity and paired uncertainty in W2 and W1b, respectively, where these issues first arise.

Critical Weaknesses

W1a. Limited evidence of generalization. The evaluation covered only 4 tasks.

A1a. To address this weakness, we ran an additional transfer experiment on two uncalibrated tasks, DMC Finger and OGBench Scene. WCPG correctly selected Global on Finger and Regional on OGBench. These tasks cover a large and a small dataset. We used the same K=4K = 4 spectral partitioning setup and applied WCPG without recalibration on either task.

TaskWCPG scoreSelectedGlobal successRegional successCorrect
DMC Finger (turn_easy)0.3488Global61.73%59.16%Yes
OGBench Scene (visual-scene-play-v0)0.5946Regional33.33%34.27%Yes
AuthorsOpenReview ↗

Response to Reviewer DhtZ(W1b)

W1b. Needs more paired uncertainty estimates.

A1b. We used 50,000 paired bootstrap replicates to estimate 95% confidence intervals for the Regional–Global planning success-rate gap in all 20 configurations. The intervals are reported in the table below and will be included in the revised paper. In the 18 configurations originally reported as correct selections, the selected branch had a strictly higher success rate in 57.6% to 98.3% of paired bootstrap replicates across configurations.

This variation in how consistently the selected branch won supports the near-tie analysis in our manuscript. The paired intervals ranged in width from 2.40 to 9.15 percentage points, and all 20 overlapped the ±0.4\pm 0.4 percentage-point near-tie band. After excluding the four configurations with observed mean gaps in that band, the same cutoff selected the observed winner in 15/16 configurations (93.75%) (Table 2 and Appendix E.3, Table E.4).

Theorem 3 (Eq. 7) gives a sufficient margin for correct selection under its assumptions: ∣b(Π)−τ⋆∣>ϵ(c)|b(\Pi)-\tau_\star|>\epsilon(c). As the paper puts it, “This threshold is shared within the stated comparison class; the required margin may vary.”

We held the selector and cutoff fixed for all 50,000 bootstrap replicates. Both branches used the same resampled training seeds and evaluation blocks. We averaged the three Regional partition seeds before resampling.

Paired results for all 20 configurations. Gap is Regional minus Global mean long-horizon success, in percentage points. Support is the share of paired bootstrap replicates in which the selected branch had strictly higher success. Ties support neither branch.

Let p^R\hat p_R and p^G\hat p_G denote the observed mean success rates of Regional and Global. Let p^S(b)\hat p_S^{(b)} and p^O(b)\hat p_O^{(b)} denote the success rates of the selected and other branches in paired bootstrap replicate bb. All success rates are proportions. Let B=50,000B=50{,}000 be the number of paired bootstrap replicates.

\mathrm{Support}=\frac{100}{B}\sum_{b=1}^{B}\mathbf{1}\left[\hat p_S^{(b)}>\hat p_O^{(b)}\right].$$ Selection is marked Success when the selected branch has the higher observed mean success rate. | Backbone | Task | K | Selected branch | Gap (pp) | Paired 95% CI (pp) | Support | Selection | | -------- | ------- | - | --------------- | -------- | ------------------ | ------- | --------- | | LeWM | TwoRoom | 2 | Regional | +1.20 | [−2.22, +3.73] | 78.4% | Success | | LeWM | TwoRoom | 3 | Regional | +3.24 | [−0.09, +5.96] | 97.0% | Success | | LeWM | PushT | 2 | Global | −0.67 | [−4.09, +2.76] | 67.0% | Success | | LeWM | PushT | 3 | Global | −1.82 | [−3.87, +0.80] | 91.9% | Success | | LeWM | Reacher | 2 | Global | −1.42 | [−4.84, +1.47] | 83.8% | Success | | LeWM | Reacher | 3 | Global | −0.76 | [−4.67, +4.09] | 66.1% | Success | | LeWM | Cube | 2 | Regional | +1.11 | [−1.47, +3.47] | 83.9% | Success | | LeWM | Cube | 3 | Regional | +1.56 | [−1.47, +4.84] | 82.5% | Success | | Sub-JEPA | TwoRoom | 2 | Global | −0.13 | [−1.47, +1.51] | 60.1% | Success | | Sub-JEPA | TwoRoom | 3 | Regional | −0.27 | [−2.13, +1.42] | 34.5% | Failure | | Sub-JEPA | TwoRoom | 4 | Regional | −0.80 | [−1.91, +0.49] | 8.4% | Failure | | Sub-JEPA | PushT | 2 | Global | −1.64 | [−4.40, +1.20] | 87.3% | Success | | Sub-JEPA | PushT | 3 | Global | −2.31 | [−4.84, −0.18] | 98.3% | Success | | Sub-JEPA | PushT | 4 | Global | −2.93 | [−6.44, +0.13] | 96.8% | Success | | Sub-JEPA | Reacher | 2 | Global | −1.96 | [−7.02, +2.13] | 84.4% | Success | | Sub-JEPA | Reacher | 3 | Global | −1.73 | [−5.60, +1.64] | 84.5% | Success | | Sub-JEPA | Reacher | 4 | Global | −0.31 | [−4.40, +3.33] | 57.6% | Success | | Sub-JEPA | Cube | 2 | Global | −0.80 | [−4.22, +2.00] | 67.1% | Success | | Sub-JEPA | Cube | 3 | Global | −1.07 | [−4.22, +2.13] | 74.0% | Success | | Sub-JEPA | Cube | 4 | Global | −0.40 | [−2.80, +1.87] | 62.7% | Success |
AuthorsOpenReview ↗

Response to Reviewer DhtZ(W2 & Q)

W2. The cutoff value is chosen from limited examples, so it's not clear if the calibration step is better than other alternatives.

A2. Selection accuracy was robust to modest cutoff changes across the four tasks. The calibrated cutoff (0.509) gave 18/20 correct selections. Cutoffs of 0.54 and 0.55 gave 19/20, while lowering the cutoff to 0.48 reduced accuracy to 14/20.

WCPG thresholdRegion-count transfer (LeWM)Backbone transfer (Sub-JEPA)Overall test accuracy
0.486/88/1214/20
0.508/89/1217/20
0.509 (calibrated)8/810/1218/20
0.528/810/1218/20
0.548/811/1219/20
0.558/811/1219/20
0.567/811/1218/20

Questions for Authors

Q1. Sensitivity of the results to cutoff value?

Addressed in our response to W2 above.

Q2. Please add confidence intervals, paired intervals to understand if the results are statistically reliable.

Addressed in our response to W1b above.

AuthorsOpenReview ↗

Follow-up to Reviewer DhtZ

Dear Reviewer DhtZ,

We appreciate your positive assessment of our work and your constructive suggestions. In our response on November 14, we provided additional experiments on two new tasks, paired confidence intervals for all 20 configurations, and a cutoff sensitivity analysis to address your concerns.

As the discussion phase is approaching its end, we would be grateful if you could let us know whether these additional results have sufficiently addressed your concerns. Your feedback would help us understand whether any points still need clarification before we incorporate these additions into the revised manuscript.

Thank you for your time and consideration. We look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.