acceptodds
Under review as a conference paper at ICLR 2027

BoundaryWorld: Decision-Regime-Sufficient World Models for Reliable Long-Horizon Planning

Abstract

World models are often optimized for state or observation fidelity, yet a planner does not need every detail of a predicted future. For a fixed downstream decision problem, the consequential distinctions are those that can change action rankings or feasibility. We formalize this requirement with Decision-Regime Trajectory Sufficiency (DRTS), which measures whether a learned model preserves the action-conditioned joint distribution of future decision regimes induced by that problem. A regime-abstraction residual measures the utility loss induced by representing a trajectory through its regime sequence and yields a continuous-plan regret bound that separates abstraction error from predictive error. BOUNDARYWORLD adds a rollout-conditioned joint regime-trajectory objective to a standard latent world model while retaining the same CEM search mechanics. An analytic diagnostic illustrates the distinction; on Safety-Gymnasium and MetaDrive, BoundaryWorld reduces \(H=20\) planning regret by 19.3% and 21.7%, respectively, relative to the strongest controlled alternative in each domain.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, decision sufficiency, state abstraction, long-horizon planning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YZYQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to train world models to preserve information relevant to planning on downstream tasks. Specifically, it introduces a quantity called Decision-Regime Trajectory Sufficiency (DRTS) to measure whether a model preserves the action-conditioned distribution of future decision-regime sequences. The proposed method, BoundaryWorld, adds a regime-prediction loss to a latent world model loss and propagates its gradients through the encoder and dynamics. The paper reports improved regret and return on Safety-Gymnasium and MetaDrive.

Critical strengths
  • The paper addresses the meaningful problem that low state-prediction error does not necessarily imply good planning decisions. The distinction between predicting the regime sequence accurately and choosing a regime abstraction that preserves utility is useful.

  • The controlled experiments include several relevant comparisons, particularly marginal regime prediction, direct sequence prediction without latent dynamics, and variants that remove gradients through the dynamics. Ablations on sharing data, counterfactual branches, and CEM settings across methods help reveal the effects of the training objectives.

Critical weaknesses
  • The main motivation is to predict the joint regime sequence, but the planning score in Eq. (16) only needs per-step regime probabilities. It is therefore unclear whether modeling correlations across time explains the gains.

  • The regime construction is still difficult to follow. The paper should explain exactly how states receive regime labels and how each regime is assigned a utility and cost.

  • Several baselines use a regime-prediction head fitted after training. This is a useful controlled comparison, but it would also help to compare against baselines that plan directly with their learned value or cost predictions.

Questions for authors
  • Why is joint sequence prediction needed for the additive planning score in Eq. (16)?

  • How exactly are regime labels and their utility/cost scores constructed?

  • How do the baselines perform when planning with their own value or cost predictions?

Reviewer_koc8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper claims that a world model for planning must keep the distribution of future decision regimes, not full state detail. The authors define this property as Decision-Regime Trajectory Sufficiency (DRTS). DRTS is a total-variation bound on action-conditioned regime sequences. A residual ηD\eta_D measures the error that the regime abstraction adds. Together, the two terms bound planning regret. BoundaryWorld trains an autoregressive regime head on latent rollouts. The regime loss also updates the encoder and the dynamics. On Safety-Gymnasium and MetaDrive, BoundaryWorld gives lower planning regret than controlled baselines with the same backbone and planner.

Critical strengths
  1. The training objective is simple and useful. The joint regime loss through predicted rollouts performs better than marginal labels, post-hoc heads, and stopped gradients (Table 5). The paper uses known results on value equivalence and bisimulation well.
  2. The paper is clear. The controlled protocol is careful.
Critical weaknesses
  1. The biggest issue with this work is its position relative to Delgrange, Nowé, and Pérez (AAAI 2022, ICLR 2023). Both works solve the same core problem. Each learns a latent model of an unknown environment that must keep the behaviour of a labelling, not the full state. In this paper, the labelling is the regime map ρD\rho_D, and DRTS is a total-variation distance between label traces. Delgrange et al. use atomic propositions as labels and bound a bisimulation distance between the real and latent models. Their bound then controls reachability and safety objectives over the labels, as Proposition 2 does for the planning objective. Their representation-quality term also matches ηD\eta_D. The main difference is that Delgrange et al. obtain PAC bounds from local losses, but this paper assumes ϵ\epsilon and does not certify it. Delgrange et al. also fix one policy, but this paper must cover many candidate plans. This difference is a strength of the paper, but the paper does not state it. I think this paper would strongly benefit from a discussion of DRTS relative to this work.
  2. A large body of work on abstraction refinement is also relevant, for example probabilistic CEGAR (Hermanns et al., CAV 2008; Kattenbelt et al., FMSD 2010). These methods refine an abstraction in a loop until it is sufficient for the property. The coarse, medium, and fine signatures in Section 5.4 are a manual version of this refinement. An automatic refinement loop can improve the method.
  3. Propositions 1 and 2 are standard results: the dual form
Questions for authors
  1. Can a PAC certificate on local losses, as in Delgrange et al., replace the empirical η^D\hat\eta_D and the assumed ϵ\epsilon? A certificate under μplan\mu_{\text{plan}} holds in expectation, but Definition 2 and Proposition 2 use a supremum over plans, and CEM selects plans where the model error is large. How do you close this gap? A union bound over the finite candidate bank at each state, with simulator forking, is one possible route.
  2. Did you consider counterexample-guided refinement of the regime signature? Simulator forking already gives the check that such a loop needs.
Reviewer_6YemOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes a world-model learning method that prioritises distinctions relevant to planning.

Critical strengths

The paper proposes DRTS that measures agreement between the true and learned action-conditioned distributions of future decision-regime sequences. The authors introduce an abstraction residual and derive a regret bound separating abstraction error from distributional prediction error. Experiments on Safety-Gymnasium and MetaDrive report lower planning regret and higher return than the controlled alternatives.

Critical weaknesses

This paper has a few major issues:

  1. The motivation is not clearly explained. Figure 1 and 2 need significant improvement to more clearly explain the issues. They are suspected to be generated by AI tool without careful designing.
  2. Section 4.4 refers readers to Appendix B for the environment-specific construction of uDu_D and cDc_D, but Appendix B only describes them as deterministic lookups derived from the task specification. It is hard to reproduce the method.
  3. The implemented planning objective does not establish the need for a joint regime-trajectory distribution.
  4. The experiments do not establish that preserving joint temporal dependence is responsible for the planning improvements.

There are some minor issues: 5. Section 4.2 introduces zt=Eϕ(ht)z_t=E_\phi(h_t) without defining hth_t. “Viol.” and “Return” need their units, normalisation and episode/step aggregation. 6. The “Diagnostic MDP” is described through nuisance and critical coordinates but lacks a complete transition and reward specification. Either provide these details or describe it as an illustrative construction. 7. Figure labels are difficult to read at normal page size. Figures 2 and 3 contain particularly small text. 8. The horizon-sweep training protocol should be stated directly. It is unclear whether models are retrained at each horizon or evaluated beyond their training horizon. This is especially relevant because the DRT loss in Equation (15) is not length-normalised, whereas the base losses are. 9. There are small formatting and wording problems.

Questions for authors

Please see the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.