acceptodds
Under review as a conference paper at ICLR 2027

Identifying Transition-Independent State Variables for Factorized World Models

Abstract

Learning a compact world model representation with factorized state variables is crucial for efficient reinforcement learning. However, existing approaches of identifiable factorization for Markov decision processes (MDPs) are either constrained to block-wise identifiability or rely on the limiting condition of control-independent factors. To extend the capability of identifiable factorization in MDP, this paper proposes Transition-Independent Factorization (TIF), a technique that can further identify factors that have dependent control influence but independent transition dynamics. We first present a theoretical analysis to establish the identifiability of controllable and transition-independent factors, followed by a practical approach to learn factorized world models from high-dimensional observations. Experiments in synthetic worlds and pixel-based grid worlds validate that TIF can accurately identify the ground-truth factors with independent transition dynamics. Experiments in variants of the MiniGrid Empty and Four-Rooms demonstrate that the resulting factorized world model representations can bring substantial benefits to the planning and policy learning in reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Factorized World Models, Identifiable Representation Learning, Model-Based Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_72mUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at learning factorised world models where each state dimension's transition is independent of the others, given the action and its own previous state. The paper presents identifiability theory for this setting (Theorem 3.1) and derives an evidence lower bound that uses the factorised transition dynamics. They evaluate their method in controlled (yet very simple) environments, where they show their world model can isolate the factors when other methods cannot. They then plug their world model into DreamerV2 and evaluate how it affects policy learning and multi-step imagination in two MiniGrid variants.

Critical strengths

I should say up front that whilst I have a lot of experience in model-based RL and world models, I am less familiar with the literature on identifiability of factorised world models.

1. Interesting and (to my knowledge) novel identifiability condition. Going from "each action affects one factor" (ACF) to "actions can affect many factors, but factors evolve independently" is a sensible extension, and the 8-action MiniGrid experiment (diagonal moves) shows nicely where ACF and SlowVAE break and TIF doesn't.

2. Well-designed controlled experiments. The experiments clearly isolate the benefit of the factorised world model. In particular, the ablations directly test the two conditions from the theory: removing transition independence (Table 3) and sweeping the rank of the action-response matrix (Fig. 5). It's good to see the theory's conditions actually tested rather than just stated.

Critical weaknesses

1. When does the factorisation assumption actually hold? And there's a gap between theory and practice. I am struggling to think of an interesting example where the main factorised assumption in this paper holds, and I can't think of a case where I'd pick this method over a more general world model. Even the paper's own environments seem to break it:

  • In FourRooms, Appendix C.2 says the walls "make the outcome of an action depend jointly on the agent's current two-dimensional position", i.e. x and y are not transition-independent. Yet this is one of only two policy-learning environments, and TIF is run with K=2.
  • In the 8-action Empty grid, what happens when a diagonal move hits a wall? If the whole move is blocked, then x's transition depends on y, so independence is violated there too.

The pixel experiments are also outside what the theory covers (the authors acknowledge "relaxed theoretical assumptions" at line 317):

  • MiniGrid states are discrete, whereas the theory assumes continuous factors, C1C^1 diffeomorphisms and positive densities.
  • A 64x64x3 image can't be a diffeomorphic image of a 2D state.
  • Each RSSM branch has a 1D stochastic state plus a 32D deterministic state, and the decoder sees all 33K dims (Table 4). So the learned "factors" aren't the scalars the theorem talks about.

On top of that, Theorem 3.1 assumes the learned and true trajectory distributions match exactly (Eq. 3.3), but TIF optimises an ELBO with a factorised posterior. The authors themselves note (line 212) that this doesn't guarantee identifiability. So the theory really only gets tested in the synthetic environment.

2. Limited empirical scope and significance. The policy-learning results (Fig. 7) are in very simple environments. At this point I'd expect to see harder ones, e.g. the environments from the DreamerV2 paper. Does the factorised world model keep up in those environments and then offer a benefit where the factorisation does hold? Or does the factorisation hurt performance when the state doesn't satisfy the assumption? I'm also not convinced by the baseline comparison. The capacity-matched DreamerV2 is tiny (K=2 stochastic, 64 deterministic dims), and DreamerV2 (large) gets to similar final returns. So the gain might just be sample efficiency at very small model sizes.

Minor:

  • The first line of the abstract says "Learning a compact world model representation with factorised state variables is crucial for efficient reinforcement learning." Is this the case? It's a bold claim for the first sentence.
  • ∂1\partial_1 is used in Eq. 3.2 but only defined in Appendix A. Please define it in the main text.
  • Line 196 refers to "the objective considered in Theorem 3.1", but there's no objective in Theorem 3.1, only the distribution-matching condition (Eq. 3.3). Please clarify (see also Weakness 1).
  • Eq. A.2 typo: it should be p~i(zi,t∣zi,t−1,at−1)\tilde{p}_i(z_{i,t} \mid z_{i,t-1}, a_{t-1}), not zi,t−1∣zi,t−1z_{i,t-1} \mid z_{i,t-1}.
  • The text/labels in Fig. 7 are too small.
Questions for authors
  1. Can the authors give an example of interest to the community where the transition-independence assumption actually holds?
  2. FourRooms seems to violate transition independence (App. C.2). How do you reconcile this with using TIF there? Does TIF still recover x/y factors in FourRooms, and if so, what does that say about how much the theory explains the results?
  3. KK is set to the number of ground truth factors (App. D.2). So I'm wondering what happens when KK is misspecified? Surely this will matter a lot for practical use?
  4. Factors are assumed scalar (Si⊆R\mathcal{S}_i \subseteq \mathbb{R}). Does this extend to vector-valued factors (e.g. object pose)?
  5. Results on harder environments (e.g. DreamerV2 benchmarks) would go a long way. Showing TIF holds up when the assumption is violated would address my main concern.
Reviewer_UbX5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose to factorize world models assuming several independent factors in the state space. The authors show that under a certain parameterezation the learned independent factors do identify the real ones. The authors show identifiability of independent factors in practice by modifying dreamerv2 to have multiple independent latent states with independent transition dynamics and the encoder acting on a shared observation, action, and the previous state. The authors conduct experiments in several MDPs varying task difficulty and show that their approach can successfully identify real environment factors in a self-supervised fashion.

Critical strengths
  • Sound idea. The authors approach an important problem of disentangling entangled dynamics which has many real world applications.
  • Theoretical support. The authors support their findings with a theoretical result that gives an insight on what conditions need to be met for identifiability. This helps understanding the problem.
Critical weaknesses
  • The central weakness of the paper is the parameterezation of the graphical model that the authors chose to demonstrate identifiability in practice. Since the theoretical result assumes that the factors are independent, the key challenge in building the practical result on top is to ensure the learned factors are truly independent. With the proposed parameterezation, there is no incentive for the world model to learn the independent factors. The authors therefore sometimes have to resort to very strong inductive biases in the encoder (e.g. parameterizing it as RealNVP which is rather an unusual choice of the encoder in world modeling literature). The main reason that there is no such incentive is that different transition models are simply trained with independent KL regularization terms and the encoder works on a shared input (often employing parameter shaing to encode all latents at once). Therefore, a particular case when all latent factors converged to the same latent variable is a valid minimizer of the variational inference objective of dreamer. Another example is when for example a single factor converged to a meaningful latent variable explaining the entire dynamics but other factors collapsed (with the encoder predicting approximately the same value for them). These two examples highlight the fact that in a challenging and realistic environment we might need to have extra assumptions about the model (for example, a stronger inductive bias). Similar kind of collapses occur very often when training world model on challenging environments. A viable solution to this problem is to incentivize competition between factors for a given observation in how much each of them contributes to the reconstruction and future prediction. Note that this does not require to abandon the independent factorization (equation 2.1) and instead can be implemented for example as another (soft) categorical latent variable that conditions each of the independent factors: if the "soft-selector" is also trained with variational inference then the encoder for it will be incentivized to provide a factors weighted selection for each individual observation and the KL term for the soft-selection latent variable will increase exploration of factors.
  • Overall, I found the paper hard to follow.

I think the two outlined concerns are serious issues which justifies my score.

Questions for authors

Do baselines models use the same RealNVP encoder (same architecture)? If no, then the comparison between models is not apples-to-apples and this component could have contributed to the improvement instead of factorization.

Reviewer_YTA7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces TIF, a factorization algorithm for MDPs that assumes “transition-independent factors”. That is, each factor is an independent system: a trivial DBN where factors do not interact. Under this assumption they derived an identifiability result based on parametric assumptions (exponential families, based on ICA proofs) and proposed an VAE-based algorithm that operationalizes this idea. They validate their algorithm empirically in a controlled grid-world with non-linear observation mixing and scale it to a pixel environment (Minigrid single’s and four room).

Critical strengths

Though the experiments were limited to very small and simple domains (few factors to identify, few actions), the authors were thorough with their baseline execution.

Critical weaknesses
  • This paper proposes an algorithm based on existing techniques in the literature using a stronger assumption than previous work. They assume that subsystems do not interact, the source of a lot of the headaches when it comes to disentanglement: it is the dynamical interactions that create the hard entanglements neural networks have a hard time separating.
  • The paper fails to position the contributions clearly and accurately with respect to previous work. In their framing they seem to be proposing a better method but in the end, TIF is just a derivative from previous work by making assumptions stronger and limiting its general applicability. Unfortunately, I don’t believe this is the expected level of novelty expected from submissions to ICLR. I would encourage the authors to build upon previous work in order to make them better and applicable to more problems, instead of less.
  • Figure 1 implies that ACF assumes that actions have a one-to-one relation to factors, which is not accurate. This would correspond to stronger assumption than the original paper suggests.
Questions for authors

I don't have any questions about this paper

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.