- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks at learning factorised world models where each state dimension's transition is independent of the others, given the action and its own previous state. The paper presents identifiability theory for this setting (Theorem 3.1) and derives an evidence lower bound that uses the factorised transition dynamics. They evaluate their method in controlled (yet very simple) environments, where they show their world model can isolate the factors when other methods cannot. They then plug their world model into DreamerV2 and evaluate how it affects policy learning and multi-step imagination in two MiniGrid variants.
Critical strengths
I should say up front that whilst I have a lot of experience in model-based RL and world models, I am less familiar with the literature on identifiability of factorised world models.
1. Interesting and (to my knowledge) novel identifiability condition. Going from "each action affects one factor" (ACF) to "actions can affect many factors, but factors evolve independently" is a sensible extension, and the 8-action MiniGrid experiment (diagonal moves) shows nicely where ACF and SlowVAE break and TIF doesn't.
2. Well-designed controlled experiments. The experiments clearly isolate the benefit of the factorised world model. In particular, the ablations directly test the two conditions from the theory: removing transition independence (Table 3) and sweeping the rank of the action-response matrix (Fig. 5). It's good to see the theory's conditions actually tested rather than just stated.
Critical weaknesses
1. When does the factorisation assumption actually hold? And there's a gap between theory and practice. I am struggling to think of an interesting example where the main factorised assumption in this paper holds, and I can't think of a case where I'd pick this method over a more general world model. Even the paper's own environments seem to break it:
- In FourRooms, Appendix C.2 says the walls "make the outcome of an action depend jointly on the agent's current two-dimensional position", i.e. x and y are not transition-independent. Yet this is one of only two policy-learning environments, and TIF is run with K=2.
- In the 8-action Empty grid, what happens when a diagonal move hits a wall? If the whole move is blocked, then x's transition depends on y, so independence is violated there too.
The pixel experiments are also outside what the theory covers (the authors acknowledge "relaxed theoretical assumptions" at line 317):
- MiniGrid states are discrete, whereas the theory assumes continuous factors, diffeomorphisms and positive densities.
- A 64x64x3 image can't be a diffeomorphic image of a 2D state.
- Each RSSM branch has a 1D stochastic state plus a 32D deterministic state, and the decoder sees all 33K dims (Table 4). So the learned "factors" aren't the scalars the theorem talks about.
On top of that, Theorem 3.1 assumes the learned and true trajectory distributions match exactly (Eq. 3.3), but TIF optimises an ELBO with a factorised posterior. The authors themselves note (line 212) that this doesn't guarantee identifiability. So the theory really only gets tested in the synthetic environment.
2. Limited empirical scope and significance. The policy-learning results (Fig. 7) are in very simple environments. At this point I'd expect to see harder ones, e.g. the environments from the DreamerV2 paper. Does the factorised world model keep up in those environments and then offer a benefit where the factorisation does hold? Or does the factorisation hurt performance when the state doesn't satisfy the assumption? I'm also not convinced by the baseline comparison. The capacity-matched DreamerV2 is tiny (K=2 stochastic, 64 deterministic dims), and DreamerV2 (large) gets to similar final returns. So the gain might just be sample efficiency at very small model sizes.
Minor:
- The first line of the abstract says "Learning a compact world model representation with factorised state variables is crucial for efficient reinforcement learning." Is this the case? It's a bold claim for the first sentence.
- is used in Eq. 3.2 but only defined in Appendix A. Please define it in the main text.
- Line 196 refers to "the objective considered in Theorem 3.1", but there's no objective in Theorem 3.1, only the distribution-matching condition (Eq. 3.3). Please clarify (see also Weakness 1).
- Eq. A.2 typo: it should be , not .
- The text/labels in Fig. 7 are too small.
Questions for authors
- Can the authors give an example of interest to the community where the transition-independence assumption actually holds?
- FourRooms seems to violate transition independence (App. C.2). How do you reconcile this with using TIF there? Does TIF still recover x/y factors in FourRooms, and if so, what does that say about how much the theory explains the results?
- is set to the number of ground truth factors (App. D.2). So I'm wondering what happens when is misspecified? Surely this will matter a lot for practical use?
- Factors are assumed scalar (). Does this extend to vector-valued factors (e.g. object pose)?
- Results on harder environments (e.g. DreamerV2 benchmarks) would go a long way. Showing TIF holds up when the assumption is violated would address my main concern.