acceptodds
Under review as a conference paper at ICLR 2027

COMB: A Factorized Object-Centric World Model, and What Its Structure Buys a Policy

Abstract

Object-centric world models decompose scenes into entities, but each entity is typically represented by a single unstructured embedding. We study whether policy learning also benefits from a second decomposition, in which each entity is represented by named attributes such as position and appearance. This factorization can, in principle, support transfer to attribute combinations that were never observed during training. We introduce COMB, an object-centric world model with explicitly factorized per-entity latents, learned without attribute supervision and trained for online model-based RL in imagination. It achieves the highest or tied-highest return on eight 2D and 3D manipulation tasks and leads the baselines on held-out attribute combinations. To separate representation format from information content, we freeze COMB and insert an invertible map between its representation and the policy heads. The map preserves all information, changes only the coordinates in which it is expressed, and can be varied continuously. On 2D and 3D tasks with held-out position-colour combinations, entangling the attributes reduces return by 31-37% at a fixed budget, with larger losses at higher doses, and reduces held-out return by up to 30% at matched in-distribution skill. Linear scrambling produces no resolvable loss. These results suggest that, once entities are separated, policy learning benefits from making task-relevant attributes linearly readable; the effect therefore depends on both the representation and the head that reads it.

Primary area:
reinforcement learning
Keywords:
object-centric representations, world models, model-based reinforcement learning, disentanglement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zv1HOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether factorizing object representations into explicit attributes helps policy learning, beyond simply separating objects. The authors introduce COMB, an object-centric world model built on particle based model and extended to online model-based RL. They evaluate it on eight 2D and 3D control tasks, including held-out attribute combinations. They also introduce an entanglement dosing experiment that applies invertible transformations to frozen representations, preserving information while changing how easily policy heads can access individual attributes. The results suggest that nonlinear entanglement hurts policy learning and compositional transfer, while linear transformations have little effect.

Critical strengths

The main strength is the controlled analysis in Sections 6 and 7. Rather than relying only on end-to-end comparisons, the authors use invertible transformations to change representation structure while preserving information content. The placebo and conditioning-matched linear controls make the results more convincing. I also appreciate the matched-skill OOD evaluation, which helps distinguish poor generalization from slower policy learning.

Also the paper offers a useful insight which is that structually disentangling attributes is not sufficient by itself but what matters is whether the downstream policy can easily read them. I find this analysis more valuable than the architectural contribution of COMB itself.

Critical weaknesses

While I find the analysis interesting, I have some concerns.

  1. Limited architectural novelty : I think COMB largely builds on existing factorized particle representations and combines them with standard components for model-based RL. The main architectural contribution seems to be extending DDLP to online policy learning. I therefore see the contribution primarily in the experimental analysis, not the proposed architecture.

  2. The interpretation of the entanglement experiments : The invertible transformations preserve information, but nonlinear scrambling also makes the representation harder for the policy heads to decode. While the linear controls address conditioning, they did not rule out increased function-approximation or optimization difficulty as an explanation for the performance gap. So I think it is less clear how broadly this conclusion holds with more expressive heads or different optimization settings.

Questions for authors
  1. The analysis focuses on position and appearance, which have relatively clear boundaries in the current representation. How would the proposed findings extend to attributes that interact more strongly, such as object geometry, contact state, or occlusion? In particular, what aspects of the current conclusions depend on the explicit factorization already provided by DDLP?

  2. The entanglement experiments show that representation format matters even when information is preserved. However, to what extent should this be interpreted as a benefit of factorization itself, rather than compatibility between the representation coordinates and the downstream policy architecture? How should this distinction affect the broader claim that factorized representations are preferable for policy learning?

  3. The entanglement intervention starts from a successfully learned COMB representation and applies an invertible nonlinear transformation. How far can the resulting performance differences be generalized to representations that are learned in an entangled form from the outset? Does the intervention primarily characterize the cost of nonlinear coordinate transformations, rather than the broader benefits of learning factorized representations?

2 comments

We thank the reviewer for their time and effort, and we appreciate the relevant questions, which we address below.

Novelty (W1). We agree that the controlled study is the paper's main contribution, as now indicated in the introduction. We see COMB as a secondary contribution: to the best of our knowledge, it is the first world model with factorized per-object attributes trained with online RL (Tab. 2). This is not a direct extension of DDLP, which is trained offline on fixed video. For online RL, the policy must act on an unordered set of particles, and reward gradients must not write relations between objects into individual particles. This required action-conditioned particle dynamics, a transformer aggregator for each head, a reward stop-gradient, and an actor–critic with MPPI (App. C). COMB achieves the best or tied-best return on all eight tasks and all six held-out splits (Fig. 2, Tab. 1).

Format or optimization difficulty? (W2).

Offline (App. H.3): Using the test from Prop. 1, we varied head size, training steps, and the amount of data. At both doses, the large head fits the training data (error ≈10−4\approx 10^{-4}), so fitting is not the issue. At the strongest dose, error on unseen combinations stops improving once the head reaches width 256 and is unchanged even with 16× more training steps. Beyond that, only more data reduces the error, and slowly: the unscrambled code trained on 1/16 of the data still transfers far better (0.004) than the strongly scrambled one trained on all of it (0.071). At the milder dose, every MLP head already transfers well (0.004–0.008, against 0.12 for a linear head). Capacity alone does not close the gap; the format determines how much data the head needs. This is consistent with Redhardt et al. (NeurIPS 2025), where large networks overcome a nonlinear encoding when training covers the task space well; in RL, that coverage is exactly what is limited.

Online (App. H.4): On OCRL-OOO-Hard, we increased the actor, critic, and reward heads by 2.75×, at the strongest dose and the paper's training budget. Both arms improved by a similar amount (+11.5 and +9.8 points), so the absolute gap remained (31.6 points, against 29.9 with the original heads; p=0.005p = 0.005). On held-out combinations at matched skill, the dosed policy still loses 37% of COMB's return (46% with the original heads), compared with 3% for COMB (p=2⋅10−4p = 2 \cdot 10^{-4}). For both head sizes, the deficit remains stable between 100k and 250k steps (Tab. 10, App. H.4).

Interacting attributes (Q1). Contact is a relation between objects: COMB's dynamics model captures it through attention across particles, and the aggregator computes it for the heads, rather than storing it in an individual particle. Geometry and rotation share the appearance code with colour, so transfer for these attributes depends on the encoder (App. G.1). Occlusion violates the one-particle-per-object assumption and is outside our scope (Sec. 8). The readability criterion itself, however, does not depend on DDLP.

Factorization or compatibility with the head? (Q2). We read it as compatibility with the head. What matters is that the task-relevant attributes are linearly readable by the head's first layer (Prop. 1b). Standard heads begin with a linear layer, so this applies broadly, and recent theory reaches a closely related requirement for linear readouts (Uselis et al., ICML 2026). This does narrow the broader claim. A factorized representation is preferable when its factorization makes the task-relevant attributes linearly readable, as COMB's does for position. A factorization the head cannot read linearly does not help: warping each attribute within its own coordinates keeps the code factorized, yet still hurts transfer (Tab. 7). We have made this explicit in the abstract and conclusion.

Scrambled vs. learned-entangled representations (Q3). The entanglement dial measures the cost of changing coordinates while keeping the information fixed; it does not measure the full benefit of learning a factorized representation. We now state this explicitly. As a link to learned representations, we applied the probe to the baselines, which are entangled from the outset (App. H.2):

R2R^2 (position / colour)OCRL-ReachOCRL-OOO-HardGA-Cube (1 object)
COMB≥0.97 / 0.76≥0.97 / 0.66≥0.97 / 0.55
SOLD0.85 / 0.670.71 / 0.540.92 / 0.49
TD-MPC20.45 / 0.470.22 / 0.330.82 / 0.49
DreamerV30.15 / 0.370.09 / 0.470.81 / 0.47

On the multi-object tasks, position readability follows the same ordering as final in-distribution return (Fig. 2). Colour is harder for all models, but COMB still reads it best. We view this as a consistency check, not a causal result: the baselines also differ in their encoders and training objectives.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the additional analyses on head capacity, training budget, and data coverage. The clarification that the main benefit lies in the linear readability of task-relevant attributes, rather than factorization alone, is also helpful.

While some limitations remain regarding architectural novelty and generalization to representations learned in an entangled form, I find the controlled analysis of representation format and its impact on policy learning to be a meaningful contribution.

Overall, the rebuttal has clarified several of my concerns, and I maintain my positive rating of 3.

Reviewer_vy9POpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether object-centric world models help policy learning merely by separating entities, or whether it also matters that each entity is represented through factorized, linearly readable attributes such as position and appearance. The authors propose COMB, a DDLP-based object-centric world model with explicit per-particle coordinates, action-conditioned dynamics, a transformer-style particle aggregator, Dreamer-like imagination training, and MPPI-based action selection.

The paper validates COMB in two ways. First, it compares end-to-end performance against DreamerV3, TD-MPC2, and SOLD on several 2D and 3D manipulation tasks, including held-out attribute-combination splits. COMB generally obtains the best or tied-best performance, especially on held-out combinations. Second, and more importantly in my view, the paper introduces a controlled “entanglement dosing” experiment: a trained COMB representation is frozen, and invertible transformations are inserted between the representation and the policy/reward/value heads. Since these transformations preserve information, this experiment aims to isolate the effect of coordinate format and linear readability. The main finding is that linear scrambling has little effect, while nonlinear scrambling degrades both in-distribution learning and transfer to unvisited attribute combinations.

Overall, I see the main contribution not so much as proposing a new RL system, but as providing a careful study of why the format of an object-centric representation matters for policy learning.

Critical strengths
  1. The entanglement dosing experiment is a strong and useful piece of evidence. The paper does not rely only on end-to-end benchmark comparisons. By freezing COMB and inserting invertible transformations, the authors can vary the coordinate system while largely preserving the information content. This makes the main claim much more convincing than a standard ablation would be. In particular, the contrast between linear scrambling, which is essentially harmless, and nonlinear scrambling, which hurts learning and transfer, gives fairly direct evidence that linear readability of attributes matters for the policy heads.
  2. The paper asks a precise question that is more informative than “are object-centric representations useful?” I liked that the paper separates entity decomposition from within-entity attribute factorization. The experiments suggest that object separation is not the whole story: once entities are separated, the way task-relevant attributes are exposed to the downstream policy also matters. This is a useful conceptual point for future work on object-centric RL and world models.
Critical weaknesses
  1. The end-to-end comparisons are still somewhat hard to interpret. COMB differs from the baselines in several ways at once: the encoder, the explicit particle attributes, the aggregator, the reward-gradient handling, and the use of MPPI for action selection. The authors are aware of this and the dosing experiments help, but the headline benchmark results should still be interpreted cautiously. In particular, it is not fully clear how much of COMB’s advantage comes from factorized attributes as opposed to other architectural or planning choices.
  2. The scope of the empirical evidence is somewhat narrow relative to the broader framing. The tasks are well chosen for studying held-out combinations of position, color, and shape, but they are also quite aligned with COMB’s inductive biases. COMB represents particles in the image plane rather than in a full 3D scene representation, and the limitation section acknowledges that occlusion-heavy or metric 3D reasoning tasks are outside the current scope. This does not invalidate the main claim, but it makes me cautious about extrapolating the conclusion to more realistic manipulation settings.
Questions for authors
  1. The end-to-end results combine several differences between COMB and the baselines, including the representation, aggregator, reward-gradient handling, and MPPI action selection. Which of these components do the authors believe is most responsible for the performance gap? Are there ablations, such as COMB without MPPI or COMB with an unstructured per-object embedding, that help separate these factors?
  2. Which aspects of the findings on attribute factorization do the authors expect to generalize to more realistic manipulation settings, and which may depend on the current task design and representation assumptions?
2 comments

We thank the reviewer for the careful reading and constructive questions.

What drives the end-to-end gap (W1, Q1). We agree that Fig. 2 and Tab. 1 alone cannot isolate the source of the gap (Sec. 5). In our view, the main factor is the representation. COMB stays ahead of the baselines even without MPPI, and the dosing study shows that changing only how the same information is written, with everything else fixed, already causes large losses. The end-to-end comparison still mixes attribute format with COMB's better perception (App. G.2), which is why the dosing study keeps the encoder fixed. Taking the components one at a time:

  • MPPI. In a new experiment, we trained and evaluated COMB using its actor alone, without MPPI. It retains about 95% of its in-distribution return on OCRL-OOO-Hard and about 90% on GA-Cube and GA-Distractor, and remains ahead of the strongest baseline on all three tasks (App. H.1). MPPI adds a few points, most on GA-Cube, where the margin over SOLD narrows (about 153 vs. 146), but COMB's lead does not depend on it.
  • Reward stop-gradient. We did not ablate it, because it protects the property the study relies on: reward depends on relations between objects, so its gradient could write those relations into individual particles and mix the object representations (App. C.4). Blocking reward gradients into the world model also had little effect in DreamerV2 (Hafner et al., ICLR 2021), so we do not expect it to be a source of COMB's advantage.
  • Aggregator. The holistic baselines do not need an aggregator. In COMB, it is an additional component that must recover the relevant relations from a set of particles; it is not a shortcut.
  • Unstructured per-object embedding. An unstructured embedding cannot simply be substituted into COMB. Its decoder, priors, and dynamics are all built around the named attributes, so making this change would amount to designing and training a different model from scratch. The entanglement dial provides a controlled alternative: it turns each object's code into an unstructured, invertible transformation of that code while keeping everything else fixed. At the highest dose, this reduces return by 31–60%. On held-out combinations at matched skill, the dosed arms lose up to 30% of COMB's return on the position–colour suites and up to 46% on OCRL-OOO-Hard (Sec. 7).

We also applied the same readability probe to the baselines. On the multi-object tasks, position readability ranks them in the same order as their in-distribution returns (App. H.2). We see this as consistent with the entanglement-dosing results, rather than as proof of a causal relationship, since the baselines differ in more than their representations.

What should carry over to realistic settings (W2, Q2). We agree that the tasks are well matched to COMB's assumptions. This is intentional: it allows us to test the central question in a controlled setting. Sec. 8 discusses the main limits; in brief:

  • What is likely to carry over: the criterion itself—that task-relevant attributes should be linearly readable at the input to the head. The probe can test this on any representation. We also expect the result to be relevant when attributes are produced by separate mechanisms, as with position and scale.
  • What may not carry over: attributes that share an appearance code, such as shape, colour, and rotation, because the encoder itself may merge unseen combinations (App. G.1). The same caution applies to heavily occluded scenes, where one particle may no longer correspond to one object (Sec. 8). 3D particle models such as 3D-DLP (Zhang et al., ICML 2026) and ParticleSplat (He et al., arXiv 2026) are natural next steps.

Thank you for the detailed response. The additional results without MPPI help address my question about the contribution of action selection to COMB’s performance, while the controlled dosing experiments provide complementary evidence for the importance of attribute readability. I also appreciate the clarification of which findings the authors expect to generalize and which may depend on the current task design and representation assumptions.

Overall, these responses address my main concerns, and I maintain my positive rating of 3.

Reviewer_eBsUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces some architectural biases into its VAE-based image encoder so that each "particle" (object) has a factored representation encodings its position, depth, scale, visibility, appearance. (This is ensured using spatial transformers with convolutional layers.) Latent dynamics are applied to each particle separately. Then the particles are aggregated before passing to the output head representing reward, actor and critic (as in Dreamer). The key claim of the paper is that this factored representation helps transfer policy performance to combinatorially novel combinations, especially if the aggregator is nearly linear. They show smll gains over methods based on more entangled representations on several trivial 2d and 3d visual environments.

Critical strengths

none

Critical weaknesses

This is a weak paper. It is obvious that factored representations in principle generalize better (P+F training examples covers PxF test cases), and that a linear readout will not entangle a factored representation as much as a nonlinear readout. The main issue is how to extract such factored representations from complex (visual) inputs, and how to know what the relevant attributes per object should be. The authors side step the hard problem by using visually trivial envirinments (mostly grid-worlds or simple colored blocks on white table) and manually specifying the attributes. So I do not think there is any value to the paper.

Questions for authors

none

1 comment

We appreciate the reviewer’s time, but we strongly disagree that the paper has no value and that its results are obvious. As Reviewers vy9P and zv1H note, the main contribution is the controlled study of Sections 6–7: with the information held fixed, it shows that what lets a policy transfer is not factorization itself, but whether the task-relevant attributes can be read linearly.

Several statements in the review do not match the paper, starting with how the model works:

  • “Latent dynamics are applied to each particle separately.” The dynamics model is a Particle Interaction Transformer that attends across all particles, so it models interactions between objects (App. C.1–C.2), as our revised Fig. 1 describes.
  • “Then the particles are aggregated [...] (as in Dreamer).” Dreamer has no aggregator; COMB gives each head its own transformer aggregator (Sec. 4).
  • “[...] especially if the aggregator is nearly linear.” We make no such claim. In every comparison, all arms share the same nonlinear aggregator; what must be linear is how the task-relevant attributes can be read from its input (Prop. 1; Sec. 7.3).

“They show small gains [...]”

On held-out combinations, COMB obtains 1.2–2.5× the return of the strongest baseline on all six splits (Tab. 1).

“It is obvious that factored representations in principle generalize better (P+F training examples covers PxF test cases), and that a linear readout will not entangle a factored representation as much as a nonlinear readout.”

The counting argument is our starting hypothesis (Sec. 3) and is close to known results for additive models with linear readouts (Lachapelle et al., NeurIPS 2023). Policy heads, however, are nonlinear, and our results show that factorization itself is not what matters. If it were, a code that stays factored should transfer and a code whose attributes are mixed should not. With the information held fixed, we find the opposite in both cases:

  • Factored, yet transfer degrades. Warping each attribute within its own coordinates keeps the code factored, yet held-out error rises to 0.25, against 0.04 for mixing across attributes at a comparable dose (Tab. 7).
  • Entangled, yet no cost. A linear map that mixes all attributes, which disentanglement metrics would score as entangled, costs nothing (Sec. 7.2).

Nor do nonlinear heads entangle a factored code: with it, they keep their performance on held-out position–colour cells (gaps of −3% to +4%, Sec. 7.3). What matters is whether each task-relevant attribute is linearly readable at the head's input. Reducing that readability costs up to 60% of return, and larger heads do not close the gap (Sec. 7.2; App. H.3–H.4).

This was not settled before. Disentanglement helped zero-shot transfer in DARLA (Higgins et al., ICML 2017), but disentangled models often fail on unseen combinations (Montero et al., ICLR 2021; Schott et al., ICLR 2022), and across 240 pretrained representations disentanglement did not predict the out-of-distribution performance of RL agents (Träuble et al., ICLR 2022). The closest work, Uselis et al. (ICML 2026), shows that compositional generalization requires linear, orthogonal representations and tests this on vision encoders, but studies neither policies nor an intervention at fixed information. If the reviewer knows of work that already establishes our results, we would be grateful for the references.

“The main issue is how to extract such factored representations from complex (visual) inputs [...]”

Extraction is a separate line of work: particle models already handle real robot video (Daniel et al., ICLR 2026) and 3D scenes (Zhang et al., ICML 2026). Our results show that extraction alone is not enough: with the encoder held fixed, the format of the representation still affects transfer.

“[...] visually trivial environments (mostly grid-worlds or simple colored blocks on white table) [...]”

None of our tasks is a grid-world: the three 2D tasks come from the published OCRL benchmark (Yoon et al., ICML 2023) and place objects at continuous positions, and the other five are 3D robot-arm tasks. The scenes are simple by design, because held-out combinations require ground-truth attributes. The transfer problem itself is not simple: on held-out GA-Cube cells, the baselines lose 36–43% of their return, compared with 6% for COMB (Tab. 1).

“[...] how to know what the relevant attributes per object should be [...] manually specifying the attributes.”

This is deliberate. Disentanglement requires inductive biases (Locatello et al., ICML 2019), and position and appearance are standard choices in object-centric models (Eslami et al., NeurIPS 2016; Lin et al., ICLR 2020). The same split is used for every task, and the heads learn which attributes matter for each one.

We kindly ask the reviewer to reevaluate the paper in light of these results.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. Reviewers vy9P and zv1H identify the entanglement-dosing study as the paper's main contribution, and we agree. COMB provides the framework that makes this study possible. Put simply, what helps a policy transfer is not that each attribute has its own coordinates, but that the policy can read the attributes linearly: warping an attribute within its own coordinates hurts transfer, while mixing attributes linearly costs nothing (Sec. 7.1–7.2). In response to the reviews, we ran three additional experiments:

  • Removing MPPI (vy9P). When COMB is trained and evaluated using only its actor, it retains about 90–95% of its in-distribution return on the three tasks we tested and remains ahead of the strongest baseline on each (App. H.1).
  • Head size, data, and training length (zv1H). In the offline setting, larger heads help only up to a moderate size and longer training does not help; beyond that, only more data reduces the cost of entanglement (App. H.3). Online, increasing the heads by 2.75× improves COMB and the dosed variant by roughly the same amount, leaving the absolute gap in return essentially unchanged (App. H.4).
  • Baseline readability (vy9P, zv1H). Applying the same probe to SOLD, TD-MPC2, and DreamerV3 ranks them by position readability in the same order as their in-distribution returns on the multi-object tasks (App. H.2). All changes in the revised PDF are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.