- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a slot-attention-based object-centric world model, termed FOCWM, where each slot is factorized into geometry (position, scale, depth), occupancy (visibility) and appearance factors. The dynamic module operates over the latent slots along with learned latent actions. The model is evaluated on video prediction and downstream imitation learning on various datasets.
Critical strengths
- Novelty: my impression is that the authors try to bring the attributes of the latent particles (DLP) to the slot-attention framework. I think that it makes a lot of sense and overall a novel idea. This mechanism can compensate, for example, when several particles overlap and can actually be aggregated to a single entity. I find this a novel contribution to the self-supervised object-centric family. I also find the latent action per-attribute a novel idea.
- Results: the proposed method outperforms PlaySlot and is comparable or slightly better than LPWM.
- Detailed appendix.
- A promise for an open-source code.
Critical weaknesses
- Writing: overall, I have a good understanding of the method; however, I believe the paper could benefit from a writing revision to make the method section clearer. For example, in line 216, a patch of size is introduced. It is not clear where it came from and what the role of this in the slot-attention framework is. Typically, the particle approach relies on glimpses/patches, so the connection should be made clearer. If this stitching part is inspired by Lin et al. or DLP, it should be clearly mentioned. In addition, up to the method section, the authors mention “action fusion” multiple times, and it is not clear what the readers should understand from this term.
- Complexity and multi-stage: to my understanding, the model is trained in two stages, where in the second stage the representation is frozen. Compared to end-to-end models such as Dreamer or LPWM, the representation is not trained to be predictable by the dynamics modules. That also means that training this model online (e.g. for model-based applications) would be challenging. I find this a major limitation of the model.
- Stochasticity: to my understanding, all the video prediction experiments in the paper are based on first extracting latent actions from GT videos, and then using them to condition the dynamics module. Can latent actions be sampled from the fixed unit Normal prior? Can this model be used for stochastic video prediction?
- Limited benchmark: the model is only evaluated on 4 datasets with very small number objects.
- Limitations: no discussion of limitations and future work. I can think of several limitations, as mentioned in my review, that I believe should be explicitly mentioned.
Questions for authors
- Stability: can the authors please address the stability of the object decomposition across runs? Slot-attention is notoriously hard to train with unstable slot decompositions between runs, and I’m interested in whether the additional attributes actually help with stable decomposition.
- Scenes with a large number of objects: how would the proposed method perform on datasets with larger numbers of objects, like LanguageTable or BAIR datasets, what would the slot decompositions look like with the proposed method?
- LPWM training: was LPWM trained with similar reconstruction losses (perceptual + MSE)?
3 comments
Thank you for your effort!
I thank the authors for their significant effort during the rebuttal phase, it is appreciated. I acknowledge that I have read the other reviews and their responses as well. My questions have been answered and my impression of the paper remains positive.
Overall, given my familiarity with the field, I find the main contribution lies in improving the slot-attention framework with stable training and additional attributes, bringing it closer to the particles representations but overcoming several limitations with it. I believe the community would benefit from access to a better slot-attention framework, as proposed in this work. However, as for the downstream applications (video prediciton, decision-making), I don't think the results (and the new additions as well) are entirely convincing, and the margins are very small. Mainly, the results on LanguageTable suggest that several objects can still be modeled with a single slot (so the decomposition is not entirely object-centric), and the fact that the model struggles with more objects (BAIR) emphasizes the limitations of the slot-attention framework, where other representations already work with complex scenes. So, the argument that the representation is more "compact" than, e.g., the particles, is nice and all, but if it does not work on more complex datasets that other methods already scale to, I don't find this a compelling argument to use the method.
Considering the limited scale of ratings this year, I believe my current score reflects my impression of the work, and I'm happy to keep it.
We sincerely thank the reviewer for their thoughtful feedback, constructive suggestions, and positive assessment of our work. We greatly appreciate the time and effort you have devoted to reviewing our paper and engaging in the discussion throughout the rebuttal process. We also appreciate your recognition of our main contributions and understand your remaining concerns regarding the downstream applications and scalability. Your feedback is valuable and will help us further improve our work and guide future research. Thank you again for your careful consideration and encouragement.
We sincerely thank the reviewer for the thorough and constructive feedback. We appreciate the careful assessment of our work and the valuable suggestions. We address each concern and question in detail below.
W1:
The local patch of size L is generated by the RGB and alpha decoders from object slots, rather than extracted as an input glimpse by Slot Attention. The spatial features determine how these patches are transformed and placed onto the image canvas for reconstruction. This design is related to patch-based object rendering approaches such as DLP, and we will clarify this connection. We will also replace the potentially ambiguous term "action fusion" with "action-oriented feature fusion," which specifically refers to adaptively combining spatial, alpha, and RGB features for latent-action inference.
W2:
Our two-stage design separates representation learning from dynamics learning. The first stage establishes functionally specialized object features through dedicated decoding pathways. Freezing the representation then provides stable feature targets for training the IDM and FDM, which are supervised by both feature prediction and image reconstruction losses. We acknowledge that this does not explicitly optimize the representation for predictability and may limit online adaptation compared with end-to-end approaches.
W3:
FOCWM supports stochastic video prediction without access to future observations.
Specifically, we introduce a learned history-conditioned Gaussian prior (Appendix A.6, Eqs. 33–34). The prior and posterior are jointly trained, with the prior learning to approximate the posterior through KL divergence. At inference time, latent actions can be sampled from this learned prior to generate future frames autoregressively, as illustrated in Figure 8.
We additionally evaluated this history-only setting on Sketchy using FVD (lower is better). FOCWM achieves 125.61, compared with 85.45 for LPWM. Although LPWM performs better in stochastic generation, these results confirm that FOCWM supports quantitative history-only video prediction. Improving its stochastic generation quality remains an important direction for future work.
W4:
Although our evaluation is limited to four datasets with relatively few objects, these benchmarks cover diverse manipulation scenarios across simulated and real-world environments, including multi-object interactions and dual-arm manipulation.
Furthermore, we have obtained stable object decompositions on the LanguageTable dataset. FOCWM achieves a PSNR of 30.5, SSIM of 0.95, and LPIPS of 0.039, compared with 29.5, 0.94, and 0.037 for LPWM, respectively. Overall, our method achieves comparable video prediction performance to LPWM on this additional benchmark. These results provide preliminary evidence that our representation can generalize beyond the evaluated benchmarks.
W5:
FOCWM has several limitations. First, although it performs well on the evaluated manipulation datasets, its object-centric decomposition may become less reliable in complex real-world scenes with densely packed objects, severe occlusions, and intricate object interactions. Second, FOCWM adopts a two-stage training strategy in which the representation model is frozen during dynamics learning. While this simplifies optimization, it limits joint adaptation between object representations and dynamics and may hinder online learning. Future work will explore more robust object decomposition in complex scenes and end-to-end optimization of representation and dynamics learning.
Q1:
In our experiments, FOCWM consistently produces stable object decompositions across independent training runs. Furthermore, Figure 5 demonstrates temporally consistent object assignments across video frames in both simulated and real-world environments.
Q2:
We additionally conducted experiments on both LanguageTable and BAIR, using 12 and 15 slots, respectively.
On LanguageTable, FOCWM achieves stable object-level slot decompositions across consecutive frames. Quantitatively, FOCWM achieves a PSNR of 30.5, SSIM of 0.95, and LPIPS of 0.039, compared with 29.5, 0.94, and 0.037 for LPWM, respectively, demonstrating comparable video prediction performance. Figure 5(c) shows the decomposition results.
On BAIR, however, we observed lower-quality slot decompositions despite increasing the number of slots to 15. We hypothesize that this is related to the greater scene complexity, including frequent occlusions and overlapping or stacked objects, which make consistent object separation more challenging.
These results suggest that FOCWM can generalize to scenes with more objects, while robust object decomposition in heavily occluded and cluttered environments remains an important direction for future work.
Q3:
LPWM was trained with both perceptual and MSE reconstruction losses, following its official implementation and training configurations to ensure a fair comparison.