acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Object-Centric Representations for Video Dynamics Modeling

Abstract

Learning to decompose videos into persistent objects is a fundamental challenge in unsupervised object-centric representation learning. Despite recent progress, existing methods struggle to simultaneously achieve accurate object segmentation, consistent identities over time, and reliable foreground–background separation. To address these challenges, we introduce UniSlot (Unified Slots), an unsupervised framework for learning robust and disentangled object-centric representations from videos. UniSlot explicitly separates object appearance from its 3D-aware geometric pose in the scene, linking object identity to appearance while leveraging depth to better distinguish objects from their surroundings. UniSlot achieves state-of-the-art performance in unsupervised object-centric video decomposition and tracking across synthetic and real-world benchmarks, yielding substantially tighter object masks and reducing background leakage while preserving object identities. Beyond decomposition and tracking, these improved representations translate directly to downstream tasks such as unsupervised object dynamics prediction, enabling more accurate forecasting of future object trajectories.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Object-centric Representation Learning, Unsupervised Object Discovery, Unsupervised Object Tracking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oBUFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniSlot learns object-centric video representations with separate appearance vectors and explicit two-dimensional position and scale. Recurrent grouping, adaptive slot activation, temporal appearance alignment, and slot separation are combined with visual reconstruction and monocular pseudo-depth reconstruction. Depth is used during training only. The method is evaluated on synthetic and real-world video decomposition, downstream SlotFormer prediction, and transfer to DAVIS.

Critical strengths

The results provide useful evidence that appearance-only temporal alignment and depth-assisted decomposition can work together across different video regimes

Critical weaknesses
  1. The central causal explanation is not cleanly separated from the architectural combination. Section 3.2 explicitly builds on ISA's slot-centric spatial reference frames, but the main disentanglement ablation replaces the mechanism with standard Slot Attention. This changes pose-normalized grouping together with the factorization. The image-method comparisons in Appendix E do not substitute for a recurrent ISA-style control under matched activation, depth, and training settings. Moreover, explicit position and scale do not by themselves establish that the appearance vector excludes geometric information. Thus, Table 4 demonstrates the utility of the complete design but provides a weaker basis for identifying appearance–geometry entanglement itself as the source of existing failures.

  2. The downstream experiment does not isolate improved dynamics learning from improved initial decomposition and a different prediction parameterization. The predictors share a Transformer architecture, but UniSlot predicts explicit position displacements and positive scales with separately weighted losses, while baseline predictors operate on entangled embeddings. The evaluation uses decoded segmentation metrics, and Table 1 already shows substantial differences in observed-frame masks. Consequently, the rollout advantage may combine better starting masks, decoder quality, and structured predictor inductive biases, rather than demonstrate more accurate temporal dynamics alone.

Questions for authors
  1. What evidence separates the benefit of appearance–geometry disentanglement from the inherited pose-normalized grouping and accompanying components?
  2. How much of the downstream rollout gain remains after accounting for initial decomposition quality and the different state-prediction parameterization?
Reviewer_vhTBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose UniSlot, an unsupervised object-centric framework for video representation learning that explicitly disentangles object appearance from geometric state. Each slot separates appearance from 2D position and scale, temporal consistency is enforced only on appearance, and monocular depth pseudo-supervision from a pretrained depth estimator is used during training to improve foreground-background separation. The method is evaluated on synthetic and real-world video benchmarks for object decomposition and tracking, transfer, and downstream object dynamics prediction, and shows consistent quantitative and qualitative improvements over recent video slot-based baselines.

Critical strengths
  • The paper addresses an important limitation of existing video object-centric methods: temporal consistency is often imposed on slot representations that entangle appearance with spatial properties, which can conflict with object motion and contribute to identity switches.
  • The central design choice is well motivated. Separating appearance from position and scale allows temporal alignment to preserve object identity while permitting geometric attributes to evolve.
  • The method is evaluated across a meaningful range of settings, from synthetic videos to real-world videos and zero-shot transfer to DAVIS.
  • The empirical results show consistent gains. UniSlot outperforms recent baselines on all metrics.
  • The ablations are informative and support the claimed contributions.
  • The downstream dynamics-prediction experiments are interesting and thus a valuable addition. These experiments show that the learned representations are useful beyond reconstruction and tracking.
  • Qualitative results clearly illustrate the intended benefits, especially the recovery of an object’s identity after temporary disappearance and re-entry.
Critical weaknesses
  • The technical novelty is moderate. UniSlot combines several existing components—pose-normalized grouping from ISA, adaptive slot activation, recurrent slot propagation, temporal slot consistency related to SlotContrast, spatial broadcast decoding, and pretrained DINO and monocular-depth priors—into a new video object-centric system.
  • The method relies heavily on pretrained priors: frozen DINO features provide the visual representation, and Depth Anything provides pseudo-depth targets. Although the paper shows robustness to different DINO encoders, the dependence on the quality and domain coverage of the depth model remains a significant limitation. The authors acknowledges this, but provide limited analysis of when biased depth estimates propagate into object masks or identity assignments.
  • The evaluation seems limited to relatively short clips and datasets with a moderate number of salient objects. It remains unclear how the method behaves on longer videos, many-object scenes, frequent and prolonged occlusions, drastic appearance changes, non-rigid deformation, or highly cluttered real-world footage.
  • The paper would benefit from more directevidence that appearance and geometry are genuinely disentangled, rather than merely represented in separate vectors. For example, interventions on position/scale or appearance, analyses of cross-frame appearance stability, or controlled tests showing that geometry can change without disrupting identity would substantiate the central claim.
  • (Minor) SlotCurri is used as the latest baseline but is not described in the related works section.
Questions for authors
  • How robust is UniSlot when the monocular depth estimator produces systematically incorrect depth, for example on transparent objects, thin structures, reflective surfaces, or domains far from the depth model’s pretraining distribution? Are there failure cases where depth pseudo-supervision harms object decomposition?
  • What evidence shows that appearance and geometry are truly disentangled rather than simply stored in separate vectors? Could the authors provide intervention experiments or quantitative measures of cross-factor independence?
  • How does UniSlot perform under longer temporal horizons, more simultaneous objects, repeated occlusions, and substantial appearance changes?
  • For adaptive activation, how sensitive is performance to the hard Gumbel-Softmax threshold of 0.5 and to the delayed introduction of sparsity and temporal/separation losses? Are these schedules dataset-specific but necessary for stable training?
Reviewer_aC1hOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes UniSlot for unsupervised object-centric learning on videos. Each slot is split into appearance and 2D position/scale, and the temporal consistency loss is applied only to appearance. The model also uses adaptive slot activation and reconstructs monocular pseudo-depth (Depth Anything v3) during training. It reports the best results among video slot methods on CLEVRER, MOVi-C/E and YouTube-VIS, along with better SlotFormer dynamics prediction and transfer to DAVIS.

Critical strengths
  • The experiments are extensive. The method is tested on both synthetic and real-world data, results are averaged over 3 seeds, and the appendix covers different encoders, decoders and slot numbers. The gains on YouTube-VIS and the transfer to DAVIS are consistent across metrics.

  • Applying temporal consistency only to appearance is a simple and reasonable idea. Depth is used only for training, so inference still needs only RGB.

Critical weaknesses
  • My main concern is the significance of this direction. Strong open-world video segmentation models already exist (e.g., SAM 2, Savvy [Su et al., 2026, arXiv:2606.15632]), and they produce object masks and tracks of much higher quality than current slot-based methods. Learning slot representations on top of such models, e.g., by pooling features within their masks, seems more efficient than learning the decomposition from scratch. This is especially true because UniSlot is not annotation-free in spirit: it already relies on large pretrained models (DINO and Depth Anything v3). The paper does not discuss this alternative or explain what end-to-end slot learning offers beyond it.
  • The value of the learned slots is not shown beyond segmentation. Object-centric representations are usually motivated by their use in downstream reasoning, such as video understanding, causal reasoning or planning. Yet the paper evaluates almost exclusively with segmentation and tracking metrics. Experiments on tasks like video QA or reasoning (e.g., the CLEVRER reasoning questions) would make the case for this representation much stronger.
  • The only downstream task is SlotFormer rollout, and even this is scored with segmentation metrics (ARI, FG-ARI, mBO, J&F) on the predicted future frames. This measures how well the masks are preserved over time, not how well the dynamics are modeled. It is weak support for the “Video Dynamics Modeling” in the title.
Questions for authors
  • How do the authors see the value of learning slots end-to-end, compared with building object representations on top of open-world video segmentation models (e.g., SAM 2, Savvy)? For example, one could pool DINO features within their masks, or use the masks to guide slot attention (e.g., as supervision for the attention maps). A comparison with such a pipeline on decomposition and dynamics prediction would directly address my main concern.
  • Can the learned slots be shown to help on tasks beyond segmentation, such as video reasoning (e.g., the CLEVRER QA tasks)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.