acceptodds
Withdrawn from ICLR 2027

Identity without Propagation or Matching: Parallel Object-Centric Video Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video object-centric learning decomposes a video into a small set of object slots. The goal is that each slot binds one object and keeps that binding in every frame. The methods that lead the video benchmarks secure that binding by propagation, initializing each frame from the previous frame's slots through a learned transition model. We present CoSlot, a fully parallel video object-centric model that binds an entire clip in a joint pass, one slot-attention competition over all of its patches, and then refines every frame independently with the same module and no explicit propagation mechanism. It is trained with weak supervision: a differentiable agreement term in the loss compares the slot assignment, pooled over the whole clip, with imperfect reference masks from an unsupervised model or a foundation segmenter; no ground-truth mask is used anywhere. The identity it obtains is competitive: against three published baselines, CoSlot leads on clip-level identity on two of three benchmarks, by to Clip FG-ARI on YouTube-VIS, and matches the strongest on the third, at comparable mask quality. We provide a theory that explains this empirical performance. The output of the joint pass defines, for each clip, convex cones in the projected feature space, one per slot and the same in every frame; an object whose patches stay in one cone keeps one slot in every frame, because every frame is partitioned by the same cones and no information passes between frames. Empirically, the cones the trained joint pass produces predict which slot owns an object significantly better than those of an untrained model.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
object-centric learning, slot attention, video representation learning, temporal consistency, weak supervision, parallel inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7KdLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoSlot, a parallel video object-centric learning framework that maintains object identity without sequential slot propagation or explicit matching. CoSlot first performs joint slot attention over all video patches to obtain clip-level object representations, followed by independent per-frame refinement using shared slot-attention weights. The framework employs a clip-level agreement loss with reference masks from pretrained segmentation models and achieves competitive performance against existing methods. The paper further provides a theoretical analysis explaining temporal identity consistency through a shared partition of the projected feature space.

Critical strengths
  1. The proposed framework eliminates sequential slot propagation and explicit matching while achieving competitive performance with existing video object-centric learning methods. This demonstrates the feasibility of an alternative approach to temporal identity preservation.

  2. The paper provides a geometric interpretation of slot assignments through convex partitions of the feature space, offering useful insights into how temporal consistency can emerge without explicit propagation.

Critical weaknesses

Heavy reliance on external supervision and limited justification of the learning paradigm.

  • The proposed method relies on reference masks generated by existing models, including SAM 3 for real-world videos. This makes the contribution appear closer to an application of foundation-model supervision than a fundamental advance in self-supervised video object-centric learning.
  • Given the availability of such powerful segmentation models, it is unclear whether the proposed slot-based learning framework is necessary. Simpler alternatives, such as constructing object representations by pooling DINOv2 features within regions identified by a foundation segmenter, should be investigated.
  • Furthermore, if the primary advantage of this approach is its 'applicability' to downstream tasks, its practical value would be more convincing with evaluations on tasks such as world modeling or future dynamics prediction, rather than segmentation metrics alone.

Insufficient motivation and experimental evidence for replacing sequential slot propagation.

  • The motivation for eliminating sequential propagation is not sufficiently justified. Existing methods explicitly propagate slot representations across frames to preserve temporal correspondence, which is beneficial for maintaining object identity. The paper does not clearly establish why this mechanism is problematic or should be replaced, beyond the advantage of parallel computation. Moreover, the computational benefits are not quantitatively evaluated.
  • The proposed framework incorporates several existing components from SlotCurri, including the slot-count curriculum and SSIM-based reconstruction objective, while also introducing additional supervision. Despite these components, the improvements over SlotCurri are relatively modest on the synthetic benchmarks.
  • More importantly, the ablation study lacks a controlled comparison in which only SlotCurri's sequential slot encoding is replaced with the proposed parallel encoding, while keeping the remaining training objectives and components identical. Therefore, it is difficult to determine whether the improvements arise from the parallel architecture itself or from the additional supervision and training mechanisms.
Questions for authors
  1. Could the authors provide an experiment that replaces only the sequential slot encoding in SlotCurri with the proposed parallel binding architecture, while keeping the other components and training objectives unchanged? Such a comparison would help isolate the contribution of the proposed architecture from that of the additional losses and supervision.
  2. Given that SAM 3 provides object-level reference masks, how does the proposed learning framework compare with simpler alternatives, such as pooling DINOv2 features within segmented object regions to construct object representations? In particular, what additional benefits does the proposed two-stage slot-attention architecture provide over such approaches?
  3. In Figure 2 (right), the performance improvement over SlotCurri appears particularly large for shorter clips, while the gap is smaller in several longer-clip groups. What explains this observation? Could it indicate that sequential slot propagation becomes more beneficial as the clip length increases? A more controlled analysis across different clip lengths would help clarify the relative advantages of sequential and parallel binding.
  4. Does the learned object-centric representation provide advantages for downstream tasks such as future frame prediction, object dynamics prediction, or world modeling? Such experiments would help establish the practical significance of the proposed learning framework, particularly given its reliance on foundation-model-generated supervision.
Reviewer_GwzMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies video object-centric learning, aiming to maintain consistent object identities across frames while processing video frames in parallel. The proposed method, CoSlot, extracts features using a frozen DINOv2 backbone and applies global Slot Attention to obtain a set of clip-level slots. These slots initialize every frame, followed by parallel per-frame refinement using Slot Attention with shared parameters. The training objective includes reconstruction, a structure term, a slot-contrastive term, an agreement term based on external reference masks, and a background penalty.

Critical strengths

1.The architecture is simple and clearly motivated. Its global aggregation, shared initialization, and per-frame refinement avoid frame-by-frame recurrence and additional explicit matching steps. 2.The method achieves competitive performance among the evaluated baselines. CoSlot obtains strong Clip FG-ARI results on MOVi-C and YouTube-VIS, and the ablation study demonstrates the importance of the agreement term. 3.The paper provides an analysis of its identity-preservation mechanism. It explains how shared initialization supports identity consistency through a partition of feature space and empirically examines the resulting predictions, helping clarify the method’s capabilities and limitations.

Critical weaknesses

1.Comparisons with recent relevant methods are incomplete. The paper does not compare against SSync, Selective Synergistic Learning for Video Object-Centric Learning, which was publicly released on June 14, 2026, and published at ECCV 2026. SSync also evaluates on MOVi-C, MOVi-E, and YouTube-VIS, and some of its reported results exceed those of CoSlot. A comparison under a consistent evaluation protocol would help establish the method’s performance advantages. 2.The ablation study does not cover all loss terms, and experiments with alternative backbones are missing. The existing experiments examine the agreement term, slot-contrastive term, and background penalty, but do not separately evaluate the contributions of reconstruction and the structure term. Their complementary effects therefore remain unclear. Furthermore, newer versions of DINO are available. A comparison between DINOv2 and DINOv3 would help assess whether the method generalizes across feature representations and benefits from newer pretrained features. 3.The novelty relative to PSB requires further clarification. Parallelized Spatiotemporal Binding (2024) already uses attention mechanisms to generate object slots for all frames in parallel while modeling cross-frame identity consistency. Although PSB employs cross-frame attention whereas CoSlot performs independent refinement following global slot initialization, the two methods substantially overlap in their research objectives and overall motivation. What are CoSlot’s substantive contributions beyond PSB in terms of architecture and identity preservation?

Questions for authors
  1. How does CoSlot compare with SSync under the same evaluation protocol?
  2. What are the individual contributions of the reconstruction loss and structure term?
  3. Does CoSlot generalize to DINOv3 or other pretrained backbones?
  4. What are CoSlot’s key advances over PSB, particularly in preserving cross-frame object identity?
Reviewer_N6DKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoSlot for discovering objects in videos and maintaining object-to-slot correspondence across frames. The model first applies joint Slot Attention to DINOv2 features from the entire video to obtain clip-level slots. These slots are then used as shared initialization for parallel per-frame refinement. During training, reference masks generated by SlotCurri or SAM 3 provide supervision through a differentiable ARI loss computed from a clip-level contingency table. The paper also analyzes the convex-cone partition induced by the first attention assignment during per-frame refinement and examines its relationship with final slot identity.

Critical strengths

S1. I think the design of joint binding followed by per-frame refinement is worth investigating. It removes the sequential dependency of updating slots frame by frame during inference, and the architecture is relatively straightforward. The motivation for the clip-level ARI loss is also clear: pooling assignments across frames before computing the agreement makes it possible to detect identity switches that per-frame metrics cannot capture.

S2. Table 1 shows that the complete system achieves competitive results on MOVi-C and YouTube-VIS, while performing similarly to SlotCurri on MOVi-E. The authors report results over three random seeds and provide per-clip comparisons and failure cases. Appendix E also explicitly shows that the average improvement on MOVi-C does not correspond to a consistent advantage across individual clips. These details help assess the actual scope of the results.

Critical weaknesses

W1. The existing experiments do not directly isolate the contribution of joint binding.

CoSlot changes both the architecture and the training objective, while the main baselines do not receive the same reference-mask supervision. This difference is particularly evident on YouTube-VIS: the SAM 3 reference masks achieve a Frame FG-ARI of 0.853, compared with approximately 0.489 for SlotCurri. When the same CoSlot architecture is trained using SlotCurri reference masks, its Clip FG-ARI reaches only approximately 0.37 (Section 4.2 and Appendix F). Therefore, Table 1 supports the effectiveness of the complete system, but does not distinguish the contributions of joint binding and the supervision source.

The authors acknowledge in Section 4.2 that distillation may allow a student to outperform its teacher, and I agree with this qualification. However, the nearby statement that the improvement cannot be attributed to a stronger training signal is still too strong. Using the baseline itself as the teacher does not mean that the student and baseline receive equivalent training constraints.

The ablations in Table 2 remove the agreement loss or untie the slot-attention weights, but neither directly removes joint initialization. A useful additional experiment would be a controlled comparison that replaces joint initialization with shared learnable slots, under the same reference masks, backbone, slot count, and comparable training budget. Other comparisons that isolate this factor would also be acceptable.

W2. Some theoretical implications are inconsistent with the stated limitations.

Sections 5.1 and 6 clearly acknowledge that the theory only constrains the first attention assignment and does not guarantee identity consistency in later iterations or decoded masks. I do not expect the authors to prove convergence of the entire network. However, Lines 150–152 in Section 3.1 state that weight sharing allows per-frame refinement to "only sharpen the region its slot already owns." Weight sharing alone does not provide this guarantee, because the updated slots depend on different frame inputs. This statement should be made consistent with the later limitations.

The implication following Proposition 2 also needs correction. Two objects that cannot be separated by a hyperplane through the origin cannot each be completely assigned to a different cone. However, this does not imply that they must share a cone. For example, consider one object with features on the positive and negative horizontal axes and another object with features on the positive and negative vertical axes in a two-dimensional space. These objects cannot be separated by a single homogeneous linear boundary, but their features can be distributed across four different cones without sharing a cone. Thus, the proposition rules out perfect one-object-one-slot assignment, but does not rule out object fragmentation. The statement in Line 422 that the observed merging failures occur where the proposition says they must therefore goes beyond the established result.

The separability probe in Appendix C also requires a more careful interpretation. Using training accuracy to examine the separability of a fixed feature set is reasonable. However, the probe samples at most 6,000 patches, so exact separability of the sampled set does not establish exact separability of the entire clip. Moreover, the training accuracy of an SVM with a finite penalty parameter is not necessarily the maximum achievable accuracy. The authors acknowledge the latter point but still describe the result as a ceiling.

Another point requiring clarification concerns MOVi-E. Appendix B acknowledges that some clips contain 19 objects while the model uses only 15 slots, but Table 4 reports 100% exact clip separability for the raw features. Please explain how these clips are handled in the probe. If the classifier allows more classes than the available slots, the result can measure instance separability in the feature space, but cannot directly establish that the binding is realizable with the given number of slots.

Questions for authors

Q1. Can the authors provide a direct controlled comparison that separates the contribution of joint initialization from the training supervision?

For example, replacing joint initialization with shared learnable slots under comparable training conditions would help establish whether joint binding is essential. If other experiments already isolate this factor, please provide those results.

Q2. Can the authors correct the implication from linear inseparability to inevitable object merging and clarify the interpretation of the separability probe?

In particular, how are sampled patches handled when assessing exact separability, and how are MOVi-E clips with more objects than available slots treated in Table 4?

Q3. Can the authors clarify the paired configurations used in Table 6?

For example, on YouTube-VIS, the restored mBO values implied by the ablation rows are 0.3376 for removing the agreement loss, 0.3325 for contiguous-frame sampling, and 0.3708 for removing the slot-contrastive loss, while the complete model in Table 1 achieves 0.3709.

The table caption already states that each change is computed relative to the configuration with the corresponding component restored, so these differences do not necessarily indicate errors. However, without listing these configurations, it is difficult to determine which model each component comparison in Section 4.4 refers to. Please provide the complete paired configurations, random seeds, and aggregation procedures, and explain how they relate to the final model.

Additional Comments:

  1. PSB already performs parallel slot binding through temporal attention. The description in Section 2 that it requires cross-frame matching should be supported by a more specific explanation of its mechanism. A clearer distinction between PSB and CoSlot in how object identity is established would be more helpful than simply expanding the baseline list.

  2. The practical efficiency benefit of the parallel architecture has not been measured. Reporting end-to-end inference time and peak GPU memory under a representative setting on the same hardware, together with video length and whether backbone encoding is included, would help assess the practical value of removing sequential dependencies.

  3. Line 470 in Section 6 states that the agreement loss acts on the first attention iteration of the per-frame pass, whereas Section 3.2 and Appendix G describe its application to joint attention and decoded masks. These descriptions should be made consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.