acceptodds
Under review as a conference paper at ICLR 2027

CoSlot: Clip-Level Joint Slot Binding for Video Object-Centric Learning

Abstract

Video object discovery requires information from multiple frames while main- taining object identities across frames. In commonly used training setups with per-frame reconstruction, adding context frames also increases the cost of dense de- coding. We propose CoSlot, which jointly binds features from multiple frames with a set of shared slots and combines two reconstruction objectives: shared prototypes reconstruct all visible frames using the binding assignments, while a spatial decoder reconstructs sampled frames from frame-level slot features. This design allows the number of frames used for binding to grow while keeping the number of spatially decoded frames fixed. Experiments show that combining the two reconstruction objectives improves object grouping in videos. In inference comparisons using fixed models, shorter independent chunks mainly reduce cross-frame consistency, and slot passing recovers most of the loss. Without a slot contrastive loss or explicit slot regularization terms, CoSlot achieves video FG-ARI scores of 72.2 and 81.9 on MOVi-C and MOVi-E, respectively. In the evaluated configurations, its cost per optimizer update on precomputed features is lower than that of SlotContrast and SRL. Experiments on YouTube-VIS also show the benefit of passing slots across chunks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Video Object-Centric Learning, Representation Learning, Unsupervised Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aRtnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoSlot studies video object-centric learning by jointly binding frozen features from multiple visible frames into shared global slots. It combines two reconstruction objectives: shared prototypes reconstruct features from all visible frames through the binding assignments, while a spatial decoder reconstructs only sampled visible frames. This allows the binding context to grow while keeping the number of densely decoded frames fixed. The full model additionally includes masked-frame prediction (MFP), slot coordination, and temporal evidence aggregation. Evaluation on MOVi-C, MOVi-E, and YouTube-VIS measures object grouping and mask overlap. Fixed-checkpoint inference comparisons distinguish the effects of joint binding scope, temporal coverage, and global-slot state passing. Training costs are measured using cached frozen features.

Critical strengths
  1. A clear implementation and empirical support for separating binding context from dense decoding cost. Shared-prototype reconstruction (ALG) covers all visible frames and can be evaluated through compact matrix statistics. Spatial decoding operates on uniformly sampled frames and provides an unbiased estimator of the visible-frame mean spatial loss for a fixed input and model. The objectives are empirically complementary: Core achieves video FG-ARI of 66.6 under Whole inference, compared with 50.6 for Only-Spatial. Table D.1 also shows relatively small quality changes from decoding more spatial frames at substantially higher computational cost, supporting a small decoding budget. The full model decodes one visible spatial-reconstruction target and one MFP target, rather than only one frame in total. (pp. 4 and 7–8; Appendix A; Table D.1)
  2. Informative fixed-checkpoint diagnostics distinguish within-frame grouping from cross-frame consistency. Shortening independent inference chunks primarily reduces video-level consistency, while image-level grouping changes relatively little. Global-slot state passing recovers much of this loss. The 48-frame YouTube-VIS experiment reports paired results across three seeds, with positive gains from state passing and whole-sequence joint binding for every seed. Full-reference alignment reindexes slots using the model’s own Whole predictions rather than ground truth, and the manuscript distinguishes offline whole-sequence access from state continuation across chunks. These controls help avoid conflating inconsistent slot identities with failures of single-frame segmentation. (pp. 5–6; Appendices B.2 and C.2)
Critical weaknesses
  1. Lower per-update cost does not yet establish a competitive quality–total-training-cost trade-off. In Table 2, CoSlot achieves video FG-ARI of 30.2±0.6 on YouTube-VIS, compared with 38.0 for SlotContrast and 42.9 for SRL, with lower mBO as well. Its mBO also trails these two principal baselines on MOVi-C and MOVi-E. Table D.3 genuinely demonstrates lower per-update time and memory: for example, 87.98 ms/update versus 149.18 ms/update for SlotContrast on YouTube-VIS, approximately a 41% reduction. However, these measurements use cached features and a physical batch size of 16, whereas comparison quality is taken from published results. Training schedules, actual batch sizes, convergence, and cumulative resource budgets are not aligned with the per-update measurements. The current evidence therefore does not establish the quality achievable under equal training time or the resources required to reach a shared quality level. The consequential concern is the absence of a direct cross-method quality–cost evaluation for an efficiency-oriented contribution. (Tables 2 and D.3; pp. 8, 15–17, and 21)
  2. The independent effect of training context remains confounded by feature exposure and randomness. The training-length sweep in Figure 3(c) fixes 100k updates and 128 clips per update while increasing context from four to 24 frames. Consequently, visible-frame feature exposure per update increases by up to sixfold, and the sweep uses only one fixed seed. The interpretation that longer joint context itself improves single-frame Reset consistency is therefore not isolated from greater feature exposure, different gradient estimation, or training-compute effects. Fixed-checkpoint inference experiments support comparisons of inference context but do not remove this training confound. The four-, eight-, and twelve-frame YouTube-VIS training sweep is also nonmonotonic, limiting the scope of the proposed interpretation. (Figure 3(c); Tables B.1 and C.2; pp. 6–7 and 15–18)
Questions for authors
  1. Using existing training logs, can you align validation quality for CoSlot, SlotContrast, and SRL with cumulative training GPU time or updates, specifying feature-caching costs, actual batch sizes, and training stages? On at least one existing dataset, can you report quality under a matched training budget or time to reach a shared quality level?
  2. Is there an existing or feasible limited MOVi-C comparison between four-frame and 24-frame training with comparable cumulative visible-feature exposure or training compute, together with seed variability? If such a control is unavailable, please describe the single-frame Reset improvement as an effect of the complete training configuration and explicitly acknowledge that context length itself has not been isolated.
  3. For chunked inference with global-slot state passing, can you report paired effects of ALG relative to Only-Spatial across matched training seeds and checkpoints? In Table C.1, Core achieves 55.7 under Chunk-1 GS passing, below 60.6 for Only-Spatial, reversing the ordering under Whole inference. Please explain this protocol-dependent benefit and MFP’s role, since it materially affects the assessment of the state-passing use case.
Reviewer_JYm5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies object-centric learning by jointly grouping the DINOv2 features into clip-level shared slots. It builds shared prototypes across sampled frames via reconstruction, which avoid densely decoding every frames to bind context. This method was evaluated on synthetic MOVI-C/E datasets and real-world YouTube-VIS, showing the effectiveness of passing slots across chunks.

Critical strengths

The new framework for video object-centric learning is very appealing compared to previous slot-attention based approaches. The prototype reconstruction to sampled spatial frames is effective and supported by the experimental results.

Critical weaknesses
  • The real-world application is limited. It shows big performance gap on real-world dataset YouTube-VIS. It's even below SlotContrast's performance. Longer context also fails to improve this setting (see Table C.2). The real-world value of this work is not enough compared with other video object-centric works such as U-OVOL and OC-MOT (Object-Centric Multiple Object Tracking, Zhao et al., ICCV 2023). Please compare and explain the real-world values in the manuscript.

  • The performance is not SOTA. Even when we benchmark on the synthetic dataset, it is still worse than SRL on all the metrics.

  • The distinctive contribution (modelling temporal information) is not fully evaluated. The binder is invariant to the frame order, how does it perform temporal positional encoding? And how does the work handle occlusion and reappearance issue in object-centric learning?

Questions for authors
  • What accounts for the substantial performance gap to SRL on YouTube-VIS (30.2 versus 42.9 video FG-ARI)? Please investigate whether the limitation comes from shared-prototype reconstruction, temporal association, or mask refinement. Maybe an ALG assignment-gradient ablation would help explain the gap.
2 comments

Thanks for the response. I think the authors answers most of my questions. The comparisons with other methods are included now. However, the limited performance on real-world dataset is still not good enough. I will keep my rating as weekly acceptance.

Dear Reviewer JYm5,

Thank you for confirming that our responses addressed most of your questions. We agree that performance on real-world videos remains a limitation. Section 5 of the revision states it, Appendix C.5 points to two likely sources of the gap, the spatial readout and instances split across several slots, and the final version will add a matched-budget comparison with SlotContrast on YouTube-VIS.

Thank you again for your time.

Reviewer_Ge2HOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed CoSlot for video object-centric learning. Clip-level slots binds features from all frames jointly; A shared-prototype loss, ALG, reconstructs all visible tokens; A decoder reconstructs only one sampled frame; and a masked prediction branch adds further self-supervision. This reduced the training overhead. The paper also analyzes several inference protocols.

Critical strengths

The idea is distinctive compared with mainstream methods. It folds the temporal dimension into one binding step instead of learning a transition. Its technique ALG has a closed-form computation, so the training cost can be lower. The analysis of inference protocols is informative.

Critical weaknesses
  1. The most important weakness is the limited performance. In Table 2, CoSlot wins only 1 of 12 comparisons against SlotContrast and SRL. The only win FG-ARI@MOVi-C uses Offline Whole inference. Under a causal protocol, the result is below SlotContrast. The mBO is inferior on all datasets, even lower than old baselines VideoSAUR.

  2. Several close related works are cited but not compared or discussed effectively.

  • The core claim is that joint binding can replace a learned transition. Thus, the paper should compare with methods that explicitly learn transition dynamics, such as RandSF.Q. GS passing is in fact an identity transition, making such comparison essential.
  • SOLV is the closest prior work. It also binds multi-frame context and decodes only the middle frame. But it is omitted. Later works adopt SOLV’s cross-frame setting with slot pruning, e.g., video version of DIAS and AdaSlot. They address redundant slots under a fixed number. This may mitigate CoSlot’s low mBO and background segmentation stripes.

These methods should be at least discussed more concretely, if quantitative comparions are not that easy.

  1. Overclaim in Contribution 1. SOLV already binded multiple frames and decoded a single frame. The key difference is that CoSlot can decode any sampled one frame, while other frames still are trained via ALG.

  2. The main table compares the authors' offline method with those online methods. This is not fair as the latter can not see all the frames while the former can, which is an uncontrolled advantage. Should unify this.


  • SOLV: Self-supervised object-centric learning for videos
  • DIAS: Slot attention with re-initialization and self-distillation
  • AdaSlot: Adaptive slot attention: Object discovery with dynamic slot number
Questions for authors

See weaknesses.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to your reviews. Please take some time to carefully examine their rebuttals and engage in the discussion by sharing your thoughts, remaining concerns, or follow-up questions.

As the discussion deadline is approaching, I would appreciate your timely participation to allow sufficient time for further exchanges with the authors.

Thank you for your time and valuable contributions.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.