acceptodds
Under review as a conference paper at ICLR 2027

Slot Count as an Ensemble Axis for Object-Centric Learning

Abstract

Object-centric models decompose a scene into a fixed number of slots, yet the right number is rarely known. Too few slots merge objects, too many fragment them, and conventional inference commits to a single decomposition. We ask whether the decompositions that one model produces at different slot counts can instead be combined into a better one. We introduce PACE (Pairwise Agreement across Cardinality Ensembles), a two-part framework that answers this question positively. Prefix training samples a slot count at every step and activates the matching prefix of an ordered bank of slot initializers, so that a single checkpoint yields decompositions at many cardinalities from one encoder pass. Horizontal-Cut Consensus, a parameter-free readout, converts each soft partition into slot-invariant pairwise co-membership, averages this evidence, and cuts the resulting hierarchy into any requested number of regions, without slot matching, labels, or additional training. On COCO, PASCAL VOC2012, and MOVi-C, PACE improves FG-ARI by 4.67, 3.67, and 5.26 points over the standard seven-slot prediction, and mBO by 1.17 and 4.57 points on COCO and VOC2012. No single slot count recovers this grouping gain, even when the best one is chosen with evaluation labels. Controlled experiments trace the benefit to complementary errors. Coarse and fine requests fail in opposite ways, repeated sampling at one slot count saturates quickly, and a weak standalone partition can contribute more than a strong one. These results establish requested slot count as an ensemble axis and decouple the granularity of the grouping evidence from that of the final segmentation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Slot attention, object-centric learning, compositionality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cTmDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACE, a framework that treats requested slot count as an ensemble axis for object-centric segmentation. Prefix training enables one model to produce decompositions at multiple slot counts. A training-free consensus readout converts soft masks into slot-invariant pairwise patch co-membership, averages this evidence, and applies average-linkage hierarchical clustering to obtain a segmentation at a chosen output region count.

Experiments on COCO, VOC2012, and MOVi-C demonstrate improved foreground grouping over single-cardinality predictions. Comparisons with single-cardinality hierarchies, repeated stochastic inference, separately trained models, and resource-matched controls support the benefit of complementary decompositions. Ablations examine population composition and readout choices. Improvements are strongest in FG-ARI; overlap gains and advantages over published methods are less consistent.

Critical strengths
  1. A meaningful perspective on slot cardinality. The paper shows that different slot counts can supply complementary evidence rather than merely represent competing hyperparameter choices. Separating evidence-generation cardinality from output granularity provides a useful framework for object-centric segmentation.

  2. Strong controls for the central empirical claim. Comparing against single-cardinality hierarchies at the same output count, repeated stochastic inference, and resource-matched alternatives helps distinguish consensus gains from postprocessing or additional computation. The consistent FG-ARI improvements across three datasets make the central finding credible.

Critical weaknesses
  1. Limited methodological novelty beyond established consensus clustering. Pairwise co-association, evidence averaging, and hierarchical consolidation closely follow existing cluster-ensemble principles, including soft clustering ensembles. The contribution therefore rests primarily on applying cardinality diversity within object-centric learning. A sharper account of what the prefix construction contributes beyond general variable-cardinality training would strengthen the novelty claim.

  2. The demonstrated benefit has a restricted scope. Most validation uses DINOv2 ViT-S/14 with a DINOSAUR-derived MLP decoder. Results establish improved foreground grouping, but do not establish broadly superior segmentation: overlap gains are inconsistent, and several published methods remain stronger on VOC2012 or overlap metrics. Limited replication of attribution experiments and the absence of qualitative failure analysis further constrain conclusions about generality.

Questions for authors

The following are clarification requests that could affect my assessment:

  1. Have you compared ordered-prefix training with variable-cardinality training using a shared Gaussian initializer or randomly selected initializer subsets? Such evidence would clarify whether the specific prefix construction is important, beyond exposure to multiple cardinalities.

  2. How stable are the Figure 5 attribution results across training seeds? The reported image-bootstrap intervals do not capture training variability, which matters for the claim that one prefix-trained model approximates separately trained cardinality-specific models.

  3. Appendix D.3 lists sampling probabilities summing to 0.70 rather than 1. Were these normalized weights, or is a value or cardinality missing? Please also clarify why COCO evaluation uses 4,946 images and identify the population configuration corresponding to the headline improvements.

  4. Are there existing results on another decoder or backbone that support the claimed applicability beyond the primary configuration? If unavailable, please clarify the intended scope of the conclusions.

5 comments

Thanks for your positive feedback on our work! We are grateful for your appreciation for using slot cardinality as an ensemble axis (shared by FHk9 and RcU3), the empirical setup (also highlighted by RcU3), and the positive evaluation on the manuscript in general. We also appreciate the constructive observations and questions raised, which we will address in the following responses.


W1. Methodological novelty. We agree that co-association and hierarchical consolidation are established techniques and explicitly credit them. Our contribution is to identify and experimentally characterize slot cardinality as a source of complementary grouping evidence. We do this in contrast with the main direction that the object-centric community has explored recently, which is unsupervised, dynamic inference of the slot count. We motivate this arguing that even when the slot count is correct, objects can still be split and merged sub-optimally. On the other hand, different cardinalities induce different merge–split errors. These multiple views can provide useful evidence about object relationships that would be discarded otherwise when committing to one decomposition.

PACE combines these alternatives from a single checkpoint through variable-cardinality training and a slot-invariant hierarchical readout that decouples evidence cardinality KK from output regions LL. The ablation in Q1 shows that ordered prefixes offer a small advantage but are not essential to consensus. On the other hand, training exposure to each cardinality is necessary: truncating a fixed-K=12K{=}12 model yields 37.0 FG-ARI, versus 42.7 for one prefix-trained checkpoint and 43.3 for three separately trained models, at one-third of the training cost (Q2).

More broadly, PACE makes complementary decompositions an explicit design consideration in object-centric learning: alongside selecting cardinality, models can learn to produce useful alternatives and consolidate their evidence instead of committing to one decomposition.


Q1. Role of prefix training. To isolate the contribution of the ordered prefix training beyond exposure to multiple cardinalities, we added the requested shared-Gaussian and random-initializer-subset comparisons. All variants use the same variable-KK schedule and differ only in initialization.

We evaluate six-cardinality PACE on COCO at L=7, using three 40,000-step training runs per variant (means ± standard deviation across training seeds; Appendix D.4).

Training variantK=7K{=}7 FG-ARIPACE FG-ARIK=7K{=}7 mBO$_i$PACE mBO$_i$
Prefix training38.1 ± 0.142.7 ± 0.526.6 ± 0.027.7 ± 0.1
Variable-cardinality training, shared Gaussian37.3 ± 0.141.7 ± 0.126.2 ± 0.027.1 ± 0.1
Variable-cardinality training, random initializer subsets37.6 ± 0.242.1 ± 0.126.4 ± 0.027.4 ± 0.0

PACE improves both metrics under every variant, showing that generating useful evidence from one checkpoint extends beyond our initialization choice. The ordered prefix training thus offers a simple, nested construction using the first KK position-specific initializers and provides a small advantage in absolute scores, but is not required for the consensus benefit.


Q2. Seed stability. Thanks for your observation; we agree that image-bootstrap intervals do not capture training variability. We therefore repeated the Figure 5 attribution experiments with three training seeds per model, using the same 40,000-step training budget. The table reports means ±\pm standard deviation on COCO at L=7L{=}7 (Appendix D.5). Every FG-ARI comparison in Fig. 5 (Fig. 6 now) retains its direction across all three seeds. Prefix training outperforms fixed-K=7K{=}7 training with the same inference cardinalities, and adding K=9,12K{=}9,12 further improves both metrics. Simply truncating a model trained at K=12K{=}12 does not reproduce these gains. One prefix-trained model remains close to the consensus of three separately trained models.

TrainingInferenceFG-ARI ↑mBO$_i$ ↑
Fixed K=7K{=}7K=7K{=}7 prediction37.9 ± 0.326.6 ± 0.1
Fixed K=7K{=}7PACE, K∈{3,4,5,7}K\in\{3,4,5,7\}39.8 ± 0.526.4 ± 0.1
Fixed K=12K{=}12Six-cardinality PACE37.0 ± 1.426.0 ± 0.4
PrefixK=7K{=}7 prediction38.1 ± 0.126.6 ± 0.0
PrefixPACE, K∈{3,4,5,7}K\in\{3,4,5,7\}41.1 ± 0.626.8 ± 0.2
PrefixSix-cardinality PACE42.7±0.5‾\underline{42.7 \pm 0.5}27.7±0.1‾\underline{27.7 \pm 0.1}
Three separate models (K=3,5,12K{=}3,5,12)PACE43.3 ± 0.328.3 ± 0.0

W2. Investigation scope. Thanks for pointing this out. We have expanded the evaluation, to clarify where PACE helps and extend the evidence that complementary decompositions can improve object-centric segmentation across multiple model configurations and evaluation criteria. The main results are reported here.

Backbones and decoders. Additional experiments extend the grouping gains to other encoders and a transformer decoder; see our response to Q4.

Training variability. We repeated the prefix-attribution experiments across three training seeds; see Q2.

Qualitative analysis. We added qualitative comparisons showing individual cardinalities and their consensus (Appendix C.2), alongside failure cases in crowded scenes (Appendix G.3).

Segmentation quality. We agree that the results do not establish broadly superior segmentation over existing methods. Our contribution is to improve a given object-centric model by combining complementary decompositions. We expanded the evaluation with instance- and class-level overlap, foreground mIoU, class-agnostic foreground PQ, boundary metrics, and SO/PO/GO analysis as suggested by FHk9 (Appendix G).

The table below report the relevant results on COCO (bold: best; underline: second best). PACE improves all five metrics over the same checkpoint’s single prediction and repeated K=7K{=}7 inference.

MethodFG-ARI ↑mBO$_i$ ↑mBO$_c$ ↑fg-mIoU ↑PQ ↑
SPOT36.834.044.231.738.1
AdaSlot38.426.630.625.238.3
DINOSAUR-prefix, single K=7K{=}739.627.431.726.038.0
DINOSAUR-prefix, 3 restarts K=7K{=}741.7‾\underline{41.7}27.431.925.939.9‾\underline{39.9}
DINOSAUR-prefix, PACE44.928.5‾\underline{28.5}33.4‾\underline{33.4}26.9‾\underline{26.9}44.3

The gains also persist in multi-object scenes: excluding single-object COCO images retains a 4.8 point FG-ARI improvement. Overlap gains vary with object count and output granularity, with adjusted hierarchy cuts recovering gains in crowded scenes. The SO/PO/GO analysis finds fewer group-of-objects regions but more part-of-object regions; area weighting shows less area in object parts at matched group-of-objects area. Boundary accuracy does not improve in the default configuration (Appendices G.1–G.2).

Across datasets, FG-ARI and PQ improve over the single-prediction baseline, while overlap improves on COCO and VOC2012 but decreases slightly on MOVi-C. Published methods retain stronger overlap scores on all three datasets (Appendix G.1).


Q3. Manuscript fixes. Thank you for identifying these ambiguities. We have clarified all three in the revised manuscript.

  • Sampling probabilities (Appendix D.3): The original text ambiguously described the training cardinalities. Training samples all ten cardinalities K∈{3,…,12}K\in\{3,\dots,12\}, whereas PACE evaluates a subset at inference, e.g., {3,4,5,7,9,12}\{3,4,5,7,9,12\}. The endpoint-weighted probabilities are p(3)=0.15p(3)=0.15, p(12)=0.25p(12)=0.25, and p(K)=0.075p(K)=0.075 for each of the eight intermediate cardinalities and sum to 1.

  • COCO evaluation (Appendix B.4): Of the 5,000 validation images, 54 contain no annotated foreground objects, which leaves 4,946 eligible for foreground evaluation.

  • Headline configuration (Appendix B.4): The reported improvements use K∈{3,7,12}K\in\{3,7,12\}, without reflections, cut at L=7L=7, compared against the raw K=7K=7 prediction from each corresponding checkpoint and averaged over three training seeds. The revised FG-ARI gains of 5.27, 4.33, and 5.61 points use uniform slot-count sampling during training; previous values used weighted sampling.


Q4. Decoder and backbone ablations. We added COCO experiments with larger DINOv2 encoders, the original DINO encoder, and a four-layer non-autoregressive transformer decoder. Each configuration uses three training seeds and 40,000 training steps. The table compares PACE (K∈{3,7,12}K\in\{3,7,12\}, L=7L{=}7) with each model’s own K=7K=7 prediction, reporting means ± standard deviation across seeds (Appendix H).

BackboneDecoderK=7K=7 FG-ARIPACE FG-ARIK=7K=7 mBO$_i$PACE mBO$_i$
DINOv2 ViT-B/14MLP37.3 ± 0.141.5 ± 0.226.4 ± 0.127.2 ± 0.1
DINOv2 ViT-L/14MLP37.2 ± 0.241.6 ± 0.126.2 ± 0.126.8 ± 0.1
DINO ViT-B/16MLP36.2 ± 0.139.7 ± 0.226.4 ± 0.027.7 ± 0.0
DINOv2 ViT-S/14Transformer37.3 ± 0.341.5 ± 0.126.5 ± 0.227.9 ± 0.0

PACE improves both grouping and overlap in every configuration, supporting applicability beyond the primary backbone and MLP decoder. It also improves FG-ARI by 1.6–4.1 points over the best single-cardinality hierarchy in each configuration, keeping the readout and output count matched (Appendix H).

Dear reviewer cTmD,

As the discussion period is approaching its end, we would appreciate any feedback on whether our responses address your concerns. Please let us know if there are remaining points we could clarify.

Thanks in advance for your time and contribution.

Reviewer_FHk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a common challenge of object-centric representation learning: How to find the best number of slots. The paper even goes further and shows that having a single, fixed number is not enough. Instead, it proposes to do the encoding with different numbers of slots and proposes a cluster-based merging algorithm to merge them. To achieve this, it first proposes a way to generate representations for various numbers of slots by using a single encoder, followed by the proposed, training-free clustering-based merging of the multi-granularity object decomposition. The strength of the method is shown mainly on the COCO dataset with respect to the FG-ARI metric, while two other datasets have also been evaluated. Further, a comparison to just using a single number of slots is also performed showing the benefit of the merging.

Critical strengths

S1: Interesting insight, that merging across multiple cardinalities provides better results than a single one.

Critical weaknesses

W1: Quantitative results are not fully convincing, the method only leads on FG-ARI for COCO dataset, while outperformed on the same dataset for the mBO metric, as well as on both metrics for the other two datasets. There are further issues with basing the evaluation mainly on the FG-ARI metric, which is known to be alone not a very suitable metric to convincingly evaluate the made claims, specifically:

  • FG-ARI is known to be an unreliable metric as it does not take into account background pixels and the shape of the predicted masks, e.g., assigning all pixels to a single slot-mask gets the highest FG-ARI on PASCAL, and therefore FG-ARI can be misleading in assessing segmentation quality, see Appendix of [a] for further discussion
  • Better is to include further metrics such as the combination of mBOc, mBOi, and mIoU than just mBO, and property prediction, e.g., as commonly done in recent OCL-papers [b, c]
  • To evaluate the claim that the method addresses the over-under-segmentation, also the metrics SO-PO-GO in [d] would be a suitable addition.
  • Also, the proposed method increases the computation cost, which is not included in the comparison to the other methods with quantitative numbers.

W2: There are no qualitative comparisons shown in the paper, making it impossible to assess the generate masks visually, e.g., whether they improve in segmentation quality and how the complementary information is formed. Also seeing the differences in individual object representations for various K and the final merged one would be valuable information.

Together, these limitations on the evaluation limit the scope of the insights on how the method performs compared to the existing related work.

[a] Kakogeorgiou et al., SPOT: Self-training with patch-order permutation for object-centric learning with autoregressive transformers, CVPR 2024

[b] Ouyang et al., QASA: Quality-Guided K-Adaptive Slot Attention for Unsupervised Object-Centric Learning, ECCV 2026

[c] Singh et al., GLASS: Guided Latent Slot Diffusion for Object-Centric Learning, CVPR 2025

[d] Fan et al., Unsupervised open-vocabulary object localization in videos, ICCV, 2023

Questions for authors

Q: How does the method perform on more commonly used OCL evaluation metrics, such as object discovery (mBOc, mBOi, and mIoU), property prediction, and, if possible, the SO-PO-GO metrics [d], compared to other existing methods?

5 comments

Thanks for your insightful review of our work! We are grateful for your positive feedback on the ensembling of multiple slot cardinalities (also noted by reviewers cTmD and RcU3) and the positive feedback on presentation and soundness. We also appreciate the constructive observations and comments featured in the Weaknesses and Questions sections; we will address them individually in the following lines.


W1 and Q1. Quantitative evaluation and segmentation quality. While we agree that FG-ARI alone is insufficient to assess segmentation quality, we would like to note that PACE is a readout applied to a trained base model rather than a new architecture. Hence, controlled comparisons mostly make sense with the same checkpoint's own prediction; external methods are shown for context.

To address the points raised, we expanded expanded the evaluation to test whether the benefits of combining cardinalities extend beyond foreground grouping, characterize the remaining errors, and quantify their computational cost.


Additional segmentation metrics. Alongside mBO$_i$, we added mBO$_c$, foreground and background-inclusive mIoU, and class-agnostic foreground Panoptic Quality (PQ), which excludes background pixels from matching. The COCO comparison below shows improvements on all five metrics over both the same checkpoint's single prediction and repeated K=7K{=}7 inference. SPOT (larger ViT-B/16 encoder, self-training, autoregressive decoder) retains stronger overlap, while PACE achieves higher FG-ARI and PQ. In the table, bold: best; underline: second best.

MethodFG-ARI ↑mBO$_i$ ↑mBO$_c$ ↑fg-mIoU ↑PQ ↑
SPOT36.834.044.231.738.1
AdaSlot38.426.630.625.238.3
DINOSAUR-prefix, single K=7K{=}739.627.431.726.038.0
DINOSAUR-prefix, 3 restarts K=7K{=}741.7‾\underline{41.7}27.431.925.939.9‾\underline{39.9}
DINOSAUR-prefix, PACE44.928.5‾\underline{28.5}33.4‾\underline{33.4}26.9‾\underline{26.9}44.3

The gain is not specific to this base model: on four further encoder/decoder configurations, PACE improves FG-ARI, mBO$_i$, and PQ over each model's own prediction (Appendix H). Across datasets, FG-ARI and PQ improve over the single-prediction baseline; overlap improves on COCO and VOC2012 but decreases slightly on MOVi-C. Published methods retain stronger overlap scores, including on VOC2012.


FG-ARI and single-object scenes. We agree with the concern: a single region covering the whole image receives perfect FG-ARI on single-object images, which constitute approximately half of VOC2012. The one-region control obtains 49.9 FG-ARI on the full VOC2012 validation set without using image information.

We therefore also evaluated images containing at least two ground-truth objects. PACE’s grouping gains persist under this restriction, while the one-region control scores zero. On these multi-object images PACE also exceeds every released checkpoint we could evaluate under the same protocol (SPOT, SPOT-M, DINOSAUR-M, AdaSlot). Background-inclusive mIoU also consistently improves on all datasets (Appendix G.1).

MethodCOCO FG-ARI ↑, ≥2 objectsVOC2012 FG-ARI ↑, ≥2 objects
One region0.00.0
Best released checkpoint (AdaSlot / SPOT-M)41.636.8
Same checkpoint, K=7K{=}743.441.7
PACE48.247.8

SO/PO/GO and boundary analysis. We added Fan et al.’s single-object, part-of-object, and group-of-objects categories, including comparisons with published methods. The COCO shares of predicted regions under the original categorization rules are (%, Appendix G.1):

MethodSO ↑PO ↓GO ↓
SPOT33.425.519.5
AdaSlot36.923.618.5
Same checkpoint, K=7K=736.921.417.8
PACE34.125.215.9

Relative to the single prediction, the per-region shares show fewer GO regions but also fewer SO and more PO regions. The extra PO regions are mostly small boundary slivers rather than split objects: fragments per object fall from 1.63 to 1.38, the lowest of all evaluated methods (SPOT: 1.49), and area weighting shows less area in PO regions at matched GO area (Appendix G.2). Boundary accuracy does not improve in the default configuration.

These analyses support consolidation with a merge–split trade-off, rather than simultaneous elimination of both error types (Appendices G.1–G.2).

W1 and Q1. Quantitative evaluation and segmentation quality (continuation).


Property prediction. We added a controlled COCO category probe with a separate linear classifier for each region source, trained under the same protocol on frozen DINOv2 features pooled within predicted regions (Appendix I). Unmatched ground-truth objects count as incorrect; the table reports image-averaged scores.

The main benefit is discovering more classifiable objects; classification accuracy conditional on matching changes little. The first row probes the learned K=7K{=}7 slot vectors; the other rows probe frozen features pooled within the predicted regions. This controlled category probe does not establish broader property-prediction superiority over external methods.

Region sourceAccuracy over all GT objects ↑Match recall, IoU ≥ 0.5 ↑Accuracy on matched objects ↑
K=7K{=}7 slot vectors19.021.090.9
Same checkpoint, K=7K{=}719.321.092.3
PACE24.025.793.1

Computational cost.

We added operation counts and end-to-end measurements including CPU clustering (Appendix F.2). PACE takes about three times the latency of a single prediction at batch size one, with a modest memory increase. At the same number of Slot Attention calls it improves FG-ARI, mBO$_i$, and PQ over three K=7K{=}7 restarts, and twelve restarts at 3.5 times its operations still reach only 41.9 FG-ARI, so the gain is not bought by compute alone. Against the released AdaSlot checkpoint, PACE uses fewer operations and less memory and scores higher on all three metrics, but is slower in wall-clock time.

We also introduce an attention-mask variant that applies the same consensus readout directly to the final Slot Attention masks, skipping the reconstruction decoder at inference without changing training. It needs about a quarter of the operations of a single prediction and less memory while still improving all three metrics over it; preprocessing, transfers, and CPU clustering keep it slower end-to-end.

InferenceGFLOPsms/image, batch 1ms/image, batch 32Peak GPU (MiB)FG-ARI ↑mBO$_i$ ↑PQ ↑
K=7K{=}7 prediction4611.73.516239.627.438.0
K=7K{=}7 prediction, attention masks1110.83.114139.728.538.1
Three K=7K{=}7 restarts11635.97.816241.727.439.9
Twelve K=7K{=}7 restarts43061.215.416241.927.440.1
PACE, decoder masks12134.37.517244.928.544.3
PACE, attention masks1231.07.114142.529.141.0
AdaSlot, released checkpoint17114.46.844438.426.638.3

W2. Qualitative results. Indeed, this was missing in our initial submission; thanks for pointing it out. The revision adds a main-text figure (Figure 3) and Appendix C.2, which compare PACE masks with the best single-KK prediction of the same checkpoint and with the released SPOT, SPOT-M, DINOSAUR-M, and AdaSlot checkpoints (where available) on all three datasets. A dedicated population-members figure shows the input image, ground-truth masks, predictions at K=3,7,12K=3,7,12, and the final PACE segmentation side by side: the coarse member merges neighbouring objects, the fine member splits objects into parts, and the consensus keeps the groupings they share. Failure cases, chosen by a fixed rule (largest mBO$_i$ loss in crowded scenes), show where consolidation merges neighbouring objects (Appendix G.3).

Dear reviewer FHk9,

As the discussion period is approaching its end, we would appreciate any feedback on whether our responses address your concerns. Please let us know if there are remaining points we could clarify.

Thanks in advance for your time and contribution.

Reviewer_RcU3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACE, a framework that combines object-centric decompositions generated with different slot counts. The method introduces prefix training to support multiple slot cardinalities within a single model and Horizontal-Cut Consensus to aggregate pairwise patch co-membership into a hierarchical segmentation. Experiments on COCO, VOC2012, and MOVi-C demonstrate improvements over single-cardinality predictions, particularly in foreground grouping. Additional analyses investigate complementary merging and fragmentation errors across different slot counts.

Critical strengths
  1. Interesting multi-cardinality formulation. The paper addresses fixed-slot limitations by combining complementary object decompositions generated with different slot counts.

  2. Consistent improvements in object grouping. PACE improves FG-ARI across three datasets, demonstrating the benefits of aggregating multiple slot configurations.

  3. Informative ablation studies. The experiments investigate complementary grouping errors, ensemble diversity, and prefix training, providing useful insights into the observed improvements.

Critical weaknesses
  1. Limited methodological novelty. The proposed consensus readout mainly applies established co-association and hierarchical clustering techniques, while prefix training is a relatively straightforward modification of Slot Attention.

  2. Limited improvements in segmentation quality. Although PACE improves FG-ARI, gains in mBO are smaller and less consistent. Several existing methods still achieve substantially better object overlap, raising questions about the practical significance of the improvements.

  3. Additional computational overhead. The method requires multiple slot inference passes and pairwise clustering, introducing computational and memory costs that are not sufficiently justified by the performance gains.

  4. Unresolved object cardinality selection. Despite addressing the fixed-slot problem, PACE still requires a predefined number of output regions. The method does not automatically determine the appropriate number of objects in each scene.

Questions for authors
  1. Can the authors clarify the methodological novelty of PACE compared with existing clustering ensemble methods and multi-slot aggregation approaches?

  2. Why are the improvements in mBO less consistent than those in FG-ARI, and can the authors provide further analysis of segmentation boundary quality and object fragmentation?

  3. Can the authors provide end-to-end computational comparisons with single-cardinality and adaptive-slot methods under matched inference budgets?

  4. Can the final output cardinality be determined automatically from the consensus hierarchy without using ground-truth information?

3 comments

Thanks for your feedback on our work! We are grateful for your positive comments on cardinality ensembling (shared by cTmD and FHk9), the empirical setup (also highlighted by cTmD), and the positive evaluation on presentation. We also appreciate the constructive observations and questions raised, which we will address in the following responses.


W1 and Q1. Methodological novelty. Thanks for giving us the chance to clarify this. We agree that co-association and hierarchical consolidation are established techniques and explicitly credit them (Section 4). Our contribution is to identify and experimentally characterize slot cardinality as a source of complementary grouping evidence.

Recently, much research in object-centric learning has focused on methods that infer the slot count dynamically. PACE asks a complementary question: what can be recovered by combining decompositions across cardinalities? Even when the slot count matches the number of objects, a prediction can still split some objects and merge others. Different cardinalities induce different merge–split errors, so their alternative decompositions can provide useful evidence about object relationships that would otherwise be discarded when selecting a single decomposition.

PACE combines these alternatives from a single checkpoint through prefix training and a slot-invariant hierarchical readout, without matching slots. The readout separates the cardinalities generating evidence (KK) from the final number of regions (LL), allowing evidence from multiple decompositions to inform an output at the desired granularity.

The gains observed in PACE comes from the cardinality axis rather than from the clustering machinery: with the same readout, the best single-KK hierarchy (chosen with labels) reaches 40.7 FG-ARI on COCO and twelve K=7K{=}7 restarts reach 41.9, against 44.9 for three cardinalities (Appendices F.2 and G.1). Unlike matching-based multi-slot aggregation (e.g., Masked Multi-Query Slot Attention), the readout needs no slot correspondence: in a controlled ablation, explicit slot matching gives 38.9 FG-ARI versus 42.9 for the slot-invariant readout; hard co-association, kk-means, and spectral clustering give 41.2, 39.4, and 39.3 (Appendix E.3).


W2 and Q2. Segmentation quality. We agree that the overlap gains are smaller and less consistent than the FG-ARI gains, and that some existing methods achieve higher overlap. We expanded the evaluation to include instance- and class-level overlap, foreground mIoU, and class-agnostic foreground Panoptic Quality (PQ), which assesses matched-region overlap and recognition quality with background pixels excluded from matching.

The COCO results below compare PACE with the same checkpoint’s single prediction, repeated inference at the same cardinality, adaptive cardinality selection, and SPOT (bold indicates the best score and underline the second best among reported values; from Appendix G.1).

MethodFG-ARI ↑mBO$_i$ ↑mBO$_c$ ↑fg-mIoU ↑PQ ↑
SPOT36.834.044.231.738.1
AdaSlot38.426.630.625.238.3
DINOSAUR-prefix, single K=7K{=}739.627.431.726.038.0
DINOSAUR-prefix, 3 restarts K=7K{=}741.7‾\underline{41.7}27.431.925.939.9‾\underline{39.9}
DINOSAUR-prefix, PACE44.928.5‾\underline{28.5}33.4‾\underline{33.4}26.9‾\underline{26.9}44.3

PACE improves all five metrics over the same checkpoint’s K=7K{=}7 prediction and repeated K=7K{=}7 inference. SPOT retains stronger overlap, while PACE achieves higher FG-ARI and PQ. Across datasets, FG-ARI and PQ improve consistently over our single-prediction baseline; overlap improves on COCO and VOC2012 but decreases slightly on MOVi-C (Appendix G.1).

We also added qualitative comparisons and failure cases (Appendices C.2 and G.3). To address your specific questions:

  • Why mBO improves less consistently. Consolidating fragments can reunite pixels belonging to the same object while incorporating pixels from another object, improving grouping without necessarily improving the best mask’s overlap. The balance varies with scene complexity: on COCO, mBO$_i$ increases by 7.5 points for single-object images but decreases by 0.9 points for images with eight or more objects. Adjusting the final hierarchy cut can recover overlap gains in crowded scenes (Appendix G.3).

  • Object fragmentation. Foreground pair recall (the fraction of same-object pixel pairs kept in one region) rises from 65.8 to 77.8 on COCO, while pair precision falls from 81.0 to 79.2: PACE keeps more same-object pixels together, with some additional merging of different objects (Appendix G.1).

  • Boundary quality. The default decoder-mask configuration does not improve boundary accuracy: COCO boundary F1 changes from 12.1 to 11.7. PACE consolidates grouping evidence on the existing patch grid without a boundary-refinement mechanism.

W3 and Q3. Computational overhead. Following your comments, we added operation counts and end-to-end latency and memory measurements, including preprocessing, transfers, and CPU clustering. Latency and memory use one RTX 3090 and a shared panel of COCO images; quality columns are three-seed COCO means (Appendix F.2).

InferenceGFLOPsms/image, batch 1ms/image, batch 32Peak GPU (MiB)FG-ARI ↑mBO$_i$ ↑PQ ↑
K=7K{=}7 prediction4611.73.516239.627.438.0
K=7K{=}7 prediction, attention masks1110.83.114139.728.538.1
Three K=7K{=}7 restarts11635.97.816241.727.439.9
Twelve K=7K{=}7 restarts43061.215.416241.927.440.1
PACE, decoder masks12134.37.517244.928.544.3
PACE, attention masks1231.07.114142.529.141.0
AdaSlot, released checkpoint17114.46.844438.426.638.3

PACE increases latency over a single prediction, with a modest memory increase. At the same number of Slot Attention calls and similar operation counts (121 versus 116 GFLOPs), it improves FG-ARI, mBO$_i$, and PQ over three K=7K{=}7 restarts, and twelve restarts at 3.5 times the operations still reach only 41.9 FG-ARI. Against the released AdaSlot checkpoint, PACE uses fewer operations and less memory and scores higher on all three metrics, but is slower in wall-clock time.

We also introduce an attention-mask variant that applies the same consensus readout directly to the final Slot Attention masks, skipping the reconstruction decoder at inference without changing training. It needs about a quarter of the operations of a single prediction (12 versus 46 GFLOPs) and less memory while still improving FG-ARI, mBO$_i$, and PQ over it. Preprocessing, transfers, and CPU clustering keep both PACE variants slower end-to-end than a single prediction and AdaSlot (Appendix F.2).


W4 and Q4. Unsupervised cardinality selection. Thank you for this question. The same consensus hierarchy can be cut at different levels without rerunning the model; selecting LL automatically additionally requires a label-free criterion. We use L=7L{=}7 in the main evaluation to match the K=7K{=}7 reference prediction, but a fixed output count is not required by the framework.

To investigate selection directly from the consensus hierarchy, we evaluated a label-free, per-image rule that chooses the cut maximizing the silhouette score computed on the consensus dissimilarity matrix 1−C1-C.

MethodCOCO FG-ARI ↑COCO mBO$_i$ ↑VOC2012 FG-ARI ↑VOC2012 mBO$_i$ ↑MOVi-C FG-ARI ↑MOVi-C mBO$_i$ ↑
K=7K{=}7 prediction39.627.423.636.670.528.6
PACE, fixed L=5L{=}544.427.229.741.472.425.3
PACE, fixed L=7L{=}744.928.528.041.176.128.0
PACE, silhouette-selected LL44.426.830.940.672.025.0

Both fixed cuts improve FG-ARI over the K=7K{=}7 prediction on all three datasets, so the gain does not depend on a tuned LL; on COCO, fixed-L=7L{=}7 PACE also exceeds the adaptive-slot AdaSlot checkpoint on all five metrics of the W2 table (AdaSlot remains ahead on MOVi-C). Silhouette selection (about five regions on average) preserves the FG-ARI gain on all three datasets and exceeds both fixed cuts on VOC2012, but lowers overlap (below the K=7K{=}7 prediction on COCO and MOVi-C) and does not reliably beat fixed counts (Appendix G.4).

The hierarchy contains better cuts than these rules find: choosing LL from the annotated object count raises COCO FG-ARI from 44.9 to 48.0, and the best cut per image reaches 57.7 (label-informed; Appendix G.3). A reliable label-free criterion to realize this headroom remains open, which the revision states as a limitation.

Dear reviewer RcU3,

As the discussion period is approaching its end, we would appreciate any feedback on whether our responses address your concerns. Please let us know if there are remaining points we could clarify.

Thanks in advance for your time and contribution.

Comments on the paper

2 comments
Area_Chair_BuGJOpenReview ↗

Author-review discussion reminder

Dear Reviewers,

Please spend some time reading the rebuttal and indicate whether the authors have clarified your concerns. Please do so before Nov 18 AoE to give authors the chance to further clarify.

Best,

AC

AuthorsOpenReview ↗

Rebuttal summary

We thank again the reviewers for their constructive feedback. In the rebuttal, we addressed the reviewers’ observations on novelty compared to established consensus methods, robustness across models and training choices, mask overlap, inference cost, and how to choose the final number of regions without labels. To complement the limited discussion, we summarize the main additions and remaining points of disagreement below.

  • Broader validation. Following the reviewers’ suggestions, we extended the training ablations across seeds and compared alternative slot initializers and consensus readouts, increasing the robustness of the original results. We also included new experiments with additional backbones and decoders that show that combining predictions from different slot counts improves object grouping across a range of architectures.

  • Segmentation and object discovery. We expanded the evaluation to cover a range of new metrics (including mask overlap and panoptic quality), alongside qualitative comparisons (showing that PACE keeps more of each object together, with some additional merging in crowded scenes) and analyses of fragmentation and object boundaries. Combining slot counts improves panoptic quality over a single prediction from the same checkpoint across all three datasets. The grouping gains persist in multi-object scenes, and a category-prediction probe shows that PACE recovers more objects that can be correctly classified.

  • Computational cost and output cardinality. We added end-to-end latency and memory measurements. At the same number of Slot Attention calls, combining slot counts outperforms fixed-count restarts on grouping, overlap, and panoptic quality. We also included an attention-mask variant uses about one-quarter of the operations of a single decoder-based prediction, although clustering still adds latency. Finally, to address the request of one reviewer, we incorporated a small analysis on label-free rules for choosing the final number of regions.

  • Remaining disagreements. The remaining disagreements with the reviewers concern methodological novelty and object overlap performance. Regarding the novelty, we are still convinced that our contribution (showing that different slot cardinalities induce complementary split and merge errors, and that combining their predictions improves grouping even when the readout and final region count are held fixed) is relevant for the community and could open new avenues for future research. Regarding the overlap performance (measured by mBO), we agree that PACE does not establish a new state of the art on all three datasets. However, we point out that 1) we never claimed it and 2) PACE still consistently outperform equivalent models that do not leverage multiple cardinalities to gather group evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.