- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes LinSlot, a method that expands a standard object-centric Slot Attention-based autoencoder with an assumption, that each object slot is a sum of transformed attribute representations called blocks, each drawn from its own Gaussian mixture prior. The method evaluated on CLEVR and CLEVR-Tex against plain slot-attention baselines.
Critical strengths
The idea is intuitive as separating object discovery from attribute discovery through an additive slot structure could make a modular approach that can be used on top of plain slot attention.
Critical weaknesses
- Novelty. The block inference module reuses SysBinder's block refinement and concept-memory projection. Meanwhile the GMM prior, from my understanding, is not implemented as a separate component, and it is enforced only implicitly through the prototype projections. The identifiability theorem assumes M·d_block ≤ d_slot, which the reported configurations violate.
- Experiment evaluation. First, the paper only considers simple synthetic datasets (CLEVR, CLEVR-Tex, and MOVi-C), and MOVi-C is used only for the linearity probe. Many relevant baselines are missing: Neural Concept Binder is cited but not compared, Object-centric disentanglement methods are neither discussed nor compared (e.g. Slot-VAE [1], NLoTM with block-level quantization [2], Slot Mixture Module [3], VQ-SA [4]. The reported gains cannot be attributed to the proposed modifications. LinSlot builds on a modified SLATE+, which alone raises FG-ARI, while SysBinder uses its own backbone. Object discovery is reported only with FG-ARI, which can be high even with poor masks, standard metrics like mBO or mIoU with Hungarian matching are missing.
Questions for authors
- The slot posterior can be any pretrained slot model. How does LinSlot perform as a plug-in on DINOSAUR (feature reconstruction) and on a generative slot model such as CODA or SlotDiffusion with real-world data?
- How does LinSlot compare with SysBinder on the same SLATE+ backbone, and with [1]–[4] and Neural Concept Binder? Can you report mBO or mIoU?
- Since the experiments use M·d_block > d_slot, does the identifiability result apply to the reported models?
5 comments
W1.3 & Q3. M·d_block ≤ d_slot.
We note that although we upper bound the inherent dimension of blocks by the number of prototypes used, the inherent dimension of blocks is much lower. This can be seen in the table below, which reports the intrinsic dimension of the block representations and of the concept memory, measured by the participation ratio (PR) and the effective rank (ERank) of the PCA spectrum. Both live in a 192-dimensional ambient space. Values are averaged across blocks and reported as mean ± std over 5 seeds.
| Dataset | Blocks: PR | Blocks: ERank | Concept memory: PR | Concept memory: ERank |
|---|---|---|---|---|
| CLEVR-Easy | 5.87 ± 0.12 | 8.39 ± 0.22 | 20.68 ± 0.43 | 29.30 ± 0.48 |
| CLEVR-Hard | 5.92 ± 0.29 | 8.64 ± 0.29 | 17.35 ± 0.76 | 23.99 ± 1.02 |
We have also added this table (Table 4) and discussed the practicality of this assumption in detail in appendix A.5 of the updated paper.
W2.1 MOVi-C results.
We agree: scaling the method to the real world would demonstrate its true potential. At the same time, we would like to note that unsupervised disentanglement is a hard problem, with no prior work showing any results on real data. To make matters worse, based on our experiments, and as observed by earlier work (Figure 2 in [B]), even unsupervised slot discovery is a challenging problem on real-world datasets.
Nevertheless, we did a preliminary experiment on MOVi-C with CODA-Base as the slot method. The results of swapping (Figure 7 (a) and fIgure 12) show evidence that our method is able to represent factors into the blocks. However, we also identify the need for a better slot method and decoder as intermediate steps towards scaling to real world images. We added the experiment details and results in subsection 5.5 and the appendix sub-section C.8 (Results on MOVi-C) of the updated paper.
W2.2 Neural Concept Binder results.
We have added two more block-slot baselines, Neural Concept Binder (NCB) and Neural Language of Thought Model (NLoTM). Our method continues to outperform these newly added baselines in D, I and FG-ARI scores, as shown in the table below (mean ± std over 5 seeds; best mean per dataset in bold).
| Dataset | Method | D ↑ | C ↑ | I ↑ | FG-ARI ↑ |
|---|---|---|---|---|---|
| CLEVR-Easy | SysBinder | 0.8515 ± 0.1647 | 0.6930 ± 0.0726 | 0.9708 ± 0.0017 | 0.9245 ± 0.0055 |
| NLoTM | 0.6488 ± 0.1855 | 0.4816 ± 0.1135 | 0.4854 ± 0.2736 | 0.1993 ± 0.3886 | |
| NCB | 0.4503 ± 0.1517 | 0.9103 ± 0.0509 | 0.7219 ± 0.0316 | 0.9245 ± 0.0055 | |
| LinSlot | 0.9686 ± 0.0221 | 0.7322 ± 0.0445 | 0.9621 ± 0.0021 | 0.9122 ± 0.0050 | |
| LinSlotB | 0.9262 ± 0.0573 | 0.6664 ± 0.0759 | 0.9713 ± 0.0004 | 0.9256 ± 0.0048 | |
| CLEVR-Hard | SysBinder | 0.7298 ± 0.0755 | 0.6196 ± 0.0475 | 0.9180 ± 0.0019 | 0.9120 ± 0.0050 |
| NLoTM | 0.7009 ± 0.2161 | 0.5443 ± 0.1887 | 0.6710 ± 0.3009 | 0.5647 ± 0.4623 | |
| NCB | 0.6026 ± 0.1142 | 0.7737 ± 0.0997 | 0.6642 ± 0.0624 | 0.9120 ± 0.0050 | |
| LinSlot | 0.8341 ± 0.0407 | 0.6514 ± 0.0398 | 0.8494 ± 0.0084 | 0.9154 ± 0.0091 | |
| LinSlotB | 0.7808 ± 0.0648 | 0.6542 ± 0.0345 | 0.9155 ± 0.0010 | 0.9416 ± 0.0025 |
These results are also added to the updated paper (Table 9, Appendix C.3).
W2.3 & Q2. Slot-VAE, SMM, NLoTM, VQ-SA.
We thank the reviewer for pointing out these relevant works. We have added the discussion on Slot-VAE, SMM and VQ-SA in the related work section of the updated paper. The comparison with NLoTM is given in W2.2 above.
W2.4 & Q2. LinSlot builds on the SLATE+ decoder.
We experimented with SysBinder with the SLATE+ decoder on the CLEVR-Easy and CLEVR-Hard datasets. The results are as follows (mean ± std over 5 seeds; best mean per dataset in bold):
| Dataset | Method | D ↑ | C ↑ | I ↑ | FG-ARI ↑ |
|---|---|---|---|---|---|
| CLEVR-Easy | SysBinder | 0.8515 ± 0.1647 | 0.6930 ± 0.0726 | 0.9708 ± 0.0017 | 0.9245 ± 0.0055 |
| SysBinder (SLATE+ decoder) | 0.9039 ± 0.0731 | 0.6654 ± 0.0524 | 0.9700 ± 0.0017 | 0.8454 ± 0.1060 | |
| CLEVR-Hard | SysBinder | 0.7298 ± 0.0755 | 0.6196 ± 0.0475 | 0.9180 ± 0.0019 | 0.9120 ± 0.0050 |
| SysBinder (SLATE+ decoder) | 0.7645 ± 0.0641 | 0.5901 ± 0.0594 | 0.8654 ± 0.0214 | 0.9030 ± 0.0203 |
We did observe an improvement in the D score, but the I, C and FG-ARI scores dropped. However, the D, C and FG-ARI scores are still better for LinSlot. The drop is consistent with the lower I score in our method LinSlot. This demonstrates that the performance gain in LinSlot is not solely due to the SLATE+ decoder. These results are also updated in the paper in appendix C.4.
W2.5 & Q2. mBO, mIoU.
We computed the scores on these metrics; the results are shown in the table below (mean ± std over 5 seeds; best mean per dataset in bold).
| Dataset | Method | FG-ARI ↑ | MIOU ↑ | mBO ↑ |
|---|---|---|---|---|
| CLEVR-Easy | Slot Attention | 0.8152 ± 0.1012 | 0.3517 ± 0.0069 | 0.3670 ± 0.0140 |
| SLATE+ | 0.9369 ± 0.0020 | 0.3960 ± 0.0074 | 0.4234 ± 0.0063 | |
| SysBinder | 0.9245 ± 0.0055 | 0.3826 ± 0.0312 | 0.4139 ± 0.0214 | |
| LinSlot | 0.9122 ± 0.0050 | 0.3671 ± 0.0085 | 0.3981 ± 0.0067 | |
| LinSlotB | 0.9256 ± 0.0048 | 0.4040 ± 0.0228 | 0.4242 ± 0.0157 | |
| CLEVR-Hard | Slot Attention | 0.9720 ± 0.0043 | 0.3761 ± 0.0151 | 0.4086 ± 0.0180 |
| SLATE+ | 0.9440 ± 0.0062 | 0.3465 ± 0.0047 | 0.3795 ± 0.0022 | |
| SysBinder | 0.9120 ± 0.0050 | 0.3488 ± 0.0056 | 0.3734 ± 0.0025 | |
| LinSlot | 0.9154 ± 0.0091 | 0.3404 ± 0.0081 | 0.3742 ± 0.0089 | |
| LinSlotB | 0.9416 ± 0.0025 | 0.3662 ± 0.0168 | 0.3976 ± 0.0124 | |
| CLEVR-Tex | Slot Attention | 0.5709 ± 0.0220 | 0.7417 ± 0.0151 | 0.6663 ± 0.0057 |
| SLATE+ | 0.8733 ± 0.0015 | 0.7154 ± 0.0166 | 0.6975 ± 0.0102 | |
| SysBinder | 0.7355 ± 0.0490 | 0.7990 ± 0.0200 | 0.7517 ± 0.0240 | |
| LinSlot | 0.8500 ± 0.0201 | 0.6420 ± 0.0216 | 0.6279 ± 0.0149 | |
| LinSlotB | 0.8443 ± 0.0169 | 0.7507 ± 0.0325 | 0.7148 ± 0.0341 |
We see that the trends in CLEVR-Easy and CLEVR-Hard are similar to that of FG-ARI. However, the CLEVR-Tex results show SysBinder with better scores, which we attribute to its better background separation. We have added the scores on these metrics in the appendix subsection C.2 (Table 8) of the updated paper.
Q1. LinSlot with different slot posteriors
This is a good suggestion. We agree that a factored method such as ours should scale to any slot posterior method. Accordingly, we have conducted the following experiments:
1. We experimented with Slot Attention as the slot posterior on the CLEVR-Easy dataset and Slot Diffusion as the slot posterior on the CLEVR-Tex dataset. We show that even with minimal hyperparameter tuning our method is able to achieve comparable results to the SLATE+ posterior and is also able to achieve the swapping of factors (see Figure 11 in the updated paper). The results are as below (mean ± std over seeds; best mean per dataset in bold):
| Dataset | Slot Posterior | D ↑ | C ↑ | I ↑ | FG-ARI ↑ |
|---|---|---|---|---|---|
| CLEVR-Easy | SLATE+ | 0.9686 ± 0.0221 | 0.7322 ± 0.0445 | 0.9621 ± 0.0021 | 0.9122 ± 0.0050 |
| Slot Attention | 0.8267 ± 0.0967 | 0.5834 ± 0.0959 | 0.9184 ± 0.0403 | 0.8826 ± 0.1036 | |
| CLEVR-Tex | SLATE+ | 0.8815 ± 0.1160 | 0.6093 ± 0.0981 | 0.8362 ± 0.0064 | 0.8500 ± 0.0201 |
| Slot Diffusion | 0.7161 ± 0.1141 | 0.4498 ± 0.0914 | 0.7336 ± 0.0064 | 0.7662 ± 0.0068 |
2. Additionally, we also experimented with CODA-Base as the slot posterior method on the naturalistic MOVi-C dataset. These preliminary results demonstrate the effectiveness of our method on any slot posterior and also on hard naturalistic datasets.
These results have also been added to the updated paper: the results with different slot posteriors are in appendix subsection C.7, and the details and qualitative results on MOVi-C are in section 5.5 and appendix subsection C.8 of the updated paper.
Q2. Neural Concept Binder results.
Addressed in W2.2.
References
[A] Zhuxi Jiang et al. Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
[B] Hongjia Liu et al. MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning
Thank you for raising these relevant points. We address them below.
W1.1 Novelty.
While it is true that our architecture builds on Sysbinder’s block refinement and concept-memory projection, there are several other fundamental differences:
Sysbinder uses a fixed chunking over object (slot) representation to discover attributes (blocks). In contrast, in our model, we have an explicit attention mechanism over the discovered slots to discover the attributes, paving way for a more flexible mechanism for attribute discovery.
Whereas Sysbinder’s block inference is coupled with slot discovery, our block discovery mechanism is decoupled, allowing for a two-stage discovery. This two stage discovery is crucial for specific experimental gains that we observe in our experiments (refer to figure 5), while also making us robust to various architectural choices (refer Table 3).
At a conceptual level, we would like to clarify the core contributions with respect to Sysbinder as follows:
We formally show the existence of LRH on various benchmarks used in our experiments. This is further used to derive a well-grounded theoretical framework for discovery of attributes in our model. Sysbinder’s formulation does not show the existence of any such linear representation of attributes in terms of the discovered slots.
Our model theoretically grounds the attribute discovery in the Linear Representation Hypothesis. It starts with a GMM prior over attributes and then develops a generative model for attribute discovery. This allows us to prove specific identifiability results for attribute discovery. We believe there is no existing prior work (including Sysbinder) which has developed such a theoretical framework for attribute discovery.
Finally, we would like to point out that while our model uses the same concept memory as Sysbinder, our ELBO-based formulation, along with the equivalence of our GMM prior with concept memory, provides a strong theoretical justification for these architectural choices in the two models. Also note that our ELBO formulation using the factored variational inference reveals three terms (a) Reconstruction (b )slot regularizer and (c) block regularizer. These loss terms in the ELBO are novel contributions of our work and further open additional pathways for further study. Systbinder provided no theoretical justification for the specific choices made in their work.
.
W1.2 GMM prior not enforced directly (used prototype projections).
The reviewer's understanding is correct that in our implementation we use prototype projections instead of the GMM prior. While we agree that training with a GMM prior would more closely match the theory developed in the paper, we found the training of such a prior to be non-trivial in our experiments. We note that this has also been pointed out by earlier work, e.g., Kivva et al. (see Figure 1 and the conclusion in their paper), as the optimization gets stuck in local maxima.
To investigate this further, and to test the effect of the assumptions on the identifiability, we performed a controlled experiment on simulated slot data. The slot data is generated from M = 3 blocks, each having 6 mixture components (K_1 = K_2 = K_3 = 6). The slot dimension is d_slot = 48 and d_block = 16. The mean and covariance matrix are randomly initialized for the components. The mixing functions f_1, f_2, f_3 are affine transformations such that f is injective (implemented as an orthonormal matrix). Given this simulated data, the inference model is designed as a simple linear transformation of the slot to give the blocks. That is, the mean and diagonal covariance matrix of the block posterior are obtained by simply taking the block-specific linear transformation of slots. We then tried a 3-stage training strategy from VaDE [A] to train the model with the GMM prior on this simulated data. The results are presented in the table below. The results are added in the subsection 5.6 and additional details are added in the appendix C.5 of the updated paper.
We observe that the GMM prior was able to identify the ground truth blocks into inferred blocks under the assumptions of the theorem. Further, we also show the presence of GMM structure in the slot space of SLATE+ (please refer to appendix subsection C.6). These results have also been added to the updated paper (see Appendix C.5).