acceptodds
Under review as a conference paper at ICLR 2027

LINSLOT: Exploiting Linear Representation hypothesis for unsupervised attribute discovery from slot based object representation

Abstract

This paper studies the problem of learning disentangled representations of ob- jects and their attributes from raw, unstructured image data. Slot-based methods have shown considerable success in unsupervised learning of object representa- tions from images. Block-slot attention-based methods extend this framework to attribute representations by assuming a uniform factorization of object representa- tions into attributes, which may be suboptimal and consequently limit the quality of the learned representations. We therefore investigate a framework for jointly discovering object and attribute representations. Our key contribution is leverag- ing the Linear Representation Hypothesis (LRH), which postulates that compos- able concepts can be represented as linearly additive subspaces in slot representa- tions. Based on this insight, we propose a probabilistic model connecting images, slots (objects), and blocks (attributes). We present an architecture that leverages block attention to connect attribute representations to slots and incorporates LRH in both object and attribute representation spaces. This architecture effectively optimizes the Evidence Lower Bound (ELBO) of the proposed graphical model. Our experiments demonstrate (i) effective discovery of disentangled object and at- tribute representations, (ii) empirical evidence for LRH in slot space, and (iii) the ability to perform image editing owing to the disentangled and interpretable nature of the learned representations. Our experiments on multiple datasets demonstrate improvements in DCI scores over state-of-the-art methods

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
object-centric learning, Slot Attention, unsupervised representation learning, disentanglement, concept learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4128OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LinSlot, a method that expands a standard object-centric Slot Attention-based autoencoder with an assumption, that each object slot is a sum of transformed attribute representations called blocks, each drawn from its own Gaussian mixture prior. The method evaluated on CLEVR and CLEVR-Tex against plain slot-attention baselines.

Critical strengths

The idea is intuitive as separating object discovery from attribute discovery through an additive slot structure could make a modular approach that can be used on top of plain slot attention.

Critical weaknesses
  1. Novelty. The block inference module reuses SysBinder's block refinement and concept-memory projection. Meanwhile the GMM prior, from my understanding, is not implemented as a separate component, and it is enforced only implicitly through the prototype projections. The identifiability theorem assumes M·d_block ≤ d_slot, which the reported configurations violate.
  2. Experiment evaluation. First, the paper only considers simple synthetic datasets (CLEVR, CLEVR-Tex, and MOVi-C), and MOVi-C is used only for the linearity probe. Many relevant baselines are missing: Neural Concept Binder is cited but not compared, Object-centric disentanglement methods are neither discussed nor compared (e.g. Slot-VAE [1], NLoTM with block-level quantization [2], Slot Mixture Module [3], VQ-SA [4]. The reported gains cannot be attributed to the proposed modifications. LinSlot builds on a modified SLATE+, which alone raises FG-ARI, while SysBinder uses its own backbone. Object discovery is reported only with FG-ARI, which can be high even with poor masks, standard metrics like mBO or mIoU with Hungarian matching are missing.
Questions for authors
  1. The slot posterior can be any pretrained slot model. How does LinSlot perform as a plug-in on DINOSAUR (feature reconstruction) and on a generative slot model such as CODA or SlotDiffusion with real-world data?
  2. How does LinSlot compare with SysBinder on the same SLATE+ backbone, and with [1]–[4] and Neural Concept Binder? Can you report mBO or mIoU?
  3. Since the experiments use M·d_block > d_slot, does the identifiability result apply to the reported models?
5 comments

Thank you for raising these relevant points. We address them below.

W1.1 Novelty.

While it is true that our architecture builds on Sysbinder’s block refinement and concept-memory projection, there are several other fundamental differences:

  1. Sysbinder uses a fixed chunking over object (slot) representation to discover attributes (blocks). In contrast, in our model, we have an explicit attention mechanism over the discovered slots to discover the attributes, paving way for a more flexible mechanism for attribute discovery.

  2. Whereas Sysbinder’s block inference is coupled with slot discovery, our block discovery mechanism is decoupled, allowing for a two-stage discovery. This two stage discovery is crucial for specific experimental gains that we observe in our experiments (refer to figure 5), while also making us robust to various architectural choices (refer Table 3).

At a conceptual level, we would like to clarify the core contributions with respect to Sysbinder as follows:

  1. We formally show the existence of LRH on various benchmarks used in our experiments. This is further used to derive a well-grounded theoretical framework for discovery of attributes in our model. Sysbinder’s formulation does not show the existence of any such linear representation of attributes in terms of the discovered slots.

  2. Our model theoretically grounds the attribute discovery in the Linear Representation Hypothesis. It starts with a GMM prior over attributes and then develops a generative model for attribute discovery. This allows us to prove specific identifiability results for attribute discovery. We believe there is no existing prior work (including Sysbinder) which has developed such a theoretical framework for attribute discovery.

  3. Finally, we would like to point out that while our model uses the same concept memory as Sysbinder, our ELBO-based formulation, along with the equivalence of our GMM prior with concept memory, provides a strong theoretical justification for these architectural choices in the two models. Also note that our ELBO formulation using the factored variational inference reveals three terms (a) Reconstruction (b )slot regularizer and (c) block regularizer. These loss terms in the ELBO are novel contributions of our work and further open additional pathways for further study. Systbinder provided no theoretical justification for the specific choices made in their work.

.

W1.2 GMM prior not enforced directly (used prototype projections).

The reviewer's understanding is correct that in our implementation we use prototype projections instead of the GMM prior. While we agree that training with a GMM prior would more closely match the theory developed in the paper, we found the training of such a prior to be non-trivial in our experiments. We note that this has also been pointed out by earlier work, e.g., Kivva et al. (see Figure 1 and the conclusion in their paper), as the optimization gets stuck in local maxima.

To investigate this further, and to test the effect of the assumptions on the identifiability, we performed a controlled experiment on simulated slot data. The slot data is generated from M = 3 blocks, each having 6 mixture components (K_1 = K_2 = K_3 = 6). The slot dimension is d_slot = 48 and d_block = 16. The mean and covariance matrix are randomly initialized for the components. The mixing functions f_1, f_2, f_3 are affine transformations such that f is injective (implemented as an orthonormal matrix). Given this simulated data, the inference model is designed as a simple linear transformation of the slot to give the blocks. That is, the mean and diagonal covariance matrix of the block posterior are obtained by simply taking the block-specific linear transformation of slots. We then tried a 3-stage training strategy from VaDE [A] to train the model with the GMM prior on this simulated data. The results are presented in the table below. The results are added in the subsection 5.6 and additional details are added in the appendix C.5 of the updated paper.

K for each blockK_i ≠ K_w K_uWeak InjectiveDCI
6, 6, 6✓✓0.53 ± 0.110.53 ± 0.110.98 ± 0.00
12, 12, 12✓✓0.39 ± 0.130.40 ± 0.130.98 ± 0.00
6, 6, 36✗✓0.33 ± 0.120.37 ± 0.110.97 ± 0.00
6, 6, 6✓✗0.04 ± 0.030.04 ± 0.030.82 ± 0.00

We observe that the GMM prior was able to identify the ground truth blocks into inferred blocks under the assumptions of the theorem. Further, we also show the presence of GMM structure in the slot space of SLATE+ (please refer to appendix subsection C.6). These results have also been added to the updated paper (see Appendix C.5).

W1.3 & Q3. M·d_block ≤ d_slot.

We note that although we upper bound the inherent dimension of blocks by the number of prototypes used, the inherent dimension of blocks is much lower. This can be seen in the table below, which reports the intrinsic dimension of the block representations and of the concept memory, measured by the participation ratio (PR) and the effective rank (ERank) of the PCA spectrum. Both live in a 192-dimensional ambient space. Values are averaged across blocks and reported as mean ± std over 5 seeds.

DatasetBlocks: PRBlocks: ERankConcept memory: PRConcept memory: ERank
CLEVR-Easy5.87 ± 0.128.39 ± 0.2220.68 ± 0.4329.30 ± 0.48
CLEVR-Hard5.92 ± 0.298.64 ± 0.2917.35 ± 0.7623.99 ± 1.02

We have also added this table (Table 4) and discussed the practicality of this assumption in detail in appendix A.5 of the updated paper.

W2.1 MOVi-C results.

We agree: scaling the method to the real world would demonstrate its true potential. At the same time, we would like to note that unsupervised disentanglement is a hard problem, with no prior work showing any results on real data. To make matters worse, based on our experiments, and as observed by earlier work (Figure 2 in [B]), even unsupervised slot discovery is a challenging problem on real-world datasets.

Nevertheless, we did a preliminary experiment on MOVi-C with CODA-Base as the slot method. The results of swapping (Figure 7 (a) and fIgure 12) show evidence that our method is able to represent factors into the blocks. However, we also identify the need for a better slot method and decoder as intermediate steps towards scaling to real world images. We added the experiment details and results in subsection 5.5 and the appendix sub-section C.8 (Results on MOVi-C) of the updated paper.

W2.2 Neural Concept Binder results.

We have added two more block-slot baselines, Neural Concept Binder (NCB) and Neural Language of Thought Model (NLoTM). Our method continues to outperform these newly added baselines in D, I and FG-ARI scores, as shown in the table below (mean ± std over 5 seeds; best mean per dataset in bold).

DatasetMethodD ↑C ↑I ↑FG-ARI ↑
CLEVR-EasySysBinder0.8515 ± 0.16470.6930 ± 0.07260.9708 ± 0.00170.9245 ± 0.0055
NLoTM0.6488 ± 0.18550.4816 ± 0.11350.4854 ± 0.27360.1993 ± 0.3886
NCB0.4503 ± 0.15170.9103 ± 0.05090.7219 ± 0.03160.9245 ± 0.0055
LinSlot0.9686 ± 0.02210.7322 ± 0.04450.9621 ± 0.00210.9122 ± 0.0050
LinSlotB0.9262 ± 0.05730.6664 ± 0.07590.9713 ± 0.00040.9256 ± 0.0048
CLEVR-HardSysBinder0.7298 ± 0.07550.6196 ± 0.04750.9180 ± 0.00190.9120 ± 0.0050
NLoTM0.7009 ± 0.21610.5443 ± 0.18870.6710 ± 0.30090.5647 ± 0.4623
NCB0.6026 ± 0.11420.7737 ± 0.09970.6642 ± 0.06240.9120 ± 0.0050
LinSlot0.8341 ± 0.04070.6514 ± 0.03980.8494 ± 0.00840.9154 ± 0.0091
LinSlotB0.7808 ± 0.06480.6542 ± 0.03450.9155 ± 0.00100.9416 ± 0.0025

These results are also added to the updated paper (Table 9, Appendix C.3).

W2.3 & Q2. Slot-VAE, SMM, NLoTM, VQ-SA.

We thank the reviewer for pointing out these relevant works. We have added the discussion on Slot-VAE, SMM and VQ-SA in the related work section of the updated paper. The comparison with NLoTM is given in W2.2 above.

W2.4 & Q2. LinSlot builds on the SLATE+ decoder.

We experimented with SysBinder with the SLATE+ decoder on the CLEVR-Easy and CLEVR-Hard datasets. The results are as follows (mean ± std over 5 seeds; best mean per dataset in bold):

DatasetMethodD ↑C ↑I ↑FG-ARI ↑
CLEVR-EasySysBinder0.8515 ± 0.16470.6930 ± 0.07260.9708 ± 0.00170.9245 ± 0.0055
SysBinder (SLATE+ decoder)0.9039 ± 0.07310.6654 ± 0.05240.9700 ± 0.00170.8454 ± 0.1060
CLEVR-HardSysBinder0.7298 ± 0.07550.6196 ± 0.04750.9180 ± 0.00190.9120 ± 0.0050
SysBinder (SLATE+ decoder)0.7645 ± 0.06410.5901 ± 0.05940.8654 ± 0.02140.9030 ± 0.0203

We did observe an improvement in the D score, but the I, C and FG-ARI scores dropped. However, the D, C and FG-ARI scores are still better for LinSlot. The drop is consistent with the lower I score in our method LinSlot. This demonstrates that the performance gain in LinSlot is not solely due to the SLATE+ decoder. These results are also updated in the paper in appendix C.4.

W2.5 & Q2. mBO, mIoU.

We computed the scores on these metrics; the results are shown in the table below (mean ± std over 5 seeds; best mean per dataset in bold).

DatasetMethodFG-ARI ↑MIOU ↑mBO ↑
CLEVR-EasySlot Attention0.8152 ± 0.10120.3517 ± 0.00690.3670 ± 0.0140
SLATE+0.9369 ± 0.00200.3960 ± 0.00740.4234 ± 0.0063
SysBinder0.9245 ± 0.00550.3826 ± 0.03120.4139 ± 0.0214
LinSlot0.9122 ± 0.00500.3671 ± 0.00850.3981 ± 0.0067
LinSlotB0.9256 ± 0.00480.4040 ± 0.02280.4242 ± 0.0157
CLEVR-HardSlot Attention0.9720 ± 0.00430.3761 ± 0.01510.4086 ± 0.0180
SLATE+0.9440 ± 0.00620.3465 ± 0.00470.3795 ± 0.0022
SysBinder0.9120 ± 0.00500.3488 ± 0.00560.3734 ± 0.0025
LinSlot0.9154 ± 0.00910.3404 ± 0.00810.3742 ± 0.0089
LinSlotB0.9416 ± 0.00250.3662 ± 0.01680.3976 ± 0.0124
CLEVR-TexSlot Attention0.5709 ± 0.02200.7417 ± 0.01510.6663 ± 0.0057
SLATE+0.8733 ± 0.00150.7154 ± 0.01660.6975 ± 0.0102
SysBinder0.7355 ± 0.04900.7990 ± 0.02000.7517 ± 0.0240
LinSlot0.8500 ± 0.02010.6420 ± 0.02160.6279 ± 0.0149
LinSlotB0.8443 ± 0.01690.7507 ± 0.03250.7148 ± 0.0341

We see that the trends in CLEVR-Easy and CLEVR-Hard are similar to that of FG-ARI. However, the CLEVR-Tex results show SysBinder with better scores, which we attribute to its better background separation. We have added the scores on these metrics in the appendix subsection C.2 (Table 8) of the updated paper.

Q1. LinSlot with different slot posteriors

This is a good suggestion. We agree that a factored method such as ours should scale to any slot posterior method. Accordingly, we have conducted the following experiments:

1. We experimented with Slot Attention as the slot posterior on the CLEVR-Easy dataset and Slot Diffusion as the slot posterior on the CLEVR-Tex dataset. We show that even with minimal hyperparameter tuning our method is able to achieve comparable results to the SLATE+ posterior and is also able to achieve the swapping of factors (see Figure 11 in the updated paper). The results are as below (mean ± std over seeds; best mean per dataset in bold):

DatasetSlot PosteriorD ↑C ↑I ↑FG-ARI ↑
CLEVR-EasySLATE+0.9686 ± 0.02210.7322 ± 0.04450.9621 ± 0.00210.9122 ± 0.0050
Slot Attention0.8267 ± 0.09670.5834 ± 0.09590.9184 ± 0.04030.8826 ± 0.1036
CLEVR-TexSLATE+0.8815 ± 0.11600.6093 ± 0.09810.8362 ± 0.00640.8500 ± 0.0201
Slot Diffusion0.7161 ± 0.11410.4498 ± 0.09140.7336 ± 0.00640.7662 ± 0.0068

2. Additionally, we also experimented with CODA-Base as the slot posterior method on the naturalistic MOVi-C dataset. These preliminary results demonstrate the effectiveness of our method on any slot posterior and also on hard naturalistic datasets.

These results have also been added to the updated paper: the results with different slot posteriors are in appendix subsection C.7, and the details and qualitative results on MOVi-C are in section 5.5 and appendix subsection C.8 of the updated paper.

Q2. Neural Concept Binder results.

Addressed in W2.2.

References

[A] Zhuxi Jiang et al. Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering

[B] Hongjia Liu et al. MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning

Reviewer_iuueOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a framework (probabilistic model) named LinSlot which connects images, slots (objects), and blocks (attributes). Concretely, it consists of a generative model where each object slot is a linear sum of transformed attribute blocks, each block drawn from its own GMM prior. Their experiments test linearity in the slot space on four datasets via the cosine similarity of delta directions, and evaluate attribute disentanglement and object discovery using DCI and FG-ARI scores. LinSlot show competitive results on these tasks compared to the baselines.

Critical strengths
  1. The linearity of attribute directions in the slot (object) representation space is a useful empirical contribution. It is interesting to study this for practice.
  2. The sequential slot-then-block design is practical. Any pretrained slot model can serve as the first stage.
Critical weaknesses
  1. The novelty of LinSlot is limited. The method appears to be a combination of existing work: It is SysBinder with the block inference moved from the image to the slot, plus a linearity-enforcing regularizer, i.e., Block-Slot Attention with a probabilistic justification.

  2. The theoretical contribution is also not clear. First, Section 4.2 Identifiability is not completed. Second, it builds on Kivva et al.. The used argument from Kivva et al. requires assumptions P1, P2 on the latent prior and F1, F4 on the decoder (please refer to their original paper). LinSlot states only P1 (GMM prior) and F1 (piecewise-affine f). P2 and F4 (injectivity) are not verified for block priors.

  3. Experimental results are narrow (only synthetic datasets) and not convincing. For instance, attribute discovery is evaluated on synthetic CLEVR datasets with only one baseline. The I score is below the best method on CLEVRs, and MOVi-C is used only for the linearity test. No attribute discovery is shown on real images.

Questions for authors
  1. The authors need to report the std along with the means in the figures or tables, and state the number of seeds.

  2. Some sentences are incomplete or broken, e.g., line 220, 288, and a missing reference in line 116.

  3. The readability of the paper can be improved, such as the conditions in the definitions (the indices i, u, w, m, M, K_i, K_j in Informal Theorem 1) and the equations.

  4. What is e_i in Fig. 2 line 9?

  5. Why are Neural Concept Binder and other recent baselines not included in the comparison?

3 comments

Thank you for raising these relevant points. We address them below.

W1. Novelty (LinSlot appears to be SysBinder with block inference moved from the image to the slot, along with a linearity-enforcing regularizer, i.e., Block-Slot Attention with a probabilistic justification).

We would like to point out that while the reviewer's understanding of our core contribution with respect to Systbinder is correct, we do not agree that these can be seen as minor or incremental contributions. Taking the block attention from directly attending over the image, to at the slot level, is in fact a core contribution at the heart of our work. While the first one thinks of attribute discovery as a direct consequence of object discovery, our architecture ensures that it is modeled as a two stage process, first doing the object discovery followed by attribute discovery. This is central to the key theme of our paper, and is central to the gains obtained in our experimental results.

Further, our work shows the existence of LRH for existing datasets, both artificial as well as the naturalistic one. This, along with the assumption of a GMM prior over attributes, is further used to develop the theory of a probabilistic generative model, and prove identifiability results. We strongly believe that this theoretical grounding is an important contribution. We would like to submit that there is no prior work whatsoever which has given any identifiability results for attribute discovery in multi object setting. Our work not only provides the first theoretical framework for this task, it also grounds our specific architectural choices in this theory via the use of LRH and GMM priors.

Further, we would like to point out that while our model uses the same concept memory as Sysbinder, our ELBO-based formulation , along with equivalence of our GMM prior with concept memory, provides a strong theoretical justification for these architectural choices in the two models. Systbinder provided no theoretical justification for the specific choices made in their work. Also note that our ELBO formulation using the factored variational inference reveals three terms (a) Reconstruction (b )slot regularizer and (c) block regularizer. These loss terms in the ELBO are novel contributions of our work and further open additional pathways for further study.

W2. Theoretical contribution and missing assumptions (Section 4.2 Identifiability is not completed; the argument from Kivva et al. requires assumptions P1, P2 on the latent prior and F1, F4 on the decoder, but LinSlot states only P1 and F1; P2 and F4 (injectivity) are not verified for block priors).

We thank the reviewer for pointing out the missing assumptions in Section 4.2. We agree that our presentation of the identifiability result was incomplete as we did not explicitly state all the assumptions required to invoke the results of Kivva et al.

Our results need the weaker F2 assumption from Kivva et al. (instead of the stronger F4 assumption) as we need to invoke Theorem 3.2 (b) from Kivva et al. We discuss P2 and F2 below.

Regarding P2: This assumption has two parts. The first part is the independence among the block (concatenated) dimensions given the underlying concept index (of concatenated blocks), which is satisfied if one uses a diagonal covariance matrix in the concept value distributions. This part of the assumption is satisfied when using the concept prototypes as block regularizer. However, the second part of this assumption needs the existence of two concept indices which have different diagonal entries in their corresponding covariance matrices of the concept value distribution. This part of the assumption cannot be enforced with the concept prototype regularizer, as it uses constant and equal covariance matrices. We leave incorporating this part of the assumption with concept prototypes as future work.

Regarding F2: For f to be weakly injective, the individual f_j s need to be weakly injective. In the case of f_j s being affine functions, this translates to the rank of W_j being d_block. Explicitly enforcing this condition within a gradient-based learning framework is non-trivial. Therefore, we leave this constraint unenforced during training.

In the revised manuscript, we have explicitly stated assumptions P2 and F2 and clarified their correspondence to our block-based formulation. We also revised Section 4.2 to clearly mention the assumptions. Additionally, we added appendix sub-section A.1 (Assumptions and Definitions) for explicitly defining the assumptions and the identifiability definition, and appendix sub-section A.5 (Practicality of the Assumptions) to discuss the practicality of the assumptions in our setting.

W3.1 Attribute discovery is evaluated on only one baseline.

We note that attribute discovery is already compared against three baselines in the submission: two slot-based methods (Slot Attention, SLATE+) and one block-slot-based method (SysBinder) (Section 5.3, Figure 5 in the original (and updated) submission).

To further strengthen this evaluation, we have added two more block-slot baselines, Neural Concept Binder (NCB) and Neural Language of Thought Model (NLoTM). Our method continues to outperform these newly added baselines in D, I and FG-ARI scores, as shown in the table below (mean ± std over 5 seeds; best mean per dataset in bold).

DatasetMethodD ↑C ↑I ↑FG-ARI ↑
CLEVR-EasySysBinder0.8515 ± 0.16470.6930 ± 0.07260.9708 ± 0.00170.9245 ± 0.0055
NLoTM0.6488 ± 0.18550.4816 ± 0.11350.4854 ± 0.27360.1993 ± 0.3886
NCB0.4503 ± 0.15170.9103 ± 0.05090.7219 ± 0.03160.9245 ± 0.0055
LinSlot0.9686 ± 0.02210.7322 ± 0.04450.9621 ± 0.00210.9122 ± 0.0050
LinSlotB0.9262 ± 0.05730.6664 ± 0.07590.9713 ± 0.00040.9256 ± 0.0048
CLEVR-HardSysBinder0.7298 ± 0.07550.6196 ± 0.04750.9180 ± 0.00190.9120 ± 0.0050
NLoTM0.7009 ± 0.21610.5443 ± 0.18870.6710 ± 0.30090.5647 ± 0.4623
NCB0.6026 ± 0.11420.7737 ± 0.09970.6642 ± 0.06240.9120 ± 0.0050
LinSlot0.8341 ± 0.04070.6514 ± 0.03980.8494 ± 0.00840.9154 ± 0.0091
LinSlotB0.7808 ± 0.06480.6542 ± 0.03450.9155 ± 0.00100.9416 ± 0.0025

It can be observed from the results that the both new added baselines for block slot attention perform worse than both versions of our method, except fot C score in the NCB. However NCB have weaker D score which is more central for disentanglement.

These results are also added to the updated paper (Table 9, Appendix C.3).

W3.2 I score below the best methods.

Thanks for your comment. We would like to clarify that our best variant, i.e., LinSlot-B, does better than any existing approaches in all metrics, including I score (refer to table 7 in the updated manuscript). We believe the reviewer’s comment is regarding our LinSlot variant, where it does not perform as well compared to other approaches.

Further, we have done additional experiments based on comments from 4128, which show additional light on this issue. We observe that specific lack of performance in I score is stemming from the choice of specific decoder. Specifically, we show that using the same decoder as the LinSlot variant, for Sysbinder, also lowers its I scores, making them comparable to LinSlot’s scores. Please refer to the W2.4 response to the reviewer 4128.

W3.3 MOVi-C is used only for linearity testing.

Based on the reviewer's suggestion, we did a preliminary experiment on MOVi-C with CODA-Base as the slot method. The results of swapping (Figure 7 (a)) show evidence that our method is able to represent factors into the blocks. However, we also identify the need for a better slot method and decoder as intermediate steps towards scaling to real world images. We added the experiment details and results in sub-section 5.5 and the additional detail in the C.8 (Results on MOVi-C) of the updated paper.

Q1. State std and number of seeds.

All results are averaged over 5 random seeds. The object discovery results with mean and standard deviation are shown in the table below (best mean per dataset in bold).

DatasetMethodFG-ARI ↑MIOU ↑mBO ↑
CLEVR-EasySlot Attention0.8152 ± 0.10120.3517 ± 0.00690.3670 ± 0.0140
SLATE+0.9369 ± 0.00200.3960 ± 0.00740.4234 ± 0.0063
SysBinder0.9245 ± 0.00550.3826 ± 0.03120.4139 ± 0.0214
LinSlot0.9122 ± 0.00500.3671 ± 0.00850.3981 ± 0.0067
LinSlotB0.9256 ± 0.00480.4040 ± 0.02280.4242 ± 0.0157
CLEVR-HardSlot Attention0.9720 ± 0.00430.3761 ± 0.01510.4086 ± 0.0180
SLATE+0.9440 ± 0.00620.3465 ± 0.00470.3795 ± 0.0022
SysBinder0.9120 ± 0.00500.3488 ± 0.00560.3734 ± 0.0025
LinSlot0.9154 ± 0.00910.3404 ± 0.00810.3742 ± 0.0089
LinSlotB0.9416 ± 0.00250.3662 ± 0.01680.3976 ± 0.0124
CLEVR-TexSlot Attention0.5709 ± 0.02200.7417 ± 0.01510.6663 ± 0.0057
SLATE+0.8733 ± 0.00150.7154 ± 0.01660.6975 ± 0.0102
SysBinder0.7355 ± 0.04900.7990 ± 0.02000.7517 ± 0.0240
LinSlot0.8500 ± 0.02010.6420 ± 0.02160.6279 ± 0.0149
LinSlotB0.8443 ± 0.01690.7507 ± 0.03250.7148 ± 0.0341

We now report the mean and standard deviation for all methods in Tables 7 and 8 (Appendix C.2) of the updated paper.

Q2. Broken sentences and missing reference.

We thank the reviewer for pointing these out. We have corrected the broken sentences at lines 220 and 288, and added the missing reference to the figure in the updated version of the paper.

Q3. Readability of the paper can be improved, such as the conditions in the definitions (the indices i, u, w, m, M, K_i, K_j in Informal Theorem 1) and the equations.

We thank the reviewer for this feedback, which also helped us catch two typos in Informal Theorem 1: the subscript i on f should have been j, and m should have been M. Both are now corrected. In addition, we have (1) added a statement defining all indices (u, w, j, i) directly before the theorem, and (2) added a dedicated assumptions and definitions section (Appendix A.1) that makes the conditions of the theorem explicit.

Q4. What is e_i in Fig. 2 line 9?

We thank the reviewer for pointing this out. It was a typo on our end. It should have been V_i. We have corrected it in the updated manuscript.

Q5. Neural Concept Binder results.

Addressed in W3.1.

Reviewer_CXk6OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is well written and presented, although soe parts of the paper are quite math heavy thus harder. Unsupervised object and concept discovery are well established and challenging fields The empirical results outperform on several standard baselines

Critical strengths

The paper is well written and presented Unsupervised object and concept discovery are well established and challenging fields The empirical results outperform on several standard baselines

Critical weaknesses

I don’t feel qualified to comment on the variational formulation or identifiability proof. However, I am concerned about the motivation. Concepts are not as flat as disentanglement benchmarks make them. They are strongly dependent – they have hierarchies (animal – dog) and similes (dog – hound). The different blocks or factors are highly correlated e.g. function and appearance are highly correlated. However, these assumptions are often made, also in this work. The academic baselines are especially created to satisfy these assumption, but such methods don’t generalize to real datasets as they do not satisfy these assumptions. Can you justify how this research program will lead to algorithms that can apply to the real world (say fast forward a couple of years)?


Having read the concerns in the other reviews regarding novelty and soundness, I will reduce my score to fit with the consensus of my peers.

Questions for authors

The core concern is high level - the assumptions here are very strong. Please explain how you believe the research program can lead to attribute and object discovery in real data.

1 comment

Thank you for raising these relevant points. We address them below.

W1.1 Concerned about motivation. Concepts are not always disentangled and are hierarchical in real datasets.

Regarding hierarchies in real data: while it is true that real data is often hierarchical, and may require additional enhancements to our method, we believe that is a natural next step. For example, instead of discovering blocks at a single level, there could be level conditioned discovery of blocks (attributes), where attributes (and their refinements) are discovered in a level ordered manner. For that manner, the two stage design of our model, where we discover objects first, and then their attributes, can be thought of as some sort of hierarchy. Extending our approach to explicitly model a notion of hierarchy, and experimenting with data (including real domains) which have such hierarchical attributes is a very promising direction for future work. We would like to note that our current approach to attribute discovery can be seen as discovering a flat hierarchy over a set of attributes.

W1.2 & Q1. Can you justify how this research program will lead to algorithms that can apply to the real world (say fast forward a couple of years)?

This is an important point that you raise. The eventual goal of the work should be (and is) to work on real datasets. At the same time, we believe that unsupervised disentanglement is a very hard problem [1]. And there is very limited literature which has shown success even on synthetic data [2]. Our goal in the current paper was to improve existing methods, which work on multi-object settings, and show that our model can indeed help in improved attribute discovery on existing benchmarks. To the best of our knowledge, none of the existing methods has attempted any experiments on real data in this regard. Another motivation for our work was also to develop strong theoretical foundations for attribute discovery, and conditions for identifiability, where we believe we have taken a significant step forward in the current work.

The natural next step would be now to try them out on real-world datasets. As has happened in case of object discovery, artificial datasets provide a natural testbest to try out various ideas in a relatively controlled setting, which can then be extended to more complex settings. We would like to refer to the literature on unsupervised object discovery in this regard. While earlier works, such as slot based methods, started with purely artificial domains, they were eventually improved to work on more complicated (real-world like) datasets [3].

Based on suggestions from other reviewers, and your inputs, we have also added some preliminary results on the naturalistic MOVi-C dataset in the subsection 5.5 of the updated paper, where we get meaningful images after swapping blocks discovered by our method (Figure 7 (a)). One of the challenges that we faced was though slot-based methods have been shown to work on real-data, they are still quite fragile, and a significant effort is required to make them work effectively in such settings. Accordingly, we see the following possibilities going into the future.

  1. Make slot based methods more robust with informed priors to further improve object discovery which is a precursor to our set-up. While current works do present results on real-data, they still suffer from problems such as over-segmentation[5].

  2. For harder settings, one could possibly think of taking additional cues in the form of textual inputs (e.g., [4]) or say segmentation masks from SAM to guide slot discovery. Such ideas can then be extended to attributes as well. For example, some lightweight information about what different kinds of attributes might be present in the data might help with attribute discovery.

  3. As suggested by the reviewer, explicitly modelling complexities in the real-world data, such as hierarchies, would also be a very natural direction to explore. As in our current work, these ideas can first be tried in an artificial setup and then eventually tested on the real-world setup.

References

[1] Francesco Locatello et. al. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

[2] Tao Yang et. al. DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models

[3] Maximilian Seitzer et. al. Bridging the Gap to Real-World Object-Centric Learning

[4] Aniket et al. CTRL-O: Language-Controllable Object-Centric Visual Representation Learning

[5] Hongjia Liu et al. MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning

Comments on the paper

4 comments

Comment regarding novelty and our core contributions (Part 1/2)

We sincerely thank the reviewers for their insightful reviews. We have addressed the concerns raised by the reviewers, including those related to novelty. Additionally, we would like to add a common comment on the novelty of our work and our core contributions.

We would like to first note that our paper is not simply proposing a training and inference architecture for the task of object and attribute discovery. Rather, it encompasses multiple aspects of this task, which includes (1) developing the first formal theory centered on graphical models for a generative model for this task (2) LRH - showing that attributes lie in a linear projection space of discovered slots (3) proposing a 2-state architecture, where the direct comparison with Sysbinder comes in. Next we throw light on each of these aspects below, and our novel contribution in each regard.

But before we do that, we would like to say a few words about the core ideas behind Sysbinder, and its drawbacks, which we try to address (and go much beyond) in our current work: SysBinder is a purely neural architecture that explicitly avoids probabilistic modeling and is trained with a reconstruction loss only with no theoretical guarantees or results on naturalistic data. The attributes are discovered as a natural fallout of slot discovery by imposing a regularizer (in the form of concept memory) over chunks of discovered slots. Further, since sysbinder ties the object discovery to factor discovery, it is not clear how any pre-trained object discovery module could be used in conjunction with Sysbinder’s attribute discovery mechanism, since they are deeply intertwined with each other. In contrast, we describe our contribution and novelty as follows:

Contributions on the theoretical front

We develop a formal probabilistic graphical model of images, objects and their attributes. Here, attribute values are represented as being sampled from a mixture of Gaussian, which are linearly related to the slot representation. Further, we develop a formal connection between concept memory and our Gaussian mixture modelling assumption for attribute representation providing a formal theoretical grounding to the concept memory which is simply used as a regularizer in the case of Sysbinder. Our theory, which builds on Linear Representation Hypothesis (LRH) for images, results in a set of assumptions for guarantee identifiability. Our theoretical framework helps establish the first result of identifiability for unsupervised attribute discovery in a generic multi-object set-up. To the best of our knowledge, no existing work has even attempted to give such theoretical guarantees for the problem of attribute discovery in case of multi-object setting. This itself is a novel and important contribution in its own regard.

Contributions Regarding Linear Representation Hypothesis (LRH) for Slots

We perform extensive experimentation on a number of benchmark datasets to show the existence of LRH, i.e., the attributes can be represented as a linear projection of slots discovered by an existing slot discovery module. This is a very important result unknown hitherto, in the context of discovering attributes from slot representation. We have shown its existence not only on slots of artificial datasets in our experiments, but even on the slots of a naturalistic dataset MOVi-C. (Please refer to section 5.1)

Contributions at the Architectural Level

While some components of our model, such as Concept memory, may seem similar to SysBinder, they are fundamentally different in the way they are used in the overall architecture proposed by us. For example, while for SysBinder, the blocks are simply read-off as chunks of the discovered slot representation, which in turn represent linear combination of learnable concepts, in our case, these blocks are discovered by explicitly performing a competitive attention over discovered slots, and both are refined sequentially (in each inference step) in a joint 2-stage pipeline. This is similar to object discovery, where competitive attention over an image was the key to discovery of effective slot representation, we argue a similar competitive attention over slots, is central to good attribute discovery in our model, an aspect completely missing in SysBinder. Finally, a formal connection between concept memory, and Gaussian prior over attributes, is also an important contribution of our work, giving theoretical justification for the use of learnable concept memory, as also highlighted eariler.

Finally, our 2-stage architecture not only results in significant experimental gains on various datasets, it also allows for a flexible use of various Slot discovery methods, more like a plug-in-play approach (as also pointed out by Reviewer 4128). This fact has already been confirmed by our latest set of experiments (please refer to Appendix C.7 in the updated manuscript).

Comment regarding novelty and our core contributions (Part 2/2)

On Experimental Front

We not not only show significant experimental gains on various metrics, on existing benchmarks, our latest set of results, extend these to naturalistic dataset MOVi-C at least of preliminary findings that attribute discovery / swapping is indeed possible in more complex settings -a result that has not been shown any existing work, in a multi-object set-up, to the best our knowledge (and we appreciate reviewers’ suggestions in this regard, who requested for results/experiments on real data).

Further, in our additional set of experiments, performed again based on reviewer inputs, we show explicitly use of GMM prior in synthetic controlled settings, and how attribute discovery in these settings validate our theoretical findings.

Overall, we believe that our contribution is multi-faceted in this paper, and comparing them with Sysbinder only on the architectural front, even where we have significant differences, may not be fair to our contributions, and work.

Summary of rebuttal and manuscript updates

Finally, in addition to novelty concerns addressed as above, we have done an extensive set of experiments, and made further improvements to the paper, in the hope of answering all reviewer concerns, and we sincerely hope reviewers take our detailed rebuttal into account while making the final decision on our paper. We also hope that these additional details help answer questions related to the soundness of our approach, raised by reviewers (iuue & 4128). Additional baseline comparisons, as well as comparison using other metrics have also been added as suggested by reviewers, and clearly our model outperforms existing baselines in all these aspects, by a significant margin in most cases.

Additionally, we are more than happy to engage further, in case reviewers have any further questions, or concerns, related to our rebuttal, or our updated manuscript.

Updated paper, with additional details, experiments, results, and clarifications, has also been updated on the portal, with all the changes highlighted in blue.

We thank all the reviewers for their insightful comments and constructive feedback. We have carefully addressed each of the concerns and questions raised in our individual responses. As the discussion phase is nearing its end, we would greatly appreciate it if the reviewers could let us know whether our responses have adequately addressed their concerns. We would be happy to provide any further clarifications or additional results if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.