acceptodds
Withdrawn from ICLR 2027

ConDe: Decomposing Representations into Manipulable Semantic Components

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Disentangling semantic factors within learned representations is critical for targeted analysis, manipulation, and improved downstream performance. However, a general disentanglement formulation applicable post-hoc to frozen representations, for multiple semantic factors, and diverse real-world data domains remains underexplored. In this work, we introduce the Contrastive Decomposer (ConDe), a weakly supervised framework designed for this general-case setting. ConDe decomposes an embedding into an arbitrary number of components whose element-wise sum reconstructs the representation. Because all components inhabit the same latent space, they can be inspected independently and directly recombined allowing for direct latent manipulation of semantic factors. ConDe supports strict and soft pairings and can be trained end to end or applied post hoc to frozen encoders by leveraging invertible networks. We further propose an evaluation protocol for supervised vector-wise disentanglement that assesses both the learned components and their downstream effects, and introduce SynBeeps, a controlled audio benchmark for testing partial disentanglement under complex, perturbed factor relationships. Experiments across vision, audio, and geological imaging show that ConDe learns informative semantic components that support direct manipulation and improved generalization under domain shifts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
disentanglement, post-hoc disentanglement, interpretability, latent manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WwBGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a disentanglement method for images that separates multiple factors in pretrained latent representations rather than raw images. The method is applied to image datasets of MNIST, 3D objects, and faces. The experiments show that the performance is comparable to competing methods.

Critical strengths

Disentanglement is now already classical, where numerous studies were conducted in late 2010's. While previous methods focus on learning decomposed representations from raw images, the present method does this on a pretrained latent representation, which can be advantagenous in practical settings. The experimental results show reasonable performance.

Critical weaknesses

My main concern is that I do not see the crucial difference in the methodology from the existing methods operating on raw images. If the representation already has sufficient information on the raw images, I would expect that a similar method can be reused. Indeed, the specific methods proposed in the paper look quite similar to existing methods.

Questions for authors

What is the crucial difference from previous disentanglement methods apart from targeting learned representations rather than raw images?

What is the relationship with other related studies, e.g., Lample et al. (2017), Mathieu et al. (2016), Hosoya (2019, 2022), Siddharth et al. (2017), Yang et al. (2015), etc.

Reviewer_oTZiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ConDe (Contrastive Decomposer), a weakly supervised framework for disentangling learned representations into factor-specific components and a residual. The method uses contrastive and reconstruction-based losses. It can be applied to pretrained representations using invertible neural networks or trained end-to-end. The authors also introduce SynBeeps, a synthetic audio benchmark, and an evaluation protocol for vector-wise disentanglement. The method is evaluated on synthetic and real-world datasets, including image style transfer, music timbre transfer, and seismic interpretation.

Critical strengths
  • The paper addresses an interesting problem of decomposing pretrained representations into interpretable, factor-specific components. The ability to perform partial decomposition while retaining residual information is relevant to representation learning and disentanglement.

  • The proposed framework combines invertible transformations with additive factor decomposition and relational supervision. Supporting both post-hoc decomposition of frozen representations and end-to-end training provides flexibility for different applications.

  • The method supports multiple factors and both strict and soft pairing, allowing decomposition under different assumptions about the relationships between samples.

  • The paper introduces SynBeeps and an evaluation protocol for vector-wise partial disentanglement. The experiments cover several synthetic and real-world applications, including seismic interpretation.

Critical weaknesses
  1. Independence of the residual representation The residual loss minimises HSIC between the residual and each semantic component separately. However, pairwise independence does not necessarily imply independence between the residual and the joint semantic representation. Therefore, higher-order dependencies may remain in the residual, potentially limiting the completeness of the disentanglement.

  2. Interaction and necessity of the loss functions The proposed method combines four loss functions, but their interaction and necessity are not entirely clear. Although the authors provide an ablation study, the performance differences are relatively small in some cases, raising questions about the contribution of each objective. In particular, the decomposition and coverage losses reconstruct different targets from the same semantic components, which may introduce competing objectives when the semantic representation differs from the full embedding.

  3. Factor-specific completeness under soft pairing Under soft pairing, the decomposition and coverage losses use the anchor components rather than positive components. Consequently, these reconstruction objectives no longer directly enforce cross-sample consistency. It remains unclear whether they encourage factor-specific completeness rather than simply preserving information in the semantic representation.

  4. Evaluation of disentanglement The main evaluation relies on Informativeness, Exclusivity, and Invariance. However, Informativeness and Exclusivity scores are consistently high across different methods, with relatively small differences in several cases. This raises questions about how informative these metrics are for distinguishing different levels of disentanglement. In addition, the specific formulations of Exclusivity and Invariance are not standard metrics, and their ability to capture information leakage and factor separation is not entirely clear.

  5. Limited validation in the real-world seismic application The paper includes seismic segmentation as a real-world application. However, the corresponding table does not provide a comparison with other methods, making it difficult to assess the benefit of ConDe in this setting. In addition, the segmentation masks presented in the supplementary material do not appear sufficiently convincing to establish the effectiveness of the method.

Minor weakness The discussion of FDEN appears somewhat simplified, as FDEN also uses a total correlation loss to encourage independence between factors. The authors could clarify whether the stated limitations of FDEN still apply when this loss is considered.

Questions for authors
  1. Evaluation metrics: Informativeness and Exclusivity scores are generally high across different methods, with relatively small differences in several cases. Could the authors clarify how well these metrics distinguish different levels of disentanglement? How sensitive is Exclusivity to PCA dimensionality and the probing model, and how should Invariance be interpreted under soft pairing, where positive samples may have different factor values?

  2. Necessity and interaction of the loss functions: Although the ablation study evaluates individual loss functions, removing some losses results in relatively small performance changes. Could the authors clarify the contribution of each loss and whether all four are necessary? How do the decomposition and coverage losses interact when the semantic representation differs from the full embedding, and how sensitive is the method to their relative weights, which are set to 1 by default?

  3. Residual independence: The residual loss minimises HSIC with each semantic component separately. However, this does not necessarily ensure independence between the residual and all semantic components jointly. Could the authors clarify whether such dependencies affect the learned decomposition?

  4. Soft pairing: When reconstruction uses anchor components alone, what prevents each semantic component from retaining information unrelated to its designated factor? How is factor-specific completeness encouraged under soft pairing?

  5. Real-world seismic application: Could the authors clarify the effectiveness of ConDe in the seismic segmentation experiment? A comparison with other methods would help establish its practical benefit, as the segmentation results in the supplementary material do not appear sufficiently convincing on their own.

Reviewer_FrYgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ConDe, a weakly supervised framework that decomposes an embedding z into k vector-valued semantic components and a gated residual, all living in the same space, so that z ≈ r + Σ s_i. Training combines a per-factor triplet loss with two cross-reconstruction terms (decomposition and coverage) and an HSIC penalty between the residual and the components; a softplus variant of the triplet loss handles soft pairing. For frozen encoders, an affine-coupling INN first reparameterizes the embedding. The authors also propose an evaluation protocol with representation-level and decoder-level metrics, and a synthetic audio benchmark (SynBeeps). Experiments cover 3D Shapes, SynBeeps, and Colored MNIST against FDEN and IIN, as well as face style transfer, music timbre transfer, DomainBed, and seismic fault segmentation.

Critical strengths

The problem setting is relevant and well motivated: post-hoc, partial, vector-wise decomposition of frozen representations for several factors at once is a practical scenario that existing methods (FDEN, IIN) only partially address. The additive formulation in a shared ambient space is a simple and appealing design, because components can be inspected, removed, or exchanged directly without committing each factor to a fixed dimensionality. Support for both strict and soft pairing is useful, and the Colored MNIST and CUFS examples illustrate the difference between recovering a label and preserving instance-specific information in an intuitive way. The evaluation protocol is a genuine contribution in its own right. In particular, the paper shows that representation-level probes and decoder-level swaps can disagree sharply (Appendix D: informativeness and exclusivity stay near 0.96–0.99 while negative swap is 0.00), which is a useful observation for the community. The breadth of modalities (vision, audio, seismic) is notable, anonymized code is provided, and the authors are candid where ConDe does not reach the state of the art, as on DomainBed.

Critical weaknesses

W1. Direct latent manipulation is presented as the main benefit of the additive design, yet the decoder-level results are mixed. On SynBeeps, ConDe's negative swap is 0.21 versus 0.78 for FDEN (Table 1). In the four-layer INN ablation, the full model reaches only 0.04, and removing the decomposition loss raises it to 0.36 (Table 6), so the complete objective performs worse on this metric than one of its ablations. Reaching 0.30 requires a 48-layer INN (Table 5). The paper attributes the FDEN advantage to its classification objective, but could the authors provide direct evidence for this explanation? More broadly, could they discuss under what conditions the additive recombination is expected to remain compatible with the pretrained decoder, given that this appears to be the limiting factor?

W2. Appendix D notes that the reported metrics "do not directly measure residual factor content," and it explains several failed swaps by possible residual leakage. Since the residual is central to the method, a direct measurement seems necessary. Could the authors report probes for each factor on r, together with the fraction of the embedding's energy assigned to r versus the components? On SynBeeps, this would also show whether the unmodelled perturbations (equalization, background noise) actually end up in the residual, which is the stated purpose of the benchmark. A related question concerns the coverage loss, which asks the positive components alone to reconstruct the full anchor embedding z^a, including information the residual is meant to hold. How do the authors reconcile this objective with the role of the residual?

W3. Because all components share the same space and are combined by addition, any vector can be added to one component and subtracted from another without changing the reconstruction. The triplet and HSIC terms constrain this freedom, but the paper does not discuss what decomposition the objective identifies, or under which pairing assumptions. Existing work gives identifiability results for paired or grouped observations, for instance content–style identification from augmented pairs [1], weakly supervised disentanglement with guarantees [2], and contrastive identifiability [3]. Positioning ConDe against these results, even informally, would clarify what the method guarantees beyond empirical behavior.

[1] Von Kügelgen, Julius, et al. "Self-supervised learning with data augmentations provably isolates content from style." Advances in neural information processing systems 34 (2021): 16451-16467. [2] Shu, Rui, et al. "Weakly supervised disentanglement with guarantees." arXiv preprint arXiv:1910.09772 (2019). [3] Zimmermann, Roland S., et al. "Contrastive learning inverts the data generating process." International conference on machine learning. PMLR, 2021.

W4. The confidence intervals in Tables 1, 5, and 6 appear to come from evaluation folds and bootstrap resamples of a single trained model. Are the results based on one training run per method? If so, could the authors report the mean and spread over several training seeds? This matters especially for the ablations, where many differences are in the 0.01–0.04 range, and for Tables 2 and 4, which report no uncertainty at all.

W5. The disentanglement comparison includes only FDEN and IIN. The comparison excludes several weakly supervised, vector-wise methods discussed in the related work, such as ML-VAE, Ada-GVAE, CSVAE, and product-manifold projection, even though some require adaptation to the post-hoc setting. IIN is reported to show "unstable optimization," and its Group MIG is 0.00 on Colored MNIST. Could the authors describe how they tuned IIN so readers can judge whether its results reflect the method or the implementation?

W6. Some metrics seem to reward degenerate solutions when read in isolation. For example, IIN obtains an exclusivity of 0.99 on Colored MNIST while its informativeness is 0.54 and its Group MIG is 0.00, presumably because components carrying little information also leak little. Similarly, invariance would be trivially high for a collapsed component. Would the authors consider a combined score or a joint analysis, for instance, exclusivity conditioned on informativeness? In addition, exclusivity aggregates non-target components by addition for ConDe and by concatenation for the baselines. Could the authors comment on whether the PCA-prefix leakage measure is comparable across these two aggregations, given their different dimensionalities?

W7. According to Appendix F, the target domain and ConDe's positive structure pairs are built with the same augmentation pipeline, whereas the source-only baseline is trained on the original images without these augmentations. ConDe therefore sees the target-domain transformations during training, and the baseline does not. Could the authors add a baseline trained with the same augmentations (for example, standard augmentation or an augmentation-consistency objective without decomposition)? Without it, it is difficult to attribute the improvement in Table 4 to the decomposition rather than to the augmentation itself. The same reasoning would apply to the training curves in Figure 6.

**W8. ** SynBeeps is listed as a contribution, but the paper provides only a short paragraph about it, and the appendix has no dedicated section. Could the authors provide the generative specification (ranges and distributions of the six factors, how the factors are defined when a signal contains several beeps, the form and strength of the unmodelled perturbations, and the splits), together with a statement about its release?

Questions for authors

See the above Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.