ConDe: Decomposing Representations into Manipulable Semantic Components

Disentangling semantic factors within learned representations is critical for targeted analysis, manipulation, and improved downstream performance. However, a general disentanglement formulation applicable post-hoc to frozen representations, for multiple semantic factors, and diverse real-world data domains remains underexplored. In this work, we introduce the Contrastive Decomposer (ConDe), a weakly supervised framework designed for this general-case setting. ConDe decomposes an embedding into an arbitrary number of components whose element-wise sum reconstructs the representation. Because all components inhabit the same latent space, they can be inspected independently and directly recombined allowing for direct latent manipulation of semantic factors. ConDe supports strict and soft pairings and can be trained end to end or applied post hoc to frozen encoders by leveraging invertible networks. We further propose an evaluation protocol for supervised vector-wise disentanglement that assesses both the learned components and their downstream effects, and introduce SynBeeps, a controlled audio benchmark for testing partial disentanglement under complex, perturbed factor relationships. Experiments across vision, audio, and geological imaging show that ConDe learns informative semantic components that support direct manipulation and improved generalization under domain shifts.