DisCoLoRA: Variational Disentanglement of Pretrained Representations

Pretrained models learn representations that can be reused across different tasks, but these representations often mix distinct sources of variation. Given attribute labels, we seek to separate variation associated with those attributes from structure shared across groups. This separation can help us understand how attributes are encoded and vary them during generation while preserving shared structure. We introduce DisCoLoRA, a generative framework that learns complementary shared and attribute-associated representations through low-rank updates to a frozen pretrained encoder–decoder. Across synthetic, image, and gene-expression data, DisCoLoRA separates these sources of variation using pretrained backbones. We identify interpretable directions in the attribute-associated representation and trace their decoded effects to observed features. These results demonstrate how pretrained representations can be adapted to separate, interpret, and manipulate attribute-associated variation across modalities.