acceptodds
Under review as a conference paper at ICLR 2027

DisCoLoRA: Variational Disentanglement of Pretrained Representations

Abstract

Pretrained models learn representations that can be reused across different tasks, but these representations often mix distinct sources of variation. Given attribute labels, we seek to separate variation associated with those attributes from structure shared across groups. This separation can help us understand how attributes are encoded and vary them during generation while preserving shared structure. We introduce DisCoLoRA, a generative framework that learns complementary shared and attribute-associated representations through low-rank updates to a frozen pretrained encoder–decoder. Across synthetic, image, and gene-expression data, DisCoLoRA separates these sources of variation using pretrained backbones. We identify interpretable directions in the attribute-associated representation and trace their decoded effects to observed features. These results demonstrate how pretrained representations can be adapted to separate, interpret, and manipulate attribute-associated variation across modalities.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Disentangled representation learning, domain invariance, LoRA, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TsrqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a disentanglement method for images or other data that exploits labels to separate multiple factors in pretrained latent representations rather than raw data. The method is applied to artificial data, Celeb-A image data, and IFN-beta genetics data. The experiments show reasonably good performance compared to competing methods.

Critical strengths

Although disentanglement has been extensively studies in late 2010's, the present method works on a pretrained latent representation rather than raw images/data. The motivation is good as learning such representation from scratch is expensive. Their experimental results show reasonably good performance.

Critical weaknesses

Apart from the fact that the model works on a frozen latent representation, I could not find prominent novelty in the method. All the techniques given in the paper seem to be rather common among existing disentanglement models.

Empirical comparison with prior methods seems to be made only for the artifical data.

Questions for authors

What is the crucial difference from previous disentanglement methods, apart from the fact that the observations are frozen latent representations? Is it mostly reuse of existing techniques?

It would be nice to clarify the relationship with other related disentaglement models, e.g., Bouchacourt et al. (2018), Lample et al. (2017), Mathieu et al. (2016), Locatello et al. (2020), Hosoya (2019, 2022), Siddharth et al. (2017), Yang et al. (2015), etc.

Reviewer_WnABOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a variational disentanglement learning method for pretrained representations. The whole framework can learn complementary shared and attribute-associated representations through low-rank updates to the pretrained encoder-decoder. The method can also identify interpretable directions in the attribute-associated representation and trace their decoded effects to observed features.

Critical strengths

(1) The motivation is well presented of the variational disentanglement learning method.

(2) The illustrations is mostly clear and intuitive of LoRA, variational disentanglement learning, and entropy regularization.

Critical weaknesses

(1) The contribution is limited since the whole framework is a straight-forward combination of a lot of methods from the literature but without considering the task at hand. For example, LoRA, variational disentanglement learning, entropy regularization, etc.

(2) On Page 3, Line 129-135, both \hat{x} and \tilde{x} are supervised by x. Since \hat{x} reconstructs from z while \tilde{x} reconstructs from z and w, the first reconstruction may fail to get shared representation where no GT, i.e., shared component of x, is provided.

(3) On Page 4, Line 191-192, a classifier trained only on \hat{x} should have difficulty in distinguishing between y labels. This conclusion has no evidence to support.

(4) On Page 5, Line 239-240, the shared encoder - mariginal decoder path is trained to make these classifiers uncertain about y. There is no explanation on how max uncertainty effects the result and there is no ablation.

(5) On Page 5, Line 254, the whole section of why a predictive entropy regularizer? Most of the content on the entropy regularization borrows from (Ganin & Lempitsky, 2015), there is no novel discussion on the reasonability behind Equations 5 & 6.

(6) On Page 3, Line 126, the authors claim, although the generative model assumes independence between z and y, the inferred representations need not satisfy it. The assumption conflicts the opinion “without a theoretical guarantee of independence from y” on Page 6, Line 281.

(7) The experiments cover a toy dataset, a small class of CelebA and a subtask of PBMCs which are insufficient.

Questions for authors

No.

Reviewer_uoBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

DISCOLORA improves upon DISCOVR by introducing additional regularization terms that encourage latent space disentanglement. Both methods concern a form of weakly supervised learning, where attribute labels are needed for representation learning.

Critical strengths
  • Nice monty python motivation
  • Very easy to follow
Critical weaknesses
  1. The hard constraint on this paper is novelty. (a) LoRA essentially adds an adapter to existing method and does not offer substantial structural change on the model level compared to DISCOVR [1]. My reading of Figure 1(B) is almost identical to Figure 2 of [1] other than additional adapter parameters (filled black circles ϕ0\phi_0 and ψ0\psi_0). (b) Entropy regularization in Eq (5,6) barely counts as a something new.
  2. Synthetic datasets like Swiss roll do not add much weight nowadays, even though it is used in [1].
  3. CelebA is used to test the method's generative capabilities, but there are no qualitative baselines comparisons in Figure 3(C) nor substantive metrics like FID and CLIP score as shown in Table S2. The paper could be stronger with datasets like CUB and PolyMNIST.
  4. To evaluate the compactness and informativeness of representations, linear probing is a common method [2,3], but it is not clear why it is not done in this work on zz or ww.
  5. If using LoRA for representation disentanglement is really a selling point, this work would be much stronger if its universality and simplicity is shown on various representation disentanglement methods, rather than only DISCOVR.

[1] Variational Learning of Disentangled Representations. Yuli Slavutsky, Ozgur Beker, David Blei, Bianca Dumitrascu. ICML, 2026. [2] An Information Criterion for Controlled Disentanglement of Multimodal Data. Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, Caroline Uhler. ICLR, 2025. [3] Disentanglement of Variations with Multimodal Generative Modeling. Yijie Zhang, Yiyang Shen, Weiran Wang. ICLR, 2026.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.