acceptodds
Under review as a conference paper at ICLR 2027

From Core to Detail: Unsupervised Disentanglement with Entropy-Ordered Flows

Abstract

The unsupervised discovery of features that are both semantically meaningful and stable across runs remains a central challenge in representation learning. We introduce entropy-ordered flows (EOFlows), a normalizing flow (NF) framework that augments standard maximum likelihood training with an orthogonality regularizer on the decoder Jacobian. The regularizer is rooted in Independent Mechanism Analysis and encourages geometric disentanglement, and a stochastic estimator makes it tractable at image scale (CelebA at and ). Learned features form near-orthogonal curvilinear coordinates and can be ordered by their after training, analogous to the ranking by explained variance in PCA, which turns EOFlows into a non-linear generalization of PCA. EOFlows identify an order of magnitude more stable features than existing methods, and these features emerge in distinguishable categories (global, local, and generic) and support tentative semantic interpretations. The local features have strikingly sparse support in pixel space, although our method never enforces this. Retaining only the most important, i.e. highest entropy, features turns the bijective flow into an autoencoder with adjustable bottleneck, rivaling the rate-distortion performance of dedicated autoencoders.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unsupervised disentanglement, representation learning, normalizing flows, independent mechanism analysis, identifiability, nonlinear PCA, manifold learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HyRWOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces entropy-ordered flows (EOFlows), which augment normalizing-flow training with a regularizer encouraging orthogonality of the decoder Jacobian. The learned coordinates are ranked by manifold entropy after training, enabling an adjustable inference-time bottleneck. The experiments investigate feature reproducibility, sparsity, disentanglement, and reconstruction.

Critical strengths
  • Clear presentation and motivation: The paper is very well written, with clean formalization and a convincing explanation of the proposed approach.

  • Strong experimental study: The experiments are thoughtfully designed and provide compelling evidence for the method’s benefits.

  • Faithful baseline comparisons: I appreciate that the reported baseline results are consistent with independent findings, which lends credibility to the empirical study.

Critical weaknesses
  • The nonlinear PCA framing needs qualification: Reconstruction-based nonlinear autoencoders are also commonly interpreted as generalizations of PCA. The abstract should more precisely distinguish what EOFlows generalize, including the role of entropy ordering versus the identification of principal directions.

  • The evidence for statistical independence could be stronger: Decoder-Jacobian orthogonality does not directly measure statistical independence between latent features. Since orthogonality is also an explicit training objective of EOFlows, superiority on this metric alone does not establish superiority on the independence objective pursued by methods such as β-TCVAE. A complementary latent-dependence evaluation like the LPS score from Fruytier et al.'s "Sculpting Latent Spaces" would strengthen these claims.

  • Broader baseline evaluation would be useful: Despite the strong experimental study, I would appreciate more evaluation of the baselines across the different experimental settings.

Questions for authors
  • 1. Polar coordinates and remaining ambiguities: The introduction states that rotating the latent coordinate system destroys orthogonality of the induced coordinates. However, polar coordinate systems admit different choices of angular reference and origin while remaining orthogonal. Could you clarify how these transformations differ from the latent rotations discussed in the paper, and which ambiguities orthogonality resolves or leaves intact?

  • 2. Connection to weakly supervised disentanglement: Varying one coordinate while holding the others fixed, as in Equation (3), seems related in spirit to the factor-wise variation considered in Locatello et al.’s "Weakly-Supervised Disentanglement Without Compromises". Could you briefly discuss this connection and the differences in assumptions?

  • 3. Soft orthogonality: Does “soft orthogonality” simply mean encouraging orthogonality through a finite-weight regularizer rather than enforcing it as an exact constraint?

  • 4. Entropy ordering: My understanding is that coordinates are rearranged post hoc according to their estimated manifold entropies, rather than occupying entropy-ordered positions during training. Is this correct? How does this ranking compare with approaches such as Matryoshka representations (Kusupati et al.'s "Matryoshka Representation Learning"), which encourage an ordering of information through the architecture and nested training objectives?

  • 5. Statistical independence: Does greater Jacobian orthogonality also produce more statistically independent features on encoded data? Could you evaluate this using a complementary measure such as Fruytier et al.’s Latent Predictability Score (LPS)?

Reviewer_a4U2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a method based on normalizing flows that learns disentangled representations by encouraging the columns of the decoder Jacobian to be orthogonal. The work builds on the theory of independent mechanism analysis (IMA) and prior empirical methods that encourage disentanglement in either normalizing flows or VAEs using IMA regularization. Their main contribution is to scale the method to higher dimensional naturalistic datasets (CelebA) by using a tractable stochastic estimate of IMA contrast/Hadamard regularizer term that encourages orthogonality. The trained model orders its latent coordinates by manifold feature entropy, and the authors demonstrate that this allows the model to be used as an autoencoder with a bottleneck that can be adjusted at inference time. They compare against existing disentanglement methods (a standard normalizing flow and variants of beta-VAE) and show using the decoder Jacobian that their representation extracts more disentangled features and exhibits emergent sparsity in pixel space.

Critical strengths

The paper is generally well written, though there are some terms and claims that are, in my opinion, slightly misleading (see weaknesses). The authors provide the reader with useful intuition over the effects of encouraging decoder orthogonality (Fig 2). Making the Hadamard regularizer tractable using stochastic estimators allows the the desired orthogonality constraint to scale independently of the input dimensionality and batch size, demonstrating the utility of this method for learning from high dimensional and complex image datasets. The empirical results also validate several mathematical claims; for example, the matched UiU_{i} stability in Section 4 support the intuition that independently trained models discover the same latent features, as such a method should if it is a property of the data rather than the model. It is also interesting to see that pixel sparsity emerges automatically from some of the learned latent coordinates.

Critical weaknesses

My main concern regards the authors' claims of disentanglement. One way the authors evaluate disentanglement performance is by measuring the degree of orthogonality between the decoder Jacobian columns, but as a property that is explicitly encouraged by the training objective, I do not think it can be used to demonstrate a desired property (of disentanglement).

More fundamentally, it is unclear whether the learned coordinates necessarily correspond to semantic features in the data in a bijective manner. The authors mention that the discovered variables have "global assignment", but the semantic meaning of one latent coordinate is not guaranteed to be preserved across different images. Though there is some globally shared coordinate system, this does not necessarily correspond to "disentanglement" in the typical sense, where one latent coordinate represents a single semantic transformation. Due to this, calling UiU_i a manifold over "feature i" seems to be a misnomer, when it may not consistently capture a single semantic feature.

This in turn casts doubt on whether it is automatically desirable to learn a representation with more active dimensions. For example, many latent coordinates may work in conjunction to represent a single semantic factor of variation in the data, which raises the number of active modes without necessarily capturing more semantic changes. This may explain why on other disentanglement metrics (e.g. MIG and DCI) besides the Jacobian orthogonality score, EOFlows performs comparably to VAEs. According to these metrics, the latents of EOFlows do not appear to capture more ground truth factors than existing methods. Finally, the state-of-the-art disentanglement scores are now achieved by diffusion-based representation learning methods, which exhibit much higher TAD, MIG, and DCI scores than VAE variants.

Another concern is the fairness of the comparison against VAE baselines. The authors state that EOFlows concentrate their entropy in a few hundred dimensions, while FactorVAE uses at most 20 active latents (lines 336–338). This does not strike me as a fair comparison. VAEs can only learn up to their allowable number of latent dimensions, and a fairer comparison would be to compare against VAE variants with up to a DD-dimensional latent space. Moreover, I would expect the number of active modes in β\beta-VAEs to depend heavily on the β\beta parameter (and/or γ\gamma, for FactorVAEs). A lower β\beta will likely recruit more active modes, but the optimal β\beta that recruits the largest number of modes is an empirical question. Although the authors report some hyperparameter tuning, I would trust the empirical results more if they performed a more extensive sweep on β\beta or γ\gamma, including smaller regularization strengths, and compared against the optimal model.

Questions for authors
  1. What condition must the network or data satisfy for the geometric coordinate system to reliably correspond to the ground truth features in the data? Is this something you could measure?
  2. How well does the Hadamard gap (in Fig 6b) correlate with other disentanglement metrics such as DCI, MIG and TAD? If there is a strong correlation it would be a sign that encouraging decoder Jacobian orthogonality would be a good proxy for promoting disentanglement.
  3. The authors establish that EOFlows recover PCA in the linear setting and provide some evidence of nonlinear dimensionality reduction on dSprites and Shapes3D. But in Appx G3 the authors find an entropy spectrum knee after 7-8 coordinates rather than the 6 ground truth factors. I would be interested in a controlled synthetic experiment that directly evaluates the extent to which EOFlows generalize PCA to nonlinear manifolds. For example, suppose we have some data with intrinsic dimensionality of 2 that are nonlinearly embedded in R3\mathbb{R}^3, and then linearly embedded into an NN-dimensional ambient space. Such a dataset would have an intrinsic dimensionality of 2 but require 3 principal components for exact linear reconstruction in the noiseless setting. Would the entropy spectrum learned by EOFlows identify C≈2C \approx 2, or would it instead reflect the higher-dimensional linear embedding? Comparing reconstruction quality against PCA as a function of CC, across noise levels and degrees of nonlinearity, would help clarify whether EOFlows recovers the intrinsic dimensionality of the underlying manifold and provide stronger empirical support for its interpretation as a nonlinear generalization of PCA.
Reviewer_N2qSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method to learning disentangled representations in a stable manner (each run should consistently give you the same disentanglement) using inductive biases of the method rather than auxiliary information about the data. In particular it proposes to add a regularizer to the normalizing flow objective. This regularizer is taken from the Independent Mechanism Analysis which proposes to force the decoder's Jacobian's columns to be orthogonal. This prevents the training objective to be invariant to rotations in the latent space which is thought to help with stability. PCF from Cunningham et al. 2022 already propose this approach but only provided a proof of concept on synthetic experiments.

Scaling up PCF requires a stochastic estimate of the Hadamard regularizer and noise addition to prevent overfitting and positive results are shown on CelebA (28x28 resolution with a proof of concept on the 64x64 resolution) and mixed results on synthetic datasets like dSprites and Shapes3D. Authors also propose a way of ordering the entropy associated with each latent which allow the selection of meaningful subsets of latents.

Critical strengths
  • Presentation is good: I think the presentation passes the acceptance bar for this venue. Enough effort was put in the presentation of results, motivation behind the method and explanation of the method and setup explored. I did not experience any major blockers when reading the paper.
  • Paper is honest about its scope and contributions: I think the paper does not overstate its contributions and is honest about prior work and limitations of the approach. It refers to prior work which is instrumental to the setup they explore (e.g., PCF and IMA for the training objective, TarFlow for the noise injection) and it states clearly the limitations of the approach (e.g., applicable to certain datasets like CelebA).
  • Results show improvements: While the empirical scope is limited (see weaknesses below), I think given the limitations of the method, the empirical setup is relatively thorough (exception discussed in question section) and do show improvements over prior work (i.e, it does better then normalizing flows and the VAE family).
Critical weaknesses
  • Incremental contribution: authors proposes to take PCF and making it more scalable. They propose to add noise to prevent overfitting (inspired from prior work like TarFlow) and formalize a way to order latents based on entropy carried by each of them which allows effective compression. They extend the experiments to CelebA dataset mostly and show there that it indeed improves prior work. EOFlows is closer to a non-linear PCA than its competitors (NF, VAE family): it is more stable than other methods, entropy spectrum approaches that of PCA with top latents capturing more entropy, it does not require a choice of C (number of latent dimensions) a priori. Now the major flaw is that, similar to many other methods in this field, the latents align with interpretable features only for certain datasets (e.g., CelebA, used by the authors) which means its appeal is limited when moving on to more complex datasets.
Questions for authors
  • If we assume the decoder's columns are orthogonal, could the ordering of latents be done by measuring the output variance (on the images) as we change a single latent at the time? how would the ranking of latents change with this approach?
  • Given that the method already limits itself to the CelebA dataset (+ mixed results on synthetic data), I would have like to see a more thorough exploration of larger resolutions for the CelebA dataset (more than the 64x64 proof-of-concept) to show that at least the method scales well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.