acceptodds
Under review as a conference paper at ICLR 2027

What Is Identified in Causal Representation Learning? Exact Answers for the Normalized iVAE from Realized Symmetries

Abstract

Causal representation learning shows that a latent representation can be recovered only up to a symmetry. This leaves a question unanswered: which claims about hidden causes are fixed by observed data? We introduce a framework that answers it by characterizing functions of latent variables that are determined by observations and unchanged by symmetries. We first show why normalization is essential: a class that admits monotone changes of every latent coordinate identifies only constant functions. For the normalized iVAE with scalar statistics and a fixed latent parameterization, we build changes of variables that realize every range-preserving permutation and sign change. This turns the classical iVAE inclusion into an exact statement within each set of models producing the same observations: the bounded functions unaffected by those permutations and sign changes are identified. We show how these statements fit together across model classes. A rigid class preserves content, while a class allowing monotone changes in every coordinate reduces it to constants. Finally, we show how contrastive content and style separation fits the same interface and sketch how CITRIS fits the picture. Experiments with fitted iVAEs show that the symmetry is real in practice, that identified functions are the stable ones across retraining, and that a stronger normalization restores individual coordinates.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
causal representation learning, exact identifiability, iVAE, nonlinear independent component analysis, latent symmetries, invariant functions, model class closure

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FjicOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Common ICA methods have identifiability results of the generative factors up to some transformations such as permutation or element-wise diffeormorphism, the paper aims to identity the generative factors up to different transformations with a normalized iVAE [1], additionnaly they want these transformations to be effectively achieved by existing models. The prove the necessity of normalization (otherwise the identified functions collapse) and they prove that under specific assumptions some equalities concerning identified functions. Finally they empirically study their claims on synthetic dataset.

Critical strengths

None

Critical weaknesses
  • The main issue with this paper is that it is really hard to read: concepts are introduces without any example of what they could represent, some notations are not explained or formally defined, some notations are not used which add unnecessary complexity, some notations are reused for different objects, some different notations represent exactly the same objects, ...
  • Another issue is that the paper do not ensure that a representation learned by a iVAE respect the very strict assumption require for the theory (see Question 1), additionally the experiments are limited and did not convince me that the method works (see Question 2)
Questions for authors
  1. My main concern lies here: assume having learned an instance JJ and one of its sufficient statistic TJT^J with a normalized iVAE, what guarantees that JJ belongs to the chart-pinned class CIC_I of the ground-truth instance II ? Particularly, what guarantees that there exists gJ∈GYg_J\in G_Y such that $ T^J=g_J\circ T^I$ which is necessary for Theorem 5 and 10 ? In the original paper introducing iVAE [1], Theorem 4 guarantees that a representation that maximizes the ELBO belong to their studied equivalence classe, is there any analogous result here ? If I understood well we have ∣CI∣=∣GY∣≤2mm!|C_I|=|G_Y|\le 2^mm! while the set of instances maximizing the ELBO seems infinite (even restricting to those satisfying A1-A6), it feels odd to imagine that a learned representation will naturally fall into CIC_I
  2. Experiments in Section 7 are very light as there are no interval confidence or correlation between h(z)h(z) and h(z^)h(\hat z) to verify if the functions are correctly approximated. It seems that more interesting results are in Appendix I, however it is really hard to read as the results are really poorly presented, could you present those results in complete sentences ? It could be interesting to know if those properties are still true for other non linear ICA methods, why is there no ablation ? The experimental setting assume to know the family of the law p(z∣s)p(z\mid s), would it still work in a more general parameterization of exponential family ?
  3. Section 3 aim to explain the necessity of normalization by using G⊇∏k Homeo +(Ik)G \supseteq \prod_k \text { Homeo }_{+}\left(I_k\right) while in the rest of the paper G⊆BmG\subseteq B_m is used. Additionnaly Theorem 3 uses the total realization of the symmetries as only assumption while the rest of the paper requires several assumption except normalization (as stated section 4.2). Could you explain how this section is related to the necessity of normalization with G⊆BmG\subseteq B_m under A1-A5 ?
  4. I don't understand why VBV^B requires to have ∫φdPZI=∫φdPZI′\int\varphi dP^I_Z=\int\varphi dP^{I'}_Z, this property seems to be implied by ∀ψ∈Aut⁡(Z), φ∘ψ=φ\forall\psi\in\operatorname{Aut}(\mathcal Z),\ \varphi\circ\psi=\varphi and by the assumption of witnesses ψ\psi between two equivalent instances. Therefore why not define VB(Π,M)V^B(\Pi, \mathcal{M}) directly as \textbraceleftφ∈L∞(Z):φ∘ψ=φ a.s. for all ψ∈Aut⁡(Z)\textbraceright\textbraceleft\varphi \in L^\infty(\mathcal Z): \varphi \circ \psi=\varphi \text { a.s. for all } \psi \in \operatorname{Aut}(\mathcal{Z})\textbraceright ? In such case VB(Π,M)⊆VA(Π,M)V^B(\Pi,\mathcal M)\subseteq V^A(\Pi,\mathcal M) would be a direct property from the assumptions and the results of the paper would be the same ?

The following issues concern unclear aspect of the text:

  1. In the definition of an instance, what represent κ\kappa and SS in real life dataset ?
    In Theorem 8, κ\kappa become skewness \textbraceleftE[Tj3]\textbracerightj\textbraceleft\mathbb E[T^3_j]\textbraceright_j do you assume that skewness is an observational variable or is it another concept with same notation ?
  2. I don't understand what represents Π\Pi, PX×SP_{X\times S} seems totally defined from an instance, therefore why introduce Π\Pi ?
  3. In the assumptions what is the difference between TIT^I and TT ? More generally, when the chart are assumed to be the sufficient statistics and when it is two separated things ?
  4. You use the same notation CC both for equivalence relation [I]∼∩M[I]_{\sim} \cap \mathcal{M} and for the chart-pinned class, this is highly confusing. For instance in Theorem 8, is CC a chart-pinned class or an equivalence class ?
  5. In Theorem 8: as several charts are admissible for the same instance, some satisfying the rule and some not, I don't understand how Cskew⁡C_{\operatorname{skew}} is defined, could you give a formal definition ?
  6. In the proof of Theorem 9: you use μj\mu_j for skewnesses while in the main text skewnesses are represented by κk\kappa_k and μj\mu_j represents means, why ?
  7. In the proof of Theorem 9: what are the YirY^r_i ? Are they related to to the chart image Y=∏iYiY=\prod_i Y_i ?
  8. In Theorem 9: what is a representation error ? Could you give a definition ?

[1] Variational Autoencoders and Nonlinear ICA: A Unifying Framework (Khemakhem et al, AISTATS 2020)

Reviewer_qFFZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates which functions of latent variables remain identifiable when representations are recoverable only up to symmetries. The authors propose a framework characterizing identifiable content through symmetry invariant functions and establish exact identifiability results for normalized iVAE models. They also show the necessity of normalization and introduce a skewness-based method for recovering individual latent coordinates. Experiments support the theoretical findings.

Critical strengths

This paper provides a framework for characterizing exactly identifiable functions beyond conventional identifability up to symmetry. The theoretical analysis of normalized iVAE models and the propose skewness-based normalization offer useful insights into latent representation identifiability.

Critical weaknesses

Although the theoretical framework is interesting, the experiments are restricted to relatively simple, low-dimensional settings. This limits the empirical support for the practical applicability of the theoretical results.

Questions for authors
  1. Can the exact identifiability results be extended to more general latent variable models?
  2. Most experiments involve only two or three latent dimensions. How does the proposed framework perform in higher-dimensional settings?
Reviewer_5se8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at which functions of the latent variables are determined by the data when the representation is only identifiable up to a group of symmetries. It gives a general test and applies it to a restricted, normalized iVAE setting with a fixed statistic chart. Under its assumptions, the identified quantities are exactly the functions unchanged by the remaining sign flips and permutations. It also studies what happens when arbitrary coordinatewise monotone changes are allowed, proposes a skewness rule for choosing signs and coordinate order, and illustrates the ideas with small synthetic experiments.

Critical strengths
  • The motivation is clear: equally valid latent representations can give different answers when we use individual coordinates.
  • The paper clearly separates knowing that ambiguity lies within a group from showing that every transformation in that group actually occurs. This is useful framing, although the second step appears straightforward for the iVAE setting studied here.
  • The experiments openly report failures of training and coordinate alignment (§7).
Critical weaknesses

My main concern is the limited advance over existing identifiability results. Once equivalent representations differ only by sign flips and permutations, and these transformations are all possible, the identified functions are simply those unchanged by them. For example, tanh⁡(∑iTi2)\tanh(\sum_i T_i^2) is unchanged by either transformation. The main characterization therefore seems to follow quite directly from the assumed ambiguity.

The construction showing that every transformation is possible also appears straightforward: transform the latents and adjust the decoder and prior to preserve the observations (§4.3). In the main full-range setting, the paper assumes the strong-identifiability step through A4 (§4.1). I therefore find it difficult to see what substantial new identification result Theorem 5 provides.

The bounded and one-sided cases also overlap with classical ICA results. For a fixed auxiliary value, both the original statistics and their affine transforms have independent coordinates. Standard consequences of Darmois–Skitovich force the affine map to be a permutation with separate coordinate scalings when at most one coordinate is Gaussian. Bounded and one-sided variables satisfy the relevant non-Gaussian condition. The paper should explain what its range-based results add beyond this existing argument.

The discussion also misses closely related work. Xi and Bloem-Reddy (AISTATS 2023) already connect identifiable downstream tasks to invariance under model ambiguities (§3) and apply their framework to iVAE (§4.2). A direct comparison is needed to establish the contribution.

Theorem 8 also appears incorrect under its stated subclass definition. The skewness rule is applied to each model’s own chart. A model with positive skewness can be reflected, while its chart and decoder are also reflected. Its chart values then retain positive skewness, and the observations remain unchanged. Both models satisfy the stated rule, even though the fixed latent coordinate changes. The claim that every fixed coordinate becomes identified therefore needs a revised definition (§5; Appendix F).

Questions for authors
  • Does the chart-pinned equivalence class contain anything beyond the signed-permutation relabelings of one model? If it does not, what substantial result does Theorem 5 add?
  • What does the framework add beyond Xi and Bloem-Reddy’s treatment of task identifiability and iVAE?
  • In Theorem 8, what excludes the reflected model described above? Must the skewness rule be imposed on the same fixed chart across all instances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.