acceptodds
Under review as a conference paper at ICLR 2027

Kinks vs. Smoothness: Identifiability of Real Analytic nICA for Laplace-like Sources

Abstract

Many machine learning systems try to explain complex data - like images or financial time series - in terms of hidden, independent factors that generated them. Recovering the true underlying factors, rather than some scrambled version of them, is the central challenge of nonlinear Independent Component Analysis (nICA). We prove identifiability (exact recovery) up to trivial ambiguities for real analytic generating functions when source probability density functions have a finite number of discontinuities in the first derivative. The Laplace distribution is the most prominent example satisfying this assumption. The proof relies on the contrast between kinks in the source distribution and the smoothness of real analytic functions. Real analytic functions comprise a broad class of generating mechanisms, and can be approximated with Normalizing Flows or Variational Autoencoders with standard activation functions (e.g., tanh, softplus, GELU), so our result applies with minimal changes to existing training pipelines. We perform experiments on real and synthetic data with both Normalizing Flows and Variational Autoencoders demonstrating their identifiability properties. In experiments on CelebA data we recover several interpretable latent factors controlling unique attributes across the dataset.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Nonlinear Independent Component Analysis, nICA, Identifiability, Real Analytic, Sparse Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PLopOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies identifiability in nonlinear ICA without auxiliary variables for cases where the generating functions are real analytic and the source probability density functions have finitely many discontinuities in their first derivatives. The Laplace distribution is the main example. Experiments on real and synthetic data with Normalising Flows and Variational Autoencoders show the identifiability properties.

Critical strengths
  1. No regulariser or auxiliary information is needed for real analytic networks, so the results plausibly apply to existing pipelines.
  2. The synthetic study varies activation functions and source distributions and shows a consistent pattern with the theory.
Critical weaknesses
  1. There are some potential gaps between the theorem and the trained models. For example, the g~\tilde{\mathbf{g}} learned by a dense flow with element-wise tanh or sigmoid activations, and softplus, is mapped onto (0,∞)k(0, ∞)^k, and thus is not a diffeomorphism onto the support of a Laplace distribution and is not covered by Theorem 1. Meanwhile, no existing nICA baselines, e.g., methods discussed in related work, are compared on synthetic data.
  2. Several steps in the theorem may need more careful statements. For example, Step 2 concludes that hl≡βmh_l\equiv\beta_m "on an open neighbourhood of AijA_{ij}". However, an open set in RkR^k would contradict invertibility.
Questions for authors
  1. Please check the experimental settings and discuss the potential gaps with the theoretical settings.
  2. It is suggested to include an empirical comparison with related nICA baselines in the controlled setting where MCC is measurable.
  3. Please carefully check theoretical statements for improved readability and rigour.
Reviewer_WPGsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new condition to guarantee the identifiability of nonlinear independent component analysis (nICA) without auxiliary information. The key idea is to exploit the contrast between the discontinuity (kink) in the source density and the real-analytic nature of the mixing and separation functions. Specifically, the paper argues that if the log-density of the source distribution exhibits first-order differential discontinuity at a finite number of breakpoints, and if the mixing and separation functions are real-analytic diffeomorphisms, then the latent sources are identifiable, except for ambiguity with respect to signed permutations and translations. The Laplace distribution is a representative example.

The theoretical argument of the paper consists of two steps. First, it construct a self-map h=g~∘fh=\tilde g\circ f in the latent space from two models that generate the same observation distribution, and use Lemma 1 to show that hh is a real analytic diffeomorphism that preserves the source distribution. Subsequently, Theorem 1, the main result, claims that for a iid source satisfying Assumption 2, such a hh is restricted to signed permutations and translations.

For numerical verification, this paper suggests normalizing flows and VAEs that use real-analytic activations and evaluate them using synthetic data, Yahoo stock returns, and CelebA. In the synthetic experiments, Laplace and Laplace mixture model sources and analytic activations demonstrate relatively high Mean Correlation Coefficient (MCC). On CelebA, some interpretable latent directions are qualitatively presented.

Critical strengths
  1. This paper approaches a key problem in nICA from a simple and intriguing perspective. Obtaining identifiability for general nonlinear mixing without auxiliary variables is a difficult problem. The "kinks vs. smoothness" perspective presented in this paper---namely, the idea that the non-analytic structure of the source density cannot be arbitrarily rearranged by analytic coordinate transformations---is intuitive and intriguing.

  2. This paper provides a form of identifiability condition that differs significantly from existing results. Unlike previous studies that utilize structural conditions such as Jacobian sparsity, conformality, auxiliary variables, or piecewise-affine mixing, this paper combines kinks in the source distribution with real analyticity. The form of the assumptions also differs significantly from the setting in Kivva et al. (2022), which the authors claim the most comparable.

  3. The connection between the theoretical conditions and actual neural network architectures is straightforward. The authors argue that by using real-analytic activations such as tanh, sigmoid, and softplus, existing normalizing-flow architectures can be utilized with minimal modification. If the theory is correct, this could be a practical advantage.

  4. Results consistent with some theoretical predictions are observed in synthetic experiments. In particular, analytic activations exhibit a higher MCC than non-analytic activations for the Laplace source, and the source recovery results for the Laplace and Laplace mixture sources are also relatively strong.

Critical weaknesses
  1. The most important issue is that Theorem 1 does not appear to hold under the current assumptions. Consider the following univariate counterexample. Define the real-analytic diffeomorphism ϕ(x)=x+x2+x3\phi(x)=x+x^2+x^3 and set
.$$ Since $\phi'(x)=1+2x+3x^2>0$, $\phi$ is a real-analytic diffeomorphism on $\mathbb{R}$. Furthermore, $\log\rho$ has two analytic branches centered at 0, and $$\alpha_-'(0)=3,\qquad \alpha_+'(0)=1 .$$ So the required first-derivative discontinuity at 0 exists. Therefore, this density appears to satisfy Assumption 2 of the paper. However, if we define $$\tau(x)=\phi^{-1}(-\phi(x)) ,$$ then $\tau$ is a real-analytic diffeomorphism that preserves $\rho$. Indeed, if $X\sim\rho$, then $\phi(X)$ follows a symmetric Laplace, so $-\phi(X)$ also has the same distribution. Therefore, $\tau(X)\sim\rho$. On the other hand, $\tau$ is not an affine reflection. To be concrete, $$\tau''(0)=-4 ,$$ not zero. Applying this to each coordinate yields a real-analytic diffeomorphism that preserves the iid product distribution but is not a signed permutation plus translation. 2. The equivalence relation was not defined in Definition 4. Definition 4 presents $$p_{\tilde f,\tilde g}=p_{f,g}\Longrightarrow g\simeq\tilde g$$ as the definition of identifiability, but does not explicitly define what $\simeq$ means. Although it is possible to infer from Theorem 1 that the intended equivalence refers to signed permutations and translations, to formalize the identifiability statement itself, it must be explicitly stated, for example, as $$g\simeq\tilde g \iff \tilde g=Pg+c ,$$ with $P$ a signed permutation. Furthermore, if the source distribution $\rho$ is known and fixed, arbitrary translations generally do not preserve the product measure $\rho^{\otimes k}$; therefore, it is necessary to describe the actual admissible ambiguity group more precisely. 3. There appears to be no guarantee that Mixture A in the synthetic data experiments is generally a diffeomorphism. In Mixture A, the componentwise transformation is defined as $$a_j(y)=\frac{1}{4}\sin(y+2\pi w_j)+y^3 .$$ However, $$a_j'(0)=\frac{1}{4}\cos(2\pi w_j) .$$ Therefore, when $w_j \sim \text{Unif}(0,1)$, there is a significant probability that $a_j'(0) < 0$, whereas for sufficiently large $|y|$, the derivative becomes positive due to the $3y^2$ term. Consequently, in such cases, $a_j$ is neither monotone nor injective, so $f$ does not satisfy the core assumption of the theory that $f$ is a global diffeomorphism. Consequently, it is questionable whether the Mixture A experiment is appropriate for verifying Theorem 1. 4. The experimental results do not sufficiently distinguish between identifiability itself and optimization bias. For example, while there is a large difference in MCC between analytic activation and LeakyReLU, there is insufficient comparison of likelihood or fit quality to show whether each model learned the observation distribution to the same degree. Therefore, it is difficult to determine whether a high MCC is due to theoretical identifiability, optimization, architecture matching, or implicit regularization. Furthermore, according to Theorem 1, triangular and truncated Laplace distributions—which should be identifiable—actually exhibit low MCC values. While the authors explain that practical identifiability may fail due to analytic approximation on compact support, this also demonstrates that there is a significant gap between exact population-level identifiability and actual recovery.
Questions for authors
  1. Can you explain the following counterexample to Theorem 1?
\rho(x)=\frac{1}{2}e^{-|\phi(x)|}\phi'(x).$$ As I understand it, $\rho$ satisfies Assumption 2. However, $$\tau(x)=\phi^{-1}(-\phi(x))$$ is a non-affine, real-analytic diffeomorphism that preserves $\rho$. If this example violates Assumption 2, please explain clearly exactly which condition it violates. 2. Can you precisely define the equivalence relation $\simeq$ in Definition 4? In particular, when the source density $\rho$ is known and fixed, which signed permutations/translations are included in the actual ambiguity group? 3. The component function of Mixture A $$a_j(y)=\frac{1}{4}\sin(y+2\pi w_j)+y^3$$ is a global diffeomorphism? It appears that the derivative may change sign depending on the randomly sampled $w_j$. 4. Can you provide evidence (e.g., held-out likelihood) from the synthetic experiments showing that different activation/source models learned the observation distribution to a similar degree? If not, how do you rule out the possibility that the MCC difference stems from optimization or model-fit differences rather than identifiability?
1 comment

We have uploaded a revision. See the paragraph starting on line 188 for an explanation of the translation ambiguity class.

Reviewer_vhpMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper considers nonlinear ICA when the latent factors/source has densities with kinks. They claim identifiability results when the nonlinear map and its inverse are real-analytic. They propose RAD, which uses real analytic activation functions in deep generative networks. They conduct empirical studies.

Critical strengths

None.

Critical weaknesses

Unsoundness of mathematical analysis: There are certain gaps in the proof.

  • In Eq (2), the Weierstrass Preparation Theorem is invoked to claim that r is a globally non-zero real analytic function. However, the Weierstrass Preparation Theorem is a local result and cannot be used to make the above global claim.

Limited Methodological Novelty: Since the premise of the theory is in doubt due to the above gap, one looks for any methodological innovation or empirical gains. The authors did propose RAD, which by virtue of its simplicity (a good thing) suffers from the fact that the amount of novelty is relatively minimal (replacing common activation functions with other existing activation functions that are analytic). The experiments/simulations done are relatively toy.

Questions for authors

N/A

1 comment

We have updated the language around the use of the Weierstrass Preparation Theorem on lines 238, 244, and 691.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.