acceptodds
Under review as a conference paper at ICLR 2027

Learnability and Competition in High-Dimensional Multi-Component ICA

Abstract

Independent Component Analysis (ICA) is a foundational tool for unsupervised representation learning, yet its high-dimensional theory remains largely limited to single-component recovery. We develop an asymptotically exact mean-field theory for multi-component online ICA, capturing the coupling induced by simultaneous learning and orthogonalization. In the high-dimensional limit, the joint empirical distribution of learned estimates and ground-truth components converges to a deterministic process, yielding a closed ODE system for the overlap matrix between learned directions and true components. This characterization reveals a genuinely multi-component, initialization-driven phase structure: a decoupled regime, where estimates align with distinct components and evolve nearly independently, and a competition regime, where overlapping initializations induce orthogonality-driven conflicts, slow reorientation, and delayed convergence. Our steady-state analysis gives explicit learnability boundaries and competition conditions linking step size, data moments, and initialization. These conditions show that larger higher-order moments and competition shrink the stable learning-rate window, increase convergence times, and predict a staircase phenomenon in which the number of recoverable components changes discretely with the learning rate. Experiments on synthetic data and hyperspectral remote sensing data validate the predicted trajectories and phase behavior.

Primary area:
learning theory
Keywords:
independent component analysis, mean-field theory, high-dimensional asymptotics, online learning dynamics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_a6GGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This study analyzes the optimization dynamics of multidimensional ICA using an asymptotically exact mean-field theory. It considers a standard projected SGD algorithm and shows that, as the observation dimension tends to infinity while the number of latent components remains finite, the joint distribution of the learned and ground-truth components converges to a deterministic process. The study then identifies two initialization-driven regimes, decoupled and competition, which explain differences in the convergence behavior of the independent components.

Critical strengths
  1. The study proposes a mean-field framework for describing the optimization dynamics of projected SGD for ICA, which appears to be new, especially in the multidimensional setting.

  2. The theoretical predictions of the optimization dynamics agree well with the numerical experiments.

Critical weaknesses
  1. The theory assumes noisy, high(infinite)-dimensional (n -> infinite) observations while the number of latent components (p) remains finite. In many conventional noiseless ICA settings, however, observations are finite-dimensional and can be reduced to p dimensions by PCA when (n>p). Although the theoretical predictions may remain accurate at finite n to some extent (Fig.11), it is unclear whether the conclusions about learning dynamics extend to such noiseless ICA settings after dimensionality reduction.

  2. The study considers standard projected SGD and does not explain more sophisticated and regularly used optimization methods, such as fixed-point algorithms (e.g., FastICA) or natural gradient methods. The phenomena identified in this study may differ under those methods.

Questions for authors

See Weaknesses.

3 comments

We thank the reviewer for their constructive assessment and recognition of the novelty of our mean-field framework and the agreement between theory and experiments. We address both concerns below.

W1. PCA dimensionality reduction and the high-dimensional setting

We agree that conventional noiseless ICA, with observations restricted to a rank-pp subspace, can be reduced to pp dimensions using PCA. Our setting addresses a different problem: learning non-Gaussian directions embedded in a high-dimensional, isotropic distribution.

Specifically, in our spiked-cumulant model (Eq. 1), the Gaussian component ak\boldsymbol{a}_k is introduced to whiten the observations rather than to model independent additive measurement noise. By construction, E[yk]=0\mathbb{E}[\boldsymbol{y}_k]=0 and E[ykyk⊤]=I\mathbb{E}[\boldsymbol{y}_k\boldsymbol{y}_k^\top]=\boldsymbol{I}. Consequently, the latent directions are invisible to population second-order statistics, and ordinary PCA cannot identify their subspace. Recovering them requires exploiting higher-order non-Gaussian structure. This is precisely the high-dimensional learning regime investigated in our paper.

Regarding dimensionality, our theoretical analysis considers the asymptotic regime n→∞n\to\infty with fixed pp. Figure 11 examines finite-dimensional convergence, showing improved theory–simulation agreement as nn increases. The additional (p=10,n=100)(p=10,n=100) experiment in Figure 12 deliberately explores a setting outside this asymptotic regime, providing complementary numerical evidence that qualitative competition persists even when the ambient dimension is relatively close to the number of components.

W2. Generality beyond projected SGD

Our primary contribution is an asymptotically exact dynamical theory for multi-component online projected SGD, which quantifies how initialization, higher-order source moments, learning rate, and orthogonalization jointly determine learning trajectories, competition, and component recovery. The behavior of other ICA optimization methods, such as FastICA and natural-gradient algorithms, may differ due to their distinct update mechanisms.

To explore this question numerically, we conducted an additional experiment using symmetric FastICA to investigate whether qualitatively similar competition can arise under a different optimization algorithm. We use the same data model with n=100n=100, (β1,β2)=(0.2,1)(\beta_1,\beta_2)=(0.2,1), and a cubic nonlinearity. Each FastICA iteration consists of a full-batch fixed-point update using a freshly sampled batch of m=106m=10^6 observations, followed by symmetric orthogonalization of the two estimates. Both initialization regimes use the same data batch at each iteration, and the results below correspond to a single realization. FastICA has no SGD learning-rate parameter.

Under decoupled initialization, both estimates rapidly recover distinct sources: their respective squared diagonal overlaps exceed 0.990.99 within two iterations, while the off-diagonal overlaps remain negligible.

Under competing initialization, however, both estimates initially pursue the second source. Their squared overlaps with this source reach approximately 0.490.49 and 0.500.50 at iteration 2. Subsequently, the second estimate captures that source while the first estimate reorients toward the remaining source.

Competition run (squared overlaps):

Squared overlapStep 0Step 2Step 4Step 6Step 9
Q112Q_{11}^20.09000.08760.27830.90680.9997
Q122Q_{12}^20.09000.48800.43660.09140.0000
Q212Q_{21}^20.09000.02230.21300.09130.0000
Q222Q_{22}^20.09000.49570.56260.90831.0000

These results exhibit qualitatively similar competition and reorientation under a batch fixed-point ICA algorithm, despite its different optimization dynamics. While our theoretical guarantees concern online projected SGD, the FastICA experiment provides complementary numerical evidence that similar competition can arise under a different optimization method.

Furthermore, new Appendix H and Figure 13 investigate symmetric Löwdin orthogonalization within the online learning framework. We derive a candidate limiting ODE, supported by simulations, and observe competition even without the sequential priority imposed by Gram--Schmidt. Although this extension remains conjectural, it further supports the robustness of the qualitative phenomenon across different orthogonalization mechanisms.


Together, these results suggest that competition among simultaneously learned components is a broader phenomenon, while the precise dynamics and learnability thresholds remain algorithm-dependent.

I appreciate the clarification of the problem setting and the additional evaluation using different optimization algorithms. I have no further questions and will increase my score accordingly.

We sincerely thank the reviewer for their feedback and for reconsidering their assessment. We are pleased that our rebuttal and additional results helped address their concerns, and we appreciate their time and consideration.

Reviewer_EnK4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The ICA is a famous unsupervised learning method and learning framework. This paper focuses on the whitened ICA in high-dimensional regime with online projected SGD algorithm. The data has the p non-gaussian sources within high dimensional space with gaussian noise, note that not sparse ICA. Their algorithm takes SGD steps and then applies Gram-Schmidt orthogonalization. Their theorem shows that the coordinates of estimates converge to the form of solving a PDE, and the overlap matrix follows a closed ODE. This paper gives two dynamics: decoupled and competition regime depending on the overlap matrix, while the first one makes the decoupled regime reduce to p independent ICA. Meanwhile, the competition regime is more interesting and may be worth more experiments. The author uses synthetic and real dataset (Indian Pines) to provide experiments.

In general the reviewer think it's an interesting and natural direction. However, the reviewer was not fully convinced for their scoping and contribution comparing to existing / classical work. I'm willing to raise my score if my concern in weakness and questions section were addressed.

Critical strengths
  1. The paper gives a multi-component limit result, which is a real technical work compared to the existing work like Wang and Lu (Neurips 2017) that only have single component results.

  2. Interesting trajectories and the learn and unlearn insight.

  3. Natural problem for high-dimensional limits ofr online ICA

Critical weaknesses
  1. The author should scope their results clearly against existing work.

In decoupled regime, the single component ODE is exactly results in Wang and Lu (Neurips 2017).

Also, it may be more helpful to present in classical ICA concept, for example: learnability vs identifiability. The reviewer would love to see discussion for the relation of it. The identifiability only requires non-Gaussian sources, however the learnability requires much stronger condition. It seems the learnable region is subset of identifiable region and when τ→0\tau \to 0. For example, a non-Gaussian source whose fourth cumulant is zero is still necessarily but not necessary learnable. It would be useful to discuss what happens for identifiable but non-learnable sources。

  1. With the Gram-Schmidt, the “competition” is actually influenced by the order that 1st estimation never influenced by others, but 2nd estimation was adapts/giving the way to estimation 1. The effects that truly co-involve multi-estimation are not clearly clarified. The reviewer suggests more experiments with theory supported.

  2. The real data experiment does not test the described setting, as hyperspectral unmixing data are poor fit for ICA. Abundances are in simplex (non-negative and sum to one) so are dependent. Such dependency will degrade ICA unmixing. Also, as n=200 was limited to dataset, the experiment can’t show what higher dimension theory has significant add-on value. The results would be more convincing with more high-dimensional ICA data.

Minor issue:

The reference “Linghuan Meng and Chuang Wang. Training dynamics of nonlinear contrastive learning model in the high dimensional limit.” Is using wrong URL: https://ieeexplore.ieee.org/document/10685361, which should be https://ieeexplore.ieee.org/document/10612238

Questions for authors
  1. Could the author clearify their contribution and novelty against existing work, include the ODE line of work.

a. The Scaling Limit of High-Dimensional Online Independent Component Analysis https://proceedings.neurips.cc/paper_files/paper/2017/file/3cfbdf468f0a03187f6cee51a25e5e9a-Paper.pdf

b. Learning Rate Should Scale Inversely with High-Order Data Moments in High-Dimensional Online Independent Component Analysis https://ieeexplore.ieee.org/document/11204212

c. Stochastic gradient descent in high dimensions for multi-spiked tensor PCA https://arxiv.org/abs/2410.18162

  1. Could the author positioning their phenomena in classical ICA and identifiability terms? For example:

a. Your GS orthogonalization vs FastICA with 'Symmetric Orthogonalization (Parallel)' and 'Deflation-based Orthogonalization (Sequential)'? b. Your methods and regimes vs classical ICA's nested identifiability and permutation ambiguity?

4 comments
AuthorsOpenReview ↗

Part 1/2 of the response

We thank the reviewer for the constructive feedback and for recognizing the technical contribution of our multi-component scaling limit and the learn-and-unlearn dynamics. We appreciate the opportunity to clarify our contributions relative to existing work, the role of orthogonalization, and the distinction between classical ICA identifiability and high-dimensional algorithmic learnability. We have also added new theoretical and numerical results to address the reviewer's concerns.

W1/Q1. Novelty and positioning relative to prior work

We agree that the decoupled regime recovers the single-component dynamics of Wang and Lu (2017). This is an expected consistency check of our framework, rather than the central novelty of our paper.

Our main contribution is the asymptotically exact characterization of the joint dynamics of simultaneously learned ICA components, including their interactions induced by orthogonalization. Specifically:

  • Joint mean-field dynamics: Theorem 4.1 establishes the convergence of the joint empirical measure of multiple learned and ground-truth components to a deterministic PDE, while Corollary 4.4 provides a closed, coupled (p×p)(p\times p) overlap ODE. Unlike the single-component setting, the resulting dynamics explicitly incorporate cross-estimator interactions through the orthogonalization operator.

  • Competition and reorientation: The coupled system identifies initialization-dependent regimes in which multiple estimates compete for the same source, potentially forcing one to unlearn an initially preferred component before recovering another. These trajectories cannot be characterized by independent single-component ODEs.

  • Multi-component learnability: Our steady-state analysis connects the learning rate, higher-order source moments, and initialization geometry to simultaneous component recovery, including the staircase phenomenon in the number of recoverable components.

Regarding the other cited works, Gültekin et al. (2025) investigates learning-rate dependence on higher-order moments in single-component online ICA, whereas our work additionally characterizes interactions and competition among multiple estimators. The multi-spiked tensor PCA work studies a related high-dimensional multi-signal recovery problem, but not the projected online ICA dynamics with orthogonalization-induced coupling analyzed here.

Thus, the central advance is not simply extending the dimension of an existing ODE, but deriving and analyzing the coupled dynamics that reveal qualitatively new multi-component learning behavior.

W2/Q2. Gram–Schmidt ordering and genuine multi-component competition

We appreciate the reviewer's observation that Gram–Schmidt introduces an asymmetric interaction structure: earlier estimates constrain subsequent ones, while the first estimate is unaffected by later estimates through orthogonalization. This is indeed captured explicitly by the lower-triangular operator (T(⋅))(\mathcal T(\cdot)) in our limiting ODE.

However, competition is not merely an artifact of this sequential ordering. To investigate this point, we have substantially extended the revised manuscript.

First, in new Appendix H, we analyze symmetric Löwdin (polar) orthogonalization, which treats all estimated components simultaneously without imposing sequential priority. By expanding the orthogonalization update, we derive a candidate limiting overlap ODE,

\tau\Psi_t^\top -\frac{\tau}{2}(M_t+M_t^\top)Q_t -\frac{\tau^2}{2}C_tQ_t.$$ Unlike the Gram–Schmidt ODE, the cross-component interactions are now symmetric. We emphasize that this is a formally derived, numerically supported conjecture rather than an additional rigorous scaling-limit theorem. **New Figure 13** compares the candidate Löwdin dynamics with simulations and contrasts them with Gram–Schmidt under identical initialization, source distributions, and learning rates. **Both schemes exhibit competition and reorientation, although the transient trajectories and eventual component assignments differ.** This provides direct evidence that competition persists without the sequential priority of Gram–Schmidt. Second, **new Figure 12 in Appendix G** considers (p=10) estimated components at (n=100). Multiple estimates develop transient alignments with the same source before one eventually dominates. This demonstrates that the qualitative competition phenomenon is observable beyond the two-component setting, including when the number of estimates is relatively large compared with the ambient dimension. We present this as finite-dimensional experimental evidence, not a theorem for proportional-growth asymptotics. Together, these additions clarify the distinction between the orthogonalization-dependent details of the dynamics and the more general phenomenon of competition among simultaneously learned components.
AuthorsOpenReview ↗

Part 2/2 of the response

W1/Q2. Classical ICA identifiability versus algorithmic learnability

Classical ICA identifiability concerns whether the independent sources can be uniquely determined from the observation distribution, up to the usual permutation and scaling ambiguities. In contrast, our learnability notion concerns the existence and stability of informative fixed points under a specified online learning algorithm, nonlinearity, and learning rate.

Consequently, identifiability does not necessarily imply learnability by a particular algorithm. For example, a non-Gaussian source with zero fourth cumulant can remain identifiable through its higher-order distributional structure, while the cubic score (f(x)=±x3)(f(x)=\pm x^3) has no fourth-cumulant-driven population learning signal for that source. A different nonlinearity may exploit other non-Gaussian statistics.

Our learnability boundaries therefore should not be interpreted as information-theoretic identifiability thresholds. Rather, they quantify algorithm-dependent recovery conditions in the high-dimensional online regime, including the additional restrictions arising from learning rate and competition.

W3. Real-data evaluation and dimensionality

We appreciate the concern regarding hyperspectral abundances, which need not satisfy the independence assumptions of classical ICA.

We clarify that the primary Indian Pines experiment (Figure 5) is semi-synthetic: real hyperspectral class signatures determine the ground-truth directions, while the training observations are generated according to our theoretical model. Its purpose is to evaluate the predicted dynamics using structured directions obtained from real data, rather than to claim that the original hyperspectral abundances are independent.

In addition, Appendix E.3 (Figure 9) evaluates the online algorithm directly on whitened hyperspectral pixels, explicitly departing from our generative-model assumptions. The observed agreement with the predicted trajectories provides empirical evidence of robustness beyond the idealized model, although it does not establish general validity for hyperspectral unmixing.

Regarding dimensionality, our asymptotic result concerns fixed (p)(p) as (n→∞)(n\to\infty). Figure 11 examines finite-dimensional convergence, showing improved theory–simulation agreement as (n)(n) increases. The additional (p=10,n=100)(p=10,n=100) experiment in Figure 12 illustrates qualitative competition in a more demanding finite-dimensional configuration. We do not claim that this experiment proves the validity of the ODE when (p/n)(p/n) remains bounded away from zero.

Minor issue

We thank the reviewer for identifying the incorrect URL info. We have fixed the bibliographic information in the revised manuscript.


We hope these clarifications and the new symmetric-orthogonalization analysis address the reviewer's concerns and better establish the novelty and scope of our contribution.

Reviewer_EnK4OpenReview ↗

Response to the author

The reviewer thanks the author for adding new theoretical and numerical results. I will raise my score to 3. And I'm looking forward to see the follow up work in this line in future.

We sincerely thank the reviewer for their feedback and for reconsidering their assessment. We are pleased that our rebuttal and additional results helped address their concerns, and we appreciate their time and consideration.

Reviewer_xNgyOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper develops an asymptotically exact mean-field theory for multi-component online ICA in high dimensions. By introducing the joint empirical measure over the coordinates of the learned and ground-truth vectors, the paper proves that such empirical measure converges to a deterministic limit governed by a PDE, as the ambient dimension diverges. The analysis reveals an initialisation-dependent phase structure as a decoupled regime of near-independent component learning and a competition regime where orthogonality-driven conflicts slow convergence. It further conducts steady-state analysis, which yields explicit learnability boundaries that predict a staircase phenomenon in the number of recoverable components as the learning rate varies. Experiments on both synthetic data and real datasets confirm that the mean-field predictions accurately track empirical learning dynamics even at moderate dimensions.

Critical strengths

The paper formulates the problem of multi-component online ICA in high dimensions in a concise manner. Specifically, it proves that the joint empirical measure converges to a deterministic PDE limit and derives a closed ODE system for the overlap matrix. The work further reveals an interesting initialisation-dependent phase structure, i.e., a decoupled regime and a competition regime. Its rigorous steady-state analysis reveals learnability boundaries and competition conditions in terms of data moments, learning rate, and initialisation.

Critical weaknesses

For deriving the explicit learnability boundaries with closed-form expressions, the work seems to rely on the case of cubic nonlinearities, which measures the skewness of signals. There are certainly more options, such as the fourth order moment and kurtosis. The speciality of the cubic nonlinearities seems to slightly weaken the contribution of the work.

Questions for authors

The paper reports in Figure 4 (c) that, as the off-diagonal/diagonal initialisation ratio increases, the largest learning rate recovering all components decreases, while convergence time increases. Did the authors use constant learning rates for SGD for comparison? Can you justify your choice? It's well known that for well-defined SGD algorithms to converge, the learning rate needs to satisfy the Robbins-Monro condition. Can the authors illustrate or demonstrate the impact of the Robbins-Monro condition on the present result?

3 comments

We thank the reviewer for their positive assessment of our work and for recognizing the contributions of the coupled mean-field dynamics, the initialization-dependent competition regimes, and the learnability analysis. We appreciate the thoughtful questions and provide clarifications below.

Q1. Cubic nonlinearity and higher-order statistics

We would like to clarify that the cubic nonlinearity f(x)=±x3f(x)=\pm x^3 is associated with fourth-order statistics (kurtosis), rather than skewness. Specifically, f(x)=x3f(x)=x^3 is the derivative of the fourth-order contrast F(x)=x4/4F(x)=x^4/4. For a standardized source, the corresponding population learning signal depends on the fourth cumulant κ4=E[c4]−3\kappa_4=\mathbb{E}[c^4]-3, as explicitly derived in Appendix C. The sixth-order moment also enters our analysis through the second moment of the stochastic gradient, thereby affecting the learning-rate-dependent stability boundaries.

Importantly, our general framework is not restricted to cubic nonlinearities. Theorem 4.1 and Corollary 4.4 apply to nonlinearities satisfying the stated regularity and moment assumptions. We focus on the cubic case in the steady-state analysis because it yields tractable closed-form conditions involving fourth- and sixth-order source moments. For other nonlinearities, the corresponding dynamics and stationary conditions can be evaluated using the general expressions, although explicit analytical boundaries may not be available. As an illustration, Appendix F (Figure 10a) also demonstrates agreement between the limiting ODE and simulations for f(x)=tanh⁡(x)f(x)=\tanh(x).

Q2. Constant versus diminishing learning rates and Robbins--Monro conditions

All experiments in Figure 4(c) use a constant learning rate τ\tau within each training run. To determine the maximum permissible learning rate τmax⁡\tau_{\max}, we vary τ\tau across separate runs while keeping it fixed throughout each run. The reported convergence times correspond to these fixed-learning-rate experiments.

This choice is consistent with our high-dimensional scaling-limit framework. In Eq. (2), the stochastic update has a dimension-dependent factor τ/n\tau/\sqrt{n}, and the limiting dynamics are obtained under the time rescaling t=k/nt=k/n, with τ\tau held fixed as n→∞n\to\infty. The resulting autonomous ODE describes the learning trajectories and learning-rate-dependent stability properties in this regime.

We agree that the classical Robbins--Monro conditions, ∑kηk=∞\sum_k\eta_k=\infty and ∑kηk2<∞\sum_k\eta_k^2<\infty, are important for almost-sure convergence in stochastic approximation under suitable additional assumptions. However, these conditions concern diminishing-step-size algorithms in the long-time limit for a fixed problem dimension, whereas our analysis considers a dimension-scaled update with fixed τ\tau over the high-dimensional time scale k=O(n)k=O(n).

Consequently, our fixed-τ\tau learnability boundaries and convergence-time predictions should not be interpreted as almost-sure convergence guarantees under Robbins--Monro schedules. A diminishing learning-rate schedule would lead to a different, generally time-dependent limiting dynamical system and may alter the long-time stability and recovery behavior. Its systematic analysis is beyond the scope of the present work.


We hope these clarifications address the reviewer's questions and make the interpretation and scope of our theoretical results clearer.

Thanks for the authors' clarification. I naturally keep my score.

We sincerely thank the reviewer for their positive assessment and continued support of our work. We appreciate their feedback and the time they devoted to reviewing our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.