- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This study analyzes the optimization dynamics of multidimensional ICA using an asymptotically exact mean-field theory. It considers a standard projected SGD algorithm and shows that, as the observation dimension tends to infinity while the number of latent components remains finite, the joint distribution of the learned and ground-truth components converges to a deterministic process. The study then identifies two initialization-driven regimes, decoupled and competition, which explain differences in the convergence behavior of the independent components.
Critical strengths
-
The study proposes a mean-field framework for describing the optimization dynamics of projected SGD for ICA, which appears to be new, especially in the multidimensional setting.
-
The theoretical predictions of the optimization dynamics agree well with the numerical experiments.
Critical weaknesses
-
The theory assumes noisy, high(infinite)-dimensional (n -> infinite) observations while the number of latent components (p) remains finite. In many conventional noiseless ICA settings, however, observations are finite-dimensional and can be reduced to p dimensions by PCA when (n>p). Although the theoretical predictions may remain accurate at finite n to some extent (Fig.11), it is unclear whether the conclusions about learning dynamics extend to such noiseless ICA settings after dimensionality reduction.
-
The study considers standard projected SGD and does not explain more sophisticated and regularly used optimization methods, such as fixed-point algorithms (e.g., FastICA) or natural gradient methods. The phenomena identified in this study may differ under those methods.
Questions for authors
See Weaknesses.
3 comments
I appreciate the clarification of the problem setting and the additional evaluation using different optimization algorithms. I have no further questions and will increase my score accordingly.
We sincerely thank the reviewer for their feedback and for reconsidering their assessment. We are pleased that our rebuttal and additional results helped address their concerns, and we appreciate their time and consideration.
We thank the reviewer for their constructive assessment and recognition of the novelty of our mean-field framework and the agreement between theory and experiments. We address both concerns below.
We agree that conventional noiseless ICA, with observations restricted to a rank-p subspace, can be reduced to p dimensions using PCA. Our setting addresses a different problem: learning non-Gaussian directions embedded in a high-dimensional, isotropic distribution.
Specifically, in our spiked-cumulant model (Eq. 1), the Gaussian component ak is introduced to whiten the observations rather than to model independent additive measurement noise. By construction, E[yk]=0 and E[ykyk⊤]=I. Consequently, the latent directions are invisible to population second-order statistics, and ordinary PCA cannot identify their subspace. Recovering them requires exploiting higher-order non-Gaussian structure. This is precisely the high-dimensional learning regime investigated in our paper.
Regarding dimensionality, our theoretical analysis considers the asymptotic regime n→∞ with fixed p. Figure 11 examines finite-dimensional convergence, showing improved theory–simulation agreement as n increases. The additional (p=10,n=100) experiment in Figure 12 deliberately explores a setting outside this asymptotic regime, providing complementary numerical evidence that qualitative competition persists even when the ambient dimension is relatively close to the number of components.
Our primary contribution is an asymptotically exact dynamical theory for multi-component online projected SGD, which quantifies how initialization, higher-order source moments, learning rate, and orthogonalization jointly determine learning trajectories, competition, and component recovery. The behavior of other ICA optimization methods, such as FastICA and natural-gradient algorithms, may differ due to their distinct update mechanisms.
To explore this question numerically, we conducted an additional experiment using symmetric FastICA to investigate whether qualitatively similar competition can arise under a different optimization algorithm. We use the same data model with n=100, (β1,β2)=(0.2,1), and a cubic nonlinearity. Each FastICA iteration consists of a full-batch fixed-point update using a freshly sampled batch of m=106 observations, followed by symmetric orthogonalization of the two estimates. Both initialization regimes use the same data batch at each iteration, and the results below correspond to a single realization. FastICA has no SGD learning-rate parameter.
Under decoupled initialization, both estimates rapidly recover distinct sources: their respective squared diagonal overlaps exceed 0.99 within two iterations, while the off-diagonal overlaps remain negligible.
Under competing initialization, however, both estimates initially pursue the second source. Their squared overlaps with this source reach approximately 0.49 and 0.50 at iteration 2. Subsequently, the second estimate captures that source while the first estimate reorients toward the remaining source.
Competition run (squared overlaps):
These results exhibit qualitatively similar competition and reorientation under a batch fixed-point ICA algorithm, despite its different optimization dynamics. While our theoretical guarantees concern online projected SGD, the FastICA experiment provides complementary numerical evidence that similar competition can arise under a different optimization method.
Furthermore, new Appendix H and Figure 13 investigate symmetric Löwdin orthogonalization within the online learning framework. We derive a candidate limiting ODE, supported by simulations, and observe competition even without the sequential priority imposed by Gram--Schmidt. Although this extension remains conjectural, it further supports the robustness of the qualitative phenomenon across different orthogonalization mechanisms.
Together, these results suggest that competition among simultaneously learned components is a broader phenomenon, while the precise dynamics and learnability thresholds remain algorithm-dependent.