Understanding Feature Learning Dynamics of SIGReg via BHEP Statistics

Self-supervised training with LeJEPA relies on the SIGReg regularizer, which pulls the embedding distribution toward an isotropic Gaussian and prevents dimensional collapse. However, its random projections make the learning dynamics hard to analyze directly. MMD-Gauss, a closed-form approximation of SIGReg, is a bandwidth mixture of BHEP statistics, and therefore we adopt BHEP as a tractable proxy that shares the same target distribution. Leveraging this theoretical proxy, we derive closed-form ODEs governing the macroscopic feature learning dynamics, exact under the BHEP proxy and our stated assumptions. Based on this formulation, we identify two phases: an initial rapid-growth $\text{Explosion Phase}$ in which leading covariance eigenvalues overshoot, followed by a self-stabilizing $\text{Shrinking Phase}$ in which they return toward the isotropic target while later features emerge one after another after increasingly long delays. We show that the eigenvalue dynamics are governed by a global $\text{engine}$ $\alpha$, which drives every dormant feature and collapses as the top features saturate, and a $\text{brake}$ $\beta$, which plateaus early, so the target eigenvalue $T=\alpha /\beta$ steps down with each saturated feature. For MMD-Gauss and SIGReg the bandwidth mixture leaves no single engine and brake, but the force on a dormant feature and the common target eigenvalue survive and decrease in the same way, so the two phases and the stepwise delay carry over.