acceptodds
Under review as a conference paper at ICLR 2027

Understanding Feature Learning Dynamics of SIGReg via BHEP Statistics

Abstract

Self-supervised training with LeJEPA relies on the SIGReg regularizer, which pulls the embedding distribution toward an isotropic Gaussian and prevents dimensional collapse. However, its random projections make the learning dynamics hard to analyze directly. MMD-Gauss, a closed-form approximation of SIGReg, is a bandwidth mixture of BHEP statistics, and therefore we adopt BHEP as a tractable proxy that shares the same target distribution. Leveraging this theoretical proxy, we derive closed-form ODEs governing the macroscopic feature learning dynamics, exact under the BHEP proxy and our stated assumptions. Based on this formulation, we identify two phases: an initial rapid-growth in which leading covariance eigenvalues overshoot, followed by a self-stabilizing in which they return toward the isotropic target while later features emerge one after another after increasingly long delays. We show that the eigenvalue dynamics are governed by a global , which drives every dormant feature and collapses as the top features saturate, and a , which plateaus early, so the target eigenvalue steps down with each saturated feature. For MMD-Gauss and SIGReg the bandwidth mixture leaves no single engine and brake, but the force on a dormant feature and the common target eigenvalue survive and decrease in the same way, so the two phases and the stepwise delay carry over.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Learning Dynamics, JEPA, LeJEPA, Regularizer, SIGReg

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r4pUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies feature learning dynamics in LeJEPA, focusing on the SIGReg regularizer. Since SIGReg is difficult to analyze directly, the authors use BHEP as a simpler theoretical proxy, connected to SIGReg through MMD-Gauss. Under simplifying assumptions, they derive an explicit ODE for the embedding covariance eigenvalues and propose an engine-brake mechanism to explain two phases of learning: rapid growth and overshooting of leading features, followed by their shrinking and delayed emergence of other features. The authors investigate how this mechanism extends to SIGReg and support their analysis with experiments on synthetic data and neural networks trained on real datasets.

Critical strengths

Original theoretical analysis.

The paper derives an explicit coupled eigenvalue ODE for BHEP and proposes an original engine-brake mechanism to explain feature-learning dynamics. The authors also establish a mathematical connection between BHEP and SIGReg through MMD-Gauss and provide numerical comparisons and experiments that qualitatively support the proposed mechanism. These results represent a potentially interesting theoretical contribution.

Critical weaknesses

1. Unclear scientific significance and scope of the proposed mechanism.

My main concern is that the paper does not convincingly establish the broader scientific significance of its theoretical contribution. Previous work (e.g., Simon et al., 2023) has already theoretically studied stepwise feature learning and empirically observed nonmonotonic eigenvalue dynamics. The authors acknowledge this literature and propose engine-brake mechanism to explain overshooting, shrinking, and delayed emergence. While this is a potentially interesting and original contribution, it remains unclear what broader understanding or testable predictions follow from the proposed mechanism beyond a qualitative explanation of these previously observed phenomena.

First, the theoretical setting is rather restrictive. The main analysis considers a linear model with Gaussian projected inputs, assumes aligned weights, and neglects the invariance term by assuming it has already saturated. Although the authors discuss these limitations and partially relax some assumptions in the appendix, the analysis remains highly idealized.

Second, while Theorem 1 provides an explicit eigenvalue ODE, the explanation of the two-phase dynamics relies heavily on idealized configurations with partially saturated features. The authors also establish theoretical properties of the equilibrium ladder and provide quantitative comparisons with toy training in Appendix J. Nevertheless, it remains unclear under what conditions this analysis accurately describes general training trajectories, particularly in nonlinear models.

Third, although the authors establish a connection between BHEP and SIGReg through MMD-Gauss and provide theoretical and empirical evidence for similar behavior, the extent to which the proposed mechanism explains practical SIGReg training remains uncertain. Section 5.2 and Appendix K investigate discrepancies between theoretical predictions and practical observations. In particular, the authors derive corrections for the unsaturated invariance term and non-Gaussian embeddings, supported by toy experiments and comparisons with practical training. However, the practical validation is primarily qualitative, and the authors acknowledge that the first-order non-Gaussian correction is not quantitatively reliable in some of their experiments. Therefore, it remains unclear how accurately the proposed mechanism can explain or predict the observed dynamics beyond the simplified theoretical setting.

Overall, I find the proposed analysis potentially interesting, but I am not yet convinced that it provides sufficiently important and generalizable insights about SIGReg or SSL feature learning. A clearer explanation of its scientific significance, predictive value, and limits of applicability would strengthen the contribution.

2. Insufficiently clear presentation of the theoretical setup and mechanism.

Although the paper is generally well organized, I found the main theoretical analysis difficult to follow. In particular, the assumptions underlying the eigenvalue dynamics are introduced rather briefly, without sufficiently explaining their meaning and role in the proposed mechanism. Moreover, Section 4.3 introduces configurations with partially saturated features and uses them to explain the two-phase dynamics, but does not clearly explain how these configurations are constructed or why they should describe actual training trajectories. The distinction between the exact eigenvalue ODE, the equilibrium-ladder analysis, and the resulting qualitative interpretation is not sufficiently clear. Finally, the extension of the mechanism to SIGReg in Section 4.4 is rather compressed, with important details deferred to the appendix. I believe the main text would benefit from a more careful explanation of the theoretical setup, the central mechanism, and the precise scope of the conclusions.

Questions for authors
  1. Scientific significance. What important new understanding does the proposed engine-brake mechanism provide beyond existing analyses of stepwise feature learning, such as Simon et al. (2023)? In particular, can the authors identify concrete, testable predictions that follow from this mechanism and demonstrate its scientific value beyond a qualitative explanation of previously observed phenomena?
  2. Scope and applicability. Appendix J provides evidence that the equilibrium-ladder analysis describes toy BHEP training, but under what conditions can this picture be expected to hold for general training trajectories and practical SIGReg models? Furthermore, although Appendix K investigates violations of the theoretical assumptions, some corrections are supported mainly by qualitative evidence. Can the authors clarify which conclusions remain reliable when these assumptions are violated, and whether the corrected theory can quantitatively explain the discrepancies observed in practical training?
2 comments

We thank the reviewer for the careful reading and for the questions on significance and scope.

Weakness

1. Restrictive Theoretical Setting, Insufficient Two-Phase Explanation, Uncertain training explanation: We agree the setting is restrictive and idealized and does not transfer quantitatively to general learning. What we observe in the real runs is the mechanism itself, for BHEP and MMD-Gauss. The engine and the target decrease during training (Figure 4), which in the theory is what produces the delayed stepwise emergence and the overshooting-shrinking behavior. For SIGReg we show eigenvalue trajectories only.

On nonlinear models: the regularizer is a function of the embeddings alone, so the encoder enters only through how its gradient reaches each eigenvalue. We derive this only for a linear encoder, with the misaligned case in Appendix K.4, and we have no theory for nonlinear encoders. For ResNet and ViT encoders our support is experimental (Section 5 and Appendix M).

The ladder describes training when features saturate one at a time. When several features grow together, the engine falls faster than the ladder predicts (Appendix J).

Significance is answered in Question 1, applicability on real runs in Question 2.

2. Insufficiently clear presentation: In the revision:

  • Assumptions: Section 4.1 now lists the four assumptions, each with what it provides to the derivation.
  • Ladder: Section 4.3 now defines the equilibrium ladder before using it: mm features saturated at a common level T(m)T(m) and the rest dormant. It is an idealization. Training does not move exactly along it, and we use it to describe how the forces change as features saturate. Theorem 2 shows that between two rungs the target and the engine stay between the rung values.
  • Section 4.4 now defines the coefficients of the MMD-Gauss force in the main text and states which part of the carry-over is proved and which is numerical. The derivations remain in Appendices H and L.

For the distinction:

  • Exact ODE: Theorem 1, exact ODE for each eigenvalue. Exact under the assumptions, with no ladder involved.
  • Equilibrium-ladder analysis: a family of idealized states indexed by the number mm of saturated features, used to show how the engine and the target change as mm grows.
  • Resulting qualitative interpretation: using the ladder analysis to read the real runs.

Question

1. Scientific Significance: Our analysis follows the framework of Simon et al. (2023): aligned weights and one ODE per eigenvalue. The difference lies in the objective. In their linear model with a simplified Barlow Twins loss the modes separate, so each eigenvalue follows its own curve toward a fixed target, and its emergence time is set by the eigenvalue of that mode alone. The SIGReg regularizer depends on the whole spectrum, so in its BHEP proxy the modes share two global quantities, the engine α\alpha and the brake β\beta, and the learning of one feature changes the force on the others.

Two consequences follow from this coupling, and both can be tested:

  • The target T=α/βT=\alpha/\beta moves as features saturate, so an early feature overshoots and then shrinks.
  • The emergence time of a feature depends on the features ahead of it, not only on its own input variance. With a shared engine that decreases as features saturate, a later feature is delayed further. The toy experiment of Section 5.1 is consistent with this: the input variances differ by about 4×4\times, while the convergence times span orders of magnitude.

As the reviewer notes, nonmonotonic eigenvalue dynamics have been observed empirically. Our analysis offers one mechanism for them in the SIGReg setting.

2. Scope and Applicability: The ladder values apply quantitatively only when the invariance term has no effect and κrad≪d2\kappa_{rad}\ll d^2. In our real runs the first fails, and the second holds only approximately for SIGReg and MMD-Gauss. On real runs we cannot correct the deviations from the idealized predictions quantitatively, which is one of the limitations of our paper. The corrections we give are qualitative, and they are in Appendix K. What we trust on real runs is the direction of the effects: sequential emergence, the band of feature levels, and the end point λˉ≈1\bar\lambda\approx1 of SIGReg in the ResNet runs, where the invariance loss is low. With the MLP encoder, where it is the highest, the mean ends at 0.850.85 (Table 1). What we do not trust is the overshoot magnitude and the band width. Each feature stops where the force equals its invariance drag, so the band lies between the drag-free target T and the level of the most-dragged feature, which is 00 for a feature that does not emerge (Appendix K).

I thank the authors for their detailed responses and revisions. The theoretical exposition has been improved, and I appreciate the additional results, particularly Theorem 2. Accordingly, I have increased my Presentation score from 1 to 2.

However, my main concerns about scientific significance and applicability to practical SIGReg training remain unresolved. Therefore, I maintain my overall score of 2: Weak rejection.

Reviewer_Kp16OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates feature learning dynamics in SIGReg using a tractable BHEP proxy. It derives coupled eigenvalue dynamics and introduces an engine–brake interpretation of feature growth, overshoot, and delayed emergence. Experiments compare these predictions with practical training and examine deviations associated with unsaturated invariance and non-Gaussian embeddings.

Critical strengths
  • This paper builds an interpretable analytical framework. The BHEP proxy yields explicit coupled eigenvalue dynamics under the stated assumptions. The engine–brake decomposition provides a useful mechanistic account of how early-emerging features influence the growth and timing of later features.
  • The paper explicitly examines departures from its idealized predictions and investigates the effects of unsaturated invariance and non-Gaussian embeddings.
Critical weaknesses
  • The scope of the explanation could be clearer. Section 4.4 states that stepwise delay and overshoot–shrinking behavior carry over from BHEP to SIGReg. However, Section 5.2 reports differences between the real-data trajectories and the equilibrium-ladder predictions. Although the authors discuss these differences, they could state more clearly which predictions still hold when the idealized assumptions are violated and under what conditions the explanation applies to practical SIGReg training.

  • The theory needs a closer quantitative comparison with the experiments. Section 4.4 predicts that the mean eigenvalue approaches 1 from below for BHEP but overshoots for SIGReg and MMD-Gauss, while Section 5.2 reports the opposite pattern. Similarly, Section 4.3.2 describes all eigenvalues converging to a common isotropic level, whereas Section 5.2 observes a band of distinct feature levels. The authors discuss corrections for these differences, but comparisons of predicted and observed overshoot magnitudes and feature-level distributions would help show how well those corrections explain the results.

  • The presentation makes the main contribution difficult to follow. The move from SIGReg to MMD-Gauss and then to BHEP has a technical rationale, but the paper could explain more directly why each objective is introduced, which results apply to each objective, and how those results connect. Making these connections clearer would help readers understand the main contribution and what they should take away from the analysis.

Questions for authors
  • Which predictions remain valid when the Gaussianity and saturated-invariance assumptions are violated, and under what conditions should they hold in practical SIGReg training?
  • Could the authors further clarify which conclusions are established for BHEP, which extend theoretically to SIGReg, and which are supported mainly by experiments?
2 comments

We thank the reviewer for pointing out where the theory and the experiments do not match.

Weakness

1. Scope of the explanation: The carry-over of Section 4.4 holds on the equilibrium ladder, under the assumptions. Section 5.2 is where real training departs from it.

Independent of the assumptions:

  • the structure of the regularizer: a push term and a pull term (Eq. 4), each a function of the whole embedding distribution, so the force on one feature depends on all the others
  • α\alpha, β\beta, and TT are measurable from P1P_1 and P2P_2 in any run

Holds when only the alignment assumption is dropped (Appendix K.4):

  • the sign of each eigenvalue's motion, the fixed point, and the engine-brake form of the ODE

Observed in the real runs, without a proof outside the assumptions:

  • the decrease of the engine and of the target during training, for BHEP and MMD-Gauss (Figure 4)
  • the delayed stepwise emergence of features (Figure 3)

Changed in a predicted direction:

  • unsaturated invariance: each feature feels an additional drag, so it settles below the target TT at its own level, giving a band of distinct feature levels instead of a single line, and features whose drag exceeds the engine do not emerge
  • non-Gaussian embeddings: for a positive radial fourth cumulant κrad>0\kappa_{\rm rad} > 0, the first-order correction raises the engine, and with it the target and the fixed point

Not clearly observed in the real runs

  • the shrinking phase. The closest case is CelebA (Appendix M).

Not predicted

  • the magnitudes on real data, namely emergence times, the size of the overshoot, and the width of the band.

2. Quantitative comparison with experiments: The reviewer is right that the observed direction of the mean eigenvalue is opposite to the ladder prediction. We attribute both deviations to the two effects above. The invariance term makes some features collapse and holds the others below the target. A toy model with augmentation then rises to 1 from below for all three objectives (Figure 7). That this lowers the mean is an observation (Table 1), not a prediction. For BHEP the first-order non-Gaussian correction has the sign of the observed overshoot, and we do not attribute its magnitude (Appendix K.2).

What we can compare quantitatively: the ladder against toy trajectories (Appendix J), and the ratio F(0)/TF(0)/T, which after a few epochs stays within [1.03,1.10][1.03, 1.10] in the every real MMD-Gauss runs (Appendix M) in our paper.

What we cannot: the overshoot magnitude of BHEP, because its measured κrad\kappa_{rad} lies outside the range of the first-order correction, and the feature-level distribution, which we do not predict. We acknowledge this limitation.

3. Move from SIGReg to MMD-Gauss and then to BHEP: SIGReg is the original regularizer, but its random projections make an ODE hard to derive. MMD-Gauss is the closed-form expectation of SIGReg over the projections. The Gamma-integral identity rewrites its inverse-root approximation as an integral over BHEP statistics, and one slice, BHEP, gives α\alpha and β\beta in closed form.

The differences are quantitative: the exact statistics differ across objectives (the gap between MMD-Gauss and SIGReg shrinks as dd grows, Appendix L), and only BHEP has closed-form factors.

Section 3.2 now ends with this chain (Eq. 8), and which results apply to which objective is listed in Question 2.

Question

1. Conditions under which the predictions hold: Weakness 1 lists which predictions remain. The ladder values are derived for the regularizer alone. With the invariance term acting, the prediction is different: each feature settles below the target at its own level, which gives the band of Weakness 1. The band does not require the invariance term to vanish: it already appears in the toy model with a fixed augmentation (Figure 6), and it narrows as the augmentation decreases (Figure 7).

The quantitative ladder values also need κrad≪d2\kappa_{rad}\ll d^2. The measured κrad\kappa_{rad} reaches about 26d26d for SIGReg and MMD-Gauss, and 90d90d under the default and 1200d1200d under the small initialization for BHEP, with d=128d=128 (Appendix K.2). It is therefore approximately met for the former and violated for the latter.

2. What is established for BHEP and what extends to SIGReg:

Established for BHEP:

  • Theorem 1
  • Closed-form ladder, the decrease of TT along it, the bounds of Eq. 14, and Theorem 2 between the rungs

Extends theoretically to MMD-Gauss:

  • Extended Theorem 1 for MMD-Gauss (Eq. 15, Section 4.4)
  • Isotropic fixed point λˉ=1\bar{\lambda}=1, shared by all three objectives
  • F(0)F(0), TT decrease on the ladder, proof as d→∞d\to\infty (Appendix Q)

For SIGReg, by exact numerical evaluation (Appendix L):

  • F(0)F(0), TT decrease on the ladder

Mainly supported by experiments:

  • The near-constancy of F(0)/TF(0)/T along training

I thank the authors for the detailed response and revisions.

W3 is mostly resolved. While W1 and W2 are not adequately addressed. It seems like the response confirms that real training departs from the theory: the mean eigenvalue moves in the opposite direction for all three objectives, and the shrinking phase is not clearly observed. The explanation relies on two corrections that push in opposite directions and whose magnitudes are not predicted. This explains the results after the fact but does not test the theory against the experiments.

Despite the revisions, the paper remains not easy to follow overall, a good paper should have a clear takeaway message in each section.

I therefore maintain my original score.

Reviewer_nfiXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the feature learning dynamics for SIGReg, a regularizer used in the LeJEPA self-supervised learning method. The authors use BHEP statistics as a tractable proxy for SIGReg and derive closed-form ODEs governing the macroscopic feature learning dynamics under this proxy.

Critical strengths

None

Critical weaknesses
  1. The paper is poorly written and does not introduce the problem formulation clearly in Section 3. Many important quantities, such as the input data, loss function, and encoder model, are not properly defined. Moreover, many notations are either undefined or introduced only after they are used. This makes the paper very difficult to follow and its main contributions difficult to evaluate. The quality of presentation is clearly below the ICLR standard.

  2. The paper does not provide rigorous theoretical guarantees for its main claims about feature learning dynamics. Only Theorem 1 provides a closed-form ODE, and the analyses in Sections 4.3 and 4.4 largely rely on approximations and idealized settings, such as assuming some eigenvalues are saturated while others are near zero.

Questions for authors
  • In Definition 1, what is the Epps-Pulley test? Do you mean the statistic EP itself? Also, BHEP and MMD-Gauss are mentioned, but they have not been defined yet. Which weight function w(t)w(t) in Definition 1 is used in the paper?

  • In line 148, what is the SIGReg objective? It has not been formally introduced yet.

  • In Definition 2, what is the exact definition of the MMD-Gauss loss? Definition 2 only gives an approximation (≈\approx), and the exact form is only referred to prior work. It would be helpful to write the exact form and briefly explain its derivation in the main text, so that the paper is self-contained. Also, Definition 1 uses xjx_j, while Definition 2 uses ziz_i and zjz_j. Are they the same as xix_i and xjx_j in Definition 1?

  • In Eq. (5), are the two terms equal (==) or only approximately equal (≈\approx)?

  • The assumptions in lines 206–212 should be formally stated as assumptions, not just described in words.

  • Sections 3.1 and 3.2 are critical parts for readers to understand the problem setting and definitions, but the paper does not clearly introduce the problem formulation and provides insufficient explanations of many technical details. This makes the mathematics in the paper very difficult to follow unless readers are already familiar with all the referenced papers.

  • In Section 4.1, what are XX and ZZ, and what is the relationship between them? How are σk\sigma_k and sks_k defined exactly? They are only described in words, and their exact mathematical definitions are not given in the main text.

  • In line 244, the paper uses the loss function L\mathcal{L}, but never clearly defines it in the main paper.

  • In Theorem 1, α\alpha and β\beta are not defined yet. They are only defined in Definition 5 after the theorem. Similarly, uku_k is defined in the bullet points after Definition 5.

  • The analyses in Sections 4.3.1 and 4.3.2 are all based on approximations, and there is no rigorous theoretical analysis of the actual learning dynamics as claimed in the paper.

  • The paper keeps referring to the appendix, but the main paper is not clearly written enough. The main paper should be sufficiently self-contained so that the readers can follow the main message.

2 comments

We thank the reviewer for the careful reading. We agree with the presentation issues raised, and we have revised the text as summarized in the official comment above.

Weakness

1. Unclear presentation

We agree. The input data, the encoder, and the loss are now defined in a Setup paragraph (Section 4.1), and the symbols are defined before they are used. The official comment lists the changes.

2. Missing theoretical guarantee

We agree that the guarantees are limited, and we state which are proved and which are not.

Proved:

  • Theorem 1, under the assumptions of Section 4.1.
  • For BHEP at any state, the target decreases whenever a feature grows (Theorem 2). This drives both phases: the target starts at order dd, which gives the explosion, and falls as features emerge, which gives the shrinking.
  • On the ladder, the closed form of T(m)T(m) and the bounds of Eq. 14 for every d≥3d\ge3 (Appendix F.2).
  • For MMD-Gaussimq\text{MMD-Gauss}_{\text{imq}}, the decrease of F(0)F(0) and TT along the ladder as d→∞d\to\infty (Appendix Q).

Observed:

  • The two phases in the toy runs (Figure 3), and the decrease of the engine and the target in the real runs (Figure 4).

Not proved:

  • That training follows the ladder exactly. In the toy setting of Appendix J it does so closely, only faster.
  • Anything outside the assumptions, where Section 5.2 and Appendix K give the direction of each effect.
  • For SIGReg, the decrease on the ladder, which is evaluated numerically.

Question

  • Q1. EP test, BHEP, MMD-Gauss: Yes, it is the Epps-Pulley statistic itself. The weight is w(t)=e−t2/2w(t)=e^{-t^2/2}, now stated in the definition. The forward references to BHEP and MMD-Gauss were removed from Definition 1.

  • Q2. SIGReg objective: SIGReg is now in Definition 2: the average of the Epps-Pulley statistic over random unit directions applied to the projected embeddings. Definition 3 gives the LeJEPA objective in which it is the regularizer.

  • Q3. MMD-Gauss loss: Definition 4 now gives the exact estimator with the Kummer kernel from KerJEPA, and Definition 5 is its inverse-root approximation for d>20d>20, written with an equality sign as a definition. In the definitions of Section 3, z∈Rdz\in\mathbb{R}^d denotes embeddings, and the statistic of Definition 1 is applied to their scalar projections. The symbol uu in Definition 1 is not a specific quantity. It is a placeholder for the scalar sample on which the statistic is defined. The sentences before Definitions 4 to 6 now explain where each kernel comes from.

  • Q4. Equality or approximation: Equal. Definition 4 is the exact MMD-Gauss, and Definition 5 is its inverse-root approximation, written MMD-Gaussimq\text{MMD-Gauss}_{\text{imq}}. Only step (ii) of Eq. 8, between these two, approximates the kernel. Step (iii) of the same chain is an equality for MMD-Gaussimq\text{MMD-Gauss}_{\text{imq}}.

  • Q5. Assumption statement: The assumptions are now a separate list in Section 4.1: saturated invariance, alignment, Gaussian inputs, and large nn and dd.

  • Q6. Section 3.1 and 3.2: Item 1 of the official comment addresses this: Section 3 now defines each object before it is used, and ends with one chain (Eq. 8) that connects SIGReg, MMD-Gauss, and BHEP.

  • Q7. XX, ZZ, σk\sigma_k, sks_k: X∈Rn×pX\in\mathbb{R}^{n\times p} holds the inputs as rows, CX=1nX⊤X=VΣ2V⊤C_X=\frac1nX^\top X=V\Sigma^2V^\top with Σ2=diag(σ12,σ22,… )\Sigma^2=\mathrm{diag}(\sigma_1^2,\sigma_2^2,\dots), and Z=XWZ=XW. Under alignment W=V(≤d)SQ⊤W=V^{(\le d)}SQ^\top with S=diag(s1,…,sd)S=\mathrm{diag}(s_1,\dots,s_d), so CZ=QS2Σ≤d2Q⊤C_Z=QS^2\Sigma^2_{\le d}Q^\top and λk=σk2sk2\lambda_k=\sigma_k^2s_k^2. These are now in Section 4.1.

  • Q8. Loss LL: LL is the BHEP statistic of Definition 6 with βbw2=1/d\beta_{bw}^2=1/d evaluated on ZZ (Setup, Section 4.1). It has the form L=P1−γP2+CL=P_1-\gamma P_2+C, where P1P_1 and P2P_2 are the push and pull terms named at Eq. 4, and under the Gaussian assumption both become products over the eigenvalues (Eq. 10).

  • Q9. Not defined α\alpha and β\beta: Fixed. P1P_1, P2P_2, c1c_1, c2c_2, γ\gamma are defined at the start of Section 4.2, α\alpha, β\beta, TT in the definition that now precedes Theorem 1, and uku_k in the theorem itself.

  • Q10. No rigorous theoretical analysis: Theorem 1 is exact under the stated assumptions. For BHEP, the ladder states, with mm features saturated and the rest dormant, are equilibria of that ODE for the saturated features, computed in closed form, and Eq. 14 gives their size for every d≥3d\ge3. The identification of ladder states with actual trajectories is an approximation, which Appendix J compares against toy runs. Section 4.3 now states the closed forms at the two ends of the ladder (Eq. 13) in place of rounded numbers, and the remaining numbers are their large-dd values. They are not approximations of the dynamics.

  • Q11. Not self-contained in main paper: We have moved the definitions and primary statements above into the main text.

Thank you for the authors' responses. However, I find the two phase learning dynamics still lack a rigorous proof and remain a qualitative explanation. The quality of the writing and presentation is also still below the standard of ICLR. I'll maintain my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Revision

We have uploaded a revised PDF, with changes marked in blue.

The major claims, Theorem 1, the experiments, and the figures are unchanged. The revision makes the main text self-contained, and its new results are the bounds of item 4 and Theorem 2 of item 5.

  1. Section 3.1 now defines the Epps-Pulley statistic (Def. 1), SIGReg (Def. 2), and the LeJEPA objective (Def. 3). Section 3.2 writes the three objectives as one kernel statistic (Eq. 4) with three kernels, the exact MMD-Gauss (Def. 4), its inverse-root approximation (Def. 5), and BHEP (Def. 6), and ends with one chain (Eq. 8) that states which steps are equalities and which one is an approximation.

  2. Section 4.1 opens with a Setup paragraph and an assumption list that define the inputs XX, the linear encoder zi=W⊤xiz_i = W^\top x_i, the embeddings Z=XWZ = XW, the covariances CXC_X and CZC_Z, and σk\sigma_k, sks_k, λk\lambda_k before they are used.

  3. Section 4.2 now writes the loss in terms of the eigenvalues (Eq. 10) and defines the engine α\alpha, the brake β\beta, and the target TT before Theorem 1, with uku_k defined in the theorem.

  4. Section 4.3 adds closed-form bounds on the ladder (Eq. 14, proof in Appendix F.2): two-sided bounds for the target TT, and the orders of the engine α\alpha and the brake β\beta in dd and mm, with explicit constants in the appendix. We added them so that the size of these quantities can be read from the main text.

  5. Section 4.3 also adds Theorem 2 (proof in Appendix F.3) on the states between two rungs of the ladder: the target decreases in every eigenvalue, and while the features in transit are below the target, the target and the engine stay between the two rung values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.