- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper studies feature learning dynamics in LeJEPA, focusing on the SIGReg regularizer. Since SIGReg is difficult to analyze directly, the authors use BHEP as a simpler theoretical proxy, connected to SIGReg through MMD-Gauss. Under simplifying assumptions, they derive an explicit ODE for the embedding covariance eigenvalues and propose an engine-brake mechanism to explain two phases of learning: rapid growth and overshooting of leading features, followed by their shrinking and delayed emergence of other features. The authors investigate how this mechanism extends to SIGReg and support their analysis with experiments on synthetic data and neural networks trained on real datasets.
Critical strengths
Original theoretical analysis.
The paper derives an explicit coupled eigenvalue ODE for BHEP and proposes an original engine-brake mechanism to explain feature-learning dynamics. The authors also establish a mathematical connection between BHEP and SIGReg through MMD-Gauss and provide numerical comparisons and experiments that qualitatively support the proposed mechanism. These results represent a potentially interesting theoretical contribution.
Critical weaknesses
1. Unclear scientific significance and scope of the proposed mechanism.
My main concern is that the paper does not convincingly establish the broader scientific significance of its theoretical contribution. Previous work (e.g., Simon et al., 2023) has already theoretically studied stepwise feature learning and empirically observed nonmonotonic eigenvalue dynamics. The authors acknowledge this literature and propose engine-brake mechanism to explain overshooting, shrinking, and delayed emergence. While this is a potentially interesting and original contribution, it remains unclear what broader understanding or testable predictions follow from the proposed mechanism beyond a qualitative explanation of these previously observed phenomena.
First, the theoretical setting is rather restrictive. The main analysis considers a linear model with Gaussian projected inputs, assumes aligned weights, and neglects the invariance term by assuming it has already saturated. Although the authors discuss these limitations and partially relax some assumptions in the appendix, the analysis remains highly idealized.
Second, while Theorem 1 provides an explicit eigenvalue ODE, the explanation of the two-phase dynamics relies heavily on idealized configurations with partially saturated features. The authors also establish theoretical properties of the equilibrium ladder and provide quantitative comparisons with toy training in Appendix J. Nevertheless, it remains unclear under what conditions this analysis accurately describes general training trajectories, particularly in nonlinear models.
Third, although the authors establish a connection between BHEP and SIGReg through MMD-Gauss and provide theoretical and empirical evidence for similar behavior, the extent to which the proposed mechanism explains practical SIGReg training remains uncertain. Section 5.2 and Appendix K investigate discrepancies between theoretical predictions and practical observations. In particular, the authors derive corrections for the unsaturated invariance term and non-Gaussian embeddings, supported by toy experiments and comparisons with practical training. However, the practical validation is primarily qualitative, and the authors acknowledge that the first-order non-Gaussian correction is not quantitatively reliable in some of their experiments. Therefore, it remains unclear how accurately the proposed mechanism can explain or predict the observed dynamics beyond the simplified theoretical setting.
Overall, I find the proposed analysis potentially interesting, but I am not yet convinced that it provides sufficiently important and generalizable insights about SIGReg or SSL feature learning. A clearer explanation of its scientific significance, predictive value, and limits of applicability would strengthen the contribution.
2. Insufficiently clear presentation of the theoretical setup and mechanism.
Although the paper is generally well organized, I found the main theoretical analysis difficult to follow. In particular, the assumptions underlying the eigenvalue dynamics are introduced rather briefly, without sufficiently explaining their meaning and role in the proposed mechanism. Moreover, Section 4.3 introduces configurations with partially saturated features and uses them to explain the two-phase dynamics, but does not clearly explain how these configurations are constructed or why they should describe actual training trajectories. The distinction between the exact eigenvalue ODE, the equilibrium-ladder analysis, and the resulting qualitative interpretation is not sufficiently clear. Finally, the extension of the mechanism to SIGReg in Section 4.4 is rather compressed, with important details deferred to the appendix. I believe the main text would benefit from a more careful explanation of the theoretical setup, the central mechanism, and the precise scope of the conclusions.
Questions for authors
- Scientific significance. What important new understanding does the proposed engine-brake mechanism provide beyond existing analyses of stepwise feature learning, such as Simon et al. (2023)? In particular, can the authors identify concrete, testable predictions that follow from this mechanism and demonstrate its scientific value beyond a qualitative explanation of previously observed phenomena?
- Scope and applicability. Appendix J provides evidence that the equilibrium-ladder analysis describes toy BHEP training, but under what conditions can this picture be expected to hold for general training trajectories and practical SIGReg models? Furthermore, although Appendix K investigates violations of the theoretical assumptions, some corrections are supported mainly by qualitative evidence. Can the authors clarify which conclusions remain reliable when these assumptions are violated, and whether the corrected theory can quantitatively explain the discrepancies observed in practical training?
2 comments
I thank the authors for their detailed responses and revisions. The theoretical exposition has been improved, and I appreciate the additional results, particularly Theorem 2. Accordingly, I have increased my Presentation score from 1 to 2.
However, my main concerns about scientific significance and applicability to practical SIGReg training remain unresolved. Therefore, I maintain my overall score of 2: Weak rejection.
We thank the reviewer for the careful reading and for the questions on significance and scope.
Weakness
1. Restrictive Theoretical Setting, Insufficient Two-Phase Explanation, Uncertain training explanation: We agree the setting is restrictive and idealized and does not transfer quantitatively to general learning. What we observe in the real runs is the mechanism itself, for BHEP and MMD-Gauss. The engine and the target decrease during training (Figure 4), which in the theory is what produces the delayed stepwise emergence and the overshooting-shrinking behavior. For SIGReg we show eigenvalue trajectories only.
On nonlinear models: the regularizer is a function of the embeddings alone, so the encoder enters only through how its gradient reaches each eigenvalue. We derive this only for a linear encoder, with the misaligned case in Appendix K.4, and we have no theory for nonlinear encoders. For ResNet and ViT encoders our support is experimental (Section 5 and Appendix M).
The ladder describes training when features saturate one at a time. When several features grow together, the engine falls faster than the ladder predicts (Appendix J).
Significance is answered in Question 1, applicability on real runs in Question 2.
2. Insufficiently clear presentation: In the revision:
For the distinction:
Question
1. Scientific Significance: Our analysis follows the framework of Simon et al. (2023): aligned weights and one ODE per eigenvalue. The difference lies in the objective. In their linear model with a simplified Barlow Twins loss the modes separate, so each eigenvalue follows its own curve toward a fixed target, and its emergence time is set by the eigenvalue of that mode alone. The SIGReg regularizer depends on the whole spectrum, so in its BHEP proxy the modes share two global quantities, the engine α and the brake β, and the learning of one feature changes the force on the others.
Two consequences follow from this coupling, and both can be tested:
As the reviewer notes, nonmonotonic eigenvalue dynamics have been observed empirically. Our analysis offers one mechanism for them in the SIGReg setting.
2. Scope and Applicability: The ladder values apply quantitatively only when the invariance term has no effect and κrad≪d2. In our real runs the first fails, and the second holds only approximately for SIGReg and MMD-Gauss. On real runs we cannot correct the deviations from the idealized predictions quantitatively, which is one of the limitations of our paper. The corrections we give are qualitative, and they are in Appendix K. What we trust on real runs is the direction of the effects: sequential emergence, the band of feature levels, and the end point λˉ≈1 of SIGReg in the ResNet runs, where the invariance loss is low. With the MLP encoder, where it is the highest, the mean ends at 0.85 (Table 1). What we do not trust is the overshoot magnitude and the band width. Each feature stops where the force equals its invariance drag, so the band lies between the drag-free target T and the level of the most-dragged feature, which is 0 for a feature that does not emerge (Appendix K).