- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper proposes -JEPA, a self-supervised representation learning framework designed to prevent dimensional collapse in backbone features rather than only in the projected representation space. The authors observe that existing explicitly regularized SSL methods can maintain high-rank projected features while the backbone representations used for downstream tasks remain concentrated in a lower-dimensional subspace. Motivated by an analysis of layer balance in two-layer linear networks, they introduce SACReg, a spectral regularizer that combines mean and covariance constraints with a log-determinant penalty to discourage collapsed feature directions. The resulting -JEPA objective applies SACReg to view-averaged backbone and projected representations, using random subspace projections for efficient covariance estimation. Experiments on ImageNet-100, ImageNet-1k, and video benchmarks demonstrate improved representation rank and downstream performance over several explicitly regularized SSL baselines. Additional experiments across six existing SSL objectives and an analysis of augmentation-dependent variation further investigate how backbone regularization affects representation geometry and transferability.
Critical strengths
- The paper identifies an important and well-motivated limitation of existing self-supervised learning methods: preventing collapse after the projection head does not necessarily guarantee informative backbone representations. The proposed connection between layer balance, representation rank, and encoder-side spectral regularization is interesting, and the theoretical analysis provides useful motivation for the design of SACReg.
- The empirical evaluation is fairly comprehensive and convincing. Beyond the large-scale ImageNet-1k and video results, the controlled ImageNet-100 experiments examine SACReg across six existing SSL objectives and show consistent improvements in backbone rank and downstream performance. The additional analysis of augmentation thickness and the comparison with backbone-level SIGReg provide useful evidence that the improvements are not simply explained by stronger invariance or by applying an arbitrary regularizer before the projection head.
Critical weaknesses
-
Although the theoretical motivation is interesting, the connection between the formal non-collapse guarantees and the actual implementation remains somewhat limited. The strongest guarantees rely on idealized linear-network assumptions or a full covariance regularizer without numerical stabilization. In practice, SACReg uses random lower-dimensional projections, a positive stabilization term, and detached memory buffers. These modifications are reasonable for scalability, but the guarantees do not directly establish non-collapse under the implemented objective. Moreover, the nonlinear regularizer is closely related to Gaussian moment matching and existing covariance-based regularization. While the comparison with backbone-level SIGReg is helpful, stronger comparisons against simpler variance/covariance regularizers would better establish the specific contribution of the proposed spectral formulation.
-
I also have some concerns about the evaluation protocol. Appendix C.4 reports the best test accuracy across 13 learning rates for downstream transfer, meaning that hyperparameter selection uses the test set rather than an independent validation split. Although this follows the adopted benchmark protocol, it weakens the interpretation of the reported transfer results as independent held-out evaluations. The video comparisons are also not entirely consistent in their feature readouts. For example, the reported Kinetics-400 result uses CLS features for -JEPA but mean-pooled features for LeVJEPA, and the paper reports a lower result for -JEPA when mean pooling is used. These issues do not invalidate the overall positive results, but they make some of the claimed improvements less conclusive.
Questions for authors
- Appendix C.4 indicates that downstream transfer accuracy is selected using the best test result across 13 learning rates. How much do the reported improvements change when learning rates are selected using a separate validation split and the test set is used only for final evaluation? Since transfer performance is a central contribution, this distinction would materially affect my assessment.
- The Kinetics-400 comparison uses different feature aggregation strategies across methods. At 1,085 epochs, -JEPA reports 45.7% with CLS features, while its mean-pooled result is 43.4%, compared with LeVJEPA's reported 44.6% using mean-pooled features. Please clarify the relative performance under identical feature aggregation and probing protocols. This would help establish whether the improvement comes from the learned representation or the evaluation configuration.
- The theoretical non-collapse guarantees rely on assumptions that differ from the practical implementation using random slices, positive numerical stabilization, and buffered representations. How sensitive are the results to these choices, particularly the slice dimension and buffer size? A clearer empirical connection between the theoretical spectrum bounds and the covariance spectrum obtained during practical training would strengthen the soundness of the proposed formulation.
- The additional backbone-level SIGReg experiment is useful, but it does not fully isolate the importance of the log-determinant penalty. How does SACReg compare against simpler backbone-level variance and covariance regularizers, such as VICReg-style constraints, under matched training settings? This comparison would help clarify whether the main improvement comes from the proposed spectral objective or from directly regularizing the backbone.
4 comments
(W2, Q1) Transfer performance evaluation
We agree that using the test set for hyperparameter selection is a limitation of the adopted VISReg evaluation protocol (Appendix C.4). We initially followed this protocol to ensure comparability with previously published baselines, without requiring a complete re-evaluation of all methods.
To address the reviewer’s concern, we have now conducted an additional evaluation in which hyperparameters are selected exclusively using a validation split. Specifically, we re-evaluated every method in Table 1 for which we performed experiments (Ours, LeJEPA, DINO, iBOT at both model sizes and epoch budgets, and VISReg-B/100). We selected the linear-probe learning rate on the official validation split where available, or otherwise on a held-out 20% subset of the training data. We then retrained the linear head at the selected learning rate using the full training split and evaluated it on the test set.
The results, reported below, show that all reported scores change by at most 0.3 percentage points, and the relative ordering of methods at matched model sizes and training budgets remains unchanged. Thus, the main conclusions drawn from Table 1 are consistent with the hyperparameter selection protocol. We will report these validation-selected results separately from the original benchmark table, since the two evaluations follow different protocols, and clarify this distinction in the camera ready.
| row | validation-selected | paper (best test) |
|---|---|---|
| ours S/100 | 77.4 | 77.6 |
| ours S/400 | 79.1 | 79.3 |
| ours B/100 | 80.8 | 80.9 |
| ours B/400 | 81.8 | 82.0 |
| LeJEPA S/100 | 68.5 | 68.7 |
| LeJEPA S/300 | 71.9 | 71.9 |
| LeJEPA B/100 | 74.1 | 74.2 |
| LeJEPA B/300 | 75.1 | 75.2 |
| DINO S/100 | 75.8 | 75.7 |
| DINO S/300 | 79.1 | 79.4 |
| DINO B/100 | 78.1 | 78.3 |
| DINO B/300 | 79.4 | 79.3 |
| iBOT S/100 | 75.6 | 75.9 |
| iBOT S/300 | 79.3 | 79.5 |
| iBOT B/100 | 80.6 | 80.7 |
| iBOT B/300 | 82.4 | 82.3 |
| VISReg-B/100 | 76.4 | 76.4 |
(W2,Q2). Kinetics-400 probe
We already report Kinetics-400 results in various settings (i) CLS representation (45.7%), mean-pooled patch features (43.4%) and attentive-probe (62.0%) in Section 4.3. For greater clarity, we will include results also directly in the main comparison table in the camera ready.
Our primary evaluation uses CLS representations because these are the features directly regularized by SACReg and retained for downstream tasks. This choice is consistent throughout the paper, including ImageNet-100, ImageNet-1k, downstream transfer and representation analysis in Figure 2. We therefore consider CLS as the most appropriate representation for evaluating the proposed method.
We would also like to note that LeVJEPA uses attentive-probing for ImageNet-1k and SSv2 datasets, but switches to a linear probe with mean-pooled features for Kinetics-400. As their pretrained checkpoints are not publicly available, we cannot evaluate their representations under a unified evaluation protocol. We therefore match the complexity of the downstream probe (linear) and transparently report both representation choices, while acknowledging that comparison does not use identical representations. We included this discussion in Appendix C.2.
(Q3) Slice and buffer sensitivity
We analyze the slice and buffer sensitivity in the ImageNet-100 setting and present the table below. The table shows that the results are relatively insensitive to the slice width and buffer size within a reasonable range. We find that slice widths of 64-128 and buffer lengths of 1-3 show similar linear and kNN accuracies. Additionally, feature rank remains high across all tested configurations except the small 32 dimensional slice setting. Performance degrades mainly at the extreme settings: namely a 32 dimensional slice without a buffer, the full width slice the longest buffer. We included this experiment in Appendix B.2.
| slice width | buffer | linear | kNN | RankMe/d |
|---|---|---|---|---|
| 32 | 0 | 67.6 | 61.0 | .57 |
| 64 | 1 | 71.2 | 65.4 | .72 |
| 128 (paper) | 3 | 71.5 | 66.2 | .78 |
| 256 | 7 | 67.2 | 59.7 | .79 |
| 128 | 1 | 71.3 | 66.2 | .80 |
| 128 | 2 | 71.9 | 66.5 | .80 |
| 128 | 4 | 69.2 | 63.8 | .74 |
References
[1] Nam, Y., Lee, S.H., Domine, C.C., Park, Y., London, C., Choi, W., Goring, N. and Lee, S., 2025. Position: Solve layerwise linear models first to understand neural dynamical phenomena (neural collapse, emergence, lazy/rich regime, and grokking). arXiv preprint arXiv:2502.21009.
[2] Anguita, N., Locatello, F., Saxe, A.M., Mondelli, M., Mancini, F., Lippl, S. and Domine, C., 2026. A theory of how pretraining shapes inductive bias in fine-tuning. arXiv preprint arXiv:2602.20062.
[3] Saxe, A.M., McClelland, J.L. and Ganguli, S., 2019. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23), pp.11537-11546.
[4] Ghosh, A., Kwon, S.M., Wang, R., Ravishankar, S. and Qu, Q., 2025. Learning dynamics of deep linear networks beyond the edge of stability. arXiv preprint arXiv:2502.20531.
[5] Kunin, D., Raventós, A., Dominé, C., Chen, F., Klindt, D., Saxe, A. and Ganguli, S., 2024. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. Advances in Neural Information Processing Systems, 37, pp.81157-81203.
Thank you for the detailed response and additional experiments.
The validation-based evaluation and new ablations address most of my concerns, and the covariance spectrum and sensitivity analyses are also very helpful. For my remaining reservations center on the gap between the theoretical guarantees and practical implementation, as well as the mismatched evaluation protocols on Kinetics-400. Overall, the rebuttal is convincing, and I will maintain my current rating.
(W1, Q3) Theory and implementation
The two-layer linear analysis provides precise results that motivate both the placement and the form of SACReg in nonlinear networks. This follows an established and fruitful approach of identifying mechanisms in simplified models and then testing whether they extend to practical nonlinear settings [1,2,3,4,5]. Beyond this motivation, we discuss the properties of a full covariance regularization in nonlinear networks in Section 3.2. Specifically, full covariance regularization creates an anti-collapse barrier (for ε=0): the loss goes to infinity as the smallest eigenvalue approaches zero. Additionally, we provide an analysis in Appendix B.1 that makes the connection between full covariance and sliced regularization we follow in practice. Specifically, our analysis shows random slicing preserves the same isotropy in expectation. To further support the SACReg’s effect on the covariance spectrum empirically, we report the table below (measured using ImageNet-100 checkpoints, included as Table 6 in the revised manuscript):
The table shows that using SACReg significantly improves the covariance spectra while the RankMe results in Figure 2 (main paper) demonstrate increased effective rank after applying SACReg. Together, these theoretical insights and empirical observations support the effectiveness of SACReg in mitigating representational collapse under the implemented objective. Please refer to the answer to reviewer 9jyu (W2,Q2) for a more detailed answer.
(W1) Relation to Gaussian moment matching
We thank the reviewer for highlighting the connection. While SACReg can be interpreted as the KL divergence between a surrogate gaussian with the same first two moments and N(0, I), we do not assume or enforce Gaussianity of the representation distribution: higher-order moments are not constrained. SACReg only constrains its mean and covariance. We deliberately compare against backbone level SIGReg in Appendix D to investigate this distinction. Enforcing Gaussianity via SIGReg improves distributional diagnostics but reduces downstream performance. We further clarify the relationship between SACReg and existing spectral regularizers (Reviewer q2HB (W1), Appendix [Related Works])], and Gaussian moment matching in the revised Appendix D.1, making the connections explicit.
(W1,Q4) Comparison against backbone regularizers
To explore this further, we evaluate SACReg and VICReg at both the backbone and projector levels. Specifically, we consider three backbone configurations (no regularizer, VICReg or SACReg) combined with two projector-level regularizers (SACReg or VICReg) yielding six configurations in total. Four combinations were already evaluated in the original paper and we additionally trained the two missing variants: (i) VICReg at both the backbone and projector levels and (ii) VICReg at the backbone with SACReg after the projector.
The table above shows the experiments (same settings as the ones in Figure 2) and reports Linear accuracy / KNN accuracy and Rankme/d respectively for each setting. Results highlight that, SACReg backbone term performs consistently better compared to VICReg at backbone under every setting. Therefore, these results support both the form and the placement of the regularizer. We included this experiment in the revised manuscript in Appendix D.2 (Table 8).