acceptodds
Under review as a conference paper at ICLR 2027

-JEPA: Self-Supervised Learning with Spectral Anti-Collapse Regularization

Abstract

Joint-embedding self-supervised learning typically combines an invariance objective across augmented views with additional mechanisms to prevent representational collapse. These objectives are often applied after a projection head, while downstream tasks use the backbone representation before the projector. We find that this mismatch does not necessarily prevent dimensional collapse in the backbone, which can retain low effective rank and potentially limit downstream transfer. To address this, we introduce SACReg, a spectral anti-collapse regularizer motivated by an analysis of -balance, which captures the relative scale of weight matrices across layers. In a two-layer linear network, we show that (i) -balance prevents collapse, and (ii) our regularizer applied to the backbone induces -balance. In the nonlinear case, this regularizer leads to anti-collapse as well and, in realistic architectures on ImageNet100, it empirically increases the representations' ranks. We apply SACReg to JEPA and propose -JEPA, which improves over LeJEPA and VISReg on ImageNet-1k classification and in average linear-probe transfer performance across eight downstream image datasets. On video self-supervised learning, -JEPA improves over LeVJEPA and V-JEPA 2 on the Something-Something-v2 and Kinetics-400 benchmarks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, joint-embedding learning, spectral regularization, anti-collapse regularization, JEPA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eWr8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes λ\lambda-JEPA, a self-supervised representation learning framework designed to prevent dimensional collapse in backbone features rather than only in the projected representation space. The authors observe that existing explicitly regularized SSL methods can maintain high-rank projected features while the backbone representations used for downstream tasks remain concentrated in a lower-dimensional subspace. Motivated by an analysis of layer balance in two-layer linear networks, they introduce SACReg, a spectral regularizer that combines mean and covariance constraints with a log-determinant penalty to discourage collapsed feature directions. The resulting λ\lambda-JEPA objective applies SACReg to view-averaged backbone and projected representations, using random subspace projections for efficient covariance estimation. Experiments on ImageNet-100, ImageNet-1k, and video benchmarks demonstrate improved representation rank and downstream performance over several explicitly regularized SSL baselines. Additional experiments across six existing SSL objectives and an analysis of augmentation-dependent variation further investigate how backbone regularization affects representation geometry and transferability.
Critical strengths
  • The paper identifies an important and well-motivated limitation of existing self-supervised learning methods: preventing collapse after the projection head does not necessarily guarantee informative backbone representations. The proposed connection between layer balance, representation rank, and encoder-side spectral regularization is interesting, and the theoretical analysis provides useful motivation for the design of SACReg.
  • The empirical evaluation is fairly comprehensive and convincing. Beyond the large-scale ImageNet-1k and video results, the controlled ImageNet-100 experiments examine SACReg across six existing SSL objectives and show consistent improvements in backbone rank and downstream performance. The additional analysis of augmentation thickness and the comparison with backbone-level SIGReg provide useful evidence that the improvements are not simply explained by stronger invariance or by applying an arbitrary regularizer before the projection head.
Critical weaknesses
  • Although the theoretical motivation is interesting, the connection between the formal non-collapse guarantees and the actual implementation remains somewhat limited. The strongest guarantees rely on idealized linear-network assumptions or a full covariance regularizer without numerical stabilization. In practice, SACReg uses random lower-dimensional projections, a positive stabilization term, and detached memory buffers. These modifications are reasonable for scalability, but the guarantees do not directly establish non-collapse under the implemented objective. Moreover, the nonlinear regularizer is closely related to Gaussian moment matching and existing covariance-based regularization. While the comparison with backbone-level SIGReg is helpful, stronger comparisons against simpler variance/covariance regularizers would better establish the specific contribution of the proposed spectral formulation.

  • I also have some concerns about the evaluation protocol. Appendix C.4 reports the best test accuracy across 13 learning rates for downstream transfer, meaning that hyperparameter selection uses the test set rather than an independent validation split. Although this follows the adopted benchmark protocol, it weakens the interpretation of the reported transfer results as independent held-out evaluations. The video comparisons are also not entirely consistent in their feature readouts. For example, the reported Kinetics-400 result uses CLS features for λ\lambda-JEPA but mean-pooled features for LeVJEPA, and the paper reports a lower result for λ\lambda-JEPA when mean pooling is used. These issues do not invalidate the overall positive results, but they make some of the claimed improvements less conclusive.

Questions for authors
  • Appendix C.4 indicates that downstream transfer accuracy is selected using the best test result across 13 learning rates. How much do the reported improvements change when learning rates are selected using a separate validation split and the test set is used only for final evaluation? Since transfer performance is a central contribution, this distinction would materially affect my assessment.
  • The Kinetics-400 comparison uses different feature aggregation strategies across methods. At 1,085 epochs, λ\lambda-JEPA reports 45.7% with CLS features, while its mean-pooled result is 43.4%, compared with LeVJEPA's reported 44.6% using mean-pooled features. Please clarify the relative performance under identical feature aggregation and probing protocols. This would help establish whether the improvement comes from the learned representation or the evaluation configuration.
  • The theoretical non-collapse guarantees rely on assumptions that differ from the practical implementation using random slices, positive numerical stabilization, and buffered representations. How sensitive are the results to these choices, particularly the slice dimension and buffer size? A clearer empirical connection between the theoretical spectrum bounds and the covariance spectrum obtained during practical training would strengthen the soundness of the proposed formulation.
  • The additional backbone-level SIGReg experiment is useful, but it does not fully isolate the importance of the log-determinant penalty. How does SACReg compare against simpler backbone-level variance and covariance regularizers, such as VICReg-style constraints, under matched training settings? This comparison would help clarify whether the main improvement comes from the proposed spectral objective or from directly regularizing the backbone.
4 comments

(W1, Q3) Theory and implementation

The two-layer linear analysis provides precise results that motivate both the placement and the form of SACReg in nonlinear networks. This follows an established and fruitful approach of identifying mechanisms in simplified models and then testing whether they extend to practical nonlinear settings [1,2,3,4,5]. Beyond this motivation, we discuss the properties of a full covariance regularization in nonlinear networks in Section 3.2. Specifically, full covariance regularization creates an anti-collapse barrier (for ε=0\varepsilon = 0): the loss goes to infinity as the smallest eigenvalue approaches zero. Additionally, we provide an analysis in Appendix B.1 that makes the connection between full covariance and sliced regularization we follow in practice. Specifically, our analysis shows random slicing preserves the same isotropy in expectation. To further support the SACReg’s effect on the covariance spectrum empirically, we report the table below (measured using ImageNet-100 checkpoints, included as Table 6 in the revised manuscript):

backbone covariance1 %10 %25 %50 %75 %90 %99 %smallest
with backbone SACReg.70.31.18.057.0123.7e-31.1e-38.7e-4
without.19.0287.0e-31.1e-32.4e-48.7e-53.5e-53.0e-5

The table shows that using SACReg significantly improves the covariance spectra while the RankMe results in Figure 2 (main paper) demonstrate increased effective rank after applying SACReg. Together, these theoretical insights and empirical observations support the effectiveness of SACReg in mitigating representational collapse under the implemented objective. Please refer to the answer to reviewer 9jyu (W2,Q2) for a more detailed answer.


(W1) Relation to Gaussian moment matching

We thank the reviewer for highlighting the connection. While SACReg can be interpreted as the KL divergence between a surrogate gaussian with the same first two moments and N(0, I), we do not assume or enforce Gaussianity of the representation distribution: higher-order moments are not constrained. SACReg only constrains its mean and covariance. We deliberately compare against backbone level SIGReg in Appendix D to investigate this distinction. Enforcing Gaussianity via SIGReg improves distributional diagnostics but reduces downstream performance. We further clarify the relationship between SACReg and existing spectral regularizers (Reviewer q2HB (W1), Appendix [Related Works])], and Gaussian moment matching in the revised Appendix D.1, making the connections explicit.


(W1,Q4) Comparison against backbone regularizers

To explore this further, we evaluate SACReg and VICReg at both the backbone and projector levels. Specifically, we consider three backbone configurations (no regularizer, VICReg or SACReg) combined with two projector-level regularizers (SACReg or VICReg) yielding six configurations in total. Four combinations were already evaluated in the original paper and we additionally trained the two missing variants: (i) VICReg at both the backbone and projector levels and (ii) VICReg at the backbone with SACReg after the projector.

backbone termSACReg after projectorVICReg after projector
Linear / KNN / Rankme/dLinear / KNN / Rankme/d
none70.5 / 58.0 / .3665.1 / 54.9 / .38
VICReg70.8 / 65.7 / .7363.8 / 58.8 / .95
SACReg71.5 / 66.2 / .7865.5 / 59.5 / .92

The table above shows the experiments (same settings as the ones in Figure 2) and reports Linear accuracy / KNN accuracy and Rankme/d respectively for each setting. Results highlight that, SACReg backbone term performs consistently better compared to VICReg at backbone under every setting. Therefore, these results support both the form and the placement of the regularizer. We included this experiment in the revised manuscript in Appendix D.2 (Table 8).

(W2, Q1) Transfer performance evaluation

We agree that using the test set for hyperparameter selection is a limitation of the adopted VISReg evaluation protocol (Appendix C.4). We initially followed this protocol to ensure comparability with previously published baselines, without requiring a complete re-evaluation of all methods.

To address the reviewer’s concern, we have now conducted an additional evaluation in which hyperparameters are selected exclusively using a validation split. Specifically, we re-evaluated every method in Table 1 for which we performed experiments (Ours, LeJEPA, DINO, iBOT at both model sizes and epoch budgets, and VISReg-B/100). We selected the linear-probe learning rate on the official validation split where available, or otherwise on a held-out 20% subset of the training data. We then retrained the linear head at the selected learning rate using the full training split and evaluated it on the test set.

The results, reported below, show that all reported scores change by at most 0.3 percentage points, and the relative ordering of methods at matched model sizes and training budgets remains unchanged. Thus, the main conclusions drawn from Table 1 are consistent with the hyperparameter selection protocol. We will report these validation-selected results separately from the original benchmark table, since the two evaluations follow different protocols, and clarify this distinction in the camera ready.

rowvalidation-selectedpaper (best test)
ours S/10077.477.6
ours S/40079.179.3
ours B/10080.880.9
ours B/40081.882.0
LeJEPA S/10068.568.7
LeJEPA S/30071.971.9
LeJEPA B/10074.174.2
LeJEPA B/30075.175.2
DINO S/10075.875.7
DINO S/30079.179.4
DINO B/10078.178.3
DINO B/30079.479.3
iBOT S/10075.675.9
iBOT S/30079.379.5
iBOT B/10080.680.7
iBOT B/30082.482.3
VISReg-B/10076.476.4

(W2,Q2). Kinetics-400 probe

We already report Kinetics-400 results in various settings (i) CLS representation (45.7%), mean-pooled patch features (43.4%) and attentive-probe (62.0%) in Section 4.3. For greater clarity, we will include results also directly in the main comparison table in the camera ready.

Our primary evaluation uses CLS representations because these are the features directly regularized by SACReg and retained for downstream tasks. This choice is consistent throughout the paper, including ImageNet-100, ImageNet-1k, downstream transfer and representation analysis in Figure 2. We therefore consider CLS as the most appropriate representation for evaluating the proposed method.

We would also like to note that LeVJEPA uses attentive-probing for ImageNet-1k and SSv2 datasets, but switches to a linear probe with mean-pooled features for Kinetics-400. As their pretrained checkpoints are not publicly available, we cannot evaluate their representations under a unified evaluation protocol. We therefore match the complexity of the downstream probe (linear) and transparently report both representation choices, while acknowledging that comparison does not use identical representations. We included this discussion in Appendix C.2.


(Q3) Slice and buffer sensitivity

We analyze the slice and buffer sensitivity in the ImageNet-100 setting and present the table below. The table shows that the results are relatively insensitive to the slice width and buffer size within a reasonable range. We find that slice widths of 64-128 and buffer lengths of 1-3 show similar linear and kNN accuracies. Additionally, feature rank remains high across all tested configurations except the small 32 dimensional slice setting. Performance degrades mainly at the extreme settings: namely a 32 dimensional slice without a buffer, the full width slice the longest buffer. We included this experiment in Appendix B.2.

slice widthbufferlinearkNNRankMe/d
32067.661.0.57
64171.265.4.72
128 (paper)371.566.2.78
256767.259.7.79
128171.366.2.80
128271.966.5.80
128469.263.8.74

References

[1] Nam, Y., Lee, S.H., Domine, C.C., Park, Y., London, C., Choi, W., Goring, N. and Lee, S., 2025. Position: Solve layerwise linear models first to understand neural dynamical phenomena (neural collapse, emergence, lazy/rich regime, and grokking). arXiv preprint arXiv:2502.21009.

[2] Anguita, N., Locatello, F., Saxe, A.M., Mondelli, M., Mancini, F., Lippl, S. and Domine, C., 2026. A theory of how pretraining shapes inductive bias in fine-tuning. arXiv preprint arXiv:2602.20062.

[3] Saxe, A.M., McClelland, J.L. and Ganguli, S., 2019. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23), pp.11537-11546.

[4] Ghosh, A., Kwon, S.M., Wang, R., Ravishankar, S. and Qu, Q., 2025. Learning dynamics of deep linear networks beyond the edge of stability. arXiv preprint arXiv:2502.20531.

[5] Kunin, D., Raventós, A., Dominé, C., Chen, F., Klindt, D., Saxe, A. and Ganguli, S., 2024. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. Advances in Neural Information Processing Systems, 37, pp.81157-81203.

Thank you for the detailed response and additional experiments.

The validation-based evaluation and new ablations address most of my concerns, and the covariance spectrum and sensitivity analyses are also very helpful. For my remaining reservations center on the gap between the theoretical guarantees and practical implementation, as well as the mismatched evaluation protocols on Kinetics-400. Overall, the rebuttal is convincing, and I will maintain my current rating.

Reviewer_q2HBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a mismatch in joint-embedding self-supervised learning. The authors analyze two-linear networks and show that negative layer imbalance provides a non-collapse guarantee. They propose SACReg to prevent dimensional collapse in the backbone. Experimental results show that the proposed method outperforms existing methods.

Critical strengths
  1. This paper presents an interesting perspective in self-supervised learning, which explores the collapse in backbone.

  2. This paper is easy to follow with clear writing.

Critical weaknesses
  1. The relationship to existing approaches for preventing dimensional collapse in self-supervised learning is insufficiently discussed, such as CorInfoMax [1] and LogDet [2].

  2. The downstream evaluation focuses on classification tasks. Results on other tasks would provide a broader assessment of the transferability of the learned representations, such as detection and segmentation.

[1] Self-Supervised Learning with an Information Maximization Criterion, NeurIPS 2022.

[2] Geometric View of Soft Decorrelation in Self-Supervised Learning, KDD 2024.

Questions for authors
  1. Why use features after projection to compute SACReg?

  2. What additional training time and memory does SACReg require?

2 comments

(W1) Related work

We thank the reviewer for highlighting these relevant works. CorInfoMax and the LogDet method of Zhang et al. both apply LogDet-based spectral regularization to prevent dimensional collapse. We acknowledge this connection and expanded the related work discussion in Appendix [Related Works] to clarify the relationship between SACReg and existing methods.

The main difference between our approach and other methods, as highlighted in the related work discussion, is where the regularization is applied. Importantly, CorInfoMax and LogDet, like VICReg, LeJEPA and VISReg, apply their anti-collapse regularization after the projector, as is standard in explicitly regularized SSL. Our central observation is that preventing collapse in the projected space does not necessarily prevent dimensional collapse in the backbone representations used downstream. This motivates applying SACReg directly to the backbone, guided by our layer-balance analysis.

To further substantiate this observation, we extended our ImageNet-100 experiments (ViT-S/16, 100 epochs) to include CorInfoMax and LogDet under matched settings. Both methods exhibit substantial backbone dimensional collapse. Adding backbone SACReg significantly increases their effective rank and improves kNN accuracy by 5-6 points and linear accuracy by 1-2 points. We will update the Figure 2, including LogDet and CorInfoMax, for the camera ready.

methodlinearkNNRankMe/d
LogDet (Zhang et al.)56.044.9.23
+ backbone SACReg57.150.7.71
CorInfoMax59.751.7.38
+ backbone SACReg61.757.2.85

(W2) Detection and segmentation

We agree that detection and segmentation would provide a broader assessment of transferability. However, these tasks extend substantially beyond our current focus on preserving the rank of global backbone representations and evaluating their classification performance. Dense prediction requires spatially local features, which would require a non-trivial extension of the regularization framework rather than simply an additional evaluation (we observed instability in our preliminary experiments). As noted in our conclusion, developing such a formulation is reserved for future work.


(Q1) Projected features

The two SACReg terms serve complementary purposes. Projector-level SACReg discourages the trivial constant map solution by the invariance objective, fulfilling the same role as anti-collapse terms in other explicitly regularized methods, such as VICReg, VISReg, LeJEPA. Without this term, the projector could collapse even if the backbone representations remained non-collapsed. Backbone-level SACReg addresses our key observation that anti-collapse objectives applied after the projector can still leave the backbone representations dimensionally collapsed. This motivates adding a regularizer directly to the backbone to preserve the rank of the representations retained for downstream tasks.


(Q2) Computational overhead

SACReg introduces only modest additional computational and memory overhead compared with the overall training procedure. In particular, the additional computation introduced by SACReg comes from random orthonormal projections O(dd′2)O(dd'^2), covariance estimation O(Bdd′+Bd′2)O(Bdd' + Bd'^2), and log-determinant computation O(d′3)O(d'^3), where d′d' is the slice width, dd the feature dimension, and BB the effective batch size. The detached ring buffer requires O(qBd)O(qBd) additional memory (typically q=3q=3) without retaining computation graphs. Thus, memory overhead is limited to storing detached features, while additional computation comes primarily from the projection and covariance operations. We included this discussion in Appendix C.2.

Thanks for the authors’ responses. I think the authors have solved most of my concerns, so I will maintain my rating.

Reviewer_9jyuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses dimensional collapse in backbone representations of joint-embedding self-supervised learning, even when projected representations remain non-collapsed. The authors propose SACReg, a spectral regularizer motivated by theoretical analysis of layer balance in two-layer linear networks, and introduce λ-JEPA by applying it to backbone and projected representations. Experiments demonstrate improved representation rank and downstream transfer across multiple SSL methods, including image and video benchmarks.

Critical strengths
  • The paper identifies an important discrepancy between projected and backbone representation collapse, providing a principled motivation for encoder-side regularization. The connection to layer balance offers an interesting theoretical perspective on representation rank.

  • The empirical evaluation is comprehensive, demonstrating consistent improvements across six SSL methods and competitive performance against established baselines on large-scale image and video benchmarks. The controlled experiments further support the effectiveness of backbone spectral regularization.

Critical weaknesses
  • The proposed covariance regularizer closely resembles existing LogDet-based anti-collapse objectives and covariance whitening methods. In particular, prior work such as Geometric View of Soft Decorrelation in Self-Supervised Learning (KDD 2024) has already explored LogDet regularization for preventing dimensional collapse. Although the connection to layer balance is interesting, it remains unclear whether SACReg offers a fundamentally new regularization principle or primarily a theoretically motivated application of existing spectral penalties to backbone representations.

  • The strongest anti-collapse guarantees are established for two-layer linear networks under restrictive rank and covariance assumptions. The nonlinear extension instead characterizes the preferred covariance of the regularizer, without establishing an analogous guarantee for trained deep networks. Furthermore, practical training relies on random covariance slices, numerical stabilization, and detached feature buffers, whose finite-training effects are not covered by the guarantees. This limits the theoretical justification of the actual implementation.

  • Some comparisons involve different training configurations and evaluation procedures. In particular, the Kinetics-400 comparison uses CLS representations for λ-JEPA but mean-pooled representations for LeVJEPA. Under mean pooling, λ-JEPA achieves 43.4 compared to LeVJEPA's 44.6, reversing the reported advantage. Similarly, matching pretraining epochs does not necessarily ensure comparable computational budgets across different augmentation and view configurations.

  • While Appendix D compares backbone SACReg against an additional SIGReg term, comparisons against simpler spectral regularizers or covariance whitening objectives remain limited. Moreover, the large-scale results do not sufficiently isolate the contributions of backbone versus projected-space regularization. The sensitivity to random slice dimensions, buffer sizes, and regularization strengths also deserves further investigation.

Questions for authors
  • How does SACReg fundamentally differ from existing LogDet-based covariance regularizers, particularly the approach of Zhang et al. (KDD 2024)? Can the authors provide a direct empirical comparison under matched conditions?

  • To what extent do the theoretical anti-collapse properties survive random slicing, finite minibatches, and numerical stabilization? Is there any guarantee or empirical evidence concerning the minimum eigenvalues of the full backbone covariance?

  • Can the authors provide strictly matched evaluation protocols for video SSL, particularly on Kinetics-400, and clarify the additional computational costs associated with multi-view training and covariance regularization?

  • What are the individual contributions of backbone SACReg, projected-space SACReg, and the invariance objective to the reported large-scale improvements? How sensitive are these gains to slice width and feature-buffer size?

7 comments

(W1, Q1) Novelty

We thank the reviewer for highlighting the connection to Zhang et al. (KDD 2024) and other anti-collapse methods. While the spectral penalty itself is not new, our contribution lies in identifying a collapse problem in self-supervised learning (SSL) and using linear network theory to establish where anti-collapse regularization should be applied.

Our key observations are that existing anti-collapse objectives, including those of Zhang et al, CorInfoMax, LeJEPA, VISReg and VICReg, are applied after the nonlinear projector, and this does not necessarily prevent dimensional collapse in the backbone representations used for downstream tasks. This motivates our approach of addressing collapse at the backbone level. Our theoretical analysis further connects layer balance and representation rank, providing a principled motivation for both the regularizer's placement and the choice of spectral penalty.

In the revised manuscript, we clarify our contribution and discuss the relationship to existing LogDet-based and covariance-whitening methods (currently, Appendix [Related Works]). To further substantiate our claims, we also include a direct empirical comparison with existing spectral regularizers under matched experimental conditions. On ImageNet-100 (ViT-S/16, 100 epochs) we trained CorInfoMax and LogDet method of Zhang et al. with their own objectives and then added backbone SACReg to each (following Figure 2 in the main manuscript). These experiments show that the backbone collapse is visible for both methods. Regularizing the representations additionally with SACReg restores the rank and improves kNN accuracy by 5-6 points and linear accuracy by 1-2 points. This supports our position that the correct placement of the penalty is an important contribution. We will update the Figure 2, including LogDet and CorInfoMax for the camera ready.

methodlinearkNNRankMe/d
LogDet (Zhang et al.)56.044.9.23
+ backbone SACReg57.150.7.71
CorInfoMax59.751.7.38
+ backbone SACReg61.757.2.85

(W2, Q2) Theory-practice gap

We thank the reviewer for asking us to clarify which theoretical properties carry over to the practical implementation.

  1. From linear to two-layer nonlinear networks (W2). The two-layer linear analysis provides precise results that motivate both the placement and the form of SACReg in nonlinear networks. This follows an established and fruitful approach of identifying mechanisms in simplified models and then testing whether they extend to practical nonlinear settings [1,2,3,4,5]. Beyond this motivation, we discuss the properties of a full covariance regularization in nonlinear networks in Section 3.2. Specifically, full covariance regularization creates an anti-collapse barrier (for ε=0\varepsilon = 0): the loss goes to infinity as the smallest eigenvalue approaches zero.
  2. Random slicing (W2, Q2). We agree that using random slicing and buffers for practical reasons could change the behavior in practice. For this reason we provide an additional analysis in Appendix B.1, proving that the sliced regularizer shares the same global minima as the full regularizer in expectation. This establishes that slicing preserves the preferred non-collapsed covariance in expectation.
  3. Numerical stabilization (Q2). Remark A.16 explicitly distinguishes the stabilized implementation from the strict barrier result, while Theorem A.13 shows an additional result when optimal covariance remains full rank.
  4. Empirical evidence (Q2). Figures 2 and Appendix A.2.3 demonstrate that backbone SACReg increases effective rank across SSL methods and nonlinear ReLU networks, respectively. In particular, the RankMe/d measurements in Figure 2 consistently improve with backbone SACReg under the controlled ImageNet-100 setting. These findings support the conclusion that SACReg mitigates dimensional collapse. To further support our results with empirical evidence, we present the eigenvalue spectrum of the backbone covariance, contrasting the cases: with and without backbone regularization. This experiment shows that SACReg significantly improves the eigenvalue spectrum making directions more available. We revise the manuscript with this new experiment (Table 6 in Appendix).
backbone covariance1 %10 %25 %50 %75 %90 %99 %smallest
with backbone SACReg.70.31.18.057.0123.7e-31.1e-38.7e-4
without.19.0287.0e-31.1e-32.4e-48.7e-53.5e-53.0e-5

(W3, Q3) Kinetics-400 comparison

We already report Kinetics-400 results in various settings (i) CLS representation (45.7%), (ii) mean-pooled patch features (43.4%) and (iii) attentive-probe (62.0%) in Section 4.3. For greater clarity, we will include results also directly in the main comparison table in the camera ready.

Our primary evaluation uses CLS representations because these are the features directly regularized by SACReg and retained for downstream tasks. This choice is consistent throughout the paper, including ImageNet-100, ImageNet-1k, downstream transfer and representation analysis in Figure 2. We therefore consider CLS as the most appropriate representation for evaluating the proposed method.

We would also like to note that LeVJEPA uses attentive-probing for ImageNet-1k and SSv2 datasets, but switches to a linear probe with mean-pooled features for Kinetics-400. As their pretrained checkpoints are not publicly available, we cannot evaluate their representations under a unified evaluation protocol. We therefore match the complexity of the downstream probe (linear) and transparently report both representation choices, while acknowledging that comparison does not use identical representations. We included this discussion in Appendix C.2.


(W4) Comparison against backbone regularizers

To explore this further, we evaluate SACReg and VICReg at both the backbone and projector levels. Specifically, we consider three backbone configurations (no regularizer, VICReg or SACReg) combined with two projector-level regularizers (SACReg or VICReg) yielding six configurations in total. Four combinations were already evaluated in the original paper and we additionally trained the two missing variants: (i) VICReg at both the backbone and projector levels and (ii) VICReg at the backbone with SACReg after the projector.

The table below shows the experiments (same settings as the ones in Figure 2) and reports Linear accuracy / KNN accuracy and RankMe/d respectively for each setting. The results highlight that the SACReg backbone term performs consistently better compared to VICReg at backbone under every setting. We included this experiment in the revised manuscript in Appendix D.2.

backbone termSACReg after projectorVICReg after projector
Linear / KNN / Rankme/dLinear / KNN / Rankme/d
none70.5 / 58.0 / .3665.1 / 54.9 / .38
VICReg70.8 / 65.7 / .7363.8 / 58.8 / .95
SACReg71.5 / 66.2 / .7865.5 / 59.5 / .92

(W4, Q4) Sensitivity to random slice dimension and buffer sizes

We analyze the requested sensitivity to the random slice dimension and buffer sizes in the ImageNet-100 setting and present the table below. The table shows that the results are relatively insensitive to the slice width and buffer size within a reasonable range. We find that slice widths of 64-128 and buffer lengths of 1-3 show similar linear and kNN accuracies. Additionally, feature rank remains high across all tested configurations except the small 32 dimensional slice setting. Performance degrades mainly at the extreme settings (a 32 dimensional slice without a buffer, the full width slice, or the longest buffer). We included this experiment in Appendix B.2.

slice widthbufferlinearkNNRankMe/d
32067.661.0.57
64171.265.4.72
128 (paper)371.566.2.78
256767.259.7.79
128171.366.2.80
128271.966.5.80
128469.263.8.74

(W4, Q4) No ablation in large scale results

We mainly used ImageNet-100 for controlled ablations allowing us to systematically evaluate six SSL objectives, different regularizers and augmentation settings. These experiments guided our choice of configuration for the larger scale ImageNet-1k experiments where we test scalability of the chosen configuration. Repeating the full ablation at ImageNet-1k scale would be impractical and prohibitive for our resources.

(Q3) Computational cost

Computing SACReg introduces only modest additional computational and memory overhead compared with the overall training procedure. In particular, multi-view training is not specific to λ\lambda-JEPA but is also used by our baselines. All ImageNet-100 experiments use identical augmentation settings with fixed number of views, for ImageNet-1k each method prefers their own custom augmentation function with varying number of views. Given this, we followed LeJEPA-style augmentation for large scale experiments and did not exceed others in terms of total number of views used.

The additional computation introduced by SACReg comes from random orthonormal projections O(dd′2)O(dd'^2), covariance estimation O(Bdd′+Bd′2)O(Bdd' + Bd'^2), and log-determinant computation O(d′3)O(d'^3), where d′d' is the slice width, dd the feature dimension, and BB the effective batch size. The detached ring buffer requires O(qBd)O(qBd) additional memory (typically q=3q=3) without retaining computation graphs. Thus, memory overhead is limited to storing detached features, while additional computation comes primarily from the projection and covariance operations. We included this discussion in Appendix C.2.


References

[1] Nam, Y., Lee, S.H., Domine, C.C., Park, Y., London, C., Choi, W., Goring, N. and Lee, S., 2025. Position: Solve layerwise linear models first to understand neural dynamical phenomena (neural collapse, emergence, lazy/rich regime, and grokking). arXiv preprint arXiv:2502.21009.

[2] Anguita, N., Locatello, F., Saxe, A.M., Mondelli, M., Mancini, F., Lippl, S. and Domine, C., 2026. A theory of how pretraining shapes inductive bias in fine-tuning. arXiv preprint arXiv:2602.20062.

[3] Saxe, A.M., McClelland, J.L. and Ganguli, S., 2019. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23), pp.11537-11546.

[4] Ghosh, A., Kwon, S.M., Wang, R., Ravishankar, S. and Qu, Q., 2025. Learning dynamics of deep linear networks beyond the edge of stability. arXiv preprint arXiv:2502.20531.

[5] Kunin, D., Raventós, A., Dominé, C., Chen, F., Klindt, D., Saxe, A. and Ganguli, S., 2024. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. Advances in Neural Information Processing Systems, 37, pp.81157-81203.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.