- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows formally that SIGReg, the regularizer used in LeJEPA, is equivalent to an MMD toward an isotropic Gaussian, with a specific heavy-tailed, dimension-dependent kernel induced by slicing. Turning this observation into algorithms, the paper derives closed-form estimators that need no random slices, one for a sliced MMD (CWMMD) and one for a sliced KSD (CWKSD). On ImageNet, the paper shows convergence of finite-slice estimators as well as performance compared to DINO and iBOT.
Critical strengths
S1: Separating the regularizer from its estimator is clarifying. Slicing implicitly induces a heavy-tailed, dimension-dependent kernel, and making this explicit removes the slice and quadrature hyperparameters. This is overall a helpful result that conceptualizes the broader landscape of SSL objectives. This is the main conceptual value and holds up independently of the remarks in W1.
S2: The experimental results are convincing at budgets of 800/400/250 epochs. This is the main supporting evidence that the method is feasible and relevant in practice. A final evaluation of this point, however, depends on W2 and W3.
Critical weaknesses
W1. The theory is partially known, and parts are incorrect as stated.
-
Most pieces already exist: Knop et al. already derive the hypergeometric Cramér–Wold kernel, the closed-form comparison with a Gaussian prior, and the large-dimensional approximation. These substantially overlap with the mathematical machinery behind Theorems 1–4.
-
The main new result, the closed-form sliced KSD, appears to contain a sign error. The paper’s own 1D Stein kernel has a minus sign, . One line in the proof of Lemma 1 (around line 1144) turns it into a plus, and the plus carries through into Lemma 1 and CWKSD. With the plus, the regularizer no longer vanishes at the prior. Therefore, we end up with an error that doesn’t shrink with . It therefore also contradicts Thm. 5’s claim that the error is only .
That doesn’t show which sign the code uses, though, because KerJEPA’s target standardization also discourages collapse. This needs to be checked carefully to determine whether it is an error in the write-up or also in the implementation (I could not check, as the code is not provided).
W2. The ImageNet gains are not directly attributed to the regularizer.
-
The experimental setup adds several components besides CWMMD, such as an EMA teacher with stop-gradient, target standardization, and so on.
-
Runs without target standardization reportedly collapse. The paper also says target standardization constrains the representation’s location and scale, which overlaps with the regularizer’s role. Overall, it is not clear what changes can actually be attributed to CWMMD.
W3. Additional experimental details
-
The analytic estimators are quadratic in batch size, but no wall-clock or memory comparison with SIGReg is given. The claimed computational benefits are therefore unverified. While this is discussed in general, numerical validation would help, as a new algorithm is suggested.
-
Table 1 has no error bars and spans under 2 points. The claim that heavy tails help rests on a single entry. The flexibility results show only that the alternatives do not hurt.
Questions for authors
Q1. Please confirm the sign of the third term in Lemma 1 and Thm. 5, and which sign the code uses.
Q2. Under the identical ImageNet recipe (ViT-S suffices), please report results with (a) finite-slice SIGReg, (b) unsliced Gaussian MMD, and (c) . Why does CWMMD alone not prevent collapse without target standardization? Please also report standard deviations here and for Table 1.
3 comments
Thank you for clarifying the theoretical results.
While I understand that extensive experiments are computationally expensive and might not always be feasible, when the message and significance of the paper rely on them, results from single runs and an incomplete ablation study are not sufficient.
I therefore retain my current rating.
We highlight that the goal of this work is to decouple the estimator from the objective and that the larger-scale results are only a means of contextualizing the method in the broader SSL landscape. Below, we show the same trends seen on ImageNette occur on ImageNet. This is expected since the variance of SIGReg depends on dimension and the number of slices. Additionally, we show that student teacher models are more prone to collapse without target standardization. This example agrees with the 0.7 delta specified when the model does not collapse.
ViT-S/16: SIGReg versus CWMMD at a fixed LR schedule
We compare SIGReg and CWMMD on ViT-S/16 at a fixed learning-rate (LR) schedule for 100 epochs, reporting ImageNet-1K validation top-1 accuracy (%). Performance is measured with an online linear probe. For SIGReg, slices denotes the number of random projection directions used to estimate the regularizer. This parameter does not apply to CWMMD, indicated by --. An x denotes training collapse rather than a reported top-1 accuracy. Numerical entries report the final online-probe accuracy.
| Regularizer | Slices | Online top-1 (%; x = collapse) |
|---|---|---|
| CWMMD | -- | 73.76 |
| SIGReg | 1,024 | 73.79 |
| SIGReg | 128 | x |
| SIGReg | 32 | x |
With 1,024 slices, SIGReg and CWMMD achieve similar online-probe accuracy under this schedule, which is expected since they converge to the same estimate of MMD. Reducing SIGReg to 128 or 32 slices leads to the collapse marked in the table. This demonstrates instability related to the variance of the estimator and confirms the results found in Figure 1 in a student-teacher setting.
ViT-B/16: teacher normalization
We compare teacher target standardization on a ViT-B/16 using CWMMD for 100 epochs and report ImageNet-1K online linear-probe top-1 accuracy (%). Best is the highest accuracy observed during pretraining, whereas Final is the accuracy at the end of pretraining. Reporting both distinguishes peak performance from whether that performance is maintained.
| Teacher-output normalization | Best online top-1 (%) | Final online top-1 (%) |
|---|---|---|
| BN | 78.96 | 78.95 |
| None | 73.81 | 45.17 |
With teacher target standardization, the model converges and hovers around it's best online probe readout. Without it, accuracy deteriorates substantially from its peak by the end of pretraining as a result of partial collapse due to instability.
We thank the reviewer for their constructive feedback.
W1. The reviewer is correct that the literature has established results deriving the hypergeometric Cram'er-Wold kernel and its high-dimensional approximation (indeed, we gave credit to these works). The purpose of this paper was to frame the recent work on Euclidean SSL in a new light, and particularly, to demonstrate what can be achieved by distinguishing the estimator of the regularizer from the regularizer itself in algorithm design, revealing a space of tradeoffs that can be navigated for profit. In addition, we provide a closed-form slice-free KSD estimator, which is novel to the best of our knowledge.
Regarding the sign error you identified, thank you for pointing this out. We have corrected it, and we note that this sign error existed only in the manuscript and \emph{not} in the code. Lastly, regarding the error term, we apologize for this; indeed you are correct, though the difference is immaterial. To be clear, the issue was with the simplification from od(1)+O(1/n) to Od(1/n), which is incorrect as you said. However, the od(1)+O(1/n) bound is correct. Indeed, for fixed finite d, there is bias, but this is to be expected. That said, with large enough d and n, the bias can be made arbitrarily small. We will amend the bound in the revision, but note that no downstream claims are materially affected.
W2. We appreciate the feedback and recognize that certain experimental details could have been made clearer. Notably, the inclusion of target standardization and self-distillation only applied to the final large-scale training run (with results shown in Table 2). The results of Figure 1 and Table 1 on Imagenette precisely evaluate the intervention on the kernel and estimator, \emph{without} self-distillation or target standardization, making for a fair comparison. The purpose of the Table 2 experiment was to demonstrate how the loss/model derived from our previous results translate to performance on a ubiquitous benchmark where strong baseline results are already well-established. This experiment was generated from a direct fork of DINO, which itself is a student-teacher model; we changed only the target standardization and the regularization components of the loss. We will be sure to amend the experimental section to clarify these points.
W3. We do not report wall-time performance because this is sensitive to the precise GPUs / utilization allocated for different training runs. However, we do note that wall clock time is very similar when training with our proposed estimator compared to e.g. LeJEPA; we are not claiming that it is \emph{better} in that regard, but it is also not worse.
You can see some evidence for example in Figure 1. Here, in order to even match performance of KerJEPA, LeJEPA required on the order of 1024 slices. So, if the batch size is comparable to 1024 or smaller, the quadratic complexity of our estimator was already being paid for. We note that many popular contrastive techniques such as SimCLR and MoCov3 also use quadratic estimators and are widely adopted.
As another point, the main purpose of our work was to introduce the concept that the estimator of the regularizer should be distinguished from the regularizer itself. With this understanding, one could also consider integrating alternative estimators of the MMD with more favorable time / memory complexity, such as the linear time MMD estimator, as mentioned in the discussion section.
Last, regarding error bars, generating such error bars would involve several independent training runs from each configuration, which was infeasible with our compute budget.
Q2. Unfortunately, due to computational limitations, we cannot carry out such an experiment. On Imagenette, we did test finite-slice SIGReg and unsliced Gaussian MMD; see Table 1. Note that LeJEPA already reports performance with ViT-L on ImageNet, whose performance we can match using KerJEPA on ViT-S. While most regualizers provide sufficient anti-collapse properties, student-teacher methods have their own collapse dynamics which can be amplified in Euclidean space, since a constraint on a student does not always carry over to the teacher due to EMA dynamics. Standardization helps control norms of the teacher and correct first and second moments issues which can arise. Removing standardization caused collapse in most self-distilled runs and applying it improved accuracy by an average of 0.7 percentage points among non-collapsed runs.