acceptodds
Under review as a conference paper at ICLR 2027

KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning

Abstract

Breakthroughs in self-supervised Joint-Embedding Predictive Architectures (JEPAs) have established that regularizing Euclidean representations toward isotropic Gaussian priors yields provable gains in training stability and downstream generalization. We introduce a new, flexible family of Euclidean joint embedding self-supervised learning algorithms with kernel-based regularizers. One instance of this family corresponds to the SIGReg Epps-Pulley regularizer which approximates a sliced maximum mean discrepancy (MMD) with a Gaussian prior and Gaussian kernel. By expanding the class of viable kernels and priors and computing the closed-form high-dimensional limit of sliced discrepancies, we develop alternative methods with improved training stability and design flexibility and leverage those techniques to train strong vision learners at scale.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, Joint-embedding architectures, kernel methods, distribution matching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PpsXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows formally that SIGReg, the regularizer used in LeJEPA, is equivalent to an MMD toward an isotropic Gaussian, with a specific heavy-tailed, dimension-dependent kernel induced by slicing. Turning this observation into algorithms, the paper derives closed-form estimators that need no random slices, one for a sliced MMD (CWMMD) and one for a sliced KSD (CWKSD). On ImageNet, the paper shows convergence of finite-slice estimators as well as performance compared to DINO and iBOT.

Critical strengths

S1: Separating the regularizer from its estimator is clarifying. Slicing implicitly induces a heavy-tailed, dimension-dependent kernel, and making this explicit removes the slice and quadrature hyperparameters. This is overall a helpful result that conceptualizes the broader landscape of SSL objectives. This is the main conceptual value and holds up independently of the remarks in W1.

S2: The experimental results are convincing at budgets of 800/400/250 epochs. This is the main supporting evidence that the method is feasible and relevant in practice. A final evaluation of this point, however, depends on W2 and W3.

Critical weaknesses

W1. The theory is partially known, and parts are incorrect as stated.

  1. Most pieces already exist: Knop et al. already derive the hypergeometric Cramér–Wold kernel, the closed-form comparison with a Gaussian prior, and the large-dimensional approximation. These substantially overlap with the mathematical machinery behind Theorems 1–4.

  2. The main new result, the closed-form sliced KSD, appears to contain a sign error. The paper’s own 1D Stein kernel has a minus sign, −(2γ/σ2+4γ2)(u−v)2-(2\gamma/\sigma^2 + 4\gamma^2)(u-v)^2. One line in the proof of Lemma 1 (around line 1144) turns it into a plus, and the plus carries through into Lemma 1 and CWKSD. With the plus, the regularizer no longer vanishes at the prior. Therefore, we end up with an O(1)O(1) error that doesn’t shrink with dd. It therefore also contradicts Thm. 5’s claim that the error is only od(1)o_d(1).

That doesn’t show which sign the code uses, though, because KerJEPA’s target standardization also discourages collapse. This needs to be checked carefully to determine whether it is an error in the write-up or also in the implementation (I could not check, as the code is not provided).

W2. The ImageNet gains are not directly attributed to the regularizer.

  1. The experimental setup adds several components besides CWMMD, such as an EMA teacher with stop-gradient, target standardization, and so on.

  2. Runs without target standardization reportedly collapse. The paper also says target standardization constrains the representation’s location and scale, which overlaps with the regularizer’s role. Overall, it is not clear what changes can actually be attributed to CWMMD.

W3. Additional experimental details

  1. The analytic estimators are quadratic in batch size, but no wall-clock or memory comparison with SIGReg is given. The claimed computational benefits are therefore unverified. While this is discussed in general, numerical validation would help, as a new algorithm is suggested.

  2. Table 1 has no error bars and spans under 2 points. The claim that heavy tails help rests on a single entry. The flexibility results show only that the alternatives do not hurt.

Questions for authors

Q1. Please confirm the sign of the third term in Lemma 1 and Thm. 5, and which sign the code uses.

Q2. Under the identical ImageNet recipe (ViT-S suffices), please report results with (a) finite-slice SIGReg, (b) unsliced Gaussian MMD, and (c) λ=0\lambda = 0. Why does CWMMD alone not prevent collapse without target standardization? Please also report standard deviations here and for Table 1.

3 comments

We thank the reviewer for their constructive feedback.

W1. The reviewer is correct that the literature has established results deriving the hypergeometric Cram'er-Wold kernel and its high-dimensional approximation (indeed, we gave credit to these works). The purpose of this paper was to frame the recent work on Euclidean SSL in a new light, and particularly, to demonstrate what can be achieved by distinguishing the estimator of the regularizer from the regularizer itself in algorithm design, revealing a space of tradeoffs that can be navigated for profit. In addition, we provide a closed-form slice-free KSD estimator, which is novel to the best of our knowledge.

Regarding the sign error you identified, thank you for pointing this out. We have corrected it, and we note that this sign error existed only in the manuscript and \emph{not} in the code. Lastly, regarding the error term, we apologize for this; indeed you are correct, though the difference is immaterial. To be clear, the issue was with the simplification from od(1)+O(1/n)o_d(1) + O(1 / n) to Od(1/n)O_d(1/n), which is incorrect as you said. However, the od(1)+O(1/n)o_d(1) + O(1 / n) bound is correct. Indeed, for fixed finite dd, there is bias, but this is to be expected. That said, with large enough dd and nn, the bias can be made arbitrarily small. We will amend the bound in the revision, but note that no downstream claims are materially affected.

W2. We appreciate the feedback and recognize that certain experimental details could have been made clearer. Notably, the inclusion of target standardization and self-distillation only applied to the final large-scale training run (with results shown in Table 2). The results of Figure 1 and Table 1 on Imagenette precisely evaluate the intervention on the kernel and estimator, \emph{without} self-distillation or target standardization, making for a fair comparison. The purpose of the Table 2 experiment was to demonstrate how the loss/model derived from our previous results translate to performance on a ubiquitous benchmark where strong baseline results are already well-established. This experiment was generated from a direct fork of DINO, which itself is a student-teacher model; we changed only the target standardization and the regularization components of the loss. We will be sure to amend the experimental section to clarify these points.

W3. We do not report wall-time performance because this is sensitive to the precise GPUs / utilization allocated for different training runs. However, we do note that wall clock time is very similar when training with our proposed estimator compared to e.g. LeJEPA; we are not claiming that it is \emph{better} in that regard, but it is also not worse.

You can see some evidence for example in Figure 1. Here, in order to even match performance of KerJEPA, LeJEPA required on the order of 1024 slices. So, if the batch size is comparable to 1024 or smaller, the quadratic complexity of our estimator was already being paid for. We note that many popular contrastive techniques such as SimCLR and MoCov3 also use quadratic estimators and are widely adopted.

As another point, the main purpose of our work was to introduce the concept that the estimator of the regularizer should be distinguished from the regularizer itself. With this understanding, one could also consider integrating alternative estimators of the MMD with more favorable time / memory complexity, such as the linear time MMD estimator, as mentioned in the discussion section.

Last, regarding error bars, generating such error bars would involve several independent training runs from each configuration, which was infeasible with our compute budget.

Q2. Unfortunately, due to computational limitations, we cannot carry out such an experiment. On Imagenette, we did test finite-slice SIGReg and unsliced Gaussian MMD; see Table 1. Note that LeJEPA already reports performance with ViT-L on ImageNet, whose performance we can match using KerJEPA on ViT-S. While most regualizers provide sufficient anti-collapse properties, student-teacher methods have their own collapse dynamics which can be amplified in Euclidean space, since a constraint on a student does not always carry over to the teacher due to EMA dynamics. Standardization helps control norms of the teacher and correct first and second moments issues which can arise. Removing standardization caused collapse in most self-distilled runs and applying it improved accuracy by an average of 0.7 percentage points among non-collapsed runs.

Thank you for clarifying the theoretical results.

While I understand that extensive experiments are computationally expensive and might not always be feasible, when the message and significance of the paper rely on them, results from single runs and an incomplete ablation study are not sufficient.

I therefore retain my current rating.

We highlight that the goal of this work is to decouple the estimator from the objective and that the larger-scale results are only a means of contextualizing the method in the broader SSL landscape. Below, we show the same trends seen on ImageNette occur on ImageNet. This is expected since the variance of SIGReg depends on dimension and the number of slices. Additionally, we show that student teacher models are more prone to collapse without target standardization. This example agrees with the 0.7 delta specified when the model does not collapse.

ViT-S/16: SIGReg versus CWMMD at a fixed LR schedule

We compare SIGReg and CWMMD on ViT-S/16 at a fixed learning-rate (LR) schedule for 100 epochs, reporting ImageNet-1K validation top-1 accuracy (%). Performance is measured with an online linear probe. For SIGReg, slices denotes the number of random projection directions used to estimate the regularizer. This parameter does not apply to CWMMD, indicated by --. An x denotes training collapse rather than a reported top-1 accuracy. Numerical entries report the final online-probe accuracy.

RegularizerSlicesOnline top-1 (%; x = collapse)
CWMMD--73.76
SIGReg1,02473.79
SIGReg128x
SIGReg32x

With 1,024 slices, SIGReg and CWMMD achieve similar online-probe accuracy under this schedule, which is expected since they converge to the same estimate of MMD. Reducing SIGReg to 128 or 32 slices leads to the collapse marked in the table. This demonstrates instability related to the variance of the estimator and confirms the results found in Figure 1 in a student-teacher setting.

ViT-B/16: teacher normalization

We compare teacher target standardization on a ViT-B/16 using CWMMD for 100 epochs and report ImageNet-1K online linear-probe top-1 accuracy (%). Best is the highest accuracy observed during pretraining, whereas Final is the accuracy at the end of pretraining. Reporting both distinguishes peak performance from whether that performance is maintained.

Teacher-output normalizationBest online top-1 (%)Final online top-1 (%)
BN78.9678.95
None73.8145.17

With teacher target standardization, the model converges and hovers around it's best online probe readout. Without it, accuracy deteriorates substantially from its peak by the end of pretraining as a result of partial collapse due to instability.

Reviewer_Jq77OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a different regularization method with kernel for self-supervised learning with JEPA. The authors replace the existing SIGReg approach by different kernels and prior distributions, and derive its high-dimensional limit of sliced MMD and KSD. They compare different regularization methods in terms of training stability, convergence and computational cost etc,. The results show that analytically sliced methods can be more stable and converge faster in certain settings, while finite slicing can be worse sometimes. They also show that Gaussian and Laplace priors give similar downstream performance, so the role of the prior distribution is not very clear yet.

Critical strengths
  1. This paper studies a different regularization method with kernel for self-supervised learning with JEPA. By exploring different kernels, target distributions and estimation methods, it gives more flexibility in designing regularizers for self-supervised learning.

  2. The authors show SIGReg, which originally uses random projections, is mathematically equivalent to a high-dimensional MMD with a specific kernel. Based on this, they derive new estimation methods that can reduce the noise caused by random projections.

  3. The experiments show that analytical slicing can improve training stability and speed up convergence in some settings. KerJEPA also achieves good classification results on ImageNet and it outperform the DINO and iBOT baselines as shown in the paper.

Critical weaknesses
  1. Much of the theoretical work builds on existing MMD and KSD theory, which have already been widely studied. Also, the connection between SIGReg and sliced MMD has been discussed in previous work. So I think the novelty here is more about extending and unifying existing theory, rather than introducing a fundamentally new regularization idea.

  2. Although the paper reports good training stability and downstream results, I would like to see a direct comparison with the original LeJEPA/SIGReg under the same training settings on ImageNet. KerJEPA also uses some additional training designs and stabilization tricks, so it is hard to tell how much of the improvement actually comes from the new kernel regularization itself.

Questions for authors
  1. The ImageNet task shows good result, but there is no direct comparison with the original LeJEPA/SIGReg under the same training setup. KerJEPA also introduces several other training tricks to improve stability. Can it be clarified that how much of the performance gain comes from the new kernel regularizer rather than these additional changes? Are there any controlled experiments that could help separate their effects?

  2. The analytically sliced methods improved training stability, but their computational cost grows quadratically with batch size. How is the actual training time and computational cost compared with the original SIGReg on large-scale tasks such as ImageNet?

4 comments

We thank the reviewer for recognizing the framework's flexibility and the benefits of analytic estimation.

W1. Relationship to existing theory and the contribution of the framework.

We agree that statistical tests, kernel discrepancies, and slicing have established connections. Related Work discusses their use in SSL, including normalized hyperspherical methods. Our contribution develops a Euclidean framework in which kernel, prior, and estimator are independently selectable, placing SIGReg within a broader family beyond a fixed-kernel, fixed-prior characteristic-function test. To the best of our knowledge, we are the first work which makes this connection in the context of Euclidean SSL.

The framework explains the heavy-tailed, dimension-dependent geometry induced by slicing and its computational tradeoffs. Although LeJEPA motivates slicing as key to overcoming the curse of dimensionality, Table 1 demonstrates that effective high-dimensional regularization does not require slicing in the settings studied here. We also derive CWKSD, our analytically sliced KSD estimator, in Lemma 1 and Theorem 5 which is new. Its ImageNet results demonstrate an algorithmic contribution beyond unification.

W2. Attribution of the ImageNet improvements.

The Imagenette experiments provide a direct comparison between SIGReg and CWMMD, without self-distillation or target standardization. In particular, the finite-slice Gaussian MMD with a Gaussian prior used in Figure 1 and Table 1 is SIGReg. Both methods reach 91.13% linear-probe accuracy when using 1,024 slices for SIGReg. However, Figure 1 also shows that using fewer slices introduces instability and slows convergence. This effect becomes more pronounced at higher representation dimensions. These results motivate using CWMMD for the ImageNet experiments rather than carrying out a finite-slice comparison at every scale. Unlike finite-slice estimation, CWMMD does not require choosing the number of slices or quadrature points, and avoids the associated sampling noise. We also use a one-dimensional Silverman bandwidth heuristic, which adapts to the representation distribution during training, rather than fixing the bandwidth to 1 as in LeJEPA. A SIGReg ImageNet experiment using the same training setup would provide a useful additional comparison, but we do not have sufficient compute to run it during the rebuttal.

Our ImageNet results show that both Euclidean regularizers yield strong representations across model scales. To compare against DINO and iBOT, we use their implementation as a starting point and replace the regularization objective, leaving the self-distillation framework unchanged. We do not use iBOT's masked image modeling objective. Thus, these experiments examine the effect of replacing the original regularization with Euclidean distribution matching within the same training framework. See Table 2 in the revision.

Both CWMMD and CWKSD outperform DINO and iBOT at all corresponding model scales. This is particularly relevant in the context of LeJEPA, which argues against the need for self-distillation and instead advocates a simpler joint-embedding formulation. However, its reported ImageNet result of 77.1% with ViT-L/14 remains below the performance of several established self-distillation-based approaches, which include our ViT-S/16 model, that reaches 77.85% and 77.65% accuracy with CWMMD and CWKSD, respectively. Finally, the comparable performance of CWMMD and CWKSD across all three model scales suggests that these improvements are not specific to a particular discrepancy, but extend to different forms of Euclidean distribution matching.

Q1. Separating regularization from the other training changes.

The controlled evidence is Figure 1 and Table 1, detailed in W2, where neither self-distillation nor standardization is used. Our ImageNet recipe shares the DINO/iBOT base implementation, with changes to the loss, targets, and learning-rate and weight-decay schedules.

Target standardization is a contribution of our Euclidean formulation, implemented with batch normalization and motivated by W-MSE. While most regualizers provide sufficient anti-collapse properties, student-teacher methods have their own collapse dynamics which can be amplified in Euclidean space, since a constraint on a student does not always carry over to the teacher due to EMA dynamics. Standardization helps control norms of the teacher and correct first and second moments issues which can arise. Removing standardization caused collapse in most self-distilled runs and applying it improved accuracy by an average of 0.7 percentage points among non-collapsed runs.

Q2. Quadratic estimation and practical training cost.

Although CWMMD uses a quadratic estimator, this does not necessarily make it more expensive in practice than SIGReg. SIGReg requires O(BSd)O(BSd) operations for its projections, where BB is batch size, SS the number of slices, and dd the embedding dimension, compared with O(B2d)O(B^2d) for CWMMD. With typical large-scale SSL batch sizes around 1024 and 1024 slices in SIGReg, the operation counts are comparable. Indeed, our Imagenette experiments show similar overall training times, even though SIGReg requires more arithmetic in the high-dimensional setting of Figure 1.

This quadratic complexity is specific to our current estimator, not the underlying discrepancy. As discussed in Section 7, linear-time MMD estimators, random Fourier features, and B-statistic estimators offer alternative computational tradeoffs while preserving the objective.

More generally, asymptotic complexity alone does not determine practical cost. Contrastive methods such as SimCLR also have quadratic complexity, while DINOv2 uses large prototype banks and distributed Sinkhorn-Knopp normalization. For PP prototypes, our B×BB \times B pairwise matrix is smaller than the B×PB \times P sample-to-prototype matrix when P>BP>B. Given the comparable training times and gains in stability and representation quality, we consider this a worthwhile tradeoff.

Thanks for the clarification, but as no extra experimental results, I want to keep my score.

We highlight that the goal of this work is to decouple the estimator from the objective and that the larger-scale results are only a means of contextualizing the method in the broader SSL landscape. Below, we show the same trends seen on ImageNette occur on ImageNet. This is expected since the variance of SIGReg depends on dimension and the number of slices. Additionally, we show that student teacher models are more prone to collapse without target standardization. This example agrees with the 0.7 delta specified when the model does not collapse.

ViT-S/16: SIGReg versus CWMMD at a fixed LR schedule

We compare SIGReg and CWMMD on ViT-S/16 at a fixed learning-rate (LR) schedule for 100 epochs, reporting ImageNet-1K validation top-1 accuracy (%). Performance is measured with an online linear probe. For SIGReg, slices denotes the number of random projection directions used to estimate the regularizer. This parameter does not apply to CWMMD, indicated by --. An x denotes training collapse rather than a reported top-1 accuracy. Numerical entries report the final online-probe accuracy.

RegularizerSlicesOnline top-1 (%; x = collapse)
CWMMD--73.76
SIGReg1,02473.79
SIGReg128x
SIGReg32x

With 1,024 slices, SIGReg and CWMMD achieve similar online-probe accuracy under this schedule, which is expected since they converge to the same estimate of MMD. Reducing SIGReg to 128 or 32 slices leads to the collapse marked in the table. This demonstrates instability related to the variance of the estimator and confirms the results found in Figure 1 in a student-teacher setting.

ViT-B/16: teacher normalization

We compare teacher target standardization on a ViT-B/16 using CWMMD for 100 epochs and report ImageNet-1K online linear-probe top-1 accuracy (%). Best is the highest accuracy observed during pretraining, whereas Final is the accuracy at the end of pretraining. Reporting both distinguishes peak performance from whether that performance is maintained.

Teacher-output normalizationBest online top-1 (%)Final online top-1 (%)
BN78.9678.95
None73.8145.17

With teacher target standardization, the model converges and hovers around it's best online probe readout. Without it, accuracy deteriorates substantially from its peak by the end of pretraining as a result of partial collapse due to instability.

Reviewer_fa9LOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new, flexible family of Euclidean joint embedding self-supervised learning algorithms with kernel-based regularizers. It interprets LeJEPA’s SIGReg as a sliced maximum mean discrepancy, then uses this connection to develop alternatives that avoid sampling random projection directions. The authors also explore different kernels and priors, propose a related Stein-discrepancy regularizer, and evaluate the resulting KerJEPA framework on Imagenette, ImageNet, and ADE20K.

Critical strengths
  • The paper offers a useful interpretation of SIGReg. Viewing it as a kernel discrepancy makes the relationship between random projections, kernel choice, and representation geometry easier to understand.

  • The distinction between a distribution-matching objective and its numerical estimator can be useful.

  • I found Figure 1 informative and useful to see how the number of slices interacts with representation dimension.

  • The discussion around the projection head is interesting.

Critical weaknesses
  • Missing key experiment: a direct comparison between SIGReg and CWMMD under the same training recipe and tuning budget. Without it, the gains over DINO and iBOT cannot be attributed specifically to the proposed regularizer.

  • Figure 1 shows convergence by epoch rather than elapsed time. Runtime, memory, and communication measurements would make the tradeoff much clearer.

  • The tables also lack uncertainty estimates. Finally, the transfer results are mixed: ADE20K performance is close to DINO but below iBOT across all reported model sizes. The evidence therefore supports competitive classification performance more strongly than broad improvements in representation quality.

  • The strongest results concern ImageNet classification. Segmentation performance is roughly comparable to DINO and below iBOT, while alternative priors and kernels are explored mainly on a subset of ImageNet. Broader transfer experiments would help establish whether the benefits extend beyond the main classification setting.

  • This paper explores several kernels, priors, and estimators. It remains unclear to me when a practitioner should prefer one option over another, or why CWMMD is selected. ( maybe some justification on this front would be helpful)

Questions for authors
  • It is unclear to me,how much of the improvement comes from CWMMD itself. Is it possible to compare it directly with SIGReg using the same teacher, target standardization, augmentations, and optimization settings?

  • The squared-distance term changes sign in the CWKSD derivation, Table 4’s IMQ Hessian appears incorrect, and Figure 2 omits the sine component of the empirical characteristic function.

  • When does removing random slices pay off in practice? At matched wall-clock time, how do the analytic and finite-slice estimators compare in accuracy, memory use, and stability?

  • How essential is target standardization? Given that training reportedly collapses without it, does the proposed regularizer improve stability beyond standardization? Are the sliced and unsliced Laplace experiments targeting the same distribution?

-Could you provide variability across seeds and explain whether the weaker segmentation results suggest a limitation of the learned features?

4 comments

We thank the reviewer for their constructive feedback.

W1. Controlled comparison with SIGReg.

Thanks for the feedback. We realize that our presentation was slightly confusing, and we'd like to take the opportunity to clarify. The source of confusion is that the self-distillation and target standardization are \emph{only} used for the large-scale ImageNet experiment, which was forked from the DINO baseline repository, which itself requires self-distillation; our target standardization is a contribution meant specifically to stabilize training in this self-distillation regime, and the experiment which only modifies the loss function demonstrates the effectiveness of our Euclidean representations above the state-of-the-art. That said, our Imagenette experiments \emph{do not} employ self-distillation or target standardization. Thus, the sliced implementation with the Gaussian kernel corresponds to SIGReg, and consequently, the requested comparison is actually given in this experiment. Replicating the comparison on the large-scale ImageNet data was beyond our computational budget.

On Imagenette, both SIGReg with 1,024 slices and slice-free CWMMD reach 91.13% linear-probe accuracy. Figure 1 shows that insufficient slicing slows convergence and introduces early instability, with more slices needed at higher dimensions. The additional stability motivate the use of CWMMD / CWKSD in downstream runs. We will better clarify this in the main text and explain how this motivates the algorithmic progression more clearly.

Returning to the large-scale ImageNet experiment, we also show that our other CWKSD drop-in Euclidean loss is competitive compared to baselines on classification tasks. See Table 2 in the revision.

W2. Computational tradeoffs.

Regarding computational complexity, we note that although our current CWMMD estimator has quadratic complexity in the batch size, this does not necessarily translate into higher computational cost in practical regimes compared with the linear-in-batch-size, slicing-based approach of SIGReg. Specifically, SIGReg has complexity O(BSd)O(BSd), where BB is the batch size, SS is the number of slices, and dd is the representation dimension, whereas our estimator has complexity O(B2d)O(B^2d). In typical large-scale SSL settings, however, B≈1024B \approx 1024, while SIGReg commonly uses S=1024S = 1024 slices, making the two approaches comparable in many practical scenarios.

More importantly, the quadratic complexity is a property of our particular estimator, not a fundamental requirement of the underlying discrepancy. As discussed in Section 7, alternative linear-time estimators, random Fourier features, and B-statistic estimators provide different computational and communication tradeoffs without changing the underlying objective and can be parallelized just like SIGReg.

Finally, quadratic scaling alone does not imply that a regularizer becomes a practical computational bottleneck. For context, many contrastive methods such as already SimCLR / Mocov3 are quadratic and are still practical.

W3. Uncertainty and transfer performance.

Repeated-seed uncertainty estimates are not feasible for the large-scale SSL runs compared here due to computation cost.

DINO is the closest objective-level ADE20K comparison because both methods use global CLS-level training without masked patch supervision. iBOT extends DINO-style alignment to masked patches. KerJEPA preserves DINO-level segmentation performance while exceeding both methods on ImageNet classification, despite using no masked objective, which can be added in future work.

W4. Scope and broader transfer.

ADE20K tests whether globally trained representations retain useful spatial features. iBOT remains stronger there as expected. The addition of masked patch alignment in KerJEPA is a natural extension which we expect to show similar lift as observed from DINO to iBOT but is outside our scope.

W5. Choosing a kernel, prior, and estimator.

The choice of kernel, prior, and estimator is an interesting question, and indeed our goal is to shed some light on what choices impact training. We first argue that a kernel bandwidth should be considered. LeJEPA uses a fixed bandwidth of 1, while we use a 1d Silverman heuristic which accounts for batch size. Overall, our results show that slicing is not necessary for beating the curse of dimensionality, and in fact our slice-free framework removes a hyperparameter, inducing lower variance (see Figure 1).

In the context of SSL, the prior constrains projector outputs, while transfer uses the backbone after discarding the projector. It therefore does not directly prescribe downstream feature geometry. Other applications may benefit from different geometries at different model layers, since these choices remain open and application dependent. We do not make any direct recommendation outside the scope of the tasks evaluated.

Q1. Matched SIGReg comparison.

W1 details the direct SIGReg--CWMMD comparison and why it motivates CWMMD for ImageNet from the lens of stability, especially given issues of collapse in self-distillation frameworks. Repeating it explicitly under the ImageNet student-teacher recipe exceeds our computation budget.

Q2. Corrections to the derivation and pseudocode.

We will correct the squared-distance sign throughout the CWKSD derivation, Lemma 1, and Theorem 5; the missing contribution and sign in Table 4's IMQ mixed-Hessian trace; and Figure 2's missing sine component. These are manuscript and pseudocode-figure errors only. There were no implementation errors. The experimental code was correct, and all results are unaffected.

Q3. Matched wall-clock time performance, memory, and stability.

W2 explains the computational tradeoffs. Figure 1 establishes convergence and stability by epoch; matched-time accuracy and peak-memory benchmarks are unavailable. For large and expensive runs, where model collapse can incur a huge cost, a stable and more reliable estimator which does not require additional parameter tuning is beneficial.

Q4. Target standardization, collapse, and Laplace experiments.

Target standardization is also a contribution of our Euclidean formulation and is introduced for self-distillation. Instability through coupled student-teacher updates can exist even with a distributional regularizer, since the regularizer is only applied to the student with no guarantees on the teacher due to EMA dynamics. Adding an explicit term on the teacher that constrains its outputs adds an additional layer of safety.

Removing standardization caused collapse in the majority, but not all, of those runs. Among non-collapsed runs, it improved ImageNet top-1 accuracy by an average of 0.7 percentage points. We will correct the categorical collapse statement.

Imagenette follows LeJEPA's non-self-distilled setting without a teacher or standardization, and we observed no collapse. Our non-self-distilled models were not competitive in representation quality, motivating self-distillation at ImageNet scale.

Both Laplace settings use the same target distribution. Sliced KSD evaluates the target score in the ambient space, then projects the score and embeddings along the same unit direction and applies the kernel to the projected embeddings which preserves its geometry. Neither setting uses standardization as the Laplace target was run on Imagenette section.

We highlight that the goal of this work is to decouple the estimator from the objective and that the larger-scale results are only a means of contextualizing the method in the broader SSL landscape. Below, we show the same trends seen on ImageNette occur on ImageNet. This is expected since the variance of SIGReg depends on dimension and the number of slices. Additionally, we show that student teacher models are more prone to collapse without target standardization. This example agrees with the 0.7 delta specified when the model does not collapse.

ViT-S/16: SIGReg versus CWMMD at a fixed LR schedule

We compare SIGReg and CWMMD on ViT-S/16 at a fixed learning-rate (LR) schedule for 100 epochs, reporting ImageNet-1K validation top-1 accuracy (%). Performance is measured with an online linear probe. For SIGReg, slices denotes the number of random projection directions used to estimate the regularizer. This parameter does not apply to CWMMD, indicated by --. An x denotes training collapse rather than a reported top-1 accuracy. Numerical entries report the final online-probe accuracy.

RegularizerSlicesOnline top-1 (%; x = collapse)
CWMMD--73.76
SIGReg1,02473.79
SIGReg128x
SIGReg32x

With 1,024 slices, SIGReg and CWMMD achieve similar online-probe accuracy under this schedule, which is expected since they converge to the same estimate of MMD. Reducing SIGReg to 128 or 32 slices leads to the collapse marked in the table. This demonstrates instability related to the variance of the estimator and confirms the results found in Figure 1 in a student-teacher setting.

ViT-B/16: teacher normalization

We compare teacher target standardization on a ViT-B/16 using CWMMD for 100 epochs and report ImageNet-1K online linear-probe top-1 accuracy (%). Best is the highest accuracy observed during pretraining, whereas Final is the accuracy at the end of pretraining. Reporting both distinguishes peak performance from whether that performance is maintained.

Teacher-output normalizationBest online top-1 (%)Final online top-1 (%)
BN78.9678.95
None73.8145.17

With teacher target standardization, the model converges and hovers around it's best online probe readout. Without it, accuracy deteriorates substantially from its peak by the end of pretraining as a result of partial collapse due to instability.

I thank the authors for responding to my concerns. I will discuss this with other reviewers and make a final decision. I dont have further questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.