- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LocaLeJEPA which replaces LeJEPA's global isotropic-Gaussian target with local distribution matching. Within each minibatch, the representations are clustered. Then the centered residuals are matched to the uniform distribution on a Euclidean ball. The authors also conduct extensive experiments with Euclidean, hyperbolic and equivariant encoders on ImageNet-1k. The experiments also cover seven transfer datasets and controlled rotation tasks.
Critical strengths
-
Simple and effective method. I like this simple idea, and the manuscript is easy-to-follow. The proposed approach itself is simple modification to distribution-matching self-supervised learning
-
Solid and substantial gains over baselines. The experimental demonstrations show the method is significantly improved over LeJEPA across Euclidean, hyperbolic and equivariant encoders, as well as seven transfer datasets.
-
The additional computation overhead is also small, while the improvements are impressive.
Critical weaknesses
I’m mostly concerned about the fairness of the experimental comparison.
- The baselines and LocaLeJEPA use different training settings like learning rates, warmup, gradient clipping and regularization weight. The improvement may not come from the regularizer itself, but a fair comparison is encouraged to solidify the contribution.
- The local SIGReg control changes both centering and the target distribution, which makes it hard to tell how much of the improvement comes from locality, centering, the uniform-ball target, or the training recipe. The K=1 vs. K>1 gap is small in Imagenette, and K=1 is not reported on ImageNet.
Questions for authors
-
The clustering is performed independently within each minibatch of 64 images, so how these neighborhoods capture meaningful local structure reliably? Would random partitions with macthed cluster sizes, using same centering + distribution matching result in similar improvements?
-
What are the ImageNet results for K=1 (with centered uniform ball) and for centered local Gaussian matching under the same training settings?
4 comments
Response to reviewer uyJV, comment 2 of 2
Q1: Neighborhoods. Each step has 256 images in four shards of 64, and -means runs within each shard, so a neighborhood at holds about 16 images. They are not classes (a shard from 1000 classes holds about 2 same-class pairs). They are images that sit close in the current embedding, and they form reliably. In fact, in the training logs, we observe 3.97 of 4 neighborhoods per shard, 99.99% of samples kept, and no shard fell back to a single neighborhood. Although we did not measure their quality explicitly (we are open to suggestions), the random-partition test below still shows that they carry the gain.
Random partitions. Thank you for suggesting this test. Keeping -means but shuffling its labels, so that each 64-image shard has random neighborhoods of the same sizes, costs 3.7 / 3.0 on ViT-S/8 (three seeds, smallest drop 3.5 / 2.6), 3.9 / 2.9 on H-ViT and 4.1 / 3.5 on -RN50. Random neighborhoods even fall below a single neighborhood at the same strength. So the gain needs data-dependent neighborhoods (Table R1 above gives the absolute numbers).
Q2: on ImageNet-1K. is in Table R1 on all four encoders, and in the revised manuscript we will insert the results at matched strength. On ViT-L/14, reads 46.5 / 54.8 at LeJEPA's published ViT-L settings (, one seed).
Centered local Gaussian. Table R2 gives it at , under the same training settings as the ball: it performs like the ball, and on H-ViT the uncentered version falls far behind. On ViT-S/8 the ball is above it on 7 of 7 transfer datasets, but the Gaussian's average lies within the range of the ball's three seeds. Our position on the ball is in W2 above.
Table R2. , , one seed each: -NN / linear / transfer.
| Target | ViT-S/8 | H-ViT |
|---|---|---|
| ball, centered (LocaLeJEPA) | 50.11 / 62.11 / 72.49 | 50.57 / 61.88 / 72.11 |
| Gaussian, centered | 50.29 / 62.09 / 72.01 | 50.70 / 62.45 / 72.52 |
| Gaussian, not centered (LocaLeJEPA-SIGReg) | not run | 35.35 / 48.15 / 62.83 |
We note we do not rerun the ViT-S/8 case corresponding to LocaLeJEPA-SIGReg already included in the previous experiments.
We sincerely thank the reviewer once again for their thoughtful and constructive feedback. We believe that the additional experiments and clarifications prompted by their suggestions strengthen our work. We kindly ask for further feedback, and we are happy to engage in more discussion.
I thank the authors for their efforts in addressing my concerns, therefore I raise my score.
Response to reviewer uyJV, comment 1 of 2
We thank the reviewer for the careful review of our work. We also appreciate the reviewer noting that our framework leads to substantial gains with only simple changes to LeJEPA’s objective. We can address the reviewer's concerns about a fair baseline in depth, as we had been working to ensure this before the rebuttal period started.
The experimental protocol we will integrate in the revised manuscript re-trains every ImageNet-1K arm in one fixed pipeline, with 256 images per step in four shards of 64 at 128 px (512 per step at 238 px for ViT-L/14) and three seeds on ViT-S/8. The full results, of which we describe the salient parts in this reply for brevity, will be integrated into the revised manuscript we will upload before the end of the rebuttal period, such that we may further integrate additional comments we will receive in the following days. In Table R1 below, the ± is the sd over seeds. In this reply, H-ViT is the hyperbolic ViT-S/8 and D4-RN50 the D4-equivariant ResNet-50. Pairs are k-NN / linear top-1 on ImageNet-1K, and transfer is the linear top-1 averaged over seven other datasets.
W1: Training settings. Relative to LeJEPA at its published settings, LocaLeJEPA gains +17.3 / +17.6 on ViT-S/8. The reviewer is right to request additional isolated components evaluation, and here we try to be as transparent as possible. In particular, we now compare at both the same settings and effective regularization strength λN/K, i.e., the weight times the number of samples per neighborhood (N=64 samples in each clustering step, see Table R1). Our training settings, with λ=0.1, lift LeJEPA by +6.1 / +7.1 on ViT-S/8. At λ=0.1, though, its regularizer is four times stronger than ours. For this reason, we also lower its weight to 0.025, and this gains it another +7.6 / +7.0. Overall, at completely matched strength and settings, LocaLeJEPA still consistently leads on all three encoders and every ViT-S/8 seed (lowest seed +3.5 / +3.2). At 128 px, the published settings are those of LeJEPA's Imagenette example, carried to ImageNet-1K.
Table R1. ImageNet-1K, k-NN / linear top-1, ViT-S/8 averaged over three seeds, the other encoders one seed each. Differences are taken before rounding.
W2: Local SIGReg control. We agree, and we now separate the four parts the reviewer names. At matched strength and settings, the centered ball at K=1 adds +1.3 / +1.1 over LeJEPA, and locality (K=4 over K=1) another +2.3 / +2.3. Table R2 below separates centering from the target's shape. Replacing the ball with a centered Gaussian moves k-NN and linear by at most 0.6 and transfer by at most 0.5 (K=1 on three encoders, K=4 on two, one seed each). Removing the centering from that local Gaussian costs 15.3 / 14.3 on H-ViT (one seed). Overall, centering and locality carry most of the effect. We note that this is aligned with our theoretical discussions in Section 3, where we believe the heat flow distribution to be the most salient aspect, and the particular choice of the ball as our local distribution is more so suggested by entropy-maximization heuristics of prior works.
K=1 against K>1. K=1 is now reported at ImageNet-1K on all four encoders (Table R1 above). At matched strength, K=4 adds +2.3 / +2.3 over K=1 on ViT-S/8 (three seeds, lowest seed +2.1 / +2.0), +2.6 / +2.0 on H-ViT and +3.0 / +2.8 on D4-RN50, and it stays positive at every tested depth (on ViT-S/8 with 6 and 3 blocks instead of 12, +1.7 / +1.9 and +1.2 / +0.6, one seed each). On Imagenette, that is a much easier dataset and results in easier downstream tasks, the reviewer is right, i.e., with λ tuned per K, K>1 and K=1 tie.