acceptodds
Under review as a conference paper at ICLR 2027

LocaLeJEPA: Maximum Entropy Representations under Geometric Priors

12 views

Abstract

Joint-Embedding Predictive Architectures (JEPA) offer a compelling approach to self-supervised learning, but are typically prone to representation collapse. LeJEPA resolves this by introducing a provably robust distribution-matching objective that drives the embedding towards an isotropic Gaussian. However, this approach is fundamentally tied to Euclidean assumptions. Therefore, it is incompatible with the incorporation of domain-specific geometric priors, such as the use of non-Euclidean embedding geometries, e.g., equivariant CNNs or hyperbolic ViTs. We thus introduce LocaLeJEPA, a framework motivated by a manifold-theoretic reformulation of LeJEPA’s entropy-maximization property. The accompanying LocaLeReg, a novel locally Euclidean distribution-matching regularizer, enables a robust, practical implementation of the LocaLeJEPA framework. We then perform extensive experiments on models trained on ImageNet-1k. LocaLeJEPA outperforms LeJEPA not only when paired with hyperbolic ViT-S/8 and equivariant ResNet-50 backbones, but even with an ordinary Euclidean ViT-S/8, both in terms of linear and kNN classification performance on ImageNet and on a series of downstream tasks tested. Overall, we view the LocaLeJEPA formalism as a principled approach to the JEPA paradigm that better respects intrinsic data geometry while also accommodating generalizations to exotic embedding geometries.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Self-Supervised Learning, Geometric Deep Learning, Entropy-Maximization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uyJVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LocaLeJEPA which replaces LeJEPA's global isotropic-Gaussian target with local distribution matching. Within each minibatch, the representations are clustered. Then the centered residuals are matched to the uniform distribution on a Euclidean ball. The authors also conduct extensive experiments with Euclidean, hyperbolic and equivariant encoders on ImageNet-1k. The experiments also cover seven transfer datasets and controlled rotation tasks.

Critical strengths
  • Simple and effective method. I like this simple idea, and the manuscript is easy-to-follow. The proposed approach itself is simple modification to distribution-matching self-supervised learning

  • Solid and substantial gains over baselines. The experimental demonstrations show the method is significantly improved over LeJEPA across Euclidean, hyperbolic and equivariant encoders, as well as seven transfer datasets.

  • The additional computation overhead is also small, while the improvements are impressive.

Critical weaknesses

I’m mostly concerned about the fairness of the experimental comparison.

  • The baselines and LocaLeJEPA use different training settings like learning rates, warmup, gradient clipping and regularization weight. The improvement may not come from the regularizer itself, but a fair comparison is encouraged to solidify the contribution.
  • The local SIGReg control changes both centering and the target distribution, which makes it hard to tell how much of the improvement comes from locality, centering, the uniform-ball target, or the training recipe. The K=1 vs. K>1 gap is small in Imagenette, and K=1 is not reported on ImageNet.
Questions for authors
  • The clustering is performed independently within each minibatch of 64 images, so how these neighborhoods capture meaningful local structure reliably? Would random partitions with macthed cluster sizes, using same centering + distribution matching result in similar improvements?

  • What are the ImageNet results for K=1 (with centered uniform ball) and for centered local Gaussian matching under the same training settings?

4 comments
AuthorsOpenReview ↗

Response to reviewer uyJV, comment 1 of 2

We thank the reviewer for the careful review of our work. We also appreciate the reviewer noting that our framework leads to substantial gains with only simple changes to LeJEPA’s objective. We can address the reviewer's concerns about a fair baseline in depth, as we had been working to ensure this before the rebuttal period started.

The experimental protocol we will integrate in the revised manuscript re-trains every ImageNet-1K arm in one fixed pipeline, with 256 images per step in four shards of 64 at 128 px (512 per step at 238 px for ViT-L/14) and three seeds on ViT-S/8. The full results, of which we describe the salient parts in this reply for brevity, will be integrated into the revised manuscript we will upload before the end of the rebuttal period, such that we may further integrate additional comments we will receive in the following days. In Table R1 below, the ±\pm is the sd over seeds. In this reply, H-ViT is the hyperbolic ViT-S/8 and D4D_4-RN50 the D4D_4-equivariant ResNet-50. Pairs are kk-NN / linear top-1 on ImageNet-1K, and transfer is the linear top-1 averaged over seven other datasets.

W1: Training settings. Relative to LeJEPA at its published settings, LocaLeJEPA gains +17.3 / +17.6 on ViT-S/8. The reviewer is right to request additional isolated components evaluation, and here we try to be as transparent as possible. In particular, we now compare at both the same settings and effective regularization strength λN/K\lambda N/K, i.e., the weight times the number of samples per neighborhood (N=64N=64 samples in each clustering step, see Table R1). Our training settings, with λ=0.1\lambda=0.1, lift LeJEPA by +6.1 / +7.1 on ViT-S/8. At λ=0.1\lambda=0.1, though, its regularizer is four times stronger than ours. For this reason, we also lower its weight to 0.025, and this gains it another +7.6 / +7.0. Overall, at completely matched strength and settings, LocaLeJEPA still consistently leads on all three encoders and every ViT-S/8 seed (lowest seed +3.5 / +3.2). At 128 px, the published settings are those of LeJEPA's Imagenette example, carried to ImageNet-1K.

Table R1. ImageNet-1K, kk-NN / linear top-1, ViT-S/8 averaged over three seeds, the other encoders one seed each. Differences are taken before rounding.

ViT-S/8H-ViTD4D_4-RN50
LeJEPA, published settings, λ=0.02\lambda=0.0232.8 / 44.632.6 / 44.034.5 / 43.4
LeJEPA, our settings, λ=0.1\lambda=0.1 (strength 6.4)38.9 / 51.639.4 / 51.639.7 / 48.8
LocaLeJEPA K=1K=1, λ=0.1\lambda=0.1 (strength 6.4)45.4 / 59.745.7 / 59.543.9 / 52.4
LeJEPA, our settings, λ=0.025\lambda=0.025 (strength 1.6)46.4 / 58.746.5 / 58.542.9 / 51.0
LocaLeJEPA K=1K=1, λ=0.025\lambda=0.025 (strength 1.6)47.7 / 59.848.0 / 59.944.5 / 52.4
K=4K=4 with random neighborhoods, λ=0.1\lambda=0.1 (strength 1.6)46.4 / 59.146.6 / 59.043.4 / 51.7
LocaLeJEPA K=4K=4, λ=0.1\lambda=0.1 (strength 1.6)50.1 / 62.150.6 / 61.947.5 / 55.2
LocaLeJEPA K=4K=4 minus LeJEPA, both at strength 1.6+3.6 / +3.5+4.0 / +3.3+4.7 / +4.2
the same, on transfer+2.3+2.3+3.6

W2: Local SIGReg control. We agree, and we now separate the four parts the reviewer names. At matched strength and settings, the centered ball at K=1K=1 adds +1.3 / +1.1 over LeJEPA, and locality (K=4K=4 over K=1K=1) another +2.3 / +2.3. Table R2 below separates centering from the target's shape. Replacing the ball with a centered Gaussian moves kk-NN and linear by at most 0.6 and transfer by at most 0.5 (K=1K=1 on three encoders, K=4K=4 on two, one seed each). Removing the centering from that local Gaussian costs 15.3 / 14.3 on H-ViT (one seed). Overall, centering and locality carry most of the effect. We note that this is aligned with our theoretical discussions in Section 3, where we believe the heat flow distribution to be the most salient aspect, and the particular choice of the ball as our local distribution is more so suggested by entropy-maximization heuristics of prior works.

K=1K=1 against K>1K>1. K=1K=1 is now reported at ImageNet-1K on all four encoders (Table R1 above). At matched strength, K=4K=4 adds +2.3 / +2.3 over K=1K=1 on ViT-S/8 (three seeds, lowest seed +2.1 / +2.0), +2.6 / +2.0 on H-ViT and +3.0 / +2.8 on D4D_4-RN50, and it stays positive at every tested depth (on ViT-S/8 with 6 and 3 blocks instead of 12, +1.7 / +1.9 and +1.2 / +0.6, one seed each). On Imagenette, that is a much easier dataset and results in easier downstream tasks, the reviewer is right, i.e., with λ\lambda tuned per KK, K>1K>1 and K=1K=1 tie.

AuthorsOpenReview ↗

Response to reviewer uyJV, comment 2 of 2

Q1: Neighborhoods. Each step has 256 images in four shards of 64, and KK-means runs within each shard, so a neighborhood at K=4K=4 holds about 16 images. They are not classes (a shard from 1000 classes holds about 2 same-class pairs). They are images that sit close in the current embedding, and they form reliably. In fact, in the training logs, we observe 3.97 of 4 neighborhoods per shard, 99.99% of samples kept, and no shard fell back to a single neighborhood. Although we did not measure their quality explicitly (we are open to suggestions), the random-partition test below still shows that they carry the gain.

Random partitions. Thank you for suggesting this test. Keeping KK-means but shuffling its labels, so that each 64-image shard has random neighborhoods of the same sizes, costs 3.7 / 3.0 on ViT-S/8 (three seeds, smallest drop 3.5 / 2.6), 3.9 / 2.9 on H-ViT and 4.1 / 3.5 on D4D_4-RN50. Random neighborhoods even fall below a single neighborhood at the same strength. So the gain needs data-dependent neighborhoods (Table R1 above gives the absolute numbers).

Q2: K=1K=1 on ImageNet-1K. K=1K=1 is in Table R1 on all four encoders, and in the revised manuscript we will insert the results at matched strength. On ViT-L/14, K=1K=1 reads 46.5 / 54.8 at LeJEPA's published ViT-L settings (λ=0.05\lambda=0.05, one seed).

Centered local Gaussian. Table R2 gives it at K=4K=4, under the same training settings as the ball: it performs like the ball, and on H-ViT the uncentered version falls far behind. On ViT-S/8 the ball is above it on 7 of 7 transfer datasets, but the Gaussian's average lies within the range of the ball's three seeds. Our position on the ball is in W2 above.

Table R2. K=4K=4, λ=0.1\lambda=0.1, one seed each: kk-NN / linear / transfer.

TargetViT-S/8H-ViT
ball, centered (LocaLeJEPA)50.11 / 62.11 / 72.4950.57 / 61.88 / 72.11
Gaussian, centered50.29 / 62.09 / 72.0150.70 / 62.45 / 72.52
Gaussian, not centered (LocaLeJEPA-SIGReg)not run35.35 / 48.15 / 62.83

We note we do not rerun the ViT-S/8 case corresponding to LocaLeJEPA-SIGReg already included in the previous experiments.

We sincerely thank the reviewer once again for their thoughtful and constructive feedback. We believe that the additional experiments and clarifications prompted by their suggestions strengthen our work. We kindly ask for further feedback, and we are happy to engage in more discussion.

I thank the authors for their efforts in addressing my concerns, therefore I raise my score.

Reviewer_g4MxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Joint-Embedding Predictive Architectures (JEPAs) have been demonstrated as promising self-supervised learning methods. In order to prevent the representations from collapsing, LeJEPA uses an additional objective to encourage the distribution towards an isotropic Gaussian, focusing on the Euclidean case. The paper instead proposes LocaLeJEPA, attempting to carry over the approach of LeJEPA to manifold valued representations. In LocaLeJEPA, the algorithms are encouraged to learn representations that lie on K different clusters, with each cluster encouraged to approach an uniform distribution on the sphere with some radius. Empirical evidence shows that the proposed method allows training models that have stronger downstream performance than the baselines.

Critical strengths

The paper studies a geometry-aware regularizer for JEPA, which is a timely research direction. I find the synergy of JEPA and Riemannian geometry interesting.

Critical weaknesses
  1. The paper cited quite some recent variants of LeJEPA. However, in practical experiments, the proposed LocaLeJEPA is only compared against the original LeJEPA, Rectifed LpJEPA and an artificial baseline, LocaLeJEPA-SIGReg.

  2. LeJEPA justified using isotropic Gaussian as it provably leads to good downstream performance. However, I did not find such guarantees stated for the proposed LocaLeJEPA method.

Questions for authors

Please see Critical Weaknesses.

  1. Generally, I am unconvinced regarding the proposed LocaLeJEPA method, which encourages the representations to lie uniformly on K clusters of spheres. There are generally 2 choices: 1. using K clusters, 2. make them uniformly on spheres. Can the authors motivate their design choice? From a theoretical perspective, does it lead to provably better performance, or is it largely empirical?

  2. In the experiments, the authors evaluate LeJEPA and LocaLeJEPA to recover the topological features of circle and torus. However, it is less clear to me whether this is meaningful, given that LeJEPA intends to learn an isotropic Gaussian, and LocaLeJEPA intends to learn K uniform spheres, since intuitively both the Euclidean space and K spheres should have different topology from a circle or a torus. Can the authors comment on this?

6 comments
AuthorsOpenReview ↗

Response to reviewer g4Mx, comment 1 of 2

We thank the reviewer for their feedback and are glad to know that they find our research direction timely and interesting. We would like to address each of the reviewer’s questions and potential weaknesses raised:

Q1: We would like to clarify this point, as LocaLeJEPA does not drive toward a uniform distribution on KK-disjoint spheres. Instead, it drives toward a uniform nn-ball (a sphere in contrast, is ‘hollow’) within each neighborhood cluster, i.e. locally, however, there is no global constraint on the distribution. Note that every manifold, by definition, is locally a nn-ball and unique global topological structures arise from the various ways these local structures are allowed to merge. Likewise, in our objective, the balls from each neighborhood cluster are allowed to merge freely, so the LocaLeJEPA objective is theoretically capable of learning any manifold. This is in contrast to the previous approaches in the literature, which are all globally formulated, and is the central insight of LocaLeJEPA.

As discussed in Section 3 on theoretical motivations, previous works on self-supervised learning have shown that downstream distributions should ideally 1. capture the structure of the underlying data manifold and 2. be entropy-maximizing. The combination of these objectives is what theoretically motivates the LocaLeReg objective, and it is backed up by our empirical performance gains.

Q2: We hope our clarification in Q1 helps better explain why LocaLeJEPA should be capable of learning a circle or torus. To clarify on the test-time torus inference task (Fig. 4), the models are not trained on the figurines dataset, but must be able to capture the dataset’s latent toroidal structure in order to perform well on the angle prediction task. In particular, performance on the task is not inherently tied to the structure of the downstream distribution used during learning and thus represents a fair comparison between LeJEPA and LocaLeJEPA. The results in Fig. 3 and Fig. 4 qualitatively show that LocaLeJEPA better learns the correct underlying structure, and this is established more quantitatively in Table 1 and Table 3.

AuthorsOpenReview ↗

Response to reviewer g4Mx, comment 2 of 2

W1: As also noted in our response to reviewer 4FDS, in addition to LeJEPA, we primarily consider comparisons to Sphere-JEPA, Rectified LpJEPA, and Sub-JEPA as they represent three distinct axes of improvement upon LeJEPA — use of geometric inductive bias (Sphere-JEPA), entropy-maximization (Rectified LpJEPA), and learning more general distributions (Sub-JEPA) — that we believe LocaLeJEPA provides a unifying perspective on. We first note that comparisons to all 4 are performed in our qualitative embedding tests (Table 2), We would also like to remark we include the synthetic LocaLeJEPA-SigReg as an ablation to show that the benefits of our method do not arise purely from locality but also from the change in target distribution from SigReg’s Gaussian to our uniform nn-ball.

To more thoroughly address the reviewer's concerns, we have since also performed additional experiments, which we will add as an appendix section to a revised version of the manuscript (to be updated before the review period ends). We trained additional variants of Rectified LpJEPA, now covering all three 128 px encoders at LeJEPA's published settings (one seed each), but find LeJEPA still leads it by 11.0 to 20.2 in kNN and 12.0 to 18.9 in linear. Most saliently, we trained SubReg (from Sub-JEPA) and SUSReg (from Sphere-JEPA) on ImageNet-1K on ViT-S/8, with several weights and one seed. At best SubReg reaches 24.6 / 41.6 and SUSReg 33.6 / 44.8, against 50.1 / 62.1 for LocaLeJEPA. At 100 epochs and 128 px, we remark that our implementations do not recover the published baselines of these methods, and these results should thus only be read as a relative but fair comparison in our setting.

W2: Our fundamental conceptual departure from LeJEPA is that we do not assume a Euclidean structure on the dataset or downstream task, which is the foundation of LeJEPA’s optimality proofs. Similarly, Sphere-JEPA instead proves that under a spherical assumption on the embedding space, the uniform distribution is optimal for downstream tasks. In section 3 of our paper, we argue that on an arbitrary manifold, the correct analog is the uniform distribution with respect to that manifold for optimal downstream performance. However, an arbitrary manifold does not have enough structure to support the proof techniques of LeJEPA and Sphere-JEPA (i.e. for an arbitrary manifold you would have to lift the linear regression arguments to local, tangent bundle-level arguments, where the necessary distributions may not be well-defined globally), and so direct analogs of those results do not exist at the level of generality we consider in the present work. As such, we instead rely upon thorough qualitative and quantitative tests of our model’s superior performance to validate our theoretical framework. Notably, we outperform LeJEPA even in the Euclidean setting where its theoretical optimality guarantees should apply, which we view as strong evidence of the manifold-theoretic perspective we put forth.

We sincerely thank the reviewer once again for their thoughtful and constructive feedback. We believe that the additional experiments and clarifications prompted by their suggestions strengthen our work. We kindly ask for further feedback, and we are happy to engage in more discussion.

Reviewer_4FDSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing work LeJEPA mitigates representation collapse via a provably robust distribution matching objective, which regularizes embeddings toward an isotropic Gaussian distribution. This work presents LocaLeJEPA, motivated by a manifold theoretic reformulation of the entropy maximization property originally from LeJEPA. Extensive experiments validate the effectiveness of the refined framework, in terms of both linear and k NN classification performance on ImageNet, as well as on a suite of downstream evaluation tasks.

Critical strengths
  1. The manuscript is well structured and well organized.
  2. LocaLeJEPA is designed to better preserve the intrinsic manifold structure of data, while being adaptable to non standard embedding geometries. These properties are empirically validated in experiments.
  3. The paper provides theoretical analysis, and empirical results demonstrate the effectiveness of the proposed approach.
Critical weaknesses
  1. The technical novelty is incremental: the proposed LocaLeJEPA builds upon and refines the existing LeJEPA framework, without substantial conceptual departures from the prior method.
  2. The best performing setting of LocaLeJEPA depends on different choices of hyper parameter K. The paper lacks sufficient discussion regarding the sensitivity and practical selection of this hyper parameter.
  3. The set of competing baselines is relatively dated; more recent state of the art self supervised representation learning approaches are missing from comparisons.
Questions for authors
  1. Why is k NN chosen as one of the evaluation classifiers? It is known that the choice of hyper parameter k can be non trivial. Is the fixed k used in experiments universally suitable across all datasets and downstream tasks considered in this paper?
  2. Can the proposed LocaLeJEPA adapt well to high dimensional feature spaces? What potential challenges would arise when embedding dimension grows large?
  3. The K means clustering is executed every training step on batch embeddings. How sensitive is training stability to batch size? Small batch sizes force small effective cluster size, which may degrade regularization quality.
4 comments
AuthorsOpenReview ↗

Response to reviewer 4FDS, comment 1 of 2

We thank the reviewer for their thorough engagement with our work and their valuable feedback. To address the reviewer’s questions and concerns,

Q1: We note that our evaluation protocol is consistent with the current standard in the self-supervised learning and JEPA literature, where papers typically report performance of a frozen backbone with respect to a linear probe and a kNN, for a fixed choice of k. For instance, this is the same protocol used in LeJEPA and Sphere-JEPA, and further, these are also precisely the two types of probes for which LeJEPA’s theoretical optimality results apply. Regarding the choice of k, we would like to refer the reviewer to Section C.4.2 of the appendix, and particularly Table 18. There we conduct precisely such an ablation over the choice of k, and find that LocaLeJEPA’s performance improvements are stable across all values of k considered, and also motivates our choice to report the k=20 case for the main body.

Q2: We note that there are two aspects to this question: the theoretical optimal embedding dimension and LocaLeReg’s robustness in potentially high dimensions. For the first, we remark that this is still an open question in the JEPA literature, but the general intuition is that the embedding dimension should be commensurate with the intrinsic dimension of the data manifold. Since the dimension of the data manifold is often quite small compared to the ambient dimension, it is likely that optimal downstream performance is actually achieved at relatively low embedding dimensions. For the second point, if we are in the case that requires a high-dimensional embedding for a particularly complex dataset or task, then we note that the fundamental algorithmic backbone of LocaLeJEPA – ie the Epps-Pulley test – is shared with LeJEPA, where it has been extensively tested for large models and large datasets. As our modifications to make the objective local and entropy maximizing still preserve the Epps-Pulley test, there is no reason for LocaLeJEPA not to inherit this robust performance in high-dimensional settings.

Q3: We would like to point the reviewer to Section B of our appendix, where we perform ablations on Imagenette in order to better understand the relationship between KK, λ\lambda, and batch size. In particular, aligned with the reviewer's intuition, Table 8, Table 10, and Table 12 of the Appendix collectively suggest that larger KK values are more effective when each cluster has access to more points in its neighborhood. Additionally, in Section A of the appendix, we consider the regularizer cost as a function of batch size in Table 7. We also note that the LocaLeReg algorithm (Algorithm 1) sets a minimum neighborhood threshold for this reason (we set nmin=4n_{min}=4 in practice), where we reject clusters that are below this threshold, and we find this sufficient to maintain stability during training. Our general intuition is that larger KK is better suited to larger and more complex datasets, which also agrees with our theoretical viewpoint wherein higher KK allows for a more complex embedding manifold. See also our response to W2 below for a further discussion of this point.

W1: We believe our work does represent a novel and significant conceptual departure from LeJEPA and previous work on the JEPA paradigm. On the theoretical side, while manifold-theoretic priors and entropy maximization have occasionally been considered in the SSL literature, we provide a unifying perspective by considering entropy maximization on an arbitrary manifold and seeing SSL training as heat flow. In this way, we gain access to techniques from optimal transport through the link between heat flow and Wasserstein gradient flow. We believe this perspective on SSL represents a novel insight, with potential applications beyond the JEPA paradigm. Within the JEPA paradigm, we then introduce the LocaLeReg objective. To the best of our knowledge, this is the first learning objective in the SSL literature that incorporates manifold-theoretic priors without fixing the base manifold to be e.g. a sphere. As we empirically demonstrate, this opens the path for the JEPA paradigm to be used in conjunction with equivariant and hyperbolic backbones, thus incorporating geometric priors which have been demonstrated as incredibly beneficial in applications such as robotics. Further, it also broadens the scope of the JEPA paradigm to fundamentally non-Euclidean structures such as graph-valued data. Altogether, we view our key contributions as providing a novel manifold-theoretic framework for SSL more broadly, and the development of LocaLeReg, which allows for the principled extension of the JEPA paradigm to geometric and non-Euclidean settings, while also advancing the state of the art in the Euclidean case.

AuthorsOpenReview ↗

Response to reviewer 4FDS, comment 2 of 2

W2: We would like to again note that Section B of our appendix precisely considers ablations on Imagenette that serve to clarify the relationship between KK, λ\lambda, and batch size. The question of optimal KK is further considered in Section C of the appendix as well, where we consider a dynamically increasing KK across training epochs, which is aligned with our intuition of training corresponding to heat flow on a latent data manifold. Based on our results, we see that in practice, a higher KK is better suited to larger and more complex datasets. Further, the dynamic KK strategy also performs quite robustly, and we recommend this in settings where there is no a priori knowledge of the dataset that can be used to infer an optimal fixed choice of KK. Finally, to better understand the relationship between regularization strength, batch size, and K, we ran an additional experiment we will include in the revised manuscript, which we will upload before the end of the rebuttal period to make sure we integrate as many comments as possible. In particular, each cell of the (B,λ,K)(B,\lambda,K) grid (three pretraining seeds) is placed by its effective strength λN/K\lambda N/K, with NN the per-step batch. For each band, the table gives the number of cells and the minimum / median / maximum of the cell means (%), on the final-epoch online probe and on the post-hoc linear probe.

Table 13: Effective regularization strength orders the Imagenette grid (800 epochs, both per-step batches).

λN/K\lambda N/KRegimeCellsOnline probe min (%)Online probe median (%)Online probe max (%)Linear probe min (%)Linear probe median (%)Linear probe max (%)
<0.4< 0.4Collapse918.6345.5364.2033.0254.0668.56
0.40.4--0.640.64Degraded573.4181.0388.5676.6982.3988.88
0.640.64--88Healthy3385.6590.2591.5286.1790.0891.44
88--2020Declining877.3388.0490.5379.2088.3690.30
>20> 20Over-regularized564.4678.1387.4469.5379.9587.56

As such, we see that in practice, we can either use the relationships between λ\lambda and NN to determine an effective choice of KK, or simply rely on the dynamic KK approach, as is most suitable. We do acknowledge that ideally, we would have a purely theory-driven criterion for the selection of an optimal KK for a given dataset, and we view this as a key direction of follow-up work.

W3: Across our various experiments, we consider comparisons to LeJEPA, Sphere-JEPA, Rectified LpJEPA, and Sub-JEPA. We believe that these models are all quite recent (all except LeJEPA being models released in 2026), and these works represent three distinct axes of improvement upon LeJEPA — use of geometric inductive bias (Sphere-JEPA), entropy-maximization (Rectified LpJEPA), and learning more general distributions (Sub-JEPA) — that LocaLeJEPA provides a unifying perspective on. As such, we believe they collectively represent the existing state of the art in the JEPA paradigm for the image setting. We will also be providing a more thorough comparison to these baselines in the revised manuscript that incorporates suggestions from reviewers g4Mx and uyJV. Of course, if the reviewer recommends other contemporary baselines, within or outside the JEPA regime, we would be happy to add these experiments to the paper.

We sincerely thank the reviewer once again for their thoughtful and constructive feedback. We believe that the additional experiments and clarifications prompted by their suggestions strengthen our work. We kindly ask for further feedback, and we are happy to engage in more discussion.

Reviewer_4FDSOpenReview ↗

LocaLeJEPA: Maximum Entropy Representations under Geometric Priors

After careful review of the responses, most of my questions have been addressed. I suggest changing weak acceptance to accept.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment Summary - comment 1 of 2

Dear Reviewers, ACs, and PCs,

We thank the reviewers for their careful assessments and constructive feedback. We appreciate the recognition of LocaLeJEPA's simple and effective formulation, substantial empirical improvements, and limited computational overhead [uyJV], the timely integration of geometric priors and Riemannian geometry into the JEPA paradigm [g4Mx], and the well-motivated theoretical framework and extensive empirical evaluation across representation geometries [4FDS].

The reviews highlighted three central questions: whether the improvements arise from the proposed local distribution-matching mechanism rather than differences in training configurations; how locality, centering, and the reference distribution individually contribute to performance; and how the theoretical motivation translates into practical neighborhood selection and robustness across training settings.

To address these points, we conducted additional controlled experiments, expanded our comparisons with recent JEPA variants, and substantially extended the theoretical and empirical analyses. These additions are now integrated into the revised manuscript, with detailed results, ablations, and implementation protocols in the main text and appendices. We summarize the principal updates below.

Experimental Results and Controlled Comparisons

  • [uyJV; 4FDS] Training-matched LeJEPA comparisons (Section 5.1; Tables 2 and 22). We extended our ImageNet-1K evaluation to compare LeJEPA and LocaLeJEPA under identical optimization settings and matched effective regularization strength, alongside LeJEPA's published configurations. Across Euclidean ViT-S/8, hyperbolic ViT-S/8, and D4D_4-equivariant ResNet-50, LocaLeJEPA achieves +3.6, +4.0, and +4.7 percentage points in k-NN accuracy, and +3.5, +3.3, and +4.2 points in linear top-1 accuracy, respectively, over the training-matched LeJEPA baseline. These improvements also extend to seven downstream transfer datasets, with average gains of +2.3, +2.3, and +3.6 points (Section 5.2; Table 3). The ViT-S/8 results are averaged over three independent pretraining seeds. We further extended the evaluation to ViT-L/14 at 238-pixel resolution, demonstrating improvements on larger architectures (Table 2).

  • [uyJV; g4Mx] Isolating locality, centering, and distribution matching (Section C.4.3; Tables 22 and 23). We introduced additional controls separating the contributions of the proposed objective. At matched effective regularization strength, the centered uniform-ball objective with K=1K=1 already improves upon LeJEPA, independently of local partitioning. Increasing KK from 1 to 4 provides an additional +2.3 points in both k-NN and linear accuracy on ViT-S/8, with positive improvements across all three seeds. Random neighborhoods of matched sizes reduce performance by 3.7 and 3.0 points, respectively, establishing the importance of data-dependent neighborhoods. Crucially, additional controls show that centered Gaussian matching performs within 0.6 points of the uniform-ball target, whereas removing centering substantially degrades performance. Indeed, replacing the ball with a centered Gaussian moves KK-NN and linear by at most 0.6 and transfer by at most 0.5 (on three encoders, on two, one seed each). Removing the centering from that local Gaussian costs 15.3 / 14.3 on H-ViT (one seed). Overall, centering and locality carry most of the effect. We note that this is aligned with our theoretical discussions in Section 3, where we believe the heat flow distribution to be the most salient aspect, and the particular choice of the ball as our local distribution is more so suggested by entropy-maximization heuristics of prior works.

  • [g4Mx; 4FDS] Comparisons with recent JEPA variants (Section 5.1; Tables 2 and 24; Section C.2). We expanded the experimental comparison to include Rectified LpJEPA (RDMReg), Sphere-JEPA (SUSReg), and Sub-JEPA (SubReg), covering alternative geometric priors and distribution-matching formulations. Our additional ImageNet-1K experiments with SUSReg and SubReg on ViT-S/8 yield, at their best tested settings, 33.6% / 44.8% and 24.6% / 41.6% k-NN / linear accuracy, respectively, compared with 50.1% / 62.1% for LocaLeJEPA. These additional experiments use one pretraining seed, and the revised manuscript explicitly documents the implementation choices and limitations in reproducing the published baseline results.

AuthorsOpenReview ↗

General Comment Summary - comment 2 of 2

Neighborhood Selection, Robustness, and Evaluation

  • [4FDS; uyJV] Neighborhood size, batch size, and regularization strength (Section 4; Appendix B, Tables 8–13). We clarified the relationship between the number of neighborhoods KK, samples per clustering step NN, and regularization weight λ\lambda, identifying λN/K\lambda N/K as the relevant effective regularization-strength parameter. Our expanded Imagenette ablations reveal a broad empirically favorable regime, 0.64≤λN/K≤80.64 \leq \lambda N/K \leq 8, while insufficient or excessive regularization degrades performance. We further show that larger per-step batches support finer neighborhood partitions and that, when regularization is tuned separately, the best local and global settings are nearly tied on Imagenette. On ImageNet-1K, locality provides additional improvements at matched strength. Dynamically increasing KK during training also achieves competitive performance without requiring a fixed neighborhood count (Tables 2 and 11). These results provide practical selection guidelines while identifying a fully theory-driven choice of KK as an open question.

  • [uyJV; 4FDS] Neighborhood reliability, representation quality, and evaluation protocols (Sections C.4.2–C.4.3 and D.3). We clarified that clustering is performed independently within 64-image shards and verified that the neighborhoods remain consistently populated: on average, 3.97 of 4 neighborhoods are retained per shard, with 99.99% of samples retained. We further specified the frozen-feature k-NN and linear-probing protocols, including checkpoint selection and hyperparameter tuning. Additional neighborhood-consistency analyses demonstrate improved semantic organization of frozen representations across all three encoder geometries, with higher same-class consistency for the vast majority of ImageNet-1K categories (Tables 26–27; Figure 11).

  • [4FDS] Architectural robustness and computational efficiency (Table 20; Appendix A.4, Tables 6–7). We extended the evaluation across ViT-S/8 encoder depths, showing that the benefits of locality persist with 3, 6, and 12 transformer blocks, although their magnitude varies. We also provide a detailed computational analysis, confirming that LocaLeJEPA preserves the scalability of LeJEPA's projection-based regularization, with only 5.7% additional end-to-end training time in the representative configuration evaluated.

Theoretical Contribution and Geometric Interpretation

  • [g4Mx; 4FDS] Local entropy maximization and manifold geometry (Sections 3–4; Appendix E). We clarified that LocaLeJEPA does not constrain representations to lie on KK disjoint spheres. Instead, it encourages locally uniform distributions within neighborhoods whose centers remain unconstrained and whose global arrangements may overlap or merge. This distinction is central to our manifold-theoretic motivation: local Euclidean structure can accommodate globally nontrivial topology without prescribing a fixed global embedding distribution. We view this flexibility as underlying the strong performance of LocaLeJEPA when incorporating geometric priors, such as equivariant or hyperbolic embedding spaces. Additionally, our theoretical motivation rests upon a general manifold hypothesis wherein we consider a unifying entropy-maximization framework (Propositions 3.1–3.3). This framework is more general than minimax-optimality guarantees under Euclidean or spherical assumptions (as in LeJEPA or Sphere-JEPA), and we believe LocaLeJEPA’s ability to outperform LeJEPA even in the Euclidean setting is strong evidence of our manifold-theoretic viewpoint.

We believe these additional experiments, theoretical clarifications, and comprehensive manuscript revisions substantially strengthen the evidence for LocaLeJEPA's contributions and directly address the reviewers' principal concerns. We sincerely thank the reviewers again for their constructive engagement and welcome any further feedback or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.