acceptodds
Withdrawn from ICLR 2027

DiJEPA: Self-Supervised Learning via Diffusion in Latent Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Joint-Embedding Predictive Architectures (JEPAs) learn semantic representations by predicting the latent embeddings of the masked target view from the visible context. Existing approaches use either direct regression in a continuous embedding space or token-softmax prediction over quantized targets. While effective, direct regression does not explicitly model a target distribution, whereas token-based prediction performs distribution matching in a discrete space; both rely on carefully designed distribution regularization to stabilize training and improve representation quality. We introduce DiJEPA, a JEPA-style framework that formulates visual representation learning as diffusion-based prediction over an online projection of the encoder's representation space. DiJEPA jointly trains a Feature Auto-Encoder (FAE) with the encoder, mapping high-dimensional representations into low-dimensional continuous latents whose distribution is optimized throughout representation learning. A diffusion predictor then reconstructs target-view latents from student-view representations through denoising, providing a distributional prediction objective in a continuous latent space. On ImageNet-1K with a ViT-L backbone, DiJEPA achieves 83.24% top-1 linear probing accuracy under a non-distilled setting, competitive with 83.3% from DINOv3. The jointly learned FAE latents can also be used for image generation, achieving 2.74 gFID without CFG in 80 epochs of latent diffusion training. Our results establish diffusion-based prediction over evolving FAE latents as a new predictive objective alongside direct regression and distribution matching over discrete tokens for self-supervised visual learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Self-Supervised Visual Learning, Diffusion Models, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AvSWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces diffusion-based prediction for SSL. The author introduces FAE that compresses the latent features of vit. The diffusion decoder receives masked features from the student-->FAE and full features from the teacher --> FAE --> noised, and trains diffusion losses. The FAE trains in a reconstruction way to reconstruct ViT features instead of images. There is an additional pixel decoder that learns pixel reconstruction on FAE-reconstructed features to decode images. The paper reports experiments that compare RAE vs FAE and compare Dijepa vs varisous ssl method.

Critical strengths
  1. Novel idea on feature autoencoder. This can potentially reduce the conflict between reconstruction and representation objectives
  2. Papaer is easy to follow with extensive ablation studies.
Critical weaknesses
  1. The benefit of the more complex pretraining objective is insufficiently established. Table 1 reports lower kNN accuracy (80.3% versus the DINOv3 result of 82.2%) and similar linear probing accuracy. The method is based on Dinov3 and cannot achieve better performance. More directly, Tables 5–6 show lower ADE20K mIoU (43.54 versus 44.39) and worse NYUv2 depth errors. The representation of DiJEPA is clearly weaker than baseline DINOv3. Table 4 reports approximately 30% higher per-step pretraining time, although DiJEPA also learns the FAE during this stage and therefore avoids the separate post-hoc FAE stage. I do not see why we should choose this method rather than training DINOv3, then freezing the encoder and training the RAE decoder. The experiments provide limited insight into what diffusion itself contributes beyond demonstrating that the complete recipe works.

  2. Representation is degraded, but generation is not improved. RAE with DINOv2 ViT-B achieves gFID 2.16, the original FAE with DINOv2 ViT-G achieves 2.08, and DiJEPA with ViT-L achieves 2.74. Comparing RAE with FAE cannot show the benefit of FAE because of the model size difference. I want to see how FAE behaves with DINOv2-B. DiJEPA also shows generation performance degradation even with a larger backbone.

Questions for authors

I would like to see RAE vs FAE under the same DINOv2-B. Moreover, given the context that the training recipe degrades representation, I would like to see a performance improvement at generation. Otherwise, joint training with extra cost results in worse performance, which I do not think can benefit the research community.

If the authors find a better training config that can achieve strong generation results where the ViT is comparable with DINOv3 baseline, I may raise my score.

Reviewer_LdMZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a framework that changes the JEPA latent prediction target to diffusion loss, called DiJEPA. Combined with DINOV3 data augmentation tricks, the authors propose several regularizations for stablizing the training to avoid collapsing, focusing on loss scale and vector norm regularizations. DiJEPA achieves non-collapsed results on ImageNet1K classification.

Critical strengths
  1. The proposed anti-collapse training tricks are good.
  2. The experiments on image generation is appreciated.
  3. The engineering work on combining DINOv3 tricks and JEPA framework SSL is good and technically sound.
  4. The failure cases are highly appreciated.
Critical weaknesses
  1. I have concerns on the motivation and practicality of using diffusion as JEPA prediction target. It doesn't seem to provide another anti-collapse starting point for JEPA, as the experiments show that many tricks are needed for the diffusion-based JEPA to work. The opening sentence on "diffusion can fit multi-mode targets " is not well-supported by either empirical results or theories. The assumption is that the target encoder, which comes from a self-developed teacher, will encode the ground-truth multi-modal input faithfully and let the diffusion objective to model it, while there is no guarantee on that in this paper. From figure 2 it seems that diffusion loss alone doesn't provide any intriguing benefits.
  2. The original I-JEPA's goal was to avoid any hand-crafted data augmentation tricks and show that no-augmentation SSL could also work well. DiJEPA seems to introduce all the complicated data augmentation back. It brings the fairness concern in the comparison part, as introducing additional tricks would, arguably, improves model performance.
  3. The method works with too many hand-crafted heuristics (e.g. different power transforms for CLS and patch losses, timestep shift, scheduled KL weight). This makes the overall framework fragile and might not be able to scale. For example, SwiGLU fails to converge indicates that advanced deep learning architecture can be non-suitable for DiJEPA. A 128-dim FAE has noticable drop in performance. Relating to the first point, it could be that the overall design in using diffusion as JEPA objective is not meaningful in its own.
  4. The overall performance, despite all the tricks, does not provide meaningful improvement over other methods when they share settings. e.g. DINOv3. Moreover, there is no comparison on finetuning the model on ImageNet1K.
Questions for authors

Please see Critical Weaknesses.

Reviewer_FFSPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a variant of JEPA models that incorporates diffusion-based prediction of the output of the EMA teacher. In the proposed method, the high-dimensional outputs of the teacher Vision Transformer are mapped into low-dimensional continuous latents by a Feature Auto-Encoder, and a diffusion predictor learns the distribution of the continuous latents conditioned on the outputs of the student Vision Transformer, in the manner of diffusion models. Performance evaluation on classification and generation tasks, as well as an ablation study on several design choices are conducted on ImageNet-1K.

Critical strengths

The idea of incorporating diffusion-based prediction into JEPA models is potentially interesting.

Critical weaknesses
  1. The presentation quality of this paper appears to leave room for improvement.

    • The problem setup is not clearly summarized before the explanation of the pilot experiments and the proposed method, which significantly hinders the readability of the paper. The inputs and outputs, the method on which the proposed method is built, and the model architecture considered are not described in detail before the main discussion starts. Although this information can be inferred from various figures and descriptions throughout the paper, the current organization may significantly hinder the reader's understanding of the paper.
    • On page 6, there are summation symbols Σ\Sigma over random variables such as ∑ϵ∼N(0,1)\sum_{\epsilon \sim \mathcal{N}(0,1)}. Those are considered to be an abuse of notation and the expectation operator should be used instead.
    • Some notations are not clearly defined, such as Dec in Line 279 and logvar in Line 293. Although the meaning of some of these notations is inferable from context, explicit definitions would improve the clarity and readability of the paper.
  2. The improvement over the baseline DINOv3 does not appear significant. In the classification task shown in Table 1, the proposed method falls behind DINOv3. In the generation results shown in Table 2, the proposed method achieved a larger sFID and showed no improvement in precision or recall compared to the DINOv3 baseline. The semantic segmentation results in Appendix G also indicate that the proposed method does not provide improvement over DINOv3. This marginal performance improvement also raises questions about the motivation that explicitly modeling the target distribution in JEPA models leads to better representation, as no theoretical grounding supporting this motivation is provided. Further discussion of the theoretical rationale of the proposed method would help support the effectiveness of the proposed method.

Questions for authors

Regarding Weakness 2, could further discussion about theoretical grounding for the proposed method be provided?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.