- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces diffusion-based prediction for SSL. The author introduces FAE that compresses the latent features of vit. The diffusion decoder receives masked features from the student-->FAE and full features from the teacher --> FAE --> noised, and trains diffusion losses. The FAE trains in a reconstruction way to reconstruct ViT features instead of images. There is an additional pixel decoder that learns pixel reconstruction on FAE-reconstructed features to decode images. The paper reports experiments that compare RAE vs FAE and compare Dijepa vs varisous ssl method.
Critical strengths
- Novel idea on feature autoencoder. This can potentially reduce the conflict between reconstruction and representation objectives
- Papaer is easy to follow with extensive ablation studies.
Critical weaknesses
-
The benefit of the more complex pretraining objective is insufficiently established. Table 1 reports lower kNN accuracy (80.3% versus the DINOv3 result of 82.2%) and similar linear probing accuracy. The method is based on Dinov3 and cannot achieve better performance. More directly, Tables 5–6 show lower ADE20K mIoU (43.54 versus 44.39) and worse NYUv2 depth errors. The representation of DiJEPA is clearly weaker than baseline DINOv3. Table 4 reports approximately 30% higher per-step pretraining time, although DiJEPA also learns the FAE during this stage and therefore avoids the separate post-hoc FAE stage. I do not see why we should choose this method rather than training DINOv3, then freezing the encoder and training the RAE decoder. The experiments provide limited insight into what diffusion itself contributes beyond demonstrating that the complete recipe works.
-
Representation is degraded, but generation is not improved. RAE with DINOv2 ViT-B achieves gFID 2.16, the original FAE with DINOv2 ViT-G achieves 2.08, and DiJEPA with ViT-L achieves 2.74. Comparing RAE with FAE cannot show the benefit of FAE because of the model size difference. I want to see how FAE behaves with DINOv2-B. DiJEPA also shows generation performance degradation even with a larger backbone.
Questions for authors
I would like to see RAE vs FAE under the same DINOv2-B. Moreover, given the context that the training recipe degrades representation, I would like to see a performance improvement at generation. Otherwise, joint training with extra cost results in worse performance, which I do not think can benefit the research community.
If the authors find a better training config that can achieve strong generation results where the ViT is comparable with DINOv3 baseline, I may raise my score.