- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AhJEPA, a single-branch way to add contextual learning to heuristic-free JEPA losses so they capture more granular details. Each global view gets a copy with some patch tokens dropped, and the model learns to predict the clean CLS embedding of the global view from it, with stop-gradient on the target. Local views are aligned to their parent global view. There is no teacher network and no EMA. Section 3 gives some theory on contextual prediction and explains stop-gradient as limiting how much the target can adapt. The authors first compare CLS and patch reconstruction on ImageNet-100 with four losses (SIGReg, VISReg, SWD and LpJEPA) and choose CLS because patch reconstruction hurts LpJEPA. They then train ViT-B/16 on ImageNet-1K with VISReg and SWD, and test on classification, segmentation, depth estimation and LeWorldModel planning. Over VISReg, AhJEPA improves in-domain accuracy by 1%, OOD accuracy by 0.29%, ADE20K mIoU by 0.89 and NYU Depth v2 RMSE from 0.688 to 0.663. It also improves LeWorldModel by 4% on OGBench-Cube and 11% on Two-Room.
Critical strengths
-
This is a simple and broadly applicable design. AhJEPA keeps the one-branch training paradigm and adds only patch-token-dropped views with CLS reconstruction, so it can be plugged into different heuristic-free JEPA losses. The design study shows its use. 1. Patch reconstruction hurts LpJEPA, which creates sparse representations, whilst CLS reconstruction helps all four losses. 2. The ablation in Table 5 also shows the gain is not only from adding more views, although the compute does not match.
-
I like the wide evaluation. The method is tested on 22 datasets covering in-domain and OOD classification, segmentation, depth estimation and latent world model planning. The TwoRoom result for LeWorldModel is a large gain, and the improvements on VOC and NYU Depth v2 for VISReg suggest contextual learning helps dense tasks for these losses.
Critical weaknesses
-
I don't think the results support the claim that AhJEPA "consistently improves" JEPA methods. The OOD gain over VISReg is only 0.29% from a single run, so it may be noise. Some results also get worse, namely KITTI, Reacher and Aircraft. VISReg is the only loss with a vanilla baseline at ViT-B on ImageNet-1K. SWD has no vanilla row, and SIGReg and LpJEPA are only tested at ViT-S on ImageNet-100. I am also worried about fairness. The CLS variant gets its learning rate, λ and dropping ratio, while Vanilla shares the Patch setting. AhJEPA-LeWM is tuned, but the LeWM numbers seem to come from the original paper. Table 8 reports the best test accuracy over 13 learning rates, and the code picks the kNN checkpoint using the same validation set it is evaluated on. Table 1 also mixes Inet1K numbers taken from other papers with the authors' 10-epoch probes, and some rows use ViT-L or ViT-H.
-
I find the novelty limited. Dropping patch tokens and aligning the CLS of the masked view to the unmasked view is the main idea of Masked Siamese Networks (Assran et al., 2022), which is not cited. The stop-gradient analysis in a shared encoder without EMA is also close to SimSiam (Chen & He, 2021) and Tian et al. (2021), which are not cited either. The theory itself is standard. Lemma 1 is the usual squared-error decomposition, Theorem 1 is the law of total variance, Proposition 1 is the chain rule, and Proposition 2 says that stop-gradient removes the target-side gradient. None of this explains why CLS reconstruction should help fine-grained tasks, so I think the claim to "mathematically characterize contextual representation learning" is too strong.
Questions for authors
Questions that could change my assessment:
- Please report the mean and standard deviation over at least three seeds for the VISReg comparison in Tables 1 to 3 and for AhJEPA-LeWM in Table 4. The OOD gain of 0.29% is hard to judge from one run.
- Is there a vanilla SWD baseline at ViT-B on ImageNet-1K? Full-scale results for SIGReg and LpJEPA would also help, since the paper claims AhJEPA is broadly applicable.
- Were Vanilla and LeWM tuned as much as AhJEPA? I would like to see vanilla VISReg trained with the same learning rate, λ and dropping ratio as the CLS variant.
- Were the linear probe learning rates and the pretraining checkpoint chosen on the test or evaluation set? If so, please report results with a separate validation split.
- How is AhJEPA different from Masked Siamese Networks, apart from not using EMA and prototypes? I would also expect SimSiam and Tian et al. (2021) to be discussed in the stop-gradient analysis.
- Why does AhJEPA make KITTI worse while it improves NYU Depth v2?
- The conclusion says AhJEPA reduces the computational overhead compared to the popular MIM methods. How does its training cost compare with iBOT? Table 5 shows 16% more FLOPs and 27% more memory than the vanilla baseline.
Minor: in train.py, weight decay is turned off for every parameter with "proj" in its name, which also includes the attention out_proj layers. Is this intended? Also, Appendix A.4 says 15 datasets but the main text says 22.
Reference:
- Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked Siamese Networks for Label-Efficient Learning. In ECCV, 2022. arXiv:2204.07141.
- Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In CVPR, pp. 15750–15758, 2021. arXiv:2011.10566.
- Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding Self-Supervised Learning Dynamics without Contrastive Pairs. In ICML, PMLR 139, pp. 10268–10278, 2021. arXiv:2102.06810.