acceptodds
Under review as a conference paper at ICLR 2027

AhJEPA: A single branch framework that captures granular details for JEPA methods

Abstract

Regularization-based self-supervised learning methods enable stable training without additional heuristic designs. However, such methods do not encourage learning of fine-grained relationships, which limits their applicability on downstream tasks. We mathematically characterize contextual prediction from incomplete observations and derive its implementation through a shared backbone, without a separately parameterized teacher or EMA updates. Our analysis interprets stop-gradient (SG) as a restriction on target-side adaptation, and an exploratory complementary-subset objective illustrates a possible implementation without SG. Inspired by previous masked-image-modeling (MIM) methods, we propose AhJEPA, a simple yet effective architecture that is broadly Applicable to heuristic-free JEPA losses to encourage learning of fine-grained relationships of the input data while maintaining the training simplicity and stability. Similar to previous works, we add a reconstruction loss to force the model to learn the fine-grained details. Different from them, AhJEPA maintains a one-branch training paradigm and applies the reconstruction loss to CLS tokens rather than patch tokens. We retain SG in the final method for its stronger empirical performance. We find that this contextual learning approach is a more general way to improve JEPA methods with different special designs (e.g., sparsity creation). This simple method consistently improves the performance of heuristic-free JEPA regularization methods across in-domain and out-of-domain classification, segmentation, depth estimation, and latent world model planning. Specifically, it helps the VISReg method achieve the best OOD accuracy and reduce the gap between heuristic-free methods and iBOT on fine-grained tasks. Also, it boosts the LeWorldModel performance to SOTA on OGBench-Cube and Two-Room datasets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, World Model, Contextual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7vJJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AhJEPA, a single-branch way to add contextual learning to heuristic-free JEPA losses so they capture more granular details. Each global view gets a copy with some patch tokens dropped, and the model learns to predict the clean CLS embedding of the global view from it, with stop-gradient on the target. Local views are aligned to their parent global view. There is no teacher network and no EMA. Section 3 gives some theory on contextual prediction and explains stop-gradient as limiting how much the target can adapt. The authors first compare CLS and patch reconstruction on ImageNet-100 with four losses (SIGReg, VISReg, SWD and LpJEPA) and choose CLS because patch reconstruction hurts LpJEPA. They then train ViT-B/16 on ImageNet-1K with VISReg and SWD, and test on classification, segmentation, depth estimation and LeWorldModel planning. Over VISReg, AhJEPA improves in-domain accuracy by 1%, OOD accuracy by 0.29%, ADE20K mIoU by 0.89 and NYU Depth v2 RMSE from 0.688 to 0.663. It also improves LeWorldModel by 4% on OGBench-Cube and 11% on Two-Room.

Critical strengths
  1. This is a simple and broadly applicable design. AhJEPA keeps the one-branch training paradigm and adds only patch-token-dropped views with CLS reconstruction, so it can be plugged into different heuristic-free JEPA losses. The design study shows its use. 1. Patch reconstruction hurts LpJEPA, which creates sparse representations, whilst CLS reconstruction helps all four losses. 2. The ablation in Table 5 also shows the gain is not only from adding more views, although the compute does not match.

  2. I like the wide evaluation. The method is tested on 22 datasets covering in-domain and OOD classification, segmentation, depth estimation and latent world model planning. The TwoRoom result for LeWorldModel is a large gain, and the improvements on VOC and NYU Depth v2 for VISReg suggest contextual learning helps dense tasks for these losses.

Critical weaknesses
  1. I don't think the results support the claim that AhJEPA "consistently improves" JEPA methods. The OOD gain over VISReg is only 0.29% from a single run, so it may be noise. Some results also get worse, namely KITTI, Reacher and Aircraft. VISReg is the only loss with a vanilla baseline at ViT-B on ImageNet-1K. SWD has no vanilla row, and SIGReg and LpJEPA are only tested at ViT-S on ImageNet-100. I am also worried about fairness. The CLS variant gets its learning rate, λ and dropping ratio, while Vanilla shares the Patch setting. AhJEPA-LeWM is tuned, but the LeWM numbers seem to come from the original paper. Table 8 reports the best test accuracy over 13 learning rates, and the code picks the kNN checkpoint using the same validation set it is evaluated on. Table 1 also mixes Inet1K numbers taken from other papers with the authors' 10-epoch probes, and some rows use ViT-L or ViT-H.

  2. I find the novelty limited. Dropping patch tokens and aligning the CLS of the masked view to the unmasked view is the main idea of Masked Siamese Networks (Assran et al., 2022), which is not cited. The stop-gradient analysis in a shared encoder without EMA is also close to SimSiam (Chen & He, 2021) and Tian et al. (2021), which are not cited either. The theory itself is standard. Lemma 1 is the usual squared-error decomposition, Theorem 1 is the law of total variance, Proposition 1 is the chain rule, and Proposition 2 says that stop-gradient removes the target-side gradient. None of this explains why CLS reconstruction should help fine-grained tasks, so I think the claim to "mathematically characterize contextual representation learning" is too strong.

Questions for authors

Questions that could change my assessment:

  1. Please report the mean and standard deviation over at least three seeds for the VISReg comparison in Tables 1 to 3 and for AhJEPA-LeWM in Table 4. The OOD gain of 0.29% is hard to judge from one run.
  2. Is there a vanilla SWD baseline at ViT-B on ImageNet-1K? Full-scale results for SIGReg and LpJEPA would also help, since the paper claims AhJEPA is broadly applicable.
  3. Were Vanilla and LeWM tuned as much as AhJEPA? I would like to see vanilla VISReg trained with the same learning rate, λ and dropping ratio as the CLS variant.
  4. Were the linear probe learning rates and the pretraining checkpoint chosen on the test or evaluation set? If so, please report results with a separate validation split.
  5. How is AhJEPA different from Masked Siamese Networks, apart from not using EMA and prototypes? I would also expect SimSiam and Tian et al. (2021) to be discussed in the stop-gradient analysis.
  6. Why does AhJEPA make KITTI worse while it improves NYU Depth v2?
  7. The conclusion says AhJEPA reduces the computational overhead compared to the popular MIM methods. How does its training cost compare with iBOT? Table 5 shows 16% more FLOPs and 27% more memory than the vanilla baseline.

Minor: in train.py, weight decay is turned off for every parameter with "proj" in its name, which also includes the attention out_proj layers. Is this intended? Also, Appendix A.4 says 15 datasets but the main text says 22.

Reference:

  1. Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked Siamese Networks for Label-Efficient Learning. In ECCV, 2022. arXiv:2204.07141.
  2. Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In CVPR, pp. 15750–15758, 2021. arXiv:2011.10566.
  3. Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding Self-Supervised Learning Dynamics without Contrastive Pairs. In ICML, PMLR 139, pp. 10268–10278, 2021. arXiv:2102.06810.
Reviewer_RNBaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AhJEPA, which introduces contextual learning into distributionally regularized JEPA methods through dropped-view CLS reconstruction and parent–local alignment with a shared encoder, demonstrating improvements in classification, dense prediction, and selected planning tasks.

Critical strengths
  1. The paper provides a simple training component with practical value. Contextual representation learning is implemented using a shared encoder and interchangeable regularizers, without maintaining an additional EMA teacher. This makes the approach suitable for integration into existing distributionally regularized JEPA pipelines. Simplicity does not diminish the contribution: if the improvements are reliable, an easily reusable component of this kind offers clear value to the community.

  2. The improvements extend beyond global classification to tasks directly relevant to the motivation. Relative to VISReg, AhJEPA improves ADE20K by 0.89 mIoU and Pascal VOC by 1.94 mIoU, while reducing NYU Depth v2 RMSE from 0.688 to 0.663. These frozen-encoder evaluations provide empirical support for improved local and spatial representations. The Two-Room planning result also merits further validation.

  3. The design study identifies meaningful boundaries of applicability. Methods producing dense representations generally benefit more from patch reconstruction, whereas LpJEPA deteriorates substantially under the evaluated patch-based configuration. The authors consequently select the more broadly reliable CLS approach without claiming that it universally outperforms patch reconstruction. The analysis of target-side adaptation under stop-gradient also provides a testable optimization-based interpretation.

Critical weaknesses
  1. Potential test-set selection bias and unclear metrics. Table 8 combines a search over 13 learning rates with reporting “best test accuracy.” If test performance determines hyperparameters or checkpoints, the results incur selection bias, particularly concerning for the 0.29-point cross-domain gain. The selection protocol requires clarification. ChestXRay is a multi-label task, yet its reported “accuracy” is not defined.

  2. Improvements are not sufficiently isolated. Table 5 controls view count, but AhJEPA still uses approximately 16.5% more FLOPs and 26.9% more memory. The method simultaneously changes view construction, local-target pairing, regularization inputs, and loss weighting. Matched ablations and equal-compute comparisons are needed to identify the contribution of dropped-view CLS reconstruction. Main-table baseline provenance and training configurations also require clarification.

  3. Positioning against closely related work is incomplete. MSN already performs masked CLS alignment through token dropping, while VICRegL combines global and local learning within a regularized framework. Neither is discussed. These methods differ from AhJEPA, but their omission leaves the incremental contribution insufficiently established. Focused comparisons should clarify the value of the shared-encoder, no-EMA implementation across regularizers.

  4. The proposed mechanisms remain insufficiently validated. The theoretical derivations appear valid under their assumptions, but establish predictability rather than preservation of useful fine-grained semantics. The sparsity explanation is confounded by differences in prediction heads, normalization, and masking; SG comparisons also use different view allocations. These interpretations should remain hypotheses unless supported by controlled experiments.

Questions for authors

The following are clarification requests or focused validation requests whose answers could materially affect my assessment. Suggestions for future extensions are provided separately in Part 2.

  1. Hyperparameter selection: What exactly does “best test accuracy” in Table 8 mean? Which splits determine the learning rate, training duration, and checkpoint selection? What are the label-processing procedure, training loss, and precise accuracy definition for ChestXRay?
  2. Attribution of improvements: With views, regularization, loss weights, and training configurations held fixed, how much improvement remains when dropped-view CLS alignment is removed or local alignment to the global-view mean is restored? Does the advantage persist under an equal total FLOPs budget?
  3. Statistical reliability and planning protocols: How many independent training runs underlie Tables 2–4? How many planning episodes are evaluated, and do methods share start–goal pairs and CEM/MPC budgets? Was LeWM given the same learning-rate and regularization-weight search budget as AhJEPA-LeWM?
  4. Contribution relative to prior work: Compared with MSN’s masked CLS learning and VICRegL’s global/local regularized learning, which objectives, constraints, or empirically established findings are new? Which results most directly demonstrate the value of these differences?
  5. Mechanism validation: Are there SG-on/off experiments with identical view allocations, and CLS-versus-patch comparisons that control sparsity, prediction heads, and normalization? If not, would the authors agree to present the corresponding causal explanations as hypotheses requiring further validation?
Reviewer_m8c9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AhJEPA adds contextual prediction to regularization-based JEPA methods. It uses a shared encoder to align CLS representations from token-dropped and local views with their parent global view. The final method retains stop-gradient for dropped-view targets but requires no separate teacher or EMA. The paper analyzes the reconstruction objective and evaluates classification, segmentation, depth estimation, and world-model planning.

Critical strengths

1 This paper is easy to follow and understand.

2 The shared-encoder design is simple and works with several regularizers. It adds contextual prediction without introducing a separate teacher or EMA updates.

3 Evaluation covers several task families. The improvements on NYU Depth v2 and Two-Room planning show that the method can benefit tasks beyond classification.

Critical weaknesses

1 AhJEPA combines token dropping, CLS alignment, and existing JEPA regularizers in a shared encoder. The theoretical results mainly describe the reconstruction objective and its gradients. The contribution is largely an adaptation of masked representation prediction.

2 The motivation for choosing CLS reconstruction is not fully convincing. Figure 2 shows better segmentation results with patch reconstruction for SWD, SIGReg, and VISReg. Yet the full-scale experiments use only SWD and VISReg, without a patch-reconstruction comparison.

3 The improvements are uneven. Average OOD accuracy increases by only 0.29 percentage points. For VISReg, KITTI depth δ1 decreases from 0.819 to 0.798. Reacher success also drops from 86 to 84. The VISReg-based model remains below iBOT on ADE20K and VOC segmentation.

4 The experiments do not fully separate the method’s benefit from extra computation. In Table 5, AhJEPA uses 16% more FLOPs and 27% more peak memory than the baseline with the same number of views. The segmentation gain is 0.32 mIoU. A comparison under equal training compute is needed.

Questions for authors

1 How does CLS reconstruction compare with patch reconstruction under the full ImageNet-1K settings used for SWD and VISReg?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.