acceptodds
Under review as a conference paper at ICLR 2027

Scalable Patch-Level Self-Supervised Learning

Abstract

Self-supervised learning (SSL) at scale produces powerful visual representations. However, most scalable SSL methods rely on ad hoc combinations of multiple objectives and stabilization mechanisms. Taking a step back, we ask if we can design a high-performing, yet principled SSL algorithm. Starting from the multi-view assumption–stipulating that task-relevant content is captured by the information common to different views–we construct an information-theoretic objective decomposing into interpretable terms. This derivation yields JEM, a student–teacher method that learns by aligning corresponding patch representations across views, explicitly regularized by information and structure preservation losses. JEM trains stably from 300M to 7B parameters, and, to our knowledge, is the first latent-space patch-level method demonstrated at 7B scale. Across all scales, JEM reaches strong performance on both global and dense probing tasks, on segmentation benchmarks consistently surpassing the DINOv2 algorithm–an influential foundation for today’s strongest visual SSL methods. Notably, at 7B parameters, it also exceeds DINOv3's performance on panoptic segmentation, despite DINOv3 being trained on a 10x larger dataset with several refinement stages. These results demonstrate that we can indeed build an SSL algorithm which is principled and stable, without sacrificing downstream performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, vision, information theory, masked prediction, multi-view InfoMax

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_QEEQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce JEM, a patch-level self-supervised learning method for computer vision. The framework is grounded in information theory: in a multi-view setting, it maximizes a lower bound on the mutual information between the representation of a student network and the target produced by a teacher network, a student–teacher paradigm shared with other self-supervised methods (e.g., BYOL, DINO).

The key modeling choice is to treat the target as a collection of elements, drawn from different patches of the view and from different layers of the network, so that the training signal is both spatially local and includes several levels of representation. From the resulting bound, the authors derive three objectives: aligning student and teacher predictions across views, preventing global collapse, and preventing structural collapse (where all patch-level targets encode the same information).

The experiments evaluate the method on standard computer vision tasks: image classification, semantic, instance and panoptic segmentation, and monocular depth estimation. JEM outperforms a DINOv2 baseline retrained under identical conditions on the dense prediction tasks, while remaining competitive on classification, at the same training cost.

Critical strengths
  • JEM trains stably up to 7B parameters with a patch-only objective and no class token, and it consistently outperforms a controlled DINOv2 baseline on semantic, instance and panoptic segmentation, especially panoptic.
  • The experiments are broad: nine benchmarks, three model scales, recent state-of-the-art baselines, and a DINOv2 baseline retrained in the same codebase on the same data.
  • Deriving the objective from the multi-view InfoMax bound is well motivated and gives an interesting reading of DINO-family heuristics as implicit entropy controls.
  • Aligning student and teacher crops on a shared patch grid is a relevant patch-level adaptation of multi-crop that unifies cropping and masking, and the ablations show it contributes clear gains.
Critical weaknesses
  • Only the alignment term of Eq. (2) is optimized directly. The anti-collapse term is replaced by a student-side estimate and TC(T) by a similarity-matching proxy, so the optimized objective differs from what the theory asks for, and the “principled” claim should thus be tempered.
  • DINOv2* is trained with its native views, where the patch loss applies only to masked tokens of the same crop the teacher sees, whereas JEM aligns patches across shifted, independently augmented crops. Since Table 1 shows that these view choices drive large dense-task gains, DINOv2 should also be evaluated with JEM’s view construction to separate the contribution of the loss from the contribution of the augmentation strategy.
  • R_struct differs from DINOv3’s Gram anchoring, which matches final-layer patch similarities to an earlier checkpoint late in training rather than to intermediate teacher layers from the start. Both, to my understanding, regularize patch-similarity structure, so an empirical comparison would be expected. Also, overall in Section 3.2.3: it is not clear why such a loss optimizes the total correlation as defined in the supplementary material. More details about why this similarity-matching proxy is used could improve a lot the readability of this section. In particular, and again, its potential connection to DINOv3's Gram anchoring should be more explicit.
  • JEM is consistently about 1 point below DINOv2* on IN-1k, plausibly because it has no class token or image-level objective. The authors should discuss this gap and ideally evaluate the impact of adding a class token term to JEM.
  • The method is presented as simpler than multi-objective approaches, yet it relies on three loss terms, one of which combines two entropies, plus several weights and schedules whose choice and influence are not analyzed.
  • Section 3.2.2 does not clearly state what loss is actually minimized, or at least it is not clear to me. It should state either the term that is implemented and minimized or clarify it. In Figure 4: the global regularization weight alpha is, I believe, not defined
Questions for authors
  1. Can the authors evaluate DINOv2 with JEM's view construction (shifted, independently augmented, grid-aligned crops, with the loss on visible and masked patches) to separate the contribution of the loss from that of the views?
  2. How does R_struct compare empirically with DINOv3's Gram anchoring, and why does matching intermediate-layer similarities control the total correlation TC(T)?
  3. What exactly is minimized in Section 3.2.2, and on which network? What is alpha in Figure 4 exactly ?
  4. Does adding a class token as an additional element of T close the classification gap with DINOv2* without hurting dense performance?
  5. How were the loss weights, target layers, and schedules chosen, and how sensitive are the results to them?
4 comments
AuthorsOpenReview ↗

Response (1/4)

We thank the review for their positive assessment of our work and for their constructive comments. We answer their questions below, and will also include the following experiments and discussion into the paper.

Q: Can the authors evaluate DINOv2 with JEM's view construction (shifted, independently augmented, grid-aligned crops, with the loss on visible and masked patches) to separate the contribution of the loss from that of the views?

We thank the reviewer for suggesting this experiment, which helps to separate the contribution of the views from that of the loss. We performed two experiments: training DINOv2 with JEM's view construction, and additionally including a CE loss on masked and visible patches on global/local crops. The results are in the table (same setting as Tab. 1: ViT-L, IN22k, 250k steps):

IN1kADE20k
JEM82.848.1
DINOv284.046.9
DINOv2 with JEM views83.547.6
DINOv2 with JEM views + masked/visible loss on global/local crops82.945.4

In JEM, student and teacher crops are shifted, but aligned on a shared patch grid. Enforcing patch-level alignment across shifted views, with independent augmentation, benefits dense tasks: DINOv2 improves on ADE20k by 0.7 mIoU using JEM’s view creation. This suggests JEM’s view creation can benefit other methods. JEM still outperforms the best DINOv2 variant by 0.5 mIoU, showing our framework's advantages. On IN-1k, JEM views reduce DINOv2's accuracy by 0.5 points. We hypothesize that grid-aligned crops remove the scale variations between local student and global teacher crops present in DINOv2’s views. These scale variations promote broader local-to-global correspondences and scale invariance, benefiting image-level classification. Surprisingly, using the full patch-level loss on visible and masked patches degrades both tasks. In contrast, both visible and mask losses improve JEM's performance (Table 1b). This may occur because visible patch loss acts as a regularizer; this mechanism might be less crucial for DINOv2.

AuthorsOpenReview ↗

Response (2/4)

Q: How does R_struct compare empirically with DINOv3's Gram anchoring, and why does matching intermediate-layer similarities control the total correlation TC(T)?

We thank the reviewer for the questions. The relation to Gram anchoring is interesting, and we discuss it in App. A.1 ("Structural regularization and Gram anchoring"). We will reference this discussion in Sec. 3.2.3 in the main text to make the connection more explicit.

Comparison with Gram anchoring. Both methods use pairwise patch similarities as targets, but they were designed to address different problems and are implemented differently.

  • Purpose. Gram anchoring addresses the local patch structure becoming increasingly noisy during long training. R_struct addresses the redundancy term TC(T) of the anti-collapse objective, i.e., structural collapse, where all patches of an image encode the same image-level content.
  • Targets. Gram anchoring requires selecting an earlier checkpoint as Gram teacher and a separate forward pass through it. R_struct reuses precomputed intermediate EMA teacher layers, requiring no extra forward pass. As noted in Appendix A.3.4, R_struct can also use targets directly from student representations, though using teacher features proved slightly more effective. Furthermore, Gram anchoring directly ties the Gram matrix of the final-layer features to the target. In R_struct, the similarities are matched after a learned transformation. The final representation is therefore not forced to replicate early-layer similarities and remains free to encode higher-level semantic structure. 
  • Schedule. Gram anchoring is a separate refinement stage. R_struct can be applied from the start of training, simplifying training pipelines and recipes.

Empirically, the table below compares R_struct with Gram anchoring applied from 400k iterations. Both runs are ViT-g on the A140M dataset, evaluated at 500k iterations. Gram anchoring uses the 200k checkpoint as Gram teacher and follows the recipe given in DINOv3. We find that Gram anchoring broadly has a similar effect to R_struct, with improvements on ADE20k (+0.4 mIoU) but drops on IN1k (-0.3 accuracy). JEM’s structural regularizer thus could serve as a computationally cheaper alternative to Gram anchoring; replacing it in DINOv3 is an interesting direction for future work.

IN1kADE20k
Structural Reg85.652.3
Gram anchoring from 400k85.352.7

Intermediate-layer similarities and TC(T). We agree that Section 3.2.3 reads dense due to space considerations. We clarify here and will improve this section in paper.

Structural collapse and intermediate layers. The issue the structural regularizer resolves is the “structural collapse”: where all patch targets T share identical information, maximizing TC(T). On an intuitive level, this can be prevented by avoiding that the patches collapse to a single (redundant) vector. Instead of simply minimizing the redundancy between patches, the structural regularizer imposes a certain (non-redundant) dependency structure, for which it uses the Gram matrix of the features of intermediate layers. This is because natural images have a strong dependency structure, and the targets should reflect this. Because we don’t want to impose a single (potentially incorrect) dependency structure, we match the structure of several intermediate layers through learned projection heads. This assumes intermediate layers are non-collapsed and capture distinct structural levels (e.g., textures at patch embeddings, part-level structure mid-network).

Gram matching and TC(T). Assume that, in both the intermediate and final layers, each patch contains the same image-wide signal and its own independent local details. In a simple Gaussian model with many independent feature dimensions, copying more of the shared signal into every patch makes the patches point in more similar directions and carry more redundant information. By matching the Gram matrix of the final layer to an intermediate layer, the structural regularizer thus limits how much more redundant the final patch features can become. Then, if the teacher assigns a category T to each patch using only that patch’s matched feature, the head can discard redundancy but cannot add more. The patch redundancy of the intermediate layers then puts a ceiling on the total correlation of the teacher’s targets TC(T) (hence “controlling” TC).

This argument relies on several assumptions. To empirically validate it, we measure structural collapse by tracking the fraction of images per batch where the teacher assigns every patch to the exact same category. Using the models from Fig. 5 (right), we find that without R_struct, 8.8% of images are collapsed, whereas with R_struct, only 4.0% are collapsed. This is consistent with the prevention of loss of spatial diversity in Fig. 5 (right).

AuthorsOpenReview ↗

Response (3/4)

Q: What exactly is minimized in Section 3.2.2, and on which network? What is alpha in Figure 4 exactly ?

Section 3.2.2 minimizes R_global (Eq. 6), which corresponds to the negative estimated per-element mutual information (Eq. 5) averaged over patch-layer elements. Minimizing R_global maximizes this estimated mutual information. Optimization occurs strictly on the student network, with gradients flowing into its head and backbone.

Alpha in Fig. 4: We thank the reviewer for pointing out this typo. Alpha refers to w_global, the weight of the global regularization term R_global in the loss. Fig. 4 illustrates performance sensitivity to this weight on IN-1k and ADE20k.

Q: Does adding a class token as an additional element of T close the classification gap with DINOv2 without hurting dense performance?

We conduct a set of experiments on ViT-L where we add a CLS token with an image-level loss, i.e. alignment loss on CLS token between student and teacher crops. We vary the relative weight of this loss compared to the patch-level loss: thus, wcls=1w_\text{cls}=1 only has the image-level loss, and no patch-level loss. We find evidence of a trade-off between classification and dense performance: with equal weight for image and patch losses, classification improves over the patch-level-only JEM baseline (+0.4 IN1k), but segmentation decreases (-1 ADE). Interestingly, using the CLS loss alone strongly drops both classification (-1.0 IN1k) and segmentation (-7.1), justifying the use of the patch-level losses. Compared to the DINOv2 baseline, adding an image-level loss to JEM closes the gap on ImageNet by 0.4 points, while maintaining slightly stronger segmentation. Whether this trade-off stems from the same negative interaction between image and local losses as reported by DINOv3 is an open question.

That said, these findings from a single trial should be interpreted with caution. JEM’s setup and hyperparameters were tuned under the patch-level loss setup, and here we performed a drop-in addition of the CLS loss. Some tweaks may also further improve the classification performance (e.g. increasing the number of CLS prototypes, adding CLS cross-view losses). Further, we find evidence that JEM’s view construction may negatively affect classification performance; switching DINOv2 to JEM views decreases IN1k performance by 0.5 (see experiment above, DINOv2 with JEM views). Thus, we believe the performance of the combined image-patch loss JEM model can likely be improved further; studying this is an interesting direction for future work.

We would also like to highlight that the patch-level objective used in DINOv2 (iBOT) obtains much worse performance on IN1k when not combined with the DINO objective (60 accuracy vs 73.8 with DINO on ImageNet, Table 9 of the iBOT paper [A]). We consider this to be a key advantage of JEM: it relies on a single objective trained on patches, yet succeeds in yielding strong global representations.

CLS weightIN1k classificationADE20k segmentation
JEM, w_cls=0.0 (only patch-level loss, JEM)82.848.1
JEM, w_cls=0.583.247.1
JEM, w_cls=1.0 (only image-level loss)81.841.0
DINOv2 baseline84.046.9

Experimental details: we train ViT-L on IN22k for 250k steps matching JEM’s ablation setup. The target of the CLS token for the local and global student crops is the corresponding teacher crop. The mutual information regularization terms are computed separately on patches and CLS token, using the weight w_cls to balance between the two. CLS and patch-losses share a head; we found little difference with a dedicated head for the CLS loss. The CLS loss is only computed on the final layer.

[A] Zhou et al. iBOT: Image BERT Pre-Training with Online Tokenizer. ICLR, 2022.

AuthorsOpenReview ↗

Response (4/4)

Q: How were the loss weights, target layers, and schedules chosen, and how sensitive are the results to them?

We agree with the reviewer on the importance of ablating hyperparameter choices and schedules. The main choices include the loss weights on masked vs. visible patches, the mutual information (MI) regularization weight (R_global), and the target layers and schedule for the structural regularizer (R_struct). We lightly tuned the MI weight and masked loss weight at ViT-L scale, transferring them unchanged to larger models. Target layers for R_struct were untuned, and its schedule was set to ramp up to counteract the structural collapse emerging progressively through training (Fig. 5). Below, we analyze sensitivity to each choice; we find JEM to be robust to these choices within a wide range of values.

Masked and visible alignment loss weights (ViT-L, IN-22k, 250k steps). Both terms are essential. Using only masked loss is catastrophic (70.8 IN-1k, 28.1 ADE20k), while using only visible loss costs 1.5 IN-1k points and 2.1 ADE20k mIoU. Other tested mixtures remain within 0.5 points on IN-1k (82.3–82.8) and 1.4 mIoU on ADE20k (46.7–48.1). The paper's ratio (0.8 masked, 0.2 visible) achieves the best performance across both benchmarks.

IN1kADE20k
w_mask=0, w_vis=1.0 (only vis. loss)81.346.0
w_mask=0.2, w_vis=0.882.546.9
w_mask=0.5, w_vis=0.582.547.5
w_mask=0.8, w_vis=0.2 (paper version)82.848.1
w_mask=0.9, w_vis=0.182.346.7
w_mask=1.0, w_vis=0 (only masking loss)70.828.1

Mutual information regularization weight. Results are robust to this weight (Fig. 4), requiring only a non-zero value; a weight of 0 causes collapse, whereas a broad range of non-zero values yields comparable performance.

Structural regularizer: target layers (ViT-g, A140M, 500k steps). The table compares different sets of target teacher layers .

  • Using multiple target layers is robust: every tested multi-layer set improves over no structural regularization by 0.4–0.7 IN-1k points and 1.8–2.2 ADE20k mIoU, with all choices yielding similar performance (84.8–85.1 IN-1k, 51.6–52.0 mIoU) across early, late, spread, and dense configurations.
  • A single target layer is less reliable: layer 0 improves ADE20k by 1.2 mIoU, whereas layer 10 reduces it by 1.4 mIoU. This supports similarity matching across multiple layers, encoding different structural levels.
  • The untuned default selection (0, 10, 20, 30) achieves the top IN-1k accuracy and falls within 0.3 mIoU of the best ADE20k score.
IN1kADE20k
No gram84.449.84
Target layer: 084.951
Target layer: 1084.848.4
Target layer: 0, 1084.952
Target layers: 0, 5, 10, 15 (first half)8551.7
Target layers: 20, 25, 30, 35 (second half)8551.9
Target layers: 0, 10, 20, 30 (full model, paper version)85.151.7
Target layers: 0, 5, 10, 15, 20, 25, 30, 35 (full model, densely spaced)84.851.6

Structural regularizer: weight and schedule (ViT-g, A140M, 500k steps). The specific schedule has little impact: a constant weight of 1 performs close to the 0.05–1 linear schedule (84.7 vs. 85.1 IN-1k, 51.6 vs. 51.7 ADE20k). What matters is the magnitude of the weight.

  • The weight must be sufficiently high to prevent structural collapse: ADE20k steadily rises from 49.8 (weight 0) to 50.4 (0.1), 51.2 (0.3), and 51.6 (1). This aligns with needing R_struct to preserve spatial diversity.
  • A much larger weight (3) over-constrains the representation and degrades both tasks (83.1 IN-1k, 48.9 ADE20k).
IN1kADE20k
w_struct=084.449.84
w_struct=0.18550.4
w_struct=0.384.851.2
w_struct=184.751.6
w_struct=383.148.9
w_struct=0.05 → 1.0 (paper version)85.151.7
Reviewer_ZL5YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JEM, a patch-level self-supervised learning method derived from multi-view information maximization objective. It combines cross-view patch alignment with global and dense regularization to prevent representation collapse. Experiments demonstrate stable scaling up to 7B parameters and strong performance across downstream tasks.

Critical strengths
  1. The paper achieves strong empirical results across multiple downstream benchmarks, particularly in dense prediction tasks, outperforming competitive baselines in several settings.

  2. The paper demonstrates favorable scaling behavior from 300M to 7B parameters, showing that scaling visual pretraining can translate into improved downstream performance, especially on dense prediction tasks.

Critical weaknesses
  1. The paper is related to prior self-supervised learning methods, for example, TWIST [R1] explores mutual information maximization, and I-JEPA[R2] learns visual representations through latent space prediction. The authors should distinguish the difference of JEM from these approaches, particularly in terms of its information-theoretic formulation and collapse prevention mechanisms.

  2. More ablation studies are needed to disentangle the contributions of data and the proposed method. Comparisons under matched data scale and computation budgets would establish whether the improvements primarily come from the proposed method.

[R1] Wang, F., Kong, T., Zhang, R., Liu, H., & Li, H. (2023). Self-supervised learning by estimating twin class distribution. IEEE Transactions on Image Processing, 32, 2228-2236.

[R2] Assran, Mahmoud, et al. "Self-supervised learning from images with a joint-embedding predictive architecture." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

Questions for authors

Refer to the weakness.

1 comment

We thank the reviewer for the positive assessment of our empirical results and scaling behavior, and for the constructive feedback.

W1: Relation to TWIST and I-JEPA

We thank the reviewer for raising this point. Both methods are already discussed in the paper, some in the appendix. We will make the discussion more clear in the main paper. I-JEPA appears in Sec. 2 and in App. A.1 ("Dense losses between partially overlapping views"). TWIST appears in App. A.1 ("Explicit entropy regularization of discrete assignments"), where we identify it as the closest image-level analogue of our Lalign+RglobalL_\text{align} + R_\text{global}. We summarize the key differences below and will make them more prominent in the main text.

TWIST differs from JEM in two main ways. First, TWIST is an image-level method, and its objective does not carry over directly to patch-level targets. Moreover, while TWIST uses similar explicit entropy regularization terms, the two objectives are derived differently. Specifically, TWIST starts from a cross-view alignment term and, to prevent collapse, adds sharpness and diversity entropy terms. It introduces these as intuitive objectives and then shows that together they equal I(X;Y)I(X;Y), the information between an image and its own class prediction (single-view InfoMax in the sense of Bridle et al., 1991). Consistency and anti-collapse are thus motivated separately. JEM instead starts from the multi-view principle, where the quantity of interest is the information shared across views. A single variational bound on I(R;T)I(R;T) (Eq. 2) yields both the alignment term and the anti-collapse term I(XT;T)I(X_T;T), as two parts of one objective. Our derivation also extends to collections of patch- and layer-level targets (Eq. 3), with consequences TWIST does not cover: per-position marginal entropies, i.e., estimating I(XT;T∣Pos)I(X_T;T|\text{Pos}), prevent positional collapse. The total-correlation term, which motivates RstructR_\text{struct}, addresses structural collapse: all patches encoding the same image-level content. Neither has a counterpart in TWIST, and RstructR_\text{struct} is needed to prevent the progressive loss of spatial diversity at ViT-g scale (Fig. 5; +2.7 ADE20k mIoU).

I-JEPA regresses continuous latents of masked target blocks and has no explicit anti-collapse term; collapse is avoided implicitly through architectural asymmetry (EMA teacher and predictor). In our framework, it optimizes only the alignment term of Eq. 2., on continuous targets. JEM makes the anti-collapse term explicit, and aligns all overlapping patches, both masked and visible, across views. In our ablation, dropping the visible patches costs 20 ADE20k mIoU (Tab. 1b). Overall, when comparing JEM and I-JEPA at the same scale and data (ViT-g, IN-22k), JEM outperforms I-JEPA by 6.7 points on IN-1k and 17.0 mIoU on ADE20k (Tab. 2).

We will move a condensed version of this discussion from the appendix into Sec. 2.

W2: On disentangling data and method

We thank the reviewer for this comment. The paper already includes the controlled comparison the reviewer asks and we summarize it below.

Matched comparison with DINOv2. We compare against DINOv2, our strongest baseline, under fully controlled conditions (Sec. 4.3–4.4; Tab. 2, rows marked DINOv2*; Figs. 6–7). We retrain the DINOv2 algorithm in our codebase with the same data, training steps, tokens seen, backbone and shared hyperparameters as JEM (App. A.3.2; Tabs. 3, 5). We do this in five settings:

  • ViT-L on IN-1k and on A140M
  • ViT-g on IN-22k and on A140M
  • ViT-7B on A140M

Results. In every setting, JEM outperforms DINOv2* on all six segmentation benchmarks (semantic, instance and panoptic). On the same data, the margin grows with model size. For example, on A140M the ADE20k panoptic gain goes from +3.0 PQ at ViT-L to +4.2 at ViT-g and +6.0 at ViT-7B (COCONut: +3.5, +4.7, +5.3). In exchange, IN-1k classification is 0.8–1.5 points lower, and depth estimation is comparable.

The gains do not come from data curation. We see the same trends on the public IN-1k and IN-22k datasets, not only on our curated A140M. Since data, backbone and training budget are matched, the remaining difference is the training objective. The ablations in Tab. 1 further show that each term of the objective improves performance.

Other SSL baselines. For the remaining methods in Tab. 2, we match pretraining data and model size:

  • ViT-L on IN-1k for MAE, MSN, data2vec 2.0 and CAPI
  • ViT-g on IN-22k for I-JEPA and Franca

All of these methods except I-JEPA are trained on more tokens than JEM, so they use a larger compute budget (Tab. 2). I-JEPA sees fewer tokens (0.12T vs. 1.39T), but it is on average 14.3 points below JEM across the six segmentation benchmarks (Tab. 2).

Reviewer_hjzqOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper starts from the multi-view assumption that information shared across different views is what matters for downstream tasks, and proposes JEM, an information-theoretic SSL method. Derived from a multi-view InfoMax lower bound, JEM formulates three core objectives: a patch-wise alignment loss between student and teacher networks, alongside global and structural regularizations that prevent representations from collapsing. The proposed method was successfully scaled to 7B parameters and demonstrated competitive performance across various segmentation tasks compared to DINOv2 and DINOv3 in probing evaluations using a fixed encoder.

Critical strengths
  1. Design and validation of a learning objective function based on information theory.
  • Rather than combining multiple independently designed learning objectives, JEM formulates its training objective based on a single information-theoretic principle, Multi-view InfoMax. The derivation and translation of this principle into practical learning objectives are well motivated. The ablation study further supports the effectiveness of the proposed design.
  1. Scalability to large-scale models and strong dense prediction performance.
  • JEM demonstrates impressive scalability, training stably from 300M to 7B parameters without structural collapse. Across all model scales, it consistently outperforms DINOv2 on dense prediction benchmarks such as semantic and panoptic segmentation. Notably, JEM achieves competitive performance with DINOv3 despite using substantially less training data and no additional refinement stages
Critical weaknesses
  • Although training on patch-level objectives alone, without a separate image-level objective, yields strong dense prediction performance, classification performance is consistently lower than DINOv2 across all model scales. This suggests that patch-level learning may involve a trade-off with global representation performance, but the paper does not sufficiently investigate the cause of this gap or whether it can be mitigated.
Questions for authors
  • JEM achieves strong dense prediction performance with patch-level training alone, but its classification performance is lower than DINOv2. The appendix states that the class token can be included in the same information-theoretic framework, so it would be important to clarify why it was excluded from the actual training. Were any experiments conducted with an image-level objective to improve classification while maintaining dense performance, or is there evidence supporting patch-level objectives alone?

  • The performance of JEM is evaluated through frozen-backbone probing. Would the improvements over DINOv2 be maintained when the entire encoder is fine-tuned for downstream tasks? Even experiments on a relatively smaller model, such as ViT-L, would help demonstrate the practical utility of the learned representations.

3 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for their positive review and constructive criticism! We respond to the reviewer’s questions below and will include the following experiments and discussions in the paper as well.

Image-level objective

It is true that the theoretical framework admits both image-level and patch-level losses. We opted for a patch-level only-objective for two reasons: simplicity (less hyperparameters), and avoiding the reported negative interactions between the global and local losses (DINOv3, Simeoni et al., 2025). That said, we are also curious about the benefit of adding a global image-level loss.

To this end, we conduct a set of experiments on ViT-L where we add a CLS token with an image-level loss, i.e. alignment loss on CLS token between student and teacher crops. We vary the relative weight of this loss compared to the patch-level loss: thus, wcls=1w_\text{cls}=1 only has the image-level loss, and no patch-level loss. We find evidence of a trade-off between classification and dense performance: with equal weight for image and patch losses, classification improves over the patch-level-only JEM baseline (+0.4 IN1k), but segmentation decreases (-1 ADE). Interestingly, using the CLS loss alone strongly drops both classification (-1.0 IN1k) and segmentation (-7.1), justifying the use of the patch-level losses. Compared to the DINOv2 baseline, adding an image-level loss to JEM closes the gap on ImageNet by 0.4 points, while maintaining slightly stronger segmentation. Whether this trade-off stems from the same negative interaction between image and local losses as reported by DINOv3 is an open question.

CLS weightIN1kADE20k
JEM, wcls=0.0w_\text{cls}=0.0 (only patch-level loss, JEM)82.848.1
JEM, wcls=0.5w_\text{cls}=0.583.247.1
JEM, wcls=1.0w_\text{cls}=1.0 (only image-level loss)81.841.0
DINOv2 baseline84.046.9

That said, these findings from a single trial should be interpreted with caution. JEM’s setup and hyperparameters were tuned under the patch-level loss setup, and here we performed a drop-in addition of the CLS loss. Some tweaks may also further improve the classification performance (e.g. increasing the number of CLS prototypes, adding CLS cross-view losses). Further, we find evidence that JEM’s view construction may negatively affect classification performance; switching DINOv2 to JEM views decreases IN1k performance by 0.5 (see response to reviewer QEEQ, DINOv2 with JEM views). Thus, we believe the performance of the combined image-patch loss JEM model can likely be improved further; studying this is an interesting direction for future work.

We would also like to highlight that the patch-level objective used in DINOv2 (iBOT) obtains much worse performance on IN1k when not combined with the DINO objective (60 accuracy vs 73.8 with DINO on ImageNet, Table 9 of the iBOT paper [A]). We consider this to be a key advantage of JEM: it relies on a single objective trained on patches, yet succeeds in yielding strong global representations.

Experimental details: we train ViT-L on IN22k for 250k steps matching JEM’s ablation setup. The target of the CLS token for the local and global student crops is the corresponding teacher crop. The mutual information regularization terms are computed separately on patches and CLS token, using the weight wclsw_\text{cls} to balance between the two. CLS and patch-losses share a head; we found little difference with a dedicated head for the CLS loss. The CLS loss is only computed on the final layer.

[A] Zhou et al. iBOT: Image BERT Pre-Training with Online Tokenizer. ICLR, 2022.

AuthorsOpenReview ↗

Response (2/2)

Fine-tuning evaluations

As per your suggestion, we ran fine-tuning evaluations on IN1k classification attention probe and ADE20k dense linear segmentation, where we train the probe jointly with the backbone for 12.5k and 40k steps respectively. We compare the ViT-L DINOv2 and JEM models trained on the A140M dataset. For each model we sweep 8/6 learning rates for IN1k/ADE20k, and report the best result.

We find that, unsurprisingly, all models improve through fine-tuning. On IN1k classification, the gap between DINOv2 and JEM reduces from 1.3 to 0.9 points; on ADE20k segmentation, DINOv2 can catch-up to JEM by 0.5 points. The results show that when fine-tuning on a specific dataset and taxonomy, differences between encoders shrink. JEM remains competitive with or better than DINOv2 under full fine-tuning, and its dense-prediction advantage is largest when the representation is used as is. We would like to add that frozen evaluation also reflects practical use-cases for pretrained features. Examples are latent world models trained on top of a frozen encoder [B,C,D], and representation autoencoders which use a frozen encoder to define the latent space of diffusion models [E,F].

ModelIN1k / frozenIN1k / FTADE20k / frozenADE20k / FT
JEM83.384.849.955.7
DINOv284.685.748.855.2

Results computed on ViT-L trained for 500k steps on A140M.

[B] Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985, 2025.

[C] Baldassarre et al. Back to the Features: DINO as a Foundation for Video World Models. arXiv:2507.19468, 2025.

[D] Karypidis et al. DINO-Foresight: Looking into the Future with DINO. NeurIPS, 2025.

[E] Zheng et al. Diffusion Transformers with Representation Autoencoders. ICLR, 2026.

[F] Singh et al. Improved Baselines with Representation Autoencoders. arXiv:2605.18324, 2026.

Thank you for conducting the additional experiments on the image-level objective and downstream fine-tuning. The CLS loss experiments help clarify the trade-off between classification and dense prediction performance, and the fine-tuning results provide further insight into JEM's downstream performance. These additional experiments and explanations address most of my earlier concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.