- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce JEM, a patch-level self-supervised learning method for computer vision. The framework is grounded in information theory: in a multi-view setting, it maximizes a lower bound on the mutual information between the representation of a student network and the target produced by a teacher network, a student–teacher paradigm shared with other self-supervised methods (e.g., BYOL, DINO).
The key modeling choice is to treat the target as a collection of elements, drawn from different patches of the view and from different layers of the network, so that the training signal is both spatially local and includes several levels of representation. From the resulting bound, the authors derive three objectives: aligning student and teacher predictions across views, preventing global collapse, and preventing structural collapse (where all patch-level targets encode the same information).
The experiments evaluate the method on standard computer vision tasks: image classification, semantic, instance and panoptic segmentation, and monocular depth estimation. JEM outperforms a DINOv2 baseline retrained under identical conditions on the dense prediction tasks, while remaining competitive on classification, at the same training cost.
Critical strengths
- JEM trains stably up to 7B parameters with a patch-only objective and no class token, and it consistently outperforms a controlled DINOv2 baseline on semantic, instance and panoptic segmentation, especially panoptic.
- The experiments are broad: nine benchmarks, three model scales, recent state-of-the-art baselines, and a DINOv2 baseline retrained in the same codebase on the same data.
- Deriving the objective from the multi-view InfoMax bound is well motivated and gives an interesting reading of DINO-family heuristics as implicit entropy controls.
- Aligning student and teacher crops on a shared patch grid is a relevant patch-level adaptation of multi-crop that unifies cropping and masking, and the ablations show it contributes clear gains.
Critical weaknesses
- Only the alignment term of Eq. (2) is optimized directly. The anti-collapse term is replaced by a student-side estimate and TC(T) by a similarity-matching proxy, so the optimized objective differs from what the theory asks for, and the “principled” claim should thus be tempered.
- DINOv2* is trained with its native views, where the patch loss applies only to masked tokens of the same crop the teacher sees, whereas JEM aligns patches across shifted, independently augmented crops. Since Table 1 shows that these view choices drive large dense-task gains, DINOv2 should also be evaluated with JEM’s view construction to separate the contribution of the loss from the contribution of the augmentation strategy.
- R_struct differs from DINOv3’s Gram anchoring, which matches final-layer patch similarities to an earlier checkpoint late in training rather than to intermediate teacher layers from the start. Both, to my understanding, regularize patch-similarity structure, so an empirical comparison would be expected. Also, overall in Section 3.2.3: it is not clear why such a loss optimizes the total correlation as defined in the supplementary material. More details about why this similarity-matching proxy is used could improve a lot the readability of this section. In particular, and again, its potential connection to DINOv3's Gram anchoring should be more explicit.
- JEM is consistently about 1 point below DINOv2* on IN-1k, plausibly because it has no class token or image-level objective. The authors should discuss this gap and ideally evaluate the impact of adding a class token term to JEM.
- The method is presented as simpler than multi-objective approaches, yet it relies on three loss terms, one of which combines two entropies, plus several weights and schedules whose choice and influence are not analyzed.
- Section 3.2.2 does not clearly state what loss is actually minimized, or at least it is not clear to me. It should state either the term that is implemented and minimized or clarify it. In Figure 4: the global regularization weight alpha is, I believe, not defined
Questions for authors
- Can the authors evaluate DINOv2 with JEM's view construction (shifted, independently augmented, grid-aligned crops, with the loss on visible and masked patches) to separate the contribution of the loss from that of the views?
- How does R_struct compare empirically with DINOv3's Gram anchoring, and why does matching intermediate-layer similarities control the total correlation TC(T)?
- What exactly is minimized in Section 3.2.2, and on which network? What is alpha in Figure 4 exactly ?
- Does adding a class token as an additional element of T close the classification gap with DINOv2* without hurting dense performance?
- How were the loss weights, target layers, and schedules chosen, and how sensitive are the results to them?
4 comments
Response (2/4)
Q: How does R_struct compare empirically with DINOv3's Gram anchoring, and why does matching intermediate-layer similarities control the total correlation TC(T)?
We thank the reviewer for the questions. The relation to Gram anchoring is interesting, and we discuss it in App. A.1 ("Structural regularization and Gram anchoring"). We will reference this discussion in Sec. 3.2.3 in the main text to make the connection more explicit.
Comparison with Gram anchoring. Both methods use pairwise patch similarities as targets, but they were designed to address different problems and are implemented differently.
- Purpose. Gram anchoring addresses the local patch structure becoming increasingly noisy during long training. R_struct addresses the redundancy term TC(T) of the anti-collapse objective, i.e., structural collapse, where all patches of an image encode the same image-level content.
- Targets. Gram anchoring requires selecting an earlier checkpoint as Gram teacher and a separate forward pass through it. R_struct reuses precomputed intermediate EMA teacher layers, requiring no extra forward pass. As noted in Appendix A.3.4, R_struct can also use targets directly from student representations, though using teacher features proved slightly more effective. Furthermore, Gram anchoring directly ties the Gram matrix of the final-layer features to the target. In R_struct, the similarities are matched after a learned transformation. The final representation is therefore not forced to replicate early-layer similarities and remains free to encode higher-level semantic structure.
- Schedule. Gram anchoring is a separate refinement stage. R_struct can be applied from the start of training, simplifying training pipelines and recipes.
Empirically, the table below compares R_struct with Gram anchoring applied from 400k iterations. Both runs are ViT-g on the A140M dataset, evaluated at 500k iterations. Gram anchoring uses the 200k checkpoint as Gram teacher and follows the recipe given in DINOv3. We find that Gram anchoring broadly has a similar effect to R_struct, with improvements on ADE20k (+0.4 mIoU) but drops on IN1k (-0.3 accuracy). JEM’s structural regularizer thus could serve as a computationally cheaper alternative to Gram anchoring; replacing it in DINOv3 is an interesting direction for future work.
| IN1k | ADE20k | |
|---|---|---|
| Structural Reg | 85.6 | 52.3 |
| Gram anchoring from 400k | 85.3 | 52.7 |
Intermediate-layer similarities and TC(T). We agree that Section 3.2.3 reads dense due to space considerations. We clarify here and will improve this section in paper.
Structural collapse and intermediate layers. The issue the structural regularizer resolves is the “structural collapse”: where all patch targets T share identical information, maximizing TC(T). On an intuitive level, this can be prevented by avoiding that the patches collapse to a single (redundant) vector. Instead of simply minimizing the redundancy between patches, the structural regularizer imposes a certain (non-redundant) dependency structure, for which it uses the Gram matrix of the features of intermediate layers. This is because natural images have a strong dependency structure, and the targets should reflect this. Because we don’t want to impose a single (potentially incorrect) dependency structure, we match the structure of several intermediate layers through learned projection heads. This assumes intermediate layers are non-collapsed and capture distinct structural levels (e.g., textures at patch embeddings, part-level structure mid-network).
Gram matching and TC(T). Assume that, in both the intermediate and final layers, each patch contains the same image-wide signal and its own independent local details. In a simple Gaussian model with many independent feature dimensions, copying more of the shared signal into every patch makes the patches point in more similar directions and carry more redundant information. By matching the Gram matrix of the final layer to an intermediate layer, the structural regularizer thus limits how much more redundant the final patch features can become. Then, if the teacher assigns a category T to each patch using only that patch’s matched feature, the head can discard redundancy but cannot add more. The patch redundancy of the intermediate layers then puts a ceiling on the total correlation of the teacher’s targets TC(T) (hence “controlling” TC).
This argument relies on several assumptions. To empirically validate it, we measure structural collapse by tracking the fraction of images per batch where the teacher assigns every patch to the exact same category. Using the models from Fig. 5 (right), we find that without R_struct, 8.8% of images are collapsed, whereas with R_struct, only 4.0% are collapsed. This is consistent with the prevention of loss of spatial diversity in Fig. 5 (right).
Response (3/4)
Q: What exactly is minimized in Section 3.2.2, and on which network? What is alpha in Figure 4 exactly ?
Section 3.2.2 minimizes R_global (Eq. 6), which corresponds to the negative estimated per-element mutual information (Eq. 5) averaged over patch-layer elements. Minimizing R_global maximizes this estimated mutual information. Optimization occurs strictly on the student network, with gradients flowing into its head and backbone.
Alpha in Fig. 4: We thank the reviewer for pointing out this typo. Alpha refers to w_global, the weight of the global regularization term R_global in the loss. Fig. 4 illustrates performance sensitivity to this weight on IN-1k and ADE20k.
Q: Does adding a class token as an additional element of T close the classification gap with DINOv2 without hurting dense performance?
We conduct a set of experiments on ViT-L where we add a CLS token with an image-level loss, i.e. alignment loss on CLS token between student and teacher crops. We vary the relative weight of this loss compared to the patch-level loss: thus, only has the image-level loss, and no patch-level loss. We find evidence of a trade-off between classification and dense performance: with equal weight for image and patch losses, classification improves over the patch-level-only JEM baseline (+0.4 IN1k), but segmentation decreases (-1 ADE). Interestingly, using the CLS loss alone strongly drops both classification (-1.0 IN1k) and segmentation (-7.1), justifying the use of the patch-level losses. Compared to the DINOv2 baseline, adding an image-level loss to JEM closes the gap on ImageNet by 0.4 points, while maintaining slightly stronger segmentation. Whether this trade-off stems from the same negative interaction between image and local losses as reported by DINOv3 is an open question.
That said, these findings from a single trial should be interpreted with caution. JEM’s setup and hyperparameters were tuned under the patch-level loss setup, and here we performed a drop-in addition of the CLS loss. Some tweaks may also further improve the classification performance (e.g. increasing the number of CLS prototypes, adding CLS cross-view losses). Further, we find evidence that JEM’s view construction may negatively affect classification performance; switching DINOv2 to JEM views decreases IN1k performance by 0.5 (see experiment above, DINOv2 with JEM views). Thus, we believe the performance of the combined image-patch loss JEM model can likely be improved further; studying this is an interesting direction for future work.
We would also like to highlight that the patch-level objective used in DINOv2 (iBOT) obtains much worse performance on IN1k when not combined with the DINO objective (60 accuracy vs 73.8 with DINO on ImageNet, Table 9 of the iBOT paper [A]). We consider this to be a key advantage of JEM: it relies on a single objective trained on patches, yet succeeds in yielding strong global representations.
| CLS weight | IN1k classification | ADE20k segmentation |
|---|---|---|
| JEM, w_cls=0.0 (only patch-level loss, JEM) | 82.8 | 48.1 |
| JEM, w_cls=0.5 | 83.2 | 47.1 |
| JEM, w_cls=1.0 (only image-level loss) | 81.8 | 41.0 |
| DINOv2 baseline | 84.0 | 46.9 |
Experimental details: we train ViT-L on IN22k for 250k steps matching JEM’s ablation setup. The target of the CLS token for the local and global student crops is the corresponding teacher crop. The mutual information regularization terms are computed separately on patches and CLS token, using the weight w_cls to balance between the two. CLS and patch-losses share a head; we found little difference with a dedicated head for the CLS loss. The CLS loss is only computed on the final layer.
[A] Zhou et al. iBOT: Image BERT Pre-Training with Online Tokenizer. ICLR, 2022.
Response (4/4)
Q: How were the loss weights, target layers, and schedules chosen, and how sensitive are the results to them?
We agree with the reviewer on the importance of ablating hyperparameter choices and schedules. The main choices include the loss weights on masked vs. visible patches, the mutual information (MI) regularization weight (R_global), and the target layers and schedule for the structural regularizer (R_struct). We lightly tuned the MI weight and masked loss weight at ViT-L scale, transferring them unchanged to larger models. Target layers for R_struct were untuned, and its schedule was set to ramp up to counteract the structural collapse emerging progressively through training (Fig. 5). Below, we analyze sensitivity to each choice; we find JEM to be robust to these choices within a wide range of values.
Masked and visible alignment loss weights (ViT-L, IN-22k, 250k steps). Both terms are essential. Using only masked loss is catastrophic (70.8 IN-1k, 28.1 ADE20k), while using only visible loss costs 1.5 IN-1k points and 2.1 ADE20k mIoU. Other tested mixtures remain within 0.5 points on IN-1k (82.3–82.8) and 1.4 mIoU on ADE20k (46.7–48.1). The paper's ratio (0.8 masked, 0.2 visible) achieves the best performance across both benchmarks.
| IN1k | ADE20k | |
|---|---|---|
| w_mask=0, w_vis=1.0 (only vis. loss) | 81.3 | 46.0 |
| w_mask=0.2, w_vis=0.8 | 82.5 | 46.9 |
| w_mask=0.5, w_vis=0.5 | 82.5 | 47.5 |
| w_mask=0.8, w_vis=0.2 (paper version) | 82.8 | 48.1 |
| w_mask=0.9, w_vis=0.1 | 82.3 | 46.7 |
| w_mask=1.0, w_vis=0 (only masking loss) | 70.8 | 28.1 |
Mutual information regularization weight. Results are robust to this weight (Fig. 4), requiring only a non-zero value; a weight of 0 causes collapse, whereas a broad range of non-zero values yields comparable performance.
Structural regularizer: target layers (ViT-g, A140M, 500k steps). The table compares different sets of target teacher layers .
- Using multiple target layers is robust: every tested multi-layer set improves over no structural regularization by 0.4–0.7 IN-1k points and 1.8–2.2 ADE20k mIoU, with all choices yielding similar performance (84.8–85.1 IN-1k, 51.6–52.0 mIoU) across early, late, spread, and dense configurations.
- A single target layer is less reliable: layer 0 improves ADE20k by 1.2 mIoU, whereas layer 10 reduces it by 1.4 mIoU. This supports similarity matching across multiple layers, encoding different structural levels.
- The untuned default selection (0, 10, 20, 30) achieves the top IN-1k accuracy and falls within 0.3 mIoU of the best ADE20k score.
| IN1k | ADE20k | |
|---|---|---|
| No gram | 84.4 | 49.84 |
| Target layer: 0 | 84.9 | 51 |
| Target layer: 10 | 84.8 | 48.4 |
| Target layer: 0, 10 | 84.9 | 52 |
| Target layers: 0, 5, 10, 15 (first half) | 85 | 51.7 |
| Target layers: 20, 25, 30, 35 (second half) | 85 | 51.9 |
| Target layers: 0, 10, 20, 30 (full model, paper version) | 85.1 | 51.7 |
| Target layers: 0, 5, 10, 15, 20, 25, 30, 35 (full model, densely spaced) | 84.8 | 51.6 |
Structural regularizer: weight and schedule (ViT-g, A140M, 500k steps). The specific schedule has little impact: a constant weight of 1 performs close to the 0.05–1 linear schedule (84.7 vs. 85.1 IN-1k, 51.6 vs. 51.7 ADE20k). What matters is the magnitude of the weight.
- The weight must be sufficiently high to prevent structural collapse: ADE20k steadily rises from 49.8 (weight 0) to 50.4 (0.1), 51.2 (0.3), and 51.6 (1). This aligns with needing R_struct to preserve spatial diversity.
- A much larger weight (3) over-constrains the representation and degrades both tasks (83.1 IN-1k, 48.9 ADE20k).
| IN1k | ADE20k | |
|---|---|---|
| w_struct=0 | 84.4 | 49.84 |
| w_struct=0.1 | 85 | 50.4 |
| w_struct=0.3 | 84.8 | 51.2 |
| w_struct=1 | 84.7 | 51.6 |
| w_struct=3 | 83.1 | 48.9 |
| w_struct=0.05 → 1.0 (paper version) | 85.1 | 51.7 |
Response (1/4)
We thank the review for their positive assessment of our work and for their constructive comments. We answer their questions below, and will also include the following experiments and discussion into the paper.
Q: Can the authors evaluate DINOv2 with JEM's view construction (shifted, independently augmented, grid-aligned crops, with the loss on visible and masked patches) to separate the contribution of the loss from that of the views?
We thank the reviewer for suggesting this experiment, which helps to separate the contribution of the views from that of the loss. We performed two experiments: training DINOv2 with JEM's view construction, and additionally including a CE loss on masked and visible patches on global/local crops. The results are in the table (same setting as Tab. 1: ViT-L, IN22k, 250k steps):
In JEM, student and teacher crops are shifted, but aligned on a shared patch grid. Enforcing patch-level alignment across shifted views, with independent augmentation, benefits dense tasks: DINOv2 improves on ADE20k by 0.7 mIoU using JEM’s view creation. This suggests JEM’s view creation can benefit other methods. JEM still outperforms the best DINOv2 variant by 0.5 mIoU, showing our framework's advantages. On IN-1k, JEM views reduce DINOv2's accuracy by 0.5 points. We hypothesize that grid-aligned crops remove the scale variations between local student and global teacher crops present in DINOv2’s views. These scale variations promote broader local-to-global correspondences and scale invariance, benefiting image-level classification. Surprisingly, using the full patch-level loss on visible and masked patches degrades both tasks. In contrast, both visible and mask losses improve JEM's performance (Table 1b). This may occur because visible patch loss acts as a regularizer; this mechanism might be less crucial for DINOv2.