acceptodds
Under review as a conference paper at ICLR 2027

Mask-supervised Object-centric Representation Learning with LeJEPA

Abstract

Self-supervised image encoders deliver strong features for downstream tasks but need many images for training. A natural remedy to counter this is to make each image count for more. A scene contains many objects, and given masks from human annotators or an off-the-shelf segmentation model, pre-training can focus on aligning per-object rather than image-wide representations, extracting more signal from every image. Existing mask-supervised methods do this through reconstruction or contrastive losses that leverage negative objects. We instead use two separate projection spaces for the alignment. In a semantic space, per-object representations from different views are aligned. To avoid collapse, instead of using negative objects, which requires category definitions, we extend the negative-free LeJEPA objective and show that its distributional anti-collapse regularizer ports naturally from whole images to the variable-sized set of objects in a scene. In an instance space, a contrastive loss separates per-object representations from their context and co-occurring instances, including those of the same category. To separate object representations from their context, we copy objects and paste them into other contexts, where each pasted copy serves as an additional view of the original object. Trained on COCO with ground-truth masks, our method outperforms image-level and mask-guided baselines on tracking (DAVIS), classification (ImageNet-1k) and re-identification (NAVI), matches the best of them on semantic segmentation (ADE20k) and keeps its lead over image-level LeJEPA and a supervision-matched alternative on COCO fractions down to 256 images.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Object-centric Representations, LeJEPA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9ffqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies mask-supervised visual pre-training under limited image diversity. A ViT produces patch features; a masked cross-attention aggregator forms one semantic representation per object, and an object-level LeJEPA loss aligns representations of the same instance across augmented views while a modified SIGReg term discourages collapse. A separate patch projection is trained with a within-view supervised contrastive loss that groups patches of the same instance and separates other objects and background. Copy-paste augmentation places donor objects into host contexts and treats the pasted occurrence as another view. Models are pre-trained on COCO with ground-truth, SAM, or Felzenszwalb–Huttenlocher masks and evaluated with frozen features on ADE20k instance/semantic probes, DAVIS tracking, NAVI re-identification, and ImageNet-1k classification. The reported point estimates generally exceed COCO-trained baselines, especially DetCon, and the paper studies dataset sizes from 256 images to full COCO.

Critical strengths
  1. The semantic/instance decomposition is coherent and task-motivated. Section 3 uses a negative-free object-level objective for invariance and a separate within-scene contrastive head for instance separation, making the intended roles of the losses easy to identify.
Critical weaknesses
  1. Initialization and subset-composition variance are most consequential in the regime that defines the paper’s contribution. “Best in all ten settings” and “down to 256 images” are therefore observations about the reported runs, not statistically established comparative claims. K-means restarts and NAVI memory-bank resampling do not replace independent pre-training runs.

  2. The proposed method can use its learned cross-attention object representation z, whereas competitors use mask-averaged patch features; both ADE20k object classification and NAVI additionally require ground-truth downstream masks.

  3. The manuscript computes ROC-AUC per image after retaining only same-category patch pairs, but does not define what happens when the retained labels contain only one class or no eligible pairs.

Questions for authors
  1. What are the mean, standard deviation or confidence interval, and paired win rate across independent training seeds and independently sampled 256/512/1024-image subsets for Ours, DetCon, and Image LeJEPA? Are the Figure 3 subsets nested, and can the image IDs and object/category counts be provided?
  2. What is the complete object-SIGReg equation and implementation: which object-view representations enter each estimate, how are repeated views and pasted copies weighted, how are variable counts reduced across GPUs, and what changes when the actual object count replaces BB?
  3. Exactly which ADE20k images and patch pairs enter the same-category ROC-AUC, how are images with only one target label handled, and how many validation images remain under that rule?
  4. Do the NAVI data-scale conclusions and the claim of superiority over all COCO-trained baselines hold when every method uses the identical mask-averaged patch readout? Which of the reported object-level settings actually require masks at deployment?
  5. Can the authors report per-method encoder-view counts, FLOPs/GPU-hours, convergence evidence, and a controlled comparison of SIGReg versus contrastive or teacher–student object alignment with masks, aggregator, augmentations, and auxiliary losses fixed?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 9ffq (1/4)

We thank the reviewer for the detailed and precise review, which led to several additions. We now report data-scale experiment results across three initialization seeds and three independently sampled subsets, report matched read-outs in Figure 3 and Table 2, present the formulation of the complete object-level SIGReg estimator in Appendix A.4, and specify the same-category ROC-AUC protocol in Appendix A.6. A controlled comparison against a contrastive object-level objective is added to Table 2. Convergence curves will follow later in the discussion period.

W1, Q1 - Seed and subset variance

We agree with the reviewer that initialization and subset-composition variance needs to be measured to support our claims. Therefore, we repeated the ViT-B data-scale experiments on the 256256, 512512 and 10241024 image subsets. Starting from the run of the submission (initialization seed 0, subset seed 0), we trained two further initialization seeds on the same subset, and trained on two further independently sampled subsets with the same initialization seed. This gives three initialization seeds and three subset seeds per subset size, with one run shared between them, i.e., 55 runs per method and subset size.

The variation in both directions is small compared to the difference between our method and the two baselines (DetCon, Image LeJEPA) for all downstream task performances. As an example, for tracking on DAVIS with 256256 images the means over the three initialization seeds are 0.4260.426, 0.4920.492 and 0.6330.633 for Image LeJEPA, DetCon and our method, with ranges (maximum minus minimum) of 0.0170.017, 0.0020.002 and 0.0090.009. Over the three subsets the means are 0.4240.424, 0.4960.496 and 0.6340.634 with ranges of 0.0120.012, 0.0060.006 and 0.0060.006. Both are much smaller than the gaps between methods. All runs are listed in Table 4 (Appendix A.7) of the revised paper. Pairing the runs by seed gives 1515 comparisons per task (55 per subset size). Our method wins 1515 of 1515 against Image LeJEPA and 1515 of 1515 against DetCon on each of the four downstream tasks, a paired win rate of 100100% (Table 5, Appendix A.7). Figure 3 now shows the mean and range over three initialization seeds for every point.

The COCO-subsets in these experiments are nested and the object/category counts are displayed in Table 3 in Appendix A.7. The number of instances remains similar across seeds for each subset size (e.g. number of instances in 256 images: 1839, 1896, 1863). The subsets of 10241024 images contain 7979, 8080 and 8080 of the 8080 categories for the three subset seeds, the subsets of 256256 images 7676, 8080 and 7878. We also provide the super-category counts for each subset in the appendix. Image IDs can be found in the code repository in the supplementary material and will be published.

AuthorsOpenReview ↗

Response to Reviewer 9ffq (2/4)

W2, Q4 - Readout asymmetry and masks at test time, which settings need masks at deployment

We agree that comparing mask-averaged patch features with semantic object representations zz is not a fair comparison. Our main results (Table 1 in main paper) already show that our mask-averaged patch features also outperform all COCO-trained models on ADE20k object classification. On NAVI re-identification they outperform all COCO-trained models except CP$^2$ (0.7140.714 vs. 0.7370.737), but it is important to note that CP$^2$ performs considerably worse on every other downstream task. We now also report mask-averaged patch features for the data-scale experiments and the ablation table. With the matched read-out our method still wins all 1515 paired comparisons on NAVI against Image LeJEPA and against DetCon (Table 5 in Appendix A.7 of the revised paper), with smallest margins of 0.2100.210 and 0.2590.259. The data-scale conclusions on subsets of 256256, 512512 and 10241024 images therefore hold with matched read-outs.

Masks at deployment. The encoder backbone never receives a mask, so patch features are obtained without one in every setting. The settings differ in whether the task built on these features needs masks. Image classification on ImageNet-1k uses none at any stage. Linear segmentation and the same-class probe need none at deployment: masks serve only as evaluation labels, and for the probe additionally to select the same-class, different-instance patch pairs on which we measure how well patch features encode instance identity beyond semantics. Tracking is given the first-frame mask that defines the task, and the propagation to later frames relies on patch features alone. Instance clustering does require masks, since we measure how well instances can be separated among foreground patches and therefore use the ground-truth masks to select these patches and to set the number of clusters. The two object-level settings, ADE20k object classification and NAVI re-identification, require one mask per object for every method and both read-outs, since the mask defines which patches are pooled. The semantic object representation z\mathbf{z} additionally takes the mask as input to the aggregator ff.

W3, Q3 - Same-category ROC-AUC edge cases, Which images and pairs enter the same-category ROC-AUC?

We thank the reviewer for pointing this out and apologize for the unclear description. We describe the evaluation protocol in more detail below and have added it to the implementation details in the revised paper. After training the quadratic probe on frozen patch features, we assign each patch to the instance that covers the largest number of its pixels, with the background as one of the candidates, and discard patches assigned to the background. From the remaining foreground patches we sample up to 256256 per image and collect their features together with their instance and semantic labels. The sampling is seeded, so the same patches are used across runs and across all methods that share the same patch grid. Each patch inherits the instance and semantic label of its assigned mask. Pairs of patches that belong to the same instance are positive pairs. Pairs of patches that share the same semantic class but belong to different instances are negative pairs. The ROC-AUC is calculated for images that have at least one positive and at least one negative pair. We report the mean over all calculated ROC-AUC metrics. For 32×3232\times32 patch feature maps (patch size 1616 on 512×512512\times512 images), this is an average over 1894 of the 2000 ADE20k validation images. For 36×3636\times36 patch feature maps (DINOv2 and FT-DINOSAUR, patch size 1414 on 504×504504\times504 images), it is 1895 images.

AuthorsOpenReview ↗

Response to Reviewer 9ffq (3/4)

Q2 - Complete object-SIGReg equation and implementation

We use the Epps-Pulley/slicing implementation of the official LeJEPA code, applied to objects instead of images. Each of the BB images yields VV views. For j∈{1,…,V}j\in\lbrace 1,\dots,V\rbrace let V(j)⊂V\mathcal{V}^{(j)} \subset \mathcal{V} be the set of jj-th views of every image in the global batch. As in LeJEPA, SIGReg is estimated per view index, here over all foreground objects present in those views:

\mathcal{S}_j = \lbrace(v,k): v \in \mathcal{V}^{(j)}, k \in \mathcal{K}, v \in \mathcal{V}_k\rbrace, \quad N_j = |\mathcal{S}_j| \end{aligned}$$ where $N_j$ corresponds to the number of object occurrences in all $B$ $j$-th views of the global batch, pasted copies included. For a unit direction $a \in \mathbb{A}$, we define the view-specific empirical characteristic function $\hat{\varphi}_{j,a}$ of the $a$-projected semantic object representations $\tilde{\mathbf{z}}$ $$\begin{aligned} \hat{\varphi}_{j,a}(t) &= \frac{1}{N_j}\sum_{(v,k) \in \mathcal{S}_j} e^{ita^\top\tilde{\mathbf{z}}_{v,k}}\\\\ &=\frac{1}{N_j}\sum_{(v,k) \in \mathcal{S}_j} \cos(ta^\top\tilde{\mathbf{z}}_{v,k}) + i \frac{1}{N_j}\sum_{(v,k) \in \mathcal{S}_j} \sin(ta^\top\tilde{\mathbf{z}}_{v,k}). \end{aligned}$$ The per-view, per-projection Epps-Pulley statistics $\text{EP}_{j,a}$ are then averaged over the directions and the views to form the SIGReg loss, $$\begin{aligned} \mathcal{L}_\text{SIGReg} = \frac{1}{V} \sum_{j=1}^V \frac{1}{|\mathbb{A}|} \sum_{a\in \mathbb{A}} \text{EP}_{j,a}, \quad\quad \text{EP}_{j,a} = B \int_{-3}^3 |\hat{\varphi}_{j,a}(t) - e^{-t^2/2}|^2 e^{-t^2/2} dt \end{aligned}$$ The integral is evaluated with the trapezoidal rule on $17$ knots over $[0,3]$, doubled by symmetry, as in LeJEPA's released implementation. We sample $|\mathbb{A}|=1024$ directions every step from a step-seeded generator shared by all GPUs. **Which objects enter?** Every foreground object present in a view (mask covering more than $16\times 16$ pixels, see Section 3.1) enters that view index’s Epps-Pulley estimate. This includes objects that are present in only one view and are therefore not in $\mathcal{K}^\mathrm{eff}$, the set of objects present in at least two views, to which the alignment term is restricted. **How are repeated views and pasted copies weighted?** Each occurrence $(v,k)$ has weight $\frac{1}{N_j}$ within its view index. A pasted object keeps its identity $k$ and $\mathcal{V}_k$ contains views of both donor and host. It therefore enters $\mathcal{S}_j$ twice, once in the donor's $j$-th view and once, in a new context, in the host's $j$-th view. This applies to at most $B/2$ of the $N_j$ samples per view index. Empirically (200 training steps, $B=256$), $95.6$ of the $128$ possible hosts receive a paste per step. The resulting second entries make up $9.1$% of the $N_j$ samples in global views ($91.8$ of $1007$) and $11.9$% in local views ($62.4$ of $524$). **How are variable counts reduced across GPUs?** LeJEPA averages the per-GPU characteristic functions, which equals the global mean because every GPU holds the same number of images. Object counts differ across GPUs, so we instead all-reduce (SUM) the presence-masked sums $\sum \cos(ta^\top\tilde{\mathbf{z}}_{v,k})$ and $\sum\sin(ta^\top\tilde{\mathbf{z}}_{v,k})$ together with the object counts, and then divide. $\hat{\varphi}_{j,a}$ is therefore the exact mean over all $N_j$ objects in the global batch. The all-reduce is autograd-aware, as in LeJEPA. **What changes when the actual object count replaces $B$?** Using the actual object count would effectively rescale SIGReg in view $j$ by $N_j/B$. The regularizer's magnitude would vary from batch to batch with scene content. Furthermore, it would regularize object representations from global views more than ones from local views, since larger crops yield higher object counts. Empirically ($B{=}256$), a global view index holds $N_j/B = 3.93$ objects per image on average and a local one $2.05$, so SIGReg would weigh global-view objects $1.92\times$ more than local-view ones. Across steps, $N_j/B$ ranges from $3.63$ to $4.31$ in global views and from $1.89$ to $2.22$ in local views (2.5-97.5th percentiles). In early development we observed more stable gradients with the constant scale $B$ than with $N_j$. We did not repeat this as a controlled ablation with the final method, so the revised paper no longer claims it and motivates the choice by the statistics above. The estimator and these details are now in Appendix A.4 of the revised paper.
Reviewer_GGZqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers the setting in which we have a dataset of unlabeled images with object masks. The goal is to learn features to support typical CV tasks: image segmentation, tracking, object and image classification.

The authors propose the following method:

  • A ViT encoder processes an image and produces features for each image patch.
  • The patches are grouped by object using the given masks.
  • There are two heads:
    1. The object-level head combines the features of one object into one vector. This head is trained with the same loss as LeJEPA.
    2. The patch-level head is trained with a contrastive loss: patches of the same object should be closer to each other, while being far from the other objects' representations.
  • The method uses copy-paste augmentation which inserts objects into other images. The method combines existing components: mask-based pooling for self-supervised training (DetCon), the LeJEPA objective, a standard supervised contrastive loss applied at the patch level, and copy-paste augmentation. The combination is new; the individual components are not.

Overall, the method seems sound and the results are promising, but the benefits are modest and the intended use case is not clearly stated.

Critical strengths
  • The method seems sound and the results are promising.
Critical weaknesses
  • The paper is hard to read. I believe I understood the main ideas, but it took considerable effort as someone familiar with the field but not its most recent papers. For example:
    • The intended use case should be stated more clearly.
    • The object pooling head needs more detail: formulas or pseudocode would be helpful.
    • Some metrics are not defined in the text (FG-ARI, balanced accuracy)
    • These are only examples; the overall flow of the presentation could be improved.
  • The intended use case is not clearly stated. My interpretation is that the primary use case would be training in new domains where good foundation models do not exist but where one can obtain object masks. This seems uncommon in practice, as masks are usually harder to produce than labels.
  • The DINO features score better in most metrics, so there is no improvement in the natural images domain.
Questions for authors
  • What is the effective number of objects per image during training? Did the authors try different setups, for example, by capping the maximum number of objects at 1, 2, 4, etc., and examine how this affects performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer GGZq

We thank the reviewer for the careful reading and the concrete suggestions on the presentation. We added pseudocode and all architectural details of the object pooling head, describe each metric where it is first used, and state the hypothesis the paper explores in the introduction. All changes are marked in blue in the revised paper.

W2 - Intended use case; masks harder than labels

Rather than presenting an explicit use case, the paper explores a hypothesis. We investigate whether and how much more image-efficient representation learning becomes when a decomposition into objects is given. This question is motivated by the finding that humans can segment objects they have never seen before (Spelke, 1990) and this might be one of the reasons why humans are sample-efficient learners (Greff et al., 2020). The experiments with natural images we presented suggest that sample or image efficiency indeed benefits from object masks. This result is the basis of our future work on assessing whether mask-guided object-centric in-domain pre-training provides better features than off-the-shelf foundation models. We have not tested this and do not claim this use case for this paper.

Regarding the cost of masks, the decomposition need not be hand-annotated: performance is largely preserved when the ground-truth masks are replaced with SAM masks (Table 2). On microscopy images, segmenters like Cellpose (Stringer et al., 2021) can provide cell instance masks, while the labels that matter come from an expert or an experiment.

W1 - Readability

We thank the reviewer for the effort spent on the paper and for the concrete examples, which we have used to revise the presentation. In the revised paper, all changes are marked in blue.

Intended use case: We now state in the introduction that the paper explores a hypothesis for image-efficiency and does not target a specific application (see our answer to the previous point).

Object pooling head: We added a description with PyTorch-style pseudocode and all architectural details in Appendix A.3 (Alg. 1), referenced from Section 3.1.

Metrics: Each metric is now briefly defined where it is first used in Section 4.1. Further details on FG-ARI and balanced accuracy are given in the corresponding downstream task paragraphs of Appendix A.6 which Section 4.1 points to.

We would be grateful if the reviewer could point us to any other specific passages that remain hard to follow.

W3 - DINO features score better

We agree that our COCO-trained model does not replace DINOv2/v3 on natural images, and we do not claim state-of-the-art performance. For a practitioner who has access to these models and works on natural images, they remain the better choice. That said, we do not think that superiority of DINOv2/v3, trained on 10310^3-10410^4 times larger datasets, counters our claim on image-efficiency. We believe research into image-efficiency in representation learning can lead to changes in how we train such models.

Q1 - Effective number of objects per image; capping

Our COCO subsets contain 7.27.2 to 7.47.4 annotated instances per image (Table 3 in Appendix A.7 of the revised paper). After cropping and the presence threshold, a global view holds 3.93.9 objects on average and a local view 2.02.0, pasted copies included (measured over 200200 training steps with B=256B=256). Due to limited compute we could not run capped variants within the rebuttal period.

AuthorsOpenReview ↗

Follow-up to Reviewer GGZq

Dear Reviewer GGZq,

we appreciate your input on our submission very much. Following your review, the revised paper states the hypothesis it explores in the introduction, describes the object pooling head with pseudocode (Appendix A.3) and defines each metric where it is first used. We would be happy to answer any remaining questions, and to revise any passage that is still hard to follow, before the discussion period ends on Nov 18 (AoE).

The authors

Thank you for your response!

As I wrote in my review, I still think that using masks in semi-supervised learning is a questionable approach. I remain slightly positive about it, so I will keep my score.

Reviewer_2JxAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a mask-supervised object-centric extension of LeJEPA for data-efficient visual representation learning. The method uses given masks to align object representations across augmented views with LeJEPA, introduces (i) an instance-level contrastive loss to separate different objects, and (ii) applies copy-paste augmentation to reduce background dependence. Experiments on several datasets demonstrate improvements over image-level and mask-supervised baselines across different tasks.

Critical strengths
  • The paper improves data efficiency by leveraging object-level information within images.
  • The proposed method achieves strong performance across multiple downstream tasks.
  • The experiments cover different dataset sizes, backbone capacities, and mask sources, providing a relatively comprehensive evaluation.
Critical weaknesses

W1. Limited methodological novelty. The proposed framework largely combines existing techniques, including mask-level alignment, instance-level contrastive learning, and copy-paste augmentation. The contribution beyond these existing components may not be sufficiently established.

W2. Insufficient justification for using LeJEPA in a mask-supervised setting. The method already relies on instance masks that provide explicit object-level supervision. While the paper compares against several mask-supervised methods, it remains unclear why LeJEPA is preferable to simpler supervised alternatives, particularly instance-level contrastive learning combined with copy-paste augmentation.

Questions for authors

Q1. Given that the method already relies on ground-truth instance masks, which are typically more expensive to annotate than category labels, how do the authors justify avoiding semantic supervision? Since COCO already provides category annotations, a comparison with conventional supervised objectives (e.g., cross-entropy or supervised contrastive learning) would help establish the practical benefit of the proposed approach.

Q2. Could the authors compare against instance-level contrastive learning with copy-paste augmentation but without object-centric LeJEPA? This would help establish whether LeJEPA provides additional benefits beyond the existing supervised components.

Q3. Table 4 shows nearly identical performance at 150 and 300 epochs, with some metrics even slightly decreasing. Why do the authors use 300 epochs for the main results? Could the authors provide convergence curves for the proposed method and key baselines to clarify whether additional training is necessary and how their convergence rates compare?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 2JxA (1/3)

We thank the reviewer for the constructive feedback. We address the concern whether object-centric LeJEPA is justified over simpler supervised alternatives by running the following comparisons: a label-supervised patch-level contrastive baseline, our method without object-centric LeJEPA, and our method with an object-level InfoNCE loss instead of the LeJEPA loss. All three are added to Table 2 and discussed in Section 4.3 of the revised paper. We discuss our findings below as well. Convergence curves will follow later in the discussion period.

Q1 - Why no category supervision

We agree with the reviewer that on COCO, where masks and mask labels are given, comparing to conventional supervised objectives would be insightful. Our reason for avoiding category labels is twofold. Class-agnostic masks are arguably easier to obtain, as sources such as SAM provide masks without labels (bottom block of Table 2 of the revised paper), and a fixed label vocabulary ties the representation to the training categories.

Nonetheless, to further analyze the direction the reviewer suggests, we perform an additional experiment, where we keep augmentations (including copy-paste augmentations) fixed and replace our loss with a patch-wise supervised contrastive loss. This means we treat all patches in a batch that belong to the same class as positives and all patches that belong to a different class as negatives. There are many more patches than objects so we sample 1/81/8 of all patches due to memory constraints in our multi-GPU training setup. With this patch-wise supervised contrastive loss the model is trained to align patches based on their mask's category (referred to as Patch SupCon from now on).

In this baseline, instance-awareness is not enforced, so we see the performance on instance-related downstream tasks decrease (Instance-Clustering; Ours 0.4170.417 vs. 0.3880.388, Same-class instance separation; Ours 0.8530.853 vs. 0.8360.836, Tracking Ours 0.6860.686 vs. 0.6520.652). Re-identification on NAVI benefits from pose- and lighting-invariant features. We hypothesize that the supervised contrastive loss enforces this invariance by aligning different instances of the same category, which would explain why our method is inferior on this downstream task (our mask-averaged patch features 0.7200.720, our semantic object representations 0.7860.786 vs. SupCon's mask-averaged patch features 0.8180.818). Our object classification probe on ADE20k indicates that our label-unsupervised approach performs better when the labels on the downstream dataset differ from the labels on the training set (Our mask-averaged patch features 0.5290.529, our semantic object representations 0.5590.559 vs. SupCon's mask-averaged patch features 0.5090.509). Linear segmentation is on par (0.4020.402 vs. 0.4000.400) and image classification on ImageNet-1k is higher for our method (0.5690.569 vs. 0.5440.544). In conclusion, we argue that pre-training with our method can have benefits over supervised contrastive learning that leverages mask labels.

AuthorsOpenReview ↗

Response to Reviewer 2JxA (2/3)

Q2 - Instance-contrastive + copy-paste without object-centric LeJEPA

We understand the reviewer's concerns about the advantage of LeJEPA over simpler instance-level contrastive learning and acknowledge that the paper lacks a clean ablation. To this end, we provide results for two additional experiments. For the first one we run our method without object-centric LeJEPA and solely train with our per-view instance-level contrastive loss ("w/o O-LeJEPA" in Table 2 of the revised paper). Without cross-augmentation alignment and batch-level SIGReg, performance decreases on linear segmentation (Ours 0.4020.402 vs. 0.3850.385), object classification (0.5290.529 vs. 0.4970.497), re-identification (0.7200.720 vs. 0.5670.567) and image classification (0.5690.569 vs. 0.5120.512), while same-class instance separation (0.8530.853 vs. 0.8590.859) and tracking (0.6860.686 vs. 0.6790.679) remain similar. This indicates that object-centric LeJEPA adds semantic and view-invariant information to the patch features that the per-view instance-level contrastive loss alone does not provide.

For our second experiment, we swap our object-centric LeJEPA loss in our framework with an instance-level InfoNCE loss ("O-LeJEPA →\to object InfoNCE" in Table 2 of the revised paper). In contrast to InfoNCE, SIGReg does not act on pairs of embeddings. It only requires the distribution of object embeddings in a batch to match an isotropic Gaussian along random projections. Two similar objects can therefore stay close as long as the batch as a whole is spread out. InfoNCE’s softmax, on the other hand, puts its largest repulsive gradient on exactly the most similar negatives. So InfoNCE provides a stronger incentive to separate two semantically similar objects. Since COCO never shows the same instance in two scenes and copy-paste positives keep the object’s original pose, it can do so by relying on pose- and lighting-specific appearance cues, which we hypothesize is harmful to object classification and re-identification, where the same object must be matched across exactly such changes. Both variants compute the semantic object representation z\mathbf{z} with the same aggregator ff, so the comparison of their z\mathbf{z} is matched. With mask-averaged patch features the two variants perform similarly. With z\mathbf{z} our method is ahead on object re-identification on NAVI (5-shot Bal. Acc. 0.7860.786, range 0.7830.783-0.7880.788, vs. 0.7080.708) and on object classification on ADE20k (Top-1 Bal. Acc. 0.5590.559, range 0.5550.555-0.5630.563, vs. 0.5270.527). Furthermore, our method demonstrates better instance separation capabilities (FG-ARI 0.4170.417, range 0.4060.406-0.4220.422, vs. 0.3420.342) indicating that object-centric LeJEPA builds a meaningful semantic representation space while still allowing patch features to encode instance identity. On the remaining tasks the InfoNCE variant lies within or close to the edge of the range of our three seeds. In conclusion, we count this as evidence that object-centric LeJEPA provides additional benefits over a contrastive objective under otherwise identical training settings.

W2 - Justification for LeJEPA in a mask-supervised setting

We hope that our answers to the reviewer's first two questions and the additional experiments described convince the reader that LeJEPA provides benefits over simpler label- and mask-supervised alternatives combined with copy-paste augmentations. We agree that a clear justification was missing in the paper; we have extended the ablation section with the mentioned experiments to remedy this missing point.

AuthorsOpenReview ↗

Response to Reviewer 2JxA (3/3)

W1 - Methodological novelty

We agree with the reviewer that our method builds on prior work. Our contribution lies in assigning each component to the role it is suited for. A contrastive loss is used only where negatives are well defined, namely between objects within a view, and the negative-free LeJEPA objective where they are not, namely for semantics across a batch in which objects may share a category. As opposed to prior work, we argue that a good image encoder should be able to capture instance identity and semantics. Based on this principle, we train features to be projected into an instance and a semantic representation space as opposed to DetCon (that tries to learn semantics by contrasting instances) or supervised contrastive learning (that relies on mask labels to learn semantics but does not enforce instance separation). In contrast, we apply an instance-level contrastive loss only on patch representations from the same view where negatives are clearly defined. In our experimental setup this results in better performance on instance-related downstream tasks (more details in Q1). For semantics, we propose porting LeJEPA to an object-level and show that with copy-paste augmentations it learns modular object representations that capture object semantics without mask labels. Compared to contrastive learning with explicit negatives this leads to better semantic object representations (more details in Q2).

Q3 - 300 vs 150 epochs; convergence curves

We appreciate the reviewer's inquiry on convergence curves and will supply further results later in the discussion period, because the experiments are still running.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their constructive feedback. We have uploaded a revised paper in which all changes are marked in blue. Based on the reviewers' comments it contains new experiments: three ablations of the training objective in Table 2 (our method without object-centric LeJEPA, with an object-level InfoNCE loss in its place, and a label-supervised patch-level contrastive baseline), three initialization seeds for every point of Figure 3 with independently sampled subsets for the small data scales (Tables 3-5), and matched read-outs in Figure 3 and Table 2. The appendix now contains the full object-level SIGReg estimator, pseudocode of the patch aggregator and descriptions of all metrics and baselines. We answer each reviewer point by point individually below. Convergence experiments are still running and we will report them during the discussion period.

AuthorsOpenReview ↗

Update: convergence analysis added

We have uploaded an updated revision of the paper. It adds the convergence analysis announced in our general response (Appendix B.2, Figures 4 and 5): the training loss and four downstream tasks for Image LeJEPA, DetCon and our method at training lengths of up to 500 epochs. It replaces the epoch ablation of the submission. All changes remain marked in blue. The related questions of Reviewer 2JxA (Q3) and Reviewer 9ffq (Q5) are answered under their reviews.

We are happy to discuss any remaining questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.