- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies mask-supervised visual pre-training under limited image diversity. A ViT produces patch features; a masked cross-attention aggregator forms one semantic representation per object, and an object-level LeJEPA loss aligns representations of the same instance across augmented views while a modified SIGReg term discourages collapse. A separate patch projection is trained with a within-view supervised contrastive loss that groups patches of the same instance and separates other objects and background. Copy-paste augmentation places donor objects into host contexts and treats the pasted occurrence as another view. Models are pre-trained on COCO with ground-truth, SAM, or Felzenszwalb–Huttenlocher masks and evaluated with frozen features on ADE20k instance/semantic probes, DAVIS tracking, NAVI re-identification, and ImageNet-1k classification. The reported point estimates generally exceed COCO-trained baselines, especially DetCon, and the paper studies dataset sizes from 256 images to full COCO.
Critical strengths
- The semantic/instance decomposition is coherent and task-motivated. Section 3 uses a negative-free object-level objective for invariance and a separate within-scene contrastive head for instance separation, making the intended roles of the losses easy to identify.
Critical weaknesses
-
Initialization and subset-composition variance are most consequential in the regime that defines the paper’s contribution. “Best in all ten settings” and “down to 256 images” are therefore observations about the reported runs, not statistically established comparative claims. K-means restarts and NAVI memory-bank resampling do not replace independent pre-training runs.
-
The proposed method can use its learned cross-attention object representation z, whereas competitors use mask-averaged patch features; both ADE20k object classification and NAVI additionally require ground-truth downstream masks.
-
The manuscript computes ROC-AUC per image after retaining only same-category patch pairs, but does not define what happens when the retained labels contain only one class or no eligible pairs.
Questions for authors
- What are the mean, standard deviation or confidence interval, and paired win rate across independent training seeds and independently sampled 256/512/1024-image subsets for Ours, DetCon, and Image LeJEPA? Are the Figure 3 subsets nested, and can the image IDs and object/category counts be provided?
- What is the complete object-SIGReg equation and implementation: which object-view representations enter each estimate, how are repeated views and pasted copies weighted, how are variable counts reduced across GPUs, and what changes when the actual object count replaces ?
- Exactly which ADE20k images and patch pairs enter the same-category ROC-AUC, how are images with only one target label handled, and how many validation images remain under that rule?
- Do the NAVI data-scale conclusions and the claim of superiority over all COCO-trained baselines hold when every method uses the identical mask-averaged patch readout? Which of the reported object-level settings actually require masks at deployment?
- Can the authors report per-method encoder-view counts, FLOPs/GPU-hours, convergence evidence, and a controlled comparison of SIGReg versus contrastive or teacher–student object alignment with masks, aggregator, augmentations, and auxiliary losses fixed?
7 comments
Response to Reviewer 9ffq (2/4)
W2, Q4 - Readout asymmetry and masks at test time, which settings need masks at deployment
We agree that comparing mask-averaged patch features with semantic object representations is not a fair comparison. Our main results (Table 1 in main paper) already show that our mask-averaged patch features also outperform all COCO-trained models on ADE20k object classification. On NAVI re-identification they outperform all COCO-trained models except CP$^2$ ( vs. ), but it is important to note that CP$^2$ performs considerably worse on every other downstream task. We now also report mask-averaged patch features for the data-scale experiments and the ablation table. With the matched read-out our method still wins all paired comparisons on NAVI against Image LeJEPA and against DetCon (Table 5 in Appendix A.7 of the revised paper), with smallest margins of and . The data-scale conclusions on subsets of , and images therefore hold with matched read-outs.
Masks at deployment. The encoder backbone never receives a mask, so patch features are obtained without one in every setting. The settings differ in whether the task built on these features needs masks. Image classification on ImageNet-1k uses none at any stage. Linear segmentation and the same-class probe need none at deployment: masks serve only as evaluation labels, and for the probe additionally to select the same-class, different-instance patch pairs on which we measure how well patch features encode instance identity beyond semantics. Tracking is given the first-frame mask that defines the task, and the propagation to later frames relies on patch features alone. Instance clustering does require masks, since we measure how well instances can be separated among foreground patches and therefore use the ground-truth masks to select these patches and to set the number of clusters. The two object-level settings, ADE20k object classification and NAVI re-identification, require one mask per object for every method and both read-outs, since the mask defines which patches are pooled. The semantic object representation additionally takes the mask as input to the aggregator .
W3, Q3 - Same-category ROC-AUC edge cases, Which images and pairs enter the same-category ROC-AUC?
We thank the reviewer for pointing this out and apologize for the unclear description. We describe the evaluation protocol in more detail below and have added it to the implementation details in the revised paper. After training the quadratic probe on frozen patch features, we assign each patch to the instance that covers the largest number of its pixels, with the background as one of the candidates, and discard patches assigned to the background. From the remaining foreground patches we sample up to per image and collect their features together with their instance and semantic labels. The sampling is seeded, so the same patches are used across runs and across all methods that share the same patch grid. Each patch inherits the instance and semantic label of its assigned mask. Pairs of patches that belong to the same instance are positive pairs. Pairs of patches that share the same semantic class but belong to different instances are negative pairs. The ROC-AUC is calculated for images that have at least one positive and at least one negative pair. We report the mean over all calculated ROC-AUC metrics. For patch feature maps (patch size on images), this is an average over 1894 of the 2000 ADE20k validation images. For patch feature maps (DINOv2 and FT-DINOSAUR, patch size on images), it is 1895 images.
Response to Reviewer 9ffq (3/4)
Q2 - Complete object-SIGReg equation and implementation
We use the Epps-Pulley/slicing implementation of the official LeJEPA code, applied to objects instead of images. Each of the images yields views. For let be the set of -th views of every image in the global batch. As in LeJEPA, SIGReg is estimated per view index, here over all foreground objects present in those views:
\mathcal{S}_j = \lbrace(v,k): v \in \mathcal{V}^{(j)}, k \in \mathcal{K}, v \in \mathcal{V}_k\rbrace, \quad N_j = |\mathcal{S}_j| \end{aligned}$$ where $N_j$ corresponds to the number of object occurrences in all $B$ $j$-th views of the global batch, pasted copies included. For a unit direction $a \in \mathbb{A}$, we define the view-specific empirical characteristic function $\hat{\varphi}_{j,a}$ of the $a$-projected semantic object representations $\tilde{\mathbf{z}}$ $$\begin{aligned} \hat{\varphi}_{j,a}(t) &= \frac{1}{N_j}\sum_{(v,k) \in \mathcal{S}_j} e^{ita^\top\tilde{\mathbf{z}}_{v,k}}\\\\ &=\frac{1}{N_j}\sum_{(v,k) \in \mathcal{S}_j} \cos(ta^\top\tilde{\mathbf{z}}_{v,k}) + i \frac{1}{N_j}\sum_{(v,k) \in \mathcal{S}_j} \sin(ta^\top\tilde{\mathbf{z}}_{v,k}). \end{aligned}$$ The per-view, per-projection Epps-Pulley statistics $\text{EP}_{j,a}$ are then averaged over the directions and the views to form the SIGReg loss, $$\begin{aligned} \mathcal{L}_\text{SIGReg} = \frac{1}{V} \sum_{j=1}^V \frac{1}{|\mathbb{A}|} \sum_{a\in \mathbb{A}} \text{EP}_{j,a}, \quad\quad \text{EP}_{j,a} = B \int_{-3}^3 |\hat{\varphi}_{j,a}(t) - e^{-t^2/2}|^2 e^{-t^2/2} dt \end{aligned}$$ The integral is evaluated with the trapezoidal rule on $17$ knots over $[0,3]$, doubled by symmetry, as in LeJEPA's released implementation. We sample $|\mathbb{A}|=1024$ directions every step from a step-seeded generator shared by all GPUs. **Which objects enter?** Every foreground object present in a view (mask covering more than $16\times 16$ pixels, see Section 3.1) enters that view index’s Epps-Pulley estimate. This includes objects that are present in only one view and are therefore not in $\mathcal{K}^\mathrm{eff}$, the set of objects present in at least two views, to which the alignment term is restricted. **How are repeated views and pasted copies weighted?** Each occurrence $(v,k)$ has weight $\frac{1}{N_j}$ within its view index. A pasted object keeps its identity $k$ and $\mathcal{V}_k$ contains views of both donor and host. It therefore enters $\mathcal{S}_j$ twice, once in the donor's $j$-th view and once, in a new context, in the host's $j$-th view. This applies to at most $B/2$ of the $N_j$ samples per view index. Empirically (200 training steps, $B=256$), $95.6$ of the $128$ possible hosts receive a paste per step. The resulting second entries make up $9.1$% of the $N_j$ samples in global views ($91.8$ of $1007$) and $11.9$% in local views ($62.4$ of $524$). **How are variable counts reduced across GPUs?** LeJEPA averages the per-GPU characteristic functions, which equals the global mean because every GPU holds the same number of images. Object counts differ across GPUs, so we instead all-reduce (SUM) the presence-masked sums $\sum \cos(ta^\top\tilde{\mathbf{z}}_{v,k})$ and $\sum\sin(ta^\top\tilde{\mathbf{z}}_{v,k})$ together with the object counts, and then divide. $\hat{\varphi}_{j,a}$ is therefore the exact mean over all $N_j$ objects in the global batch. The all-reduce is autograd-aware, as in LeJEPA. **What changes when the actual object count replaces $B$?** Using the actual object count would effectively rescale SIGReg in view $j$ by $N_j/B$. The regularizer's magnitude would vary from batch to batch with scene content. Furthermore, it would regularize object representations from global views more than ones from local views, since larger crops yield higher object counts. Empirically ($B{=}256$), a global view index holds $N_j/B = 3.93$ objects per image on average and a local one $2.05$, so SIGReg would weigh global-view objects $1.92\times$ more than local-view ones. Across steps, $N_j/B$ ranges from $3.63$ to $4.31$ in global views and from $1.89$ to $2.22$ in local views (2.5-97.5th percentiles). In early development we observed more stable gradients with the constant scale $B$ than with $N_j$. We did not repeat this as a controlled ablation with the final method, so the revised paper no longer claims it and motivates the choice by the statistics above. The estimator and these details are now in Appendix A.4 of the revised paper.
Response to Reviewer 9ffq (1/4)
We thank the reviewer for the detailed and precise review, which led to several additions. We now report data-scale experiment results across three initialization seeds and three independently sampled subsets, report matched read-outs in Figure 3 and Table 2, present the formulation of the complete object-level SIGReg estimator in Appendix A.4, and specify the same-category ROC-AUC protocol in Appendix A.6. A controlled comparison against a contrastive object-level objective is added to Table 2. Convergence curves will follow later in the discussion period.
W1, Q1 - Seed and subset variance
We agree with the reviewer that initialization and subset-composition variance needs to be measured to support our claims. Therefore, we repeated the ViT-B data-scale experiments on the 256, 512 and 1024 image subsets. Starting from the run of the submission (initialization seed 0, subset seed 0), we trained two further initialization seeds on the same subset, and trained on two further independently sampled subsets with the same initialization seed. This gives three initialization seeds and three subset seeds per subset size, with one run shared between them, i.e., 5 runs per method and subset size.
The variation in both directions is small compared to the difference between our method and the two baselines (DetCon, Image LeJEPA) for all downstream task performances. As an example, for tracking on DAVIS with 256 images the means over the three initialization seeds are 0.426, 0.492 and 0.633 for Image LeJEPA, DetCon and our method, with ranges (maximum minus minimum) of 0.017, 0.002 and 0.009. Over the three subsets the means are 0.424, 0.496 and 0.634 with ranges of 0.012, 0.006 and 0.006. Both are much smaller than the gaps between methods. All runs are listed in Table 4 (Appendix A.7) of the revised paper. Pairing the runs by seed gives 15 comparisons per task (5 per subset size). Our method wins 15 of 15 against Image LeJEPA and 15 of 15 against DetCon on each of the four downstream tasks, a paired win rate of 100% (Table 5, Appendix A.7). Figure 3 now shows the mean and range over three initialization seeds for every point.
The COCO-subsets in these experiments are nested and the object/category counts are displayed in Table 3 in Appendix A.7. The number of instances remains similar across seeds for each subset size (e.g. number of instances in 256 images: 1839, 1896, 1863). The subsets of 1024 images contain 79, 80 and 80 of the 80 categories for the three subset seeds, the subsets of 256 images 76, 80 and 78. We also provide the super-category counts for each subset in the appendix. Image IDs can be found in the code repository in the supplementary material and will be published.