acceptodds
Withdrawn from ICLR 2027

Revisiting Jigsaw Objectives for Fine-Grained Visual Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Current self-supervised models focus on semantic invariance, capturing similarity of content while ignoring fine-grained variations in the input. This is useful for tasks such as semantic segmentation and retrieval, but physical scene understanding, anomaly detection, and the guidance and evaluation of generative models require representations that capture fine details and spatial structure of the input. This motivates unsupervised objectives that teach models more about the structure of what they encode. Jigsaw-style objectives are a natural candidate. Decomposing an image and asking a model to reassemble it requires attending to boundaries and local structure, and the supervision comes entirely from the decomposition. Such objectives were explored in early self-supervised learning but largely abandoned after contrastive methods proved stronger on classification, and they have not been revisited at the scale or in the architectures now standard. In this paper, we revisit this family in a modern self-distillation framework. We study a range of formulations of the Jigsaw puzzle objective, several of which are new, and evaluate them on several benchmarks. Across tasks requiring fine detail, including object discovery, tracking, and anomaly detection, these objectives outperform prior self-supervised models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, vision transformers, pretext tasks, position prediction, jigsaw puzzles, spatial representation learning, dense prediction

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_upiYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors revisit jigsaw-style self-supervised objectives in a modern DINOv3 self-distillation framework. They compare relative position, absolute position, and permutation classification objectives, and introduce two new formulations: blockwise relative position and region matching. The approach is evaluated on a broad suite of frozen-feature benchmarks spanning dense prediction, correspondence, tracking, object discovery, saliency, boundary detection, anomaly localization, and global recognition. The results show that explicit spatial supervision can improve spatially sensitive tasks, with region matching being the only objective that improves both spatial and semantic performance relative to DINOv3.

Critical strengths
  • The paper addresses an important question in self-supervised representation learning: whether explicitly supervising spatial arrangement, rather than relying only on semantic invariance or masked prediction, improves fine-grained and spatially structured representations.
  • The study is well motivated and provides a useful controlled comparison of several jigsaw formulations under a common DINOv3 training setup, backbone, pretraining checkpoint, refinement schedule, and evaluation protocol.
  • The two proposed objectives, blockwise relative position and region matching, are interesting. Region matching in particular is a sensible formulation because it uses soft spatial targets over image regions rather than requiring exact tile-to-cell assignment.
  • The evaluation is broad and thus convincing, covering object discovery, saliency, boundary detection, anomaly localization, segmentation, depth, correspondence, video segmentation, point tracking, and global recognition.
  • The finding that different jigsaw formulations favor different capabilities is valuable: permutation classification appears strongest for figure-ground separation and object discovery, while relative-position variants are stronger for boundaries, anomaly localization, and tracking.
Critical weaknesses
  • Although the paper introduces two new objectives, the technical novelty is moderate. The core idea is a careful modernization and systematic empirical study of jigsaw objectives within DINOv3, rather than a fundamentally new learning principle. The contribution would be strengthened by clearer analysis of why each objective produces its observed representational trade-off.
  • The paper claims that jigsaw objectives improve fine-grained visual representations, but the mechanisms behind the improvements are not directly validated. For example, it remains unclear whether gains arise from better boundary sensitivity, better local correspondence, reduced feature collapse, or simply an additional dense prediction signal. Additional quantitative probing to clarify the mechanisms would be helpful.
  • The comparison is limited to a ViT-L/16 backbone, ImageNet-1k pretraining, and a relatively short 20k-iteration refinement phase. It is unclear whether the conclusions generalize to other backbone scales, longer training schedules, larger datasets, or models where the dense-feature degradation problem may be more or less pronounced.
  • Some results appear mixed and deserve more discussion. For example, several jigsaw variants clearly improve object discovery and anomaly detection but underperform DINOv3 or NeCo on SPair correspondence and depth estimation. The paper would benefit from a clearer explanation of when spatial supervision helps and when it harms downstream transfer.
  • The shortcut-suppression analysis in Table 5 is interesting but potentially confusing: adding augmentations changes the effective task difficulty, so it is difficult to isolate whether the observed shifts are due to removal of low-level shortcuts or to a change in the optimization landscape.
Questions for authors
  • What is the main source of the observed improvement: stronger boundary information, improved local feature discriminability, better object/background separation, or reduced degradation of dense features during refinement? Could this be disentangled using feature-similarity, boundary-localization, or patch-correspondence diagnostics?
  • Why does region matching improve both semantic and spatial benchmarks, whereas the other jigsaw objectives show a clearer trade-off? Is this because its soft region-level target avoids overfitting to exact tile positions, or because it preserves more global context?
  • How sensitive are the conclusions to the refinement duration and loss weight λ=0.3? 
In particular, would longer refinement cause the jigsaw objectives to over-suppress semantic invariance or eventually recover the semantic performance of DINOv3?
  • For the shortcut-suppression experiments, could you share the results for separating the effect of grayscale conversion, chromatic shifting, and per-tile normalization in order to better understand the effects of each?
Reviewer_joXxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits jigsaw supervision during DINOv3-style refinement of ViT-L/16. It compares relative position, absolute position, permutation classification, and new blockwise relative-position and region-matching variants. Most variants match shuffled student content to intact-image teacher features. Region predicts where a shuffled crop belongs using soft area-overlap labels. Models share an ImageNet-1k starting checkpoint and refinement duration. Frozen-feature evaluations cover object discovery, tracking, anomaly detection, dense prediction, and recognition.

Critical strengths
  1. The gains are substantial on several relevant tasks. In Table 1, Perm improves VOC07 CorLoc from DINOv3's 36.28 to 48.17 and DUTS saliency from 29.75 to 43.99. Blockwise RelPos reaches 86.28 MVTec AUPRO versus 81.11. Table 3 gives 56.06 versus 49.21 on DAVIS and 25.77 versus 19.97 on TAP-Vid. These are useful results across distinct feature-based evaluations, rather than an improvement confined to classification.
  2. The study reports informative tradeoffs. Table 2 shows that most jigsaw variants lose semantic segmentation and depth quality, while Region improves all five semantic segmentation columns. Tables 4-5 examine grid size and shortcut suppression instead of assuming that finer grids or stronger suppression always help. For example, augmenting RelPos 12x12 improves ADE20K from 19.06 to 20.56 but reduces MVTec AUPRO from 86.21 to 82.11. Shared initialization and refinement duration make these comparisons more informative than unrelated pretrained-model comparisons.
Critical weaknesses
  1. The comparison identifies effective complete variants more clearly than it isolates the spatial loss formulation. Appendix A.2 says Region retains full three-patch tiles while other objectives use two-patch tiles with gaps; Perm encodes tiles independently; relative-position variants use a periodically refreshed frozen teacher. Table 7 also lists Region-specific jitter/grayscale settings. These differences are legitimate design choices, but they complicate attributing Region's particular semantic/spatial balance to its target alone. Table 5 already shows that augmentation can materially change that balance. The conclusions should distinguish recipe-level findings from an explanation of which ingredient causes them.
  2. Generality beyond the tested training regime remains uncertain. Section 4 uses one 100k-iteration ImageNet-1k pretraining run at batch 64, followed by 20k refinement iterations, with one training run per model. Table 6's DINOv3 baseline reaches 49.80 ImageNet validation accuracy. Thus, these are comparisons under a specific short-training regime, rather than comparisons to released foundation-model checkpoints. The broad task coverage is valuable, but it does not establish that the same objective ranking persists with more mature representations. Small advantages, such as Region versus NeCo on several segmentation columns, are also difficult to interpret without run variability.
Questions for authors
  1. Which existing controls distinguish the effects of Region's loss target from tile coverage and augmentation? Please clarify which teacher and input settings are shared, and which conclusions concern complete recipes.
  2. In Eq. 11, writing projected tokens as a_j and b_h gives r_k = mean_j(a_j)^T sum_{h in R_k}(b_h) / sqrt(P). Is this the implemented score? If so, how should Appendix A.2's statement about an unpooled conjunction be understood, separately from interactions inside the encoder?
  3. Do existing checkpoint or repeated-run results indicate whether the principal gains persist later in training, or how variable the smaller differences are? Please distinguish available evidence from an extrapolation beyond this regime.
Reviewer_eTTKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits jigsaw-based self-supervised learning for fine-grained visual representations. The authors compare five spatial supervision objectives within a DINOv3-based framework, including two proposed objectives: Blockwise Relative Position and Region Matching. Experiments across diverse downstream tasks show that different objectives lead to different trade-offs between spatial and semantic representations, with Region Matching providing a favorable balance across several tasks.

Critical strengths

(1) The paper compares five spatial supervision objectives within a common DINOv3-based framework and evaluates them across diverse downstream tasks. This provides a useful empirical study of how different objective designs affect learned visual representations.

(2) Potentially useful insights into spatial-semantic trade-offs. The results suggest that different jigsaw objectives favor different downstream capabilities, rather than consistently improving all tasks. This observation is useful for understanding the limitations of spatial supervision and may help guide the design of future self-supervised learning objectives.

Critical weaknesses

(1) Limited novelty and insufficient differentiation from prior work. The paper builds on established jigsaw-based self-supervised objectives, while its main methodological contributions are Blockwise Relative Position and Region Matching. However, the distinction between the proposed Region Matching objective and existing region-matching approaches, such as EsViT, is not sufficiently established. The paper would benefit from a more explicit technical comparison and stronger evidence that the proposed formulations offer a substantial advance beyond prior work. This limits my confidence in the paper's originality.

(2) The paper reports grid-like artifacts in the feature maps produced by some jigsaw objectives, but their implications for the learned representations remain unclear. In particular, it is not established whether these artifacts reflect reliance on artificial patch-layout cues or whether they affect generalization to downstream tasks requiring fine-grained spatial understanding. Since the paper aims to improve fine-grained visual representations, a more systematic analysis of these artifacts and their relationship to downstream performance would strengthen the central claims.

Questions for authors

(1) Novelty relative to prior region-matching methods: Could the authors clarify the precise technical differences between the proposed Region Matching objective and existing approaches such as EsViT, particularly in the construction of supervision targets and feature matching? What evidence supports the claim that the proposed formulation provides a meaningful advance over these methods?

(2) Impact of grid artifacts on learned representations: The paper observes grid-like artifacts in the feature maps of some jigsaw objectives. Could the authors clarify whether these artifacts reflect the model learning artificial patch-layout cues rather than meaningful object structure? Is there evidence linking the severity of these artifacts to downstream performance, particularly on dense prediction tasks? Such an analysis would help clarify the limitations of the proposed objectives and strengthen the paper's claims about fine-grained representations.

Reviewer_4dErOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits jigsaw objectives for learning spatially informative visual representations within a DINOv3-style self-distillation framework. It compares relative position prediction, absolute position matching, a Sinkhorn variant, and permutation classification, and introduces blockwise relative position prediction and region matching. The former combines permutations within and across blocks, while the latter predicts a crop’s overlap distribution over regions of an intact reference image. All models start from a shared ViT-L/16 checkpoint pretrained on ImageNet-1k and undergo additional refinement, with the jigsaw objective replacing Gram anchoring. Evaluation uses frozen features across object discovery, saliency, segmentation, depth estimation, correspondence, tracking, anomaly detection, and recognition. The results indicate substantial differences between objectives. Permutation classification performs particularly well on object discovery and saliency, while blockwise relative position prediction is effective for tracking. Region matching provides a favorable balance between semantic and spatial performance among the default configurations, although its improvements do not extend to every task. Additional experiments examine grid size and augmentations intended to suppress low-level shortcuts. The main contribution is a comparative empirical study of spatial supervision, supplemented by two objective variants.

Critical strengths
  1. The paper evaluates spatial supervision across an unusually broad range of downstream tasks using frozen representations. The shared initialization and refinement schedule provide a useful basis for comparing objectives within the studied setting. Several reported improvements are substantial, including the increase in VOC07 CorLoc from 36.28 to 48.17 with permutation classification and the increase in DAVIS J&F from 49.21 to 56.06 with blockwise relative position prediction. These results support further investigation of jigsaw objectives beyond classification.

  2. The grid-size and shortcut-suppression experiments provide useful evidence about the sensitivity of learned representations to the pretext task. In particular, removing low-level cues can improve semantic segmentation while reducing gains in anomaly localization or object discovery. Reporting these competing effects makes the study more informative and helps explain why no single objective dominates across tasks.

Critical weaknesses
  1. The distinct contribution of the proposed objectives is not sufficiently established against closely related spatial pretraining methods. LOCA by Caron et al. published at WACV 2024, already combines query-to-reference patch localization with local representation learning, yet it is neither discussed nor evaluated. Spatial Prediction by Shen et al. is cited but not compared experimentally. Targeted controls are also missing for the new designs, particularly region matching without crop shuffling and blockwise prediction without within-block permutation. Moreover, Eq. 11 factorizes into the dot product between the mean projected crop feature and the summed projected region feature. This does not support the explanation in Appendix A.2 that the scoring mechanism preserves a conjunction of contents distinct from feature averaging. These omissions leave the benefit attributable to the proposed mechanisms insufficiently separated from established localization supervision and differences in input construction.

  2. The experimental evidence does not yet establish the reliability and scope of the main conclusions. All models use one shared ImageNet-1k pretraining run, followed by 20k refinement iterations, and each configuration is evaluated from a single training run. The reproduced DINOv3 baseline obtains 49.80 ImageNet linear-probe accuracy, making it important to establish whether the observed advantages persist with more fully trained representations. Small differences between leading methods are particularly difficult to interpret without uncertainty estimates. Appendix A.3 also reports the TAP-Vid occlusion threshold that maximizes AJ for each model and the best dense-probe checkpoint, without clearly identifying independent selection data. These choices require clarification before the corresponding gains can be interpreted as performance on held-out data.

Questions for authors
  1. Clarification of region matching. As written, Eq. 11 is equivalent to comparing a mean projected crop descriptor with a summed projected region descriptor. Does the implementation include an operation omitted from the equation that prevents this factorization? How should the explanation in Appendix A.2 be interpreted in light of this equivalence?

  2. Evidence for the proposed mechanisms. How much of the improvement remains when crop shuffling is removed from Region Matching and when within-block permutation is removed from Blockwise RelPos, with other settings fixed?

  3. Clarification of the training regime. Is the reported batch size of 64 global or per device, and how was the common pretraining checkpoint initialized? What evidence establishes that the baseline is sufficiently trained for the claimed comparison? Results for representative objectives from a released, well-trained checkpoint, or evidence that their relative gains persist with longer training, would help determine the scope of the findings.

  4. Clarification of evaluation and uncertainty. Which splits are used to select the TAP-Vid occlusion threshold and dense-probe checkpoints? Are the reported scores computed on data independent of those choices?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.