acceptodds
Withdrawn from ICLR 2027

Learning to Perceive “Where”: Spatial Pretext Tasks for Robust Self-Supervised Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing self-supervised learning (SSL) methods primarily learn object-invariant representations but often neglect the spatial structure and relationships among object parts. To address this limitation, we introduce Spatial Prediction (SP), a spatially aware pretext regression task that predicts the relative position and scale between a pair of disentangled local views from the same image. By modeling part-to-part relationships in a continuous geometric space, SP encourages representations to capture fine-grained spatial dependencies beyond invariant categorical semantics, thereby learning the compositional structure of visual scenes. SP is implemented as a decoupled plug-in and can be seamlessly integrated into diverse SSL frameworks. Extensive experiments show consistent improvements across image recognition, fine-grained classification, semantic segmentation, and depth estimation, as well as substantial gains in out-of-distribution robustness for object recognition. To evaluate spatial reasoning, we introduce (1) a position and scale prediction task on image patch pairs and (2) a jigsaw understanding task requiring patch reordering and recognition after reconstruction. Strong performance on these tasks indicates improved spatial structure and geometric awareness. Overall, explicitly modeling spatial information provides an effective inductive bias for SSL, leading to more structured representations and better generalization. Code and models will be released.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ELmwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Spatial Prediction (SP), a spatially-aware self-supervised pretext task that learns the relative position and scale between local views of the same image. SP can be seamlessly integrated into existing frameworks such as MAE, MoCo v3, and DINO without changing their original objectives. Experiments show that SP generally improves tasks such as spatial-relationship prediction, jigsaw, image recognition, segmentation, and depth estimation, etc. It suggests that explicit geometric supervision can produce more spatially structured visual representations.

Critical strengths

This paper presents a comprehensive experimental evaluation. The proposed Spatial Prediction (SP) module is tested across three representative self-supervised learning frameworks, MAE, MoCo v3, and DINO, covering reconstruction-based, contrastive, and self-distillation approaches. The evaluation span diverse tasks, including in-domain image classification, out-of-distribution transfer learning in image classification, semantic segmentation, and depth estimation. The authors also introduce dedicated spatial reasoning evaluations involving relative position and scale prediction, as well as jigsaw reconstruction.

Critical weaknesses

1. Limited methodological novelty. The pretext task of predicting spatial relationships between image regions has been explored in some closely-related work. LOCA [1] uses a ViT and cross-attention to locate query patches relative to a reference view, mainly targeting dense tasks such as semantic segmentation. More importantly, PART [2] is highly similar to the proposed method. It samples random off-grid patch pairs and targets the prediction of the same four-dimensional relation between the two patches: (Δx,Δy,Δw,Δh)(\Delta x, \Delta y, \Delta w, \Delta h). PART also evaluates these representations using ViT-S and ViT-B backbones on image classification and COCO object detection. Since the methodology and evaluation tasks in this paper are extremely similar to these previou work, the authors need explain the major novelty of their approach.

2. Insufficient comparison with stronger baselines. Most experiments only compare the proposed approach with the baseline model without SP supervision. The paper includes one comparison with PART on CIFAR-100 in the appendix, which is far from convincing as CIFAR-100 is an in-domain dataset. Some more pretext-based methods (e.g., SiameseIM [3], DropPos [4]) have reported substantially stronger performance in similar tasks (e.g., ImageNet-1K, COCO segmentation) after training on models like MoCo-v3 and MAE. Though the results are not comparable due to different evaluation protocols, the paper should also inclulde comparison with these stronger pretext-based baselines.

[1]: Location-Aware Self-Supervised Transformers for Semantic Segmentation

[2]: How PARTs assemble into wholes: Learning the relative composition of images

[3]: Siamese Image Modeling for Self-Supervised Vision Representation Learning

[4]: DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions

Questions for authors
  1. Are the models sufficiently converged after 100 epochs? The MAE baseline and MAE+SP achieve 44.6% and 52.6% on ImageNet-1K linear-probing accuracy, whereas the MAE baseline reaches approximately 68.0% (Table 12 MAE+ViT-B in the paper of MAE [1]). I fully understand that the number training epochs is different in your settings. But does the reported improvement only reflect faster early-stage optimization or could it has has a better converged representation?If the improvement cannot be witnessed on a converged model, why don't I just use the off-the-shelf MAE baseline?

  2. What is the practical advantage of SP over established SSL pretext methods? Earlier methods already report stronger or near-saturated ImageNet performance, such as MaskFeat [2], SiameseIM, and DropPos (see weakness 2). Several also perform strongly on detection, segmentation, etc. The authors need to clarify their advantage. (Is SP intended to improve final accuracy, training efficiency, robustness, or spatial transfer? )

  3. As described in Abstract, this paper aims to address the lack of SSL supervision targeting spatial structure and relationships. However, I believe that this is a question already solved. From LOCA and PART (see weakness 1) targeting traditional computer-vision tasks, to Visual Jigsaw [3] and Spatial-SSRL [4] targeting MLLM spatial reasoning, a handful of proposed SSL pretext tasks focus on spatial structure and relationships. So I wonder whether it is a bit overclaimed in the Abstract and what the major contribution of this paper is.

[1]: Masked Autoencoders Are Scalable Vision Learners

[2] Masked Feature Prediction for Self-Supervised Visual Pre-Training

[3]: Visual Jigsaw Post-Training Improves MLLMs

[4]: Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning

Reviewer_omaFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spatial Prediction (SP), an auxiliary self-supervised objective that regresses the relative position and scale of two local views of an image. SP combines a rejection-based crop sampler, projection-free attention pooling, and a small regression head, and is trained jointly with MAE, MoCo v3, or DINO. The evaluation covers classification, distribution shifts, transfer learning, segmentation, depth estimation, and two spatial diagnostics. The main results show substantial gains for MAE and MoCo v3, especially in linear probing and some transfer tasks, alongside more mixed results for DINO.

Critical strengths
  1. The paper offers a straightforward way to combine auxiliary geometric supervision with existing SSL objectives. Its integration into three different training frameworks, without adding a component to downstream inference.
Critical weaknesses
  1. The novelty over PART appears limited and needs a more precise characterization. PART already uses continuous relative-position regression and extends its targets to relative width and height. Neither relative-position prediction nor adding relative-scale targets is new.
  2. The experimental validation is incomplete. Although the extra-crop baseline is useful, the paper lacks a matched comparison with MoCo v3 + a PART-style spatial branch and a separate ablation of rejection sampling.
Questions for authors
  1. Could you provide the missing experiments identified in the weaknesses?
  2. Could you discuss how SP relates to spatial self-supervised learning methods such as CroCo, which also use target and reference views, and provide experiments evaluating SP in a 3D spatial self-supervised learning setting?
Reviewer_NRV5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims existing SSL mostly learn object-invariant representations but often ignore spatial structure and propose spatial prediction (SP) as a general regression pretext task. SP predicts relative position and scale between a pair of disentangled local views from the same image. SP is evaluated on image recognition, fine-grained classification, semantic segmentation and object recognition, across base models such as MAE, MoCov3 and DINO and show general improvements.

Critical strengths

The method is simple and clean. The method is generally applied to MAE, MoCov3 and DINO across most benchmarks. Useful controlled study.

Critical weaknesses

Some arguments in the paper are too strong: In abstract, "often neglect the spatial structure and relationships among object parts": MAE and DINO's random masking can learn these features as masking can partially remove parts of multiple object and reconstruct. The augmented view is constructed from human observation and not generally scalable (similarly MoCo etc. is not scalable). Can the author show the scalability of this method?

Questions for authors

Does the proposed method scalable? The augmented view and prediction are typically human observed and may not able to scale up and keep the gain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.