- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Spatial Prediction (SP), a spatially-aware self-supervised pretext task that learns the relative position and scale between local views of the same image. SP can be seamlessly integrated into existing frameworks such as MAE, MoCo v3, and DINO without changing their original objectives. Experiments show that SP generally improves tasks such as spatial-relationship prediction, jigsaw, image recognition, segmentation, and depth estimation, etc. It suggests that explicit geometric supervision can produce more spatially structured visual representations.
Critical strengths
This paper presents a comprehensive experimental evaluation. The proposed Spatial Prediction (SP) module is tested across three representative self-supervised learning frameworks, MAE, MoCo v3, and DINO, covering reconstruction-based, contrastive, and self-distillation approaches. The evaluation span diverse tasks, including in-domain image classification, out-of-distribution transfer learning in image classification, semantic segmentation, and depth estimation. The authors also introduce dedicated spatial reasoning evaluations involving relative position and scale prediction, as well as jigsaw reconstruction.
Critical weaknesses
1. Limited methodological novelty. The pretext task of predicting spatial relationships between image regions has been explored in some closely-related work. LOCA [1] uses a ViT and cross-attention to locate query patches relative to a reference view, mainly targeting dense tasks such as semantic segmentation. More importantly, PART [2] is highly similar to the proposed method. It samples random off-grid patch pairs and targets the prediction of the same four-dimensional relation between the two patches: . PART also evaluates these representations using ViT-S and ViT-B backbones on image classification and COCO object detection. Since the methodology and evaluation tasks in this paper are extremely similar to these previou work, the authors need explain the major novelty of their approach.
2. Insufficient comparison with stronger baselines. Most experiments only compare the proposed approach with the baseline model without SP supervision. The paper includes one comparison with PART on CIFAR-100 in the appendix, which is far from convincing as CIFAR-100 is an in-domain dataset. Some more pretext-based methods (e.g., SiameseIM [3], DropPos [4]) have reported substantially stronger performance in similar tasks (e.g., ImageNet-1K, COCO segmentation) after training on models like MoCo-v3 and MAE. Though the results are not comparable due to different evaluation protocols, the paper should also inclulde comparison with these stronger pretext-based baselines.
[1]: Location-Aware Self-Supervised Transformers for Semantic Segmentation
[2]: How PARTs assemble into wholes: Learning the relative composition of images
[3]: Siamese Image Modeling for Self-Supervised Vision Representation Learning
[4]: DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
Questions for authors
-
Are the models sufficiently converged after 100 epochs? The MAE baseline and MAE+SP achieve 44.6% and 52.6% on ImageNet-1K linear-probing accuracy, whereas the MAE baseline reaches approximately 68.0% (Table 12 MAE+ViT-B in the paper of MAE [1]). I fully understand that the number training epochs is different in your settings. But does the reported improvement only reflect faster early-stage optimization or could it has has a better converged representation?If the improvement cannot be witnessed on a converged model, why don't I just use the off-the-shelf MAE baseline?
-
What is the practical advantage of SP over established SSL pretext methods? Earlier methods already report stronger or near-saturated ImageNet performance, such as MaskFeat [2], SiameseIM, and DropPos (see weakness 2). Several also perform strongly on detection, segmentation, etc. The authors need to clarify their advantage. (Is SP intended to improve final accuracy, training efficiency, robustness, or spatial transfer? )
-
As described in Abstract, this paper aims to address the lack of SSL supervision targeting spatial structure and relationships. However, I believe that this is a question already solved. From LOCA and PART (see weakness 1) targeting traditional computer-vision tasks, to Visual Jigsaw [3] and Spatial-SSRL [4] targeting MLLM spatial reasoning, a handful of proposed SSL pretext tasks focus on spatial structure and relationships. So I wonder whether it is a bit overclaimed in the Abstract and what the major contribution of this paper is.
[1]: Masked Autoencoders Are Scalable Vision Learners
[2] Masked Feature Prediction for Self-Supervised Visual Pre-Training
[3]: Visual Jigsaw Post-Training Improves MLLMs
[4]: Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning