acceptodds
Withdrawn from ICLR 2027

What Have Vision Models Learned in the Deep Learning Era?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

What have vision models learned in the deep learning era, and how general are their representations? We systematically revisit the decade of visual self-supervised learning (VSSL) and geometry, generation, and video models, with references including supervised models, vision-language models (VLMs), multimodal world models (MMWMs), and frontier models. Through over 2,500 training and evaluation experiments, we distinguish original-setting comparisons, controlled pre-training, and common downstream evaluation on the Basic Five: image classification, object detection, segmentation, depth estimation, and video classification, complemented by an extended transfer suite. All scores are computed from our own environment, never copied from prior reports. Our main findings connect the past, present, and future directions of visual representation learning. (Past) Over the past decade, vision-native representations have become substantially more general, supporting transfer across semantics, dense prediction, and geometry, although progress and model strengths remain task-dependent. (Present) Modern vision-native representations, exemplified by DINOv3, outperform the evaluated VLM and MMWM references on multiple visual tasks, demonstrating broad transfer from vision-native learning itself. (Future direction) These achievements are not fixed endpoints: learning-design changes improve both classical and modern VSSL approaches, while scaling yields further gains across all five core tasks. Moreover, a separate matched-subset comparison with GPT-6 Astra reveals complementary strengths: DINOv3 performs better on ADE20K semantic-point accuracy and comparably on NYUv2 relative-depth accuracy. Overall, these results connect historical progress to the current frontier and future directions, providing an empirical account of what vision models have learned and a foundation for developing more general vision-native representations.

Primary area:
datasets and benchmarks
Keywords:
Vision Models, Visual Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ypa3OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducts an extensive evaluation of various “vision-native” models across a suite of benchmarks. There are 3 main settings: 1) a taxonomy of vision-native models, and how the representations compare on classic computer vision tasks (classification, object detection, semantic segmentation, depth, video classification) against each other and supervised/vlm/mllm models as a reference; 2) an evaluation of 4 vision native models against two references on an extended suite of tasks and under different probing settings; 3) a controlled study of vision native models, and re-examining specific techniques in a controlled setting.

Critical strengths
  1. The evaluation and scale of the study is comprehensive. I appreciate the inclusion of both the ASIS (original/released setting) protocol and CTRL (controlled pretraining) protocols. The study is useful to assess the representation of models like DINO-v3 against VLMs and frontier models: this is an important open question.
  2. Amongst the several other takeaways, the controlled evaluation section shows the importance of revisiting early techniques, e.g. the improved performance of Instance Discrimination, DINO-v3’s shared head, and improved performance under multicrop. The paper is also usually upfront about the scope of these takeaways (e.g., L329, pointing out that the gains from multi-crop may stem from additional compute).
Critical weaknesses
  1. A hyperparameter sensitivity analysis is largely missing. This affects the significance of the overall findings and rankings. While I understand a full hyperparameter sweep is computationally expensive, that is the tradeoff of having several models and experiments. Some results look like collapses (e.g., V-JEPA 2.1 on ADE20K under AP). Is this a failure of optimization rather than representation quality?
  2. The object detection mAP values on COCO (e.g, 80+ for DINOv3) seem extremely high. Very strong models score under 70 [1].

[1] DETRs with Collaborative Hybrid Assignments Training

Questions for authors
  1. Related to the weakness, a smaller scale hyperparameter sweep on a set of models would be informative.
  2. Why does the performance of V-JEPA 2.1 fall so much under attentive probing in ADE20K (Figure 3)? I find this surprising, since most models seem to improve under AP.
  3. Please clarify the metric used for object detection.
  4. The frontier model comparison is useful. It has been noted that MLLMs are sensitive to the prompt [1-3] on fine grained tasks, so I would weaken the claim that DINOv3 performs better.

I am happy to increase my score if these points (particularly 1-3) are clarified.

[1] Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

[2] DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM

[3] How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks

Reviewer_No9DOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper analyzes many vision SSL model, generative and video models with VLMs, multi-modal world models and frontier models with more than 2500 training and evaluations with their own run. Typical eval tasks are: image classification, object detection, segmentation, depth estimation and video classification. The paper story sets the finding with past (vision native model progresses significantly), present (vision-native model eg DINOv3 stands out), future (learning-design changes improve both classical and modern SSL methods and scaling gets further gains).

Critical strengths

The paper is comprehensive and with very long and detailed appendix. The models being tested are broad and covers various learning objectives.

Critical weaknesses

The comparison in this paper is more on empirical/system level side and the conclusion is not convincing. For example, models differ significantly on training data, not just training objectives etc. One main reason DINOv3 shines because the benchmarks used in this paper are used as seed by DINOv3 for curation (Page 8) and DINOv2 (Page 30 Table 15). On the other hand, WebLI used by SigLIP is curated from Google Image Search. So some model may overfit benchmarks used in this paper and others may not.

Though the author present CTRL in future part, the setup is just ViT-B/16 on ImageNet, which is too small to draw conclusion for 32 methods and methods may be tuned for ImageNet already.

The story of past/present/future is not meaningful: any analysis research paper can have a view of the progress of a model (eg DINO series model). Not sure the motivation of split into 3.

The title has no concrete information about what this paper aims to sell. 'in the deep learning era' is a default choice in 2026? and those benchmarks used in this paper are not significant enough to show generally 'What have vision models learned'.

Questions for authors

N/A

Reviewer_PKgGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the development of visual representations through a large collection of experiments spanning historical methods and recent foundation models. It combines evaluations under original and controlled training settings with linear probing, attentive probing, and fine-tuning across multiple downstream tasks. The goal is to characterize progress in visual learning and understand the strengths and limitations of different approaches.

Critical strengths
  • The paper addresses an important question and brings together an unusually broad range of models and tasks. Evaluating beyond image classification is valuable for understanding how representations transfer.
  • Thanks for open sourcing the code! It is quite commendable.
Critical weaknesses
  • The comparisons do not adequately support the broader conclusions about progress in visual learning. The evaluated foundation models differ in parameter count, pretraining data, supervision, and training compute. Their scores describe particular checkpoints under particular evaluation recipes, but do not isolate the contribution of the learning approach. The controlled experiments partly address this concern, yet downstream heads and training schedules are not fully standardized. Consequently, it remains difficult to determine which conclusions reflect better representations and which reflect differences in resources or evaluation choices. This is central to the paper’s contribution, rather than a minor limitation
  • Several results require validation before the rankings can be interpreted reliably. For example, V-JEPA 2.1 obtains 36.10 mIoU with linear probing but only 0.22 with attentive probing on ADE20K, and its fine-tuned COCO AP is 0.07. These kind of inconsistencies could be found across the paper.
Questions for authors
    1. Which central conclusions remain supported after separating checkpoint comparisons from comparisons of learning approaches? Within the controlled experiments, which comparisons also hold the downstream head, optimizer, and schedule fixed?
    1. Could the author revisit the evaluation protocol to identify if there are any probing issues?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.