- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper conducts an extensive evaluation of various “vision-native” models across a suite of benchmarks. There are 3 main settings: 1) a taxonomy of vision-native models, and how the representations compare on classic computer vision tasks (classification, object detection, semantic segmentation, depth, video classification) against each other and supervised/vlm/mllm models as a reference; 2) an evaluation of 4 vision native models against two references on an extended suite of tasks and under different probing settings; 3) a controlled study of vision native models, and re-examining specific techniques in a controlled setting.
Critical strengths
- The evaluation and scale of the study is comprehensive. I appreciate the inclusion of both the ASIS (original/released setting) protocol and CTRL (controlled pretraining) protocols. The study is useful to assess the representation of models like DINO-v3 against VLMs and frontier models: this is an important open question.
- Amongst the several other takeaways, the controlled evaluation section shows the importance of revisiting early techniques, e.g. the improved performance of Instance Discrimination, DINO-v3’s shared head, and improved performance under multicrop. The paper is also usually upfront about the scope of these takeaways (e.g., L329, pointing out that the gains from multi-crop may stem from additional compute).
Critical weaknesses
- A hyperparameter sensitivity analysis is largely missing. This affects the significance of the overall findings and rankings. While I understand a full hyperparameter sweep is computationally expensive, that is the tradeoff of having several models and experiments. Some results look like collapses (e.g., V-JEPA 2.1 on ADE20K under AP). Is this a failure of optimization rather than representation quality?
- The object detection mAP values on COCO (e.g, 80+ for DINOv3) seem extremely high. Very strong models score under 70 [1].
[1] DETRs with Collaborative Hybrid Assignments Training
Questions for authors
- Related to the weakness, a smaller scale hyperparameter sweep on a set of models would be informative.
- Why does the performance of V-JEPA 2.1 fall so much under attentive probing in ADE20K (Figure 3)? I find this surprising, since most models seem to improve under AP.
- Please clarify the metric used for object detection.
- The frontier model comparison is useful. It has been noted that MLLMs are sensitive to the prompt [1-3] on fine grained tasks, so I would weaken the claim that DINOv3 performs better.
I am happy to increase my score if these points (particularly 1-3) are clarified.
[1] Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
[2] DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM
[3] How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks