- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work investigates how to improve the accuracy, computational efficiency, and downstream transfer performance of supervised Vision Transformers (ViTs). It explores three aspects: optimization, model architecture, and training strategy. First, it extends AdaBelief with stochastic updates informed by the mean and variance of historical gradients. Second, it introduces a cross-attention classifier in which learnable class-specific queries attend to relevant image regions, followed by gating and cosine-based scoring. Third, it combines these components with multi-scale training, varying the input image resolution during training.
Critical strengths
A key strength of the paper is its extensive empirical evaluation. The authors assess the proposed components across image classification, object detection, and vision-language tasks, and provide detailed ablations of the optimizer, classifier design, and training strategy. Additional evaluations across input resolutions and distribution shifts, together with analyses of convergence and computational cost, help establish both the effectiveness of the approach and the conditions under which its benefits arise.
Critical weaknesses
-
The proposed approach appears to be primarily an engineering integration of established ideas: adaptive optimization with gradient perturbations, attention-based feature aggregation, gating, cosine classification, and multi-scale training. Engineering contributions can be valuable, but the paper does not sufficiently clarify the fundamentally new insight that motivates these particular modifications or distinguishes them from prior approaches. While the extensive experiments demonstrate the effectiveness of the resulting recipe, they provide a stronger case for the usefulness of the combination than for the originality of the individual contributions. Especially, there are only modest gains from the proposed components. it is difficult to determine whether these gains are consistent and exceed normal training variability. The stronger results of the complete recipe also incorporate several established techniques, making them less conclusive evidence for the proposed components.
-
Insufficient motivation and unclear relevance to broader ViT pretraining paradigms. The paper does not clearly identify a specific limitation of existing training recipes or an unresolved research question that necessitates revisiting supervised ViT training. The main methodological ablations focus on supervised ImageNet training, yet the paper provides limited discussion of how its contributions relate to self-supervised approaches such as DINOv3 or image–text contrastive pretraining. Although downstream and vision-language experiments are included, they do not establish whether the central design extends to these pretraining paradigms. The paper therefore needs a stronger justification for why this particular setting warrants renewed attention and what broader lessons the proposed modifications offer beyond improving a supervised ImageNet recipe. In other words, a paper that aims to “revisit ViT” should examine whether its findings generalize across major pretraining paradigms rather than primarily refining a supervised ImageNet recipe.
Questions for authors
The paper states, “We hope that our recipe can be adopted and foster further research in vision transformers.” If the authors aim for a recipe-level advance comparable to what DeiT brought to ViT or ConvNeXt brought to ResNet, I would expect more substantial improvements to give the community a compelling reason to adopt it. Given the modest gains from the proposed components, what evidence supports the practical advantage of adopting this recipe over established alternatives?