acceptodds
Under review as a conference paper at ICLR 2027

Revisiting supervised transformers: An optimized recipe

Abstract

We revisit supervised Vision Transformers training and study two components: the optimizer and the classifier. We combine gradient sampling (GS), with AdaBelief optimizer and show that gradient perturbation using running gradient statistics improves downstream performance and fine-tuning convergence. We further propose a lightweight cross-attention classifier that trains end-to-end with the backbone using multi-scale training. We show that multi-scale training with cross-attention improves accuracy, efficiency and robustness. Our models transfers well to object detection and beat MAE-pretrained detectors using less epochs. We further combine our model with modern architectural components, and our final recipe reaches 86.5% in ImageNet-1K using ViT-L and 55.8 mAP in COCO using ViT-B.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
supervised image classification, optimization, vision transformer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uYXUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates how to improve the accuracy, computational efficiency, and downstream transfer performance of supervised Vision Transformers (ViTs). It explores three aspects: optimization, model architecture, and training strategy. First, it extends AdaBelief with stochastic updates informed by the mean and variance of historical gradients. Second, it introduces a cross-attention classifier in which learnable class-specific queries attend to relevant image regions, followed by gating and cosine-based scoring. Third, it combines these components with multi-scale training, varying the input image resolution during training.

Critical strengths

A key strength of the paper is its extensive empirical evaluation. The authors assess the proposed components across image classification, object detection, and vision-language tasks, and provide detailed ablations of the optimizer, classifier design, and training strategy. Additional evaluations across input resolutions and distribution shifts, together with analyses of convergence and computational cost, help establish both the effectiveness of the approach and the conditions under which its benefits arise.

Critical weaknesses
  1. The proposed approach appears to be primarily an engineering integration of established ideas: adaptive optimization with gradient perturbations, attention-based feature aggregation, gating, cosine classification, and multi-scale training. Engineering contributions can be valuable, but the paper does not sufficiently clarify the fundamentally new insight that motivates these particular modifications or distinguishes them from prior approaches. While the extensive experiments demonstrate the effectiveness of the resulting recipe, they provide a stronger case for the usefulness of the combination than for the originality of the individual contributions. Especially, there are only modest gains from the proposed components. it is difficult to determine whether these gains are consistent and exceed normal training variability. The stronger results of the complete recipe also incorporate several established techniques, making them less conclusive evidence for the proposed components.

  2. Insufficient motivation and unclear relevance to broader ViT pretraining paradigms. The paper does not clearly identify a specific limitation of existing training recipes or an unresolved research question that necessitates revisiting supervised ViT training. The main methodological ablations focus on supervised ImageNet training, yet the paper provides limited discussion of how its contributions relate to self-supervised approaches such as DINOv3 or image–text contrastive pretraining. Although downstream and vision-language experiments are included, they do not establish whether the central design extends to these pretraining paradigms. The paper therefore needs a stronger justification for why this particular setting warrants renewed attention and what broader lessons the proposed modifications offer beyond improving a supervised ImageNet recipe. In other words, a paper that aims to “revisit ViT” should examine whether its findings generalize across major pretraining paradigms rather than primarily refining a supervised ImageNet recipe.

Questions for authors

The paper states, “We hope that our recipe can be adopted and foster further research in vision transformers.” If the authors aim for a recipe-level advance comparable to what DeiT brought to ViT or ConvNeXt brought to ResNet, I would expect more substantial improvements to give the community a compelling reason to adopt it. Given the modest gains from the proposed components, what evidence supports the practical advantage of adopting this recipe over established alternatives?

Reviewer_jA3DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an optimized training recipe for Vision Transformers (ViTs) with two main components: (i) a Gradient Sampling (GS) optimizer that adaptively adjusts the optimization update through gradient sampling and (ii) a gated cross-attention classifier combined with multi-scale training to improve performance while reducing computational cost. Extensive empirical evaluations across multiple datasets, downstream tasks, and ViT architectures demonstrate improvements over standard design choices, such as the AdamW optimizer and conventional classification heads. Overall, the paper is well organized, and the proposed methods are straightforward and easy to follow.

However, despite the extensive empirical evaluation, the improvements attributable to the proposed components are generally marginal, and the lack of variance estimates makes it difficult to assess whether these gains are statistically meaningful and consistent across training runs. Additionally, the two proposed components, Gradient Sampling and the cross-attention classifier, offer limited methodological novelty relative to existing approaches. While the optimized training recipe has practical value, these limitations reduce the significance and strength of the demonstrated contribution. Considering these factors, I lean toward rejection.

Critical strengths

S1. Simple and practical methodology. The proposed recipes are straightforward and easy to understand and can be readily integrated into existing ViT training pipelines without substantial modification.

S2. Extensive empirical evaluation. The authors provide comprehensive experiments across multiple datasets, ViT architectures, and downstream tasks, from object detection to VQA. These evaluations demonstrate the applicability of the proposed methods across different settings.

S3. Improved training efficiency. The proposed recipe achieves competitive performance while reducing training time (Table 6f).

Critical weaknesses

W1. Limited theoretical justification. The proposed Gradient Sampling optimizer lacks sufficient theoretical justification, particularly regarding the relationship between gradient variance and local minima. The analysis in Section 3.1 does not adequately explain why stochastic gradient sampling improves optimization.

W2. Marginal empirical improvements. Despite extensive experiments, the improvements attributable to the proposed components are generally marginal (e.g., 0.1–0.2 percentage points in Tables 2, 3, and 6). Considering such small margins, the lack of variance estimates across multiple runs makes it difficult to assess the statistical significance and reliability of these improvements, limiting the contribution.

W3. Minor writing and presentation issues. Despite the generally clear writing, the paper contains several minor grammatical errors (e.g., “Our models transfers well” in the abstract). Some tables (e.g., Tables 1, 4, and 8) could also be presented more clearly. However, these issues do not affect my overall recommendation.

Questions for authors

Q1. Can the authors provide additional justification or evidence supporting the relationship between historical gradient variance and local minima, which motivates the proposed Gradient Sampling optimizer? Furthermore, the proposed update introduces both an additional deterministic drift and a stochastic perturbation, but their individual contributions are not clearly isolated. An ablation that separately evaluates the momentum drift and zero-mean Gaussian perturbation would help clarify the contribution of each component.

Q2. Can the authors report variance estimates across multiple training runs for the main comparisons, particularly those showing improvements of only 0.1–0.2 percentage points?

Reviewer_BhFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a a revised training procedure for Vision Transformers using gradient sampling together with an AdaBelief optimizer. Further, a cross-attention classifier is used that leverages multi-scale training by pooling multi-resolution features. The proposed optimizer and classifier are evaluated in a diverse set of tasks, with small but consistent improvements.

Critical strengths

S1. Experiments across a wide range of large datasets and tasks.

S2. Simple method, that can be integrated in a wide range of existing ViT tasks.

Critical weaknesses

W1. Small overall improvements in performance and convergence in all but one benchmark (Table 4 shows clear benefits).

W2. No measures of uncertainty in the results. Given that the results are all very close, often differing by only 0.1 pts, either results over multiple seeds (at least for a subset of experiments) or bootstrapped estimates should be reported.

W3. The paper presentation and write-up seem rushed, without a clear story outline. Together with having two proposed contributions (classifier and gradient sampling), this makes it difficult to see the clear advantages of the method. Further, important analyses, like the computational analysis, are in the appendix.

W4. The overall novelty is limited, given that it mostly combines existing parts. This would not be an issue for me, if the empirical improvements would be stronger.

Minor weaknesses

  • The paper uses overly informal and subjective language, e.g., in the caption of Figure 1: "Our proposed pipeline combines strong architectural components, ..." What does "strong" mean here, or what does it add to the understanding of the figure? Or in the caption of Figure 3: "Our optimizer clearly converges faster in all cases," which is not quantitatively measured, and AdamW seems to be very close.
  • The presentation should be improved: using version names like v5, v6, etc. for the different methods is difficult to follow.
  • Overall, the table placement should be better aligned, e.g., Tables 6 and 9. Further, the captions of figures and tables should be improved to be more descriptive.
Questions for authors

1. Can you expand the discussion on the convergence analysis in Figure 3? I have two concerns here that should be addressed in particular.

  • 1.a. Can you quantify and/or theoretically analyse the faster convergence behaviour?
  • 1.b. How relevant is the faster convergence if the overall training is short anyway (20 epochs)? Especially since the plots show that the final performance is better than the early one, where the sampling approach has an early lead.

2. Can you provide standard deviations for your key comparison results against competitors, either by running multiple seeds or via bootstrapped estimates? I understand that this is not commonly done in CV, but given the small improvements you report, I think it is necessary to justify the new method over established methods like AdamW.

3. To my understanding, your multi-scale training is more compute efficient than training on a fixed resolution of, e.g., 224. Unfortunately, this point is not well presented in the paper, and the analysis in the appendix is limited to one experiment. Could you extend this analysis to provide a comprehensive comparison of compute efficiency (FLOPs, wall-clock time, per-epoch train time, etc.) on the main evaluation tasks against a subset of the strongest competitors in each setup? A comprehensive and positive evaluation in this respect would lead me to increase my score, as it would justify the method despite the small performance improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.