Revisiting supervised transformers: An optimized recipe

We revisit supervised Vision Transformers training and study two components: the optimizer and the classifier. We combine gradient sampling (GS), with AdaBelief optimizer and show that gradient perturbation using running gradient statistics improves downstream performance and fine-tuning convergence. We further propose a lightweight cross-attention classifier that trains end-to-end with the backbone using multi-scale training. We show that multi-scale training with cross-attention improves accuracy, efficiency and robustness. Our models transfers well to object detection and beat MAE-pretrained detectors using less epochs. We further combine our model with modern architectural components, and our final recipe reaches 86.5% in ImageNet-1K using ViT-L and 55.8 mAP in COCO using ViT-B.