- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper revisits the design of Vision Transformers by combining established architectural components from recent vision and language models. Several designs are investigated, i.e., RMSNorm, RoPE, QK-Norm, LayerScale, and registers. Based on the investigation, the authors proposed the "ViT-5", which is claimed as the next generation of ViT. Reported results are also good, and covering image classification, segmentation, and generation.
Critical strengths
-
Extensive experiments spanning classification, semantic segmentation, and image generation.
-
Systematic component-wise ablations and generally consistent performance improvements across model scales.
Critical weaknesses
While the experimental coverage across classification, segmentation, and generation is very good, I find the novelty and significance insufficient for ICLR.
Below are my major concerns:
- Most components, including RMSNorm, RoPE, QK-Norm, LayerScale, and registers. They have already been explored in prior vision or language Transformer literature. The resulting architecture appears largely to be a combination of established design choices rather than a fundamentally new contribution.
- Revisiting component choices in conventional ViT backbones is not a particularly compelling research question in 2026. In my observation, this paper provides limited new insights beyond identifying a nice configuration.
- The title "ViT-5" suggests a new generation of Vision Transformers, which is disproportionate to the actual technical contributions. Even the proposed "over-gating" explanation lacks direct mechanistic evidence. Besides, where is ViT-2/3/4?
- Despite extensive experiments are claimed, the evaluation remains centered on conventional architectures and benchmarks, without demonstrating the claimed relevance to modern large VLMs.
- There are no object detection / instance segmentation comparisons. It is not acceptable for claiming a new generation vision backbone w/o object detection and instance segmentation experiments.
Overall, despite the strong empirical results, they do not sufficiently compensate for the limited conceptual novelty and the overly ambitious claims. Thus I lean toward clear REJ.
Questions for authors
Please Refer to weakness part.