acceptodds
Withdrawn from ICLR 2027

ViT-5: Vision Transformers for The Mid-2020s

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This work presents a systematic investigation into modernizing Vision Transformer backbones by leveraging architectural advancements from the past five years. While preserving the canonical Attention–FFN structure, we conduct a component-wise refinement involving normalization, activation functions, positional encoding, gating mechanisms, and learnable tokens. These updates form a new generation of Vision Transformers, which we call ViT-5. Extensive experiments demonstrate that ViT-5 consistently outperforms state-of-the-art plain Vision Transformers across both understanding and generation benchmarks. On ImageNet-1k classification, ViT-5-Base reaches 84.2% top-1 accuracy under comparable compute, exceeding DeiT-III-Base at 83.8%. ViT-5 also serves as a stronger backbone for generative modeling: when plugged into an SiT diffusion framework, it achieves 1.84 FID versus 2.06 with a vanilla ViT backbone. Beyond headline metrics, ViT-5 exhibits improved representation learning and favorable spatial reasoning behavior, and transfers reliably across tasks. With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformer, Neural architecture design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8tkuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits the design of Vision Transformers by combining established architectural components from recent vision and language models. Several designs are investigated, i.e., RMSNorm, RoPE, QK-Norm, LayerScale, and registers. Based on the investigation, the authors proposed the "ViT-5", which is claimed as the next generation of ViT. Reported results are also good, and covering image classification, segmentation, and generation.

Critical strengths
  • Extensive experiments spanning classification, semantic segmentation, and image generation.

  • Systematic component-wise ablations and generally consistent performance improvements across model scales.

Critical weaknesses

While the experimental coverage across classification, segmentation, and generation is very good, I find the novelty and significance insufficient for ICLR.

Below are my major concerns:

  • Most components, including RMSNorm, RoPE, QK-Norm, LayerScale, and registers. They have already been explored in prior vision or language Transformer literature. The resulting architecture appears largely to be a combination of established design choices rather than a fundamentally new contribution.
  • Revisiting component choices in conventional ViT backbones is not a particularly compelling research question in 2026. In my observation, this paper provides limited new insights beyond identifying a nice configuration.
  • The title "ViT-5" suggests a new generation of Vision Transformers, which is disproportionate to the actual technical contributions. Even the proposed "over-gating" explanation lacks direct mechanistic evidence. Besides, where is ViT-2/3/4?
  • Despite extensive experiments are claimed, the evaluation remains centered on conventional architectures and benchmarks, without demonstrating the claimed relevance to modern large VLMs.
  • There are no object detection / instance segmentation comparisons. It is not acceptable for claiming a new generation vision backbone w/o object detection and instance segmentation experiments.

Overall, despite the strong empirical results, they do not sufficiently compensate for the limited conceptual novelty and the overly ambitious claims. Thus I lean toward clear REJ.

Questions for authors

Please Refer to weakness part.

Reviewer_tZKvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the architecture of plain Vision Transformers by incorporating several components developed in recent years. ViT-5 preserves the standard attention–FFN backbone while combining LayerScale, RMSNorm, QK normalization, absolute and rotary positional embeddings, position-encoded register tokens, and bias-free QKV projections. The authors retain GeLU, as their experiments show that SwiGLU interacts poorly with LayerScale under the evaluated training setup. Across several model sizes, ViT-5 improves performance on ImageNet classification, SiT-based image generation, and ADE20K semantic segmentation. For example, ViT-5-B achieves 84.2% ImageNet accuracy, compared with 83.8% for DeiT-III-B. On image generation, the XL model improves FID from 2.06 to 1.84 after 7M training steps. The main contribution is therefore not a new individual component, but a practical architectural recipe supported by an empirical study of how these components interact.

Critical strengths
  • The gains are consistent across multiple tasks, suggesting that the proposed recipe is useful beyond ImageNet classification. On ADE20K, Table 6 reports improvements of 2.3, 1.1, and 2.7 mIoU points for the Small, Base, and Large models, respectively. Table 5 also shows better generation FID at all four model scales, including the XL model after extended training. In addition, the classification baselines are closely matched in parameter count and FLOPs. This consistency across tasks and model sizes is the main reason I lean toward acceptance.
  • The ablations on component interactions offer useful practical guidance. Table 2 evaluates all four combinations of LayerScale and SwiGLU and shows that using both together performs worse than using either one alone, for both classification and generation. Table 3 provides a similar observation for register tokens: standard registers can degrade performance, whereas the proposed positional encoding treatment improves the results. Although the individual components are already known, the paper helps clarify which combinations work well under the evaluated setup.
Critical weaknesses
  • Some of the paper’s explanations are stronger than the available evidence supports. Table 2 establishes a negative interaction between LayerScale and SwiGLU under the tested recipe, but the claim that excessive sparsity causes this degradation is not backed by activation or gate statistics. Likewise, Section 3.4 does not define the patch transformation in Figure 2 precisely enough, nor does it clearly state the positional assumptions under which a RoPE-only model should produce an equivalent output. Table 8 ablates RoPE but does not include a corresponding APE ablation. These omissions make it difficult to turn the empirical observations into broader design principles, though they do not undermine the reported performance gains themselves.
  • Several individual design choices would benefit from stronger evidence or more complete implementation details. Removing the QKV bias, for instance, improves accuracy by only 0.04–0.06 points in Table 8, yet no variation across independent runs is reported. Repeated runs for a representative setting would help determine whether effects of this size are reliable. The treatment of register tokens under RoPE is also not specified in enough detail, particularly with respect to token coordinates and frequency settings. Finally, Table 9(b) identifies (10^{-4}) as the best or default frequency base, whereas the accompanying text states (10^{-5}). These issues mainly affect the interpretation and reproducibility of the finer architectural choices.
Questions for authors
  • In Table 2, what hidden dimensions are used for the GeLU and SwiGLU FFNs? Are the two variants matched in parameter count and FLOPs? If available, activation or gate statistics would also help evaluate the proposed sparsity-based explanation for the observed degradation.
  • What exact patch transformation is used in Figure 2? Under which positional convention should a RoPE-only model produce an equivalent output? A more precise explanation, or a direct comparison of the resulting outputs, would strengthen this argument. A controlled ablation comparing RoPE-only with APE+RoPE would also better support the decision to retain APE.
  • Please specify the token coordinates and RoPE frequency settings used for patch, register, and class tokens. The discrepancy between (10^{-4}) in Table 9(b) and (10^{-5}) in the text should also be resolved. For small effects such as QKV-bias removal, are uncertainty estimates or results from repeated runs available?
Reviewer_SVc3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents ViT-5, a systematic modernization of the Vision Transformer architecture by incorporating recent advances from both vision and language Transformers. The authors retain the canonical Attention–FFN structure while revisiting several architectural components, including normalization, activation functions, positional encoding, gating mechanisms, and learnable tokens, resulting in a backbone that combines LayerScale, RMSNorm, 2D RoPE, QK-Norm, register tokens, and bias-free QKV projections. Extensive experiments demonstrate consistent improvements over existing plain ViT variants across image classification, image generation, and semantic segmentation, with component-wise ablations further analyzing the contribution of individual design choices. Overall, the paper argues that carefully modernizing the components of the original ViT, without fundamentally changing its macro-architecture, can provide a stronger and more versatile vision backbone.

Critical strengths
  1. Systematic architectural study. The paper provides a systematic investigation of several recent Transformer design choices and integrates them into a unified ViT-5 architecture, offering a useful perspective on how the canonical ViT can be modernized without changing its fundamental Attention–FFN structure.

  2. Broad and convincing empirical evaluation. The experiments cover multiple tasks, including ImageNet classification, image generation, and semantic segmentation, as well as different model scales and training settings. The consistent improvements across these settings strengthen the empirical evidence for the proposed design.

  3. Useful component-wise analysis. The paper conducts detailed ablations to quantify the contribution of individual components such as 2D RoPE, Registers, QK-Norm, and QKV bias removal. The analysis also reveals non-trivial interactions between architectural components, providing useful insights beyond simply reporting the final ViT-5 performance.

Critical weaknesses
  1. Limited architectural novelty. Most components adopted by ViT-5, including LayerScale, Registers, 2D RoPE, QK-Norm, and related Transformer refinements, have been proposed in prior work. As a result, the main contribution appears to be the systematic combination and empirical evaluation of existing techniques, rather than a fundamentally new architectural idea. The paper should better clarify its novelty relative to prior modern ViT and Transformer designs.

  2. Insufficient analysis of component interactions. The experiments mainly rely on single-component ablations, which do not fully characterize interactions among the different design choices. Since the final architecture combines multiple components and the paper itself observes non-trivial interactions such as LayerScale and gated activations, a more systematic study of component combinations would be valuable.

  3. Modest gains on ImageNet classification. The improvements over strong existing plain ViT baselines are relatively small in some settings, e.g., only 0.4 percentage points for ViT-5-Base over DeiT-III-Base. Given the paper's broad claim that conventional ViTs are substantially under-optimized, the significance of these improvements should be discussed more carefully.

  4. Potentially overstated claims about spatial reasoning. The paper attributes part of the improvement to better spatial reasoning and spatial modeling, but the presented evaluations are primarily classification, segmentation, generation, and qualitative analyses. These results provide indirect evidence for improved spatial representations, but do not directly establish stronger spatial reasoning capabilities.

  5. The “drop-in upgrade” claim is not fully validated. Although ViT-5 preserves the overall ViT structure, it introduces several simultaneous architectural modifications. The experiments do not sufficiently demonstrate that the resulting backbone is genuinely a drop-in replacement across a broad range of existing vision or multimodal systems.

  6. Unclear distinction between principled design and architecture tuning. The paper presents ViT-5 as a systematic modernization of ViT, but it remains unclear whether the final configuration reflects general design principles or extensive empirical tuning. A clearer account of the model-selection process, including how the final combination was determined and how much search was performed, would strengthen the scientific contribution.

Questions for authors
  1. How was the final combination of components in ViT-5 selected, and did the authors perform any systematic search over different component combinations?
  2. Could the authors provide a more detailed comparison of training and inference costs between ViT-5 and the strongest baseline models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.