acceptodds
Withdrawn from ICLR 2027

Raw Image Patches in the Residual Stream: A Data-Efficient Isotropic Vision Transformer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In cases where data or compute is constrained, state-of-the-art computer vision models have tended to be anisotropic, i.e. architecturally constrained to regard a different granularity of image detail in different layers. These are Convolutional Neural Networks (CNNs) or Transformer-based models incorporating CNN stems or inductive biases inspired by CNNs. This contrasts with the original ViT, which is isotropic in the sense that every layer represents the image as the same number of constant-size patches. Here we present an ablation study leading to a modernised, post-norm ViT that is isotropic and yet competitive with anisotropic Transformers in data- and compute-constrained settings. In particular, we match or exceed previous Transformer-based models when training for 300 epochs on CIFAR-100 and are competitive with similarly-sized SOTA models when training for 300 epochs on ImageNet-1k. The strength of our model comes partly from incorporating ideas proven effective since the original ViT paper 2020_dosovitskiy_vit and partly from three novel architectural choices. First, our model uses a novel re-weighting of the add-and-norm step of Transformers, allowing stable training across a wide range of depths. Second, we use a Transformer feed-forward block as a stem in place of the usual patch embedding and, third, we pass RMSNormed image patches directly into the residual stream.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
isotropic, transformer, post-norm, computer vision, imagenet, cifar

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2w4UOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript describes a project aiming to improve the performance of isotropic ViTs (i.e. without spatial inductive bias) for image classification, in particular in the data/compute-constrained regime. The two primary architectural innovations are the following: First, RMSNorm is applied frequently, to the point where the output of every layer with weights goes through subsequent norming. Second, patch embeddings are replaced by a feedforward sub-network with the same structure as the MPL within a transformer block. These interventions are tested in an ablation-style experiment probing classification accuracy on CIFAR-100, where the performance gains due to the inclusion of specific components is tested. The proposed architecture with both interventions performs the best in terms of CIFAR-100 classification. A smaller-scope experiment is run on ImageNet, where the model achieves performance that is comparable to its competitors.

Critical strengths

The design choices that are proposed are described rigorously and clearly. Further, they are very well-embedded in the context of recent literature, and the manuscript demonstrates nicely how they relate to prior work and points out their historical roots. Therefore, the global aspects of the presentation of the work are strong; this is also reflected by the well-calibrated section on the work's limitations. The discussion of model components in section 3 is interesting and holds value for practitioners, especially in cases in which architectural interventions are necessary to squeeze out additional performance (such as the low-data setting, or commercial applications). The experiments on CIFAR-100 (Fig. 3) show the contribution of different general model components in the low-data regime; the choice of dataset, as well as the experimental parameters is sensible.

Critical weaknesses

The motivation for the work is not entirely clear to me. I don't see a specific reason for why an isotropic transformer model is preferable. While they are used for big foundation models, which are trained on large datasets, I don't see why we would want that to extend to the small-data regime. Since the proposed method does not reach the performance of models with a more vision-specific inductive bias on ImageNet, additional arguments are necessary for why the isotropic model is desirable.

While the design choices are described well in the context of related work, theoretical justification is limited. Section 3.1.3 describes one aspect, but the heavy norming strategy and the FF-First approach seem somewhat arbitrary apart from empirical gains. This makes the manuscript read more like an engineering report, rather than a scientific paper that aims to answer specific scientific questions. Further, not all contributions stated in the 'Methods' section are represented properly in the experiment. The influence of the large number of norming layers, which are featured heavily in Section 3', is not studied as they are subsumed in the baseline model. The FF-First approach is evaluated there, but fails to demonstrate an improvement in performance. This leaves the 'image in residual stream' as the main contribution, which is not explored in-depth in the 'Methods' section. The other ablations relate to model components from the literature and are not directly tied to the work at hand. I would argue that the experiments should be connected more closely with the approaches discussed in Section 3.

Finally, the experimental results are limited to one small-scale dataset. This is discussed explicitly in the text as a limitation, which is commendable. I also acknowledge the argument that compute was limited for this project, but am of the opinion that larger-scale experiments are of high importance for a contribution that relies primarily on empirical arguments. The findings on ImageNet indicate that the results may not directly from CIFAR to other datasets.

Overall, this manuscript offers some interesting engineering-heavy findings for the clever design of ViT models. In terms of scientific contribution, I believe that in the current form it does not quite meet the high standards for this venue. I would recommend submitting this work to a more specialized journal / conference.

Questions for authors

No specific questions.

Reviewer_qCSzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a data-efficient isotropic ViT. It combines mostly known components (SequencePool, SwiGLU, RoPE, registers, extra RMSNorms) with three proposals. These are a depth-scaled post-norm residual weighting, an FF-block stem, and raw image patches written into the residual stream. It reports 83.1/85.0% on CIFAR-100 with 3.8M/14M parameters and 83.5% on ImageNet-1k with 63M parameters.

Critical strengths
  1. Writing raw image patches into the residual stream is simple and almost free. It gives a clear +2.0-point gain on CIFAR-100.
  2. The ρ=1/√S residual weighting is well motivated at initialization. It lets a 100-layer post-norm model train with the same hyperparameters, where ρ=1 fails.
Critical weaknesses
  1. The proposed components are not cleanly evaluated. The residual weighting is already in the baseline, and its only accuracy evidence is a single 100-layer comparison. It is not compared with pre-norm, DeepNorm or LayerScale in accuracy. The FF stem is added together with a switch to overlapping 4x4/stride-2 patches, so its effect cannot be separated from the tokenization change.
  2. All design choices were tuned on CIFAR-100 test accuracy. At 14M parameters, the margin over MSCVIT-S is only 0.3 points. Baseline numbers are taken from prior papers rather than reproduced under one recipe. No FLOPs or throughput are reported, despite the compute-constrained framing.
  3. The ImageNet evidence is thin. The result (83.5%) is 0.1 below PVT v2-B4 and 0.4 below two of the three CNNs. None of the components is ablated at this scale, and data efficiency is never tested directly. Strong plain-ViT and small-dataset ViT baselines, e.g. Beyer et al. (2022) and Lee et al. (2021), are not discussed.
Questions for authors
  1. With 4x4/stride-2 tokenization fixed, how does the FF stem compare with a linear patch embedding? How does ρ compare with pre-norm and DeepNorm in accuracy at the depths actually used?
  2. Can you report FLOPs and throughput for all models, and the full CIFAR-100 training recipe?
Reviewer_fWk3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to use several ViT normalization and training strategies to improve the small ViT performance on limited data settings. Unlike previous methods that try to use convolution training and data processing strategies, the authors instead tried to use a feedforward model first with a full-size image to get token input to the transformer. A post-norm is proposed with a specific weighting strategy to improve performance. In addition to this, various training strategies are analyzed in this paper for a better comparison. The results show that the proposed method could have relatively better results than the original small ViT models and other competitors.

Critical strengths
  1. This paper has made a contribution that many existing techniques should be properly applied and paired to make smaller ViT training in a limited-data setting to be effective.
  2. There is limited research on training transformers in limited-data settings, and this is an interesting idea for research to follow and contribute to.
Critical weaknesses
  1. This paper has made many arguments and claims, and there are many different techniques used in the method. However, it lacks a clear and organized logic or structure to discuss, making the paper very confusing to readers. For example, the authors could have a more strategic way to represent different network training settings in the experiments.
  2. In Figure 3, the authors have made an ablation study. However, the authors only add components step by step, instead of doing an “ablation”. For example, when removing components A and B, how would the performance change?
  3. Furthermore, the ablation results need a detailed analysis. It seems that adding the initial feedforward layer does not help with the performance, despite this being one of the contributions in this paper.
  4. The authors claimed that their model is efficient, but only network parameters are provided, no computation time efficiency is presented.
Questions for authors

Please see my detailed comments above. I like this paper’s idea, and I think the paper's findings could be important and beneficial for the community. However, the experimental evidence provided in the paper alone could not back up the main results. I encourage the authors to provide more solid and complete results to further strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.