- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This manuscript describes a project aiming to improve the performance of isotropic ViTs (i.e. without spatial inductive bias) for image classification, in particular in the data/compute-constrained regime. The two primary architectural innovations are the following: First, RMSNorm is applied frequently, to the point where the output of every layer with weights goes through subsequent norming. Second, patch embeddings are replaced by a feedforward sub-network with the same structure as the MPL within a transformer block. These interventions are tested in an ablation-style experiment probing classification accuracy on CIFAR-100, where the performance gains due to the inclusion of specific components is tested. The proposed architecture with both interventions performs the best in terms of CIFAR-100 classification. A smaller-scope experiment is run on ImageNet, where the model achieves performance that is comparable to its competitors.
Critical strengths
The design choices that are proposed are described rigorously and clearly. Further, they are very well-embedded in the context of recent literature, and the manuscript demonstrates nicely how they relate to prior work and points out their historical roots. Therefore, the global aspects of the presentation of the work are strong; this is also reflected by the well-calibrated section on the work's limitations. The discussion of model components in section 3 is interesting and holds value for practitioners, especially in cases in which architectural interventions are necessary to squeeze out additional performance (such as the low-data setting, or commercial applications). The experiments on CIFAR-100 (Fig. 3) show the contribution of different general model components in the low-data regime; the choice of dataset, as well as the experimental parameters is sensible.
Critical weaknesses
The motivation for the work is not entirely clear to me. I don't see a specific reason for why an isotropic transformer model is preferable. While they are used for big foundation models, which are trained on large datasets, I don't see why we would want that to extend to the small-data regime. Since the proposed method does not reach the performance of models with a more vision-specific inductive bias on ImageNet, additional arguments are necessary for why the isotropic model is desirable.
While the design choices are described well in the context of related work, theoretical justification is limited. Section 3.1.3 describes one aspect, but the heavy norming strategy and the FF-First approach seem somewhat arbitrary apart from empirical gains. This makes the manuscript read more like an engineering report, rather than a scientific paper that aims to answer specific scientific questions. Further, not all contributions stated in the 'Methods' section are represented properly in the experiment. The influence of the large number of norming layers, which are featured heavily in Section 3', is not studied as they are subsumed in the baseline model. The FF-First approach is evaluated there, but fails to demonstrate an improvement in performance. This leaves the 'image in residual stream' as the main contribution, which is not explored in-depth in the 'Methods' section. The other ablations relate to model components from the literature and are not directly tied to the work at hand. I would argue that the experiments should be connected more closely with the approaches discussed in Section 3.
Finally, the experimental results are limited to one small-scale dataset. This is discussed explicitly in the text as a limitation, which is commendable. I also acknowledge the argument that compute was limited for this project, but am of the opinion that larger-scale experiments are of high importance for a contribution that relies primarily on empirical arguments. The findings on ImageNet indicate that the results may not directly from CIFAR to other datasets.
Overall, this manuscript offers some interesting engineering-heavy findings for the clever design of ViT models. In terms of scientific contribution, I believe that in the current form it does not quite meet the high standards for this venue. I would recommend submitting this work to a more specialized journal / conference.
Questions for authors
No specific questions.