- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LENS (Learnable Normalization States), a hardware-oriented Transformer modification that replaces both LayerNorm and attention Softmax with small learnable MLPs. The motivation is that LayerNorm and Softmax rely on operations such as reduction, square root, division, and exponentiation that are less natural for GEMM-centric accelerators, whereas small MLPs can be mapped more directly onto matrix-multiplication hardware. The authors formulate existing LayerNorm families into a single ax*x+bx formulation, and use it as the basis for justifying why substituting Norms with MLP is plausible. On small-scale ViTs and language models, LENS is able to achieve mostly comparable performance to LayerNorms.
Critical strengths
- I like the formulation where the authors characterize all Norms into a single equation.
Critical weaknesses
- It is hard for me to see the novelty of this work beyond previous learnable normalization works. Previous works like [1] and [2] introduce this concept already. I'm not sure how this is conceptually different from existing learnable Norms.
[1] Luo et. al., Differentiable Dynamic Normalization for Learning Deep Representation, PMLR 2019
[2] Zhang et. al, SAL-ViT: Towards Latency Efficient Private Inference on ViT using Selective Attention Search with a Learnable Softmax Approximation, ICCV 2023.
- The authors mention a few Norm variants in the intro/related works, but the experiments only compare with LayerNorm. I'm not sure why the authors do not compare with other Norm variants.
Questions for authors
-
Please address the novelty and experimental weaknesses above.
-
In Table4, I see that you need to do official then LENS to get comparable accuracy, whereas pure LENS has severe accuracy degradation. Please comment on this.
-
The main motivation of the paper seems to be that GEMM is more efficient, but I do not see any efficiency documentation in the experiments. Do you have any results to show on that?
5 comments
Response for Q2 about the Official Checkpoints
Q2: We adopted an official pretrained checkpoint as a foundation in our experiments just for simulating the progress of deploying a trained model on hardware:
We first downloaded or trained a baseline to simulate the checkpoint that is prepared to be deployed on hardware (In Table 4, that is the official checkpoint of DeiT) and converted it to the LENS architecture. Current results support that the LENS-replaced model can recover or even exceed the original performance with sufficient joint retraining.
Therefore, the official checkpoint just acts as the workload we plan to deploy on hardware.
We explored the LENS-based model training from scratch just to explore which method is cheaper and faster: adapting the existing model to LENS or training a new model adopting LENS from scratch. The results show that if we limited the training budget to only 50 epochs, retraining from the existing official checkpoint can recover most of the accuracy of the original model. The new model adopting LENS training from scratch cannot achieve the same accuracy as the official checkpoint because 50 epochs are too few for DeiT+ImageNet. The official checkpoints are trained over 300 epochs. Therefore, this is also guidance for applying LENS to deploy a workload on hardware: adapting from a well-trained checkpoint is faster and cheaper than training ourselves from scratch.
Response for Q3 about the Hardware Motivation
Q3: We want to clarify that this work is not intended to improve GEMM efficiency. It’s a hardware-design simplification optimization based on the modification and replacement of LayerNorm and Softmax in the Transformer architecture.
LENS replaces LayerNorm and Softmax with MLPs. As a result, in the design of a Transformer hardware accelerator, we can eliminate the corresponding specific arithmetic units from the hardware architecture to reduce the chip area for ASIC design or the hardware resource consumption for FPGA designs.
In principle, fully connected linear layers in LENS can be executed in matrix-computation cores in current hardware, such as the systolic array in NPUs, and do not require new hardware. ReLU and power-of-two scaling operations are simple comparators and bitwise shifters (for fixed-point data) or integer subtraction (for floating-point data). Sigmoid approximation based on thresholds and bitwise shifting is also a very mature and low-cost hardware design, as in Amin et al. (1997) [1].
Therefore, the chip area and hardware resource costs of LayerNorm and Softmax units in the current design, which can be eliminated, translate into hardware gains of our work. LENS may involve additional computation in MLPs when reusing the matrix computation cores, but can also trade this additional computation for a smaller chip area or lower hardware cost in the meantime.
So, considering that the manufacturing cost and power consumption are always directly proportional to the chip area of ASICs or the hardware resource consumption of FPGAs, if LENS can eliminate the corresponding hardware of LayerNorm and Softmax, the associated costs can be reduced, which may especially benefit the design of edge accelerators with limited area, hardware resources, or power budgets. We are actually working on one of the related hardware accelerator designs and will release it in the future. We updated the description and definition of LENS in the Introduction and Discussion Sections and highlighted them in yellow.
[1] H. Amin, K.M. Curtis, and B.R. Hayes-Gill. Piecewise linear approximation applied to nonlinear function of a neural network. IEE Proceedings - Circuits, Devices and Systems, 1997
Response for Q1 about Novelty and Weakness
We thank you so much for your valuable comments and for taking the time to review. :) The following is our reply to the questions and weaknesses:
Q1: Our work is indeed not the first research on learnable normalization, and we have added the two papers you offered in the discussion of Section 2.1, highlighted in yellow.
However, prior works, including the two new related works, still require some hand-designed nonlinear computation more or less, such as the reciprocal, mean, square root, etc. Most of them typically only learn the parameters for their normalization formula.
Our work is trying to push learnable normalization to an extreme design that adopts only MLPs to learn suitable normalization computations without human-defined mathematical formulas. Therefore, as shown in Figure 1 of our paper, our LENSMax learned a completely different function compared with Softmax while training.
As a result, this design also offers a hardware-design-friendly architecture to simplify the design of an AI processor or accelerator by eliminating the related hardware of LayerNorm and Softmax. More details about the hardware motivation can be found in our answer for question 3.
In addition, we added the experimental results of DyT+Softmax, Derf+Softmax, and RMSNorm+Softmax on CIFAR-10/100 in Appendix A.5. Experimental results show that they perform similarly to LayerNorm+Softmax after being replaced by LENS. We have not tested the other alternative in our original experiments because this work is a hardware-oriented design to eliminate the specific normalization arithmetic modules from hardware design, not to achieve a higher accuracy than other alternatives. We chose LayerNorm and Softmax as the baseline because they are the most widely applied methods in currently released pretrained checkpoints.
All values are Top-1 accuracy (%), using seed 42. Baselines use Softmax and are trained for 400 epochs. LENS variants are retrained from their respective baselines for 50 epochs. Derf results are reported in Appendix A.5.