acceptodds
Under review as a conference paper at ICLR 2027

LENS: Replacing LayerNorm and Softmax in Transformers with Learnable Normalization States

Abstract

LayerNorm and attention Softmax are critical operations in Transformers. However, they cannot be efficiently mapped to General Matrix Multiplication (GEMM) computations on hardware. Therefore, modern hardware accelerator designs, such as NPUs on ASIC/FPGA, consume a large amount of hardware resources or chip area to implement their specific arithmetic units. Building on previous work about LayerNorm, Softmax, and their alternatives, we introduce Learnable Normalization States (LENS), a framework that reformulates LayerNorm as tiny MLPs by abstracting a common structure from related methods and extends this method to Softmax, which moves more of the computation in Transformers to GEMM as a hardware-friendly architecture. We evaluate LENS on image classification with Vision Transformers (ViT) and autoregressive language modeling with Pythia models, including generalization tests on WikiText and LAMBADA. Across these tasks, LENS matches or exceeds baseline performance in several vision settings and retains most of the baseline performance in the remaining vision and language experiments. These findings may suggest that hand-designed normalization and attention functions are not the only choice as alternatives to LayerNorm and Softmax. LENS also provides a basis for exploring Transformer architectures for resource-limited accelerators.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Transformers, LayerNorm, Softmax, Learnable Normalization, Hardware-friendly network architecture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J8w1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LENS (Learnable Normalization States), a hardware-oriented Transformer modification that replaces both LayerNorm and attention Softmax with small learnable MLPs. The motivation is that LayerNorm and Softmax rely on operations such as reduction, square root, division, and exponentiation that are less natural for GEMM-centric accelerators, whereas small MLPs can be mapped more directly onto matrix-multiplication hardware. The authors formulate existing LayerNorm families into a single ax*x+bx formulation, and use it as the basis for justifying why substituting Norms with MLP is plausible. On small-scale ViTs and language models, LENS is able to achieve mostly comparable performance to LayerNorms.

Critical strengths
  • I like the formulation where the authors characterize all Norms into a single equation.
Critical weaknesses
  • It is hard for me to see the novelty of this work beyond previous learnable normalization works. Previous works like [1] and [2] introduce this concept already. I'm not sure how this is conceptually different from existing learnable Norms.

[1] Luo et. al., Differentiable Dynamic Normalization for Learning Deep Representation, PMLR 2019

[2] Zhang et. al, SAL-ViT: Towards Latency Efficient Private Inference on ViT using Selective Attention Search with a Learnable Softmax Approximation, ICCV 2023.

  • The authors mention a few Norm variants in the intro/related works, but the experiments only compare with LayerNorm. I'm not sure why the authors do not compare with other Norm variants.
Questions for authors
  • Please address the novelty and experimental weaknesses above.

  • In Table4, I see that you need to do official then LENS to get comparable accuracy, whereas pure LENS has severe accuracy degradation. Please comment on this.

  • The main motivation of the paper seems to be that GEMM is more efficient, but I do not see any efficiency documentation in the experiments. Do you have any results to show on that?

5 comments
AuthorsOpenReview ↗

Response for Q1 about Novelty and Weakness

We thank you so much for your valuable comments and for taking the time to review. :) The following is our reply to the questions and weaknesses:

Q1: Our work is indeed not the first research on learnable normalization, and we have added the two papers you offered in the discussion of Section 2.1, highlighted in yellow.

However, prior works, including the two new related works, still require some hand-designed nonlinear computation more or less, such as the reciprocal, mean, square root, etc. Most of them typically only learn the parameters for their normalization formula.

Our work is trying to push learnable normalization to an extreme design that adopts only MLPs to learn suitable normalization computations without human-defined mathematical formulas. Therefore, as shown in Figure 1 of our paper, our LENSMax learned a completely different function compared with Softmax while training.

As a result, this design also offers a hardware-design-friendly architecture to simplify the design of an AI processor or accelerator by eliminating the related hardware of LayerNorm and Softmax. More details about the hardware motivation can be found in our answer for question 3.

In addition, we added the experimental results of DyT+Softmax, Derf+Softmax, and RMSNorm+Softmax on CIFAR-10/100 in Appendix A.5. Experimental results show that they perform similarly to LayerNorm+Softmax after being replaced by LENS. We have not tested the other alternative in our original experiments because this work is a hardware-oriented design to eliminate the specific normalization arithmetic modules from hardware design, not to achieve a higher accuracy than other alternatives. We chose LayerNorm and Softmax as the baseline because they are the most widely applied methods in currently released pretrained checkpoints.

DatasetOriginal normBaselineLENSNorm onlyLENSMax onlyBoth
CIFAR-10DyT76.7576.9776.6678.15
CIFAR-10RMSNorm80.1581.4380.3380.84
CIFAR-100DyT45.8449.3946.2749.66
CIFAR-100RMSNorm51.9551.7251.9952.43

All values are Top-1 accuracy (%), using seed 42. Baselines use Softmax and are trained for 400 epochs. LENS variants are retrained from their respective baselines for 50 epochs. Derf results are reported in Appendix A.5.

AuthorsOpenReview ↗

Response for Q2 about the Official Checkpoints

Q2: We adopted an official pretrained checkpoint as a foundation in our experiments just for simulating the progress of deploying a trained model on hardware:

We first downloaded or trained a baseline to simulate the checkpoint that is prepared to be deployed on hardware (In Table 4, that is the official checkpoint of DeiT) and converted it to the LENS architecture. Current results support that the LENS-replaced model can recover or even exceed the original performance with sufficient joint retraining.

Therefore, the official checkpoint just acts as the workload we plan to deploy on hardware.

We explored the LENS-based model training from scratch just to explore which method is cheaper and faster: adapting the existing model to LENS or training a new model adopting LENS from scratch. The results show that if we limited the training budget to only 50 epochs, retraining from the existing official checkpoint can recover most of the accuracy of the original model. The new model adopting LENS training from scratch cannot achieve the same accuracy as the official checkpoint because 50 epochs are too few for DeiT+ImageNet. The official checkpoints are trained over 300 epochs. Therefore, this is also guidance for applying LENS to deploy a workload on hardware: adapting from a well-trained checkpoint is faster and cheaper than training ourselves from scratch.

AuthorsOpenReview ↗

Response for Q3 about the Hardware Motivation

Q3: We want to clarify that this work is not intended to improve GEMM efficiency. It’s a hardware-design simplification optimization based on the modification and replacement of LayerNorm and Softmax in the Transformer architecture.

LENS replaces LayerNorm and Softmax with MLPs. As a result, in the design of a Transformer hardware accelerator, we can eliminate the corresponding specific arithmetic units from the hardware architecture to reduce the chip area for ASIC design or the hardware resource consumption for FPGA designs.

In principle, fully connected linear layers in LENS can be executed in matrix-computation cores in current hardware, such as the systolic array in NPUs, and do not require new hardware. ReLU and power-of-two scaling operations are simple comparators and bitwise shifters (for fixed-point data) or integer subtraction (for floating-point data). Sigmoid approximation based on thresholds and bitwise shifting is also a very mature and low-cost hardware design, as in Amin et al. (1997) [1].

Therefore, the chip area and hardware resource costs of LayerNorm and Softmax units in the current design, which can be eliminated, translate into hardware gains of our work. LENS may involve additional computation in MLPs when reusing the matrix computation cores, but can also trade this additional computation for a smaller chip area or lower hardware cost in the meantime.

So, considering that the manufacturing cost and power consumption are always directly proportional to the chip area of ASICs or the hardware resource consumption of FPGAs, if LENS can eliminate the corresponding hardware of LayerNorm and Softmax, the associated costs can be reduced, which may especially benefit the design of edge accelerators with limited area, hardware resources, or power budgets. We are actually working on one of the related hardware accelerator designs and will release it in the future. We updated the description and definition of LENS in the Introduction and Discussion Sections and highlighted them in yellow.

[1] H. Amin, K.M. Curtis, and B.R. Hayes-Gill. Piecewise linear approximation applied to nonlinear function of a neural network. IEE Proceedings - Circuits, Devices and Systems, 1997

Reviewer_DKsCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed to use learnable MLPs to replace the layer norm and softmax in transformers, which can increase the hardware utilization efficiency of the models. The adapted model, tested on both vision and language tasks, shows comparable quality to the original model.

Critical strengths
  1. The methods elaboration is straightforward and clear
  2. The effectiveness of the shown experiments is good, based on the shown experiments. The effectiveness is also shown across both vision and language tasks.
  3. Negative results are reported openly.
Critical weaknesses
  1. For the starting equation, where a(x)⊙x + b(x) . I think any equation can be written that way ... the per-token constraint put on a(x) will make the methods not generalizable to DyT or Derf; not technically a claimed supersets of these methods.
  2. The paper's motivation is to increase hardware efficiency. Qualitatively, I get that replacing softmax will probably increase hardware efficiency, but I just don't see any benchmark results for the replaced modules. That would give us a clearer idea.
  3. In this work, the output appears to scale with the token input range (as shown in Figure 1) rather than being normalized to some range. If that's the case, then the proposed method is more like per-token gain ... which could be interesting ... but I have concerns that it will only work for the tested models? It’s also deviating from what the work intended to do: replacing LayerNorm and softmax. If that’s the case, it’s more like removing them rather than replacing them. And the question is whether the model, or the training recipe, can make the model tolerant to such a removal.
  4. I do not see any comparison with the relevant baselines, except for the base models with standard softmax or layernorm. At least with the reviewed works in the related works section, I don't even see a comparison against them.
  5. Joint adaption: I don't quite get the uniqueness regarding this finding. I mean, if you swap some modules and do the training, you would expect to see other weights change.
Questions for authors

Please provide justifications for the weakness points.

3 comments
AuthorsOpenReview ↗

Response for Weaknesses 1 and 2

We thank you so much for your valuable comments and for taking the time to review. :) The following is our reply to the questions and weaknesses:

Weakness 1: We agree that the format of a(x)⊙x+b(x) is a relatively generic format for different functions. We adopted this format, inspired by prior learnable normalization works, just to represent the computation of LayerNorm and related alternatives as a standard normalization with input-dependent scaling/bias parameters, and then converted related works to this format. Therefore, this formula serves as a design heuristic. The current scalar implementation does not purport to strictly encompass the full functional form of DyT/Derf.

Weakness 2: Sorry that we have not provided enough detail about the hardware motivation. The other reviewer has the same question. First, we want to explain that this work is a hardware-oriented modification and replacement of LayerNorm and Softmax in the Transformer architecture. However, it’s not a hardware-inference-efficiency optimization, but a hardware-design simplification optimization.

LENS replaces LayerNorm and Softmax with MLPs. As a result, in the design of a Transformer hardware accelerator, we can eliminate the corresponding specific arithmetic units from the hardware architecture to reduce the chip area for ASIC design or the hardware resource consumption for FPGA designs.

In principle, fully connected linear layers in LENS can be executed in matrix-computation cores in current hardware, such as the systolic array in NPUs, and do not require new hardware. ReLU and power-of-two scaling operations are simple comparators and bitwise shifters (for fixed-point data) or integer subtraction (for floating-point data). Sigmoid approximation based on thresholds and bitwise shifting is also a very mature and low-cost hardware design, as in Amin et al. (1997) [1].

Therefore, the chip area and hardware resource costs of LayerNorm and Softmax units in the current design, which can be eliminated, translate into hardware gains of our work. LENS may involve additional computation in MLPs when reusing the matrix computation cores, but can also trade this additional computation for a smaller chip area or lower hardware cost in the meantime.

Considering that the manufacturing cost and power consumption are always directly proportional to the chip area of ASICs or the hardware resource consumption of FPGAs, if LENS can eliminate the corresponding hardware of LayerNorm and Softmax, the associated costs can be reduced, which may especially benefit the design of edge accelerators with limited area, hardware resources, or power budgets. We are actually working on one of the related hardware accelerator designs and will release it in the future. We updated the description and definition of LENS in the Introduction and Discussion Sections and highlighted them in yellow.

[1] H. Amin, K.M. Curtis, and B.R. Hayes-Gill. Piecewise linear approximation applied to nonlinear function of a neural network. IEE Proceedings - Circuits, Devices and Systems, 1997

AuthorsOpenReview ↗

Response for Weaknesses 3, 4, and 5

Weakness 3: Thanks for your question, and you are correct. This work is more likely to eliminate the LayerNorm and Softmax because our target is to eliminate their corresponding hardware in the design of AI processors or accelerators. We have not used the elimination of LayerNorm and Softmax in the title and paper, just because we tried to avoid being considered as overclaiming. In LENSNorm, it indeed scales with token-wise inputs. Considering the format of a(x)⊙x+b(x), the normalization parameters a and b are input-dependent. Therefore, it is, in principle, a per-token input-dependent affine transformation. Considering that we have tested the LENS in both language and image models with six different models and four datasets, we believe that, with sufficient recovery retraining, LENS can recover the performance of a pretrained model by replacing the LayerNorm and Softmax with MLPs to support a simplified hardware design. If the reviewer has any specific concerns about the experimental results, we would be happy to answer them.

Weakness 4: We added the experimental results of DyT+Softmax, Derf+Softmax, and RMSNorm+Softmax on CIFAR-10/100 in Appendix A.5. Experimental results show that they perform similarly to LayerNorm+Softmax after being replaced by LENS. We have not tested the other alternative in our original experiments because this work is a hardware-oriented design to eliminate the specific normalization arithmetic modules from hardware design, not to achieve a higher accuracy than other alternatives. We chose LayerNorm and Softmax as the baseline because they are the most widely applied methods in currently released pretrained checkpoints.

DatasetOriginal normBaselineLENSNorm onlyLENSMax onlyBoth
CIFAR-10DyT76.7576.9776.6678.15
CIFAR-10RMSNorm80.1581.4380.3380.84
CIFAR-100DyT45.8449.3946.2749.66
CIFAR-100RMSNorm51.9551.7251.9952.43

All values are Top-1 accuracy (%), using seed 42. Baselines use Softmax and are trained for 400 epochs. LENS variants are retrained from their respective baselines for 50 epochs. Derf results are reported in Appendix A.5.

Weakness 5: We present the joint adaptation targeting to provide guidance for deploying the LENS-replace model on a hardware accelerator (Especially an area/resource-limited edge accelerator without the specific arithmetic unit for LayerNorm and Softmax). At the beginning of the research, we tried to freeze the weights and train only the new LENS MLPs to save GPU memory and speed up training, expecting the LENS MLPs to simply approximate the original LayerNorm and Softmax functions. However, the results show a huge gap in accuracy. As a result, we have to adopt joint retraining of the weights and LENS MLPs. The progress is: We first downloaded or trained a baseline to simulate the checkpoint that is prepared to be deployed on hardware and converted it to the LENS architecture. Current results support that the LENS-replaced model can recover or even exceed the original performance with sufficient joint retraining.

AuthorsOpenReview ↗

Additional Information on Hardware Costs and Trade-offs

We would like to provide additional quantitative information regarding your concerns about hardware costs:

  1. Parameter Storage: Using DeiT-Tiny as an example: the hidden dimension is 192, the token length is 197, and there are 12 Transformer blocks. Therefore, one LENSNorm has one MLP with 192→32→2192\rightarrow32\rightarrow2 neurons, and one LENSMax has one MLP with 197→32→1197\rightarrow32\rightarrow1 neurons and one MLP with 2→16→12\rightarrow16\rightarrow1 neurons. As a result, one LENSNorm indicated 6242 additional parameters, and one LENSMax indicated 6434 parameters. In one DeiT-Tiny, there are 25 LENSNorm and 12 LENSMax, which have 233258 parameters. LENS contains as many as 4.08% of the parameters from the original model and is as large as 0.93 MB in FP32 format. Therefore, LENS is not very heavy in parameter storage.
  2. Number of MAC Operations: We want to offer a rough estimate of the additional MAC operations introduced by LENS. In DeiT-Tiny, we denote the hidden dimension by d=192, the number of tokens by N=197, and the number of attention heads by H=3. Single LENSNorm has N(d×32+32×2)=1222976N(d\times32+32\times2)=1222976 MAC operations. Single LENSMax has HN(N×32+32×1)+HN2(2×16+16×1)=9333072HN(N\times32+32\times1)+HN²(2\times16+16\times1)=9333072 MAC operations. Across 25 LENSNorm and 12 LENSMax modules, LENS introduces 142571264 MACs, accounting for 10.21% of the total MACs in the LENS-replaced DeiT-Tiny model.
  3. Reduction of Hardware Cost: Removing the LayerNorm and Softmax units from the LLM-FPGA design cited in Table 1 would release resources corresponding to 5.795.79\\% of its LUTs, 6.486.48\\% of its FFs, 11.1211.12\\% of its DSPs, and 12.3412.34\\% of its BRAM. However, their hardware is designed for BERT. Therefore, this rough estimate of hardware resource reduction is for reference only.
Reviewer_Ur8SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Learnable Normalization States (LENS), a framework that replaces LayerNorm and attention Softmax in Transformers with small learnable MLPs. The motivation is to make Transformer computation more GEMM-oriented and reduce the reliance on specialized arithmetic units in hardware accelerators.

LENS consists of two components. LENSNorm replaces LayerNorm with an input-dependent affine transformation controlled by two learned scalar states. LENSMax replaces attention Softmax using a learned row-state function and an element-wise mapping function, both implemented with small MLPs.

The authors evaluate the method on CIFAR-10/100, ImageNet-1K with DeiT-Tiny and DeiT-Small, and autoregressive language modeling using Pythia-14M/31M/70M. The results suggest that LENS can recover much of the original model performance after retraining, although the outcomes vary across architectures and tasks. The paper also examines training trajectories and parameter changes to better understand how the modified models adapt.

Critical strengths
  • Strength 1: Interesting and practically motivated architectural direction

The paper considers a relevant problem in hardware-oriented Transformer design. LayerNorm and Softmax can require specialized operations that are less convenient for certain hardware accelerators, so exploring alternatives based on learnable matrix computations is a reasonable direction.

I find the idea of replacing these operations with small MLPs interesting, especially in the context of customized ASIC and FPGA designs. The paper provides a concrete framework for investigating whether more GEMM-oriented Transformer architectures can maintain competitive predictive performance.

  • Strength 2: Evaluation across vision and language tasks, including useful negative results

The authors test LENS on several vision benchmarks and small autoregressive language models. The CIFAR experiments are particularly useful because they examine LENSNorm and LENSMax separately, allowing the effects of the two replacements to be compared.

I also appreciate that the authors report cases where performance decreases, rather than focusing only on successful replacements. The varying results across DeiT and Pythia models give a more realistic picture of the method's current limitations. The training-trajectory and weight-change analyses provide additional context for understanding how the models respond to the architectural modifications.

Critical weaknesses
  • Weakness 1: The main hardware-efficiency motivation is not directly supported by the experiments.

My main concern is that the paper motivates LENS primarily through potential hardware savings, but does not provide direct measurements demonstrating these savings.

The hardware-resource comparison in Table 1 comes from a previous FPGA accelerator rather than an actual implementation of LENS. The paper also does not report hardware area, inference latency, energy consumption, or end-to-end throughput for the proposed architecture.

This is important because replacing LayerNorm and Softmax does not automatically imply lower computational or hardware cost. LENSMax introduces a row-wise MLP and an element-wise MLP that operate on attention scores. These additional computations have their own costs, and nonlinear operations such as Sigmoid are still required.

Without a more detailed cost analysis, it is difficult to tell whether the proposed replacements would reduce total hardware resources or improve inference performance. Even an operator-level comparison or a resource estimate under a specific hardware configuration would help.

Since hardware friendliness is the primary motivation of the work, I think this claim needs stronger evidence.

  • Weakness 2: The experimental setup does not sufficiently separate the effects of LENS from those of additional retraining.

Another concern is that several of the main results involve replacing operators in pretrained models followed by additional retraining. However, the original architectures are not always evaluated under the same additional training conditions.

For example, the reported 1.45 percentage-point improvement for DeiT-Tiny is obtained after 50 epochs of retraining with knowledge distillation. The comparison is made against the original pretrained checkpoint, without showing how the unmodified DeiT-Tiny would perform after the same retraining and distillation procedure.

A similar issue appears in the Pythia Scratch-to-LENS experiments, where the modified models receive an additional training stage relative to the one-epoch baseline runs. This makes it hard to determine whether the performance recovery is due to the architectural design or simply the additional optimization.

The performance trends are also somewhat inconsistent. DeiT-Small remains below its original baseline even with knowledge distillation, and replacing the operators in pretrained Pythia-14M leads to substantial degradation on WikiText and LAMBADA.

These results suggest that the effectiveness of LENS depends considerably on the training setup and model configuration. I would find the experimental evidence more convincing if the authors included compute-matched retraining baselines and more systematic component ablations on the larger models.

Questions for authors
  • Q1. Hardware efficiency and computational overhead

Could the authors provide a more concrete analysis of the hardware costs of LENS compared with the original LayerNorm and Softmax implementations?

For example, operator-level runtime measurements, FLOPs, parameter overhead, or estimated FPGA/ASIC resource usage would be helpful. I am particularly interested in how the additional MLP computations and the remaining Sigmoid operation affect the expected hardware savings.

This would help establish whether the proposed architecture offers an actual efficiency advantage.

  • Q2. Controlled retraining comparisons

Could the authors compare LENS with the original Transformer architecture using the same additional training budget, learning-rate schedule, and knowledge-distillation settings?

For the DeiT experiments, I would especially like to see whether the reported improvements remain when the original model receives the same 50 epochs of retraining. It would also be helpful to evaluate LENSNorm and LENSMax separately on the larger DeiT and Pythia models, since the CIFAR experiments suggest that the two components can have different effects.

  • Q3. Sequence-length generalization and normalization properties of LENSMax

LENSMax uses a learned row-state MLP whose input dimension depends on the length of the attention row. How does this design handle variable sequence lengths, causal masking, and inference at sequence lengths beyond the predefined maximum?

I am also curious about the normalization behavior of LENSMax. Unlike Softmax, the proposed mapping does not explicitly ensure that the attention weights sum to one. Have the authors examined how much the attention-row sums vary and whether this affects numerical stability or generalization, especially for longer sequences?

4 comments
AuthorsOpenReview ↗

Response for Q1 about Hardware Efficiency

We thank you so much for your valuable comments and for taking the time to review. :) The following is our reply to the questions:

Q1: This work is a hardware-oriented modification and replacement of LayerNorm and Softmax in the Transformer architecture. However, it’s not a hardware-inference-efficiency optimization, but a hardware-design simplification optimization.

LENS replaces LayerNorm and Softmax with MLPs. As a result, in the design of a Transformer hardware accelerator, we can eliminate the corresponding specific arithmetic units from the hardware architecture to reduce the chip area for ASIC design or the hardware resource consumption for FPGA designs.

In principle, fully connected linear layers in LENS can be executed in matrix-computation cores in current hardware, such as the systolic array in NPUs, and do not require new hardware. ReLU and power-of-two scaling operations are simple comparators and bitwise shifters (for fixed-point data) or integer subtraction (for floating-point data). Sigmoid approximation based on thresholds and bitwise shifting is also a very mature and low-cost hardware design, as in Amin et al. (1997) [1].

Therefore, the chip area and hardware resource costs of LayerNorm and Softmax units in the current design, which can be eliminated, translate into hardware gains of our work. LENS may involve additional computation in MLPs when reusing the matrix computation cores, but can also trade this additional computation for a smaller chip area or lower hardware cost in the meantime.

So, considering that the manufacturing cost and power consumption are always directly proportional to the chip area of ASICs or the hardware resource consumption of FPGAs, if LENS can eliminate the corresponding hardware of LayerNorm and Softmax, the associated costs can be reduced, which may especially benefit the design of edge accelerators with limited area, hardware resources, or power budgets. We are actually working on one of the related hardware accelerator designs and will release it in the future. We updated the description and definition of LENS in the Introduction and Discussion Sections and highlighted them in yellow.

[1] H. Amin, K.M. Curtis, and B.R. Hayes-Gill. Piecewise linear approximation applied to nonlinear function of a neural network. IEE Proceedings - Circuits, Devices and Systems, 1997

AuthorsOpenReview ↗

Response for Q2 about the Continuous Training for DeiT-Tiny

Q2: Thanks a lot for your suggestions. We have added the continuous training results for DeiT-Tiny across three seeds, 50 epochs, without KD. The final accuracy is Top-1 74.30% ± 0.04% and Top-5 92.06% ± 0.01%, which exceeds our experimental results of LENS. We highlighted the discussion of this in Section 3.2 in yellow.

SettingAdditional epochsKDTop-1 accuracy (%)Top-5 accuracy (%)
Official checkpoint0—72.2191.10
Continued training50No74.30 ± 0.0492.06 ± 0.01
LENS (best configuration)50Yes73.66 ± 0.2691.91 ± 0.11

However, we want to provide more details about our experimental setting in this work. Our experiments with single-head ViT, DeiT, and Pythia aim to simulate deploying a pretrained model on a hardware accelerator based on LENS (Especially an area/resource-limited edge accelerator without the specific arithmetic unit for LayerNorm and Softmax) and explore whether we can recover the original model performance through retraining after replacing LayerNorm and Softmax with LENSNorm and LENSMax.

To this end, we downloaded or trained a baseline and converted it to the LENS architecture. For instance, the official checkpoint of DeiT-Tiny just acts as the workload we plan to deploy on hardware. Current results support that the LENS-replaced model can recover or even exceed the original performance with sufficient retraining. However, the main goal of retraining is not for higher accuracy, but to adapt the pretrained checkpoint for hardware based on LENS. That’s why we did not perform continuous training on the original checkpoints in our old version, because the current retraining achieved our target.

Therefore, although the new continuous training results for DeiT-Tiny that we added to our manuscript achieve higher accuracy than our LENS-replaced model, they still do not violate our experimental target for LENS. The LENS-replaced model has recovered most of the performance from the original baseline.

In addition, we have not added the experiment for LENSNorm-Only or LENSMax-Only in DeiT/Pythia because, if we cannot convert both LayerNorm and Softmax to MLPs, they will still require specialized hardware designs in accelerators, which violates our design target in this work. Related experiments on CIFAR-10/100 have provided preliminary results for analysis. The same experiments in larger models do have value. However, due to the limited GPU resources currently available, we planned these experiments for our future work.

AuthorsOpenReview ↗

Response for Q3 about the Sequence Length

Q3: It’s true that our current design is limited by the maximum input length. The LENS variant for a variable-length model is still under ongoing research. Our current work may mainly target edge hardware accelerators, where the input length may always be set to a maximum length to save on-board hardware and memory resources. And for the causal mask, we currently fill zeros into the inputs, and future positions were masked before LENSMax. Their output weights were set to zero. If the input exceeds our maximum input length, it cannot be supported by our current design.

In addition, we explored some different designs of the attention row sum at the beginning of our research. We did not apply any division to the attention score in the early stage of our experiments, which means no limitation on the row sum of attention. This design works fine in the related experiment of single-head ViT on CIFAR-10/100. However, in the DeiT experiments, because the scores in some extreme cases on the ImageNet-1K dataset became too large, multiplying by Value caused numerical overflow and caused training to fail. That's why we adopted power-of-two scaling for each score in the current LENS design to limit the sum of each row to the range [0, 2].

AuthorsOpenReview ↗

Additional Information on Hardware Costs and Trade-offs

We would like to provide additional quantitative information regarding your concerns about hardware costs:

  1. Parameter Storage: Using DeiT-Tiny as an example: the hidden dimension is 192, the token length is 197, and there are 12 Transformer blocks. Therefore, one LENSNorm has one MLP with 192→32→2192\rightarrow32\rightarrow2 neurons, and one LENSMax has one MLP with 197→32→1197\rightarrow32\rightarrow1 neurons and one MLP with 2→16→12\rightarrow16\rightarrow1 neurons. As a result, one LENSNorm indicated 6242 additional parameters, and one LENSMax indicated 6434 parameters. In one DeiT-Tiny, there are 25 LENSNorm and 12 LENSMax, which have 233258 parameters. LENS contains as many as 4.08% of the parameters from the original model and is as large as 0.93 MB in FP32 format. Therefore, LENS is not very heavy in parameter storage.
  2. Number of MAC Operations: We want to offer a rough estimate of the additional MAC operations introduced by LENS. In DeiT-Tiny, we denote the hidden dimension by d=192, the number of tokens by N=197, and the number of attention heads by H=3. Single LENSNorm has N(d×32+32×2)=1222976N(d\times32+32\times2)=1222976 MAC operations. Single LENSMax has HN(N×32+32×1)+HN2(2×16+16×1)=9333072HN(N\times32+32\times1)+HN²(2\times16+16\times1)=9333072 MAC operations. Across 25 LENSNorm and 12 LENSMax modules, LENS introduces 142571264 MACs, accounting for 10.21% of the total MACs in the LENS-replaced DeiT-Tiny model.
  3. Reduction of Hardware Cost: Removing the LayerNorm and Softmax units from the LLM-FPGA design cited in Table 1 would release resources corresponding to 5.795.79\\% of its LUTs, 6.486.48\\% of its FFs, 11.1211.12\\% of its DSPs, and 12.3412.34\\% of its BRAM. However, their hardware is designed for BERT. Therefore, this rough estimate of hardware resource reduction is for reference only.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summray of Our Rebuttal

Dear Reviewers and AC,

We thank you so much for your time and effort in reviewing our submission, and also thanks for the reply from Reviewer 3. Here, we want to summarize our rebuttal and the revisions to our paper. We hope this addresses your concerns and helps AC make the final decision.

  1. All reviewers are concerned about the same issues regarding hardware efficiency. As we explained in our response, we want to point out that this work aims to simplify the design of a new ASIC/FPGA-based hardware accelerator by eliminating the LayerNorm/Softmax arithmetic units. Therefore, in principle, this work is not a hardware-inference-efficiency optimization, but a hardware-design simplification optimization. Therefore, the acceleration of the existing GPU or TPU, or supporting FlashAttention, is out of the scope of this work. In addition, we also provide a rough estimate of the trade-off in our work. In summary, using DeiT as an example, the MLPs of LENS will increase the weights by about 4.08% from the original model (0.93MB) and account for 10.21% of the total MACs in the LENS-replaced DeiT-Tiny model. Using previous hardware work as a rough estimate, removing LayerNorm/Softmax units from the hardware accelerator can reduce 5.79% of its LUTs, 6.48% of its FFs, 11.12% of its DSPs, and 12.34% of its BRAM. The complete hardware prototype and hardware cost from synthesis will be our future work.
  2. We want to explain the experimental setup in this work. The downloaded official checkpoint of DeiT and the trained model of Pythia are all acting as the workloads to be deployed on hardware. Therefore, our experiments simulate the hardware processing that converts the existing model by replacing the LayerNorm and Softmax with LENS and retraining it to recover the performance. Then, the processed model can be used to deploy on hardware. Therefore, our experiments are not two-stage training or continuous training. Furthermore, the accuracy loss in DeiT trained from scratch of the LENS-replaced model is just proof that converting from an existing checkpoint by retraining the LENS-replaced model is cheaper and faster than training a model that directly adopts LENS from scratch.
  3. We added experiments of DyT/Derf/RMSNorm+Softmax models and their LENS-replaced models. The results show that after replacing DyT/Derf/RMSNorm and Softmax with LENS, models can still recover accuracy with sufficient retraining.
  4. We extended the revised paper to 10 pages and added one more page of the Appendix. All the edited parts in the main content have been highlighted in yellow.

In the end, we want to thank you again for the constructive comments and for helping us improve our current work. More details about the rebuttal can be found in our replies to reviewers below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.