acceptodds
Under review as a conference paper at ICLR 2027

Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation

Abstract

Vision Transformers (ViTs) achieve state-of-the-art performance on challenging vision tasks, but their deployment on edge devices is hindered by the computational complexity and global reduction bottleneck imposed by layer normalization. Recent methods attempt to bypass this by replacing normalization layers with hardware-friendly scalar approximations. However, these homogeneous replacements do not optimally fit to all layers' behaviour and rely on expensive model retraining. In this work, we propose a highly efficient, hardware-aware framework that utilizes genetic programming (GP) to evolve heterogeneous, layer-specific scalar functions directly from pre-trained weights. Coupled with a novel post-training re-alignment strategy, our approach eliminates the need to retrain models from scratch entirely. Our evolved expressions accurately approximate the target normalization behaviours, capturing 90–93% of the variance () compared to only 70–76% for homogeneous baselines, allowing our modified architecture to recover 84.32% Top-1 ImageNet-1K accuracy on ViT-B and 85.74% on ViT-L in only 20 epochs. By retaining near-baseline accuracy while eliminating the global reduction bottleneck, our approach achieves a strict reduction in both arithmetic complexity and off-chip memory traffic compared to standard LayerNorm, removing a primary barrier to the efficient deployment of ViTs on edge accelerators.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
vision transformers, layer normalization, efficient inference, genetic programming, symbolic regression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KrFcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework for replacing LayerNorm in pretrained Vision Transformers with layer-specific scalar functions discovered through genetic programming. The search balances approximation error, tail regularization, and estimated computational cost. The replacements inherit pretrained affine parameters and undergo 20 epochs of adaptation. Experiments on ViT-B and ViT-L evaluate functional alignment, ImageNet-1K accuracy recovery, and analytical computational and memory costs. The distillation-based variants achieve Top-1 accuracies of 84.32% and 85.74%, respectively.

Critical strengths
  1. A practically motivated framework for adapting pretrained models. The approach addresses the relevant problem of replacing input-dependent normalization while preserving useful pretrained representations. Extracting layer-specific activation mappings, retaining pretrained affine parameters, and explicitly considering expression cost provide a coherent framework for adapting existing ViTs without training them from scratch.

  2. Meaningful accuracy recovery with statistical reporting. The GP-based full fine-tuning and distillation variants outperform their corresponding DyT variants on both ViT-B and ViT-L. GP-D reaches 84.32% and 85.74% Top-1 accuracy, compared with 83.36% and 84.85% for DyT-D. The evaluation includes five independent runs with standard deviations, multiple adaptation schemes, independently tuned configurations, and complete recovery trajectories. These results support the effectiveness of the approach within the tested settings.

Critical weaknesses
  1. The calibration data and the generalization of the replacement functions are insufficiently characterized. Section 2.1 states that a batch of images is passed through the pretrained model and 50,000 activation points are sampled from each layer, but does not specify the source, number, or sampling procedure of the calibration images. It is also unclear whether the 90/10 split ensures image-level disjointness. If both splits contain activations from the same images, the current validation does not establish the applicability of the fitted functions to independent images. Although the ImageNet classification results after fine-tuning support the effectiveness of the overall framework, they do not disentangle the generalization of the replacement functions from the compensatory effects of subsequent adaptation. The calibration protocol should be clarified, with fitting errors and classification performance reported on independent images and across different calibration subsets.

  2. The coverage of directly relevant replacement baselines is limited. The evaluation primarily compares LayerNorm and DyT, without including other directly relevant pointwise normalization replacements such as Derf (Stronger Normalization-Free Transformers). Derf includes ViT experiments and is therefore a reasonable comparison. Evaluating accuracy and cost using the same pretrained models, calibration data, and adaptation budget would help establish the additional value of GP search over existing alternatives.

  3. Key ablations are missing, leaving the contributions of structure search, layer-specific calibration, and the search objectives insufficiently disentangled. The existing A/F/D comparisons examine the adaptation scope and the role of distillation, but the following controls remain important: First, Directly fit simple clipping functions and compare them with GP-based structure search. The expressions shown in the appendix predominantly use clipping and simple combinations, so the necessity of searching over function structures remains unclear. Second, Compare a shared functional structure with layer-specific parameters against independently searched structures for different layers, separating the benefits of layer-specific parameter calibration from structural heterogeneity. Third, Separately remove the computational-cost objective and the tail penalty, reporting fitting error, training success rate, final accuracy, and expression cost. The authors describe potential instability when optimizing MSE alone, but do not provide systematic quantitative comparisons. In particular, the cost objective and tail penalty directly underpin the efficiency and stability claims and should receive priority in the ablation analysis.

  4. The description of which parameters are trained in DyT-A is inconsistent between the main text and the appendix. Section 2.3 states that affine-only adaptation optimizes only the inherited scale and bias parameters, (w, b), whereas Tables 3 and 4 in Appendix C also specify a learning rate of 8 × 10⁻³ for α in DyT-A. The authors should clarify whether α is trained and reconcile the method description, configurations, and implementation. Since α controls the input scale of tanh, this discrepancy also affects the interpretation of the comparison between DyT-A and GP-A.

  5. Presentation and organization could be improved. There is a noticeable blank space beneath the Discussion heading on page 9, which should be removed to improve the layout. The paper also lacks a dedicated Conclusion section.

Questions for authors
  1. Calibration protocol and generalization. What are the source, number, and sampling procedure of the calibration images? Does the 90/10 split separate images, or only activation points? Could you report fitting errors on image-disjoint evaluation data and examine performance across different calibration subsets, both before and after adaptation?

  2. Comparison with relevant alternatives. How does the proposed method compare with Derf when adapted from the same pretrained ViT checkpoints using matched calibration data and a comparable 20-epoch fine-tuning or distillation budget? Please compare both classification accuracy and estimated computational cost.

  3. Contribution of the search components. Could you compare GP search with directly fitted clipping functions and a shared functional structure with layer-specific parameters? Could you also separately remove the computational-cost objective and the tail penalty under matched search and adaptation budgets, reporting fitting error, training success rate, accuracy, and expression cost?

  4. Trainable parameters in DyT-A. Is α updated during DyT-A adaptation? Please clarify whether the reported results follow the “only (w, b)” description in Section 2.3 or the α learning-rate settings in Appendix C. Please specify the exact trainable parameter sets for both DyT-A and GP-A and explain any resulting difference in their adaptation capacity.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KrFc (1/2)

We sincerely thank the reviewer for their time and effort, and especially appreciate the recognition of our framework as a practical way to adapt existing pre-trained ViTs without training from scratch. We address your concerns and questions below.

Below we summarize the new ViT-B baselines and ablations, together with the LN, DyT-D, and GP-D results from Table 2. We discuss these results under W2 and W3. Cost is given in FLOPs per token, summed over all 25 normalization layers (Appendix F).

MethodFit MSETrainsTop-1 (%)Cost
LN (reference)–5/584.99 ± 0.02125d + 75
DyT-D0.3005/583.36 ± 0.04600d
Derf-D0.2755/583.55 ± 0.08900d
Tanh-D0.0801/183.45 (n=1)625d
Clip-D0.1225/584.22 ± 0.0350d
GP-D (ours)0.1005/584.32 ± 0.0131d
GP-D, no tail penalty0.0664/564.10 ± 33.0842d
GP-D, no cost objective0.0855/584.25 ± 0.02474d

W1/Q1 Calibration protocol: The mappings come from a single batch of 64 randomly drawn ImageNet-1K validation images (no labels are used), with 50,000 input-output points sampled per LN layer at random across tokens and features. The 90/10 split is over these points, not over images, and serves only for the GP metrics. We will clarify the image source in Section 2.1.

To check how well the equations generalize to independent images, we extracted new mappings from 64 ImageNet-1K training images and re-ran the full GP search on them (5 seeds, same configuration). The table below shows the fit of both sets of equations on both sets of mappings, and their top-1 accuracy directly after replacing LayerNorm (before adaptation) and after adaptation with distillation (GP-D, same settings as in the paper):

Equations evolved onMSE, validation-image mappingsMSE, training-image mappingsTop-1 before adaptation (%)Top-1 after adaptation, GP-D (%)
Validation images (paper)0.100 ± 0.0040.107 ± 0.00351.0 ± 3.684.32 ± 0.01
Training images (new)0.099 ± 0.0030.102 ± 0.00351.1 ± 3.284.29 ± 0.05

Both sets fit the mappings they were not evolved on about as well as their own, and give the same accuracy before and after adaptation. This means that the choice of images used to extract the mappings does not affect the fit or final accuracy. We will add these results to the appendix.

W2/Q2 Comparison with alternatives: We have run the requested additional experiment using the Derf method, adapted from the same pre-trained ViT-B checkpoint and using the same re-alignment settings and data, with distillation (Derf-D, 5 seeds, see the table above).

Derf-D improves slightly over DyT-D, but remains well below GP-D. Derf is also more expensive than DyT. Unlike tanh, erf cannot be written directly in terms of exp. We therefore use the Abramowitz–Stegun approximation of erf, which needs an exp and a polynomial, and evaluate it in Horner form as we do for tanh and sigmoid (Appendix F). Under our cost model, Derf then costs 36 FLOPs per element, compared to 24 for DyT. We agree that Derf is a relevant alternative and will include it as a baseline in the revision.

AuthorsOpenReview ↗

Response to Reviewer KrFc (2/2)

W3/Q3 Contribution of the search components: We have run two directly fitted baselines on the same activation mappings and with the same re-alignment settings as GP-D (Clip-D and Tanh-D in the table above).

For the clip baseline (Clip-D, 5 runs), we fit a⋅clip(bx)a \cdot \mathrm{clip}(bx) per layer. The fitted clip closes most of the gap between DyT-D and GP-D, while GP-D stays consistently ahead (all five Clip-D runs score below the GP-D mean). However, we see this clip family as an outcome of our method rather than only a competing baseline: it was discovered by the GP search from the layer mappings, whereas DyT and Derf assume a tanh or erf shape beforehand. This is a general strength of our setup: given the mappings of a pre-trained model, the GP search finds cheap, deployable functions adapted to each layer, and their shapes can be reused directly, as Clip-D shows. We will add this comparison to the appendix and discuss the clip family as a finding of the search in the Discussion.

Clip-D and Tanh-D (atanh⁡(bx)a \tanh(bx) per layer, i.e. DyT with a per-layer calibrated input scale and output affine) together form the shared-structure control you suggest: one functional form in every layer, with parameters fitted per layer. Calibrating tanh per layer adds little over DyT-D (+0.09), even though Tanh-D fits the mappings better than our GP solutions. Changing the shape to a clip gives most of the gain (+0.77), and the per-layer structure found by the search adds a further +0.10.

We have also run the requested two ablations, with the same search and adaptation budgets. Removing the tail penalty improves the fit (MSE 0.066 vs 0.100) at a slightly higher cost (42d vs 31d) but makes training unstable. With the fp16 mixed precision of our GP-D recipe, three of the five equation sets fail within the first two epochs with a NaN loss. To rule out fp16 overflow as the only cause, we reran all five in bf16 (the 4/5 in the table): four sets train but reach only 78.77 ± 4.94, and one diverges to chance level, giving 64.10 ± 33.08 over all five against 84.32 ± 0.01 with the penalty. This is the activation explosion described in Section 2.2, which the tail penalty was introduced to prevent.

Without the computational-cost objective (using standard node count in GP instead of FLOPs and selecting the best-fitting expression per layer), the fit is slightly better (MSE 0.085 vs 0.100) and all five runs train, reaching a similar accuracy to GP-D (84.25 ± 0.02 vs 84.32 ± 0.01). However, the selected expressions cost 474d, about 15 times more than with the cost objective (31d) and almost 4 times more than LayerNorm itself. The cost objective therefore gives the efficiency gain without any loss in accuracy. We will add both ablations to the appendix.

W4/Q4 Trainable parameters in DyT-A: α is not updated during the DyT-A adaptation. Only each DyT layer’s inherited weight and bias parameters are updated. This keeps the number of trainable parameters equal for both DyT-A and GP-A (38,400 parameters), so both have the same adaptation capacity. The α learning rate settings in Appendix C will be corrected in the revision to reflect this. However, since α is a learnable parameter in the original DyT implementation, we have also trained DyT-A with α trainable. Our results showed that training α does not improve DyT-A performance (83.22 with α trained vs 83.23 frozen, one seed).

W5 Presentation: We thank the reviewer for pointing this out. We will remove the blank space on page 9 and check the revision for similar layout issues. The Discussion already summarizes our contributions and closes with limitations and future work, so it serves as the paper’s conclusion.

We hope this addresses your concerns and are happy to clarify any further questions.

Thank you for the response and additional experiments, which address several of my concerns. However, the new ViT-B results show that Clip-D achieves most of the accuracy gain, with GP-D adding only 0.10 percentage points. My concern about the limited additional methodological contribution remains, so I maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer KrFc

Thank you for the follow-up, and we are glad the additional experiments address several of your concerns. We agree that on ViT-B most of the gain over DyT-D comes from the clip shape (Clip-D 84.22% vs DyT-D 83.36%). However, this shape is itself a result of our search. As you noted in your review, the expressions in the appendix predominantly use clipping, and Clip-D was constructed from exactly these expressions. Without the search, the existing replacements use a fixed smooth shape, and on ViT-B these remain close to DyT-D, both with erf (Derf-D 83.55%) and with tanh calibrated per layer (Tanh-D 83.45%). The clip shape is therefore a contribution of the search itself. On top of this, the ablations you requested (W3 in our response) show that its other components are needed as well, as the cost objective lowers the cost about 15 times and the tail penalty keeps training stable.

We hope this clarifies the contribution of the different parts of our method.

Reviewer_KZQeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes replacing LayerNorm in pretrained Vision Transformers with layer-specific scalar functions discovered through genetic programming. The search balances approximation error, a tail regularization penalty, and estimated operation cost. The resulting functions inherit the original affine parameters, followed by 20 epochs of re-alignment through affine-only tuning, full fine-tuning, or knowledge distillation. Evaluation covers ViT-B and ViT-L on ImageNet-1K across five independent runs, including functional approximation, classification recovery, and analytical hardware cost comparisons. The best variants achieve 84.32% and 85.74% Top-1 accuracy, respectively. Under the proposed cost model, the replacements reduce normalization arithmetic and input-read traffic relative to LayerNorm.

Critical strengths
  1. Practical adaptation of pretrained models. The framework replaces all LayerNorm layers and recovers classification performance within 20 fine-tuning epochs. GP-D exceeds the corresponding DyT-D results by 0.96 and 0.89 percentage points on ViT-B and ViT-L, respectively. Five-run reporting supports the consistency of these gains, making the approach relevant to adapting existing models.
  2. Interpretable, inexpensive replacement functions. The search produces explicit, mostly piecewise-linear expressions with layer-dependent coefficients and structure. Reporting these expressions alongside approximation quality and analytical costs makes the proposed replacements inspectable and provides a concrete basis for subsequent implementation.
Critical weaknesses
  1. Limited validation of hardware benefits. The hardware comparison relies on analytical costs, and the assumed two off-chip reads for LayerNorm are not justified for a specific buffering and tiling setup. Consequently, the reported savings do not establish practical latency or energy improvements, limiting support for the deployment claims.
  2. The contribution of evolutionary search is not sufficiently isolated. Most discovered expressions are simple clipping-based functions, but no directly fitted clipped-affine baseline is evaluated under matched settings. This leaves uncertainty about whether evolutionary structure search provides substantial benefits over simpler calibrated replacements.
Questions for authors
  1. Clarification of the hardware model. What target memory hierarchy, on-chip capacity, and tiling schedule justify the two off-chip reads assumed for LayerNorm in Appendix F.5? How would the comparison change if a token’s feature vector remained on-chip during statistics computation and normalization? Please clarify which efficiency conclusions follow from the analytical assumptions and whether any are supported by deployment measurements.
  2. Additional validation that could change the assessment. Are there matched results for a directly fitted layer-specific clipped-affine baseline and for DyT initialized by jointly calibrating its input scale and output affine parameters? Comparisons using the same activation data and re-alignment settings would help establish the additional value of evolutionary structure search. If unavailable, these constitute additional controlled experiments rather than clarification of an existing result.
4 comments
AuthorsOpenReview ↗

Response to Reviewer KZQe (1/2)

We sincerely thank the reviewer for their time and effort, and especially appreciate the recognition of the interpretability of our discovered expressions and the practical relevance of adapting existing pre-trained models. We address your concerns and questions below.

W1/Q1 Hardware validation: We have run cycle-accurate RTL simulations of LN, DyT, and GP on two chips and measured the cycles and energy per normalization layer. The first (RISC-V) is Caravel, an open-source SoC in the SkyWater 130 nm process that has been proven in silicon, with a VexRiscv RV32I core. This core has no floating-point unit, so most of the cost is computation. The second (ASIC-NAU) adds a programmable nonlinear accelerator unit to the same SoC, a pipelined datapath with shift, multiply, add, compare, and lookup-table units. The computation is then cheap, and reading the inputs from off-chip memory becomes the main cost. Energy is estimated from the chip netlists using the switching activity of the simulation. It does not include the off-chip memory itself, so the real cost of each off-chip read is higher than reported here.

The two off-chip reads in Appendix F.5 apply when a token’s feature vector does not fit in the on-chip buffer and is read in tiles. LN needs the mean and variance of the whole vector before it can normalize any element. It therefore reads every tile once for the statistics and a second time for the normalization, while DyT and GP read each tile only once. The on-chip SRAM of this SoC holds fewer values than a ViT-B feature vector, so vectors are stored in off-chip flash memory and read in tiles. To keep the whole-chip simulations tractable, we use a buffer of 32 values. We compare a vector that fits in this buffer (d = 32) with one that spans two tiles (d = 64), and larger vectors only add further tiles of the same pattern. The table gives the cycles and energy of LN and DyT relative to GP (ViT-B GP expressions, averaged over the layers).

ChipVector sizeLN (cycles / energy)DyT (cycles / energy)
RISC-Vd = 32 (fits on-chip)3.19× / 3.15×12.04× / 12.05×
RISC-Vd = 64 (two tiles)3.14× / 3.10×12.03× / 12.03×
ASIC-NAUd = 32 (fits on-chip)1.06× / 1.09×1.28× / 1.38×
ASIC-NAUd = 64 (two tiles)2.03× / 2.06×1.27× / 1.38×

GP is the cheapest method on both chips and at both vector sizes. Its computation is about 3× cheaper than that of LN and more than 10× cheaper than that of DyT. On the RISC-V chip, where the computation dominates, this carries over to the total cost. On ASIC-NAU, memory access dominates instead. While the vector fits on-chip, all methods read it once and GP’s advantage comes from its computation alone. Once the vector spans two tiles, LN reads every input twice and needs about twice the cycles and energy of GP, a factor that stays the same for any number of tiles. These measurements support both efficiency conclusions of Section 3.3 per normalization layer: the lower arithmetic cost at every vector size, and the halved off-chip reads once the vector exceeds the on-chip buffer. We will add the setup and specific measurements of these simulations to the results and appendix in the revision.

AuthorsOpenReview ↗

Response to Reviewer KZQe (2/2)

W2/Q2 Baseline comparisons: We have run both suggested baselines on the same activation mappings and with the same re-alignment settings as GP-D (on ViT-B).

MethodTop-1 (%)
DyT-D83.36 ± 0.04
Tanh-D83.45 (n=1)
Clip-D84.22 ± 0.03
GP-D84.32 ± 0.01

For the clipped-affine baseline (Clip-D, 5 runs), we fit a⋅clip(bx)a \cdot \mathrm{clip}(bx) per layer, which together with the inherited affine parameters gives a layer-specific clipped-affine function. The fitted clip closes most of the gap between DyT-D and GP-D, while GP-D stays consistently ahead (all five Clip-D runs score below the GP-D mean). However, we see this clip family as an outcome of our method rather than only a competing baseline: it was discovered by the GP search from the layer mappings, whereas DyT assumes a tanh shape beforehand. This is a general strength of our setup: given the mappings of a pre-trained model, the GP search finds cheap, deployable functions adapted to each layer, and their shapes can be reused directly, as Clip-D shows. We will add this comparison to the appendix and discuss the clip family as a finding of the search in the Discussion.

Our original DyT baselines follow the published implementation (Zhu et al., 2025), where only α is learned. To test whether calibration alone explains the gap, we fit atanh⁡(bx)a \tanh(bx) per layer (Tanh-D), which corresponds to DyT initialized with a jointly calibrated input scale and output affine. Tanh-D fits the mappings better than our GP solutions (MSE 0.080 vs 0.100), but still ends level with DyT-D. This indicates that calibration alone does not explain the gain over DyT, and that the shape of the function matters.

We hope this addresses your concerns and are happy to clarify any further questions.

Thank you for the additional experiments. The RTL simulations clarify the memory assumptions, and the matched baselines strengthen the evaluation. However, Clip-D (84.22%) nearly matches GP-D (84.32%), leaving limited evidence for the added value of evolutionary search. The hardware results also do not establish full-model latency or energy gains. I therefore maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer KZQe

Thank you for the follow-up, and we are glad the RTL simulations clarify the memory assumptions and that the matched baselines strengthen the evaluation. We agree that our hardware measurements are per normalization layer and do not establish full-model latency or energy gains. We will state this scope wherever the efficiency gains are discussed in the revision.

With regard to Clip-D, we would like to clarify where its clip function comes from. Our paper does not propose a fitted clip as an alternative to the GP search. Clip-D was constructed from the expressions returned by our search, as almost all of them contain a clip. Existing replacements use one smooth function with a fixed shape, such as tanh (DyT) or erf (Derf). On ViT-B these remain close to DyT-D (83.36%), both with erf (Derf-D 83.55%) and with tanh calibrated per layer (Tanh-D 83.45%). The clip shape found by the search reaches 84.22% (Clip-D), and the full per-layer expressions reach 84.32% (GP-D). We therefore see the strong result of Clip-D as evidence for the added value of the evolutionary search, since it is the search that found this shape.

We hope this clarifies how Clip-D relates to our method.

Reviewer_BLhLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Addressing the LayerNorm global-reduction bottleneck when deploying ViTs on edge accelerators, the paper proposes using genetic programming to evolve, layer by layer, heterogeneous elementwise scalar functions from the input-output mappings of the normalization layers in pretrained models, combined with a realignment strategy that inherits the pretrained affine parameters.

Critical strengths

The composite fitness design is a highlight. Eq. 3 directly targets the activation explosion caused by unbounded linear fitting, which is an empirically valuable engineering contribution.

The experiments are relatively rigorous, with complete code provided.

Critical weaknesses

The paper claims a hardware-related contribution, but this is barely reflected in the experiments. The main contribution lies in cost-performance improvement rather than accuracy gains; the accuracy does not actually reach SOTA. Moreover, since LayerNorm layers are generally not the bottleneck in terms of overall cost, the effectiveness of the contribution is questionable.

If the deployment premise is that "a single reconfigurable activation unit can evaluate these symbolic primitives" (Discussion section), this point is crucial, because it seems not to be a claim that can obviously be taken for granted — yet it determines the validity of your method. The explanation of this is too brief to allow judgment. The theoretical derivations are almost entirely relegated to the appendix, making it very hard to grasp the main points of the paper from the main text alone.

Questions for authors

In Figure 3 on ViT-L, why does your method start with such high accuracy? Judging by the improvement slope, the GP-D curve appears to rise almost as much as your method — how do you explain this?

How is the claimed hardware contribution measured? How can you demonstrate end-to-end latency/energy benefits in real-world scenarios, particularly when bandwidth is actually constrained?

Please explain in greater detail how a single reconfigurable activation unit can evaluate these symbolic primitives.

4 comments
AuthorsOpenReview ↗

Response to Reviewer BLhL (1/2)

We sincerely thank the reviewer for their time and effort, and especially appreciate the recognition of our composite fitness design (Eq. 3). We address your concerns and questions below.

Q1 Figure 3: We realize the figure can be read ambiguously and hope to clarify it properly. The black LN curve is a reference that keeps the original LayerNorm, not our method, which is why it starts high at the pre-trained accuracy. Our method consists of the three GP curves, where GP-D is our best performing variant. All of these start at roughly 22.3% (for ViT-L) before fine-tuning and the GP-D rise is the model’s recovery after replacing LayerNorm. In the revision we will remove the LN curve and instead show the LN fine-tuning result from Table 2 as a dotted reference line, to prevent confusion.

W1/Q2 Hardware contribution: In the paper, the hardware cost is derived analytically (Section 3.3). We have now also run cycle-accurate RTL simulations of LN, DyT, and GP on two chips and measured the cycles and energy per normalization layer. The first (RISC-V) is Caravel, an open-source SoC in the SkyWater 130 nm process that has been proven in silicon, with a VexRiscv RV32I core. This core has no floating-point unit, so most of the cost is computation. The second (ASIC-NAU) adds a programmable nonlinear accelerator unit to the same SoC, a pipelined datapath with shift, multiply, add, compare, and lookup-table units. The computation is then cheap, and reading the inputs from off-chip memory becomes the main cost. Energy is estimated from the chip netlists using the switching activity of the simulation, and excludes the off-chip flash memory, so the extra reads of LN are understated.

The on-chip SRAM of this SoC holds fewer values than a ViT-B feature vector, so vectors are stored in off-chip flash memory and read in tiles. LN needs the mean and variance of the whole vector before it can normalize any element, and therefore reads every tile twice (once for the statistics and once for the normalization), while DyT and GP read each tile only once. To keep the whole-chip simulations tractable, we use a buffer of 32 values and compare a vector that fits in it (d = 32) with one that spans two tiles (d = 64), and larger vectors only add further tiles of the same pattern. ASIC-NAU, where reading the inputs is the main cost, is the bandwidth-constrained case the reviewer asks about. The table gives the cycles and energy of LN and DyT relative to GP (ViT-B GP expressions, averaged over the layers).

ChipVector sizeLN (cycles / energy)DyT (cycles / energy)
RISC-Vd = 32 (fits on-chip)3.19× / 3.15×12.04× / 12.05×
RISC-Vd = 64 (two tiles)3.14× / 3.10×12.03× / 12.03×
ASIC-NAUd = 32 (fits on-chip)1.06× / 1.09×1.28× / 1.38×
ASIC-NAUd = 64 (two tiles)2.03× / 2.06×1.27× / 1.38×

GP is the cheapest method on both chips and at both vector sizes. On ASIC-NAU, where memory access dominates, LN needs about twice the cycles and energy of GP once the vector spans two tiles, a factor that stays the same for any number of tiles. In a real system the memory bandwidth is shared with other components, which would make the second read of LN even more costly. We measure per normalization layer, not a full model, but on bandwidth-bound edge hardware it is this data movement that makes LN costly (see the Introduction). We will add the setup and measurements of these simulations to the results and appendix in the revision.

AuthorsOpenReview ↗

Response to Reviewer BLhL (2/2)

W1 Accuracy vs SOTA: The reviewer correctly notes that our method’s accuracy does not reach SOTA. However, this was not the aim of this work: as stated in the abstract and Section 2.3, our goal is restoring the performance of a given pre-trained model after replacing LayerNorm. GP-D performance is restored to within 0.67% (ViT-B) and 0.43% (ViT-L) of the LN reference (Table 2), at lower arithmetic and memory cost (Section 3.3, and the measurements above).

W2/Q3 Single reconfigurable activation unit: We agree that the Discussion explains this too briefly. All GP expressions are built from the same small operator set (Section 2.2): addition, multiplication, negation, clipping, tanh, and sigmoid. Each layer’s expression is therefore a short program over these primitives with a few layer-specific constants. A single unit that supports this operator set can evaluate every layer by loading that layer’s program and constants, without any change to the hardware.

The two chips in our measurements already work this way: LN, DyT, and every GP layer run on the same hardware and differ only in the program they execute (W1/Q2 above). ASIC-NAU is an example of such a unit, a single programmable datapath whose instructions drive its shift, multiply, add, compare, and lookup-table units. Published accelerators follow the same principle. COMPACT (Ou et al., 2023) builds each supported activation function as a sequence of a few instructions (addition, subtraction, multiplication, division, exponential, and logarithm), Flex-PE (Lokhande et al., 2025) reuses one processing element for several activation functions and selects the function at run time, and LoRA (Dai et al., 2026) loads the coefficients of its nonlinear functions as configuration data. In all of these designs, a new function changes only the operation sequence or the stored coefficients, not the hardware.

We will expand this explanation in the Discussion with these designs and our measurements. The appendix only contains the per-operator FLOP derivations (Appendix F), which Section 3.3 summarizes, while the method and the main results are in the main text.

We hope this addresses your concerns and are happy to clarify any further questions.

References

  • Ou, W., Wu, Z., Wang, Z., Chen, C., and Yang, Y. COMPACT: Co-processor for Multi-mode Precision-adjustable Non-linear Activation Functions. DATE, 2023.
  • Lokhande, M., Raut, G., and Vishvakarma, S. K. Flex-PE: Flexible and SIMD Multiprecision Processing Element for AI Workloads. IEEE TVLSI, 2025.
  • Dai, Y., Zou, G., Yang, Y., et al. LoRA: Towards Improved Applicability of Reconfigurable Architecture for Versatile Nonlinear Functions. ISCA, 2026.
Reviewer_BLhLOpenReview ↗

Response to submission 9256 authors

Dear authors, Thanks for the new experiments they address my concerns on parts i mentioned, although i'm still unsure about whether this approach has enough improvement over baselines with regards to it novelty, i will maintain my overall recommendation.

AuthorsOpenReview ↗

Response to Reviewer BLhL

Thank you for the follow-up, and we are glad the new experiments address your concerns. With regard to the improvement over baselines, GP-D outperforms DyT-D on all four architectures (ViT-B 84.32% vs 83.36%, ViT-L 85.74% vs 84.85%, Swin-B 84.92% vs 84.45%, ConvNeXt-B 85.21% vs 84.66%). On ViT-B it also outperforms Derf-D (83.55%) and a fitted per-layer clip (84.22%). On top of this, GP uses fewer cycles and less energy than DyT on both simulated chips (up to 12× on the RISC-V chip). With regard to novelty, DyT and Derf use one fixed function for every layer, whereas our method searches a separate hardware-efficient function for each layer, based on the mappings of the pre-trained model. The clip-based function of the fitted baseline above was itself found by this search.

We hope this clarifies the improvement of our method over the baselines, and are happy to discuss any specific comparison that would help assess this further.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer xwCi (1/2)

We sincerely thank the reviewer for their time and effort, and especially appreciate the recognition of our layer-specific perspective and the careful reporting of our accuracy-recovery results. We address your concerns and questions below.

W1/Q1 Implementation on hardware: We have run cycle-accurate RTL simulations of LN, DyT, and GP on two chips and measured the cycles and energy per normalization layer. The first (RISC-V) is Caravel, an open-source SoC in the SkyWater 130 nm process that has been proven in silicon, with a VexRiscv RV32I core, where all floating-point arithmetic runs in software. The second (ASIC-NAU) adds a programmable nonlinear accelerator unit to the same SoC, a pipelined datapath with shift, multiply, add, compare, and lookup-table units. Energy is estimated from the chip netlists using the switching activity of the simulation, and excludes the off-chip flash memory, so the extra reads of LN are understated.

Each GP expression is a short sequence of primitive operations (e.g., clip, multiply, and add). On both chips, LN, DyT, and every GP layer run on the same hardware and differ only in the program they execute. Since the simulations are cycle-accurate, the measured cost includes all instruction overhead, not only the FLOPs counted in Section 3.3.

The on-chip SRAM of this SoC holds fewer values than a ViT-B feature vector, so vectors are stored in off-chip flash memory and read in tiles. To keep the whole-chip simulations tractable, we use a buffer of 32 values and compare a vector that fits in it (d = 32) with one that spans two tiles (d = 64), and larger vectors only add further tiles of the same pattern. The table gives the cycles and energy of LN and DyT relative to GP (ViT-B GP expressions, averaged over the layers).

ChipVector sizeLN (cycles / energy)DyT (cycles / energy)
RISC-Vd = 32 (fits on-chip)3.19× / 3.15×12.04× / 12.05×
RISC-Vd = 64 (two tiles)3.14× / 3.10×12.03× / 12.03×
ASIC-NAUd = 32 (fits on-chip)1.06× / 1.09×1.28× / 1.38×
ASIC-NAUd = 64 (two tiles)2.03× / 2.06×1.27× / 1.38×

GP is the cheapest method on both chips and at both vector sizes. Its computation is about 3× cheaper than that of LN and more than 10× cheaper than that of DyT. On the RISC-V chip, where the computation dominates, this carries over to the total cost. On ASIC-NAU, memory access dominates instead. While the vector fits on-chip, all methods read it once and GP’s advantage comes from its computation alone. Once the vector spans two tiles, LN reads every input twice and needs about twice the cycles and energy of GP, a factor that stays the same for any number of tiles. We will add the setup and measurements of these simulations to the results and appendix in the revision.

AuthorsOpenReview ↗

Response to Reviewer xwCi (2/2)

W2/Q2 Baseline comparison: We have run the suggested per-layer clipped-linear baseline on the same activation mappings and with the same re-alignment settings as GP-D (on ViT-B).

MethodTop-1 (%)
DyT-D83.36 ± 0.04
Clip-D84.22 ± 0.03
GP-D (ours)84.32 ± 0.01

For this clip baseline (Clip-D, 5 runs), we fit a⋅clip(bx)a \cdot \mathrm{clip}(bx) per layer, i.e., a clipped-linear function with a layer-specific input and output scale, whose output is not fixed to ±1. As the reviewer expected, this per-layer clip closes most of the gap between DyT-D and GP-D, while GP-D stays consistently ahead (all five Clip-D runs score below the GP-D mean). However, we see this clip family as an outcome of our method rather than only a competing baseline: it was discovered by the GP search from the layer mappings, whereas DyT assumes a tanh shape beforehand. This is a general strength of our setup: given the mappings of a pre-trained model, the GP search finds cheap, deployable functions adapted to each layer, and their shapes can be reused directly, as Clip-D shows. We will add this comparison to the appendix and discuss the clip family as a finding of the search in the Discussion.

W3/Q3 Generalization of results: We have extended our results to two further pre-trained architectures on ImageNet-1K: Swin-B (a hierarchical vision transformer) and ConvNeXt-B (a convolutional network that also uses LN). Both use the same pipeline as ViT (mapping extraction, GP search, and 20-epoch re-alignment, 5 runs each).

MethodSwin-BConvNeXt-B
LN (pre-trained)85.2885.81
DyT-D84.45 ± 0.0584.66 ± 0.06
GP-D (ours)84.92 ± 0.0485.21 ± 0.01

GP-D again outperforms DyT-D on both architectures, and stays within 0.36% (Swin-B) and 0.60% (ConvNeXt-B) of the pre-trained model. GP also outperforms DyT for the full fine-tuning and affine-only variants (by 0.81% and 1.05% on Swin-B, and by 0.92% and 0.82% on ConvNeXt-B). We will add the Swin-B and ConvNeXt-B results to the main results of the paper.

On top of this, we evaluated the GP-D and DyT-D models of all four architectures on shifted input distributions (ImageNet-V2, -Sketch, -R, and -A). GP-D outperforms DyT-D on every set, and the margin grows on the harder sets, up to 5.4% (ViT-B), 8.9% (ViT-L), 4.0% (Swin-B), and 4.1% (ConvNeXt-B) on ImageNet-A. We will add the robustness results to the appendix.

For the baseline tuning, each variant was tuned with its own grid search (Section 2.3 and Appendix C), and the DyT-F/D training curves are shown in Appendix E. The GP search budget is given in Section 2.2, and the fine-tuning GPU hours in Appendix B. We will refer to these more clearly from the main text in the revision.

We hope this addresses your concerns and are happy to clarify any further questions.

AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their time and feedback. Based on the reviews, we have run the following additional experiments (details in the individual replies):

  • Hardware measurements: cycle-accurate RTL simulations of LN, DyT, and GP on two chips, a RISC-V SoC where computation dominates and the same SoC with a nonlinear accelerator unit where memory access dominates. GP uses fewer cycles and less energy than both LN and DyT in every configuration.
  • More baselines on ViT-B: a directly fitted per-layer clip (Clip-D), a per-layer calibrated tanh (Tanh-D), and Derf (Derf-D), all with the GP-D re-alignment settings. GP-D stays ahead of all three.
  • Ablations of the search objectives: without the tail penalty, training becomes unstable, and without the computational-cost objective, accuracy stays the same but the selected expressions cost about 15 times more.
  • More architectures: Swin-B and ConvNeXt-B, where GP outperforms DyT in all three adaptation variants. On top of this, GP-D outperforms DyT-D under distribution shift (ImageNet-V2, -Sketch, -R, and -A) on all four architectures.

We will add these results to the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.