- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework for replacing LayerNorm in pretrained Vision Transformers with layer-specific scalar functions discovered through genetic programming. The search balances approximation error, tail regularization, and estimated computational cost. The replacements inherit pretrained affine parameters and undergo 20 epochs of adaptation. Experiments on ViT-B and ViT-L evaluate functional alignment, ImageNet-1K accuracy recovery, and analytical computational and memory costs. The distillation-based variants achieve Top-1 accuracies of 84.32% and 85.74%, respectively.
Critical strengths
-
A practically motivated framework for adapting pretrained models. The approach addresses the relevant problem of replacing input-dependent normalization while preserving useful pretrained representations. Extracting layer-specific activation mappings, retaining pretrained affine parameters, and explicitly considering expression cost provide a coherent framework for adapting existing ViTs without training them from scratch.
-
Meaningful accuracy recovery with statistical reporting. The GP-based full fine-tuning and distillation variants outperform their corresponding DyT variants on both ViT-B and ViT-L. GP-D reaches 84.32% and 85.74% Top-1 accuracy, compared with 83.36% and 84.85% for DyT-D. The evaluation includes five independent runs with standard deviations, multiple adaptation schemes, independently tuned configurations, and complete recovery trajectories. These results support the effectiveness of the approach within the tested settings.
Critical weaknesses
-
The calibration data and the generalization of the replacement functions are insufficiently characterized. Section 2.1 states that a batch of images is passed through the pretrained model and 50,000 activation points are sampled from each layer, but does not specify the source, number, or sampling procedure of the calibration images. It is also unclear whether the 90/10 split ensures image-level disjointness. If both splits contain activations from the same images, the current validation does not establish the applicability of the fitted functions to independent images. Although the ImageNet classification results after fine-tuning support the effectiveness of the overall framework, they do not disentangle the generalization of the replacement functions from the compensatory effects of subsequent adaptation. The calibration protocol should be clarified, with fitting errors and classification performance reported on independent images and across different calibration subsets.
-
The coverage of directly relevant replacement baselines is limited. The evaluation primarily compares LayerNorm and DyT, without including other directly relevant pointwise normalization replacements such as Derf (Stronger Normalization-Free Transformers). Derf includes ViT experiments and is therefore a reasonable comparison. Evaluating accuracy and cost using the same pretrained models, calibration data, and adaptation budget would help establish the additional value of GP search over existing alternatives.
-
Key ablations are missing, leaving the contributions of structure search, layer-specific calibration, and the search objectives insufficiently disentangled. The existing A/F/D comparisons examine the adaptation scope and the role of distillation, but the following controls remain important: First, Directly fit simple clipping functions and compare them with GP-based structure search. The expressions shown in the appendix predominantly use clipping and simple combinations, so the necessity of searching over function structures remains unclear. Second, Compare a shared functional structure with layer-specific parameters against independently searched structures for different layers, separating the benefits of layer-specific parameter calibration from structural heterogeneity. Third, Separately remove the computational-cost objective and the tail penalty, reporting fitting error, training success rate, final accuracy, and expression cost. The authors describe potential instability when optimizing MSE alone, but do not provide systematic quantitative comparisons. In particular, the cost objective and tail penalty directly underpin the efficiency and stability claims and should receive priority in the ablation analysis.
-
The description of which parameters are trained in DyT-A is inconsistent between the main text and the appendix. Section 2.3 states that affine-only adaptation optimizes only the inherited scale and bias parameters, (w, b), whereas Tables 3 and 4 in Appendix C also specify a learning rate of 8 × 10⁻³ for α in DyT-A. The authors should clarify whether α is trained and reconcile the method description, configurations, and implementation. Since α controls the input scale of tanh, this discrepancy also affects the interpretation of the comparison between DyT-A and GP-A.
-
Presentation and organization could be improved. There is a noticeable blank space beneath the Discussion heading on page 9, which should be removed to improve the layout. The paper also lacks a dedicated Conclusion section.
Questions for authors
-
Calibration protocol and generalization. What are the source, number, and sampling procedure of the calibration images? Does the 90/10 split separate images, or only activation points? Could you report fitting errors on image-disjoint evaluation data and examine performance across different calibration subsets, both before and after adaptation?
-
Comparison with relevant alternatives. How does the proposed method compare with Derf when adapted from the same pretrained ViT checkpoints using matched calibration data and a comparable 20-epoch fine-tuning or distillation budget? Please compare both classification accuracy and estimated computational cost.
-
Contribution of the search components. Could you compare GP search with directly fitted clipping functions and a shared functional structure with layer-specific parameters? Could you also separately remove the computational-cost objective and the tail penalty under matched search and adaptation budgets, reporting fitting error, training success rate, accuracy, and expression cost?
-
Trainable parameters in DyT-A. Is α updated during DyT-A adaptation? Please clarify whether the reported results follow the “only (w, b)” description in Section 2.3 or the α learning-rate settings in Appendix C. Please specify the exact trainable parameter sets for both DyT-A and GP-A and explain any resulting difference in their adaptation capacity.
4 comments
Response to Reviewer KrFc (2/2)
W3/Q3 Contribution of the search components: We have run two directly fitted baselines on the same activation mappings and with the same re-alignment settings as GP-D (Clip-D and Tanh-D in the table above).
For the clip baseline (Clip-D, 5 runs), we fit per layer. The fitted clip closes most of the gap between DyT-D and GP-D, while GP-D stays consistently ahead (all five Clip-D runs score below the GP-D mean). However, we see this clip family as an outcome of our method rather than only a competing baseline: it was discovered by the GP search from the layer mappings, whereas DyT and Derf assume a tanh or erf shape beforehand. This is a general strength of our setup: given the mappings of a pre-trained model, the GP search finds cheap, deployable functions adapted to each layer, and their shapes can be reused directly, as Clip-D shows. We will add this comparison to the appendix and discuss the clip family as a finding of the search in the Discussion.
Clip-D and Tanh-D ( per layer, i.e. DyT with a per-layer calibrated input scale and output affine) together form the shared-structure control you suggest: one functional form in every layer, with parameters fitted per layer. Calibrating tanh per layer adds little over DyT-D (+0.09), even though Tanh-D fits the mappings better than our GP solutions. Changing the shape to a clip gives most of the gain (+0.77), and the per-layer structure found by the search adds a further +0.10.
We have also run the requested two ablations, with the same search and adaptation budgets. Removing the tail penalty improves the fit (MSE 0.066 vs 0.100) at a slightly higher cost (42d vs 31d) but makes training unstable. With the fp16 mixed precision of our GP-D recipe, three of the five equation sets fail within the first two epochs with a NaN loss. To rule out fp16 overflow as the only cause, we reran all five in bf16 (the 4/5 in the table): four sets train but reach only 78.77 ± 4.94, and one diverges to chance level, giving 64.10 ± 33.08 over all five against 84.32 ± 0.01 with the penalty. This is the activation explosion described in Section 2.2, which the tail penalty was introduced to prevent.
Without the computational-cost objective (using standard node count in GP instead of FLOPs and selecting the best-fitting expression per layer), the fit is slightly better (MSE 0.085 vs 0.100) and all five runs train, reaching a similar accuracy to GP-D (84.25 ± 0.02 vs 84.32 ± 0.01). However, the selected expressions cost 474d, about 15 times more than with the cost objective (31d) and almost 4 times more than LayerNorm itself. The cost objective therefore gives the efficiency gain without any loss in accuracy. We will add both ablations to the appendix.
W4/Q4 Trainable parameters in DyT-A: α is not updated during the DyT-A adaptation. Only each DyT layer’s inherited weight and bias parameters are updated. This keeps the number of trainable parameters equal for both DyT-A and GP-A (38,400 parameters), so both have the same adaptation capacity. The α learning rate settings in Appendix C will be corrected in the revision to reflect this. However, since α is a learnable parameter in the original DyT implementation, we have also trained DyT-A with α trainable. Our results showed that training α does not improve DyT-A performance (83.22 with α trained vs 83.23 frozen, one seed).
W5 Presentation: We thank the reviewer for pointing this out. We will remove the blank space on page 9 and check the revision for similar layout issues. The Discussion already summarizes our contributions and closes with limitations and future work, so it serves as the paper’s conclusion.
We hope this addresses your concerns and are happy to clarify any further questions.
Thank you for the response and additional experiments, which address several of my concerns. However, the new ViT-B results show that Clip-D achieves most of the accuracy gain, with GP-D adding only 0.10 percentage points. My concern about the limited additional methodological contribution remains, so I maintain my original rating.
Response to Reviewer KrFc
Thank you for the follow-up, and we are glad the additional experiments address several of your concerns. We agree that on ViT-B most of the gain over DyT-D comes from the clip shape (Clip-D 84.22% vs DyT-D 83.36%). However, this shape is itself a result of our search. As you noted in your review, the expressions in the appendix predominantly use clipping, and Clip-D was constructed from exactly these expressions. Without the search, the existing replacements use a fixed smooth shape, and on ViT-B these remain close to DyT-D, both with erf (Derf-D 83.55%) and with tanh calibrated per layer (Tanh-D 83.45%). The clip shape is therefore a contribution of the search itself. On top of this, the ablations you requested (W3 in our response) show that its other components are needed as well, as the cost objective lowers the cost about 15 times and the tail penalty keeps training stable.
We hope this clarifies the contribution of the different parts of our method.
Response to Reviewer KrFc (1/2)
We sincerely thank the reviewer for their time and effort, and especially appreciate the recognition of our framework as a practical way to adapt existing pre-trained ViTs without training from scratch. We address your concerns and questions below.
Below we summarize the new ViT-B baselines and ablations, together with the LN, DyT-D, and GP-D results from Table 2. We discuss these results under W2 and W3. Cost is given in FLOPs per token, summed over all 25 normalization layers (Appendix F).
W1/Q1 Calibration protocol: The mappings come from a single batch of 64 randomly drawn ImageNet-1K validation images (no labels are used), with 50,000 input-output points sampled per LN layer at random across tokens and features. The 90/10 split is over these points, not over images, and serves only for the GP metrics. We will clarify the image source in Section 2.1.
To check how well the equations generalize to independent images, we extracted new mappings from 64 ImageNet-1K training images and re-ran the full GP search on them (5 seeds, same configuration). The table below shows the fit of both sets of equations on both sets of mappings, and their top-1 accuracy directly after replacing LayerNorm (before adaptation) and after adaptation with distillation (GP-D, same settings as in the paper):
Both sets fit the mappings they were not evolved on about as well as their own, and give the same accuracy before and after adaptation. This means that the choice of images used to extract the mappings does not affect the fit or final accuracy. We will add these results to the appendix.
W2/Q2 Comparison with alternatives: We have run the requested additional experiment using the Derf method, adapted from the same pre-trained ViT-B checkpoint and using the same re-alignment settings and data, with distillation (Derf-D, 5 seeds, see the table above).
Derf-D improves slightly over DyT-D, but remains well below GP-D. Derf is also more expensive than DyT. Unlike tanh, erf cannot be written directly in terms of exp. We therefore use the Abramowitz–Stegun approximation of erf, which needs an exp and a polynomial, and evaluate it in Horner form as we do for tanh and sigmoid (Appendix F). Under our cost model, Derf then costs 36 FLOPs per element, compared to 24 for DyT. We agree that Derf is a relevant alternative and will include it as a baseline in the revision.