- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work is motivated by noticing that spectral finetuning methods are not well defined after a LayerNorm. Specifically, standardized LayerNorm activations are orthogonal with respect to the all-ones vector, which makes the row-constant compoent of the folded weigths invisible to the network output, while still affecting the SVD and therefore the trainable directions. The authors propose NormFold, which fixes this by folding the LayerNorm scale into the weights and projecting out the all-ones direction before decomposing the result. Experiments on few-shot adaptation of CLIP ViT-B/16 and ViT-B/32 on CIFAR-100, DTD, and Oxford Pets, plus Swin-T on CIFAR-100 and Pets, show that the resulting adapters lead to improved consistency under function-preserving rewrites of the checkpoint with respect to the authors own spectral controls
Critical strengths
The theoretical finding of LayerNorm ambiguity is interesting, and the fix shows to work at making the learned predictor invariant to function-preserving rewrites of the checkpoint
Critical weaknesses
- Paper's clarity needs to be improved. The way sentences are structured, and the lack of proper definition or contextualization of several terms, makes the paper hard to read
- Relevance is limited. It doesn't seem that the proposed fix is of actual usefulness. As accuracies do not improve over the spectral baselines, the method does not seem to be useful in practice. In addition, all non-trivial A/B pairs are created by an artificial rewrite, and the paper states that it doesn't estimate how often naturally produced checkpoints contain such null components (the null-component ratio of the checkpoints used is not reported either). The method should provide stronger evidence where base models visibly fail (accuracy, reproducibility, convergence, generality, ...) and where NormFold proves useful
- Generality is lacking. With only one layer being adapted, on vision models only. It is not clear how results would hold for more architectures, for example language models of different sizes where LayerNorm is present. Plus, there are no comparison with other published spectral methods such as CLIP-SVD, or a standard baselines like LoRA or full-finetuning to be used as reference
- Novelty is limited. Folding LayerNorm into the following projection and removing the all-ones direction is already done by SITH (Gentile et al., From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition, CVPR2026) for CLIP. The new contribution seems to mainly be the identifiability analysis applied to spectral PEFT methods
Questions for authors
- The 16-shots gains equal to 4-shot gains. This suggests that the training budget might be the limiting factor. How do conclusions change in different setups that lead to the expected differences between these? (longer training, higher learning rate, ...)
- Did the authors try adapting multiple layers, to try leading to a larger effect? How do the gains vary with more layers being adapted, up to all linear layers that directly use a LayerNorm output (QKV and MLP up-projection in every block)?
- Can the authors discuss the differences with respect to SITH (as mentioned in W4)