acceptodds
Withdrawn from ICLR 2027

NORMFOLD: Function-Identifiable Spectral Adaptation at LayerNorm Interfaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spectral adapters derive trainable coordinates from a stored pretrained matrix, but storage coordinates need not be uniquely determined by the function. We expose this ambiguity at a LayerNorm followed by an affine map. The complete local invariant is , where and . We prove that two parameter tuples with the same can induce spectral tangent spaces of different dimensions, so scale folding and offset preservation alone do not make spectral adaptation well defined on the pretrained function. NORMFOLD instead decomposes the unique minimum-Frobenius representative , preserves , and ties exactly repeated singular blocks. Its adapted family and matched optimizer trajectory are therefore determined by the local function in exact arithmetic. A controlled 9,120-fit study spans two CLIP checkpoints, three datasets, two shot counts, five support seeds, seven nonzero rewrite strengths, and five rewrite directions. Across all 2,100 matched nonzero task-direction conditions, NORMFOLD has lower final-logit discrepancy than the closest offset-preserving control. Its mean discrepancy remains between and , while that control ranges from to . NORMFOLD changes 9 of 10.88 million repeated paired predictions, compared with 27,145 for the control. Swin-T, seen-to-unseen transfer, fresh-image replay, merged export, and finite-precision analyses support the same function-identifiability account. The claim concerns whether the learning rule is representation independent, not whether NORMFOLD improves task accuracy over every spectral baseline.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
spectral adaptation, function identifiability, parameter-efficient fine-tuning, LayerNorm, reparameterization invariance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AsKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work is motivated by noticing that spectral finetuning methods are not well defined after a LayerNorm. Specifically, standardized LayerNorm activations are orthogonal with respect to the all-ones vector, which makes the row-constant compoent of the folded weigths invisible to the network output, while still affecting the SVD and therefore the trainable directions. The authors propose NormFold, which fixes this by folding the LayerNorm scale into the weights and projecting out the all-ones direction before decomposing the result. Experiments on few-shot adaptation of CLIP ViT-B/16 and ViT-B/32 on CIFAR-100, DTD, and Oxford Pets, plus Swin-T on CIFAR-100 and Pets, show that the resulting adapters lead to improved consistency under function-preserving rewrites of the checkpoint with respect to the authors own spectral controls

Critical strengths

The theoretical finding of LayerNorm ambiguity is interesting, and the fix shows to work at making the learned predictor invariant to function-preserving rewrites of the checkpoint

Critical weaknesses
  1. Paper's clarity needs to be improved. The way sentences are structured, and the lack of proper definition or contextualization of several terms, makes the paper hard to read
  2. Relevance is limited. It doesn't seem that the proposed fix is of actual usefulness. As accuracies do not improve over the spectral baselines, the method does not seem to be useful in practice. In addition, all non-trivial A/B pairs are created by an artificial rewrite, and the paper states that it doesn't estimate how often naturally produced checkpoints contain such null components (the null-component ratio of the checkpoints used is not reported either). The method should provide stronger evidence where base models visibly fail (accuracy, reproducibility, convergence, generality, ...) and where NormFold proves useful
  3. Generality is lacking. With only one layer being adapted, on vision models only. It is not clear how results would hold for more architectures, for example language models of different sizes where LayerNorm is present. Plus, there are no comparison with other published spectral methods such as CLIP-SVD, or a standard baselines like LoRA or full-finetuning to be used as reference
  4. Novelty is limited. Folding LayerNorm into the following projection and removing the all-ones direction is already done by SITH (Gentile et al., From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition, CVPR2026) for CLIP. The new contribution seems to mainly be the identifiability analysis applied to spectral PEFT methods
Questions for authors
  1. The 16-shots gains equal to 4-shot gains. This suggests that the training budget might be the limiting factor. How do conclusions change in different setups that lead to the expected differences between these? (longer training, higher learning rate, ...)
  2. Did the authors try adapting multiple layers, to try leading to a larger effect? How do the gains vary with more layers being adapted, up to all linear layers that directly use a LayerNorm output (QKV and MLP up-projection in every block)?
  3. Can the authors discuss the differences with respect to SITH (as mentioned in W4)
Reviewer_grXFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study investigates the dependence of spectral parameter-efficient adaptation on different parameter representations of the same pretrained function. At LayerNorm–affine interfaces, the authors found that different weight matrices implementing the same local function can lead to distinct singular-vector bases and therefore to different trainable spectral directions.

To resolve the above issue, the researchers developed NormFold. It creates spectral adaptation coordinates using a canonical function-visible operator. Specifically, the proposed approach incorporates the LayerNorm scale into the weight matrix, eliminates the unobservable mean direction, preserves the complete affine offset, and ties the multipliers corresponding to exactly repeated nonzero singular values. The theoretical developments consist of characterizing the complete local function invariant, providing an example of why different parameterizations may yield spectral tangent spaces of varying dimensions, and demonstrating that the adapted functions and optimization trajectories will be identical under matched optimization conditions in exact arithmetic.

The authors use a controlled study with 9,120 adaptation fits involving different combinations of CLIP models, datasets, support sets, and function-preserving parameter transformations to evaluate NormFold. They report that the results of these experiments demonstrate substantially lower prediction discrepancies between functionally equivalent parameter representations than those observed with conventional spectral adaptation controls. The authors also perform additional evaluations on Swin-T, seen-to-unseen classification, corrupted images, fresh-image inference, and finite-precision settings.

Critical strengths

An exact theoretical characterization of representation dependence in spectral adaptation.

In this paper, the authors discuss an important problem that can occur when spectral adaptation is used with model weights that have already been trained. While a single function can be represented by multiple sets of parameters, the different singular value decompositions of those sets of parameters can lead to different trainable directions.

Theorem 1 establishes a complete local invariant for LayerNorm-affine interfaces; Proposition 2 shows that equivalent matrices can result in spectral tangent spaces with different dimensions. Therefore, this explains why preserving the initial function by itself will not ensure that the subsequent adaptation process is representation-independent.

The NormFold approach resolves this problem within the stated setting by decomposing the canonical function-visible operator while preserving the consistency of spectral coordinate systems. The adapted-family and optimization-trajectory results obtained from the decomposition of the canonical function-visible operator offer a mathematical basis for the construction of NormFold.

A comprehensive and carefully controlled experimental validation.

The experimental design of this study is one of its strengths. Instead of evaluating the model solely based upon how accurately it performs in downstream tasks, the researchers examine whether parameters that represent the same function will result in consistent learned predictors.

The primary experiments evaluated multiple CLIP checkpoints, datasets, support sets, rewrite magnitudes, and rewrite directions. The evaluations of the various methods were performed with matched optimization settings and independently prepared spectral factors. NormFold achieved lower final prediction discrepancies between equivalent representations across all 2,100 evaluated matched nonzero task–rewrite-strength–direction conditions as compared to the closest offset-preserving control method.

The additional evaluations on Swin-T, seen-to-unseen classification, fresh-image inference, merged-weight export, and finite-precision behavior also support the empirical analysis. They are consistent with the claim that the representative-consistency property holds in the tested scenarios.

Critical weaknesses

The extent to which representation-dependent adaptation is significant for practical applications is still not well understood.

While this study clearly shows that transforming a model's parameters in a way that preserves its functionality can affect how conventional spectral adapters learn, all of the main experiments in this research were performed with artificially constructed transformations that affected the unobservable components of the pretrained weights.

These transformations are both mathematically valid and suitable for examining the theoretical claims. However, since the studies did not determine how often similar variations occur in naturally trained models or during typical model-conversion processes, it is unclear whether the practical implications of the identified failure mode have been demonstrated. Although the experiments are extensive, they primarily examine the same type of algebraically constructed transformation within a limited set of interfaces. Consequently, the number of experimental configurations does not necessarily establish the practical importance of the identified problem.

The improvements reported by the authors primarily relate to the consistency of different parameter representations of the same function. The downstream performance of NormFold is comparable to that of the strongest control as reported by the authors; therefore, there is no strong evidence of improved predictive capability beyond the scope of their original goal.

A quantitative examination of naturally occurring unobservable weight components, along with some examples of real-world checkpoint transformations, could demonstrate when the suggested correction will have a significant impact. Without such evidence, it is difficult to determine whether the proposed method addresses a practically important limitation of existing spectral adaptation techniques.

The methodological scope is limited, and the differences between this method and other existing centering-aware spectral methods should be more clearly defined.

The primary theoretical guarantees apply to a LayerNorm-affine interface with fixed standardized-input coordinates. In addition, the implemented adaptation method trains spectral multipliers with fixed singular vectors.

This is a reasonable setting to establish the principle being proposed, but it does not describe more general spectral adaptation approaches based on rank truncation, sparse spectral coordinate systems, or updating the singular-vector subspaces. Additionally, rank selection and numerical grouping thresholds can create further representation-dependent choices that are not fully addressed by the exact singular-block analysis.

In addition, previous studies have used LayerNorm folding and removed the mean direction before applying spectral analysis (e.g., SITH). Although this study provides a theoretical foundation for function-identifiable adaptation and how it affects optimization, a clearer comparison could be drawn between what this study does differently and what has already been established by other studies that incorporate centering-aware spectral analysis. In particular, the main construction relies heavily on existing normalization properties and spectral decomposition operations. The paper has not sufficiently demonstrated that their combination constitutes a substantial methodological advance beyond the formalization of these known properties.

The limitations listed above do not invalidate the theoretical results of the work; however, they substantially limit its methodological novelty and practical significance. As a result, I do not find the current contribution sufficiently compelling as a general approach to spectral parameter-efficient fine-tuning.

Questions for authors

Practical relevance of function-preserving transformations (experimental validation).

Can the authors estimate the magnitude of the unobservable components in naturally occurring pretrained weights and determine whether common checkpoint transformations alter the way the model adapts?

It would be especially beneficial to see whether the representative dependence observed occurs during real-world model-conversion processes, as opposed to only under explicitly constructed null-direction transformations. If the findings show that this occurs in real-world applications, they would provide evidence of the practical significance of the proposed method.

Extension to broader spectral adaptation methods (theoretical and experimental clarification).

How can the proposed principle of function identifiability be applied to spectral adaptation techniques that either select a small subset of singular directions or alter the subspaces spanned by the singular vectors?

Could the authors please clarify how a fixed-rank selection is affected when it intersects a repeated or nearly repeated singular-value block? Also, additional experiments using different spectral budgets would help establish whether the authors' results continue to hold under increasingly constrained parameter-efficient adaptation configurations.

Relationship to existing centering-aware methods (technical clarification).

What differentiates NormFold from the direct application of LayerNorm folding, centering, and spectral decomposition prior to spectral adaptation?

I believe that making a clearer separation between the canonicalization process itself and the newly established function-identifiability and optimization properties would provide better insight into how each can be considered an independent methodological contribution.

Reviewer_nEsbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a representation-dependence problem in spectral adaptation at LayerNorm–affine interfaces: weights implementing the same initial function can yield different SVD-based adaptation directions and training trajectories. NORMFOLD addresses this problem by constructing a spectral adaptation rule uniquely determined by the local function. Under matched training conditions, it guarantees identical adapted functions and optimization trajectories. Controlled experiments on functionally equivalent checkpoints show that NORMFOLD substantially reduces differences in model outputs after adaptation while maintaining comparable task accuracy.

Critical strengths
  1. The paper shows that functionally equivalent weight representations can induce spectral adaptation tangent spaces of different dimensions, and thus different adaptation trajectories. This is interesting and I appreciate this mechinical analysis.

  2. The experiments isolate the effect of weight representation by comparing functionally equivalent checkpoints with independently computed SVDs under matched training conditions.

  3. NORMFOLD requires only a preprocessing change to spectral adaptation, without introducing additional trainable parameters or inference modules.

Critical weaknesses
  1. The paper addresses a narrow issue within spectral adaptation. While the controlled experiments clearly demonstrate representation dependence, they do not establish whether such dependence commonly arises in practical workflows. This limits the significance of the proposed correction.

  2. SITH already folds LayerNorm affine parameters and removes the all-ones direction before spectral analysis. The main additional contributions seems to be the tangent-space counterexample, the formal consistency guarantees, and the controlled audit, rather than a brand new procedure.

  3. The primary experiments adapt only one LayerNorm–linear interface in the CLIP model. This isolates the proposed mechanism effectively, but leaves its impact on complete, multilayer spectral PEFT methods uncertain.

Questions for authors
  1. Beyond the deliberately constructed rewrites, can the authors provide an example of a common checkpoint-processing workflow that introduces comparable representation changes while preserving the model’s function? How much would such changes affect subsequent spectral adaptation in practice?

  2. Neural networks admit function-preserving reparameterizations beyond the LayerNorm null direction, such as compensating changes across consecutive linear maps. Can these also change the adaptation directions selected by SVD-based methods? If so, what makes the LayerNorm case particularly important to resolve, and to what extent does handling this local ambiguity suffice when other representation dependencies remain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.