acceptodds
Under review as a conference paper at ICLR 2027

Parameter Symmetries Are Not Task Symmetries: Which Invariances Can Weight-Space Learning Afford?

Abstract

Weight-space learning predicts properties of neural networks from their parameters, and predictors are commonly made invariant to all parameter symmetries, i.e., to every transformation that preserves the network function. We show that the right invariance depends on the target. Each target has its own task-symmetry group, and quotienting by anything larger incurs an irreducible error, which we compute exactly for compact groups. We then derive task-symmetry groups for broad target classes. In LoRA, a single parameterization exhibits a strict ladder: function-level targets are -invariant, Euclidean sensitivity is only -invariant, gradient sparsity is only invariant under signed permutations, and distance from initialization has no symmetry. The two middle levels are selected by the optimizer, since gradient descent and Adam measure progress by the and gradient norms, respectively. Finally, neuron-permutation symmetry, which most weight-space architectures quotient, is lost by any target that refers to a reference model, such as drift from a pretrained base. Experiments on synthetic zoos and LoRA-fine-tuned language models confirm the predicted separations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Weight-space learning, parameter symmetries, expressive power, Low-Rank Adaptation (LoRA), parameter sensitivity, gradient sparsity.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_54DGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers the case of meta-networks that take as input model weights with the goal of predicting some property of the model. In this case its main argument and contribution is that weight symmetries (i.e. symmetries that preserve the model’s function) and task symmetries (i.e. the subset of weight symmetries that preserve the quantity to predict) should be treated separately. Specifically, the authors argue and show that a meta-network should be invariant exactly to task symmetries (and not necessarily to all weight symmetries). Otherwise, i.e., if it is invariant to a larger set of symmetries, there is an irreducible error.

Critical strengths

Positive points:

  • The argument that weight symmetries and task symmetries should be distinguished, although kind of obvious in retrospect, is indeed a useful observation. The authors also quantify the irreducible error that occurs when a larger set of invariances is enforced.
  • They prove the strict LoRA ladder and show the optimizer dependent symmetry in LoRA
Critical weaknesses

Negative points:

  • Concerning the theorems in section 3, they do not bring any really new theoretical results. The new thing here is the framing of how they are used (i.e. in the context of meta-networks that take as input network weights and predict network properties). The same applies to the optimizer equivariance proposition 5.1. The theoretical novelty is thus mainly concentrated on the LoRA results.
  • The synthetic zoo LoRA experiments use very small weight matrices. Moreover, each LoRA adapter is randomly reparameterized, whereas in practice the initialization of the second LoRA matrix is typically zero, which, to a large extent, constrains the parameterization of the LoRA adapters in conjunction with training dynamics. This raises the question of how realistically these experiments reflect the conditions encountered in the case of large scale models. The authors need to show results on a real zoo of LoRA adapters to convincingly demonstrate how important the issue actually is in practice (i.e. a real zoo + meta-model experiment). Note that this is not shown by the reparameterization tests reported in section 7.4.
Questions for authors

The authors should address the issues raised about experimental evaluation and practical importance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 54DG (1/2)

We thank the reviewer for the careful reading of our paper and their valuable comments. Here's our response:

Weakness 2

Thanks for your suggestion. We ran exactly the experiment the reviewer asked for in the revised version. The new results are in Sec 7.4 (Table 1) and App. D of the paper.

  • Real zoo with standard training. 671 LoRA adapters (2,013 checkpoints) of rank 8 on RoBERTa-base (d=768d = 768, 24 adapted modules), trained on six GLUE tasks. We use the standard PEFT recipe. BB starts at zero. AA is Kaiming-uniform (AdamW) or Gaussian (SGD). The other hyperparameters are standard and randomized. We apply no synthetic reparameterization anywhere.
  • Meta-models on this zoo. We train the same backbone and change only the imposed symmetry (Sec 7.4, Table 1 and App. D, Table 7). On function-level targets the GLr\mathrm{GL}_r-invariant model beats the unconstrained one (validation-loss R2R^2 0.39 vs −0.01). On Gaussian-perturbation robustness the order flips (0.27 vs 0.41). Feeding gauge statistics as an extra input recovers the gap (0.51).
  • A control with no deep network. We also use nested invariant features over 5 splits (App. D, Table 6). The information that a GLr\mathrm{GL}_r-invariant predictor throws away is worth +0.21+0.21 to +0.41+0.41 in R2R^2 on noise robustness and +0.14+0.14 to +0.24+0.24 on gradient sparsity. It is worth nothing on function-level targets.
  • The symmetry structure holds exactly on these real checkpoints (App. D, Table 5). On 90 checkpoints, the ceilings are 1.000 wherever the theory predicts it. GLr\mathrm{GL}_r changes leave the function unchanged up to 4⋅10−64 \cdot 10^{-6}. Rotations remove 46–62% of the variance of quantization robustness.

Why B=0B = 0 does not make the problem go away. In fact, standard training is a direct case of the theory.

  • The trained adapters have an invariant law. Every group element keeps B=0B = 0 fixed. An i.i.d. Kaiming-uniform AA has an SPr\mathrm{SP}_r-invariant law. An i.i.d. Gaussian AA has an O(r)\mathrm{O}(r)-invariant law. Adam is SPr\mathrm{SP}_r-equivariant and SGD is O(r)\mathrm{O}(r)-equivariant (Prop 5.1). So the trained adapters have an exactly SPr\mathrm{SP}_r-invariant law under Adam. Under SGD from a Gaussian AA, the law is exactly O(r)\mathrm{O}(r)-invariant. This still holds with minibatches, learning-rate schedules, weight decay, clipping and LoRA+ ratios. It also holds for any mix of hyperparameters. So Theorem 3.3 applies to standard pipelines for these groups, with no artificial randomization. This is now in the paper as Remark 5.3, with the full statement and proof in Prop. B.2.
  • Training does not randomize the GLr\mathrm{GL}_r gauge, but it does not fix it either. The gauge changes with standard choices such as the learning rate, LoRA+ ratio, alpha and init scale. Adapter hubs mix these choices.
  • The cost grows with the gauge spread, as predicted, and vanishes without it (S1, App. E, Fig. 5a). In a controlled zoo we increase the gauge spread step by step. The cost ρH\rho_H computed before training matches the measured extra error of the GLr\mathrm{GL}_r-invariant model almost exactly. For sensitivity it is 0.117 vs 0.123, 0.708 vs 0.709 and 0.930 vs 0.914 (predicted vs measured). With no reparameterization the invariant model is close to the unconstrained one (R2R^2 0.946 vs 0.954 on the ℓ1\ell_1 gradient norm). The loss, which is GLr\mathrm{GL}_r-invariant, stays at R2=0.976R^2 = 0.976 at every spread.

We agree that the reparameterization tests in the old Sec 7.4 (now Sec 7.5) do not involve a learned predictor. They now serve only as supporting evidence. The new real-zoo experiment is the main evidence.


AuthorsOpenReview ↗

Response to Reviewer 54DG (2/2)

Weakness 1

We agree that the results in Section 3 use standard tools, namely projection onto invariant functions and orbit averaging. The revised paper now presents Section 3 as the framework. It credits these tools at the start of the section. What we claim there is a new use of these results in weight-space learning. They give a criterion that can be computed and that selects the architecture. The ceiling 1−ρH1 - \rho_H can be estimated on a zoo before any meta-model is trained. It also grows monotonically as the group gets larger. Reviewer 1kyk points to this as a strength, and we now test it on real checkpoints. There the ceilings are exact wherever predicted, and they rank the targets by how much they need non-invariant information (App. D).

The technical results beyond Section 3 are the following.

  • The strict LoRA ladder GLr⊋O(r)⊋SPr⊋{I}\mathrm{GL}_r \supsetneq \mathrm{O}(r) \supsetneq \mathrm{SP}_r \supsetneq \{I\}. It finds the largest subgroup that survives for each class of targets (Thm 4.3).
  • The curvature ladder (Prop A.3).
  • The result that the optimizer selects the symmetry. It links the init law and the update rule to the invariance of the zoo (Sec 5, Remark 5.3).
  • The counting result for permutations. Permutation-invariant linear order-kk targets are only a Bell(k)/hk\mathrm{Bell}(k)/h^k fraction of all such targets. For symmetric tensors the fraction is at most k! Bell(k)/hkk!\,\mathrm{Bell}(k)/h^k. This shows how much information permutation quotienting removes for reference targets (Thm A.5, Sec 6).

We have rewritten the contributions in the introduction in these terms. The paper also has two new practical parts. One is a procedure for choosing or adapting the symmetry of a metanetwork (Procedure 1). The other extends the criterion and Thm 3.3 to equivariant weight-to-weight outputs (App. C).

Reviewer_1kykOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies which parameter symmetries should be imposed on weight-space predictors. It distinguishes model symmetries, which preserve the represented function, from task symmetries, which preserve the prediction target. For compact groups and invariant data distributions, it characterizes the minimum prediction error caused by imposing an incompatible invariance. It then identifies task-symmetry groups for several target families, including derivative-based, optimizer-dependent, and reference-based targets, with LoRA as the main example.

Critical strengths
  1. I particularly appreciate Theorem 3.3. Under its stated assumptions, it provides a way to estimate the irreducible error imposed by an invariance on a given model zoo. This makes the theoretical analysis potentially useful for choosing the symmetry of a weight-space architecture before training it.

  2. The LoRA hierarchy illustrates how different targets can require different invariances. The connection to optimizer dynamics also helps explain why these distinctions matter.

Critical weaknesses
  1. From the current experiments, I cannot tell whether imposing the wrong invariance also hurts prediction on real distributions of trained models. The prediction experiments use synthetic zoos with random reparameterizations, while the language-model experiments do not evaluate learned predictors. A comparison between invariant and unconstrained predictors on models obtained through ordinary training or fine-tuning would help clarify this.

  2. The motivation for predicting function-level properties, such as accuracy or loss, is intuitive. For the parameterization-level targets considered here, the benefit of learning a predictor is less clear. Section 5 connects gradient quantities to optimization progress, but I would appreciate more concrete downstream applications for predicting sensitivity, curvature, gradient sparsity, and robustness to parameter perturbations. This concern is particularly relevant to the distance from a fixed anchor or initialization. When both the weights and the anchor are available, this distance can be computed directly in time linear in the number of parameters. A weight-space model that reads all parameters also incurs at least linear computational cost, making the practical advantage of predicting this quantity unclear. The paper should therefore clarify a setting in which learning to predict this distance offers a practical benefit over direct computation.

Minor Comments:

  • In Theorem 3.1, please add a space between "If" and W/H.
  • In Theorem 3.3, item 1, h denotes both the predictor and the group element in the orbit average. Using g for group elements would make the statement easier to read.
Questions for authors
  1. A related distinction between function-level and parameterization-level targets is discussed in On the Expressive Power of Permutation-Equivariant Weight-Space Networks (Dayan et al., ICML 2026), which also considers equivariant operators. Could the authors clarify whether the framework can be extended from scalar functionals to weight-to-weight operators with equivariant outputs? Appendix B identifies equivariant outputs as future work. Could the authors outline whether the task-symmetry criterion and the error characterization in Theorem 3.3 admit analogous formulations, and what additional assumptions would be needed?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1kyk (1/2)

We thank the reviewer for the careful reading of our paper and their valuable comments. Here's our response:

Weakness 1

We ran this comparison in the new version. The new results are in Sec 7.4 (Table 1) and App. D of the paper. Here's a summary:

  • Zoo. 671 LoRA adapters (2,013 checkpoints) on RoBERTa-base (rank 8, 24 modules, d=768d = 768), trained on six GLUE tasks with standard PEFT settings only. We use B=0B = 0 at init and AdamW or SGD. We randomize the learning rate, LoRA+ ratio, alpha, init scale, weight decay and training length. We apply no synthetic reparameterization.
  • Predictors. One backbone, where only the input invariance changes.
target (GTG_T)unconstrainedGLr\mathrm{GL}_r-invariantGLr\mathrm{GL}_r-inv. + gauge stats
validation loss (GLr\mathrm{GL}_r)−0.010.390.40
probe loss (GLr\mathrm{GL}_r)0.130.410.45
Gaussian robustness (O(r)\mathrm{O}(r))0.410.270.51
∥∇L∥1/∥∇L∥2\Vert \nabla L\Vert_1 / \Vert \nabla L\Vert_2 (SPr\mathrm{SP}_r)0.120.090.41

The table shows test R2R^2 over 3 seeds. On function-level targets the invariant model is far better. On noise robustness, imposing GLr\mathrm{GL}_r-invariance costs R2R^2. Feeding the gauge statistics as an extra input recovers it.

A control with no deep network gives the same picture (App. D, Table 6). It uses nested invariant feature sets GLr⊂O(r)⊂SPr\mathrm{GL}_r \subset \mathrm{O}(r) \subset \mathrm{SP}_r with ridge regression and gradient-boosted trees, over 5 splits. Since the sets are nested, a gain can only come from the extra information.

  • There is no gain on function-level targets (within ±0.02\pm 0.02).
  • The gain is +0.21±0.03+0.21 \pm 0.03 (trees) and +0.41±0.07+0.41 \pm 0.07 (ridge) on Gaussian robustness.
  • The gain is +0.14±0.02+0.14 \pm 0.02 and +0.24±0.09+0.24 \pm 0.09 on ∥∇L∥1/∥∇L∥2\Vert \nabla L\Vert_1 / \Vert \nabla L\Vert_2.
  • On this SPr\mathrm{SP}_r-only ratio, SPr\mathrm{SP}_r features add a further +0.025±0.009+0.025 \pm 0.009 over O(r)\mathrm{O}(r).

As the reviewer suggests, we also used Theorem 3.3 directly. We computed the Monte Carlo ceilings 1−ρH1 - \rho_H on 90 real checkpoints, before training any meta-model (App. D, Table 5).

  • Where the target is HH-invariant. The ceilings are exactly 1.000, on all 90 checkpoints.
  • Where it is not. The ceilings rank the targets correctly. Gaussian robustness has the lowest GLr\mathrm{GL}_r ceiling (0.74), and it shows the largest measured gain from non-invariant information (+0.21+0.21 in R2R^2). Targets with ceilings close to 1, like ∥∇L∥2\Vert \nabla L\Vert^2 (0.991) and ∥∇L∥1\Vert \nabla L\Vert_1 (0.995), show no gain.
  • Quantization robustness. Rotations remove 46–62% of its variance (O(r)\mathrm{O}(r) ceiling 0.38–0.54), while signed permutations remove none.

The theorem applies to real pipelines for a simple reason. With B=0B = 0 and an i.i.d. init of AA, the trained adapters have an exactly SPr\mathrm{SP}_r-invariant law under Adam. Under SGD from a Gaussian AA, the law is exactly O(r)\mathrm{O}(r)-invariant. This follows from Prop 5.1 and the invariance of the init law. We added this to the paper as Remark 5.3, with the full statement and proof in Prop. B.2.


Weakness 2

We agree with the reviewer on the anchor distance, as when the anchor is known, a predictor cannot replace direct computation. We use it as a diagnostic target. Its task symmetry Stab(w0)\mathrm{Stab}(w_0) is known in closed form, so it gives a clean test of the theory. The paper now says this clearly in Sec 2 and in the new App. B.1. Reference targets become worth predicting when the anchor is unknown, as in model-lineage recovery. They are also worth predicting when the reference quantity itself is expensive.

For sensitivity, curvature, gradient sparsity and robustness to noise or quantization, the case rests on cost and on access to data.

  • Cost. Computing these targets directly needs data and backward passes, or Hessian-vector products. A Hutchinson trace needs many of them over a dataset. A weight-space predictor reads only the weights, once, with no data.
  • Access to data. This matters when we screen adapter hubs, since the training data is usually not available. For example, we may want adapters that survive low-bit quantization, that are robust to weight noise, or that are safe to merge.

AuthorsOpenReview ↗

Response to Reviewer 1kyk (2/2)

Question 1

Yes! Both the criterion and the error result extend, as they need one extra assumption about the metric. We added this as a new App. C. Let TT be a target with values in a space VV, and let ψ\psi be an action of the group on VV.

  • Criterion (Prop. C.1). An equivariant predictor with output action ψ\psi can be lossless exactly when T(gw)=ψ(g) T(w)T(gw) = \psi(g)\,T(w) for all g∈Hg \in H. So the task symmetry GT,ψG_{T,\psi} is the set of gg with this property. It depends on the chosen output action. When ψ\psi is orthogonal, the pointwise bound of Thm 3.2 also carries over.

  • Error (Thm. C.2). Take HH compact, an HH-invariant zoo, and ψ\psi orthogonal on HH. The best equivariant predictor is the twirl

    PHT(w)=Eg ψ(g)−1 T(gw).P_H T(w) = \mathbb{E}_g\, \psi(g)^{-1}\, T(gw).

    The irreducible error is

    Ew Eg∥ψ(g)−1 T(gw)−PHT(w)∥2.\mathbb{E}_w\, \mathbb{E}_g \big\Vert \psi(g)^{-1}\, T(gw) - P_H T(w) \big\Vert^2 .

    This is the spread of the target over each orbit, after we map it back to a common frame with ψ(g)−1\psi(g)^{-1}. Monotonicity, the zero-cost result and the R2R^2 ceiling all carry over. This is the exact analogue of Theorem 3.3, with the same proof. The orthogonality of ψ\psi is what makes the twirl a projection. One thing is new. A constant is equivariant only if ψ(H)\psi(H) fixes it, so the normalized cost can be larger than 1.

  • The extra assumption is about the metric. For compact HH it costs nothing. We can always average the inner product over ψ(H)\psi(H), so that ψ\psi becomes orthogonal, and the theorem holds in that metric. For non-compact HH it matters. Gradients transform contragrediently. Under A→CAA \to CA and B→C−⊤BB \to C^{-\top}B, the gradients map as ∇A→C−⊤∇A\nabla_A \to C^{-\top}\nabla_A and ∇B→C ∇B\nabla_B \to C\,\nabla_B. So the gradient is exactly GLr\mathrm{GL}_r-equivariant with the right output action ψ=ρ⋆\psi = \rho^\star. With the naive action ψ=ρ\psi = \rho, the group is exactly O(r)\mathrm{O}(r) (Prop. C.3). The gradient norm is also only O(r)\mathrm{O}(r)-invariant. This is a general pattern. If we apply a function Φ\Phi to an equivariant target, only the gg with Φ∘ψ(g)=Φ\Phi \circ \psi(g) = \Phi survive. This is how the scalar ladder of Thm 4.3 comes from an equivariant target. For GLr\mathrm{GL}_r acting by ρ⋆\rho^\star there is no invariant metric. So, as in the scalar case, the error result then needs a prior over the group.

  • Link to Prop 5.1. Optimizer updates are themselves weight-to-weight operators. SGD steps are O(r)\mathrm{O}(r)-equivariant and sign or Adam steps are SPr\mathrm{SP}_r-equivariant. Over all admissible losses these groups are exact (Prop. C.4, which follows from Thm 5.2). So Prop 5.1 is one instance of the equivariant criterion.

  • Relation to the paper (Dayan et al., ICML 2026). They ask what a permutation-equivariant network can represent, and show that the main architectures have the same expressive power, and that they are universal under a general-position assumption. They treat targets that depend on the function and targets that depend on the weights, with invariant and equivariant outputs. We ask a different question. Which group should we impose for a given target, and what does a mismatch cost? The two views fit together. Where their universality holds, the ceiling of Thm 3.3 or C.2 with H=ShH = S_h is the only limit left. It is not a limit of expressive power. Our analysis also finds weight-dependent targets where ShS_h itself is too large, namely reference targets (Sec 6). And beyond permutations, for example in scaling-equivariant metanetworks, outputs like gradients need the contragredient output action.

The new experiment S4 tests this (App. E, Table 9 of the paper).

  • Gradient target. We predict the gradient with equivariant models built by canonicalization. The GLr\mathrm{GL}_r model with the contragredient ψ\psi is the best model, with R2=0.978R^2 = 0.978. The same model with the naive ψ=ρ\psi = \rho reaches only 0.270. The O(r)\mathrm{O}(r) and SPr\mathrm{SP}_r models with ψ=ρ\psi = \rho reach 0.957 and 0.946, close to their ceiling of 1, as predicted.
  • Adam-like update (only SPr\mathrm{SP}_r-equivariant). The O(r)\mathrm{O}(r)-equivariant model reaches 0.708 and stays below its twirl ceiling of 0.881. The SPr\mathrm{SP}_r model is not capped and is the best model (0.752).

This extension, with the discussion of Dayan et al., is now App. C of the paper. It replaces the short future-work remark in the old Appendix B (now App. F). We also cite Dayan et al. in the related work of the main text.


Minor comments. Thank you. We fixed both.


Reviewer_xYqtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper distinguishes transformations preserving a neural network’s function from those preserving a weight-space prediction target. It characterizes lossless invariant prediction and, for compact groups with invariant sampling distributions, expresses unavoidable squared error as within-orbit target variance, yielding an R2R^2 ceiling. For LoRA, it derives a hierarchy from general linear invariance through orthogonal and signed-permutation invariance to trivial symmetry, depending on derivative geometry or a fixed reference. It connects these groups to gradient and sign-based optimization and analyzes neuron permutations. Synthetic model-zoo experiments, optimizer continuation tests, and reported language-model measurements illustrate both the benefits and costs of imposing invariance.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The paper gives principled guidance on which weight differences a predictor can safely ignore depending on the task. It suggests more practical applications of weight space learning.

  3. The experiments show both when invariance helps and when it hurts, which makes the practical trade-off clear.

Critical weaknesses
  1. The practical implications for model design could be clearer. The paper identifies when invariance can discard useful information, but gives less guidance on how to adapt an existing metanetwork (or meta-model) for a particular task. In particular, it is unclear when the solution is to change the enforced symmetry rather than the model’s inputs or readout.

  2. The experiments seem to support the theoretical mechanism more clearly than the practical benefit. Most meta-learning evidence comes from controlled settings, leaving unclear how much the proposed symmetry choices improve prediction on normal checkpoints. This limits the evidence for using the framework to guide real weight-space learning applications.

Questions for authors
  1. How should we choose which symmetries to enforce when we have only a small number of training checkpoints?

  2. Can we estimate how much information invariance would discard from a real set of checkpoints, even if it does not meet the theory’s assumptions?

  3. How much of the performance difference is due to the choice of symmetry, and how much is due to the architectures or how they were trained?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xYqt (1/3)

We thank the reviewer for the careful reading of our paper and their valuable comments. Here's our response:

Weakness 1 and Question 1

ANSWER: Thanks for your suggestion. Indeed, our framework gives a simple procedure for this. We added it to the paper as "Procedure 1" in Sec 3, please check our revised version. Also, App. B.2 gives more details and examples on it. Here's a summary of this procedure:

  1. First check the task symmetry GTG_T of the target, such as what we did in Sec 4 to 6 to give GTG_T for derivative, optimizer and reference targets. Then, if the group HH of the metanetwork is already inside GTG_T, nothing needs to change.

  2. But if HH is larger than GTG_T, we first check how much this costs. A larger group always pays a fixed extra error ρH\rho_H, and this error does not shrink with more data. We can estimate ρH\rho_H from the checkpoints we have, without training any model. In return, a larger group needs fewer checkpoints to learn. So with few checkpoints we compare the two quantities, and if ρH\rho_H is smaller than the error that a GTG_T-invariant model still makes at our number of checkpoints, we keep HH. This is our answer to Q1, and we explain it in more detail below.

  3. Otherwise there are three solutions, and the right one depends on what breaks the symmetry.

    • (a) Change the inputs. Use this when the target depends on something outside ww that moves with ww. One example is a reference model w0w_0, such as a base model, an initialization or a second network. We feed w0w_0 as an extra input and transform it together with ww. The target is then invariant again, so we can keep the same HH-equivariant network. In the same way, we can add simple statistics that HH removes but GTG_T keeps. For O(r)\mathrm{O}(r)-level LoRA targets, the singular values of AA and BB are one example.

    • (b) Relax the layers. Use this when the target depends on how ww itself is parameterized. Derivative and trajectory targets are like this. For example, we replace GLr\mathrm{GL}_r-equivariant layers by O(r)\mathrm{O}(r)-equivariant ones for Euclidean and SGD targets, or by SPr\mathrm{SP}_r-equivariant ones for ℓ1\ell_1 and Adam targets.

    • (c) Add a non-invariant readout. Use this when GTG_T is trivial or unknown, or when the backbone cannot be retrained. It is the cheapest fix. But it loses the sample-efficiency gain on the invariant part of the target.

Few checkpoints (Q1). With few checkpoints there is a bias–variance trade-off, and our framework makes it exact. For any HH-invariant learner h^\hat h, the error splits into two parts:

E (T−h^)2=BH(T)+E (TˉH−h^)2.\mathbb{E}\,(T - \hat h)^2 = B_H(T) + \mathbb{E}\,(\bar T_H - \hat h)^2 .

The first part is the irreducible error ρH⋅Var(T)\rho_H \cdot \mathrm{Var}(T) from Thm 3.3. It does not go down with more data. The second part is the estimation error. A larger group makes it smaller. The key point is that we can estimate ρH\rho_H before we train any meta-model (see Q2). This gives a simple rule. Enforce a group HH larger than GTG_T only if ρH\rho_H is small compared with the validation error (1−R2)(1 - R^2) of a GTG_T-invariant model at our nn. As nn grows, the rule moves towards GTG_T.

New experiments show the procedure at work. The controlled ones are in the new App. E of the paper, and the real zoo is in the new Sec 7.4 and App. D. Please check them, and here we have a summary of the results:

  • S2, few checkpoints (App. E, Fig. 5b–c). We vary nn from 50 to 6400 checkpoints. Imposing exactly GTG_T is much more sample-efficient than imposing nothing. The O(r)\mathrm{O}(r)-invariant model reaches with 100 checkpoints what the unconstrained model needs about 1600 checkpoints for. A group larger than GTG_T stays at or above the floor ρGL\rho_{\mathrm{GL}} that we computed before training. A larger group helps only a little, and only at very small nn with a small ρGL\rho_{\mathrm{GL}}. For example, for the ℓ1\ell_1 gradient norm with ρGL=0.046\rho_{\mathrm{GL}} = 0.046, the GLr\mathrm{GL}_r model is slightly better at n≤100n \le 100 (0.427 vs 0.468 NMSE at n=50n = 50). From n=200n = 200 on, the O(r)\mathrm{O}(r) model wins. For sensitivity, the larger group never wins. This matches the rule above.

  • S3a, fix (a) for reference targets (App. E, Table 8). We predict drift from a base model under neuron permutations. An ShS_h-invariant DeepSet on ww alone reaches R2=0.076R^2 = 0.076, which is at its ceiling. The same DeepSet on neuron pairs (wi,w0,i)(w_i, w_{0,i}) reaches R2=1.000R^2 = 1.000. It also keeps the permutation-invariant targets, while the unconstrained model fails on them.

  • S3b, fix (a) for LoRA (App. E, Table 8). A GLr\mathrm{GL}_r-invariant model that predicts sensitivity reaches R2=0.189R^2 = 0.189. Adding the singular values of AA and BB raises it to 0.856. Adding the Gram matrices A⊤AA^\top A and B⊤BB^\top B raises it to 0.980. This equals the full O(r)\mathrm{O}(r)-invariant model (0.977).

AuthorsOpenReview ↗

Response to Reviewer xYqt (2/3)

  • Fix (a) on the real zoo (Sec 7.4, Table 1 and App. D, Table 7). We add gauge statistics to the input of the GLr\mathrm{GL}_r-invariant model. This raises noise-robustness R2R^2 from 0.27 to 0.51, which is above the unconstrained model (0.41). It also raises gradient-sparsity R2R^2 from 0.09 to 0.41.

Weakness 2 + Question 3

ANSWER: Here's our new sets of experiments:

Real checkpoints (W2). We built a zoo of 671 LoRA adapters (2,013 checkpoints) on RoBERTa-base (rank 8, 24 modules, d=768d = 768) on six GLUE tasks. We used standard training only (B=0B = 0 at init, AdamW or SGD) and no synthetic reparameterization. We then trained one backbone and changed only the imposed symmetry (Sec 7.4, Table 1 and App. D).

  • Function-level targets. Here the GLr\mathrm{GL}_r-invariant model beats the unconstrained one by a wide margin (R2R^2 0.39 vs −0.01 on validation loss), as the theory predicts.
  • Noise robustness. Here the order flips (0.27 vs 0.41). Adding gauge statistics as an extra input recovers the gap (0.51).
  • Nested features. With nested invariant features and no deep network (5 splits), the information that GLr\mathrm{GL}_r-invariance throws away is worth +0.21 to +0.41 R2R^2 on noise robustness. It is worth +0.14 to +0.24 on gradient sparsity, and nothing on function-level targets (App. D, Table 6).
  • Exact structure. The symmetry structure holds exactly on these real checkpoints. The ceilings are 1.000 wherever the theory says so (90 checkpoints, App. D, Table 5).

Symmetry is the only thing that changes (Q3). We designed this experiment so that the symmetry is the only variable.

  • Everything else is the same. All models share the same backbone, optimizer, early stopping, data split (by run) and input projection.
  • Symmetry comes only from the input. We canonicalize the factors of each module before the network, and then apply the same fixed random projection. For GLr\mathrm{GL}_r we use the balanced factorization of ΔW\Delta W. Each input is exactly invariant, and we check this numerically. The deep O(r)\mathrm{O}(r) model with an SVD frame learns poorly, which is a known issue with discontinuous canonicalizations. So for O(r)\mathrm{O}(r) and SPr\mathrm{SP}_r we rely on the next control.
  • A control with no deep network. We use nested hand-made feature sets GLr⊂O(r)⊂SPr\mathrm{GL}_r \subset \mathrm{O}(r) \subset \mathrm{SP}_r with ridge regression and gradient-boosted trees (App. D, Table 6). Since the sets are nested, any gain can only come from the extra non-invariant information. The extra information gives no gain on function-level targets (within ±0.02\pm 0.02). It gives +0.21±0.03+0.21 \pm 0.03 (trees) and +0.41±0.07+0.41 \pm 0.07 (ridge) on noise robustness, and +0.14±0.02+0.14 \pm 0.02 and +0.24±0.09+0.24 \pm 0.09 on gradient sparsity. On the SPr\mathrm{SP}_r-only ratio ∥∇L∥1/∥∇L∥2\|\nabla L\|_1 / \|\nabla L\|_2, SPr\mathrm{SP}_r features add a further +0.025±0.009+0.025 \pm 0.009 over O(r)\mathrm{O}(r).
  • The same picture across models. Trees, ridge and the deep models agree, over 5 splits and 3 seeds. The absolute R2R^2 values are moderate, because the targets span six tasks and two optimizers. The evidence is in the differences.

AuthorsOpenReview ↗

Response to Reviewer xYqt (3/3)

Question 2

ANSWER: Yes. This is also the tool behind step 2 of the procedure above. The Monte Carlo quantity

ρH=Ew Varg∼Haar T(gw)Var T\rho_H = \frac{\mathbb{E}_w \, \mathrm{Var}_{g \sim \mathrm{Haar}} \, T(gw)}{\mathrm{Var}\, T}

can be computed on any zoo. It only needs the target at reparameterized checkpoints. When the zoo is HH-invariant, it equals the cost in Thm 3.3. When the zoo is not invariant, it is still exact for a closely related zoo. This is the zoo where the same checkpoints come in random gauges, as in a hub that mixes many training pipelines (new Prop. B.1). For the zoo at hand it is then a diagnostic and not a bound. The true cost can be smaller or larger. We report both cases below.

  • On real checkpoints (App. D, Table 5). We computed the ceilings on 90 real checkpoints, before training any meta-model.
    • For rotations, 1−ρO=1.0001 - \rho_{\mathrm{O}} = 1.000 for gradient norm, Hessian trace and noise robustness, since they are exactly invariant. For quantization robustness it is only 0.38 (int4) and 0.54 (int3).
    • For signed permutations, 1−ρSP=1.0001 - \rho_{\mathrm{SP}} = 1.000 for every target.
    • For GLr\mathrm{GL}_r, the ceiling ranges from 0.995 (∥∇L∥1\|\nabla L\|_1) down to 0.74 (noise robustness) for the targets other than quantization.
    • The ceilings rank the targets well. The largest measured gains come on the targets with the lowest ceilings, and there is no gain where the ceiling is close to 1 (Spearman 0.8 over five targets).
  • The estimate predicts the cost (S1, App. E, Fig. 5a). In controlled zoos we increase the gauge spread step by step. The cost ρH\rho_H computed before training matches the measured extra error of the invariant model almost exactly. For sensitivity it is 0.708 predicted vs 0.709 measured at one setting, and 0.930 vs 0.914 at another.
  • When the zoo is not invariant (S5, App. E, Table 10). We change how concentrated the stored frames are. When the frames are aligned, the invariant model can use this and does better than the estimate (R2R^2 0.493 vs an estimated ceiling of 0.258). So on such zoos the estimate is a diagnostic and not a bound, as Prop. B.1 says.
  • A check with no assumptions. We also report the measured gap between an invariant model and an unconstrained model with the same backbone. In S5 this gap is largest when the frames are partly aligned (0.219). On the real zoo the unconstrained model is 0.14 R2R^2 better on noise robustness, while the invariant model is 0.40 better on validation loss (App. D, Table 7).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to reviewers

We thank all reviewers for their careful reading of our paper and their helpful comments on it. To answer the reviewers concerns, we ran many new experiments (e.g., a real zoo of ordinarily trained LoRA adapters). we also clarified some points in the main text according to reviewers suggestions. The main message of the paper remains the same, but now it has much stronger empirical support.

Indeed, every concern now has a clear answer, supported by new empirical evidence. We give these answers in the response to each reviewer below. All changes are in the revised manuscript, marked in red. The revised version includes a more comprehensive set of new experiments compared to the below subset that we show as a message to reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.