Parameter Symmetries Are Not Task Symmetries: Which Invariances Can Weight-Space Learning Afford?

Weight-space learning predicts properties of neural networks from their parameters, and predictors are commonly made invariant to all parameter symmetries, i.e., to every transformation that preserves the network function. We show that the right invariance depends on the target. Each target has its own task-symmetry group, and quotienting by anything larger incurs an irreducible error, which we compute exactly for compact groups. We then derive task-symmetry groups for broad target classes. In LoRA, a single parameterization exhibits a strict ladder: function-level targets are ${\mathrm{GL}}_r$-invariant, Euclidean sensitivity is only $O(r)$-invariant, gradient sparsity is only invariant under signed permutations, and distance from initialization has no symmetry. The two middle levels are selected by the optimizer, since gradient descent and Adam measure progress by the ${\ell }_2$ and ${\ell }_1$ gradient norms, respectively. Finally, neuron-permutation symmetry, which most weight-space architectures quotient, is lost by any target that refers to a reference model, such as drift from a pretrained base. Experiments on synthetic zoos and LoRA-fine-tuned language models confirm the predicted separations.