Same Function, Different Gradients: Evaluating Gradient Reliability under Reparameterization

Gradients are no longer used only to train models: Gradient magnitudes are now measurement instruments, used to rank attention heads for pruning, to decide per-example clipping in private training, to allocate adapter rank budgets, and to debug models. Across these workflows, larger gradient magnitudes are taken to mean greater importance. Even for a fixed input–output function, these gradients are computed in one of many possible parameter coordinate systems. For example, attention scores depend on the query and key projections only through the paired product $W_Q{W}_K^{\top }$. Scaling a head's query projection by $c$ and the corresponding key projection by $1/c$ therefore leaves every output logit unchanged. This change in parameter coordinates rescales the corresponding query and key gradient norms by $1/c$ and $c$, respectively. To test whether gradient-based decisions remain unchanged under these coordinate changes, we introduce Gradient Reliability under Reparameterization (GRR). GRR rewrites a trained model in new parameter coordinates that compute exactly the same function, verifies that predictions are unchanged, and then tests whether the decisions derived from the gradients are unchanged too. Our audits span GLUE classifiers, a ViT on CIFAR-10, GPT-2-medium and GPT-2-large, a 1.1B-parameter Llama-style model on WikiText-2, and six trained LoRA adapters. Empirically, predictions are preserved throughout, but both the gradients and the gradient-based decisions change. Across 66 GLUE attention audits, the median top-3 overlap of raw head-gradient rankings is only $1/3$, and rankings and pruning masks change even under the mildest rescalings we test. In the largest of 14 global-pruning audits, the shifted ranking selects head masks that differ in four of 22 pruned heads and reach $50.92\%$ and $94.84\%$ validation accuracy on one ELECTRA SST-2 model. We prove that same-function reparameterizations can realize any component ranking and make gradient scores arbitrarily large, so these gradient-based measurements are not properties of the learned function at all. Beyond diagnosis, we propose symmetry-aware remedies: Function-space controls and canonical balancing restore identical decisions in every audit we run, and a quotient-space score is provably invariant under any invertible change of basis in the low-rank factors. Gradient-based evidence must therefore be verified to remain unchanged under same-function reparameterization, or be independent of coordinate choice by definition; otherwise, the evidence cannot be trusted.