acceptodds
Under review as a conference paper at ICLR 2027

Same Function, Different Gradients: Evaluating Gradient Reliability under Reparameterization

Abstract

Gradients are no longer used only to train models: Gradient magnitudes are now measurement instruments, used to rank attention heads for pruning, to decide per-example clipping in private training, to allocate adapter rank budgets, and to debug models. Across these workflows, larger gradient magnitudes are taken to mean greater importance. Even for a fixed input–output function, these gradients are computed in one of many possible parameter coordinate systems. For example, attention scores depend on the query and key projections only through the paired product . Scaling a head's query projection by and the corresponding key projection by therefore leaves every output logit unchanged. This change in parameter coordinates rescales the corresponding query and key gradient norms by and , respectively. To test whether gradient-based decisions remain unchanged under these coordinate changes, we introduce Gradient Reliability under Reparameterization (GRR). GRR rewrites a trained model in new parameter coordinates that compute exactly the same function, verifies that predictions are unchanged, and then tests whether the decisions derived from the gradients are unchanged too. Our audits span GLUE classifiers, a ViT on CIFAR-10, GPT-2-medium and GPT-2-large, a 1.1B-parameter Llama-style model on WikiText-2, and six trained LoRA adapters. Empirically, predictions are preserved throughout, but both the gradients and the gradient-based decisions change. Across 66 GLUE attention audits, the median top-3 overlap of raw head-gradient rankings is only , and rankings and pruning masks change even under the mildest rescalings we test. In the largest of 14 global-pruning audits, the shifted ranking selects head masks that differ in four of 22 pruned heads and reach and validation accuracy on one ELECTRA SST-2 model. We prove that same-function reparameterizations can realize any component ranking and make gradient scores arbitrarily large, so these gradient-based measurements are not properties of the learned function at all. Beyond diagnosis, we propose symmetry-aware remedies: Function-space controls and canonical balancing restore identical decisions in every audit we run, and a quotient-space score is provably invariant under any invertible change of basis in the low-rank factors. Gradient-based evidence must therefore be verified to remain unchanged under same-function reparameterization, or be independent of coordinate choice by definition; otherwise, the evidence cannot be trusted.

Primary area:
interpretability and explainable AI
Keywords:
gradient reliability, reparameterization invariance, neural network symmetries, gradient-based diagnostics, Transformer pruning, LoRA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_djAgOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper recognizes a scale-representation issue in using weight gradients to make decisions about which parts of the model are important and which ones are not. The idea is that if a product ABAB is preserved by scaling (cA×c−1BcA \times c^{-1}B), then the gradients change without the input-output function changing. If we just measure the gradients at ABAB then we lose reproducibility. We should seek a scoring system that is invariant to such scaling transformations. The paper recommends some alternatives and performs experiments to verify their fidelity.

Critical strengths

The paper recognizes an interesting perspective on how many interventions are made to machine learning models. Gradient-based decision making is common, yet is performed under a moving ground, which threatens its reproducibility. If the measurement is not invariant to transformations that preserve the function, how can we trust it? The paper does a good job building the argument and is quite convincing as to its validity.

A variety of balancing methods are proposed. From simple rescaling to a two-sided projection mechanism. These are interesting techniques that solidify the coordinate system in which gradients are measured and ensure reproducible decision-making.

Experiments are done in a sound and coherent way.

Critical weaknesses
  1. The writing is a bit dense. Most of the messages are coming across fine, but I would reduce the verbosity in certain parts so that the arguments are simplified.
  2. I am a bit uncertain about the correct way to think about this problem. I agree with the premise: a balanced model is the correct starting point if we want gradient-based decision making to be reproducible. But one thing that concerns me is whether balancing changes the gradients in ways that can negatively affect our decisions. The model was not trained with dynamics that these new gradients outline, so should we be making decisions based on them?
  3. It is not clear to me which scoring method to choose from the experiments. I think having an evaluation metric of how good a scoring method is would be nice. That metric now is just success / failure in a task. But that metric changes for each model / task, so it is hard to know when to try what.
  4. I would include some of the figures from the appendix to the main section. They helped me understand and digest the results.
Questions for authors

N/A

4 comments

Thank you for the questions about balancing and score selection. For head pruning, we recommend mask gradients (MG) for fast scoring and function-space ablation (FS) for directly measuring complete head removal. The added controlled experiments test the effect of balancing, compare the scores under matched conditions, and support a practical guide to score selection and cost.

W2. Does balancing disconnect the measurement from the model that was trained?

Balancing preserves the trained function; pruning asks how that function responds when a head is weakened or removed. Canonical balancing (CB) puts paired weights on a common scale before measuring their gradients. Changing units changes the gradient entries, while expressing the same perturbation in those units preserves its first-order loss effect.

Multiply head hh's output by mhm_h: 1 keeps the head, and 0 removes it. MG measures how sensitive the loss is to slightly reducing mhm_h from 1. FS sets mhm_h to 0 and measures how much the output logits change. Our transformations preserve the function at every value of mhm_h, so these scores measure the same operation on the trained checkpoint before and after balancing.

We verified this in 1,400 paired loss/derivative comparisons across seven checkpoints, five suppression levels, and scalar, balanced, orthogonal, and general-basis coordinates. A separate 128-example training subset evaluates the curves. Predictions agree throughout; maximum mean-loss discrepancy is 5.55×10−75.55\times10^{-7} nats/example. The agreement holds from keeping the head at full strength to removing it completely.

We also tested the effect on actual decisions. CB preserves the raw checkpoint mask in 15/16 original encoder/ViT settings; the remaining case swaps one head (0.34 pp accuracy change). In the three-draw, layer-normalized study below, CB-minus-raw accuracy is −0.03 ± 0.06 pp, averaged over eight checkpoints. Balancing therefore closely tracks the checkpoint criterion while making its scalar units reproducible.

ELECTRA shows how the audit leads to an effective correction. Raw checkpoint gradients and CB choose the same mask, which removes all final-layer heads and yields 50.92%. With the checkpoint, calibration data, and budget fixed, changing only the per-layer score normalization raises CB to 95.18%. MG and FS reach 94.72% and 94.95% under the original global rule. This controlled ablation isolates the contribution of layer scale; W3(b) then gives every method the same normalization to isolate the scoring rule.

W3(a). A common metric for evaluating a score.

We evaluate a score with three questions:

  1. Does it select the same heads across equivalent coordinates? Mask Jaccard measures agreement between the sets of removed heads; 1 means identical selections.
  2. Does it predict the effect of deleting a head? Held-out KL fidelity checks whether higher scores identify heads whose removal changes the output distribution more.
  3. Does the selected pruning preserve accuracy? Full-validation accuracy measures performance after the selected heads are removed together.

The fidelity measure has the same interpretation across tasks: rank heads by their scores, delete each head on held-out inputs, and compare the score ranking with the ranking of the resulting KL changes:

q_h &= \mathbb{E}_{x\in D_{\mathrm{heldout}}} D_{\mathrm{KL}}\\!\left(p_{\theta}(\cdot\mid x)\\,\Vert\\,p_{\theta\setminus h}(\cdot\mid x)\right), \\\\ \rho_{\mathrm{KL}} &= \operatorname{Spearman}_{h}(s_h,q_h). \end{aligned}$$ Here $s_h$ is the head score, $q_h$ is the measured change in output distribution, and higher $\rho_{\mathrm{KL}}$ means better agreement between their rankings. The reference uses separate inputs and KL divergence; FS uses calibration inputs and logit changes. Fidelity tests the score's predictions about individual heads; accuracy tests the complete pruning decision. The global 15% study uses seven GLUE checkpoints and 128 training examples per calibration draw (seeds 7, 11, 13), with no recovery. A separate 128-example reference set measures fidelity and never selects masks. We first average Q–K/V–O within checkpoint, then weight checkpoints equally. ± denotes sample standard deviation across the three calibration-draw averages, for fixed checkpoints and evaluation splits; the same convention applies to the macro table below. PN denotes path normalization.
ScoreMinimum scalar-gauge mask JaccardHeld-out KL fidelity ↑Accuracy drop from dense (pp) ↓
Raw, checkpoint coordinate0.1890.623 ± 0.0285.87 ± 2.39
Raw, rescaled coordinate0.1890.504 ± 0.0280.45 ± 0.01
CB1.0000.620 ± 0.0285.73 ± 2.09
PN1.0000.634 ± 0.0283.35 ± 1.30
QS1.0000.583 ± 0.0296.97 ± 1.31
MG1.0000.712 ± 0.0300.27 ± 0.08
FS1.0000.724 ± 0.0130.20 ± 0.12

MG and FS perform strongly on all three criteria: identical selections across the tested scalar coordinates, the highest mean held-out fidelity, and the smallest mean accuracy drops. FS retains higher mean accuracy than raw scoring in either tested coordinate, in all seven checkpoint means.

W3(b). Controlled ablations that isolate the scoring rule.

We added controlled experiments to separate the effects of the scoring rule, layer scale, and selection policy. To isolate the scoring rule, every method uses the same checkpoint, calibration examples, deletion budget, and per-layer L2 normalization, following Michel et al. (2019; https://arxiv.org/abs/1905.10650). We divide each layer's head scores by their L2 norm before global ranking. This separate study uses all eight encoder/vision checkpoints, with 128 GLUE training examples or 256 CIFAR-10 training images per draw (seeds 7, 11, 13), full validation/test evaluation, and no recovery.

Each entry averages Q–K/V–O within a draw, then reports mean ± sample standard deviation over the three draws. Dense accuracy is fixed. Random uses ten uniform masks and reports their mean ± sample standard deviation; the same masks are shared across calibration draws.

Held-out accuracy (%) after 15% head removal.

ModelDenseRawCBPNQSMGFSRandom
DistilBERT SST-291.0690.60 ± 0.1790.62 ± 0.1490.52 ± 0.1890.58 ± 0.1890.56 ± 0.5390.98 ± 0.2990.45 ± 0.26
BERT SST-292.4392.13 ± 0.0992.14 ± 0.0692.03 ± 0.2692.18 ± 0.1892.32 ± 0.1192.24 ± 0.0791.63 ± 0.36
RoBERTa SST-294.0494.06 ± 0.1293.98 ± 0.0093.86 ± 0.3293.98 ± 0.2093.96 ± 0.1894.15 ± 0.2093.50 ± 0.56
ELECTRA SST-295.1894.61 ± 0.2194.71 ± 0.1294.76 ± 0.2494.97 ± 0.1794.69 ± 0.1395.26 ± 0.1894.62 ± 0.43
RoBERTa MNLI88.1087.33 ± 0.1987.30 ± 0.1187.07 ± 0.2187.13 ± 0.3887.63 ± 0.0687.48 ± 0.0985.46 ± 1.76
RoBERTa QNLI92.6892.15 ± 0.1892.08 ± 0.3091.95 ± 0.1891.95 ± 0.4192.51 ± 0.0292.48 ± 0.0889.52 ± 1.85
RoBERTa MRPC91.1890.69 ± 0.4490.48 ± 0.4389.34 ± 0.3289.87 ± 0.1990.69 ± 0.2589.30 ± 0.5184.90 ± 5.94
ViT CIFAR-1098.5279.17 ± 0.7079.17 ± 0.7087.31 ± 5.1479.00 ± 0.5997.81 ± 0.1097.56 ± 0.1194.05 ± 7.25

The benefit of measuring head suppression persists with per-layer score normalization held fixed. On ViT, accuracy is 97.81 ± 0.10% for MG and 97.56 ± 0.11% for FS, versus 79.17 ± 0.70% for raw scoring. The paired MG-minus-raw gain is 18.64 ± 0.74 pp across the same three draws. MG and FS also exceed the random-control mean on every checkpoint.

The budget comparison uses the same normalization and averages the seven GLUE checkpoints equally within each draw. The final two columns compute method-minus-raw differences within the same draw, using unrounded accuracies, before calculating their mean and standard deviation.

GLUE mean accuracy (%), with paired gains in percentage points.

Heads removedRawCBMGFSMG − Raw (pp)FS − Raw (pp)
10%91.74 ± 0.1491.78 ± 0.1791.83 ± 0.1591.82 ± 0.02+0.10 ± 0.13+0.08 ± 0.12
15%91.65 ± 0.0791.62 ± 0.0691.76 ± 0.0991.70 ± 0.11+0.11 ± 0.05+0.05 ± 0.08
30%90.48 ± 0.4190.54 ± 0.3791.02 ± 0.1191.28 ± 0.27+0.54 ± 0.32+0.80 ± 0.64

MG and FS achieve higher mean GLUE accuracy than raw scoring at all three budgets. The gains increase at 30% removal, where FS improves all seven checkpoint means, with positive macro gains in every calibration draw (0.30, 0.59, and 1.52 pp).

A further controlled experiment varies the selection policy while keeping the underlying scores fixed. With per-layer score normalization fixed, changing coordinates still changes raw masks in 42/42 comparisons. A policy that retains at least one head per layer also leaves all 42/42 raw masks changed across coordinates and allows an 8.58 pp accuracy gap. MG and FS preserve every mask under both policies, removing the coordinate dependence that remains after these safeguards.

W3(c). Which score to choose, and its cost.

For head pruning, start with MG. Use FS when you want to measure complete deletion directly. Use QS when your question concerns loss sensitivity to small changes in the effective matrix or adapter.

Your questionRecommended scoreMeasurement and cost
How sensitive is the loss to slightly weakening a head?MGMean absolute per-example derivative with respect to the head-output multiplier at 1. One forward/backward pass per calibration minibatch scores all heads.
What changes if I remove a whole head?FSOutput-logit change after deletion. One dense forward evaluation plus one per head.
How sensitive is the loss to a small change in the effective matrix or adapter?QSLargest local loss response to a unit product change. Factor gradients and two small Gram-system solves.
How can I compare parameter gradients in consistent scalar units?CB or PNCB equalizes paired weight norms; PN cancels reciprocal scales in the score. No per-head backward passes.

The guarantees match these uses: MG/FS are invariant under transformations preserving the same head intervention; QS covers admissible invertible factor-basis changes under Theorem 2; CB/PN cover reciprocal scalar changes. QS extends the guide to general factor-basis changes: in the six-adapter study reported in our response to jjmy, Q3, it preserves the selected modules and post-pruning accuracy across all 204 equivalent factorizations.

On one A100 40GB, scoring 128 examples takes 0.11–0.39 seconds for MG and 2.73–19.39 seconds for FS across the seven GLUE checkpoints (median of three repeats after warm-up; loading/tokenization excluded). Batched MG preserves per-example absolute values and reproduces serial-gradient masks. These costs make the choice concrete: MG scores all heads at low cost; FS directly measures complete head removal through additional forward evaluations.

W1 / W4. Simpler writing and figures in the main text.

We have shortened Secs. 2–3, introduced the framework with rewrite → verify outputs → compare decisions, and explained each remedy's purpose before its formula. Submitted Figure 3 (summary diagnostics) and Figure 6 (ELECTRA mask substitutions) are now Figures 2 and 3 in the main text. The absolute-performance and score-selection tables appear beside the claims they support, with full results in Appendix D.18–D.21.

Thank you for your responses! The rebuttal clears up my concerns on the scoring method and I think including such a discussion in the main paper would help a lot. I do appreciate the discussion on the effect of balancing on the potential disconnect from what the model trained on. I am not fully convinced that balancing is a completely neural operation, but the initial experiments do seem favorable. I will maintain my slightly positive score due to the above, as I do further find the writing and presentation as requiring substantial simplification and improvement towards publication.

Reviewer_jjmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work identifies gradient sensitivities to functional symmetries of neural network parameters as a confounder in various gradient-based algorithms for pruning, fine-tuning, and debugging. The paper presents a thorough, coherent set of experiments demonstrating that functionally equivalent networks can have varied statistical performance with the same algorithm when the algorithm relies solely on gradient information. The paper also theoretically shows that, in a simplified setting, functionally equivalent networks can exhibit arbitrarily ordered gradient magnitudes.

Critical strengths

S1. Strong Empirical Evidence

The paper conducts a thorough and sound empirical study of its claims. The experimental methodology is rigorous, and the technical content is, to the best of my knowledge, correct. The results presented in this work may be of interest to a broad range of algorithm designers.

Critical weaknesses

W1. Poor Significance

Despite the thorough empirical validation, the stated result, that algorithms that run on functionally equivalent networks in an orbit exhibit different properties, is not surprising. Analyzing the symmetries of neural parameterizations is well studied, as noted in the related work section of this paper (Appendix B). The paper provides no insight into why these inconsistencies occur, nor does it offer rigorous remedies for the problems raised in this work.

Questions for authors

Clarifications

  1. Will the authors release a reproducible codebase to enable replication of the results in their work?
  2. Can the authors demonstrate the same observed phenomenon under different symmetries, such as rotations, rather than scaling? Algorithms like SliceGPT [1] already take these symmetries into account during pruning and avoid the use of gradients.
  3. Can the authors clarify Section 3.4? What are the motivations for the gauge-invariant product-manifold score? What is its significance to the rest of this work?

Nitpicks not affecting the overall assessment

  1. Excessive whitespace on page 3.
  2. S(c) is undefined in line 239 and is only defined later in Theorem 1, harming the readability of the work.

Score Justification

Given the short review period for this ICLR, I may not have fully understood a few nuances of this work, and my review remains constrained by this ICLR's review criteria. I am eager to engage with the authors in the rebuttal period to improve their manuscript and revise my score.

[1] SliceGPT: Compress Large Language Models by Deleting Rows and Columns, Ashkboos et. al., ICLR 2024

4 comments

Thank you for the questions about significance, mechanism, and remedies. GRR asks whether rewriting the same trained model changes which components a pruning rule removes. We prove which selections are possible, isolate why a changed selection can cause a large accuracy loss, and give guarantees for measurements that remain unchanged.

W1(a). What GRR adds to existing symmetry and invariance results.

We agree that coordinate dependence is established, including for sharpness (Dinh et al., 2017; https://proceedings.mlr.press/v70/dinh17b.html) and geometric measurements (Kristiadi et al., 2023; https://proceedings.neurips.cc/paper_files/paper/2023/hash/395371f778ebd4854b88521100af30ad-Abstract-Conference.html). GRR characterizes the resulting component choices at a fixed trained model.

Choose any kk heads: the raw-gradient norm rule can be made to delete exactly those heads. To make them the bottom-kk, keep their scales fixed and raise all other head scores above theirs by reciprocal rescaling. Theorem 1 proves that such scales exist and, more generally, constructs every strict ranking. The model's function stays fixed, and deleting a given head still has the same effect on its output. The assumptions are independent reciprocal rescalings, unrestricted positive scales, and nonzero gradients for both factors of each component. Standard attention heads satisfy the symmetry, including their biases; grouped-query attention permits independent shared-group choices. Figure 4 in the revised manuscript (submitted Figure 2) executes the construction on trained DistilBERT: all 12 head ranks reverse with zero prediction flips.

We also determine when deleting a component still means the same operation. Mixing LoRA's internal basis can change what “delete component 1” removes, even though the full update stays fixed. Appendix C.5 proves that all rank components keep their identities, up to relabeling, exactly under diagonal rescaling and permutation. Under general mixing, the norm of one deleted component can take every value from the update's smallest nonzero singular value to infinity (full-rank factors, rank at least two). This separates a changed importance score from a changed deletion target, and motivates whole-module tests under general basis changes.

These results sharpen the comparison with the closest prior work:

Prior resultWhat GRR adds
Sundararajan et al. (2017; https://proceedings.mlr.press/v70/sundararajan17a.html): implementation invariance for input attribution.For internal deletions, an exact LoRA condition for the component being removed to remain the same after a basis change.
Gonon et al. (2025; https://proceedings.mlr.press/v267/gonon25a.html) and IPPRO (https://arxiv.org/abs/2507.14171): pruning criteria designed to remove scale dependence.The complete set of attainable raw-gradient selections, and tests showing that passing a scalar check can still leave a criterion vulnerable to rotations.
BayesLoRA / LRVD (https://arxiv.org/abs/2506.22809): variational training selects a preferred rank basis for relevance estimation.A condition for preserving existing component deletions and their exact range of possible effects on the update matrix, at a fixed trained adapter.
LoRA-RITE (https://arxiv.org/abs/2410.20625) and PRISM (https://arxiv.org/abs/2606.00944): invariant LoRA optimization and private training.An audit of component selections at fixed checkpoints, plus the exact boundary between raw-gradient clipping decisions that coordinates can and cannot reverse (Appendix C.7).

The audit holds the function, data, candidate deletions, and budget fixed: rewrite → verify outputs → compare selections → evaluate the selected deletions. The controls below identify how the scoring rule changes accuracy. We have added these direct comparisons with prior work to Appendix B.

W1(b). Why the scores change, and why one changed head can matter.

A raw gradient norm measures loss sensitivity per unit change in a stored parameter. Changing the parameter units changes the score. For example, doubling BB and halving AA leaves BABA unchanged, but doubles the AA-factor gradient and halves the BB-factor gradient in the new coordinates. The combined norm of the two gradients after (B′,A′)=(cB,c−1A)(B',A')=(cB,c^{-1}A) is

S(c):=(c2∥∇AL∥F2+c−2∥∇BL∥F2)1/2.S(c) := \left(c^2\lVert\nabla_A\mathcal{L}\rVert_F^2 + c^{-2}\lVert\nabla_B\mathcal{L}\rVert_F^2\right)^{1/2}.

With both factor gradients nonzero, each head's score can be made arbitrarily large. Independent scales let us place the scores in any chosen order. A small change near the pruning cutoff can then exchange two heads whose removal has very different effects.

We tested this mechanism by changing just one head in ELECTRA's 22-head deletion list. We redistributed the checkpoint's existing paired-weight imbalances among heads within each layer, preserving their distribution. The first Q–K permutation to change the list requires at most 7.4% reciprocal branch rescaling. We selected it by the mask change, without using validation accuracy. All 872 predictions before pruning remain unchanged.

Holding the other removals fixed, we obtain:

Controlled interventionHeads removedFinal-layer heads retainedAccuracy after pruning (%)
Apply the original deletion list22050.92
Restore one final-layer head21194.84
Delete the replacement head, returning to the same budget22194.72

The original list removes every final-layer head; the changed list keeps one. Restoring that head recovers performance, and deleting its replacement preserves the recovery at the same budget. Each fixed deletion list gives identical predictions in both coordinates, isolating the changed selection as the source of the gap. Across all 16 encoder/vision settings, a separate cutoff audit changes the bottom-15% list with a median reciprocal rescaling of 4.4%, verified by recomputed gradients and zero prediction flips before pruning.

W1(c). Constructive guarantees for the remedies.

Each remedy makes the measurement independent of a specified change of coordinates. The choice depends on what is being measured:

RemedyHow it removes coordinate dependenceGuarantee
Canonical balancing (CB)Equalize the paired weight norms before computing gradients.Reciprocal scalar rescaling; Appendix C.6.
Path normalization (PN)Combine weight and gradient norms so their scale factors cancel.Reciprocal scalar rescaling; Appendix C.6.
Quotient score (QS)Measure sensitivity to changes in the shared product BABA.Every admissible invertible factor-basis change, under Theorem 2's full-rank and product-loss assumptions; Corollary 1.
Mask gradient (MG)Measure loss sensitivity to slightly weakening a head's output.Transformations preserving the same head suppression at every strength.
Function-space ablation (FS)Measure the output change caused by deleting a whole head.Transformations preserving the same whole-head deletion.

For MG and FS, the proof is direct. Let mhm_h multiply head hh's output: 1 keeps it, 0 deletes it. Our admissible within-head transformations preserve the model at every suppression level:

fT(θ)(x;m)=fθ(x;m)for every m.f_{T(\theta)}(x;m) = f_{\theta}(x;m) \qquad \text{for every } m.

The loss derivative with respect to mhm_h is therefore unchanged (MG), as is the output difference between keeping and deleting the head (FS). Michel et al.'s established mask-gradient criterion (https://arxiv.org/abs/1905.10650) is a positive control that passes GRR. These guarantees establish reproducible measurements. We then evaluate deletion fidelity and retained accuracy to select a score for pruning (response to djAg). Q3 gives the product-sensitivity interpretation and adapter results for QS.

Q2. Rotations and the relationship to SliceGPT.

Yes: rotations change coordinate-Taylor selections. We hold each GLUE checkpoint, 128 training-calibration examples, candidate head deletions, and 15% budget fixed, varying only the within-head basis. Proper orthogonal rotations and general bases with singular values in [0.5, 2] give 112 comparisons: seven checkpoints × two factor pairs × two transformations × four basis draws. Calibration uses seed 17; basis seeds are 7, 11, 13, and 17. Dense predictions never flip; maximum logit drift is 7.13×10−57.13\times10^{-5}.

Changed pruning masks / comparisons. Counts measure sensitivity to coordinate changes.

TransformationRaw normCoordinate-TaylorPNQSMGFS
Orthogonal rotation0/5623/560/560/560/560/56
Non-diagonal basis, condition number 421/5628/5629/560/560/560/56

Rotations preserve Frobenius norms, so raw norms pass that control. Coordinate-Taylor adds absolute coordinate contributions, ∑i∣θi∂iL∣\sum_i \lvert\theta_i\partial_i\mathcal{L}\rvert. Rotation redistributes those contributions and changes their absolute sum. This explains how a score can pass scalar rescaling yet fail rotation. Non-orthogonal changes also alter raw norms and PN.

Two directly replicated examples show the consequences (basis seed 17 for ELECTRA and 13 for MNLI). Gaps use unrounded accuracies.

SettingTransformation and scorePruned accuracy using original / transformed-coordinate selections (%)Gap (pp)
ELECTRA SST-2, Q–KPure rotation; coordinate-Taylor95.18 / 50.9244.27
RoBERTa MNLI, Q–KNon-diagonal basis; raw norm83.42 / 87.684.26

The ELECTRA rotation deletes the last surviving final-layer head. The 44.27-point gap repeats under all four basis draws at this fixed calibration subset. Score correlation is still 0.996 for basis seed 17: a nearly unchanged ranking can cross a consequential decision boundary. QS, MG, and independently recomputed FS each preserve all 112 masks.

SliceGPT (https://arxiv.org/abs/2401.15024) uses rotations and PCA to choose residual subspaces for slicing. GRR contributes a test for any chosen diagnostic: does it select the same functional interventions from equivalent representations? The LoRA characterization above explains why preserving the intervention is essential. Our head tests preserve whole-head deletion; the adapter test below preserves whole-module deletion.

We also tested an element-wise first-order Taylor criterion reimplemented from LLM-Pruner (https://github.com/horseee/LLM-Pruner/blob/main/LLMPruner/pruner/hf_llama_pruner.py). Eligible query-head/MLP-channel groups, data, and 30% removal budgets stay fixed across residual/value-head rotations. Scoring uses 16 WikiText-2 validation blocks; evaluation uses 64 test blocks of 128 tokens, without recovery. Canonicalization absorbs RMSNorm gains into adjacent weights and selects residual PCA and within-value-head singular bases before applying the same criterion.

Perplexity under 11 starting coordinates per model; lower is better. Ranges describe coordinate sensitivity.

ModelCheckpoint coordinateRange over starting coordinatesAfter canonicalization
TinyLlama-1.1B41.6140.46–46.5140.96
Mistral-7B28.5228.52–30.3728.85

Every canonicalized start rounds to the displayed perplexity. Head masks agree exactly; channel-mask Jaccard is 1.000 for TinyLlama and at least 0.99996 for Mistral. The gain here is stable decisions with retained performance, demonstrated up to 7B parameters. We have stated the intervention-preservation condition explicitly in Sec. 2 and the checklist (Appendix F).

Q3. What the product-manifold score measures and why it matters.

QS measures how much the loss can change when the matrix product used by the model changes slightly. An attention product or LoRA update M=BAM=BA has many factorizations (BG,G−1A)(BG,G^{-1}A). They all describe the same matrix. QS measures sensitivity in that matrix, using the same units for every factorization.

Specifically, consider all local directions that keep the product's rank fixed, and give each direction unit Frobenius norm. QS is the largest first-order loss change per unit movement among these directions. This is the meaning of the “product-manifold gradient norm.” Theorem 2 assumes full-rank factors and a differentiable loss that depends on them through MM.

The geometry is established in fixed-rank matrix optimization (https://arxiv.org/abs/1209.0430); PRISM (https://arxiv.org/abs/2606.00944) also uses this tangent-gradient norm for private LoRA clipping. In GRR, it provides a common measurement for comparing the same attention product or adapter across factorizations. Theorem 2 expresses the norm directly in the existing factor gradients, using two small r×rr\times r Gram-system solves. The formula adds both factors' contributions and subtracts their overlap, counting each product direction once.

This gives a direct selection rule. QS measures loss sensitivity to small changes in the product; MG measures loss sensitivity to slightly weakening a head; FS measures the output change from deleting a head. MG and FS target the head intervention used in pruning. QS measures local product sensitivity consistently across every admissible factorization, with the adapter results below demonstrating its value for decisions across different bases.

We tested QS on six adapters trained with the submitted recipes: rank-16 DistilBERT/SST-2 and rank-8 RoBERTa/MRPC, each with training seeds 7, 11, and 13. Each receives 34 transformations—diagonal rescaling, SVD refactorization, and general bases—for 204 equivalent factorizations, with zero full-validation prediction flips. Bottom/top-25% whole-module deletion keeps the intervention and budget fixed.

Minimum mask agreement and maximum coordinate-induced accuracy gap over SVD and non-diagonal changes. Jaccard 1 means identical selections.

Module scoreMinimum mask JaccardMaximum accuracy gap (pp)
Raw norm0.2004.70
Coordinate-Taylor0.1252.87
Per-rank canonical balancing0.3333.10
Path normalization0.2003.33
QS1.0000.00

QS selects the same modules and gives the same post-pruning accuracy from every tested factorization. Appendix C.7 further gives the exact range of raw factor-gradient norms: a minimum with no finite upper bound. A clipping threshold below that minimum always clips; at or above it, a coordinate change can switch clipping on or off. This result requires full-rank factors and factor gradients. A threshold on QS has the same outcome across factorizations because QS itself is unchanged. Sec. 3.4 now explains product sensitivity before the QS formula and gives each remedy's target, guarantee, and cost in Table 1 and the accompanying text.

Q1. Reproducible code release.

Yes. The accompanying grr_code_and_results.zip includes the audit library, experiment drivers, recorded checkpoint revisions, calibration indices and seeds, masks, per-example results, and table/validation scripts, including the additional studies. The CPU validator reconstructs the added numerical tables and 2,332 accuracies from saved per-example predictions. Appendix C.2 provides an executable attention audit; Appendix E gives the original recipes and additional artifact instructions.

Presentation

We have removed the page-3 whitespace, defined S(c)S(c) before its first use as above, and shortened Secs. 2–3 around the audit, attainable-decision theorem, and remedy-selection guide.

Reviewer_ZQXjOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proves that gradient magnitudes and gradient-derived quantities, and subsequent decisions (e.g. structured pruning) based on those quantities, depend heavily on parametrization. The authors introduce GRR (Gradient Reliability under Reparametrization), an operational audit that holds the represented input-output fixed, changes the parameter coordinates using architecture symmetries (for example, rescaling keys and values of attention mechanism by a factor of c and 1/c respectively), verifies prediction numerical identity and evaluates if gradients and decisions change. The authors theoretically prove that function-preserving re-parametrization can arbitrarily change gradients-based component ranking. An extensive empirical audit, spanning GLUE classifiers, a ViT on CIFAR-10, GPT-2-medium and GPT-2-large, a 1.1B-parameter LLama-style model on WikiText-2 and six trained LoRA adapters, shows that predictions are preserved, but both the gradients and the gradients-based decision change. In the largest audit on one ELECTRA on SST-2 model, the shifted ranking-based pruning resulted in 50.92% vs 94.84% validation accuracy. The paper proposes symmetry aware remedies: measurements alternative to pure gradients that are invariant to function-preserving reparametrization. In conclusion, the gradient-based evidence needs to be verified to remain unchanged under same-function parametrization.

Critical strengths

The authors rigorously prove and empirically verify that model parametrization significantly affects gradient based measurements and pruning (in an extreme case leading to measured validation accuracy drop from 94.84% to 50.92%) and provide provable remedies.

Critical weaknesses

I couldn’t find absolute post-pruning accuracies for remedies-based pruned models, it would be interesting to see where the lie between the best and the worst re-parametrized gradient-based pruned models.

Questions for authors

Would it be possible to add accuracies measured for any remedies-based pruned models?

3 comments

Thank you for requesting the absolute comparisons. Function-space pruning (FS) retains 94.95% accuracy in the ELECTRA example, versus 95.18% before pruning. Head-mask gradients (MG) retain 94.72%. Both select exactly the same heads before and after rescaling. Below, we place each remedy within the raw-coordinate accuracy range and use controlled ablations to explain the differences between remedies.

W1 / Q1. Absolute accuracy relative to the raw-coordinate range.

We use the submitted global bottom-15% budget: 11 of 72 heads for DistilBERT and 22 of 144 for the other models. Scores use the original 128-example validation prefix for GLUE and 256 test images for ViT; accuracy uses the full validation/test split. All methods share the checkpoint, calibration examples, and deletion budget, with no recovery training. This controlled comparison isolates the scoring rule and parameter coordinates.

For each Q–K/V–O setting, the raw range covers 66 representations of the same model: the checkpoint, the submitted rescaling, 32 mild rescalings in [0.5, 2], and 32 aggressive rescalings in [0.125, 10]. Biases are transformed consistently. These ranges measure coordinate sensitivity at fixed calibration. Independent calibration draws and their standard deviations are reported below and in our response to djAg.

The five remedies work as follows:

MethodWhat it does
Canonical balancing (CB)Make the two paired weight norms equal, then measure their gradients.
Path normalization (PN)Multiply each weight norm by its gradient norm so reciprocal scale changes cancel.
Quotient scoring (QS)Measure loss sensitivity to a small change in the matrix product represented by the paired weights.
Head-mask gradients (MG)Use a head-output multiplier to measure how sensitive the loss is to slightly weakening a head.
Function-space ablation (FS)Remove one head and measure how much the model's output logits change.

MG uses the mean absolute per-example head-mask derivative of Michel et al. (2019; https://arxiv.org/abs/1905.10650). All five scores are invariant to the tested head-wise scalar transformations. MG and FS have identical Q–K and V–O entries because both operate on the same whole-head output.

Post-pruning accuracy (%); higher is better. Dense is accuracy before pruning; raw min / median / max are over the 66 representations.

ModelPairDenseRaw min / median / maxCBPNQSMGFS
DistilBERT SST-2Q–K91.0689.91 / 90.60 / 91.1790.2590.7190.2590.7190.94
DistilBERT SST-2V–O91.0689.56 / 90.71 / 91.4090.9490.9490.8390.7190.94
BERT SST-2Q–K92.4391.17 / 91.63 / 92.3291.6391.8691.7492.4392.32
BERT SST-2V–O92.4391.51 / 92.09 / 92.5591.8692.0991.9792.4392.32
RoBERTa SST-2Q–K94.0493.00 / 93.69 / 94.1593.6993.6993.2394.1594.15
RoBERTa SST-2V–O94.0492.55 / 93.35 / 94.5093.1293.4692.0994.1594.15
ELECTRA SST-2Q–K95.1850.92 / 94.15 / 95.3050.9250.9250.9294.7294.95
ELECTRA SST-2V–O95.1850.92 / 94.15 / 95.0794.2794.7250.9294.7294.95
RoBERTa MNLIQ–K88.1086.70 / 87.72 / 88.1087.7387.7387.7287.7987.88
RoBERTa MNLIV–O88.1071.03 / 86.06 / 87.9273.0173.0173.9387.7987.88
RoBERTa QNLIQ–K92.6892.22 / 92.59 / 92.7592.5792.5992.6892.7792.75
RoBERTa QNLIV–O92.6891.96 / 92.51 / 92.8692.0692.0292.0692.7792.75
RoBERTa MRPCQ–K91.1887.25 / 89.95 / 91.6790.6990.4490.2090.6990.20
RoBERTa MRPCV–O91.1868.38 / 87.50 / 90.9368.3868.3868.3890.6990.20
ViT CIFAR-10Q–K98.5240.97 / 73.12 / 95.6659.4062.1279.1598.2798.32
ViT CIFAR-10V–O98.5284.86 / 95.30 / 98.4390.6894.8290.1798.2798.32

FS reaches or exceeds the raw median in all 16 settings; in 14, it is no more than 0.5 percentage points below the best sampled raw result. These are eight model–task combinations, each audited through Q–K and V–O. For ViT Q–K, FS reaches 98.32%, above the raw maximum of 95.66%. These results establish strong absolute performance with a reproducible selection, without searching over coordinates.

The comparisons identify which measurement to use for each decision. CB and PN put parameter gradients in consistent scalar units; QS measures sensitivity to small product changes. MG and FS directly measure the effect of weakening or removing a head, and their retained accuracies support this choice for pruning. The controlled studies below and in our response to djAg separate the effects of the scoring rule, per-layer score normalization, and selection policy.

In ELECTRA Q–K, CB repeats the checkpoint's raw mask, which removes every final-layer head and yields 50.92% accuracy. Holding the checkpoint, calibration examples, and deletion budget fixed, adding only per-layer score normalization raises CB to 95.18%. This isolates the role of layer scale in that failure. MG and FS preserve useful final-layer attention and reach 94.72% and 94.95% under the original global rule.

Separate training-calibrated studies compare all scores with identical per-layer normalization and test a policy that retains at least one head per layer (details in our response to djAg). In the safeguard controls, MG and FS preserve every selection under both policies, while raw selections change in 42/42 coordinate comparisons per policy. Together with the accuracy results, these controls support a practical contribution of GRR: identify the source of an unreliable decision, then choose a score that measures the intended intervention and retains accuracy.

On GPT-2-medium/WikiText-2, we remove 58 of 384 heads without recovery, score on 32 validation blocks, and evaluate on 128 test blocks of 128 tokens. Dense perplexity is 46.75. The 97 raw coordinates comprise the checkpoint and 32 each of existing-imbalance permutations, mild rescalings, and aggressive rescalings.

Post-pruning perplexity; lower is better. Fixed calibration, with ranges over coordinates.

PairRaw min / median / maxCBPNQSMGFS
Q–K52.59 / 61.24 / 121.1262.0455.4577.8949.0552.24
V–O49.16 / 51.25 / 52.4451.3651.3251.3049.0552.24

MG gives the lowest perplexity for both pairs. The Q–K improvement is substantial (49.05 versus a raw minimum of 52.59); the V–O results are close (49.05 versus 49.16). A single coordinate-invariant MG selection therefore retains strong language-model performance across both audits.

A separate evaluation uses 128 training examples per calibration draw, with seeds 7, 11, and 13, and tests on full GLUE validation splits. At the same 15% budget, mean accuracy drops are 0.20 ± 0.12 pp for FS and 0.27 ± 0.08 pp for MG. The sample standard deviation is across three seed-level averages over seven fixed checkpoints, averaging Q–K/V–O within each checkpoint first. FS exceeds both raw-coordinate averages in all seven checkpoint means. Our response to djAg gives the full comparisons; we have added them to Sec. 4.3 and Appendix D.19.

Thank you for your satisfying explanations. We therefore keep our score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

Thank you for the constructive reviews. GRR asks whether rewriting the same trained model changes which components a gradient-based rule selects. We hold the function, candidate deletions, data, and pruning budget fixed. The paper proves which selections coordinates can induce, traces their consequences, and provides scores with explicit invariance guarantees. The added controlled experiments connect these guarantees to accuracy: they locate harmful head choices, isolate layer-scale effects, and identify useful scores for head removal. They answer the main review questions:

Review questionMain result
Do invariant remedies retain useful accuracy? (ZQXj)Function-space pruning reaches or exceeds the median raw-coordinate accuracy in all 16 encoder/vision settings. In ELECTRA, it retains 94.95% accuracy, versus 50.92% / 94.84% for the two submitted raw-gradient decisions.
What is the mechanism, and does it extend to rotations? (jjmy)A controlled head-substitution experiment explains the ELECTRA gap. Pure rotations change coordinate-Taylor masks in 23/56 comparisons; quotient, mask-gradient, and function-space masks remain identical in all 112 rotation/general-basis comparisons.
How should a practitioner choose a score? (djAg)MG measures how sensitive the loss is to slightly weakening a head; FS measures the output change from removing it. Across three calibration draws, their mean accuracy drops at 15% global pruning are 0.27 ± 0.08 and 0.20 ± 0.12 percentage points.

We have incorporated these comparisons and the simpler explanations into the revised manuscript, moved the requested figures into the main text, and included the code and raw results in the accompanying supplementary archive. Changes to the manuscript are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.