- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors show that function-preserving changes of coordinates can redistribute per-neuron conductance arbitrarily, subject to its fixed total. As a replacement, they propose intervening on the dominant eigenspace of the conductance matrix. This intervention is coordinate-invariant. In small models it has a larger logit effect than a same-rank spectral comparator.
Critical strengths
- I find the matrix formulation useful. Writing conductance as the diagonal of a path-integrated matrix turns completeness into a trace identity and makes the omitted cross-coordinate terms explicit. The similarity law then lets the authors apply Fillmore's theorem to characterize all attainable coordinate attributions for nonscalar . This gives a precise account of the ambiguity, rather than just an example of scores changing under a particular transformation.
- I also appreciate the care in the experimental protocol. The selection rules, comparators, and thresholds are fixed in advance, and the authors use held-out seeds and compute uncertainty over paths rather than treating gauge draws as independent evidence. Tracking eigengaps, projector norms, and conditioning makes the numerical results easier to assess. The mixed and negative results are reported too, including inconsistent directions on attack paths and the two cells that fail the isolation criterion.
Critical weaknesses
-
My main conceptual reservation is the choice of equivalence class. The analysis allows any at the hidden cut. As the authors acknowledge in Section 3.2, doing this at a nonlinear layer may require inserting an affine interface; it need not be a reparameterization within the original architecture. I accept the theorem for this enlarged class, but I am not yet convinced that it is the right class for assessing claims about the network's native neurons.
Under the usual permutation and positive-rescaling symmetries of ReLU networks, corresponding neurons retain their conductance scores. For a permutation matrix and a positive diagonal matrix ,
= \Pi\,\operatorname{diag}(C).$$ So the coordinate ambiguity established under full $\mathrm{GL}(d)$ does not arise under these transformations. I also think there are substantive reasons to study the native basis. Godfrey et al. (2022) relate neuron axes to the activation's symmetry group, and their Section 6, together with the results of Bau et al. (2017), shows that rotating post-activation representations can reduce unit interpretability. Whittington et al. (2023) give another reason to expect axis-aligned structure: single-factor selectivity can arise under nonnegativity and energy constraints. There is empirical evidence of reproducible native units across models as well (Li et al., 2016; Dravid et al., 2023). More recently, Dravid et al. (2026) report that the shared Rosetta population becomes more selective with scale even as its fraction of all neurons decreases. None of this contradicts the affine theorem. It does make native-neuron organization a reasonable object to explain, though. I would like the authors to say which uses of neuron attribution need full affine invariance, and why invariance under the architecture's own symmetries would not suffice. The same question applies to the proposed replacement: smooth invertible nonlinear recodings can preserve $F$ while generally changing the spectral scores and projector intervention. The spectral construction therefore also depends on a choice of equivalence class, and I think that choice needs more motivation.
-
I am not sure the utility experiments distinguish identifying a useful internal feature from simply undoing most of the displacement from baseline. For a straight hidden path, writing for the gradient averaged along that path gives
If , the projector onto its nonzero eigenvalue is
\qquad P\Delta z = \Delta z.$$ The intervened state $z_0+(I-P)\Delta z$ is then exactly $z_0$. In this case, a rank-one intervention removes the entire change from baseline to input. The paper notes the related rank-one case for affine downstream $G$ in Appendix A.2.3. I do not think this makes the general finite-ablation comparison circular. But it gives a concrete alternative explanation that I would want to rule out in the experiments, especially the text results. Table 16 reports dominant-to-runner-up effect ratios of roughly 195 to 1456, and the separate complement experiment gives median effects of 2.21 versus 0.004 logits. Those numbers make me wonder how much of the dominant component's effect comes from nearly resetting the representation to baseline. A useful check would be to report the residual displacement $$\frac{\lVert(I-P)\Delta z\rVert}{\lVert\Delta z\rVert}$$ in the native basis, together with the difference between the intervened output and the baseline output. I would also find the signed ratio $\operatorname{tr}(PC)/\operatorname{tr}(C)$ informative where the denominator is not close to zero. The comparisons I am missing are a full baseline reset, matched-rank native units selected by conductance, and directions selected by other attribution methods. Reporting intervention magnitudes alongside ranks would help establish how much of the advantage comes from the chosen subspace rather than the size of the perturbation.
Questions for authors
- How much of the dominant-component ablation effect comes from approximately returning to baseline? On the existing evaluation paths, could you report in the native basis and check whether the utility advantage remains when this residual is substantial? I would find comparisons with a full baseline reset and matched-rank native units selected by conductance particularly helpful here, with intervention magnitudes reported as well.
- I would like to understand the intended interpretability task more concretely. In what setting is full affine invariance necessary, beyond invariance under the original architecture's symmetries? Could you give an example where the spectral explanation tells us something useful that native-neuron conductance misses?
7 comments
Response to Question 1
Tables R1 and R2 report the requested diagnostics across all 120 vision holdout paths (layer3 on ResNet-18, ; layer2 on WRN-28-2, ; features[:14] on PlainConvNet, ), 72 deep AG News Layer-0 paths (64 tokens), and 144 shallow CLS paths. Note that because is non-normal, is an oblique projector (), so can slightly exceed ; likewise can slightly exceed via spectral cancellation. Hidden IG integrates along the straight hidden segment (max relative completeness error 0.85%), and uses summed over positions.
Table R1: Native-basis residual displacements, signed trace ratios, perturbation magnitudes, and baseline reset diagnostics
| Setting & Cut (, median rank ) | Paths (Residual ) | Native Residual Ratio [IQR] | Signed Trace Ratio [IQR] | Perturbation Norms (Dom. / RU / Native) | Full Baseline Reset | Intervened-to-Baseline Gap (Paired %) |
|---|---|---|---|---|---|---|
| Vision Pooled (5 cells) | 120 (120) | 0.999 [0.928, 1.103] | 0.722 [0.570, 0.906] | 10.16 / 9.38 / 2.48 | 10.31 logits | 1.80 logits (20.6%) |
| ResNet-18 CIFAR-10 () | 24 (24) | 0.938 [0.885, 1.053] | 0.711 [0.605, 0.955] | 10.09 / 7.94 / 2.70 | 9.32 logits | 1.71 logits (20.6%) |
| WRN-28-2 CIFAR-10 () | 24 (24) | 1.037 [0.945, 1.111] | 0.623 [0.548, 0.691] | 18.05 / 15.80 / 4.95 | 14.24 logits | 3.06 logits (22.5%) |
| PlainConvNet CIFAR-10 () | 24 (24) | 1.071 [0.935, 1.166] | 0.395 [0.207, 0.844] | 7.05 / 9.48 / 1.66 | 18.13 logits | 7.55 logits (40.0%) |
| ResNet-18 CIFAR-100 () | 24 (24) | 0.958 [0.928, 1.027] | 0.796 [0.658, 0.998] | 11.58 / 10.08 / 2.62 | 11.79 logits | 0.93 logits (8.6%) |
| ResNet-18 SVHN () | 24 (24) | 1.051 [0.947, 1.121] | 0.820 [0.709, 1.009] | 8.24 / 6.16 / 1.97 | 7.64 logits | 1.40 logits (20.4%) |
| NLP Layer-0 Cut (Deep, ) | 72 (72) | 1.024 [0.997, 1.137] | 0.910 [0.729, 1.144] | 32.11 / 45.71 / 7.05 | 2.23 logits | 0.76 logits (34.4%) |
| NLP CLS Cut (Shallow, ) | 144 (0) | 0.041 [0.022, 0.060] | 1.003 [0.991, 1.023] | 7.87 / 0.29 / 1.55 | 2.34 logits | 0.0039 logits (0.2%) |
Table R2: Target-logit effects (medians) and paired win rates vs. norm-matched, native, and attribution directions
| Setting | Dominant Spectral | vs. Runner-Up (Raw / Norm-Matched) | vs. Native Top- Cond. | Dom. at Native Norm vs. Native Top- | vs. IG Top- | vs. Top- | vs. Path-Grad Subspace | vs. Full Complement (, rank ) |
|---|---|---|---|---|---|---|---|---|
| Vision Pooled (5 cells) | 8.30 | 2.61 (90.8%) / 3.02 (93.3%) | 1.18 (97.5%) | 1.45 vs. 1.18 (61.7%) | 1.48 (95.8%) | 1.02 (97.5%) | 5.00 (87.5%) | 6.80 (70.8%) |
| ResNet-18 CIFAR-10 | 6.78 | 1.37 (87.5%) / 1.85 (83.3%) | 0.28 (95.8%) | 0.32 vs. 0.28 (66.7%) | 0.67 (91.7%) | 0.38 (95.8%) | 2.04 (87.5%) | 1.86 (83.3%) |
| WRN-28-2 CIFAR-10 | 9.54 | 3.23 (95.8%) / 4.37 (100.0%) | 1.79 (100.0%) | 1.80 vs. 1.79 (54.2%) | 1.94 (100.0%) | 1.16 (100.0%) | 4.36 (87.5%) | 9.78 (50.0%) |
| PlainConvNet CIFAR-10 | 12.63 | 4.32 (70.8%) / 4.20 (83.3%) | 2.05 (91.7%) | 1.93 vs. 2.05 (41.7%) | 3.59 (87.5%) | 2.80 (91.7%) | 6.84 (79.2%) | 13.44 (50.0%) |
| ResNet-18 CIFAR-100 | 11.25 | 3.28 (100.0%) / 3.86 (100.0%) | 1.12 (100.0%) | 1.76 vs. 1.12 (75.0%) | 1.53 (100.0%) | 1.06 (100.0%) | 7.17 (91.7%) | 7.21 (87.5%) |
| ResNet-18 SVHN | 6.08 | 1.50 (100.0%) / 2.57 (100.0%) | 0.81 (100.0%) | 1.23 vs. 0.81 (70.8%) | 1.03 (100.0%) | 0.60 (100.0%) | 4.67 (91.7%) | 4.74 (83.3%) |
| NLP Layer-0 Cut | 1.76 | 0.41 (93.1%) / 0.25 (91.7%) | 0.15 (98.6%) | 0.27 vs. 0.15 (75.0%) | 0.15 (98.6%) | 0.03 (100.0%) | 1.03 (91.7%) | 0.37 (91.7%) |
| NLP CLS Cut | 2.34 | 0.0046 (100.0%) / 0.12 (100.0%) | 0.14 (100.0%) | 0.38 vs. 0.14 (94.4%) | 0.14 (100.0%) | 0.08 (100.0%) | 2.20 (93.8%) | 0.0047 (100.0%) |
Response to Question 2
Full invariance is required when transferring subspace attributions across checkpoints aligned by a dense linear representation map (), while even in a single unmodified checkpoint () the spectral projector identifies a distributed multi-channel coalition that requires ablating 11.0–32.5 native channels to match (Table R3):
- What spectral attribution captures at : Because any dense non-axis-aligned operator in dimension naturally has high off-diagonal Frobenius mass (98.4%–99.6%), the stronger test at is how many native channels must be ablated together to match a rank-1 spectral intervention. Across 16 paths per vision model and 24 Layer-0 paths, the leading eigenvector spans channels, and cumulative native top- ablation () requires a median of 11.0–32.5 native channels among matched paths to reach the rank-1 spectral effect (with 5/16 PlainConvNet and 7/24 Layer-0 paths never matching at any ).
- Where full invariance matters in practice (Seed transfer): Procrustes-regularized stitching maps fitted on held-out development activations are dense (, 74.4%–93.7% non-peak row mass). While exact commutation across inside the stitched model follows directly from our theorem (100% top-1 flip vs. 0.00 spectral discrepancy), the practical test is transferring source explanations between the two original checkpoints by transporting both the source spectral projector () and the source native top- projector () through . Raw transfer vs. wins 95.8%–97.2% across all four models, and norm-inflated
A-upretains 66.7%–100.0% wins on ResNet-18, WRN-28-2, and AG News (50.0% on PlainConvNet); conversely, scaling down to the single-channel norm (S-down) retains 75.0% wins on ResNet-18 and 100.0% on AG News, but drops to 41.7% on WRN-28-2 and 33.3% on PlainConvNet (showing directional alignment on ResNet-18/AG News vs. multi-channel norm dominance on WRN/PlainConvNet).
Table R3: Native-basis () off-diagonal coalition analysis and cross-seed linear stitching (, Seed )
| Model & Cut | Dim | Native Off-Diagonal Frobenius Ratio | Native Leading-Eigenvector PR | Dominant vs. Native Top-1 / Top-5 / Top-10 Effect | First Matching Native (Matched Paths) | Median Non-Peak Row Mass () | Interface Flip / Jaccard / Spectral Effect Discrepancy | Transfer / (Raw; A-up; S-down Win %) |
|---|---|---|---|---|---|---|---|---|
ResNet-18 (layer3) | 256 | 99.6% | 27.4 ch. | 6.51 vs. 0.22 / 0.79 / 1.21 | 32.5 ch. (16/16 matched) | 84.0% () | 100.0% / 0.000 / 0.00 | 7.16 / 0.22 (95.8%; 91.7%; 75.0%) |
WRN-28-2 (layer2) | 64 | 98.4% | 19.9 ch. | 10.34 vs. 1.54 / 4.54 / 8.53 | 14.0 ch. (16/16 matched) | 79.3% () | 100.0% / 0.000 / 0.00 | 6.58 / 1.00 (95.8%; 66.7%; 41.7%) |
PlainConvNet (features[:14]) | 128 | 99.5% | 43.8 ch. | 12.76 vs. 1.99 / 7.31 / 11.09 | 11.0 ch. (11/16 matched) | 74.4% () | 100.0% / 0.000 / 0.00 | 9.84 / 3.16 (95.8%; 50.0%; 33.3%) |
| AG News (CLS stitch / L0 audit) | 128 | 99.5% | 45.5 ch. | 2.44 vs. 0.096 / 0.51 / 0.86 | 31.0 ch. (17/24 matched) | 93.7% () | 100.0% / 0.000 / 0.00 | 2.00 / 0.057 (97.2%; 100.0%; 100.0%) |
Thank you for your careful review and mathematical analysis. We appreciate your positive comments on the matrix formulation of conductance, the Fillmore orbit characterization, and the experimental design. Below we address each of the weaknesses and questions you raised.
Response to Critical Weakness 1
We agree that the equivalence class must match the interpretability question (concrete empirical comparison in Response to Question 2, Table R3):
Response to Critical Weakness 2
Your rank-one derivation for an uncontracted straight path (C=Δzgˉ⊤⟹PΔz=Δz when tr(C)=0) is exact, and from an output-logit perspective our data partially support your concern while clarifying where the spectral subspace adds value beyond a reset (full diagnostics in Response to Question 1, Tables R1–R2):