acceptodds
Under review as a conference paper at ICLR 2027

Attribution Evaluation Scores Depend on the Network Being Explained

Abstract

Attribution methods are often compared using faithfulness scores based on interventions and localisation scores based on human annotation. We show that both can depend strongly on the network being explained, even when the attribution method is unchanged. In fixed-width CIFAR-10 ResNets, rank-based faithfulness tracks how concentrated the measured behaviour is across channels (within-seed Spearman ). An extreme-weight-decay model with one effective channel reaches only 18% accuracy yet receives a score of 1.000, because almost all channel–image pairs are tied at zero attribution and zero ablation effect. On ImageNet, using the same attribution and the same images, faithfulness rises from 0.595 to 0.880 across the first three stages of ResNet-18 and reaches 1.000 at the final stage, where the linear readout makes the metric exact by construction; activation magnitude alone scores 0.887 there. In a separate localisation study, two publicly available diabetic-retinopathy classifiers with the same ViT-B/16 architecture differ by in lesion localisation, more than the variation across five attribution methods within either classifier; one falls below a matched random baseline under every method. Gain-over-random normalisation, reproducibility normalisation, and support restriction do not remove the concentration effect. We therefore recommend reporting the explained layer and its distance from the readout, the concentration of the measured effect, and results across more than one layer and model.

Primary area:
interpretability and explainable AI
Keywords:
feature attribution, attribution evaluation, interpretability, explainability, faithfulness metrics, saliency maps, neural network representations, model explanations

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_9EsBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission investigates how the evaluation of explanations, i.e. faithfulness of layer attribution and localisation of input attribution, changes depending on the convolutional network being explained. It considers image classification tasks such as CIFAR-10, ImageNet, and medical imaging. CIFAR-10 and ImageNet experiments show that the same attribution method can get a higher faithfulness score simply because it is explaining a simpler part of the network. A separate study with retinal images finds substantial differences in lesion localisation between two checkpoints with the same model architecture.

Critical strengths

None

Critical weaknesses

Two critical weaknesses of this paper are (1) lack of novelty and (2) poor quality.

  1. The main claim given in the paper's title, that explanation evaluation depends on the model, has been studied in depth. The four "findings" presented in the intro do not constitute a substantial contribution to machine learning research; studying ResNets on CIFAR-10 is not informative. The paper should distinguish from and relate to findings from the relevant literature (only for example):
    • Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations. KDD 2025
    • Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics. NeurIPS 2024
    • M4: A Unified XAI Benchmark for Faithfulness Evaluation of Feature Attribution Methods across Metrics, Modalities and Models. NeurIPS 2023
    • Sanity Simulations for Saliency Methods. ICML 2022
    • Do Feature Attribution Methods Correctly Attribute Features? AAAI 2022
    • Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanations. WACV 2022
    • Benchmarking saliency methods for chest X-ray interpretation. Nature Machine Intelligence, 2022
    • When Comparing to Ground Truth is Wrong: On Evaluating GNN Explanation Methods. KDD 2021
    • Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging. Radiology: Artificial Intelligence, 2021
    • Evaluating local explanation methods on ground truth. Artificial Intelligence, 2021
    • Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2020
    • Sanity Checks for Saliency Metrics. AAAI 2020
    • Fooling Neural Network Interpretations via Adversarial Model Manipulation. NeurIPS 2019
  2. This paper reads as AI-generated to the point where it is hard to parse. The "AI use statement" says only that "GPT was used only to aid or polish the writing of this manuscript," and it feels dishonest, as judged by the amount of weirdness given in its text, e.g. many unclear/uncommon words/phrases, ": ;" signs, 12! sections. Please read published ICLR papers to understand the expected standards for terminology, structure, and clarity. Only for example:
    • The second sentence says "Rank-based faithfulness" as if it were obvious what it means. It is also unclear what an attribution method is at this point.
    • In fact, there is only a single reference to the literature in the introduction.
    • What is an "interference penalty"?
    • What does "rank-based faithfulness tracks the number of channels carrying the behavior at within-seed Spearman −0.83" mean?
    • The 1st contribution reads as "The score rises toward the readout." Which score? What is readout? It also shouldn't reference Appendix E.
    • "At fixed width, concentration strongly predicts the score." What is the width? What is the concentration?
    • "Simple scalar corrections do not remove the effect." Again, what are "scalar corrections"? Which effect?
    • "We conclude with a reporting protocol in §10." What does the protocol mean, and what is it for?
    • L82: What are "the magnitudes of the individual input–output paths through channel jj"?
    • Section 7 does not specify how attribution maps are computed or which representation it attributes.
    • None of the six Tables in the paper have a caption.

Regarding the paper's soundness, no code is attached to verify its findings.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

response to reviewer 9EsB (1/2)

(1/2) We thank the reviewer for the list of related works and for pointing out the passages that needed definitions. The revision rewrites the paper: eight sections instead of twelve, definitions (§3) before any result, self-contained captions on every table and figure, and anonymised code.

Relation to the literature (new §2). All 13 works are now discussed; each entry was checked against the publisher's page.

Benchmarks (M4, NeurIPS 2023; LATEC, NeurIPS 2024; Saliency-Bench, KDD 2025) apply many metrics to many methods and models, and report that method rankings change with the model and the metric. We hold method and metric fixed and identify which property of the explained network changes the score, by how much and in which direction. For rank faithfulness this is an exact formula (Proposition 1, §4).

Tomsett et al. (AAAI 2020) test the reliability of the metrics psychometrically. We add a failure of a different kind: in a concentrated layer a rank correlation is maximised by reproducing which components have no effect, so the metric ranks an attribution that knows only which channels are active above the exact ablation effects, in every planted network once three quarters of the pairs are silent (§5.2, Table 2).

Work on ground truth for explanations (Guidotti, AIJ 2021; Zhou et al., AAAI 2022; Kim et al., ICML 2022) plants reasoning to test attribution methods. We plant concentration to test the evaluation metric, with attributions whose quality is known by construction. Faber et al. (KDD 2021) show that a model need not use planted evidence; our ground truth is the measured effect of each component on the trained network, so this cannot arise.

Work on the dependence of explanations on the trained model (Heo et al., NeurIPS 2019; Schramowski et al., NMI 2020; Watson et al., WACV 2022; Arun et al., Radiology: AI 2021; Saporta et al., NMI 2022) concerns the maps themselves: they change across seeds, under adversarial fine-tuning, and between retrained chest X-ray models. We measure the consequence for the evaluation outcome: in retinopathy lesion localisation the checkpoint accounts for as much of the score variance as the attribution method (51% vs 30% with the main maps, 30% vs 31% with exact gradients; §7).

The new contributions are therefore the exact characterisation (§4), the planted-ground-truth demonstration that the metric reverses the order of attributions of known quality (§5.2), an effect-weighted concordance that orders these pairs correctly, and a skill score that stays within 0.04 while planted concentration raises the raw score by 0.29 (§4, §5.2). CIFAR-10 is the controlled setting in which concentration can be planted exactly; layer and checkpoint effects are measured on ImageNet ResNets, ConvNeXts, ViTs and DeiTs (§6, App. E) and on ten retinopathy graders (§7).

Continued in part 2/2 below.

AuthorsOpenReview ↗

response to reviewer 9EsB (2/2)

(2/2)

Clarity. The passages the reviewer cites are now defined or rewritten. The attribution method and rank-based faithfulness are defined in §1 (first two paragraphs) and formally in §3. The readout is defined in §3 (global average pooling followed by the linear layer). For concentration and width, §3 defines the effective component count n_eff, and §3 ("Networks") defines the fixed width of 16 channels. The "within-seed Spearman −0.83" is now explained in §5.1 as the correlation across training conditions, computed separately for each seed and averaged.

For the interference penalty, §3 states the training term λ · mean_j(log M_j − log |C_j|), where M_j and C_j are the absolute and signed sums of the contributions of all paths through channel j. This term rewards paths that agree in sign. The phrase "magnitudes of the input–output paths" (L82) is rewritten in §3 ("Networks") as the sum of absolute path contributions.

"Scalar corrections" is replaced by the named procedures (subtracting a random ordering, dividing by the reproducibility of the target, and restricting to the support of the effect in App. D), and "Protocol" is replaced by five concrete recommendations tied to quantities defined in the paper (§8). The introduction now cites the evaluation literature, and the contributions refer only to main-text sections.

How the maps of §7 are computed. All maps are input-space maps for the logit of the predicted grade, on the full fundus image resized to 224 × 224: gradient magnitude, input × gradient (signed sum over colour channels), absolute input × gradient, Integrated Gradients (black baseline, 32 steps), SmoothGrad (16 samples), and Grad-CAM on the tokens entering the last attention block. The submitted Figure 4 uses gradients with attention patterns and LayerNorm statistics held at their forward values, together with two path-magnitude maps. §7 now states this convention, and App. F reports both conventions on every checkpoint. The score is average precision on the union of lesion pixels, relative to a uniformly random map on the same images.

Code and verification. The supplementary zip contains the anonymised code for every experiment, the saved experimental records, and the scripts that reproduce every number, table, and figure directly from those records. We sincerely apologise for the missing table captions and overly fragmented sections in our initial submission. All experiments, proofs, and analyses were carried out by the authors, with language models used solely for stylistic wording polish as stated.

We are glad to answer any further questions during the discussion.

I will maintain my clear reject rating. This has practically become a new paper that needs a comprehensive re-review. It remains poorly written, and its claims are not necessarily novel.

Thank you for looking at the revision. For the benefit of the discussion record and the Area Chair, we address each of the three points in your comment:

  1. Scope of the revision ("practically a new paper"). The title, research question, and all three empirical pillars from the initial submission (CIFAR-10 representation concentration in §5.1, ImageNet layer position across 14 models in §6, and IDRiD retinopathy lesion localisation in §7) remain intact with the same underlying findings. Merging twelve short sections into eight, defining all terms before results (§3), and adding captions fulfilled your own presentation requests, while every new result directly answers a specific question from the other reviewers on the existing setups: Proposition 1, the planted-channel test, and effect-weighted concordance (§4, §5.2) answer Reviewer 55nw's Q2 and Reviewer KmSv's Q3, and the additional retinopathy seeds, larger image sweeps, and no-retraining ROAR control (§5.3, §6, §7) answer Reviewer KmSv's Q1 and Q2. Addressing the reviewers' own questions on the submitted setups does not make this a different paper.

  2. Writing quality ("remains poorly written"). Every concrete clarity issue listed in your initial review (undefined terms, missing table captions, fragmented sections, and §7 map specifications) has been resolved in the revised PDF. Because the new comment states that the paper "remains poorly written" without pointing to any passage in the revised text, please let us know which specific sentence or section still reads unclearly so we can address it directly.

  3. Novelty ("claims are not necessarily novel"). Section 2 compares our work directly against all 13 papers you listed. While prior benchmarks and sensitivity studies show that attribution maps or method rankings vary across models and metrics, none derives the closed-form mechanism by which zero-effect pairs dominate rank-based faithfulness (Proposition 1), shows that the metric ranks output-agnostic or wrong-class attributions above exact ablation effects in networks of known ground truth (§5.2, Table 2, including 22 of 78 unplanted networks), or resolves this reversal with an effect-weighted concordance from the same ablations (§4). If a specific prior paper already establishes these results, we would welcome the citation so we can compare against it directly.

We appreciate the feedback that helped us strengthen and clarify the manuscript.

Reviewer_KmSvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how attribution evaluation scores depend on the properties of the network being explained, rather than solely on the quality of the attribution method. The authors show that rank-based faithfulness metrics can be influenced by the concentration of attribution effects and the position of the explained layer within a network. Through experiments on CIFAR-10 ResNets, the authors demonstrate that highly concentrated representations can achieve nearly perfect faithfulness scores despite poor predictive accuracy. Experiments on ImageNet-pretrained models further show that faithfulness scores tend to increase toward the output layer, sometimes reaching perfect correlation by construction due to a linear readout. Finally, the paper evaluates several normalisation strategies and retraining-based metrics, finding that these approaches do not fully resolve the identified issues.

Critical strengths
  1. The observation that high faithfulness scores may reflect properties of the underlying network rather than the attribution method itself is relevant to research on explainability and mechanistic interpretability.

  2. The authors investigate the phenomenon using fixed-width CIFAR-10 ResNets, ImageNet-pretrained CNNs and vision transformers, and medical image classifiers.

Critical weaknesses
  1. The localisation experiments use only two diabetic-retinopathy classifiers with the same architecture. Although the observed difference is substantial, it is difficult to determine whether the results reflect a general checkpoint-dependent phenomenon or differences in the specific models' predictive behavior, training data, or learned representations.

  2. The ROAR experiments indicate that removing most channels from the first residual block has little impact on accuracy after retraining. However, this result may reflect architectural redundancy and the ability of retraining to compensate for removed channels. It does not necessarily establish that the original attribution scores contain little useful information.

  3. All the tables are lack of captions, which makes them difficult to reference and interpret independently. Besides, the main findings of this paper appear relatively straightforward and largely follow from the properties of the evaluation metrics themselves. While the empirical observations are useful, the paper provides limited new theoretical insights beyond these relatively straightforward properties. Furthermore, the work does not propose a new evaluation method or an effective solution to address the identified limitations. The final reporting recommendations are reasonable but remain relatively general.

Questions for authors
  • Given that retraining can compensate for removed channels, how should the near-zero ROAR scores be interpreted? Do these results indicate that the attribution rankings are uninformative, or primarily that the network architecture provides substantial redundancy?
  • Some experiments, particularly the cross-model ImageNet comparison and ROAR evaluation, rely on small evaluation sets or limited random seeds. How stable are the reported correlations and conclusions under larger evaluation sets and additional seeds?
  • Could the authors clarify which findings constitute fundamentally new insights beyond these expected behaviors, and what concrete implications they have for developing more reliable attribution evaluation methods?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KmSv

(1/2) We thank the reviewer for the constructive questions. Each one led to a new experiment in the revision.

More than two retinopathy classifiers (§7, App. F). We added eight graders that we fine-tuned ourselves: five ViT-B/16 and three ResNet-50, trained from ImageNet weights on the public APTOS-2019 grading images. Within each architecture the data and recipe are identical and only the seed changes; validation kappa is 0.886–0.904 for ViT-B/16 and 0.856–0.871 for ResNet-50. We scored them with the five maps of the submitted Figure 4 on the same 81 IDRiD images, against the same random baseline, with bootstrap CIs (Table 6). App. F adds Integrated Gradients, SmoothGrad and Grad-CAM with exact gradients.

Over all ten checkpoints, the checkpoint explains 51% of the variance of log(AP / random) and the map 30%. The five ViT-B/16 seeds differ by 2.2–2.9× under four of the five maps; gradient magnitude ranges from 1.89× to 4.08× random. That is as large as the 2.71× spread across maps within public model A. Across all checkpoints the same map varies 4.0–4.8×, and public model B stays below random under all five maps (0.78–0.90×). With exact gradients, the checkpoint and the map contribute about equally (30% and 31%), and Grad-CAM varies 3.5× across the ViT seeds.

Checkpoint dependence therefore appears among seeds of one recipe, not only between models with different data or training. This answers the reviewer's question: the localisation score is a property of the trained model being explained.

Interpretation of near-zero ROAR scores (§5.3, Table 3). The revision measures the two contributions separately. On the fully trained networks (three seeds), we apply the same masks with and without retraining.

Without retraining, masking the four channels ranked highest by gradient times activation lowers accuracy from 0.850 to 0.305, against 0.631 for four random channels. Masking the top eight gives 0.165 against 0.279. The ranking therefore identifies the channels the trained network depends on.

After retraining from scratch, the network reaches 0.840 accuracy with only two of the 16 channels, whichever are removed. ROAR stays within ±0.005 at every removal fraction (4, 8, 12, 14 channels), and KAR (keep the top-k channels and retrain) also stays within ±0.005.

Near-zero ROAR on this layer thus measures how completely retraining compensates for removed channels, a property of the architecture. The no-retraining comparison on the same masks measures the information in the ranking. Recommendation (v) in §8 asks for both to be reported together.

Continued in part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer KmSv (2/2)

(2/2)

Captions. Every table and figure now has a self-contained caption.

New insights beyond expected properties of the metrics, and a concrete method (§4, §5.2). The revision turns the property into an exact, testable statement and uses it to build and validate two scores.

Proposition 1 gives the rank-faithfulness score in closed form as a function of the fraction of zero-effect pairs and of the attribution's agreement on the remaining pairs. With 120 of 128 pairs at zero, any zero-respecting attribution scores at least 0.997, whatever order it gives the other eight. An attribution that is exact on every pair of different effect but breaks the ties scores 0.420.

In the planted-ground-truth experiment (§5.2, Fig. 2), silencing a controlled number of channels in trained networks makes the metric rank gradient times activation, a wrong-class attribution, activation magnitude, and a ranking that knows only which channels are active above the exact ablation effects, in every planted network once enough pairs are silent. In 22 of 78 trained networks without planting, all with neff≤9.07n_{\mathrm{eff}} \le 9.07, it prefers an attribution that is exact only on negligible channels to one that is exact on the channels carrying 90% of the effect.

The effect-weighted concordance (§4), computed from the same ablations, weights each pair of components by the difference of their effects. It orders these pairs correctly in all 78 trained networks and in 63 of 65 planted ones, and never ranks any attribution above the exact effects (Table 2).

The skill score is taken relative to activation magnitude, an output-agnostic reference fixed before the experiments. Under planting it stays at 0.452, 0.475 and 0.485 while the raw score of gradient times activation rises from 0.670 to 0.957. On ImageNet (Table 5) it shows that Integrated Gradients at the first two ResNet-18 stages adds 0.005 and 0.024 to what activations alone provide, despite raw scores of 0.467 and 0.548.

The recommendations in §8 are now tied to these quantities. For building evaluation methods the implication is concrete: a score that compares attributions must give no weight to pairs of equal effect, as the effect-weighted concordance does, and a score compared across layers or networks must be referenced to an output-agnostic baseline, as the skill score is. A per-model normalisation such as normalised AOPC (Edin et al., ACL 2025), now discussed in §2, keeps the order of attributions within a model, whereas the reversals we report occur within single networks.

Stability under larger evaluation sets and more seeds. On CIFAR, the second sweep has 80 networks (16 conditions × 5 seeds) evaluated on 256 images with all 16 channels per image, i.e. 4,096 pairs per network instead of 128. The within-seed correlation between rank faithfulness and neffn_{\mathrm{eff}} is −0.917 ± 0.073 over five seeds (submitted −0.83 on eight images). Within the accuracy band, now 50 networks at accuracy 0.800–0.859, it is −0.821 pooled and −0.774 after partialling out accuracy (submitted −0.662 and −0.595 on 20 networks).

On ImageNet, the fourteen-model comparison was rerun with the same attribution code on 100 images per model (App. E). Family ranges are ResNet layers 0.88–0.95, ConvNeXts 0.36 and 0.45, and ViT/DeiT blocks 0.03–0.17. The 100-image scores correlate with the six-image scores at Spearman 0.969, and the cross-model correlation of neffn_{\mathrm{eff}} with the score is +0.824 (+0.820 with six images).

ROAR now has three seeds per condition, with the controls above.

We are glad to answer any further questions during the discussion.

Reviewer_55nwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how attribution evaluation metrics depend on the properties of the neural network being explained rather than exclusively on the quality of the attribution method. The authors study this phenomenon through several experiments (attribution concentration, layer and checkpoint dependence).

Critical strengths
  • The reliability of attribution evaluation remains an important open problem in explainable AI, especially across different models
  • Interesting empirical observations
Critical weaknesses
  • The paper looks more like a collection of diagnostic experiments rather than a paper contribution. The paper repeatedly demonstrates that attribution scores depend on network properties, but it does not develop a sufficiently precise hypothesis explaining when this dependence is problematic, why existing evaluation approaches fail or what new scientific understanding emerges from the observations.
  • The paper argues that attribution evaluation metrics can produce misleadingly high scores for networks with concentrated representations. However, a higher faithfulness score does not necessarily indicate a failure of the evaluation metric. If a network genuinely relies on a single channel and an attribution method correctly identifies it, then a perfect faithfulness score is justified, regardless of the network's classification accuracy. The authors demonstrate that some networks are easier to explain, but they do not establish that the evaluation metrics are incorrectly assessing attribution quality. In other words, the paper confuses the difficulty of the attribution task with the reliability of the evaluation metric.
Questions for authors
  • How exactly are attribution scores and intervention effects paired when disjoint image sets are used? Does the resulting metric measure per-input faithfulness or cross-image channel-importance stability?
  • Can the authors demonstrate that concentration causes a ranking metric to prefer an objectively worse attribution method, rather than simply making the underlying ranking problem easier?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 55nw

(1/2) We thank the reviewer for two precise questions. Both led to new results, now the core of the revision (§4 and §5.2).

A precise hypothesis of when the dependence on the network is a problem. The revision states it as an exact result (Proposition 1, §4; proof in App. A). Pool the nn component–input pairs, let mm of them have zero ablation effect, and let ρ+\rho_+ be the attribution's Spearman correlation on the q=n−mq = n - m pairs with positive effect. An attribution that assigns zero to exactly the zero-effect pairs scores

S0=3mn+ρ+(q2−1)3mn+q2−1,S_0 = \frac{3mn + \rho_+ (q^2 - 1)}{3mn + q^2 - 1},

and one that orders every pair of different effect correctly but breaks the ties among the zero-effect pairs scores Stb=(3mnq+q(q2−1))/(3mnq+q(q2−1)+m(m2−1))S_{\mathrm{tb}} = \sqrt{(3mnq + q(q^2-1)) / (3mnq + q(q^2-1) + m(m^2-1))} (Proposition 1 with ρ+=1\rho_+ = 1). For the degenerate CIFAR network (n=128n = 128, m=120m = 120), every zero-respecting attribution scores at least 0.997, even one that reverses the order of all remaining pairs, while the tie-breaking exact attribution scores 0.420.

The dependence is a problem whenever the score is used for its purpose, choosing between attributions: in a concentrated layer the score is dominated by whether an attribution reproduces which pairs have no effect, a question that can be answered without any information about the explained output. Scalar corrections that do not use an output-agnostic reference (subtracting a random ordering, dividing by the reproducibility of the target) remain associated with concentration (App. D). §4 derives two predictions, tested in §5.2.

Difficulty of the attribution task versus reliability of the metric. A planted-ground-truth experiment (§5.2, Fig. 2) separates the two. In five fully trained networks we set kk of the 16 channels of the explained layer to zero, leaving every other channel unchanged, and compare attributions of known quality: the measured ablation effects ("exact effects") and a ranking that knows only which channels are active ("activity only").

The exact effects score 1.000 before planting, 0.760 when three quarters of the pairs are silent, and 0.420 when one channel remains. Over the same range, the activity-only ranking rises from 0.003 to 0.972 and 0.999. From half of the pairs silent onward, the metric ranks gradient times activation above the exact effects in every planted network, and from three quarters onward it also ranks the wrong-class attribution, activation magnitude and the activity-only ranking above the exact effects in every planted network.

Both curves follow Proposition 1. The case the reviewer describes, a network that relies on one channel, is the end point of this experiment (k=15k = 15). There every zero-respecting attribution scores 1, including one computed for a different class, so a score of 1 no longer distinguishes a correct explanation from an incorrect one. Well before that end point, the metric already ranks attributions with less information above the exact effects.

Continued in part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer 55nw (2/2)

(2/2)

Q2 (whether concentration makes the metric prefer an objectively worse attribution). It does, also in trained networks without any planting (§5.2, Table 2). From the measured effects of each input we build two attributions. One is exact on the channels that carry 90% of the effect and random on the rest. The other is exact on the rest and randomly permutes the channels that carry the effect. The first is the better explanation.

Rank faithfulness prefers the second in 22 of 78 trained networks. These are exactly the networks with neff≤9.07n_{\mathrm{eff}} \le 9.07; every network with neff≥9.12n_{\mathrm{eff}} \ge 9.12 is ordered correctly. It prefers the second in all 65 planted networks.

An effect-weighted concordance, computed from the same ablations, weights each pair of components by the difference of their effects. It ranks this pair correctly in all 78 trained networks and in 63 of 65 planted ones. It never ranks any attribution above the exact effects (Table 2: 0/65 in every comparison with the exact effects, against 35–50/65 for rank faithfulness).

Q1 (how attribution scores and intervention effects are paired). Rank faithfulness pairs the attribution of channel jj on input xx with the effect of ablating jj on the same xx, and pools these pairs. It therefore measures per-input faithfulness (§3, "Rank-based faithfulness").

The disjoint-set analysis answers a second, dataset-level question: whether an attribution identifies the channels that matter for the model in general. Attributions averaged over one image set are correlated, across channels, with effects averaged over a disjoint set. This dataset-level score also tracks concentration (Spearman −0.776 with neffn_{\mathrm{eff}}, −0.701 after partialling out accuracy; App. C). Disjoint sets matter here because averages over the same images share sampling noise (0.950 against 0.444 in one condition).

The second CIFAR sweep uses 80 networks, 256 images and five seeds; there the within-seed correlation of rank faithfulness with neffn_{\mathrm{eff}} is −0.917 ± 0.073 (−0.83 in the submitted eight-image sweep). Code is in the supplementary material.

We are glad to answer any further questions during the discussion.

Dear authors,

Thank you for your response, but I will maintain my score

AuthorsOpenReview ↗

Thank You for the Update & Follow-up Question

Dear Reviewer 55nw,

Thank you for taking the time to read our rebuttal and post an update.

In our response and revised manuscript, we worked carefully to address each of your initial questions and concerns with detailed clarifications and supporting evidence. Since there are still a few days remaining in the discussion period, could you kindly let us know which specific points or concerns from your initial review remain unresolved or unconvincing to you?

We value your feedback and would greatly appreciate the opportunity to clarify any remaining questions before the discussion window closes.

Best regards, Authors of Submission 9354

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers and the AC. The reviews asked for four things: a precise statement of when the dependence of attribution scores on the network is a problem, evidence that the metric itself fails rather than the task becoming easier, a wider basis for the localisation result, and a much clearer paper. The revised PDF addresses each with new results; the data and the core claim are unchanged. The abstract was revised to state the main result explicitly; the title is unchanged.

An exact account of what rank faithfulness rewards (§4, App. A). Proposition 1 gives the pooled Spearman score in closed form as a function of the fraction of component–input pairs with zero ablation effect and of the attribution's agreement on the other pairs. With 120 of 128 pairs at zero, as in the degenerate CIFAR network, any attribution that zeroes those pairs scores at least 0.997, whatever order it gives the rest, while an attribution that orders every pair of different effect correctly but breaks the ties scores 0.420. In a concentrated layer the score is therefore dominated by reproducing the layer's zero structure, which requires no information about the explained output.

Planted ground truth: the metric prefers worse attributions (§5.2, Fig. 2, Table 2). Silencing a controlled number of channels in trained CIFAR-10 ResNets plants concentration while leaving every other channel unchanged. The exact ablation effects then fall from 1.000 to 0.420, while a ranking that knows only which channels are active rises from 0.003 to 0.999. Once enough pairs are silent, the metric ranks gradient times activation, a wrong-class attribution, activation magnitude and the activity-only ranking above the exact effects, in every planted network. In 22 of 78 trained networks without planting, all with neff≤9.07n_{\mathrm{eff}} \le 9.07, it prefers an attribution that is exact only on negligible channels to one that is exact on the channels carrying 90% of the effect.

Two scores from the same ablations (§4, §5.2, §6). An effect-weighted concordance, which weights each pair of components by the difference of their effects, orders these pairs correctly in all 78 trained networks and in 63 of 65 planted ones. A skill score relative to activation magnitude, an output-agnostic reference, changes by less than 0.04 when planted concentration raises the raw score from 0.670 to 0.957. On ImageNet it shows that Integrated Gradients at the first two ResNet-18 stages adds almost nothing to what activations alone provide (skill 0.005 and 0.024).

Larger and more varied evidence. On CIFAR, a second sweep of 80 networks uses 256 evaluation images and five seeds. The within-seed correlation of rank faithfulness with neffn_{\mathrm{eff}} is −0.917 ± 0.073, and −0.774 after partialling out accuracy within the accuracy band (50 networks).

On ImageNet, we reran the fourteen-model comparison on 100 images per model. The family ordering (ResNets 0.88–0.95, ConvNeXts 0.36 and 0.45, ViT/DeiT 0.03–0.17) and the cross-model correlation with neffn_{\mathrm{eff}} (+0.824, against +0.820 with six images) are unchanged (App. E).

The retraining-based evaluation now has three seeds, KAR, and a no-retraining control on the same masks. Masking the top four channels without retraining drops accuracy to 0.305, against 0.631 for random channels, so near-zero ROAR scores on this layer reflect retraining compensation.

For localisation, eight newly fine-tuned retinopathy graders (five ViT-B/16 seeds, three ResNet-50 seeds) join the two public ones. Over the ten checkpoints the checkpoint explains 51% of the variance of the localisation score and the map 30% (30% and 31% with exact gradients, App. F), and seeds of one recipe differ by up to 2.9× under the same map (§7, Table 6, App. F).

Writing. The paper now has eight sections instead of twelve, with a definitions section before any result. A related-work section engages all 13 works listed by Reviewer 9EsB, together with the closest work on metric reliability (monotonicity correlation, MetaQuantus, normalised AOPC, retraining-based evaluation). Every table and figure has a self-contained caption, §7 gives a full specification of its attribution maps, and the supplementary material contains anonymised code and records for every experiment.

A table mapping each reviewer point to the location of the change is included in the supplementary material (supp/CHANGES.md).

We are glad to answer any further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.