- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This submission investigates how the evaluation of explanations, i.e. faithfulness of layer attribution and localisation of input attribution, changes depending on the convolutional network being explained. It considers image classification tasks such as CIFAR-10, ImageNet, and medical imaging. CIFAR-10 and ImageNet experiments show that the same attribution method can get a higher faithfulness score simply because it is explaining a simpler part of the network. A separate study with retinal images finds substantial differences in lesion localisation between two checkpoints with the same model architecture.
Critical strengths
None
Critical weaknesses
Two critical weaknesses of this paper are (1) lack of novelty and (2) poor quality.
- The main claim given in the paper's title, that explanation evaluation depends on the model, has been studied in depth. The four "findings" presented in the intro do not constitute a substantial contribution to machine learning research; studying ResNets on CIFAR-10 is not informative. The paper should distinguish from and relate to findings from the relevant literature (only for example):
- Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations. KDD 2025
- Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics. NeurIPS 2024
- M4: A Unified XAI Benchmark for Faithfulness Evaluation of Feature Attribution Methods across Metrics, Modalities and Models. NeurIPS 2023
- Sanity Simulations for Saliency Methods. ICML 2022
- Do Feature Attribution Methods Correctly Attribute Features? AAAI 2022
- Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanations. WACV 2022
- Benchmarking saliency methods for chest X-ray interpretation. Nature Machine Intelligence, 2022
- When Comparing to Ground Truth is Wrong: On Evaluating GNN Explanation Methods. KDD 2021
- Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging. Radiology: Artificial Intelligence, 2021
- Evaluating local explanation methods on ground truth. Artificial Intelligence, 2021
- Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2020
- Sanity Checks for Saliency Metrics. AAAI 2020
- Fooling Neural Network Interpretations via Adversarial Model Manipulation. NeurIPS 2019
- This paper reads as AI-generated to the point where it is hard to parse. The "AI use statement" says only that "GPT was used only to aid or polish the writing of this manuscript," and it feels dishonest, as judged by the amount of weirdness given in its text, e.g. many unclear/uncommon words/phrases, ": ;" signs, 12! sections. Please read published ICLR papers to understand the expected standards for terminology, structure, and clarity. Only for example:
- The second sentence says "Rank-based faithfulness" as if it were obvious what it means. It is also unclear what an attribution method is at this point.
- In fact, there is only a single reference to the literature in the introduction.
- What is an "interference penalty"?
- What does "rank-based faithfulness tracks the number of channels carrying the behavior at within-seed Spearman −0.83" mean?
- The 1st contribution reads as "The score rises toward the readout." Which score? What is readout? It also shouldn't reference Appendix E.
- "At fixed width, concentration strongly predicts the score." What is the width? What is the concentration?
- "Simple scalar corrections do not remove the effect." Again, what are "scalar corrections"? Which effect?
- "We conclude with a reporting protocol in §10." What does the protocol mean, and what is it for?
- L82: What are "the magnitudes of the individual input–output paths through channel "?
- Section 7 does not specify how attribution maps are computed or which representation it attributes.
- None of the six Tables in the paper have a caption.
Regarding the paper's soundness, no code is attached to verify its findings.
Questions for authors
None
4 comments
response to reviewer 9EsB (2/2)
(2/2)
Clarity. The passages the reviewer cites are now defined or rewritten. The attribution method and rank-based faithfulness are defined in §1 (first two paragraphs) and formally in §3. The readout is defined in §3 (global average pooling followed by the linear layer). For concentration and width, §3 defines the effective component count n_eff, and §3 ("Networks") defines the fixed width of 16 channels. The "within-seed Spearman −0.83" is now explained in §5.1 as the correlation across training conditions, computed separately for each seed and averaged.
For the interference penalty, §3 states the training term λ · mean_j(log M_j − log |C_j|), where M_j and C_j are the absolute and signed sums of the contributions of all paths through channel j. This term rewards paths that agree in sign. The phrase "magnitudes of the input–output paths" (L82) is rewritten in §3 ("Networks") as the sum of absolute path contributions.
"Scalar corrections" is replaced by the named procedures (subtracting a random ordering, dividing by the reproducibility of the target, and restricting to the support of the effect in App. D), and "Protocol" is replaced by five concrete recommendations tied to quantities defined in the paper (§8). The introduction now cites the evaluation literature, and the contributions refer only to main-text sections.
How the maps of §7 are computed. All maps are input-space maps for the logit of the predicted grade, on the full fundus image resized to 224 × 224: gradient magnitude, input × gradient (signed sum over colour channels), absolute input × gradient, Integrated Gradients (black baseline, 32 steps), SmoothGrad (16 samples), and Grad-CAM on the tokens entering the last attention block. The submitted Figure 4 uses gradients with attention patterns and LayerNorm statistics held at their forward values, together with two path-magnitude maps. §7 now states this convention, and App. F reports both conventions on every checkpoint. The score is average precision on the union of lesion pixels, relative to a uniformly random map on the same images.
Code and verification. The supplementary zip contains the anonymised code for every experiment, the saved experimental records, and the scripts that reproduce every number, table, and figure directly from those records. We sincerely apologise for the missing table captions and overly fragmented sections in our initial submission. All experiments, proofs, and analyses were carried out by the authors, with language models used solely for stylistic wording polish as stated.
We are glad to answer any further questions during the discussion.
I will maintain my clear reject rating. This has practically become a new paper that needs a comprehensive re-review. It remains poorly written, and its claims are not necessarily novel.
Thank you for looking at the revision. For the benefit of the discussion record and the Area Chair, we address each of the three points in your comment:
-
Scope of the revision ("practically a new paper"). The title, research question, and all three empirical pillars from the initial submission (CIFAR-10 representation concentration in §5.1, ImageNet layer position across 14 models in §6, and IDRiD retinopathy lesion localisation in §7) remain intact with the same underlying findings. Merging twelve short sections into eight, defining all terms before results (§3), and adding captions fulfilled your own presentation requests, while every new result directly answers a specific question from the other reviewers on the existing setups: Proposition 1, the planted-channel test, and effect-weighted concordance (§4, §5.2) answer Reviewer 55nw's Q2 and Reviewer KmSv's Q3, and the additional retinopathy seeds, larger image sweeps, and no-retraining ROAR control (§5.3, §6, §7) answer Reviewer KmSv's Q1 and Q2. Addressing the reviewers' own questions on the submitted setups does not make this a different paper.
-
Writing quality ("remains poorly written"). Every concrete clarity issue listed in your initial review (undefined terms, missing table captions, fragmented sections, and §7 map specifications) has been resolved in the revised PDF. Because the new comment states that the paper "remains poorly written" without pointing to any passage in the revised text, please let us know which specific sentence or section still reads unclearly so we can address it directly.
-
Novelty ("claims are not necessarily novel"). Section 2 compares our work directly against all 13 papers you listed. While prior benchmarks and sensitivity studies show that attribution maps or method rankings vary across models and metrics, none derives the closed-form mechanism by which zero-effect pairs dominate rank-based faithfulness (Proposition 1), shows that the metric ranks output-agnostic or wrong-class attributions above exact ablation effects in networks of known ground truth (§5.2, Table 2, including 22 of 78 unplanted networks), or resolves this reversal with an effect-weighted concordance from the same ablations (§4). If a specific prior paper already establishes these results, we would welcome the citation so we can compare against it directly.
We appreciate the feedback that helped us strengthen and clarify the manuscript.
response to reviewer 9EsB (1/2)
(1/2) We thank the reviewer for the list of related works and for pointing out the passages that needed definitions. The revision rewrites the paper: eight sections instead of twelve, definitions (§3) before any result, self-contained captions on every table and figure, and anonymised code.
Relation to the literature (new §2). All 13 works are now discussed; each entry was checked against the publisher's page.
Benchmarks (M4, NeurIPS 2023; LATEC, NeurIPS 2024; Saliency-Bench, KDD 2025) apply many metrics to many methods and models, and report that method rankings change with the model and the metric. We hold method and metric fixed and identify which property of the explained network changes the score, by how much and in which direction. For rank faithfulness this is an exact formula (Proposition 1, §4).
Tomsett et al. (AAAI 2020) test the reliability of the metrics psychometrically. We add a failure of a different kind: in a concentrated layer a rank correlation is maximised by reproducing which components have no effect, so the metric ranks an attribution that knows only which channels are active above the exact ablation effects, in every planted network once three quarters of the pairs are silent (§5.2, Table 2).
Work on ground truth for explanations (Guidotti, AIJ 2021; Zhou et al., AAAI 2022; Kim et al., ICML 2022) plants reasoning to test attribution methods. We plant concentration to test the evaluation metric, with attributions whose quality is known by construction. Faber et al. (KDD 2021) show that a model need not use planted evidence; our ground truth is the measured effect of each component on the trained network, so this cannot arise.
Work on the dependence of explanations on the trained model (Heo et al., NeurIPS 2019; Schramowski et al., NMI 2020; Watson et al., WACV 2022; Arun et al., Radiology: AI 2021; Saporta et al., NMI 2022) concerns the maps themselves: they change across seeds, under adversarial fine-tuning, and between retrained chest X-ray models. We measure the consequence for the evaluation outcome: in retinopathy lesion localisation the checkpoint accounts for as much of the score variance as the attribution method (51% vs 30% with the main maps, 30% vs 31% with exact gradients; §7).
The new contributions are therefore the exact characterisation (§4), the planted-ground-truth demonstration that the metric reverses the order of attributions of known quality (§5.2), an effect-weighted concordance that orders these pairs correctly, and a skill score that stays within 0.04 while planted concentration raises the raw score by 0.29 (§4, §5.2). CIFAR-10 is the controlled setting in which concentration can be planted exactly; layer and checkpoint effects are measured on ImageNet ResNets, ConvNeXts, ViTs and DeiTs (§6, App. E) and on ten retinopathy graders (§7).
Continued in part 2/2 below.