- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CFIP estimates uncertainity in ROI-level neuroimaging explanations. It defines an ROI's effect as the change in model logit after replacing that ROI. It applies a Laplace approximation to model parameters and derives a Gaussian posterior over ROI effects. Tests include ADNI, OASIS, HCP-Aging, synthetic data, and comparisons with ensembles and HMC.
Critical strengths
- Influence posterior is simple to compute, and the exact last-layer case is clearly derived. Joint covariance across ROI effects is useful.
- Evaluation goes beyond attribution maps: retraining stability, HMC comparisons, calibration, and runtime measurements.
Critical weaknesses
- Methodological novelty seems limited for ICLR.
- With fixed features and the last-layer assumption, this is a standard Gaussian pushforward of a Laplace posterior.
- Posterior mean is exactly the deterministic ROI-removal effect.
- So better deletion ranking compared with gradient/SHAP methods does not show a benefit from posterior uncertainty itself.
- Reliability is inconsistent in harder settings. The cognitively normal to mild cognitive impairment task has 0.708 coverage for nominal 90% intervals in the main analysis, and Laplace intervals underestimate HMC variability.
- Most experiments fix the feature extractor too, leaving out an important source of uncertainty.
Questions for authors
- What does CFIP add over applying standard last-layer Laplace directly to deterministic ROI-removal effects? Please separate gains from the attribution definition and gains from uncertainty estimation.
- Can the authors explain the undercoverage against retraining/HMC and show when all-layer uncertainty improves reliability in the clinical tasks?
3 comments
Dear Reviewer RKmm,
We hope we’ve addressed all your concerns. Since the discussion period ends in <6 hours, please let us know if you need any further clarification. If no further clarifications are needed, may we kindly request that you revise your scores upwards?
Thank you, The Authors
Thank you very much for the detailed response and the additional results. I appreciate authors' clarifications and the effort to address my concerns. A few points remain:
- Methodological novelty: The authors clarify that the influence functional is not part of the contribution. The proposed closed-form posterior is still a standard Gaussian pushforward of a Laplace approximation. Its methodological novelty remains unclear.
- Attribution versus uncertainty: The rebuttal separates these contributions more clearly. However, the posterior mean equals the deterministic ROI-removal effect, the reported gains in attribution faithfulness is not a benefit from uncertainty estimation itself.
- Fixed feature extractor: The given references support the practical use of simpler neuroimaging models, but do not show that uncertainty in the learned feature extractor can be neglected.
- Calibration and reliability: The new coverage results are nive, particularly for CN-MCI. The calibration protocol and the independence of calibration and evaluation data still need clarification. The all-layer approach also produces relatively conservative intervals.
Overall, the rebuttal partly addresses my reliability concerns, but the central concern about methodological novelty remains. I therefore maintain my original rating of 2 (Weak Rejection).
We thank Reviewer RKmm for the careful review. We appreciate the reviewer's recognition that the exact last-layer case is clearly derived and that the evaluation goes beyond attribution maps.
W1 + W3 — novelty; fixed feature extractor
Reply: Our contribution is the uncertainty estimation in brain neuroimaging tasks: a closed-form posterior over regional influence that needs no repeated sampling, for neuroimaging models whose explanations are otherwise reported as single point maps. The influence functional is not part of our contribution. The deletion-based functional is a simple starting point; the same construction applies to other influence definitions (Table 14 already applies it to a gradient functional), and we will explore better ones in future work. On the fixed feature extractor, using extracted regional features is standard practice in brain neuroimaging (Fischl, 2012; Cuingnet et al., 2011; Rathore et al., 2017), and training from scratch does not beat it: a reproducible ADNI benchmark showed no advantage for image-level CNNs over an SVM (Wen et al., 2020), and UK Biobank benchmarking gave deep models no edge over linear ones on brain scans (Schulz et al., 2020). CFIP is also not limited to a fixed extractor: the full-network tangent variant (Sec. 4.5) includes all layers in closed form.
Q1 — separating gains from the attribution definition and from uncertainty estimation
Reply: We separate the two by fixing one and changing the other. With the uncertainty fixed, the functional sets the ranking: under the same Laplace posterior, the deletion-based functional gives AOPC 0.177 against 0.127 for the gradient one (Table 14). With the functional fixed, the gain from our uncertainty estimation is reliability information at closed-form cost. It matches the ranking quality of a ten-model ensemble from a single trained model (AOPC 0.177 vs 0.172, Table 14; 0.056 vs 0.340 ms per subject, Table 7), it reproduces 4,000-draw posterior sampling to within 0.001 with no sampling, and it preserves the variance ordering of the HMC reference (ρ=1.000, Table 5). Its variances also track retraining variability, with rank agreement above 0.90 on all three ADNI tasks (Tables 3 and 18), which is what lets it flag unstable regions: ROIs that enter the top ten only intermittently across retrainings carry 1.24 times the uncertainty of those always selected (Sec. 5.6).
W2 + Q2 — under-coverage on harder tasks; all-layer version
Reply: The under-coverage appears on harder tasks such as CN–MCI because the signal is weaker, so different training runs settle on different solutions: ranking stability across retrains drops from 0.652 on CN–AD to 0.541 on CN–MCI (Table 12). A posterior built from one trained model does not include this between-run variation, so its intervals are narrower than the spread of retrained models. Against HMC, which uses the same trained model, the gap is small: Laplace keeps the variance ordering (ρ=1.000) and its variances are 92–95% of HMC’s (Table 5). We have therefore introduced a calibration step: the uncertainty hyperparameter is selected on a held-out set so that the 90% coverage target is met, and on held-out subjects this holds across all three ADNI tasks (table). All-layer uncertainty matters when the extractor itself varies across training runs: by Proposition 2 it adds the extractor’s contribution on top of the last-layer variance, and Table 2 shows this contribution grows with extractor depth. The last row reports it on the three clinical tasks. On the clinical tasks (last row) it removes the under-coverage on CN–MCI, raising coverage from 0.793 to 0.964, but it is conservative on all three tasks, with wider intervals.
References: Wen et al., 2020 (https://arxiv.org/abs/1904.07773v6); Schulz et al., 2020 (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7447816/).