- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how replacement distributions affect feature-removal explanations. The primary study compares ridge-based conditional Gaussian and moment-matched marginal Gaussian replacements on Dry Bean across three MLP runs on a common data split. Conditional replacement achieves lower coordinate-prediction negative log-likelihood and covariance discrepancy, yet produces substantially smaller loss-based importance scores and different feature rankings. Complementary experiments on CelebA, Adult, Spambase, and Bank Marketing examine accuracy sensitivity to stochastic draws, deterministic means, and label-informed replacements. The main contribution is a controlled empirical characterization of replacement sensitivity, showing that stronger observational fidelity can coexist with substantial disagreement between explanations.
Critical strengths
The observed differences affect substantive interpretation, beyond numerical score scales. On Dry Bean, conditional replacement achieves better coordinate-prediction and covariance diagnostics, yet conditional and marginal replacement select different top-ranked features in all three runs, with Kendall’s τ_b between 0.12 and 0.32. These results provide a concrete illustration that stronger performance on distributional diagnostics can coexist with substantial disagreement in feature prioritization. This is consequential for interpreting and reporting feature-removal explanations.
Critical weaknesses
-
The novelty is limited The fact that conditional and marginal replacement define different estimands, and can therefore produce different importance scores for dependent features, is already recognized in the literature discussed by the paper. The paired study quantifies this distinction clearly, but provides limited additional analysis of the mechanisms or conditions governing the magnitude and ranking changes. Consequently, the manuscript’s advance beyond a controlled illustration of an established principle remains modest, which limits the strength of its contribution.
-
The direct evidence for the central finding is narrow. The conditional–marginal comparison uses one Dry Bean split, three MLP runs, and two Gaussian replacement constructions. The secondary experiments use different replacement contrasts and an accuracy endpoint, so they do not directly establish the robustness of the primary score and ranking findings. Moreover, the ranking results are point estimates, and the reported bootstrap conditions on fitted models and realized draws. These limitations leave uncertainty about how strongly the reported disagreements depend on the particular dataset, replacement approximations, and experimental realization.
Questions for authors
How stable are the top-ranked feature identities and Kendall’s τ_b values when increasing the Monte Carlo draw count or resampling the evaluation examples? Since ranking disagreement is central to the paper’s contribution, quantifying its uncertainty would help establish the strength of this finding.