- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the change of the feature attributions with the updating of a model under distribution shift and asks how to distinguish the changes caused by appropriate adaptation from changes beyond expectation. The authors define an expected-change reference method with the bayes-optimal predictors in the source and target domains. Besides, they make the introduction about one matched control: given the same model, they use the same training procedure on the newly sampled source data rather than target data. This provides a reference for how much attributions could change after the continued training without distribution shift. The authors examine when stable predictions will bring stable attribution for different methods, including Integrated Gradients, local gradient methods, and Shapley-based methods. As for the experiments, they use synthetic data, census data to make image classification, and text classification, also a reanalysis of an earlier study. Based on 681 runs, the matched control reproduces a median of 82% of the attribution movement observed after the model parameters updating. Also, the results show that the choice of control can have huge impact on the conclusions about update strength, even bring reversal of the conclusion. Finally, the relationship between observed and expected attribution changes will vary with how far adaptation has progressed.
Critical strengths
- The authors set up a meaningful control through targeted experiments. The matched-operator null addresses an important issue: attribution changes after an update may also be caused by training without distribution shift. They made different experiments to support this control, including the no-shift placebo, comparison with the seed-retraining floor, update-budget sweeps, experiments across model classes, and EMA analysis. This can be the strongest contribution of this paper, because it provides a practical way to make explanation-change evaluations more informative.
- The authors made broad checks of the robustness to evaluation choices. They developed a great examination with a wide range of evaluation choices, including alternative distances, normalization schemes, prediction-preservation thresholds, attribution baselines, background sets, and probe-sampling strategies. These results show that the main observations are not driven by a single implementation choice.
Critical weaknesses
-
With the single evidence of movement magnitude cannot tell us whether the model has appropriate adaptation. The authors use both UEC and the proposed adaptation-completeness ratio to compare scalar distances, but they don’t check whether attribution changes move toward the target reference. For example, the initial attributions are (0.5, 0.5). They should change to (0.8, 0.2) in the ideal case, but the actual update changes them to (0.2, 0.8). Both these two kinds of changes have the same distance of 0.3 from the initial point, but they move in opposite directions. If the null movement and noise floor are zero, UEC would still be zero, and the completeness ratio would be one. Therefore, these quantities can help measure excess movement, but we cannot say whether adaptation is appropriate or complete with them. The author should better make direction-sensitive checks and independent measures of proximity to the target Bayes predictor
-
The comparisons with prior methods are not sufficient. The authors use their own distance to replace the original SSIM, Spearman, and Jaccard components in Appendix F’s FASS-style baseline. Here the linear-regression “spurious residual” attributed to Delta-Audit lacks an explicit connection to the original method, which includes BAC and DCE metrics. Besides, the re-audit compares configuration B fitted on the original data with configuration A fitted on bootstrap samples, so it seems that it does not use the identical-operator matching used in the main experiments. These comparisons may still be informative, but they are not sufficient to support the claims about the specific limitations of the original method.
Questions for authors
- Is it possible for UEC to distinguish the appropriate adaptation from an update in which case that will have same distance in the wrong direction? Please make an evaluation on a controlled case and show if the attribution changes match the target changes.
- Could you please evaluate “adaptation completeness” that is independent of the ratio E[Δ]/E[ω], with target-domain excess risk or prediction error which is relative to the known Bayes-optimal predictor? This would help determine whether a ratio near one can be corresponding with successful adaptation.
- Please clarify which comparisons use the original methods and which are modified methods. Particularly how do you calculate Delta-Audit’s “spurious residual” using regression? And are the conclusions still meaningful in the case that you use the original FASS components and Delta-Audit metrics? As for the re-audit, please also justify with configuration A’s bootstrap variability as the reference for an A-to-B configuration change.