acceptodds
Under review as a conference paper at ICLR 2027

THE CONTROL DECIDES THE ANSWER: A MATCHED-OPERATOR NULL FOR EXPLANATION CHANGE UNDER MODEL UPDATES

Abstract

When a deployed model is updated in response to a distribution shift, its feature attributions move. Existing work measures how far they move and treats the residual, after conditioning on prediction change, as instability. That is not yet a criterion, because it carries no account of how far an explanation should have moved. We supply one. Using shift families whose Bayes-optimal predictor is available in closed form, we define a warranted-change reference w and decompose attribution change across an update into warranted, unwarranted and explainer-noise parts. The design turns on a control prior work does not use: a matched-operator null, applying the identical update to fresh source data, so treatment and control differ in the distribution alone and not in the amount of training. This changes the conclusion. Across 681 runs the null reproduces a median 82% of the movement that follows an update. On a no-shift placebo, where the answer is fixed by construction, it returns 1.02 on an MLP and 0.98 on gradient-boosted trees with exact TreeSHAP, while the seed-retraining floor prior work uses implicitly returns 0.32 and 1.90 on the same data, disagreeing with itself sixfold. We state the consequence for our own results instead of leaving it to be found: against that seed floor our headline effect is inverted (median 0.31), so the case for the matched null rests on the placebo. Applying the missing control to a published audit, 64% of its 45 settings produce movement smaller than refitting on a bootstrap resample of the same data. Unwarranted change is real but bounded: present across MLPs over a 609x parameter range, trees and a small ResNet, weaker on real census covariates, absent on DistilBERT. Magnitude indicates legitimacy only once an update completes the adaptation it implies, so it fails in the light-update regime auditors work in.

Primary area:
interpretability and explainable AI
Keywords:
feature attributions, explanation change, model updates, distribution shift, matched-operator null

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PqPMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the change of the feature attributions with the updating of a model under distribution shift and asks how to distinguish the changes caused by appropriate adaptation from changes beyond expectation. The authors define an expected-change reference method with the bayes-optimal predictors in the source and target domains. Besides, they make the introduction about one matched control: given the same model, they use the same training procedure on the newly sampled source data rather than target data. This provides a reference for how much attributions could change after the continued training without distribution shift. The authors examine when stable predictions will bring stable attribution for different methods, including Integrated Gradients, local gradient methods, and Shapley-based methods. As for the experiments, they use synthetic data, census data to make image classification, and text classification, also a reanalysis of an earlier study. Based on 681 runs, the matched control reproduces a median of 82% of the attribution movement observed after the model parameters updating. Also, the results show that the choice of control can have huge impact on the conclusions about update strength, even bring reversal of the conclusion. Finally, the relationship between observed and expected attribution changes will vary with how far adaptation has progressed.

Critical strengths
  1. The authors set up a meaningful control through targeted experiments. The matched-operator null addresses an important issue: attribution changes after an update may also be caused by training without distribution shift. They made different experiments to support this control, including the no-shift placebo, comparison with the seed-retraining floor, update-budget sweeps, experiments across model classes, and EMA analysis. This can be the strongest contribution of this paper, because it provides a practical way to make explanation-change evaluations more informative.
  2. The authors made broad checks of the robustness to evaluation choices. They developed a great examination with a wide range of evaluation choices, including alternative distances, normalization schemes, prediction-preservation thresholds, attribution baselines, background sets, and probe-sampling strategies. These results show that the main observations are not driven by a single implementation choice.
Critical weaknesses
  1. With the single evidence of movement magnitude cannot tell us whether the model has appropriate adaptation. The authors use both UEC and the proposed adaptation-completeness ratio to compare scalar distances, but they don’t check whether attribution changes move toward the target reference. For example, the initial attributions are (0.5, 0.5). They should change to (0.8, 0.2) in the ideal case, but the actual update changes them to (0.2, 0.8). Both these two kinds of changes have the same distance of 0.3 from the initial point, but they move in opposite directions. If the null movement and noise floor are zero, UEC would still be zero, and the completeness ratio would be one. Therefore, these quantities can help measure excess movement, but we cannot say whether adaptation is appropriate or complete with them. The author should better make direction-sensitive checks and independent measures of proximity to the target Bayes predictor

  2. The comparisons with prior methods are not sufficient. The authors use their own distance to replace the original SSIM, Spearman, and Jaccard components in Appendix F’s FASS-style baseline. Here the linear-regression “spurious residual” attributed to Delta-Audit lacks an explicit connection to the original method, which includes BAC and DCE metrics. Besides, the re-audit compares configuration B fitted on the original data with configuration A fitted on bootstrap samples, so it seems that it does not use the identical-operator matching used in the main experiments. These comparisons may still be informative, but they are not sufficient to support the claims about the specific limitations of the original method.

Questions for authors
  1. Is it possible for UEC to distinguish the appropriate adaptation from an update in which case that will have same distance in the wrong direction? Please make an evaluation on a controlled case and show if the attribution changes match the target changes.
  2. Could you please evaluate “adaptation completeness” that is independent of the ratio E[Δ]/E[ω], with target-domain excess risk or prediction error which is relative to the known Bayes-optimal predictor? This would help determine whether a ratio near one can be corresponding with successful adaptation.
  3. Please clarify which comparisons use the original methods and which are modified methods. Particularly how do you calculate Delta-Audit’s “spurious residual” using regression? And are the conclusions still meaningful in the case that you use the original FASS components and Delta-Audit metrics? As for the re-audit, please also justify with configuration A’s bootstrap variability as the reference for an A-to-B configuration change.
Reviewer_EHhUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to interpret changes in feature attributions after a model update. The authors propose a control that applies the same update to fresh source data. This separates change associated with the distribution shift from change caused by continued training. They also define warranted change using the Bayes-optimal predictor. This reference is exact in their synthetic settings. The experiments show that the matched control reproduces much of the observed attribution movement. Additional movement remains in several settings, but not on DistilBERT. The paper also shows that attribution movement does not consistently track warranted change.

Critical strengths
  • Comparing a short fine-tune with a full retrain changes several factors at once. The no-shift experiment shows that this comparison can mislead in either direction. I find the proposed matched-update control a sensible alternative.
  • Removing a shortcut should change a model's explanation, even when the underlying causal mechanism stays fixed. The Bayes-optimal reference handles this case correctly. This makes the controlled experiments more informative than comparisons of attribution distance alone.
  • The paper covers different explainers and model classes. It also tests smoothing, explainer noise and different image shifts. I particularly appreciate the brightness-only result, which shows that the method does not report excess change for every shift.
Critical weaknesses
  • When source and target distributions are identical, the treatment and matched control follow the same procedure. So similar attribution movement is expected. Under shift, their differnce measures the effect of changing the update-data distribution. Calling that difference unwarranted needs explanation. I would like the paper to distinguish these two claims more clearly.
  • Under covariate shift, the Bayes-optimal predictor stays fixed. A limited model can still benefit from changing its approximation to fit the target distribution better. The matched control does not necessarily remove this effect. So can Bayes reference can classify useful changes in a finite model as unwarranted?
  • The warranted reference uses IG but the measured changes use several explainers. Wouldn’t it matter under concept shift? The paper should explain this comparison and report the IG-only findings clearly.
  • The paper needs a rewrite to improve its clarity. Long result paragraphs combine many numbers, caveats and implementation details. The writing lacks coherence.
Questions for authors
  • How do you distinguish excess attribution movement from useful adaptation within a limited model class?
  • Do the conclusions about warranted change hold when both the measured change and the reference use IG?
  • What evidence would clarify why the effect is absent on DistilBERT?
Reviewer_Jeo5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to answer the question of when a model is updated, how much should its feature attribution explanations change?

Methodology: updating the model with data from a new distribution vs another from the original distribution. Both with the same training settings. The goal is to distinguish changes associated with the new environment from changes that occur simply because training continues.

Results: changes in the explanation can occur without a change in the distribution. In controlled experiments, they additionally compare observed changes with the change expected from an ideal predictor.

Critical strengths

The question is an interesting and important one that has practical implications.

The idea of updating the model with data from a new distribution vs another from the original distribution is quite interesting.

Critical weaknesses

The paper is not well written.

The main idea is buried.

Too many complicated terms: Warranted change, matched-operator null, seed floor, completeness and legitimacy.

The paper is often defensive and rethorical: the placebo is decisive, instead of leaving it to be found and we ran it rather than argue for it

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.