acceptodds
Under review as a conference paper at ICLR 2027

When Do Explanations Predict Model Changes?

Abstract

Can an explanation predict a model's response to an internal change it has not observed? Matching the original outputs is insufficient: two models can agree on every input yet react differently to the same internal change. We study what finite measurements must retain to predict specified changes, and what to improve when those predictions fail. In a linear response space, a necessary and sufficient condition distinguishes inaccessible information from information discarded by compression; a stability bound separates both from unreliable recovery. Controlled experiments make this distinction actionable: ordinary least squares recovers local responses, whereas principal component analysis (PCA) can lose the required differences despite preserving almost all variance. Retaining those differences repairs prediction. Larger MLPs and pretrained GPT-2 show why local recovery is only part of the problem: new inputs and finite changes introduce different errors. In GPT-2, measuring more inputs improves prediction when local fitting is already accurate. Finite attention-head replacements expose a sharper trade-off: interaction measurements greatly improve predictions of unqueried combinations on the fitting inputs, yet can lose to simpler measurements on more inputs at the same budget. Together, the theory and experiments distinguish when to change the measurements, the representation, or the inputs used to predict model edits.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, explanation evaluation, activation patching, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jmgMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether model explanations can predict the effects of previously unobserved internal interventions. The authors propose a response-matrix framework and derive conditions under which measured and compressed information is sufficient for predicting model changes. Experiments on small-scale neural networks, MLPs, and GPT-2 demonstrates that conventional methods of compression could remove the information useful for making predictions, and it can be more beneficial to cover more inputs than to model interactions between them.

Critical strengths
  1. This paper is clear-writing, and it contains a distinct investigation into whether mechanisms that can predict the behavior of models under novel interventions can be used, which is important to model interpretability.

  2. They provide many experiments and theory analysis in this paper.

Critical weaknesses
  1. The predictions from the GPT-2 experiments involve changes in the margin of template-averaged logit values. The modifications made via additive edits are minor, and the authors acknowledge that the study provides no evidence of a reversal in the decision outcome. Uncertainty remains regarding whether this method can induce behavioral changes in real-world applications.

  2. In Section 4.2, the authors explained how the direction matrix itself encompasses the local operation space. So obtaining additional directions for the same input yields no further information.

Questions for authors
  1. The proposed framework studies response matrices constructed from predefined interventions, while existing explanation methods are not directly evaluated under the same measurement. I would like to know how this framework provides insights into evaluating the reliability of existing explanation methods.

  2. In Appendix G, single-head prediction outperforms pairwise prediction on new inputs when both use 12 fitting pairs. Can the authors further distinguish the benefit of broader input coverage from differences in cross-input generalization between the two methods?

Reviewer_FCwtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies: If only several internal interventions can help predicts the influence of new interventions.

Author use linear reponse model to study the problem, and saperate three cases

  1. not messured
  2. lost by copression
  3. unstable recovery

Use PCA、MLP、GPT-2、Attension-head during the experiment

Critical strengths
  1. Good ablation on pca, showing that preserve most covariance does not preserve the information needed to predict certain internal changes.
  2. Attention head experiment consider both the measured input and new input. Interaction measurements works better on the measured inputs, but more input with simpler method works petter on new input at larger budget.
Critical weaknesses

The GPT-2 experiment is kinda narraw, it is unclear whether the result would can generalize to other model and other inputs.

Questions for authors
  1. The experiment mainly based on GPT-2, is the result still generalizable to other models
  2. with fixed model and budget, after resample inputs, will the single-head method has advantage?

other suggestions: should better define the "response" once near the beginning.

Reviewer_1YP1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when a small set of measured model responses can predict the effect of internal edits that were not tried. It gives a linear condition that tells whether the needed information was never measured, lost in compression, or not stably recoverable, and it separates the error on new inputs into three sources. Experiments on small networks, MLPs and GPT-2 show that PCA can lose the needed information, that errors on new inputs come mainly from differences between input sets, and that measuring pairwise head interactions does not pay off on new inputs under a fixed budget.

Critical strengths

Strengths

  1. The paper turns the question of whether an explanation can predict model edits into a diagnosis that says what to fix when the prediction fails. It separates three causes, namely missing measurements, information lost in compression, and differences between input sets. Each cause has its own remedy, which is to measure more, to store the same measurements differently, or to measure on more inputs.

  2. The main empirical finding is supported by controlled and repeated experiments. In Section 3.3 all methods receive the same measurements, so the failure of PCA can be traced to what it keeps rather than to the estimator. The authors explain the cause of this failure and repeat the result on three new groups of models.

Critical weaknesses

Weaknesses

  1. (Partially Solved) The practical need for predicting untried edits is not established. The paper does not show a setting where prediction is cheaper or more useful than simply running the edits. In Section 5, for example, running the 24 predicted head combinations directly would cost about a third of the forward passes needed to measure the pairwise interactions.

  2. (Unsolved) The theoretical contribution is limited. Theorem 1, which the paper lists as its first contribution, is the standard condition in linear models for when a quantity can be recovered from linear measurements [1]. Appendix A.3 itself describes these arguments as standard.

  3. (Partially Solved) The compression result depends on a setup that the paper does not motivate. In Section 3.3, PCA is applied before the positive and negative responses are subtracted. At that stage most of the variance is a shared offset unrelated to the edit, so PCA keeps the offset and drops the useful difference. That PCA can drop directions needed for prediction is also already known [2].

  4. (Solved) The experiments do not compare with existing methods for the same problem. Section 5 predicts the effect of replacing several heads at once, which is the problem studied by COAR [3]. COAR fits a linear model to the effects of randomly chosen groups of components and has been applied to GPT-2. Without this comparison, it is unclear whether measuring every single head and every pair is a good use of the budget.

references:

[1] Shayle R. Searle. Linear Models. John Wiley & Sons, New York, 1971.

[2] Ian T. Jolliffe. A note on the use of principal components in regression. Journal of the Royal Statistical Society Series C (Applied Statistics), 31(3):300–303, 1982.

[3] Harshay Shah, Andrew Ilyas, and Aleksander Madry. Decomposing and editing predictions by modeling model computation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 44244–44292. PMLR, 2024. URL https://proceedings.mlr.press/v235/shah24a.html.

Questions for authors
  1. Under the same forward budget, how do the single head and pairwise predictors in Section 5 compare with a COAR style regression on randomly sampled head combinations?

  2. Can you give a practical task where the stored responses must answer many more edits than they cost to collect? How many edits are needed before prediction becomes cheaper than running them?

  3. Does PCA still fail if it is applied after the positive and negative responses are subtracted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.