- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates whether model explanations can predict the effects of previously unobserved internal interventions. The authors propose a response-matrix framework and derive conditions under which measured and compressed information is sufficient for predicting model changes. Experiments on small-scale neural networks, MLPs, and GPT-2 demonstrates that conventional methods of compression could remove the information useful for making predictions, and it can be more beneficial to cover more inputs than to model interactions between them.
Critical strengths
-
This paper is clear-writing, and it contains a distinct investigation into whether mechanisms that can predict the behavior of models under novel interventions can be used, which is important to model interpretability.
-
They provide many experiments and theory analysis in this paper.
Critical weaknesses
-
The predictions from the GPT-2 experiments involve changes in the margin of template-averaged logit values. The modifications made via additive edits are minor, and the authors acknowledge that the study provides no evidence of a reversal in the decision outcome. Uncertainty remains regarding whether this method can induce behavioral changes in real-world applications.
-
In Section 4.2, the authors explained how the direction matrix itself encompasses the local operation space. So obtaining additional directions for the same input yields no further information.
Questions for authors
-
The proposed framework studies response matrices constructed from predefined interventions, while existing explanation methods are not directly evaluated under the same measurement. I would like to know how this framework provides insights into evaluating the reliability of existing explanation methods.
-
In Appendix G, single-head prediction outperforms pairwise prediction on new inputs when both use 12 fitting pairs. Can the authors further distinguish the benefit of broader input coverage from differences in cross-input generalization between the two methods?