- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the limitations of some existing attribution axioms. It shows that equivariance and completeness do not uniquely determine an attribution rule. It also evaluates whether frame-correct or finer-grained attributions lead to better downstream decisions. Experiments on the docking and reward models show that removing frame dependence or adding detailed attribution features does not necessarily improve intervention outcomes.
Critical strengths
The paper goes beyond standard attribution evaluation metrics to evaluate attribution through downstream interventions. This is a useful direction for testing whether explanations provide practical value.
Critical weaknesses
-
The main theoretical contribution is that equivariance and completeness are not sufficient to determine a unique attribution rule. However, the fact that existing axioms are necessary but not sufficient is not new to the community, making the main contribution of this work quite limited. Besides, the paper also acknowledges that non-uniqueness of attribution is already well known.
-
The link from attribution to downstream action is not rigorously justified. In the docking study, attribution scores are used to select a DOF to modify, while in the reward-model study they are used to choose a restart step. The paper does not rigorously establish that higher attribution should correspond to better interventions in these settings. Therefore, poor downstream outcomes do not clearly show that the attribution itself is unreliable or unactionable.
-
The paper mainly focuses on revealing the limitations of attribution methods, while offering limited constructive progress. It shows that common attribution properties are insufficient, but it does not provide a better attribution rule or an effective way to connect attribution quality with actionability.
-
The presentation and organization of this work are difficult to follow. The key contributions and the role of each experiment are not easy to identify quickly.
Questions for authors
-
What is the new contribution of this work beyond existing findings on the limitations or non-uniqueness of attribution methods?
-
How do the authors justify that the designed downstream experiments are rigorously correlated with the reliability of an attribution method?
-
What is an actionable way to use attribution in practice, and how should attribution scores be connected to downstream interventions?