acceptodds
Under review as a conference paper at ICLR 2027

Attention Tracks Reaction Centers, But Doesn't Need Them: A Causal Study of a Retrosynthesis Transformer

Abstract

Attention weights in transformer-based chemistry models are widely treated as an interpretability signal — attention concentrated on a molecule's reaction center is commonly cited as evidence that a model has "learned" the relevant chemistry. We test this assumption directly using causal intervention. In a retrosynthesis Chemformer trained on USPTO-50k, encoder attention correlates strongly with ground-truth reaction centers (within-molecule AUC 0.902 vs. 0.836 for hand-engineered structural descriptors), replicating the correlational finding that motivates this interpretability practice. However, zero-ablating attention heads and measuring the resulting change in generation likelihood reveals that causal necessity is inversely related to this correlation: the heads most predictive of reaction centers are among the least necessary for generation, while 74.9% of total ablation damage concentrates in the encoder's first layer — the layer with the lowest reaction-center correlation of any layer in the network. We show this necessity is not decomposable into individual head contributions: the sum of single-head ablation effects in layer 1 is 115× smaller than the effect of ablating all eight heads jointly, and which single head survives a near-total ablation determines whether the model recovers 84% or 31% of baseline performance. We trace this structure to a measurable property of each head: attention entropy predicts sole-survivor sufficiency (ρ = −0.857) in the opposite direction from necessity, explaining why sharp, low-entropy positional heads are individually redundant yet individually sufficient, while diffuse, high-entropy heads are collectively necessary yet cannot reconstruct function alone. Our results extend the attention-is-not-explanation literature from NLP classification into generative chemistry models, and show that correlational interpretability claims in this domain — as in concurrent findings on 3D-geometry alignment in other chemical language models — do not reliably predict where causal computation actually occurs.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, attention, causal intervention, chemistry, transformers, retrosynthesis, head ablation, attention entropy, Chemformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_pyN2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether attention to reaction centers reflects causal importance in a retrosynthesis Transformer. Using head-level ablation on Chemformer, the authors find that heads strongly correlated with reaction centers have little impact on generation when removed, while most ablation damage is concentrated in the first encoder layer. They further identify non-additive interactions among attention heads and a correlation between attention entropy and head sufficiency.

Critical strengths

Understanding how retrosynthesis models represent and utilize reaction-center information is an important problem, given the extensive research on explicitly incorporating reaction centers into retrosynthesis prediction.

Critical weaknesses
  1. Limited novelty and mechanistic insight. Although the paper presents extensive empirical analyses, I find the main observations largely consistent with existing understanding of Transformer behavior. The distinction between attention correlation and causal importance is already well established, and the redundancy and interactions among attention heads are not particularly surprising. More importantly, the analyses do not lead to a substantially new understanding of how the model actually performs retrosynthesis or utilizes chemical information. As a result, I find the contribution limited beyond confirming previously known interpretability concerns in a chemistry-specific setting.

  2. Unclear connection between the observed phenomena and reaction-center information. The finding that the first encoder layer dominates ablation damage may simply reflect the general importance of early representations in Transformers, rather than anything specific to retrosynthesis. Similarly, the strong non-additivity of attention-head ablation could arise from generic redundancy and interactions among heads. These observations do not necessarily reveal any distinctive mechanism related to reaction centers. Consequently, it is unclear what additional conclusions about retrosynthesis or chemical reasoning can be drawn from these experiments.

  3. The central claim appears overstated. The fact that ablating reaction-center-correlated attention heads causes little performance degradation does not establish that the model does not need reaction-center information. The model may still encode and utilize such information through other attention heads, layers, or implicit representations that the current analysis does not identify. In particular, the experiments intervene on attention heads rather than reaction-center information itself. Therefore, the results support a narrower conclusion—that attention correlation is not a reliable indicator of head-level causal necessity—rather than the stronger implication suggested by the title that reaction-center information is unnecessary for retrosynthesis prediction.

Questions for authors
  1. What evidence supports the conclusion that reaction-center information is unnecessary, rather than being encoded and utilized through other components of the model?

  2. To what extent are the first-layer dominance and non-additive head interactions specific to retrosynthesis, rather than general properties of Transformers?

Reviewer_wk6GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests whether attention on reaction centers in a chemistry transformer indicates where the model's computation happens. In a Chemformer fine-tuned for USPTO-50k retrosynthesis, encoder attention predicts reaction centers better than structural descriptors, yet zero-ablating heads shows that the most reaction-center-correlated heads are among the least necessary, while the first encoder layer, the least correlated one, accounts for most of the damage measured by teacher-forced likelihood. The paper further reports that layer-1 heads behave non-additively and that attention entropy is associated with whether a single surviving head can restore function.

Critical strengths

Testing attention-based explanations of reaction models with interventions addresses a real weakness in chemistry-model interpretability, and the finding is interesting but reported honestly.

Single-head, joint, cumulative and leave-one-in ablations reveal behavior that single-head ablation alone would miss.

Critical weaknesses

W1. Necessity is concentrated in layer 1 by about two orders of magnitude, and the compared head sets differ in layer composition. The high-salience heads sit in a later layer, while the most damaging low-salience set contains several layer-1 heads. The salience–necessity inversion can therefore be explained by a layer effect alone, and outside layer 1 the differences are very small.

Zero-ablation of all layer-1 heads pushes downstream activations off-distribution, which alone could produce the layer-1 concentration and the non-additive threshold. The only metric is teacher-forced likelihood with effects around 1e-3, so "generation performance" overstates what is measured.

W2. Instead of removing reaction-center-correlated heads, the author should also try to intervene on the information itself, such as knocking out attention to reaction-center tokens across all heads. I am not sure whether this is available, but 1 attention head usually contains too many things and it is hard to say a head is "reaction-center-correlated heads"

Minor

Ablating layers 1–3 is more damaging than ablating all heads. How to explain this?

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.