- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper explains the problem in using attention as a evidence for GFM's biological understanding. The authors point out that those attention is not a causal evidence, and show "AG" at the splice acceptor as an example. Attention is enriched at AG, but removing those sources does not reduce AUROC. To test this they build their own model (LIND), a frozen Mamba backbone with an added k-mer memory and a local read head, so the attended sources can be removed exactly.
The paper explains the problem in using attention as a evidence for GFM's biological understanding. The authors point out that those attention is not a causal evidence, and show "AG" at the splice acceptor as an example. Attention is enriched at AG, but removing those sources does not reduce AUROC. To test this they build their own model (LIND), a frozen Mamba backbone with an added k-mer memory and a local read head, so the attended sources can be removed exactly.
My main concern is the scope. That attention is correlation and not causal evidence is already widely known in the NLP community. So the problem itself is narrow and largely settled, and the contribution is also narrow. The whole result rests on one case, AG on one checkpoint. AG was the only one of the five families that passed their enrichment test. It does not hold on other seeds. Also, the model does not use AG even when the input is mutated. So the null result was close to expected.
I also question the novelty and validity of the method. The components already exist k-mer/n-gram memory (Engram, Gengram), Backpack-style attention, attention knockout and Mamba knockout. The local, attention-matched test (source and value removal) is never shown to detect dependence at a site where dependence is known to exist, since there is no positive control. The paper never validates the site-level test on a known case. Without this, I cannot tell whether "not depending" means the attention is misleading or the test is not sensitive enough.
Finally, the method is not applied to any open GFM such as DNABERT, NT or SpliceBERT, even though these motivate the paper. I am not convinced this approach can tell us whether biological understanding exists in those models. SAE-based analysis (e.g. Evo 2, InterPLM) already works on real models at the concept level and has shown that ablating features matters. The paper should justify or discuss these methods and explain what its approach provides that SAE cannot.
Critical strengths
The paper states clearly what each operation changes (source removal, value removal, output masking). This matters because it is the paper's main methodological contribution --- a clean way to connect an attention measurement to an intervention on the same object, which most attention-based analyses in genomics lack.
Critical weaknesses
The method is not validated. There is no positive control. Whole-memory removal and some output-masking windows do change predictions. However, the attention-matched site-level test is never shown to detect dependence at a site where dependence is known to exist. A null result at AG therefore cannot be distinguished from limited sensitivity or from redundancy (e.g., the frozen backbone still receives the raw bases).
The motivating problem in open GFMs is never evaluated. The paper is motivated by attention-based claims in open GFMs such as DNABERT, Nucleotide Transformer and SpliceBERT, yet none of these models is tested. The analysis covers only the attention of a read head the authors added to their own small pretrained Mamba model. It therefore cannot directly test whether the attention-based claims made about open GFMs hold.
Limited novelty. The core problem, that attention is not causal evidence, is well established in NLP (e.g., Jain & Wallace, 2019). The method's components also have direct precedents; (1) hashed k-mer/n-gram memory (Engram; Gengram for DNA); (2) attention over non-contextual per-token vectors (Backpack LMs); (3) targeted removal of attention contributions (Attention Knockout https://arxiv.org/pdf/2304.14767; Mamba Knockout). The slot-vs-address framing and the address-level dependence test are useful, but the contribution is mainly an application of existing ideas to genomics.
Questions for authors
-
Positive control. Can you show a case where source or value removal at an annotated site does reduce performance, for example a planted label-determining motif? Without one, how do you distinguish "no dependence" from an insensitive test? Plausible redundant routes include the frozen backbone, which still receives the raw bases, and other splice signals such as the polypyrimidine tract.
-
Open GFMs. Your motivation is attention-based claims in DNABERT, Nucleotide Transformer and SpliceBERT, but these models are not tested. Could fixed-attention value zeroing be applied to the heads those works interpreted, or at least to first-layer heads, where values reflect token identity? If not, what specifically prevents adapting your test to them?
-
Comparison with SAEs. Sparse-autoencoder analyses of Evo 2 identify features for exon–intron boundaries and report that ablating biologically aligned features increases model loss. What does your addressable memory establish that ablating such features in an existing DNA model cannot? Could an SAE feature ablation at acceptor sites serve as the positive control your test currently lacks?