acceptodds
Desk rejected at ICLR 2027

Looking Is Not Depending: Testing Positional Attention with an Addressable k-mer Memory in DNA Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained DNA models are widely used to predict regulatory activity, chromatin state, and splicing. As accuracy improves, it remains unclear whether predictions rely on task-relevant biological signals or on incidental correlations in the data. One common form of evidence, attention concentrated at functional elements, shows where the model looks but not what the prediction depends on. Testing this distinction requires connecting attention at annotated positions with interventions on the corresponding internal representations. To enable this, we present LIND, a frozen Mamba backbone with an explicit -mer memory and a local read head. Attention in LIND corresponds to memory sources selected by genomic address, so matched interventions and controls test positional explanations while the input sequence and parameters remain unchanged. To avoid the influence of dataset artifacts, the aforementioned tests were conducted under conditions where LIND achieved relative improvements across 16 Nucleotide Transformer tasks. At splice acceptors, attention is enriched 1.54-fold at the canonical AG, yet removing the underlying sources leaves AUROC essentially unchanged, opposite to the prespecified direction, while full memory removal costs 0.281. Attention looks at the AG in this Mamba case, yet silencing these sources does not reduce AUROC. In this model, looking is not depending. Our code can be found in the Supplementary Materials accompanying this paper.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
attention interpretability, DNA sequence models, genomic language models, intervention-based evaluation, k-mer memory, state space models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6yuPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper explains the problem in using attention as a evidence for GFM's biological understanding. The authors point out that those attention is not a causal evidence, and show "AG" at the splice acceptor as an example. Attention is enriched at AG, but removing those sources does not reduce AUROC. To test this they build their own model (LIND), a frozen Mamba backbone with an added k-mer memory and a local read head, so the attended sources can be removed exactly.

The paper explains the problem in using attention as a evidence for GFM's biological understanding. The authors point out that those attention is not a causal evidence, and show "AG" at the splice acceptor as an example. Attention is enriched at AG, but removing those sources does not reduce AUROC. To test this they build their own model (LIND), a frozen Mamba backbone with an added k-mer memory and a local read head, so the attended sources can be removed exactly.

My main concern is the scope. That attention is correlation and not causal evidence is already widely known in the NLP community. So the problem itself is narrow and largely settled, and the contribution is also narrow. The whole result rests on one case, AG on one checkpoint. AG was the only one of the five families that passed their enrichment test. It does not hold on other seeds. Also, the model does not use AG even when the input is mutated. So the null result was close to expected.

I also question the novelty and validity of the method. The components already exist k-mer/n-gram memory (Engram, Gengram), Backpack-style attention, attention knockout and Mamba knockout. The local, attention-matched test (source and value removal) is never shown to detect dependence at a site where dependence is known to exist, since there is no positive control. The paper never validates the site-level test on a known case. Without this, I cannot tell whether "not depending" means the attention is misleading or the test is not sensitive enough.

Finally, the method is not applied to any open GFM such as DNABERT, NT or SpliceBERT, even though these motivate the paper. I am not convinced this approach can tell us whether biological understanding exists in those models. SAE-based analysis (e.g. Evo 2, InterPLM) already works on real models at the concept level and has shown that ablating features matters. The paper should justify or discuss these methods and explain what its approach provides that SAE cannot.

Critical strengths

The paper states clearly what each operation changes (source removal, value removal, output masking). This matters because it is the paper's main methodological contribution --- a clean way to connect an attention measurement to an intervention on the same object, which most attention-based analyses in genomics lack.

Critical weaknesses

The method is not validated. There is no positive control. Whole-memory removal and some output-masking windows do change predictions. However, the attention-matched site-level test is never shown to detect dependence at a site where dependence is known to exist. A null result at AG therefore cannot be distinguished from limited sensitivity or from redundancy (e.g., the frozen backbone still receives the raw bases).

The motivating problem in open GFMs is never evaluated. The paper is motivated by attention-based claims in open GFMs such as DNABERT, Nucleotide Transformer and SpliceBERT, yet none of these models is tested. The analysis covers only the attention of a read head the authors added to their own small pretrained Mamba model. It therefore cannot directly test whether the attention-based claims made about open GFMs hold.

Limited novelty. The core problem, that attention is not causal evidence, is well established in NLP (e.g., Jain & Wallace, 2019). The method's components also have direct precedents; (1) hashed k-mer/n-gram memory (Engram; Gengram for DNA); (2) attention over non-contextual per-token vectors (Backpack LMs); (3) targeted removal of attention contributions (Attention Knockout https://arxiv.org/pdf/2304.14767; Mamba Knockout). The slot-vs-address framing and the address-level dependence test are useful, but the contribution is mainly an application of existing ideas to genomics.

Questions for authors
  1. Positive control. Can you show a case where source or value removal at an annotated site does reduce performance, for example a planted label-determining motif? Without one, how do you distinguish "no dependence" from an insensitive test? Plausible redundant routes include the frozen backbone, which still receives the raw bases, and other splice signals such as the polypyrimidine tract.

  2. Open GFMs. Your motivation is attention-based claims in DNABERT, Nucleotide Transformer and SpliceBERT, but these models are not tested. Could fixed-attention value zeroing be applied to the heads those works interpreted, or at least to first-layer heads, where values reflect token identity? If not, what specifically prevents adapting your test to them?

  3. Comparison with SAEs. Sparse-autoencoder analyses of Evo 2 identify features for exon–intron boundaries and report that ablating biologically aligned features increases model loss. What does your addressable memory establish that ablating such features in an existing DNA model cannot? Could an SAE feature ablation at acceptor sites serve as the positive control your test currently lacks?

Reviewer_Jzw7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LIND, a frozen Mamba-based DNA model with an explicit k-mer memory and a local attention read head. Address-matched interventions test whether attended sources support predictive performance. The read head improves performance across 16 retained NT tasks. In the main splice-acceptor case, attention is enriched at AG, yet removing the corresponding sources or values does not reduce AUROC. Additional controls distinguish source removal, fixed-attention value ablation, and output masking.

Critical strengths
  1. The addressable k-mer memory enables precise positional interventions. The fixed-attention value ablation provides a useful control for the effects of source removal.
  2. The read head improves performance across all 16 retained tasks, with additional evidence of predictive benefit on a convolutional backbone.
Critical weaknesses
  1. The conceptual advance over prior work is limited. Serrano and Smith (2019) already examined intermediate representation erasure, while Kobayashi et al. (2020) showed why attention weights alone do not capture contributions. The addressable memory provides a clearer genomic intervention target, but the paper does not sufficiently establish the new explanatory capability or mechanistic insight it adds.
  2. The complete alignment–intervention finding remains concentrated in one splice-acceptor setting within the proposed architecture. Additional seeds do not consistently reproduce AG enrichment, and the convolutional experiment establishes predictive transfer rather than replication of the full finding (Appendices G.1 and J). This limits the broader significance of the central result.
  3. The performance comparison does not isolate the architectural contribution. The read-head arm has more trainable parameters than the classifier-only control, and the main task table uses one training seed. The gains establish utility under this setup, but not an advantage over adaptations of comparable capacity.
Questions for authors
  1. What explanatory capability does the addressable memory add beyond existing representation-erasure methods, apart from its explicit genomic mapping?
  2. Do existing analyses help explain why AG enrichment varies across seeds and why local value removal has different effect directions between the Mamba and convolutional backbones?
Reviewer_mQk6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines interpretability in deep learning for genomics, asking whether enrichment of attention on functional elements, such as the canonical AG dinucleotide at the splice acceptor, demonstrates that a prediction truly depends on that position. The authors propose a frozen Mamba backbone combined with explicit k-mer memory and a local attention read head. In this design, every attention read source carries a genomic address, which makes it possible to perform targeted interventions on the exact object that attention measures while keeping both the input sequence and the model parameters unchanged. Together with a preregistered evaluation protocol based on four criteria, which the authors call PROTOCOL-F, this setup yields an unambiguous picture. The authors find that attention at the splice acceptor AG is enriched by a factor of 1.54, and that this enrichment replicates on independent samples, yet silencing these attention sources leaves AUROC essentially unchanged. Removing the memory altogether, in contrast, costs 0.281 AUROC. The authors take these findings as a warning against the widespread practice in genomics of treating attention alignment as biological evidence.

Critical strengths
  1. Exploring the interpretability of genomic foundation models is interesting and important. Compared with the current trend of benchmarking on a few existing datasets, I believe that investigating the interpretability and counterfactual reasoning abilities of genomic models is considerably more important.

  2. The author attempts to demonstrate that certain biologically interpretable analyses are worth encouraging.

Critical weaknesses

I think the topic this paper attempts to address interesting, and I would very much like to see more work in the field discussing the interpretability of genomic models. Unfortunately, the experiments the authors present, the biological mechanisms their work is built on, and the authors' understanding of genomics (at least as demonstrated in the paper) do not match the conclusions they intend to draw, the points they aim to make, or the ambition of the paper.

In details:

  1. The choice of genomic models is inappropriate. Purely sequence-based genomic models contain only sequence information and no information on epigenetic modifications, so they can naturally only memorize certain key sequence features. The authors should in fact include, as controls, models that carry genomic epigenetic modification information, such as GET and scDNAm-GPT [1, 2] (ATAC data for the former and DNA methylation for the latter). In addition, models with ultra long inputs, such as Enformer [3], which takes DNA sequences of roughly 200 kbp as input, should also be part of the discussion. The interpretability of these models (the association between attention and key biological regions) may well be stronger. At present, the authors discuss only short pure DNA sequences, with inputs of only around 600 bp.

  2. The complete analysis pipeline the authors present (namely, attention enrichment → replication on independent samples → sequence controls → removal of internal memory sources and input replacement interventions) is carried out only for splice sites and the "AG" dinucleotide within them, so the evidence is too weak. Far too few biological elements receive the full analysis.

  3. These two points reduce to the following: first, too few models are analyzed (the study lacks genomic models with epigenetic modification information as controls), and second, too few functional elements receive the full analysis. Yet on top of this limited evidence, the authors go on to elaborate some very sweeping concepts, for example by putting forward grand notions in the title and main text that attempt to define the entire field, such as "Looking Is Not Depending in DNA Models" and the "identity dependence gap."

[1] A foundation model of transcription across human cell types [2] scDNAm-GPT Captures Genome-wide CpG Dependencies in Single-cell DNA methylomes to Revolutionize Epigenetic Analysis [3] Effective gene expression prediction from sequence by integrating long-range interactions

Questions for authors

Please the authors reply to the Critical Weaknesses which I propose. I think the authors should add experiments, that are extending the analysis to more models and more genomic elements. And the claims made in the paper should also be scaled back. I will then assess the overall score of the paper based on the authors' response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.