acceptodds
Under review as a conference paper at ICLR 2027

TAFP-DNA: TEMPERATURE-ANNEALED FOCUSED POOLING FOR DNA FOUNDATION MODELS

Abstract

Fixed pooling strategies in current DNA foundation models poorly adapt to diverse Transcription Factor (TF) binding patterns and can make predictions vulnerable to sequence-level statistical biases such as Guanine-Cytosine (GC) content. To tackle these issues, we propose Temperature-Annealed Focused Pooling (TAFP), an attention pooling head with a temperature-scaled softmax. Inspired by simulated annealing, TAFP gradually lowers the temperature during fine-tuning to sharpen the attention distribution, moving from broad global aggregation to a focus on informative local regions; at inference, the temperature is kept fixed. When a known motif is available, an optional weak supervision loss aligns attention with motif sites. By exploiting the Reverse-Complement (RC) symmetry of DNA, RC training augmentation combined with averaging of forward- and RC-strand predictions yields RC-invariant predictions without an extra RC consistency loss. On ten TF binding prediction tasks from the Genome Understanding Evaluation (GUE) benchmark, the resulting TAFP-DNA achieves a state-of-the-art average Matthews Correlation Coefficient (MCC) of 0.7381 over seven baselines, and under the same backbone, focused pooling outperforms CLS and mean pooling by 0.044 and 0.034 MCC. With motif supervision, the motif recovered from attention weights matches the known MAF profile. TAFP is a lightweight module that can be added to pretrained DNA models without architectural changes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
DNA foundation models, focused pooling, Boltzmann attention, transcription factor binding, motif discovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bq41OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper fine-tunes Nucleotide Transformer v2-500M (NT-v2) for transcription-factor (TF) binding prediction. It replaces the usual CLS-token or mean readout with attention pooling: a linear scorer gives each token a weight, and the sequence vector is the weighted sum of token embeddings. The softmax temperature of these weights is annealed, i.e. lowered linearly from 1.0 to 0.01 during training, so attention starts spread out and ends focused. The full system (TAFP-DNA) also uses reverse-complement (RC) handling: training on both DNA strands and averaging forward- and RC-strand logits at test time. It is reported as a 3-seed ensemble. An optional loss pushes attention toward known motif sites. Evaluation is on the 10 GUE TF-binding tasks (5 human, 5 mouse), scored by MCC (Matthews correlation coefficient, a balanced accuracy measure from −1 to 1):

Benchmark (Table 1): average MCC 0.7381 vs 0.7359 for the runner-up SPACE, claimed as state of the art. Ablations on the same backbone: focused pooling beats CLS by 0.044 and mean pooling by 0.034 (Fig. 2). RC handling adds 0.045 (Fig. 3). Annealing beats a fixed low temperature ("cold start") by 0.018 (Fig. 4). Motifs (mouse_3 only): with motif supervision, motifs recovered from attention match the known MAF profile (Fig. 6).

Critical strengths

Simple, cheap head with the right ablation design. The head is one linear scorer and needs no change to the backbone architecture. Swapping only the head on a fixed backbone is the correct way to test pooling.

Critical weaknesses
  1. The core gains are confounded. Pooling vs RC and ensembling: the focused-pooling value in Fig. 2 (0.7381) is the full system with RC handling and a 3-seed ensemble. Focused pooling without RC scores 0.6936 (Sec. 4.3), below mean pooling (0.7044) and equal to CLS (0.6938). The paper does not say whether the CLS and mean rows also used RC handling. If not, the "pooling gain" is the RC gain. No seed-level variance is reported anywhere. Annealing rests on one small task: from Fig. 4, annealing is worse than cold start on all five human tasks (−0.004 to −0.014). It gains +0.189 on mouse_3, whose test set has 239 sequences. Without mouse_3, annealing is on average slightly worse than cold start. Table C1 agrees on human tasks (0.7069 annealed vs 0.7175 cold start). Motif evidence is circular or negative: without supervision, annealed attention finds a shortcut motif (AGKGAMA), while cold start finds a MAF-like one (ACTCAGC). So annealing hurts interpretability in the only case examined. With supervision, attention is trained toward MAF regex hits, so recovering MAF (TOMTOM E = 0.0026) is expected by construction. It also costs MCC (0.803 vs 0.867). Even where attention aligns with motifs, attention weights are not evidence that predictions depend on those positions (Jain & Wallace, NAACL 2019).

  2. The SOTA claim is unsupported (an overclaim), and significance is narrow. The margin over SPACE is 0.0022 MCC with no uncertainty, and SPACE wins 5 of 10 tasks. The paper does not say whether baselines were re-run or copied, and several cells differ from published values. ILNN's human MCCs (−0.038 to 0.076) look like failed runs, which also distorts the "Std. ↓" row. TAFP-DNA alone gets RC test-time averaging and a seed ensemble. Separately, "can be added to pretrained DNA models" is shown on one backbone only. Each component is also established: attention pooling (Ilse et al., ICML 2018; Enformer's softmax pooling), temperature annealing (Gumbel-Softmax, ICLR 2017), RC augmentation with test-time averaging (Caduceus-Ph, ICML 2024) and attention supervision (Bao et al., EMNLP 2018). The contribution is their combination as a readout on one backbone for one task family.

Even if both are fixed, the remaining contribution is a readout head built from established components, evaluated on one task family with a modest gain. I would consider this a sound but incremental result, below the bar for ICLR, unless it generalized broadly across task families and backbones, or its attention recovered known motifs without supervision.

Questions for authors

Clarifications and requests that could change my assessment:

  1. Can you give a per-task table of {CLS, mean, multi, focused} × {no RC, RC train + test-time averaging}, each as a single-model seed mean ± sd (no ensemble), plus a learned head at fixed T = 1? This would separate the head, the schedule and RC handling.

Suggestions (not part of my decision):

  1. Showing the head comparison on a second backbone (e.g., DNABERT-2) would support the "drop-in for pretrained DNA models" claim; otherwise, please narrow it to NT-v2.
  2. For each Table 1 baseline, please state whether numbers were re-run or copied, and under which protocol. If they cannot be reconciled with published values, please withdraw "state of the art" and present the work as a pooling study.
  3. Is the mouse_3 cold-start drop a single-seed training failure? Please give per-seed results, and confirm that T_min and checkpoints were chosen on the GUE dev split, not the test split.
  4. Please cite and discuss Feng et al. (Nature Communications 2025), who benchmark summary-token, mean and max pooling of frozen embeddings for five DNA foundation models, including NT-v2-500M, across 52 binary classification datasets that include TF-binding tasks. For NT-v2, mean pooling was significantly better in 42 of 52 datasets (DeLong, p < 0.01). That readout choice matters for DNA models is therefore known; the paper's contribution is a learned, annealed head under fine-tuning.
  5. Position the head against its established components: attention-based pooling (Ilse et al., ICML 2018) and Enformer's learned softmax pooling (Avsec et al., Nature Methods 2021); temperature annealing (Gumbel-Softmax, ICLR 2017); attention supervision (Bao et al., EMNLP 2018); and attention as explanation (Jain & Wallace, NAACL 2019). Recast the RC module as adopted practice (post-hoc averaging, Zhou et al.; Caduceus-Ph, ICML 2024).
  6. State which tokens the sequence-end shortcut occupies.
Reviewer_YfBMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the sequence-level aggregation issue of DNA foundation models in transcription factor (TF) binding prediction and proposes a token aggregation training strategy with a variable temperature, allowing the model to focus more on effective tokens during aggregation and thereby improving performance.

Critical strengths
  1. For the TF binding classification task on 101 bp, it's true that usually only a few local segments actually determine the label. Both CLS pooling and mean pooling mix in a lot of tokens unrelated to binding, so the author's motivation makes sense.
  2. Annealing the softmax temperature from high to low is logical. If you set a super low softmax temperature right from the start, the model would be forced to bet on very few positions while the token representations are still unstable, which can easily lead to bad local optima.
  3. Ablation experiments show that the boost brought by focused pooling is relatively clean, and its effectiveness has been somewhat verified.
Critical weaknesses
  1. The overall framework is more like a reasonable combination of existing training tricks. Its practicality is beyond doubt, but it doesn’t really bring many new insights. The core of TAFP is linear scoring attention pooling plus softmax temperature annealing. RC augmentation and RC test-time averaging are also mature strategies. Motif weak supervision just adds existing knowledge into the attention objective. The combination of the three can be quite effective. It’s a lightweight, plug-and-play pooling recipe suitable for DNA sequence classification, but it’s not a fundamental advance in modeling DNA foundation models or TF binding itself.
  2. The temperature scheduling strategy lacks thorough mechanistic justification. Currently, a global fixed schedule is used that linearly drops from 1.0 to 0.01. This design is intuitive, but the implementation details need more support, for example: a. Why must the endpoint be 0.01 and not a milder temperature? b. Why linear annealing, instead of cosine, piecewise, or performance-driven annealing? c. Do different TFs, dataset sizes, or motif densities require different degrees of focus?
  3. The paper points out in the introduction that DNA classification models may exploit statistical shortcuts like GC content, repeat sequences, or sequence ends. But the current method mainly shows that attention becomes more concentrated, and in one case weak supervision can pull attention back to known motifs. It doesn’t sufficiently show that focused pooling itself systematically reduces reliance on GC or other non-causal features. Sharper attention doesn’t necessarily mean more biologically sensible. Extremely sharp attention can easily lock onto a high GC segment, a padded boundary, or some dataset-specific pattern. In mouse 3, attention without weak supervision focuses on the sequence end, which exactly illustrates this risk.
  4. The weak supervision loss directly requires attention weights to be close to positions found via known motif scanning. So, after adding this loss, the model’s attention naturally matches the motif more closely, and using these high-attention positions to recover similar motifs is expected. In Fig. 6, this should not be described as de novo biological discovery.
Questions for authors

Please refer to the weaknesses.

Reviewer_jw7oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAFP-DNA improves transcription-factor binding prediction by changing how a DNA foundation model combines information across sequence positions. It gradually sharpens attention during fine-tuning and averages predictions across both DNA strands. Its strongest evidence is improvement over standard pooling methods, but its interpretability and overall state-of-the-art claims need more qualification.

Critical strengths

The method is simple to implement and inexpensive in added parameters, and annealing can prevent poor early attention choices.

Critical weaknesses
  1. The performance advantage over the strongest baseline is tiny. The difference from SPACE is minor. The paper does not provide confidence intervals or a significance test establishing that this gap is reliable. Its reported standard deviation measures variation across tasks, not uncertainty across training runs.

  2. The annealing advantage is heavily concentrated in one dataset. At the default endpoint, fixed low-temperature training outperforms annealing on all five human tasks. Using the reported averages, excluding mouse_3 makes annealing approximately 0.0013 MCC worse than cold start across the remaining nine tasks. The evidence therefore supports a task-dependent benefit rather than a universal improvement.

  3. A key baseline is missing for isolating the temperature schedule. The paper compares against mean pooling and an extreme fixed temperature of 0.01, but does not present a clear standalone comparison with ordinary attention pooling at a well-tuned fixed temperature. That would help determine how much benefit comes specifically from annealing.

  4. The figures in this paper can be improved.

Questions for authors

See above weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.