- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper fine-tunes Nucleotide Transformer v2-500M (NT-v2) for transcription-factor (TF) binding prediction. It replaces the usual CLS-token or mean readout with attention pooling: a linear scorer gives each token a weight, and the sequence vector is the weighted sum of token embeddings. The softmax temperature of these weights is annealed, i.e. lowered linearly from 1.0 to 0.01 during training, so attention starts spread out and ends focused. The full system (TAFP-DNA) also uses reverse-complement (RC) handling: training on both DNA strands and averaging forward- and RC-strand logits at test time. It is reported as a 3-seed ensemble. An optional loss pushes attention toward known motif sites. Evaluation is on the 10 GUE TF-binding tasks (5 human, 5 mouse), scored by MCC (Matthews correlation coefficient, a balanced accuracy measure from −1 to 1):
Benchmark (Table 1): average MCC 0.7381 vs 0.7359 for the runner-up SPACE, claimed as state of the art. Ablations on the same backbone: focused pooling beats CLS by 0.044 and mean pooling by 0.034 (Fig. 2). RC handling adds 0.045 (Fig. 3). Annealing beats a fixed low temperature ("cold start") by 0.018 (Fig. 4). Motifs (mouse_3 only): with motif supervision, motifs recovered from attention match the known MAF profile (Fig. 6).
Critical strengths
Simple, cheap head with the right ablation design. The head is one linear scorer and needs no change to the backbone architecture. Swapping only the head on a fixed backbone is the correct way to test pooling.
Critical weaknesses
-
The core gains are confounded. Pooling vs RC and ensembling: the focused-pooling value in Fig. 2 (0.7381) is the full system with RC handling and a 3-seed ensemble. Focused pooling without RC scores 0.6936 (Sec. 4.3), below mean pooling (0.7044) and equal to CLS (0.6938). The paper does not say whether the CLS and mean rows also used RC handling. If not, the "pooling gain" is the RC gain. No seed-level variance is reported anywhere. Annealing rests on one small task: from Fig. 4, annealing is worse than cold start on all five human tasks (−0.004 to −0.014). It gains +0.189 on mouse_3, whose test set has 239 sequences. Without mouse_3, annealing is on average slightly worse than cold start. Table C1 agrees on human tasks (0.7069 annealed vs 0.7175 cold start). Motif evidence is circular or negative: without supervision, annealed attention finds a shortcut motif (AGKGAMA), while cold start finds a MAF-like one (ACTCAGC). So annealing hurts interpretability in the only case examined. With supervision, attention is trained toward MAF regex hits, so recovering MAF (TOMTOM E = 0.0026) is expected by construction. It also costs MCC (0.803 vs 0.867). Even where attention aligns with motifs, attention weights are not evidence that predictions depend on those positions (Jain & Wallace, NAACL 2019).
-
The SOTA claim is unsupported (an overclaim), and significance is narrow. The margin over SPACE is 0.0022 MCC with no uncertainty, and SPACE wins 5 of 10 tasks. The paper does not say whether baselines were re-run or copied, and several cells differ from published values. ILNN's human MCCs (−0.038 to 0.076) look like failed runs, which also distorts the "Std. ↓" row. TAFP-DNA alone gets RC test-time averaging and a seed ensemble. Separately, "can be added to pretrained DNA models" is shown on one backbone only. Each component is also established: attention pooling (Ilse et al., ICML 2018; Enformer's softmax pooling), temperature annealing (Gumbel-Softmax, ICLR 2017), RC augmentation with test-time averaging (Caduceus-Ph, ICML 2024) and attention supervision (Bao et al., EMNLP 2018). The contribution is their combination as a readout on one backbone for one task family.
Even if both are fixed, the remaining contribution is a readout head built from established components, evaluated on one task family with a modest gain. I would consider this a sound but incremental result, below the bar for ICLR, unless it generalized broadly across task families and backbones, or its attention recovered known motifs without supervision.
Questions for authors
Clarifications and requests that could change my assessment:
- Can you give a per-task table of {CLS, mean, multi, focused} × {no RC, RC train + test-time averaging}, each as a single-model seed mean ± sd (no ensemble), plus a learned head at fixed T = 1? This would separate the head, the schedule and RC handling.
Suggestions (not part of my decision):
- Showing the head comparison on a second backbone (e.g., DNABERT-2) would support the "drop-in for pretrained DNA models" claim; otherwise, please narrow it to NT-v2.
- For each Table 1 baseline, please state whether numbers were re-run or copied, and under which protocol. If they cannot be reconciled with published values, please withdraw "state of the art" and present the work as a pooling study.
- Is the mouse_3 cold-start drop a single-seed training failure? Please give per-seed results, and confirm that T_min and checkpoints were chosen on the GUE dev split, not the test split.
- Please cite and discuss Feng et al. (Nature Communications 2025), who benchmark summary-token, mean and max pooling of frozen embeddings for five DNA foundation models, including NT-v2-500M, across 52 binary classification datasets that include TF-binding tasks. For NT-v2, mean pooling was significantly better in 42 of 52 datasets (DeLong, p < 0.01). That readout choice matters for DNA models is therefore known; the paper's contribution is a learned, annealed head under fine-tuning.
- Position the head against its established components: attention-based pooling (Ilse et al., ICML 2018) and Enformer's learned softmax pooling (Avsec et al., Nature Methods 2021); temperature annealing (Gumbel-Softmax, ICLR 2017); attention supervision (Bao et al., EMNLP 2018); and attention as explanation (Jain & Wallace, NAACL 2019). Recast the RC module as adopted practice (post-hoc averaging, Zhou et al.; Caduceus-Ph, ICML 2024).
- State which tokens the sequence-end shortcut occupies.