- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper distills three frozen DNA language models (GENERator-1.2B, GenomeOcean-4B, NT-2.5B-multi) into compact students with 136M and 377M parameters (ModernGENA backbone). The teachers use different tokenizers, so teacher and student tokens are matched by the nucleotide spans they share. Training combines three losses --- per-token feature matching, relational, and summary. The distillation targets come from intermediate teacher layers, chosen as the first layer to reach 95% of peak linear-probe performance. A second stage post-trains on 3.29M labeled sequences from 56 datasets. Its contrastive target is an tilted version of an EMA reference model's similarity distribution, held to a KL budget.
Critical strengths
A strong, compact DNA encoder with careful evaluation. A 136M model has the best average across 54 tasks under one consistent frozen-feature protocol. Benchmark test regions are masked from distillation, and BEND-overlapping intervals are removed from post-training (0 overlaps after filtering). This is practically useful, and the leakage control is great.
The post-training recipe transfers to other backbones. The same post-training improves BEND for DNABERT-2, NT-v2 and ModernGENA (Table A11). This contribution stands independently of the distillation stage.
Critical weaknesses
- The paper does not show that distillation solves the problem it poses. Distillation from three billion-scale teachers is costly. To justify it, the paper must show something a directly trained compact model would not deliver. There are two ways, and neither is demonstrated:
- Strong in every task family. The stated problem is that no single model dominates. Yet where a teacher specializes, the student still trails it: BEND disease variants: .7523 vs NT-multi .7723 (Table 2). Fine-tuned splice tasks: L_d .935–.942 vs GENERator .961–.973 (Table A12). BEND mean score: a tie with NT-multi (.6363 vs .6373). The result implies better average, not consolidated specialties.
- Better than a matched student trained without external teachers. The gain over the initial checkpoint (ModernGENA-L .5086 → .5864, Revised-NT) mixes three changes: (1)the teachers, (2) 1.1B new training tokens on a 348-Gbp corpus, (3) and a new positional encoding. The ablation (Table 3) removes one distillation loss at a time (L_MTD, L_rel or L_sum); every variant still learns from all three teachers through the remaining losses. The losses themselves may also drive the gain, whoever the teacher is: matching intermediate-layer features, relational structure and pooled summaries can improve a model even when it is its own teacher (Born-Again Networks, ICML 2018). This matters because, by the paper's own evidence, size predicts DNA LM performance weakly. Caduceus-PS (7.7M) beats NT-v2 (500M) on Revised-NT (Table 1), and the paper cites Caduceus at 2M param as within 3% of Evo-1 at 7B. "Outperforms teachers 9–29× larger" is therefore weak evidence for distillation specifically.
- The teachers are scored at their final layer, but the student learns from earlier layers (L10 ...), the earliest to reach 95% of peak linear-probe performance. Where a teacher's probe performance falls after its peak, as for GENERator on splice tasks (Fig. A8), its final-layer score understates it. Under full fine-tuning the lead does not hold on Revised-NT (Table A12): B_d (.6680) is below GENERator (.6744) and NT-multi (.6746), and L_d (.6811) ties them. The paper attributes this to model capacity. An alternative is that the linear-probe lead reflects probe-friendly geometry inherited from the teachers' intermediate layers. Question 2 would distinguish the two.
This weakness determines my rating. With the controls in Questions 1 and clear per-family parity with the teachers, I would move to weak accept.
Questions for authors
- For the 136M Base model only, can you run two controls? Both should use the same data, token budget, positional change and all three losses as the three-teacher run:
- Self-distillation: the frozen initial ModernGENA-base checkpoint as the only teacher, read at its own earliest layer reaching 95% of peak (your readout rule). This holds the objective fixed and removes only the external knowledge.
- Single-teacher distillation: GENERator alone, your strongest teacher on Revised-NT and GUE (Table 1).
Each run costs no more than your existing distillation run. Together they would show what the external teachers, and their combination, contribute. Please report results per task family on all four benchmarks for these two runs and the three-teacher model. Please also report the GPU-hours of the original distillation run.
- Can you report teacher results at their distillation layers, pooled the same way as the student, and give a per-task-family comparison against the best teacher? This needs linear probes only.
10 comments
Rebuttal for Reviewer jASY [2/3]
(2) Matched controls (136M Base)
All runs share the data, the 1.1B-token budget and nucleotide-coordinate RoPE; only the teacher signal and the initialization differ. We report the three linear-probing benchmarks here; BEND is our held-out benchmark for evaluating post-training (Sec. 4.1).
Table R1. Revised-NT family means (MCC). Per-task results in Appendix Tables A15–A16.
| Family (# tasks) | ModernGENA-B | Continued MLM | Scratch distill. | (a) Self-distill. | (b) GENERator only | 3 teachers (Ours) |
|---|---|---|---|---|---|---|
| Regulatory (5) | 0.6081 | 0.6185 | 0.6089 | 0.6052 | 0.6438 | 0.6401 |
| Splice (3) | 0.5024 | 0.5744 | 0.4956 | 0.4989 | 0.6199 | 0.6420 |
| Histone (10) | 0.4768 | 0.4867 | 0.5264 | 0.4682 | 0.5164 | 0.5254 |
| Overall | 0.5175 | 0.5379 | 0.5442 | 0.5114 | 0.5690 | 0.5767 |
Table R2. GUE family means (MCC). Overall = mean of the 6 category means (virus_covid excluded).
| Family | ModernGENA-B | Continued MLM | Scratch distill. | (a) Self-distill. | (b) GENERator only | 3 teachers (Ours) |
|---|---|---|---|---|---|---|
| EMP | 0.4308 | 0.4909 | 0.4794 | 0.4467 | 0.5346 | 0.5358 |
| TF-M | 0.6219 | 0.6507 | 0.5749 | 0.6462 | 0.6843 | 0.6836 |
| TF-H | 0.5984 | 0.6340 | 0.6072 | 0.6179 | 0.6531 | 0.6570 |
| PD | 0.7563 | 0.7838 | 0.7823 | 0.7642 | 0.8052 | 0.8181 |
| CPD | 0.5695 | 0.6082 | 0.5939 | 0.6010 | 0.6250 | 0.6260 |
| SSP | 0.4233 | 0.4772 | 0.4369 | 0.3632 | 0.6004 | 0.6031 |
| Overall | 0.5667 | 0.6075 | 0.5791 | 0.5732 | 0.6504 | 0.6539 |
Table R3. Genomic Benchmarks per-task results (ACC). Overall = mean over all 9 tasks. Note that Genomic Benchmark does not have pre-defined family.
| Task | ModernGENA-B | Continued MLM | Scratch distill. | (a) Self-distill. | (b) GENERator only | 3 teachers (Ours) |
|---|---|---|---|---|---|---|
| Mouse EE | 0.7959 | 0.8529 | 0.8174 | 0.8017 | 0.8603 | 0.8570 |
| CI | 0.9227 | 0.9326 | 0.9140 | 0.9178 | 0.9405 | 0.9404 |
| EC | 0.7441 | 0.7554 | 0.7436 | 0.7409 | 0.7568 | 0.7571 |
| EE | 0.7466 | 0.7595 | 0.7746 | 0.7625 | 0.7746 | 0.7779 |
| ER | 0.6419 | 0.6529 | 0.7375 | 0.7184 | 0.6983 | 0.7289 |
| non-TATA | 0.8661 | 0.8770 | 0.8622 | 0.8706 | 0.8800 | 0.8788 |
| OCR | 0.6741 | 0.6843 | 0.7007 | 0.6846 | 0.7015 | 0.7039 |
| Human–Worm | 0.9607 | 0.9695 | 0.9635 | 0.9655 | 0.9765 | 0.9752 |
| ES | 0.7074 | 0.7240 | 0.7216 | 0.7116 | 0.7316 | 0.7361 |
| Overall | 0.7844 | 0.8009 | 0.8039 | 0.7971 | 0.8133 | 0.8173 |
- (a) Self-distillation from the frozen ModernGENA-base falls slightly below its own initialization (0.5114 vs. 0.5175), and DNA-RADIO-B$_d$ exceeds it on all 18 tasks. The losses, tokens and positional change alone do not produce the gain, so a Born-Again effect does not explain it.
- (b) GENERator alone reaches 0.5690. Three teachers reach 0.5767 and are higher on 12/18 tasks, gaining on splice (+0.022) and histone (+0.009). GENERator alone is slightly higher on regulatory (0.6438 vs. 0.6401).
Together, these controls attribute the gain to the external teacher signal rather than to the losses, data, or positional change. These results are added to Appendix Sec. E.6 (Ablation on initialization and teacher signal, Table A15; GENERator-only results on Revised-NT in Table A16).
(3) Compute
Each distillation run uses 8 NVIDIA A100-80GB GPUs, 80 GPU-hours per base-sized run, 100 GPU-hours for large sized model distillation and 30 GPU-hours for post-training (added to Appendix Sec. D.1.5).
Rebuttal for Reviewer jASY [3/3]
[W2, Q2] Teacher evaluation at their distillation layers. The teachers are scored at their final layer, but the student learns from their earlier layers, the earliest to reach 95% of peak linear-probe performance. Where a teacher's probe performance falls after its peak (e.g., GENERator on splice tasks, Fig. A8), its final-layer score understates it. Moreover, under full fine-tuning the student's lead does not hold on Revised-NT. The paper attributes this to model capacity, but an alternative is that the linear-probe lead reflects probe-friendly geometry inherited from the teachers' intermediate layers. Could the authors report teacher results at their distillation layers, pooled the same way as the student, with a per-task-family comparison against the best teacher?
We report every teacher at its distillation layer, pooled the same way as the student. In short:
- Distillation layers. DNA-RADIO-L$_d$ outperforms every teacher at its distillation layer on all three benchmarks (Table R4).
- Per task family. L$_d$ leads the best teacher on Revised-NT regulatory and histone tasks.
- Probe-friendly geometry. If the student's lead were inherited from the teachers' intermediate layers, the teachers read out at those layers would match it. They do not.
- Full fine-tuning. L$_d$ still has the highest Revised-NT average (Table A14).
1. Teachers at their distillation layers
Every teacher remains below L$_d$ on all three benchmarks. Table R4. Linear probing of each teacher at its final layer and at its distillation layer.
| Model (size) | Layer | Revised-NT | GUE | GenomicBench. |
|---|---|---|---|---|
| GENERator (1.2B) | final(L26)/readout(L10) | 0.5744/0.5788 | 0.6391/0.6471 | 0.8030/0.8076 |
| GenomeOcean (4B) | final(L24)/readout(L8) | 0.5586/0.5531 | 0.5721/0.5877 | 0.8013/0.8029 |
| NT-multi (2.5B) | final(L32)/readout(L15) | 0.5547/0.5724 | 0.6066/0.6218 | 0.8113/0.8068 |
| DNA-RADIO-B$_d$ (136M) | final | 0.5767 | 0.6539 | 0.8173 |
| DNA-RADIO-L$_d$ (377M) | final | 0.5864 | 0.6628 | 0.8194 |
2. Per-task-family comparison against the best teacher
- Revised-NT (Table R5). L$_d$ leads the best teacher on regulatory and histone.
- GUE (Table R6). L$_d$ leads the best teacher on 4 of 6 families (TF-M, TF-H, PD, CPD).
- Genomic Benchmark (Table R7). L$_d$ has the highest average. The best teacher varies by task.
Table R5. Revised-NT family means (MCC), teachers at their distillation layers.
| Family(#tasks) | GENERator(L10) | GenomeOcean(L8) | NT-multi(L15) | DNA-RADIO-B$_d$ | DNA-RADIO-L$_d$ |
|---|---|---|---|---|---|
| Regulatory(5) | 0.6400 | 0.6204 | 0.6437 | 0.6401 | 0.6560 |
| Splice(3) | 0.6444 | 0.5677 | 0.6316 | 0.6420 | 0.6426 |
| Histone(10) | 0.5286 | 0.5151 | 0.5190 | 0.5254 | 0.5347 |
| Overall | 0.5788 | 0.5531 | 0.5724 | 0.5767 | 0.5864 |
Table R6. GUE per task family (MCC), teachers at their distillation layers.
| Family | GENERator(L10) | GenomeOcean(L8) | NT-multi(L15) | DNA-RADIO-B$_d$ | DNA-RADIO-L$_d$ |
|---|---|---|---|---|---|
| EMP | 0.5151 | 0.5647 | 0.5391 | 0.5358 | 0.5444 |
| TF-M | 0.6428 | 0.5446 | 0.6005 | 0.6836 | 0.6880 |
| TF-H | 0.6432 | 0.6023 | 0.6065 | 0.6570 | 0.6505 |
| PD | 0.8027 | 0.7526 | 0.7895 | 0.8181 | 0.8217 |
| CPD | 0.6214 | 0.5498 | 0.6130 | 0.6260 | 0.6390 |
| SSP | 0.6573 | 0.5123 | 0.5824 | 0.6031 | 0.6330 |
| Average | 0.6471 | 0.5877 | 0.6218 | 0.6539 | 0.6628 |
Table R7. Genomic Benchmarks per task (accuracy), teachers at their distillation layers.
| Task | GENERator(L10) | GenomeOcean(L8) | NT-multi(L15) | DNA-RADIO-B$_d$ | DNA-RADIO-L$_d$ |
|---|---|---|---|---|---|
| MouseEE | 0.8384 | 0.8496 | 0.8426 | 0.8570 | 0.8702 |
| CI | 0.9492 | 0.9449 | 0.9313 | 0.9404 | 0.9457 |
| EC | 0.7376 | 0.7367 | 0.7270 | 0.7571 | 0.7593 |
| EE | 0.7937 | 0.7701 | 0.7850 | 0.7779 | 0.7814 |
| ER | 0.6860 | 0.6599 | 0.7739 | 0.7289 | 0.7141 |
| non-TATA | 0.8866 | 0.8714 | 0.8603 | 0.8788 | 0.8834 |
| OCR | 0.7280 | 0.7028 | 0.7089 | 0.7039 | 0.7127 |
| Human–Worm | 0.9685 | 0.9637 | 0.9607 | 0.9752 | 0.9772 |
| ES | 0.6806 | 0.7272 | 0.6712 | 0.7361 | 0.7306 |
| Average | 0.8076 | 0.8029 | 0.8068 | 0.8173 | 0.8194 |
3. Probe-friendly geometry and full fine-tuning
- Geometry. The teachers' distillation layers are exactly where any inherited probe-friendly geometry would be. Read out there, every teacher remains below L$_d$ on all three benchmarks (Table R4).
- Full fine-tuning. L$_d$ has the highest Revised-NT average (0.6811, vs. 0.6746 for NT-multi and 0.6744 for GENERator; Table A14). By family, it leads on histone (0.5904 vs. 0.5722 for GENERator), and trails NT-multi on regulatory (0.7072 vs. 0.7261) and GENERator on splice (0.9397 vs. 0.9679).
- Capacity. Fine-tuning updates the whole encoder, so its results also reflect task-specific adaptation and model capacity (Appendix Sec. E.5). Splice is where nearly all models gain the most and rankings change the most. Linear probing keeps the encoder frozen, so it isolates representation quality; under it, L$_d$ leads every teacher.
Tables R5–R7 and per-task Revised-NT results are added to Appendix Sec. E.2 (Table A10).
Rebuttal for Reviewer jASY [1/3]
We thank the reviewer for the thoughtful and constructive comments. We are encouraged that the reviewer views DNA-RADIO as a strong, compact, and practical DNA encoder supported by careful evaluation, appreciated our thorough leakage control, and recognized that our post-training recipe transfers across different backbones. We address each of the reviewer's concerns and questions in detail below.
We address the two points in order, then the reporting requests. In short:
(1) Strength across task families
No single teacher is best across Revised-NT families: NT-multi leads on regulatory, GENERator on splice and histone (W2/Q2, Table R5). At each teacher's distillation layer, DNA-RADIO-L$_d$ leads the best teacher on regulatory and histone, and trails GENERator on splice by 0.0018. Where the student trails a specialized teacher, the gap is small relative to the size gap:
Appendix Sec. F already states that the student does not surpass every teacher on every task.
(continued in [2/3])