acceptodds
Under review as a conference paper at ICLR 2027

DNA-RADIO: Towards an Agglomerative DNA Embedding Model

Abstract

DNA language models (LMs) achieve strong genomic prediction performance, yet their strengths remain distributed across models, with no single model consistently dominating. Through layer-wise linear probing and centered kernel alignment, we find that downstream performance often saturates early and deeper layers contribute little to additional downstream-relevant signal, suggesting that these complementary capabilities can be consolidated into a compact representation. We introduce DNA-RADIO, an agglomerative embedding model that distills heterogeneous DNA LMs into a single student up to 29.4 smaller than its teachers. We then enrich its representations through post-training on large-scale labeled sequences while preserving inherited knowledge of distillation. Under a frozen-feature linear evaluation protocol, distillation using only unlabeled sequences improves average performance by up to 15.85% over its teachers and other sequence-only DNA LMs on held-out sequences, establishing state-of-the-art results despite its substantially smaller size. Post-training yields a further improvement on BEND, a benchmark held out in its entirety with its sequences excluded from training. These results establish multi-model distillation and post-training as an effective route to compact, unified, general-purpose DNA representations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
DNA Embedding Model, DNA language models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_jASYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper distills three frozen DNA language models (GENERator-1.2B, GenomeOcean-4B, NT-2.5B-multi) into compact students with 136M and 377M parameters (ModernGENA backbone). The teachers use different tokenizers, so teacher and student tokens are matched by the nucleotide spans they share. Training combines three losses --- per-token feature matching, relational, and summary. The distillation targets come from intermediate teacher layers, chosen as the first layer to reach 95% of peak linear-probe performance. A second stage post-trains on 3.29M labeled sequences from 56 datasets. Its contrastive target is an tilted version of an EMA reference model's similarity distribution, held to a KL budget.

Critical strengths

A strong, compact DNA encoder with careful evaluation. A 136M model has the best average across 54 tasks under one consistent frozen-feature protocol. Benchmark test regions are masked from distillation, and BEND-overlapping intervals are removed from post-training (0 overlaps after filtering). This is practically useful, and the leakage control is great.

The post-training recipe transfers to other backbones. The same post-training improves BEND for DNABERT-2, NT-v2 and ModernGENA (Table A11). This contribution stands independently of the distillation stage.

Critical weaknesses
  1. The paper does not show that distillation solves the problem it poses. Distillation from three billion-scale teachers is costly. To justify it, the paper must show something a directly trained compact model would not deliver. There are two ways, and neither is demonstrated:
  • Strong in every task family. The stated problem is that no single model dominates. Yet where a teacher specializes, the student still trails it: BEND disease variants: .7523 vs NT-multi .7723 (Table 2). Fine-tuned splice tasks: L_d .935–.942 vs GENERator .961–.973 (Table A12). BEND mean score: a tie with NT-multi (.6363 vs .6373). The result implies better average, not consolidated specialties.
  • Better than a matched student trained without external teachers. The gain over the initial checkpoint (ModernGENA-L .5086 → .5864, Revised-NT) mixes three changes: (1)the teachers, (2) 1.1B new training tokens on a 348-Gbp corpus, (3) and a new positional encoding. The ablation (Table 3) removes one distillation loss at a time (L_MTD, L_rel or L_sum); every variant still learns from all three teachers through the remaining losses. The losses themselves may also drive the gain, whoever the teacher is: matching intermediate-layer features, relational structure and pooled summaries can improve a model even when it is its own teacher (Born-Again Networks, ICML 2018). This matters because, by the paper's own evidence, size predicts DNA LM performance weakly. Caduceus-PS (7.7M) beats NT-v2 (500M) on Revised-NT (Table 1), and the paper cites Caduceus at 2M param as within 3% of Evo-1 at 7B. "Outperforms teachers 9–29× larger" is therefore weak evidence for distillation specifically.
  1. The teachers are scored at their final layer, but the student learns from earlier layers (L10 ...), the earliest to reach 95% of peak linear-probe performance. Where a teacher's probe performance falls after its peak, as for GENERator on splice tasks (Fig. A8), its final-layer score understates it. Under full fine-tuning the lead does not hold on Revised-NT (Table A12): B_d (.6680) is below GENERator (.6744) and NT-multi (.6746), and L_d (.6811) ties them. The paper attributes this to model capacity. An alternative is that the linear-probe lead reflects probe-friendly geometry inherited from the teachers' intermediate layers. Question 2 would distinguish the two.

This weakness determines my rating. With the controls in Questions 1 and clear per-family parity with the teachers, I would move to weak accept.

Questions for authors
  1. For the 136M Base model only, can you run two controls? Both should use the same data, token budget, positional change and all three losses as the three-teacher run:
  • Self-distillation: the frozen initial ModernGENA-base checkpoint as the only teacher, read at its own earliest layer reaching 95% of peak (your readout rule). This holds the objective fixed and removes only the external knowledge.
  • Single-teacher distillation: GENERator alone, your strongest teacher on Revised-NT and GUE (Table 1).

Each run costs no more than your existing distillation run. Together they would show what the external teachers, and their combination, contribute. Please report results per task family on all four benchmarks for these two runs and the three-teacher model. Please also report the GPU-hours of the original distillation run.

  1. Can you report teacher results at their distillation layers, pooled the same way as the student, and give a per-task-family comparison against the best teacher? This needs linear probes only.
10 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer jASY [1/3]

We thank the reviewer for the thoughtful and constructive comments. We are encouraged that the reviewer views DNA-RADIO as a strong, compact, and practical DNA encoder supported by careful evaluation, appreciated our thorough leakage control, and recognized that our post-training recipe transfers across different backbones. We address each of the reviewer's concerns and questions in detail below.

[W1, Q1] Unclear whether distillation itself drives the gains. Distilling from three billion-scale teachers is costly, so the paper needs to show something a directly trained compact model would not deliver, and neither of two possible cases is shown. (1) Consolidated specialties: where a teacher specializes, the student still trails it (e.g., BEND disease variants, fine-tuned splice tasks), which points to a better average rather than combined strengths. (2) Gain over a matched non-distilled student: the improvement over the initial checkpoint mixes teachers, new training tokens, and a new positional encoding. The Table 3 ablation also still learns from all three teachers through the remaining losses. Because the losses alone may account for the gain (cf. Born-Again Networks) and model size predicts DNA LM performance only weakly, “outperforms teachers 9–29× larger” is weak evidence for distillation specifically. For the 136M Base model, could the authors run two controls with the same data, token budget, positional change, and losses: (a) self-distillation from the frozen initial ModernGENA-base checkpoint, and (b) single-teacher distillation from GENERator alone? Please report per-task-family results on all four benchmarks for both controls and the three-teacher model, plus the GPU-hours of the original distillation run.

We address the two points in order, then the reporting requests. In short:

  • (1) The student consolidates specialties, though not on every task. No single teacher is best across task families. At each teacher's distillation layer, the student leads the best teacher on two of three Revised-NT families. Where it trails a specialized teacher, the gap is small.
  • (2) The gain comes from the external teachers. (a) Self-distillation under the same configuration does not improve the initialization consistently. (b) Three teachers outperform GENERator alone. Both controls rule out the losses, tokens and positional change as the source of the gain.

(1) Strength across task families

No single teacher is best across Revised-NT families: NT-multi leads on regulatory, GENERator on splice and histone (W2/Q2, Table R5). At each teacher's distillation layer, DNA-RADIO-L$_d$ leads the best teacher on regulatory and histone, and trails GENERator on splice by 0.0018. Where the student trails a specialized teacher, the gap is small relative to the size gap:

  • BEND variant-disease: 0.7523 vs. 0.7723 for NT-multi (97.4%), with 18.4× fewer parameters.
  • Fine-tuned splice: 0.9397 vs. 0.9679 for GENERator, while L$_d$ has the best fine-tuned Revised-NT average Table A14 (Table A12 in the original submission).

Appendix Sec. F already states that the student does not surpass every teacher on every task.

(continued in [2/3])

AuthorsOpenReview ↗

Rebuttal for Reviewer jASY [2/3]

(2) Matched controls (136M Base)

All runs share the data, the 1.1B-token budget and nucleotide-coordinate RoPE; only the teacher signal and the initialization differ. We report the three linear-probing benchmarks here; BEND is our held-out benchmark for evaluating post-training (Sec. 4.1).

Table R1. Revised-NT family means (MCC). Per-task results in Appendix Tables A15–A16.

Family (# tasks)ModernGENA-BContinued MLMScratch distill.(a) Self-distill.(b) GENERator only3 teachers (Ours)
Regulatory (5)0.60810.61850.60890.60520.64380.6401
Splice (3)0.50240.57440.49560.49890.61990.6420
Histone (10)0.47680.48670.52640.46820.51640.5254
Overall0.51750.53790.54420.51140.56900.5767

Table R2. GUE family means (MCC). Overall = mean of the 6 category means (virus_covid excluded).

FamilyModernGENA-BContinued MLMScratch distill.(a) Self-distill.(b) GENERator only3 teachers (Ours)
EMP0.43080.49090.47940.44670.53460.5358
TF-M0.62190.65070.57490.64620.68430.6836
TF-H0.59840.63400.60720.61790.65310.6570
PD0.75630.78380.78230.76420.80520.8181
CPD0.56950.60820.59390.60100.62500.6260
SSP0.42330.47720.43690.36320.60040.6031
Overall0.56670.60750.57910.57320.65040.6539

Table R3. Genomic Benchmarks per-task results (ACC). Overall = mean over all 9 tasks. Note that Genomic Benchmark does not have pre-defined family.

TaskModernGENA-BContinued MLMScratch distill.(a) Self-distill.(b) GENERator only3 teachers (Ours)
Mouse EE0.79590.85290.81740.80170.86030.8570
CI0.92270.93260.91400.91780.94050.9404
EC0.74410.75540.74360.74090.75680.7571
EE0.74660.75950.77460.76250.77460.7779
ER0.64190.65290.73750.71840.69830.7289
non-TATA0.86610.87700.86220.87060.88000.8788
OCR0.67410.68430.70070.68460.70150.7039
Human–Worm0.96070.96950.96350.96550.97650.9752
ES0.70740.72400.72160.71160.73160.7361
Overall0.78440.80090.80390.79710.81330.8173
  • (a) Self-distillation from the frozen ModernGENA-base falls slightly below its own initialization (0.5114 vs. 0.5175), and DNA-RADIO-B$_d$ exceeds it on all 18 tasks. The losses, tokens and positional change alone do not produce the gain, so a Born-Again effect does not explain it.
  • (b) GENERator alone reaches 0.5690. Three teachers reach 0.5767 and are higher on 12/18 tasks, gaining on splice (+0.022) and histone (+0.009). GENERator alone is slightly higher on regulatory (0.6438 vs. 0.6401).

Together, these controls attribute the gain to the external teacher signal rather than to the losses, data, or positional change. These results are added to Appendix Sec. E.6 (Ablation on initialization and teacher signal, Table A15; GENERator-only results on Revised-NT in Table A16).

(3) Compute

Each distillation run uses 8 NVIDIA A100-80GB GPUs, 80 GPU-hours per base-sized run, 100 GPU-hours for large sized model distillation and 30 GPU-hours for post-training (added to Appendix Sec. D.1.5).

AuthorsOpenReview ↗

Rebuttal for Reviewer jASY [3/3]

[W2, Q2] Teacher evaluation at their distillation layers. The teachers are scored at their final layer, but the student learns from their earlier layers, the earliest to reach 95% of peak linear-probe performance. Where a teacher's probe performance falls after its peak (e.g., GENERator on splice tasks, Fig. A8), its final-layer score understates it. Moreover, under full fine-tuning the student's lead does not hold on Revised-NT. The paper attributes this to model capacity, but an alternative is that the linear-probe lead reflects probe-friendly geometry inherited from the teachers' intermediate layers. Could the authors report teacher results at their distillation layers, pooled the same way as the student, with a per-task-family comparison against the best teacher?

We report every teacher at its distillation layer, pooled the same way as the student. In short:

  • Distillation layers. DNA-RADIO-L$_d$ outperforms every teacher at its distillation layer on all three benchmarks (Table R4).
  • Per task family. L$_d$ leads the best teacher on Revised-NT regulatory and histone tasks.
  • Probe-friendly geometry. If the student's lead were inherited from the teachers' intermediate layers, the teachers read out at those layers would match it. They do not.
  • Full fine-tuning. L$_d$ still has the highest Revised-NT average (Table A14).

1. Teachers at their distillation layers

Every teacher remains below L$_d$ on all three benchmarks. Table R4. Linear probing of each teacher at its final layer and at its distillation layer.

Model (size)LayerRevised-NTGUEGenomicBench.
GENERator (1.2B)final(L26)/readout(L10)0.5744/0.57880.6391/0.64710.8030/0.8076
GenomeOcean (4B)final(L24)/readout(L8)0.5586/0.55310.5721/0.58770.8013/0.8029
NT-multi (2.5B)final(L32)/readout(L15)0.5547/0.57240.6066/0.62180.8113/0.8068
DNA-RADIO-B$_d$ (136M)final0.57670.65390.8173
DNA-RADIO-L$_d$ (377M)final0.58640.66280.8194

2. Per-task-family comparison against the best teacher

  • Revised-NT (Table R5). L$_d$ leads the best teacher on regulatory and histone.
  • GUE (Table R6). L$_d$ leads the best teacher on 4 of 6 families (TF-M, TF-H, PD, CPD).
  • Genomic Benchmark (Table R7). L$_d$ has the highest average. The best teacher varies by task.

Table R5. Revised-NT family means (MCC), teachers at their distillation layers.

Family(#tasks)GENERator(L10)GenomeOcean(L8)NT-multi(L15)DNA-RADIO-B$_d$DNA-RADIO-L$_d$
Regulatory(5)0.64000.62040.64370.64010.6560
Splice(3)0.64440.56770.63160.64200.6426
Histone(10)0.52860.51510.51900.52540.5347
Overall0.57880.55310.57240.57670.5864

Table R6. GUE per task family (MCC), teachers at their distillation layers.

FamilyGENERator(L10)GenomeOcean(L8)NT-multi(L15)DNA-RADIO-B$_d$DNA-RADIO-L$_d$
EMP0.51510.56470.53910.53580.5444
TF-M0.64280.54460.60050.68360.6880
TF-H0.64320.60230.60650.65700.6505
PD0.80270.75260.78950.81810.8217
CPD0.62140.54980.61300.62600.6390
SSP0.65730.51230.58240.60310.6330
Average0.64710.58770.62180.65390.6628

Table R7. Genomic Benchmarks per task (accuracy), teachers at their distillation layers.

TaskGENERator(L10)GenomeOcean(L8)NT-multi(L15)DNA-RADIO-B$_d$DNA-RADIO-L$_d$
MouseEE0.83840.84960.84260.85700.8702
CI0.94920.94490.93130.94040.9457
EC0.73760.73670.72700.75710.7593
EE0.79370.77010.78500.77790.7814
ER0.68600.65990.77390.72890.7141
non-TATA0.88660.87140.86030.87880.8834
OCR0.72800.70280.70890.70390.7127
Human–Worm0.96850.96370.96070.97520.9772
ES0.68060.72720.67120.73610.7306
Average0.80760.80290.80680.81730.8194

3. Probe-friendly geometry and full fine-tuning

  • Geometry. The teachers' distillation layers are exactly where any inherited probe-friendly geometry would be. Read out there, every teacher remains below L$_d$ on all three benchmarks (Table R4).
  • Full fine-tuning. L$_d$ has the highest Revised-NT average (0.6811, vs. 0.6746 for NT-multi and 0.6744 for GENERator; Table A14). By family, it leads on histone (0.5904 vs. 0.5722 for GENERator), and trails NT-multi on regulatory (0.7072 vs. 0.7261) and GENERator on splice (0.9397 vs. 0.9679).
  • Capacity. Fine-tuning updates the whole encoder, so its results also reflect task-specific adaptation and model capacity (Appendix Sec. E.5). Splice is where nearly all models gain the most and rankings change the most. Linear probing keeps the encoder frozen, so it isolates representation quality; under it, L$_d$ leads every teacher.

Tables R5–R7 and per-task Revised-NT results are added to Appendix Sec. E.2 (Table A10).

Reviewer_Lg8SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DNA-RADIO consolidates three heterogeneous DNA language models into a compact encoder through multi-teacher feature distillation. It aligns tokens by nucleotide overlap and combines token-level, relational, and sequence-level objectives. A subsequent supervised stage adjusts embedding neighborhoods using KL-constrained targets and an EMA reference. The distilled models achieve competitive results across three benchmarks, and post-training improves performance on BEND.

Critical strengths
  1. Consolidating complementary DNA models into a compact embedding model is a useful direction. The layer-wise probing analysis provides an interesting empirical motivation.
  2. The evaluation covers multiple task types, and the compact students remain competitive with substantially larger teachers. Additional full fine-tuning and cross-backbone post-training results strengthen the practical evidence.
Critical weaknesses
  1. The methodological advance is not sufficiently isolated. Much of the distillation framework builds on RADIO and PHI-S. Without single-teacher distillation controls or a comparison against standard supervised contrastive learning under matched conditions, it remains unclear how much benefit comes specifically from teacher complementarity and the proposed post-training objective.
  2. The teacher comparison does not fully establish superiority over teacher representations. Distillation uses selected intermediate teacher layers, whereas the main comparisons read from the final layer. Given the paper’s own evidence of intermediate-layer advantages, validation-selected teacher readouts are needed to assess this claim more convincingly.
  3. The evidence for general-purpose transfer is promising but not yet sufficient. The benchmarks cover several task types, but there is no direct motif-sensitivity evaluation. Such evidence is particularly relevant because overlap-based token aggregation may smooth fine-grained information; aggregate downstream performance alone does not establish that this information is preserved.
  4. Some reporting and isolation claims need clarification. Table 2 lists the post-trained model as 377M, while the text and Table A9 indicate the 136M Base model. Appendix C.2.2 also states that BEND chromatin accessibility was not included in coordinate-based overlap filtering. These details should be reconciled with the model-size and held-out evaluation claims.
Questions for authors
  1. Are there existing single-teacher, validation-selected teacher-layer, or standard supervised contrastive results that help separate the contributions of the proposed components?
  2. Please clarify the post-trained model size and the scope of BEND overlap protection, particularly for chromatin accessibility and records without genomic coordinates.
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Lg8S [1/3]

We thank the reviewer for the thoughtful and constructive review. We are encouraged that the reviewer found value in consolidating complementary DNA language models into a single compact embedding model, recognized our evaluation across diverse task types, and noted that our smaller model remains competitive with substantially larger models, with the full fine-tuning results further strengthening the practical evidence. We address each of the reviewer's concerns and questions below.

[W1, Q1] The methodological advance is not sufficiently isolated. Much of the distillation framework builds on RADIO and PHI-S. Without single-teacher distillation controls or a comparison against standard supervised contrastive learning under matched conditions, it remains unclear how much benefit comes specifically from teacher complementarity and the proposed post-training objective.

We ran both requested controls. In short:

  • (A) Teacher complementarity. Each single-teacher student excels on different tasks, and combining the teachers gives the best overall performance (Table R1).
  • (B) Post-training objective. Our objective outperforms SupCon on all BEND tasks (Table R2).

Relation to RADIO and PHI-S. We adopt the RADIO-style agglomerative setup and PHI-S normalization by design. Two components are specific to our setting. First, DNA teachers use incompatible tokenizers (k-mer, BPE), so token indices are not comparable across models; we align them by the nucleotide intervals each token covers (Eq. 1). Second, the information-constrained relational post-training stage (Sec. 3.3) has no counterpart in RADIO. The two controls below isolate the contribution of the teacher signal and of this post-training objective.

(A) Single-teacher distillation

The ablation supports the complementarity (each teacher has complementary roles) hypothesis. Each single-teacher student excels on different tasks, and combining the teachers gives the best overall performance (Table R1).

  • Complementarity. Among the single-teacher students, the best one differs from task to task. The GENERator student leads on 8/18 tasks, including all three promoter tasks. The NT-2.5B-multi student leads on 8/18, including half of the histone marks. The GenomeOcean student leads on the remaining 2/18.
  • Consolidation. Multi-teacher distillation achieves the best overall MCC (0.5767, vs. 0.5696 for the best single teacher). It is also best on 11/18 tasks, more than any single-teacher student.

Table R1. Single- vs. multi-teacher distillation on Revised-NT (MCC). We distill the student from each of the three teachers used for DNA-RADIO (GENERator, GenomeOcean and NT-2.5B-multi) individually, and compare it with multi-teacher distillation (DNA-RADIO, ours). Best per row in bold.

GENERatorGenomeOceanNT-2.5B-multiMulti-teacher (Ours)
H2AFZ0.43160.44210.44010.4419
H3K27ac0.46300.46090.46930.4839
H3K27me30.57520.56290.56640.5790
H3K36me30.56470.55980.59260.5977
H3K4me10.44970.44030.44660.4575
H3K4me20.50190.51640.52040.5240
H3K4me30.61780.62270.61680.6233
H3K9ac0.52490.50150.52280.5277
H3K9me30.42510.38230.42550.3964
H4K20me10.60980.60900.62040.6231
Enhancers0.49790.49720.50600.4973
Enhancers types0.38130.35700.36890.3744
Promoter all0.75960.73320.74040.7581
Promoter no TATA0.78180.76320.76040.7608
Promoter TATA0.79850.74210.77010.8101
Splice sites acceptors0.70020.66200.70340.7072
Splice sites all0.43220.45740.47230.4969
Splice sites donors0.72740.69120.70980.7218
Overall0.56900.55560.56960.5767

(B) Supervised contrastive learning (SupCon)

Both objectives start from DNA-RADIO-B$_d$ with the same corpus, batches, τ\tau, optimizer and steps, evaluated on held-out BEND. SupCon pulls all same-label pairs together; ours revises the distilled neighborhood toward labels within a KL budget δ\delta, preserving teacher-derived structure the labels do not capture. Our objective outperforms SupCon on all tasks. SupCon even falls below the distilled model on gene finding and both variant-effect tasks, suggesting that it overwrites structure learned from the teachers (variant-effect tasks are zero-shot, so it supports our claim). We added these results as an additional ablation study in Sec. 4.3 and Table 3.

Table R2. Post-training objectives under matched conditions on BEND. MCC for gene finding, AUPRC for enhancer annotation, AUROC otherwise; variant-effect tasks are zero-shot.

SupConInformation budget (Ours)
Gene finding0.52020.6137
Enhancer annot.0.03830.0433
Chromatin access.0.76170.7647
Histone mod.0.77910.7826
CpG methyl.0.93010.9303
Var. effect expr.0.48380.5671
Var. effect disease.0.64980.7523
AuthorsOpenReview ↗

Rebuttal for Reviewer Lg8S [2/3]

[W2] The teacher comparison does not fully establish superiority over teacher representations. Distillation uses selected intermediate teacher layers, whereas the main comparisons read from the final layer. Given the paper’s own evidence of intermediate-layer advantages, validation-selected teacher readouts are needed to assess this claim more convincingly.

Table R3 below reports each teacher at both layers. DNA-RADIO-L$_d$ outperforms every teacher at either layer on all three benchmarks. DNA-RADIO-B$_d$ does so on GUE and Genomic Benchmark. These results are added to the revised Appendix Sec. E.2 with paragraph Teacher performance on pooling layers. and Table A10 with detailed sub-category performance.

Table R3. Revised-NT, GUE and Genomic Benchmark performance of each teacher, measured at the intermediate layer used as the distillation target and final layer for reference.

Model (size)LayerRevised-NTGUEGenomic Bench.
GENERator (1.2B)final (L26) / readout (L10)0.5744 / 0.57880.6391 / 0.64710.8030 / 0.8076
GenomeOcean (4B)final (L24) / readout (L8)0.5586 / 0.55310.5721 / 0.58770.8013 / 0.8029
NT-multi (2.5B)final (L32) / readout (L15)0.5547 / 0.57240.6066 / 0.62180.8113 / 0.8068
DNA-RADIO-B$_d$ (136M)final0.57670.65390.8173
DNA-RADIO-L$_d$ (377M)final0.58640.66280.8194

[W3] The evidence for general-purpose transfer is promising but not yet sufficient. The benchmarks cover several task types, but there is no direct motif-sensitivity evaluation. Such evidence is particularly relevant because overlap-based token aggregation may smooth fine-grained information; aggregate downstream performance alone does not establish that this information is preserved.

Two results in the submission test information preservation directly. (Appendix Sec. D.1.1 and Sec. C.4.4)

(A) Composition-controlled tasks: does motif information survive aggregation?

In these tasks, the negatives share the positives' composition or motif, so composition alone cannot separate the classes. Each pair shares the tokenizer and initial weights and differs only by the distillation stage (incl. nucleotide-coordinate RoPE, Appendix Sec. D.1.1). In that stage, every token-level target is built by Eq. 1. If Eq. 1 smoothed out fine-grained information, these scores should drop; all six rise. On TF-M, where composition is identical, DNA-RADIO-Ld and -Bd rank 1st and 2nd of 18 models and beat the best teacher (GENERator, 0.6149).

Table R4. Distillation-only models vs. their ModernGENA initialization (same tokenizer). Frozen linear probe, MCC, mean of 5 seeds.

Task (negative construction)ModernGENA-B → DNA-RADIO-BdModernGENA-L → DNA-RADIO-LdSource
GUE TF-M (dinucleotide-shuffled, same composition)0.6219 → 0.68360.6480 → 0.6880Table 1, A8
GUE TF-H (length/GC-matched)0.5984 → 0.65700.6359 → 0.6505Table 1, A8
GUE PD-tata (negatives also contain TATA)0.5454 → 0.65960.5190 → 0.6625Table A8

(B) Zero-shot single-nucleotide test: is fine-grained information in the representation itself?

Ref/alt embeddings are compared by cosine distance at the variant position (Appendix C.4.4). This directly measures whether a single-base change is reflected in the representation, without any training. The same-tokenizer baseline is at chance level. After distillation, the model surpasses two of its three teachers and trails only NT-multi 2.5B. Distillation through Eq. 1 does not erase nucleotide-level information; it strengthens it.

Table R5. BEND Var. effect disease, zero-shot AUROC. DNA-RADIO-Bd is distillation-only; BEND is held out from all training (Appendix C.2.2).

ModernGENA-B → DNA-RADIO-Bd → DNA-RADIO-BNT-multi 2.5B (teacher)GENERator (teacher)GenomeOcean (teacher)Source
0.4921 → 0.7368 → 0.75230.77230.67370.5227Table 2, A9, A11
AuthorsOpenReview ↗

Rebuttal for Reviewer Lg8S [3/3]

[W4, Q2] Some reporting and isolation claims need clarification. Table 2 lists the post-trained model as 377M, while the text and Table A9 indicate the 136M Base model. Appendix C.2.2 also states that BEND chromatin accessibility was not included in coordinate-based overlap filtering. These details should be reconciled with the model-size and held-out evaluation claims.

Typo in model size. Thank you for the careful reading. This is a typo: the post-trained DNA-RADIO has 136M parameters. We have corrected its parameter count in Table 2.

Data leakage on BEND. Every BEND record has genomic coordinates, and among our post-training sources, the only one that could plausibly collide with BEND is ENCODE cCRE V4. Chromatin accessibility is the one task distributed in GRCh37 and is therefore outside our GRCh38 interval-overlap gate (Appendix C.2.2). And it does not use cCRE: its labels are processed following the Basset pipeline, and the data differ from cCRE V4 in both version and scale. In addition, every post-training record, including those from cCRE, passes our exact-sequence filter, which removes any record whose canonical DNA (accounting for reverse complements) appears in any BEND evaluation split.

We have added this clarification to the Corpus scope paragraph in Appendix Sec. C.2 and BEND source exclusion and exact-sequence filtering paragraph in Appendix Sec. C.2.2.

AuthorsOpenReview ↗

Reminder to Reviewer Lg8S

Dear Reviewer Lg8S,

This is a gentle reminder that we have posted our responses to your comments. We took each point seriously and put considerable effort into addressing it, including additional experiments and the corresponding revisions to the paper. If any concern remains or anything needs clarification, we would be glad to address it in the remaining discussion period.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_3gZQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DNA-RADIO, an agglomerative embedding model that distills multiple heterogeneous DNA language models into a single compact student. The motivation comes from two analyses: different DNA LMs each have their own strengths, with no single model dominating across the board; and layer-wise linear probing plus CKA shows that the downstream performance of existing models saturates at early layers, with deeper layers contributing very little (unused capacity). The method proceeds in two stages: the first stage performs multi-teacher distillation from three billion-scale teachers (GENERator-1.2B, GenomeOcean-4B, NT-2.5B), aligning heterogeneous tokenizers by nucleotide intervals, applying PHI-S normalization and teacher-specific adapters, and combining pointwise, relational, and summary losses; the second stage performs information-constrained relational post-training on 3.29 million labeled sequences (constructing target distributions within a KL budget through exponential tilting of an EMA baseline). Results: the distillation-only 136M/377M students surpass all baselines, including the 4B model, on linear probing across 54 tasks drawn from Revised-NT, GUE, and Genomic Benchmark, and the post-trained version leads by a further margin on the fully held-out BEND.

Critical strengths
  1. The direction is novel, and the work is driven by a mechanistic motivation rather than a mere accumulation of results. This is the first work to bring the multi-teacher agglomerative distillation of the RADIO family from the vision domain into DNA representation learning. What is more commendable is that the motivation rests on the authors' own analyses: layer-wise probing plus CKA systematically characterizes the "95%-of-peak saturation depth" of 9 DNA LMs, providing a testable premise for the claim that the knowledge of large models can be compressed into small ones, rather than simply asserting that such compression is feasible. A 136M model matching a 4B model has real value for practical deployment costs.

  2. Data hygiene and leakage prevention reach a level of rigor that is rare in this field. BEND is excluded from training in its entirety: 334,527 GRCh38 intervals totaling 263.2 Mb are merged, an interval overlap gate removes 470,336 post-training records, and exact deduplication is performed on canonical DNA, with reverse complements collapsed to their lexicographically smallest form, finally verifying zero overlap against 1.85 million coordinate records; the distillation corpus uses chromosome-level hold-out plus masking of benchmark loci. The authors also proactively identify and clean up cross-source contamination, such as GUE reusing NT samples and the class collapse of prom_300.

Critical weaknesses
  1. The most critical control arm is missing: "teacher-free, same-corpus, same-budget continued pretraining." The student is initialized from the ModernGENA checkpoint and continues training on 1.1B tokens of the 348 Gbp corpus. In Table 1, the advantage of DNA-RADIO over ModernGENA mixes three factors: (a) multi-teacher distillation, (b) continued pretraining on a different corpus, and (c) training budget; the existing ablations only remove individual loss terms or post-training components (Table 3) and cannot decouple these factors. Therefore, the core and counterintuitive claim that "the student surpasses the teachers thanks to multi-teacher synergy" (in Section 5 the authors themselves admit they "hypothesize... have not conducted experiments to verify") has no direct evidence; a control arm of "ModernGENA with pure MLM continued training on the same data and the same budget" is a must-add.

  2. The inference that "deep layers are underutilized" goes beyond what linear probing can support. The 95%-of-peak criterion is defined on linear probing averaged across tasks; CKA greater than 0.7 only indicates high representational similarity to the final layer under the authors' own CKA kernel choice, and it does not follow that deep layers are redundant for uses such as generation, few-shot learning, or fine-tuning. Moreover, the last layers of the two AR decoder teachers (GENERator, GenomeOcean) inherently serve the LM head, so pooling their intermediate layers for probing puts them at a natural disadvantage; using their "best probing layers" as teachers and then claiming the student surpasses the teachers tilts the comparison in the student's favor. The cited weak scale-performance correlation (Ledneva et al.) is correlational evidence and is not sufficient on its own to support the causal narrative of "capacity redundancy."

  3. Several advantage margins are thin, benchmark ceilings are low, and the aggregate statements lean optimistic. On Genomic Benchmark, Ld 0.8194 vs NT-multi 0.8113 (+0.008); on the binary enhancer task of Revised-NT, all models achieve only about 0.48 to 0.52 MCC, the weakest task family in the benchmark, so the gains reported there carry little practical meaning; the post-trained version's average rank on BEND is 2.14 vs NT-multi's 2.29, and Var. effect disease still falls below NT-multi (0.7523 vs 0.7723). The abstract's "up to 15.85% / 29.4× smaller" picks the best slice and diverges from the actual per-task distribution.

Questions for authors

Please refer to the Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 3gZQ [1/4]

We thank the reviewer for the time and effort devoted to reviewing DNA-RADIO. We are encouraged that the reviewer recognized the novelty of our approach, its grounding in a mechanistic motivation rather than an accumulation of results, and the systematic nature of our analysis. We are also glad that the reviewer highlighted the practical value of our 136M-parameter model, which matches a 4B-parameter model at a fraction of the deployment cost, and our data leakage prevention, which the reviewer noted is rarely seen in this field. We address each of the reviewer's concerns and questions in detail below.

[W1] Missing teacher-free, same-corpus, same-budget continued pretraining control. The student is initialized from the ModernGENA checkpoint and continues training on 1.1B tokens of the 348 Gbp corpus, so the advantage of DNA-RADIO over ModernGENA in Table 1 mixes three factors: (a) multi-teacher distillation, (b) continued pretraining on a different corpus, and cc training budget. The existing ablations (Table 3) cannot decouple these factors, leaving no direct evidence for the claim that the student surpasses the teachers thanks to multi-teacher synergy. A control arm of ModernGENA with pure MLM continued training on the same data and budget should be added.

To separate the effect of multi-teacher distillation from the benefit of additional training alone, we continually trained the same initialization with an MLM objective under an identical training data budget and batch size (Table R1). Continued MLM training yields only a small gain, which is expected: ModernGENA has already been pretrained on 1.5T tokens, and the additional 1.1B tokens amount to only about 0.07% of that. Under the same small budget, multi-teacher distillation delivers clear gains: DNA-RADIO-B$_d$ outperforms continued MLM on all three benchmarks (0.5767 vs. 0.5379 on Revised-NT, 0.6539 vs. 0.6075 on GUE, 0.8173 vs. 0.8009 on Genomic Benchmark), and on every task of Revised-NT and Genomic Benchmark and every GUE family. This indicates that the improvement comes from the teacher signal rather than from additional data. These results are added to the revised Appendix Sec. E.6 with paragraph Ablation on initialization and teacher signal and Table A15.

Table R1. Comparison with continued MLM training on the ModernGENA-base architecture. Continued MLM continues training the pretrained ModernGENA with its masked language modeling objective on the same 1.1B-token budget as DNA-RADIO. Best per row in bold; per-task Revised-NT results are in Appendix Table A15.

a) Revised-NT family means (MCC).

Family (# tasks)ModernGENA-BContinued MLMScratch distill.DNA-RADIO-B$_d$ (Ours)
Regulatory (5)0.60810.61850.60890.6401
Splice (3)0.50240.57440.49560.6420
Histone (10)0.47680.48670.52640.5254
Overall0.51750.53790.54420.5767

b) GUE family means (MCC). Overall = mean of the 6 family means.

FamilyModernGENA-BContinued MLMScratch distill.DNA-RADIO-B$_d$ (Ours)
EMP0.43080.49090.47940.5358
TF-M0.62190.65070.57490.6836
TF-H0.59840.63400.60720.6570
PD0.75630.78380.78230.8181
CPD0.56950.60820.59390.6260
SSP0.42330.47720.43690.6031
Overall0.56670.60750.57910.6539

c) Genomic Benchmarks per task (accuracy). Overall = mean over all 9 tasks; this benchmark has no predefined task families.

TaskModernGENA-BContinued MLMScratch distill.DNA-RADIO-B$_d$ (Ours)
Mouse EE0.79590.85290.81740.8570
CI0.92270.93260.91400.9404
EC0.74410.75540.74360.7571
EE0.74660.75950.77460.7779
ER0.64190.65290.73750.7289
non-TATA0.86610.87700.86220.8788
OCR0.67410.68430.70070.7039
Human–Worm0.96070.96950.96350.9752
ES0.70740.72400.72160.7361
Overall0.78440.80090.80390.8173
AuthorsOpenReview ↗

Rebuttal for Reviewer 3gZQ [2/4]

[W2] Overreaching inference that deep layers are underutilized. The 95%-of-peak criterion is defined on linear probing averaged across tasks, and CKA greater than 0.7 only indicates high representational similarity to the final layer under the authors' own kernel choice; it does not follow that deep layers are redundant for uses such as generation, few-shot learning, or fine-tuning. Moreover, the last layers of the AR decoder teachers (GENERator, GenomeOcean) inherently serve the LM head, so using their best probing layers as teachers and then claiming the student surpasses the teachers tilts the comparison in the student's favor. The cited weak scale-performance correlation (Ledneva et al.) is correlational evidence and is not sufficient on its own to support the causal narrative of capacity redundancy.

The concern contains two questions: (A) whether Sec. 3.1 supports its claim, and (B) whether the student–teacher comparison is fair. In short:

  • (A) Sec. 3.1 claims only what linear probing shows, and it is motivation, not our main result. Frozen-feature linear probing is the primary measure of representation quality for foundation models, and our notion of under-utilization is defined with respect to it. Generation and few-shot learning are outside this scope, and we address fine-tuning directly in Appendix Table A14, full fine-tuning. CKA and Ledneva et al. corroborate the probing result rather than support a causal claim. Our main claim, that the student outperforms its teachers, is tested directly.
  • (B) The student outperforms every teacher even when each teacher is read out at its distillation layer. Tab. 1 evaluates all models at their final layer. The distillation layer is the earliest layer reaching 95% of peak, not the best one. DNA-RADIO-L$_d$ outperforms every teacher at either layer on all three benchmarks (Table R2), and also under full fine-tuning, where no layer is chosen.

A. Scope of Sec. 3.1

  1. What we claim. We define "unused capacity" operationally in Sec. 1: "deeper layers contribute little additional downstream-relevant signal." Sec. 3.1 states that depth "may be functionally under-utilized" as a "plausible explanation." Layer-wise linear probing is the standard tool for analyzing what intermediate layers encode [R1], and frozen-feature linear probing is the primary measure of representation quality for foundation models [R2-3] and for DNA LMs [R4-6]. Our notion of under-utilization is defined with respect to this measure.
  2. Other uses. DNA-RADIO is a representation model. Generation and few-shot learning are not among our claims, and they are not part of these benchmarks' protocols, so they are out of scope. For full fine-tuning, see Appendix Tab A14 (Table A12 in the original submission).
  3. CKA and Ledneva et al. corroborate the claim; it does not rest on them. Probing saturates early across encoder, SSM and AR-decoder models and on two benchmarks (Fig. 2, A4–A9). CKA, computed with the RBF kernel of Kornblith et al. (2019), adds that representations after that point change little relative to the final layer. We cite Ledneva et al. as consistent with this finding, not as causal evidence.

(continued to [3/4])

AuthorsOpenReview ↗

Rebuttal for Reviewer 3gZQ [3/4]

B. Fairness of the student–teacher comparison

  1. Evaluation layer. Every model in Tab. 1 is read out from its final layer, with the last (EOS) token used for AR models (Sec. 4.1). Using the last token is standard for causal models, because it is the only position that has seen the whole sequence. This protocol is common practice for AR DNA LMs, including in the papers that introduced them [R7-9].
  2. Distillation layer. This is the earliest layer reaching ≥95% of peak, chosen on a calibration set disjoint from both training stages (Sec. 4.1, Implementation details; Appendix Sec. D.1.1). Table R2 reports each teacher at both layers. DNA-RADIO-L$_d$ outperforms every teacher at either layer on all three benchmarks. DNA-RADIO-B$_d$, which is 8.98×8.98\times smaller than GENERator, does so on GUE and Genomic Benchmark and is 0.0021 below GENERator's readout layer on Revised-NT. We added these results to Appendix Sec. E.2 (Table A10, with per-task results).

Table R2. Linear-probing performance of each teacher at its final layer and at the intermediate layer used as the distillation target (readout), compared with DNA-RADIO.

Model (size)LayerRevised-NTGUEGenomic Bench.
GENERator (1.2B)final (L26) / readout (L10)0.5744 / 0.57880.6391 / 0.64710.8030 / 0.8076
GenomeOcean (4B)final (L24) / readout (L8)0.5586 / 0.55310.5721 / 0.58770.8013 / 0.8029
NT-multi (2.5B)final (L32) / readout (L15)0.5547 / 0.57240.6066 / 0.62180.8113 / 0.8068
DNA-RADIO-B$_d$ (136M)final0.57670.65390.8173
DNA-RADIO-L$_d$ (377M)final0.58640.66280.8194
  1. Full fine-tuning (no layer choice). Full fine-tuning is one of the protocols for evaluating DNA LMs, and it uses the whole model. Under full fine-tuning (Appendix Table A14), DNA-RADIO-L$_d$ averages 0.6811, against 0.6746 for NT-multi and 0.6744 for GENERator.

References
[R1] Alain & Bengio, Understanding intermediate layers using linear classifier probes. arXiv, 2016.
[R2] Radford et al., Learning transferable visual models from natural language supervision. ICML, 2021.
[R3] Oquab et al., DINOv2: Learning Robust Visual Features without Supervision. TMLR, 2024.
[R4] Dalla-Torre et al., Nucleotide Transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 2025.
[R5] Tang et al., Evaluating the representational power of pre-trained DNA language models for regulatory genomics. Genome Biology, 2025.
[R6] Ledneva et al., GENEB: Why Genomic Models Are Hard to Compare. ICML, 2026.
[R7] Brixi et al., Genome modelling and design across all domains of life with Evo 2. Nature, 2026.
[R8] Wu et al., GENERator: a long-context generative genomic foundation model. arXiv, 2025.
[R9] Zhou et al., GenomeOcean: An efficient genome foundation model trained on large-scale metagenomic assemblies. bioRxiv, 2025.

Reviewer_hhShOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a smaller model that by distillation learns to combine the strengths of existing DNA foundation models. The authors also investigate the depth-vs-learning correlation of existing models, showing that models appear saturated at an earlier depth. They use this information to motivate a smaller distillation model. The DNAradio model combines DNA-FMs with very different tokenisation and setup and the student model is trained to retain the different representations as well as relational information between them. The authors the post-trains the model with functional labels to improve it. The results on downstream benchmarks (using frozen embedding model) demonstrate that the student model outperforms the teachers showing the potential of much smaller models.

Critical strengths

Combining different DNA language models is interesting and useful for several reasons. Current DNA models are difficult to compare since they are often so different and perform so differently on downstream tasks for reasons that can be hard to explain, combining them and doing rigorous ablations can show the contributions from the different teachers. It can also result in an overall both stronger and smaller model as demonstrated by the benchmarks used here and removes the need to choose between several models with unknown strengths when performing a task.

The investigation of what the representations depending on model depth learn, are very interesting and useful (could be explored more).

Critical weaknesses

lack of ablations concerning the specific contributions of the different teacher models and the weight initialisation used (e.g. use random weights with the modernGENA backbone). This is also acknowledged by the authors themselves in the discussion ( around line 468-471). the authors show ablations for the components of the distillation loss but not the actual models used. This would be very interesting, especially in combination with the investigation of the models' learning capabilities done in section 3.1.

Questions for authors
  1. contribution of different teachers (already mentioned in weaknesses). The authors address this themselves so I don't think a long elaboration is needed here. but a comparison/ablation using individual teacher models would be good when one of the central hypothesis is that the teacher models have learned potentially complementary representation.
  2. comparison with other models trained functional labels (e.g. alpha genome, enformer, NTV3, segmentNT) since the model also post trains on functional labels.
  3. the distillation uses intermediate teacher layers but the results reported are from the final layers, could the authors potentially report the same selected intermediate teacher layers performance in the benchmarks?
  4. How was the checkpoint selected (D.2.3)
  5. it would be interesting to understand how much the representations change after post-training and how much of the original structure is retained. The authors probe the performance and separation of the post-trained model in fig A10/A11 but they do not compare representation similarity before and after post training. Would it be possible to do a similar analysis to figure 3.
3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer hhSh [1/3]

We thank the reviewer for the careful review of DNA-RADIO. We are encouraged that the reviewer recognized the value of combining multiple DNA language models: the resulting model is both stronger and more compact, and it removes the need to choose among several models with unknown task-specific strengths. We are also glad that the reviewer found our layer-wise analysis of what representations at different depths learn to be interesting and useful. We address each of the reviewer's concerns and questions in detail below.

[W1] Missing ablations on teacher models and weight initialization. The paper lacks ablations concerning the specific contributions of the different teacher models and the weight initialization used. This would be particularly interesting in combination with the investigation of the models' learning capabilities in Section 3.1.

We have added both ablations. In short:

  • Teacher models. Each teacher contributes different strengths, and combining them performs best overall (Table R2; see our response to Q1).
  • Initialization. Multi-teacher distillation is the main source of the gains, and pretrained initialization adds to them. Distillation from scratch with only 1.1B tokens outperforms the pretrained ModernGENA (1.5T tokens) on all three benchmarks, and continued MLM training on the same budget on Revised-NT and Genomic Benchmark (Table R1).

Initialization ablation (Table R1). We compare four settings, all evaluated with the same protocol:

  1. ModernGENA: the pretrained initialization (1.5T tokens), without further training.
  2. MLM Continued: ModernGENA trained further with its MLM objective on the same 1.1B-token budget as our distillation.
  3. Distillation from scratch: random initialization, then multi-teacher distillation on 1.1B tokens.
  4. DNA-RADIO (ours): ModernGENA initialization, then multi-teacher distillation on 1.1B tokens.

Continued MLM gives only a modest gain over ModernGENA (0.5379 vs. 0.5175 overall MCC on Revised-NT, +0.020), about a third of the gain from DNA-RADIO (+0.059). This is expected, because the 1.1B additional tokens amount to only about 0.07% of ModernGENA's 1.5T pretraining tokens. Under the same budget, distillation from scratch outperforms both on Revised-NT (0.5442) and Genomic Benchmark (0.8039), so the gains come from the teacher signal rather than from additional data. Starting from the pretrained ModernGENA improves results further (0.5767 / 0.6539 / 0.8173 on Revised-NT / GUE / Genomic Benchmark): DNA-RADIO outperforms both ModernGENA and continued MLM on every task of Revised-NT and Genomic Benchmark and on every GUE family. The two sources of gain are complementary. Distillation from scratch already matches DNA-RADIO on histone marks (0.5264 vs. 0.5254), but falls below ModernGENA on splice sites (0.4956 vs. 0.5024). Pretrained initialization recovers these tasks (0.6420).

These results are added to the revised Appendix Sec. E.6 with paragraph Ablation on initialization and teacher signal and Table A15.

Table R1. Comparison of training strategies on the ModernGENA-base architecture. Continued MLM continues training the pretrained ModernGENA with its masked language modeling objective; Scratch distill. trains ModernGENA from random initialization following the DNA-RADIO distillation procedure. Both use the same 1.1B-token budget as DNA-RADIO. Best per row in bold; per-task Revised-NT results are in Appendix Table A15.

a) Revised-NT family means (MCC).

Family (# tasks)ModernGENA-BContinued MLMScratch distill.DNA-RADIO-B$_d$ (Ours)
Regulatory (5)0.60810.61850.60890.6401
Splice (3)0.50240.57440.49560.6420
Histone (10)0.47680.48670.52640.5254
Overall0.51750.53790.54420.5767

b) GUE family means (MCC). Overall = mean of the 6 family means.

FamilyModernGENA-BContinued MLMScratch distill.DNA-RADIO-B$_d$ (Ours)
EMP0.43080.49090.47940.5358
TF-M0.62190.65070.57490.6836
TF-H0.59840.63400.60720.6570
PD0.75630.78380.78230.8181
CPD0.56950.60820.59390.6260
SSP0.42330.47720.43690.6031
Overall0.56670.60750.57910.6539

c) Genomic Benchmarks per task (accuracy). Overall = mean over all 9 tasks; this benchmark has no predefined task families.

TaskModernGENA-BContinued MLMScratch distill.DNA-RADIO-B$_d$ (Ours)
Mouse EE0.79590.85290.81740.8570
CI0.92270.93260.91400.9404
EC0.74410.75540.74360.7571
EE0.74660.75950.77460.7779
ER0.64190.65290.73750.7289
non-TATA0.86610.87700.86220.8788
OCR0.67410.68430.70070.7039
Human–Worm0.96070.96950.96350.9752
ES0.70740.72400.72160.7361
AuthorsOpenReview ↗

Rebuttal for Reviewer hhSh [2/3]

[Q1] Contribution of individual teacher models. Since one of the central hypotheses is that the teacher models have learned potentially complementary representations, a comparison or ablation using individual teacher models would be helpful.

Thank you for suggesting an insightful experiment. The ablation supports the complementarity hypothesis. Each single-teacher student excels on different tasks, and combining the teachers gives the best overall performance (Table R2).

  • Complementarity. Among the single-teacher students, the best one differs from task to task. The GENERator student leads on 8/18 tasks, including all three promoter tasks. The NT-2.5B-multi student leads on 8/18, including half of the histone marks. The GenomeOcean student leads on the remaining 2/18.
  • Consolidation. Multi-teacher distillation achieves the best overall MCC (0.5767, vs. 0.5696 for the best single teacher). It is also best on 11/18 tasks, more than any single-teacher student.

These results are added to the revised Appendix Sec. E.6 with paragraph Single- vs. multi-teacher distillation. and Table A16.

Table R2. Single- vs. multi-teacher distillation on Revised-NT (MCC). We distill the student from each of the three teachers used for DNA-RADIO (GENERator, GenomeOcean and NT-2.5B-multi) individually, and compare it with multi-teacher distillation (DNA-RADIO, ours). Best per row in bold.

GENERatorGenomeOceanNT-2.5B-multiMulti-teacher (Ours)
H2AFZ0.43160.44210.44010.4419
H3K27ac0.46300.46090.46930.4839
H3K27me30.57520.56290.56640.5790
H3K36me30.56470.55980.59260.5977
H3K4me10.44970.44030.44660.4575
H3K4me20.50190.51640.52040.5240
H3K4me30.61780.62270.61680.6233
H3K9ac0.52490.50150.52280.5277
H3K9me30.42510.38230.42550.3964
H4K20me10.60980.60900.62040.6231
Enhancers0.49790.49720.50600.4973
Enhancers types0.38130.35700.36890.3744
Promoter all0.75960.73320.74040.7581
Promoter no TATA0.78180.76320.76040.7608
Promoter TATA0.79850.74210.77010.8101
Splice sites acceptors0.70020.66200.70340.7072
Splice sites all0.43220.45740.47230.4969
Splice sites donors0.72740.69120.70980.7218
Overall0.56900.55560.56960.5767

[Q2] Comparison with models trained on functional labels. Since the model is also post-trained on functional labels, a comparison with other models trained on functional labels (e.g., AlphaGenome, Enformer, NTv3, SegmentNT) would be helpful.

We compare post-trained DNA-RADIO-B with SegmentNT, NTv3 (650M, post-trained), AlphaGenome and Enformer on BEND (Table R3), which is held out from our post-training (see Sec. 4.1 and Appendix Sec. C.2.2). DNA-RADIO-B ranks first on CpG methylation (0.9303 vs. ≤0.9050) and variant effect, expression (0.5671 vs. ≤0.5363), where no compared model was trained on the evaluation data. On histone modification, the compared models were trained on K562 histone used in BEND. Thus, this task has partial overlap with their training data. Despite this overlap, DNA-RADIO-B remains within 0.016 AUROC of the best model.

We exclude four tasks from Table R3 because the compared models were trained on their evaluation labels or the same assays:

  • Gene finding: the GENCODE gene-structure labels (e.g., ~96% of test genes lie on SegmentNT's training chromosomes).
  • Enhancer annotation: the K562 DNase/H3K27ac/CAGE tracks underlying the CRISPRi enhancer labels.
  • Chromatin accessibility: the ENCODE DNase-seq tracks.
  • Variant effect (disease): splice-site data are overlapped for the models trained on functional labels (SegmentNT, NTv3 and AlphaGenome).

The full comparison on all tasks, with per-model overlap details, is added to the revised Appendix Sec. E.2 with paragraph Comparison with models trained on functional labels. and Table A11.

Table R3. Comparison with models trained on functional labels on BEND (AUROC). Best per row in bold.

SegmentNTNTv3-650M-postAlphaGenomeEnformerDNA-RADIO-B
model size562M650M450M236M136M
CpG methylation0.89210.90500.90160.90120.9303
Variant effect (expression)0.53630.51160.53340.53090.5671
Histone modification0.76440.79860.79610.78560.7826
AuthorsOpenReview ↗

Rebuttal for Reviewer hhSh [3/3]

[Q3] Performance of the selected intermediate teacher layers. The distillation uses intermediate teacher layers, but the reported teacher results are from the final layers. Could the authors report the benchmark performance of the same selected intermediate teacher layers?

Table R4 below reports each teacher at both layers. DNA-RADIO-L$_d$ outperforms every teacher at either layer on all three benchmarks. DNA-RADIO-B$_d$ does so on GUE and Genomic Benchmark. These results are added to the revised Appendix Sec. E.2 with paragraph Teacher performance on pooling layers. and Table A10 with detailed sub-category performance.

Table R4. Revised-NT, GUE and Genomic Benchmark performance of each teacher, measured at the intermediate layer used as the distillation target and final layer for reference.

Model (size)LayerRevised-NTGUEGenomic Bench.
GENERator (1.2B)final (L26) / readout (L10)0.5744 / 0.57880.6391 / 0.64710.8030 / 0.8076
GenomeOcean (4B)final (L24) / readout (L8)0.5586 / 0.55310.5721 / 0.58770.8013 / 0.8029
NT-multi (2.5B)final (L32) / readout (L15)0.5547 / 0.57240.6066 / 0.62180.8113 / 0.8068
DNA-RADIO-B$_d$ (136M)final0.57670.65390.8173
DNA-RADIO-L$_d$ (377M)final0.58640.66280.8194

[Q4] How was the checkpoint selected?

The reference to a 4,000-step checkpoint is a typo; we use the checkpoint trained for 48,000 steps, and we have corrected this in Appendix Sec. D.2.3. The total number of training steps was selected using the validation set of the Revised-NT Benchmark without access to its test set. We thank the reviewer for the careful reading.

[Q5] Representation change after post-training. It would be interesting to understand how much the representations change after post-training and how much of the original structure is retained. Figures A10 and A11 probe the performance and separation of the post-trained model but do not compare representation similarity before and after post-training. Would it be possible to conduct an analysis similar to Figure 3?

We thank the reviewer for this helpful suggestion. We have added a layer-wise Centered Kernel Alignment (CKA) analysis in Appendix Sec. E.1.4 (Fig. A12), comparing pretrained ModernGENA-base, distillation-only DNA-RADIO-B$_d$, and post-trained DNA-RADIO-B. The results show that post-training largely preserves the distilled representation, with changes concentrated in the upper layers, whereas distillation causes substantially greater representational reorganization.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion period

We thank the AC and all reviewers for their time and constructive feedback. Reviewers found the direction novel and well-motivated (3gZQ, Lg8S), the early-saturation analysis of DNA LMs useful (hhSh, jASY), the leakage control rigorous (3gZQ, jASY), and the compact model practically valuable (all). Below, we summarize the main concerns and how we addressed them. All changes are in the revised paper.

1. Do the gains come from the teachers? (3gZQ, jASY, hhSh)

  • Concern: The gains could come from extra tokens, the new positional encoding, or the losses themselves rather than from the teachers.
  • Response: We added three controls with the same data and 1.1B-token budget: continued MLM, distillation from scratch, and self-distillation (App. E.6, Table A15). Self-distillation does not improve over the initialization. DNA-RADIO outperforms continued MLM on every Revised-NT and Genomic Benchmark task and on every GUE family. The gain comes from the external teacher signal.

2. Contribution of multiple teachers (hhSh, Lg8S, jASY)

  • Concern: It is unclear whether the teachers are complementary and whether multiple teachers are needed.
  • Response: We added single-teacher distillation for each teacher on all three benchmarks (App. E.6, Table A16). Three teachers give the highest average on every benchmark, and the best single teacher varies by task. As the reviewers noted, the multi-teacher gain is consistent but modest: a single strong teacher recovers 87–96% of the improvement. The revised abstract, Discussion and App. E.6 now state this.

3. Teachers evaluated at their distillation layers (hhSh, Lg8S, jASY, 3gZQ)

  • Concern: Teachers are scored at their final layer, but the student learns from their intermediate layers, so the comparison may understate the teachers.
  • Response: We report each teacher at both its final and its distillation layer (App. E.2, Table A10). DNA-RADIO-L outperforms every teacher at either layer on all three benchmarks.

4. Post-training objective (Lg8S, hhSh)

  • Concern: It is unclear what the objective adds over standard supervised contrastive learning, and how much it changes the distilled representation.
  • Response: Under matched conditions, our objective outperforms supervised contrastive learning on all 7 BEND tasks (Sec. 4.3, Table 3). A layer-wise CKA analysis shows that post-training largely preserves the distilled representation (App. E.1.4, Fig. A12).

5. Comparison with models trained on functional labels (hhSh)

  • Concern: Since DNA-RADIO is post-trained on functional labels, it should be compared with models trained on such labels.
  • Response: We compare against SegmentNT, NTv3, AlphaGenome and Enformer on BEND (App. E.2, Table A11). DNA-RADIO-B (136M) ranks first on CpG methylation and variant effect (expression). On histone modification, where those models were trained on overlapping data, it is within 0.016 AUROC of the best model.

6. Scope of the layer-wise analysis (3gZQ)

  • Concern: Linear probing and CKA do not show that deep layers are redundant for other uses, such as fine-tuning.
  • Response: We clarified that under-utilization is defined with respect to frozen-feature linear probing and serves as motivation. We also point to the full fine-tuning results (Table A14), where DNA-RADIO-L has the highest Revised-NT average.

7. Preservation of fine-grained information (Lg8S)

  • Concern: Overlap-based token aggregation may smooth out motif-level information.
  • Response: Distillation improves composition-controlled GUE tasks (TF-M, TF-H, PD-tata). It also raises zero-shot single-nucleotide variant AUROC from 0.49 to 0.74, which shows that nucleotide-level information is preserved.

8. Claims and reporting (3gZQ, Lg8S, hhSh)

  • Concern: The abstract highlights the best case, and several details are unclear or inconsistent: the model size in Table 2, the BEND leakage scope, checkpoint selection, and compute cost.
  • Response:
    • The abstract now reports the full range of results (1.0–15.85% improvement; 3.24–29.4× smaller).
    • We fixed the model-size typo in Table 2 (136M).
    • We clarified the BEND leakage protection, including chromatin accessibility (App. C.2, C.2.2).
    • We clarified that the checkpoint was selected on the Revised-NT validation split only (App. D.2.3).
    • We added the compute cost: 80 GPU-hours per base distillation run (App. D.1.5).

We thank the AC and reviewers again. The requested controls clarified where the gains come from and made our claims more precise, and the paper is stronger as a result.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.