acceptodds
Under review as a conference paper at ICLR 2027

Which Layer to Read? The Consequences of Layer Choice in DNA Foundation Models

Abstract

Frozen DNA foundation models are conventionally used through final-layer representations, under the implicit assumption that the final hidden state provides the most informative features. We investigate how this convention affects performance and model comparisons, and how useful layers should be selected. Across ten models from five architectural families and two task suites (280 model–task cells), selecting a layer via held-out validation raises mean test AUROC from to on the primary suite, with similar gains on the external suite. The validation-based layer selection also changes model rankings in both suites. We then examine how to select layers without probing every stored state. Selecting layers directly with representation and transferability metrics yields lower mean test AUROC than full-layer validation. Useful depth varies across models and tasks, yet every model's median validation-selected depth lies in the first half of the stack in both suites. Searching only the first half of stored states nearly matches full-layer validation in mean test AUROC across both suites while requiring approximately half as many layer-specific probe fits. When screening only of stored states, LogME-guided nomination achieves higher mean test AUROC than the evaluated metric-free rules. These findings establish layer choice as a consequential component of frozen model evaluation and support explicit reporting of the extracted state and selection protocol.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
DNA Foundation Models, Representation Analysis, Layerwise Probing, Layer Selection, Model Evaluation, Transferability, Frozen Probing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3eRjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies which layer should be used when frozen DNA foundation models are evaluated. The authors argue that the common habit of using the last layer can miss better features in earlier layers and can change how models are ranked. The main challenge is to choose a good layer fairly and cheaply across very different models. The authors probe every layer of ten models on two task suites, choose layers with a validation set, and test cheap scores and simple rules as shortcuts. They find that choosing a layer improves results and changes model rankings, and that good layers usually lie in the first half of the model.

Critical strengths

S1. The evaluation is careful and transparent.

  • The test set is never used to choose a layer, and uncertainty is estimated properly (Sec. 3, App. A.6).
  • Several definitions of "last layer" are compared (Sec. 4.3, App. B.3).
  • The paper gives enough detail to reproduce the work and states its limits openly (App. A.1, App. C, Sec. 7).

Why this matters to my score: The results can be trusted, and the setup can serve future benchmarks.

S2. The results are directly useful.

  • The last layer underestimates most models, not only the Evo models (Sec. 4.3).
  • LogME is the best cheap score, and checking only the first half of the layers is almost as good as checking all of them (Table B4, Tables B5–B6).
  • The reporting advice is simple and easy to follow (Sec. 7).

Why this matters to my score: For an analysis paper, this practical value is what could justify publication.

Critical weaknesses

W1. The paper adds little new insight, and the ranking change comes mainly from Evo and Evo 2.

  • It is already known, in NLP and for DNA models, that middle layers often beat the last layer. The Evo 2 code itself recommends a middle layer.
  • Most rank changes involve Evo or Evo 2, and they mostly disappear under the norm-based definition of "last layer" (Sec. 4.2–4.3, Table B3).
  • Models with more and wider layers are favoured, and the paper does not control for this (Table A2).

Why this matters to my score: Beyond careful measurement, there is little new here. This is the main reason my score is below the acceptance line.

W2. It is unclear whether the "first half" advice holds beyond these tasks.

  • Most tasks are short and local, and the CpG task can be solved from GC content alone (App. A.2, Table A5). Early layers may simply suit such tasks.
  • There is no k-mer baseline, and no test with other pooling methods.
  • The cost saving is small because all layers come from one forward pass (Sec. 3.1), and the rules were designed after seeing both task sets (Sec. 6.2).

Why this matters to my score: The "first half" advice may not hold for longer or harder tasks.

Questions for authors
  1. Do the ranking changes still hold when all models are given the same feature width (for example, PCA to a common width) and the same number of layers to choose from?
  2. How well does a k-mer-count or one-hot baseline do on each task type? How often is L0 chosen?
  3. Do the chosen layers change if you use the center token or max pooling instead of mean pooling?
  4. What happens to the results if Evo 2's "last layer" is set to the layer its developers recommend (blocks.28.mlp.l3)?
  5. In 54% of cases the chosen C is at the edge of the search grid, and in 39% it is at the lowest value, C = 10⁻³ (L1004–1006). Do the conclusions hold, especially for Evo and Evo 2, with a wider grid?
  6. How much time does the "first half" rule save if the forward pass is stopped early? Does the rule still work on models or tasks that were not used to design it (for example, other DNA models, BEND, or DART-Eval)?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 3eRj

We thank the reviewer for the constructive comments.

W1. Contribution. Intermediate-layer superiority is known, yet current benchmarks (BEND, DART-Eval, Feng et al., 2025) still read the final layer. We quantify how this convention changes evaluation conclusions and test the reliability of selection strategies. On the external suite, the pretrained-model mean difference from a validation-tuned k-mer baseline changes from −0.0063 [−0.0350, +0.0197] at the final layer to +0.0354 [0.0120, 0.0643] after selection. Thus the extraction convention changes the conclusion against this baseline. A fixed recommended layer also does not transfer across tasks. Evo 2's README state ranks second externally but ninth on the primary suite (Q4). Finally, scores tracking layerwise performance can still select poorly (§6.1). These findings go beyond the known observation.

W2. Scope. Ranking changes concentrate in Evo/Evo 2 and depend on the endpoint and pooling rule. Excluding them removes supported reversals and leader changes, but retains mean gains of +0.0258/+0.0214 (primary/external). Depth patterns in both main suites motivated the screening rules. External results show consistency across suites rather than independent validation of the rules.

Q1. Width and candidate count (Appendix D.3). At native width, using the first ten or ten log-spaced states retains the primary leader change and ten CI-supported reversals. Equalizing candidate count therefore does not by itself remove the effect. Across 15 PCA-width and candidate-count controls, gain CIs remain positive. Leaders change in 11/15 primary and all 15 external configurations. Width-only controls leave 4–7/2–5 supported reversals (primary/external), while joint width/even-grid controls leave 0–2/0–1. All four primary exceptions use even grids that omit early states. Candidate composition and feature width both matter.

Q2. Baselines and L0 (Appendix D.6). One-hot reaches 0.99 AUROC on the primary motif tasks. These local tasks assess feature accessibility under a specified readout, with comparisons conditional on task construction. With k=3–6 chosen on validation, the primary pretrained-model mean is below the k-mer baseline at the final layer (−0.0648 [−0.1170, −0.0147]). After selection, its difference is −0.0211 [−0.0653, +0.0206]. Externally, the mean advantage over one-hot (AUROC 0.7735) has a 95% CI excluding zero only after selection (+0.0858 [0.0095, 0.1619]). Token-embedding L0 is selected in 2/120 primary and 0/160 external cells. NT-v3's additional 5/120 and 1/160 L0 selections are convolutional outputs.

Q3. Pooling (Appendix D.1). Across six alternatives, 76–88% of primary cells select first-half states. Under center-token pooling, all model medians remain in the first half. Under max pooling, the Caduceus medians move to 0.81/0.59. For Evo 2 externally, max/midpoint pooling moves the median from 0.29 to 0.83/0.77. With layers selected per task and pooling rule, last-token and max pooling lower primary AUROC relative to mean pooling for all ten models, by 0.0963/0.0442 on average, but remove the primary leader change. Center pooling also removes it and gains 0.1790 on primary, consistent with the centered task construction. Midpoint pooling loses 0.0790 for Evo/Evo 2 externally. We retain mean pooling as a reference and qualify the ranking claim accordingly. Mean selection gains remain positive under every tested pooling rule.

Q4. README state (Appendix D.1). The exact blocks.28.mlp.l3 module gives Evo 2 primary AUROC 0.709 versus 0.830 with per-task selection, ranking ninth. Externally it ranks second, with regret 0.046 versus 0.004. This fixed state does not reproduce validation-selected performance.

Q5. Regularization (Appendix A.3). Expanding C to {1e-5, …, 1e3} across all 280 cells reduces boundary selections to 8.3%/8.9%. Layers change in 6/120 and 11/160 cells. Mean gains change by <0.0005 and both leader changes remain. Evo/Evo 2 have no primary changes and two external changes, both on DNase I (605 training sequences).

Q6. Cost and new tasks (Appendices D.5 and D.7). On the primary 1 kb workload with extraction unchanged, halving probe-fitting time would save approximately 0.6 hours, or 2% of the 27.3-hour full-validation estimate excluding disk reads. Stopping the forward pass after the first half of stored states reduces that total to 14.0 hours, a 49% reduction. These are component estimates, not timed end-to-end runs. Predictions and thresholds were fixed before BEND histone extraction. The gain is +0.0142 [0.0028, 0.0289]. Screening meets all planned loss criteria, with mean loss 0.0006 [−0.0004, +0.0019] and 15/180 cells losing >0.005. The depth prediction fails, with seven models having first-half medians versus eight required. On BEND CpG methylation, screening fails the same criteria, with loss +0.0030 [0.0002, 0.0067] and 14/70 cells losing >0.005. These findings cover the ten evaluated models and inputs ≤1 kb.

Reviewer_dq4mOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the impact of layer selection on frozen DNA foundation model evaluation. Across 10 models and 28 downstream tasks, the authors show that intermediate layers often outperform the final layer and that layer selection can significantly change model rankings. They further evaluate several layer selection strategies.

Critical strengths
  1. Extensive empirical evaluation. The paper evaluates 10 DNA foundation models across different architectures and downstream tasks.
  2. Practical implications. The results highlight the importance of layer selection in foundation model evaluation and demonstrate its impact on model rankings.
Critical weaknesses
  1. Limited novelty. The observation that intermediate layers provide better representations for certain tasks has been extensively studied in CNNs and Transformers. The paper offers limited new methodological or conceptual insights.
  2. Lack of multi-layer aggregation baselines. The paper focuses exclusively on single-layer representations without comparing against established multi-layer fusion strategies, such as feature concatenation or learnable layer weighting.
  3. Limited downstream task diversity. Most primary tasks focus on local motif detection, with sequence lengths limited to 1 kb. The findings may not generalize to more complex genomic tasks requiring long-range dependencies.
  4. Limited practical efficiency gains. The proposed layer screening strategies mainly reduce the number of linear probe fits, without evaluating end-to-end computational costs, including feature extraction, metric computation, and validation.
  5. Sensitivity to validation splits. Layer selection relies heavily on validation performance, yet the experiments use fixed train/validation/test splits. The selected layers and model rankings may be sensitive to the validation data distribution. Experiments with multiple chromosome-level splits are needed to establish the robustness of these findings.
Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dq4m

We thank the reviewer for the constructive comments. The revised manuscript addresses each concern and clarifies the implications of the additional analyses. The text changes in the manuscript are marked in blue. Our responses follow:

W1. Novelty. Intermediate-layer superiority is established and acknowledged in the submission. Final-layer representations are used in published frozen-model evaluations, including DART-Eval and Feng et al. (2025). We quantify how this convention changes evaluation conclusions and test the reliability and cost of alternatives. On the external suite, the pretrained-model mean difference from a validation-tuned k-mer baseline changes from −0.0063 [−0.0350, +0.0197] at the final layer to +0.0354 [0.0120, 0.0643] after selection. Thus the extraction convention affects whether an average advantage over this baseline is established. The selection analysis also distinguishes profile correlation from successful layer selection. Even the strongest standalone score, LogME, leaves primary mean regret 0.0110 versus 0.0034 for full validation (§6.1). These findings identify consequences of evaluation choices and assess practical alternatives.

W2. Multi-layer aggregation. The primary leader change is absent under all-state PCA concatenation and learned mixing, both of which retain NT-v2 as leader. Thus model comparisons depend on aggregation as well as depth. Appendix D.2 adds concatenation, learned scalar mixing and uniform averages. All tested concatenation and learned-mix variants improve on the final layer. Against selected single layers, all-state PCA concatenation gives −0.0059 [−0.0243, +0.0098] primary and +0.0042 [−0.0047, +0.0138] external. Raw concatenation improves by +0.0123/+0.0167 for five smaller models, with positive intervals. The all-state learned mix gives −0.0132 [−0.0228, −0.0056] primary and −0.0046 [−0.0111, +0.0003] external. Its overall ranking remains close to single-layer selection (τ=0.87/0.91), although its primary leader differs (bootstrap top-1 share 0.586). We do not claim that single-layer selection is universally optimal.

W3. Task diversity. One-hot reaches 0.99 AUROC on the primary motif tasks. These controlled local tasks assess the linear accessibility of features under specified readouts. Model comparisons remain conditional on task construction. The external suite adds 16 tasks in seven families and five species, but inputs remain at most 1 kb. The screening rules were derived from depth patterns on both main suites, so the external results show consistency rather than independent validation of the rules. Appendix D.7 adds two BEND tasks. For histone modification, predictions and thresholds were fixed before extraction. Layer selection gains +0.0142 [0.0028, 0.0289], and first-half screening loses +0.0006 [−0.0004, +0.0019]. The planned loss criteria are met, but the depth prediction fails with seven models having first-half medians, versus eight required. On CpG methylation, the screening loss of +0.0030 [0.0002, 0.0067] fails the same criteria. Model–label bootstrap intervals may understate uncertainty because labels share sequences. Screening is therefore task dependent. Long-range genomic tasks remain outside our evidence.

W4. Cost. On the primary 1 kb workload with extraction unchanged, halving the 1.17-hour probe-fitting component would save approximately 0.6 hours, or 2% of the 27.3-hour full-validation estimate excluding disk reads. Final-layer evaluation costs 26.2 hours. Including cache reads gives 31.8 versus 26.3 hours for full versus final-layer evaluation. Stopping the forward pass after the first half of stored states reduces the total excluding disk reads to 14.0 hours, a 49% reduction. LogME 10% screening costs about 1.2 times full validation. These are component estimates, not a single timed end-to-end run (Appendix D.5).

W5. Split sensitivity. Across seven alternative chromosome splits, every split retains a significant mean gain (+0.041 to +0.046), the NT-v2-to-Evo-2 leader change, and 8–11 CI-supported reversals. Mean first-half screening loss ranges from −0.0004 to +0.0004. Exact selected layers agree with the reference split S0 in only 47–58% of cells. However, among the five splits retaining the test chromosomes, 186/275 changed selections fall in S0's near-optimal layer set, defined by test AUROC within 0.005 of the S0 optimum. This is a retrospective diagnostic, not a selection criterion. Thus individual layer choices vary more than aggregate performance and ranking conclusions (Appendix D.4).

Reviewer_PDGQOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how reading frozen DNA foundation models at their final layer affects evaluation. It probes every stored state of 10 models from 5 architectural families on a 12-task primary suite (mostly PWM-motif detection) and a 16-task external suite. Selecting the layer on validation data raises mean test AUROC (0.726→0.770 primary, 0.818→0.859 external) and changes model rankings. Useful depth varies by model and task but concentrates in the first half of the stack. Screening only that half nearly matches full validation, while standalone metrics such as LogME do not.

Critical strengths
  • Rigorous protocol. Test data never enter selection (A.6), splits are chromosome-disjoint, and confidence intervals use a crossed bootstrap. Endpoint and aggregation choices are stress-tested (Tables B2–B3), and the paper reports where its own claims weaken. For an evaluation-methodology paper, this credibility is the main value.
  • Robust performance gap. The final layer is within 0.005 AUROC of the test optimum in only 11.7% of cells (section 4.1). A mean regret of about 0.029 remains even for the eight models other than Evo/Evo 2 (section 4.3).
Critical weaknesses

These weaknesses do not lead me to recommend rejection, but they are why my score is weak rather than clear acceptance.

  1. The ranking-change claim is driven largely by Evo/Evo 2 and may reflect a read-out artifact rather than the final-layer convention itself. Evo or Evo 2 is involved in 12 of the 16 primary-suite reversals and 15 of the 22 external-suite reversals. Under the norm-based endpoint, Kendall’s τ\tau rises from 0.289 to 0.600 (primary) and from 0.022 to 0.689 (external). CI-supported reversals drop from 10 to 2 and from 8 to 0. Evo 2’s token norms grow by roughly 6×10^13 relative to L1 (Table C1). Norms this extreme suggest that a few tokens or channels dominate the mean-pooled embedding, which per-feature standardization across samples does not correct. Mean pooling over causal hidden states is also a questionable read-out for these models. This matters because “layer choice changes model rankings” is a headline contribution. By contrast, the performance-gap claim looks robust: mean regret stays around 0.029 for the other eight models under every endpoint definition.

  2. The paper’s novelty rests on its prescriptive conclusions, whose generality is unclear. That intermediate layers can outperform the final layer has already been reported (Skean et al., 2025; Dalla-Torre et al., 2025; Brixi et al., 2026). The contribution therefore lies in the prescriptive findings: useful depth concentrates in the first half of the stack, and screening that half suffices. Several features of the setup plausibly favor shallow layers:

    • 10 of the 12 primary tasks are local PWM-motif detection on motif-centered windows.
    • CpG detection is saturated (a GC-fraction probe alone reaches 0.991).
    • External inputs are 40–600 bp.
    • All results use linear probes on mean-pooled embeddings.

    The Early Layers and Depth Prior rules were also formulated after the depth pattern had been observed on both suites (Section 6.2). Finally, the obvious practical alternative, a learned mixture or concatenation of layers, is not evaluated. It therefore remains unclear whether practitioners should select a single layer at all.

Questions for authors

Clarifications

  1. Do the ranking and leader changes persist (a) when Evo and Evo 2 are excluded, and (b) under a robust read-out for all models? Examples would be per-token L2 normalization or outlier clipping before mean pooling, and last-token pooling for causal models. If the reversals largely persist, I would raise my score. If they largely disappear, I would view the ranking claim as model-specific.
  2. How does a learned scalar mix (or concatenation) over all stored states compare with the validation-selected single layer, in both test AUROC and model rankings?
  3. All states are cached from a single forward pass, so screening only saves probe fits on cached features. What is the wall-clock cost of probe fitting relative to feature extraction, and how much end-to-end time does 50% screening actually save?
  4. Selected C values lie on the grid boundary in 54% of configurations. Does applying the expanded grid to all cells, not only the nine checked, change the selected layers or the rankings?

Suggestions for future work:

  1. Nonlinear (MLP) probes and tasks with longer inputs (≥10 kb), to test whether the first-half concentration holds beyond motif-scale tasks.
1 comment
AuthorsOpenReview ↗

Response to Reviewer PDGQ

We thank the reviewer for the constructive comments. We revised the manuscript to address the reviewer’s concerns. The text changes in the manuscript are marked in blue. Our responses follow:

Q1/W1. Rankings and readout. The reversals persist under your proposed per-token L2 readout, but their strength depends on the models and pooling rule. Mean selection gains are more consistent. With token normalization for all ten models, gains are +0.0440/+0.0432 (primary/external), with positive 95% intervals. Both leader changes remain, with 11/8 CI-supported reversals. This argues against token magnitude as the sole explanation. The 6×10^13 norm ratio concerns an internal state. For the final-normalized state, the ratio to L1 is approximately 217. Excluding Evo/Evo 2 leaves no supported reversal or leader change, although mean gains remain +0.0258/+0.0214. Applying last-token pooling only to the causal models removes the primary leader change and supported reversals. Max and center pooling also remove the primary leader change. Externally, changing only Evo/Evo 2 to last-token pooling retains seven supported reversals but no leader change. Using max or midpoint pooling for those two models leaves neither. Norm-based endpoints likewise weaken reversals, although the primary leader change persists (§4.3). At each rule's validation-selected layer, last-token and max pooling lower primary task-averaged AUROC relative to mean pooling for all ten models, by 0.0963/0.0442 on average. Center pooling gains 0.1790 on primary, consistent with the centered task construction. Midpoint pooling loses 0.0790 for Evo/Evo 2 externally. We retain mean pooling as a reference and qualify the ranking claim accordingly. Mean selection gains remain positive under every readout examined (Appendix D.1).

W2. Scope and screening. One-hot reaches 0.99 on the primary motif tasks. These tasks assess the linear accessibility of local features under a specified readout. Both main suites use inputs ≤1 kb. Excluding saturated CpG detection preserves the primary gain (+0.0476) and ten supported reversals. Depth patterns in both main suites motivated the screening rules. External results show consistency across suites rather than independent validation of the rules. On BEND histone modification, predictions and thresholds were fixed before extraction. The gain is +0.0142 [0.0028, 0.0289]. First-half screening loss is +0.0006 [−0.0004, +0.0019], with 15/180 cells losing >0.005. This meets the planned limits of 0.002 mean loss, a CI upper bound below 0.005, and at most 10% of cells losing >0.005. The depth prediction fails with seven models having first-half medians, versus eight required. Model–label bootstrap intervals may understate uncertainty because labels share sequences. On BEND CpG methylation, screening fails the same criteria, with loss +0.0030 [0.0002, 0.0067] and 14/70 cells losing >0.005. Screening is therefore task dependent (Appendix D.7).

Q2. Aggregation. All-state PCA concatenation and learned mixing retain NT-v2 as primary leader (bootstrap top-1 share 0.586 for the learned mix). Against single-layer rankings, their τ values are 0.38/0.42 and 0.87/0.91, respectively. Relative to selected single layers, PCA concatenation gives −0.0059 [−0.0243, +0.0098] primary and +0.0042 [−0.0047, +0.0138] external. The all-state learned mix gives −0.0132 [−0.0228, −0.0056] and −0.0046 [−0.0111, +0.0003]. Raw concatenation gains +0.0123/+0.0167 for five smaller models, with positive intervals. All tested concatenation and learned-mix variants improve on the final layer. Single-layer selection is not universally optimal, and model comparisons depend on aggregation as well as depth (Appendix D.2).

Q3. Cost. On the primary 1 kb workload with extraction unchanged, halving the 1.17-hour probe-fitting component would save approximately 0.6 hours, or 2% of the 27.3-hour full-validation estimate excluding disk reads. Final-layer evaluation costs 26.2 hours. With cache reads, full/final-layer evaluation takes 31.8/26.3 hours. Stopping the forward pass after the first half of stored states reduces the total excluding disk reads to 14.0 hours, a 49% reduction. For Evo 2 on Human TFBS 1 (101 bp), the forward pass with pooling takes 475 seconds and probing all states takes 416 seconds, making probing a substantial cost in this example. For the primary workload, LogME 10% screening costs about 1.2 times full validation. These are component estimates, not timed end-to-end runs (Appendix D.5).

Q4. Regularization. Expanding C to {1e-5, …, 1e3} across all 280 cells reduces boundary selections to 8.3%/8.9% and changes layers in 6/120 and 11/160 cells. Gains become +0.0439/+0.0413. Both leader changes remain. Supported reversals stay at ten primary and rise from eight to ten external (Appendix A.3).

Additional suggestions. MLP probes on three models retain primary mean first-half losses ≤0.0004 (Appendix D.6). Inputs ≥10 kb remain outside our evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.