- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies which layer should be used when frozen DNA foundation models are evaluated. The authors argue that the common habit of using the last layer can miss better features in earlier layers and can change how models are ranked. The main challenge is to choose a good layer fairly and cheaply across very different models. The authors probe every layer of ten models on two task suites, choose layers with a validation set, and test cheap scores and simple rules as shortcuts. They find that choosing a layer improves results and changes model rankings, and that good layers usually lie in the first half of the model.
Critical strengths
S1. The evaluation is careful and transparent.
- The test set is never used to choose a layer, and uncertainty is estimated properly (Sec. 3, App. A.6).
- Several definitions of "last layer" are compared (Sec. 4.3, App. B.3).
- The paper gives enough detail to reproduce the work and states its limits openly (App. A.1, App. C, Sec. 7).
Why this matters to my score: The results can be trusted, and the setup can serve future benchmarks.
S2. The results are directly useful.
- The last layer underestimates most models, not only the Evo models (Sec. 4.3).
- LogME is the best cheap score, and checking only the first half of the layers is almost as good as checking all of them (Table B4, Tables B5–B6).
- The reporting advice is simple and easy to follow (Sec. 7).
Why this matters to my score: For an analysis paper, this practical value is what could justify publication.
Critical weaknesses
W1. The paper adds little new insight, and the ranking change comes mainly from Evo and Evo 2.
- It is already known, in NLP and for DNA models, that middle layers often beat the last layer. The Evo 2 code itself recommends a middle layer.
- Most rank changes involve Evo or Evo 2, and they mostly disappear under the norm-based definition of "last layer" (Sec. 4.2–4.3, Table B3).
- Models with more and wider layers are favoured, and the paper does not control for this (Table A2).
Why this matters to my score: Beyond careful measurement, there is little new here. This is the main reason my score is below the acceptance line.
W2. It is unclear whether the "first half" advice holds beyond these tasks.
- Most tasks are short and local, and the CpG task can be solved from GC content alone (App. A.2, Table A5). Early layers may simply suit such tasks.
- There is no k-mer baseline, and no test with other pooling methods.
- The cost saving is small because all layers come from one forward pass (Sec. 3.1), and the rules were designed after seeing both task sets (Sec. 6.2).
Why this matters to my score: The "first half" advice may not hold for longer or harder tasks.
Questions for authors
- Do the ranking changes still hold when all models are given the same feature width (for example, PCA to a common width) and the same number of layers to choose from?
- How well does a k-mer-count or one-hot baseline do on each task type? How often is L0 chosen?
- Do the chosen layers change if you use the center token or max pooling instead of mean pooling?
- What happens to the results if Evo 2's "last layer" is set to the layer its developers recommend (
blocks.28.mlp.l3)? - In 54% of cases the chosen C is at the edge of the search grid, and in 39% it is at the lowest value, C = 10⁻³ (L1004–1006). Do the conclusions hold, especially for Evo and Evo 2, with a wider grid?
- How much time does the "first half" rule save if the forward pass is stopped early? Does the rule still work on models or tasks that were not used to design it (for example, other DNA models, BEND, or DART-Eval)?
Response to Reviewer 3eRj
We thank the reviewer for the constructive comments.
W1. Contribution. Intermediate-layer superiority is known, yet current benchmarks (BEND, DART-Eval, Feng et al., 2025) still read the final layer. We quantify how this convention changes evaluation conclusions and test the reliability of selection strategies. On the external suite, the pretrained-model mean difference from a validation-tuned k-mer baseline changes from −0.0063 [−0.0350, +0.0197] at the final layer to +0.0354 [0.0120, 0.0643] after selection. Thus the extraction convention changes the conclusion against this baseline. A fixed recommended layer also does not transfer across tasks. Evo 2's README state ranks second externally but ninth on the primary suite (Q4). Finally, scores tracking layerwise performance can still select poorly (§6.1). These findings go beyond the known observation.
W2. Scope. Ranking changes concentrate in Evo/Evo 2 and depend on the endpoint and pooling rule. Excluding them removes supported reversals and leader changes, but retains mean gains of +0.0258/+0.0214 (primary/external). Depth patterns in both main suites motivated the screening rules. External results show consistency across suites rather than independent validation of the rules.
Q1. Width and candidate count (Appendix D.3). At native width, using the first ten or ten log-spaced states retains the primary leader change and ten CI-supported reversals. Equalizing candidate count therefore does not by itself remove the effect. Across 15 PCA-width and candidate-count controls, gain CIs remain positive. Leaders change in 11/15 primary and all 15 external configurations. Width-only controls leave 4–7/2–5 supported reversals (primary/external), while joint width/even-grid controls leave 0–2/0–1. All four primary exceptions use even grids that omit early states. Candidate composition and feature width both matter.
Q2. Baselines and L0 (Appendix D.6). One-hot reaches 0.99 AUROC on the primary motif tasks. These local tasks assess feature accessibility under a specified readout, with comparisons conditional on task construction. With k=3–6 chosen on validation, the primary pretrained-model mean is below the k-mer baseline at the final layer (−0.0648 [−0.1170, −0.0147]). After selection, its difference is −0.0211 [−0.0653, +0.0206]. Externally, the mean advantage over one-hot (AUROC 0.7735) has a 95% CI excluding zero only after selection (+0.0858 [0.0095, 0.1619]). Token-embedding L0 is selected in 2/120 primary and 0/160 external cells. NT-v3's additional 5/120 and 1/160 L0 selections are convolutional outputs.
Q3. Pooling (Appendix D.1). Across six alternatives, 76–88% of primary cells select first-half states. Under center-token pooling, all model medians remain in the first half. Under max pooling, the Caduceus medians move to 0.81/0.59. For Evo 2 externally, max/midpoint pooling moves the median from 0.29 to 0.83/0.77. With layers selected per task and pooling rule, last-token and max pooling lower primary AUROC relative to mean pooling for all ten models, by 0.0963/0.0442 on average, but remove the primary leader change. Center pooling also removes it and gains 0.1790 on primary, consistent with the centered task construction. Midpoint pooling loses 0.0790 for Evo/Evo 2 externally. We retain mean pooling as a reference and qualify the ranking claim accordingly. Mean selection gains remain positive under every tested pooling rule.
Q4. README state (Appendix D.1). The exact blocks.28.mlp.l3 module gives Evo 2 primary AUROC 0.709 versus 0.830 with per-task selection, ranking ninth. Externally it ranks second, with regret 0.046 versus 0.004. This fixed state does not reproduce validation-selected performance.
Q5. Regularization (Appendix A.3). Expanding C to {1e-5, …, 1e3} across all 280 cells reduces boundary selections to 8.3%/8.9%. Layers change in 6/120 and 11/160 cells. Mean gains change by <0.0005 and both leader changes remain. Evo/Evo 2 have no primary changes and two external changes, both on DNase I (605 training sequences).
Q6. Cost and new tasks (Appendices D.5 and D.7). On the primary 1 kb workload with extraction unchanged, halving probe-fitting time would save approximately 0.6 hours, or 2% of the 27.3-hour full-validation estimate excluding disk reads. Stopping the forward pass after the first half of stored states reduces that total to 14.0 hours, a 49% reduction. These are component estimates, not timed end-to-end runs. Predictions and thresholds were fixed before BEND histone extraction. The gain is +0.0142 [0.0028, 0.0289]. Screening meets all planned loss criteria, with mean loss 0.0006 [−0.0004, +0.0019] and 15/180 cells losing >0.005. The depth prediction fails, with seven models having first-half medians versus eight required. On BEND CpG methylation, screening fails the same criteria, with loss +0.0030 [0.0002, 0.0067] and 14/70 cells losing >0.005. These findings cover the ten evaluated models and inputs ≤1 kb.