- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+2)
- Soundness
- 2
Read review
Paper summary
The paper examines how biological readout and next-base prediction diverge across depth in Evo 2 (7B, 40B). At one block in each model (b28 in 7B, b21 in 40B), state norms jump more than 200-fold and the standardised effective rank collapses. Across this transition, linear-probe AUROC for gene-structure annotations falls more steeply than in the preceding blocks, while next-base probe accuracy keeps improving.
The key experiment projects the transition state onto its top-16 principal directions and runs the rest of the frozen model. Native prediction stays within prespecified tolerances, but annotation AUROC drops by 0.086 (7B) and 0.059 (40B). The same edit before the transition breaks prediction. The authors call this pattern “predictive summarisation”.
Practically, earlier layers (7B b24, 40B b17) beat final embeddings on annotation, gene finding and conservation. Some variant-effect readouts favor the final layer instead.
Critical strengths
-
This paper proposes concrete layer guidance for Evo 2 users. Extracting from b24 (7B) or b17 (40B) instead of the final embedding improves annotation. It also improves BEND gene-structure and phyloP readouts. Many people use Evo 2 as a frozen feature extractor, so this is directly actionable.
-
This paper found the best layer depends on the target. Earlier layers are better for annotation and conservation. The final embedding is better for several variant-effect readouts (ClinVar, GPN-MSA). The preference can also flip with feature construction (BRCA1 position-level vs. pooled features). Showing this experimentally is more useful than the general claim that intermediate layers are better.
Critical weaknesses
- Novelty is not positioned against directly relevant prior work. The main practical message, that intermediate layers outperform final embeddings, is already established for genomic LMs.
- The Nucleotide Transformer paper (Dalla-Torre et al., 2025) reports that the best probing performance is never achieved with the final layer.
- The Evo 2 paper selects intermediate blocks for its downstream tasks [VERIFY: blocks.26 for exon classification; 40B block 20 for BRCA1], and its repository recommends intermediate embeddings.
The paper cites this work only in passing and does not explain what it adds. A new term, “predictive summarisation,” is not warranted for a largely known phenomenon.
- The transition is described but not explained. The paper shows what happens at the transition: norms explode, rank collapses, annotation readout falls and next-base decoding improves. It does not show why.
- No causal link. Restriction shows that annotation information can be removed without hurting prediction, not that the transition removes it for prediction. Attenuating the transition does not recover annotation. The authors acknowledge that the causal link is unresolved.
- No training account. The 7B b28 MLP write is 43–113× larger than its input, even for random sequence (Appendix E.2). Why training produces this is not addressed.
The term “predictive summarisation” implies an explanation the evidence does not provide.
- The annotation scope is narrow.
- Only four gene-structure classes are tested (GENCODE v44, each against intron). Regulatory elements, repeats and ncRNAs are absent. Regulatory function is a main use of gLM embeddings and may change with depth differently from gene structure.
- Nair et al. (2026) show that the best layer and representation type differ across genomic concepts. Here, every transition contrast with intervals is a four-task average. Per-task results are shown only as curves without intervals.
- The background class is ill-defined. The “intergenic” label is the default for any position without another retained label, so it includes genic positions such as noncoding exonic bases.
- Held-out evaluation is weak.
- Section 3 uses only development chromosomes. The main results are replicated only on new chr22 windows, and reconstruction adds chr17.
- The only prespecified panel (chr18) covers 7B alone and only the layer comparison.
- Chr19 is retrospective. BEND uses chr17, chr19 and chr22. All variant panels (ClinVar, GPN-MSA, BRCA1) are on chr17.
- Generality beyond Evo 2 is unclear.
- No comparable transition appears in the other screened models (HyenaDNA, Caduceus, NT-v2; maximum norm ratio about 3). The transition sits at very different relative depths (29/32 in 7B, 22/50 in 40B).
- The comparison models differ in size, data, architecture and objective, and were selected after inspecting diagnostics. They cannot isolate any single factor.
The finding may therefore be specific to Evo 2 and may not carry over to future gLMs. Combined with the missing mechanism (point 2), its implications for model design are limited.
- Weak clarity.
Core terms (“panel,” “window,” “panel D/C2,” “excess change,” “native retention”) are used before they are defined, or are defined only in the appendix. Many sentences stack several qualifications, which makes the main argument hard to follow.
Reference: Akira A. Nair et al. Interpreting Genomic Language Models using Sparse Autoencoders. ICML 2026.
Questions for authors
-
What is the core contribution beyond prior work? Nucleotide Transformer and Evo 2 already show that intermediate layers outperform final embeddings. What does this paper establish that those works did not?
-
What should gLM developers do differently because of this work? For example, does it suggest changes to the training objective, the architecture, normalization, or how embeddings are exposed to users? A concrete, testable recommendation would strengthen the paper considerably.
-
Is the transition specific to Evo 2? No comparable transition appears in HyenaDNA, Caduceus or NT-v2. If it is Evo 2–specific, what is the general lesson for future gLMs?
-
Does every annotation class decline at the transition? Please report per-class transition contrasts with intervals for the four gene-structure tasks, not only the four-task average.
-
Does the result extend beyond gene structure? Would regulatory elements (cCREs, enhancers, TF binding sites, CpG islands), repeats or ncRNAs show the same decline? Concept-level analyses such as Nair et al. (2026) suggest that the best layer varies by concept.
7 comments
Response to Reviewer tDWC (Part 1/4)
W1 / Q1. What is the core contribution beyond prior work?
Response: Our contribution is to show, by intervening inside gLMs, that what a model needs to predict the next nucleotide is not what we need to read biology from it. Prediction quality and biological readout can be dissociated, and the readout lost late in the network is not required for accurate prediction. Prior work established that intermediate layers read better; it could not say why, or whether it has to be so.
We agree that the submission did not position itself clearly against the Nucleotide Transformer and Evo 2 papers. Those papers, and the choices the reviewer cites (blocks.26 in 7B and block 20 in 40B, both before the transition at blocks 28 and 21), tell a user where to read. A comparison of layers cannot go further, because two layers differ in many ways at once.
The question. gLMs are trained on the premise that a model which predicts DNA well must come to represent the biology behind it. Most of that biology cannot be requested in the output; it is read from hidden states. The premise is therefore testable: does the model's prediction depend on the biology its states encode? We test it by intervention, measuring the model's own prediction and a probe readout side by side.
What the interventions show.
- Prediction does not need everything a probe reads. Keeping 16 of 4,096 directions after the transition changes perplexity by less than 1%, while annotation AUROC falls from 0.931 to 0.836 (−0.095 [−0.116, −0.073]). Sixteen random directions, or the same edit one block earlier, do not preserve the prediction. Prediction survives the same edit in all five models, from three families (Part 3).
- Better prediction does not mean better readout. A module trained away from next-nucleotide prediction predicts far worse (perplexity 4.6 vs. 1.6 in Genos-1.2B) and leaves a final layer that is easier to read (AUROC 0.74 vs. 0.69).
- The loss of readout is not the price of predicting well. After the late blocks are reset and retrained with the same objective, perplexity returns (2.71 vs. 2.68) while final-layer AUROC is 0.92 against 0.78, in both of two runs.
What this changes. Two models whose perplexities differ by 1% can differ this much in what can be read from them. Perplexity, by which these models are trained and compared, therefore cannot stand in for readout, which has to be evaluated and designed for in its own right (Part 4).
Nair et al. (2026) report that the best layer differs across genomic concepts; our per-class contrasts (Part 2) agree, and relate each class to the transition.
Revision: Sections 1 and 6 state the contribution, Section 5 separates it from the layer-wise observations, and the interventions are in Sections 4.1 to 4.3.
W2. The transition is described but not explained; the term implies an explanation.
Response: Accurate prediction does not require the loss of readout: in a new experiment, retraining the late blocks recovers the prediction with the readout intact. We removed the term, which implied an explanation we had not tested. Why pretraining produces the transition remains open, and the revision says so.
We agree with both points: the submission described the transition without explaining it, and the term implied more than the evidence supported.
The term. The revision no longer uses "predictive summarisation", in the title or in the text. It states what is measured: under the same edit of a state, the model's prediction is preserved and annotation readout falls. That the state after the transition is organized around the directions that set the next-base distribution is now stated as a hypothesis and tested by intervention (Section 4).
The explanation. We added retraining experiments that ask whether the loss of readout is the price of predicting well. In the base checkpoint of Evo 2 7B we reset the last four blocks, from the transition block on, to a fresh initialization and retrain them with next-nucleotide prediction (two runs of 1,000 steps).
| NLL | Annotation AUROC, final layer | |
|---|---|---|
| Original model | 0.988 | 0.779 |
| Reset and retrained, 1,000 steps (two runs) | 0.994, 0.994 | 0.932 [0.914, 0.945], 0.925 [0.906, 0.939] |
NLL returns close to the original value, and final-layer annotation AUROC is 0.93 and 0.92, close to the 0.95 read before the transition. The amount lost at the final layer is therefore not fixed by prediction performance. The state still narrows in the retrained model (effective rank 311 to about 7), but gradually, over three blocks. Whether still longer training restores the abrupt contraction, and why pretraining produces the large write of the transition block, as the reviewer asks, remain open, and the revision says so.
Revision: Section 4.3, Figure 5 and Appendix H.
Response to Reviewer tDWC (Part 2/4)
W3 / Q4. Does every annotation class decline at the transition? Please report per-class contrasts with intervals.
Response: Every task declines, but unevenly. The loss is concentrated in the splice sites, while coding exons change by less than 0.01 at the transition, and most of the loss to the final layer occurs after the transition block. All contrasts are now reported per task with intervals.
As the reviewer notes, a four-task average hides these differences.
Table. Change in AUROC per task (five chromosomes, 95% intervals, all probes refitted with one setting).
| Task | 7B, across the transition block | 7B, to the final layer | 40B, across the transition block | 40B, to the final layer |
|---|---|---|---|---|
| Splice donor | −0.033 [−0.040, −0.025] | −0.151 [−0.166, −0.139] | −0.053 [−0.062, −0.045] | −0.216 [−0.234, −0.201] |
| Splice acceptor | −0.034 [−0.042, −0.027] | −0.156 [−0.174, −0.144] | −0.042 [−0.050, −0.035] | −0.155 [−0.173, −0.140] |
| Coding exon | −0.003 [−0.005, −0.001] | −0.057 [−0.067, −0.050] | −0.007 [−0.009, −0.005] | −0.037 [−0.043, −0.032] |
| Mean of the three | −0.023 [−0.027, −0.020] | −0.121 [−0.131, −0.115] | −0.034 [−0.039, −0.030] | −0.136 [−0.144, −0.130] |
For the three-task mean, about 19% of the total decline occurs across the transition block in 7B (0.023 of 0.121 AUROC) and 25% in 40B (0.034 of 0.136); the remaining 81% and 75%, respectively, accumulate through the subsequent blocks and final normalization. These are approximate shares of the point-estimate decline. The region label stays readable, while its single-nucleotide boundaries become harder to read from the transition on.
Most of the remainder falls in the blocks that read the transition block's output: in Evo 2 7B (base checkpoint), annotation AUROC is 0.896 at the transition block, 0.807 one block later and 0.761 after the next two, against 0.774 at the final layer (Appendix G). The transition block starts the decline and the blocks right after it complete it.
The decline is not confined to labels that coincide with a fixed dinucleotide. LTR repeats (−0.092), L1 repeats (−0.043) and enhancer-like elements (−0.025) are read less well across the transition too (Q5), and the decline remains when positive and negative positions are matched on the model's own next-base prediction (−0.072 in 7B, −0.074 in 40B; Appendix E.3).
The background class. The reviewer is right. Our "intergenic" label was a default class: of 4,000 such positions on chr22, 253 (7B) and 263 (40B) lie inside transcripts. We have removed this class from the reported average, which is now the mean of the three tasks above. We also defined a clean intergenic class that excludes every transcript with a 2 kb margin and all cCREs. Under the original four-task average the decline is the same, across twelve probe settings (Appendix E.3).
Revision: Section 3 and Figure 2; per-task values are in Appendix E.2, and the four-task average across probe settings in Appendix E.3.
Q5. Does the result extend beyond gene structure?
Response: Yes, for most classes. Of 18 added annotation classes, including cCREs, transcription-factor binding, CpG islands, repeats and ncRNA, 13 decline at the transition in 7B. The direction depends on the concept, as the reviewer anticipated from Nair et al. (2026).
We agree that gene structure alone was too narrow a scope.
Table. Change in AUROC across the transition block (each class against intron positions; 95% pointwise intervals).
| Annotation | Evo 2 7B | Evo 2 40B |
|---|---|---|
| Distal enhancer-like cCRE | −0.025 [−0.033, −0.017] | −0.017 [−0.025, −0.010] |
| Promoter-like cCRE | +0.031 [+0.021, +0.044] | −0.033 [−0.044, −0.022] |
| CpG island | −0.014 [−0.019, −0.011] | −0.010 [−0.016, −0.005] |
| TF binding: REST | −0.021 [−0.033, −0.008] | −0.015 [−0.026, −0.002] |
| TF binding: YY1 | −0.010 [−0.022, +0.001] | +0.001 [−0.010, +0.013] |
| L1 repeat | −0.043 [−0.050, −0.038] | −0.023 [−0.027, −0.019] |
| LTR repeat | −0.092 [−0.106, −0.083] | −0.056 [−0.070, −0.047] |
| lncRNA exon | −0.007 [−0.024, +0.018] | −0.045 [−0.062, −0.027] |
Enhancer-like elements, CpG islands and repeats decline in both sizes. Promoter-like elements increase in 7B and decline in 40B, and several transcription-factor classes do not change.
We also evaluated a separate regulatory task: classifying ATAC-seq peaks by the cell line in which they are active (five cell lines, Evo 2 7B). Macro AUROC is 0.797 in the block before the transition, 0.773 at the transition block (difference −0.024 [−0.028, −0.019]) and 0.681 at the final layer.
Revision: the regulatory task is in Section 3 and Figure 2; all 18 classes, with their intervals and negative-class definitions, are in Appendix E.5.
Response to Reviewer tDWC (Summary)
We sincerely thank the reviewer for the detailed and constructive review, and for valuing the layer guidance for users of Evo 2 and the finding that the best layer depends on the target. Each of the six weaknesses led to a concrete change: new experiments where evidence was missing, and rewritten text where the presentation obscured the contribution. In brief:
Throughout, annotation AUROC is the mean over coding exon, splice donor and splice acceptor; the background class is discussed under W3.
The detailed responses follow in 4 parts.