acceptodds
Under review as a conference paper at ICLR 2027

Predictive Summarisation in Evo 2

Abstract

Genomic causal language models learn representations through next-nucleotide prediction, yet these representations are also used to infer biological properties of DNA. We investigate how biological readout and predictive function diverge across depth in Evo 2 7B and 40B. Both models exhibit an abrupt representational transition at which variation across genomic positions concentrates in fewer directions. Across this transition, the decline in genomic annotation readout steepens while next-base probe performance improves, including when both are evaluated at the same positions. Restricting standardised transition states to their 16 leading principal directions and continuing pretrained computation preserves native next-base prediction within prespecified NLL and output-KL tolerances, while reducing annotation AUROC by in 7B and in 40B relative to intact transition states. The same restriction before the transition fails to preserve prediction. Annotation decline also persists across probe controls and among positions with similar predictive distributions. We interpret these findings as : a reorganisation of contextual representations for next-base prediction that reduces the accessibility of genomic annotations. Earlier representations selected during development outperform final embeddings by in mean annotation AUROC in a retrospective chromosome 19 evaluation, with benefits extending to gene-structure and conservation readouts. Some variant-difference readouts instead favour final embeddings, showing that layer preference depends on the biological target and feature construction. Next-nucleotide prediction provides indirect supervision for biological properties; representations organised for this objective need not be the most useful for biological analysis.

Primary area:
interpretability and explainable AI
Keywords:
Genomic language models, predictive summarisation, representation analysis, biological readout, layer selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tDWCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+2)
Soundness
2
Read review
Paper summary

The paper examines how biological readout and next-base prediction diverge across depth in Evo 2 (7B, 40B). At one block in each model (b28 in 7B, b21 in 40B), state norms jump more than 200-fold and the standardised effective rank collapses. Across this transition, linear-probe AUROC for gene-structure annotations falls more steeply than in the preceding blocks, while next-base probe accuracy keeps improving.

The key experiment projects the transition state onto its top-16 principal directions and runs the rest of the frozen model. Native prediction stays within prespecified tolerances, but annotation AUROC drops by 0.086 (7B) and 0.059 (40B). The same edit before the transition breaks prediction. The authors call this pattern “predictive summarisation”.

Practically, earlier layers (7B b24, 40B b17) beat final embeddings on annotation, gene finding and conservation. Some variant-effect readouts favor the final layer instead.

Critical strengths
  1. This paper proposes concrete layer guidance for Evo 2 users. Extracting from b24 (7B) or b17 (40B) instead of the final embedding improves annotation. It also improves BEND gene-structure and phyloP readouts. Many people use Evo 2 as a frozen feature extractor, so this is directly actionable.

  2. This paper found the best layer depends on the target. Earlier layers are better for annotation and conservation. The final embedding is better for several variant-effect readouts (ClinVar, GPN-MSA). The preference can also flip with feature construction (BRCA1 position-level vs. pooled features). Showing this experimentally is more useful than the general claim that intermediate layers are better.

Critical weaknesses
  1. Novelty is not positioned against directly relevant prior work. The main practical message, that intermediate layers outperform final embeddings, is already established for genomic LMs.
  • The Nucleotide Transformer paper (Dalla-Torre et al., 2025) reports that the best probing performance is never achieved with the final layer.
  • The Evo 2 paper selects intermediate blocks for its downstream tasks [VERIFY: blocks.26 for exon classification; 40B block 20 for BRCA1], and its repository recommends intermediate embeddings.

The paper cites this work only in passing and does not explain what it adds. A new term, “predictive summarisation,” is not warranted for a largely known phenomenon.

  1. The transition is described but not explained. The paper shows what happens at the transition: norms explode, rank collapses, annotation readout falls and next-base decoding improves. It does not show why.
  • No causal link. Restriction shows that annotation information can be removed without hurting prediction, not that the transition removes it for prediction. Attenuating the transition does not recover annotation. The authors acknowledge that the causal link is unresolved.
  • No training account. The 7B b28 MLP write is 43–113× larger than its input, even for random sequence (Appendix E.2). Why training produces this is not addressed.

The term “predictive summarisation” implies an explanation the evidence does not provide.

  1. The annotation scope is narrow.
  • Only four gene-structure classes are tested (GENCODE v44, each against intron). Regulatory elements, repeats and ncRNAs are absent. Regulatory function is a main use of gLM embeddings and may change with depth differently from gene structure.
  • Nair et al. (2026) show that the best layer and representation type differ across genomic concepts. Here, every transition contrast with intervals is a four-task average. Per-task results are shown only as curves without intervals.
  • The background class is ill-defined. The “intergenic” label is the default for any position without another retained label, so it includes genic positions such as noncoding exonic bases.
  1. Held-out evaluation is weak.
  • Section 3 uses only development chromosomes. The main results are replicated only on new chr22 windows, and reconstruction adds chr17.
  • The only prespecified panel (chr18) covers 7B alone and only the layer comparison.
  • Chr19 is retrospective. BEND uses chr17, chr19 and chr22. All variant panels (ClinVar, GPN-MSA, BRCA1) are on chr17.
  1. Generality beyond Evo 2 is unclear.
  • No comparable transition appears in the other screened models (HyenaDNA, Caduceus, NT-v2; maximum norm ratio about 3). The transition sits at very different relative depths (29/32 in 7B, 22/50 in 40B).
  • The comparison models differ in size, data, architecture and objective, and were selected after inspecting diagnostics. They cannot isolate any single factor.

The finding may therefore be specific to Evo 2 and may not carry over to future gLMs. Combined with the missing mechanism (point 2), its implications for model design are limited.

  1. Weak clarity.

Core terms (“panel,” “window,” “panel D/C2,” “excess change,” “native retention”) are used before they are defined, or are defined only in the appendix. Many sentences stack several qualifications, which makes the main argument hard to follow.

Reference: Akira A. Nair et al. Interpreting Genomic Language Models using Sparse Autoencoders. ICML 2026.

Questions for authors
  1. What is the core contribution beyond prior work? Nucleotide Transformer and Evo 2 already show that intermediate layers outperform final embeddings. What does this paper establish that those works did not?

  2. What should gLM developers do differently because of this work? For example, does it suggest changes to the training objective, the architecture, normalization, or how embeddings are exposed to users? A concrete, testable recommendation would strengthen the paper considerably.

  3. Is the transition specific to Evo 2? No comparable transition appears in HyenaDNA, Caduceus or NT-v2. If it is Evo 2–specific, what is the general lesson for future gLMs?

  4. Does every annotation class decline at the transition? Please report per-class transition contrasts with intervals for the four gene-structure tasks, not only the four-task average.

  5. Does the result extend beyond gene structure? Would regulatory elements (cCREs, enhancers, TF binding sites, CpG islands), repeats or ncRNAs show the same decline? Concept-level analyses such as Nair et al. (2026) suggest that the best layer varies by concept.

7 comments
AuthorsOpenReview ↗

Response to Reviewer tDWC (Summary)

We sincerely thank the reviewer for the detailed and constructive review, and for valuing the layer guidance for users of Evo 2 and the finding that the best layer depends on the target. Each of the six weaknesses led to a concrete change: new experiments where evidence was missing, and rewritten text where the presentation obscured the contribution. In brief:

  • W1 / Q1 (contribution): Prediction quality and biological readout can be dissociated inside one model, and the readout lost late in the network is not required for accurate prediction. Three interventions show this; comparisons of layers cannot (Part 1).
  • W2 (term, explanation): The term is removed. A new retraining experiment shows that accurate prediction does not require the loss of readout: with the same objective, perplexity returns (2.71 vs. 2.68) while final-layer AUROC is 0.92 against 0.78 (Part 1).
  • W3 / Q4 / Q5 (per-class results): Every task declines, most of all the splice sites; contrasts are reported per task with intervals, and the background class is removed. Of 18 new annotation classes, 13 decline at the transition in 7B (Part 2).
  • W4 (held-out evaluation): The results hold on data not used before: both sizes on new chromosomes (+0.113 in 7B, +0.137 in 40B), and variants outside chr17 (Part 3).
  • W5 / Q3 (generality): The transition is present in all three model families examined, and prediction survives the same edit in each (Part 3).
  • Q2, W6 (implications, clarity): Keeping biological features accessible is a goal separate from learning them, with three testable recommendations for developers. The paper is rewritten, with every term defined before use (Part 4).

Throughout, annotation AUROC is the mean over coding exon, splice donor and splice acceptor; the background class is discussed under W3.

The detailed responses follow in 4 parts.

AuthorsOpenReview ↗

Response to Reviewer tDWC (Part 1/4)

W1 / Q1. What is the core contribution beyond prior work?

Response: Our contribution is to show, by intervening inside gLMs, that what a model needs to predict the next nucleotide is not what we need to read biology from it. Prediction quality and biological readout can be dissociated, and the readout lost late in the network is not required for accurate prediction. Prior work established that intermediate layers read better; it could not say why, or whether it has to be so.

We agree that the submission did not position itself clearly against the Nucleotide Transformer and Evo 2 papers. Those papers, and the choices the reviewer cites (blocks.26 in 7B and block 20 in 40B, both before the transition at blocks 28 and 21), tell a user where to read. A comparison of layers cannot go further, because two layers differ in many ways at once.

The question. gLMs are trained on the premise that a model which predicts DNA well must come to represent the biology behind it. Most of that biology cannot be requested in the output; it is read from hidden states. The premise is therefore testable: does the model's prediction depend on the biology its states encode? We test it by intervention, measuring the model's own prediction and a probe readout side by side.

What the interventions show.

  • Prediction does not need everything a probe reads. Keeping 16 of 4,096 directions after the transition changes perplexity by less than 1%, while annotation AUROC falls from 0.931 to 0.836 (−0.095 [−0.116, −0.073]). Sixteen random directions, or the same edit one block earlier, do not preserve the prediction. Prediction survives the same edit in all five models, from three families (Part 3).
  • Better prediction does not mean better readout. A module trained away from next-nucleotide prediction predicts far worse (perplexity 4.6 vs. 1.6 in Genos-1.2B) and leaves a final layer that is easier to read (AUROC 0.74 vs. 0.69).
  • The loss of readout is not the price of predicting well. After the late blocks are reset and retrained with the same objective, perplexity returns (2.71 vs. 2.68) while final-layer AUROC is 0.92 against 0.78, in both of two runs.

What this changes. Two models whose perplexities differ by 1% can differ this much in what can be read from them. Perplexity, by which these models are trained and compared, therefore cannot stand in for readout, which has to be evaluated and designed for in its own right (Part 4).

Nair et al. (2026) report that the best layer differs across genomic concepts; our per-class contrasts (Part 2) agree, and relate each class to the transition.

Revision: Sections 1 and 6 state the contribution, Section 5 separates it from the layer-wise observations, and the interventions are in Sections 4.1 to 4.3.


W2. The transition is described but not explained; the term implies an explanation.

Response: Accurate prediction does not require the loss of readout: in a new experiment, retraining the late blocks recovers the prediction with the readout intact. We removed the term, which implied an explanation we had not tested. Why pretraining produces the transition remains open, and the revision says so.

We agree with both points: the submission described the transition without explaining it, and the term implied more than the evidence supported.

The term. The revision no longer uses "predictive summarisation", in the title or in the text. It states what is measured: under the same edit of a state, the model's prediction is preserved and annotation readout falls. That the state after the transition is organized around the directions that set the next-base distribution is now stated as a hypothesis and tested by intervention (Section 4).

The explanation. We added retraining experiments that ask whether the loss of readout is the price of predicting well. In the base checkpoint of Evo 2 7B we reset the last four blocks, from the transition block on, to a fresh initialization and retrain them with next-nucleotide prediction (two runs of 1,000 steps).

NLLAnnotation AUROC, final layer
Original model0.9880.779
Reset and retrained, 1,000 steps (two runs)0.994, 0.9940.932 [0.914, 0.945], 0.925 [0.906, 0.939]

NLL returns close to the original value, and final-layer annotation AUROC is 0.93 and 0.92, close to the 0.95 read before the transition. The amount lost at the final layer is therefore not fixed by prediction performance. The state still narrows in the retrained model (effective rank 311 to about 7), but gradually, over three blocks. Whether still longer training restores the abrupt contraction, and why pretraining produces the large write of the transition block, as the reviewer asks, remain open, and the revision says so.

Revision: Section 4.3, Figure 5 and Appendix H.

AuthorsOpenReview ↗

Response to Reviewer tDWC (Part 2/4)

W3 / Q4. Does every annotation class decline at the transition? Please report per-class contrasts with intervals.

Response: Every task declines, but unevenly. The loss is concentrated in the splice sites, while coding exons change by less than 0.01 at the transition, and most of the loss to the final layer occurs after the transition block. All contrasts are now reported per task with intervals.

As the reviewer notes, a four-task average hides these differences.

Table. Change in AUROC per task (five chromosomes, 95% intervals, all probes refitted with one setting).

Task7B, across the transition block7B, to the final layer40B, across the transition block40B, to the final layer
Splice donor−0.033 [−0.040, −0.025]−0.151 [−0.166, −0.139]−0.053 [−0.062, −0.045]−0.216 [−0.234, −0.201]
Splice acceptor−0.034 [−0.042, −0.027]−0.156 [−0.174, −0.144]−0.042 [−0.050, −0.035]−0.155 [−0.173, −0.140]
Coding exon−0.003 [−0.005, −0.001]−0.057 [−0.067, −0.050]−0.007 [−0.009, −0.005]−0.037 [−0.043, −0.032]
Mean of the three−0.023 [−0.027, −0.020]−0.121 [−0.131, −0.115]−0.034 [−0.039, −0.030]−0.136 [−0.144, −0.130]

For the three-task mean, about 19% of the total decline occurs across the transition block in 7B (0.023 of 0.121 AUROC) and 25% in 40B (0.034 of 0.136); the remaining 81% and 75%, respectively, accumulate through the subsequent blocks and final normalization. These are approximate shares of the point-estimate decline. The region label stays readable, while its single-nucleotide boundaries become harder to read from the transition on.

Most of the remainder falls in the blocks that read the transition block's output: in Evo 2 7B (base checkpoint), annotation AUROC is 0.896 at the transition block, 0.807 one block later and 0.761 after the next two, against 0.774 at the final layer (Appendix G). The transition block starts the decline and the blocks right after it complete it.

The decline is not confined to labels that coincide with a fixed dinucleotide. LTR repeats (−0.092), L1 repeats (−0.043) and enhancer-like elements (−0.025) are read less well across the transition too (Q5), and the decline remains when positive and negative positions are matched on the model's own next-base prediction (−0.072 in 7B, −0.074 in 40B; Appendix E.3).

The background class. The reviewer is right. Our "intergenic" label was a default class: of 4,000 such positions on chr22, 253 (7B) and 263 (40B) lie inside transcripts. We have removed this class from the reported average, which is now the mean of the three tasks above. We also defined a clean intergenic class that excludes every transcript with a 2 kb margin and all cCREs. Under the original four-task average the decline is the same, across twelve probe settings (Appendix E.3).

Revision: Section 3 and Figure 2; per-task values are in Appendix E.2, and the four-task average across probe settings in Appendix E.3.


Q5. Does the result extend beyond gene structure?

Response: Yes, for most classes. Of 18 added annotation classes, including cCREs, transcription-factor binding, CpG islands, repeats and ncRNA, 13 decline at the transition in 7B. The direction depends on the concept, as the reviewer anticipated from Nair et al. (2026).

We agree that gene structure alone was too narrow a scope.

Table. Change in AUROC across the transition block (each class against intron positions; 95% pointwise intervals).

AnnotationEvo 2 7BEvo 2 40B
Distal enhancer-like cCRE−0.025 [−0.033, −0.017]−0.017 [−0.025, −0.010]
Promoter-like cCRE+0.031 [+0.021, +0.044]−0.033 [−0.044, −0.022]
CpG island−0.014 [−0.019, −0.011]−0.010 [−0.016, −0.005]
TF binding: REST−0.021 [−0.033, −0.008]−0.015 [−0.026, −0.002]
TF binding: YY1−0.010 [−0.022, +0.001]+0.001 [−0.010, +0.013]
L1 repeat−0.043 [−0.050, −0.038]−0.023 [−0.027, −0.019]
LTR repeat−0.092 [−0.106, −0.083]−0.056 [−0.070, −0.047]
lncRNA exon−0.007 [−0.024, +0.018]−0.045 [−0.062, −0.027]

Enhancer-like elements, CpG islands and repeats decline in both sizes. Promoter-like elements increase in 7B and decline in 40B, and several transcription-factor classes do not change.

We also evaluated a separate regulatory task: classifying ATAC-seq peaks by the cell line in which they are active (five cell lines, Evo 2 7B). Macro AUROC is 0.797 in the block before the transition, 0.773 at the transition block (difference −0.024 [−0.028, −0.019]) and 0.681 at the final layer.

Revision: the regulatory task is in Section 3 and Figure 2; all 18 classes, with their intervals and negative-class definitions, are in Appendix E.5.

Reviewer_q2S6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors perform a targeted analysis of Evo2 embeddings, to understand how information about biological annotations and next nucleotide prediction varies in the model. Through this, the authors identify a phenomenon they call predictive summarisation, ie that there are transitions where the model preserves information about next nucleotide prediction whilst degrading performance in biological annotations. Practically, this suggests that earlier layers in the model are more useful for certain biological tasks compared to later layers in the model.

Critical strengths

This is a thorough empirical analysis of the embeddings within Evo 2. The paper goes beyond evaluating layerwise performance for certain tasks, and looks at other properties, such as changes in the activation norms and rank to identify sharp transitions in the representations. This offers additional insights beyond just performance of individual embedding layers within Evo 2.

Critical weaknesses

The key weakness is that the insights and techniques do not seem to offer much additional novelty. The main practical finding demonstrated is that the earlier layer embeddings are more useful for certain downstream biological tasks, whereas later layers are more specialised for next-nucleotide prediction. However, the importance of embedding depth and the observation that intermediate layers can outperform final layer embeddings in DNA models have already been demonstrated in prior work, including the Nucleotide Transformer v2 and original Evo 2 papers. Furthermore, the technical novelty of the analysis is limited. Previous studies have investigated changes in representation geometry across layers of language models, and their relationship to downstream representation quality (e.g. [1]). It remains unclear whether the proposed techniques and concepts represent a fundamentally new phenomenon or primarily extend existing observations and techniques to Evo 2.

The analysis and insights are limited to Evo 2, so the authors do not establish if predictive summarisation is a general phenomenon. This is an important weakness, as it limits the broad utility of these insights across the field of DNA models.

[1] Skean et al Layer by layer: Uncovering hidden representations in language models. ICML, 2025.

Questions for authors

What do the authors consider to be the key new insight established by predictive summarisation compared to other work? In particular, how does this phenomenon differ from existing work that observes layerwise specialisation towards the pretraining objective?

To what extent do the authors expect the insights to apply to models other than Evo 2?

6 comments
AuthorsOpenReview ↗

Response to Reviewer q2S6 (Summary)

We sincerely thank the reviewer for the clear summary, for recognizing that the analysis goes beyond a ranking of layers, and for raising the positioning question directly. The two questions led us to new experiments, and both now have a direct answer. In brief:

  • Q1 (new insight): Prediction quality and biological readout can be dissociated inside one model, and the readout lost late in the network is not required for accurate prediction. Three interventions show it: editing a state (perplexity within 1%, annotation AUROC 0.93 to 0.84), training a module away from next-nucleotide prediction (worse perplexity, better readout), and, in a new experiment, retraining the late blocks (perplexity 2.71 vs. 2.68, final-layer AUROC 0.92 against 0.78). Perplexity therefore cannot stand in for readout (Part 1).
  • Q2 (generality): The result is not specific to Evo 2. In new experiments on CENO and Genos, two independently developed families, the same edit preserves prediction in all five models and lowers readout in four (Part 2).

The detailed responses follow in 2 parts.

AuthorsOpenReview ↗

Response to Reviewer q2S6 (Part 1/2)

Q1. What is the key new insight compared to prior work on layerwise specialization?

It remains unclear whether the proposed techniques and concepts represent a fundamentally new phenomenon or primarily extend existing observations and techniques to Evo 2.

Response: The new insight is that what a gLM needs to predict the next nucleotide is not what we need to read biology from it. By intervening inside the model, we show that prediction quality and biological readout can be dissociated, and that the readout lost late in the network is not required for accurate prediction. Prior work shows where to read; it could not say why, or whether it has to be so.

We agree with the reviewer that better intermediate-layer embeddings would not, by themselves, constitute a new finding. The work the reviewer cites compares layers: the Nucleotide Transformer and Evo 2 papers report that intermediate layers give better features, and Skean et al. (2025) relate the geometry of each layer to the quality of its representations in language models. These studies tell a user where to read. A comparison of layers cannot go further, because two layers differ in many ways at once.

The question. gLMs are trained on the premise that a model which predicts DNA well must come to represent the biology behind it. Most of that biology cannot be requested in the output; it is read from hidden states. The premise is therefore testable: does the model's prediction depend on the biology its states encode? We test it by intervention, measuring the model's own prediction and a probe readout side by side.

Does prediction need what a probe reads? We keep the 16 leading principal directions of the state right after the transition block of Evo 2 and run the remaining blocks unchanged. On chromosomes not used in any earlier analysis, NLL changes by +0.007 / +0.004 nats in 7B while annotation AUROC at that state falls from 0.931 to 0.836 (−0.095 [−0.116, −0.073]); in 40B it falls by 0.060 [0.047, 0.073]. The result is specific: 16 random directions raise NLL by 0.45 to 0.56 nats, and the same edit one block earlier does not preserve the prediction.

Does better prediction mean better readout? Not necessarily. A module trained away from next-nucleotide prediction predicts far worse (perplexity 4.6 vs. 1.6 in Genos-1.2B) and leaves a final layer that is easier to read (AUROC 0.74 vs. 0.69).

Is the loss necessary? After the last four blocks of Evo 2 7B are reset and retrained with the same objective, perplexity returns to within 1% of the original (2.71 vs. 2.68) while annotation AUROC at the final layer is 0.92, against 0.78 in the original model. A second seed gives the same result, and at 1,000 steps both runs have a lower NLL and an AUROC of 0.93 and 0.92.

Comparable predictive performance can therefore coexist with substantially different biological readout, which layer rankings cannot show. Perplexity, by which these models are trained and compared, cannot stand in for readout; readout has to be evaluated and designed for in its own right.

Revision: Sections 1 and 6 state the contribution, Section 5 separates it from the established layer-wise observations, and the interventions are in Sections 4.1 to 4.3. Annotation AUROC is now a three-task mean (see the General response), so its level differs from the 0.744 →\to 0.659 reported in the submission.

AuthorsOpenReview ↗

Response to Reviewer q2S6 (Part 2/2)

Q2. To what extent do the insights apply to models other than Evo 2?

The analysis and insights are limited to Evo 2, so the authors do not establish if predictive summarisation is a general phenomenon.

Response: The dissociation is not specific to Evo 2. Under the same state edit, it recurs in two further, independently developed families of next-nucleotide models.

We agree that a result confined to Evo 2 would be of limited use, and we have therefore tested it in other models. CENO (1B) and Genos (1.2B and 10B) were developed independently of Evo 2. We identify the transition from the state itself: the block at which the effective rank falls sharply and is not restored by any later block. The norm grows with it in Evo 2 and CENO but only about fourfold in Genos, so the norm is not our criterion.

Table. The transition block in five models.

ModelRelative depthGrowth of the state normEffective rank after the transition, relative to one block earlier
Evo 2 7B0.91more than 100-fold4%
Evo 2 40B0.44more than 100-fold3%
CENO0.95more than 1,000-fold1%
Genos-1.2B1.00about 4-fold6%
Genos-10B1.00about 4-fold29%

We then repeated the state edit of the submission in each model. The edited state is passed through the remaining pretrained blocks, so the NLL is the model's own. The number of directions was fixed for each model before this evaluation.

Table. State edit in five models on chr6 and chr13 (not used before). The AUROC change is the mean over the two chromosomes, with probes retrained on the edited state. The random-direction column is the range over three draws on chr5 and chr11.

Model (state edited)Directions kept, of widthNLL change, leading directionsNLL change, random directionsAnnotation AUROC change [95% interval]
Evo 2 7B (after transition)16 of 4,096+0.007 / +0.004+0.45 to +0.46−0.095 [−0.116, −0.073]
Evo 2 40B (after transition)16 of 8,192−0.0001 / −0.0003+0.54 to +0.56−0.060 [−0.073, −0.047]
CENO (last block)64 of 1,024+0.002 / +0.003+0.40 to +0.43−0.035 [−0.042, −0.029]
Genos-10B (after transition)64 of 4,096+0.0008 / +0.0005+1.47 to +1.59−0.040 [−0.062, −0.020]
Genos-1.2B (after transition)256 of 1,024+0.004 / +0.002+0.73 to +0.84−0.007 [−0.020, +0.005]

In all five models the leading directions carry what the rest of the model needs to predict, and the same number of random directions does not. Part of the random-direction cost reflects the larger perturbation: with the size matched, random directions still cost about ten times more in Evo 2 7B and Genos-1.2B and almost nothing in the other three (Appendix F.1). What is specific to the leading directions is the pair of outcomes: in Evo 2 7B, 16 random directions lower the readout about as much (0.07 to 0.15) but raise NLL by 0.45 nats, whereas on the 16 leading directions the prediction survives.

In four models the same edit lowers annotation readout. Three differences between the models are reported in the revision:

  • In Genos the contraction is the mildest of the three families and sits at the last block (rank to 6% and 29% of its previous value, norm about fourfold), so for Genos our claim rests on the state edit and not on the shape of the transition. In Genos-1.2B the readout does not change under the edit; there annotation AUROC declines at the final normalization (0.71 to 0.67).
  • In CENO the result holds at the last block. Right after its transition block, 64 directions are not enough to predict (NLL +0.38 / +0.26).
  • One block before the transition, the same edit costs +0.446 / +0.483 nats in Genos-10B and +0.847 / +0.625 nats in Genos-1.2B (Appendix F.2), as it does in Evo 2.

Nucleotide Transformer v2, a masked model, shows no such late fall in rank. The finding is therefore not specific to Evo 2 or to its architecture: in all three families the transition is present and the same predictive pattern recurs.

Revision: Sections 3 and 4.1, Table 1 and Figure 3 cover all five models; all values are in Appendix F.1, and the full profiles and the control models are in Appendix E.1.

Reviewer_NAdcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Investigation of how biological information changes across the layers of Evo 2 (7B and 40B)
  • Analysis of hidden states using probes and PCA
  • Identification of an abrupt shift in representations within specific layers
  • Next-nucleotide prediction performance improves, while the extractability of biological annotations declines
  • As few as 16 PCA components are sufficient to largely preserve sequence prediction capabilities
  • Earlier layers yield better results for certain downstream biological tasks than the final embeddings
  • Introduction of the concept of predictive summarisation to describe this shift in representations-
  • It remains unclear whether biological information is truly lost and what causes this shift in representations
Critical strengths

1. The paper explores an important question: how does the accessibility of biological information change within genomic foundation models? The authors show that next-nucleotide prediction improves, while the decodability of biological annotations decreases. This finding is relevant for selecting appropriate hidden states for downstream biological tasks.

2. The authors combine geometric analyses of the hidden states with linear and non-linear probes, as well as targeted PCA interventions. It is particularly compelling that reducing the representation to 16 PCA components preserves the original nucleotide prediction within defined tolerances, while the annotation AUROC drops significantly. Additional control experiments examine various probe configurations, sequence lengths, and genomic positions.

3. The authors demonstrate that, for tasks involving gene structure annotation and evolutionary conservation, earlier layers can yield better results than final embeddings. The evaluation includes, among others, GENCODE, BEND, ClinVar, and the effects of BRCA1 variants. Notably, the authors also identify tasks for which final embeddings are superior, thereby avoiding the claim that earlier layers are universally better.

Critical weaknesses

1. The authors demonstrate a connection between changes in hidden-state representations, improved next-base decodability, and a reduced Auroc for biological annotations. While the PCA interventions show that biological information can become less decodable without significantly compromising the model's original predictions, it remains unclear whether this natural shift in representations is actually caused by optimization for next-nucleotide prediction, or if it merely accompanies it. Analyzing various training checkpoints could help to better understand the emergence of this phenomenon.

2. The core experiments focus on Evo 2 7B and 40B, two model sizes from the same architectural family. Most analyses utilize human DNA sequences with a length of 6 kb, although Evo 2 is capable of processing longer genomic contexts as well. While the authors do examine additional models and longer sequences, they do not conduct the primary intervention experiments for these cases with the same level of depth. Consequently, it remains an open question whether "predictive summarisation" is a general phenomenon among autoregressive genomic foundation models or a specific characteristic of Evo 2.

3. The optimal earlier layers were selected based on developmental analyses conducted on chromosomes 17 and 22. The subsequent evaluation on chromosome 19 is, according to the authors, retrospective, as earlier results had already been taken into account during the development process. Furthermore, the reported confidence intervals do not account for the uncertainty introduced by the layer selection process and the retraining of the probes. The additional evaluation on chromosome 18 supports the findings for Evo 2 7B. However, a comparable, independent validation for both model sizes would be desirable.

Questions for authors
  1. How can the authors demonstrate that the observed change in representation is actually caused by the optimization of next-nucleotide prediction, rather than merely being correlated with it?

2.To what extent can these findings be extended to other genomic foundation models, different model architectures, and longer DNA sequences?

  1. How robust are the reported advantages of earlier layers with respect to layer selection and the retraining of probes? Could the authors conduct a fully independent evaluation for both model sizes that accounts for these uncertainties?
6 comments
AuthorsOpenReview ↗

Response to Reviewer NAdc (Summary)

We sincerely thank the reviewer for the supportive and precise assessment, and for recognizing both the 16-direction intervention and our reporting of the tasks on which the final layer performs better. Each of the three questions asked for evidence the submission did not have; we have run the corresponding experiments, and each question now has a direct answer. In brief:

  • W1 / Q1 (cause): A new retraining experiment. When the last four blocks are reset and retrained with next-nucleotide prediction, perplexity returns (2.71 vs. 2.68) but the loss of readout does not (final-layer AUROC 0.92 against 0.78, in both of two runs). The objective does not require the loss (Part 1).
  • W2 / Q2 (generality): The intervention now runs in two further model families and at 24 kb. Prediction is preserved in all five models, and readout falls in four (Part 2).
  • W3 / Q3 (independent evaluation): Both sizes on chromosomes not used before, with layer selection and probe refitting inside the bootstrap: +0.113 [+0.098, +0.138] in 7B and +0.137 [+0.118, +0.165] in 40B (Part 3).

The detailed responses follow in 3 parts.

AuthorsOpenReview ↗

Response to Reviewer NAdc (Part 1/3)

W1 / Q1. Is the change in representation caused by optimization for next-nucleotide prediction, or does it merely accompany it?

Analyzing various training checkpoints could help to better understand the emergence of this phenomenon.

Response: The objective causes the state to narrow, but it does not require the loss of readout. When the late blocks are reset and retrained with next-nucleotide prediction, the state narrows again in every run, while the abrupt contraction and the loss of readout do not return.

As the reviewer notes, the submission could not separate cause from accompaniment. We therefore ran a retraining experiment and, as suggested, compared the released checkpoints. How the abrupt form first arises during pretraining remains open, and the revision states this as a limitation.

Checkpoints. The public Evo 2 releases are training stages, not a dense trajectory. We compared the ones that exist: the transition is already present in the base checkpoints of both sizes, before the context extension, with a norm increase of more than 300-fold at the same block.

Resetting and retraining the late blocks. In the base checkpoint of Evo 2 7B we reset the transition block and every block after it, four blocks in all, to a fresh initialization, and retrain them with next-nucleotide prediction, keeping the first 28 blocks fixed (two runs of 1,000 steps).

Table. Reset and retraining of the last four blocks (384 test sequences; annotation AUROC is the mean over coding exon, splice donor and splice acceptor).

NLLEffective rank after the four blocksAnnotation AUROC, final layer
Original model0.9882.10.779
Full reset, 500 steps (two runs)0.997, 0.9976.7 (first run)0.924, 0.921; first run +0.145 [+0.126, +0.171] over the original
The same two runs at 1,000 steps0.994, 0.9947.7, 5.80.932 [0.914, 0.945], 0.925 [0.906, 0.939]
Reset keeping a quarter of the weights, 250 steps (two runs)0.996, 0.9962.6, 3.30.755, 0.762
The same two runs at 500 steps0.993, 0.9933.1, 3.00.781, 0.776

NLL returns close to the original value and the state narrows again, but the loss of readout does not return: final-layer AUROC is 0.92 at 500 steps and 0.93 and 0.92 at 1,000, close to the 0.95 read before the transition. The retrained and original models share their first 28 blocks, and their perplexity is 2.71 against 2.68 at 500 steps, within 1%. The retrained model narrows the state over three blocks, with the norm growing eight- to tenfold, where the original transition block alone multiplies it 339-fold.

Where the cost comes from. When the reset keeps a quarter of the weights at their pretrained values, the convolution filters among them (two runs), retraining reconnects the convolution branch of one block within 100 steps: the effective rank collapses to about 3 in a single block, AUROC falls from 0.947 to 0.76 and 0.74 right after it, and final-layer AUROC stays about where the original model was (third row of the table). The contraction disappears when that branch is switched off. Resetting only the transition block gives an intermediate case: the next block contracts the state instead, and final-layer AUROC is 0.844 at NLL 0.992 (three runs).

Is the readout bought with the remaining NLL? Three observations argue against it. (i) Across the retrained models NLL lies within 0.005 nats (0.992 to 0.997) while final-layer AUROC ranges from 0.76 to 0.93. (ii) Within a run, further training lowers NLL without lowering readout: between 500 and 1,000 steps of the full reset, NLL falls from 0.997 to 0.994 in both runs while AUROC rises from 0.924 to 0.932 and from 0.921 to 0.925. (iii) Where the readout is lost, it is lost in one identifiable branch, and switching that branch off removes the contraction. What these runs do not cover is the last 0.006 nats to the original model.

The amount lost at the final layer is therefore not fixed by prediction performance; it depends on how the model narrows the state. The full reset is two runs of 1,000 steps; whether still longer training brings the abrupt contraction back, and how the transition first forms during pretraining, remain open.

Is the information lost, or only harder to read? The reviewer's summary raises this. Both occur. At the final layer of Evo 2 a nonlinear probe reads only 0.01 to 0.02 better than the linear one, so a stronger probe does not bring the readout back. In CENO the last block restores the readout that the transition lowered, so there the information is present but harder to read (Appendix E.3).

Revision: Section 4.3 and Figure 5; the reset, the variant that keeps pretrained filters and the component ablation are in Appendix H.1, the single-block reset in Appendix H.3, and the base checkpoints in Appendix E.1.

AuthorsOpenReview ↗

Response to Reviewer NAdc (Part 2/3)

W2 / Q2. To what extent do the findings extend to other genomic foundation models, architectures, and longer sequences?

While the authors do examine additional models and longer sequences, they do not conduct the primary intervention experiments for these cases with the same level of depth.

Response: The findings extend to both. In two further families of next-nucleotide models and at a 24 kb context, the same intervention gives the same result: prediction runs on a few leading directions of the state, so what a model predicts with is separate from what can be read from it.

As the reviewer points out, the submission ran the primary intervention only in Evo 2 and at 6 kb. We have now run it with the same protocol in other models and at a longer context.

Other models. We identify the transition from the state itself: the block at which the effective rank falls sharply and is not restored by any later block (to 4% and 3% of its previous value in Evo 2 7B and 40B, 1% in CENO, 6% in Genos-1.2B and 29% in Genos-10B). We repeated the state edit of the submission in each model, continuing the pretrained computation after the edit. The number of directions was fixed for each model before this evaluation.

Table. State edit in five models on chr6 and chr13 (not used before). Annotation AUROC is the mean over coding exon, splice donor and splice acceptor; its change is the mean over the two chromosomes, with probes retrained on the edited state. The random-direction column is the range over three draws on chr5 and chr11.

Model (state edited)Directions kept, of widthNLL change, leading directionsNLL change, random directionsAnnotation AUROC change [95% interval]
Evo 2 7B (after transition)16 of 4,096+0.007 / +0.004+0.45 to +0.46−0.095 [−0.116, −0.073]
Evo 2 40B (after transition)16 of 8,192−0.0001 / −0.0003+0.54 to +0.56−0.060 [−0.073, −0.047]
CENO (last block)64 of 1,024+0.002 / +0.003+0.40 to +0.43−0.035 [−0.042, −0.029]
Genos-10B (after transition)64 of 4,096+0.0008 / +0.0005+1.47 to +1.59−0.040 [−0.062, −0.020]
Genos-1.2B (after transition)256 of 1,024+0.004 / +0.002+0.73 to +0.84−0.007 [−0.020, +0.005]

In all five models the leading directions carry what the rest of the model needs to predict, and the same number of random directions does not; with the perturbation size matched, random directions still cost about ten times more in Evo 2 7B and Genos-1.2B (Appendix F.1). In four models the same edit lowers annotation readout. In Genos-1.2B the readout does not change under the edit; in that model the decline occurs at the final normalization. In CENO the result holds at the last block, while right after its transition block 64 directions are not enough to predict (NLL +0.38 / +0.26).

Longer sequences. We ran the 16-direction edit at 6 kb and at 24 kb on the same sequences, in both sizes. The criteria of the submission for the model's own prediction (NLL change within $\pm$0.01 nats, KL at most 0.01) are met in all eight combinations, and the decline in annotation readout at 24 kb is close to that at 6 kb.

Table. Annotation AUROC change at the edited state [95% interval].

Model, chromosome6 kb24 kb
Evo 2 7B, chr5−0.036 [−0.064, −0.017]−0.031 [−0.058, −0.009]
Evo 2 7B, chr11−0.079 [−0.110, −0.058]−0.065 [−0.093, −0.045]
Evo 2 40B, chr5−0.046 [−0.064, −0.031]−0.033 [−0.055, −0.019]
Evo 2 40B, chr11−0.030 [−0.049, −0.014]−0.026 [−0.039, −0.009]

The megabase contexts that Evo 2 supports are beyond what we could run for the intervention.

Revision: Sections 3 and 4.1, Table 1 and Figure 3 cover all five models, with all values in Appendix F.1. The 24 kb results are also in Appendix F.1, where AUROC is the mean of four tasks; the table above uses the three-task mean.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for asking us to clarify what the paper contributes and what it means for the development of genome language models (gLMs). Guided by these comments, we rewrote the manuscript so that the question, the contribution and the scope of each claim are stated up front, and we extended the evidence.

Our contribution is to show, by intervening inside pretrained gLMs, that prediction quality and biological readout can be dissociated, and that the readout lost at the final layer need not be an unavoidable cost of accurate prediction.

This distinction matters because autoregressive gLMs are trained to predict the next nucleotide, whereas most of what users want from them, such as annotation, cannot be requested in the output and is read from hidden states. Perplexity, by which these models are trained and compared, does not show what can be read. Three interventions test how the two relate (Sections 4.1 to 4.3).

  • Editing a state. In Evo 2 7B, keeping 16 of 4,096 directions changes perplexity by less than 1%, while annotation AUROC falls from 0.93 to 0.84. The model has learned biology that the rest of its computation does not need in order to predict.
  • Changing what one module is trained on. Continuing to train a single module on a different target makes next-nucleotide perplexity far worse and leaves a final layer that is easier to read (Genos-1.2B: perplexity 4.6 vs. 1.6, AUROC 0.74 vs. 0.69). A shuffled-target control gives a similar gain, so the gain does not depend on what the new target teaches. A lower perplexity does not by itself mean a more readable model.
  • Retraining the late blocks. With the same objective, the model recovers near-original perplexity (2.71 vs. 2.68), while annotation AUROC at the final layer rises from 0.78 to 0.92. A second seed and 1,000 steps of training give the same result.

The dissociation is not specific to Evo 2. Autoregressive gLMs that tokenize single nucleotides are recent and few; we examined five public models from three independently developed families. In all five, the leading directions of the state preserve the prediction, and in four the same edit lowers annotation readout.

For developers, the distinction is between learning biological features and keeping them accessible. Readout should be evaluated alongside perplexity, and how the late layers retain what a model has learned is a design target in its own right.

We state the scope of each claim. How the transition first emerges during pretraining remains open, as does whether retraining beyond the 1,000 steps we ran brings the abrupt contraction back.

Where to start in the revised manuscript. The reviews showed us that the submission did not make its question clear, and we believe the rewritten text now does. The first three pages (the Introduction with Figures 1 and 2) give the whole argument, each part of Section 4 answers one question with one intervention, and Table 3 in Appendix A maps every claim to its evidence. We would be grateful if the reviewers could look at these pages when judging whether the concerns are resolved.

Changes to the manuscript. All findings of the submission are retained. The revision differs from it in three ways, each following a review.

  • Same results, clearer presentation. The transition, the state edit, the comparison of layers and the variant results are those of the submission. The text is reorganised around one question (clarity: tDWC W6), so a pdfdiff marks most pages even where the content is the same.
  • Experiments the reviews asked for. The state edit in two further model families and at 24 kb (q2S6, NAdc, tDWC); evaluation on chromosomes not used before (NAdc, tDWC); per-task and per-class results (tDWC); and the training-target and retraining experiments, which test the explanation that NAdc and tDWC asked about.
  • Wording and metric. The term "predictive summarisation" implied an untested explanation (tDWC), so we removed it, and the title is now "Prediction and Biological Readout Come Apart in Autoregressive Genome Language Models". Annotation AUROC is now the mean over coding exon, splice donor and splice acceptor, without the background class, so its level differs from the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.