acceptodds
Under review as a conference paper at ICLR 2027

Emergent Codon Frame Representations Reveal Mechanisms of Translation in Genomic Foundation Models

Abstract

Interpreting scientific foundation models requires both identifying scientific variables in model representations and showing that the model uses those variables in its computation. As a test case with known ground truth, we first investigate how the triplet structure of coding RNA emerges in the representations of four genomic foundation models. Across all four models, representations of RNA coding regions exhibit stronger period-3 structure than non-coding regions, and in three of the four, coding region representations projected onto their top two principal components reorganize from nucleotide identity toward codon phase with depth. We use these findings as motivation to look more closely at the RNA-to-protein translation mechanism in MIMIC, a multimodal encoder-decoder foundation model trained jointly on RNA and protein modalities, and show that the codon phase representation is causally used for protein decoding. Rotating RNA representations from MIMIC's encoder by in a two-dimensional codon phase subspace shifts the model’s reading frame for protein decoding by one nucleotide. The inverse rotation induces the opposite shift and repeated rotations compose according to the expected modulo-three structure. Tracing the computation downstream, we find that amino acid identity becomes accessible through the decoder's own cross-attention channel late in encoder representations. A single cross-attention head in the decoder's final block reads the amino acid from the encoder's output representation at the codon's middle nucleotide, which it preferentially attends to, and supplying this position alone recovers the correct amino acid in the single-residue-masked assay. In a natural transcript with overlapping protein-coding reading frames, phase rotation and start-site conditioning separately control translation register and origin, together recovering the alternative protein. These results characterize mechanisms of protein translation in MIMIC and link an emergent biological coordinate to the computations that determine protein output.

Primary area:
interpretability and explainable AI
Keywords:
genomic foundation models, mechanistic interpretability, representation learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_7YaZOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This study looks at genomic foundation models and how they represent a nucleotide’s position within its codon (codon phase). Across four models, coding regions show stronger three-nucleotide periodicity than non-coding regions, and in three of them, deeper layers organize around codon phase. Rotating RNA models like MIMIC by 120 degrees in a fitted phase plane actually shifts the decoded reading frame by just one nucleotide while three full rotations restore everything. The authors trace one decoder attention head reading amino acid identity mostly from the very middle nucleotide of each single codon. Finally, on the POLG gene, combining rotation with start-site conditioning produces an alternative protein encoded in an overlapping frame.

Critical strengths
  • The reading-frame steering result is clean and well controlled. A 120 degree rotation shifts the frame by one nucleotide, rotations compose as expected, and the effect stays local when applied to a single exon, while random-plane, scaling and UTR controls have little effect. Earlier work showed that codon phase can be decoded from genomic models; this result shows that the model actually uses it.
  • I also found the readout built only from the model’s own weights useful, because it separates what a probe can decode from what the decoder actually reads. Amino acid identity is decodable early in the encoder but reaches the decoder only in the late blocks, which is a good lesson for interpreting biological models more generally.
Critical weaknesses
  • The mechanism is traced only at its two ends. Steering works up to blocks 15 to 17 and amino acid identity reaches the decoder after blocks 18 to 19, but the step in between, where phase and nucleotides are combined into an amino acid, is not examined. The single-head result also seems limited to the assay where only one residue is hidden, and the Discussion itself notes that other routes compensate in less constrained settings. The POLG case is weaker than the abstract suggests: conditioning on the real POLGARF start fails, and the version that works uses a start one nucleotide off, supplies 15 protein residues, and is tested on a single transcript. Since both offsets were chosen with the answer known, I read this as a demonstration that the two controls compose, and not as recovery of the alternative protein.
  • Some claims need stronger baselines. Because the phase plane is built from the three phase centroids, a 120 degree rotation roughly moves each token to the next centroid. The three plateaus and the three-fold composition would follow from the model reading phase as three categories, so they do not by themselves show a circular geometry. In the periodicity analysis, coding DNA is already period-3 at the sequence level, so the 85x enrichment needs the raw sequence as a reference, and the layer reported in Figure 2c is chosen to maximize the contrast. Finally, HyenaDNA’s weak result is attributed to its lack of attention, although it is also far smaller than the other three models.
Questions for authors
  • How many protein residues are visible to the decoder in the steering experiments, and does steering still work with the entire protein hidden?
  • What happens in encoder blocks 17 to 19 when a rotation is applied?
  • Does combining start-site conditioning with rotation recover alternative proteins in other transcripts with overlapping frames? (without choosing the start offset by hand)
4 comments

We thank the reviewer for their helpful clarification questions and specific suggestions to explore the mechanism further.

The mechanism is traced only at its two ends. Steering works up to blocks 15 to 17 and amino acid identity reaches the decoder after blocks 18 to 19, but the step in between, where phase and nucleotides are combined into an amino acid, is not examined. What happens in encoder blocks 17 to 19 when a rotation is applied?

Thanks to the reviewer’s comments, we applied our weight-only readout to rotated encoders. After a rotation at the output of block 17, the readout at the shifted codon's middle nucleotide gives the shifted codon's amino acid for only 0.16 of residues if the encoder is cut there, rising to 0.77 after block 18, 0.89 after block 19's attention and 0.98 after its feed-forward layer. In the same run without rotation (whole protein hidden), the original codon's amino acid is already partly readable at block 17 (0.52 → 0.86 → 0.98 → 1.00). After rotation, blocks 18–19 compute the amino acid again, now for the codon the rotated phase defines, while the rotated phase itself persists (+118° in 99.8% of tokens). Blocks 18–19 therefore compute the amino acid of whichever codon the phase variable defines, which is also why a rotation applied after these blocks no longer changes the protein.

We also examined how nucleotides are combined into an amino acid. An isolated block-19 feed-forward write selects the genetic code's amino acid for 0.97 of held-out codons, whereas an additive model of its inputs reaches only 0.64, so the conversion is a gated, nonlinear combination of nucleotide evidence computed in blocks 18–19. We will add these results to section 6.

The single-head result also seems limited to the assay where only one residue is hidden ...

We will update the manuscript to emphasize that in both the single-residue-masked setting and the whole-protein-masked setting, replacing this head's output with the value at the middle nucleotide of the target codon yields the correct amino acid, showing that head 19 is still a sufficient readout route when given the correct position.

Does combining start-site conditioning with rotation recover alternative proteins in other transcripts with overlapping frames? (without choosing the start offset by hand)

We agree with the reviewer’s assessment of the POLG/POLGARF result and will update the abstract and result wording to reflect compositional multimodal conditioning and causal steering, and not unconditional generation of the alternative protein.

To further characterize this behavior, we test 162 additional natural transcripts with alternate reading frames. We perform the same rotation intervention and multimodal CDS start conditioning as in POLG, and annotate each alternative ORF separately at its natural start and at the nearest position in phase with the main start (one nucleotide after the CUG for POLG). We find that composition success varies across samples.

The rotation alone steers the output to the alternative reading frame (fraction of transcripts 0.92), but does not move to the alternate start location (0.00). Providing the alternative start site as multimodal conditioning without rotating almost never produces the alternate protein (0.01), but composing the two operations produces the desired alternate protein (0.36). This composition works most often when the alternative start lies downstream of the main start (0.67 vs 0.07 upstream) and where it is an AUG codon (0.71 vs 0.27 for other start codons, 1.00 for downstream AUGs vs 0.57 for other starts downstream). POLGARF is a CUG start upstream of POLG, so it is a rare functional example in the cohort as a whole. Conditioning instead on the in-phase position, as in POLG, also composes with rotation but slightly less often (0.30 vs 0.36), largely on the same transcripts. In POLG itself, only the in-phase position works.

Full results can be found in Table 2 in the general response. These expanded results allow us to quantitatively assess how a multimodal model honors modality conditioning based on causal interventions or on the presence of other modalities, and can inform training paradigms for future multimodal foundation model development.

Because the phase plane is built from the three phase centroids, a 120 degree rotation roughly moves each token to the next centroid. The three plateaus and the three-fold composition would follow from the model reading phase as three categories, so they do not by themselves show a circular geometry.

We agree with the reviewer. Our interventions show a planar, direction-coded three-state variable on which ±120° rotations act as the cyclic group C₃, read out categorically. Sweeping the rotation angle snaps the output to one of the three frames. We will update our wording in the revised manuscript to reflect this description.

In the periodicity analysis, coding DNA is already period-3 at the sequence level, so the 85x enrichment needs the raw sequence as a reference, and the layer reported in Figure 2c is chosen to maximize the contrast.

Coding DNA itself has a 3.25× coding/non-coding period-3 contrast when its four nucleotide tracks (A, C, G, U one-hot vectors) are scored with the same spectral pipeline as Fig 2c. Each model's input layer sits at about this level (MIMIC 3.7×, Evo 2 3.5×). MIMIC's final layer exceeds the raw sequence 26-fold, Evo 2's block 14 38-fold and NTv3 7.6-fold, so the enrichment is built by the networks. HyenaDNA (0.8×) does not exceed the sequence. To check that picking each model's strongest layer does not inflate its contrast reading, we chose the layer on one half of the molecules and measured the contrast on the other half. This reproduces every result within 9%, and for MIMIC picks the final layer. We will add the raw-sequence reference to Fig 2c.

HyenaDNA's weak result is attributed to its lack of attention, although it is also far smaller than the other three models.

We agree that we cannot separate model size and architecture effects. Across all public HyenaDNA checkpoints (0.44–6.55M parameters, 16k–1M context), the period-3 contrast stays at the level of the raw sequence (0.8–1.1× the sequence reference), with no trend in size. We will therefore no longer attribute the weak result to the absence of attention.

How many protein residues are visible to the decoder in the steering experiments, and does steering still work with the entire protein hidden?

In the rotation-only steering tests in section 5, we do not provide any visible protein residues, so all reported steering is with the entire protein hidden. Those results describe rotating the latent representation of the embedded RNA alone.

For the natural transcript experiment results presented above, we condition with 15 residues from the alternate protein (which are excluded from scoring), and while we note that protein prefix conditioning does improve compositional steering (0.36 vs 0.22 without), interestingly the specific residues given do not greatly impact the performance (0.35 with the main protein's residues / 0.31 with random residues).

We would like to thank the reviewer again for taking the time to assess our work and providing feedback to help us make it more generalizable and robust. We hope we have addressed any remaining concerns with these updated results and clarifications.

I thank the authors for addressing my questions and concerns. I think the revised abstract should also reflect the new 162 transcripts results. I raised my score to 4.

Thank you for your thoughtful comments and critiques. We agree that the abstract should emphasize the cohort-level natural transcript results and will revise the manuscript to reflect this.

Reviewer_dH7ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether genomic foundation models represent and use codon phase. In four frozen models, coding-region representations carry stronger period-3 structure than non-coding regions, and in three of them the leading PCA geometry shifts from nucleotide identity toward codon phase with depth. The paper then tests causal use in MIMIC, the only model that decodes protein: rotating one encoder layer by 120 degrees in a two-dimensional subspace fitted to the three codon phases shifts the protein reading frame by one nucleotide, the inverse rotation induces the opposite shift, and three rotations largely restore the original frame. Tracing the decoder, a single final-block cross-attention head reads amino acid identity from the codon's middle nucleotide. On the POLG transcript, a phase rotation combined with CDS-start conditioning recovers the overlapping POLGARF protein.

Critical strengths
  1. The paper shows a full chain from representation to behavior. It fits a two-dimensional plane in which the three codon phases occupy three directions, then rotates one encoder layer by 120 degrees inside that plane; the protein reading frame moves by one nucleotide. The inverse rotation moves it the other way, three rotations return to the original frame, and random-plane, PCA-plane, scaling and UTR rotations do not move it. The model uses the phase representation to decide which three bases form a codon, not only that a classifier can read phase out.

  2. The paper also separates "can be read out" from "is used by the model". A linear probe recovers codon identity at 96-100% from encoder block 2 onward, but the decoder's own pathway gets amino acid information much later: a readout built directly from the head's weights rises from 51.0% after block 17 to 86.7% after block 18 and 100% after block 19. Generalization to a held-out synonymous codon also reaches 100% only at block 18. The probe succeeds early; the model starts using the information late.

Critical weaknesses
  1. The causal claims rest on one model and one decoding setting. The rotation experiments are run only on MIMIC, and the four-model analysis is descriptive. Inside MIMIC, the head-19 pathway is necessary when one residue is hidden (88.5% to 53.4%) but not when the whole protein is hidden (85.5% vs 85.0%). The mechanism result therefore covers one model.

  2. The natural-sequence check uses one transcript, POLG, with no resampling or confidence intervals (A.7), and the 16 transcripts in the codon assay are not verified to be absent from MIMIC's pretraining data (A.5). The mechanism is therefore not shown on sequences the model has not seen. Without a known-held-out set, the readout cannot be separated from prior exposure to those transcripts.

Questions for authors
  1. Please fit the two-dimensional phase plane from permuted phase labels and apply the same plus or minus 120 degree rotation. Does the reading frame still shift? A.4 lists random-plane, PCA-plane, scaling and UTR controls, but not a label-permuted plane.

  2. Does the weight-only native readout still reach high accuracy on a randomly initialized MIMIC with the same architecture? This separates the learned head weights from the geometry of the output embedding.

  3. A.7 reports one transcript and no resampling or confidence intervals. Please report the per-position variation of the POLG/POLGARF match rates, and state how mark 105 was chosen; it is one of the 13 swept marks from 98 to 110.

  4. The non-coding phase label is an arithmetic continuation of the CDS phase. Please repeat the non-coding decodability analysis with a random phase offset or permuted labels, to show that the above-chance result is not driven by position or periodicity.

  5. A.5 states that exposure of the codon-assay transcripts during pretraining is unknown. Please state which of the 16 transcripts are known to be absent from MIMIC's training data, or discuss how prior exposure could produce the readout.

  6. The random-plane control retains about half of the original-displacement match rate. Please report the raw match rates for the real and random planes, so the effect size can be read without the floor subtraction.

2 comments

We thank the reviewer for their detailed questions regarding additional baselines and experiment extensions.

The causal claims rest on one model and one decoding setting.

We focused our initial causal experiments on MIMIC as it is the only foundation model we studied that can natively translate RNA sequence to protein sequence. We agree wholeheartedly with the reviewer that extending this exploration to other models strengthens our findings by characterizing how the observed behavior generalizes.

To address this concern, we trained a small 2-block transformer decoder on frozen embeddings from MIMIC, Evo 2 and NTv3 and find that it learns to translate mature mRNA to protein sequence effectively (accuracy 0.93 / 0.84 / 0.84, vs the native MIMIC decoder 1.00). We perform the same ±120° rotations as in section 5 and report raw match rates on the predicted shifted reading averaged over both rotation directions. This setup shows moderate steering efficacy with MIMIC (0.67, native decoder 0.91), weak steering with Evo 2 (0.32), and with NTv3 rotation collapses the protein output to chance (0.10). In comparison, HyenaDNA and a one-hot encoded nucleotide baseline do not learn to locate the start of the coding sequence and translate (0.11 / 0.10), so we cannot evaluate steering. We will add these results to a revised section 5 of the manuscript and include detailed findings in the appendix. Full results can be found in Table 1 in the general response.

Inside MIMIC, the head-19 pathway is necessary when one residue is hidden (88.5% to 53.4%) but not when the whole protein is hidden (85.5% vs 85.0%).

We agree with the reviewer's comment that the head-19 pathway is only necessary when a single amino acid is hidden, and not in the full protein decoding setting. We will update the manuscript to emphasize that in both the single-residue-masked setting and the whole-protein-masked setting, replacing this head's output with the value at the middle nucleotide of the target codon yields the correct amino acid, showing that head 19 is still a sufficient readout route when given the correct position.

Q1 (phase plane fit from permuted phase labels)

As suggested by the reviewer, we repeated the ±120° rotations with a plane fit from permuted phase labels, and the reading frame does not shift. We observe 0.06 of residues matching the shifted reading frame on both microbial and metazoan transcripts, about chance. In the same runs, the true plane shifts 0.99 (microbial) and 0.91 (metazoan) of residues.

Q2 (weight-only readout on a randomly initialized MIMIC)

On a randomly initialized MIMIC with the same architecture, the weight-only readout recovers the codon's amino acid for only 0.02 of codons, below chance (0.098). Since random initialization also randomizes the encoder, we additionally kept the trained encoder and output embedding and replaced only head 19's value or output matrix with a random matrix of equal norm. Accuracy drops from 1.00 to 0.05, and applying the output embedding directly to the encoder state, with no head, gives 0.055. The readout therefore reflects head 19's learned weights, not the geometry of the output embedding.

Q3 (per-position POLG/POLGARF variation, how the highlighted start position was chosen)

We appreciate the reviewer’s request for clarification of this point. We first tested three adjacent positions (one per frame, including the natural CUG start) and were surprised that the in-phase position relocated the decoder. This motivated the full 13-position sweep and informed the in-phase start selection. Without rotation, none of the 8 positions out of phase with POLG's start moves the decoder. Of the 5 in phase, the one adjacent to the natural start does cleanly (0.92) and one other partially (0.47).

We computed the match rate at each residue position, with block-bootstrap confidence intervals, for every POLG condition and will add these to the appendix. Under start annotation plus rotation, the decoder matches POLGARF at nearly every position (0.96 [0.95, 1.00] of residues overall).

Q4 (non-coding decodability controls)

We repeated the non-coding decodability analysis on UTR tokens with alternative labels. The paper's phase label, which continues the CDS phase into the UTRs, is decoded at 0.618 (chance 0.333). Offsetting the label randomly per transcript, which keeps each transcript's period-3 structure but breaks its alignment to the CDS, gives 0.329 ± 0.020. Token-permuted labels give 0.334 ± 0.008.

Q5 (training exposure)

10 of the 16 codon-assay transcripts appear in the training split, the other 6 do not (4 only in validation or test, 2 absent from the corpus). Among the 12 confirmation transcripts, the weight-only readout reaches the same accuracy for trained-on and held-out transcripts at every depth (0.50 vs 0.52 after encoder block 17, 0.87 vs 0.86 after block 18, and 1.00 vs 1.00 after block 19). Steering is likewise unaffected by exposure. In a separate set of human transcripts, held-out transcripts shift the decoded frame at least as much as training transcripts (0.66 [0.60, 0.71] vs 0.59 [0.56, 0.61]).

Q6 (raw match rates)

At snapshot 18 (after encoder block 17), with no intervention, all decoded residues match the original reading frame. A ±120° rotation moves 0.95 of residues onto the predicted shifted reading, with only 0.07 still matching the original. The random-plane rotation instead leaves 0.36 on the original reading and moves only 0.07 onto either shifted reading, near chance (0.06). The remaining 0.56 match no reading frame. Here the random plane is displacement-matched exactly per transcript. The submitted control used a smaller displacement (about 0.6 retained). We will include the full results as a new appendix table (see Table 3 below).

Table 3. Raw match rates, MIMIC's own decoder, snapshot 18 (after encoder block 17)

Averaged over microbial (227) and metazoan (162) transcripts. 95% CI from resampling transcripts.

Arm−10+1No frame
No rotation0.062 [0.060, 0.064]1.000 [0.999, 1.000]0.056 [0.055, 0.058]0.000 [0.000, 0.001]
+120° (expect +1)0.062 [0.061, 0.064]0.066 [0.064, 0.069]0.966 [0.960, 0.972]0.021 [0.016, 0.027]
−120° (expect −1)0.936 [0.922, 0.950]0.073 [0.070, 0.076]0.062 [0.061, 0.064]0.047 [0.035, 0.060]
Scale in plane (control)0.062 [0.060, 0.064]0.998 [0.996, 0.999]0.056 [0.055, 0.058]0.002 [0.001, 0.004]
Random plane (control)0.057 [0.055, 0.059]0.359 [0.350, 0.368]0.089 [0.086, 0.092]0.555 [0.546, 0.564]
Rotate 5′UTR only (control)0.062 [0.060, 0.064]0.999 [0.999, 1.000]0.056 [0.055, 0.058]0.000 [0.000, 0.001]

We thank the reviewer again for their constructive comments and hope these additional experiments and controls addressed any outstanding concerns.

Reviewer_gx5SOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This is a mechanistically focused interpretability paper that asks whether genomic foundation models represent reading-frame/codon-phase information and whether that information is actually used for RNA-to-protein decoding. Across MIMIC, Evo 2, NTv3, and HyenaDNA it finds stronger period-3 signals in coding regions and analyzes codon phase via PCA geometry and linear probes, but its central contribution is causal: in MIMIC, rotating a codon-phase-aligned latent plane shifts the decoding reading frame in the expected direction, and the authors then trace the decoder’s cross-attention/readout channels and test conditioned phase and start-site effects in POLG/POLGARF. The evidence chain—cross-model representation analysis, controlled phase-plane interventions, decoder readout manipulations, and a natural overlapping-reading-frame case—is relatively complete and supports the mechanistic claims for MIMIC, while the paper appropriately acknowledges that cross-model results are mainly representational and that the POLG/POLGARF case demonstrates conditioned compositional behavior rather than broad biological autonomy

Critical strengths
  1. The core contribution is clear and substantively advances the field. The paper does not stop at showing that phase information is decodable; it advances phase representations to causal intervention. In MIMIC, rotating a codon-phase-aligned latent plane changes the decoding reading frame in the expected direction, and the authors further trace how the decoder reads out amino-acid information. This forms a relatively complete chain from representation to intervention, downstream behavior, and readout mechanism.
  2. The evidence chain is relatively complete and the controls are well designed. The cross-model period-3 analysis, PCA geometry and linear probes, cross-layer interventions, random-plane/scaling/UTR controls, evaluation on 389 transcripts, and the single-head readout, head ablation, channel-content transfer, and readout-address interventions together make the core conclusions stronger than mere decodability evidence.
  3. Boundary awareness is good and key concepts are clearly distinguished. The authors explicitly distinguish “phase information is decodable” from “phase organizes the dominant geometry,” and distinguish representational correlation from computational mechanism. They also acknowledge MIMIC’s unique conditions, possible decoder redundancy, and the conditional limits of the POLG/POLGARF case, which makes the overall claims appropriately cautious.
Critical weaknesses
  1. The mechanistic conclusion has limited model scope. Cross-model results show that related representations appear in several models, but the causal manipulation and decoder-mechanism analysis are concentrated in MIMIC. The paper acknowledges this limitation, but the findings cannot be directly generalized into a claim that genomic foundation models commonly use the same translation mechanism.

  2. The POLG/POLGARF evidence supports conditioned compositional behavior rather than strong biological autonomy. The case is limited by being a single transcript, depending on a protein-prefix condition, and using a start-marker position that is not the natural POLGARF start site. The natural CUG start condition does not produce the same address shift. Therefore, the more accurate conclusion is that the model can compositionally control reading frame and readout position under specific conditions, not that it can unconditionally and autonomously identify and generate natural POLGARF translation.

Questions for authors

How to ensure the generalization beyond MIMIC?

1 comment

We thank the reviewer for their insightful questions and comments. We appreciate the suggestions and think that the additional results motivated by these concerns make our findings more general across models and natural transcripts.

The mechanistic conclusion has limited model scope…

How to ensure the generalization beyond MIMIC?

We focused our initial causal experiments on MIMIC as it is the only foundation model we studied that can natively translate RNA sequence to protein sequence. We agree wholeheartedly with the reviewer that extending this exploration to other models strengthens our findings by characterizing how the observed behavior generalizes.

Following the suggestion of the reviewer, we trained a small 2-block transformer decoder on frozen embeddings from MIMIC, Evo 2 and NTv3 and find that it learns to translate mature mRNA to protein sequence effectively (accuracy 0.93 / 0.84 / 0.84, vs the native MIMIC decoder 1.00). We perform the same ±120° rotations as in section 5 and report raw match rates on the predicted shifted reading averaged over both rotation directions. This setup shows moderate steering efficacy with MIMIC (0.67, native decoder 0.91), weak steering with Evo 2 (0.32), and with NTv3 rotation collapses the protein output to chance (0.10). In comparison, HyenaDNA and a one-hot encoded nucleotide baseline do not learn to locate the start of the coding sequence and translate (0.11 / 0.10), so we cannot evaluate steering. We will add these results to a revised section 5 of the manuscript and include detailed findings in the appendix. Full results can be found in Table 1 in the general response.

The POLG/POLGARF evidence supports conditioned compositional behavior rather than strong biological autonomy…

We agree with the reviewer’s assessment of the POLG/POLGARF result and will update the abstract and result wording to reflect compositional multimodal conditioning and causal steering, and not unconditional generation of the alternative protein.

To further characterize this behavior, we test 162 additional natural transcripts with alternate reading frames. We perform the same rotation intervention and multimodal CDS start conditioning as in POLG, and annotate each alternative ORF separately at its natural start and at the nearest position in phase with the main start (one nucleotide after the CUG for POLG). We find that composition success varies across samples.

The rotation alone steers the output to the alternative reading frame (fraction of transcripts 0.92), but does not move to the alternate start location (0.00). Providing the alternative start site as multimodal conditioning without rotating almost never produces the alternate protein (0.01), but composing the two operations can yield the desired alternate protein (0.36). This composition works most often when the alternative start lies downstream of the main start (0.67 vs 0.07 upstream) and where it is an AUG codon (0.71 vs 0.27 for other start codons, 1.00 for downstream AUGs vs 0.57 for other starts downstream). POLGARF is a CUG start upstream of POLG, so it is a rare functional example in the cohort as a whole. Conditioning instead on the in-phase position, as in POLG, also composes with rotation but slightly less often (0.30 vs 0.36), largely on the same transcripts. In POLG itself, only the in-phase position works.

The results presented above are performed with 15 residues from the alternate protein provided (which are excluded from scoring), and while we note that protein prefix conditioning does improve compositional steering (0.36 vs 0.22 without), interestingly the specific residues given do not greatly impact the performance (0.35 with the main protein's residues / 0.31 with random residues).

Full results can be found in Table 2 in the general response. These expanded results allow us to quantitatively assess how a multimodal model honors modality conditioning based on causal interventions or on the presence of other modalities, and can inform training paradigms for future multimodal foundation model development.

We’d like to thank the reviewer again for raising these important points and hope that the additional results presented here resolve any concerns.

Comments on the paper

5 comments

Dear Reviewers and Area Chair,

We sincerely appreciate the thoughtful comments and suggestions. In response, we have performed experiments in three main areas to address the points raised:

  1. We extend our causal steering experiments to all models present in the study.
  2. We explore additional natural sequences under the same rotation and multimodal conditioning scheme used in the manuscript with POLG/POLGARF.
  3. We performed the suggested baseline experiments and clarified other reviewer questions.

We believe that these additions strengthen the robustness and generalizability of our findings and we will revise the manuscript to reflect these newly added experiments and baselines. We hope that if these updates address your concerns you will consider increasing your score.

Sincerely,

The Authors

Table 1. Raw match rates across models, trained 2-block decoder

162 held-out metazoan transcripts, averaged over decoder seeds. 95% CI from resampling seeds and transcripts. "No frame" is the share of residues matching none of the three readings.

−10+1No frame
MIMIC, own decoder
No rotation0.06 [0.06, 0.06]1.00 [1.00, 1.00]0.06 [0.06, 0.06]0.00
+120° (expect +1)0.06 [0.06, 0.07]0.08 [0.07, 0.08]0.94 [0.93, 0.95]0.04
−120° (expect −1)0.89 [0.86, 0.91]0.08 [0.08, 0.08]0.07 [0.06, 0.07]0.09
Random plane0.07 [0.07, 0.07]0.37 [0.35, 0.38]0.10 [0.10, 0.11]0.54
MIMIC, trained decoder (5 seeds)
No rotation0.06 [0.06, 0.07]0.93 [0.87, 0.99]0.06 [0.06, 0.06]0.06
+120° (expect +1)0.06 [0.06, 0.07]0.07 [0.06, 0.07]0.74 [0.62, 0.85]0.22
−120° (expect −1)0.60 [0.54, 0.65]0.07 [0.07, 0.07]0.07 [0.06, 0.07]0.35
Random plane0.07 [0.07, 0.08]0.41 [0.38, 0.44]0.10 [0.10, 0.11]0.50
Evo 2, block 18 (5 seeds)
No rotation0.07 [0.06, 0.07]0.84 [0.77, 0.91]0.07 [0.06, 0.07]0.13
+120° (expect +1)0.08 [0.07, 0.08]0.24 [0.17, 0.32]0.33 [0.28, 0.37]0.44
−120° (expect −1)0.31 [0.28, 0.35]0.18 [0.13, 0.24]0.08 [0.07, 0.09]0.48
Random plane0.05 [0.05, 0.05]0.35 [0.32, 0.38]0.05 [0.05, 0.06]0.58
NTv3, stem (3 seeds)
No rotation0.06 [0.06, 0.07]0.84 [0.74, 0.92]0.06 [0.06, 0.07]0.14
+120° (expect +1)0.09 [0.09, 0.10]0.09 [0.08, 0.10]0.11 [0.10, 0.12]0.73
−120° (expect −1)0.08 [0.08, 0.09]0.09 [0.09, 0.10]0.11 [0.10, 0.12]0.74
Random plane0.07 [0.07, 0.08]0.40 [0.32, 0.48]0.08 [0.08, 0.09]0.51
HyenaDNA, layer 1 (3 seeds)
No rotation0.09 [0.08, 0.09]0.11 [0.11, 0.12]0.10 [0.09, 0.11]0.73
+120° (expect +1)0.09 [0.09, 0.10]0.10 [0.10, 0.11]0.11 [0.10, 0.11]0.73
−120° (expect −1)0.09 [0.08, 0.10]0.10 [0.10, 0.11]0.11 [0.10, 0.11]0.73
Random plane0.09 [0.08, 0.09]0.11 [0.10, 0.11]0.10 [0.09, 0.11]0.73
One-hot nucleotides (3 seeds)
No rotation0.08 [0.08, 0.09]0.10 [0.10, 0.11]0.10 [0.09, 0.10]0.74
+120° (expect +1)0.06 [0.04, 0.07]0.06 [0.05, 0.07]0.05 [0.04, 0.06]0.85
−120° (expect −1)0.06 [0.05, 0.08]0.06 [0.05, 0.08]0.07 [0.06, 0.08]0.82
Random plane0.06 [0.04, 0.07]0.06 [0.05, 0.07]0.06 [0.05, 0.07]0.84

Table 2. Steering to alternative proteins across a balanced transcript cohort

Fraction of the 162 transcripts that pass the per-transcript test under each prefix setting. 95% CI from resampling transcripts.

nAlternative-protein prefixNo prefixMain-protein prefixRandom prefix
Rotation only: alternative frame, from the main start1620.92 [0.88, 0.96]0.94 [0.90, 0.98]0.93 [0.88, 0.96]0.94 [0.90, 0.98]
Rotation only: alternative protein, from its own start1620.000.000.000.00
Natural start annotation only: start moves (main frame)1620.01 [0.00, 0.03]0.01 [0.00, 0.02]0.01 [0.00, 0.02]0.01 [0.00, 0.03]
Natural start annotation only: alternative protein1620.01 [0.00, 0.02]0.01 [0.00, 0.02]0.01 [0.00, 0.02]0.01 [0.00, 0.02]
Rotation + natural start annotation: all1620.36 [0.28, 0.43]0.22 [0.16, 0.29]0.35 [0.28, 0.43]0.31 [0.24, 0.39]
— downstream starts780.67 [0.56, 0.77]0.42 [0.31, 0.53]0.65 [0.54, 0.76]0.59 [0.48, 0.70]
— upstream starts840.07 [0.02, 0.13]0.04 [0.00, 0.08]0.07 [0.02, 0.13]0.06 [0.01, 0.11]
— AUG starts310.71 [0.55, 0.86]0.48 [0.31, 0.67]0.68 [0.50, 0.84]0.68 [0.50, 0.84]
— non-AUG starts1310.27 [0.20, 0.35]0.16 [0.10, 0.23]0.27 [0.20, 0.35]0.23 [0.16, 0.30]
— downstream AUG starts171.00 [1.00, 1.00]0.71 [0.47, 0.92]0.94 [0.80, 1.00]0.94 [0.80, 1.00]
— upstream non-AUG starts (POLGARF-like)700.01 [0.00, 0.05]0.000.01 [0.00, 0.05]0.00
In-phase start annotation only: start moves (main frame)1620.07 [0.04, 0.12]0.03 [0.01, 0.06]0.04 [0.01, 0.08]0.04 [0.01, 0.07]
In-phase start annotation only: alternative protein1620.000.000.000.00
Random plane + in-phase start annotation1620.000.000.000.00
Rotation + in-phase start annotation: all1620.30 [0.23, 0.38]0.15 [0.10, 0.21]0.27 [0.20, 0.34]0.23 [0.17, 0.30]
— downstream starts780.55 [0.44, 0.66]0.28 [0.18, 0.39]0.51 [0.40, 0.62]0.41 [0.30, 0.52]
— upstream starts840.07 [0.02, 0.13]0.04 [0.00, 0.08]0.05 [0.01, 0.10]0.06 [0.01, 0.12]
— AUG starts310.39 [0.22, 0.56]0.29 [0.14, 0.46]0.29 [0.13, 0.46]0.26 [0.11, 0.42]
— non-AUG starts1310.28 [0.21, 0.36]0.12 [0.07, 0.18]0.27 [0.19, 0.34]0.22 [0.15, 0.29]
— downstream AUG starts170.59 [0.33, 0.82]0.35 [0.12, 0.60]0.41 [0.18, 0.67]0.35 [0.12, 0.60]
— upstream non-AUG starts (POLGARF-like)700.06 [0.01, 0.12]0.000.03 [0.00, 0.07]0.04 [0.00, 0.10]

"Start moves": without rotation, the decoder reads the main frame starting at the annotated position. "Alternative protein": decoded from its own start. "Alternative frame": ≥ 0.6 of residues match the alternative-frame translation read from the main start.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.