acceptodds
Withdrawn from ICLR 2027

Mechanistically Understanding How a Biological Language Model Predicts Beneficial Mutations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models have shown impressive results when applied to biological tasks. Concurrently, recent advancements in interpretability have produced methods that enable the computation of language models to be understood mechanistically. This paper examines what language models learn when trained on proteins. Specifically, we analyze CovFit, protein language model introduced by Ito et al. [2025] trained to predict viral fitness from input SARS-CoV-2 sequences. We apply interpretability methods to (i) identify a MLP "look-up" computation that generalizes across several SARS-CoV-2 mutations, (ii) identify a particular biological circuit for the L455F mutation and (iii) show that highly conserved sites in SARS-CoV-2 are used as "attention sinks" over the course of computation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
CovFit, ESM-2, Biological AI, SARS-CoV-2

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wtcMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates CovFit, an ESM-2-based model for predicting viral fitness. The authors select 10 biologically relevant SARS-CoV-2 spike protein mutations that the model predicts to be beneficial and construct original, single-point-mutant, and masked sequences for each mutation. Through activation patching and component ablation, they identify similar information-flow pathways across several mutations: attention heads in later layers aggregate mutation information at specific positions, MLPs process this information, and final-layer attention heads transfer it to the <cls> representation for fitness prediction. The authors then construct and evaluate a detailed computational circuit for L455F. They also find that some highly conserved positions consistently receive substantial attention and exhibit large representation norms, suggesting that these positions may serve as attention sinks.

Critical strengths
  1. Interventional experiments provide evidence beyond attention visualization.
    The authors combine activation patching, path patching, and MLP-output ablation to test whether specific components affect prediction differences. This provides a stronger basis for assigning computational roles than high attention weights alone. The original and mutated sequences differ only at the target position, which also reduces background differences at the input level and makes internal interventions easier to interpret.
  2. The analysis examines both shared patterns across mutations and a detailed circuit for an individual mutation.
    The paper investigates shared computational structures in later layers across multiple mutations while also examining L455F, which is not fully explained by the simpler pathway. This design reveals both common mechanisms and case-specific differences, extending the analysis beyond a single example.
  3. The conserved-site analysis offers an observation worth further investigation.
    The authors connect site conservation with attention concentration and high representation norms, examining this relationship in both CovFit and the intermediate model ESM_coronaviridae. This observation provides a starting point for investigating how the statistical properties of protein sequences shape internal Transformer computations.
Critical weaknesses
  1. The intervention-score formula contradicts its textual interpretation, limiting the verifiability of the core results.
    According to the formula in the appendix, the score is −1 when an intervention has no effect and 0 when the mutation effect is completely removed. However, the main text and appendix state the opposite: “0 indicates no effect, and −1 indicates complete removal of the effect.”
  2. Mutation selection limits the generalizability of the conclusions.
    All 10 mutations are predicted by the model to be beneficial, and nine lie within the receptor-binding domain. Without neutral mutations, deleterious mutations, or cases in which the model makes incorrect predictions, it is difficult to determine whether the identified pathway specifically evaluates beneficial effects or instead performs general mutation encoding or information readout.
Questions for authors
  1. How is the intervention score actually implemented?
    The formula in Appendix A.8 contradicts the interpretation that “0 indicates no effect and −1 indicates complete removal.” Please provide the formula used in the code and clarify whether the relevant figures, component rankings, or conclusions require correction.
  2. What is the testable definition of an “MLP look-up”?
    What are the keys and values, and where are they represented? Which experiments distinguish a look-up operation from general nonlinear feature processing? Can targeted modifications to the relevant representations alter the predicted mutation effect in a predictable way?
  3. Does the identified pathway generalize to independent mutations?
    Please validate it on mutations not used during mechanism discovery, covering different structural domains, beneficial/neutral/deleterious mutations, different substitutions at the same position, and cases in which the model makes incorrect predictions. Please report the proportions of cases that the pathway successfully explains and those it does not.
  4. Does the L455F circuit contain many general-purpose components?
    Please explain the criteria for including components and terminating the search, provide control circuits of comparable size, and explain the substantial deviations in Figure 3 when individual MLPs are removed. Does the circuit retain similar faithfulness on independent sequences or lineage backgrounds?
  5. Are the path-patching conclusions robust to the intervention method?
    Please compare different reference activations or ablation methods and explain how LayerNorm, the residual stream, and frozen components are handled. How does freezing intermediate components justify the claim that it “removes all confounding elements”?
  6. Do conserved sites actually perform the proposed computational functions?
    Can targeted interventions compare these sites with matched nonconserved positions? Does the enrichment persist after controlling for amino acid identity, position, and structural domain? Are the findings robust to the thresholds and definition of conservation?
Reviewer_NQTnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes CovFit, a finetuned version of the ESM-2 model that is trained on variants of SARS-CoV-2. Specifically, they apply activation patching and other techniques

Critical strengths
  • This work applies standard tools from mechanistic interpretability to a very interesting and meaningful setting. They find that MLP 31, and the attention heads there are important mediators for key downstream mutations.

  • This work uses causal interventions to move beyond correlational analyses of attention patterns or activations.

  • The work does include a nice finding about L455F circuit. In this part of the paper, they also perform faithfulness and completeness evaluations.

  • Lastly, this work identifies a relationship between conserved amino acid positions and attention sinks.

Critical weaknesses

While this work was insightful to read, it mostly is a very nice examination of a specific model trained on a specific set of variants. It is unclear whether the findings for this would imply something interesting other of the variants considered.

Figure 1 (right) has very big error bars. It is unclear that we can take much away from this given the variation.

I have more specific questions that I have asked in the next section about the identifiability of the results.

Questions for authors

There is a recent work that the authors should be aware of, it is the paper on "The dead salmons of AI interpretability". The paper indicates challenges with estimation and identifiability that plague tools like activation patching and others.

Here are the experiments I would suggest the authors perform:

  • Random mutation controls: do the identified circuits specifically explain the beneficial mutations, or do they random similarly to arbitrary mutations? To do this, do the following: sample random amino acid substitutions, apply the same activation and path patching analysis, compare the effects of L31.H7, MLP31 and the layer 31/32 heads across the mutation categories. The goal of this is to separate the circuit specific effect to random ones.
  • Random circuit controls: Are the attention heads that you have identified special? Or would a randomly chosen circuit of comparable size give similar results?
  • Test on held-out mutations and lineages: Does the explanation generalize beyond the data used to discover it?
Reviewer_sASROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how CovFit, a protein language model fine-tuned to predict SARS-CoV-2 variant fitness, processes ten selected spike mutations that the model predicts to be beneficial. Using activation patching, absolute path-patching, and MLP-output ablations, the authors propose an MLP31-centered computation involving token 679, reconstruct a circuit for L455F, and report an enrichment of highly conserved sites among positions with high attention-sink and residual-norm scores. The work focuses on the model's internal computations rather than experimentally validating the biological effects of the mutations.

Critical strengths
  1. Important research question. The paper addresses a meaningful problem in mechanistic interpretability: understanding how biological language models arrive at fitness predictions.

  2. Use of causal interventions. Activation patching, absolute path-patching, and MLP-output ablations provide stronger mechanistic evidence than attention visualization alone.

  3. Substantial circuit-level analysis. The authors examine ten mutations and conduct a detailed circuit analysis for L455F, including quantitative faithfulness and completeness evaluations.

Critical weaknesses
  1. Limited generality of the MLP31/token-679 pathway. The ablation results vary substantially across mutations. For F456L and S486P, ablating token 679 alone yields scores of 0.001 and -0.001, respectively, whereas ablating the mutation site alone yields 0.382 and 0.172, nearly matching joint ablation scores of 0.383 and 0.172. Although the authors acknowledge these exceptions, the results do not consistently support a general token-679-mediated lookup mechanism. The path-patching experiments also use only 80 paired sequences per mutation, making robustness estimates across sequence backgrounds valuable.

  2. Insufficient controls for mutation specificity. All ten mutations were selected because they are biologically relevant and predicted by CovFit to increase fitness. Without matched neutral or deleterious mutation controls, it remains unclear whether the identified pathway is specific to beneficial fitness predictions or reflects a more general mechanism for processing mutations. This limits the scope of the paper's generalization.

  3. Incomplete validation of the broader mechanistic interpretations. The L455F circuit's faithfulness and completeness scores are useful evidence, but they do not establish that the circuit is minimal or uniquely explanatory. Similarly, the enrichment of conserved sites among high-norm, high-sink positions is correlational and does not establish the proposed garbage-collection function. Comparisons with smaller or size-matched baseline circuits, formal enrichment tests with matched controls, and targeted interventions on candidate sink positions would help distinguish the proposed mechanisms from alternative explanations.

Questions for authors
  1. Can the authors clarify which aspects of the MLP31/token-679 pathway generalize across mutations, particularly for F456L and S486P, and provide robustness estimates across sequence backgrounds?

  2. Do matched neutral or deleterious mutations exhibit the same pathway, or is the proposed computation specific to beneficial fitness predictions?

  3. Can the authors compare the L455F circuit with smaller or size-matched baseline circuits and provide stronger statistical or causal evidence for the proposed functional role of conserved attention-sink positions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.