acceptodds
Under review as a conference paper at ICLR 2027

Tertiary structure is directly accessible in protein language models

Abstract

Protein language models are trained on sequence alone, yet they learn enough three-dimensional structure that contact maps can be read out of them without structural supervision. The categorical Jacobian pioneered fully unsupervised readout, recovering contacts by substituting every amino acid at every position and measuring the response. We show the same contacts can be read directly from attention, across eight model families and fourteen models including encoder-decoder and causal architectures. Selecting a model's attention heads on a handful of labeled chains — or, to keep the pipeline unsupervised, bootstrapping off the Jacobian's own predictions, which select the same heads — and averaging the top few unweighted costs one forward pass per chain. On ESM-2 650M this exceeds the Jacobian on four evaluation sets by 2.9 to 27.0 percentage points of long-range precision (P@L/2), with the margin widening as training-leakage screening gets stricter. How accessible that structure is depends on the architecture: under both readouts every masked and encoder-decoder model above the smallest reads at least 0.47, while the three causal decoders reach at most 0.114. On the unsupervised path the Jacobian runs once per model, on ten chains, after which attention reads every protein 150× to 1,626× faster and, on designed repeat proteins, 27.9 to 45.9 points more precisely. It lets us ask how accessibly tertiary structure is encoded across frontier protein language models and architectures, and interrogate how the Jacobian conflates contact signal with additional information.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language models, residue contact prediction, attention head selection, unsupervised contact readout, categorical Jacobian, pretraining data leakage

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HQmcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The categorical Jacobian pioneered fully unsupervised readout, recovering contacts by substituting every amino acid at every position and measuring the response. This paper shows the same contacts can be read directly from attention, across eight model families and fourteen models including encoder-decoder and causal architectures. Selecting a model’s attention heads on a handful of labeled chains — or, to keep the pipeline unsupervised, bootstrapping off the Jacobian’s own predictions, which select the same heads — and averaging the top few unweighted costs one forward pass per chain.

Critical strengths
  1. Clear motivation and practical relevance. The paper addresses the computational inefficiency of unsupervised contact extraction from protein language models. By performing attention head selection only once, the proposed approach reduces the per-protein inference cost from thousands of Jacobian perturbations to a single forward pass, offering clear practical benefits.
  2. Extensive experiments with promising results. The authors evaluate multiple PLM families, model scales, and benchmarks, including datasets designed to reduce pretraining contamination. The results demonstrate that simple attention head fusion can outperform the Categorical Jacobian in several settings while substantially reducing computational costs.
  3. Valuable analysis of the limitations of the Categorical Jacobian. The paper identifies an interesting failure mode of the Categorical Jacobian on designed repeat proteins, where sequence periodicity can be mistaken for spatial contacts. The case studies provide useful insights into the differences between attention-based and perturbation-based structural readouts.
Critical weaknesses
  1. Limited methodological novelty. Top-K attention head selection and unweighted averaging were already proposed by Bhattacharya et al. (2022). The main novelty here is replacing ground-truth contact labels with Jacobian-generated pseudo-labels, which appears to be a relatively straightforward extension.
  2. Incomplete computational cost comparison. The reported speedups exclude the expensive Jacobian computations required for unsupervised head selection. An end-to-end cost comparison including calibration would be more informative.
  3. Lack of causal evidence for contact-related heads. The concentration of contact signals in late attention heads is primarily an observation. Head ablation experiments against random or larger head subsets could help establish their functional importance.
  4. Limited analysis of calibration robustness. Head selection relies on Jacobian-generated pseudo-labels, which can be unreliable for repeat proteins. More experiments on different calibration sets and pseudo-label quality would help establish the method's robustness.
Questions for authors

see aboved

2 comments

Thank you for the review. Novelty is answered in the general response: the averaging is Bhattacharya et al.'s, and our addition is label-free selection, which on ESM-2-650M and 150M picks the same heads labels do.

End-to-end cost. The unsupervised path runs the Jacobian once per model, on the ten selection chains, and the labeled path not at all. Including that one-off cost, the unsupervised path costs about ten Jacobians plus one forward pass per protein, against one Jacobian per protein. It is cheaper from the eleventh protein onward. On ESM-2-3B, selection costs about 15 minutes once, after which each protein takes 0.05 s instead of 87 s. The revision reports this amortized comparison next to the per-protein speedup. We note, though, that given that we have selected heads for all the models described in the paper, any future use of these models will not need to pay this cost.

Functional importance of the heads. We ran the ablation. On ESM-2-650M we replaced the output of ten heads with its average over the selection chains (mean-ablation) and measured the model's own behavior on 50 held-out Zhang chains. Contact precision is the model's Jacobian, long-range P@L/2, which reads contacts from the output logits and not from attention; higher is better. MLM loss is masked-token cross-entropy; lower is better.

Heads ablatedContact precisionChangeMLM lossChange
None (intact model)0.7291.466
The ten selected heads0.373−0.3551.627+0.161
Ten random heads, same layers: draw 10.728−0.0011.511+0.045
draw 20.730+0.0011.499+0.033
draw 30.726−0.0031.519+0.053
draw 40.732+0.0031.504+0.038
draw 50.720−0.0091.523+0.058
The ten lowest-ranked heads0.731+0.0031.476+0.010

Ablating the selected ten halves the model's contact precision (95% interval on the drop [−0.396, −0.315]); ten other heads from the same layers barely move it. The model's contact signal therefore depends on these heads specifically, not on late-layer capacity in general. Growing the fused set to 20 heads raises precision by only 0.005 (Appendix G). The revision adds this table and replaces the sentence on how the model "learns to allocate" its heads with this narrower claim.

Calibration robustness. Appendix F and Section 5.3 test this along four axes.

  • Number of labeled chains: one structure recovers 8.9 of the ten heads.
  • Label noise: selection stays within 0.5 points of full labels until nine of ten labels are wrong.
  • Teacher quality: a 4.3 times smaller teacher selects the same heads, and an 18.6 times smaller one recovers seven of ten.
  • Repeat chains in the selection set: with four of ten, selection still reads 0.702; with five it collapses to 0.346.

The repeat failure is why the label-free path needs the repeat screen we describe.

We also drew a second set of ten chains, disjoint by homology from the first and from every evaluation set. Labels on it select the same ten heads. The Jacobian's pseudo-labels select the same ten at budget L/2 and nine of ten at budgets L and 2L; the swapped head lowers P@L/2 by 0.6 to 1.0 points, and fusion still beats the Jacobian on all four sets. The revision moves this analysis into the main text.

thanks for your response, i keeped my score

Reviewer_4tpiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether residue contacts can be recovered from protein language model attention without training a structure predictor. The readout averages a small set of attention heads after symmetrization and APC correction. Heads are ranked by long-range contact precision, usually retaining the top ten. The main methodological addition is to use Categorical Jacobian pseudo-contacts for this ranking in place of experimental contacts. Once the heads have been chosen, inference requires a single forward pass. On ESM-2-650M, attention fusion outperforms the Categorical Jacobian on four evaluation sets, with gains in long-range P@L/2 ranging from 2.9 to 27.0 percentage points. The largest gain occurs on a filtered set containing many designed repeat proteins. The authors trace the Jacobian's failures on these proteins to sequence periodicity. They also compare fourteen checkpoints from eight model families and construct a pretraining-aware benchmark, on which both readouts perform substantially worse than on the unfiltered data. I lean toward acceptance because the method is inexpensive and the experiments give useful insight into when these readouts work. The repeat-protein analysis is particularly convincing. My reservations concern the interpretation: the experiments establish access to long-range contacts, while the title makes a broader claim about tertiary structure. The architecture comparison is also difficult to separate from differences in training and readout. These issues limit the conclusions more than they diminish the practical value of the method.

Critical strengths
  1. Substantial computational savings Once heads are selected, fusion needs a single forward pass instead of the Categorical Jacobian's 20L perturbation passes. Reported speedups on one A100 range from 150x for ESM-2-35M to 1,626x for ESM-2-3B. This makes the readout useful for large protein collections.
  2. Consistent gains across datasets On ESM-2-650M, fusion improves long-range P@L/2 over the Jacobian on all four evaluation sets, by 2.9 to 27.0 percentage points. Paired bootstrap intervals and the appendix's Wilcoxon/Holm analyses support the comparison, although the size of the gain depends on benchmark composition.
  3. Convincing analysis of repeat proteins The authors show that the Jacobian can rank pairs according to the sequence repeat period and its harmonics rather than physical contacts. Every zero-scoring Jacobian chain in the two filtered sets is a repeat chain. Distinguishing periodicity-sensitive heads from contact-sensitive heads gives a plausible explanation for this failure.
  4. Useful label-free head selection For ESM-2-650M and ESM-2-150M, Jacobian pseudo-contacts recover the same ten heads as structural labels across the tested budgets. The small-label and noisy-label experiments suggest that accurate head selection does not require extensive supervision.
  5. Careful evaluation and broad coverage The study covers fourteen checkpoints from eight model families and includes a pretraining-aware benchmark. The substantial accuracy drop after filtering is a useful warning about pretraining overlap. The appendices also examine the choice of K, alternative fusion operators, teacher quality, and readout variants, while reporting failures that qualify the main conclusions.
Critical weaknesses
  1. The structural claim is too broad The experiments measure 8 A C-beta contacts, mainly through long-range P@L/2. They do not show recovery of a 3D fold. The title and abstract should refer more specifically to contact information unless a reconstruction experiment is added. Results at another distance threshold and additional precision cutoffs would also help establish robustness.
  2. The methodological novelty is limited Ranking and averaging selected attention heads follows Bhattacharya et al. (2022). The methodological addition is Jacobian-based pseudo-label selection, alongside the empirical study. That distinction should be clearer. Selection also depends on teacher quality and protein composition: ESM-2-35M performs worse than label-based selection, and repeat-rich selection sets can cause failure. Transfer to an independent natural-protein dataset needs a cleaner test.
  3. The architecture comparison is confounded The models differ in objective, pretraining data, tokenizer, scale, and readout geometry, so the observed gap cannot be attributed to architecture alone. Hidden-state and logit Jacobians agree on masked ESM models, but comparable validation is missing for causal and encoder-decoder models. Unequal pretraining overlap with Zhang eval-200 adds further uncertainty to the cross-model ranking.
  4. Several results need reconciliation The stored and fresh ProGen2-large Jacobians have pooled Pearson r = -0.263. For ProGen2-xlarge, K=10 gives 0.014 in the sweep and 0.111 in the standalone run despite the same ranking set and head list. These discrepancies need a diagnosis. The manuscript should also clarify the Ankh-large results reported for 50 chains in Table H1 and 200 chains in the main text.
  5. Generalization and practical limits remain unclear PTA has 83 chains and filtered Boltz only 24, half of which are repeats. The largest gain therefore comes from a small, compositionally unusual set; non-repeat gains are roughly 2.9 to 8.5 points. A larger leakage-controlled natural-protein benchmark would strengthen the conclusion. Peak memory, sequence-length limits, and attention-extraction settings are also needed to assess the practical cost of the method.
Questions for authors
  1. Could the title and abstract refer specifically to long-range residue contacts? If the authors retain the tertiary-structure claim, can they demonstrate structure reconstruction from the fused contacts?
  2. Relative to Bhattacharya et al. (2022), which improvements depend on pseudo-label selection, and which would also follow from top-head averaging with a small labeled set?
  3. Can a Jacobian teacher from one family select useful heads in another? For example, would ESM-2 pseudo-contacts work for ESMC, ProtT5, or Ankh without computing a Jacobian for those models?
  4. Is a controlled masked-versus-causal comparison feasible with matched data, tokenization, and scale, or with different attention masks applied to the same backbone?
  5. What explains the negative correlation between the stored and fresh ProGen2-large Jacobians?
  6. Can the ProGen2-xlarge K=10 runs be repeated from a clean pipeline to resolve the 0.014 versus 0.111 discrepancy?
  7. Do heads selected on one natural-protein dataset transfer to an independent leakage-controlled natural-protein set without target-informed screening?
  8. Do the main findings hold at an alternative contact threshold and at P@L and P@L/5?
  9. Can the filtered evaluation be expanded, or can uncertainty and effect sizes be reported separately for natural proteins, designed non-repeat proteins, and designed repeat proteins?
  10. How much pretraining overlap does each model have with Zhang eval-200? A nearest-neighbor similarity analysis, even if approximate, would help interpret the cross-model differences.
  11. Which Ankh-large evaluation, the 50-chain or 200-chain run, is used in Figure 1 and the architecture comparison?
  12. What peak memory and sequence-length limits apply to attention extraction for the largest checkpoints, and how do these affect the measured speedup?
2 comments

Thank you for the careful reading. The title and the Bhattacharya distinction are answered in the general response.

What depends on pseudo-labels. Nothing in the accuracy numbers. On ESM-2-650M and 150M, Jacobian pseudo-labels select the same ten heads as labels, so every Table 1 cell is the same under top-head averaging with ten labeled chains. One labeled structure already recovers 8.9 of the ten heads (Appendix F). Pseudo-labels buy freedom from only labels; the revision will make that clearer.

Cross-model teachers. A smaller teacher in the same family works: ESM-2-150M's Jacobian, 4.3 times smaller, selects the same ten heads as ESM-2-650M's own, while ESM-2-35M's recovers seven. Across families it also works, with no Jacobian computed for the student. ESM-2-650M's pseudo-labels select ProtT5-XL's label-selected ten heads at every budget, so its fusion is unchanged (0.727). For ESMC-600M and Ankh-large they select nine of the ten, and fused P@L/2 falls by 0.6 points (0.772 against 0.778, 200 chains) and 0.5 points (0.781 against 0.786, 50 chains). The revision adds these rows.

Transfer. Every head set in the paper is ranked on ten Zhang chains and applied unchanged to Boltz-1 and PTA. Those chains are disjoint from every evaluation set by homology as well as by ID (best match 42 to 49% identity over at most 18% coverage). Every non-Zhang row is therefore a transfer to an independent set. Split by origin, the non-repeat PTA margin is +8.1 points on natural chains (N = 33, [+4.5, +12.7]) and +10.1 on designed ones (N = 29, [+6.8, +13.7]); repeat chains are reported separately. The revision reports these three strata with intervals, deduplicated and bootstrapped over sequence clusters. We cannot enlarge the filtered sets within the discussion period.

Architecture. We agree the comparison is confounded, and a matched masked-versus-causal model is beyond this paper. Applying a causal mask at inference to a masked backbone would test a model outside its training distribution, not a causally trained one. The revision describes the gap as what these checkpoints expose under these readouts and states that the hidden-state Jacobian is validated only within masked models. We will treat the cross-model ranking as descriptive.

Reconciliations.

  • ProGen2-large: the appendix mislabels r = −0.263. It compares an archived attention-head grid with a fresh run, not two Jacobians. We withdrew that grid; ProGen2-large's fusion cell does not use it.
  • ProGen2-xlarge: the 0.014 came from a sweep run on a later transformers release with patched model loading. Rerun on the pinned library version of the main pipeline, the sweep gives 0.112 at K = 10, matching the standalone 0.111, and its best K is 1 to 2 (0.193). Table G1 will report the corrected row.
  • Ankh-large: Figure 1 and Table H1 use the 50-chain run, the only one with a Jacobian. The 200-chain run, fusion 0.774, is the number in the Section 5.4 text. The revision makes this clearer.

Threshold, cutoffs, memory. At P@L, P@L/2 and P@L/5 and at short, medium and long range, fusion leads the Jacobian in all 36 comparisons across the four sets, every paired interval above zero; the revision adds the tables. At 10 Å Cα on 50 Zhang chains, fusion still leads (0.877 against 0.827 on ESM-2-3B). We will report peak memory and the sequence-length limit, ESM-2's 1,026-position context, for every timed checkpoint.

Thank you for the detailed responses. The additional explanations and experiments address most of my concerns.

In particular, the clarification on pseudo-label selection, cross-model teacher transfer, and the corrected ProGen2 results make the methodology and experimental findings clearer. The additional transfer analysis and robustness results also strengthen the paper.

Some limitations remain, especially the relatively small filtered evaluation sets and the lack of a fully controlled cross-architecture comparison. However, the authors have acknowledged these limitations and appropriately narrowed their claims.

Overall, I am satisfied with the rebuttal. But I still maintain my original rating.

Reviewer_wJ8SOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The core claim is that the 3D structural information in pLMs — though strictly speaking what's being read out is a 2D contact matrix, not 3D structure — lives in a very low-dimensional subspace of attention: averaging the scores of a handful of heads (top-10) is enough to recover it. In practice the proposed readout is cheap and more accurate than the Categorical Jacobian (CJ) baseline.

Critical strengths
  • Real and substantial speedup over CJ. One forward pass per chain vs. ~20L is a meaningful practical improvement, and the timing numbers are reported carefully.

  • The method is very intuitive, and I think the interpretability intuition behind it may actually be the more important part.

  • The homology / repeat-protein analysis of CJ is a genuinely interesting observation. In my view this is where most of the empirical value of the paper lies: showing that CJ's coupling signal tracks the repeat period rather than the fold, that the failing picks land exactly on the chain's self-similarity period and its harmonics, and that the "periodicity heads" are a separate population from the contact heads. That whole line of analysis is the strongest contribution here.

Critical weaknesses
  • Title oversells. What's recovered is a contact map, not tertiary structure. The gap is not cosmetic. Either retitle or add a 3D evaluation.

  • The headline finding is largely known. Rao et al. 2021 and Vig et al. 2020 already established that contacts are decodable from attention and concentrated in a small number of heads. The paper doesn't argue why this was still open.

  • Novelty of the readout is unclear. Precision-ranked unweighted head averaging is close to prior practice. If the real contribution is the head-selection procedure rather than the averaging, the framing should say so — and needs an ablation separating the two.

  • P@L/2 is a weak metric for the paper's own claim. The paper argues that fine structural detail is what matters (90% backbone agreement ≠ same function), but then evaluates with a coarse top-K precision metric that is insensitive to exactly those details and saturates easily. At minimum: report P@L, P@L/5, short/medium/long-range breakdowns, and ideally fold the maps and report lDDT/TM-score.

  • Baselines and splits. CJ as essentially the only baseline is puzzling — ESMFold and comparable pLM structure predictors should be reference points. And both Boltz-1 and PTA are PDB subsets; a strict temporal holdout (most recent depositions) would be far more convincing. Filtering is sequence-level only; Foldseek-based structural dedup is now standard and matters here, since the sequence filter appears to enrich for designed repeat proteins.

Questions for authors
  • Why is CJ the only baseline? Can you add ESMFold / other pLM folding results on the same splits?

  • Can you report absolute wall-clock against a single plain forward pass, and confirm CJ was batched?

  • Can you add metrics beyond P@L/2 that are sensitive to local/functional detail?

  • Can you build a strict temporal split and re-run the main comparison?

1 comment

Thank you for the detailed review. The title and novelty points are answered in the general response: we retitle to contacts, and we state that the averaging is Bhattacharya et al.'s and that our addition is label-free selection.

Why the comparison was still open. Rao et al. and Vig et al. showed that contacts are decodable from attention, using labeled structures. The Categorical Jacobian later became the standard label-free readout. To our knowledge, no prior work asks whether attention, read without labels, matches it, on which proteins, or across current checkpoints. The answer we find is that attention beats it on every ESM-2-650M set at one forward pass instead of 20L, and that the Jacobian collapses on designed repeats. You identify that repeat analysis as the main empirical value. The revision puts it in the framing rather than in Section 5.1 alone.

Baselines. ESMFold is trained on structures. It belongs with the supervised reference we already report, the fitted head of Rao et al. refit on ESM-2, rather than with the label-free readouts we compare. We ran ESMFold on all four sets and scored the contact probabilities from its distogram with the same criterion. Its long-range P@L/2 is 0.997, 0.916, 0.934 and 0.852 on Zhang eval-200, Boltz-1, PTA and filtered Boltz-1, 24 to 44 points above fusion. Zhang eval-200 is almost certainly in its structural training data. The gap measures what training on structures adds beyond what the language model exposes without it. The revision adds the row to Table 1 as a supervised reference.

Temporal split and Foldseek. PTA is already a temporal holdout. It keeps only PDB chains released on or after 2022-09-01, ten months after ESM-2's pretraining cutoff, and then also removes chains with a pre-cutoff relative in UniRef90 or the PDB (Appendix D). We agree that a structural screen matters, especially since the sequence filter enriches for designed proteins, so we ran one. We searched every PTA chain against the PDB with Foldseek and dated each hit by the earliest release in its identical-sequence cluster. Fold-level overlap is real: 23 of the 74 chains have a pre-cutoff structural neighbor at TM-score 0.7 or above, and both readouts score higher on them. Fusion's margin holds in every bin: +11.0 points on the 22 chains with no pre-cutoff neighbor above TM 0.5, +16.0 on the 29 between 0.5 and 0.7, and +9.0 on the 23 above 0.7, every paired interval above zero. The revision adds this screen to Appendix D.

Metrics. We computed P@L, P@L/2 and P@L/5 at short, medium and long range for fusion and the Jacobian on all four sets. Fusion leads in all 36 comparisons, and every paired 95% interval excludes zero. On Zhang eval-200 and the non-repeat strata, the long-range margin is 2.5 to 6.4 points at P@L and 2.7 to 11.1 at P@L/5. At short range the margin is larger, 9.5 to 12.0 points at P@L/2 on the four full sets. The revision adds these tables. We also checked an alternative criterion, 10 Å Cα, on 50 Zhang chains. Fusion still leads there, 0.877 against 0.827 on ESM-2-3B and 0.787 against 0.763 on ESM-1b. On repeat chains the gap widens.

Timing. The Jacobian was batched at 32 perturbed sequences per forward pass (Section 5.2). Fusion's 0.03 to 0.05 s per protein is one forward pass plus attention post-processing, against a median 4.7 s (ESM-2-35M) and 87 s (ESM-2-3B) for the batched Jacobian on one A100. We will tabulate the plain forward pass separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.