Tertiary structure is directly accessible in protein language models

Protein language models are trained on sequence alone, yet they learn enough three-dimensional structure that contact maps can be read out of them without structural supervision. The categorical Jacobian pioneered fully unsupervised readout, recovering contacts by substituting every amino acid at every position and measuring the response. We show the same contacts can be read directly from attention, across eight model families and fourteen models including encoder-decoder and causal architectures. Selecting a model's attention heads on a handful of labeled chains — or, to keep the pipeline unsupervised, bootstrapping off the Jacobian's own predictions, which select the same heads — and averaging the top few unweighted costs one forward pass per chain. On ESM-2 650M this exceeds the Jacobian on four evaluation sets by 2.9 to 27.0 percentage points of long-range precision (P@L/2), with the margin widening as training-leakage screening gets stricter. How accessible that structure is depends on the architecture: under both readouts every masked and encoder-decoder model above the smallest reads at least 0.47, while the three causal decoders reach at most 0.114. On the unsupervised path the Jacobian runs once per model, on ten chains, after which attention reads every protein 150× to 1,626× faster and, on designed repeat proteins, 27.9 to 45.9 points more precisely. It lets us ask how accessibly tertiary structure is encoded across frontier protein language models and architectures, and interrogate how the Jacobian conflates contact signal with additional information.