- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The categorical Jacobian pioneered fully unsupervised readout, recovering contacts by substituting every amino acid at every position and measuring the response. This paper shows the same contacts can be read directly from attention, across eight model families and fourteen models including encoder-decoder and causal architectures. Selecting a model’s attention heads on a handful of labeled chains — or, to keep the pipeline unsupervised, bootstrapping off the Jacobian’s own predictions, which select the same heads — and averaging the top few unweighted costs one forward pass per chain.
Critical strengths
- Clear motivation and practical relevance. The paper addresses the computational inefficiency of unsupervised contact extraction from protein language models. By performing attention head selection only once, the proposed approach reduces the per-protein inference cost from thousands of Jacobian perturbations to a single forward pass, offering clear practical benefits.
- Extensive experiments with promising results. The authors evaluate multiple PLM families, model scales, and benchmarks, including datasets designed to reduce pretraining contamination. The results demonstrate that simple attention head fusion can outperform the Categorical Jacobian in several settings while substantially reducing computational costs.
- Valuable analysis of the limitations of the Categorical Jacobian. The paper identifies an interesting failure mode of the Categorical Jacobian on designed repeat proteins, where sequence periodicity can be mistaken for spatial contacts. The case studies provide useful insights into the differences between attention-based and perturbation-based structural readouts.
Critical weaknesses
- Limited methodological novelty. Top-K attention head selection and unweighted averaging were already proposed by Bhattacharya et al. (2022). The main novelty here is replacing ground-truth contact labels with Jacobian-generated pseudo-labels, which appears to be a relatively straightforward extension.
- Incomplete computational cost comparison. The reported speedups exclude the expensive Jacobian computations required for unsupervised head selection. An end-to-end cost comparison including calibration would be more informative.
- Lack of causal evidence for contact-related heads. The concentration of contact signals in late attention heads is primarily an observation. Head ablation experiments against random or larger head subsets could help establish their functional importance.
- Limited analysis of calibration robustness. Head selection relies on Jacobian-generated pseudo-labels, which can be unreliable for repeat proteins. More experiments on different calibration sets and pseudo-label quality would help establish the method's robustness.
Questions for authors
see aboved
2 comments
thanks for your response, i keeped my score
Thank you for the review. Novelty is answered in the general response: the averaging is Bhattacharya et al.'s, and our addition is label-free selection, which on ESM-2-650M and 150M picks the same heads labels do.
End-to-end cost. The unsupervised path runs the Jacobian once per model, on the ten selection chains, and the labeled path not at all. Including that one-off cost, the unsupervised path costs about ten Jacobians plus one forward pass per protein, against one Jacobian per protein. It is cheaper from the eleventh protein onward. On ESM-2-3B, selection costs about 15 minutes once, after which each protein takes 0.05 s instead of 87 s. The revision reports this amortized comparison next to the per-protein speedup. We note, though, that given that we have selected heads for all the models described in the paper, any future use of these models will not need to pay this cost.
Functional importance of the heads. We ran the ablation. On ESM-2-650M we replaced the output of ten heads with its average over the selection chains (mean-ablation) and measured the model's own behavior on 50 held-out Zhang chains. Contact precision is the model's Jacobian, long-range P@L/2, which reads contacts from the output logits and not from attention; higher is better. MLM loss is masked-token cross-entropy; lower is better.
Ablating the selected ten halves the model's contact precision (95% interval on the drop [−0.396, −0.315]); ten other heads from the same layers barely move it. The model's contact signal therefore depends on these heads specifically, not on late-layer capacity in general. Growing the fused set to 20 heads raises precision by only 0.005 (Appendix G). The revision adds this table and replaces the sentence on how the model "learns to allocate" its heads with this narrower claim.
Calibration robustness. Appendix F and Section 5.3 test this along four axes.
The repeat failure is why the label-free path needs the repeat screen we describe.
We also drew a second set of ten chains, disjoint by homology from the first and from every evaluation set. Labels on it select the same ten heads. The Jacobian's pseudo-labels select the same ten at budget L/2 and nine of ten at budgets L and 2L; the swapped head lowers P@L/2 by 0.6 to 1.0 points, and fusion still beats the Jacobian on all four sets. The revision moves this analysis into the main text.