- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper identifies scoring variants of a given amino acid sequence as a task where exact likelihoods are not needed to approximate the same likelihood. The authors propose simply grouping together multiple single-amino-acid mutations into a single forward pass into the model, and inspecting the likelihoods of masked out tokens at each mask position. They find empirically that the corresponding likelihoods are similar to the ungrouped case, which is then sufficient to produce approximate scores.
Critical strengths
None
Critical weaknesses
As a disclaimer, I am not familiar with the protein language modeling domain. But since compute is entirely spent on prefill, from the presented experiments it does not make sense to provide such speedups. For example, the assay “influenza HA doud” has 565 length and 564 positions, producing total tokens for computing exact per-token marginals. With a 650M model, this is roughly 414 TFLOPs, which can run in tens of seconds on A100 gpus with a reasonable implementation. The speedup from tens of seconds to a few seconds seems negligible in terms of compute cost, if this computation is performed just a single time for each assay. If indeed this inference cost is substantial in practical settings (on the order of, say, multiple hours that is often an experimental bottleneck), then it would be helpful to see comparisons between ESM-2 150M exact g=1 and ESM-2 650M g=4, which correspond to roughly the same FLOPs. It would also be interesting if the described kernel optimizations were more broadly applicable, but they appear to be customized to ESM-2 650M.
The paper optimizes for Spearman correlation, which is reasonable for optimizing against DMS regret. But there is little analysis on the failure modes of the proposed method, e.g. a pairwise analysis shows that pairwise interference is somewhat predictable (at correlation 0.416). Thus, it is difficult to understand precisely in what settings the proposed method disagrees with exact inference.
Questions for authors
None