- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper trains protein encoders at 7.5 and 33M parameter sizes with two main experimental decisions:
- Corruption mode (mask vs mutation)
- Loss supervision through all tokens vs only through the corrupted positions.
The authors ultimately argue that dense supervision over all tokens is useful when using mutation based corruption but not when using standard masking. They validate using zero-shot likelihood benchmarks from ProteinGym.
Critical strengths
Given the objective, the experimental design is clear and for the most part reasonable.
Critical weaknesses
The primary issue is the paper does not address a problem of any real significance. In practice, protein language model training mixes mask and token corruption, and the question of how exactly that interacts with the loss function is just not a particularly consequential problem. Frontier pLMs train with a loss over just the masked version and it works fine. The paper does not convince me that the careful isolation of the problem actually has any significance.
This would be more compelling if the empirical results demonstrated a significant advantage under realistic large-scale pretraining, or revealed something surprising about representation learning. Instead, the paper largely confirms an intuitive property of denoising objectives using very small models and a narrow downstream evaluation. To me this reads more like a single ablation, not something that merits a conference submission.
The main thing that would convince me to raise my score is if I thought this had an implication for practical pLM training. Some barriers to this are: (1) unsupervised performance on ProteinGym is an okay but not great signal, (2) the models train on 300k swissprot predictions and the conventional MLM baselines are signifiacntly worse than ESM2 checkpoints. I don't expect the authors to train a 3B model, but there needs to be some evidence these design choices hold up at bigger scales.
As an aside, to my eye the paper strongly reads of being AI written almost in its entirety. This has not affected my review or rating, but I would strongly urge the authors to consider avoiding this in future.
Questions for authors
- What makes this problem a useful one?
- In practice, the point of masked language model training is to provide a strong embedding for downstream methods. Does this method have any claims for how large, practical pLMs should be trained?
I thank the authors for conducting the full 75k-step comparison, corpus overlap analysis, and teacher budget experiments. I will maintain my score.