acceptodds
Under review as a conference paper at ICLR 2027

Corrupted Inputs Make Dense Supervision Informative for Zero-Shot Variant Scoring: A Controlled Factorial Study of Compact Protein Encoders

Abstract

Masked language modeling, the default pretraining objective for protein encoders, fixes two separable choices: the input (a sequence with mask tokens, or a complete sequence with some residues replaced by substitutes) and the positions at which the loss is computed (selected or all positions). We cross them with architecture, corpus, optimizer and training length fixed, train compact encoders (7.5M and 33.5M parameters) from scratch on 300k Swiss-Prot sequences, and evaluate zero-shot variant effect prediction on 201 ProteinGym substitution assays. The factors interact: over three seeds at 7.5M, computing the loss at all positions increases mean Spearman by +0.0095 with masked input and by +0.0830 with corrupted input. With masked input almost every visible residue is correct, and the added loss is minimized by copying. Most of the aggregate interaction arises in assays mixing substitution counts, where the count alone predicts fitness; among variants with the same number the interaction is smaller (+0.0194) but positive in every seed, and it remains (+0.0178) when the corruption rate is matched and no pretrained model is used. In the matched design, with loss at all positions, corrupted input outperforms masked input by +0.0890. Under a supervised probe of frozen embeddings, the interaction is not detected in the original design and small in the matched one, so the conclusions concern zero-shot scoring, not learned representations. Sampling substitutes from BLOSUM62 recovers most of the improvement over uniform sampling that a frozen ESM-2 650M teacher gives, and an entropy-preserving permutation of the matrix recovers little.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language models, pretraining objectives, masked language modeling, supervision density, corrupted inputs, variant effect prediction, ProteinGym, replaced token detection

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_kCmvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper trains protein encoders at 7.5 and 33M parameter sizes with two main experimental decisions:

  1. Corruption mode (mask vs mutation)
  2. Loss supervision through all tokens vs only through the corrupted positions.

The authors ultimately argue that dense supervision over all tokens is useful when using mutation based corruption but not when using standard masking. They validate using zero-shot likelihood benchmarks from ProteinGym.

Critical strengths

Given the objective, the experimental design is clear and for the most part reasonable.

Critical weaknesses

The primary issue is the paper does not address a problem of any real significance. In practice, protein language model training mixes mask and token corruption, and the question of how exactly that interacts with the loss function is just not a particularly consequential problem. Frontier pLMs train with a loss over just the masked version and it works fine. The paper does not convince me that the careful isolation of the problem actually has any significance.

This would be more compelling if the empirical results demonstrated a significant advantage under realistic large-scale pretraining, or revealed something surprising about representation learning. Instead, the paper largely confirms an intuitive property of denoising objectives using very small models and a narrow downstream evaluation. To me this reads more like a single ablation, not something that merits a conference submission.

The main thing that would convince me to raise my score is if I thought this had an implication for practical pLM training. Some barriers to this are: (1) unsupervised performance on ProteinGym is an okay but not great signal, (2) the models train on 300k swissprot predictions and the conventional MLM baselines are signifiacntly worse than ESM2 checkpoints. I don't expect the authors to train a 3B model, but there needs to be some evidence these design choices hold up at bigger scales.

As an aside, to my eye the paper strongly reads of being AI written almost in its entirety. This has not affected my review or rating, but I would strongly urge the authors to consider avoiding this in future.

Questions for authors
  1. What makes this problem a useful one?
  2. In practice, the point of masked language model training is to provide a strong embedding for downstream methods. Does this method have any claims for how large, practical pLMs should be trained?
Reviewer_9yNvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides a systematic comparison of input corruption and supervision density, with additional controls for corruption rate and teacher information. The analysis of substitution order also helps clarify where the reported improvements come from.

Critical strengths

The factorial comparisons are reasonably systematic, and the authors include additional controls for corruption rate and teacher information. I also appreciate the analysis of substitution order and the supervised probe, which help clarify the limitations of the main results. The paper is generally clear and provides detailed experimental configurations.

Critical weaknesses

1.The main improvement is largely driven by the evaluation protocol and substitution-count effects. My biggest concern is that the reported improvement does not convincingly demonstrate better variant effect prediction. As shown in Appendix L, 82% of the interaction in the original design comes from assays that contain variants with different numbers of substitutions. This increases to 97% in the fully matched design. After controlling for substitution count, the interaction decreases from +0.0734 to +0.0194 in the original design, and from +0.1907 to +0.0178 in the fully matched design.

This is a substantial reduction. The authors also report that simply ranking variants by the negative number of substitutions gives a mean Spearman correlation of +0.3780 on the 69 mixed-order assays. I therefore question how much of the headline improvement reflects meaningful discrimination between variants rather than a general penalty for additional substitutions.

There is also a problem with the sparse corrupted-input baseline. It is trained to reconstruct residues at substituted positions, but evaluated on an uncorrupted wild-type sequence. In the fully matched design, this model assigns higher scores to the mutant residue for 99.5% of single substitutions. Its poor performance is therefore not surprising. Adding supervision at unaltered positions corrects this behavior, but this does not necessarily establish that the model has learned more useful information about protein fitness.

The authors acknowledge these limitations and report positive within-order effects. However, I do not think the remaining results justify the significance placed on the large aggregate improvements.

2.The contribution is incremental, and the experiments do not establish a broader benefit of the training objective.

The main observation is closely related to existing work. ELECTRA already studied all-token supervision with replaced inputs and discussed why predicting visible tokens can reduce to copying. ProteinBERT also used amino acid substitution corruption with full-sequence reconstruction. The paper provides a more controlled comparison in the protein setting, but I find the additional insight fairly limited.

More importantly, the improvement appears specific to the output distribution used for wild-type-marginal scoring. In the supervised probe on frozen embeddings, the interaction is not detected in the original design (+0.0028, p = 0.34) and is very small in the matched design (+0.0058). This raises the question of whether dense supervision actually improves the learned protein representations or mainly makes the output head more suitable for the chosen scoring protocol.

The experimental setting is also quite limited. The models are trained on only 300k Swiss-Prot sequences, and the authors acknowledge that they are undertrained. For example, the 7.5M MASK baseline scores 0.1341, compared with 0.2578 for the released ESM-2 8M checkpoint under wild-type marginals. The longer-training experiments do not repeat the complete factorial comparison. In addition, the training sequences are not filtered against the benchmark proteins.

I do not require a controlled empirical study to introduce a new architecture or achieve state-of-the-art performance. However, given the limited novelty of the objective, I would expect stronger evidence that the observed effect reveals a useful general principle rather than a property of this particular scoring setup.

Overall, I am not convinced that the current results establish a sufficiently significant contribution for ICLR. The main improvement is substantially reduced after controlling for substitution order, and the broader contribution beyond existing pretraining objectives remains limited.

Questions for authors

1 How much of the improvement remains beyond simple substitution-based priors? Could the authors compare against BLOSUM62-only substitution scores and the context-free predictor considered in Appendix H, especially for single substitutions and within-order ranking? For mixed-order assays, a comparison with the substitution-count baseline is also important.

2 Does the interaction persist when all four configurations are trained for longer? Appendix N extends MASK and CORRUPT-T to 75,000 steps, but does not repeat the complete matched factorial experiment. Since the models are still undertrained, it is unclear whether the interaction remains of similar magnitude after more extensive training.

3 How much of the improvement comes from the output head rather than the encoder representations? The supervised probe shows little or no interaction. Could the authors provide further evidence that dense supervision improves the information captured by the encoder, rather than primarily correcting the output probabilities used by wild-type-marginal scoring?

Reviewer_rLfuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how input corruption and supervision density interact in compact protein encoders for zero-shot variant scoring. It compares masked versus substitution-corrupted inputs with losses computed at selected versus all positions. Experiments on 201 ProteinGym assays include matched corruption rates, teacher-free controls, multiple seeds, and two model sizes. The results support combining substitution corruption with dense supervision, while showing that much of the aggregate interaction reflects substitution-count effects. Additional experiments examine teacher-based sampling, BLOSUM62 corruption, and frozen representations.

Critical strengths
  1. The controlled comparisons provide useful evidence about the interaction between corruption and supervision density. Matching corruption rates and removing teacher information strengthen the conclusions beyond the initial comparison. The teacher-free dense configuration also retains a substantial advantage over masked input when evaluated within substitution counts.

  2. The evaluation examines alternative explanations rather than relying only on aggregate performance. The substitution-count analysis, frozen-representation probe, and shuffled-BLOSUM control help distinguish scoring improvements from broader representation gains and clarify the contribution of biologically plausible corruption.

Critical weaknesses
  1. The empirical scope remains limited to compact, relatively undertrained models and one benchmark family. Longer training compares only two configurations, so it does not establish whether the full factorial interaction persists with more converged models.

  2. The central interaction becomes considerably smaller after controlling for substitution count, and corresponding gains in frozen representations are limited. Together with the unexplained masked-marginal results, this restricts the contribution to the particular zero-shot scoring setting. The paper appropriately acknowledges these limitations, but they temper its significance.

Questions for authors
  1. Can the authors provide evidence that the interaction persists under longer training for all four configurations, particularly in the fully matched, teacher-free design? This would help distinguish a persistent benefit from differences in learning speed.

  2. Why does the standard MLM baseline perform substantially worse under masked marginals than under wild-type marginals? Have training convergence or implementation-specific factors been investigated beyond validating the scoring code on released checkpoints?

Reviewer_1aooOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1(−1)
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper investigates how input corruption and supervision density interact in masked language modeling for protein encoders. The authors compare masked inputs and substitution-corrupted inputs, together with sparse and dense loss computation, using compact ESM-2-style models trained from scratch on 300k Swiss-Prot sequences. The evaluation covers 201 ProteinGym substitution assays. The main finding is that dense supervision provides substantially larger improvements when the input contains substituted amino acids rather than mask tokens. The authors further investigate the contribution of BLOSUM62 corruption, the role of unaltered positions, and the influence of mutation order on evaluation results. The study provides a controlled empirical analysis of pretraining objectives for zero-shot protein variant scoring.

Critical strengths
  1. A well-controlled experimental design The factorial comparison provides a useful way to separate the effects of input corruption and supervision density. In particular, the fully matched, teacher-free experiment using BLOSUM62 helps address potential confounding from different corruption rates and pretrained teacher information. The additional loss analysis also gives a reasonable explanation for why dense supervision behaves differently under masked and corrupted inputs.
  2. Careful treatment of evaluation confounding The authors recognize that the number of substitutions can itself predict fitness in assays containing both single- and multiple-substitution variants. Rather than relying entirely on aggregate Spearman correlation, they introduce an order-stratified evaluation. This makes the conclusions more credible and demonstrates good experimental awareness.
Critical weaknesses
  1. The main improvement is partly driven by substitution count The reported interaction is relatively large when all variants are evaluated together. However, mixed-order assays account for 82% of the interaction in the original design and 97% in the fully matched design. Ranking variants simply by the negative number of substitutions already achieves a mean Spearman correlation of +0.378 in these assays. After controlling for substitution order, the interaction becomes much smaller: +0.0194 in the original design and +0.0178 in the fully matched design. Although these effects remain statistically significant, their practical magnitude is considerably more limited than the headline results suggest.
  2. Limited evidence for broader generalization The study focuses on one benchmark family and two model sizes. The 7.5M models use three seeds, whereas the 33.5M experiments use only one seed. Furthermore, the authors acknowledge that the training corpus was not filtered against benchmark proteins. The supervised probe also does not detect the original interaction in frozen representations. Consequently, the conclusions should remain restricted to zero-shot scoring rather than being generalized to protein representation learning more broadly.
Questions for authors
  1. Can the authors provide an additional evaluation with sequence-level separation between the training corpus and ProteinGym proteins? This would help determine whether the observed improvement is affected by training-set overlap.
  2. Does the interaction between corruption type and supervision density persist with multiple seeds at 33.5M parameters and under longer training schedules?

Comments on the paper

2 comments

I thank the authors for conducting the full 75k-step comparison, corpus overlap analysis, and teacher budget experiments. I will maintain my score.

Thank you for reviewing the additional results. Your suggestions helped us strengthen the experimental evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.