- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Diffusion MSA Pairformer (DMP) combines evolutionary representation learning and family-conditioned generation through structured masked diffusion over multiple sequence alignments. Building on the MSA Pairformer backbone, it mixes token-, row-, column-, and query-level masking tasks. Evaluation covers variant-effect prediction, intra-protein and interface contacts, MSA augmentation for structure prediction, and family-conditioned enzyme design. The main contribution is the structured training objective and its use within a common representation-and-generation model.
Critical strengths
- The central representation result has a controlled comparison. Appendix B.3 holds the architecture and training pipeline fixed while varying task probabilities. Against uniform token diffusion, the full mixture improves ProteinGym by 0.0206 in weighted Spearman correlation, PPI-27 by 0.140 in mean P@K, and CASP15 by 0.046 in mean P@L. All three comparisons remain significant under a Bonferroni correction across the twelve tests in Table 9. This provides substantial evidence for structured masking at the evaluated MSA depth.
- The method builds on a compact, established MSA backbone. The 111M-parameter model uses OpenProteinSet-derived Uniclust30 MSAs, and its comparison with MSA Pairformer is closely matched in backbone scale and corpus source. Competitive performance against much larger single-sequence models is noteworthy, although parameter count alone does not account for the cost of obtaining and processing MSAs.
- The evaluation reuses existing frozen readouts. Contact and interface predictions use MSA Pairformer's pretrained heads without refitting them for DMP, avoiding additional task-specific head training as an explanation for the gains. Paired tests and bootstrap intervals further improve the transparency of the evaluation.
- The generation experiments extend beyond a single application. The multi-EC benchmark and ligand-aware structural evaluation provide useful evidence for family-conditioned generation. The authors also report the unfavorable pLDDT-ranked MSA-augmentation results and explicitly distinguish sequence-level functional prediction from structural validation.
Critical weaknesses
-
The proposed roles of the masking tasks are only partially validated. Row masking is motivated by homolog generation, query masking by family-conditioned protein generation, and column masking by co-evolutionary learning. However, the task-removal ablations are evaluated only on representation benchmarks. Evaluating the existing ablation checkpoints on MSA augmentation and enzyme design would test whether the tasks support their intended applications. A perfectly diagonal pattern is not required, since tasks can share information, but the present evidence establishes the benefit of the mixture more directly than the proposed division of roles. The removal variants also reassign probability to token masking, so their results describe substitution by token diffusion rather than removal in isolation.
-
The MLM-versus-diffusion comparison needs matched MSA depth. The structured-versus-uniform ablation uses depth 32, whereas the headline comparison with MSA Pairformer uses the full-depth setting. Comparing uniform diffusion at one depth against masked language modeling at another cannot establish the effect of replacing the objective. Evaluating MSA Pairformer at depth 32, or the ablation checkpoints under the headline depth setting, would clarify the separate contributions of diffusion and structured masking. This would refine the interpretation without undermining the controlled structured-versus-uniform result already reported. Relatedly, the headline gains over MSA Pairformer are 0.016 on ProteinGym and 0.057 on CASP15. Please report paired confidence intervals for these two baseline differences; the ablation tests in Table 9 do not quantify their uncertainty.
-
The practical MSA-augmentation advantage is weaker than the oracle results suggest. Under pLDDT selection, DMP gives mean TM-scores of 0.3268 versus 0.3466 for AF2 with the original shallow MSA, and 0.4440 versus 0.4571 for single-sequence ESMFold2. The conditional-NLL selector improves these results and is usable without native structures, which is valuable. However, on ESMFold2 its mean TM-score is 0.4606 versus 0.4589 using the original shallow MSA, a small difference without a reported paired uncertainty estimate. Both win on 25 of 52 cases against the single-sequence baseline, although equal win counts do not imply equal performance. Since only DMP receives the additional selector, the comparison does not isolate generation quality from selection quality. Oracle TM is an informative retrospective ceiling, not a deployable selection rule. Comparable selector treatment and explicit candidate-pool sizes would make the practical contribution clearer.
Questions for authors
- How do the existing task-ablation checkpoints perform on MSA augmentation and enzyme design? Can the interpretation of their roles distinguish task substitution from task removal?
- At matched MSA depth, how do MSA Pairformer, uniform masked diffusion, and the structured mixture compare? Is DMP trained from scratch or initialized from MSA Pairformer, and how do the training budgets compare?
- Can the generation methods be compared under a common selection procedure, or with analogous method-specific selectors where available, with paired uncertainty estimates for selected TM-score differences?
- Please clarify the candidate pool for each oracle baseline, the separate contributions of Generate-to-32 and Fill-to-32, and how MSA information is supplied in the ESMFold2 evaluation.
5 comments
Response to Reviewer CCbN (2/4)
W1 for another page
The results demonstrate task-specific contributions aligned with their design, validating the structured decomposition. MSA augmentation generates additional homolog rows—removing row masking significantly reduces ESMFold2 oracle TM from 0.5217 to 0.5082. Enzyme design generates query sequences conditioned on family—removing query masking causes severe degradation: CLEAN All-EC drops from 0.916 to 0.423, and FoldBench success plummets from 42.2% to 4.2%. Uniform masking completely fails at functional prediction.
This diagonal pattern—row masking benefits row generation, query masking benefits query generation—demonstrates the intrinsic correctness of the multi-task design at the data-structure level. The alignment between each task's training design and its downstream contribution validates that our structured masking paradigm captures the inherent organization of MSA data. MSA augmentation which depends on row generation relies on row mask task and enzyme design is also sensitive to query task. Together with our representation ablations, these generation results show that all structured tasks make significant, complementary contributions to both representation learning and generation, establishing the necessity of the complete task mixture beyond what uniform token diffusion can provide. We will add this generation ablation study to the ablation part of the revised manuscript.
W2. Training depth and paired confidence intervals.
Apologies for any confusion. All ablation comparisons use matched MSA training depth (=32). Our main model is trained with =512 and =128; due to computational limitations, all five ablation variants (Full, Uniform, w/o query/row/column) use =32 during training. The N32 designation refers to training depth, not an inference restriction. This ensures that the structured-versus-uniform comparison isolates the effect of the masking objective rather than confounding it with depth differences.
For the headline comparison against MSA Pairformer, we have computed paired 95% confidence intervals as requested:
Table E. Paired benchmark improvements over MSA Pairformer. ProteinGym reports weighted Spearman across 211 assays; CASP15 reports long-range P@L across 57 targets. Confidence intervals are paired 95% CIs.
| Benchmark | MSA Pairformer | DMP | Difference [paired 95% CI] |
|---|---|---|---|
| ProteinGym | 0.4334 | 0.4492 | +0.0158 [0.0105, 0.0213] |
| CASP15 | 0.5016 | 0.5587 | +0.0571 [0.0339, 0.0827] |
Response to Reviewer CCbN (3/4)
W3. Generation quality and ranking quality.
Thank you for this important suggestion to separate these two concepts. We conceptualize MSA generation performance as comprising two orthogonal components: generator quality, measured by oracle TM-score (the best achievable performance in a candidate pool, independent of selection), and ranking quality, measured by how well practical selectors identify high-quality candidates from that pool.
DMP demonstrates superior generator quality. Oracle TM directly measures generator quality: DMP achieves 0.4178 on ColabFold and 0.5312 on ESMFold2, surpassing MSAGPT (0.4163 and 0.5205) and ProfileBFN (0.3805 and 0.4826). This establishes DMP's advantage in candidate-pool quality regardless of which selector is subsequently applied.
To enable fair comparison of ranking quality across generators, we conducted two selector experiments applying consistent protocols to all methods:
- Shared DMP-NLL: We use DMP as a unified NLL calculation method for all models' generated MSAs. This yields significant improvement only for DMP itself, likely due to distribution differences and model preferences in the generated MSAs.
- Own meanP: Each model evaluates its own generated MSAs using forward-pass mean probability combined with pLDDT as a ranking score, providing a fairer demonstration of selector effectiveness based on model confidence plus prediction confidence. DMP shows the most significant improvement under both protocols (Table F).
Table F. Selector comparison on Hard52. Values are mean TM-score. Selector stars indicate significantly higher TM than same-pool pLDDT top-1: * , ** , *** .
| Generator | pLDDT top-1 TM | Shared DMP-NLL TM | Own meanP TM | Oracle TM |
|---|---|---|---|---|
| ColabFold | ||||
| ProfileBFN | 0.3023 | 0.3014 | 0.3033 | 0.3805 |
| MSAGPT | 0.3490 | 0.3434 | 0.3530 | 0.4157 |
| DMP | 0.3268 | 0.3581** | 0.3591* | 0.4178 |
| ESMFold2 | ||||
| ProfileBFN | 0.3976 | 0.4013 | 0.4006 | 0.4826 |
| MSAGPT | 0.4572 | 0.4542 | 0.4580 | 0.5207 |
| DMP | 0.4440 | 0.4606 | 0.4600 | 0.5312 |
For the requested ESMFold2 comparison, the original NLL selector gives 0.4606 versus 0.4589 for the shallow-MSA baseline. The paired difference is +0.00175 TM, with a 95% CI of [-0.01362, +0.02168]. We report this small mean improvement together with its uncertainty. Though we cannot claim statistical significance, it shows comparable performance to the shallow-MSA baseline in practical experiment. As a result, DMP's generator quality is superior but how to improve ranking quality is a future direction to explore.
Table G. Paired ESMFold2 comparison of the original NLL selector with the shallow-MSA baseline (=52).
| Shallow-MSA TM | NLL selector TM | TM | Paired 95% CI | |
|---|---|---|---|---|
| 0.458854 | 0.460608 | +0.001754 | [-0.013620, +0.021684] | 0.895946 |
Response to Reviewer CCbN (1/4)
Thank you for your constructive review. Your feedback has helped us strengthen both the evaluation and the interpretation of our results. We address each concern below and will incorporate these additions into the revision.
W1. Generation-task ablations.
Thank you for this helpful suggestion. We have evaluated five variants—Full, Uniform only, w/o query, w/o row, and w/o column—on MSA augmentation and enzyme design. Given the training-time limitation, we use checkpoints trained with a maximum sequence length of 512 and a maximum MSA depth of 32 for all five variants, with the same training budget and matched sampling budgets and selection rules within each task.
Table A. MSA-augmentation ablation on Hard52, pLDDT top-1 selector. Stars mark oracle-TM or selected-TM values significantly below Full: * padj<0.05.
Table B. Enzyme-generation ablation on CLEAN sequence-level outcomes. DMP generates 100 sequences per case. Asterisks mark values significantly below Full in two-sided paired tests: * padj<0.05, ** padj<0.01, *** padj<0.001.
Table C. Enzyme-generation ablation on protein-only task with each case having 100 predictions.
Table D. Enzyme-generation ablation on with-ligand task with 100 sequences per case.