acceptodds
Under review as a conference paper at ICLR 2027

Diffusion MSA Pairformer: Structured Masked Diffusion for Protein Representation and Generation

Abstract

Multiple sequence alignments (MSAs) capture evolutionary constraints on protein structure and function through patterns of conservation and coevolution. However, explicit modeling of MSA compositional structure—namely queries, homologous rows, and aligned columns—is notably lacking in current MSA language models. Relying on uniform token masking, existing approaches fail to fully exploit evolutionary information encoded within MSAs. We introduce Diffusion MSA Pairformer, a unified generative model that decomposes MSA diffusion into biologically motivated masking tasks and captures evolutionary dependencies through coupled MSA-pair representations. Empirically, this structured multi-task approach significantly enhances representation learning, with ablation studies confirming its crucial advantage over uniform masking. Our robust approach also enhances generation capabilities: in MSA augmentation and family-conditioned enzyme design, the model achieves superior performance by precisely capturing intra- and interchain structural constraints while balancing sequence diversity, structural consistency, and functional accuracy. These results establish structured MSA diffusion as a principled paradigm unifying evolutionary representation and protein design.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
MSA, protein design, pLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CCbNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Diffusion MSA Pairformer (DMP) combines evolutionary representation learning and family-conditioned generation through structured masked diffusion over multiple sequence alignments. Building on the MSA Pairformer backbone, it mixes token-, row-, column-, and query-level masking tasks. Evaluation covers variant-effect prediction, intra-protein and interface contacts, MSA augmentation for structure prediction, and family-conditioned enzyme design. The main contribution is the structured training objective and its use within a common representation-and-generation model.

Critical strengths
  1. The central representation result has a controlled comparison. Appendix B.3 holds the architecture and training pipeline fixed while varying task probabilities. Against uniform token diffusion, the full mixture improves ProteinGym by 0.0206 in weighted Spearman correlation, PPI-27 by 0.140 in mean P@K, and CASP15 by 0.046 in mean P@L. All three comparisons remain significant under a Bonferroni correction across the twelve tests in Table 9. This provides substantial evidence for structured masking at the evaluated MSA depth.
  2. The method builds on a compact, established MSA backbone. The 111M-parameter model uses OpenProteinSet-derived Uniclust30 MSAs, and its comparison with MSA Pairformer is closely matched in backbone scale and corpus source. Competitive performance against much larger single-sequence models is noteworthy, although parameter count alone does not account for the cost of obtaining and processing MSAs.
  3. The evaluation reuses existing frozen readouts. Contact and interface predictions use MSA Pairformer's pretrained heads without refitting them for DMP, avoiding additional task-specific head training as an explanation for the gains. Paired tests and bootstrap intervals further improve the transparency of the evaluation.
  4. The generation experiments extend beyond a single application. The multi-EC benchmark and ligand-aware structural evaluation provide useful evidence for family-conditioned generation. The authors also report the unfavorable pLDDT-ranked MSA-augmentation results and explicitly distinguish sequence-level functional prediction from structural validation.
Critical weaknesses
  1. The proposed roles of the masking tasks are only partially validated. Row masking is motivated by homolog generation, query masking by family-conditioned protein generation, and column masking by co-evolutionary learning. However, the task-removal ablations are evaluated only on representation benchmarks. Evaluating the existing ablation checkpoints on MSA augmentation and enzyme design would test whether the tasks support their intended applications. A perfectly diagonal pattern is not required, since tasks can share information, but the present evidence establishes the benefit of the mixture more directly than the proposed division of roles. The removal variants also reassign probability to token masking, so their results describe substitution by token diffusion rather than removal in isolation.

  2. The MLM-versus-diffusion comparison needs matched MSA depth. The structured-versus-uniform ablation uses depth 32, whereas the headline comparison with MSA Pairformer uses the full-depth setting. Comparing uniform diffusion at one depth against masked language modeling at another cannot establish the effect of replacing the objective. Evaluating MSA Pairformer at depth 32, or the ablation checkpoints under the headline depth setting, would clarify the separate contributions of diffusion and structured masking. This would refine the interpretation without undermining the controlled structured-versus-uniform result already reported. Relatedly, the headline gains over MSA Pairformer are 0.016 on ProteinGym and 0.057 on CASP15. Please report paired confidence intervals for these two baseline differences; the ablation tests in Table 9 do not quantify their uncertainty.

  3. The practical MSA-augmentation advantage is weaker than the oracle results suggest. Under pLDDT selection, DMP gives mean TM-scores of 0.3268 versus 0.3466 for AF2 with the original shallow MSA, and 0.4440 versus 0.4571 for single-sequence ESMFold2. The conditional-NLL selector improves these results and is usable without native structures, which is valuable. However, on ESMFold2 its mean TM-score is 0.4606 versus 0.4589 using the original shallow MSA, a small difference without a reported paired uncertainty estimate. Both win on 25 of 52 cases against the single-sequence baseline, although equal win counts do not imply equal performance. Since only DMP receives the additional selector, the comparison does not isolate generation quality from selection quality. Oracle TM is an informative retrospective ceiling, not a deployable selection rule. Comparable selector treatment and explicit candidate-pool sizes would make the practical contribution clearer.

Questions for authors
  1. How do the existing task-ablation checkpoints perform on MSA augmentation and enzyme design? Can the interpretation of their roles distinguish task substitution from task removal?
  2. At matched MSA depth, how do MSA Pairformer, uniform masked diffusion, and the structured mixture compare? Is DMP trained from scratch or initialized from MSA Pairformer, and how do the training budgets compare?
  3. Can the generation methods be compared under a common selection procedure, or with analogous method-specific selectors where available, with paired uncertainty estimates for selected TM-score differences?
  4. Please clarify the candidate pool for each oracle baseline, the separate contributions of Generate-to-32 and Fill-to-32, and how MSA information is supplied in the ESMFold2 evaluation.
5 comments
AuthorsOpenReview ↗

Response to Reviewer CCbN (1/4)

Thank you for your constructive review. Your feedback has helped us strengthen both the evaluation and the interpretation of our results. We address each concern below and will incorporate these additions into the revision.

W1. Generation-task ablations.

Thank you for this helpful suggestion. We have evaluated five variants—Full, Uniform only, w/o query, w/o row, and w/o column—on MSA augmentation and enzyme design. Given the training-time limitation, we use checkpoints trained with a maximum sequence length of 512 and a maximum MSA depth of 32 for all five variants, with the same training budget and matched sampling budgets and selection rules within each task.

Table A. MSA-augmentation ablation on Hard52, pLDDT top-1 selector. Stars mark oracle-TM or selected-TM values significantly below Full: * padj<0.05p_{\mathrm{adj}}<0.05.

ConfigurationpLDDT top-1 (ColabFold)Oracle (ColabFold)pLDDT top-1 (ESMFold2)Oracle (ESMFold2)
Full0.31380.41690.44300.5217
w/o query0.30820.41240.43050.5163
w/o row0.29950.40700.42860.5082*
w/o column0.31120.41580.44170.5182
Uniform0.28600.3876*0.44020.5107

Table B. Enzyme-generation ablation on CLEAN sequence-level outcomes. DMP generates 100 sequences per case. Asterisks mark values significantly below Full in two-sided paired tests: * padj<0.05p_{\mathrm{adj}}<0.05, ** padj<0.01p_{\mathrm{adj}}<0.01, *** padj<0.001p_{\mathrm{adj}}<0.001.

ConfigurationAny-ECAll-ECPairwise identityUnique
Full0.93800.91600.55981.0000
w/o query0.5013***0.4227***0.26491.0000
w/o row0.93530.92330.58751.0000
w/o column0.93670.91000.53661.0000
Uniform0.0000***0.0000***0.06211.0000

Table C. Enzyme-generation ablation on protein-only task with each case having 100 predictions.

ConfigurationpLDDTProtein TMAny-ECAll-ECPairwise identityUnique
Full92.280.94631.00000.96830.57211.0000
w/o query69.930.79540.37000.30670.26631.0000
w/o row92.970.94760.99830.99670.60591.0000
w/o column91.970.94311.00000.94500.56221.0000
Uniform30.880.28050.00000.00000.06221.0000

Table D. Enzyme-generation ablation on with-ligand task with 100 sequences per case.

ConfigurationSuccess (%)LRMSD (Å)lDDT-PLIlDDT-LPTM
Full42.225.16680.62800.67340.9125
w/o query4.259.79990.33900.40510.7588
w/o row40.725.19240.62350.67390.9215
w/o column41.655.04360.62480.67010.9045
Uniform0.0019.44010.00930.10100.2828
AuthorsOpenReview ↗

Response to Reviewer CCbN (2/4)

W1 for another page

The results demonstrate task-specific contributions aligned with their design, validating the structured decomposition. MSA augmentation generates additional homolog rows—removing row masking significantly reduces ESMFold2 oracle TM from 0.5217 to 0.5082. Enzyme design generates query sequences conditioned on family—removing query masking causes severe degradation: CLEAN All-EC drops from 0.916 to 0.423, and FoldBench success plummets from 42.2% to 4.2%. Uniform masking completely fails at functional prediction.

This diagonal pattern—row masking benefits row generation, query masking benefits query generation—demonstrates the intrinsic correctness of the multi-task design at the data-structure level. The alignment between each task's training design and its downstream contribution validates that our structured masking paradigm captures the inherent organization of MSA data. MSA augmentation which depends on row generation relies on row mask task and enzyme design is also sensitive to query task. Together with our representation ablations, these generation results show that all structured tasks make significant, complementary contributions to both representation learning and generation, establishing the necessity of the complete task mixture beyond what uniform token diffusion can provide. We will add this generation ablation study to the ablation part of the revised manuscript.

W2. Training depth and paired confidence intervals.

Apologies for any confusion. All ablation comparisons use matched MSA training depth (Nmax⁡N_{\max}=32). Our main model is trained with Lmax⁡L_{\max}=512 and Nmax⁡N_{\max}=128; due to computational limitations, all five ablation variants (Full, Uniform, w/o query/row/column) use Nmax⁡N_{\max}=32 during training. The N32 designation refers to training depth, not an inference restriction. This ensures that the structured-versus-uniform comparison isolates the effect of the masking objective rather than confounding it with depth differences.

For the headline comparison against MSA Pairformer, we have computed paired 95% confidence intervals as requested:

Table E. Paired benchmark improvements over MSA Pairformer. ProteinGym reports weighted Spearman across 211 assays; CASP15 reports long-range P@L across 57 targets. Confidence intervals are paired 95% CIs.

BenchmarkMSA PairformerDMPDifference [paired 95% CI]
ProteinGym0.43340.4492+0.0158 [0.0105, 0.0213]
CASP150.50160.5587+0.0571 [0.0339, 0.0827]
AuthorsOpenReview ↗

Response to Reviewer CCbN (3/4)

W3. Generation quality and ranking quality.

Thank you for this important suggestion to separate these two concepts. We conceptualize MSA generation performance as comprising two orthogonal components: generator quality, measured by oracle TM-score (the best achievable performance in a candidate pool, independent of selection), and ranking quality, measured by how well practical selectors identify high-quality candidates from that pool.

DMP demonstrates superior generator quality. Oracle TM directly measures generator quality: DMP achieves 0.4178 on ColabFold and 0.5312 on ESMFold2, surpassing MSAGPT (0.4163 and 0.5205) and ProfileBFN (0.3805 and 0.4826). This establishes DMP's advantage in candidate-pool quality regardless of which selector is subsequently applied.

To enable fair comparison of ranking quality across generators, we conducted two selector experiments applying consistent protocols to all methods:

  • Shared DMP-NLL: We use DMP as a unified NLL calculation method for all models' generated MSAs. This yields significant improvement only for DMP itself, likely due to distribution differences and model preferences in the generated MSAs.
  • Own meanP: Each model evaluates its own generated MSAs using forward-pass mean probability combined with pLDDT as a ranking score, providing a fairer demonstration of selector effectiveness based on model confidence plus prediction confidence. DMP shows the most significant improvement under both protocols (Table F).

Table F. Selector comparison on Hard52. Values are mean TM-score. Selector stars indicate significantly higher TM than same-pool pLDDT top-1: * padj<0.05p_{\mathrm{adj}}<0.05, ** padj<0.01p_{\mathrm{adj}}<0.01, *** padj<0.001p_{\mathrm{adj}}<0.001.

GeneratorpLDDT top-1 TMShared DMP-NLL TMOwn meanP TMOracle TM
ColabFold
ProfileBFN0.30230.30140.30330.3805
MSAGPT0.34900.34340.35300.4157
DMP0.32680.3581**0.3591*0.4178
ESMFold2
ProfileBFN0.39760.40130.40060.4826
MSAGPT0.45720.45420.45800.5207
DMP0.44400.46060.46000.5312

For the requested ESMFold2 comparison, the original NLL selector gives 0.4606 versus 0.4589 for the shallow-MSA baseline. The paired difference is +0.00175 TM, with a 95% CI of [-0.01362, +0.02168]. We report this small mean improvement together with its uncertainty. Though we cannot claim statistical significance, it shows comparable performance to the shallow-MSA baseline in practical experiment. As a result, DMP's generator quality is superior but how to improve ranking quality is a future direction to explore.

Table G. Paired ESMFold2 comparison of the original NLL selector with the shallow-MSA baseline (nn=52).

Shallow-MSA TMNLL selector TMΔ\Delta TMPaired 95% CIpp
0.4588540.460608+0.001754[-0.013620, +0.021684]0.895946
Reviewer_QzTmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Diffusion MSA Pairformer (DMP), a generative protein language model that extends the original MSA Pairformer architecture by incorporating a "structured masked discrete diffusion" mechanism.

Critical strengths
  1. The paper addresses a significant limitation in existing MSA-based modeling approaches. MSAs contain structured evolutionary information spanning homologous sequences and aligned residue positions. By explicitly distinguishing between reconstruction tasks at the token, row, column, and query-sequence levels, this method leverages these complementary dependencies more effectively.
  2. The proposed framework establishes a coherent link between representation learning and conditional generation. By integrating structured masked diffusion into the MSA Pairformer architecture, the model achieves multiple functionalities—such as mutation effect prediction, structural contact prediction, MSA augmentation, and protein generation conditioned on protein families—within a unified framework.
  3. Compared to the original MSA Pairformer, this model demonstrates superior performance in terms of Spearman correlation coefficients on ProteinGym tasks and in intra-chain contact prediction. Figure 3 presents ablation study results, confirming that this structured combination of tasks is more advantageous than uniform token masking.
Critical weaknesses
  1. The methodological innovation is limited. DMP builds upon the existing MSA Pairformer architecture, incorporating masked discrete diffusion and structured masking objectives. While the task decomposition is biologically motivated, the individual components—both in terms of architecture and training—are largely based on existing techniques. Consequently, the primary innovation lies in the specific combination of masking strategies, rather than the proposal of an entirely novel generative architecture.
  2. Improvements in representation learning are uneven. As noted in Section 5.1, the Spearman correlation coefficient on ProteinGym increased from 0.433 to 0.449, while the P@K metric for protein-protein interaction (PPI) interfaces improved from 0.521 to 0.538. While the improvement in intrachain contacts is quite notable, the gains on the other two benchmarks are relatively limited. Figure 3 provides statistical evidence regarding the contribution of structured masking; however, a deeper discussion on the practical significance and variability of these improvements would help establish the broader value of the proposed training strategy.
  3. The results obtained through MSA augmentation depend heavily on the selection of candidate samples. In Table 1, DMP achieves the highest "oracle" TM-score of 0.4178; however, when selection is based on the pLDDT metric, the TM-score drops to 0.3268—a figure lower than those of MSAGPT (0.3490) and the full-scale AF2 baseline (0.3466). Although the pLDDT–NLL selector proposed in the paper improves this metric to 0.3581, this evaluation process employs a selection strategy unique to DMP—one that is not applied to the other generative models used for comparison. Even though Appendix C.2 clarifies that this selector does not utilize information from native structures or TM-scores, this approach effectively conflates "generation quality" with "differences in candidate ranking." Evaluating alternative selection methods more consistently across different approaches would help to better isolate the specific contributions attributable to the generator itself. 4.The ablation study fails to sufficiently demonstrate the specific contribution of "structured masking" to generative performance. It remains unclear whether all four masking objectives contribute substantively to the generation task, or if certain specific objectives play a dominant role. Comparing this approach against a "uniform masking" strategy—or evaluating variants where individual masking objectives are removed—would provide stronger evidence for the paper’s core methodological claims. 5.Functional validation of the generated enzymes relies heavily on computational prediction tools. In the multi-EC (Enzyme Commission number) benchmark, functional accuracy is assessed based on predictions from CLEAN rather than experimentally measured enzymatic activity. Although Section 5.4.3 provides further evaluation of predicted protein structures and ligand-binding geometries, these analyses are likewise dependent on computational structure prediction. 6.The generalizability of the conclusions regarding enzyme design is constrained by the scope of the benchmarks. The multi-EC evaluation covers only 15 enzyme families, while the ligand-based evaluation is limited to just 6. Although the authors acknowledge this limitation in Section 7, the reported performance may be highly dependent on the specific characteristics of the selected enzyme families. Further evaluation across a more diverse range of enzyme families—particularly those with distinct catalytic mechanisms and evolutionary constraints—would provide more robust support for the authors' claims regarding the model's generalizability.
Questions for authors

Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer QzTm (1/5)

Thank you for your review and for recognizing the value of modeling MSA structure within a unified framework. We appreciate your careful evaluation and address each concern below.

W1. Methodological innovation.

Thank you for this comment. We clarify how our contribution addresses a fundamental modeling question that prior work has not solved.

Existing masked diffusion methods for proteins (DPLM, EvoDiff) employ uniform token masking, which works well for single sequences where each position relates only to other positions in the same sequence. MSAs, however, organize evolutionary information across two structural dimensions: rows (evolutionarily related homologs) and columns (aligned positions under functional constraints). Uniform token masking does not explicitly model these structured dependencies.

Our contribution is not a simple combination of existing techniques, but a biologically-motivated post-training paradigm with synergistic effects. We decompose learning into four tasks aligned with MSA organization: token-level (local variation), row-level (homolog patterns), column-level (positional conservation), and query-level (family-conditioned generation). This is not an arbitrary combination—the MSA's biological structure necessitates structured task decomposition to capture complementary dependencies that uniform masking cannot learn.

Our ablation studies demonstrate this is not mere combination but synergy. In representation (Figure 3), the full task mixture consistently outperforms uniform masking: ProteinGym Spearman +0.021 (0.442 vs 0.421), PPI median P@K +0.174 (0.471 vs 0.297), CASP15 long-range P@L +0.046 (0.531 vs 0.485). Critically, removing individual tasks degrades performance:column masking alone is responsible for maintaining PPI at 0.471 versus dropping to 0.297, showing each task contributes unique and non-redundant information. In generation (W4 tables below), we observe task-specific patterns: row masking improves MSA augmentation; query masking is essential for enzyme design; uniform masking fails completely.

We build on the MSA Pairformer backbone and masked discrete diffusion framework. What we contribute is identifying that MSA structure necessitates structured training, and validating through comprehensive ablation that each decomposed task captures complementary co-evolutionary dependencies. The consistent task-specific effects across both representation and generation establish this as a transferable paradigm: our framework provides principled guidance for training any MSA-based diffusion model to better extract co-evolutionary information, enabling stronger representation and generation capabilities beyond what single-sequence or uniform-masking approaches can achieve.

We will revise the abstract and introduction to clarify our contribution and innovation.

AuthorsOpenReview ↗

Response to Reviewer QzTm (2/5)

W2. Representation improvements: magnitude and distribution.

Thank you for raising this important point. We provide statistical analysis and case-level distributions to clarify the practical significance.

All three improvements are statistically significant. ProteinGym: +0.0158 weighted Spearman (211 assays, paired t-test pp<0.001, 95% CI [0.0105, 0.0213]). PPI: +0.017 median P@K (35 interfaces, Wilcoxon signed-rank pp=0.032). CASP15: +0.0571 long-range P@L (57 targets, paired t-test pp<0.001, 95% CI [0.0339, 0.0827]). All confidence intervals are statistically significant.

Gains are broadly distributed, not driven by outliers. We examined case-level improvements:

Table A. Case-level improvement distribution: DMP vs MSA Pairformer.

BenchmarkBetterWorseTiedMedian Δ\Delta
ProteinGym (211 assays)131764+0.018
PPI (35 interfaces)22130+0.021
CASP15 (57 targets)38190+0.063

The majority of cases improve rather than degrade, and median values track mean values closely. This confirms improvements are distributed across cases rather than concentrated in outliers.

Practical significance of the benchmarks. Each measures distinct capabilities with direct applications. ProteinGym evaluates variant effect prediction—identifying disease mutations and guiding protein engineering. Contact prediction (PPI interfaces and CASP15 intra-chain) proxies structural modeling: correctly predicted contacts guide structure prediction models toward better prediction performance, informing drug design and interaction analysis. The consistent improvements across all three establish broad representational benefits.

AuthorsOpenReview ↗

Response to Reviewer QzTm (3/5)

W3. MSA augmentation: generator quality versus selector quality.

Thank you for this important distinction. We clarify how we separate generator quality from ranking quality. We conceptualize MSA generation performance as comprising two components: generator quality, measured by oracle TM-score (the best achievable performance across all candidates without selector influence), and ranking quality, measured by how well selectors identify high-quality candidates using pLDDT or other confidence metrics. As shown in Table 1 of the main paper and Table B below, our oracle TM-scores demonstrate generator quality that surpasses other models.

The reason we explore alternatives to pLDDT as a universal selector is that for MSA augmentation methods like ProfileBFN and DMP, increasing MSA depth induces pLDDT inflation unrelated to quality improvement, degrading pLDDT's ranking ability. This creates an unfair benchmark comparison. We therefore developed alternative selector methods to enable fairer cross-model comparison.

We conducted two comparison experiments across models:

  • Shared DMP-NLL: We use DMP as a unified NLL calculation method for all models' generated MSAs. This yields significant improvement only for DMP itself, likely due to distribution differences and model preferences in the generated MSAs.
  • Own meanP: Each model evaluates its own generated MSAs using forward-pass mean probability combined with pLDDT as a ranking score, providing a fairer demonstration of selector effectiveness based on model confidence plus prediction confidence. DMP shows the most significant improvement under both protocols (Table B).

Table B. Selector comparison on Hard52. Values are mean TM-score. Selector stars indicate significantly higher TM than same-pool pLDDT top-1: * padj<0.05p_{\mathrm{adj}}<0.05, ** padj<0.01p_{\mathrm{adj}}<0.01, *** padj<0.001p_{\mathrm{adj}}<0.001.

GeneratorpLDDT top-1 TMShared DMP-NLL TMOwn meanP TMOracle TM
ColabFold
ProfileBFN0.30230.30140.30330.3805
MSAGPT0.34900.34340.35300.4157
DMP0.32680.3581**0.3591*0.4178
ESMFold2
ProfileBFN0.39760.40130.40060.4826
MSAGPT0.45720.45420.45800.5207
DMP0.44400.46060.46000.5312

DMP demonstrates superior generator quality across competing models. Most critically, DMP achieves the highest oracle TM-scores on ColabFold and ESMFold2, surpassing MSAGPT and ProfileBFN. This oracle performance directly measures generator quality independent of any selection strategy.

DMP also achieves competitive ranking performance with various selectors. With the support of both selector protocols, DMP effectively improves the poor performance under the pLDDT top-1 selector and achieves similar ranking performance with MSAGPT, demonstrating that DMP achieves competitive practical performance in MSA augmentation. These selector comparison results will be incorporated into Section 5.3 (Generation) of the revised manuscript.

Reviewer_482WOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Diffusion MSA Pairformer (DMP), which adds token, row, column, and query-level masked-diffusion objectives to the MSA Pairformer backbone, coupling MSA and pair representations for both representation learning and family-conditioned generation. Evaluation that spans variant-effect prediction, intra-chain and interface contacts, MSA augmentation for orphan proteins, and multi-EC enzyme design demonstrates the effectiveness of the proposed method.

Critical strengths

The structured v.s. uniform ablation is a controlled comparison. This paper holds architecture and pipeline fixed and only varies task probabilities, and the full mixture improves all three representation benchmarks, with paired permutation tests and bootstrap intervals rather than just point estimates. Reusing MSA Pairformer's frozen contact heads without refitting them is also a good choice, since it rules out task-specific head training as an explanation for the gains. The enzyme evaluation goes beyond sequence-level annotation into ligand-aware geometry, and family-macro ligand success improves a lot over ProfileBFN, from 31.1% to 42.2%. Besides, reporting the pLDDT-ranked augmentation numbers, instead of only the oracle ones, also counts in the paper's favour.

Critical weaknesses
  1. The novelty of the unification claim is not separated from the single-sequence case. The first contribution is stated as "a structured multi-task masked-diffusion framework that unifies evolutionary representation learning and generation," and the abstract goes further, calling it "a principled paradigm unifying evolutionary representation and protein design." But to my knowledge, the fact that masked discrete diffusion can unify representation learning and generation for protein language models was already established in the single-sequence modeling by [1]. What I want to know is the delta. Does moving this unification to MSAs actually change how representation and generation relate, because rows and columns bring in dependencies that single sequences don't have? Or does the MSA setting mostly inherit the result? The paper should discuss this directly.

  2. Structured masking is not ablated on generation. The task-removal checkpoints are only evaluated on representation benchmarks. Comparing against other generators cannot separate the objective from architecture, decoding, and candidate selection. Please run the existing full, uniform, and drop-one variants on MSA augmentation and enzyme design under matched depth, sampling budget, and selection rule.

  3. The augmentation gain is not significant and tangled up with a selector only DMP gets. Under pLDDT selection DMP reaches 0.3268 TM, which is below both the shallow-MSA baseline (0.3466) and MSAGPT (0.3490), and its oracle 0.4178 barely beats MSAGPT's 0.4157. The NLL selector lifts DMP to 0.3581, but no competing generator is scored with it, so generator quality and ranking quality does not get separated.

[1] Diffusion Language Models Are Versatile Protein Learners.

Questions for authors

Is DMP initialized from a pretrained MSA Pairformer checkpoint or trained from scratch, and how do the training budgets compare?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 482W (1/3)

Thank you for your thoughtful feedback and suggestions, they will be invaluable in helping us improve our work. We address the raised concerns as follows:

W1. Contribution beyond single-sequence mask discrete diffusion.

We appreciate your suggestion to clarify the scope of our contribution. We will carefully limit our descriptions of contribution and novelty to the MSA-based model. Our unification is the first application of structured multi-task masked discrete diffusion to the MSA data format, and we emphasize that the evolutionary representation learning here refers specifically to learning based on MSA structure. We will revise the abstract and introduction to clarify our descriptions of MSAs and co-evolutionary representations.

We agree with you that the delta comes from dependencies between rows and columns in MSAs. That is precisely why we proposed structured multi-task masking beyond uniform masking. In single-sequence protein language models, uniform token masking suffices because each position depends only on other positions in the same sequence. MSAs, however, encode structured evolutionary information across two dimensions: (1) homologous sequences (rows) sharing evolutionary ancestry, and (2) aligned residue positions (columns) under functional constraints. Our complete ablation study demonstrates that structured masking is necessary: uniform masking alone achieves inferior performance on both representation (Figure 3) and generation tasks (Tables A–D below), while removing individual structured tasks degrades performance, confirming that each task captures complementary dependencies specific to MSA organization.

Thank you for raising this important point. We will further refine the description in the paper to more clearly convey our contribution and insights regarding diffusion modeling on the MSA modality.

AuthorsOpenReview ↗

Response to Reviewer 482W (2/3)

W2. Structured masking ablations on generation.

This is an excellent suggestion to further separate the contribution of multi-task. Due to training time limitations, we conducted experiments using the LmaxL_{\mathrm{max}}=512, NmaxN_{\mathrm{max}}=32 MSA with identical sampling budgets and selection rules across five settings: Full (baseline), Uniform only, w/o query, w/o row, and w/o column. Tables A–D present the results.

Table A. MSA-augmentation ablation on Hard52, pLDDT top-1 selector. Stars mark oracle-TM or selected-TM values significantly below Full: * padj<0.05p_{\mathrm{adj}}<0.05.

ConfigurationpLDDT top-1 (ColabFold)Oracle (ColabFold)pLDDT top-1 (ESMFold2)Oracle (ESMFold2)
Full0.31380.41690.44300.5217
w/o query0.30820.41240.43050.5163
w/o row0.29950.40700.42860.5082*
w/o column0.31120.41580.44170.5182
Uniform0.28600.3876*0.44020.5107

Table B. Enzyme-generation ablation on CLEAN sequence-level outcomes. DMP generates 100 sequences per case. Asterisks mark values significantly below Full in two-sided paired tests: * padj<0.05p_{\mathrm{adj}}<0.05, ** padj<0.01p_{\mathrm{adj}}<0.01, *** padj<0.001p_{\mathrm{adj}}<0.001.

ConfigurationAny-ECAll-ECPairwise identityUnique
Full0.93800.91600.55981.0000
w/o query0.5013***0.4227***0.26491.0000
w/o row0.93530.92330.58751.0000
w/o column0.93670.91000.53661.0000
Uniform0.0000***0.0000***0.06211.0000

Table C. Enzyme-generation ablation on protein-only task with each case having 100 predictions.

ConfigurationpLDDTProtein TMAny-ECAll-ECPairwise identityUnique
Full92.280.94631.00000.96830.57211.0000
w/o query69.930.79540.37000.30670.26631.0000
w/o row92.970.94760.99830.99670.60591.0000
w/o column91.970.94311.00000.94500.56221.0000
Uniform30.880.28050.00000.00000.06221.0000

Table D. Enzyme-generation ablation on with-ligand task with 100 sequences per case.

ConfigurationSuccess (%)LRMSD (Å)lDDT-PLIlDDT-LPTM
Full42.225.16680.62800.67340.9125
w/o query4.259.79990.33900.40510.7588
w/o row40.725.19240.62350.67390.9215
w/o column41.655.04360.62480.67010.9045
Uniform0.0019.44010.00930.10100.2828

The results demonstrate task-specific contributions aligned with generation applications:

  • MSA augmentation: Removing row masking reduces ESMFold2 oracle TM from 0.5217 to 0.5082 , as row masking directly models homolog generation.
  • Enzyme design: Removing query masking causes severe degradation—CLEAN All-EC drops from 0.916 to 0.423, protein TM from 0.946 to 0.795, and FoldBench success from 42.2% to 4.2%.
  • Uniform masking: Fails at both tasks—0% EC recovery and substantially lower oracle TM.

These ablation results will be incorporated into Section 5.3 (Generation) of the revised manuscript, demonstrating that structured masking tasks provide generation contributions beyond uniform token diffusion.

AuthorsOpenReview ↗

Response to Reviewer 482W (3/3)

W3. MSA augmentation of generator quality versus selector quality.

Thank you for pointing out this omission. We conceptualize MSA generation performance as comprising two components: generator quality, measured by oracle TM-score (the best achievable performance across all candidates without selector influence), and ranking quality, measured by how well selectors identify high-quality candidates using pLDDT or other confidence metrics. As shown in Table 1 of the main paper and Table E below, our oracle TM-scores demonstrate generator quality that surpasses other models.

The reason we explore alternatives to pLDDT as a universal selector is that for MSA augmentation methods like ProfileBFN and DMP, increasing MSA depth induces pLDDT inflation unrelated to quality improvement, degrading pLDDT's ranking ability. This creates an unfair benchmark comparison. We therefore developed alternative selector methods to enable fairer cross-model comparison.

We conducted two comparison experiments across models:

  • Shared DMP-NLL: We use DMP as a unified NLL calculation method for all models' generated MSAs. This yields significant improvement only for DMP itself, likely due to distribution differences and model preferences in the generated MSAs.
  • Own meanP: Each model evaluates its own generated MSAs using forward-pass mean probability combined with pLDDT as a ranking score, providing a fairer demonstration of selector effectiveness based on model confidence plus prediction confidence. DMP shows the most significant improvement under both protocols (Table E).

Table E. Selector comparison on Hard52. Values are mean TM-score. Selector stars indicate significantly higher TM than same-pool pLDDT top-1: * padj<0.05p_{\mathrm{adj}}<0.05, ** padj<0.01p_{\mathrm{adj}}<0.01, *** padj<0.001p_{\mathrm{adj}}<0.001.

GeneratorpLDDT top-1 TMShared DMP-NLL TMOwn meanP TMOracle TM
ColabFold
ProfileBFN0.30230.30140.30330.3805
MSAGPT0.34900.34340.35300.4157
DMP0.32680.3581**0.3591*0.4178
ESMFold2
ProfileBFN0.39760.40130.40060.4826
MSAGPT0.45720.45420.45800.5207
DMP0.44400.46060.46000.5312

DMP demonstrates superior generator quality across competing models. Most critically, DMP achieves the highest oracle TM-scores on ColabFold and ESMFold2, surpassing MSAGPT and ProfileBFN. This oracle performance directly measures generator quality independent of any selection strategy.

DMP also achieves competitive ranking performance with various selectors. With the support of both selector protocols, DMP effectively improves the poor performance under the pLDDT top-1 selector and achieves similar ranking performance with MSAGPT, demonstrating that DMP achieves competitive practical performance in MSA augmentation. These selector comparison results will be incorporated into Section 5.4 (Generation) of the revised manuscript.

Q1.

Our DMP is initialized from pretrained MSA Pairformer and trained with the same training data of MSA Pairformer. This training paradigm is consistent with DPLM and ProfileBFN, which makes diffusion post-training more stable and efficient.

These tables and other details of experiment setup will be included in a revised manuscript.

References

[1] Wang et al., "Diffusion Language Models Are Versatile Protein Learners," ICML, 2024, pp. 52309–52333.

[2] Alamdari et al., "Protein generation with evolutionary diffusion: sequence is all you need," bioRxiv, 2023, doi:10.1101/2023.09.11.556673.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Summary of Revisions

We thank the reviewers for their constructive feedback. We summarize their main concerns below and indicate where we have addressed them in the revised manuscript, with further details in our individual replies.

Main Concerns and Revisions

1. Novelty and contribution scope (Reviewers 482W, QzTm). Does DMP merely apply single-sequence masked diffusion to MSAs, or address a fundamental challenge unique to MSAs? We clarified that MSAs organize evolutionary information across two structural dimensions—rows (homologs) and columns (positions)—creating dependencies that uniform masking cannot capture. Our structured decomposition mirrors this organization, validated through ablations showing task-specific effects. We revised the abstract (lines 15-28) and introduction (lines 45-91).

2. Generation task ablations (Reviewers 482W, CCbN, QzTm). Task-removal checkpoints were only evaluated on representation. Do these tasks support their proposed generation roles? We evaluated all five ablation configurations on MSA augmentation and enzyme design. Results show diagonal effectiveness: removing row masking reduces MSA augmentation oracle TM, while removing query masking collapses enzyme All-EC accuracy from 0.916 to 0.423. Uniform masking achieves 0% EC recovery. We added generation ablation results to the main text (lines 400-425), expanded Figure 3 showing representation and generation side-by-side, and provide complete tables in new Appendix E.2.

3. Generator quality versus selector design (Reviewers 482W, CCbN, QzTm). DMP's conditional-NLL selector conflates generation quality with ranking quality, making cross-method comparison unfair. We now separate these explicitly. Oracle TM isolates generator quality: DMP achieves 0.418 (ColabFold) and 0.531 (ESMFold2), exceeding MSAGPT and ProfileBFN. For fair ranking, we introduced two alternative selectors applicable to all generators: Shared DMP-NLL and Own meanP. Under these selectors, DMP's selected TM improves substantially. We revised Section 5.3 (lines 340-365) to distinguish generator versus selector quality, added Appendix C.3 detailing the Own meanP selector with formulation and algorithm, and provide complete comparison tables in Appendix C.4.

4. Statistical significance and practical impact (Reviewers QzTm, CCbN). Are representation improvements broadly distributed or outlier-driven? All improvements are statistically significant with narrow confidence intervals: ProteinGym +0.0158 Spearman (p<0.001, 95% CI [0.0105, 0.0213]), PPI +0.017 P@K (p=0.032), CASP15 +0.0571 P@L (p<0.001). Case-level analysis confirms broad distribution: 131/211 ProteinGym cases improve, 22/35 PPI, 38/57 CASP15. New Appendix B.1 reports paired tests, confidence intervals, and case-level distributions. Main text tables now include significance markers throughout.

5. Enzyme benchmark generalizability (Reviewer QzTm). The original 15-family enzyme evaluation is too small to establish generalizability. We expanded to 94 multi-EC families from Swiss-Prot using consistent filtering criteria. DMP achieves 97.4% Any-EC and 51.5% All-EC across all 94 families, exceeding ProfileBFN (96.2%, 50.9%) and PoET (95.8%, 49.1%). From these, 26 families were selected for structure evaluation. New Appendix D.5 documents the expanded benchmark with three result tables spanning sequence evaluation (94 families), protein-only structure (26 families), and ligand-aware co-folding (6 families).

6. Depth matching and baseline comparison (Reviewer CCbN). Structured-versus-uniform ablation uses depth 32, but MSA Pairformer comparison uses full depth. Does this conflate diffusion benefits with depth effects? The depth-32 ablation isolates task decomposition effects under matched conditions. The full-depth comparison demonstrates practical utility. Training budgets are matched: DMP initializes from baseline checkpoint, consistent with DPLM and ProfileBFN. Implementation details appear in Appendix A (lines 567-670), and ablation protocols with matched hyperparameters in Appendix B.2 (lines 773-831).


We believe these revisions address all substantive concerns while preserving the paper's core contributions. The structured decomposition framework is now positioned as a necessary response to MSA's architectural properties, validated through comprehensive ablation spanning both representation and generation tasks. We thank the reviewers for their constructive feedback and hope further disccusion.

Dear reviewers,

Please be aware that little time remains for the author-reviewer discussion. As the deadline is approaching, we kindly request that you refrain from asking for additional experiments. Please summarize your final position concisely.

Sincerely, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.