acceptodds
Withdrawn from ICLR 2027

PROGEA: Unifying Protein Sequence, Structure, and Language with Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We envision a general-purpose protein foundation model as a unified, interactive model of sequence, structure, and function, capable of understanding, generation, and cross-modal translation while retaining the instruction-following and reasoning capabilities of modern LLMs. Toward this vision, we introduce **PROGEA**, the first native, unified, multimodal diffusion language model (dLLM) for proteins. **PROGEA** is built on two key design choices: (1) a unified discrete token space that jointly represents sequence, structure, and natural language; and (2) a dLLM backbone that replaces strict left-to-right factorization with bidirectional or semi-bidirectional token interactions, enabling global residue communication and long-range contact modeling that are naturally suited to structure modeling. This formulation unifies six core capabilities: folding, inverse folding, sequence understanding, structure understanding, sequence design, and structure design, while retaining the native language capabilities of general-purpose LLMs. On established benchmarks, **PROGEA** achieves broadly competitive performance. In folding, PROGEA-MDLM improves backbone TM-score over existing tri-modal models by 0.30/0.16 on CAMEO22/PDB split. For sequence understanding, PROGEA-BDLM closely matches the best evaluated frontier LLM, Claude Opus 4.7 (62.37%). These results provide strong evidence for effective sequence–structure alignment, extending the sequence–language representation space established in prior work into the 3D physical space of proteins.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Foundation Models, Diffusion Language Models, Unified Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jUAzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents PROGEA, a unified diffusion language model that integrates protein sequences, discretized backbone structures, and natural language through a shared token vocabulary. The authors compare MDLM and BDLM across cross-modal tasks, providing an empirical assessment of their respective strengths in structural modeling, sequence design, and protein-domain language interaction.

Critical strengths
  1. The native integration of sequence, structure, and language within a shared dLLM backbone is a valuable systems contribution. The MDLM-BDLM comparison provides evidence of task-dependent trade-offs between fully noncausal and blockwise generation.

  2. PROGEA-MDLM achieves substantially better folding metrics than the reevaluated trimodal AR baseline, while PROGEA-BDLM attains GeneralQA accuracy numerically comparable to the evaluated strong general-purpose LLMs.

Critical weaknesses
  1. Multi-task coverage does not establish positive transfer. The experiments lack controlled single-task, bimodal, and trimodal comparisons and task or modality removal ablations. Without matching the backbone, target-task data exposure, and training budget, the marginal benefit of joint training remains unidentified. The reported gains therefore cannot be disentangled from additional supervision or regularization effects.

  2. The contribution of diffusion-based generation remains confounded. The MDLM-BDLM comparison and attention analysis support the proposed mechanism, but do not establish that diffusion-based generation itself explains the improvements over the AR baseline.

  3. The incremental methodological contribution of fusion is unclear. Token-level early fusion and staged modality alignment are described, but their novelty and independent performance contributions relative to prior shared-vocabulary models remain unresolved. Figure 4 provides descriptive evidence about representation geometry, rather than causal evidence of task-level gains.

  4. The incremental utility of a unified trimodal system remains insufficiently validated. Evaluation primarily covers single-step mappings between modality pairs. Tasks jointly constrained by sequence, structure, and text are underexplored, and multi-stage generation lacks systematic evaluation of cross-modal consistency, error accumulation, compositional generalization, and end-to-end success.

  5. The framework figure (Figure 2) has limitations in both scientific accuracy and methodological clarity. Several regions in the protein illustrations appear structurally implausible, and the folding landscape risks conflating conditional backbone prediction with physical folding dynamics. The figure also insufficiently highlights the framework’s distinctive features, particularly the attention visibility patterns and denoising procedures that distinguish MDLM from BDLM. More accurate structural illustrations and explicit depictions of these mechanisms would better communicate the framework’s technical contributions.

Questions for authors
  1. Does joint training yield positive transfer beyond multi-task coverage? Please provide controlled single-task, bimodal, and trimodal comparisons with matched backbone, target-task supervision, and training compute, together with task or modality removal ablations.

  2. Which components constitute the incremental contribution over prior shared-vocabulary trimodal models, and what controlled ablations isolate their effects on downstream performance?

  3. Could the authors extend the structure prediction evaluation to CASP14 and CASP15 alongside the reported CAMEO22 results? Comparisons with established structure prediction methods on these benchmarks would provide a more comprehensive assessment of the model’s structural prediction capability.

  4. Given that PROGEA’s sequence-text supervision and GeneralQA both derive from Swiss-Prot, could the authors clarify the train-test decontamination procedures and provide the complete evaluation prompts, specifying any information supplied beyond the sequence and question?

  5. For the Claude and GPT baselines listed in Table 3, could the authors clarify the prompting strategy, reasoning and output token budgets, decoding settings, and whether tool use or external retrieval was enabled?

Reviewer_wChDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PROGEA, a unified protein diffusion language model that represents protein sequences, backbone structures, and natural language within a shared discrete token space. The framework builds on a pretrained language model and employs a multi-stage training procedure to support six sequence–structure–language tasks. The authors investigate two variants, masked diffusion (MDLM) and block diffusion (BDLM), and evaluate their performance on protein folding, inverse folding, understanding, and design benchmarks. The results show substantial improvements in folding over a prior unified tri-modal model, while the block diffusion variant achieves stronger performance on protein understanding and text-conditioned sequence generation.

Critical strengths
  1. A unified framework for diverse protein modeling tasks. The paper presents a coherent approach to modeling protein sequences, structures, and natural language within a single generative architecture.

  2. Substantial improvements in folding over prior unified tri-modal models. PROGEA-MDLM achieves backbone TM-scores of 0.80 and 0.84 on CAMEO 2022 and the PDB date split, respectively, compared with 0.50 and 0.68 for the re-evaluated BioMatrix baseline.

Critical weaknesses
  1. The practical benefit of a unified model is not clearly demonstrated.

While combining six protein modeling tasks within a single model is interesting, I am not convinced that this provides a clear advantage over existing specialized models. In practice, researchers often focus on a specific task, such as folding, inverse folding, or function-guided protein design. The paper does not demonstrate a use case where combining multiple capabilities within one model leads to better results or a more effective workflow.

  1. The contribution of the diffusion formulation is not sufficiently isolated.

The proposed method largely builds on existing components, including Qwen3, the DPLM2 structure tokenizer, and diffusion language modeling. Although the MDLM–BDLM comparison is useful, I do not think it is sufficient to attribute the folding improvements to non-causal modeling. The comparison with BioMatrix involves different training setups, and it is unclear how much of the improvement comes from the architecture itself. A controlled comparison with an autoregressive baseline using the same backbone and training data would be helpful.

  1. PROGEA does not consistently match the strongest existing models on individual tasks.

Despite improvements over the prior unified BioMatrix model, PROGEA still falls behind existing models on several reported metrics. For example, DPLM2 performs better on several folding and inverse folding metrics, while AMix2 achieves better results on multiple sequence design metrics. Pinal also reports higher foldability in text-conditioned sequence design, although the sampling settings differ.

These results make it difficult to identify a clear performance advantage over existing methods. The paper would be more convincing if it demonstrated either stronger results on individual tasks or measurable benefits from jointly training multiple tasks.

Questions for authors
  1. Could the authors clarify how overlap between the multimodal training corpus and the folding evaluation sets is controlled, including sequence homology? This is particularly relevant given the use of large-scale sequence–structure paired data during fine-tuning.

  2. Could the authors provide a quantitative evaluation of text-to-backbone generation across a larger set of functional conditions? It would also be helpful to demonstrate whether interleaved generation offers measurable benefits over independently applying the corresponding tasks.

  3. Does training all six tasks jointly actually improve performance on any individual task? I would like to see a comparison with models trained on individual tasks using the same backbone and comparable training budgets. This would help establish whether the unified training brings any benefit beyond supporting multiple tasks in one model.

  4. MDLM performs better on folding, while BDLM performs considerably better on protein understanding and sequence design. Could the authors explain why the two variants behave so differently?

  5. Given the substantial differences in sequence generation success rates, could the authors compare model quality on a common subset of successfully generated prompts? It would also be helpful to report results under matched candidate sampling budgets, particularly for the comparison with Pinal.

Reviewer_kHHfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PROGEA, a unified diffusion language model that extends a Qwen3-4B backbone with protein-sequence and quantized-structure tokens and trains masked-diffusion and block-diffusion variants to handle folding, inverse folding, understanding, and design within a single model. The paper tackles an important integration problem with an auditable, multi-task evaluation, but its novelty framing, the strength of its capability claims, and several comparison protocols need to be tightened.

Critical strengths
  • This paper is easy to follow.

  • The paper investigates whether sequence, structure, and biological language can genuinely share one model rather than being bridged by separate systems.

  • This paper evaluates both fully masked diffusion and block diffusion.

  • The evaluation spans six task directions and includes both specialized protein models and unified multimodal baselines.

Critical weaknesses
  • The novelty and the technical contributions of this paper are limited. The main technical novelty appears to be the combination of an existing language-model backbone, an existing structure tokenizer, diffusion training, and a staged multimodal training recipe.

  • It's important to attribute the folding gains specifically to the diffusion formulation rather than to the full training/evaluation recipe.

  • The experiments section can be further enhanced. For example, in structure understanding, either re-run baselines under matched input modalities/procedures or explicitly flag which rankings are cross-procedure and therefore indicative only. The ablation studies can also be improved.

Questions for authors

see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.