acceptodds
Under review as a conference paper at ICLR 2027

Exploring the Alignment of Generation and Understanding in Protein Structure Modeling

Abstract

Understanding and generation are often treated as two separate paradigms in training deep neural networks, despite the fact that both are trained with closely related objectives such as denoising and masked prediction. While prior studies have shown that generative models often learn suboptimal representations for understanding tasks in vision, it is less understood whether a similar gap exists in the protein domain. In this work, we systematically investigate this question by benchmarking state-of-the-art protein generative models on widely-used protein understanding tasks, and observe that these models exhibit consistently poor performance compared to existing protein encoders. Furthermore, inspired by the Representation Alignment (REPA) framework, we propose to explicitly align generative protein diffusion models with pretrained protein understanding models during training. Experiments on the MotifBench demonstrate that representation alignment significantly improves functional protein generation, boosting the MotifBench score of Protpardelle-1c from 39.2 to 47.1, corresponding to a 20% relative improvement. Our results suggest that representation alignment provides a general and effective mechanism for bridging understanding and generation in protein structure modeling. Model weights and code are available at https://anonymous.4open.science/r/reapro-AE43/

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EuiBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors ask whether protein generative models learn representations useful for understanding tasks, and whether aligning a generator to such a model improves what it generates. The authors first probe two structure tokenizers (La-Proteina VAE, Kanzi VQ-VAE) and two diffusion models (Protpardelle-1c, RFDiffusion) on EC and GO classification, and find all four behind ESM2 and ProteinMPNN. Citing that gap and the REPA framework from vision, they add an alignment loss to Protpardelle-1c: at layer 5 of its U-ViT trunk an MLP projects the hidden state onto the embedding a frozen encoder assigns to the clean structure, under a squared-L2 loss at weight λ = 2. The head is discarded at inference, so sampling is unchanged. The resulting model, ReaPro-1c, raises the MotifBench score from 39.2 to 47.1, improves on the baseline on both the backbone and all-atom RFDiffusion benchmarks, and lowers Fréchet Protein Distance (FPD) against CATH 4.4-S40.

Critical strengths
  • The question is under-explored in proteins, and Section 3 stands on its own as an analysis contribution.
  • The main gains are on MotifBench, and hold across two further settings on the RFDiffusion benchmark. Both score unique successes after clustering, so they are not explained by collapse onto a few easily designable folds.
  • The method is cheap: one loss term, no architecture change, no inference cost.
Critical weaknesses
  • Unconditional generation is assessed only by FPD in ProteinMPNN embedding space, the space the model is trained to match, with no designability, diversity or novelty.
  • Of three alignment layers tested, only layer 5 improves on the baseline, which can harm the generalization of proposed method.
Questions for authors
  1. Why does the baseline appear as both 39.2 and 39.4, and was it evaluated under the same best-of-three-epoch selection and eight-sequence protocol?
  2. Can you report designability, diversity and novelty for unconditional generation, and FPD in an embedding space other than ProteinMPNN's?
  3. As a robustness check, can you run motif scaffolding once with an inverse-folding model outside the MPNN family?
  4. What is FA-MPNN's score, and why are the main results reported with ProteinMPNN?
  5. Does the gain transfer to a second base model? That is what "a general and effective mechanism" would require.
1 comment

Thank you for the suggestions on independent evaluation and the scope of the method's generality.

Q1

The baseline MotifBench score is 39.24, which rounds to 39.2. We are sorry for the reporting error of 39.4. We will correct it.

The baseline and ReaPro-1c use the same sampling settings and eight-sequence evaluation protocol. For the baseline, we use the released checkpoint trained for 416 epochs, whose original checkpoint-selection procedure was not explicitly reported. For ReaPro-1c, we selected among epochs 399, 400, and 401 using one sequence per structure before running the eight-sequence evaluation.

We now report eight-sequence results for all three ReaPro-1c checkpoints: 46.08, 45.56, and 47.08, with a mean of 46.24 ± 0.77. Each checkpoint exceeds the baseline, supporting consistency beyond the selected final checkpoint.

Q2

We have added the following evaluations under the same length-matched sampling setup:

ModelDesignability ↑\uparrowDiversity ↑\uparrowFPD (ESM3) ↓\downarrowFPD (ProtDomainSegmentor) ↓\downarrow
Protpardelle-1c64.45%13.84%50.585.86
ReaPro-1c73.48%12.13%39.255.77

Relative to the baseline, designability improves by 14.0%, while diversity decreases by 12.4%. The lower FPD in both ESM3 and ProtDomainSegmentor [1] embedding spaces provides evidence beyond the ProteinMPNN space used for alignment.

Q3

We repeat motif scaffolding with ESM-IF for sequence design, using the same ProteinMPNN-aligned ReaPro-1c model.

ModelMotifBench score (ESM-IF) ↑\uparrow
Protpardelle-1c23.88
ReaPro-1c27.44

Although the absolute scores are lower than with ProteinMPNN sequence design, ReaPro-1c retains a 14.9% relative improvement. This result supports robustness to an inverse-folding model outside the MPNN family.

Q4

Aligning ReaPro-1c with FA-MPNN yields a MotifBench score of 48.77. Our framework is model-agnostic and can use representations from different pretrained protein understanding models for alignment. Our main question is which types of representations benefit protein structure generation, with a particular focus on the difference between sequence-based and structure-based representations.

The FA-MPNN result provides further evidence that structure-aware understanding models can improve structure generation through representation alignment. The comparison with ESM2 also suggests that functional classification performance alone does not determine the most useful representations for generation; the modality gap between sequence representations and 3D structures matters as well.

Among models within the same modality that yield comparable generation performance, their relative ranking is secondary to this broader question. We therefore use ProteinMPNN-aligned ReaPro-1c for the main results and report the higher FA-MPNN score in the ablation on align model choice.

Q5

We agree that evaluating representation alignment on additional base models would strengthen the evidence for its generality. Due to limited computational resources, we leave this extension to future work.

[1] Miao, Yangyang, et al. “Leveraging protein representations to explore uncharted fold spaces with generative models.” bioRxiv (2025).

Reviewer_bRufOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between protein understanding and generation by examining whether protein generative models learn representations that are useful for downstream functional prediction. The authors first compare representations from protein generative models, structure tokenizers, and pretrained protein understanding models on Enzyme Commission (EC) and Gene Ontology (GO) classification tasks. The results indicate that the evaluated generative models generally underperform dedicated protein encoders. Motivated by this observation, the authors introduce ReaPro-1c, which incorporates a representation alignment objective into the Protpardelle-1c diffusion model. Intermediate representations of the generative model are aligned with those from frozen protein understanding models, including ProteinMPNN and ESM2, through an auxiliary projection head and alignment loss. Experiments cover unconditional protein generation, motif scaffolding, and all-atom generation. The proposed method improves the reported MotifBench score from 39.2 to 47.1 and shows improved structural distribution coverage. Ablation studies investigate the choice of understanding model, alignment strength, and alignment layer.

Critical strengths
  1. An interesting connection between protein understanding and generation.

The paper addresses a relevant question in biomolecular machine learning: whether pretrained protein understanding models can provide useful representations for improving protein generation. The adaptation of representation alignment to protein structure diffusion is conceptually straightforward and potentially useful. The comparison between sequence-based and structure-based alignment teachers also provides insight into the importance of representation modality.

  1. Promising performance improvements across multiple generation settings.

The reported improvement on MotifBench is substantial, increasing from 39.2 to 47.1. The authors additionally evaluate unconditional structure generation, backbone motif scaffolding, and all-atom generation. The ablation studies examining teacher choice, alignment strength, and alignment layers provide supporting evidence that the observed improvements are related to representation alignment.

Critical weaknesses
  1. Evaluation independence and training-budget comparability are insufficiently established.

The manuscript describes using MotifBench performance to select checkpoints, while MotifBench is also used to evaluate alignment configurations and report the main results. It is unclear whether model selection and final evaluation are sufficiently separated. If the same benchmark targets were used for both purposes, the reported improvements could be affected by benchmark-specific tuning.

Furthermore, the manuscript emphasizes faster convergence and reduced training requirements, but the relationship between the main results and the different training budgets is not sufficiently clear. Comparisons between aligned and unaligned models at matched optimization budgets would strengthen the evidence for improved training efficiency.

  1. The representation-gap analysis and teacher-dependent evaluation do not fully isolate the proposed mechanism.

The initial comparison between generative and understanding models involves architectures with substantially different training objectives, input modalities, model capacities, and training data. Consequently, the observed differences in functional classification performance cannot be attributed solely to generative versus discriminative learning objectives.

In addition, ProteinMPNN serves as a representation teacher and is also used in the downstream sequence-design pipeline, while unconditional generation is evaluated using ProteinMPNN-derived embeddings. Although these choices follow established evaluation practices, additional independent evaluations would help establish whether the improvements reflect broader generative quality rather than increased compatibility with the teacher's representation space.

Questions for authors
  1. Evaluation independence and model selection.

Were the alignment hyperparameters and checkpoints selected using validation targets independent of those used for the final MotifBench evaluation? Could the authors clarify the checkpoint-selection procedure and, if the same benchmark targets were used, provide an independently selected evaluation?

  1. Training efficiency and fair comparison.

Can the authors provide a direct comparison between aligned and unaligned models under identical training budgets, checkpoint-selection criteria, and computational resources? In particular, how do the results at 200 epochs compare with the final reported results, and which checkpoints produced the MotifBench score of 47.1?

  1. Representation alignment and independent evaluation.

Does the performance improvement persist when using evaluation procedures independent of the alignment teacher? Additionally, can the authors clarify how differences in input modality, model capacity, and training data affect their conclusion that generative models learn inferior representations for protein understanding tasks?

2 comments

Thank you for highlighting evaluation independence, training-budget comparability, and the scope of the representation analysis.

Q1

Using the same generation metric for checkpoint selection and final reporting is common in diffusion models, as with FID in image generation. We likewise use MotifBench performance for both purposes, since the denoising loss does not directly measure the quality of generated designs.

For the main model, we first evaluated epochs 399, 400, and 401 with one sequence per generated structure, then evaluated the selected checkpoint with eight sequences per structure. Alignment hyperparameters were also compared on MotifBench, so selection and final evaluation used the same motif targets.

We understand the reviewer's concern about potential overfitting of metrics in the selection process. To assess how much the reported improvement depends on checkpoint selection, we now report eight-sequence MotifBench results for all three final checkpoints:

ModelMotifBench score ↑\uparrow
Protpardelle-1c39.24
ReaPro-1c (epoch 399)46.08
ReaPro-1c (epoch 400)45.56
ReaPro-1c (epoch 401)47.08
ReaPro-1c (mean ± sample SD)46.24 ± 0.77

All three exceed the baseline score of 39.24, and the mean represents a 17.8% relative improvement. This supports consistency across the final checkpoints. These results assess checkpoint sensitivity; they do not constitute a held-out MotifBench evaluation.

The backbone RFDiffusion benchmark also evaluates the same checkpoint selected on MotifBench, without selecting it again on that benchmark. ReaPro-1c outperforms Protpardelle-1c on 22 of 26 motifs in total successes and 17 of 26 motifs in unique successes, providing additional evidence of transfer across benchmarks.

Q2

The reported MotifBench score of 47.1 is rounded from 47.08, obtained by the ProteinMPNN-aligned model at epoch 401 under the eight-sequence evaluation protocol. The released Protpardelle-1c baseline checkpoint was trained for 416 epochs; its checkpoint-selection procedure was not explicitly reported.

The 200-epoch experiments use the same selection procedure as the final model: we evaluate checkpoints at epochs 199, 200, and 201 with one sequence per generated structure, then run the full eight-sequence evaluation for the selected checkpoint. The comparison at roughly half the baseline's training duration applies to the separate early-training experiments and the all-atom setting, where our model is trained for 201 epochs and the released baseline for 383 epochs.

We use the same training dataset and base training configuration as Protpardelle-1c. Relative to the baseline, the additional computation consists of one-time preprocessing of the teacher embeddings and the alignment head with its auxiliary loss during training. The embeddings are precomputed and reused, so the frozen teacher requires no forward pass within each training step. The alignment head is lightweight and adds only a small overhead to training time. We will clarify these additional costs in the training description.

Q3

We have added evaluation using models outside the MPNN family, including ProtDomainSegmentor [1]:

ModelMotifBench score (ESM-IF) ↑\uparrowFPD (ESM3) ↓\downarrowFPD (ProtDomainSegmentor) ↓\downarrow
Protpardelle-1c23.8850.585.86
ReaPro-1c27.4439.255.77

These results show that the improvement persists when the sequence-design model and evaluation embedding space are independent of the alignment teacher. The teacher and projection head are used only during training and are discarded for sampling. The generated structures are therefore obtained without teacher guidance at inference.

Regarding the representation analysis, the evaluated pretrained models differ in input modality, architecture, capacity, and training data. Our main question is whether a functional-representation gap exists between current protein generation and understanding models. Explaining why the gap arises, including how modality, model capacity, training data, and training objectives contribute to functional-representation quality, is an interesting question for a further controlled study. We will expand the discussion to include these questions and identify a controlled analysis of these factors as future work.

Thank you to the authors for the response—most of my concerns have been addressed, and I still believe my current score is appropriate and will not make further changes.

Reviewer_sLLtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ReaPro-1c, a model based on Protpardelle-1c but trained with its internal representation aligned to an external protein understanding model. The paper demonstrates that representation alignment in protein generative model training helps to improve model capability in motif scaffolding. The paper also performs many ablation experiments to search for the optimal strategy of aligning the representations.

Critical strengths
  1. Good idea of aligning the internal representation of a protein generative model with an external model's representation.
  2. Comprehensive ablation study.
Critical weaknesses
  1. Presentation of this paper could be improved: the title is missing from manuscript; no explicitly listed contributions; methods precede background and related work; identical figure captions.
  2. Questionable choice of baseline: La-Proteina's VAE encodes mostly side-chain information. See Question section for details.
  3. Evaluation is not comprehensive: see questions for detail.
Questions for authors
  1. In Table 1, the authors compared the representations extracted from different models to show that diffusion model representations are much worse than "understanding" model representations for downstream classification tasks. One thing I want to point out is that the VAE representation of La-Proteina most encodes side-chain structural information conditioned on CαC_{\alpha} coordinate. It contains much less global structure context than ProteinMPNN's representation does, and has only 8 channels. If the authors want to use La-Proteina's representation, why not take the representation from its joint denoiser layers for a fairer comparison? Also, the VAE latent space of La-Proteina has 8 channels, but the number of channels mentioned in this paper is 16. Can the authors elaborate on how they extracted representations from La-Proteina?

  2. For unconditional generation tasks, related work normally report metrics such as designability (co-designability for all-atom protein generative models) and diversity calculated using Foldseek. Can the authors also report these metrics for ReaPro-1c and compare them with ProtPardelle-1c?

  3. Did the authors train ReaPro-1c from scratch with auxiliary representation alignment, or fine-tune ProtPardelle-1c from the published checkpoint? Appendix C.3 reads as training from scratch, but could the authors please confirm?

  4. Which model's representation is the main ReaPro-1c aligned to? I assume it's ProteinMPNN, but can the authors clarify this and state it explicitly in the paper. If the main model is aligned to ProteinMPNN, why not use the FA-MPNN-aligned checkpoint, given its superior MotifBench score shown in Figure 7?

  5. Comment: Panels (A) and (B) of Figure 3 have the same description.

  6. The representation ReaPro-1c aligned to, including ProteinMPNN and ESM2, are either sequence-based model or backbone-structure based models. Does ReaPro-1c have improved side-chain generation capability compared to Protpardelle-1c?

  7. The ProteinMPNN representation is shown to be worse than ESM in classification tasks in Table 1. What might explain why it improves the ReaPro-1c model more than ESM does after alignment training?

2 comments

Thank you for the suggestions on representation comparison, generation evaluation, and presentation. We address each point below.

Q1

We included La-Proteina's VAE alongside Kanzi in the tokenizer comparison to examine whether representations that support accurate structural reconstruction also capture functional information. As you point out, La-Proteina's VAE primarily encodes side-chain information conditioned on the backbone, so its latent alone provides limited global structural context.

Following your suggestion, we additionally evaluate intermediate representations from La-Proteina's joint denoiser:

ModelEC ↑\uparrowGO-MF ↑\uparrowGO-BP ↑\uparrowGO-CC ↑\uparrow
La-Proteina joint denoiser0.37700.30840.30610.3530
ProteinMPNN0.45280.42300.33750.4089
ESM20.63760.61170.43170.4821

The denoiser representations perform better than the VAE latents, while remaining below ProteinMPNN and ESM2 on all four tasks. This extends the observed functional-representation gap beyond the side-chain tokenizer.

For the VAE evaluation, we use the per-residue posterior mean returned by the encoder, with shape L×8L \times 8. The stated dimension of 16 in Appendix B.1 is a typographical error, which we will correct to 8.

Q2

We have added these metrics under the same length-matched sampling setup for both models:

ModelDesignability ↑\uparrowDiversity ↑\uparrow
Protpardelle-1c64.45%13.84%
ReaPro-1c73.48%12.13%

Relative to the baseline, ReaPro-1c improves designability by 14.0%, with a slight decrease in diversity. These results show a designability–diversity tradeoff and complement the original distribution-coverage analysis. We also report all-atom co-designability in our response to Question 6.

Q3

ReaPro-1c is trained from scratch with the diffusion objective and auxiliary representation-alignment loss. We use the same base architecture, training dataset, and training configuration as Protpardelle-1c. Code and model weights are provided for reproducing the results. We are sorry for the potential misunderstanding for our presentation, we will make the initialization procedure explicit in the training description.

Q4

The main ReaPro-1c model aligns its intermediate representations with those of ProteinMPNN. Aligning with FA-MPNN yields a MotifBench score of 48.77.

We use ProteinMPNN for representation alignment in the main experiments and compare ESM2 and FA-MPNN in the ablation on alignment target. Our central question is whether semantic representations from pretrained protein understanding models can improve generation, and which representations are most useful for structure generation. The FA-MPNN result provides further evidence that structure-aware understanding models are effective for representation alignment. We will state explicitly which understanding model is used in the main experiments and report the FA-MPNN score in the ablation.

Q5

We are sorry for the duplicated description. In Figure 3, panel A shows structures from regions where Protpardelle-1c has higher sampling density, while panel B shows structures from regions where ReaPro-1c has higher sampling density. We will correct the caption accordingly.

Q6

We have added an all-atom co-designability evaluation:

ModelCo-designability ↑\uparrowDiversity ↑\uparrow
Protpardelle-1c44.52%27.95%
ReaPro-1c60.98%27.45%

Relative to the baseline, co-designability improves by 37.0%, while diversity decreases by 1.8%. This provides evidence that the benefit of alignment extends to joint sequence–structure generation in the all-atom setting, complementing the original backbone-based evaluation.

Q7

Functional classification performance alone does not determine how useful a representation is for protein structure generation. As discussed in Section 6.1, we hypothesize that the modality gap between sequence representations and 3D structures helps explain why ProteinMPNN improves generation more than ESM2. ESM2 learns from sequences, whereas ProteinMPNN operates directly on structural coordinates.

To examine this hypothesis, we also align ReaPro-1c with FA-MPNN, another structure encoder, and obtain a MotifBench score of 48.77. This result supports the importance of matching modalities between the pretrained understanding model and the generation task. The ablation therefore suggests that both semantic information and the modality gap affect the benefit of representation alignment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General reply and more evaluation results.

We thank the reviewers for their thoughtful feedback. We have added the following evaluations to further examine the effects of representation alignment on protein generation. In the tables, ↑\uparrow indicates higher is better and ↓\downarrow indicates lower is better.

1. Functional representations from La-Proteina's joint denoiser. We extend the EC and GO evaluation to intermediate representations from La-Proteina's joint denoiser. The Fmax scores are:

ModelEC ↑\uparrowGO-MF ↑\uparrowGO-BP ↑\uparrowGO-CC ↑\uparrow
La-Proteina joint denoiser0.37700.30840.30610.3530
ProteinMPNN0.45280.42300.33750.4089
ESM20.63760.61170.43170.4821

These scores remain below those of ESM2 and ProteinMPNN on all four tasks, extending the observed functional-representation gap to La-Proteina's joint denoiser.

2. Additional evaluation of unconditional generation. Under the same length-matched sampling setup, we report designability, diversity, and FPD computed using ESM3 and ProtDomainSegmentor [1], two encoders outside the MPNN family:

ModelDesignability ↑\uparrowDiversity ↑\uparrowFPD (ESM3) ↓\downarrowFPD (ProtDomainSegmentor) ↓\downarrow
Protpardelle-1c64.45%13.84%50.585.86
ReaPro-1c73.48%12.13%39.255.77

Designability improves by 14.0%, while diversity decreases by 12.4%, relative to the baseline. FPD is lower with both additional encoders, providing evidence of improved distributional coverage beyond the ProteinMPNN embedding space.

3. All-atom co-designability. We add co-designability and diversity measurements for the all-atom models:

ModelCo-designability ↑\uparrowDiversity ↑\uparrow
Protpardelle-1c44.52%27.95%
ReaPro-1c60.98%27.45%

Co-designability improves by 37.0%, with a 1.8% decrease in diversity, relative to the baseline. This supports improved joint sequence–structure designability in the all-atom setting.

4. Consistency across final checkpoints. We report MotifBench results for all three final checkpoints:

ModelMotifBench score ↑\uparrow
Protpardelle-1c39.24
ReaPro-1c (epoch 399)46.08
ReaPro-1c (epoch 400)45.56
ReaPro-1c (epoch 401)47.08
ReaPro-1c (mean ± sample SD)46.24 ± 0.77

The standard deviation is computed across the three checkpoints. All three exceed the baseline score, and their mean gives a 17.8% relative improvement, showing that the improvement persists across the final checkpoints.

5. Motif scaffolding with ESM-IF. We repeat motif scaffolding using ESM-IF for sequence design:

ModelMotifBench score (ESM-IF) ↑\uparrow
Protpardelle-1c23.88
ReaPro-1c27.44

ReaPro-1c achieves a 14.9% relative improvement. The advantage therefore persists with an inverse-folding model outside the MPNN family.

We also clarify the following points raised across the reviews.

6. Choice of understanding model. The main ReaPro-1c model uses ProteinMPNN for representation alignment, while alignment with FA-MPNN yields a MotifBench score of 48.77. Our framework is model-agnostic, and the main question is which types of representations benefit structure generation, particularly sequence-based versus structure-based representations. Among models within the same modality that yield comparable generation performance, their relative ranking is secondary to this question. We therefore use ProteinMPNN-aligned ReaPro-1c for the main results and report the higher FA-MPNN score in the ablation on alignment target.

7. Checkpoint selection and evaluation protocol. The main model is trained from scratch. We select among epochs 399, 400, and 401 using one sequence per generated structure, then evaluate the selected checkpoint with eight sequences per structure. Selection and final evaluation use the same MotifBench targets; the additional results for all three checkpoints assess consistency across this final training window. The baseline uses the released checkpoint trained for 416 epochs, whose checkpoint-selection procedure was not explicitly reported. Both models use the same sampling settings and eight-sequence evaluation protocol. The exact scores are 39.24 for the baseline and 47.08 for the selected ReaPro-1c checkpoint, rounded to 39.2 and 47.1.

8. Representation alignment and independent evaluation. Representation alignment is used during training: the frozen pretrained understanding model provides the target embeddings, and the alignment head is discarded during inference. Sampling therefore incurs no additional cost from alignment and uses no additional guidance from the understanding model. The ESM-IF sequence-design evaluation and FPD computed using ESM3 and ProtDomainSegmentor assess generation with models different from ProteinMPNN, providing additional evidence beyond the embedding space used for alignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.