acceptodds
Under review as a conference paper at ICLR 2027

Stroma: A Unified Benchmark for Motif Scaffolding Across Protein Generative Paradigms

Abstract

Motif scaffolding, the design of a protein that preserves a predefined motif while generating the surrounding scaffold, is central to protein engineering. Models that perform this task have advanced rapidly across three generative paradigms: backbone, all-atom and sequence generation. Existing benchmarks were built alongside particular model families, assume incompatible input representations, and cover a narrow range of motifs, so models from different paradigms cannot be compared on equal terms. We introduce Stroma, a unified, extensible benchmark of 100 motif scaffolding tasks, each specified at up to five motif resolutions at once, from residue identity to catalytic side-chain geometry, so that every paradigm receives the same task rather than a translation of it. Nearly a quarter are new, covering protein families no prior benchmark included, among them membrane proteins, nanobodies and nucleic-acid-binding motifs. Where an architecture supports them, we evaluate two conditioning modes (indexed and unindexed) and three motif representations (all-atom, tip-atom and theozyme). Success is defined by explicit thresholds that a single predicted structure must satisfy jointly; we test whether rankings survive re-folding under different structure predictors. We benchmark 12 models (15 checkpoints) spanning all three paradigms. Even though all-atom and backbone models perform similarly on average, only all-atom models can take on the more demanding tasks, such as theozyme scaffolding, which backbone models cannot represent. Under minimal-harness conditioning, protein language models underperform both. Thirteen tasks are solved by no model, pointing to opportunities for further work in motif scaffolding.

Primary area:
datasets and benchmarks
Keywords:
motif scaffolding, protein design, benchmark, generative models, diffusion models, protein language models, proteins structure prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mKcSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Stroma, a benchmark of 100 protein motif scaffolding tasks. It combines 77 existing tasks with 23 new ones and compares 12 models that generate backbones, all-atom structures, or sequences. The evaluation covers motif preservation, predicted structure quality, diversity, and novelty. RFdiffusion3 leads on the 89 tasks used for the main comparison. Backbone and all-atom models perform similarly on average, while sequence models perform worse.

Critical strengths

This paper addresses an important problem. There is still no single strong benchmark for comparing motif scaffolding models across different model families. I appreciate that the authors unite together many (100) tasks and evaluate a broad set of models. Aslo mirroing motifs analysis looks interesting. The paper is clearly written and have well-made illustration.

Critical weaknesses

Main concerns

  1. The benchmark brings together tasks from existing benchmarks. In my view, aggregation alone is not a sufficiently novel contribution for a conference paper. I suggest a more careful review of the task set itself. Some tasks may be very similar, while others may be difficult to pass under the chosen evaluation protocol. Please assess redundancy and feasibility, and explain how the final set was selected to increase diversity, task coverage.
  2. Some models cannot attempt all 100 tasks, so the main comparison uses only 89 tasks. This leaves the original comparability problem partly unresolved. The paper aims to address the fact that different models are tested on different tasks and metrics, yet some task types still cannot be evaluated across all the models in this benchmark.
  3. The motivation for the 23 new tasks needs more detail. The paper explains which broad gaps they fill, but it is less clear why these particular proteins and motifs were chosen. Please describe the selection criteria and explain why these examples were preferred over other candidates.
  4. Enzymes still account for 51 of the 100 tasks. This seems at odds with the claim that the benchmark reduces bias toward any one task type. Please justify this imbalance.
  5. I am concerned that some tasks may not have a positive control under the evaluation protocol. The reported median Cα RMSD of 2.57 Å for native proteins raises this question, although it does not answer it directly. Please use the native sequences and structures as positive controls under the same prediction and success criteria applied to generated designs. How many native proteins pass all the filters? If a native protein fails, it is hard to tell whether a model failed or the validation pipeline failed. This question is especially important for multi-PDB tasks, whose motifs come from different source structures. What evidence shows that these combined tasks are feasible?
Questions for authors

Additional comments and questions

  1. Why were two inverse-folded sequences and two structure predictions used per design? Please justify these choices or show how sensitive the results are to them.
  2. Seems that Geommotif benchmark [1] can be usefull complement to your direction. Its goals are similar to those of Stroma, and it seems relevant to the choice and coverage of motif scaffolding tasks.
  3. I would like to see more analysis by functional and biochemical category. Do any models perform especially well on particular types of tasks? It would also help to group motifs by properties such as secondary structure and compare model performance within those groups.
  4. Success rate measures how often a model produces a passing design, while the Stroma score also rewards distinct solutions. Please report a simple, separate measure of diversity among successful designs. For example, this could be the number of distinct structural clusters at a fixed number of successful samples. This would make the contributions of quality and diversity easier to see for each model.
  5. Please report novelty relative to the source protein from which each motif was taken, in addition to novelty against reference databases.
  6. The analysis of GPU time per successful design is interesting. I would also like a comparison under the same total compute budget for every model. A plot showing the number of tasks with at least one successful design as a function of GPU time would be helpful. Please include the time spent on inverse folding and structure prediction, and report the validation cost separately.
  7. The cutoff of six consecutive identical residues seems arbitrary. The continuous complexity analysis in the supplement is useful, but it does not fully explain why this particular cutoff was chosen. Could you justify the arguments for this parameter?
  8. For ligand-bearing tasks, which models receive ligand information during generation or inverse folding? I understand that Boltz2 receives ligand-related information during prediction. If a generator does not see the ligand, please explain how the no-clash criterion should be interpreted for that model.
  9. Could some of the difference between all-atom and backbone models be related to inverse folding with LigandMPNN? Where possible, please compare the performance of all-atom models evaluated with LigandMPNN-generated sequences and with their own generated sequences. Please discuss how this choice might affect the comparison.

[1] Strashnov, P., Shevtsov, A., Meshchaninov, V., Kardymon, O., and Vetrov, D. P. (2026). GeomMotif: A Benchmark for Arbitrary Geometric Preservation in Protein Generation. International Conference on Learning Representations (ICLR).

3 comments

We thank the reviewer for their thorough and very insightful comments. It is highly appreciated, and addressing all concerns has helped us improve the manuscript enormously.

We address the points below:

Weaknesses

  1. We have added 60 new tasks ( response to reviewer DLVT) and 3 evaluations to the benchmark to incorporate novel functionalities, totalling 160, 149 of which amenable to all models. In addition, we provide a detailed analysis of why models fail on specific tasks, and make recommendations on architecture and training dataset improvements based on the results of the benchmark ( response to reviewer 61b3). We have now assessed feasibility and task redundancy. Of 4 950 task pairs, only 10 pairs are redundant (analysis now detailed in SI). For feasibility, 153 of 160 new tasks have at least one passing design. The final 23 tasks stratum set was selected to increase functional and geometrical coverage based on these criteria: high-quality data available in the literature, absence of this category in other benchmarks, and unexpected results for these tasks in our first version of the benchmark.

  2. Thanks for pointing this out. We are aware that the 11 tasks that span several chains cannot be attempted by some models. We included them because, in our view, it is important to see how the multi-chain capable models perform on them for the subset that can, even if it came at the cost of losing homogeneity.

  3. The criteria are (i) it belongs to a functional class absent from previous benchmarks (ii) its function is assigned to specific residues in the primary literature; (iii) an experimental structure resolves every motif residue; and (iv) the motif can be specified at all five resolutions of the benchmark. We have made our best effort to create a framework where adding new tasks is straightforward for users, as shown now with the addition of 60 more tasks.

  4. We expanded the original task set to reduce that imbalance. Still, about a quarter to a third of all proteins in an organism are enzymes, and we try to mirror that distribution.

  5. We have added a feasibility audit in Supplementary D.1 along with positive controls for all native scaffolds. A feasibility assessment has been performed and detailed in the SI; all 7 tasks that are not solved by models are thoroughly assessed for biological plausibility.

Questions:

  1. We varied M and K from 1 to 8 (now in SI). The mean success rate rises with more samples (0.32 at 2×2, 0.40 at 4×4, 0.47 at 8×8), but the ranking of tasks by difficulty barely changes (Spearman ρ ≥ 0.99 against 2×2 on the backbone criterion, ≥ 0.94 on the heavy-atom criterion). We kept M = K = 2 as the best trade-off between compute and accuracy; users can raise both through the configuration without changing the code.

  2. We adapted some of the analysis introduced in GeomMotif (now cited in Related work) and we now find similar results: helix-rich motifs are easier, β-rich and buried motifs are harder, and success falls with fragment count (Section 5; Supplementary D.10).

  3. We have added a broad category comparison to check the model's performance on different categories. Ranking depends more on segment number than protein type. We also analyzed results based on motif properties and present the full analysis along with the protein category analysis.

  4. Very neat suggestion. We implemented d@k measure that does exactly what is suggested. Results show high diversity in designs from structural models.

  5. We now report novelty relative to the native sequence together with database novelty and d@k metric mentioned above.

  6. Two new figures on GPU time have been added to the manuscript based on this suggestion, and the validation time is now reported separately.

  7. In the supplementary section, we added cut-off calibration to detail how we set that threshold using complexity analysis from a sample of UniProtDB.

  8. Of the generators, RFdiffusion2, RFdiffusion3, RFD4 (added upon release), ODesign and Proteina-Complexa see the ligand during design, and LigandMPNN also receives it for RFdiffusion2, RFdiffusion3 and ODesign; for every other model the ligand first appears when Boltz-2 co-folds it, so the identical no-clash check means "the model designed a pocket" for ligand-aware generators but only "the predictor could fit the ligand" for ligand-blind ones, a difference we now state in the Methods.

  9. We tested this in two ways (Supplementary D.9). Refolding the same designs with the other sequence source makes no significant difference for the models that write their own sequences (La Proteina 25.0% → 29.0%, Protpardelle-1C 18.5% → 19.0%, RFD4 37.1% → 33.5%; all p ≥ 0.08, Supplementary D.9), and all-atom models score about the same with their own sequences (18.7%) as with LigandMPNN sequences (17.2%), so the gap between all-atom and backbone models is not an artefact of LigandMPNN.

I thank the authors for their responses. Several of my questions have been addressed, including those concerning LigandMPNN, the use of two structure predictions per design, functional categories, diversity, and the threshold of six consecutive identical residues. However, my main concerns remain, so i would like to keep my score.

The benchmark still lacks a clear conceptual motivation or a novel underlying idea. It combines existing benchmarks and adds new tasks, which is certainly useful to the community. However, this is currently too straightforward an extension to constitute a sufficient contribution for publication on its own. As a result, the benchmark does not offer a new perspective on the models; it mainly ranks them using established evaluation principles. The motif-mirroring analysis is perhaps the only genuinely new insight, but it seems more like an insight that could have emerged from analysing existing benchmarks than a result enabled by creating a new one.

Moreover, the authors’ redundancy and feasibility analyses indicate that redundant and potentially infeasible tasks remain in the benchmark. Thus, in addition to the lack of a new conceptual contribution, technical concerns remain unresolved.

The mentioned redundancy analysis seems to concern only the original 100 tasks, since (as i understand 100 × 99 / 2 = 4950), whereas the feasibility analysis covers all 160 tasks. The authors state: “Of 4 950 task pairs, only 10 pairs are redundant (analysis now detailed in SI). For feasibility, 153 of 160 new tasks have at least one passing design.” Simply reporting these counts is insufficient. Also, the author response also does not explain how redundancy was determined or how the seven unsolved tasks were assessed for feasibility.

My question about novelty concerned structural novelty for structure-generating models: sequences can differ substantially while adopting essentially the same structure. Novelty relative to the native sequence therefore does not fully address my request for comparison with the source protein’s structure.

Technical comments:

  1. The revised manuscript is not available, i therefore cannot verify the results reported in the rebuttal that refer to newly added sections.
  2. Some author responses refer me to replies to other reviewers without identifying the relevant point. This makes the author rebuttal harder to follow and assess.

Thank you for the response and for giving us the chance to discuss during the revision period; it is highly appreciated. We are grateful you find that several of the previous concerns were addressed.

Regarding the concern about novelty, we realize the previous answer pointed entirely to our answer to Reviewer 1, and it made it hard to review what we actually meant. We now lay out our response here. Indeed, our manuscript focused extensively on the technical details of the benchmark and not on the actual purpose behind building it. Our major motivation was to identify the limits of what the state-of-the-art models are capable of performing, characterize where they fail, and provide the community with clear next steps on how to build the next generation of frontier models for motif scaffolding. In this sense, we identified two failure modes: 1) mirroring of the original motifs, as the reviewer points out, and 2) all models currently fail on tasks with motifs with >4 segments (<6% success, except RFD4 at 15%). We analyze why the second instance happens, and observe two main reasons. First: models that receive the motif only as pairwise distances are among the first to fail. A good example of this is illustrated by Odesign’s 2 checkpoints, Rigid and Flex, two modes of the same model: Rigid fixes the motif’s 3D coordinates; Flex only receives its pairwise distances. For motifs with multiple segments, Rigid performs significantly better (better on 33 vs 9 tasks, p < 0.001). Second, failures mostly happen after refolding, which implies that models are not capable of co-optimizing residue identity and backbone coordinates under motif constraints when building a scaffold.

Observing this failure mode, we offer the community recommendations on how to train the next generation of models. To showcase this, we have therefore post-trained RFD4 to avoid failures after re-folding. Specifically, we have post-trained RFD4 using reinforcement learning (direct preference optimization, DPO), using refolding outcomes as a training signal. Using 1,000 new motifs from AlphaFold DB (4–7 segments, excluding all Stroma source proteins and their homologs), we have paired RFD4 designs that pass after Boltz-2 refolding with designs for the same motif that fail, and aligned RFD4 on these pairs using DPO. By doing so, the overall Stroma score increases from 38.4% to 42.8%, with improvements on 51 tasks and decreases on 14. For motifs with ≥4 segments, the score increases from 29.6% to 33.8%, and for the held-out synthetic motifs, it increases from 3.3% to 9.0%. The improvement was reproduced in a second training run and we also observed it when designs were refolded with OpenFold3 instead of Boltz-2. We did not observe designs becoming simpler or less diverse. We will make the model weights available to the community in the camera-ready version. We believe post-training strategies such as the one applied here for RFD4 open a new alternative to improve performance in other models as well.

Regarding redundancy and feasibility, thank you for analysing our data with such scrutiny. Due to space constraints before, we had not commented appropriately in this regard.

Redundancy: For the 160 motifs, we superimpose the 12,720 pairs. Out of those, 7 presented an RMSD < 1 Å and can be considered redundant, with 3 corresponding to submotifs of the same motif (RMSD = 0), from the original MotifBench benchmark. Regarding feasibility, thank you for pointing out the discrepancy in numbers. Indeed, we had only reported it for the original benchmark; now we have extended it to the entire dataset. Now, we show that 147/160 tasks have passing designs, with 11/13 failing tasks missing the 1-angstrom criterion by 0.5 angstroms or less and all of them having structures predicted with pLDDT > 70. All of these hard tasks are multi-segment motifs and 8 have n > 4 segments. Three of these hard tasks show an RMSD to GT with native structure prediction of more than 1 angstrom and we agree they might not be solvable under the 1-angstrom criterion. We think nonetheless that they are interesting to include because of their difficulty. These hard tasks that fail by a small margin with the best models are the ones we think the community can use to assess new frontier models.

Indeed, we have not included the PDF with our changes. Assuming ICLR was like NeurIPS and ICML (where the upload is not possible), we failed to upload it in time. We are working to have it as soon as possible.

Thank you again for revising our manuscript; it is highly appreciated.

Reviewer_DLVTOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Stroma is a unified benchmark for AI protein motif scaffolding. It emphasizes a generalized paradigm of five resolutions to enable uniform benchmarking of models across different atomistic design choices. This work introduced new protein families and evaluated under multiple structure predictors.

Critical strengths
  • Extremely thorough benchmarking. This is really rigorous work that takes a lot of time and resources and is quite valuable -Clear well defined and useful unification of important motif scaffolding task
Critical weaknesses

Overall this work is interesting but its not all that novel as it essentially takes known prior benchmarks runs them together and adds 26 new tasks. I think this type of benchmarking is really important but beyond the unification there are not really new findings.

  • Many of the key take aways can be seen in prior work on the smaller subset like idx vs unindex and task vs model -Folding models play a big role it would be very valuable to have this be a central focus as new structure prediction models come out so often and each behave quite differently and it would be important to understand how various models that use different aspects of synthetic data impact the results
  • No inutions on why certain models fail or the design choices that cause it. Benchmark makes sense but what do I use this for beyond trying to optimize for it.
Questions for authors
  • How does Proteina-Complexa for AME and RFD4-Proteina compare on this unifed benchmark?
  • Is it possible to share the detailed results for all folding models included AlphaFold2 for the non AME tasks
  • Which tasks are nanobody and membrane protein focused and is there a dedicated section discussion how models perform there
1 comment

We thank the reviewer for their careful assessment and for acknowledging the time and effort that went into this thorough comparison. We address all points below:

Insights from the data. We agree that the submitted manuscript focused on the technical details of the benchmark and left out why we built it in firs place: to find where SOTA models succeed and fail under the same conditions, and to give guidelines on how to train the next generation of models to solve the tasks that remain unsolved.

The hardest tasks are motifs with more than 4 segments: every model stays below 6% success except RFD4 (15%). We asked why some models perform better than others. Seeing many-segment motifs during training does not explain it (ρ = 0.04 across 12 models). Protpardelle is trained mostly on such motifs, yet on motifs with 4 or more segments it keeps only 21% of its success on 1-2 segment motifs. La-Proteina, never trained on more than 4 segments, keeps 38%, and RFD4 60%. Both generate a Cα trace plus a compact per-residue latent, which we think simplifies the problem: 3 coordinates per residue instead of ~30 for an all-atom model. New models therefore benefit from: (i) motif residues entering as fixed 3D coordinates rather than pairwise distances (as in the Genie family), (ii) building the coarse Cα trace and the atomic details separately, and (iii) training on many-segment functional motifs with both contiguous and non-contiguous spans, which helps reach hard tasks but does not replace (i) and (ii).

Synthetic data and new tasks. Along with 3 new evaluations (including motif dynamics and pocket accessibility), we have added 60 tasks: 48 synthetic catalytic motifs, to test generalization to idealized, unseen geometries, and 12 membrane protein motifs, to ask whether models can satisfy both the motif geometry and the sequence constraints of the lipid bilayer. The synthetic motifs separate two abilities. When the motif is 3-5 isolated catalytic residues built from idealized chemistry, RFD4 reaches 37% success, as in natural enzyme sites (32%). When each catalytic residue sits on a rigid 4-8 residue backbone fragment, in an arrangement not found in nature, it drops to 3%. We find models place isolated residues well but cannot yet join rigid fragments in unseen arrangements.

Questions:

  1. We use the Proteina-Complexa AME checkpoint and have added RFD4, released during the review period. Proteina-Complexa supports only unindexed, single-chain motifs and was trained on small motifs (1-8 residues in 1-4 segments). This explains its results: in unindexed all-atom mode, it solves none of the 59 tasks outside that range, and all 11 tasks it solves lie inside it. On the 91 tasks both models attempt in unindexed mode, RFD4 solves 80 (31.0% mean success) and Proteina-Complexa 11 (1.7%). Over all 100 tasks, RFD4 reaches a Stroma score of 62.0 in unindexed all-atom mode, while Proteina-Complexa reaches 4.7:
ModelModeStroma scoreTasks solvedClusters
RFD4unindexed all-atom62.0832,425
RFD4indexed all-atom60.6772,572
RFD4unindexed tip-atom33.9551,232
RFD4indexed tip-atom33.1511,320
RFD4unindexed theozyme6.415221
RFD4indexed theozyme6.211225
Complexaunindexed tip-atom15.428333
Complexaunindexed theozyme6.210185
Complexaunindexed all-atom4.79132
  1. Success rate (%) on the 12 non-AME tasks of our predictor-comparison subset, 10 designs per task and model (motif RMSD < 1 Å, scRMSD < 2 Å, pLDDT > 70):
ModelAF2AF3Boltz-2OF3AF2 +MSAAF3 +MSABoltz-2 +MSAOF3 +MSA
Genie216.717.525.017.521.723.329.224.2
Genie315.021.730.818.323.325.832.526.7
La-Proteina20.027.530.025.833.336.735.035.0
ODesign-Flexn/a13.310.811.79.213.313.310.0
ODesign-Rigidn/a8.312.58.313.315.016.715.8
Protpardelle-1C13.316.723.314.220.022.522.524.2
RFdiffusion17.523.330.821.725.027.532.530.0
RFdiffusion211.712.514.211.715.020.016.716.7
RFdiffusion313.322.526.720.022.525.830.026.7
All models15.4 (7 models)18.122.716.620.423.325.423.2
  1. The first version had 2 membrane (GPCR) and 2 nanobody tasks. The membrane tasks are among the easiest, while the nanobody tasks are almost unsolved: only RFD4 (1%) and La-Proteina (5%) solve one of them. Per-task results are now in the supplement. We have now also added 12 membrane tasks and 3 specific evaluations for membrane motifs: the free energy of the lipid-facing surface, the depth of the motif (whether the protein would span the membrane), and an implicit bilayer fit of each design.
Reviewer_61b3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript proposes a new benchmark set of motif scaffolding problems, largely derived from existing benchmark sets but also introducing some new benchmark problems that are more diverse in nature (about 25% of total). This benchmark suite is then used to assess a broad range of protein design models.

Critical strengths
  • A new benchmark set is moderately useful, especially with an expanded and more diverse set of problems.
  • Independent assessment of a broad range of protein design models is useful as well, as self-benchmarking by various groups often exaggerates their preferred model's performance.
Critical weaknesses
  • Contribution is ultimately somewhat minor, especially in the age of LLM-derived science. Most benchmark problems come from other papers, and there aren't really any new ideas or techniques.
Questions for authors

The claim that Boltz-2 significantly outperforms AlphaFold3 and OpenFold3 on experimental structures (very large delta of 2.57A vs. 9.73A and 12.88A, respectively) seems highly suspect. Are the authors sure this is not representing data leakage because Boltz-2 is trained on a more recent dataset than AlphaFold3 and OpenFold3?

4 comments

We thank the reviewer for the careful reading, appreciating the clarity of the manuscript and for highlighting the usefulness of this independent assessment. We address all points below:

Novelty: We are grateful for the reviewer's comment, as it made us realize our manuscript focused on the technical details of the benchmark and not on the actual purpose behind building it. Our major motivation was to identify the limits of what the state-of-the-art models are capable of performing, characterize where they fail, and provide the community clear next steps on how to build the next generation of frontier models for motif scaffolding. To this end, we needed a modular and technically rigorous benchmark that fairly evaluates different model families (pLMs, diffusion all-atom and backbone) and modalities (unindexed, indexed, tip atom) across a functionally diverse set of tasks. This was not possible with current benchmarks, which, while providing excellent insights, did fall short in one or more of the categories for a broad comparison.

In our analysis, we have identified two failure modes for current models: 1) mirroring of the original motifs occurs for models conditioned on distances alone (16% of Genie2 designs), and 2) all models currently fail on tasks with motifs with >4 segments (<6% success, except RFD4 at 15%).

In this regard, we investigated the mechanisms of failure. Models that receive the motif only as pairwise distances are among the first to fail. A good example of this is illustrated by Odesign's 2 checkpoints Rigid and Flex, two modes of the same model: Rigid fixes the motif's 3D coordinates, Flex only receives its pairwise distances. For motifs with multiple segments, Rigid performs significantly better (better on 33 vs 9 tasks, p < 0.001), as Flex keeps the motif within 1 Å in only 26% of designs for >3 segments.

We also find that the representation matters more than how often models see fragmented motifs in training. Across 12 models, the share of training motifs with ≥4 segments does not predict how much success a model keeps on them (ρ = 0.04). Protpardelle trains mostly on such motifs (82%) yet keeps 21% of its 1-2 segment success at ≥4 segments, while La-Proteina, which never sees 5+ segments, keeps 38%. RFD4 (added upon publication), which like La-Proteina generates Cα coordinates with a per-residue latent, keeps 60% and outperforms Protpardelle (37.0% vs 20.4%, p < 10^-12). What fails for multi-segment motifs is the arrangement of these segments relative to each other. Interestingly, failures mostly happen after refolding: for n_segments > 3, 8 of 10 models hold the motif at ≤0.5 Å in the design but at 2.4-4.9 Å after refolding. This implies that models aren't capable of co-optimizing residue identity and backbone coordinates under motif constraints when building a scaffold.

We therefore make the following recommendations for the training of new models: (i) having the motif residues enter as fixed in 3D coordinates rather than using pairwise distances (genie model family) (ii) build the coarse C-alpha and atomic details separately (iii) add many segments functional motifs in training with a mix of contiguous and non-contiguous spans, which helps reach hard tasks but does not replace (i)-(ii).

In addition, we added in new version 60 new motif tasks and 3 new evaluations: 48 synthetic catalytic site motifs and 12 membrane protein motifs. We now also evaluate motif flexibility change through short MD simulation and check whether catalytic and binding sites are not only held in their position in the designs but also accessible to their ligands. We also added membrane protein-specific evaluations.

Data leakage concern: We thank the reviewer for their comment. Indeed, such a big gap between the structure predictors calls for scrutiny. We emphasize that the numbers reported in the manuscript refer to the predictions without MSA and without templates. The reason is that for many designs we find no meaningful MSAs. We therefore evaluated the model in the low-homology regime they would have to work in.Of the 60 tasks with less than 300 residues, 52 were released before September 2021 (AF3 training set cut-off date). If we restrict the analysis to these 52 PDBs pre-dating both Boltz2's and AF3's training data set cut-off date, the numbers barely change for no-MSA predictions: 2.57 Å, 7.34 Å and 11.61 Å for Boltz-2, AlphaFold3 and OpenFold3. The reviewer's question does, however, expose a limitation we had not stated. Because nearly all source structures are in every model's training set, this control measures how well each predictor reproduces known structures from a single sequence. To further motivate our choice of using Boltz2 as a predictor, we ran the 60 native experimental structure predictions with MSAs for all models. We obtained median Cα RMSDs of 0.90 Å for Boltz2, 1.34 Å for AlphaFold3 and 1.30 Å for OpenFold3.

The new insights the authors describe are interesting and I commend them for the detailed study. I am going to keep my score as-is however as the totality of the work is still a little too light for a major conference like ICLR. If the authors had incorporated their recommendations into a model and shown that it raises performance, then that would've cleared the bar in my opinion of an ICLR paper.

Thank you for this clear and prompt suggestion! This is a good idea, and we can certainly do something along these lines. We are continuing to train RFD4-Proteina from its latest released checkpoint to test two of our recommendations: (a) more exposure to fragmented motifs (4–8 segments) during training, and (b) fine-tuning on the model’s own designs that keep the motif after refolding, which is the other failure point our analysis identified.

We hope to have results within 24h-48h (currently training running on 64 GH200 GPUs). We will evaluate both on the Stroma tasks with >=4 segments and on the held-out synthetic motifs. We are, of course, excluding all Stroma source proteins and their homologs from the fine-tuning data. We will report as soon as we have them.

We have now done the experiment. Following your suggestion, we incorporated one of our recommendations into RFD4.

We have observed an increase in performance. Specifically, we post-trained RFD4 using reinforcement learning (direct preference optimization, DPO) to improve motif preservation after refolding. One of our main findings was that multi-segment motifs are often not preserved after refolding, so we used refolding outcomes as a training signal. Using 1,000 new motifs from AlphaFold DB (4–7 segments, excluding all Stroma source proteins and their homologs), we paired RFD4 designs that pass after Boltz-2 refolding with designs for the same motif that fail, and aligned RFD4 on these pairs using DPO.

The overall Stroma score increased from 38.4% to 42.8%, with improvements on 51 tasks and decreases on 14. For motifs with ≥4 segments, the score increased from 29.6% to 33.8%, and for the held-out synthetic motifs, it increased from 3.3% to 9.0%. The improvement was reproduced in a second training run and we also observed it when designs were refolded with OpenFold3 instead of Boltz-2. We did not observe designs becoming simpler or less diverse.

We believe these results provide direct evidence that incorporating refolding outcomes into post-training can improve motif-scaffolding performance, and it could be a data-efficient strategy for the improvement of other models. We are releasing the model weights to the community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.