- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces Stroma, a benchmark of 100 protein motif scaffolding tasks. It combines 77 existing tasks with 23 new ones and compares 12 models that generate backbones, all-atom structures, or sequences. The evaluation covers motif preservation, predicted structure quality, diversity, and novelty. RFdiffusion3 leads on the 89 tasks used for the main comparison. Backbone and all-atom models perform similarly on average, while sequence models perform worse.
Critical strengths
This paper addresses an important problem. There is still no single strong benchmark for comparing motif scaffolding models across different model families. I appreciate that the authors unite together many (100) tasks and evaluate a broad set of models. Aslo mirroing motifs analysis looks interesting. The paper is clearly written and have well-made illustration.
Critical weaknesses
Main concerns
- The benchmark brings together tasks from existing benchmarks. In my view, aggregation alone is not a sufficiently novel contribution for a conference paper. I suggest a more careful review of the task set itself. Some tasks may be very similar, while others may be difficult to pass under the chosen evaluation protocol. Please assess redundancy and feasibility, and explain how the final set was selected to increase diversity, task coverage.
- Some models cannot attempt all 100 tasks, so the main comparison uses only 89 tasks. This leaves the original comparability problem partly unresolved. The paper aims to address the fact that different models are tested on different tasks and metrics, yet some task types still cannot be evaluated across all the models in this benchmark.
- The motivation for the 23 new tasks needs more detail. The paper explains which broad gaps they fill, but it is less clear why these particular proteins and motifs were chosen. Please describe the selection criteria and explain why these examples were preferred over other candidates.
- Enzymes still account for 51 of the 100 tasks. This seems at odds with the claim that the benchmark reduces bias toward any one task type. Please justify this imbalance.
- I am concerned that some tasks may not have a positive control under the evaluation protocol. The reported median Cα RMSD of 2.57 Å for native proteins raises this question, although it does not answer it directly. Please use the native sequences and structures as positive controls under the same prediction and success criteria applied to generated designs. How many native proteins pass all the filters? If a native protein fails, it is hard to tell whether a model failed or the validation pipeline failed. This question is especially important for multi-PDB tasks, whose motifs come from different source structures. What evidence shows that these combined tasks are feasible?
Questions for authors
Additional comments and questions
- Why were two inverse-folded sequences and two structure predictions used per design? Please justify these choices or show how sensitive the results are to them.
- Seems that Geommotif benchmark [1] can be usefull complement to your direction. Its goals are similar to those of Stroma, and it seems relevant to the choice and coverage of motif scaffolding tasks.
- I would like to see more analysis by functional and biochemical category. Do any models perform especially well on particular types of tasks? It would also help to group motifs by properties such as secondary structure and compare model performance within those groups.
- Success rate measures how often a model produces a passing design, while the Stroma score also rewards distinct solutions. Please report a simple, separate measure of diversity among successful designs. For example, this could be the number of distinct structural clusters at a fixed number of successful samples. This would make the contributions of quality and diversity easier to see for each model.
- Please report novelty relative to the source protein from which each motif was taken, in addition to novelty against reference databases.
- The analysis of GPU time per successful design is interesting. I would also like a comparison under the same total compute budget for every model. A plot showing the number of tasks with at least one successful design as a function of GPU time would be helpful. Please include the time spent on inverse folding and structure prediction, and report the validation cost separately.
- The cutoff of six consecutive identical residues seems arbitrary. The continuous complexity analysis in the supplement is useful, but it does not fully explain why this particular cutoff was chosen. Could you justify the arguments for this parameter?
- For ligand-bearing tasks, which models receive ligand information during generation or inverse folding? I understand that Boltz2 receives ligand-related information during prediction. If a generator does not see the ligand, please explain how the no-clash criterion should be interpreted for that model.
- Could some of the difference between all-atom and backbone models be related to inverse folding with LigandMPNN? Where possible, please compare the performance of all-atom models evaluated with LigandMPNN-generated sequences and with their own generated sequences. Please discuss how this choice might affect the comparison.
[1] Strashnov, P., Shevtsov, A., Meshchaninov, V., Kardymon, O., and Vetrov, D. P. (2026). GeomMotif: A Benchmark for Arbitrary Geometric Preservation in Protein Generation. International Conference on Learning Representations (ICLR).
3 comments
I thank the authors for their responses. Several of my questions have been addressed, including those concerning LigandMPNN, the use of two structure predictions per design, functional categories, diversity, and the threshold of six consecutive identical residues. However, my main concerns remain, so i would like to keep my score.
The benchmark still lacks a clear conceptual motivation or a novel underlying idea. It combines existing benchmarks and adds new tasks, which is certainly useful to the community. However, this is currently too straightforward an extension to constitute a sufficient contribution for publication on its own. As a result, the benchmark does not offer a new perspective on the models; it mainly ranks them using established evaluation principles. The motif-mirroring analysis is perhaps the only genuinely new insight, but it seems more like an insight that could have emerged from analysing existing benchmarks than a result enabled by creating a new one.
Moreover, the authors’ redundancy and feasibility analyses indicate that redundant and potentially infeasible tasks remain in the benchmark. Thus, in addition to the lack of a new conceptual contribution, technical concerns remain unresolved.
The mentioned redundancy analysis seems to concern only the original 100 tasks, since (as i understand 100 × 99 / 2 = 4950), whereas the feasibility analysis covers all 160 tasks. The authors state: “Of 4 950 task pairs, only 10 pairs are redundant (analysis now detailed in SI). For feasibility, 153 of 160 new tasks have at least one passing design.” Simply reporting these counts is insufficient. Also, the author response also does not explain how redundancy was determined or how the seven unsolved tasks were assessed for feasibility.
My question about novelty concerned structural novelty for structure-generating models: sequences can differ substantially while adopting essentially the same structure. Novelty relative to the native sequence therefore does not fully address my request for comparison with the source protein’s structure.
Technical comments:
- The revised manuscript is not available, i therefore cannot verify the results reported in the rebuttal that refer to newly added sections.
- Some author responses refer me to replies to other reviewers without identifying the relevant point. This makes the author rebuttal harder to follow and assess.
Thank you for the response and for giving us the chance to discuss during the revision period; it is highly appreciated. We are grateful you find that several of the previous concerns were addressed.
Regarding the concern about novelty, we realize the previous answer pointed entirely to our answer to Reviewer 1, and it made it hard to review what we actually meant. We now lay out our response here. Indeed, our manuscript focused extensively on the technical details of the benchmark and not on the actual purpose behind building it. Our major motivation was to identify the limits of what the state-of-the-art models are capable of performing, characterize where they fail, and provide the community with clear next steps on how to build the next generation of frontier models for motif scaffolding. In this sense, we identified two failure modes: 1) mirroring of the original motifs, as the reviewer points out, and 2) all models currently fail on tasks with motifs with >4 segments (<6% success, except RFD4 at 15%). We analyze why the second instance happens, and observe two main reasons. First: models that receive the motif only as pairwise distances are among the first to fail. A good example of this is illustrated by Odesign’s 2 checkpoints, Rigid and Flex, two modes of the same model: Rigid fixes the motif’s 3D coordinates; Flex only receives its pairwise distances. For motifs with multiple segments, Rigid performs significantly better (better on 33 vs 9 tasks, p < 0.001). Second, failures mostly happen after refolding, which implies that models are not capable of co-optimizing residue identity and backbone coordinates under motif constraints when building a scaffold.
Observing this failure mode, we offer the community recommendations on how to train the next generation of models. To showcase this, we have therefore post-trained RFD4 to avoid failures after re-folding. Specifically, we have post-trained RFD4 using reinforcement learning (direct preference optimization, DPO), using refolding outcomes as a training signal. Using 1,000 new motifs from AlphaFold DB (4–7 segments, excluding all Stroma source proteins and their homologs), we have paired RFD4 designs that pass after Boltz-2 refolding with designs for the same motif that fail, and aligned RFD4 on these pairs using DPO. By doing so, the overall Stroma score increases from 38.4% to 42.8%, with improvements on 51 tasks and decreases on 14. For motifs with ≥4 segments, the score increases from 29.6% to 33.8%, and for the held-out synthetic motifs, it increases from 3.3% to 9.0%. The improvement was reproduced in a second training run and we also observed it when designs were refolded with OpenFold3 instead of Boltz-2. We did not observe designs becoming simpler or less diverse. We will make the model weights available to the community in the camera-ready version. We believe post-training strategies such as the one applied here for RFD4 open a new alternative to improve performance in other models as well.
Regarding redundancy and feasibility, thank you for analysing our data with such scrutiny. Due to space constraints before, we had not commented appropriately in this regard.
Redundancy: For the 160 motifs, we superimpose the 12,720 pairs. Out of those, 7 presented an RMSD < 1 Å and can be considered redundant, with 3 corresponding to submotifs of the same motif (RMSD = 0), from the original MotifBench benchmark. Regarding feasibility, thank you for pointing out the discrepancy in numbers. Indeed, we had only reported it for the original benchmark; now we have extended it to the entire dataset. Now, we show that 147/160 tasks have passing designs, with 11/13 failing tasks missing the 1-angstrom criterion by 0.5 angstroms or less and all of them having structures predicted with pLDDT > 70. All of these hard tasks are multi-segment motifs and 8 have n > 4 segments. Three of these hard tasks show an RMSD to GT with native structure prediction of more than 1 angstrom and we agree they might not be solvable under the 1-angstrom criterion. We think nonetheless that they are interesting to include because of their difficulty. These hard tasks that fail by a small margin with the best models are the ones we think the community can use to assess new frontier models.
Indeed, we have not included the PDF with our changes. Assuming ICLR was like NeurIPS and ICML (where the upload is not possible), we failed to upload it in time. We are working to have it as soon as possible.
Thank you again for revising our manuscript; it is highly appreciated.
We thank the reviewer for their thorough and very insightful comments. It is highly appreciated, and addressing all concerns has helped us improve the manuscript enormously.
We address the points below:
Weaknesses
We have added 60 new tasks ( response to reviewer DLVT) and 3 evaluations to the benchmark to incorporate novel functionalities, totalling 160, 149 of which amenable to all models. In addition, we provide a detailed analysis of why models fail on specific tasks, and make recommendations on architecture and training dataset improvements based on the results of the benchmark ( response to reviewer 61b3). We have now assessed feasibility and task redundancy. Of 4 950 task pairs, only 10 pairs are redundant (analysis now detailed in SI). For feasibility, 153 of 160 new tasks have at least one passing design. The final 23 tasks stratum set was selected to increase functional and geometrical coverage based on these criteria: high-quality data available in the literature, absence of this category in other benchmarks, and unexpected results for these tasks in our first version of the benchmark.
Thanks for pointing this out. We are aware that the 11 tasks that span several chains cannot be attempted by some models. We included them because, in our view, it is important to see how the multi-chain capable models perform on them for the subset that can, even if it came at the cost of losing homogeneity.
The criteria are (i) it belongs to a functional class absent from previous benchmarks (ii) its function is assigned to specific residues in the primary literature; (iii) an experimental structure resolves every motif residue; and (iv) the motif can be specified at all five resolutions of the benchmark. We have made our best effort to create a framework where adding new tasks is straightforward for users, as shown now with the addition of 60 more tasks.
We expanded the original task set to reduce that imbalance. Still, about a quarter to a third of all proteins in an organism are enzymes, and we try to mirror that distribution.
We have added a feasibility audit in Supplementary D.1 along with positive controls for all native scaffolds. A feasibility assessment has been performed and detailed in the SI; all 7 tasks that are not solved by models are thoroughly assessed for biological plausibility.
Questions:
We varied M and K from 1 to 8 (now in SI). The mean success rate rises with more samples (0.32 at 2×2, 0.40 at 4×4, 0.47 at 8×8), but the ranking of tasks by difficulty barely changes (Spearman ρ ≥ 0.99 against 2×2 on the backbone criterion, ≥ 0.94 on the heavy-atom criterion). We kept M = K = 2 as the best trade-off between compute and accuracy; users can raise both through the configuration without changing the code.
We adapted some of the analysis introduced in GeomMotif (now cited in Related work) and we now find similar results: helix-rich motifs are easier, β-rich and buried motifs are harder, and success falls with fragment count (Section 5; Supplementary D.10).
We have added a broad category comparison to check the model's performance on different categories. Ranking depends more on segment number than protein type. We also analyzed results based on motif properties and present the full analysis along with the protein category analysis.
Very neat suggestion. We implemented d@k measure that does exactly what is suggested. Results show high diversity in designs from structural models.
We now report novelty relative to the native sequence together with database novelty and d@k metric mentioned above.
Two new figures on GPU time have been added to the manuscript based on this suggestion, and the validation time is now reported separately.
In the supplementary section, we added cut-off calibration to detail how we set that threshold using complexity analysis from a sample of UniProtDB.
Of the generators, RFdiffusion2, RFdiffusion3, RFD4 (added upon release), ODesign and Proteina-Complexa see the ligand during design, and LigandMPNN also receives it for RFdiffusion2, RFdiffusion3 and ODesign; for every other model the ligand first appears when Boltz-2 co-folds it, so the identical no-clash check means "the model designed a pocket" for ligand-aware generators but only "the predictor could fit the ligand" for ligand-blind ones, a difference we now state in the Methods.
We tested this in two ways (Supplementary D.9). Refolding the same designs with the other sequence source makes no significant difference for the models that write their own sequences (La Proteina 25.0% → 29.0%, Protpardelle-1C 18.5% → 19.0%, RFD4 37.1% → 33.5%; all p ≥ 0.08, Supplementary D.9), and all-atom models score about the same with their own sequences (18.7%) as with LigandMPNN sequences (17.2%), so the gap between all-atom and backbone models is not an artefact of LigandMPNN.