- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Proteocraft proposes mutation though gradient based mutataion proposal and MCTS. Compute gradients, identify possible mutations, generate, run cofolding model, evaluate the confidence and then use mcts to choose which sequence to expand then repeat. It first represents a.a as continuous probabilities then calculates the gradient signal using distogram, seq prior and one-step diffusion. core novelty is from the one-step diffusion gradient,
Critical strengths
- Performance seems good, however, i leave a question regarding how fair performance comparison is
- Wet-lab results seem promising this supports that the method works
- One-step diffusion contribution seems clear
Critical weaknesses
- Novelty is questioned. What is already established: Continuous amino acid logits and gradient-based optimization, using protein language models as sequence priors, distogram-based sequence optimization, evaluating mutated sequences with structure predictors. UCB/PUCT-based tree search, iterative sequence refinement. All above already exist. One-step diffusion signal is not novel and coarse-to-fine is just an algorithmic choice.
- I question the headline claims that ProteoCraft significantly outperforms existing binder-design methods. Different computational budgets. Table 1 compares 210 candidates per method. But ProteoCraft performs substantial optimization before selecting its final candidates: 70 Stage-A search iterations, evaluating up to 8 candidates per iteration, 40 additional Stage-B refinement iterations, Multiple initial sequences and repeated full ESMFold2 evaluations.Meanwhile, competing methods use their native generation procedures. You must report BoN(for all design methods) or TTA methods(complexa) in each paper along with GPU-h or budget analysis. Otherwise, it's hard to accept this claim. Figure 1's quality-versus-runtime comparison is therefore not a standardized comparison of the cost to produce one final high-quality binder. The authors should report GPU-hours per design passing the same independent evaluation threshold for every method
Questions for authors
- paper does not provide enough information about the diffusion noise schedule to determine exactly how many denoising steps occur before τ = 2. Please elaborate regarding the one-step diffusion gradient.
- The method assumes: Gradients from approximate structural objectives provide useful directions for discovering sequences with better full-model confidence. But the paper does not directly demonstrate how reliably that assumption holds. For example, suppose the gradient recommends 100 amino acid substitutions. How many actually improve pair iPTM after complete structure prediction? And how does this compare with random mutations, ESMC-based mutations, or distogram-only gradients? That experiment would tell us whether their gradient proposal mechanism is genuinely informative. Without it, we cannot easily distinguish two explanations for their success:
- The main benchmark contains 14 target–hotspot settings, but only eight unique proteins. In particular nine of the fourteen settings come from just three proteins. This can inflate the apparent diversity of the benchmark.The authors also do not provide convincing uncertainty estimates across independent design runs or target proteins. I would prefer evaluation on a substantially larger and more diverse target set, with uncertainty reported at the protein level. For more targets, refer to ProtDBench targets and please add Boltzgen hard targets(in examples). include protein binders and nanobodies.
- More cofolding model confidence results, use opendde for instance and protenix-v2.
- Where are the TTA methods from complexa paper? Beam/MCTS/BoN/Feynman-kac sampling? Why exclude these in the performance? The authors state that the internal variants use matched candidate/verifier budgets. This provides evidence that MCTS improves the authors' own proposal-based optimization pipeline. However, it still does not establish whether MCTS is better than beam search or a conventional evolutionary algorithm using the same proposals.
- I would require a compute-matched benchmark: Fix the same target structures, hotspots, and binder lengths. Give every method the same GPU-hour budget. Permit each method to generate and filter candidates using its own pipeline. Evaluate the resulting candidates using independent structure predictors. Compare the number of high-confidence designs obtained per GPU-hour, along with structural diversity. Report results separately for each protein, with confidence intervals across independent runs. If this is too much, refer and use ProtDBench. Please add TTA methods as i questioned in 5.
- Report diversity of generated binders using foldseek or other tools. report how diverse, threshold-passing binders are made. Efficiency should focus on that.
- Please compare performance with RFOptimization.
13 comments
Reply to Reviewer ftSc (Sec II)
(1) ProtDBench [A] itself provides direct evidence for this concern. Its analysis of experimentally annotated designs shows substantial dependence on the verifier, confidence metric, and target. It further shows that different verifiers recover different subsets of experimentally confirmed binders, and that recall decreases rapidly as more verifiers are required to agree. Consequently, using a common confidence threshold standardizes the computational evaluation, while the relationship between that threshold and experimental success still requires validation. In particular, a higher number of filter-passing designs per GPU-hour cannot, by itself, establish a higher number of experimentally functional binders per GPU-hour.
[A] ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation. ICML 2026
(2) Further evidence [B] analyzed 3,766 experimentally tested de novo binders across 15 targets. Although AF3 ipSAE_min was the strongest individual predictor, its predictive performance varied substantially across targets. Even at target-specific thresholds selected to maximize F1, the corresponding precision ranged from approximately 0.1 to 1.0, with limited positive counts contributing uncertainty for some targets. Combining confidence scores with physicochemical interface descriptors, including Rosetta δG/δSASA and shape complementarity, further improved prediction. These findings demonstrate that a confidence threshold does not establish a consistent experimental success probability across targets. They support evaluating design methods through complementary computational measures and prospective binding experiments.
[B] Predicting Experimental Success in De Novo Binder Design: A Meta-Analysis of 3,766 Experimentally Characterised Binders. 2025.09
(3) We also investigated this issue further through our retrospective analysis of 1,230 experimentally tested designs from the Anthropic dataset (https://huggingface.co/datasets/Anthropic/claude-protein-binder-design) [C]. Discrimination varied substantially by target: for ipSAE-based screening, the median AUC across predictors was approximately 0.481 for BHRF1 and 0.497 for Cas9, compared with 0.880 for Latent GDF-8. On the 12 targets with at least 5 experimental positives, selecting the top 10% of designs within each target by ipSAE yielded reported hit rates of 48.8%–61.9% across individual predictors. Thus, approximately 38%–51% of these highly ranked candidates still failed experimental binding assessment. Combining all ten predictors increased the reported hit rate to 63.1%, leaving approximately 37% experimental failures. Confidence ranking enriched hits relative to random selection, but substantial uncertainty remained even among the highest-ranked candidates. These results illustrate why confidence-filter pass rates require experimental calibration before they can be interpreted as functional binder yields.
[C] Autonomous de novo protein binder design with Claude. 2026.08
(4) Complementary evidence [D] from 106 experimentally determined nanobody–antigen complexes and 11,342 shuffled non-cognate pairings showed that AF3, Boltz-2, and Chai-1 frequently produced geometrically plausible complexes, while their ipTM scores had limited ability to distinguish cognate from non-cognate pairings. Increased sampling improved structural refinement but provided limited improvement in pairing discrimination. This finding is particularly relevant to our VHH setting, and further motivates experimental binding and specificity measurements when assessing design success.
[D] Structural Plausibility Without Binding Specificity. 2026.03
(5) Experimental binding affinity introduces a further distinction. The Adaptyv EGFR competition analysis [E] found no statistically significant correlations between measured affinity and the metrics used to rank designs, including ipTM and iPAE. The POISK antibody benchmark [F] similarly reported only a weak relationship between structural confidence and affinity across 5,207 records with usable affinity or potency measurements. These studies show why increases in computational confidence alone provide limited evidence about improvements in binding strength.
[E] Crowdsourced Protein Design: Lessons From the Adaptyv EGFR Binder Competition. 2025.04
[F] Can Folding Models Tell Binders from Bluffers? Evidence from POISK: The Patent-Derived Antibody Dataset. NeurIPS 2026
To summarize, a larger confidence-based target panel would not be treated as a substitute for prospective experimental validation or as direct evidence of improved binding success. We therefore consider prospective experimental validation essential when assessing whether a design method produces functional binders.
Reply to Reviewer ftSc (Sec III)
For PD-L1, 29 of 35 tested miniproteins showed measurable binding, yielding an 82.9% hit rate, the highest observed rate among the studies listed in our comparison. This compares with reported rates of 82.4% for PXDesign, 77.8% for BindCraft, 72.0% for Chai-2, and 10.0%–22.0% for the other listed methods. The best ProteoCraft binder achieved a fitted of 0.879 nM, demonstrating subnanomolar affinity. Our IFNAR2 experiments additionally yielded 4 binding hits among 9 successfully expressed VHH candidates, with a best fitted of 26.29 nM. Together, these prospective measurements demonstrate experimental binding across two targets and two binder formats, complementing our computational evaluations.
| Method | Hit rate | Hits / Tested | Best K_d (nM) |
|---|---|---|---|
| Proteocraft | 82.90% | 29/35 | 0.879 |
| PXDesign | 82.40% | 14/17 | 21 |
| BindCraft | 77.80% | 7/9 | 615 |
| Chai-2 | 72.00% | 18/25 | 0.982 |
| Latent-X | 22.00% | 22/100 | 0.27 |
| AlphaProteo | 15.00% | NA | 0.18 |
| RFdiffusion | 12.60% | NA | 1.6 |
| BoltzGen | 10.00% | 2/20 | 96 |
Additional evaluation on ProtDBench
Benchmark construction and implementation. To address the reviewer’s request for broader computational coverage and fair comparison with the same computational budget, we nevertheless evaluated ProteoCraft on all ten ProtDBench targets: BHRF1, PDL1, IR, TrkA, IL7RA, SC2RBD, VEGFA, H1, IL17A, and TNFα. To cover multiple binder lengths within the supplementary evaluation budget, we selected five fixed lengths (80, 90, 100, 110, and 120 residues) shared by all targets. For each target–length combination, ProteoCraft contributes four backbones, yielding 20 backbones per target and 200 backbones overall. We extracted baseline results for the corresponding target–length combinations from the official ProtDBench release. BindCraft contributes 291 backbones across these combinations, while each other method contributes 200.
We sampled 8 ProteinMPNN sequences per backbone and evaluated them using the shared independent AF2-IG protocol. This produces 1,600 evaluated sequences per method for the 200-backbone cohorts, and 2,328 for BindCraft. The table below reports sequence-level pass rates under the AF2-IG Easy and Strict filters. We first calculate the pass rate for each target–length combination, average the five lengths with equal weight within each target, and then average the ten targets with equal weight. This aggregation prevents targets or length settings with more evaluated samples from dominating the reported means.
Independent AF2-IG evaluation. ProteoCraft achieves mean Easy and Strict sequence pass rates of 54.50% and 27.06%, respectively. The strongest baseline results are BindCraft’s 33.88% Easy pass rate and PXDesign’s 11.94% Strict pass rate. Using the unrounded values, ProteoCraft improves these by 20.62% and 15.13%, respectively.
The improvement extends across most targets. ProteoCraft has the highest Easy pass rate on nine of ten targets and the highest Strict pass rate on nine targets. On H1, all methods have a Strict pass rate of zero. ProteoCraft’s strongest results include PDL1, with Easy/Strict pass rates of 91.88%/69.38%, BHRF1 at 76.88%/54.38%, IR at 67.50%/52.50%, and TrkA at 75.00%/49.38%.
| Method (Easy / Strict) | BHRF1 | PDL1 | IR | TrkA | IL7RA | SC2RBD | VEGFA | H1 | IL17A | TNFα | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| ProteoCraft | 76.88 / 54.38 | 91.88 / 69.38 | 67.50 / 52.50 | 75.00 / 49.38 | 71.88 / 1.88 | 56.88 / 30.63 | 31.25 / 8.13 | 3.75 / 0.00 | 26.25 / 1.25 | 43.75 / 3.13 | 54.50 / 27.06 |
| BindCraft | 44.46 / 17.74 | 70.42 / 42.02 | 53.66 / 19.96 | 47.96 / 25.45 | 44.52 / 0.00 | 17.67 / 3.50 | 10.36 / 0.50 | 30.46 / 0.00 | 19.24 / 0.42 | 0.00 / 0.00 | 33.88 / 10.96 |
| PXDesign | 52.50 / 40.00 | 48.13 / 36.25 | 43.13 / 22.50 | 33.75 / 16.25 | 26.25 / 0.00 | 7.50 / 1.25 | 18.75 / 3.13 | 20.00 / 0.00 | 0.00 / 0.00 | 2.50 / 0.00 | 25.25 / 11.94 |
| BoltzDesign-1 | 6.25 / 2.50 | 29.38 / 7.50 | 11.88 / 0.00 | 40.63 / 8.75 | 6.25 / 0.00 | 6.25 / 0.63 | 0.63 / 0.00 | 6.88 / 0.00 | 8.75 / 0.00 | 0.00 / 0.00 | 11.69 / 1.94 |
| BoltzGen | 13.75 / 13.13 | 20.00 / 10.00 | 15.00 / 6.88 | 7.50 / 0.00 | 2.50 / 0.00 | 1.25 / 0.00 | 6.88 / 0.00 | 27.50 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 9.44 / 3.00 |
| ODesign | 38.75 / 11.88 | 3.75 / 0.00 | 8.13 / 3.75 | 10.00 / 0.00 | 18.75 / 0.00 | 5.63 / 2.50 | 1.88 / 0.00 | 6.88 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 9.38 / 1.81 |
| RFdiffusion-3 | 35.00 / 15.00 | 10.00 / 5.00 | 12.50 / 8.75 | 3.13 / 1.88 | 11.25 / 0.00 | 0.63 / 0.00 | 2.50 / 0.00 | 1.88 / 0.00 | 0.63 / 0.00 | 0.00 / 0.00 | 7.75 / 3.06 |
| Protpardelle-1c | 0.00 / 0.00 | 5.00 / 0.63 | 1.25 / 0.00 | 1.25 / 0.00 | 0.00 / 0.00 | 5.00 / 0.00 | 0.00 / 0.00 | 1.88 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 1.44 / 0.06 |
Reply to Reviewer ftSc (Sec I)
Thank you for the detailed assessment and for recognizing the promising performance, experimental binding results, and contribution of one-step diffusion guidance. Your questions motivated additional mutation-level controls, a beam-search comparison, and a broader ProtDBench evaluation under a measured A100 GPU budget. Below, we clarify the methodological contribution and report the additional evidence on proposal quality, computational efficiency, and structural diversity.
Q1: Exact diffusion schedule and the one-step gradient
In our ProteoCraft implementation, setting the truncation noise level to τ=2 with the 50-point clipped schedule produces 22 reverse-sampling updates before the terminal differentiable denoiser call. This includes the update that reaches τ=2. All 22 updates run without gradient tracking. At every one-step gradient evaluation, the sampler starts from Gaussian noise and recomputes the trajectory using the current soft sequence conditioning. We then apply stop-gradient to the newly sampled Xτ, evaluate the frozen denoiser once at τ=2, and differentiate the structural loss through this terminal call and the soft sequence conditioning with respect to the sequence logits. We will include the schedule construction and the distinction between forward sampling and backward differentiation in the method description.
Q2: Whether gradient proposals identify beneficial discrete mutations
Thanks for your question. Our existing No One-step Diffusion ablation retains the distogram and ESMC signals. Before tree search, adding the one-step diffusion signal increases mean pair-chain iPTM from 0.637 to 0.660 and the number of candidates above 0.7 from 85/210 to 118/210, supporting its contribution to proposal quality.
To directly assess individual substitutions, we additionally compared five proposal strategies: random mutations, ESMC-only, distogram-only, distogram+ESMC, and the full method. Each strategy was evaluated on 128 single-residue substitutions using complete structure prediction. We report both the mean change in pair-chain iPTM and the fraction of mutations whose improvement exceeds 0.01.
The full method produces 23/128 mutations with improvement above 0.01, compared with 13/128 for random mutations and 18/128 for distogram+ESMC. The improvement over random is 7.8 percentage points, with a paired 95% confidence interval of +0.8 to +16.4 percentage points. The difference from distogram+ESMC is 3.9 percentage points, with an interval of −7.8 to +14.8 percentage points.
At the 20 soft updates used in Stage B, the mean changes across all substitutions are +0.002096 for random mutations, +0.001905 for ESMC-only, +0.001744 for distogram-only, +0.003458 for distogram+ESMC, and +0.004560 for the full method.
These results provide direct evidence that the full proposal mechanism enriches mutations with larger improvements compared with random sampling. Its incremental advantage over distogram+ESMC remains uncertain in this experiment. We will state this distinction alongside the existing end-to-end ablation.
Q3 / Q6 / W2: Broader target coverage and standardized quality–cost evaluation
We agree that the original benchmark contains 14 target–hotspot settings from 8 unique proteins, and that these settings should not be treated as 14 independent proteins. We will state both counts consistently.
Moreover, we agree that broader target coverage is valuable, and we have added the requested ProtDBench evaluation. However, assessing practical binder-design performance primarily through confidence-filter pass rates requires an additional assumption: that passing the chosen filter is sufficiently predictive of experimental binding across targets and design methods.
The available evidence shows substantial limitations to this assumption.