- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates whether the performance of protein motif scaffolding can be improved by changing how candidates are sampled, rather than retraining the underlying generative model. Using pretrained RFdiffusion, the authors focus on two practical objectives: obtaining more designs that successfully recover the intended motif after sequence design and refolding, and exploring a broader range of successful scaffold structures within a limited candidate budget.
The proposed framework, S³-Scaffold, intervenes at different stages of the generation process. During denoising, it uses geometric gradients to steer scaffold residues toward better motif preservation, with an error-dependent gate to limit excessive corrections. At an intermediate sampling step, it evaluates candidate structures using ProteinMPNN and ESMFold, then assigns more independent continuations to promising candidates. In parallel, a stratified sampling scheme called EMPS distributes trajectories across scaffold layouts and diffusion-noise regions to reduce redundant exploration. The authors also analyze the conditions under which these interventions improve local geometry, sampling allocation, and coverage of distinct successful structures.
On 30 MotifBench tasks, S³-Scaffold increases the benchmark score from 21.95 to 30.27, the number of solved tasks from 17 to 22, and successful structural clusters from 205 to 225, using 100 final designs per task. The ablation results reveal a distinction between generating more successful candidates and discovering more distinct successful structures: rollout reweighting primarily increases the former, whereas EMPS improves the latter. Overall, the study presents an inference-time approach to improving the utilization of a pretrained protein generator, while requiring additional computation for geometric guidance and intermediate candidate evaluation.
Critical strengths
1. Effective integration of complementary sampling strategies
The paper presents a coherent sampling framework that addresses distinct failure modes in protein motif scaffolding, including motif geometry errors, downstream refolding failures, and limited structural diversity. The component-wise ablations provide useful evidence of their complementary effects. Notably, rollout reweighting increases the number of successful designs but reduces structural diversity, whereas EMPS improves structural coverage without increasing the number of successful designs. The combined framework improves the MotifBench score from 21.95 to 30.27 without retraining RFdiffusion.
2. Rigorous mechanism-level analysis and implementation transparency
The paper goes beyond reporting benchmark improvements by providing mathematical analyses and targeted diagnostics of the proposed sampling mechanisms. In particular, the restored geometric gradients are validated against finite differences, and local guidance interventions reduce motif errors in 31 of 32 tested cases. The EMPS component is accompanied by theoretical results on marginal preservation and expected coverage under explicitly stated assumptions. Detailed implementation settings and computational cost measurements further support the technical credibility and reproducibility of the work.
Critical weaknesses
1. Insufficient methodological novelty and significance beyond RFdiffusion
Although the proposed framework improves RFdiffusion's performance, its methodological novelty and broader significance appear insufficient for a main-track ICLR contribution. The approach primarily combines existing sampling strategies with implementation-level modifications tailored to a single pretrained model. Without demonstrating effectiveness across different diffusion architectures or establishing sufficiently novel model-independent insights, the work remains closer to an RFdiffusion-specific optimization than a broadly applicable methodological advance. While EMPS provides a theoretical contribution, its practical significance beyond the RFdiffusion setting has not been established.
2. Lack of comparisons under matched computational budgets
Although S³-Scaffold improves motif-scaffolding performance at a fixed number of final candidates, it requires substantially more computation than standard RFdiffusion. The authors provide detailed runtime measurements, but do not compare the proposed framework against generating additional independent RFdiffusion samples under an equivalent computational budget. In the reported pilot comparison, gradient guidance increases per-trajectory generation time by approximately 3.7–3.9 times relative to standard RFdiffusion, while rollout reweighting introduces additional sequence-design, refolding, and diffusion-continuation computations. Consequently, it remains unclear whether the reported improvements translate into better computational efficiency. A compute-matched comparison of successful designs and structural coverage is necessary to establish the practical advantages of the framework.
3. Compromised evaluation independence due to benchmark-informed hyperparameter selection
A major concern is the lack of independence between method development and final evaluation. As explicitly stated in Appendix A.3, all 30 MotifBench tasks were used during development, including the selection of the rollout boundary, and the same tasks were subsequently used for final performance evaluation. No held-out evaluation tasks were reserved. This creates a risk of benchmark-specific hyperparameter selection and potentially optimistic performance estimates, limiting the evidence for generalization to unseen motif-scaffolding tasks. This concern is further amplified by the reliance on a single benchmark and the absence of repeated independent runs with uncertainty estimates. Evaluation on genuinely held-out tasks or an independent benchmark would be necessary to establish the robustness and generalizability of the reported improvements.
Questions for authors
Q1. Methodological novelty and generalizability
The proposed framework is evaluated exclusively on the original RFdiffusion model. Could the authors clarify which methodological contributions are expected to generalize beyond RFdiffusion, particularly given the model-specific implementation of geometric gradient restoration? Have the authors evaluated any of the proposed sampling strategies on other diffusion-based protein generators, such as RFdiffusion2 or RFdiffusion3? If not, what evidence supports their broader applicability?
Q2. Computational efficiency under matched budgets
The proposed framework incurs substantial additional computational costs from gradient guidance, intermediate sequence design and refolding, and diffusion continuations. Could the authors compare S³-Scaffold against standard RFdiffusion using an equivalent GPU-time budget, allowing the baseline to generate additional independent samples? In particular, how do the methods compare in terms of successful designs and successful structural clusters as a function of computational cost?
Q3. Independence of hyperparameter selection and evaluation
Appendix A.3 explicitly states that all 30 MotifBench tasks participated in method development and that no held-out tuning split was used. Could the authors clarify the extent to which the final hyperparameters were selected based on performance on these same tasks? Given this overlap, how can the reported benchmark improvements be distinguished from potential benchmark-specific tuning effects? Could the authors provide an evaluation on previously unseen motif-scaffolding tasks or an independent public benchmark?
Q4. Statistical robustness of the reported improvements
The main experimental comparisons appear to rely on a single generated pool per configuration. Could the authors provide results from multiple independent random seeds, including mean performance and variability for MotifBench Score, solved tasks, successful designs, and successful structural clusters? This would help determine whether the reported improvements are stable across independent sampling runs.
5 comments
Rebuttal by Authors (2/2)
Q2. Computational efficiency under matched budgets (continued)
-
Structural and task coverage on multi-segment motifs. S³-Scaffold's coverage advantage is most pronounced on the ten tasks with at least three motif segments (Table 20):
Configuration Final designs/run Successful designs Successful clusters Solved /10 S³-Scaffold (3 runs) 1,000 148.00 ± 1.00 17.33 ± 4.04 4.67 ± 0.58 RF150 1,500 164 6 2 RFtime 2,837 305 8 2 All three S³-Scaffold runs outperform both baselines in successful clusters (22, 15, and 15) and solved tasks (5, 4, and 5). In particular, S³-Scaffold solves 24_1QY3 and 27_4XOJ in every run, whereas neither RFdiffusion baseline succeeds on either task (Table 22 and Appendix G.3).
The additional yield of RFtime does not translate into broader structural coverage. From RF150 to RFtime, 185 of the 199 additional successful clusters (93.0%) come from single-segment tasks. On tasks with at least three segments, successful designs increase from 164 to 305, but solved tasks remain at two. Notably, 150 and 284 of these designs, respectively, come from task 22_1BCF and belong to a single cluster in each pool. Thus, RFdiffusion's increased designs fail to translate into increased structural diversity or task coverage.
-
Performance as a function of GPU cost. We further evaluate RFdiffusion at 25%, 50%, 75%, and 100% of the per-task GPU allocations, counting only designs whose final evaluation finishes within each budget (Appendix H, Table 21). At the full allocation, RFdiffusion produces more successful designs and clusters in total; RF150 also yields 725 successful designs and 265 clusters. However, S³-Scaffold distributes its successful designs across more tasks and achieves substantially greater coverage on complex, multi-segment motifs, highlighting the distinction between sampling yield and effective coverage.
Configuration Final designs/run GPU-hours Successful designs Successful clusters Solved /30 RFdiffusion / 25% allocation 2,108 58.825 335 146 17 RFdiffusion / 50% allocation 4,236 118.174 683 253 17 RFdiffusion / 75% allocation 6,367 177.498 1,027 350 17 RFdiffusion / 100% allocation 8,501 236.769 1,353 464 19 S³-Scaffold (3 runs) 3,000 237.179 685.33 ± 17.21 227.00 ± 5.29 21.67 ± 1.53
Q3. Hyperparameter selection and external evaluation
1. Hyperparameter selection. We selected the rollout boundary based on all 30 MotifBench tasks. Other sampling hyperparameters were not optimized (Appendix A.3).
2. External evaluation on GeomMotif. Without further hyperparameter tuning, we evaluated S³-Scaffold and RFdiffusion on all 57 GeomMotif tasks (Strashnov et al., 2026), generating 100 designs per task:
| Configuration | SUN (%) | Successful designs | Successful clusters |
|---|---|---|---|
| RFdiffusion | 37.9792 | 3,269 | 2,517 |
| S³-Scaffold | 40.0894 | 3,952 | 2,620 |
S³-Scaffold achieves higher SUN, successful-design yield, and structural diversity than RFdiffusion, demonstrating its effectiveness on an external benchmark without benchmark-specific tuning. The full experimental protocol and per-task results are provided in Appendix K (Table 35).
Q4. Statistical robustness across independent runs
We conducted three independent S³-Scaffold runs, each generating 100 designs per task across all 30 tasks (Appendix G.1; per-task results in Tables 16–18):
| Run | Score | Solved /30 | Successful designs | Successful clusters |
|---|---|---|---|---|
| Run 1 (original) | 30.2732 | 22 | 678 | 225 |
| Run 2 | 28.3759 | 20 | 673 | 223 |
| Run 3 | 31.7615 | 23 | 705 | 233 |
| Mean ± sample SD | 30.14 ± 1.70 | 21.67 ± 1.53 | 685.33 ± 17.21 | 227.00 ± 5.29 |
The three runs show consistent performance. Each independently outperforms the public RFdiffusion reference, demonstrating that the improvements are reproducible instead of driven by a single run.
References
Wu, L., Trippe, B. L., Naesseth, C. A., Blei, D. M., and Cunningham, J. P. (2023). Practical and asymptotically exact conditional sampling in diffusion models. NeurIPS, 36, 31372–31403.
Hartman, E., Wallin, J., Malmström, J., and Olsson, J. (2025). Controllable protein design with particle-based Feynman–Kac steering. arXiv:2511.09216v2.
Strashnov, P., Shevtsov, A., Meshchaninov, V., Kardymon, O., and Vetrov, D. P. (2026). GeomMotif: A benchmark for arbitrary geometric preservation in protein generation. ICLR.
I thank the authors for their detailed rebuttal and substantial additional experiments. The new results address several of my original concerns and improve the empirical support for the proposed approach. Nevertheless, I remain inclined to maintain my original Weak Reject recommendation for the following reasons.
Q1 — Generalizability
The RFdiffusion3 experiments substantially address my concern. The improvements from EMPS on a stronger generator provide meaningful evidence that its benefits are not confined to the original RFdiffusion implementation. I appreciate this additional validation and no longer consider the absence of experiments on a stronger generator a major concern.
Q2 — Compute-matched evaluation and practical contribution
I appreciate the addition of both refolding-matched and GPU-time-matched comparisons. These experiments directly address my original request.
However, the results materially affect my assessment of the paper's practical contribution. At approximately equal GPU cost, S³-Scaffold produces substantially fewer successful designs (685 vs. 1,353) and successful structural clusters (227 vs. 464) than expanded RFdiffusion sampling.
I acknowledge that S³-Scaffold achieves higher per-candidate success rates and solves more tasks, particularly among challenging multi-segment motifs. This is a meaningful advantage and is consistent with one of the objectives stated in the original submission.
Nevertheless, the original paper also emphasized improvements in successful-design yield and structural diversity under a fixed candidate budget. The compute-matched results reveal an important trade-off: these improvements do not translate into higher total design yield or structural coverage under a fixed computational budget.
While this does not invalidate the original fixed-candidate results, it substantially limits the demonstrated practical benefit of the method. I remain unconvinced that the improvement in difficult-task coverage sufficiently compensates for the substantial reduction in overall sampling throughput.
Q3 — Hyperparameter selection and independent evaluation
The authors acknowledge that the rollout boundary was selected using all 30 MotifBench tasks, which compromises the independence of the original benchmark evaluation.
The additional GeomMotif results provide valuable evidence of transfer beyond MotifBench. However, they do not remove the potential selection bias in the original evaluation. The extent of structural overlap between the benchmarks is also unclear.
Furthermore, the GeomMotif comparisons use matched final candidate counts rather than matched computational budgets. Thus, while the results support generalization to additional tasks, they do not establish superior computational efficiency.
I therefore consider this concern partially, but not fully, addressed.
Q4 — Reproducibility
The three independent runs provide useful evidence of stability, and I consider my original reproducibility concern substantially addressed. Although repeated baseline runs would allow a stronger statistical comparison, the additional experiments are a meaningful improvement.
##############################
I acknowledge the substantial effort invested in the rebuttal and the positive evidence for improved coverage of challenging motif-scaffolding tasks. My remaining concern is not primarily the absence of additional experiments, but the scope and significance of the contribution established by those experiments.
In particular, the compute-matched results reveal a substantial trade-off between difficult-task coverage and total successful-design yield and structural diversity. Combined with the benchmark-specific hyperparameter selection and the remaining uncertainty regarding the empirical advantages of joint stratification, I do not find the evidence sufficiently compelling to change my original recommendation.
I therefore maintain my Weak Reject rating, while recognizing that the rebuttal has substantially strengthened the paper.
Rebuttal by Authors (1/2)
Thank you for the questions on transferability, computational value, evaluation independence, and sampling variability.
Q1. Methodological novelty and generalizability
S³-Scaffold provides a three-stage sampling strategy: Steer trajectories through geometry-aware, error-gated guidance; Select continuations using dual-criterion refolding feedback; and Stratify exploration through joint layout-noise sampling to improve structural diversity. Despite building on RFdiffusion, our method improves over all its versions and serves as a strong complementary module.
Steer: Gradient construction and error-gated guidance. Wu et al. (2023) reported unstable translation and rotation gradients when applied to RFdiffusion, often causing trajectory divergence. We address this by restoring coordinate derivatives through RFdiffusion's geometric operations while preserving its native forward predictions. Finite-difference tests validate the restored gradients at the tested states (Appendices B.2 and B.5.2). We further introduce an error gate that adapts guidance strength to predicted-motif error, reducing unnecessary scaffold perturbations once motif accuracy is sufficient. Full-benchmark ablations demonstrate improvements in Score, successful designs, and successful clusters, both with and without rollout (Appendices B.3, B.6, and I.1.2).
Select: Dual-criterion refolding feedback. Hartman et al. (2025) use sequence-design and relaxation rewards to resample RFdiffusion trajectories. Our approach introduces a joint refolding score that explicitly enforces both scaffold consistency and original-target motif recovery. Specifically, it takes the maximum of the two threshold-normalized errors from each refold and averages the resulting exponential scores across four refolds. This prevents a low error in one criterion from compensating for a high error in the other. The resulting scores guide the allocation of independent continuations from saved states (Section 4.2; Appendix C).
Stratify: Joint stratification for structural diversity. We introduce a joint stratification of scaffold layouts and geometry-dependent Gaussian-noise paths, requiring no additional denoiser evaluations. Our analysis establishes trajectory-marginal preservation before rollout reweighting and an expected-coverage ordering among joint, single-axis, and independent sampling under a fixed successful-category map and matched one-continuation reference, while preserving the expected success count (Theorems 1–2; Appendices D–E). Appendix F further establishes full-framework coverage lower bounds under retention and continuation conditions. Empirically, EMPS improves successful-structure coverage on both RFdiffusion (205 to 212 clusters, with 481 successful designs in both pools) and RFdiffusion3 (329 to 351 clusters, with Score increasing from 43.23 to 45.64), demonstrating its applicability beyond the original RFdiffusion model (Appendices I.2 and J).
RFdiffusion3 natively preserves fixed motif coordinates in its denoised predictions, so we transfer EMPS and rollout to RFdiffusion3. MotifBench evaluation is unchanged. Each configuration uses one 30-task run with 100 designs per task (main Table 1; Appendix J, Tables 33–34):
EMPS improves all four outcomes at the same 24,000 refolding calls. Adding rollout further increases successful-design yield and Score, but reduces structural diversity at a higher cost of 36,000 calls. EMPS requires access to scaffold layouts and Gaussian-noise paths, whereas rollout requires resumable states and downstream evaluation. These results demonstrate that our training-free sampling framework generalizes beyond RFdiffusion and improves performance on the stronger RFdiffusion3 generator.
Q2. Computational efficiency under matched budgets
We compare S³-Scaffold against two expanded RFdiffusion baselines under complementary budget constraints: RF150 matches refolding calls, while RFtime matches GPU time. Across all 30 tasks, S³-Scaffold achieves a higher success rate (22.84 ± 0.57) than RF150 (16.11) and RFtime (15.92), and solves more tasks on average (21.67 ± 1.53 versus 17 and 19, respectively).
RF150 uses the same 1,200 refolding calls per task to evaluate 150 final backbones. RFtime uses the per-task GPU allocations measured from a complete S³-Scaffold run, totaling 237.179 GPU-hours on NVIDIA A800 80GB GPUs. S³-Scaffold results are reported as mean ± sample SD over three independent runs. All counts use complete evaluated pools, without 50-design subsampling.