- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BindGRPO post-trains a target-conditioned protein binder generator using AlphaFold3 interface confidence as a reward. It constructs conditional Gaussian transitions from BoltzGen’s EDM sampler and applies a binder-coordinate GRPO surrogate to fine-tune the denoiser. On the Cao benchmark, computational success increases from 24.4% to 36.7% after ProteinMPNN sequence reassignment, with improvements also observed for directly generated sequences and under an AlphaFold2 evaluator. The paper additionally examines threshold sensitivity, success-criterion components, and transfer to seven further targets, where the improvement is smaller and statistically inconclusive.
Critical strengths
-
The main comparison convincingly isolates the benefit of post-training. BoltzGen and BindGRPO differ only in denoiser weights under the same evaluation protocol. The improvement after ProteinMPNN sequence reassignment supports better generated backbones, rather than merely more favorable decoded sequences.
-
Validation goes beyond the optimized reward. AlphaFold2 corroboration, threshold sensitivity, paired per-target tests, and decomposition of the success criterion strengthen the evidence for improved computational design quality. The authors also report unfavorable results and explicitly acknowledge the absence of experimental binding validation.
Critical weaknesses
-
The generator could improve by producing successful candidates across multiple structural families or by concentrating on a few high-scoring solutions. The reported success rates, evaluator checks, and threshold analyses do not distinguish these possibilities. Evidence about sequence and structural diversity among successful candidates would clarify whether post-training expands useful design options or mainly increases repeated sampling of similar solutions.
-
Appendix D supports the combined training-pool filter through improved validation ipTM, but does not isolate the contribution of the reward-standard-deviation criterion from mean-reward filtering. Given GRPO’s within-group normalization, the role of this criterion remains unclear. Moreover, the reference-regularization ablation demonstrates that validation ipTM can diverge from benchmark success, limiting what validation reward alone establishes about the filter’s downstream benefit.
-
The two-step AlphaFold3 reward approximation is validated using ProteinMPNN sequences on pretrained BoltzGen backbones, whereas RL uses co-designed sequences from an evolving generator. The reported ranking agreement therefore does not establish approximation fidelity on the samples actually driving optimization. This leaves uncertainty about whether improvements under the accelerated reward consistently reflect improvements under the default evaluator.
Questions for authors
-
How does post-training affect sequence and structural diversity among successful candidates, and does the improvement correspond to more distinct successful designs?
-
What contribution does the standard-deviation criterion make beyond mean-reward filtering, and does the filtering benefit also appear in downstream design success rather than only validation ipTM?
-
Does the agreement between two-step and default AlphaFold3 evaluation remain high for co-designed sequences from later RL checkpoints, particularly those showing the largest reward improvements?
6 comments
W2 — Isolating the spread criterion and validation selection
1. Why group normalization requires a spread filter. The training reward has its internal noise (unlike a reward in math problem) (), the observed group variance is . For targets whose designs have uniform true quality (), dividing by in GRPO scales pure evaluator noise up to unit variance, turning noise into full-magnitude policy gradients. A mean filter () cannot catch low-spread targets anywhere inside that interval. Instead, our threshold () is set directly above AF3's seed-to-seed reward noise floor (), retaining only targets with signal above evaluator noise.
2. Isolating mean vs. spread filters. While Appendix D showed the combined filter improves validation ipTM over no filtering, we further isolated each filter at 500 updates (1,000 targets):
| Training pool filter (1,000 targets) | Best val ipTM | Cao AF3 success (%, best-val ckpt) | Cao AF3 success (%, update 500) |
|---|---|---|---|
| Mean + spread filter (Ours) | 0.478 | 37.7 | 35.8 |
| Spread filter only | 0.485 | 34.5 | 35.6 |
| Mean filter only | 0.460 | 36.8 | 36.3 |
Combining both filters yields the strongest best-validation Cao success (37.7%), whereas spread-only achieves the highest validation ipTM (0.485) and mean-only matches at update 500.
3. Validation ipTM vs. three-gate success for checkpoint selection.
The three-gate criterion of Zambaldi et al. (2024) was specifically calibrated for the Cao benchmark protocol—61–65 residue miniproteins evaluated via any-of-eight ProteinMPNN sequence redesign. Applying those strict binary cutoffs directly to raw co-designed sequences on our diverse validation set yields an extremely sparse pass rate (, or only passing designs per checkpoint).
To verify this, we scored 14 saved checkpoints (Run 3) on the validation split under both criteria:
| Validation selection criterion | Selected update | Val design success (%)* | Cao AF3 success (%) |
|---|---|---|---|
| Highest validation ipTM (reported protocol) | 440 | 3.4 | 37.4 (411/1,100) |
| Highest validation three-gate success | 220 | 3.8 | 31.8 (350/1,100) |
*Note: Validation success is per raw co-designed sequence, whereas Cao success is backbone-level any-of-eight with ProteinMPNN
Because the Cao-calibrated binary gates are overly stringent for un-redesigned PDB validation samples, they separate trained checkpoints () from pretrained BoltzGen () but are dominated by counting noise when ranking trained checkpoints. Under KL regularization (), continuous validation ipTM avoids this threshold sparsity and reliably identifies the stronger checkpoint.
W3 — approximation fidelity on the samples that drive optimisation
We have expanded the paired comparison (2-step vs. 200-step AlphaFold3, keeping sequences, MSAs, and seeds fixed) to cover the generator's co-designed sequences across RL checkpoints (boltzgen, final, and the unregularized run), across both the Cao panel and a 100-target training subset:
| Sequence Source & Checkpoint | Dataset | Pooled Spearman | Within-Target Median | Mean ipTM Offset (2-step − 200-step) | Median Binder RMSD (, binder-only fit) |
|---|---|---|---|---|---|
| ProteinMPNN (boltzgen) | Cao (1,100) | 0.971 | 0.967 | 0.50 | |
| Co-designed (boltzgen)* | Cao (1,100) | 0.960 | 0.931 | 0.66 | |
| Co-designed (BindGRPO) | Cao (1,100) | 0.976 | 0.968 | 0.46 | |
| Co-designed (unregularized, ) | Cao (1,100) | 0.977 | 0.965 | 0.39 | |
| Co-designed (boltzgen)* | Train subset | 0.943 | 0.932 | 2.28 | |
| Co-designed (BindGRPO) | Train subset | 0.978 | 0.971 | 0.88 |
Ranking agreement does not degrade under the cheaper reward: the within-target median on co-designed sequences rises from 0.931 before RL to 0.968 after, matching the 0.967 the paper reported for ProteinMPNN sequences, and from 0.932 to 0.971 on training targets. The two-step ipTM sits 0.03–0.06 below the full-step value in every row, a near-constant offset that leaves rankings intact. Re-folding the same sequences under a different AlphaFold3 seed gives a pooled of 0.95–0.96, so these values are at the evaluator's own noise ceiling.
In short: the two-step and 200-step settings do not disagree in any way that affects training. They rank co-designed sequences the same, before and after RL alike; what disagreement remains is a near-constant offset of about 0.05 ipTM, which cancels in the group-relative advantage; and the residual is no larger than full-step AlphaFold3's disagreement with itself across seeds.
We sincerely thank the reviewer for their thoughtful feedback and insightful suggestions. Based on your comments, we have conducted additional experiments and analyses, and re-evaluated the models across our benchmarks. We summarize the new results and address each point in detail below.
W1 — multiple structural families versus repeated sampling
We cluster the successful designs on the 11 Cao targets (1,100 designs per method, with 1100 structural proposals and 8800 proteinMPNN sequences) by sequence (50 % identity) and by binder backbone (4 Å Kabsch RMSD on index-aligned Cα atoms. ). For sequence cluster, we cluster the passing sequences and one successful design contributes anywhere from one to eight of them.
We further compute the sequence and structural diversity regardless of success or not for BoltzGen and BindGRPO.
Findings:
In short: post-training buys more distinct successful designs in absolute terms and leaves sequence diversity intact, at the cost of a modest narrowing of the backbone distribution.