acceptodds
Under review as a conference paper at ICLR 2027

PEPFM-TANRL: FINE-TUNING A MULTI-MANIFOLD FLOW MATCHING MODEL FOR PEPTIDE DESIGN VIA TANGENT-SPACE GRPO

Abstract

*De novo* design of peptides that bind a target protein is a discrete–continuous generative problem requiring the joint modeling of structure and sequence. Flow-matching-based generative models faithfully reproduce the distribution of native peptide–receptor complexes, yet likelihood training provides no guarantee that generated samples exhibit high binding affinity. We introduce PepFM-TanRL, a framework that brings Group Relative Policy Optimization (GRPO) to deterministic-ODE flow models operating over multi-modal continuous state spaces. Injecting time-decaying noise converts the ODE sampler into a stochastic policy, from which we derive factored importance ratios and analytic KL penalties across four modalities — , , torus, and logit space. As an architectural change that enables reinforcement learning, we predict torsion angles for all 20 amino acid types simultaneously and aggregate side-chain features via probability-weighted expectations, eliminating discrete sampling so that every generation step admits a closed-form Gaussian log-likelihood in the tangent space. Fine-tuning with a reward that splices the Vinardo docking score with a steric-clash penalty improves the reward distribution on all four targets (best-target median reward ; clash-free fraction 39% 80%); on the one target that starts with a very high clash rate, the gain stops at fewer clashes without reaching a Vinardo-scorable population. On the other three, Vinardo scores among clash-free samples also shift higher as energetically unfavorable conformations are suppressed, while native-like backbone geometry and structural diversity are preserved. We further analyze how reinforcement learning alters the similarity of generated structures to the native complex.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
flow matching, reinforcement learning, GRPO, peptide design, protein-peptide interaction, structure-based drug design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eKgMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PepFM-TanRL, which uses a reward-based finetuning of a flow matching model. The method introduces time-decaying noise into ODE sampling and extends GRPO objectives to different modalities. The main result independently fine-tunes models for four peptide-target complexes and shows improved reward and clash-free rates. The paper also shows a separate 128-target evaluation of pretrained models; however, the 128 target evaluation does not evaluate the proposed RL methods.

Critical strengths

I think combining noise injection during sampling with policy regularization is interesting. The noise gives the sampler some kind of stochastic transition property, and the KL term explicitly penalizes the pretrained reference policy. I think it's a potentially useful way to adapt pretrained generative models.

Critical weaknesses
  1. The presentation makes the method and its contribution hard to read and follow. The paper needs to give a clearer explanation of the method, rather than showing too many implementation details. I would also like to see what has changed and why we need these changes for PepFlow. The case studies are in the Appendix, which makes it hard to understand the results. Also, there are some weird names, such as pepflowww_repro; I suggest the authors do some proofreading of the paper.
  2. The experimental results use very small datasets, and it's unclear how to find the relationships between different experimental sections.
Questions for authors
  1. Could you provide a self-contained explanation of how the method modifies PepFlow, and clearly explain each part of the method?
  2. Can the current method show consistently better results on larger and more diverse datasets?
  3. How does the final fine-tuned model compare with PepFlow pretrained weights? I also think we need more ablation studies on different parts of the method.
4 comments

Response to Reviewer eKgM

Thank you - the presentation criticisms were fair and we acted on all of them. Our general response summarizes the revision; below we answer your points.

Q1 (self-contained explanation of how the method modifies PepFlow). Added as Appendix B, which can be read without the PepFlow paper. It opens with what we keep (the flow-matching formulation, dataset, pocket definition, loss terms and weights, IPA backbone), and Table 2 then lists all five changes - torsion state, side-chain feature aggregation, sequence target per step, side-chain feature content, time conditioning - each with its reason and a pointer to the subsection that details it (App. B.1-B.4). The first three exist for one reason, which we now state plainly: an importance ratio needs the probability of each sampled transition, and PepFlow draws a discrete amino acid type inside the generation loop, which makes that density unavailable in closed form. The last two are independent architectural choices.

Weird names. Removed. pepflowww_orig / _oh / pepflowww_repro / _oh are now "PepFlow (released)" and "PepFlow (retrained)", with a separate column for whether the sequence target is continuous or one-hot - the two axes the suffixes encoded. The dataset file name and the code-style typesetting of MSE and CE are gone too; the only monospaced text left is peptide sequences.

Q2 (larger and more diverse datasets). Added nine Vilya Riptides targets - the entries our model can represent (linear unmodified peptides of 3-25 standard residues, standard-residue pocket, absent from the PepFlow training split). Deposited June 2023 to January 2024, after the PepFlow dataset, they act as a temporal hold-out, and all nine were fine-tuned under one shared configuration with no per-target tuning. Six of nine improve on both clash-free rate and the Vinardo score of the clash-free samples, e.g. 7YC2 10.6 -> 68.5% and 8GW4 16.5 -> 56.9% (Sec. 5.3, Table 1). The other three fail outright: no pretrained sample clears the clash-penalty threshold there, so the advantage only ranks degrees of failure; we trace it to how enclosed the initialization is on those receptors (App. E.5).

Q3 (comparison with PepFlow's pretrained weights). Table 13 now includes the released checkpoint alongside our retrained PepFlow and our own pretrained model, under the clash-free definition used elsewhere, so it reads directly against Table 1. It is strongly target-dependent: 78.1% clash-free on 1SSH, 21.9% on 5UZW, 6.3% on 2R02, 0.0% on 6V63. The fine-tuned checkpoints exceed it on all four:

TargetClash-free (fine-tuned vs released)R_95% (same order)
1SSH79.8 vs 78.1%+0.70 vs -1.30
2R0274.3 vs 6.3%+0.31 vs -13.44
5UZW61.5 vs 21.9%+0.00 vs -2.99
6V636.6 vs 0.0%-15.03 vs -26.32

Caveats are in App. E.7: the architectures differ, the fine-tuned numbers come from 1,024 noise-injected samples against 64 deterministic ones, and our method cannot be applied to the released weights at all. Not a controlled comparison, then; what it shows is that the optimization passes the published model's reward statistics from a weaker start.

Q4 (more ablations). Partly. Appendix C still ablates the sequence parameterization and the advantage/reward rescaling strategy. We added the reasoning for the factored importance ratio (Sec. 4.2) and for the probability-weighted torsion ratio, now stated to be an approximation rather than an exact density ratio (App. A.2.1). A run that replaces the four per-modality ratios with a single joint-likelihood ratio is in progress; if it finishes during the discussion period we will report it.

Relationships between the experimental sections. Section 5 now reads as one argument: 5.1 setup and definitions, 5.2 the four individually configured targets, 5.3 nine held-out targets under one configuration, 5.4 what that is worth against oversampling the pretrained model, 5.5 what it costs in similarity to the reference. The two analyses carrying the claim are in the main text - the Vinardo score of the clash-free samples, now a column of Table 1 with the distributions in App. Fig. 4, and the best-of-N comparison - while the per-structure case studies stay in the appendix as the qualitative check they are. The abstract was also shortened and no longer claims improved affinity.

A short note to flag that the run we mentioned in our reply to Q4 has finished.

The factored importance ratio is now ablated against a single joint-likelihood ratio, so Appendix C ablates three things rather than two: the sequence parameterization, the advantage/reward rescaling strategy, and the construction of the importance ratio itself. The new material is a table of both runs at 3,000 and 6,000 steps, in the format of the per-checkpoint tables and with the equivalent best-of-N added, and a figure of the clip fraction over training.

The numbers and the caveats are in the general follow-up comment on this submission. If anything there raises further questions, we would be glad to answer them in this thread.

I appreciate the authors’ response and their careful answers to my questions. I have increased the presentation and soundness scores. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

Thank you for the careful reading in both rounds, and for revisiting the presentation and soundness scores after our response. We appreciate the reconsideration, and we respect your decision.

One addition since your comment: the joint-likelihood-ratio ablation, which our earlier follow-up reported on 7YC2 alone, finished in time on a second target (1SSH, a PepFlow-split target evaluated with the other reward version), so Appendix C now covers both. We added it so that the ablation does not rest on a single target; the numbers are in our general follow-up comment on this submission.

Thank you again for the questions you raised — they sharpened several parts of the paper.

Reviewer_YLYGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PepFM-TanRL, which fine-tunes a multi-manifold flow-matching model (PepFlow) with reinforcement learning for peptide binder design. The motivation is that likelihood-trained flow models reproduce native complexes but do not optimize binding, and many generated samples contain steric clashes that make docking scores unusable.

The method adapts GRPO to a deterministic-ODE flow over a product of manifolds (ℝ³, SO(3), torus, logit space). Injecting time-decaying Gaussian noise at each Euler step turns the sampler into a stochastic policy with a closed-form tangent-space likelihood, which gives per-modality factored importance ratios and an analytic per-modality KL to the frozen reference. To make this tractable, the authors retrain PepFlow with an architectural change: predicting torsion angles for all 20 amino-acid types and aggregating side-chain features by a probability-weighted expectation (Eq. 1), so no discrete sampling breaks the likelihood. The reward combines the Vinardo docking score for clash-free samples with a decaying clash penalty otherwise.

The authors fine-tune four targets independently (about two GPU-days each). They report that median reward and clash-free rate improve on all four, that diversity, secondary structure, and bond geometry are preserved, and that unrelaxed Rosetta ΔG_bind improves on all targets as independent validation. A relaxed ΔG_bind analysis is given in the appendix.

Critical strengths

A competent, reusable construction for RL on a product of manifolds. Factoring the importance ratio per modality so the σ-dependent normalization cancels (Eq. 12), and reducing the KL to a squared logarithmic-map distance that grows near t→1 (Eq. 14), is a thoughtful treatment of a real failure mode (scale/dimension imbalance across ℝ³/SO(3)/𝕋/logit). I note this honestly — but it does not redeem the submission, because the construction is (a) acknowledged to “share the same intuition as Flow-GRPO,” and (b) never shown to outperform a naive joint-likelihood or single-SDE alternative, so its necessity is unestablished.

Critical weaknesses

W1. The main affinity claim is contradicted by the authors’ own data. The reward scores only unrelaxed structures, and the authors validate with unrelaxed ΔG_bind (App. F.4), which shares that bias and so agrees with the reward (ρ > 0.83). On relaxed ΔG_bind, the metric that matters in practice, binding worsens on 1SSH (−18.20 to −12.79) and 2R02 (−13.51 to −10.71) and is uncorrelated with the reward on all four targets (|ρ| ≤ 0.24, p > 0.08). The authors concede this and did not pursue a fix. The headline contribution is therefore not just unproven but contradicted.

W2. The optimized endpoint is a non-binder on the method’s own metric. Since clashing samples score in [−30, −20], the reported median jump (−23.93 to −0.41) is just the exit from the clash floor; reward near zero means a Vinardo score near zero, i.e. no predicted binding. The best decile is only +0.69 on 1SSH, far from the −8 to −12 kcal/mol of real binders, and the authors admit the rewards plateau (Sec. 5.3). The result is clash removal, not affinity.

W3. No baseline, and the obvious cheap alternative is untested. The reward already runs OpenMM minimization (Sec. 4.4), so the key question is whether two GPU-days of GRPO per target beats simply oversampling the pretrained model and repacking to clear clashes. The authors never check this, so it is unclear the RL does any useful work.

W4. The “framework” components are never ablated. The authors note the construction “shares the same intuition as Flow-GRPO,” yet neither the factored importance ratio nor the per-modality KL is tested against a joint-likelihood or single-SDE variant. App. C ablates only the sequence parameterization and rescaling.

W5. Weak base model, thin statistics, failure on the realistic target. The pretrained model is weaker than released PepFlow (AAR 0.286 vs 0.490; clash-free 16% to 19% vs 27.1%), so the problem is partly self-inflicted. With four targets, 64 samples, and no error bars, generality is unestablished. On 6V63, the one realistic 19-mer, the method essentially fails (3.1% to 10.9% clash-free). On 1SSH the authors weaken the KL to β = 10⁻⁴, after which BSR drops (0.976 to 0.717) and relaxed binding worsens, which looks like mild reward hacking.

Questions for authors

Q1. If the authors oversample the pretrained model (or PepFlow) to match the RL budget and run the minimization already in your reward, what clash-free rate and relaxed ΔG_bind result, and does RL beat it?

Q2. Given that relaxed ΔG_bind worsens on 1SSH and 2R02 and is uncorrelated with the reward, how should readers read the affinity claim? Would you reframe it as improved physical validity?

Q3. Could the authors report raw Vinardo scores (not reward) for clash-free samples before and after RL? Do any reach the −8 to −12 kcal/mol range?

Q4. Can the authors show the factored importance ratio and per-modality KL actually help, versus a joint-likelihood or single-SDE baseline?

Q5. Could the authors add diagnostics that the closed-form likelihood is well-behaved (per-step ratio distribution, PPO clip fraction, effective sample size), given the approximations in App. A?

4 comments

Response to Reviewer YLYG

Thank you - your central objection was correct: the affinity claim was contradicted by our own data. We have withdrawn it and reframed the paper; our general response summarizes the revision, and below we answer your points.

W1, W2, Q2 (affinity contradicted; reframe as physical validity). We agree, and improved physical validity is now the claim. The paper says the reward traverses the range where a docking score reports steric feasibility rather than discriminating binders, the unrelaxed Rosetta agreement is a consistency check between two scores dominated by the same steric term, and the relaxed energy does not improve (Sec. 6, App. E.12). The reward also scores the unrelaxed structure the model emits (Sec. 4.4), one reason the absolute values are pessimistic.

Q3 (raw Vinardo for clash-free samples; anything reaching -8 to -12?). Table 9 and App. Fig. 4. Medians, pretrained -> RL:

TargetVinardo median (kcal/mol)n
1SSH+9.16 -> +0.05311 -> 817
2R02+7.08 -> +1.12375 -> 761
5UZW+10.36 -> +2.42135 -> 630
6V63+14.58 -> +11.0019 -> 68

The scorable samples themselves improve, so the gain is not only that more became scorable. The answer to your question is no: the best single sample over 1,024 draws is -1.4 kcal/mol on the PepFlow-split targets and -2.1 across all 13.

Q1, W3 (oversampling baseline). Sec. 5.4, App. Fig. 5 and Table 11: how many of 32,768 pretrained draws best-of-N needs to match a quantile the checkpoint reaches. Final checkpoints, median (90th percentile): 1SSH 137.3 [106.8, 177.1] (138.0), 5UZW 27.0 [19.2, 39.3] (313.8), 2R02 13.0 [10.6, 15.4] (38.7), 7YC2 257.8 [164.2, 398.1] (1150.8). Two stay near the baseline: 6V63 (3.1) and 8C1L (2.3), already 90.7% clash-free before RL. RL therefore does work that oversampling does not, and the gap widens in the tail. We did not add repacking or a relaxed dG_bind: everything we report scores the unrelaxed output (Sec. 4.4).

W4, Q4 (framework components never ablated). Still open and we will not claim otherwise. The reasoning for the factored surrogate is now in Sec. 4.2: it keeps PepFlow's per-modality loss weights, lets the other modalities keep learning when one leaves the trust region, and avoids the high clip rate of a ratio built from 3 + 3 + 100 + 20 coordinates per residue. A joint-likelihood-ratio run is in progress; if it finishes during the discussion period we will report it.

Q5 (likelihood diagnostics). We monitor the clip fraction during training, as a batch average per update rather than resolved by t: it sits at roughly 5-35% depending on the modality, so clipping does not dominate the updates and training is stable. On the approximation side, App. A.2.1 now states that the torsion ratio is an approximation rather than an exact density ratio, why we use it, and that the PPO surrogate bounds nothing there; the other three ratios and the KL are exact.

W5 (weak base model; thin statistics; 6V63; reward hacking on 1SSH).

  • Statistics: 64 -> 1,024 samples, 4 -> 13 targets, bootstrap intervals on the best-of-N estimates.
  • Base model: Table 13 now includes the released PepFlow checkpoint (78.1% clash-free on 1SSH, 0.0% on 6V63). Read against Table 1, the fine-tuned checkpoints exceed it on all four targets from a weaker start - though the architectures differ and our method cannot be applied to the released weights at all (App. E.7). Now an explicit limitation.
  • 6V63: still essentially fails (1.9% -> 6.6%), reported as such.
  • 1SSH / BSR: quantified over 1,024 samples in App. E.9. BSR falls on nine of the ten targets whose reward carries signal, by 0.085 at the median and by 0.257 on 1SSH, the largest drop. It never approaches zero, though - the lowest post-RL value over all thirteen targets is 0.569 - and RMSD does not grow with it. We read this as the ensemble moving off the crystallographic contact set, not out of the pocket.
  • 1SSH / beta: you are right that only 1SSH loosens the KL. It was exploratory: the reward distribution looked like it was plateauing at 12,000 steps and we wanted to know whether a weaker constraint would let it keep improving, so we tried it on our lightweight validation target. It did (clash-free 71.5% -> 79.8%), so we report the run as it is. Not a free gain, though: BSR keeps falling and reward-hacking-like behaviour cannot be excluded, even if the fall does not begin at the change (0.978 -> 0.789 over the 12,000 steps at beta = 1e-3, 0.789 -> 0.721 over the 6,000 after) and so tracks the reward rather than the constraint. Whether the better reward distribution amounts to a better binder needs its own evaluation, which is outside this paper's scope. Intent and caveat: App. D and App. E.9.

New negative result: three of the nine Riptides targets fail outright - no pretrained sample there clears r_0, so the advantage has nothing to separate (App. E.5).

A short note to flag that the run we mentioned in our reply has finished, since it bears on two of your points.

  • W4, Q4, "the framework components are never ablated". The factored importance ratio is now ablated against a single joint-likelihood ratio, and the comparison is in Appendix C of the revision with a table and a figure. It is one target, so it does not close the question, but it is no longer only an argument.
  • Q5, likelihood diagnostics. The same appendix plots the clip fraction over training for both runs, per weight-update loop and per modality.

The numbers, and the caveat that our hyperparameters were tuned for the factored run, are in the general follow-up comment on this submission. If anything there raises further questions, we would be glad to answer them in this thread.

Thank you for the thorough response and new experiments.

I appreciate you admitting the binding affinity claim wasn't supported and narrowing your focus to physical validity instead. Your expanded tests and oversampling comparisons also successfully proved that reinforcement learning finds valid structures faster than random sampling, which resolves some of my main baseline concerns.

However, my core scientific doubts remain:

Poor realistic scores: Docking scores are still weak, and realistic energy tests (relaxed interface energy) show no improvement.

Failures and gaming the system: Your larger tests revealed major failures on several targets and signs of the model gaming the reward system (reward hacking).

While I appreciate these improvements, there still isn't enough evidence to raise my score, so my recommendation remains unchanged.

Thank you for reading our response so carefully and for engaging with the new experiments. We are grateful that you found the narrowed claim and the oversampling comparison convincing, and we take your remaining concerns — the absolute docking scores, the relaxed interface energy, the failures on several targets and the signs of reward hacking — as a fair reading of what our current evidence supports. They are stated as limitations in the revision, and we respect your assessment rather than arguing them further here.

One addition since your comment: the joint-likelihood-ratio ablation, which our earlier follow-up reported on 7YC2 alone, finished in time on a second target (1SSH, a PepFlow-split target evaluated with the other reward version), so Appendix C now covers both. We added it so that the ablation does not rest on a single target; the numbers are in our general follow-up comment on this submission.

Thank you again for the time and care you gave this paper across both rounds — your review has made it a better and more honest piece of work.

Reviewer_TL8hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies GRPO to a PepFlow-style multi-modal flow matching model for receptor-conditioned peptide binder design, where the state lives on a product of R3, SO(3), a torus and a sequence logit space. Three obstacles are addressed: the deterministic ODE sampler has a degenerate policy density, so time-decaying isotropic Gaussian noise is injected in each manifold's tangent space at every Euler step and pushed onto the manifold by the exponential map, decaying to zero so the sampler becomes deterministic at the terminal time; treating the joint log-likelihood as one scalar is unstable across modalities of very different dimension, so importance ratios are factored per modality and per residue from the normalized tangent-space noise and combined with PPO clipping and a per-modality analytic KL to the frozen reference policy; and the reward is undefined for self-intersecting structures, so a Vinardo docking score is spliced with an exponentially saturating clash penalty.

An architectural change enables this. Instead of PepFlow's single shared torsion vector plus discrete amino-acid sampling at each step, the model predicts 20 by 5 torsions per residue and aggregates side-chain features as a probability-weighted expectation over types, so every generation step admits a closed-form Gaussian tangent-space log-likelihood.

Four complexes are fine-tuned independently. Clash-free rates rise from 39.1, 48.4, 17.2 and 3.1 percent to 79.7, 70.3, 60.9 and 10.9 percent, and median rewards improve on all four. Unrelaxed Rosetta binding free energy improves on all four, while the relaxed binding free energy is reported as showing limited improvement. Ablations cover the cross-entropy versus MSE sequence parameterization and the advantage-normalization and reward-rescaling switch.

Critical strengths
  • Projecting tangent-space Gaussians through the exponential map keeps the kernel Gaussian, so the normalization cancels in the ratio and the KL reduces to a squared logarithmic map between means (L703-740).
  • Per-modality factoring is the right call when the logit space contributes 20 dimensions per residue and the torus one per angle. Predicting all 20 by 5 torsions removes the discrete sampling step that breaks the closed-form density, and is independently justified by cross-type torsion confusion (L148-168).
  • Negative results are reported rather than buried: relaxed binding energy worsens on two targets and does not correlate with the reward (L1562-1568).
  • The paper also states that rewards plateau below high-affinity levels (L425-429) and that the unused cross-entropy initialization outperforms the one the main results use (L776-786).
Critical weaknesses
  • The reward's clash branch lies between -30 and -20, so the recurring pretrained median of -23.93 is a penalty value and "median reward -23.9 to -0.4" (L041-044) restates that the clash-free fraction crossed 50 percent. Unrelaxed binding free energies of 1691 to 17 kcal/mol are steric-repulsion readouts, and the endpoint remains thermodynamically unfavorable.
  • The relaxed metric regresses on half the targets with no correlation to the reward (L1562-1568), contradicting the claim that Rosetta confirms improved affinity on every target (L080-084). Binding-site ratio drops from 0.976 to 0.717 and 0.718 to 0.628, which the ablation attributes to exploration away from the reference pose (L800-802).
  • The only evidence against pocket escape is two hand-picked structures per target, which the authors themselves call qualitative (L411-415).
  • The sole baseline is the model's own initialization, chosen per target as the highest clash-free checkpoint (L346-348); no matched-budget best-of-N, repacking, or RL baseline is run.
  • Four documented per-target deviations plus a per-target reward-rescaling switch (L804-848) mean nothing is held out across the four targets.
Questions for authors
  • Please report, per target, the Vinardo score distribution restricted to clash-free samples before and after RL, with sample counts. Without it there is no way to separate "more samples became scorable" from "scorable samples score better", which is the claim made at L418-429.
  • Can you add an ensemble-level pocket-engagement statistic over all 64 samples, such as buried surface area or receptor contact count, for pretrained versus fine-tuned? The binding-site-ratio drops and the finding at L800-802 mean the two-structure analysis at L411-415 cannot rule out partial pocket vacation.
  • What clash-free rate and median reward does best-of-N sampling from the pretrained model reach under the same number of reward evaluations consumed by RL, and what does a restrained side-chain repack of pretrained samples achieve?
  • The importance-ratio exponent divides a difference of squared tangent-vector norms by twice the squared noise scale, which diverges as that scale goes to zero unless the means coincide at the same rate; the stability argument at L712-716 does not establish boundedness. What fraction of per-residue ratios hit the PPO clip, and how is the learning signal distributed over the trajectory?
  • The torsion ratio weights per-type squared-norm differences by the predicted amino-acid probability (L242-248), but the kernel updates all 100 angles jointly, so the exact density ratio is the unweighted masked sum. Is the weighted form exact for some kernel, or a heuristic, and if a heuristic does the PPO surrogate still bound anything for that modality?
  • The cross-entropy initialization reaches a higher reward median and clash-free rate than MSE at the same RL step count while also improving RMSD (L776-786). Why were the main results run from the MSE initialization?
  • Your pretrained model reaches AAR 0.286 against 0.490 for the released PepFlow checkpoint and 15 to 16 percent clash-free against 27.1 percent (L337-348). Does operating on a materially weaker generator limit what the RL result implies?
3 comments

Response to Reviewer TL8h

Thank you - you were right on the central point: we claimed affinity where the data supports only physical validity. The claim is retracted throughout and most of the experiments you asked for are in the revision, summarized in our general response.

W1, W2 (reward floor; relaxed metric). Agreed; the paper now says it: the median crossing r_0 is about physical validity, an unrelaxed dG_bind of 10^3 kcal/mol is a repulsion readout, and the relaxed energy does not improve (Sec. 5.2, App. E.12). On scope: we deliberately score the unrelaxed, unrepacked output, since relaxation would collapse distinct samples onto one minimum and flatten the reward where the advantage must separate them (Sec. 4.4). Absolute scores are pessimistic as a result; a post-relaxation energy is a different objective, outside our scope.

Q1 (Vinardo restricted to clash-free, with n). Table 9 and App. Fig. 4. Medians, pretrained -> RL:

TargetVinardo median (kcal/mol)n
1SSH+9.16 -> +0.05311 -> 817
2R02+7.08 -> +1.12375 -> 761
5UZW+10.36 -> +2.42135 -> 630
6V63+14.58 -> +11.0019 -> 68

So it is not only that more became scorable. But the best single sample over 1,024 draws is -1.4 kcal/mol (-2.1 over all 13): suppression of high-energy conformations, not binding.

W3, Q2 (pocket engagement at the ensemble level). App. E.9, over 1,024 samples. Of the ten targets whose reward carries signal - the other three never leave the penalty branch - BSR falls on nine, but never approaches zero: the lowest post-RL value over all thirteen is 0.569, and RMSD does not grow with the drop. The targets with no reward signal drift far less, so the fall is reward-driven. We read it as the ensemble moving off the crystallographic contact set rather than out of the pocket; the paper notes that BSR cannot exclude relocation within the pocket.

Q3 (best-of-N). Sec. 5.4, App. Fig. 5 and Table 11: how many of 32,768 pretrained draws best-of-N needs to match a quantile the checkpoint reaches. Final checkpoints, median (90th percentile): 1SSH 137.3 [106.8, 177.1] (138.0), 5UZW 27.0 [19.2, 39.3] (313.8), 2R02 13.0 [10.6, 15.4] (38.7), 7YC2 257.8 [164.2, 398.1] (1150.8). Two stay near the baseline: 6V63 (3.1) and 8C1L (2.3), already 90.7% clash-free. We did not run restrained repacking, for the reason in W1/W2.

Q4 (ratio divergence; clip fraction). The divergence belongs to the continuous schedule, not to what we run: sampling uses a fixed grid of N = 100 steps with sigma taken at each interval's left endpoint, so its argument never exceeds t = 0.99 and the exponent stays bounded (App. A.2). We monitor the clip fraction as a batch average per update rather than resolved by t: it sits at roughly 5-35% by modality, so clipping does not dominate the updates and training is stable.

Q5 (torsion ratio). It is an approximation and the revision says so (App. A.2.1). We weight by p_t(a) because the side-chain feature already enters weighted by p_t(a): torsions of a low-probability type barely affect the prediction and get the least supervision, so unweighted they dominate the exponent and the modality is almost always clipped. The cost is stated too: the PPO surrogate bounds nothing there, while the other three ratios and the KL stay exact.

Q6 (why MSE). Only MSE defines a flow-matching velocity: its target is a finite point in logit space, whereas CE's optimum lies at infinity, so a velocity must be manufactured by rescaling the softmax. The PepFlow paper specifies MSE while its released code trains with CE, so we ran both: they pretrain to comparable quality (AAR 0.286 for MSE vs 0.280 for CE), so we keep MSE, the one consistent with the rest of the model (App. B.3, App. C).

Q7 (weaker generator). Table 13 now includes the released PepFlow checkpoint: much the best on 1SSH (78.1% clash-free), collapsing on 2R02 (6.3%) and 6V63 (0.0%). Against the matched retrained run ours is ahead on 1SSH and 2R02, behind on 5UZW, level on 6V63. Read against Table 1, the fine-tuned checkpoints exceed the released one on all four in clash-free rate and R_95% from a weaker start - but it is not a controlled comparison, and our method cannot be applied to the released weights (App. E.7). The weak base model is now an explicit limitation.

W4 (nothing held out). The nine Riptides targets run under one shared configuration with no per-target tuning; six of nine improve on both measures, and three fail because no pretrained sample there clears r_0 (App. E.5). On "no RL baseline": DDPO, DPOK, Flow-GRPO and Adjoint Matching are all formulated for a single Euclidean state space, and DRaFT additionally needs a differentiable reward, so none runs unmodified on a state in R^3 x SO(3) x torus x logit space with an external scorer. The closest in-scope baseline is our own construction with the four ratios collapsed into one - the run now in progress.

A short note to flag that the run we mentioned in our reply has finished, since it bears on two of your points.

  • W4, "no RL baseline". The in-scope baseline we described - our own construction with the four per-modality ratios collapsed into a single joint-likelihood ratio - is done and is now in Appendix C of the revision, with a table and a figure.
  • Q4, clip fraction. The same appendix plots the clip fraction over training for both runs, per weight-update loop: the four factored ratios separately and the joint ratio.

The numbers, and our reading of them, are in the general follow-up comment on this submission. If anything there raises further questions, we would be glad to answer them in this thread.

The joint-ratio ablation has now finished on a second target, 1SSH, and Appendix C has been extended to cover both targets. The results are in our general follow-up comment on this submission. If anything there raises further questions, we would be glad to answer them in this thread.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

General Response to All Reviewers

We thank the reviewers for the careful reading. Two criticisms go to the substance of the paper and we accept both: the affinity claim was not supported by our own data, and four targets at 64 samples each were too thin a basis for the conclusions. The revision retracts the first and fixes the second. This comment summarizes the changes; the per-reviewer replies answer individual points.

What changed in the revision

1. The claim. We have withdrawn the binding-affinity claim from the abstract, introduction, discussion and conclusion, and reframed the paper as a proof of concept: GRPO is practicable on the product of manifolds representing peptide structure, and a physics-based reward moves the generated distribution in a physically meaningful direction. The Rosetta re-scoring is now presented as a consistency check between two unrelaxed scores, not as validation (App. E.12).

2. Evaluation. Every checkpoint was re-evaluated with 1,024 samples instead of 64. Clash-free is now defined as reward > r_0 = -20, i.e. the reward is the Vinardo score itself; the old definition (zero clashing heavy-atom pairs) also counted unevaluable and clipped samples.

3. Benchmark. Nine targets from the Vilya Riptides benchmark were added - the entries our model can represent, deposited after the PepFlow dataset (June 2023 to January 2024), hence a temporal hold-out. All nine were fine-tuned under a single shared configuration with no per-target tuning; thirteen targets in total.

4. New analyses.

  • Vinardo score restricted to the clash-free samples, separating "more samples became scorable" from "scorable samples score better" (Table 9, App. Fig. 4).
  • An equivalent best-of-N baseline against 32,768 pretrained samples per target (Sec. 5.4, App. Fig. 5, Table 11).
  • Pocket engagement (BSR) over all 1,024 samples, including what the KL coefficient does and does not explain (App. E.9).
  • Why three Riptides targets fail outright, with an enclosure measure that predicts the pretrained clash-free rate across targets (App. E.5).
  • A comparison against the released and retrained PepFlow checkpoints (Table 13, App. E.7).

5. Presentation. Appendix B is new: a self-contained account of every difference from PepFlow and why each is needed, indexed by Table 2. The implementation-derived names (pepflowww_orig and the like) and the code-style typesetting are gone. Table 1 now reports all thirteen targets, and the figures are consolidated into one reward box plot in the main text with curves and histograms in the appendix. Section 5 now reads as one argument, and the abstract was shortened.

6. What we now state explicitly. The reward scores the all-atom structure reconstructed from the generated frames and torsions, with no relaxation or repacking; this is deliberate and one reason the scores are pessimistic (Sec. 4.4). The torsion importance ratio is an approximation rather than an exact density ratio, with the reason and the cost stated (App. A.2.1). The limitations now include the weak base model, the three targets with no reward signal, and the precision of thirteen single runs.

What the revision shows

Physical validity improves wherever the reward is well-posed: the clash-free fraction rises on all four PepFlow-split targets (1SSH 30.4 -> 79.8%, 2R02 36.6 -> 74.3%, 5UZW 13.2 -> 61.5%, 6V63 1.9 -> 6.6%) and on six of nine Riptides targets (e.g. 7YC2 10.6 -> 68.5%, 8GW4 16.5 -> 56.9%). The already-scorable samples improve as well - on 1SSH the median Vinardo score of the clash-free samples moves from +9.16 to +0.05 kcal/mol - so the gain is not an artifact of more samples becoming evaluable. Measured against oversampling: to match the median reward of a fine-tuned ensemble, best-of-N on the pretrained model needs 137.3 [106.8, 177.1] draws on 1SSH and 257.8 [164.2, 398.1] on 7YC2 (95% bootstrap interval), 10 to 10^3 across the targets and more in the tail.

The ensemble pays little for this: bond lengths stay near the natural 1.33 A, diversity is almost unchanged (0.679 -> 0.687 on 1SSH), and the binding-site ratio never approaches zero (lowest post-RL value 0.569). What does not improve is affinity: the scores plateau near 0 kcal/mol and the best single sample over 1,024 draws reaches only -2.1 kcal/mol, far from the -8 to -12 kcal/mol of confirmed binders. On three Riptides targets the pretrained model produces no scorable sample at all, so RL has nothing to work with there. We therefore claim improved physical plausibility and reward statistics, not affinity.

Still open

An ablation that replaces the four per-modality importance ratios with a single joint-likelihood ratio is running; if it finishes during the discussion period we will report it.

Code. The submitted supplementary covers the original version; the scripts and configurations added for this revision will be released publicly on acceptance, as the Reproducibility Statement says.

AuthorsOpenReview ↗

Follow-up to all reviewers: the joint-ratio experiment

In our replies we said that a run replacing the four per-modality importance ratios with a single joint-likelihood ratio was under way, and that we would report it if it finished during the discussion period. It has finished, and the revised paper now carries it in Appendix C, with a table and a figure.

Setup. The four modalities' likelihood ratios are multiplied into one ratio per residue, so the advantage contributes a single term per residue and the PPO clipping acts on all four modalities at once. The torsion factor keeps the probability-weighted heuristic of App. A.2.1. The per-modality loss weights have no counterpart in this form - weighting the log-ratios rescales the ratio itself rather than reweighting a loss - so the joint run uses none. Hyperparameters, schedule and the 1,024-sample evaluation are otherwise identical. Target 7YC2, 6,000 steps.

Reward. The two are close. At 6,000 steps the joint run moves slightly more samples across r_0 (43.8% vs 39.6% clash-free), while the factored run is ahead in the upper tail and in score quality: R_95% -1.78 vs -2.38, median Vinardo over the clash-free samples +6.76 vs +7.71 kcal/mol, and an equivalent best-of-N at the 90th percentile of 30.5 vs 25.9 pretrained draws.

Clip fraction. Here the two separate. Over the second half of training the factored ratios sit at a median of 6.9-11.0% per modality, with 90th percentiles of 10.2-26.1% (the torsion factor highest, as its dimension would suggest). The joint ratio has a median of 14.6%, a 90th percentile of 44.0% and a maximum of 54.7%: once any one modality leaves the trust region, the whole residue is clipped. The appendix figure plots this per weight-update loop for both runs.

What we read into it. The comparison is not neutral, and we say so in the paper: our hyperparameters were chosen so that the factored clip fractions stay in a workable range, which disadvantages the joint form. A joint run with a shorter update interval, a larger batch or a smaller learning rate would clip less and might reach a better reward, at the cost of sample efficiency. We therefore do not claim that the joint estimator is worse - as an estimator of the policy gradient it is the more faithful one. What the comparison supports is the practical argument of Sec. 4.2: with factored ratios the objective can carry per-modality weights, as pretraining does, and the clip fraction can be controlled per modality instead of being set by whichever factor is least stable.

One housekeeping note. The appendix additions shifted some numbers after our earlier replies. Tables 1 and 2 are unchanged; Table 9 is now Table 10, Table 11 is now Table 12, and Table 13 is now Table 14. App. Fig. 4 (Vinardo of the clash-free samples) is now App. Fig. 5, and App. Fig. 5 (equivalent best-of-N) is now App. Fig. 6. Section and appendix numbers are unchanged.

AuthorsOpenReview ↗

Follow-up to all reviewers: the joint-ratio experiment on a second target

Our earlier follow-up reported the joint-likelihood-ratio run on 7YC2. The same ablation has now finished on 1SSH, a PepFlow-split target with a different reward version, and Appendix C of the revision has been extended to both: the table carries all four runs, and the clip-fraction figure now has one column per target.

1SSH, at 6,000 steps (factored vs joint). Clash-free 60.1% vs 54.2%, R_95% -0.17 vs -0.30, median Vinardo over the clash-free samples +4.79 vs +4.61 kcal/mol, equivalent best-of-N at the 90th percentile 12.9 vs 9.5 pretrained draws. The clip fraction over the second half of training is 7.0-9.8% at the median per factored modality against 14.7% for the joint ratio, whose 90th percentile is 44.9% and whose maximum is 53.9%.

What the second target changes. Nothing in the conclusion, and it sharpens one part of it. The reward distributions remain close, with neither construction dominating: on 1SSH the factored run leads on the clash-free fraction and on the tail while the joint run has the marginally better restricted Vinardo median, and on 7YC2 the first two go the other way. What is consistent across both targets is the equivalent best-of-N, which favours the factored run (12.9 vs 9.5 on 1SSH, 30.5 vs 25.9 on 7YC2), and the clip fraction, where the joint ratio sits at roughly twice the factored median and peaks above 50% on both. The caveat from the earlier comment still stands: our hyperparameters were tuned so that the factored clip fractions stay workable, so this is evidence for the practical convenience of per-modality control rather than a ranking of the two estimators.

The table and the figure we pointed to for 7YC2 keep their numbers, Table 4 and App. Fig. 2; both now cover 1SSH as well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.