- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces the FlowR.NFT, which incorporates DiffusionNFT method to optimize Flowr.root model with binding affinity, drug-likeness, and synthetic-accessibility score. The major objective of this work is addressing the validity collapse problem and find the reward compositions for Flowr.root finetuning. In experiments, authors evaluate the proposed method on various reward compositions and compare it with various diffusion and flow-based methods.
Critical strengths
None
Critical weaknesses
- The primary motivation of this paper is the validity collapse during optimization. However, it is not validated that this validity collapse issue is raised not only Flowr.root but also various diffusion-based methods. The authors' experiments on Flowr.root is the sole evidence of this claim. In this paper, there is no additional reference which addresses or measures the validity collapse issue of diffusion-based sbdd methods. In particular, in Table 1, MolJo and AliDiff show higher validity than
Flowr.NFT+Vina. This indicates that they do not suffer the same issue which author observed in Flowr.root. - The raw Vina scores are poor. The main rationale for developing 3D pocket-based molecule generators is the understanding that protein-ligand interactions are crucial for target-aware drug design—specifically regarding target specificity. If the generator does not aim to produce feasible binding conformations, then 3DSynthFlow [1]—which jointly generate 2D molecular graph with its 3D binding pose —demonstrates superior performance compared to most baselines reported in this paper (Vina: -9.42 kcal/mol; QED: 0.73; AiZynthFinder retrosynthesis success rate: 36%). Furthermore, compared to MolJO, the difference in redocked Vina scores is smaller than the reported standard deviation (-9.10 ± 0.09 kcal/mol vs. -9.07 kcal/mol), and the QED score is slightly lower (0.52 vs. 0.55). Considering the trade-off between QED and Vina docking scores, it is difficult to claim that the proposed method is superior to MolJO.
References:
- Shen, Tony, et al. "Compositional flows for 3D molecule and synthesis pathway co-design." ICML 2025.
Questions for authors
- Is this issue of 'validity collapse' a phenomenon commonly observed across various diffusion models? If this is a general problem, can the proposed method resolve it for other models?
- Do the trained FlowR.NFT models retain the target specificity? This can be evaluated via Delta score [1]
- Please explain the reason why the
Vinascore of FlowR.NFT is poor compared to other baselines? - Please report the protein-ligand interaction profile using PoseCheck [2] -- the number of H-Bond and Van der Waals interaction -- and other drug-like properties, such as molecular weight, number of heavy atoms, number of rings, and number of aromatic rings, and compare the metric with baseline models. It is important since it is easy to hack the docking score by increasing the number of atoms or hydrophobic groups [3].
References:
- Gao, Bowen, et al. "Rethinking specificity in sbdd: Leveraging delta score and energy-guided diffusion." ICML 2024.
- Harris, Charles, et al. "Posecheck: Generative models for 3d structure-based drug design produce unrealistic poses." (2023).
- Qu, Yanru, et al. "MolCRAFT: structure-based drug design in continuous parameter space." ICML 2024.
5 comments
Potential exploitation of the Vina reward: We agree that docking improvements should be assessed alongside molecular composition and protein–ligand interactions. Our sampling protocol retains the same atom-count prior, and the observed heavy-atom distributions remain unchanged after fine-tuning. However, optimisation with Vina reward increases ring counts, and PoseCheck results show fewer hydrogen-bond interactions than the pretrained model, while van der Waals contact counts remain comparable or higher (using redocked poses for docked-reward model due to poor original-pose validity). Further, FlowR.NFT-multi with score-only Vina reward shows higher hydrogen-bond and VdW contact counts and lower aromatic-ring counts.
FlowR.NFT optimises the supplied reward and therefore inherits its limitations. The framework can accommodate alternative affinity estimators without requiring differentiable rewards. Evaluating whether estimators better correlated with experimental affinity reduce these biases is an important direction for future work. We will include the descriptor and interaction comparisons in the manuscript.
| Model | PC-HBAcceptor | PC-HBDonor | PC-VdWContact | Heavy atoms | Rings | Aromatic rings |
|---|---|---|---|---|---|---|
| MolJO | 0.54 | 0.19 | 4.41 | 22.92 | 3.88 | 2.26 |
| Flowr.root | 1.11 | 0.41 | 5.41 | 22.60 | 2.73 | 1.69 |
| FlowR.NFT-multi (docked Vina) | 0.20 ± 0.02 | 0.08 ± 0.00 | 5.03 ± 0.12 | 22.61 ± 0.00 | 4.88 ± 0.22 | 2.30 ± 0.30 |
| FlowR.NFT-multi (score-only Vina) | 0.37 ± 0.02 | 0.19 ± 0.00 | 6.31 ± 0.01 | 22.60 ± 0.01 | 4.78 ± 0.15 | 0.97 ± 0.16 |
Thank you for the clarifications and additional experiments. I still have the following concerns.
-
Trade-offs relative to pretrained Flowr.root. The score-only-trained FlowR.NFT-multi improves upon the original fine-tuned model, but remains worse than pretrained Flowr.root on CrossDocked in raw Vina (−4.49 versus −5.96), intermolecular PB validity (0.80 versus 0.93), and overall PB validity (0.73 versus 0.91). Thus, even the directly optimized score-only objective does not improve over the pretrained baseline in this transfer setting. The Vina+QED+Geom configuration improves Vina to −7.15, but lowers SA from 0.71 to 0.53. Please provide the complete results, including variability, to clarify what is gained and lost through fine-tuning. Fewer H-Bond and more VdW interactions relative to the pretrained model also raise concerns about altered interaction patterns and potential false positives [1].
-
Target specificity. My question remains unanswered. Please provide Delta scores or an equivalent intended-target/off-target evaluation.
-
Clarification regarding 3DSynthFlow. I introduced this method to highlight the importance of directly generated pose quality, not to request an additional baseline. Improving molecular proposals for subsequent docking differs from improving directly generated binding poses.
References:
- Qu, Yanru, et al. "MolCRAFT: structure-based drug design in continuous parameter space." ICML 2024
We thank the reviewer for the detailed review. Our responses below include new experimental results, clarifying the findings and limitations of our approach.
Scope of Validity collapse: Our experiments establish validity collapse for Flowr.root under valid-only online NFT fine-tuning. Our experiments do not establish how common this issue is across molecular generators or alignment methods. It is, however, a common issue across online post-training methods for molecular generators that molecular rewards can be undefined on chemically invalid samples. Thus, any online fine-tuning method that encounters chemically invalid outputs must specify how to handle these undefined rewards. In contrast, MolJO uses inference-time guidance, and AliDiff uses offline preference alignment. Their validity scores therefore do not test the online fine-tuning failure mode studied here. We will clarify this scope and identify evaluation across additional generators and online alignment algorithms as future work.
Poor score-only VINA scores on Crossdocked: Our original reward uses redocked Vina, which evaluates affinity after an external pose search and therefore does not directly reward the initial ligand placement. While our models fine-tuned on SPINDR achieve good Vina score-only results and intermolecular PoseBusters validity remains at least 0.90 on the SPINDR test set (Table 3), performance on zero-shot transfer to CrossDocked might be limited by this choice.
To address this, we additionally trained FlowR.NFT using score-only Vina, which evaluates the generated pose directly instead of redocked Vina. For FlowR.NFT_multi, this improves score-only Vina from −1.14 to −4.49 kcal/mol on CrossDocked2020, intermolecular PB validity from 0.58 to 0.80, and overall PB validity from 0.56 to 0.73. However, we want to emphasize that when comparing these new results and the results presented in Table 3 to the baseline models in Table 3, one has to take into account that, unlike our FlowR.NFT, all the baselines have been finetuned on CrossDocked2020. The Vina+QED+Geom configuration achieves a stronger pose-quality trade-off on CrossDocked2020: score-only Vina reaches −7.15 kcal/mol, while intermolecular validity reaches 0.9 and redocked Vina reaches −9.10 kcal/mol, although SA is lower at 0.53.
On SPINDR, score-only Vina of FlowR.NFT_multi improves from −7.43 to −8.73 kcal/mol and intermolecular validity from 0.90 to 0.98 when using Vina score-only in the reward.
Thus, direct pose feedback can mitigate the pose deterioration on zero-shot CrossDocked transfer. We will include the full comparisons across reward configurations in the revised manuscript.
Comparison to previous methods: We thank the reviewer for highlighting 3DSynthFlow. While its reported −9.42 kcal/mol is Vina docked, rather than score-only Vina, this strong docking, QED and retrosynthesis results make it a relevant comparison, which we will discuss in the manuscript. A direct numerical ranking would require evaluation under a common protocol. We also agree that the small difference in mean redocked Vina does not establish superiority over MolJO. We will describe the original results as competitive docking performance with different molecular-property trade-offs. This comparison also involves different training distributions: FlowR.NFT is pretrained and fine-tuned only on SPINDR and evaluated on CrossDocked2020 without further training or checkpoint selection, whereas MolJO and 3DSynthFlow use CrossDocked2020 training data. Our results therefore demonstrate transfer across datasets, although this difference prevents us from attributing performance differences solely to the alignment method.