acceptodds
Withdrawn from ICLR 2027

Controlling Property Trade-Offs in Reward-Alignment of 3D Molecular Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Structure-based drug design requires balancing predicted binding affinity, synthetic accessibility, and molecular validity. However, models trained to reproduce the distribution of training ligands do not necessarily satisfy these objectives jointly. We adapt Flow-GRPO to a pretrained, pocket-conditioned 3D ligand generative model to jointly optimize multiple molecular rewards such as Vina docking and synthetic accessibility scores. Optimizing either objective in isolation improves the target metric at the expense of the other. By varying their respective reward weights, we identify the trade-offs between these conflicting objectives and relate them to the composition, topology, and flexibility of generated ligands. We find that, depending on the reward composition, the aligned model learns to balance these ligand descriptors to maximize the reward. This insight allows us to control the aligned models’ trade-off between Vina docking score and synthetic accessibility, with multiple configurations improving both over the baseline model, while also increasing drug-likeness and maintaining PoseBusters validity. Our results connect the trade-offs between objectives to interpretable changes in generated ligands’ chemistry, providing a basis for understanding and selecting useful compromises across protein targets.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
3D Molecule generation, Reinforcement learning, Multi-objective alignment, GRPO, Molecular validity, Property Trade-offs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pZyFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the multi-objective trade-off between binding affinity and synthetic accessibility in Structure-Based Drug Design (SBDD).

The authors point out that pretrained 3D molecular generation models mainly learn the distribution of training data and do not necessarily satisfy multiple target properties simultaneously. In addition, during reinforcement learning post-training, directly dropping invalid molecules may lead to a decrease in the valid molecule generation rate, while different reward objectives may also compete with each other. To address these issues, the authors adapt the existing Flow-GRPO method to the pretrained 3D molecular generation model FLOWR.ROOT. By penalizing invalid generation trajectories and adjusting the reward weights of different properties, including Vina, SA, and QED, they achieve post-training alignment for multi-objective molecular generation.

The authors conduct post-training and evaluation on the SPINDR dataset and test cross-dataset transfer performance on CrossDocked2020. The experimental results show that adjusting reward weights can change the property preferences of generated molecules. Some configurations can simultaneously improve Vina, SA, and QED while maintaining a high molecular validity rate. In addition, by analyzing changes in molecular descriptors such as rotatable bonds, ring structures, stereocenters, and logP, the authors investigate the statistical correlations between reward configurations and changes in molecular structures and properties.

Critical strengths

S1. The paper identifies and effectively mitigates the decline in molecular validity during reinforcement learning post-training.

The authors compare two approaches: directly dropping invalid generation trajectories and assigning low rewards to them. The experiments show that the latter can better maintain the valid molecule generation rate of the pretrained model while improving the target properties. This reveals an important issue in molecular reward alignment: optimizing properties only among valid generated samples may lead to a decline in overall generation capability.

S2. The paper provides relatively systematic multi-objective reward-weight experiments and cross-dataset validation.

The authors conduct systematic experiments with different Vina–SA reward weights and validate the corresponding trends across multiple training seeds and two datasets, SPINDR and CrossDocked2020. The experiments show that different reward configurations can produce different molecular property preferences, and some configurations can simultaneously improve multiple objectives relative to the pretrained baseline. In addition, the authors analyze changes in the structural and physicochemical characteristics of generated molecules, providing some empirical evidence for understanding the behavior of reward alignment.

Critical weaknesses

W1. The methodological novelty is relatively limited, and the paper does not sufficiently demonstrate substantive differences from existing multi-objective optimization methods.

The core framework of this paper is mainly built upon the existing FLOWR.ROOT and Flow-GRPO methods, while multi-objective property control is primarily achieved by adjusting linearly weighted rewards. Although this combination has practical value, the paper lacks a new multi-objective optimization algorithm, an adaptive reward control strategy, or other clearly distinctive methodological designs.

Therefore, the main contribution of the current work is closer to an application and empirical study of existing reinforcement learning methods in 3D molecular generation, rather than the proposal of a new multi-objective control mechanism.

Although the authors provide comparison results with methods such as MolJO, MolForm-DPO, and PILOT in the appendix, these methods differ in training data, generative backbones, and optimization objectives. Therefore, these comparisons are insufficient to demonstrate the advantages of Flow-GRPO itself.

A more convincing validation would compare other reasonable post-training or reward optimization strategies, such as reward-weighted fine-tuning, rejection sampling, or other applicable preference alignment methods, under the same pretrained backbone, training data, and comparable computational budgets. Such experiments would help clarify the unique value and necessity of the current framework compared with existing methods.

W2. The mechanistic analysis of molecular structures and property trade-offs mainly remains at the level of statistical correlations.

In Figure 4, Figure 5, and the appendix, the authors analyze the relationships between molecular descriptors, including rotatable bonds, ring structures, stereocenters, and logP, and improvements in Vina, SA, and QED.

These analyses can demonstrate certain statistical correlations between reward alignment and molecular structural changes. However, they lack targeted intervention experiments or mechanistic ablations, making it difficult to further verify which structural changes actually drive the trade-offs between different objectives.

In addition, some observed relationships are consistent with the existing preferences of scoring functions such as Vina and SA for molecular flexibility or structural complexity. Therefore, the extent to which these statistical results constitute new scientific findings still requires further discussion.

If the authors could use more targeted controlled experiments to distinguish the effects of molecular descriptors from the direct preferences of the scoring functions, it would significantly strengthen the persuasiveness of their mechanistic analysis.

Questions for authors

Q1. Has the Trade-off Frontier in Figure 3 been sufficiently validated?

Figure 3 presents an empirical trade-off curve by connecting the average Vina–SA results corresponding to different reward weights. However, some reward configurations outperform others in both reported average metrics. For example, on SPINDR, the configuration with w_Vina = 0.25 achieves better average Vina and SA scores than the configuration with w_Vina = 0. This means that, based on the reported mean values, not all points lie on a Pareto-optimal frontier in the strict sense.

Could the authors further analyze the Pareto efficiency of different reward configurations while considering the uncertainty introduced by random seeds? In addition, can these results support a stable and predictable relationship between reward weights and final molecular properties, rather than merely providing several empirical operating points obtained by training with different weights?

Q2. How do the authors explain the deterioration of some molecular geometric distribution metrics after reward alignment?

Although some reward configurations can simultaneously improve Vina, SA, and QED, Table 1 and Appendix Table 5 show that the Wasserstein distances of bond-angle and bond-length distributions deteriorate to varying degrees after post-training compared with the pretrained model.

This suggests that property reward optimization may also impair some of the original generative model's ability to match geometric distributions, while this additional trade-off has not been sufficiently discussed.

Could the authors further explain the sources of these distributional changes and their relationship with PoseBusters validity? In particular, do these observations suggest that the current Vina–SA trade-off analysis does not fully reflect the effects of reward alignment on overall molecular quality?

Q3. Could the probability calculation and normalization of the mixed continuous-discrete policy be further clarified?

Appendix B.2 normalizes the log-likelihoods of different molecular modalities by the number of atoms or valid atom pairs to reduce the influence of molecular size on policy updates. However, the normalized likelihood ratio is not exactly equivalent to the importance ratio of the original joint transition distribution.

Could the authors further explain the influence of this modification on the GRPO optimization objective and the rationale for choosing the current normalization approach?

In addition, the stochastic coordinate sampling in Appendix A becomes deterministic when t >= 0.9, while Appendix B still uses Gaussian transition likelihood and KL formulas. Could the authors clarify how zero-variance coordinate transitions are handled at this stage in the actual implementation to ensure the numerical stability of the probability calculations?

Reviewer_YzxBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts Flow-GRPO to post-train FLOWR.ROOT, a pocket-conditioned 3D molecular generator, on SPINDR. It sweeps Vina/SA reward weights, compares invalid-rollout penalization versus dropping, and analyzes changes in molecular descriptors after alignment. Zero-shot transfer to CrossDocked2020 is also reported.

Critical strengths
  1. Invalid-rollout handling is practically useful. The paper shows that dropping invalid samples substantially reduces RDKit-valid yield, whereas penalizing them preserves the pretrained level. This is a concrete and transferable finding for online RL fine-tuning of molecular generators. Multiple seeds and two datasets yield a controllable trade-off frontier. Comparisons against the same-dataset pretrained FLOWR.ROOT are fair and credible.

  2. The matched-pocket descriptor analysis adds interpretability. By controlling for pocket identity, the correlation analysis relates alignment-induced descriptor shifts to score changes more carefully than raw aggregate comparisons. The mixed continuous–discrete likelihood and KL derivations in the appendix are also reasonably complete.

Critical weaknesses
  1. Limited algorithmic novelty. The core contribution is an application of an existing method (Flow-GRPO) plus reward-weight tuning. Moreover, Vina and SA are proxy metrics, not gold standards, so in my opinion, when new objectives are introduced, reward weights and clipping thresholds will likely need retuning. The paper should position itself as a controlled empirical study of online post-training behavior rather than as a new method.

  2. Cross-dataset baseline comparison is not well controlled. MolJO and MolForm-DPO in Tables 3 and 5 are trained/aligned on CrossDocked2020, whereas Flow-GRPO is fine-tuned on SPINDR. Side-by-side presentation is potentially misleading even if labeled as reference points. These comparisons should be explicitly downgraded to non-comparable references, and all superiority claims should be restricted to comparisons against the same-dataset pretrained FLOWR.ROOT.

Questions for authors

Refer to the weakness.

Reviewer_TcodOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts Flow-GRPO to FLOWR.ROOT, a pocket-conditioned 3D ligand generator. It optimizes Vina docking score, synthetic accessibility (SA) and QED, alone and in weighted combinations. There are three results. Penalizing invalid rollouts preserves the valid-molecule yield, and masking them makes it collapse. The reward weights trace a controllable Vina-SA frontier. The trade-off maps onto interpretable ligand descriptors: rotatable bonds, rings, stereocenters and logP.

Critical strengths
  • The evaluation hygiene is well above average. Pocket-matched paired t-tests with p-values, a pose-optimized Vina cross-check, PoseBusters physical validity, bond-length and bond-angle Wasserstein distances, diversity and uniqueness. Many alignment papers report one metric and stop.
  • Zero-shot transfer is tested. The SPINDR-selected checkpoints are evaluated on CrossDocked2020 with no refitting and no new checkpoint selection.
  • Several configurations beat the baseline on more than one axis at the same time. At wVina = 0.75: Vina −7.35 to −7.92, SA 0.70 to 0.72, QED 0.48 to 0.53, and PB validity held at 0.96.
  • Diversity and uniqueness rose in every configuration tested, 0.53 to 0.56–0.60 and 0.91 to 0.92–0.94. This is a useful counterpoint to the usual mode-collapse worry about RL fine-tuning.
  • The limitations section is honest, and the paper states plainly that no single multi-objective configuration dominates all metrics.
Critical weaknesses
  • I have some concerns regarding the novelty. I checked the source method. Flow-GRPO (Liu et al., 2026) is text-to-image on SD3.5 and works on continuous latents. This means that the ODE-to-SDE conversion is Flow-GRPO's contribution and is inherited here, but §3.1 and Appendix A read as if it is part of the adaptation.

  • The mixed continuous-discrete transition likelihood is the genuine extension. Also, image latents have no categorical part. Invalid rollouts also cannot happen in image generation, because every latent decodes to some image, so the invalid-handling work solves a problem the source method never had. -As written, "we adapt Flow-GRPO" undersells the likelihood work and oversells everything around it. Please add one paragraph that separates what Flow-GRPO provides from what this paper adds. Credit the ODE-to-SDE conversion to Liu et al., and put the mixed-type likelihood and the invalid-rollout handling forward as your own.

  • The oracle should not also be the metric. Uni-Dock score-only Vina is the training reward. The "Vina min." column is the same docking engine run with pose optimization, so it is not an independent check. Docking scores correlate only loosely with measured affinity. PoseBusters validity is a genuinely independent axis and a real strength of the paper, but it measures physical plausibility, not binding. This should be discussed in the paper.

  • GRPO exists to avoid training a critic, and it substitutes a group-relative advantage. The word "critic" does not appear in the paper, and no actor-critic or PPO-with-critic baseline is run. So the paper cannot show that the GRPO mechanism, and not online RL in general, is what works here. Please add an actor-critic ablation.

  • Single backbone. Only FLOWR.ROOT is used, and §5 acknowledges this. It still means that none of the findings are shown to be properties of the method rather than of this one backbone. Please add a second backbone, for example a diffusion model such as TargetDiff or DiffSBDD.

  • "Controllable frontier" and "no configuration dominates all metrics" are standard multi-objective statements, and standard quantitative tools exist for them. Please report Pareto dominance counts or hypervolume for the weight sweep.

  • V_g and V_G are used inconsistently around Eq. 4. Please fix this.

Questions for authors

How does your approach differ from Graph-GRPO, which uses analytical transition probabilities and may not require injected noise?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.