acceptodds
Under review as a conference paper at ICLR 2027

Steering Flexible-Size Molecular Generation via RL on Trans-Dimensional Flows

Abstract

Flexible-size generative models for structure-based molecular design allow size to adapt to the target objective, opening opportunities to out-of-distribution scientific discovery. However, steering such trans-dimensional processes with reinforcement learning (RL) requires accounting for actions that can change the state's dimension along the generative trajectory, such as insertions and deletions. We introduce Transdimensional Flow Policy Optimization (TFPO), formulating RL finetuning of flexible-size generative models as a parameterized-action Markov decision process, in which discrete structural decisions select an action-space component and where variables such as continuous positions and discrete atom types and bonds are sampled conditionally within it. We instantiate TFPO on Morph, a trans-dimensional 3D molecular generator and derive its corresponding per-step policy density. Across three settings, reward optimization steers both molecular properties and the model’s structural decisions. On QM9, we steer the model far beyond the pretraining size support, generating viable structures larger than the training support mean. On GEOM-Drugs, RL with verifiable rewards increases strict validity from 92.9% to 98.0% while decreasing the strain energy. For pocket-conditioned ligand generation, fine-tuning improves binding efficiency and successfully adapts ligand size to the target protein pocket.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
generative models for molecules, reinforcement learning, 3D molecular generation, reward-guided fine-tuning, flow matching, trans-dimensional generative models, structure-based drug design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jxcBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces TFPO, which applies GRPO to steer flexible-size molecular generation through a parameterized-action formulation. The method is applied to Morph, and the experiments are conducted on QM9, GEOM-Drugs, and CrossDocked2020, mainly evaluated in terms of property scores, structural quality, and diversity.

Critical strengths

The paper is generally well written and organized. The presentation is relatively clear, and the visualizations provide useful support for understanding the proposed framework and experimental results.

Critical weaknesses
  1. The novelty of this study may be limited. The work does not appear to introduce new algorithms, theories, frameworks, or insights. Instead, this work applies existing standard parameterized-action MDP and GRPO to Morph, which may be methodologically incremental. Additionally, although this is an application-driven study, it does not appear to clearly propose a new application-driven method or direction to address a specific challenging question in the field and demonstrate its necessity. Thus, the new knowledge identified from this work that can contribute to the community may be limited.

  2. Several major concerns may be raised according to the experimental evidence provided. (1) There are no key ablations to show trans-dimensional policy formulation is necessary, adaptive cardinality is effective (molecular size is not the only variable), and fixed-variance Gaussian noise design is reasonable and robust. (2) OOD experiments on QM9 may imply severe 3D generation failure. Only ~2.8% of the raw generated molecules pass all checks, while the reported result of 65% is after MMFF force-field minimization. This may mean the claimed OOD generation ability is supported only after external force-field repair and the generation of molecules with large size may not be reliable. Additionally, the reward for QM9 experiments appears to be based only on atom count. This may just encourage the model to generate larger molecules rather than OOD discovery. (3) Comparing optimization-based frameworks with amortized generative models (e.g., DiffSBDD, MolPilot) may be unfair. For the relatively fair baseline comparison (DePPA vs. the proposed method), the key metric (vina score) of the proposed method is uncompetitive. In addition, the diversity in the table is also reduced a lot and not competitive (DePPA vs. Morph vs. the proposed method: 0.78 vs. 0.58 vs. 0.41), implying potential mode collapse. Given this, it may be unclear if the proposed method is more worthwhile than the existing methods, since the computational cost of the proposed method may also be relatively large.

Questions for authors
  1. What are the substantial contributions or novelties of this work?

  2. Can the authors provide more convincing evidence to show the effectiveness and necessity of the proposed method?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Q1 - Novelty and contributions

The novelty of this study may be limited. The work does not appear to introduce new algorithms, theories, frameworks, or insights. Instead, this work applies existing standard parameterized-action MDP and GRPO to Morph, which may be methodologically incremental.

Q1 What are the substantial contributions or novelties of this work?

We thank the reviewer for raising this important point and agree that the methodological contribution of TFPO should be more clearly articulated.

Our contribution is a general formulation connecting trans-dimensional generative modeling with parametrized-action reinforcement learning, together with a concrete realization that enables reward driven optimization of a dimension changing generative process.

  • We identify a common RL formulation for a broader class of flexible-size generators. We observe that a parametrized-action structure naturally arises in trans-dimensional generation and provide a common perspective on otherwise distinct generative models as illustrated in Appendix A, Edit Flows (Havasi et al., 2025), jump diffusion models (Campbell et al., 2023), and Morph (Franke et al., 2026) can all be represented through structural decisions selecting state-dependent action-space components
  • We turn this connection into a concrete tractable policy-optimization procedure for mixed discrete-continuous molecular graph generation. We derive the complete per-step policy density for a generator coupling continuous 3D coordinate flows, discrete atom and bond transitions, and insertion/deletion jumps. This enables standard likelihood-ratio policy optimization to operate directly on trans-dimensional trajectories, without requiring a fixed-size representation or an external cardinality controller.
  • We demonstrate experimental evidence that structural optimization matters and leads to measurable gains. Our new same-backbone, same-reward, and same-budget ablations show that removing the insertion/deletion policy-gradient terms substantially limits cardinality extrapolation: full TFPO reaches 77.7 ± 3.8 atoms on QM9, compared with 36.8 ± 4.3 when the structural gradient is removed. On CrossDocked, freezing insertion/deletion rates reduces the binding-affinity improvement by 0.35 kcal/mol, demonstrating that structural optimization contributes to a downstream objective that does not explicitly reward size.

We acknowledge the underlying RL principles are established, the new contribution lies in identifying and operationalizing their connection with trans-dimensional geometric generators, deriving the required likelihood for a non-trivial jump-flow process and establishing the value of optimizing structural decisions directly. We have strengthened this contribution presentation in the paper.

AuthorsOpenReview ↗

Response to Weakness 2 - Structural ablations and adaptive cardinality

there are no key ablations to show trans-dimensional policy formulation is necessary, adaptive cardinality is effective (molecular size is not the only variable)

We agree with the reviewer that the contribution of structural optimization should be isolated experimentally. We therefore conduct controlled ablations on CrossDocked2020 and QM9, testing both implicit size adaptation and explicit cardinality extrapolation.

CrossDocked2020: We compare full TFPO against ablation A2 (reference structural rates), where insertion/deletion rates are taken from the frozen pretrained model. Experiments use the same backbone, reward, and budget across 100 pockets and 3 training seeds, with a new backbone trained for 100 epochs.

MethodVS Avg ↓VM Avg ↓VD Avg ↓QED ↑SA ↑Div ↑SizeComplete ↑PB-valid ↑
Reference (published)-6.36-6.71-7.450.480.73—22.8100.0%—
Morph base (no RL)-5.72-6.83-7.970.550.710.7322.897.4%0.81
TFPO (ours)-7.14 ± 0.02-8.07 ± 0.03-8.98 ± 0.030.61 ± 0.0010.80 ± 0.0030.75 ± 0.00321.3 ± 0.297.7% ± 0.0%0.85 ± 0.003
TFPO, A2 ref-struct-6.79 ± 0.08-7.72 ± 0.08-8.67 ± 0.060.61 ± 0.0040.80 ± 0.0020.76 ± 0.00320.6 ± 0.297.7% ± 0.2%0.85 ± 0.004

On CrossDocked, where size adapts indirectly to the downstream objective, full TFPO improves Vina Score from −5.72 (base) to −7.14 ± 0.02 kcal/mol. Using reference insertion/deletion rates (A2) yields −6.79 ± 0.08. Thus, learned structural adaptation yields a 0.35 kcal/mol advantage over A2, approximately 25% of the total improvement, demonstrating that structural optimization contributes to binding affinity even without an explicit cardinality reward.

QM9 - Cardinality extrapolation: We additionally evaluate two matched Morph ablations: A1 removes structural log-probabilities from the policy gradient, while A2 uses frozen reference insertion/deletion rates. We also compare against fixed-cardinality SemlaFlow with an externally optimized size distribution (Thompson-sampling bandit) based on Wang et al. (2026) [1]. All use the same reward and RL budget, with 3 seeds and 3 curriculum rounds of 600 steps.

MethodR1 atomsR2 atomsR3 atomsR3 MaxR3 ValidityR3 PoseBusters
Full TFPO44.4 ± 2.759.5 ± 4.777.7 ± 3.8860.866 ± 0.0350.823 ± 0.049
A1: stop-grad30.5 ± 0.134.4 ± 3.336.8 ± 4.3500.976 ± 0.0040.974 ± 0.005
A2: ref rates28.5 ± 0.230.3 ± 1.631.5 ± 1.8410.952 ± 0.0180.947 ± 0.018
SemlaFlow + bandit28.5 ± 2.814.9 ± 1.713.4 ± 2.4210.001 ± 0.0010.000

The gap between the full TFPO and both ablations increases over the curriculum rounds. Full TFPO achieves +60.1 atoms over the pretrained mean, compared with +19.2 (A1) and +13.9 (A2). The externally controlled SemlaFlow baseline initially reaches 28.5 atoms but collapses to 0.001 validity by round 3, whereas TFPO reaches 77.7 atoms at 0.866 validity.

Together, these ablations demonstrate that directly optimizing structural decisions benefits both cardinality extrapolation (QM9) and downstream binding affinity without an explicit size reward (CrossDocked).

[1] Wang, K., Janet, J. P., & Tibo, A. (2026). Controllable molecular generation with fine-tuned flow-matching model. Communications Chemistry, 9, 291. https://doi.org/10.1038/s42004-026-02188-z

and fixed-variance Gaussian noise design is reasonable and robust.

We agree with the reviewer that the robustness and design of the fixed variance gaussian noise is not ablated. The design of the gaussian kernel follows the RL for flow policies literature (Liu et al., 2025; Zhang et al., 2025) to give a tractable transition density to densities.

We evaluated different noise scales, linear decay, and learned noise (initialized at σ=0.05) on QM9 curriculum round 3, over 3 training seeds:

σ exploreScheduleMean atomsMaxValidityDiversityPoseBusters
0.01Fixed68.2 ± 3.0770.896 ± 0.0140.723 ± 0.0060.795 ± 0.102
0.05 (ours)Fixed75.3 ± 4.3860.867 ± 0.0310.722 ± 0.0080.835 ± 0.066
0.10Fixed72.1 ± 5.1830.903 ± 0.0460.720 ± 0.0040.896 ± 0.040
0.05Linear decay70.2 ± 5.0800.891 ± 0.0270.731 ± 0.0090.854 ± 0.068
Learned—76.7 ± 4.0860.885 ± 0.0490.716 ± 0.0080.814 ± 0.107

Performance of the gaussian noise remains robust across the tested configurations. Larger fixed noise improves geometric validity, while learned noise achieves comparable cardinality extrapolation, suggesting that fixed variance is a reasonable but not uniquely optimal choice.

AuthorsOpenReview ↗

Response to QM9 OOD validity and reward formulation

OOD experiments on QM9 may imply severe 3D generation failure. Only ~2.8% of the raw generated molecules pass all checks, while the reported result of 65% is after MMFF force-field minimization. This may mean the claimed OOD generation ability is supported only after external force-field repair and the generation of molecules with large size may not be reliable.

We agree with the reviewer that the submitted QM9 results show raw geometric degradation, and attribute the gap to an undertrained backbone and to reward mis-specification rather than the method.

We reran the experiment with a longer-trained backbone, evaluating 3 independent training seeds (3 × 1000 molecules per seed) without any force-field relaxation. The same TFPO procedure reaches 77.7 ± 3.8 atoms (4.4× the training mean, max 86), with RDKit validity of 0.866 ± 0.035 and raw PoseBusters validity of 0.823 ± 0.049, compared with 0.028 in the submitted runs. Conditional on chemical validity, PoseBusters validity reaches 0.950. Thus, the improved extrapolation does not depend on external geometry repair.

We would add that the QM9 reward contains no geometric term and the gate is RDKit validity, which stays flat while size grows. Geometry was never rewarded, so its drift reflects an unconstrained quantity rather than an inability to generate reliably at large size. In some other cases where a terminal reward does impact geometry like on GEOM-Drugs (Section 3.2) PoseBusters increases from 0.878 to 0.943 while PoseCheck strain falls.

Additionally, the reward for QM9 experiments appears to be based only on atom count. This may just encourage the model to generate larger molecules rather than OOD discovery.

We agree with the reviewer that the QM9 reward is simply size-extrapolation. It was built as such, to test the trans-dimensional machinery without any other property confounds. The property-driven claims rest on the two other experiments which both don’t directly reference size, showcasing validity increase at a preserved size distribution on GEOM-Drugs and a five term binding reward on CrossDocked under which size adapts bidirectionally.

Reviewer_4xN8OpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TFPO, which formulates a variable-cardinality 3D molecular generator as a parameterized-action MDP and applies GRPO to jointly optimize insertion, deletion, substitution, and coordinate decisions. Experiments on QM9, GEOM-Drugs, and CrossDocked demonstrate size extrapolation, improved molecular validity, and pocket-specific ligand optimization.

Critical strengths
  • The policy-density derivation is clear and technically useful. In particular, the treatment of variable action components and marginalization of latent mixture labels provides a concrete recipe for applying policy gradients to mixed continuous/discrete jump processes.

  • The empirical scope and reporting are strong. The paper evaluates three distinct objectives and provides unusually detailed metric definitions, raw-versus-relaxed geometry results, and distribution-shift analyses; the observed bidirectional size adaptation across pockets is an interesting capability.

Critical weaknesses

The central trans-dimensional contribution is not isolated experimentally. There is no matched Morph baseline with insertion/deletion disabled, the structural-policy term removed, or size controlled only through the initial size distribution. Consequently, the experiments show that RL can improve Morph, but not that explicitly optimizing dimension-changing actions is necessary or superior to ordinary molecular RL.

Several conclusions are overstated and the training evidence is insufficiently robust. At the largest QM9 size, raw PoseBusters validity collapses from 92.0% to 2.8% and reaches 65.0% only after MMFF relaxation; in CrossDocked, diversity is an optimized reward component but decreases from 0.584 to 0.411. Moreover, independent RL training seeds are not clearly reported, and Eq. (10) is undefined when the within-group reward variance is zero—a frequent case for the near-binary GEOM reward—while B=140B=140 cannot be partitioned into groups of G=32G=32 as stated.

Questions for authors

Can the authors provide a same-backbone, same-reward, and same-budget ablation in which insertion/deletion is disabled or logπstructlog \pi_\text{struct} is stop-gradiented? Without this counterfactual, what evidence attributes the GEOM and CrossDocked improvements specifically to trans-dimensional size control rather than ordinary RL over atom, bond, and coordinate outputs?

2 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Question 1 - Isolating the trans-dimensional contribution

We thank the reviewer for the constructive feedback and the actionable experiments

The central trans-dimensional contribution is not isolated experimentally. There is no matched Morph baseline with insertion/deletion disabled, the structural-policy term removed, or size controlled only through the initial size distribution. Consequently, the experiments show that RL can improve Morph, but not that explicitly optimizing dimension-changing actions is necessary or superior to ordinary molecular RL.

Can the authors provide a same-backbone, same-reward, and same-budget ablation in which insertion/deletion is disabled or logPi_struct is stop-gradiented?

Moreover, independent RL training seeds are not clearly reported

Without this counterfactual, what evidence attributes the GEOM and CrossDocked improvements specifically to trans-dimensional size control rather than ordinary RL over atom, bond, and coordinate outputs?

We completely agree with the reviewer that the role of the insertions and deletions needs to be isolated. We therefore present the following experiments:

Experiment 1: we ablate our TFPO method on the Morph architecture on the QM9 molecular size and validity task in the following manner, with the ablations suggested by the reviewer:

  • A1 (structural stop-gradient): Morph TFPO with the insertion deletion policy terms stop-gradiented, which effectively is removes their logprobabilities from the policy gradient
  • A2 (reference insertion/deletion rates): Samples insertion/deletion decisions from the frozen reference policy.

We additionally evaluate SemlaFlow with an externally optimized size distribution (Thompson-sampling bandit) based on Wang et al. (2026) [1], using the same reward and RL budget. We choose this baseline comparison over pre-training a fixed-size Morph backbone because of time and compute constraints.

Training is conducted on a stronger pretrained backbone on QM9, (3 seeds, 3 curriculum rounds of 600 steps):

MethodRound 1 atomsRound 2 atomsRound 3 atomsR3 ValidityR3 PoseBusters
Full TFPO44.4 ± 2.759.5 ± 4.777.7 ± 3.80.866 ± 0.0350.823 ± 0.049
A1: stop-grad30.5 ± 0.134.4 ± 3.336.8 ± 4.30.976 ± 0.0040.974 ± 0.005
A2: ref rates28.5 ± 0.230.3 ± 1.631.5 ± 1.80.952 ± 0.0180.947 ± 0.018
SemlaFlow + bandit28.5 ± 2.814.9 ± 1.713.4 ± 2.40.001 ± 0.0010.000

Conclusion : The gap grows across curriculum rounds: full TFPO achieves +60.1 atoms over the pretrained mean, versus +19.2 (A1) and +13.9 (A2), demonstrating the importance of directly optimizing structural decisions for cardinality extrapolation. The externally controlled SemlaFlow baseline initially reaches 28.5 atoms but collapses to 0.001 validity by round 3, whereas TFPO reaches 77.7 atoms at 0.866 validity.

Experiment 2 - CrossDocked (implicit size adaptation): To test whether structural optimization benefits objectives beyond explicit cardinality control, we additionally compare full TFPO against A2 (frozen reference insertion/deletion rates) on 100 CrossDocked pockets, using the same (new, better-trained) backbone, reward, and budget over 3 training seeds.

MethodVS Avg ↓VM Avg ↓VD Avg ↓QED ↑SA ↑Div ↑SizeComplete ↑PB-valid ↑
Reference (published)-6.36-6.71-7.450.480.73—22.8100.0%—
Morph base (no RL)-5.72-6.83-7.970.550.710.7322.897.4%0.81
TFPO (ours)-7.14 ± 0.02-8.07 ± 0.03-8.98 ± 0.030.61 ± 0.0010.80 ± 0.0030.75 ± 0.00321.3 ± 0.297.7% ± 0.0%0.85 ± 0.003
TFPO, A2 ref-struct-6.79 ± 0.08-7.72 ± 0.08-8.67 ± 0.060.61 ± 0.0040.80 ± 0.0020.76 ± 0.00320.6 ± 0.297.7% ± 0.2%0.85 ± 0.004

Full TFPO improves Vina Score from −5.72 (base) to −7.14 ± 0.02 kcal/mol, compared with -6.79 ± 0.08 for A2. Thus, learned structural adaptation contributes a 0.35 kcal/mol advantage over A2, approximately 25% of the total improvement, demonstrating that structural optimization contributes to binding affinity even without an explicit cardinality reward.

[1] Wang, K., Janet, J. P., & Tibo, A. (2026). Controllable molecular generation with fine-tuned flow-matching model. Communications Chemistry, 9, 291. https://doi.org/10.1038/s42004-026-02188-z

AuthorsOpenReview ↗

Response to Weakness 2 - Robustness and validity

Several conclusions are overstated and the training evidence is insufficiently robust. At the largest QM9 size, raw PoseBusters validity collapses from 92.0% to 2.8% and reaches 65.0% only after MMFF relaxation

QM9 geometric validity. We agree with the reviewer that the submitted results show raw geometric degradation, partly attributable to an undertrained backbone and the absence of a geometry term in the reward. With a longer trained QM9 backbone the same TFPO procedure reaches 77.7 ± 3.8 atoms (4.4× the training mean, max 86) at raw PoseBusters (no relaxation) validity 0.823 ± 0.049 (3 training seeds, 3 × 1000 molecules each) against 0.028 raw posebusters for the submitted runs under the identical protocol.

In CrossDocked, diversity is an optimized reward component but decreases from 0.584 to 0.411.

CrossDocked diversity. We acknowledge the diversity decrease in the submitted results. As discussed in Appendix G.5, this is partly related to the pretrained backbone: pocket-level diversity decreases during pretraining, consistent with learning a more target-specific distribution (real ligands sharing a target have diversity 0.597) and RL further concentrates the distribution.

We trained a stronger backbone on 100 training epochs on CrossDocked rather than 650+, for which diversity is much stronger. On this backbone, TFPO increases diversity from 0.726 to 0.751 ± 0.003 over three independent seeds, with uniqueness remaining at 1.000.

Eq. (10) is undefined when the within-group reward variance is zero—a frequent case for the near-binary GEOM reward—while cannot be partitioned into groups of as stated

Technical correction We thank the reviewer for catching both errors. The implementation carries the standard stabilizer, A^i=r(gTi)−μiσi+ϵ\hat{A}_i = \frac{r(g_T^i)-\mu_i}{\sigma_i+\epsilon}, with ϵ=10−6\epsilon=10^{-6}. A group whose rewards are all equal has a numerator exactly zero, so A^i=0\hat{A}_i=0 and the group contributes no policy-gradient term. And the group partition was a transcription error as we used 160/40 for the experiment. We have updated the manuscript accordingly.

We thank the reviewer again for the constructive feedback and hope these additional experiments and clarifications address their concerns.

Reviewer_Pz6sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional RL-guided methods for molecular generation usually target fixed-size (fixed-cardinality) generators. This paper sets out to solve a big challenge: how to RL fine-tune 3D molecular generation models that handle flexible-size (trans-dimensional) geometric graphs. To tackle this, the authors introduce a framework called TFPO (Transdimensional Flow Policy Optimization) and demonstrate it using a trans-dimensional 3D molecular generator named Morph.

Critical strengths

It lifts the restriction of fixed-size models, making molecular size an optimization variable for RL and opening up a brand-new dimension for exploring uncharted chemical space (OOD discovery).

Critical weaknesses

1.RL fine-tuning can't break through the fundamental capacity ceiling of the pre-trained generator. For instance, over-extrapolating on QM9 leads to a drop in validity, and its absolute performance in pocket-conditional generation still lags behind some specialized, stronger baselines.

2.Long sampling trajectories make RL rollouts and updates super time-consuming, especially when the reward function involves expensive calculations like molecular docking scores.

Questions for authors
  1. In molecular generation, rewards (like Vina docking scores or strict validity) are usually only given at the very end of the trajectory as terminal rewards. In a trans-dimensional MDP, early insertion/deletion decisions have a massive impact on the final molecule's size and topology. Since TFPO relies solely on group-level relative advantage from GRPO, is it prone to high variance or poor credit assignment? Could we consider introducing auxiliary dense rewards (such as intermediate geometric penalties) or training a Value Function to stabilize training?

  2. In the QM9 experiments, the model successfully pushed molecular size up to nearly 4 times the training mean, but PoseBusters' geometric validity dropped significantly (from 0.919 to 0.650). Doesn't this suggest that relying purely on terminal scalar RL rewards isn't enough to make up for the lack of prior knowledge the pre-trained model has in large molecular conformation spaces?

  3. If I introduce a new atom type (a dummy atom) into a diffusion or flow generation algorithm—meaning it generates right alongside the other atoms, but I just strip away the generated dummy atoms in the final output, kind of like language diffusion models—doesn't this approach let us directly plug and play existing RL methods?

3 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 & 2 : Pretrained capacity, Pocket conditional performance & computational cost

We thank the reviewer for the constructive feedback. We have run new experiments and ablations addressing each point.

Weakness 1a : RL fine-tuning can't break through the fundamental capacity ceiling of the pre-trained generator.

Pretrained capacity: We agree with the reviewer on the dependence on pretrained model capacity and acknowledge this in Section 5 (Limitations: empirical scope). RL reweights the prior to the KL-regularized optimum π∗(x)∝πref(x)exp⁡(r(x)/β)\pi^*(x)\propto\pi_{\mathrm{ref}}(x)\exp(r(x)/\beta) and in practice, learning still depends on encountering informative samples with sufficient frequency. This limitation however applies to all RL-finetuning generally (DDPO, Flow-GRPO, DePPA), and is not specific to TFPO.

We would like to make the point that while the support of fixed-size models is structurally constrained, for a trans-dimensional generator, every molecule has non-zero prior probability and the limit is statistical, now only depending on whether a region is reached often enough to generate a signal. TFPO's contribution is therefore being able to optimize on a strictly larger reachable set than what a fixed-size model can attain.

Weakness 1b : For instance, over-extrapolating on QM9 leads to a drop in validity, and its absolute performance in pocket-conditional generation still lags behind some specialized, stronger baselines.

Pocket-conditional performance: We agree that the submitted affinity gap warrants a more comparable evaluation. We showcase that some of the gap can be explained by reward mis-specification in the optimization objective. TFPO's original CrossDocked reward includes a PoseBusters geometry gate (App. G.3), unlike DePPA. We reran TFPO with and without this gate on an improved Morph backbone (3 RL seeds):

MethodVina Score ↓Vina Min ↓Vina Dock ↓QED ↑SA ↑Div ↑SizeComplete ↑PB-valid ↑
Reference (published)-6.36-6.71-7.450.480.73—22.8100.0%—
DePPA (published)-8.50-8.91-9.380.570.690.7821.295.3%—
Morph base (no RL)-5.72-6.83-7.970.550.710.7322.897.4%0.81
TFPO, gate ON-7.14 ± 0.02-8.07 ± 0.03-8.98 ± 0.030.61 ± 0.0010.80 ± 0.0030.75 ± 0.00321.3 ± 0.297.7% ± 0.0%0.85 ± 0.003
TFPO, gate OFF-8.71 ± 0.06-9.55 ± 0.06-10.22 ± 0.030.58 ± 0.0030.70 ± 0.0030.70 ± 0.00623.0 ± 0.292.9% ± 0.1%0.50 ± 0.02

Conformational quality: Strain Energy (SE), steric clashes, and Redocking RMSD.

MethodSE 25% ↓SE 50% ↓SE 75% ↓Clash ↓RMSD < 2 Å ↑
Reference (published)341071965.5134.0%
DePPA (published)1082997025.5951.8%
TFPO, gate ON31.5 ± 2.461.3 ± 3.6119.6 ± 6.55.81 ± 0.0436.0% ± 0.7%
TFPO, gate OFF128.6 ± 3.3312.0 ± 21.4867.0 ± 68.45.97 ± 0.0847.8% ± 1.3%

Without the geometry gate, TFPO achieves a Vina Score of −8.71 ± 0.06 kcal/mol, outperforming DePPA (−8.50 ± 0.12). With the gate, TFPO achieves substantially better geometric quality, including a PoseBusters validity of 0.85. The two configurations therefore illustrate different operating points in the affinity–geometry trade-off, rather than an intrinsic limitation of trans-dimensional policy optimization.

Weakness 2 : Long sampling trajectories make RL rollouts and updates super time-consuming, especially when the reward function involves expensive calculations like molecular docking scores.

Computational cost: We thank the reviewer for highlighting the computational cost. We distinguish the costs inherent to reward-guided molecular optimization from the additional overhead introduced by our trans-dimensional formulation.

TFPO uses 100 integration steps, compared with 500 steps in DePPA (with policy scoring every fifth step). To quantify costs, we measured the wall-clock time of a complete RL update on one protein pocket:

ComponentTime/updateShare
Total wall-clock20.45 s100%
Docking oracle (32 calls)12.46 s60.9%
Policy replay (2 forward passes)4.60 s22.5%
Rollout (100 steps, no gradient)2.91 s14.2%
Log-density replay (2 passes)0.48 s2.3%
↳ Structural scoring (included above)0.019 s<0.1%

The Vina docking oracle dominates the update cost (60.9%), which is method independent, and the TFPO specific component is negligible at <0.1% relative to the overall workload. The expensive rollouts are therefore more a property of RL for structure based design generally than a cost introduced by our method. We have added this breakdown in the revised manuscript

AuthorsOpenReview ↗

Response to Q1 - Variance, credit assignment, and reward shaping

Since TFPO relies solely on group-level relative advantage from GRPO, is it prone to high variance or poor credit assignment? Could we consider introducing auxiliary dense rewards (such as intermediate geometric penalties) or training a Value Function to stabilize training?

We thank the reviewer for raising these important questions regarding stability and credit assignment. We agree that these are important challenges, and address them through new analyses below:

Variance : We analyze variance related metrics over 3 training seeds on the QM9 size extrapolation curriculum. Training is stable while mean size grows 4.4x with final atom count standard deviations of ±2.7 to ±4.7 on means of 44–78 (≈6%). The mean importance-sampling ratio stays very close to unity (0.99989-0.99998) while the PPO ratio clipping fraction remains low (0.082%-0.420%) suggesting that the policy updates are stable.

Credit assignment : We ran two ablations on the QM9 task: A1 (structural stop-gradient) removes insertion/deletion log-probabilities from the policy gradient and A2 (reference ins/del rates) goes further and samples those structural decisions from the frozen reference. Mean generated atom counts are as follows :

RoundFull TFPOA1 stop-gradA2 ref rates
144.4 ± 2.730.5 ± 0.128.5 ± 0.2
259.5 ± 4.734.4 ± 3.330.3 ± 1.6
377.7 ± 3.836.8 ± 4.331.5 ± 1.8

The flexible size policy enables growth up to 75+ atoms (+60 over the base), with the gap to the ablations growing monotonically. The ablations never approach full TFPO size and show that there is real credit assignment in the insert and deletion and the structural score.

Dense rewards : We assess the possibility of dense rewards and measure RDKit validity (parseable, one connected component) of the generated structures over the trajectory in rounds 1 and 3 of the QM9 task. We observe that in the case of Morph, validity is resolved very late.

Round 1: tAtomsValidityRound 3: tAtomsValidity
0.4442.00.0000.6876.30.000
0.6042.40.1130.8876.50.098
0.7242.40.4800.9276.20.211
0.8842.40.7770.9675.90.355
1.0042.40.9221.0075.60.812

An intermediary geometric reward before t = 0.9 would score a molecule whose chemistry is not yet well defined. We agree that dense reward shaping would be a promising direction, but the structural obstacle here means it would have to be informative at a time where the quantity it measures does not yet exist, so we leave it to further work.

Value Function : Since all trajectories in a group share the same initial graph, the group-relative baseline provides a parameter-free estimate of the initial-state expected reward, avoiding the need to train an additional critic. We agree that a learned state-dependent value function could provide finer-grained credit assignment along the trajectory, however, our ablations demonstrate that group-relative terminal rewards already provide an effective learning signal for insertion and deletion decisions, without this additional complexity.

AuthorsOpenReview ↗

Response to Q2 & Q3 - QM9 geometric validity and alternative cardinality control

In the QM9 experiments, the model successfully pushed molecular size up to nearly 4 times the training mean, but PoseBusters' geometric validity dropped significantly (from 0.919 to 0.650). Doesn't this suggest that relying purely on terminal scalar RL rewards isn't enough to make up for the lack of prior knowledge the pre-trained model has in large molecular conformation spaces?

Q2 - QM9 extrapolation and geometric validity. We agree with the reviewer that the prior has an important impact on the reachable frontier. We report new runs using a longer trained QM9 backbone under the same unchanged TFPO procedure. All numbers are 3 training seeds, evaluated on 3 × 1000 molecules, and are raw PoseBusters scores on the generated coordinates with no force-field relaxation:

ArmMean atomsMaxValidityPoseBustersDiversity
Base (QM9)17.6270.9900.9360.897
Round 144.4 ± 2.7530.905 ± 0.0330.900 ± 0.0380.779
Round 259.5 ± 4.7710.871 ± 0.0950.859 ± 0.1080.739
Round 377.7 ± 3.8860.866 ± 0.0350.823 ± 0.0490.714

At 4.4× the training mean size, raw PoseBusters validity reaches 0.823, against 0.028 for the submitted backbone. Conditional on chemical validity the rate is 0.950.

We also note that the QM9 reward contains no geometric term as it optimizes size subject to RDKit validity, which stays flat across the curriculum while molecule size grows 4.4×. The observed geometric drift is therefore not directly penalized by the objective, rather than necessarily reflecting a limitation of terminal rewards.

On GEOM-Drugs (Section 3.2), the same terminal-reward mechanism improves PoseBusters reduces PoseCheck strain energy, showing that it can transfer to geometric quality when the optimization has headroom in that direction.

If I introduce a new atom type (a dummy atom) into a diffusion or flow generation algorithm—meaning it generates right alongside the other atoms, but I just strip away the generated dummy atoms in the final output, kind of like language diffusion models—doesn't this approach let us directly plug and play existing RL methods?

Q3 - Dummy-atom padding and fixed-size RL. We agree that dummy-atom padding is a viable alternative enabling standard fixed-dimensional RL. However, this requires a generator pretrained to handle dummy atoms and sufficient slot capacity for the desired molecular sizes, potentially introducing computational overhead from unused slots.

TFPO instead directly optimizes the insertion/deletion mechanisms of an existing trans-dimensional generator, without requiring a padded representation. Both approaches can be understood within our general parameterized-action formulation, which we clarify in Appendix A.

We also evaluate a related alternative, SemlaFlow (Irwin et al., 2025) with an externally optimized size distribution, fine-tuned with the same machinery, with cardinality handled by a Thompson-sampling bandit over the size distribution, following Wang et al. (2026) [1]. SemlaFlow conditions the model on a size embedding, so cardinalities outside the training support are warm-started from the N=28 size embedding.

MethodRoundMean atomsMaxValidityPoseBusters
Full TFPO144.4 ± 2.7530.905 ± 0.0330.900 ± 0.038
Full TFPO259.5 ± 4.7710.871 ± 0.0950.859 ± 0.108
Full TFPO377.7 ± 3.8860.866 ± 0.0350.823 ± 0.049
SemlaFlow + size bandit128.5 ± 2.8410.932 ± 0.0160.824 ± 0.152
SemlaFlow + size bandit214.9 ± 1.7220.004 ± 0.0040.000
SemlaFlow + size bandit313.4 ± 2.4210.001 ± 0.0010.000

The fixed-size baseline reaches 28.5 atoms but collapses to 0.001 validity by round 3, whereas TFPO reaches 77.7 atoms at 0.866 validity. While this does not directly evaluate dummy-atom padding, it illustrates the difficulty of extrapolating cardinality through external size control. We have included this comparison and the alternatives in the manuscript.

We thank the reviewer for the constructive feedback and thoughtful questions, which motivated additional experiments and helped us clarify the scope and contributions of TFPO.

[1] Wang, K., Janet, J. P., & Tibo, A. (2026). Controllable molecular generation with fine-tuned flow-matching model. Communications Chemistry, 9, 291. https://doi.org/10.1038/s42004-026-02188-z

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Remarks

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions, which motivated extensive new experiments, ablations, and analyses that have significantly strengthened our paper. We believe these additions address the main concerns raised during the review process, and hope they will be taken into account in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.