- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a Reinforcement Learning (RL)-based forgetting framework for pretrained 3D molecular flow matching models, which can hinder undesired generation while preserving the original generation capability and preventing the forgotten behavior from being easily recovered through subsequent fine-tuning. The experimental results suggest the effectiveness of the proposed method.
Critical strengths
- This method sounds technical and interesting. Avoiding the model generating undesired molecules is also a realistic application problem.
- Post-training with the sampling intermediate state is also a unique perspective, and the core method is novel.
Critical weaknesses
- The presentation is not good. The algorithm 1 and 2 overlap with the Equation.
- This post-training uses the data sampled from its own model, which can be viewed as self-distillation. Therefore, the model might heavily rely on the pretrained model's performance. Especially when the pretrained model generate the the samples with unbalance property (like most of them are non-toxic). Further, this model will use the intermediate state G_t in the sampling trajectory, but use G_T as the label only. The potential risk is that before getting the final state G_T, many molecules with different properties might share the same intermediate state. This will also make the fine-tuning unstable and cause the model to be confused.
- This method constructs the SDE version of the original ODE version in order to utilize DPO fine-tuning. However, this will make the original flow matching distinct from the fine-tuning model. Therefore, I was wondering if the model has a discrepancy between ODE and SDE sampling after fine-tuning.
- The baselines appear to be too weak to support the proposed method. It would be better to consider some SOTA general unlearning methods like [1]. Further, the baselines should include references, since the details of the baselines are missing in the main text.
[1] Kim, Hoki, et al. "Unlearning-aware minimization." Advances in Neural Information Processing Systems 38 (2026): 93806-93829.
Questions for authors
Did the authors try to directly use the known undesirable molecule dataset to construct Flow Matching intermediate states, and then perform PPO-based forgetting on these states?
5 comments
Response to Reviewer crVZ (Part 2/3 - W3)
Table R1: Direct Use of Known Undesirable Molecules for Forgetting
| Method | Validity (%) ↑ | Uniqueness (%) ↑ | Novelty (%) ↑ | FCD ↓ | UR (%) ↓ | Recovery@20 (%) ↓ |
|---|---|---|---|---|---|---|
| GA | 96.91 ± 0.58 | 99.95 ± 0.03 | 99.93 ± 0.04 | 6.47 ± 0.27 | 3.84 ± 0.34 | 5.18 ± 0.57 |
| Ours | 98.92 ± 0.30 | 100.00 ± 0.00 | 100.00 ± 0.00 | 5.31 ± 0.19 | 1.43 ± 0.05 | 1.52 ± 0.15 |
W3:
The constructed SDE preserves the marginal distributions of the original ODE, establishing their theoretical consistency. During PPO post-training, the deterministic ODE does not define a stochastic transition density for the continuous coordinates required to construct the PPO transition likelihood, whereas the SDE does. In Appendix B.3, we show how the marginal-preserving SDE is constructed and explain why it is consistent with the original ODE. For the reviewer’s convenience, we summarize the key derivation below.
Let denote the marginal distribution of the original flow, where contains continuous coordinates and discrete molecular features. The coordinate ODE is , and the joint distribution satisfies , where represents the discrete CTMC dynamics. Keeping these discrete dynamics unchanged, we construct the SDE as , with .
Let denote the distribution induced by this SDE. Its forward equation is . Since , we have . Consequently, substituting makes the score correction and diffusion terms cancel: . The remaining equation is exactly the same as that of the original flow.
Therefore, during fine-tuning, the SDE introduces stochasticity to support exploration while preserving the original flow's marginal distributions. After fine-tuning, this additional stochasticity is no longer needed, so we retain the original ODE for generation. The ODE and SDE are defined by the same flow matching model and share the same parameters . PPO updates these shared parameters through the SDE trajectories, and generation with the original ODE follows the updated velocity field and discrete predictions. In the main paper, the results in Tables 1-3 are all obtained with ODE and show the learned forgetting effect. For example, Table 2 shows that UR decreases from 9.02% to 1.70% for toxicity and from 10.76% to 2.13% for poor permeability.
In addition, to directly assess consistency after fine-tuning, we compare the ODE and SDE results for the same fine-tuned model in Table R2 below. Their similar UR and validity further support consistent forgetting effectiveness and generation quality. Small differences can arise from stochastic variation across a finite number of generated molecules and numerical discretization.
Table R2. Additional SDE results on the same fine-tuned model (toxicity).
| UR (%) | Validity (%) | |
|---|---|---|
| SDE | 1.46 | 98.87 |
| ODE | 1.43 | 98.92 |
Response to Reviewer crVZ (Part 3/3 - W4)
W4:
We add Unlearning-Aware Minimization (UAM) (Kim et al., 2025) as an additional baseline and report its results under the same protocol, as shown in Table R3. Compared with UAM, our method achieves lower UR and Recovery@20 on both tasks while maintaining comparable generation quality. We also include references for baselines in the revised manuscript, including Retained (Bourtoule et al., 2021), GA and GA+Retain (Maini et al., 2024), REINFORCE (Williams, 1992), Global KL (Lu et al., 2022), Smoothness (Fan et al., 2025), and UAM (Kim et al., 2025).
Table R3: Comparison with UAM.
| Task / Method | Validity (%) ↑ | Uniqueness (%) ↑ | Novelty (%) ↑ | FCD ↓ | UR (%) ↓ | Recovery@20 (%) ↓ |
|---|---|---|---|---|---|---|
| Toxicity | ||||||
| UAM | 98.60 ± 0.20 | 100.00 ± 0.00 | 100.00 ± 0.00 | 5.47 ± 0.21 | 1.80 ± 0.22 | 3.30 ± 0.34 |
| Ours | 98.92 ± 0.30 | 100.00 ± 0.00 | 100.00 ± 0.00 | 5.31 ± 0.19 | 1.43 ± 0.05 | 1.52 ± 0.15 |
| Permeability | ||||||
| UAM | 98.50 ± 0.35 | 100.00 ± 0.00 | 100.00 ± 0.00 | 5.45 ± 0.22 | 2.30 ± 0.28 | 3.80 ± 0.40 |
| Ours | 98.82 ± 0.31 | 99.99 ± 0.01 | 100.00 ± 0.00 | 5.26 ± 0.14 | 1.83 ± 0.19 | 2.49 ± 0.22 |
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021). Machine Unlearning. IEEE Symposium on Security and Privacy (SP), 141-159.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., & Kolter, J. Z. (2024). TOFU: A Task of Fictitious Unlearning for LLMs. ICLR 2024 Workshop.
Williams, R. J. (1992). Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8, 229-256.
Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., & Choi, Y. (2022). Quark: Controllable Text Generation with Reinforced Unlearning. NeurIPS 35.
Fan, C., Jia, J., Zhang, Y., Ramakrishna, A., Hong, M., & Liu, S. (2025). Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond. ICML 2025.
Kim, H., Kim, K., Chae, S., & Yoon, S. (2025). Unlearning-Aware Minimization. NeurIPS 38.
We once again thank the reviewer for the thoughtful feedback.
Response to Reviewer crVZ (Part 1/3 - W1 & W2Q1)
We thank the reviewer for recognizing the novelty, practical significance, effectiveness and technical soundness of our work.
Below we address the reviewer’s questions about presentation (W1), intermediate-state handling (W2Q1), ODE–SDE consistency (W3), and an additional baseline (W4).
W1:
We sincerely apologize for the presentation issue and appreciate the reviewer pointing it out. We have carefully reformatted Algorithms 1 and 2 and updated the paper. Now, all equations and pseudocode are clearly presented. We also thoroughly checked the entire manuscript to prevent similar issues.
W2Q1:
We first clarify that sampling trajectories from the current model does not make the forgetting objective self-distillation. Self-distillation typically requires the model’s own predictions, or those of a previous/frozen version, to serve as training targets. Here, the current model provides the sampled trajectories, not the training targets. In our forgetting objective, the supervision instead comes from the external undesirability estimator through the terminal reward. PPO then uses this reward to reduce the likelihood of transitions leading to undesired molecules.
Balanced desired and undesired samples are not required by our formulation. In our reward design, desired molecules receive zero forgetting reward and therefore do not provide a forgetting signal, whereas undesired molecules receive a negative reward that directly drives forgetting. Thus, as long as undesired molecules are sampled during rollout, the model receives a valid forgetting signal. Although most generated molecules may be non-toxic, this only affects the sampling efficiency of the forgetting signal. As a practical strategy, we reuse the sampled trajectories for multiple updates to improve their utilization. As forgetting progresses, fewer undesired trajectories are expected as the forgetting objective is being satisfied, rather than indicating instability. Our QM9 results further support this: despite a pretrained undesired generation rate of only 1.62% for fluorine-containing molecules, our method reduces it to 0.00%, showing effective forgetting even with sparse undesired generations.
Independently sampled trajectories almost surely do not share the same intermediate state. Each intermediate state is Gt=(Xt,At,Ct,Et), where Xt contains continuous atomic coordinates. Under our stochastic transition, Xs=μψ,t→s+Vt,sϵ, with ϵ∼N(0,I) and Vt,s>0. For two independently sampled trajectories, Xs(1)−Xs(2) therefore follows a non-degenerate continuous Gaussian distribution, implying Pr(Xs(1)=Xs(2))=0. Since two full molecular states can be identical only if their continuous coordinates are identical, Pr(Gs(1)=Gs(2))=0 as well. Therefore, except for numerical precision, two independently sampled trajectories will not encounter exactly the same intermediate molecular state.
Even if two trajectories pass through the same intermediate state, different terminal rewards do not mean that PPO receives contradictory supervision. In the standard stochastic policy-gradient framework, Q(Gt,at)=E[R(GT)∣Gt,at], which represents the expected terminal reward after taking transition at from intermediate state Gt. The policy is optimized to maximize the expected return, rather than assigning a unique terminal property to each Gt. The corresponding on-policy policy-gradient identity is ∇ϕJ(ϕ)=Eτ∼pϕ[R(GT)∑t=0T−1∇ϕlogpϕ(Gt+1∣Gt)], while our method uses the corresponding clipped PPO surrogate. Therefore, even if the same or similar Gt leads to different terminal molecules GT, the algorithm can still estimate the expected reward through stochastic trajectories. Different terminal rewards do not constitute conflicting labels.
We had considered directly using a known undesirable molecule dataset to construct flow-matching intermediate states for PPO-based forgetting. However, such states follow the data-conditioned interpolation path from known terminal molecules, rather than on-policy trajectories sampled from the behavior policy ϕold. Our PPO likelihood ratio is defined on the same transitions actually sampled by ϕold. Directly applying it to externally constructed states would therefore no longer follow the on-policy PPO formulation. As an alternative, we include GA as a baseline, which directly uses known undesirable molecules as terminal states, constructs their flow-matching intermediate states through the original interpolation path, and performs gradient ascent on the corresponding flow-matching objective. As shown in Table R1, it reduces the toxicity UR to 3.84%, compared with 1.43% for our method.