- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces DySCo, a lightweight data curation framework for molecular RL that adaptively constructs training coresets from current and prior samples. The experiments are mainly conducted on DrugEx v2, DrugEx v3, REINVENT 4, and PMO. The method is evaluated according to the reward, diversity, and computational efficiency.
Critical strengths
The experimental scope in the paper is relatively broad, which includes multiple generation frameworks, tasks, and ablation studies. The implementation details are clearly discussed in the Appendix.
Critical weaknesses
-
The methodological novelty may be somewhat limited. The key contributions of this paper might be "adaptive data curation for molecular RL" and "semi-relaxed OT". However, these might be incremental extensions or interesting tricks inspired by existing works. It is unclear if these could be considered as the substantial methodological novelty or new formulations.
-
The experimental results may not be clearly supportive. (1) SR version seems to already obtain very strong results. Its reward is even higher than complex DGS/RTS and the runtime is shorter, while the results of other metrics are close. It might be unclear if SR basically is the most effective part. This may also further raise the concern of the necessity of other components, say OT part, which appears to be one of the major contributions in the paper. Similarly, the results on PMO show DySCo’s improvement on diversity and runtime, but its AUC is mostly similar or lower than Vanilla RL, which may not fully support the advantage of the proposed method. And, DySCo uses an offline prior pool, but it is unclear if the cost of pool construction is considered in the reported runtime. If not, then the comparison for computational cost with baselines may be unfair. (2) The paper motivates DySCo as a solution for RL mode collapse, but the vanilla RL results may not clearly support the existence of the problem.
-
Potential key inconsistencies and errors in the methodology. (1) The optimization formulation may be inconsistent. Eq. 2 defines the RL objective using samples from the current policy, but Eq. 13 updates the model using a reward-selected mixture of samples from the current and prior policies, without any correction. So, the gradient in Eq. 13 may not be optimizing the objective defined in Eq. 2. Also, the REINVENT regularization may be written with the wrong optimization direction. If so, it may increase the regularization error. (2) The proposed adaptive signal may not measure structural diversity but measure whether high-reward samples come from the current or prior policy. A high value may not reflect the mode collapse as intended.
Questions for authors
-
Could the authors clarify what the substantial methodological novelties of this work are?
-
Is there more convincing evidence to support the advantage and necessity of the proposed method?
-
What is the correct formulation and interpretation for the method?
3 comments
Dear Reviewer qW6h,
Thank you for your thoughtful comments. We have carefully addressed your questions in our rebuttal and would greatly appreciate it if you had an opportunity to review our responses before the discussion period ends.
Sincerely,
The Authors of Paper #21503
I thank the authors for their efforts in the rebuttal. It helps clarify the notation and diversity interpretation issue. However, the major concerns in my initial review still remain.
-
The methodological novelty is still limited and the gains are somewhat modest and task-dependent.
-
OT may be useful in some settings, but its necessity is not sufficiently demonstrated with supportive evidence.
-
On PMO, the AUCs on some tasks seem to be lower than Vanilla RL, although the diversity and runtime may be better. So, this might be a trade-off advantage.
-
The runtime seems to exclude the cost of constructing and scoring the prior pool. The fairness of its comparison with baselines is still not fully demonstrated.
-
The method may need a stronger theoretical justification to show why the proposed biased/non-on-policy update can optimize the stated objective. This concerns optimization correctness.
Also, some key formulas in the rebuttal box are not displayed properly, which makes them very hard to read.
Overall, I will keep my original assessment.
Response to Reviewer qW6h
Thank you for the review. We clarify DySCo’s framework-level contribution and empirical trade-offs, provide evidence on structural concentration, and explain the update formulation, including the corrected REINVENT sign notation, and have revised the relevant sections of the manuscript accordingly.
W1/Q1 - Methodological novelty
DySCo is motivated by the observation that, in the self-evolving sample–score–update loop, generated molecules play two linked roles: they first reveal the current policy’s search state, while the scored molecules supply the training inputs for the next update. This feedback motivates a model-independent curation framework that couples when and how much correction is needed with which molecules provide the correction. The lightweight source ratio γt determines the correction size, while adaptive curation and the SR, DGS, and RTS selectors determine how the corrective data are selected. RTS further gives a geometric interpretation of corrective curation through candidate-specific, reward-aware transport costs and a closed-form semi-relaxed OT score. The after-reward-evaluation variants are compared in Table 1. During rebuttal, following the Reviewer 6VYF's request for a direct comparison, we added Appendix E.2 to compare this post-scoring curation with diverse mini-batch selection before reward evaluation under a comparable update budget, with oracle-call costs reported separately.
W2/Q2 - Empirical evidence and trade-offs
SR and the role of OT. SR, DGS, and RTS share adaptive correction but occupy different reward-diversity-runtime Pareto points. SR is the low-overhead end of this trade-off, whereas RTS-Tanimoto raises IntDiv from 0.9009 to 0.9120. On DrugEx v3 Lipinski, RTS-Hamming improves both reward and IntDiv over SR (0.775/0.928 vs. 0.724/0.909; Table 9). RTS therefore adds reward-aware structural selection when that trade-off is useful; Fig. 3 evaluates the shared adaptive mechanism against fixed correction.
PMO results. Compared with Vanilla RL, reward AUC@10 remains close overall (with lower values on some tasks), while top-100 IntDiv is higher on 7/8 tasks and runtime is lower on all 8 (Table 11). These results support a task-dependent Pareto choice rather than a claim that one selector is universally best.
Prior-pool cost and runtime fairness. For a fixed task and reward configuration, prior molecules and scores are prepared offline and reused across online epochs. In practical use, this one-time preparation can be amortized over repeated online epochs and runs sharing the same pool. We therefore report it separately from recurring online scoring and update costs, which isolate per-epoch costs.
Evidence of structural concentration. DySCo pursues faster updates while preserving reward and diversity through model-independent curation. “Does not guarantee” a useful population does not imply inevitable full-batch collapse; we test whether compact reward-only updates concentrate feedback. With the task reward and native update fixed, greedy Top-10% updates give IntDiv 0.7565 versus 0.8953 for full-batch Vanilla RL (Appendix Table 3); a separate structural characterization of the reward-greedy region in Appendix J.7 reports scaffold coverage of 468 versus 882 and top-10 scaffold shares of 28.7% versus 8.4%. The 1,000-epoch experiment is an auxiliary long-horizon check, not evidence that full-batch RL must collapse.
W3/Q3 - Formulation and interpretation
DySCo is model-independent and changes only which scored examples enter each native update; the task reward, framework-specific loss, regularization mechanism, and optimizer remain unchanged. In the revised manuscript, Eq. (2) defines the task-reward objective JR(θ)=E[R(x)], while Eq. (13) denotes the average native per-example update evaluated on the curated set Dt, containing selected current elites and, when needed, prior-policy corrective samples. Thus, Eq. (13) is not claimed to be the full-batch on-policy estimator of Eq. (2).
Regarding the reviewer’s concern, the sign of the REINVENT term in the submitted manuscript was reversed due to a typographical error. This is a notation error only and does not affect our experiments. We used the original REINVENT implementation without modifying its code: REINVENT 4 minimizes its native positive squared-error augmented-likelihood loss ΩREINVENT with its native optimizer. We have corrected this sign in the revised manuscript.
γt is a lightweight source-imbalance signal computed from source labels, not a direct diversity metric. Across 16 independent runs, higher γt correlates with lower elite IntDiv (ρ=−0.665±0.150) and scaffold entropy (ρ=−0.556±0.146) (Appendix J.1), supporting its use as a practical concentration proxy without making a candidate-level guarantee.