- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AVID, a two-stage RL framework for molecular discovery: a pre-scored archive guides prefix generation, followed by an identity-weighted PPO update. Across 23 tasks the authors report large improvements in candidate yield.
Critical strengths
The evaluation scale is unusual for this area and the paper honestly reports unfavorable results such as reduced pairwise fingerprint diversity.
Critical weaknesses
The main comparison is not like-for-like: AVID receives task-specific offline reward supervision while the baselines have no equivalent information, so the gains cannot be attributed to the proposed method.
Beyond that, the guided PPO objective is non-standard and updates may be clipped before taking effect; there is no ablation with the archive alone and without identity weighting to isolate the second component, and the headline metric H_32768 is coupled to a specific archive quantile threshold tau.
Questions for authors
Can the baselines be given equivalent offline reward information? How often are updates clipped in practice? Can you report the sensitivity of H_32768 to tau, together with paired significance tests?
3 comments
Thank you for the response and additional analyses, which help address my concerns about clipping and threshold sensitivity. However, differences in access to task-specific offline reward information remain a concern in the main baseline comparisons, limiting attribution of the reported gains to the proposed method. I therefore maintain my rating of 2.
Thank you for raising the question of access to offline rewards. When PPO and Augmented Memory (AM) learned from the same task-specific reward archive as AVID, AVID still found 3,480 more qualifying molecules per task than PPO and 2,287 more than AM, on average.
Comparisons with the same archive. We evaluated all 23 tasks with five paired seeds per task. The table averages the seeds within each task and gives each task equal weight.
| Baseline | Baseline mean per task | AVID mean per task | AVID gain per task | Tasks with a higher AVID mean |
|---|---|---|---|---|
| PPO trained on the same archive | 3,634 | 7,114 | +3,480 | 22/23 |
| AM with replay seeded from the same archive | 4,827 | 7,114 | +2,287 | 18/23 |
PPO trained its starting policy for eight epochs on the archive, giving higher-scoring examples more weight. AM seeded its replay buffer with molecules and scores from the same archive, using its usual buffer size. AVID used that archive to train its guide.
The common budget was 2,048 offline scores per task and up to 10,000 new online scores per run, for a total cap of 12,048 including the archive. A qualifying molecule is a distinct, newly scored molecule that meets the paper's task-specific cutoff (the 90th percentile of archive rewards). We count these molecules within a separate cap of 32,768 proposals recorded by the evaluation ledger.
What identity weighting adds to guided PPO. Both conditions used the same archive-trained guide and the same active-token loss normalization. We changed only whether repeat molecules contributed directly to the policy loss. Masking repeats added 3,257 qualifying molecules per task on average, raising the five-seed mean on all 23 tasks.
In the revision, we will report both archive-informed baselines in the main results and present the fixed-guide comparison alongside them.
Thank you for your careful reading and thoughtful feedback. Below, we respond to your questions point by point and share the results of our additional analyses.
Q1: Access to offline reward information. When we charge the full archive on the saved runs, AVID finds 2,251 more qualifying molecules per task than standard PPO (Holm-adjusted p = 0.00609) and covers 1,476 more qualifying Bemis-Murcko scaffolds. Across 23 tasks and five paired seeds, we apply the same caps of 10,000 total labels and 32,768 proposals counted by the identity ledger. AVID pays for all 2,048 offline scores, leaving at most 7,952 new online scores; a baseline pays for an archived score the first time it uses it. The table gives the estimated differences at the paper's 90th-percentile cutoff.
The table addresses label cost rather than training the external baselines on the same archive. To hold offline information fixed, we compare AVID with Q-Steer-only: both use the same archive-trained guide and loss reduction. Adding the mask yields 3,475 more qualifying candidates per task on average across the 23 tasks.
Q2: Actual PPO clipping frequency. Across 112 completed diagnostic runs spanning all 23 tasks, the median run clipped 1.664% of retained tokens (run-level range: 0.817% to 6.656%). Parameters changed in all 995 logged update checks. The ratio is the current base actor's token probability divided by the stored guided behavior probability. We counted clipping only when it took effect given the sign of the advantage, not every ratio outside the allowed range.
Q3: Threshold sensitivity and paired tests. Here, H32768 counts distinct newly scored molecules with reward at least τ, the task-specific archive reward cutoff (the 90th percentile in the paper); 32,768 is the cap on generation proposals counted by the ledger, not the number of oracle calls. AVID finds more qualifying molecules than AM at the 80th, 85th, and 90th percentiles after Holm correction. At the original online budget, we average the five paired seeds within each task and run a paired sign-flip test across the 23 tasks. The p-values below are adjusted across the threshold and structural comparisons.
At the 90th percentile, the gain also holds against PPO (adjusted p = 0.00051) and REINVENT (adjusted p = 0.04172).