- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RetroTAC predicts a route-derived synthetic-accessibility score directly from PROTAC structures to prioritize large virtual libraries before running explicit retrosynthesis. The authors obtain labels for 18,076 PROTACs using ShallowTree and eMolecules building blocks, combining normalized route depth with the average heavy-atom balance of reactants at bimolecular steps. Purchasable targets and unsuccessful searches receive fixed endpoint scores. They compare XGBoost, an MLP, and a pretrained Chemprop graph network using warhead-scaffold-grouped cross-validation, then select a 27-member ensemble that achieves a reported held-out RMSE of 0.132 and . The ensemble also scores approximately 1.30 million enumerated PROTACs, from which 1,600 candidates undergo additional planning after selection by predicted score and model disagreement. High-scoring candidates achieve mean route scores of 0.733 versus 0.313 for low-scoring candidates, with planner-resolution rates of 98.75% versus 51.00%. I see the main contribution in the PROTAC route resource and the practical screening workflow, with further validation needed to connect the score to experimental synthetic effort.
Critical strengths
-
From my experience, PROTAC libraries can grow quickly through recombination of warheads, linkers, and E3 ligands, making explicit planning for every candidate quite expensive. A structure-based surrogate that identifies promising candidates for route inspection addresses a concrete bottleneck in computational degrader design.
-
The additional planning experiment provides meaningful evidence for prioritization. Across 800 high-scoring candidates, the planner resolves 98.75%, compared with 51.00% among 800 low-scoring candidates. The high-scoring groups also yield shorter routes with more balanced fragment joining, showing that the predictor transfers its intended ranking to newly enumerated molecules.
-
The route dataset could support work beyond this particular ensemble. I also appreciate the component-overlap analysis and warhead-scaffold grouping, which address an important source of overly optimistic validation for modular molecules. These choices make the resource and evaluation more useful than a random molecular split would.
Critical weaknesses
-
The methodological contribution is modest, which places more weight on demonstrating practical improvement over existing tools. The regressors and ensemble selection are established methods, and RSPred (Parrot et al., 2023) already learns a continuous score from retrosynthesis outputs. Appendix E includes DeepPSA, but measures agreement between scores rather than comparative screening performance. I would be comfortable with a straightforward model if the authors showed that it selects more useful candidates than existing scores under the same planning budget.
-
Fragment-size balance does not account for coupling difficulty, protecting groups, selectivity, or building-block cost. Appendix C provides useful sensitivity analyses, but agreement among score variants does not establish chemical validity. The labels also inherit the first route returned by one planner, including its stated difficulty with macrocycles. A focused comparison against documented syntheses or blinded chemist assessments would make the practical interpretation much more convincing for me.
-
There were several implementation details I could not locate. The paper reports 2,232 held-out molecules, then 463 for ensemble selection and 1,854 for evaluation, which sum to 2,317. Appendix E instead reports 378 selection molecules. Figure 3b also shows RMSE 0.128 and , whereas Table 2 reports 0.132 and 0.647. These may be reporting errors, but they need correction to establish which results come from untouched evaluation data. The uncertainty estimates also need calibration before supporting confidence-based filtering.
Questions for authors
-
Could you compare the top-ranked candidates from RetroTAC, DeepPSA, PROTAC-adapted FSscore, RAscore, and random selection under an identical planning budget? Please report resolution rate, route depth, and retained chemical diversity. The current high-versus-low comparison demonstrates separation, but does not establish the improvement over available screening methods.
-
Is the ordering induced by Eq. 1 consistent with chemists’ preferences? A one-step route with no bimolecular reactions receives because the fallback fragment balance is , whereas a two-step route with perfect balance, , receives . Please examine such cases using documented syntheses or blinded expert rankings, and compare against a depth-only score.
-
How sensitive are the labels to the first route returned, search budget, and building-block stock? Please repeat planning on a representative subset across seeds and budgets, including macrocycles and zero-labeled molecules, and report changes in resolution and ranking, and clarify how the 22 partial routes in the prospective low-score group contribute to the reported scores.
-
Was the ensemble-selection subset 463 or 378 molecules? Please confirm that the final evaluation subset influenced neither ensemble weights nor strategy selection.
-
How does performance vary when the warhead, linker, or E3 ligand is unseen? The warhead-grouped cross-validation is useful, but the final held-out and enumerated sets need corresponding component-overlap statistics. Please also confirm whether prospective deduplication excluded both development and held-out molecules.
-
Can ensemble disagreement improve selection after controlling for predicted score? Compare errors within narrow score bins and report error as increasingly uncertain predictions are excluded. The current interval spanning two ensemble standard deviations covers only 72.9% of held-out cases.
-
What are the wall time and hardware requirements for scoring the 1.30 million candidates? Please compare the 27-member ensemble with the single GNN in throughput and top-ranked candidate quality. The RMSE improvement from 0.1378 to 0.1320 is modest, so the additional inference cost does matter for the proposed use case.
-
How reliable are low predictions across the full library? The selected low-score candidates have predictions below -0.058, despite labels lying in , yet 51% still have fully resolved routes. A randomly sampled planning panel would help measure calibration and the fraction of useful candidates discarded at different filtering thresholds. This would clarify how practitioners should use the score.
4 comments
Q3. Seeds, budgets and stock. We sampled 25 molecules per stratum: macrocycles (ring size ), and low, medium and high predicted scores (tertiles) among non-macrocycles. Each molecule received three ShallowTree runs with different seeds at both 10- and 30-minute budgets:
| Stratum | 30 min: mean score ± mean SD | 10 min: mean score ± mean SD |
|---|---|---|
| Macrocycle | 0.333 ± 0.007 | 0.300 ± 0.019 |
| Low | 0.300 ± 0.006 | 0.224 ± 0.038 |
| Medium | 0.495 ± 0.011 | 0.415 ± 0.035 |
| High | 0.575 ± 0.019 | 0.562 ± 0.030 |
These are route scores, not predictions. SD means the mean within-molecule SD across seeds: . It is 0.0107 at 30 minutes and 0.0306 at 10 minutes. At 30 minutes, 70 molecules resolve in all three runs and 30 in none, while pairwise run-ranking correlations are 0.9795-0.9890. At 10 minutes, 58 resolve in every run, 36 in none and six inconsistently.
Averaging across seeds, resolution rises from 61% to 70%: +9.0 percentage points (95% bootstrap interval 4.3-14.7). Mean route score rises by 0.0505 (0.0252-0.0794). Here we resample molecules within strata 5,000 times, preserving both budgets and all seeds per molecule. The correlation between mean scores at the two budgets is . Equal stratum sampling means these pooled rates do not estimate library-wide prevalence.
The 22 partial routes receive the same Eq. 1 score, , from their returned reaction trees, even though some terminal building blocks are not in stock. They contribute to the uncertain-low mean of 0.330 but are not counted as resolved. We will replace the claim that all searches failing to reach stock receive zero with: “Unresolved searches with no returned reaction steps receive ; nonempty partial routes retain their depth-and-balance score.”
Additional notes:
- stock sensitivity remains untested;
- repeated first-route searches assess repeatability, not optimality across alternative routes;
- macrocycles were excluded from development and held-out sets but remain in the enumerated library.
Q4. Counts and evaluation integrity. The corrected split is : selection and evaluation. The ensemble has 32 members, RMSE 0.12775 and . The older 27-member ensemble has RMSE 0.13202. We will correct the stale counts, composition and metrics.
Caruana weights use only the 446 selection cases. However, the original script selects “best single” and “best backend” using evaluation RMSE and compares strategies there. Strategy choice was therefore not independent. Reselecting baselines on selection cases alone gives evaluation RMSE 0.13355 for a single GNN and 0.14008 for the MLP average. This reanalysis does not provide a fresh untouched test set. We will update Table 2.
The DeepPSA comparison has cases: 14 of the 446 selection cases were already among the 73 training overlaps. Pearson/Spearman/thresholded Kendall correlations are 0.2577/0.3892/0.2555 for route scores and 0.3313/0.4944/0.2478 for RetroTAC. Kendall uses thresholds 0.70 and 0.50 for RetroTAC/route scores and DeepPSA, respectively, and the other correlations use continuous outputs.
Q5. Unseen components. Component-grouped CV (mean ± SD over 25 folds):
| Grouping | Model | RMSE | |
|---|---|---|---|
| Warhead | XGB | 0.177 ± 0.018 | 0.237 ± 0.121 |
| Warhead | MLP | 0.171 ± 0.016 | 0.290 ± 0.114 |
| Linker | XGB | 0.160 ± 0.006 | 0.384 ± 0.046 |
| Linker | MLP | 0.150 ± 0.007 | 0.453 ± 0.054 |
| E3 ligand | XGB | 0.174 ± 0.020 | 0.249 ± 0.061 |
| E3 ligand | MLP | 0.160 ± 0.019 | 0.362 ± 0.071 |
GNN models training is ongoing and its results remain pending. Compared with warhead and E3 ligand grouping, linker grouping performs higher in for both models, possibly suggesting a memorization of repeating PROTAC components to predict the score.
Canonical isomeric SMILES give these exact component overlaps (% of query molecules):
| Query → reference | Warhead | Linker | E3 ligand |
|---|---|---|---|
| Held-out → development | 79.57 | 71.01 | 76.79 |
| Enumerated → development | 52.51 | 67.31 | 56.39 |
| Enumerated → held-out | 8.68 | 13.19 | 23.07 |
Whole-molecule overlap is zero for held-out versus development and enumerated versus development. However, three of 1,303,154 enumerated molecules overlap the held-out set.
Q7. Cost and prediction quality. We scored the same 10,000 PROTACs, sampled uniformly without replacement from the enumerated library, three times per model. Jobs used A100 80-GB GPUs, eight EPYC 7742 CPU cores, batch size 256 and chunks of 2,000. Timings include featurization and inference but exclude loading/warm-up.
| Model | Time (s), mean ± SD | Molecules/s | Predicted score, mean ± SD | RMS difference vs 32-member | Random-panel RMSE |
|---|---|---|---|---|---|
| Single GNN | 23.83 ± 0.08 | 419.66 ± 1.34 | 0.565 ± 0.194 | 0.136 | 0.330 |
| 27-member ensemble | 384.00 ± 3.51 | 26.04 ± 0.24 | 0.531 ± 0.188 | 0.058 | 0.294 |
| 32-member ensemble | 430.76 ± 2.84 | 23.22 ± 0.15 | 0.544 ± 0.182 | 0 | 0.305 |
SD is across runs for time and across molecules for scores. RMS difference measures disagreement with the 32-member predictions, not error against reference labels. The ensembles share 72 top-100 picks, versus 25 for GNN and the 32-member ensemble.
For quality, RMSE compares predictions on the random 100-molecule panel against mean route scores from three 30-minute searches. These estimates concern averaged route scores on 100 molecules, not all 1.3M. The ensembles cost 16-18 times more and have lower estimated error, although this small panel leaves their ranking uncertain. Library timing is still unmeasured.
Q8. Filtering low predictions. Negative regression outputs are not probabilities or proof of infeasibility. In the random 100-molecule panel, discarding RetroTAC predictions below 0.5 removes 33% of candidates and loses 15.1% of resolved observations. At 0.7, it removes 84% and loses 77.3% of resolved observations. Estimates average three seeds per molecule. We recommend prioritization rather than treating 0.7 as a safe rejection threshold. This small panel does not establish library-wide calibration.
We will include the additional experiments and results of this rebuttal response to the manuscript. We hope our answers address the reviewer's concerns and that they consider raising their score.
I sincerely thank the authors for doing a strong job addressing my concerns, and I understand that not all experiments could be performed in the limited time window. I just wanted to provide enough opportunities to improve the manuscript. I especially appreciate the Q1 response to see how RetroTAC performs against other methods, which despite lower diversity (not the most informative metric anyway) its overall performance is strong compared to othhers. I will maintain my score of 3 (simply because the paper would be better with additional experiments) but will advocate for its acceptance during the AC/Reviewer discussion.
We thank the reviewer for the interesting questions and insights to improve the manuscript. We will respond with several comments.
We agree that practical screening utility and chemical validity require separate evidence. These analyses assess prioritization under a fixed planner and stock; they do not establish experimental synthesis success.
Q1. Comparison under identical planning budgets. Learning a continuous retrosynthesis-derived score has precedent in RSPred. Its underlying Spaya score is proprietary, although experimental code is open source. Our contribution is a PROTAC-focused target, surrogate and evaluation.
We sampled 10,000 enumerated PROTACs uniformly without replacement, selected the top 100 from RetroTAC, DeepPSA, RAscore and generic FSscore, and sampled 100 random controls. Their union has 479 molecules. Each received three ShallowTree runs at 30 minutes (shared candidates reuse planning results). FSscore here is the generic model, not SCScore or the PROTAC-adapted model in the FSscore paper. RetroTAC uses the 27-member RetroTAC ensemble.
Resolution includes all 100 candidates per method per seed, and depth averages resolved molecule-seed observations. Diversity is 1−T, where T is pairwise Morgan-fingerprint Tanimoto similarity (radius 2, 2,048 bits, chirality), excluding self-pairs. Scaffolds are non-generic whole-molecule Bemis-Murcko scaffolds: atom and bond types, and specified scaffold stereochemistry, are retained.
We can see that RetroTAC improves mean resolution by 35.7 percentage points over random and 11.0 over RAscore, with shallower routes. However, its fingerprint diversity is lower than every method, despite retaining 100 distinct scaffolds. Its picks contain 58 warheads, 81 linkers and 28 E3 ligands, versus 91, 95 and 66 for random selection. Higher planning yield comes with reduced component diversity. To estimate uncertainty, we resample molecules with replacement 2,000 times, keeping each molecule's three seed outcomes together. The 2.5th and 97.5th percentiles give 95% bootstrap intervals: 94-100% for RetroTAC and 52.3-70.7% for random.
Q2. Chemical interpretation. The example is correct: y=(L+B)/2, L=max(0,1−(D−1)/10), and fallback B=0.5 give 0.75 for the one-step case and 0.95 for the balanced two-step case. Balance does not encode coupling difficulty, protecting groups, selectivity or cost. Against depth alone, Appendix C reports ρ=0.723 and top-decile Jaccard overlap 0.289 over 16,318 routes.
The exact example occurs infrequently: the development set contains one one-step fallback route and four perfectly balanced two-step routes, while neither case occurs in the held-out set. Nevertheless, this measures changed ranking, and the score can favor longer, more balanced routes, so rarity of the fallback does not establish a broader chemical validity.
Due to the limited time frame, we were unable to perform systematic experts or documented-synthesis validation.