- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper curates a dataset and benchmark for molecular glue degrader activity prediction. The authors trace records from four databases back to primary sources, producing 19,815 records (5,170 exact-label, 4,723 interval, 9,922 unlabeled) across four endpoints, then evaluate 45 representation×model configurations under random, scaffold, and target-cluster splits. Mean AUROC drops 0.895 → 0.839 → 0.561; mean R² 0.646 → 0.499 → −0.190. I review as an ML researcher without domain expertise, the value of this paper lies in a set of diagnostic analyses of the generalization gap.
Critical strengths
- The diagnostic analyses go well beyond typical benchmark papers: a 1-NN retrieval control, modality-randomization controls, label-precision and cell-line ablations, and a separation of calibration from selective-prediction utility.
- The support-stratified AUROC decomposition — attributing 74.4% of EC50’s net discrimination to pairs where both samples have high chemical support — is an original and reusable protocol for any work claiming OOD generalization.
Critical weaknesses
-
No variance is reported anywhere, and the target-cluster setting has too few independent entities to support its conclusions.
-
Curation quality is asserted but never quantified. The central claim is that manual curation improves on four existing databases; the evidence is four case studies, with no inter-annotator agreement, error rate from independent re-annotation, or systematic diff against sources. For a dataset paper this is the most important missing experiment.
-
The benchmark discards its most distinctive property in the first preprocessing step: §4.1 collapses interval and censored measurements to midpoints or reported bounds (79% of IC50 records). Intervals should not be collapsed to point values — ML already has the right tools for this and here the censoring is informative rather than incidental: a compound is written as “> c” precisely because it is too weak to quantify. Substituting the bound therefore piles the negative class onto reporting conventions like 1000 and 10000 nM, which amounts to manufacturing labels rather than recording them.
-
The benchmark has no headroom gradient, and the paper is not positioned in the ML literature. All 45 configurations fall in ~[0.5, 0.7] AUROC on target-cluster with uniformly negative R². Trivial baseline ranks first even cannot measure progress. Separately, there is no engagement with DrugOOD, TDC/WILDS, or cold-start splits in drug-synergy prediction, which is often used in ML community.
Questions for authors
-
Please report cross-fold variance for all main tables and per-fold target-cluster results; and for that split, report Spearman/MAE plus a predict-the-training-mean baseline.
-
Could you quantify curation quality?
5 comments
Comment 2/5
Q2: The target-cluster setting has too few independent entities to support its conclusions.
Response:
Thank you for highlighting target coverage. In public data, relatively few molecular glue degrader targets have both endpoint-specific quantitative measurements and complete molecular and protein information. DC50/Dmax/EC50/IC50 cover 37/22/28/47 targets and 1,626/1,077/1,177/2,641 distinct molecules, respectively. Thus, the supervised data contain many molecules across relatively few targets. These data reflect our best efforts to curate available public records; we will continue adding measurements for new targets.
Records without quantitative activity labels are retained as Type 3 data and used for pretraining. Type 3 covers 168 targets, including 112 absent from all four supervised endpoints.
Target-cluster splitting, a scaffold split at the target-protein level, represents our hardest task with strong translational implications. To strengthen our conclusions, we added a less stringent three-fold Target-ID split (corresponds to “random” splitting of target proteins, where the training and test sets contain distinct target IDs without enforcing cluster-based grouping) and a Leave-one-target-out evaluations (all compounds of a single target are reserved for testing in turn). Target-ID keeps all samples for each target in one fold while allowing similar targets across folds. Mean AUROCs for Scaffold/Target-ID/Target-cluster were 0.839/0.622/0.561. Leave-one-target-out evaluation, training on all remaining targets, yielded 0.633. Both evaluations support our main finding that generalizing to new targets is more challenging than to new scaffolds. Complete fold-level results have been added to the anonymous repository.
Q3: Please report cross-fold variance for all main tables and per-fold target-cluster results; and for that split, report Spearman/MAE plus a predict-the-training-mean baseline.
Response:
For space reasons, the original manuscript reported only fold-averaged results. At submission, the anonymous repository already included cross-fold SDs, Spearman correlation, RMSE, and MAE. We added per-fold results for all splits, cross-fold variances and SDs, and all regression metrics to the repository. We also added SDs to the revised manuscript tables.
(1) Cross-fold variance. To illustrate cross-fold variance for the reviewer, we report example AUROC results (mean ± SD). For each endpoint, we select the configuration with the highest mean AUROC on the Random split. Our goal is to show how its performance drops across generalization settings of varying difficulty.
| Endpoint | Random ↑ | Scaffold ↑ | Target-ID ↑ | Target-cluster ↑ |
|---|---|---|---|---|
| DC50 | .886 ± .007 | .816 ± .041 | .623 ± .097 | .567 ± .171 |
| Dmax | .899 ± .019 | .816 ± .047 | .604 ± .042 | .570 ± .195 |
| EC50 | .954 ± .009 | .929 ± .043 | .809 ± .131 | .585 ± .266 |
| IC50 | .936 ± .011 | .849 ± .048 | .631 ± .083 | .564 ± .115 |
(2) Per-fold Target-cluster results: Spearman ρ, MAE, and R². We show examples of per-fold regression results. For each endpoint, we select the configuration with the lowest mean MAE on the Random split. Values are listed in fold order 1/2/3.
| Endpoint | Spearman ρ ↑ | MAE ↓ | R² ↑ |
|---|---|---|---|
| DC50 | .277 / .419 / −.354 | .591 / .694 / 1.083 | .088 / .075 / −.256 |
| Dmax | −.016 / .432 / .083 | 36.954 / 21.901 / 21.944 | −2.676 / −.084 / −.088 |
| EC50 | .694 / .304 / −.028 | .653 / 1.065 / .953 | .287 / .085 / −.079 |
| IC50 | .140 / .112 / .324 | .925 / .907 / .890 | .016 / −.376 / −.062 |
(3) Predict-the-training-mean baseline. For each Target-cluster fold, the baseline predicts the mean of the training labels.
| Metric | DC50 | Dmax | EC50 | IC50 |
|---|---|---|---|---|
| R² ↑ (mean ± SD) | −.041 ± .030 | −.353 ± .377 | −.811 ± .624 | −.290 ± .088 |
| MAE ↓ (mean ± SD) | .790 ± .171 | 24.040 ± 1.970 | 1.540 ± .260 | .954 ± .075 |
MAE is in pActivity units for concentration endpoints and percentage points for Dmax.
Our responses continue in Comment 3/5.
Comment 3/5
Q4: The benchmark has no headroom gradient. All 45 configurations fall in ~[0.5, 0.7] AUROC on target-cluster with uniformly negative R². Trivial baseline ranks first even cannot measure progress.
Response:
Thank you for highlighting the importance of measuring progress on this benchmark.
In our initial design, we implemented 3 evaluation schemes: molecular Random/Scaffold split and Target-cluster split. The scaffold split simulates the discovery of novel high-activity molecular glues against known disease targets for which some labeled molecular glue compounds are already available. By contrast, the Target-cluster, can be viewed as the “scaffold” split at the target-protein level. Target proteins are grouped by sequence and structural similarity and partitioned by clusters, which mimics the discovery of molecular glue candidates for novel disease targets with no prior known molecular glues. This setting resembles a zero-shot learning setup for new targets, represents our most challenging evaluation task and carries substantial practical implications for advancing healthcare. The weak predictive performance observed under this cross-target setting computationally corroborates the inherent difficulty of this task. Our MGDEval establishes cross-target activity prediction as a challenging benchmark task for the ML community, and highlights that the molecular glue paradigm opens up a vast technical frontier for addressing novel, traditionally undruggable targets, with ample room for future methodological advances. We believe this is the core significance and value of our benchmarking work.
Although quantitative prediction under Target-cluster remains challenging, the evaluated models outperform the training-mean baseline on average, with mean R² of −0.190 versus −0.374. This shows that the models retain predictive information even in the most demanding setting, and may inspire future researchers to develop more sophisticated AI models for cross-target molecular glue prediction.
Furthermore, inspired by your comment, to provide a clearer difficulty gradient, we added the milder Target-ID (corresponds to “random” splitting of target proteins, where the training and test sets contain distinct target IDs without enforcing cluster-based grouping) and Leave-one-target-out evaluations (all compounds of a single target are reserved for testing in turn) described in Q2. Their mean AUROCs of 0.622 and 0.633, compared with 0.561 under Target-cluster, provide complementary evaluation settings with different levels of difficulty. These settings allow future methods to be assessed across both related-target transfer and more demanding target-cluster generalization.
In brief, our MGDEval provides curated data, a common evaluation protocol, and baselines for measuring progress. Covering five distinct evaluation scenarios ranging from molecular-level Random split and Scaffold split to target-centric Leave-one-target-out, random Target-ID split and Target-cluster split, this benchmark — refined based on your professional comment — establishes a spectrum of difficulty gradients for molecular glue discovery. The biomedical challenges and practical relevance increase progressively across these settings, which naturally leads to a gradual decline in model performance. This trend clearly reveals the substantial room and strong necessity for further research in these cross-target tasks. We believe this multi-granularity evaluation framework can serve as a solid benchmark and reference for future AI-driven molecular glue discovery.
In addition, our failure analyses and ablations examine how protein representations, label quality, and cellular information affect performance. We also test pretraining on unlabeled molecule–protein associations. These results point to learning transferable interactions among molecules, targets, and effectors as a direction for improving cross-target generalization (§5.1–5.3).
Our responses continue in Comment 4/5.
Comment 1/5
Response:
We thank the reviewer for this question. While existing databases collect broad MG records, MGDEval focuses on MG degraders with activity labels about DC50/Dmax/IC50/EC50. Through manual verification, completion and standardization, we convert raw entries into an AI-ready dataset with unified prediction tasks, data splits and baselines. Our curation contributions fall into five areas.
(1) Organizing curated records by supervision signals. Raw MG records from MGDB, MGTbind, MolGlueDB and TPDdb are mixed with inconsistent activity labels and cannot be directly used for AI. We conducted extensive curation to stratify them into 3 tiers (19,815 records). Type 1 has exact activity labels for supervised regression and classification (5170 records). Type 2 includes interval/censored values for weak supervision (4723 records), and offers clear classification labels when activity class is conclusive (≥100nM or ≤100nM). Type 3 covers unlabeled target–molecule–effector associations for unsupervised pretraining (9922 records). This multi-supervision hierarchy enables AI modeling across diverse supervision regimes.
(2) Recovering patent activity labels and adding quantitative activity information. In MGTbind and TPDdb, 318 DC50/Dmax records reported activity only as B/C/D grades. We revisited the experimental results and grade definitions in the original patents to recover numerical or interval labels suitable for supervised learning, including 138 records from WO2024092039A1. Another 206 records used “+” symbols whose numerical ranges could not be verified; these were included in the unsupervised data. Overall, by reviewing papers, supplementary materials, and patents, we added exact-value activity data absent from the 4 source databases for 817 distinct records, e.g., we retrieved 7 IC50 measurements from WBC100 across varied cellular backgrounds from a single patent.
(3) Separating mixed records, completing protein sequences, and rectifying molecular structures. We split 74 multi-effector entries in MGDB and 64 multi-target entries in MolGlueDB by tracing original papers to match each measurement with its target and effector. We filled missing target/effector sequences for 1,461 supervised samples from MolGlueDB and TPDdb. Among all sources, 1,481 records had invalid SMILES incompatible with feature extraction. Via RDKit validation, desalting, standardization and literature cross-check, we rectified and standardized 1,038 molecular structures. For unsupervised data, we further supplemented 2,872 target sequences, 2,352 effector sequences and rectified the structures of 661 molecules.
(4) Collecting experimental conditions and building standardized context annotations. The 4 source databases lacked experimental-condition descriptions linked to individual activity measurements. We manually reviewed the source papers and patents, matching conditions scattered across the main text, methods, figure captions, and supplementary materials to the corresponding measurements. This completed 1,739 cell-line fields and 941 treatment-duration fields in the labeled data. Based on the retrieved information, we organized 32 categories of keyword fields, including detection methods, assay readouts, treatment duration, dosing, and experimental constructs. These fields standardized fragmented and inconsistently worded descriptions into a uniform context format, producing experimental-context annotations for 13,696 records.
(5) Correcting activity values, and standardizing/completing units. We checked the experimental descriptions and results in the source papers and patents to correct 36 records from IC50 to EC50, perform 439 concentration-unit conversions, and supply standardized unit annotations for 1,581 records without explicit units. We also corrected activity values verified against the original sources, e.g., JWJ-01–306’s reported degradation was corrected from 0.6% (MGDB) to 60%.
Quantified improvements in field coverage compared with the four source databases.
For items not reaching 100%, the required information is absent from original papers and patents, and we have made our best attempt at data recovery.
Annotation error rate in the random audit. During curation, all records underwent multiple rounds of checks. In the final random audit, we checked 6,848 of the 13,696 records with experimental-context annotations against the original papers or patents and identified annotation errors in nine records (0.13%). We corrected the flagged records, screened the full dataset, and incorporated the resulting corrections into the final version.
Our responses to the remaining questions continue in Comments 2/5–5/5.