- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OpenDDI, a comprehensive benchmark for drug–drug interaction prediction. It integrates six existing DDI datasets and three newly constructed large-scale datasets, together with multiple drug representations including a five-modality representation. The benchmark covers twenty representative prediction methods and three task formulations: binary, multiclass, and multilabel DDI prediction. Beyond standard predictive performance, the benchmark evaluates model generalization to unseen drugs, computational efficiency, and robustness to noise and sparsity. The authors further summarize a number of empirical observations regarding dataset scale, multimodal representations, model performance, generalization, and robustness.
Critical strengths
S1. The benchmark has broad coverage. It brings together multiple DDI datasets, twenty representative methods, several drug representations, and three commonly studied task formulations within a unified evaluation framework.
S2. The evaluation goes beyond conventional in-distribution predictive accuracy. The analyses of unseen-drug generalization, robustness to noise and sparsity, and computational efficiency provide useful complementary perspectives on current DDI prediction methods.
S3. The unified implementation and data resources could be useful to the DDI community by reducing inconsistencies caused by heterogeneous datasets, task definitions, and experimental protocols.
Critical weaknesses
W1. The quality of the LLM-augmented interactions is not fully established. The pipeline uses MRCGNN for candidate screening followed by Qwen3 Medical GRPO for validation, which may bias the augmented set toward interactions favored by the screening model. Independent expert review or external database validation would strengthen confidence in these labels.
W2. The evaluation metrics could better reflect the strong class imbalance in DDI prediction. Accuracy for binary prediction and Macro-AUC for multilabel prediction may obscure performance on rare interactions. Precision–recall-based metrics would provide a more informative evaluation.
W3. The unseen-drug setting is not described in enough detail. It is unclear how many test pairs contain one unseen drug versus two unseen drugs, and how relational or textual features for held-out drugs are constructed.
Questions for authors
Q1. Could the authors independently validate the LLM-augmented interactions, for example an external database? Results on curated and augmented data separately would also be helpful.
Q2. In the unseen-drug setting, how many test pairs contain one unseen drug versus two unseen drugs? Reporting these cases separately would better characterize generalization difficulty.
Q3. How much overlap is there between the three new OpenDDI datasets and the six existing datasets in terms of drugs and interaction pairs?
Q4. How were hyperparameter-search budgets controlled across the twenty methods? Are the main model rankings stable across tuning budgets and random seeds?