acceptodds
Under review as a conference paper at ICLR 2027

OpenDDI: A Comprehensive Benchmark for DDI Prediction

Abstract

Drug–Drug Interactions (DDIs) significantly influence therapeutic efficacy and patient safety. As experimental discovery is resource-intensive and time-consuming, efficient computational methodologies have become essential. The predominant paradigm formulates DDI prediction as a drug graph-based link prediction task. However, further progress is hindered by two fundamental challenges: (1) lack of high-quality data: most studies rely on small-scale DDI datasets and single-modal drug representations; (2) lack of standardized evaluation: inconsistent scenarios, varied metrics, and diverse baselines. To address the above issues, we propose OpenDDI, a comprehensive benchmark for DDI prediction. Specifically, (1) from the data perspective, OpenDDI unifies 6 widely used DDI datasets and 2 existing forms of drug representation, while additionally contributing 3 new large-scale LLM-augmented datasets and a new multimodal drug representation covering 5 modalities. (2) From the evaluation perspective, OpenDDI unifies 20 SOTA model baselines across 3 downstream tasks, with standardized protocols for data quality, effectiveness, generalization, robustness, and efficiency. Based on OpenDDI, we conduct a comprehensive evaluation and derive 11 valuable insights for DDI prediction while exposing current limitations to provide critical guidance for this rapidly evolving field. Our code is available at https://anonymous.4open.science/r/anonym1234

Primary area:
datasets and benchmarks
Keywords:
Drug–Drug Interactions, Benchmark, Multimodal drug representations, DDI datasets

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xfeaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenDDI, a comprehensive benchmark for drug–drug interaction prediction. It integrates six existing DDI datasets and three newly constructed large-scale datasets, together with multiple drug representations including a five-modality representation. The benchmark covers twenty representative prediction methods and three task formulations: binary, multiclass, and multilabel DDI prediction. Beyond standard predictive performance, the benchmark evaluates model generalization to unseen drugs, computational efficiency, and robustness to noise and sparsity. The authors further summarize a number of empirical observations regarding dataset scale, multimodal representations, model performance, generalization, and robustness.

Critical strengths

S1. The benchmark has broad coverage. It brings together multiple DDI datasets, twenty representative methods, several drug representations, and three commonly studied task formulations within a unified evaluation framework.

S2. The evaluation goes beyond conventional in-distribution predictive accuracy. The analyses of unseen-drug generalization, robustness to noise and sparsity, and computational efficiency provide useful complementary perspectives on current DDI prediction methods.

S3. The unified implementation and data resources could be useful to the DDI community by reducing inconsistencies caused by heterogeneous datasets, task definitions, and experimental protocols.

Critical weaknesses

W1. The quality of the LLM-augmented interactions is not fully established. The pipeline uses MRCGNN for candidate screening followed by Qwen3 Medical GRPO for validation, which may bias the augmented set toward interactions favored by the screening model. Independent expert review or external database validation would strengthen confidence in these labels.

W2. The evaluation metrics could better reflect the strong class imbalance in DDI prediction. Accuracy for binary prediction and Macro-AUC for multilabel prediction may obscure performance on rare interactions. Precision–recall-based metrics would provide a more informative evaluation.

W3. The unseen-drug setting is not described in enough detail. It is unclear how many test pairs contain one unseen drug versus two unseen drugs, and how relational or textual features for held-out drugs are constructed.

Questions for authors

Q1. Could the authors independently validate the LLM-augmented interactions, for example an external database? Results on curated and augmented data separately would also be helpful.

Q2. In the unseen-drug setting, how many test pairs contain one unseen drug versus two unseen drugs? Reporting these cases separately would better characterize generalization difficulty.

Q3. How much overlap is there between the three new OpenDDI datasets and the six existing datasets in terms of drugs and interaction pairs?

Q4. How were hyperparameter-search budgets controlled across the twenty methods? Are the main model rankings stable across tuning budgets and random seeds?

Reviewer_5AtGOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a Drug-Dug Interaction benchmark, incorperating more data and wider range of evaluation than previous benchmark. In specific, OpenDDI unifies 6 widely used DDI datasets and 2 existing forms of drug representation and 3 new large-scale LLM-augmented datasets and a new multimodal drug representation covering 5 modalities. The benchmark

Critical strengths

This work fill the gap that current work lack of large scale and multimodal evaluation on DDI prediction and reveal that the existing work and previous benchmark not perform very well under large scale experiment.

The experiment is also insightful as the authors discover that the SMILES has the highest gains in performance, this provides a potential guide for later works.

The benchmark covers multiple experiment and evalaution setting and shows grounded evidence to support their claims.

Critical weaknesses

The LLM augmented dataset may require calibration and cross-validate to increase reliability of dataset. Although the paper report LLM annotation has 7.6% FPR and 11.7% FNR, such FPR is still very high for DDI.

Only one MRCGNN network is used to select potential DDI pairs, then an LLM is used for annotation, however, it may introduce inductive bias from MRCGNN. The potential DDI pairs should be selected by multiple pretrained models to reduce such bias. Or external source database is used to validate DDI pairs such as DrugBank.

DDI prediction is naturally a long-tailed distribution problem, but the benchmark seems ignore this experiment setting. In particular, MRCGNN is designed to address long-tailed distribution of DDI. it is less likely to claim that the benchmark is comprehensive and practical without comparison of long-tailed setting.

There are additional line works:

[1] Devil in the tail: a multi-modal framework for drug-drug interaction prediction in long tail distinction. [2] MHAFR-DDI: A multimodal hierarchical attention fusion and relation-aware architecture for drug–drug interaction event prediction [3] ZSCAN-DDIE: an interpretable zero-shot learning method for the prediction of drug-drug interaction events using biomedical text [4] ProAlign-DDI: A Zero-Shot Drug–Drug Interaction Event Prediction Framework with Prototype Alignment

This line work aligns multimodal scope of the benchmark and it provide long-tailed distribution evaluation. An additional evaluation on long-tailed distribution would be a plus for this benchmark. In particular, [1] provides different setting of long-tailed modelling that should be discussed in the benchmark.

Questions for authors

The prompt for LLM annotation is simply ask the model to answer YES/NO, how can the author ensure that the model reasons based on pathology evidence and evaluate the LLM hallucination?

LLM annotation shows high FPR rate. with large scale dataset, have the authors sample and check the reliability of synthetic data with human expert? This particularly important for medicine domain.

Reviewer_sSvmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OpenDDI, a benchmark for drug drug interaction prediction. It integrates six existing datasets and constructs three new datasets with LLM-assisted annotations. The benchmark includes five drug feature modalities and 20 baseline methods across three prediction tasks. The authors evaluate model effectiveness, generalization, robustness, and efficiency. The experiments provide useful insights into multimodal drug representations and the limitations of existing DDI prediction methods.

Critical strengths

1.The integration of 9 datasets and 20 baseline methods into a unified framework is valuable. It provides a useful resource for standardized evaluation and comparison of DDI prediction methods. 2.The experiments cover multiple aspects of model performance. The multimodal ablation studies and generalization analysis provide useful findings for future DDI prediction research. 3.OpenDDI provides an open-access data and code platform with a unified interface, which could contribute to further research and development in DDI prediction.

Critical weaknesses

1.OpenDDI primarily integrates existing DDI datasets, drug representations, and baseline methods into a unified benchmark, without offering substantial theoretical insights or method innovation. 2. The authors treat interactions inferred by predictive models and LLMs as benchmark ground truth. I consider this practice methodologically questionable without independent biological validation. In particular, the reported FPR of 7.6% and FNR of 11.7% in Table 5 make me doubt about the benckmark. 3.The dataset statistics and experimental results are not reported with sufficient rigor. The reported 5.3 million DDIs and 34,000 drugs are aggregated across three task-specific datasets of Table 2, but the authors do not specify whether overlapping drugs and drug pairs were identified and excluded when calculating these totals. Moreover, some improvement rates in Table 3 are inconsistent with the reported values. 4. Although OpenDDI claims to integrate 20 SOTA baseline models, only 6–12 models are included in most key experiments (Tables 3–4 and Figures 3–6). The absence of many baselines from these evaluations undermines the claimed comprehensiveness of the benchmark.

Questions for authors

1.What specific feature encoders are used for the five drug modalities ? How is the modality fusion function in Equations (1)–(2) implemented? 2.How are models that rely on molecular graph structures or specific chemical substructures, such as DSNDDI, CASTER, and GoGNN, adapted to the unified feature representations? Does this adaptation preserve their original architectures and core mechanisms, thereby ensuring a fair comparison? 3.Why are the OpenDDI datasets excluded from generalization and robustness evaluations? 4.Does the data splitting protocol explicitly prevent information leakage during graph construction and unseen-drug evaluation? Are test interactions excluded from the training graph? 5.How are negative samples constructed for binary DDI prediction? Are unobserved drug interactions directly treated as negative sample?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.