acceptodds
Under review as a conference paper at ICLR 2027

PlantMetBench: A Multimodal Knowledge Graph Benchmark for Biosynthesis Prediction

Abstract

Plant metabolism produces a vast diversity of molecules, yet identifying the enzymes that catalyse biosynthetic reactions remains a major open problem. Existing benchmarks typically reduce this problem to isolated enzyme-reaction pairs, overlooking pathway and metabolite context that determines biological plausibility, leaving no standardised way to evaluate approaches that exploit this richer multimodal and relational structure. We present PlantMetBench, a multimodal graph benchmark built on PlantMetWiki, a curated knowledge graph of plant metabolism spanning 424 organisms. The resulting benchmark graph comprises 13,682 proteins, 2,741 genes, 4,577 metabolites, and 11,980 experimentally curated enzyme-reaction pairs. PlantMetBench frames enzyme-reaction prediction as an organism-disjoint link-prediction task and provides graph-native dataset splits, precomputed multimodal node features, and evaluation protocols. Using PlantMetBench, we evaluate heterogeneous graph neural networks, standard architectures (HeteroSAGE, GAT, HGT, and R-GCN), alongside residual and jumping-knowledge variants, against non-graph models. Most heterogeneous graph architectures consistently outperform non-graph baselines on our rank-based evaluation, suggesting the value of relational context for enzyme-reaction prediction. Yet, performance remains far from saturated, indicating substantial room for improved representations and modelling strategies. Also, we show that benchmark design choices (shortcuts, splits, and metrics) are key to aligning evaluation with biologically meaningful tasks. PlantMetBench provides a standardised, reproducible framework to evaluate multimodal biosynthetic data, establishing a foundation for future work on graph-based architectures for enzyme-reaction prediction. Code and data are publicly available.

Primary area:
datasets and benchmarks
Keywords:
knowledge graph, GNN, multimodal, reaction prediction, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_krqzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce PlantMetBench, a multimodal knowledge graph benchmark for predicting enzyme–reaction associations in plant metabolism, with a focus on organism-disjoint link prediction. Built upon PlantMetWiki, the benchmark integrates proteins, genes, metabolites, and biochemical reactions, providing multimodal node features, reproducible dataset splits, and evaluation protocols. The authors evaluate heterogeneous GNN architectures against non-graph baselines, showing that relational information can improve enzyme–reaction prediction. Additional experiments investigate generalization across species and reactions, structural shortcuts, and pathway reconstruction. The results highlight both the potential of graph-based approaches and the remaining challenges in biosynthesis prediction

Critical strengths
  • PlantMetBench addresses a gap in biosynthesis prediction by providing a standardized multimodal knowledge graph benchmark. The integration of biological entities, domain-specific node features, reproducible data splits, and evaluation protocols makes it a potentially valuable resource for the computational biology and graph learning communities
  • The paper provides a comprehensive comparison of several GNN architectures and non-graph baselines. Beyond standard performance comparisons, the authors consider the evaluations also on different data-splitting strategies and includes ablation studies investigating the impact of graph structure and potential structural shortcuts
Critical weaknesses
  • A major challenge in molecular machine learning is designing dataset splits that reliably assess generalization beyond the training data. Although PlantMetBench adopts an organism-disjoint split, the substantial performance drop on unseen reactions highlights the importance of distinguishing cross-species transfer from generalization to novel reactions. While the authors acknowledge this limitation, the strong dependence on previously observed reaction information raises concerns about the extent to which the default evaluation captures broader generalization capabilities
  • A substantial number of nodes in the dataset lack feature representations and are instead assigned zero vectors. This may affect the performance of the evaluated architectures, making it difficult to determine whether the reported results reflect the inherent complexity of the task or the limited availability of input information
Questions for authors
  • Considering the substantial performance drop on unseen reactions, why is the organism-disjoint split preferred as the default evaluation setting? Would an unseen-reaction split provide a more comprehensive assessment of generalization to novel enzyme–reaction associations?
  • Due to the use of zero vectors to represent missing node features, how does this choice affect the performance of the evaluated architectures? It would be interesting to distinguish limitations arising from missing input information from those related to the inherent difficulty of the prediction task

Minor comments

  • The paper includes GAT among the heterogeneous GNN baselines, although the original GAT architecture was designed for homogeneous graphs. A brief clarification in the main text on how GAT is adapted to handle different node and edge types would be helpful
  • Some tables, including Tables 1, 2, and 3, contain vertical lines, which could be removed to improve readability
Reviewer_WusqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present PlantMetBench, a benchmark for enzyme-reaction discovery in plants based on the PlantMetWiki knowledge graph. They then continue to use this benchmark to test several discovery algorithms and propose that the knowledge gained from semantic connection improves performance, depending on the architecture used. In addition, the authors show that careless benchmark design can make models look good unreliably.

Critical strengths

The paper presents a much needed benchmark for these prediciton tasks, which can be used out-of-the-box to test models developed by labs working on diverse aspects of plant natural prodcts. The authors compare graph and non-graph models, which, while a bit contrived for the data type that they present (i.e., GNNs are not evaluated on the same data as the dual encoders), is an important contribution and done properly, as far as this reviewer was able to discern.

Critical weaknesses

As this reviewer is not an expert in this specific field, but rather in adjacent fields (genomics, ecology, some ML), I will voice my concerns as questions.

Questions for authors
  1. The benchmark uses a taxonomic holdout to test cross-species transfer. Does test performance degrade with phylogenetic distance between the held-out species and the training species, and if so, how? This is important if we want to interpert taxa holdout as a generalization rather than reshuffling.
  2. Performance on reactions absent from training is 12.6% P-H@50 while reactions seed during training under a different catalyst get 66%. The authors report 48.0% which is an aggregate metric. Which number should the reader use when judging if the benchmark is relevant for a truly novel reaction?
  3. About 2700 of 13682 proteins have a gene product, and PlantCaduceus covers about half of those, so only about 10% of the protein set have genomic information overall. What does this information contribute? Is there an ablation that shows that?
Reviewer_ugUKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PlantMetBench, a multimodal knowledge-graph benchmark for predicting which enzymes catalyse which reactions in plant metabolism. The authors argue that existing enzyme–reaction benchmarks treat the task as matching isolated enzyme–reaction pairs, ignoring the pathway, metabolite, gene, and organism context that determines biological plausibility, so there is no standard way to evaluate models that exploit this relational and multimodal structure. The benchmark is built from the PlantMetWiki knowledge graph and frames the task as ranking all candidate proteins for each reaction, with entire species held out for testing. The authors remove several structural shortcuts from the raw graph and provide alternative splits and metrics. They evaluate four heterogeneous GNNs and three variants against BLASTp and a dual-encoder baseline, and report that most GNNs outperform the non-graph baselines on the default split, though the dual encoder performs better on the A. thaliana reaction holdout. They conclude that relational context is valuable and that the benchmark is far from saturated.

Critical strengths
  • The authors identify specific structural shortcuts in the raw PlantMetWiki graph (pathway co-membership, the reverse catalysis edge, one-hop supervision) and ablate each one: the reverse-catalysis edge alone shifts test P-H@50 from 48.0% to 36.3% (Table 4). They also quantify what the default split actually measures, showing that 63% of test edges target reactions already catalysed by a training-organism protein, and that performance is 66.0% on these versus 12.6% on novel reactions (Appendix D.8). A benchmark's value depends on whether it leads to correct conclusions; by auditing leakage and reporting the hard subset openly, the paper gives future users the information needed to interpret results correctly, and its findings on how splits and shortcuts change conclusions are insightful beyond this dataset.
  • The benchmark frames enzyme–reaction prediction as ranking against the full pool of 8,445 proteins with whole species held out, which matches how candidate enzymes would be prioritised in practice, and the authors show why sampled-negative metrics mislead (GAT's CP-AUC is within 0.04 of the best model despite a fourfold lower P-H@50). It adds a pathway-level reconstruction metric, an external check against Rhea, and releases code, splits, and precomputed multimodal features. It fills a real gap left by pair-based enzyme–reaction benchmarks and is ready to use.
Critical weaknesses

The paper claims that relational context adds signal beyond sequence and reaction representations, but the models differ in more than graph structure. Most importantly, the GNNs receive PlantCaduceus gene embeddings, which the dual encoder and BLASTp do not, so the statement in Appendix H.3 that "the only variable … is the architecture" does not hold. This matters in practice: when held-out proteins lose their within-organism edges, which connect them mainly to their own gene products, the best GNN drops from 48.0% to 26.8% test P-H@50 (Table 15), below both BLASTp (32.4%) and the dual encoder (29.6%). The GNN's advantage may therefore come partly from the extra gene modality rather than from relational context. This paper's central conclusion is about the value of graph context, and the resulting leaderboard ordering, rest on this comparison; a non-graph baseline given the same input features would be needed to support it.

Questions for authors
  • How does a non-graph model perform when given the same protein-side inputs as the GNNs, i.e., ESM-C plus the encoding gene's PlantCaduceus embedding? If it closes most of the gap, the conclusion that relational context drives the improvement would need revising.
  • Table 1 reports the "best-performing depth" for each architecture. Which data split was used to make this selection? The same question applies to the hidden dimension in Figure 3 and to the residual and jumping-knowledge design in Section 4.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.