- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BiOE for predicting many-to-many relations. It uses separate hyperbolic spaces to model the two entailment directions. The full model also adds order information estimated from the data and a residual component. Experiments cover recommendation, protein-ligand retrieval, and protein function prediction.
Critical strengths
- Using separate spaces for the two directions is a reasonable design. It avoids forcing one type of entity to always be above the other in the hierarchy.
- The method is tested on three different tasks and improves several metrics. The recommendation results provide useful evidence that modeling both directions can help.
Critical weaknesses
- The reason for using hyperbolic space needs to be clearer. Many-to-many relations do not necessarily form a hierarchy. The authors should explain why the relations in these datasets are suited to hierarchical modeling in hyperbolic space, and provide evidence from the data to support this choice.
- The comparisons could better show the benefit over existing graph models. LightGCN is already included. The authors should also consider comparisons with heterogeneous GNNs, which can model different node types and relations. Using the same input information and similar model sizes would help clarify what the proposed method adds.
Questions for authors
- The reason for using hyperbolic space needs to be clearer. Many-to-many relations do not necessarily form a hierarchy. The authors should explain why the relations in these datasets are suited to hierarchical modeling in hyperbolic space, and provide evidence from the data to support this choice.
- The comparisons could better show the benefit over existing graph models. LightGCN is already included. The authors should also consider comparisons with heterogeneous GNNs, which can model different node types and relations. Using the same input information and similar model sizes would help clarify what the proposed method adds.
3 comments
Response to Reviewer ohmL (Part 2 of 2)
(Q2, W2) Comparison with heterogeneous GNNs
We thank the reviewer for this suggestion. BiOE models are based on HICF, which is a hyperbolic LightGCN variant, and BiOE loss terms operate only on the observed user item interactions, so it uses a single type of relation and requires no additional node types, edge types, attributes or knowledge graph relations. We therefore compare BiOE to two additional heterogeneous GNNs, Simple-HGN[b] and R-GCN[c], on exactly the same interaction graph and data splits, with the two interaction directions (user to item and item to user) as their only relation types. Both heterogeneous GNNs use embedding dimension 256, the same as HICF and BiOE. The results are average over 3 runs, where STD is omitted for presentation clarity. The results are shown below, where BiOE outperforms both heterogeneous GNNs in nearly all cases, demonstrating that BiOE can outperform heterogeneous GNNs even with a simpler backbone. These results are added to Table 1 and Section 5.1 of the revision.
| Model | Amazon-book | Yelp | Steam | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@5 | R@20 | N@5 | N@20 | R@5 | R@20 | N@5 | N@20 | R@5 | R@20 | N@5 | N@20 | |
| R-GCN | 0.0444 | 0.1118 | 0.0444 | 0.0668 | 0.0302 | 0.0852 | 0.0352 | 0.0533 | 0.0461 | 0.1229 | 0.0514 | 0.0772 |
| + LLM Alignment | 0.0502 | 0.1139 | 0.0462 | 0.0669 | 0.0298 | 0.0888 | 0.0357 | 0.0560 | 0.0463 | 0.1262 | 0.0523 | 0.0804 |
| Simple-HGN | 0.0871 | 0.1767 | 0.0883 | 0.1104 | 0.0448 | 0.1213 | 0.0519 | 0.0772 | 0.0548 | 0.1396 | 0.0601 | 0.0889 |
| + LLM Alignment | 0.0915 | 0.1801 | 0.0906 | 0.1129 | 0.0449 | 0.1249 | 0.0511 | 0.0803 | 0.0560 | 0.1461 | 0.0647 | 0.0920 |
| BiOE$_{Bi}$ | 0.0810 | 0.1761 | 0.0825 | 0.1135 | 0.0467 | 0.1317 | 0.0548 | 0.0824 | 0.0603 | 0.1484 | 0.0636 | 0.0996 |
| + LLM Alignment | 0.0833 | 0.1819 | 0.0852 | 0.1170 | 0.0491 | 0.1347 | 0.0570 | 0.0850 | 0.0651 | 0.1536 | 0.0696 | 0.1026 |
| BiOE$_{PO}$ | 0.0841 | 0.1788 | 0.0828 | 0.1145 | 0.0491 | 0.1330 | 0.0559 | 0.0837 | 0.0631 | 0.1495 | 0.0651 | 0.1010 |
| + LLM Alignment | 0.0862 | 0.1849 | 0.0859 | 0.1183 | 0.0503 | 0.1366 | 0.0581 | 0.0870 | 0.0688 | 0.1551 | 0.0722 | 0.1047 |
To ensure a fair comparison, we tune both heterogeneous GNN baselines extensively with grid search, with the search ranges listed in Appendix B.4.1 of the revision
[b] Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, Jie Tang. Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks. KDD 2021.
[c] Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, Max Welling. Modeling Relational Data with Graph Convolutional Networks
Follow Up and Kind Reminder
Dear Reviewer ohmL,
Thank you again for your detailed review and insightful suggestions. As the discussion period is coming to an end, we would like to kindly follow up on our response. We have added additional heterogenous GNN baselines and demonstrated BiOE advantages in comparison, and added several numerical and intuitive analysis demonstrating the presence of hierarchical structures in all datasets used in the paper to justify the use of hyperbolic geometry. We hope they have fully addressed your concerns; if so, we would greatly appreciate it if you would consider updating your assessment. We truly appreciate your time and effort in reviewing our paper.
Response to Reviewer ohmL (Part 1 of 2)
We thank the reviewer for validating the design and breadth of our method and experiments. We provide clarifications and additional results below.
(Q1, W1) The reason for using hyperbolic space needs to be clearer
We’d like to first highlight that hierarchy and many-to-many structure are two separate properties that motivate two separate design choices. The hierarchical structure present in these datasets (discussed below) is what motivates hyperbolic models in the first place. The many-to-many nature of the relations is what motivates BiOE as prior hyperbolic baselines face limitations: as neither side is uniformly more general than the other, a single entailment direction imposes the wrong inductive bias, which BiOE corrects by modeling both directions. The motivation for using hyperbolic geometry in these tasks come from the structures of the underlying dataset and domain.
In recommender systems, interactions are long-tailed, where there are a few popular items that are consumed by most users with a long tail of niche items, and a few highly active users interact broadly while most users are much less active. This gives a tree-like structure, with few general nodes and many specific ones, on both sides. Below we show δ-hyperbolicity values for all three recommendation datasets used in the paper. δ-hyperbolicity measures how similar a dataset is to a perfect tree, with lower values meaning more tree like.
As shown, the 3 datasets have low hyperbolicity values, indicating the relationships are tree like. Furthermore, to quantify the long-tail distribution of items and users, we employ the Gini coefficient, which is 0 when every node has the same number of interactions, and approaches 1 as interactions concentrate on a few nodes. Additionally, we measure the nestedness index (NODF) [a] to test that this long tail induces hierarchical structure. NODF measures how often the interactions of tail nodes are contained in those of head nodes, with high nestedness means the relations form a general-to-specific hierarchy. We compare each dataset with a random graph with the same numbers of users, items and interactions.
As shown above, the interactions in the 3 datasets form non-trivial long-tail distributions as shown by the Gini coefficients and the long-tail distributions also induce a general-specific hierarchical structure as shown by the NODF ratios.
In protein function annotation, the GO labels are themselves organized as a curated hierarchy. It is a directed acyclic graph of general to specific functions, and a protein annotated with a term is also annotated with all of its ancestors. As a result, the proteins of a specific function are contained within those of its more general ones, while multifunctional proteins carry annotations that cover those of more specialized proteins, so the hierarchy extends to both sides of the relation. Below shows the δ-hyperbolicity, Gini coefficients, and NODF values for the protein annotation dataset, demonstrating that the protein-GO dataset is tree-like, exhibits long-tail distributions, and the long-tail distribution induces a general-specific hierarchy
Protein ligand binding also exhibits hierarchical structure, which prior work has used to motivate hyperbolic models of binding (Wang et al., 2026, Hypseek). Proteins are organized into families and superfamilies with conserved binding sites, so a ligand that binds a conserved site can bind many members of a family, while a selective ligand binds only a few of them. Likewise, a target can bind a broad range of ligands that covers the binders of more specialized targets. We have added these measurements and this discussion to Appendix C of the revision (Table 9), and refer to them at the beginning of Section 5.
[a] Mário Almeida-Neto, Werner Ulrich. A straightforward computational approach for measuring nestedness using quantitative matrices. Environmental Modelling & Software, Volume 26, Issue 2, 2011.