- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a method for learning hierarchical embeddings in hyperbolic space termed HoloWedge, which is based on the following procedure. Each node is represented by a wedge, which is a hyperbolic half-space associated with a spherical cap on the boundary. The node’s direction in the embedding sets the cap’s center, while its distance from the origin determines the cap’s size. Ancestor-descendant relations are represented by wedge nesting. Inspired by holographic entropy, the method combines a score for the relative arrangement of two wedges with a penalty that encourages parents to have larger wedges than their children. Node embeddings are learned from observed relations and sampled negative pairs using cross-entropy or hinge loss. Experimental results show accurate graph reconstruction and mixed link prediction performance, with less concentration of embedded points near the boundary than some existing Poincare embedding methods.
Critical strengths
-
The paper incorporates holographic ideas into learning hierarchical embeddings.
-
Nesting wedges gives a clear way to represent hierarchy.
-
The method trains from random initialization without pretraining or excluding a region around the center.
-
Reconstruction results are favorable across several datasets and embedding dimensions.
-
The resulting embeddings are less concentrated near the boundary than some existing Poincare embedding methods.
Critical weaknesses
-
The comparisons to baselines are somewhat limited. For example, box embeddings (Boratko et al. NeurIPS 2021) and order embeddings (Vendrov et al ICLR 2016), which are cited in the paper, are not tested.
-
Link prediction results are mixed. On WordNet, the method performs worse than entailment cones in some cases.
-
The benefit of the entropy-inspired score is unclear. There is no comparison with a simpler containment loss using the same wedges.
-
Lower concentration near the boundary is shown, but its practical benefit is not clear. The paper does not show that it improves numerical stability or drives the accuracy gains.
-
There are no theoretical guarantees for hierarchy recovery, embedding accuracy, or optimization convergence.
-
The optimization procedure is not described in detail.
-
Computational complexity and runtime are not reported.
Questions for authors
-
Does the entropy-inspired score improve on a simple containment loss using the same wedges?
-
Does reduced concentration near the boundary improve numerical stability or accuracy?
-
Which optimizer and update rule was used?
-
What are the training time and memory usage and how do they compare with the baselines?
4 comments
Official Comment by Authors (Part 2)
Which optimizer and update rule was used?
We used RSGD. In an upcoming revision, we will shift to RAdam which was used in the recent paper by Yu et al. (2024 ICLR) which improves our performance slightly over RSGD.
Regarding the update rules, for the RSGD we followed the additive Riemannian-gradient step of Nickel and Kiela (2017). Our optimizer updates each embedding as , where is the learning rate and is the gradient adjusted for the Poincaré geometry. For RAdam, we are following the HTorch implementation adopted for the Poincaré ball in Shadow cones (Yu et al. (2024)). The following equation summarizes the position update implemented in that code: , where is the moving average of Riemannian gradients, is the moving average of their squared Riemannian norms, and is the learning rate including bias correction. The optimizer rescales the averaged gradient adaptively, then uses the exponential map . The numerical projection keeps the updated point inside the Poincaré ball, while stabilizes the denominator. We will include this information in the upcoming revision.
What are the training time and memory usage and how do they compare with the baselines?
The training time of our models is larger than the baselines by a factor of three due to our more complex loss function, and the peak RAM usage is comparable. For instance, we obtain the following figures for the reconstruction task on the mammals dataset:
- Poincaré embeddings: training time 0.326 s/epoch, walltime 193.7 s, peak RAM usage: 381.9 MB
- Entailment cones: training time 0.310 s/epoch, walltime 177.7 s, peak RAM usage: 380.5 MB
- HoloWedge CE loss: training time 0.997 s/epoch, walltime 531.0 s, peak RAM usage: 384.9 MB
- HoloWedge Hinge loss: training time 0.869 s/epoch, walltime 461.9 s, peak RAM usage: 383 MB
These numbers were obtained on a MacBook Pro with an M3 Pro chip while running on the CPU. We will provide further details in the upcoming revision of our paper.
We would also like to comment on the following point.
Limited baselines. The central experimental question of our work was how HoloWedge compares with existing, gradient based, hyperbolic approaches to learning hierarchical representations, which explains our baseline selection. Box and order embeddings, while interesting in their own right utilize Euclidean space. Yu et al. (ICLR 2024) https://openreview.net/forum?id=zbKcFZ6Dbp¬eId=BTgyuPZlSZ provide a list of testruns that is comparable with our results on the Mammals subtree of Wordnet presented in appendix B. Comparing the numbers we see that the performance of our model is better than GBC and VBC, except for GBC in dim 5 at 50% and 90%. For comparison, shadow cones by Yu et al. which improves over Entailment cones also performes worse for d=5, 90%. While we have not included box embeddings in our baseline, we would like to emphasize the conceptual connection to our work (nesting implies hierarchical relation) and since, as explained in the discussion, our model can be seen as spherical embedding on the boundary of hyperbolic space, from a different point of view our HoloWedge and box embeddings indeed seem to be closer related than the difference in underlying geometry suggests.
I thank the authors for the detailed responses to my comments. The responses address well many of comments. In their response, the authors refer to future changes in the paper, which I believe are substantial and need to be reviewed, however, I did not find a revision.
Thank you for your reply. We are just waiting for the results of a few final runs to finalize the revised version of the manuscript. We will upload it as soon as possible.
Official Comment by Authors (Part 1)
Thank you for your comments and the questions.
Does the entropy-inspired score improve on a simple containment loss using the same wedges?
As a first attempt, we in fact investigated a simple containment loss defined on the same RT wedges, before introducing the entropy-inspired score, but were not satisfied with the performance. We therefore decided not to include the results in the manuscript. In an upcoming revision, we will add the material, including a quantitative comparison, to the appendix.
In our first approach, for a child c and a candidate parent p, we used the score
E(c,p)=max(0,γ+θc−θp),
together with a hinge containment loss (inspired by the one used by Ganea et al. (2018)):
Lcont=K+11[E++∑j=1Kmax(0,m−Ej−)],m>0.
Here, γ is the angular separation between the cap centers, θc and θp are their half-opening angles. This energy vanishes when the child cap is fully contained in the parent cap. Let E+ denote the energy of the positive pair and let Ej−, with j=1,…,K, denote the energies of the K corresponding negative pairs. This encourages positive pairs to satisfy containment, while negative pairs must violate it by a margin. Both the wedge parametrization and the containment condition are identical to those underlying HoloWedge. Although this containment loss performed better than the hyperbolic entailment-cones baseline in the initial reconstruction experiments, the embeddings remained concentrated near the origin of the Poincare ball when trained from random initialization, c.f. the 2d plot using the mammals subtree: https://anonymous.4open.science/r/rebuttal-figures-holowedge-575D/rt_containment_on_mammals_rsgd_2d.png.
This suggested the need for a preliminary distance-based pretraining as in Ganea et al. which we wanted to avoid.
We ran some sample computations, which show that our first approach consistently underperforms HoloWedge, see for example the following table for Mammal reconstruction:
Training settings: Results are reported as mean ± standard deviation over three seeds (42, 43, 44).
Starting from this containment baseline, we explored energy scores based on entropic quantities to more directly reflect the physical intuition underlying our approach. The final score presented in the paper yielded the best empirical performance among the variants we tested.
Does reduced concentration near the boundary improve numerical stability or accuracy?
While we do not claim that the lower concentration near the boundary drives accuracy gains (as was suggested in the weaknesses), its impact is twofold. First, one practical application of hyperbolic embeddings is visualization, and lower concentration near the boundary makes it possible to represent the hierarchical structure better, see also Figure 3 in Ganea et al. (ICML 2018). Second, the numerical precision needed for hyperbolic embeddings is set by requiring that (radial) distances can be resolved. For isometric embeddings it is shown in Sala et al. (ICML 2018) that the required numerical precision scales with the logarithm of distance to the boundary. This is of course only a rough upper bound on being able to tell apart two embedded data points. Our embeddings are (by design) not isometric, but we still think that the general point stands and carries over to any embeddings in hyperbolic space.