- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes GeFLoSM, a neural machine translation framework that incorporates hyperbolic geometry and Riemannian conditional flow matching into a Transformer-based translation system. Its central motivation is that conventional NMT models do not explicitly represent the geometric structure of sentence trajectories, while hyperbolic spaces may provide a useful inductive bias for hierarchical linguistic representations. The experiments cover five translation benchmarks across four language-pair directions, and the reported results show improvements over several supervised baselines, and the ablation study suggests that the trajectory bottleneck and flow-matching objective contribute to the final performance.
Critical strengths
- First I think the paper presents an interesting integration of geometric modeling and neural machine translation. The paper explores a potentially useful combination of hyperbolic representations, continuous sequence trajectories, and Riemannian flow matching. The attempt to incorporate these components into a standard autoregressive translation architecture is technically interesting. Also the framework does not rely on external syntactic or semantic parsers. Unlike graph-augmented translation systems, GeFLoSM derives its trajectory representation from encoder hidden states. This avoids parser-dependent preprocessing and makes the approach easier to integrate into existing NMT pipelines.
Critical weaknesses
- I don't quite like the presentation of this paper, quite a few tables and figures are too small to see, like Figure2 and Table4. Please make it better.
- The central geometric motivation is not sufficiently validated: The paper argues that hyperbolic geometry is a natural choice because linguistic structures are hierarchical and hyperbolic spaces accommodate exponentially branching structures. However, this motivation does not establish that the proposed trajectory actually captures syntactic or semantic hierarchy. In particular, connecting token embeddings in their original sequence order creates a geometric curve, but the existence of such a curve does not imply that its curvature represents syntactic dependencies, compositional structure, or semantic relationships. The claimed correspondence between trajectory geometry and linguistic structure remains largely an assumption.
- There are some flaws in the experimental comparisons: The paper reports improvements over several supervised baselines. For example, on WMT16 EN–DE, the full-split result is 27.2 BLEU for GeFLoSM versus 26.0 for the vanilla Transformer. However, these results do not establish that the improvements arise specifically from hyperbolic geometry or Riemannian flow matching. The proposed system introduces an additional neural ODE, a decoder-conditioning pathway, a velocity network, and auxiliary vocabulary-prediction losses. Any of these could improve optimization or representation learning independently of the geometric interpretation.
Questions for authors
see weakness part
5 comments
Reply 2 to Reviewer Tgmj
Critical weakness 2, continued (does the trajectory capture the hierarchy?). “However, this motivation does not establish that the proposed trajectory actually captures syntactic or semantic hierarchy.”
Reply. The revision shows that the trajectory GeFLoSM builds uses the capacity that the exponential volume growth of hyperbolic space (Eq. 2) gives: it measures, with a structural probe, whether the trajectory carries the hierarchy of the sentence, and we set out how the measurement answers the question.
What a trajectory that captures hierarchy must do. The trajectory is a chain of geodesics through its waypoints, which are the encoder states placed on the ball by Eq. 5. The paper derives (Sec. 3.1; Appendix A.3, Eq. 21) that for points placed on the ball as a tree is placed, the distance between two points equals the number of edges on the tree path between them, times a constant, up to a bounded error, and that the geodesic between them dips toward the centre to the depth of their lowest common ancestor. If the trajectory captures syntactic hierarchy, two things must therefore hold: (i) the distances between its waypoints reproduce the distances between the corresponding words in the sentence’s dependency tree, so that the closest pairs of waypoints are the head-dependent pairs; and (ii) the distance of a waypoint from the centre tracks the depth of its word in the tree. Both are measurable.
How the probe measures them (Table 9, Appendix H). The structural probe of Hewitt and Manning (2019), in the Poincare form of Chen et al. (2021), tests (i) and (ii) directly. A 64- dimensional map is fitted on the UD English Web Treebank and evaluated on held-out sentences; the distance probe asks whether the squared distances between the mapped waypoints equal the gold tree distances (UUAS: whether the minimum spanning tree of those distances recovers the dependency arcs; distance Spearman: whether the ordering of distances is right), and the depth probe asks whether the norm of a mapped waypoint equals the depth of its word (root accuracy; depth Spearman). The length of a trajectory segment and the depth of its dip are functions of these waypoint distances, so a high score says that the length and radial profile of the curve follow the parse tree.
Why the comparison shows that the geometry is responsible. Transformer encoders carry some syntax in any geometry, so to isolate the contribution of the geometry we read the same quantities from three encoders: GeFLoSM, the Euclidean twin (every module and parameter kept, only the geometry made flat; Table 3, Appendix G), and the vanilla Transformer. The Poincare probe recovers 61.6 UUAS from the GeFLoSM waypoints, against 59.1 from the twin and 57.9 from the vanilla Transformer, and GeFLoSM is highest on all four measures (distance Spearman 0.738, 0.710, 0.698; root accuracy 67.3, 64.6, 63.4; depth Spearman 0.768, 0.743, 0.734). Since the twin differs from GeFLoSM only in geometry, the additional tree structure is attributable to building the trajectory on the ball. The gain from reading the states with a hyperbolic instead of a Euclidean probe is also largest for GeFLoSM (3.5 UUAS, against 0.7 for the twin and 1.2 for the vanilla encoder).
Why the score is not produced by the probe. Three controls that carry no trained syntax (token embeddings, positional encodings, an untrained encoder) score 22.7 to 44.1 UUAS, and the Poincare probe differs from the Euclidean probe on them by at most 1.1 UUAS points, so the probe does not act as a parser by itself (Chen et al. make the same observation for BERT); the three trained encoders score at least 13.8 UUAS points above every control, and that margin is contributed by training.
Conclusion. For dependency structure the answer to the reviewer’s question is yes: the waypoints through which the trajectory passes carry more of the parse tree’s distances and depths than the waypoints of the same model in flat geometry or of the vanilla Transformer, and the Euclidean twin traces the difference to the geometry. The sentiment probe (Table 10) gives the corresponding measurement for semantic structure, reported in the reply below.
Reply 3 to Reviewer Tgmj
Critical weakness 2, continued (second paragraph of the review). “In particular, connecting token embeddings in their original sequence order creates a geometric curve, but the existence of such a curve does not imply that its curvature represents syntactic dependencies, compositional structure, or semantic relationships. The claimed correspondence between trajectory geometry and linguistic structure remains largely an assumption.”
Reply. The curvature of the trajectory does encode the syntactic, compositional and semantic structure of the sentence, and the structural probe is the measurement that shows it. The curve is the chain of geodesics through the waypoints (the encoder states placed on the ball by Eq. 5), so its length and the depth of its dips are fixed by the hyperbolic distances between waypoints (Appendix A.3, Eq. 21), and these are the quantities the probe reads against linguistic structure. Dependencies and compositional structure: the distance probe recovers the dependency arcs from the waypoints (61.6 UUAS and distance Spearman 0.738 for GeFLoSM), and the depth probe recovers the depth of each word in the tree (root accuracy 67.3, depth Spearman 0.768); the depth of a word in the tree is its position in the nesting of the sentence, so these two scores are measurements of its compositional structure (Table 9). Curvature is the cause: the Euclidean twin keeps every module and the parameter count and removes only the curvature (c to 0), and its waypoints give lower scores on all four measures (59.1 UUAS, 0.710, 64.6, 0.743), as does the vanilla Transformer (57.9, 0.698, 63.4, 0.734); the same change in curvature also changes translation quality, by 0.73 to 1.46 SacreBLEU on three benchmarks (Table 3). Semantic relationships: the sentiment probe defined by Chen et al. on Movie Review reads the meaning of the sentence from the waypoints (Table 10): with trainable class representations the Poincare probe reaches 75.6% on GeFLoSM’s states, against 74.4% for the twin and 73.9% for the vanilla Transformer, and with class representations fixed the Euclidean probe loses 2.8 points while the Poincare probe loses 0.3, the direction Chen et al. report for BERT. The probe controls (token embeddings 70.1%, positional encodings 50.3%) show that this margin is not a lexical effect alone, and the structural controls (22.7 to 44.1 UUAS) show that the probe does not produce the syntactic scores by itself. The correspondence between the geometry of the trajectory and linguistic structure is therefore a measured result and not an assumption.
Reply 1 to Reviewer Tgmj
Paper summary. “The paper proposes GeFLoSM, a neural machine translation framework that incorporates hyperbolic geometry and Riemannian conditional flow matching into a Transformer- based translation system . . . the ablation study suggests that the trajectory bottleneck and flow-matching objective contribute to the final performance.”
Reply. Thank you for the accurate summary and for the helpful detailed, technical review. Your critical three points are answered below, and your summary of the ablation is borne out by the new rows of Table 3: the trajectory bottleneck and the flow-matching objective each contribute, and the geometry contributes a further measurable increment. Kindly find attached revised manuscript.
Critical strength 1. “The paper presents an interesting integration of geometric modeling and neural machine translation . . . Also the framework does not rely on external syntactic or semantic parsers.”
Reply. Thank you. This is also what the probe uses: the trajectory is built from the encoder’s own states, and the probe measures how much tree structure those states carry without any parser in the model (Tables 9 and 10, Appendix H).
Critical weakness 1. “I don’t quite like the presentation of this paper, quite a few tables and figures are too small to see, like Figure 2 and Table 4. Please make it better.”
Reply. Figure 2 now spans the page width, with larger block titles, and the equations that crowded the blocks are moved into a key beneath the diagram (Table 4, Appendix B.1). The large result tables are reflowed into stacked panels. The geometry primer (Sec. 3.1) and the architecture walkthrough (Sec. 4) are in the main text, and the implementation details are in Appendix D.
Critical weakness 2. “The central geometric motivation is not sufficiently validated: The paper argues that hyperbolic geometry is a natural choice because linguistic structures are hierarchical and hyperbolic spaces accommodate exponentially branching structures. However, this motivation does not establish that the proposed trajectory actually captures syntactic or semantic hierarchy.”
Reply. The motivation is that language is hierarchical and that a hyperbolic space holds a hierarchy with low distortion. We set out why language is hierarchical and why hierarchy helps translation, why the Poincare ball represents it without an external graph, and why this suits a sequence of tokens; the replies below then validate the motivation with measurements.
Language is hierarchical, and hierarchy helps translation. Words group into phrases and clauses, and meaning is composed along that nesting, so the number of nodes at depth k grows roughly exponentially in k. Translation systems that inject this structure improve over a plain Transformer, whether they encode syntactic or semantic graphs with graph neural networks (Bastings et al., 2017; Marcheggiani and Titov, 2018; Song et al., 2019; Yao et al., 2020) or feed AMR graphs produced by an external parser (Li and Flanigan, 2022); in our comparison the AMR-augmented Transformer scores 28.2 against 27.6 for the vanilla Transformer on EN-VI (Table 6) and 22.1 against 20.3 on NC-v11 (Table 5).
The Poincare ball holds a hierarchy without an external graph. The volume of a hyperbolic ball grows exponentially with its radius (Eq. 2), whereas a Euclidean ball’s volume grows polynomially, so a tree can be placed in the ball at fixed dimension with low distortion (Sarkar, 2012; Nickel and Kiela, 2017), and hyperbolic word and sentence embeddings learned from text capture notions of hierarchy such as phrase constituency (Dhingra et al., 2018). For points placed on the ball as a tree is placed, the distance between two nodes equals the tree distance up to a constant factor and a bounded error, and the geodesic between them dips toward the origin to the depth of their lowest common ancestor (Sec. 3.1; derivation in Appendix A.3, Eqs. 19-21). GeFLoSM therefore holds the hierarchy in the positions of the encoder’s own token states and the geodesics between them, with no parser and no graph construction, and it scores 29.1 against 28.2 for the AMR-augmented Transformer on EN-VI (Table 6) and 23.4 against 22.1 on NC-v11 (Table 5).
Why this suits a sequence of tokens. The trajectory visits the tokens in reading order, but its length and its radial profile are set by the hyperbolic distances between its waypoints, so they record the nesting of the sentence and not only the order of its words. The claim is tested and not assumed: the next replies set out a structural probe of the trajectory’s waypoints (Table 9, Appendix H), the Euclidean twin that differs from GeFLoSM only in geometry (Table 3, Appendix G), and a sentiment probe (Table 10).