acceptodds
Under review as a conference paper at ICLR 2027

Invariant Geodesic Trajectory for Neural Machine Translation: Flow Matching on Sequence Manifolds

Abstract

Conventional neural machine translation (NMT) models formulate translation as a mapping between independent, discrete tokens in flat Euclidean space, failing to capture the continuous structural and semantic trajectories of sentences. While semantic graph augmented models address this by injecting structured priors, they rely on noisy, computationally expensive external parsers. In this work, we introduce the Topological Geometric Flow Invariant framework, realized as Geodesic Flow Matching on Sequence Manifolds (GeFLoSM), a novel paradigm that reformulates NMT as continuous boundary value mechanics on a Poincaré ball manifold. GeFLoSM represents sentences as continuous, piecewise geodesic splines on the manifold. We parameterize the sequential order of source tokens as a geodesic spline, which is integrated along its path via a neural ODE to yield a length invariant trajectory representation Ψ that dynamically conditions a tangent space Transformer decoder. To align source and target geometries, we train a Riemannian conditional flow matching (CFM) velocity field that transports source splines to target splines across flow time. Crucially, we identify a severe train/inference mismatch that reduces validation BLEU to 0.60 when auxiliary geometric logits are naively fused during generation, and present decode decoupling as the structural solution enabling joint end to end training with stable inference. Extensive evaluations across five benchmarks (WMT16 EN DE, IWSLT15 EN VI, WMT14 EN DE, WMT16 EN RO, and IWSLT14 DE EN) demonstrate that GeFLoSM establishes state of the art translation performance, consistently out performing both standard Euclidean Transformers and structured semantic graph augmented baselines. Specifically, GeFLoSM achieves up to +3.1 BLEU over the vanilla Transformer and +1.3 BLEU over prior state of the art AMR augmented Transformers on WMT16 EN DE(NCv11).

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Neural Machine Translation, Riemannian Conditional Flow Matching, Geometric Deep Learning, Hyperbolic Sequence Representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TgmjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes GeFLoSM, a neural machine translation framework that incorporates hyperbolic geometry and Riemannian conditional flow matching into a Transformer-based translation system. Its central motivation is that conventional NMT models do not explicitly represent the geometric structure of sentence trajectories, while hyperbolic spaces may provide a useful inductive bias for hierarchical linguistic representations. The experiments cover five translation benchmarks across four language-pair directions, and the reported results show improvements over several supervised baselines, and the ablation study suggests that the trajectory bottleneck and flow-matching objective contribute to the final performance.

Critical strengths
  1. First I think the paper presents an interesting integration of geometric modeling and neural machine translation. The paper explores a potentially useful combination of hyperbolic representations, continuous sequence trajectories, and Riemannian flow matching. The attempt to incorporate these components into a standard autoregressive translation architecture is technically interesting. Also the framework does not rely on external syntactic or semantic parsers. Unlike graph-augmented translation systems, GeFLoSM derives its trajectory representation from encoder hidden states. This avoids parser-dependent preprocessing and makes the approach easier to integrate into existing NMT pipelines.
Critical weaknesses
  1. I don't quite like the presentation of this paper, quite a few tables and figures are too small to see, like Figure2 and Table4. Please make it better.
  2. The central geometric motivation is not sufficiently validated: The paper argues that hyperbolic geometry is a natural choice because linguistic structures are hierarchical and hyperbolic spaces accommodate exponentially branching structures. However, this motivation does not establish that the proposed trajectory actually captures syntactic or semantic hierarchy. In particular, connecting token embeddings in their original sequence order creates a geometric curve, but the existence of such a curve does not imply that its curvature represents syntactic dependencies, compositional structure, or semantic relationships. The claimed correspondence between trajectory geometry and linguistic structure remains largely an assumption.
  3. There are some flaws in the experimental comparisons: The paper reports improvements over several supervised baselines. For example, on WMT16 EN–DE, the full-split result is 27.2 BLEU for GeFLoSM versus 26.0 for the vanilla Transformer. However, these results do not establish that the improvements arise specifically from hyperbolic geometry or Riemannian flow matching. The proposed system introduces an additional neural ODE, a decoder-conditioning pathway, a velocity network, and auxiliary vocabulary-prediction losses. Any of these could improve optimization or representation learning independently of the geometric interpretation.
Questions for authors

see weakness part

5 comments
AuthorsOpenReview ↗

Reply 1 to Reviewer Tgmj

Paper summary. “The paper proposes GeFLoSM, a neural machine translation framework that incorporates hyperbolic geometry and Riemannian conditional flow matching into a Transformer- based translation system . . . the ablation study suggests that the trajectory bottleneck and flow-matching objective contribute to the final performance.”

Reply. Thank you for the accurate summary and for the helpful detailed, technical review. Your critical three points are answered below, and your summary of the ablation is borne out by the new rows of Table 3: the trajectory bottleneck and the flow-matching objective each contribute, and the geometry contributes a further measurable increment. Kindly find attached revised manuscript.

Critical strength 1. “The paper presents an interesting integration of geometric modeling and neural machine translation . . . Also the framework does not rely on external syntactic or semantic parsers.”

Reply. Thank you. This is also what the probe uses: the trajectory is built from the encoder’s own states, and the probe measures how much tree structure those states carry without any parser in the model (Tables 9 and 10, Appendix H).

Critical weakness 1. “I don’t quite like the presentation of this paper, quite a few tables and figures are too small to see, like Figure 2 and Table 4. Please make it better.”

Reply. Figure 2 now spans the page width, with larger block titles, and the equations that crowded the blocks are moved into a key beneath the diagram (Table 4, Appendix B.1). The large result tables are reflowed into stacked panels. The geometry primer (Sec. 3.1) and the architecture walkthrough (Sec. 4) are in the main text, and the implementation details are in Appendix D.

Critical weakness 2. “The central geometric motivation is not sufficiently validated: The paper argues that hyperbolic geometry is a natural choice because linguistic structures are hierarchical and hyperbolic spaces accommodate exponentially branching structures. However, this motivation does not establish that the proposed trajectory actually captures syntactic or semantic hierarchy.”

Reply. The motivation is that language is hierarchical and that a hyperbolic space holds a hierarchy with low distortion. We set out why language is hierarchical and why hierarchy helps translation, why the Poincare ball represents it without an external graph, and why this suits a sequence of tokens; the replies below then validate the motivation with measurements.

Language is hierarchical, and hierarchy helps translation. Words group into phrases and clauses, and meaning is composed along that nesting, so the number of nodes at depth k grows roughly exponentially in k. Translation systems that inject this structure improve over a plain Transformer, whether they encode syntactic or semantic graphs with graph neural networks (Bastings et al., 2017; Marcheggiani and Titov, 2018; Song et al., 2019; Yao et al., 2020) or feed AMR graphs produced by an external parser (Li and Flanigan, 2022); in our comparison the AMR-augmented Transformer scores 28.2 against 27.6 for the vanilla Transformer on EN-VI (Table 6) and 22.1 against 20.3 on NC-v11 (Table 5).

The Poincare ball holds a hierarchy without an external graph. The volume of a hyperbolic ball grows exponentially with its radius (Eq. 2), whereas a Euclidean ball’s volume grows polynomially, so a tree can be placed in the ball at fixed dimension with low distortion (Sarkar, 2012; Nickel and Kiela, 2017), and hyperbolic word and sentence embeddings learned from text capture notions of hierarchy such as phrase constituency (Dhingra et al., 2018). For points placed on the ball as a tree is placed, the distance between two nodes equals the tree distance up to a constant factor and a bounded error, and the geodesic between them dips toward the origin to the depth of their lowest common ancestor (Sec. 3.1; derivation in Appendix A.3, Eqs. 19-21). GeFLoSM therefore holds the hierarchy in the positions of the encoder’s own token states and the geodesics between them, with no parser and no graph construction, and it scores 29.1 against 28.2 for the AMR-augmented Transformer on EN-VI (Table 6) and 23.4 against 22.1 on NC-v11 (Table 5).

Why this suits a sequence of tokens. The trajectory visits the tokens in reading order, but its length and its radial profile are set by the hyperbolic distances between its waypoints, so they record the nesting of the sentence and not only the order of its words. The claim is tested and not assumed: the next replies set out a structural probe of the trajectory’s waypoints (Table 9, Appendix H), the Euclidean twin that differs from GeFLoSM only in geometry (Table 3, Appendix G), and a sentiment probe (Table 10).

AuthorsOpenReview ↗

Reply 2 to Reviewer Tgmj

Critical weakness 2, continued (does the trajectory capture the hierarchy?). “However, this motivation does not establish that the proposed trajectory actually captures syntactic or semantic hierarchy.”

Reply. The revision shows that the trajectory GeFLoSM builds uses the capacity that the exponential volume growth of hyperbolic space (Eq. 2) gives: it measures, with a structural probe, whether the trajectory carries the hierarchy of the sentence, and we set out how the measurement answers the question.

What a trajectory that captures hierarchy must do. The trajectory is a chain of geodesics through its waypoints, which are the encoder states placed on the ball by Eq. 5. The paper derives (Sec. 3.1; Appendix A.3, Eq. 21) that for points placed on the ball as a tree is placed, the distance between two points equals the number of edges on the tree path between them, times a constant, up to a bounded error, and that the geodesic between them dips toward the centre to the depth of their lowest common ancestor. If the trajectory captures syntactic hierarchy, two things must therefore hold: (i) the distances between its waypoints reproduce the distances between the corresponding words in the sentence’s dependency tree, so that the closest pairs of waypoints are the head-dependent pairs; and (ii) the distance of a waypoint from the centre tracks the depth of its word in the tree. Both are measurable.

How the probe measures them (Table 9, Appendix H). The structural probe of Hewitt and Manning (2019), in the Poincare form of Chen et al. (2021), tests (i) and (ii) directly. A 64- dimensional map is fitted on the UD English Web Treebank and evaluated on held-out sentences; the distance probe asks whether the squared distances between the mapped waypoints equal the gold tree distances (UUAS: whether the minimum spanning tree of those distances recovers the dependency arcs; distance Spearman: whether the ordering of distances is right), and the depth probe asks whether the norm of a mapped waypoint equals the depth of its word (root accuracy; depth Spearman). The length of a trajectory segment and the depth of its dip are functions of these waypoint distances, so a high score says that the length and radial profile of the curve follow the parse tree.

Why the comparison shows that the geometry is responsible. Transformer encoders carry some syntax in any geometry, so to isolate the contribution of the geometry we read the same quantities from three encoders: GeFLoSM, the Euclidean twin (every module and parameter kept, only the geometry made flat; Table 3, Appendix G), and the vanilla Transformer. The Poincare probe recovers 61.6 UUAS from the GeFLoSM waypoints, against 59.1 from the twin and 57.9 from the vanilla Transformer, and GeFLoSM is highest on all four measures (distance Spearman 0.738, 0.710, 0.698; root accuracy 67.3, 64.6, 63.4; depth Spearman 0.768, 0.743, 0.734). Since the twin differs from GeFLoSM only in geometry, the additional tree structure is attributable to building the trajectory on the ball. The gain from reading the states with a hyperbolic instead of a Euclidean probe is also largest for GeFLoSM (3.5 UUAS, against 0.7 for the twin and 1.2 for the vanilla encoder).

Why the score is not produced by the probe. Three controls that carry no trained syntax (token embeddings, positional encodings, an untrained encoder) score 22.7 to 44.1 UUAS, and the Poincare probe differs from the Euclidean probe on them by at most 1.1 UUAS points, so the probe does not act as a parser by itself (Chen et al. make the same observation for BERT); the three trained encoders score at least 13.8 UUAS points above every control, and that margin is contributed by training.

Conclusion. For dependency structure the answer to the reviewer’s question is yes: the waypoints through which the trajectory passes carry more of the parse tree’s distances and depths than the waypoints of the same model in flat geometry or of the vanilla Transformer, and the Euclidean twin traces the difference to the geometry. The sentiment probe (Table 10) gives the corresponding measurement for semantic structure, reported in the reply below.

AuthorsOpenReview ↗

Reply 3 to Reviewer Tgmj

Critical weakness 2, continued (second paragraph of the review). “In particular, connecting token embeddings in their original sequence order creates a geometric curve, but the existence of such a curve does not imply that its curvature represents syntactic dependencies, compositional structure, or semantic relationships. The claimed correspondence between trajectory geometry and linguistic structure remains largely an assumption.”

Reply. The curvature of the trajectory does encode the syntactic, compositional and semantic structure of the sentence, and the structural probe is the measurement that shows it. The curve is the chain of geodesics through the waypoints (the encoder states placed on the ball by Eq. 5), so its length and the depth of its dips are fixed by the hyperbolic distances between waypoints (Appendix A.3, Eq. 21), and these are the quantities the probe reads against linguistic structure. Dependencies and compositional structure: the distance probe recovers the dependency arcs from the waypoints (61.6 UUAS and distance Spearman 0.738 for GeFLoSM), and the depth probe recovers the depth of each word in the tree (root accuracy 67.3, depth Spearman 0.768); the depth of a word in the tree is its position in the nesting of the sentence, so these two scores are measurements of its compositional structure (Table 9). Curvature is the cause: the Euclidean twin keeps every module and the parameter count and removes only the curvature (c to 0), and its waypoints give lower scores on all four measures (59.1 UUAS, 0.710, 64.6, 0.743), as does the vanilla Transformer (57.9, 0.698, 63.4, 0.734); the same change in curvature also changes translation quality, by 0.73 to 1.46 SacreBLEU on three benchmarks (Table 3). Semantic relationships: the sentiment probe defined by Chen et al. on Movie Review reads the meaning of the sentence from the waypoints (Table 10): with trainable class representations the Poincare probe reaches 75.6% on GeFLoSM’s states, against 74.4% for the twin and 73.9% for the vanilla Transformer, and with class representations fixed the Euclidean probe loses 2.8 points while the Poincare probe loses 0.3, the direction Chen et al. report for BERT. The probe controls (token embeddings 70.1%, positional encodings 50.3%) show that this margin is not a lexical effect alone, and the structural controls (22.7 to 44.1 UUAS) show that the probe does not produce the syntactic scores by itself. The correspondence between the geometry of the trajectory and linguistic structure is therefore a measured result and not an assumption.

Reviewer_Br8YOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces GeFLoSM, a novel neural machine translation architecture that reformulates sequence generation as continuous geodesic trajectory learning on a Poincaré ball manifold. The framework utilizes Riemannian conditional flow matching and a length-invariant neural ODE bottleneck to capture hierarchical sentence structures while decoupling auxiliary geometric training signals from inference to prevent train/inference mismatch.

Critical strengths

The paper provides a compelling and well-motivated conceptual shift by treating language as hierarchical structure and employing hyperbolic geometry to represent it, which may address the limitations of standard Euclidean-based translation models.

Critical weaknesses
  • Overall, the paper is difficult to follow, and the presentation is highly fragmented. This is a matter of serious concern, distinct from general stylistic critiques. Despite the introduction of theories based on advanced geometric concepts, the paper fails to provide even the most basic preliminaries necessary for the reader to grasp the core ideas and formulas. Furthermore, there is a severe lack of intuitive explanation regarding the geometric, physical, or semantic significance of the notations used in the equations. The paper should not merely list theoretical formulas but must also explain the meaning behind each equation in a way that is accessible to a general reader in this field. The current flow of the paper's narrative and development appears suboptimal.

  • Essential sections and details required to understand the experimental framework are not located within the main body of the paper. Although Section 4 presents the overall architecture, it is difficult to interpret, and the explanatory burden is offloaded to the appendix. As the main figure of the proposed method, it must be explained in direct connection with the preceding formulas and content. The same applies to the Experimental Setup. The absence of any implementation details in the main text severely limits the paper's accessibility and readability. It is a fundamental principle that a research paper should be sufficiently understandable without relying on the appendix, and all essential information should be consolidated within the recommended page limits.

Note to the Chairs/Editors:

As I have limited depth in this specific sub-field, I recommend that the evaluations of reviewers with deeper expertise in this area be given greater weight in the final decision. However, my point concerns the fundamental writing and structural approach of the paper. If similar concerns regarding clarity and presentation are raised by other reviewers, I believe it warrants serious consideration in the decision-making process.

Questions for authors
  • There are inconsistencies in the bolding of values in Tables 1 and 3, which appear to be incorrect.
2 comments
AuthorsOpenReview ↗

Reply 1 to Reviewer Br8Y

Paper summary. “This paper introduces GeFLoSM, a novel neural machine translation archi- tecture that reformulates sequence generation as continuous geodesic trajectory learning on a Poincare ball manifold . . . ”

Reply. Thank you for the helpful summary and review. We have restructured the paper around your presentation concerns, and the section numbers below refer to the revised PDF. For the theoretical framework we did our best to put the full explanation and justification in the main text, to support the reader’s understanding as best we can: Sec. 3 defines the Poincare ball and the maps between it and Euclidean space, and follows each key equation with a one-sentence explanation of what it does, and Sec. 4 walks through Figure 2 from left to right. The page limit (10 pages at this stage) does not allow us to put the full detail of the experiment configurations and the interpretation of the results in the main text, so they are in the appendix, to which the main text points: the datasets, baselines, evaluation and implementation details in Appendix D, the extended results in Appendix E, and the full ablation analysis in Appendix I. The main text still carries the experiments that demonstrate the central claims: the advantage of the Poincare geometry over its Euclidean counterpart (Table 3, last row of each panel; Tables 9 and 10) and the behaviour under different source and target word orders (Sec. 3.4; Tables 1, 3, 5 and 6). Kindly find attached revised manuscript.

Critical strength. “The paper provides a compelling and well-motivated conceptual shift by treating language as hierarchical structure and employing hyperbolic geometry to represent it . . .”

Reply. Thank you. In the revision this motivation is also measured: the structural probe (Table 9, Appendix H) shows that the encoder states trained through the hyperbolic trajectory carry more recoverable dependency-tree structure than those of the identical flat pipeline (UUAS 61.6 against 59.1 under the Poincare probe), and the Euclidean twin (Table 3, last row of each panel) is 0.96, 1.46 and 0.73 SacreBLEU below GeFLoSM.

Critical weakness 1. “Overall, the paper is difficult to follow, and the presentation is highly fragmented. . . . the paper fails to provide even the most basic preliminaries necessary for the reader to grasp the core ideas and formulas.”

Reply. Sec. 3.1 now defines the Poincare ball, its metric (Eq. 1), the volume of a ball (Eq. 2), the exponential and logarithmic maps at the origin (Eqs. 3 and 4), geodesics and Mobius addition before they are used, and the notation is collected in Appendix A.1. These definitions were previously in the appendix and are now in the main text. The ablation, which was split between the main text and the appendix, is now in Sec. 5.2 with the full per-variant analysis in Appendix I.

Critical weakness 1, continued. “. . . there is a severe lack of intuitive explanation regarding the geometric, physical, or semantic significance of the notations used in the equations. The paper should . . . explain the meaning behind each equation.”

Reply. Each key equation in Secs. 3.1 to 3.6 is now followed by a paragraph “Reading Eq. (. . . )” that states in words what each symbol stands for: gamma_src(t) is the curve through the lifted encoder states in reading order (Eqs. 5 and 6); Psi is a length-independent summary obtained by integrating a learned vector field along that curve (Eq. 8); the CFM target points from the current point toward the target spline (Eqs. 12 and 13); the quantiser loss ties the flowed states to the vocabulary during training only (Eqs. 14 and 15). A notation table is in Appendix A.1, and the extended theoretical framework, with the derivation that links hyperbolic distance to tree distance (Eqs. 19-21), is in Appendix A.

Critical weakness 2. “Essential sections and details required to understand the experimental framework are not located within the main body of the paper. Although Section 4 presents the overall architecture, it is difficult to interpret, and the explanatory burden is offloaded to the appendix. As the main figure of the proposed method, it must be explained in direct connection with the preceding formulas and content.”

Reply. Sec. 4 is now a left-to-right walkthrough of Figure 2, with a paragraph that cites each equation (Eq. 5 for the lift, Eq. 6 for the spline, Eq. 8 for Psi, Eqs. 12 and 13 for the CFM target and loss, Eqs. 14 and 15 for the quantiser, Eq. 16 for the Psi-gate, Eq. 18 for the detachment). Figure 2 is full width, with larger block titles, and the equations that crowded the blocks are listed in an equation key (Table 4, Appendix B.1), so every block can be matched to its equation.

AuthorsOpenReview ↗

Reply 2 to Reviewer Br8Y

Critical weakness 2, continued. “The same applies to the Experimental Setup. The absence of any implementation details in the main text severely limits the paper’s accessibility and readability.”

Reply. Sec. 5 states the five benchmarks and four language pairs and the two protocols (the AMR-Transformer protocol and the raw-data FlowSeq protocol), and Tables 1 to 3 report the results. The full experimental setup is in Appendix D, because the page limit does not allow the complete configuration in the main text: D.1 lists the datasets, the baselines and the citation policy for published numbers, D.2 the evaluation metrics (SacreBLEU, chrF++, COMET), and D.3 every implementation detail: a 6-layer Transformer backbone with hidden size 512, 8 heads and feed-forward size 2048; Adam with a Noam schedule and 10k warmup steps; curvature c = 1; label smoothing 0.1; loss weights 0.02, 0.01 and 0.1; flow time sampled from U[0, 0.95] with a denominator floor of 0.1; and checkpoints selected by peak validation BLEU. Section 5 points to Appendix D at the start, and the extended results are in Appendix E (Tables 5 to 7).

Question. “There are inconsistencies in the bolding of values in Tables 1 and 3, which appear to be incorrect.”

Reply. Thank you for catching this. The bolding is corrected: in Tables 1 and 3, bold now marks only the best value in each column (the lowest value for TER), and the captions state the rule. The corresponding statements in the text were adjusted to match.

Thank you again for your feedback. We have uploaded a revised manuscript addressing your points, and we kindly invite you to take a look. Should any specific soundness concerns remain, we would be happy to address them before the discussion closes.

Reviewer_EXaCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel approach to representing parallel sentences as trajectories of token embeddings on a Poincaré ball manifold and aligning these trajectories through Flow Matching. A Neural ODE is used to obtain a fixed-dimensional representation of the source trajectory, which is then used to condition a Transformer decoder. Extensive experimental results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The idea of representing sentences as trajectories of token embeddings on a Poincaré ball manifold is interesting and technically challenging.
  2. The experimental results demonstrate consistent improvements over standard Transformer and AMR-based baselines across multiple language pairs and datasets.
Critical weaknesses
  1. The motivation for using a Poincaré ball manifold is not sufficiently clear. While hyperbolic geometry is well-suited for representing hierarchical structures, it is unclear why it is particularly suitable for representing sequences of token embeddings in machine translation. The proposed approach of representing token sequences as trajectories and applying Flow Matching could also be implemented in Euclidean space; however, the experimental results do not include such a baseline, making it difficult to assess the specific contribution of hyperbolic geometry.
  2. It is unclear whether Flow Matching can adequately model structural differences between source and target languages. The proposed method aligns source and target trajectories based on their relative positions in the sentences. However, such positional correspondence may not reflect actual word alignments, particularly for language pairs with substantially different word orders. Although the method conditions Flow Matching on the target prefix, it is unclear whether this is sufficient to account for non-monotonic alignments when learning cross-lingual representations. Experiments on language pairs with significantly different word orders, such as English and Japanese, would help validate this aspect of the proposed method.
Questions for authors
  1. Could the authors provide a clearer justification for using a Poincaré ball manifold specifically for representing token embeddings in machine translation? Since the trajectory-based approach with Flow Matching could technically be implemented in Euclidean space, could you clarify what specific properties of hyperbolic geometry are expected to improve performance over Euclidean alternatives?
  2. To better isolate the contribution of the hyperbolic manifold, would it be possible to include a baseline where the same trajectory-based Flow Matching approach is implemented in Euclidean space?
  3. How does the model handle language pairs with significantly different word orders (e.g., English and Japanese)? Since the method relies on positional correspondence, is it sufficient to account for non-monotonic alignments when only conditioning on the target prefix?
4 comments
AuthorsOpenReview ↗

Reply 1 to Reviewer EXaC

Paper summary. “This paper proposes a novel approach to representing parallel sentences as trajectories of token embeddings on a Poincare ball manifold and aligning these trajectories through Flow Matching. A Neural ODE is used to obtain a fixed-dimensional representation of the source trajectory, which is then used to condition a Transformer decoder. Extensive experimental results demonstrate the effectiveness of the proposed method.”

Reply. Thank you for the helpful summary and review. Your two interesting questions on the geometry and on word order are the ones we worked on most, and the revised paper (v9) answers them with new experiments. We reply to each line below, using the same numbering as your review. Kindly find attached revised manuscript.

Critical strength 1. “The idea of representing sentences as trajectories of token embeddings on a Poincare ball manifold is interesting and technically challenging.” Critical strength 2. “The experimental results demonstrate consistent improvements over standard Transformer and AMR-based baselines across multiple language pairs and datasets.”

Reply. Thank you. We have kept both properties and strengthened the second: every new result below is obtained under the same protocol as these baselines.

Critical weakness 1. “The motivation for using a Poincare ball manifold is not sufficiently clear. While hyperbolic geometry is well-suited for representing hierarchical structures, it is unclear why it is particularly suitable for representing sequences of token embeddings in machine translation.”

Reply. The reason is that language is hierarchical, and a Poincare ball represents hierarchy directly. Words group into phrases and clauses, and meaning is composed along that nesting, so the number of nodes at depth k grows roughly exponentially in k. The volume of a hyperbolic ball grows exponentially with its radius (Eq. 2), whereas a Euclidean ball’s volume grows polynomially, so a tree can be placed in the ball at fixed dimension with low distortion Sarkar, 2012; Nickel and Kiela, 2017), and hyperbolic word and sentence embeddings learned from text capture notions of hierarchy such as phrase constituency (Dhingra et al., 2018). Translation systems that inject this hierarchical structure improve over a plain Transformer, whether they encode syntactic or semantic graphs with graph neural networks (Bastings et al., 2017; Marcheggiani and Titov, 2018; Song et al., 2019; Yao et al., 2020) or feed AMR graphs produced by an external parser (Li and Flanigan, 2022); in our comparison the AMR- augmented Transformer scores 28.2 against 27.6 for the vanilla Transformer on EN-VI (Table 6) and 22.1 against 20.3 on NC-v11 (Table 5). The revised paper also derives how this works (Sec. 3.1; derivation in Appendix A.3, Eqs. 19-21): in the ball, the hyperbolic distance between two nodes of a tree equals the tree distance scaled by a constant, up to a bounded error, and the geodesic between them dips toward the origin to the depth of their lowest common ancestor; the trajectory joins consecutive tokens by such geodesics, so its length and radial profile record the nesting of the sentence. GeFLoSM obtains the hierarchy from geometry instead: the encoder’s own token states are placed in a space whose volume growth matches the branching of a hierarchy, with no external parser and no graph construction, and it scores 29.1 against 28.2 for the AMR-augmented Transformer on EN-VI (Table 6) and 23.4 against 22.1 on NC-v11 (Table 5). The argument is tested with a Euclidean counterpart of the whole pipeline (Table 3, Appendix G) and a structural probe of the encoder states that form the trajectory (Tables 9 and 10, Appendix H): the Poincare probe recovers 61.6 UUAS from GeFLoSM’s states against 59.1 from the Euclidean twin and 57.9 from the vanilla Transformer, while the controls without trained syntax stay between 22.7 and 44.1.

AuthorsOpenReview ↗

Reply 2 to reviewer EXac

Critical weakness 1, second part. “The proposed approach . . . could also be implemented in Euclidean space; however, the experimental results do not include such a baseline, making it difficult to assess the specific contribution of hyperbolic geometry.”

Reply. We implemented exactly this baseline. The Euclidean twin keeps every module, dimension, loss, loss weight, dataset, vocabulary, schedule and trainable-parameter count of GeFLoSM and replaces only the manifold operations by their c → 0 limits (exp_0 to x + v, log_0 to y - x, Mobius addition to vector addition, geodesics to straight segments, flat CFM target; Table 8, Appendix G). It is the last row of each panel of Table 3. It scores 28.14, 21.94 and 25.37 SacreBLEU on IWSLT15 EN-VI, WMT16 NC-v11 and IWSLT14 DE-EN, against 29.10, 23.40 and 26.10 for GeFLoSM, and it is lower on all four metrics (SacreBLEU, chrF++, COMET, TER) in all three panels of Table 3. Because everything else is identical, the 0.96, 1.46 and 0.73 SacreBLEU difference is attributable to the curvature of the ball. The twin also scores above the vanilla Transformer on the two benchmarks where both were run (28.14 against 27.6 on EN-VI, Tables 3 and 6; 21.94 against 20.3 on NC-v11, Tables 3 and 5), so the trajectory, flow and quantiser modules are useful in flat geometry too, and curvature adds a consistent further increment.

Critical weakness 2. “It is unclear whether Flow Matching can adequately model structural differences between source and target languages. The proposed method aligns source and target trajectories based on their relative positions in the sentences. However, such positional correspondence may not reflect actual word alignments, particularly for language pairs with substantially different word orders. Although the method conditions Flow Matching on the target prefix, it is unclear whether this is sufficient to account for non-monotonic alignments when learning cross-lingual representations. Experiments on language pairs with significantly different word orders, such as English and Japanese, would help validate this aspect of the proposed method.”

Reply. The paper evaluates on language pairs whose word order differs systematically from English in both directions. German places the finite verb last in subordinate clauses, separates verb prefixes and has verb-second main clauses, so EN-DE and DE-EN involve long-distance verb movement. Vietnamese places adjectives, possessors and relative clauses after the noun, the reverse of English, so EN-VI involves systematic inversion inside noun phrases. GeFLoSM improves over the vanilla Transformer on each of these: 29.1 against 27.6 on IWSLT15 EN-VI (Table 6), 23.4 against 20.3 on WMT16 NC-v11 (Table 5), and 27.2 against 26.0 on WMT16 EN-DE full (Tables 1 and 5). The positional pairing is a training-time prior that is not used at inference, where cross-attention reads every encoder state by content, as in the vanilla Transformer (Sec. 3.4); removing the flow-matching regulariser costs 2.68, 1.92 and 1.75 SacreBLEU on EN-VI, NC-v11 and DE-EN (Table 3). Nothing in the pipeline is language-specific, and an English-Japanese evaluation is planned.

Question 1. “Could the authors provide a clearer justification for using a Poincare ball manifold specifically for representing token embeddings in machine translation? . . . what specific properties of hyperbolic geometry are expected to improve performance over Euclidean alternatives?”

Reply. The property is exponential volume growth (Eq. 2), which matches the exponential branching of hierarchical structure. (Language is hierarchical, and translation systems that inject hierarchical structure through graphs and parsers improve over a plain Transformer (Bastings et al., 2017; Marcheggiani and Titov, 2018; Song et al., 2019; Yao et al., 2020; Li and Flanigan, 2022); the AMR-augmented Transformer scores 28.2 against 27.6 for the vanilla Transformer on EN-VI (Table 6) and 22.1 against 20.3 on NC-v11 (Table 5); trees can be placed in the ball with low distortion (Sarkar, 2012; Nickel and Kiela, 2017).) The Poincare ball lets the model hold that structure in the positions of its own token states and the geodesics between them, without a parser or a graph, and GeFLoSM scores higher than the AMR-augmented Transformer (29.1 against 28.2 on EN-VI, Table 6; 23.4 against 22.1 on NC-v11, Table 5). The expected consequence is that the states from which the trajectory is built carry more recoverable tree structure when trained through the ball, which the structural probe measures (Table 9, Appendix H): under the Poincare probe, GeFLoSM’s states give 61.6 UUAS, 0.738 distance Spearman, 67.3 root accuracy and 0.768 depth Spearman, against 59.1, 0.710, 64.6 and 0.743 for the Euclidean twin. The effect on translation is measured by the twin (0.73 to 1.46 SacreBLEU, Table 3).

AuthorsOpenReview ↗

Reply 3 to Reviewer EXaC

Question 2. “To better isolate the contribution of the hyperbolic manifold, would it be possible to include a baseline where the same trajectory-based Flow Matching approach is implemented in Euclidean space?”

Reply. Yes, and it is included: the Euclidean twin is the last row of each panel of Table 3, and its definition, validity checks and protocol are in Appendix G (Table 8 lists every operation that differs). (The twin keeps every module, dimension, loss, loss weight, dataset, vocabulary, schedule and parameter count of GeFLoSM and replaces only the manifold operations by their c → 0 limits. It scores 28.14, 21.94 and 25.37 SacreBLEU on IWSLT15 EN-VI, WMT16 NC-v11 and IWSLT14 DE-EN, against 29.10, 23.40 and 26.10 for GeFLoSM, and it is lower on all four metrics in all three panels of Table 3.)

Question 3. “How does the model handle language pairs with significantly different word orders (e.g., English and Japanese)? Since the method relies on positional correspondence, is it sufficient to account for non-monotonic alignments when only conditioning on the target prefix?”

Reply. The positional pairing is a training-time prior that is not used at inference, where cross- attention handles reordering as in the vanilla Transformer (Sec. 3.4). (The paper evaluates on German and Vietnamese, whose word order differs systematically from English, and GeFLoSM improves over the vanilla Transformer on each: 29.1 against 27.6 on IWSLT15 EN-VI, Table 6; 23.4 against 20.3 on WMT16 NC-v11, Table 5; 27.2 against 26.0 on WMT16 EN-DE full, Tables 1 and 5.) Nothing in the pipeline is language-specific, and an English-Japanese evaluation is planned.

Thank you for adding the Euclidean twin and the probes; the twin addresses my second question. However, the motivation for the Poincaré ball is still not fully convincing to me, and the word-order concern remains, as the evaluated language pairs do not include strongly non-monotonic ones (e.g., head-final languages). I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.