acceptodds
Under review as a conference paper at ICLR 2027

A Geometric Theory of Semantic Relation Operators

Abstract

Analogies such as man : woman :: king : queen demonstrate that two relations can be applied in either order with the same result. This property is formalized as the flatness of a discrete connection, where tokens are represented as vertices in a graph and edges are labeled by relation types, each acting on embeddings via a specific affine operator. An analogy square corresponds to a closed walk whose holonomy is defined by the commutator of the two operators. Exactly fitted squares enforce flatness if their base points affinely span the space. Flat squares ensure path independence of composite analogies when they generate the fundamental group. For contextual representations, operators may depend on context through softmax mixtures of Lie-algebra generators, such that curvature is uniformly bounded by their brackets. In an evaluation of 1,154 mined morphological analogy squares in GPT-2, affine operators outperform vector offsets by ten points on two-hop analogies, but display path dependence in 11–25% of held-out compositions. Application of curvature regularization reduces path dependence to 3–6% across all relation pairs, and ensures consistency in unpenalized four-hop detours, validating our theoretical predictions. When syntactic frames shift, static operators lose 8–15 points, while contextual operators recover to within two points of per-context oracles and exhibit context dependence.

Primary area:
interpretability and explainable AI
Keywords:
Semantic Analogy, Learned Representations, Contextual Consistency, holonomy, topology

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NJDNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a geometric and topological framework for modeling semantic relations in static word embeddings and contextual language model representations. By viewing tokens as vertices on a directed graph and relations as shared affine operators in Aff(D)GL(D+1), the authors formalize analogical consistency (e.g., applying gender then royalty vs. royalty then gender) as the flatness of a discrete relation-typed connection. The holonomy around an analogy square is given by the group commutator rs=s-1r-1sr.

The authors prove several foundational properties: 1. exact fit on affinely spanning base points forces flatness; 2. vector offsets correspond to the trivially flat case; and 3. trivial holonomy on elementary squares guarantees global path independence iff the induced square 2-complex KQ is simply connected.

For contextual representations, relation operators are parameterized as matrix exponentials of softmax mixtures of Lie-algebra basis generators, bounding curvature across contexts via basis brackets.

Empirically, across GloVe and a newly mined benchmark of 1,154 morphological squares in GPT-2, the authors demonstrate that affine operators substantially improve retrieval accuracy over vector offsets (+10 points) at the cost of path dependence (11–25% inconsistency), and that adding a curvature regularization penalty restores path consistency (94–97%) and generalizes to unpenalized 4-hop detours with negligible accuracy loss.

Critical strengths
  1. Formalizing shared relation operators as a discrete connection where analogy closure is a fixed-point condition on the commutator cleanly unifies vector-offset models (Corollary 3) and linear/affine relational probes

  2. Theorem 2(v) and Proposition 4 provide exact geometric intuition for when local closure on training points fails to force global commutativity.

  3. Using the square complex and proving that the empirical graph of 1,154 squares is a forest of attached discs (hence simply connected, Appendix J) yields a strong, falsifiable prediction: penalizing 2-hop squares automatically enforces consistency on unpenalized 4-hop detours, which is borne out empirically (reducing detour inconsistency from 21–24% to 5–6%).

  4. Contextual Lie-Algebra Parametrization and kernel initialization (Remark 11) elegantly solves the generator collapse issue and allows bounding unseen context curvature via the convex hull of basis Lie brackets.

Critical weaknesses
  1. All language model experiments are conducted exclusively on GPT-2 (124M) with representations projected via PCA to D=48 or D=64. Modern open-weight LLMs (e.g., Llama-3, Qwen, Mistral) use Rotary Position Embeddings, SwiGLU, and grouped-query attention, which may induce different geometric properties in intermediate layers.

  2. As acknowledged in the Appendix, the 1,154 mined morphological squares are constructed via string prefix/suffix matching, which includes non-compositional semantics. While filtering to 18 compositional prefixes mitigates this, morphological derivation in a single token is the easiest regime for affine operators compared to multi-token factual or encyclopedic relations.

  3. The authors note that using the frame prefix hidden state alone yields much weaker context dependence (0.498 vs. 0.491 for frozen). Requiring a forward pass of all 4,592 candidate words in a new context just to compute the context descriptor is impractical for real-time inference on a single sentence.

  4. Repetitive Related Work (Appendix B), suggesting an editing oversight when merging sections.

Questions for authors

Have you tested whether the accuracy-vs-consistency trade-off holds in modern LLMs? Or without PCA dimensionality reduction? Does PCA projection artificially smooth out high-frequency curvature?

Why does the prefix-only hidden state fail to capture the syntactic frame shift, whereas the centroid of the 4,592 target words succeeds?

Reviewer_WbR3OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The submission proposes to assign linear operators to relations between embeddings. Each relation between two words is treated as transformation. In a conventional word analogy, the transformations can be followed in a four-word/token/morphological unit loop.

Critical strengths

The idea to present semantic relations as transformations on a graph and treat path inconsistencies with holonymy is potentially interesting, even though the broader idea might not be entirely novel.

Critical weaknesses

Abstract: The abstract is generally very opaque and gives little explicit motivation on why the proposed approach is interesting. It would benefit from more clarity of proposed method, motivation, and obtained results. Even after reading the intro and rest of the paper it seems still hard to fully understand or just follow the argumentation of the abstract.

Intro: The mentioned contributions seem to relate more to methodological details than to actual contributions and the proposed test bed is unclear. The reported results are consequently equally incomprehensible until much later, including reading the appendix.

The paper seems to collapse lexical semantic relations and knowledge graph relations even in case of KG embeddings into one and the same category.

The paper is extremely opaque and seems to rely too strongly on the appendices. Instead, it should be easy to read and self-contained.

The dataset being used is not explicitly described sufficiently and the entire experimental setup is as opaque as the remainder of the paper.

The basic constraint says an operator can approximately map one endpoint embedding to the other. A single input-out pair does not identify a unique linear map. How are operators established and shared across examples of the same relations?

Questions for authors

In the abstract, there seems to be a mixture of words, tokens, and morphological elements. So the focus is not on words or tokens but mined morpheme analogies? What do they look like and where do they come from?

Why GPT-2?

"In this paper, we establish a correspondence between vocabulary embeddings and contextual representations in language models." What exactly do you mean by vocabulary embeddings and how is this "correspondence" established and what for?

Reviewer_DZa4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Since the seminal work of Mikolov, analogies have been used as an important tool for evaluating the quality of word embeddings. Building on this idea, this paper establishes a geometric framework to study semantic relations and analogical consistency in word embeddings and contextual language-model representations. Semantic relations are represented by affine operators acting on embedding vectors, and analogies are interpreted as commutative diagrams in which applying two relations in different orders leads to the same result.

From a theoretical point of view, the authors formalize this interpretation using concepts from differential geometry and topology, particularly discrete connections, holonomy, and curvature. They establish theoretical results connecting the closure of analogy squares to operator commutativity and characterize conditions under which local consistency guarantees path independence for more complex compositions. They also propose curvature-based regularization and extend their framework to context-dependent operators.

From an empirical point of view, the authors evaluate the improvement of relational prediction using affine operators on GloVe and GPT-2 representations, using morphological analogy squares constructed from combinations of prefixes and suffixes.

Critical strengths

The authors propose an original and mathematically interesting framework. I particularly appreciate the perspective opened by this representation of semantic relations through affine transformations and the study of their compositions using holonomy and curvature. The theoretical development is relatively extensive and proposes and interesting treatment of compositional consistency. The paper goes beyond conventional analogy completion by investigating whether different sequences of relational transformations produce consistent results. The distinction between fitting individual relations and ensuring consistency across compositions is particularly relevant. Moreoever, the experiments demonstrate substantial improvements over translation-based approaches in relational prediction, while curvature regularization significantly reduces disagreement between alternative composition paths with limited impact on accuracy. The contextual extension provides additional evidence that relation operators can benefit from adapting to linguistic context.

Critical weaknesses

My main weakness is that the paper completely ignores the literature on analogies, and proposes a work in isolation from decades of research made on the topic. It is appreciable to see some interest beyond the parallelogram rule (the use of which has already been discarded / strongly criticized in word embeddings), but it is frustrating to see that the work of specialists is ignored in this attempt to go beyond. In particular, the analogy squares considered in the paper are closely related to the central permutation property of analogical proportions, while global commutativity of the corresponding operators constitutes a substantially stronger requirement. It is unclear whether this stronger condition is necessary for analogical reasoning, or whether it primarily characterizes a restricted class of compositional relations. A clearer distinction between analogical proportions, their representation through relational operators, and the additional requirement of path independence would considerably strengthen the theoretical positioning of the work. I would suggest in particular the following readings:

  • "Analogical Proportions: Why They Are Useful in AI." Prade and Richard, 2021
  • "Analogy and formal languages" Lepage, 2004
  • "Analogical dissimilarity: definition, algorithms and two experiments in machine learning" Miclet et al., 2008
  • "Analogical proportions" Antic 2022
  • "Opening the parallelogram: Considerations on non-Euclidean analogies", Murena et al., 2018

The main experiments rely on morphological constructions involving prefixes and suffixes, where the commutativity of transformations is largely determined by the construction of the dataset. These examples provide a useful controlled setting for evaluating compositional consistency, but they do not necessarily demonstrate meaningful semantic or analogical reasoning. In particular, the experiments do not establish whether the framework generalizes to semantic analogies involving richer relational structures, or whether the learned operators capture relational meaning beyond regularities in word formation. Evaluation on established semantic analogy datasets or more complex relational reasoning tasks would significantly strengthen the contribution.

Questions for authors
  • What is the relation between your proposed approach and the framework of analogical reasoning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.