acceptodds
Under review as a conference paper at ICLR 2027

One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces

Abstract

Embedding spaces are fundamental to natural language processing, translating text into high-dimensional vectors that encode rich semantic relationships. Yet, their internal structures remain opaque, with existing approaches often sacrificing semantic coherence for structural regularity or incurring high computational overhead to improve interpretability. To address these challenges, we introduce the Semantic Field Subspace (SFS), a geometry-preserving, context-aware representation that captures local semantic neighborhoods within the embedding space. We also propose SAFARI (SemAntic Field subspAce deteRmInation), an unsupervised algorithm that uncovers hierarchical semantic structures using a novel metric called Semantic Shift, which quantifies how semantics evolve as SFSes evolve. To ensure scalability, we develop an efficient approximation of Semantic Shift that replaces costly SVD computations, achieving a 1530 speedup with average errors below 0.01. Extensive evaluations on five text datasets show that SFSes outperform standard classifiers in classification and nuanced tasks such as political bias detection, while SAFARI consistently reveals interpretable semantic hierarchies, with additional validation on image embeddings confirming its generalizability. This work presents a unified framework for structuring, analyzing, and scaling semantic understanding in embedding spaces.

Primary area:
interpretability and explainable AI
Keywords:
Embedding spaces, Semantic interpretability, Unsupervised semantic structure discovery, Semantic field subspaces, Embedding geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iT3xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies semantic structure through changes in local embedding neighbourhoods. It represents neighbourhoods using Semantic Field Subspaces (SFSes), based on singular directions and singular values, and introduces Semantic Shift to measure changes during cluster merging. SAFARI combines agglomerative clustering with a dynamic threshold to select SFSes at notable transitions. Experiments examine hierarchical structure discovery, classification, and an efficient approximation of Semantic Shift across text and image embeddings.

Critical strengths
  • The perspective of examining how neighbourhood geometry evolves across scales is interesting. The qualitative examples provide interpretable illustrations of semantic groupings and motivate further investigation of this approach.
  • The paper evaluates the framework across text and image embeddings and reports competitive classification results against several standard classifiers.
  • The approximation offers a potentially useful computational improvement. The reported speedups and correlation with the exact Semantic Shift suggest that it can capture similar trends at substantially lower computational cost.
Critical weaknesses

The contribution beyond standard hierarchical clustering is not sufficiently isolated. SAFARI constructs its hierarchy by merging clusters according to the cosine distance between their centroids; Semantic Shift subsequently determines which nodes are retained as SFSes. However, neither the main experiments nor the appendix compares against existing hierarchical clustering methods. Table 1 shows classification utility but does not isolate the proposed method's benefits. Likewise, Figure 4 and the qualitative examples do not establish whether comparable semantic structures would already emerge from standard hierarchical clustering using the same embeddings.

Questions for authors
  • Could the authors compare SAFARI against hierarchical clustering baselines using the same embeddings, and isolate the effect of Semantic Shift-based node selection from that of the underlying clustering procedure?
  • What evidence connects threshold crossings to meaningful semantic boundaries?
1 comment

We thank the reviewer for the careful reading and for the two precise questions.

Critical Weaknesses.

The contribution beyond standard hierarchical clustering is not sufficiently isolated.

However, neither the main experiments nor the appendix compares against existing hierarchical clustering methods.

We ran this comparison. It uses existing hierarchical clustering methods on the same embeddings, and it separates the effect of Semantic Shift from the effect of the clustering procedure. The results are given in Table R4.

Table 1 shows classification utility but does not isolate the proposed method's benefits.

Table 1 tests whether SFSes preserve meaningful semantics (L411-412). The benefit of Semantic Shift is isolated in the new experiment.

Likewise, Figure 4 and the qualitative examples do not establish whether comparable semantic structures would already emerge from standard hierarchical clustering using the same embeddings.

The new experiment tests this on the same embeddings. Standard hierarchical clustering does not produce comparable structures. The clusters it returns are less pure than the SFSes at every level.

Questions For Authors.

Could the authors compare SAFARI against hierarchical clustering baselines using the same embeddings, and isolate the effect of Semantic Shift-based node selection from that of the underlying clustering procedure?

We ran this experiment on the AG-News embeddings. We built four hierarchies on the same embeddings. They are SAFARI's hierarchy and three standard hierarchical clustering methods from the literature, average, complete and Ward linkage. On each hierarchy, clusters are selected in three ways. Semantic Shift (Eq. 1) with the dynamic threshold of Algorithm 1 gives the SFSes of that hierarchy. The largest merge distances give the same number of clusters. A flat cut, the usual way a hierarchical clustering method returns clusters, gives the same number again. Each cell of Table R4 gives the impurity of the selected clusters at Lv1, Lv2 and Lv3. Lower is better.

Table R4

HierarchyClustersSemantic ShiftLargest merge distancesFlat cut
SAFARI720.48/0.48/0.200.70/0.64/0.360.64/0.55/0.29
Average linkage720.51/0.48/0.170.72/0.63/0.340.60/0.50/0.26
Complete linkage830.48/0.46/0.210.72/0.67/0.350.66/0.58/0.29
Ward linkage600.49/0.46/0.200.73/0.64/0.320.62/0.53/0.24

Table R4 isolates the two effects. Along a row the hierarchy is fixed and only the node selection changes. Semantic Shift gives the lowest impurity in every row, at every level. Down the Semantic Shift column the node selection is fixed and only the clustering procedure changes. The impurity stays almost the same, between 0.48 and 0.51 at Lv1. So the gain comes from Semantic Shift-based node selection and not from the underlying clustering procedure. The SFSes of SAFARI are also purer than the flat cut of each standard method. In the revised manuscript, we will add this experiment to Sec. 5.2.

What evidence connects threshold crossings to meaningful semantic boundaries?

We tested this against reference labels that the method never sees. A meaningful boundary lies where two different classes meet. So we measured the share of merges that join two different reference classes, for the merges at threshold crossings and for all merges (Table R5). Higher is better.

Table R5

DatasetMergesLv1Lv2Lv3
AG-NewsAt threshold crossings56%51%29%
AG-NewsAll merges52%44%19%
MIT-StatesAt threshold crossings36%21%13%
MIT-StatesAll merges5%2%1%

At every level on both datasets, a merge at a threshold crossing joins two different classes more often than a merge in general. On MIT-States it does so seven times more often at Lv1.

The clusters formed at threshold crossings are also purer than the same number of clusters taken at the largest merge distances (Table R6). Lower is better.

Table R6

DatasetSelected clustersLv1Lv2Lv3
AG-NewsSFSes0.480.480.20
AG-NewsLargest merge distances0.700.640.36
MIT-StatesSFSes0.260.130.08
MIT-StatesLargest merge distances0.510.280.18

The case study in the paper gives an example. Semantic Shift shows notable evolutions at two iterations of the Sports category (L459-460), and Figure 7 visualises the U.S. university sports groupings. In the revised manuscript, we will add both measurements to Sec. 5.2.

Reviewer_Pit2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at what happens to the local neighbourhood of an embedding as that neighbourhood grows. A neighbourhood ("semantic field", basically an \epsilon-ball in cosine distance) is represented by the span of its members, computed via SVD and weighted by singular values. During plain agglomerative clustering (centroid linkage, cosine) the authors compute a "semantic shift" between the subspace of the larger child and the subspace of the merged cluster If the shift exceeds a sliding-window mean + 3 std, the merged subspace is recorded as an SFS. The set of recorded SFSes, ordered by the dendrogram, is the "hierarchy". Because the SVD per merge is expensive they replace Eq. 1 by ||A_y||_2 * sigma_max(A_x) and justify this with Weyl's inequality plus a condition-number argument.

Critical strengths
  • The question is a good one and I like the framing: a point in embedding space has no meaning on its own, the structure of its neighbourhood is what carries the semantics, and one should look at how that structure changes under coarsening. Representing a neighbourhood by a (weighted) subspace rather than a centroid is reasonable, it absorbs near-duplicates for free.
  • The honest discussion in App. E (horse racing example, Fig. E7) that the recovered hierarchy is the model's and not an ontology is the kind of thing I wish more papers did. The authors clearly looked at their output.
Critical weaknesses
  1. The evaluation does not test the actual contribution. The hierarchy experiment (Sec. 5.2, Fig. 4) plots impurity against clustering iterations, not against the detected SFSes, and there is no baseline (Table A3 says N/A). Impurity going up with iterations and Lv0 > Lv1 > Lv2 > Lv3 is what you get from any agglomerative clustering with a coarsening of the labels, it says nothing about whether the semantic-shift criterion picks out good cut points. The obvious comparison would be the same dendrogram with standard cut selection (e.g. the inconsistency coefficient in scipy's linkage, or dynamic tree cut), which is incidentally almost the same rule as mu + 3 tau on merge heights.
  2. The semantic shift and its approximation are not well founded. Eq. 1 pairs the i-th singular value of two matrices of different size and sums absolute differences, so the quantity scales with cluster size, which is exactly why the "baseline grows gradually" and a dynamic threshold is needed. The directional term uses an ad-hoc nearest-counterpart cosine instead of principal angles / a Grassmannian distance, which would be the standard, basis-independent way to compare subspaces. The approximation ||A_y|| sigma_max(A_x) does not contain any geometric information about the merge beyond the sizes/norms of the two blocks. Theorem 4.1 is just Weyl's inequality applied to [A_x | 0] + [0 | A_y], and the step from "directional change is O(kappa(A_x))" to "approx sigma_max(A_x)" in App. F is not an argument.
Questions for authors
  • Please add a baseline for Sec. 5.2: same linkage, same dendrogram, cut by a standard criterion. How many SFSes does SAFARI return on AG-News / MIT-States and what is the impurity of exactly those clusters (not of the whole iteration sequence)?
  • What does "average error below 0.01" measure (Fig. 5, Sec. 5.4)? Given the scale difference in Fig. 6, is it a rank or normalised error?
  • Does the Semantic Shift criterion enter the classification experiments in any way, or is Table 1 a nearest-subspace classifier with one subspace per class? If the latter, which prior nearest-subspace methods were compared?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Pit2 (1/4)

We thank the reviewer for the careful reading and for the concrete experiments suggested.

Critical Weaknesses.

The hierarchy experiment (Sec. 5.2, Fig. 4) plots impurity against clustering iterations, not against the detected SFSes, and there is no baseline (Table A3 says N/A).

We ran the experiment the reviewer asks for. It scores exactly the detected SFSes, on the same dendrogram, against standard cut selection. The comparison uses two standard criteria, merge distance and the inconsistency coefficient of scipy. To compare at equal size, each standard criterion selects as many merges as SAFARI. The table gives the share of selected merges that join two different reference classes. Higher is better.

Table R1

DatasetSelected mergesLv1Lv2Lv3
AG-NewsSemantic Shift (SAFARI)56%51%29%
AG-NewsMerge distance79%68%51%
AG-NewsInconsistency coefficient45%34%8%
AG-NewsAll merges52%44%19%
MIT-StatesSemantic Shift (SAFARI)36%21%13%
MIT-StatesMerge distance76%50%36%
MIT-StatesInconsistency coefficient7%3%1%
MIT-StatesAll merges5%2%1%

A good cut point has to lie at a class boundary, and it has to return a coherent cluster. Both requirements come from the paper. An SFS marks the boundary at which a Semantic Field starts to represent a new concept (L181-184). Impurity measures the semantic coherence within the clusters (L357-358). Semantic Shift meets both requirements on both datasets. Neither standard criterion does. The impurity of the selected clusters is given in our response under Questions For Authors.

Impurity going up with iterations and Lv0 > Lv1 > Lv2 > Lv3 is what you get from any agglomerative clustering with a coarsening of the labels, it says nothing about whether the semantic-shift criterion picks out good cut points.

The experiment above tests whether the criterion picks out good cut points. It scores exactly the merges that Semantic Shift selects, and it compares them with all merges of the same agglomerative clustering. The row for all merges is what the agglomerative clustering gives without any criterion. Semantic Shift is at class boundaries more often than all merges, at every level and on both datasets. So the criterion picks out cut points that the agglomerative clustering alone does not give. Figure 4 answers a different question. It measures how well the clusters align with a reference hierarchy that is generated separately from the clustering (L351-353).

The obvious comparison would be the same dendrogram with standard cut selection (e.g. the inconsistency coefficient in scipy's linkage, or dynamic tree cut), which is incidentally almost the same rule as μ+3τ\mu + 3\tau on merge heights.

We ran this comparison, and the two rules are not the same. Applied to the merge heights or to the inconsistency coefficient of scipy, μ+3τ\mu + 3\tau selects no cluster on either dataset. Semantic Shift selects 72 SFSes on AG-News and 234 on MIT-States. Table R1 gives the comparison at equal size.

Semantic Shift is above all merges and above the inconsistency coefficient, at every level on both datasets. The inconsistency coefficient does not separate itself from all merges, and on AG-News it is below them at every level. Such a criterion adds nothing to the clustering.

Merge distance scores highest because it only selects the last 1% merges of the hierarchy. Its share reflects where its merges lie, and it finds no fields below the top. Semantic Shift selects 43 of 72 and 167 of 234 below the top. The clusters of merge distance are also the most mixed of the three at every level, as Table R3 shows.

Neither standard criterion decides how many clusters to select. With the μ+3τ\mu + 3\tau rule both select none, and in this comparison both receive the number from SAFARI. SAFARI determines the number itself from the semantic structure (L290-291). In the revised manuscript, we will add this comparison to Sec. 5.2.

AuthorsOpenReview ↗

Response to Reviewer Pit2 (2/4)

Eq. 1 pairs the ii-th singular value of two matrices of different size and sums absolute differences, so the quantity scales with cluster size, which is exactly why the "baseline grows gradually" and a dynamic threshold is needed.

The detection in SAFARI does not depend on the scale of Eq. 1, and Eq. 1 follows cluster size less closely than the Grassmannian distance does, as Table R2 shows.

The different size leaves no singular value without a partner. Filling AxA_x with zeros to the size of AnewA_{\mathrm{new}} leaves its singular values unchanged (L1144), so Eq. 1 compares two matrices of the same size. The pairing by index is the standard one. Singular values are ordered by magnitude, so the ii-th value gives the importance of the ii-th strongest direction before and after the merge. Weyl's theorem and Theorem 4.1 use the same pairing, and Theorem 4.1 bounds every paired difference by ∥Ay∥2\|A_y\|_2. Eq. 1 sums these differences, each weighted by the directional change of the same direction (Definition 4.2).

SAFARI does not threshold the raw value of Eq. 1. It compares each value with the mean and the standard deviation of the last ww values (L280-283). A gradual growth of the baseline raises this mean with it, so only a spike above the recent values creates an SFS. The paper reports the growth and gives it as the reason for this design (L285-289).

A relation between Semantic Shift and cluster size follows from the semantics. Expanding fields lead to broader, more abstract concepts (L244-246), and a large shift indicates a more abstract concept (L263-264). But cluster size does not determine Semantic Shift. The two are only partly correlated, and the SFSes are not the largest clusters, as most of them lie below the last 1% of the hierarchy.

The directional term uses an ad-hoc nearest-counterpart cosine instead of principal angles / a Grassmannian distance, which would be the standard, basis-independent way to compare subspaces.

We thank the reviewer for this insightful suggestion, and we tested it. Neither principal angles nor the Grassmannian distance can replace the directional term of Eq. 1. On the full spans, all principal angles are zero for every merge. On the leading directions, the Grassmannian distance follows cluster size more closely than Eq. 1 does, and it selects far fewer clusters. We computed the Grassmannian distance between the leading kk directions of SxS_x and SnewS_{\mathrm{new}}, for k=5k = 5, 10 and 20, and applied the same threshold. Table R2 gives the rank correlation with the size of the merged cluster and the number of clusters selected.

Table R2

CriterionCorrelation with size, AG-NewsCorrelation with size, MIT-StatesSelected, AG-NewsSelected, MIT-States
Semantic Shift (Eq. 1)0.520.3772234
Principal angles, k=5k = 50.780.6640
Principal angles, k=10k = 100.880.80253
Principal angles, k=20k = 200.920.874352

This holds for every kk and on both datasets, so Eq. 1 depends less on cluster size than the standard distance. For k=5k = 5 on MIT-States the Grassmannian distance selects no cluster and returns no SFS. It selects more clusters only as kk grows, and its correlation with size grows with kk as well. In the revised manuscript, we will add this comparison to Appendix C.

We used the leading kk directions because principal angles are zero on the full spans, for every merge. The members of CxC_x are also members of CnewC_{\mathrm{new}}, so the span of CxC_x lies inside the span of CnewC_{\mathrm{new}} (Definition 4.1). A basis-independent distance uses only the two spans, so it cannot detect a transition. This is why an SFS is represented by its singular directions weighted by their singular values (L211-214), and Semantic Shift compares these weighted directions. The SVD determines them from the data, so they are not an arbitrary basis.

The nearest counterpart is not ad hoc. A direction of SxS_x can take a different index in SnewS_{\mathrm{new}} after a merge, so pairing by index would report a large change for a direction that did not move. The nearest counterpart pairs each direction with the direction it became. The change is measured with the semantic distance of Definition 3.1 (Definition 4.2). This is the same distance that defines a Semantic Field (Definition 3.2) and selects the two clusters to merge (L277-278), so the directional term uses the one distance on which the whole method is built.

AuthorsOpenReview ↗

Response to Reviewer Pit2 (3/4)

The approximation ∥Ay∥ σmax⁡(Ax)\lVert A_y \rVert \, \sigma_{\max}(A_x) does not contain any geometric information about the merge beyond the sizes/norms of the two blocks.

The approximation contains geometric information about the merge, and the measurements show it. The exact Semantic Shift is computed from the SVD of the merged matrix, so it contains the full geometry of the merge. The approximation follows it. Over all merges, the Pearson correlation of the approximation with the exact Semantic Shift is 0.95 on AG-News and 0.80 on MIT-States. On AG-News it recovers 65 of the 72 SFSes. The paper reports a correlation of 0.92 on the Sports category (L462-463).

The position of AyA_y relative to AxA_x enters through the choice of the merge. SAFARI merges the two clusters with the smallest semantic distance (L277-279). Eq. 2 is evaluated only on such a pair. Theorem 4.1 then ties ∥Ay∥2\lVert A_y \rVert_2 to the merge. When AyA_y is merged into AxA_x, no singular value of AxA_x changes by more than ∥Ay∥2\lVert A_y \rVert_2, wherever AyA_y lies. So ∥Ay∥2\lVert A_y \rVert_2 controls the dimensional importance shift of the merge.

The spectral norm is also not a size. Two clusters with the same number of members can have very different spectral norms. For embedding vectors of the same length, ∥Ay∥2\lVert A_y \rVert_2 is large when the members of CyC_y point in the same direction and small when they spread out. The same holds for σmax⁡(Ax)\sigma_{\max}(A_x). So the approximation carries how concentrated each block is.

Theorem 4.1 is just Weyl's inequality applied to [Ax∣0]+[0∣Ay][A_x \mid 0] + [0 \mid A_y]

Theorem 4.1 builds on Weyl's Theorem, as the paper states (L321), and extends it to a merge. Weyl's Theorem compares a matrix with a perturbed matrix of the same size (Theorem F1). A merge adds the members of CyC_y to AxA_x, so AxA_x and AnewA_{\mathrm{new}} differ in size. Theorem 4.1 is the statement for this case. Its proof writes AnewA_{\mathrm{new}} as [Ax∣O]+[O∣Ay][A_x \mid O] + [O \mid A_y], and uses that the zero block leaves the singular values of AxA_x unchanged and that the spectral norm of [O∣Ay][O \mid A_y] equals that of AyA_y (L1138-1144).

The theorem establishes the foundation of Eq. 2, and Weyl's Theorem alone does not state it. After a merge, every singular value of the larger cluster changes by at most the spectral norm of the smaller cluster. The bound holds for every index ii. It needs no SVD of the merged matrix. So the dimensional importance shift is controlled by a quantity that is computed from AyA_y alone. Appendix C shows that this shift is the dominant component of Semantic Shift (Table C4). The theorem therefore gives a proven bound for the dominant component of Eq. 1, and it is what allows the speedup of 15 to 30 times over full SVD (L448-449).

The bound is also what Eq. 2 needs. An SFS is a spike in Semantic Shift. By Theorem 4.1, a merge with a small ∥Ay∥2\lVert A_y \rVert_2 cannot produce a large dimensional importance shift at any index. So ∥Ay∥2\lVert A_y \rVert_2 marks the merges at which a spike can occur. The bound gives the scale of the change at every index, and Eq. 2 uses this scale for the dimensional importance shift.

the step from "directional change is O(κ(Ax))O(\kappa(A_x))" to "≈σmax⁡(Ax)\approx \sigma_{\max}(A_x)" in App. F is not an argument.

Appendix F gives the argument for this step. The directional change grows with the condition number of AxA_x, which is σmax⁡(Ax)\sigma_{\max}(A_x) divided by σmin⁡(Ax)\sigma_{\min}(A_x) (L1146-1150). This follows the sensitivity analysis of Meyer (2000) and Belsley et al. (2005). Real embedding matrices are "noisy and numerically full-rank" (L211-212). For such matrices the smallest singular value is governed by the noise and not by the signal that the matrix carries (Bai and Yin, 1993; Rudelson and Vershynin, 2009). AxA_x, AyA_y and AnewA_{\mathrm{new}} come from the same embedding space and carry the same noise, so their minimum singular values are comparable (L1151-1153). The denominator of the condition number is then comparable from one merge to the next. The condition number therefore varies with σmax⁡(Ax)\sigma_{\max}(A_x), and so does the directional change (L1153-1155). Eq. 2 needs no more than this. The detection compares each value with the recent values of the same curve (L280-283), so a constant factor does not change the detected SFSes.

The experiments confirm the step. The exact Semantic Shift includes the directional change, which contributes 40.7% of it (Table C4). So the agreement between the approximate and the exact Semantic Shift reported above covers the directional change and not only the singular values.

AuthorsOpenReview ↗

Response to Reviewer Pit2 (4/4)

Please add a baseline for Sec. 5.2: same linkage, same dendrogram, cut by a standard criterion.

We added this baseline. It uses the same linkage, the same dendrogram and the two standard criteria of Table R1, with as many clusters as SAFARI returns SFSes. Table R3 reports the impurity of the selected clusters. For each cluster, we take the share of the members that lie outside the cluster's most frequent label class. Impurity is the average of this share over the clusters. Lower is better.

Table R3

DatasetSelected clustersMedian sizeLv1Lv2Lv3
AG-NewsSFSes307.50.480.480.20
AG-NewsMerge distance3790.700.640.36
AG-NewsInconsistency coefficient9.50.490.370.14
MIT-StatesSFSes380.260.130.08
MIT-StatesMerge distance4200.510.280.18
MIT-StatesInconsistency coefficient110.030.010.01

The SFSes are purer than the clusters of merge distance at every level, on both datasets. On AG-News at Lv1 they are the purest of the three. The inconsistency coefficient selects clusters of about ten members. Clusters this small cannot serve the goal of the paper, which is to discover a hierarchy of SFSes "at multiple granularities" (L401-402). A cluster of ten members is nearly pure by default. It can only be a fine group at the bottom of the hierarchy, and it cannot represent a broader field. So its low impurity comes from its small size and does not show good cut points. The same small size gives its low share of merges that join different classes. The SFSes on AG-News are thirty times larger and are still purer at Lv1.

Tables R1 and R3 must be read together, since they measure the two requirements of a good cut point. Merge distance meets the first and not the second. The inconsistency coefficient meets the second only because its clusters have about ten members, and such clusters do not form a hierarchy of fields. Semantic Shift does both. Neither standard criterion is ahead of Semantic Shift on both measures, and Semantic Shift is ahead of each of them on one measure at every level and on both datasets. The two standard criteria fail at opposite ends of the hierarchy. Merge distance returns only the largest clusters at the top, and the inconsistency coefficient returns only the smallest at the bottom. Semantic Shift returns fields across the hierarchy.

How many SFSes does SAFARI return on AG-News / MIT-States and what is the impurity of exactly those clusters (not of the whole iteration sequence)?

SAFARI returns 72 SFSes on AG-News and 234 on MIT-States. The impurity of exactly those clusters is given in the SFSes rows of Table R3. In the revised manuscript, we will add these results to Sec. 5.2.

What does "average error below 0.01" measure (Fig. 5, Sec. 5.4)? Given the scale difference in Fig. 6, is it a rank or normalised error?

It is the absolute difference between the exact Semantic Shift (Eq. 1) and the approximate Semantic Shift (Eq. 2). It is neither a rank error nor a normalised error.

The scale difference in Fig. 6 does not affect the result. Fig. 6 draws the exact and the approximate curves on two vertical axes and compares them by Pearson correlation, which does not depend on scale. The detection does not depend on scale either, as multiplying a curve by a constant leaves the detected SFSes unchanged. In the revised manuscript, we will state in Sec. 5.4 that the error is absolute.

Does the Semantic Shift criterion enter the classification experiments in any way, or is Table 1 a nearest-subspace classifier with one subspace per class? If the latter, which prior nearest-subspace methods were compared?

Semantic Shift enters the classification experiments. Table 1 is not a classifier with one subspace per class. A test embedding is compared with "all identified SFSes" and receives the label of the nearest one (L416-418). These SFSes are the ones that SAFARI identifies, and SAFARI selects them with Semantic Shift. The baselines in Table 1 are SVM, KNN, RF and MLP (L413-416).

References

Z. D. Bai and Y. Q. Yin. Limit of the smallest eigenvalue of a large dimensional sample covariance matrix. The Annals of Probability, 21(3), 1275-1294, 1993.

Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics, 62(12), 1707-1739, 2009.

Reviewer_8vdzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes SAFARI, an unsupervised algorithm that aims to summarize the geometrical structure from local to global embedding neighborhoods. For this, the authors introduce Semantic Field Subspace and Semantic Shift, which characterizes geometric structure and evolution as changes in semantic subspaces, yielding in interpretable measures of the embedding geometry. They tested across different datasets and classifier methods revealing good balance in efficiency and recall in comparison with baselines.

Critical strengths

The idea of exploring geometric embedding space as it evolves from local to global geometry is very compelling, and the computed metrics overall make methodological sense. Geometric representations have potential utility and interpretability, with competitive classification performance. Furthermore, this being a text + vision approach is a big contribution in the comprehension regarding geometric representations of meaning in embedding spaces in both domains.

Critical weaknesses

In its current form, the manuscript has important conceptualization issues that are relevant for the link between the theoretical and the methodological formulation. I feel that, at times, this conceptualization issues makes it difficult to understand what the authors are really looking for with their proposed metrics, which also makes the results difficult to interpret. Specifically, it seems to me that the concepts of concreteness and semantic granularity/specificity are not clear nor well-defined. Also, methodological choices as the “impurity” metric need further clarification, as the current explanation of “semantic coherence” is theoretically broad. Finally, I think some comparisons with human semantic cognition could be better grounded and referenced.

Questions for authors

I will organize my concerns mainly in three:

  1. Theoretical clarifications of the framework A. When calling a concept “concrete” or “abstract” (concreteness), in some definitions we typically refer to the grounding of its representations in sensorimotor or physical reference (for example, the difference between dog and freedom). In the other hand, talking about a concept being “specific” or “generic” (specificity/granularity), we may refer to how superordinate the concept is (for example, the difference between dog and animal). Given this brief description, it is hard to me to clearly understand what the authors are trying to capture when talking about semantic shifts, as it has being referred in the manuscript in instance as something that evolves form something concrete into something generic (thus mixing concreteness and specificity), in others as something that evolves from being more concrete to being more abstract, and in others as marking the semantic change from one field to another. Could the authors clarify which one the definitions are the intended one with the metrics they are computing?

  2. Methodological clarifications and baselines A. In its current form, I feel the computation of “Impurity” is not very clear regarding to its contribution in the evaluation of the resulting semantic clusters. When the authors state that it measures “semantic coherence within clusters”, it is not automatically clear to me what semantic coherence means. This is connected with my previous question. At times it feels like semantic coherence may refer to semantically similar concepts, taxonomically related ones, or concepts that pertain to similar hierarchies. Following this, could the authors further explain what the Impurity metric is computing? How do this evaluation metric prove that clusters are organized in the way the authors intended to organize? What constitutes a coherent Semantic Field, and what semantic property changes (specificity, concreteness, semantic distance) when a Semantic Shift occurs?

B. Although the authors compare their approach with several benchmarks, I think it would be valuable to compare their clustering results with other ways of computing geometric relationships in embeddings, or hierarchical clustering methodologies in literature.

  1. Clarification regarding relation to the framework and theories of human semantic representations A. I am aware that the authors do not intend to compare embedding with human semantics as their main goal. However, in several parts of the manuscript, this comparison is made. I think that the link with cognitive science theories of semantics is a very valuable addition for this work, and it makes a lot of sense, particularly in terms of interpretability of the framework. Furthermore, there are several places in the manuscript in which I think this relation has been established with assumptions that are not correctly referenced, nor further developed. For example: “However, since semantics naturally form hierarchies, there exists a boundary beyond which the Semantic Field contains embeddings diverse enough for it to transcend from a concrete concept to a more abstract one.” This is not automatically true. Concrete concepts do not automatically transform into abstract concepts as they have more associations. In fact, some concrete concepts may have more association than abstract ones as an effect of familiarity (e.g., “dog” vs. “subreptitious”). It seems to me that, the question here is when a target concept has neighbors that are related to that concept and when there exists a “critical jump” of meaning that makes necessary to build another semantic field. If I am correctly following the authors’ intentions, the important question should be whether there is a detectable change in the semantic structure, not necessarily whether concepts become more abstract. To my view, exploring these theories may help with the interpretations the authors are trying to make, for example the one in section E.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8vdz (1/4)

We thank the reviewer for the constructive feedback and respond to each concern below.

it seems to me that the concepts of concreteness and semantic granularity/specificity are not clear nor well-defined.

We clarify each concept. Semantic Shift measures the change in the semantic structure of a growing Semantic Field. When it is large, the field has become broader. Its specificity/granularity has decreased. Concreteness is not measured. By a coherent Semantic Field we mean one whose members refine the interpretation of its target, such as the models of one product or the teams of one sport. Impurity measures how far each cluster is concentrated in one reference label class, and it is 0 when the concentration is perfect.

Theoretical clarifications of the framework, A.

Could the authors clarify which one the definitions are the intended one with the metrics they are computing?

The intended definition is the semantic change from one field to another, and a large Semantic Shift marks such a change. Semantic Shift measures the change in the semantic structure of a growing Semantic Field (Eq. 1). After a large shift the field is broader, so its specificity/granularity has decreased. The step from dog to animal in the reviewer's example is a change of this kind. The difference between dog and freedom is concreteness in the sensorimotor sense, which we do not measure.

Example 4.1 (L293-300, Figure 3) shows this. Merging Macbook Air with Macbook Pro causes no significant shift. Merging Apple with the Macbook cluster triggers a significant Semantic Shift and forms a new SFS, Apple (IT Company). This SFS and the one for Microsoft are later covered by the broader SFS IT Companies.

In the revised version we will use the terms specific and generic consistently in place of concrete and abstract. L263-265 will read as follows.

"Semantic Shift acts as a boundary criterion. A large shift suggests that the new subspace represents a broader field that subsumes the previous one, whereas small values reflect refinements within the same semantic field."

Methodological clarifications and baselines, A.

could the authors further explain what the Impurity metric is computing?

Impurity measures how far each cluster is concentrated in one reference label class. The reference labels form a hierarchy with four levels (Lv0 to Lv3, L351-352), and impurity is computed at one level at a time. LiL_i is label class ii and CjC_j is cluster jj (L356). The term ∣Li∩Cj∣|L_i \cap C_j| counts the members that label class ii and cluster jj share. For each cluster, impurity takes its largest overlap with a label class and divides it by the size of the cluster. One minus this ratio is the share of the cluster that lies outside that label class. Impurity is the average of this share over the clusters. Lower is better, with 0 denoting "perfect label concentration within clusters" (L357-358).

How do this evaluation metric prove that clusters are organized in the way the authors intended to organize?

The intended organisation is that the detected fields correspond to classes of the reference hierarchy, at several levels of granularity. We test this against labels that the method never sees.

Figure 4 tracks impurity over the merging process at each of the four levels. Impurity is lowest at the most generic level (Lv3) and highest at the most specific one (Lv0), on both datasets. This is the expected ordering (L358-360). The paper also expects impurity to grow "due to the merging of semantically broader categories" (L358-360), and Figure 4 shows this growth.

To test this on the detected fields directly, we ran an additional experiment on the SFSes themselves. We computed the impurity of each SFS. As a baseline, we took the same number of clusters from the same dendrogram at the largest merge distances, which are the clusters that standard hierarchical clustering returns. The results are below, and lower is better.

DatasetSelected clustersLv1Lv2Lv3
AG-NewsSFSes0.480.480.20
AG-NewsLargest merge distances0.700.640.36
MIT-StatesSFSes0.260.130.08
MIT-StatesLargest merge distances0.510.280.18

The SFSes are purer than the baseline clusters at every level reported, on both datasets. On MIT-States their impurity is about half that of the baseline. On MIT-States, an average SFS has 92% of its members under one Lv3 label. We will add this experiment to Sec. 5.2.

AuthorsOpenReview ↗

Response to Reviewer 8vdz (2/4)

What constitutes a coherent Semantic Field, and what semantic property changes (specificity, concreteness, semantic distance) when a Semantic Shift occurs?

A Semantic Field is the neighbourhood of a target embedding (Definition 3.2). We use semantic coherence as a descriptive expression (L357). We call a field coherent as long as expanding it only adds context that refines the interpretation of the target. In that case the Semantic Shift stays small and the field remains the same field. The field stops being coherent at the boundary where it starts to represent a new concept. A large Semantic Shift marks this boundary, and SAFARI then records a new SFS. Example 4.1 shows both cases. Macbook Air and Macbook Pro merge without forming an SFS. Merging Apple with the Macbook cluster triggers a significant Semantic Shift and forms a new SFS.

By coherent we mean "semantically similar concepts". The members of a field are neighbours of its target under semantic distance (Definitions 3.1 and 3.2). We do not mean a taxonomy or a hierarchy. The reference hierarchy serves only as the test. Members that refine the interpretation of one target should share a reference class.

When the Semantic Shift is large, the specificity of the field decreases. The field becomes a broader one that subsumes the previous field. Concreteness in the sensorimotor sense is not measured. Semantic distance is the criterion by which clusters are merged (L277-279). It changes at every merge and does not mark boundaries, and we compare it with Semantic Shift in our response to Methodological clarifications and baselines, B.

Methodological clarifications and baselines, B.

I think it would be valuable to compare their clustering results with other ways of computing geometric relationships in embeddings, or hierarchical clustering methodologies in literature.

We ran this comparison against hierarchical clustering from the literature. On each of four hierarchies, Semantic Shift selects purer clusters than the largest merge distances and than a flat cut, at Lv1, Lv2 and Lv3.

We built the four hierarchies on the same AG-News embeddings. They are SAFARI's hierarchy and three standard hierarchical clustering methods from the literature, average, complete and Ward linkage. On each hierarchy, Semantic Shift (Eq. 1) with the dynamic threshold of Algorithm 1 gives its SFSes. The largest merge distances give the same number of clusters. A flat cut, the usual way a hierarchical clustering method returns clusters, gives the same number again. Each cell gives impurity as measured for the SFSes in our response to Methodological clarifications and baselines, A, at Lv1, Lv2 and Lv3. Lower is better.

HierarchyClustersSemantic ShiftLargest merge distancesFlat cut
SAFARI720.48/0.48/0.200.70/0.64/0.360.64/0.55/0.29
Average linkage720.51/0.48/0.170.72/0.63/0.340.60/0.50/0.26
Complete linkage830.48/0.46/0.210.72/0.67/0.350.66/0.58/0.29
Ward linkage600.49/0.46/0.200.73/0.64/0.320.62/0.53/0.24

In a second comparison, SAFARI's clusters are compared with the clusters that standard agglomerative clustering returns on the same embeddings. Its two standard criteria are merge distance and the inconsistency coefficient. Semantic Shift is the only one of the three criteria that, on both datasets, both finds fields below the top of the hierarchy and selects merges that join different reference classes more often than merges in general. The summary and the full numbers follow.

CriterionSelects merges below the top 1% of the hierarchySelected merges join different classes more often than merges in general
Semantic Shift (SAFARI)YesYes, on both datasets
Merge distanceNoYes, at the top only
Inconsistency coefficientYesLess often on AG-News, about as often on MIT-States

SAFARI builds its tree in the same way, so the only difference is Semantic Shift. The three linkages and both baselines are also other ways of computing geometric relationships in embeddings. They use distances, while Semantic Shift uses subspaces. Each criterion selects the same number of merges, 72 on AG-News and 234 on MIT-States.

The tables give the share of selected merges that join two different reference classes, for AG-News and then MIT-States. Higher is better.

Selected mergesLv1Lv2Lv3
Semantic Shift (SAFARI)56%51%29%
Merge distance79%68%51%
Inconsistency coefficient45%34%8%
All merges52%44%19%
Selected mergesLv1Lv2Lv3
Semantic Shift (SAFARI)36%21%13%
Merge distance76%50%36%
Inconsistency coefficient7%3%1%
All merges5%2%1%
AuthorsOpenReview ↗

Response to Reviewer 8vdz (3/4)

At every level on both datasets, Semantic Shift is at class boundaries more often than the inconsistency coefficient and than all merges. The gap is large on MIT-States, 36% against 7% and 5% at Lv1. On AG-News it is 11 to 21 points and 4 to 10 points.

Merge distance scores highest because it only selects the last merges. All 72 and 234 of them are in the last 1% of the hierarchy. The largest and most different clusters join there. So it finds no fields below the top. Its clusters are also less pure than the SFSes. At Lv1 this is 0.70 against 0.48 on AG-News and 0.51 against 0.26 on MIT-States. Semantic Shift selects most of its merges below the top. These are 43 of 72 on AG-News and 167 of 234 on MIT-States.

Semantic Shift is thus the only one of the three criteria that does both. It finds fields below the top of the hierarchy. It places their boundaries where reference classes meet. We will add both comparisons to Sec. 5.2.

Clarification regarding relation to the framework and theories of human semantic representations, A.

there are several places in the manuscript in which I think this relation has been established with assumptions that are not correctly referenced, nor further developed.

We agree that the quoted sentence needs a reference and a more careful wording. The linguistic basis of our framework is semantic field theory and componential analysis (Ullmann, 1957; Nida, 2015), cited at L130-132, which argue that "meaning arises only through contextual associations". This supports our Semantic Field (Definition 3.2). It does not support the claim that a field goes from the concrete to the abstract. We will remove this claim from the sentence. In the revised version the sentence (L181-183) will read as follows.

"As a Semantic Field expands, there may be a boundary beyond which its members no longer share the same structure, and the field starts to represent a new, broader concept."

At L245-246, where the paper refers to "the hierarchical nature of language semantics", we will also cite Rosch et al. (1976) on levels of categorisation and Mirman et al. (2017) on taxonomic and thematic relations, so that this statement is well supported.

Concrete concepts do not automatically transform into abstract concepts as they have more associations. In fact, some concrete concepts may have more association than abstract ones as an effect of familiarity (e.g., “dog” vs. “subreptitious”).

We fully agree. The sentence was meant to describe the Semantic Field, the neighbourhood around a target embedding, and not a single concept. The example that follows it in the paper (L186-188, Figure 2) shows this. The nearest neighbour of Coca-Cola is Coke, and a larger neighbourhood also contains Sprite and Pepsi. The neighbourhood then represents a broader concept, while Coca-Cola itself is unchanged. The reworded sentence above no longer permits the other reading.

If I am correctly following the authors’ intentions, the important question should be whether there is a detectable change in the semantic structure, not necessarily whether concepts become more abstract.

We thank the reviewer for the insightful comment. The reviewer follows our intention correctly, and we adopt this formulation. Semantic Shift (Definition 4.2) measures the change in the semantic structure of a growing Semantic Field, and a large value signals a detectable change. As stated in L183-185 of the submission, the aim is to determine the boundary at which a Semantic Field starts to represent a new concept. To avoid confusion, we will remove the wording about concepts becoming more abstract in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 8vdz (4/4)

exploring these theories may help with the interpretations the authors are trying to make, for example the one in section E.

We thank the reviewer for this suggestion, and we have taken it up. In the revised version we will relate our framework to three lines of work in cognitive science. The first is the work on basic-level categories (Rosch et al., 1976). Its subordinate, basic and superordinate levels offer an analogy for our narrower and broader fields. The second is the work on concreteness norms (Brysbaert et al., 2014). It makes precise the notion of concreteness that the reviewer separates from specificity and that we do not measure. The third is the work on taxonomic and thematic relations (Mirman et al., 2017), which separates the two ways in which our fields broaden. Appendix E will be rewritten in these terms and will open as follows.

"Human taxonomies organise concepts into subordinate, basic and superordinate levels (Rosch et al., 1976). The fields found by SAFARI often broaden in a comparable way, as when U.S. basketball teams are first grouped with U.S. football teams and later with sports teams from various locations. They also broaden by association, as when racing horses are merged with the clubs, companies and events of horse racing (Figure E7). The second kind corresponds to thematic relatedness in human semantics and not to a taxonomy (Mirman et al., 2017). Such cases show how the model organises these entities, and they are the reason why the hierarchies found in an embedding space should not be expected to mirror a human ontology."

References

Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46(3), 904-911, 2014.

Daniel Mirman, Jon-Frederick Landrigan, and Allison E Britt. Taxonomic and thematic semantic systems. Psychological Bulletin, 143(5), 499-520, 2017.

Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braem. Basic objects in natural categories. Cognitive Psychology, 8(3), 382-439, 1976.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.