One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces

Embedding spaces are fundamental to natural language processing, translating text into high-dimensional vectors that encode rich semantic relationships. Yet, their internal structures remain opaque, with existing approaches often sacrificing semantic coherence for structural regularity or incurring high computational overhead to improve interpretability. To address these challenges, we introduce the Semantic Field Subspace (SFS), a geometry-preserving, context-aware representation that captures local semantic neighborhoods within the embedding space. We also propose SAFARI (SemAntic Field subspAce deteRmInation), an unsupervised algorithm that uncovers hierarchical semantic structures using a novel metric called Semantic Shift, which quantifies how semantics evolve as SFSes evolve. To ensure scalability, we develop an efficient approximation of Semantic Shift that replaces costly SVD computations, achieving a 15$\sim$30$\times$ speedup with average errors below 0.01. Extensive evaluations on five text datasets show that SFSes outperform standard classifiers in classification and nuanced tasks such as political bias detection, while SAFARI consistently reveals interpretable semantic hierarchies, with additional validation on image embeddings confirming its generalizability. This work presents a unified framework for structuring, analyzing, and scaling semantic understanding in embedding spaces.