OySTER: Ortho-Spherical Transformation to Preserve Semantics in Embedding Compression

The impetus towards larger language models, stemming from better world knowledge along with enhanced semantic and contextual understanding, has led to diverse text and content representation via vector embeddings of extremely high dimensionality. However, this poses severe challenges in terms of embedding storage (in vector databases) and computational inefficiencies (for similarity and nearest neighbors) in downstream applications like search and retrieval. This paper presents OySTER, a novel embedding compression method combining orthogonal eigen-space projection of high dimensional text embedding (to align with informative dimensions) along with truncated hyper-spherical coordinate transformation. Extensive experiments on MTEB benchmark datasets depict improved performance of our approach, in terms of semantic understanding and retrieval tasks, even under 64x compression ratio. Additionally, we show that OySTER inherently provides Matryoshka embeddings as a by-product – performing better than the models trained with Matryoshka learning objective.