acceptodds
Withdrawn from ICLR 2027

OySTER: Ortho-Spherical Transformation to Preserve Semantics in Embedding Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The impetus towards larger language models, stemming from better world knowledge along with enhanced semantic and contextual understanding, has led to diverse text and content representation via vector embeddings of extremely high dimensionality. However, this poses severe challenges in terms of embedding storage (in vector databases) and computational inefficiencies (for similarity and nearest neighbors) in downstream applications like search and retrieval. This paper presents OySTER, a novel embedding compression method combining orthogonal eigen-space projection of high dimensional text embedding (to align with informative dimensions) along with truncated hyper-spherical coordinate transformation. Extensive experiments on MTEB benchmark datasets depict improved performance of our approach, in terms of semantic understanding and retrieval tasks, even under 64x compression ratio. Additionally, we show that OySTER inherently provides Matryoshka embeddings as a by-product – performing better than the models trained with Matryoshka learning objective.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Embedding Compression, Dimensionality Reduction, Spherical Transformation, Matryoshka Embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MeBNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a dimension-reduction approach called OySTER, which consists of four steps: First, an SVD decomposition is used to obtain an orthogonal basis; next, the embedding is projected onto each orthogonal basis and converted to spherical coordinates; then, the first K angles are retained as the compressed spherical coordinate embedding; finally, when calculating similarity, the vector is reconstructed from the spherical coordinate embedding. The authors claim that their method outperforms PCA and UMAP.

Critical strengths
  1. The research question has practical significance, and the method requires no pre-training.
  2. The study considers multiple embedding tasks, as well as models with different architectures and different compression ratios.
Critical weaknesses
  1. After reading the paper, I had a question: following the authors’ method of retaining the first k angles and reconstructing the first k coordinates seems to yield the same results as directly retaining the first k coordinates of the orthogonal projection. I confirmed this with an AI, which corroborated my understanding. I also had the AI implement Python code to perform numerical simulations, which validated my hypothesis. If the paper’s description is accurate, then I believe the method proposed by the authors is essentially not a new method.
  2. Based on the previous point, I do not understand why there is a discrepancy between the authors’ method and PCA. However, the authors do not appear to report which data set was used for the SVD decomposition, nor do they provide implementation code. Therefore, I am actually unclear as to why the authors’ method is effective, and it is also because the description of the proposed method in the main text is incorrect.
  3. The experiments lack a sufficient number of baselines, which should at least include (1) the product quantization methods listed by the authors themselves, and (2) a basic baseline obtained by directly extracting the first K dimensions of the original embeddings (since Qwen3 Embedding has already been trained using matryoshka representation learning, the embeddings formed by the first K-dimensional vectors should also be of high quality).
  4. The organizational structure of the paper’s Introduction is problematic; the “Related Work” section takes up a significant amount of space while lacking a description of the authors’ own method.
  5. The paper contains several instances of imprecise wording, such as claiming that GPT-6 Astra has 16,384 dimensions without providing evidence, and asserting that their method can generate “Matryoshka” embeddings—which should actually be the name of a supervised training method.
Questions for authors
  1. Can you provide the exact algorithmic implementation of OySTER and explain how it differs from retaining the first k coordinates after a direct orthogonal projection?
  2. Why is there only one result for Qwen3 Embedding 0.6B? Since it already used matryoshka representation learning during training, it could have reported low-quality results.
Reviewer_FTMvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a post-hoc method, named OySTER, to reduce the dimension of text embeddings. It projects each embedding onto the singular vectors of an embedding matrix from a training set. It converts the projected vector to hyper-spherical angles with Equation 1. It keeps the first kk angles as the compressed embedding. At query time it reconstructs a kk-dimensional vector with Equation 2 and computes the cosine similarity. The paper says that the ordered angles also give Matryoshka embeddings without MRL training. Experiments on a subset of MTEB with two models show small gains over PCA and UMAP, and gains over MRL truncation of the base models.

Critical strengths

Motivation is good and writting is easy to follow.

Critical weaknesses

The method is equal to PCA truncation. In Equation 2, the product of the sines equals the norm of the tail vector from index jj, and cos⁡θj\cos\theta_j equals xjx_j divided by the same norm. The two cancel, so the reconstruction from the first kk angles returns the first kk PCA coordinates with no error. The reconstructed vector is the PCA vector, and the cosine similarities are identical. Observation 2 says that the angles keep information from the truncated dimensions, but that information cancels in the reconstruction. Observation 1 is also not correct, because the product of sines is the tail norm and its decay comes from the sorted variance of PCA. Equation 1 as written uses arctan of a ratio, which gives an angle with a positive cosine. For a negative xix_i, the round trip through Equations 1 and 2 does not return xix_i, so the lossless claim in Section 2 does not hold as written. If the code uses atan2 instead, the method is PCA. If the code uses arctan as written, the method is PCA with scrambled signs.

Narrow MTEB evaluation. The abstract says that the MTEB experiments are extensive. But the paper does not include the Classification and Clustering tasks. These tasks are a large part of MTEB. They are the primary application of embeddings. Datasets are limited in the experiments.

Questions for authors

Please check the weaknesses

Reviewer_vXDQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OySTER compresses embeddings by projecting them onto an SVD eigenbasis fitted on training embeddings, converting each projected vector into hyper-spherical angles, retaining the top-(k) angles, and computing cosine similarity on (k)-dimensional vectors reconstructed from those angles. It is evaluated on six MTEB task families with two 1024-dimensional models at 16×–128× compression against PCA, UMAP, and Matryoshka truncation, and claims Matryoshka-style behavior without MRL training.

Critical strengths

1 The pipeline is simple, training-free, and applicable to frozen embedders, which makes it directly useful for reducing storage and retrieval costs in existing systems. This is consequential because drop-in practicality is the paper's main appeal.

2 The evaluation spans six MTEB task families, two 1024-dimensional models, and four compression ratios, and it candidly reports cases where PCA or UMAP wins. This is consequential because the breadth and honesty of the evaluation make the negative results credible and provide a useful empirical reference for embedding compression.

Critical weaknesses

1 The hyper-spherical transform appears algebraically equivalent to direct top-(k) truncation of the normalized SVD projection: for a unit vector, the cascading (\sin/\cos) terms in Eq. 2 telescope, so reconstructing the first (k) coordinates from (\theta_1,\dots,\theta_k) recovers the exact prefix (x_1,\dots,x_k). If so, cosine similarity on the reconstructed vectors is identical to truncating the eigen-projected embedding, contradicting Observation 2. The decisive ablation—direct top-(k) truncation in the same SVD basis with matched normalization—is missing, so the reported differences from PCA may come from preprocessing, centering, or sign handling rather than the spherical transform. This is consequential because it removes the method's core novelty and undermines the comparison with PCA.

2 The empirical claims are not yet established: gains over PCA are small and inconsistent (PCA wins on Summarization at 8/16 dimensions and on several STS settings), baseline protocols are underspecified (centering, (\ell_2) normalization, fit split, UMAP at test time), several tables contain unexplained numerical discrepancies for identical settings, and no runtime/throughput/memory measurements support the efficiency claim. The Matryoshka-without-MRL claim is also untested because both backbones were already MRL-pretrained. These issues are consequential because the paper's practical case rests on efficiency and on outperforming PCA, and neither is currently established.

Questions for authors

1 Please provide a controlled ablation of OySTER vs. direct top-(k) truncation of the SVD-projected embeddings using the same basis, centering, and normalization. Since Eq. 2 telescopes back to (x_1,\dots,x_k) for a unit vector, what implementation step makes OySTER differ from truncated eigen-projection? Also state whether Eq. 1 uses (\mathrm{atan2})/(\arccos) or literal (\arctan), since literal (\arctan) cannot preserve quadrant signs. 2 What are the exact preprocessing and fitting protocols for OySTER, PCA, and UMAP—mean centering, (\ell_2) normalization before/after truncation, and the data split used to fit each transformation? Why do Snowflake scores at 64 dimensions differ between Table 1 and Table 2, and why do the NFCorpus/SprintDupQ rows in Table 6b match Snowflake numbers despite the Qwen3-Embedding-0.6B caption? 3 Can you provide measured encoding, reconstruction, and query latency/memory across dimensions, and results on a backbone not pre-trained with MRL?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.