- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Matryoshka Representation Learning (MRL) learns flexible-size representations via nested prefix optimization but lacks constraints for high-quality compact encoding. To solve this problem, the authors introduce SpecMRL, which augments MRL with a parameter-based spectral basis ordered by feature importance and carries out prefix readout in the learned spectral space. Experiments on MTEB show SpecMRL beats original MRL at all tested compressed dimensions, with bigger gains under heavier compression. It even exceeds full-dimensional MRL using only half the embedding size.
Critical strengths
- As a drop-in modification to MRL, SpecMRL leverages learnable spectral bases to sort dimensions by information importance and adopts efficient, stable approximate solvers, achieving better performance under embedding compression.
- Experiments are conducted on multiple backbones including Qwen and Llama, and consistent results demonstrate that SpecMRL outperforms vanilla MRL across all compressed dimensions. The performance gain becomes more pronounced under stronger compression, verifying that the improvements are not incidental to a single model.
Critical weaknesses
- The experiments are mainly conducted on relatively small models, which limits the generalizability of the observed improvements to larger-scale LLMs.
- The inherent performance drop under full-dimensional embedding, as its spectral basis optimized for low-dimensional compression damages the isotropy of the full representation space
Questions for authors
- Can the advantages of SpecMRL be maintained on larger model scales such as 7B or 13B models?
- How does the SpecMRL method perform on downstream tasks?
- Will the learned spectral basis change significantly with shifts in the training data distribution?
- Is the final performance sensitive to hyperparameters?
5 comments
Thank you for the additional experiments which partially address my concerns. My remaining concern is how reliably those gains extend to embeddings near full dimensionality. Full-dimensional results vary across the reported settings, and removing the last 5% of spectral directions mitigates the drop where it occurs. Considering the concern above, I am inclined to maintain my original score.
Rebuttal by Authors
I appreciate the reviewers' quick responses. However, I would like to question the reviewers' concern about whether "the gains reliably extend to embeddings near full dimensionality."
I would like to emphasize the following two points, and hope that the reviewers and AC will take them into consideration.
(1) First, the performance degradation at full dimensionality is very marginal. As shown in our multi-seed experiments (see our Response to Reviewer Y3wb), this degradation is not observed in many more runs.
(2) Second, we have already identified the cause of this performance drop. We further demonstrate that removing only 5% of the spectral edge dimensions is sufficient to recover the performance. In representation learning, achieving stronger performance with a smaller representation space is essentially a free lunch, and can even be considered an advantage. I believe that researchers with a basic understanding of this field would recognize this.
While I respect the reviewers' opinions, I still have reservations about this criticism. I will further discuss this point with the AC. Thank you.
Rebuttal by Authors
We thank the reviewer for the thoughtful feedback and hope our responses and additional results can address the concerns. If so, we would be grateful if you could reconsider your overall recommendation.
W1/Q1: Scaling to larger model.
We thank the reviewer for this comment. First, we want to clarify that our original submission already includes scalability results on Qwen3-4B, which is a moderately sized model for practical embedding applications.
During the rebuttal phase, we conduct additional experiments on Qwen3-8B-Base to further address the concern. The results on the MTEB subset are reported below.
These results provide evidence that the advantages of SpecMRL can be maintained on larger backbone model.
In practice, embedding models often need to balance model capacity with inference efficiency, making relatively smaller models more suitable for many deployment scenarios. Accordingly, our main experiments focus on compact models to reflect these practical considerations.
Overall, the consistent gains from 0.5B to 8B demonstrate the generalizability of SpecMRL across a broad range of commonly used embedding model scales.
W2: Full-dimensional performance and isotropy.
This is an insightful question, and we have already discussed this issue in our original submission. Specifically, performance can be recovered by removing only the last 5% of directions, which exhibit a disproportionate concentration of energy.
We suspect that this performance drop is associated with the introduction of trainable linear matrices Wproj, rather than evidence of actual representation degradation. [1] also reports similar observation in its analysis of unembedding matrix, a large-scale linear parameter, suggesting that this is a general side effect of large trainable linear rather than an issue specific to SpecMRL. As shown above, this effect can be effectively mitigated by removing only the last 5% of directions. This simple mitigation suggests that the observed performance drop does not reflect a fundamental disruption of isotropy across the entire representation space.
Q2: Downstream task performance.
Our evaluation covers 56 datasets across seven downstream task categories in MTEB. We wonder if this question may stem from a misunderstanding of our evaluation scope. We will improve our presentation to further clarify the evaluation scenarios in the revised manuscript.
Q3: Sensitivity to training data distribution.
We thank the reviewer for raising this question. We use the publicly released training data from [2] to retrain both MRL and SpecMRL.
Following Appendix D, we find that the leading 512-dimensional subspaces learned from the two distributions have 71.42% overlap, substantially above the 50% random reference, indicating considerable shared structure despite distributional differences.
We also conducted MTEB evaluation on these models. The results demonstrate the robustness of SpecMRL to changes in training data distribution.
Q4: Hyperparameter sensitivity.
The original submission already includes hyperparameter sensitivity experiments for MRL. In the revised manuscript, we extend this analysis in Section 4.5 to fewer supervised dimensions. SpecMRL maintains its advantage under these alternative MRL hyperparameter.
As for the training hyperparameters, we further evaluated both methods with a learning rate of 2×10−5 and a temperature of 0.02, keeping other settings unchanged. SpecMRL retains its advantage across all compressed dimensions:
[1] Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings
[2] Repetition Improves Language Model Embeddings