acceptodds
Under review as a conference paper at ICLR 2027

SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

Abstract

Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, MRL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GFkeOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Matryoshka Representation Learning (MRL) learns flexible-size representations via nested prefix optimization but lacks constraints for high-quality compact encoding. To solve this problem, the authors introduce SpecMRL, which augments MRL with a parameter-based spectral basis ordered by feature importance and carries out prefix readout in the learned spectral space. Experiments on MTEB show SpecMRL beats original MRL at all tested compressed dimensions, with bigger gains under heavier compression. It even exceeds full-dimensional MRL using only half the embedding size.

Critical strengths
  1. As a drop-in modification to MRL, SpecMRL leverages learnable spectral bases to sort dimensions by information importance and adopts efficient, stable approximate solvers, achieving better performance under embedding compression.
  2. Experiments are conducted on multiple backbones including Qwen and Llama, and consistent results demonstrate that SpecMRL outperforms vanilla MRL across all compressed dimensions. The performance gain becomes more pronounced under stronger compression, verifying that the improvements are not incidental to a single model.
Critical weaknesses
  1. The experiments are mainly conducted on relatively small models, which limits the generalizability of the observed improvements to larger-scale LLMs.
  2. The inherent performance drop under full-dimensional embedding, as its spectral basis optimized for low-dimensional compression damages the isotropy of the full representation space
Questions for authors
  1. Can the advantages of SpecMRL be maintained on larger model scales such as 7B or 13B models?
  2. How does the SpecMRL method perform on downstream tasks?
  3. Will the learned spectral basis change significantly with shifts in the training data distribution?
  4. Is the final performance sensitive to hyperparameters?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the thoughtful feedback and hope our responses and additional results can address the concerns. If so, we would be grateful if you could reconsider your overall recommendation.

W1/Q1: Scaling to larger model.

We thank the reviewer for this comment. First, we want to clarify that our original submission already includes scalability results on Qwen3-4B, which is a moderately sized model for practical embedding applications.

During the rebuttal phase, we conduct additional experiments on Qwen3-8B-Base to further address the concern. The results on the MTEB subset are reported below.

Method409620481024512256
MRL62.6062.3761.9961.3760.00
SpecMRL62.9864.0363.8563.5162.90
Δ+0.38+1.65+1.86+2.14+2.90

These results provide evidence that the advantages of SpecMRL can be maintained on larger backbone model.

In practice, embedding models often need to balance model capacity with inference efficiency, making relatively smaller models more suitable for many deployment scenarios. Accordingly, our main experiments focus on compact models to reflect these practical considerations.

Overall, the consistent gains from 0.5B to 8B demonstrate the generalizability of SpecMRL across a broad range of commonly used embedding model scales.

W2: Full-dimensional performance and isotropy.

This is an insightful question, and we have already discussed this issue in our original submission. Specifically, performance can be recovered by removing only the last 5% of directions, which exhibit a disproportionate concentration of energy.

Full spectrumRemove last 5%
Qwen3-0.6B-0.03+0.41
Qwen2-0.5B-0.22+0.20
Llama3.2-1B-0.05+0.44

We suspect that this performance drop is associated with the introduction of trainable linear matrices WprojW_{\mathrm{proj}}, rather than evidence of actual representation degradation. [1] also reports similar observation in its analysis of unembedding matrix, a large-scale linear parameter, suggesting that this is a general side effect of large trainable linear rather than an issue specific to SpecMRL. As shown above, this effect can be effectively mitigated by removing only the last 5% of directions. This simple mitigation suggests that the observed performance drop does not reflect a fundamental disruption of isotropy across the entire representation space.

Q2: Downstream task performance.

Our evaluation covers 56 datasets across seven downstream task categories in MTEB. We wonder if this question may stem from a misunderstanding of our evaluation scope. We will improve our presentation to further clarify the evaluation scenarios in the revised manuscript.

Q3: Sensitivity to training data distribution.

We thank the reviewer for raising this question. We use the publicly released training data from [2] to retrain both MRL and SpecMRL.

Following Appendix D, we find that the leading 512-dimensional subspaces learned from the two distributions have 71.42% overlap, substantially above the 50% random reference, indicating considerable shared structure despite distributional differences.

We also conducted MTEB evaluation on these models. The results demonstrate the robustness of SpecMRL to changes in training data distribution.

Method102451225612864
MRL52.0851.5250.2948.3245.34
SpecMRL52.3052.4251.8750.8848.64
Δ+0.22+0.90+1.58+2.56+3.30

Q4: Hyperparameter sensitivity.

The original submission already includes hyperparameter sensitivity experiments for MRL. In the revised manuscript, we extend this analysis in Section 4.5 to fewer supervised dimensions. SpecMRL maintains its advantage under these alternative MRL hyperparameter.

As for the training hyperparameters, we further evaluated both methods with a learning rate of 2×10−52\times10^{-5} and a temperature of 0.02, keeping other settings unchanged. SpecMRL retains its advantage across all compressed dimensions:

Method102451225612864
MRL60.0859.2457.3655.1050.40
SpecMRL59.8661.4461.0360.3658.54
Δ-0.22+2.20+3.67+5.26+8.14

[1] Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings

[2] Repetition Improves Language Model Embeddings

Thank you for the additional experiments which partially address my concerns. My remaining concern is how reliably those gains extend to embeddings near full dimensionality. Full-dimensional results vary across the reported settings, and removing the last 5% of spectral directions mitigates the drop where it occurs. Considering the concern above, I am inclined to maintain my original score.

AuthorsOpenReview ↗

Rebuttal by Authors

I appreciate the reviewers' quick responses. However, I would like to question the reviewers' concern about whether "the gains reliably extend to embeddings near full dimensionality."

I would like to emphasize the following two points, and hope that the reviewers and AC will take them into consideration.

(1) First, the performance degradation at full dimensionality is very marginal. As shown in our multi-seed experiments (see our Response to Reviewer Y3wb), this degradation is not observed in many more runs.

(2) Second, we have already identified the cause of this performance drop. We further demonstrate that removing only 5% of the spectral edge dimensions is sufficient to recover the performance. In representation learning, achieving stronger performance with a smaller representation space is essentially a free lunch, and can even be considered an advantage. I believe that researchers with a basic understanding of this field would recognize this.

While I respect the reviewers' opinions, I still have reservations about this criticism. I will further discuss this point with the AC. Thank you.

Reviewer_Y3wbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpecMRL, which improves Matryoshka Representation Learning by learning an ordered spectral basis for embedding compression. The method uses eigenvectors derived from a learnable parameter matrix and applies the original MRL objective to the resulting representations. The authors also introduce an efficient basis update using power iteration. Experiments on MTEB show consistent improvements over MRL, particularly at smaller embedding dimensions.

Critical strengths
  1. The idea is simple and well motivated. Instead of introducing additional losses, the authors improve how the embedding space is organized for prefix truncation.
  2. The comparisons with learned linear and orthogonal readouts help support the role of spectral ordering. The additional experiments with a larger backbone and longer training also strengthen the results.
Critical weaknesses
  1. The experimental results are based on a single training seed, without reporting variance across runs. Some improvements, particularly at half the embedding dimension, are relatively small. Additional runs would help establish how stable these gains are.
  2. The training cost of the additional parameter matrix is not fully evaluated. With R = 128D, the method introduces a substantial number of trainable parameters. Although the reported throughput overhead is small, GPU memory usage is not discussed. Table 3 also suggests diminishing returns as R increases, raising questions about the choice of such a large parameterization.
Questions for authors
  1. How stable are the improvements across different training seeds, especially at half the embedding dimension? Reporting the mean and standard deviation over several runs would help assess the reliability of these results.
  2. Could the authors provide the training memory overhead compared with MRL? Given the results in Table 3, is R = 128D necessary, or could a smaller parameter matrix offer a better trade-off?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for raising these questions and hope the additional results can help address your concerns.

W1 & Q1: Stability across training seeds.

We thank the reviewer for raising this question. To assess the variability across training seeds, we evaluated MRL and SpecMRL across five independent runs. The mean and standard deviation, together with the paired improvements on MTEB, are reported below.

Method102451225612864
MRL60.89 ± 0.1360.42 ± 0.1559.37 ± 0.1457.79 ± 0.1354.74 ± 0.16
SpecMRL61.04 ± 0.1761.73 ± 0.1661.42 ± 0.1960.59 ± 0.2058.60 ± 0.27
Δ+0.15 ± 0.20+1.31 ± 0.19+2.04 ± 0.17+2.81 ± 0.17+3.86 ± 0.13

SpecMRL shows consistent improvements across runs with different training seeds, with a notable gain of 1.31 points even at half dimension.

We thank the reviewer for the helpful suggestion and have included these additional results in the revised manuscript.

W2/Q2: Training memory overhead and the choice of RR.

1. Training Memory Overhead.

On Qwen3-0.6B, the peak allocated GPU memory is 48.93 GiB for MRL and 50.18 GiB for SpecMRL, resulting in an additional 1.25 GiB overhead. This represents a modest memory overhead of only 2.6%.

Furthermore, this overhead can be further reduced with a more memory-efficient implementation. During the rebuttal phase, we implement a sharding design that reduces the overhead to 0.25 GiB on Qwen3-0.6B, accounting for just 0.5% overhead. On Qwen3-4B, the overhead is also substantially reduced from 9.39 GiB to 1.59 GiB.

Specifically, the strategy partitions WprojW_{\mathrm{proj}} across PP GPUs:

G=\sum_{i=1}^{P}W_iW_i^\top.$$ By distributing the shards across GPUs, the additional memory overhead per GPU is reduced from $O(DR)$ to $O(DR/P)$. We have included this implementation in our codebase and thank the reviewer for raising this question. **2. Choice of $R$.** As stated in the original submission, we choose $R=128D$ as a practical default, as performance consistently improves with increasing $R$ but begins to converge around $128D$. As shown in Table 3, SpecMRL consistently achieves gains across different choices of $R$, **indicating that its benefits are relatively insensitive to the exact value of $R$.** **We also clarify that the associated overhead is manageable.** As discussed above, the additional training memory overhead is not substantial. The training-time evaluation in Section 4.1.3 also shows only modest overhead. Moreover, since the derived basis $U$ is precomputed after training, $R$ introduces no additional inference overhead. Therefore, we recommend a larger $R$ when training resources permit, while smaller $R$ remains a viable choice under resource constraints.

Thank you for the response and additional experiments. The new results address my concerns about training stability and memory overhead. I will keep my original score.

AuthorsOpenReview ↗

Reply Rebuttal Comment by Authors

We are glad that the additional experiments have addressed your concerns. Thank you for your time and helpful feedback, which have been valuable in strengthening our work.

Reviewer_LBe8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper extends traditional Matrioshka Representation Learning by adding a trainable unitary as additional, last layer. Results are experimentally evaluated using small (~1B) open-weights models, on MTEB tasks.

Critical strengths

Empirical results show improvements over standard MRL.

Critical weaknesses

The novelty of the approach is limited: incorporating orthogonal linearities is well known to improve e.g. optimization of deep networks. The authors do not provide in depth analysis of factors behind the improvements that would show the gains go beyond prior observations.

Questions for authors

Adding trainable Wproj increases the number of parameters, how much does this explain improved model performance?

How much does orthogonality matter? E.g. will using XU as in eq. 5, but without constraining U to be orthogonal, or only initializing U to be orthogonal, lead to inferior results compared to SpecMRL?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the suggestions and feedback. We hope that our clarifications and additional analyses address the concerns, and would appreciate it if the reviewer could reconsider the score.

W1: Clarification on the Benefits of Orthogonal Linearity.

We appreciate the reviewer’s comment and would like to clarify that the concern about "orthogonal linearity" may stem from a misunderstanding of MRL. In fact, orthogonal linearity is inherent to MRL's readout mechanism, rather than an additional constraint introduced by our method.

Specifically, the readout projects embeddings onto an orthogonal basis and uses the resulting prefixes as low-dimensional representations. Vanilla MRL simply fixes this orthogonal basis to the identity matrix.

In this paper, we consider this fixed orthogonal linear transformation to be suboptimal for nested prefix readout. We replace it with a learnable, spectrally ordered orthogonal basis, aiming to better organize informative directions into earlier prefixes.

In the original submission, we have shown that a learnable orthogonal linear transformation alone brings limited benefits. Specifically, we introduced MRL w. Orth, which replaces the spectral basis with an arbitrary learnable orthogonal basis. As shown in Table 5, it underperforms our method. Furthermore, we investigated the choice of spectral basis (parameter-derived vs. feature-derived) and the effects of additional parameters and their scale in Sections 4.1 and 4.2, respectively.

These results further demonstrate that the gains of SpecMRL are not simply due to introducing additional parameters or a learnable orthogonal transformation, but also depend on the construction of the spectral basis.

Overall, the gains of SpecMRL stem from the carefully constructed spectral basis, rather than simply introducing a learnable orthogonal linear transformation. We hope this addresses the reviewer’s concern.

Q1: Effect of additional parameters.

Thank you for the question. In the original submission, we have already included this analysis to test whether learnable orthogonal parameters or additional parameters alone explain the gains.

Specifically, MRL w. Gram directly uses the same matrix to project backbone outputs, without spectral decomposition, while MRL w. Orth replaces it with QR decomposition to enforce orthogonality. Despite equal parameter counts, neither variant matches SpecMRL's low-dimensional performance. These comparisons support the contribution of the spectral readout beyond the additional parameter.

Methoddim=512dim=64
MRL60.5454.73
MRL w. Gram60.8054.11
MRL w. Orth60.9157.19
SpecMRL61.6158.53

Q2: Role of orthogonality.

As clarified in W1, orthogonality is inherent to MRL. The key addition in SpecMRL is a learnable spectral readout derived from parameter. To further examine its role, we also evaluated an unconstrained linear readout, which yields limited gains. Specifically, MRL w. Linear jointly learns an unconstrained linear readout and the backbone, but still underperforms SpecMRL, especially under stronger compression.

Methoddim=512dim=64
MRL60.5454.73
MRL w. Linear60.7257.03
SpecMRL61.6158.53
AuthorsOpenReview ↗

Summary of Responses and Clarifications

Dear Reviewer LBe8,

Thank you for your time and consideration. As the discussion period is approaching its close, we briefly summarize our clarifications and would appreciate any further feedback.


Clarification on the Role of Orthogonality in SpecMRL

We would like to clarify that "incorporating orthogonal linearities" is a misunderstanding, which is inherent to MRL's readout mechanism rather than a newly introduced constraint. In this paper, SpecMRL makes two key innovations: exploiting the ordered spectral structure and making readout trainable. If the reviewer concerns on the gains merely from a trainable orthogonal transformation, we have also conducted rigorous experiments to isolate its effect.

We want to highlight that the advantage of SpecMRL stems primarily from spectral ranking, rather than orthogonality itself. As a natural comparison, we have already conducted experiments comparing SpecMRL against a trainable orthogonal readout in the original submission. These results further confirm that the gains cannot be explained by the trainability of the readout alone.

To the best of our knowledge, few works have explored training the readout while retaining the MRL objective. Moreover, the spectral ranking structure highlighted by SpecMRL also received limited attention in related fields. While our implementation also differs from existing orthogonal optimizers or other orthogonal transformation designs, we respectfully consider it debatable to claim a lack of novelty based solely on the superficial overlap in the term "orthogonal linearity".

Therefore, we kindly ask the reviewer to reconsider the lack-of-novelty assessment and we will also bring this clarification to the AC's attention.


Thank you for your time and thoughtful consideration of our submission.

Best regards,

Authors of Submission 43298

Thank you for the clarification about what the authors consider the key innovations. I will keep my score; I don't see the manuscript as meeting the novelty and significance threshold for acceptance. As the rebuttal highlights, in the hard, low d=64 case where the difference between MRL and SpecMRL is highest, vanilla orthogonal or even unconstrained linear readout covers 60% or more of the improvement SpecMRL has over MRL.

Thank you for the follow-up. After we clarified the original comment, we appreciate the reviewer for further examining the comparison with alternative experiments.

As mentioned in our previous response, considering the orthogonal readout to be trainable and jointly optimizing it with backbone is also proposed in our work. By considering the vanila MRL's fixed readout suboptimal, we would like the gains from the Orth varient to further support our attempt to draw attention to the design principles of MRL.

Moreover, introducing spectral structure pushes the performance frontier with fewer dimensions. To complement our extensive ablations, we also include an unconstrained linear baseline. Across experiments, SpecMRL shows consistent advantages at reduced dimensions and achieves a higher performance frontier. As shown in Table 5, it surpasses full-dimensional MRL by 0.64 points using only half the dimensions, making dimensionality compression almost a free lunch.

We hope these clarifications help convey the contribution of SpecMRL, and would appreciate the reviewer taking it into consideration when reassessing our work.

Comments on the paper

2 comments

Dear reviewers,

This is a friendly reminder to read the authors’ rebuttals and post your responses by Nov 18. Please briefly indicate which concerns have been addressed and which remain unresolved. Even if your overall assessment remains unchanged, please provide a brief response to the authors. Please also update your reviews and scores as appropriate. Thank you for your time and effort!

AC.

AuthorsOpenReview ↗

Kind Request for Further Discussion

Dear Reviewers LBe8 and Y3wb,

Thank you for your time and effort during the rebuttal phase. As the discussion period is set to close in 48 hours, we would appreciate it if you could let us know whether our rebuttal has addressed your concerns.

If there are any remaining concerns or additional feedback you would like us to consider, please do not hesitate to let us know.

Sincerely,

Authors of Submission 43298

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.