acceptodds
Under review as a conference paper at ICLR 2027

Semi-Continuous Representation Learning: Mixed-Width Contrastive Training for Calibrated Matryoshka Embeddings

Abstract

Matryoshka Representation Learning (MRL) has become a widely adopted approach for preserving semantic structure across embedding dimensionalities. Its premise is that each Matryoshka prefix is a coarser representation of the same input within one common embedding space. However, while all nested embeddings share the same space, their similarity scores are miscalibrated across prefix sizes. We demonstrate theoretically and experimentally that the miscalibration stems from MRL's loss formulation where each prefix is optimised under its own partition function. Therefore, we propose Semi-Continuous Representation Learning (SCRL), which replaces MRL's decomposed loss with a single contrastive objective over randomly sampled prefix sizes under a shared softmax. We prove that SCRL results in calibrated prefixes and show empirically that it ensures consistent similarity scores across nested dimensions, improving mixed-size retrieval performance by up to 8.8 nDCG@10 points on average across eight BEIR benchmarks. Furthermore, SCRL improves cross-size neighborhood consistency and increases the score distribution overlap between 64- and 768-dimensional embeddings from 71.7% to 84.9%. SCRL can be applied as a direct training replacement for MRL, as a post-training step for existing MRL embedding models, and in MRL-adaptor settings on top of any general-purpose embeddings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
matryoshka representation learning, contrastive learning, embedding calibration, text embeddings, adaptive embedding dimensionality, nested embeddings

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jM2QOpenReview ↗
Rating
3
Confidence
2(−2)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper finds the issue of uncalibrated similarity scores across embeddings of different dimensions in Matryoshka Representation Learning. The authors point out that traditional MRL lacks an explicit mechanism to align scores across different dimensions. To address this issue, the authors propose Semi-Continuous Representation Learning. By independently sampling different embedding dimensions and performing contrastive learning within a unified softmax, the method allows representations from different dimensions to compete directly, thereby improving the comparability of similarities across dimensions.

Critical strengths
  1. This paper focuses on the comparability of scores when performing hybrid retrieval using embeddings of different dimensions. It is a scope that extends beyond the single-dimension retrieval performance typically addressed by traditional Matryoshka Representation Learning. This issue is grounded in a practical and well-justified application context.

  2. SCRL achieves cross-dimensional interaction through mixed-width contrastive learning; it requires no complex additional modules and can be easily integrated into existing embedding training frameworks.

  3. The author explains the differences between the two objective functions through offset invariance and Hessian analysis, and validates the approach using metrics such as mixed-size retrieval, score distribution overlap, and neighborhood consistency.

Critical weaknesses
  1. Theorem 10 proves that with the encoder parameters held fixed, the additionally introduced free offset parameters possess a unique relative optimal solution under the SCRL objective and satisfy a calibration condition. But the practical implementations do not explicitly optimize these offset parameters. This theorem does not provide a direct guarantee that the resulting embeddings will satisfy the corresponding calibration condition, leaving a gap between the theoretical analysis and the actual algorithm.

  2. The author defines calibration by requiring that the expected softmax probability mass aligns with the corresponding sampling probabilities. However, this condition does not guarantee that the similarity score distributions remain consistent across dimensions, nor does it directly ensure the quality of the ranking in multi-dimensional retrieval.

  3. The authors did not fully disentangle the respective contributions of random-dimension sampling and shared-softmax optimization, and they lack controlled ablation studies targeting these core design elements.

Questions for authors
  1. Theorem 10 considers the optimization of free offset parameters while keeping the encoder fixed. Could the authors provide further explanation on how this theoretical result generalizes to the actual training of encoder parameters? Furthermore, is it possible to prove that the embeddings obtained via SCRL training satisfy the defined calibration condition even in the absence of additional offset parameters?

  2. Given that MRL combined with affine calibration can already achieve retrieval performance comparable to that of SCRL, could the authors elaborate on the specific advantages of SCRL over such post-hoc calibration? In particular, does SCRL offer stable and significant gains in terms of cross-dataset generalization, training or inference overhead, and practical deployment scenarios involving mixed-dimensionality embeddings?

  3. Could the authors include additional controlled experiments to disentangle the individual contributions of random-dimension sampling and the mixed-width shared softmax? Furthermore, could results be provided across multiple independent training seeds? Additionally, it would be valuable to test whether SCRL remains effective in scenarios where the assignment of document dimensions is correlated with relevance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jM2Q

We thank the reviewer for the careful review, and in particular for the precise question about the trained encoder, which led to a new result.

Answer to Question 1 and Weakness 1: From Theorem 10 to the trained encoder. Theorem 10 (now Proposition 10) concerns a fixed encoder with explicit offsets, and the revised version now states this scope explicitly (L212–213, L283–285). To answer your question, we added Proposition 11 (Calibration at stationary points), which applies to SCRL as trained, without offsets. At any stationary point, the calibration gap ∣E[Pw]−π(w)∣|\mathbb{E}[P_w] - \pi(w)| is zero if the encoder can shift the scores of width ww relative to the others, and otherwise bounded by how well it can approximate such a shift, with errors weighted by softmax mass. So the answer is yes, under an explicit condition. The condition cannot be dropped, since an encoder that cannot move one width's scores relative to the others can be stationary at any gap. Empirically it appears to hold approximately: a fitted per-width affine correction adds about 2 points to SCRL, against 6 to 9 points for MRL (L385–388).

Answer to Weakness 2: Calibration does not guarantee matching distributions or ranking quality. Our calibration condition is the first-order optimality condition of the SCRL objective with respect to per-width offsets. Because the positive's width is also drawn from π\pi, it states that the softmax neither over- nor under-selects candidates of any width, which is the failure that lets truncated documents crowd out relevant full-width ones. In a new remark after Definition 1, we now state that it does not imply matching score distributions, which would not be the right target since a smaller prefix carries less information about relevance, and that good rankings also require within-width quality. We therefore report distribution overlap, KL divergence, fixed-width retrieval and neighbourhood consistency as separate measures.

Answer to Weakness 3 and Question 3: Ablation, seeds and relevance-correlated assignment.

Ablation (L183–184). We ran the nested-dropout ablation with exactly the SCRL recipe and the same random width sampling, but with one width per pass for all candidates and a separate softmax, so widths never compete.

Mixed-width nDCG@10 (discrete / continuous protocol), mean over eight BEIR domains:

BackboneMRLNested dropoutSCRL
BGE-base36.74 / 46.2436.92 / 46.5145.58 / 52.00
E5-base28.51 / 36.6528.40 / 36.7839.19 / 48.28

Random sampling alone gives no mixed-width gain on either backbone. With discrete π, its fixed-width quality stays within 0.6 points of MRL at every width on both backbones, so it is a healthy model. The gains therefore come from the shared softmax, as predicted by the offset invariance of the MRL objective.

Seeds. We trained MRL and SCRL with three training seeds each on both backbones. Across seeds, the mixed-width gain of SCRL over MRL is +8.47 ± 0.31 (discrete) and +5.54 ± 0.27 (continuous) nDCG@10 on BGE-base, and +11.83 ± 0.93 and +11.36 ± 1.01 on E5-base. SCRL improves on MRL in all eight domains for every seed, and nested dropout stays at the MRL level.

Relevance-correlated assignment. We are running two more realistic assignments and will report them as soon as they are available:

  • Relevance-correlated: relevant documents are stored at 64 dimensions with probability p∈{0,0.5,1}p \in \lbrace 0, 0.5, 1 \rbrace and at 768 otherwise, while all other documents are split evenly. This covers the favourable case (all relevant documents at full width), a split independent of relevance, and the adversarial case (all relevant documents truncated).
  • Popularity-based: the 10% most-retrieved documents (by BM25) are kept at 768 dimensions and the rest at 64, matching the deployment example in L361–363.

Answer to Question 2: Advantages over post-hoc calibration.

AspectPost-hoc affine calibrationSCRL
Cross-dataset generalisationFitted on the dev sets of FiQA and NFCorpus, its gain over MRL ranges from +24.0 points on TREC-COVID to −5.5 on FEVER (Table 7)Improves over MRL in every domain
Training / inference overheadNeeds calibration data and a refit for every checkpoint, and possibly every domainSame training cost as MRL (one encoder pass per batch, L336–345), no calibration data, no inference overhead
Mixed-width deploymentCannot change which documents are retrieved at a given widthImproves cross-width neighbourhood consistency (top-1 agreement with full width at 32d from 46.1% for MRL to 54.3% for SCRL), which matters for cascaded retrieval and mixed-width indexes

We hope that the new proposition, the clarified definition and the ablation address your concerns. We would be glad to discuss any remaining questions.

Thank authors for the detailed response and my concerns have been resolved. I find it is a really interesting topic. Though I am not an expert in this field, it is still a useful idea for me. I will maintain the score of acceptance at current stage.

AuthorsOpenReview ↗

Thank you, and additional results

We thank the reviewer for the reply and are glad that the response resolved their concerns. For completeness, we share two results completed since then, on cross-dataset generalization (Question 2) and relevance-correlated width assignment (Question 3). The first shows that SCRL generalizes to new domains better than MRL with post-hoc affine calibration.

Training on MS MARCO only. We fine-tuned BGE-base with MRL and SCRL on MS MARCO alone and evaluated on the eight BEIR test sets, none of which enters fine-tuning, checkpoint selection or calibration. The affine map is fitted on 50k MS MARCO dev pairs, i.e. in the training domain, as it would be in practice.

Mixed-width nDCG@10 (discrete / continuous protocol), mean over eight BEIR domains, ± over five width assignments:

MethodDiscreteContinuous
MRL34.48 ± 0.1746.15 ± 0.29
MRL + affine37.59 ± 0.6347.56 ± 0.19
SCRL42.03 ± 0.3150.22 ± 0.23
SCRL + affine36.88 ± 0.7046.58 ± 0.21

Trained on a single source domain, SCRL generalizes to unseen domains better than both MRL and MRL with post-hoc calibration, improving on MRL in all eight domains without any calibration step. SCRL improves on MRL by +7.55 / +4.06 and outperforms MRL + affine by +4.44 / +2.66. The affine map does not transfer reliably across domains. Its effect on MRL ranges from −11.74 to +22.35 points per domain (discrete), and applied to SCRL it lowers nDCG@10 in six of eight domains in both protocols. Fixed-width quality is preserved when trained on MS MARCO, with SCRL within 0.6 points of MRL at every width (+0.58 at 32d, −0.25 at 768d).

Relevance-correlated assignment (Question 3). With widths 64 and 768, SCRL improves on MRL under every assignment, by +6.88 nDCG@10 when relevant documents are kept at full width, +3.09 when width is independent of relevance, +1.37 when relevant documents are truncated, and +2.94 under a popularity-based assignment, averaged over all eight BEIR domains. The gain is largest when relevant documents are kept at full width, since this is where truncated non-relevant documents with inflated scores crowd them out. The margins are smaller than under random assignment because only two widths compete, rather than the full range down to 32. We trained with uniformly random width assignment as an unbiased baseline that assumes no particular deployment, but π\pi can be chosen to fit a given use case, e.g. by sampling widths according to document popularity, which represents a promising research direction for future work.

We will include these experiments in the appendix of the revised version.

We also noticed that the reviewer's confidence changed from 4 to 2 with this update. We greatly valued the reviewer's careful engagement with the theory, in particular the question on the trained encoder that led to Proposition 11. Was this change in confidence intended?

We believe we have now addressed all of the reviewer's questions and concerns. Are there any further follow-up questions about our work that we could answer? We hope these additional results further strengthen the submission, and would be grateful if the reviewer would consider them in their final assessment. We thank the reviewer again for their feedback, which has really helped us improve our submission and better position our work.

Reviewer_fc9xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper observes that Matryoshka Representation Learning (MRL) leaves similarity scores uncalibrated across prefix sizes, because each size is trained with its own softmax, which is invariant to per-size score offsets.

It proposes Semi-Continuous Representation Learning (SCRL), which samples a prefix size independently for each candidate and puts all candidates in a single shared softmax, so scores at different sizes land on a common scale.

SCRL is then evaluated with BGE-base, E5-base and a Qwen3-0.6B adapter on eight BEIR datasets.

Critical strengths
  • The problem is important and has real consequences for retrieval systems: if scores from different prefix sizes aren’t comparable, any index mixing sizes ranks documents wrongly.
  • The method is novel and elegant. Mixing sizes inside one softmax is a simple, one-line change at the same training cost, and it fixes the issue directly in the embeddings.
Critical weaknesses
  1. The theory is overstated and should be toned down. Proposition 3 is the shift invariance of softmax, Theorem 9 is the standard softmax Hessian, and Theorem 10 is convexity of log-sum-exp in the offsets, all holding only for a fixed encoder with explicit offsets that SCRL never learns. These are useful remarks, not theorems.

  2. The key ablation is missing, and the authors acknowledge it (L183–184). SCRL combines random width sampling with mixing widths in one softmax, and without a homogeneous random-width baseline (one width per batch, i.e., nested dropout, ICML 2014) it’s impossible to tell which ingredient drives the gains. Comparisons with current alternatives are also missing: SMEC (EMNLP 2025), a matched Matryoshka-Adaptor (EMNLP 2024), and simple fixes such as MRL with a learnable per-width bias or temperature.

  3. The experiments are small-scale and not fully convincing. Only two ~110M models are fully trained, and Qwen3-0.6B gets just an adapter with no matched MRL baseline. In fixed-width retrieval the differences are within about one point and go both ways (e.g., −0.65 at 768d on BGE, −1.26 on E5), while the reported standard deviations cover only the random width assignment, not training seeds. These gains could fall within run-to-run variance.

  4. (Minor.) The calibration claim is weaker than presented. A per-width affine correction of MRL matches SCRL in mixed retrieval (45.84 vs 45.58) and beats it on E5 (40.85 vs 39.19). The same correction still adds +1.96 points to SCRL, so SCRL’s scores appear to be not fully calibrated. The paper’s notion of calibration (equal average softmax mass per size) also doesn’t imply matching score distributions, and this should be stated clearly.

Questions for authors

The problem is essentially compatible representation learning. A mixed-width index is a partially backfilled gallery, where items embedded by different models (here, prefix sizes) must produce comparable scores in one ranking (BCT, CVPR 2020; FastFill, ICLR 2023; λ-Orthogonality, NeurIPS 2025). How does SCRL relate to this literature?

Reviewer_qaiHOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Semi-Continuous Representation Learning (SCRL) to address similarity score miscalibration across different embedding dimensions in Matryoshka Representation Learning (MRL). Its main contribution is a shared contrastive objective that jointly optimizes embeddings of different prefix sizes, supported by theoretical analysis. Experiments on eight BEIR benchmarks and multiple embedding models show improvements of up to 8.8 nDCG@10 points in mixed-size retrieval, along with better score calibration and cross-size ranking consistency.

Critical strengths

One big strenth is the clear theoretical motivation identified by the paper. It identifies a meaningful limitation of standard MRL and provides a theoretical explanation for cross-dimensional score miscalibration. The proposed shared-softmax objective is simple and well motivated. It also shows strong empirical improvements of SCRL improves mixed-size retrieval while maintaining competitive fixed-size retrieval performance. The experiments also demonstrate improvements in embedding neighborhood consistency that cannot be achieved through score calibration alone.

Critical weaknesses

The biggest weakness of the paper is having a compounded effect in the experiments of the proposed mechanism and not showing a practical setting. The paper does not isolate the effect of random prefix sampling from that of the shared softmax. Additionally, mixed-size retrieval is evaluated using randomly assigned dimensions, which may not reflect realistic deployment scenarios.

Questions for authors

Could the authors provide an ablation that separates the contribution of random prefix sampling from the shared-softmax objective? This would help establish which component drives the improvements. Given that post-hoc affine calibration achieves comparable performance, what are the main practical advantages of SCRL over this simpler alternative?

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer fc9x (2/2)

Answer to Weakness 3: Scale and variance.

  • Training seeds. We trained MRL, SCRL and nested dropout with three training seeds each on BGE-base, and MRL and SCRL with the same seeds on E5-base. Across seeds, the mixed-width gain of SCRL over MRL is +8.47 ± 0.31 (discrete) and +5.54 ± 0.27 (continuous) nDCG@10 on BGE-base, and +11.83 ± 0.93 and +11.36 ± 1.01 on E5-base. SCRL improves on MRL in all eight domains for every seed, and nested dropout stays at the MRL level (37.59 ± 0.28 and 46.89 ± 0.24, against 37.44 ± 0.31 and 46.69 ± 0.31 for MRL).
  • Fixed-width differences. These differences are small and go both ways, and we revised L404–410 to present them as a trade-off controlled by π rather than a gain. The ablation supports that this trade-off is systematic: nested dropout matches MRL at every width (within 0.22 points on BGE), whereas SCRL gains 0.9 points at 32d and gives up 0.7 at 768d relative to it, so the shift comes from the shared softmax. On E5 the drop at the larger widths is somewhat bigger (−1.26 at 768d), but consistent with the same trade-off.
  • Model scale. To test scale, we are training MRL and SCRL on BGE-large-en-v1.5 (335M parameters, about three times BGE-base, native width 1024) with the same recipe, and will report the results as soon as they finish.
  • Adapter baseline. In the adapter setting, the frozen Qwen3-Embedding-0.6B backbone is itself MRL-trained, so the no-adapter baseline is an MRL model, and the matched Matryoshka-Adaptor described above will provide a trained adapter baseline on the same backbone.

Answer to Weakness 4: Calibration claim. SCRL's scores are not fully calibrated, and the revised text now says SCRL largely removes the width-dependent offset. On E5, MRL + affine (40.85) is above raw SCRL (39.19). SCRL's advantages lie elsewhere: no fitting step or calibration data, robustness across domains (the post-hoc fit's gain over MRL ranges from +24.0 points on TREC-COVID to −5.5 on FEVER, Table 7), and better geometry that no score transform can provide (top-1 agreement at 32d from 46.1% for MRL to 54.3% for SCRL).

We now also state explicitly, in a new remark after Definition 1 (L191–198), that mass calibration does not imply matching score distributions, and that matching them would not be the right target. A smaller prefix carries less information about relevance, so scores for which a given value implies the same probability of relevance at every width must be distributed differently across widths.

Answer to Question 1: Compatible representation learning. Thank you for this suggestion. We have added this line of work to the related work section to better contextualise SCRL. A mixed-width index is indeed a partially backfilled gallery whose "models" are prefixes of one encoder. Backward-compatible training aligns a new model to a fixed old one; SCRL instead trains all widths jointly in one shared softmax, so compatibility is symmetric, requires no separately trained models, and covers a continuum of widths. Our relevance-correlated assignment experiment, which we are evaluating now, includes a setting in which half of the corpus is stored at 64 and half at 768 dimensions independently of relevance. This mirrors a 50% backfill in which the order of re-embedding is unrelated to relevance, and we will report it as soon as possible.

We hope that the additional experiments and the revised theory strengthen the paper, and we would appreciate it if the reviewer would consider updating their assessment. We would be glad to clarify any remaining points and will share the results of the running experiments as soon as they are available.

AuthorsOpenReview ↗

Additional results: larger backbones, out-of-domain training and relevance-correlated assignment

We thank the reviewer again for the detailed review. Since our response we have completed the larger-backbone training (Weakness 3), an out-of-domain experiment that bears on the calibration claim (Weakness 4), and the relevance-correlated assignment, which includes the partial-backfill setting from the question on compatible representation learning. We have also updated our earlier response (2/2) with the results across three training seeds (Weakness 3).

Larger backbones. We trained MRL and SCRL on BGE-large-en-v1.5 and E5-large (335M parameters, native width 1024) with the same training setup as the base models and one training seed.

BackboneMRLSCRLGain
BGE-large36.48 / 45.0945.59 / 52.08+9.11 / +6.99
E5-large36.79 / 45.5045.67 / 52.59+8.88 / +7.09

The mixed-width gains are as large as on the base models. At fixed widths, SCRL gains 1.26 (BGE) and 1.60 (E5) points at 32d and gives up about 1.1 points at 1024d, the same trade-off as on the base models.

Training on MS MARCO only. We fine-tuned BGE-base with MRL and SCRL on MS MARCO alone and evaluated on the eight BEIR test sets, none of which enters fine-tuning, checkpoint selection or calibration. The affine map is fitted on 50k MS MARCO dev pairs, i.e. in the training domain, as it would be in practice.

Mixed-width nDCG@10 (discrete / continuous protocol), mean over eight BEIR domains, ± over five width assignments:

MethodDiscreteContinuous
MRL34.48 ± 0.1746.15 ± 0.29
MRL + affine37.59 ± 0.6347.56 ± 0.19
SCRL42.03 ± 0.3150.22 ± 0.23
SCRL + affine36.88 ± 0.7046.58 ± 0.21

Trained on a single source domain, SCRL generalizes to unseen domains better than both MRL and MRL with post-hoc calibration, improving on MRL in all eight domains without any calibration step. SCRL improves on MRL by +7.55 / +4.06 and outperforms MRL + affine by +4.44 / +2.66. The affine map does not transfer reliably across domains. Its effect on MRL ranges from −11.74 to +22.35 points per domain (discrete), and applied to SCRL it lowers nDCG@10 in six of eight domains in both protocols. Fixed-width quality is preserved when trained on MS MARCO, with SCRL within 0.6 points of MRL at every width (+0.58 at 32d, −0.25 at 768d).

Relevance-correlated assignment. With widths 64 and 768, SCRL improves on MRL under every assignment, by +6.88 nDCG@10 when relevant documents are kept at full width, +3.09 when width is independent of relevance, +1.37 when relevant documents are truncated, and +2.94 under a popularity-based assignment, averaged over all eight BEIR domains. The gain is largest when relevant documents are kept at full width, since this is where truncated non-relevant documents with inflated scores crowd them out. The setting where width is independent of relevance corresponds to a 50% backfill in which the order of re-embedding is unrelated to relevance. The margins are smaller than under random assignment because only two widths compete, rather than the full range down to 32. We trained with uniformly random width assignment as an unbiased baseline that assumes no particular deployment, but π\pi can be chosen to fit a given use case, e.g. by sampling widths according to document popularity, which represents a promising research direction for future work.

We will include these experiments in the appendix of the revised version. The matched Matryoshka-Adaptor on Qwen3-Embedding-0.6B is still training, and we will report it as soon as it is available.

We hope that these results help address the reviewer's concerns, and would appreciate it if the reviewer would consider raising their score. We are grateful to the reviewer for raising the quality bar, which has helped us improve our paper substantially.

Reviewer_fc9xOpenReview ↗

Official Comment by Reviewer fc9x

Thank you for the detailed response and the additional experiments. The nested-dropout ablation, the results across training seeds and larger backbones, and the out-of-domain comparison with affine calibration address my main concerns, and the revised theory and calibration claims are now presented at the right level. I also appreciate the link to compatible representation learning.

I am raising my score to 3.

AuthorsOpenReview ↗

Thank you to the reviewer

We thank the reviewer for engaging with our response and for raising their score. Their feedback on the scope of the theory, the ablation and the link to compatible representation learning has improved the submission. All additional results are included in the revised version, including a matched MRL adapter on Qwen3-Embedding-0.6B, which gives no mixed-width gain over the frozen backbone, while the SCRL adapter improves on it by +5.58 / +8.49 nDCG@10 (Appendix F).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.