- Rating
- 3
- Confidence
- 2(−2)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper finds the issue of uncalibrated similarity scores across embeddings of different dimensions in Matryoshka Representation Learning. The authors point out that traditional MRL lacks an explicit mechanism to align scores across different dimensions. To address this issue, the authors propose Semi-Continuous Representation Learning. By independently sampling different embedding dimensions and performing contrastive learning within a unified softmax, the method allows representations from different dimensions to compete directly, thereby improving the comparability of similarities across dimensions.
Critical strengths
-
This paper focuses on the comparability of scores when performing hybrid retrieval using embeddings of different dimensions. It is a scope that extends beyond the single-dimension retrieval performance typically addressed by traditional Matryoshka Representation Learning. This issue is grounded in a practical and well-justified application context.
-
SCRL achieves cross-dimensional interaction through mixed-width contrastive learning; it requires no complex additional modules and can be easily integrated into existing embedding training frameworks.
-
The author explains the differences between the two objective functions through offset invariance and Hessian analysis, and validates the approach using metrics such as mixed-size retrieval, score distribution overlap, and neighborhood consistency.
Critical weaknesses
-
Theorem 10 proves that with the encoder parameters held fixed, the additionally introduced free offset parameters possess a unique relative optimal solution under the SCRL objective and satisfy a calibration condition. But the practical implementations do not explicitly optimize these offset parameters. This theorem does not provide a direct guarantee that the resulting embeddings will satisfy the corresponding calibration condition, leaving a gap between the theoretical analysis and the actual algorithm.
-
The author defines calibration by requiring that the expected softmax probability mass aligns with the corresponding sampling probabilities. However, this condition does not guarantee that the similarity score distributions remain consistent across dimensions, nor does it directly ensure the quality of the ranking in multi-dimensional retrieval.
-
The authors did not fully disentangle the respective contributions of random-dimension sampling and shared-softmax optimization, and they lack controlled ablation studies targeting these core design elements.
Questions for authors
-
Theorem 10 considers the optimization of free offset parameters while keeping the encoder fixed. Could the authors provide further explanation on how this theoretical result generalizes to the actual training of encoder parameters? Furthermore, is it possible to prove that the embeddings obtained via SCRL training satisfy the defined calibration condition even in the absence of additional offset parameters?
-
Given that MRL combined with affine calibration can already achieve retrieval performance comparable to that of SCRL, could the authors elaborate on the specific advantages of SCRL over such post-hoc calibration? In particular, does SCRL offer stable and significant gains in terms of cross-dataset generalization, training or inference overhead, and practical deployment scenarios involving mixed-dimensionality embeddings?
-
Could the authors include additional controlled experiments to disentangle the individual contributions of random-dimension sampling and the mixed-width shared softmax? Furthermore, could results be provided across multiple independent training seeds? Additionally, it would be valuable to test whether SCRL remains effective in scenarios where the assignment of document dimensions is correlated with relevance.
3 comments
Thank authors for the detailed response and my concerns have been resolved. I find it is a really interesting topic. Though I am not an expert in this field, it is still a useful idea for me. I will maintain the score of acceptance at current stage.
Thank you, and additional results
We thank the reviewer for the reply and are glad that the response resolved their concerns. For completeness, we share two results completed since then, on cross-dataset generalization (Question 2) and relevance-correlated width assignment (Question 3). The first shows that SCRL generalizes to new domains better than MRL with post-hoc affine calibration.
Training on MS MARCO only. We fine-tuned BGE-base with MRL and SCRL on MS MARCO alone and evaluated on the eight BEIR test sets, none of which enters fine-tuning, checkpoint selection or calibration. The affine map is fitted on 50k MS MARCO dev pairs, i.e. in the training domain, as it would be in practice.
Mixed-width nDCG@10 (discrete / continuous protocol), mean over eight BEIR domains, ± over five width assignments:
| Method | Discrete | Continuous |
|---|---|---|
| MRL | 34.48 ± 0.17 | 46.15 ± 0.29 |
| MRL + affine | 37.59 ± 0.63 | 47.56 ± 0.19 |
| SCRL | 42.03 ± 0.31 | 50.22 ± 0.23 |
| SCRL + affine | 36.88 ± 0.70 | 46.58 ± 0.21 |
Trained on a single source domain, SCRL generalizes to unseen domains better than both MRL and MRL with post-hoc calibration, improving on MRL in all eight domains without any calibration step. SCRL improves on MRL by +7.55 / +4.06 and outperforms MRL + affine by +4.44 / +2.66. The affine map does not transfer reliably across domains. Its effect on MRL ranges from −11.74 to +22.35 points per domain (discrete), and applied to SCRL it lowers nDCG@10 in six of eight domains in both protocols. Fixed-width quality is preserved when trained on MS MARCO, with SCRL within 0.6 points of MRL at every width (+0.58 at 32d, −0.25 at 768d).
Relevance-correlated assignment (Question 3). With widths 64 and 768, SCRL improves on MRL under every assignment, by +6.88 nDCG@10 when relevant documents are kept at full width, +3.09 when width is independent of relevance, +1.37 when relevant documents are truncated, and +2.94 under a popularity-based assignment, averaged over all eight BEIR domains. The gain is largest when relevant documents are kept at full width, since this is where truncated non-relevant documents with inflated scores crowd them out. The margins are smaller than under random assignment because only two widths compete, rather than the full range down to 32. We trained with uniformly random width assignment as an unbiased baseline that assumes no particular deployment, but can be chosen to fit a given use case, e.g. by sampling widths according to document popularity, which represents a promising research direction for future work.
We will include these experiments in the appendix of the revised version.
We also noticed that the reviewer's confidence changed from 4 to 2 with this update. We greatly valued the reviewer's careful engagement with the theory, in particular the question on the trained encoder that led to Proposition 11. Was this change in confidence intended?
We believe we have now addressed all of the reviewer's questions and concerns. Are there any further follow-up questions about our work that we could answer? We hope these additional results further strengthen the submission, and would be grateful if the reviewer would consider them in their final assessment. We thank the reviewer again for their feedback, which has really helped us improve our submission and better position our work.
Response to Reviewer jM2Q
We thank the reviewer for the careful review, and in particular for the precise question about the trained encoder, which led to a new result.
Answer to Question 1 and Weakness 1: From Theorem 10 to the trained encoder. Theorem 10 (now Proposition 10) concerns a fixed encoder with explicit offsets, and the revised version now states this scope explicitly (L212–213, L283–285). To answer your question, we added Proposition 11 (Calibration at stationary points), which applies to SCRL as trained, without offsets. At any stationary point, the calibration gap ∣E[Pw]−π(w)∣ is zero if the encoder can shift the scores of width w relative to the others, and otherwise bounded by how well it can approximate such a shift, with errors weighted by softmax mass. So the answer is yes, under an explicit condition. The condition cannot be dropped, since an encoder that cannot move one width's scores relative to the others can be stationary at any gap. Empirically it appears to hold approximately: a fitted per-width affine correction adds about 2 points to SCRL, against 6 to 9 points for MRL (L385–388).
Answer to Weakness 2: Calibration does not guarantee matching distributions or ranking quality. Our calibration condition is the first-order optimality condition of the SCRL objective with respect to per-width offsets. Because the positive's width is also drawn from π, it states that the softmax neither over- nor under-selects candidates of any width, which is the failure that lets truncated documents crowd out relevant full-width ones. In a new remark after Definition 1, we now state that it does not imply matching score distributions, which would not be the right target since a smaller prefix carries less information about relevance, and that good rankings also require within-width quality. We therefore report distribution overlap, KL divergence, fixed-width retrieval and neighbourhood consistency as separate measures.
Answer to Weakness 3 and Question 3: Ablation, seeds and relevance-correlated assignment.
Ablation (L183–184). We ran the nested-dropout ablation with exactly the SCRL recipe and the same random width sampling, but with one width per pass for all candidates and a separate softmax, so widths never compete.
Mixed-width nDCG@10 (discrete / continuous protocol), mean over eight BEIR domains:
Random sampling alone gives no mixed-width gain on either backbone. With discrete π, its fixed-width quality stays within 0.6 points of MRL at every width on both backbones, so it is a healthy model. The gains therefore come from the shared softmax, as predicted by the offset invariance of the MRL objective.
Seeds. We trained MRL and SCRL with three training seeds each on both backbones. Across seeds, the mixed-width gain of SCRL over MRL is +8.47 ± 0.31 (discrete) and +5.54 ± 0.27 (continuous) nDCG@10 on BGE-base, and +11.83 ± 0.93 and +11.36 ± 1.01 on E5-base. SCRL improves on MRL in all eight domains for every seed, and nested dropout stays at the MRL level.
Relevance-correlated assignment. We are running two more realistic assignments and will report them as soon as they are available:
Answer to Question 2: Advantages over post-hoc calibration.
We hope that the new proposition, the clarified definition and the ablation address your concerns. We would be glad to discuss any remaining questions.