acceptodds
Withdrawn from ICLR 2027

CSM-VL: Conditional Subspace Matryoshka for Vision-Language Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Matryoshka Representation Learning (MRL) enables a single encoder to support multiple dimensionality budgets through nested embedding prefixes. However, conventional MRL uses the same dimensional ordering for every input, limiting its ability to adapt to varying information needs in vision-language embeddings. We introduce CSM-VL (Conditional Subspace Matryoshka for Vision-Language Embeddings), which replaces the fixed hierarchy with an input-dependent ordering of learned functional subspaces while preserving nested representations. CSM-VL partitions the embedding space into functional dimension groups and uses a conditional router to determine their order for each multimodal input. We further introduce Utility-Guided Group Allocation, which measures each group's contribution to the multimodal contrastive objective, and Conditional Multimodal Interaction, which encourages subspaces to capture informative cross-modal interactions. This allows different inputs to prioritize different subspaces under the same dimensional budget. Experiments across multimodal embeddings benchmarks and multiple vision-language backbones show that CSM-VL consistently improves the accuracy-dimensionality trade-off, particularly under constrained embedding budgets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Conditional Computation, Adaptive Representation Learning, Matryoshka Representation Learning, Vision-Language Models, Subspace Learning, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nvYVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CSM-VL (Conditional Subspace Matryoshka for Vision-Language Embeddings). It extends Matryoshka Representation Learning with input-dependent ordering of learned dimension groups, preserving nested vision-language embeddings across dimensional budgets. Its two auxiliary objectives prioritize groups by their contribution to contrastive learning and reduce redundant cross-modal interactions. Experiments on 11 MMEB datasets across four vision-language backbones generally demonstrate improved accuracy–dimensionality trade-offs over MRL, ESE, and MIPIC, especially at compact budgets. Ablations support the complementary benefits of both objectives, with reported training overhead of approximately 5–14% over MRL.

Critical strengths
  1. A useful framework for adaptive multimodal embeddings. CSM-VL combines learned dimension groups, input-dependent ordering, and nested representations within a unified framework. This addresses a meaningful limitation of fixed-prefix embeddings: different multimodal inputs may require different information under the same dimensional budget.
  2. Encouraging empirical coverage. Evaluation across four backbones and 11 datasets generally shows improved performance at compact embedding budgets. The auxiliary-objective ablations provide additional support for the framework, suggesting potential benefits across multiple model architectures.
Critical weaknesses
  1. Insufficient differentiation from closely related work. DIME [1] and Learning to Select [2] already explore query-dependent dimension selection, while SMEC [3] introduces learnable selection into Matryoshka compression. Without discussing or comparing these approaches, the paper does not sufficiently establish the novelty of its specific combination of grouping, routing, and multimodal objectives.
  2. Limited evidence isolating the central contribution. The experiments support the complete framework, but do not clearly establish how much improvement comes from input-dependent allocation versus auxiliary training objectives. Stronger comparisons with simpler selection and ordering strategies would help demonstrate why the proposed mechanism is necessary.
  3. Practical efficiency benefits remain insufficiently demonstrated. The motivation emphasizes reduced storage and retrieval costs, whereas the main evidence concerns accuracy at different embedding dimensions. Evaluation of actual memory usage, inference latency, and large-scale retrieval would be needed to establish the practical advantages of conditional subspace selection.

[1] Guglielmo Faggioli, Nicola Ferro, Raffaele Perego, and Nicola Tonellotto. 2024. Dimension Importance Estimation for Dense Information Retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1318–1328. ACM.

[2] Zhanyu Wu, Richong Zhang, and Zhijie Nie. 2026. Learning to Select: Query-Aware Adaptive Dimension Selection for Dense Retrieval. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18666–18677. Association for Computational Linguistics.

[3] Biao Zhang, Lixin Chen, Tong Liu, and Bo Zheng. 2025. SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26209–26222. Association for Computational Linguistics.

Questions for authors

See weakness.

Reviewer_YPuEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CSM-VL, an input-adaptive extension of Matryoshka Representation Learning for compact vision-language embeddings. Instead of using a fixed dimensional ordering, the method partitions embedding dimensions into functional groups and employs a conditional router to prioritize different groups for different inputs while preserving nested representations. It further introduces Utility-Guided Group Allocation and Conditional Multimodal Interaction objectives to encourage informative and complementary subspace selection. The authors evaluate the method on classification and VQA tasks from the MMEB benchmark using four VLM backbones, comparing against MRL, ESE, and MIPIC. The results demonstrate performance improvements in many settings, particularly at smaller embedding dimensions, although the gains are not consistent across all models and tasks. Additional ablation studies and analyses investigate the contributions of the proposed objectives, routing behavior, and computational overhead.

Critical strengths
  1. A practically relevant research problem. The paper addresses the limitations of fixed dimensional ordering in Matryoshka representation learning. Improving the quality of compact multimodal embeddings is a meaningful problem with practical applications.

  2. Moderate methodological novelty. The paper extends conventional Matryoshka learning through input-dependent subspace allocation. The combination of functional grouping, conditional routing, and utility- and redundancy-aware objectives provides a reasonable methodological extension, although the underlying ideas are not new.

Critical weaknesses
  1. Serious concerns about the reliability of the experimental results. The experimental results contain numerous repetitions that raise serious concerns about their reliability. Several examples are particularly concerning:
  • Table 4 (Qwen3-VL-8B, VQA): MRL and ESE report exactly identical means and standard deviations across all seven embedding dimensions (e.g., 59.34±0.69 at 64 dimensions and 68.59±0.48 at 2048 dimensions). The same pattern appears throughout Table 17, where their results are identical for every dataset and dimension.

  • Table 13 (B3-Qwen2-2B, Classification): At 256 dimensions, CSM-VL reports exactly 78.62±0.16 on five different datasets: ImageNet-1K, VOC2007, N24News, SUN397, and Hateful-Memes. At 512 dimensions, all five datasets again report exactly 79.66±0.16.

  • Table 14 (B3-Qwen2-2B, VQA): At 256 dimensions, CSM-VL reports exactly 53.62±0.27 across all six datasets: OK-VQA, InfographicsVQA, A-OKVQA, ChartQA, DocVQA, and Visual7W. Similar repetitions occur at other dimensions.

  • Table 16 (Qwen3-VL-8B, Classification): At 512 dimensions, CSM-VL reports exactly 78.64±0.27 across all five classification datasets. The same pattern extends to multiple other dimensions and also appears in the MIPIC results.

  • Table 4 shows that CSM-VL consistently underperforms MRL on Qwen3-VL-8B VQA, contradicting the paper's claim of improved performance across settings.

These extensive and systematic anomalies are difficult to reconcile with independent experimental evaluations. They substantially undermine confidence in the reported results and, consequently, the validity of the paper's main empirical claims.

  1. Fundamental inconsistencies between the proposed method and its reported evaluation. The method explicitly partitions embeddings into eight equal-sized groups and constructs lower-dimensional representations by selecting complete groups. However, many evaluated dimensional budgets, such as 64 and 128 dimensions for a 2048-dimensional embedding, cannot be obtained under this formulation. More seriously, the Jaccard overlap reported in Table 7 is mathematically inconsistent with the dimensionality constraints. For FastVLM-0.5B, two 768-dimensional subsets of an 896-dimensional space must have a Jaccard overlap of at least 0.714, whereas the paper reports 0.48. These inconsistencies raise fundamental questions about whether the experiments were conducted according to the described methodology.
Questions for authors

Please clarify Weaknesses 1 and 2.

Reviewer_mNHQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CSM-VL studies variable-dimensional vision-language embeddings. It partitions coordinates into equally sized, disjoint functional groups and uses an input-conditioned router to order them into nested subspaces. Training combines contrastive learning across budgets, supervision from marginal contrastive utility, and a penalty on redundant interaction residuals. For each query, all candidates are compared using the same query-selected coordinates. Experiments cover classification and candidate-based VQA on four backbones, with analyses of the auxiliary losses, allocation diversity, and training overhead.

Critical strengths
  1. Conditional subspace selection is a worthwhile problem. The proposed approach lets each input determine the priority of functional groups, making subspace selection responsive to the input. Supervising the ordering through marginal improvements in task loss is also more closely tied to retrieval than activation magnitude alone.
  2. The evaluation considers both performance and training cost. Four backbones, multiple dimensional budgets, two task families, and auxiliary-loss ablations form a reasonably broad evaluation framework. The appendix also reports roughly 5%–14% additional training time. These are sensible evaluation choices, although the performance claims remain subject to the numerical concerns below.
Critical weaknesses
  1. The algorithm, dimensional budgets, and overlap statistics are not yet consistent. Section 3 and Appendix B.1 specify M=8M=8 equal-sized groups and budgets of dk=kD/8d_k=kD/8. For FastVLM, D=896D=896 implies 112 coordinates per group, so selecting whole groups cannot produce the reported 64, 128, 256, 512, or 768 dimensions. Table 7 also reports a group-set Jaccard overlap of 0.48 at 768 dimensions without explaining which groups realize that budget. If the statistic instead compares two sets of 768 coordinates, it must satisfy J≥(2×768−896)/896=5/7J\geq(2\times768-896)/896=5/7. Any within-group truncation or additional projection needs to be specified, with the statistic interpreted accordingly. As written, the main experiments cannot be reconstructed from the method description.
  2. The main results need to be checked, and the contribution of conditional ordering remains unresolved. In Table 13, CSM-VL has identical means and standard deviations across all five B3 classification datasets at every budget from 256 dimensions onward, including 78.62±0.1678.62\pm0.16 at 256 dimensions. In Table 4, MRL and ESE also have identical VQA entries at all seven budgets. CSM-VL is below MRL throughout that VQA table, including 66.29 versus 68.59 at full dimensionality. These may be table-construction errors, but the current entries do not establish consistent gains across backbones. Moreover, removing UG or CMI changes both representation and router learning; there is no fixed-order control on the same representations. Even after correcting the tables, the low-dimensional gains need to be separated into those due to conditional ordering and those due to better representations overall.
Questions for authors
  1. Budgets and statistics: Please give the actual coordinate construction for FastVLM at 64 and 768 dimensions, together with the set definition and a worked example for Table 7. How do these correspond to eight equal-sized groups?
  2. Result verification: Please check the repeated entries in Tables 4 and 13, provide the corresponding per-run results, and recompute the averages. Do the corrected results change the comparison with the strongest baseline for each backbone and task?
  3. Routing mechanism: Eqs. (8)–(9) compute scores once, so the relative probabilities of unselected groups do not change with the selected set; the marginal utilities in Eqs. (12)–(13) can change. How close is the actual router to selection that recomputes utility at each step? With the same groups and representations, does conditional ordering outperform a shared fixed order? These comparisons would clarify the link between the idealized greedy analysis and the implemented method.
  4. Reproducibility: How are the functional signatures, grouping objective, and balanced hard assignment implemented? Which states supply the three modality summaries in CMI, and is the shared projection trainable? These details are needed to reproduce the grouping and residual constraint.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.