- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ResComEmb addresses the trade-off between embedding quality and storage cost in universal multimodal embeddings. It encodes global, intermediate-scale, and fine-grained image views, then compresses contextualized embeddings using learned importance signals and cross-stage novelty signals. Nested supervision supports coarse-to-fine prefix representations, while bidirectional Top-K mean matching scores query-candidate pairs. With 384 visual vectors, the paper reports Precision@1 of 67.4 on MMEB and NDCG@5 of 90.4/61.6 on ViDoRe V1/V2, respectively (Tables 1-3). These results demonstrate useful compact representations, but the broader efficiency claims and comparative conclusions require clarification.
Critical strengths
-
Useful compact representations. The reported improvements span MMEB and both versions of ViDoRe. Using only a prefix comprising its first 128 vectors, the same trained model achieves 66.7 on MMEB and 60.8 on ViDoRe V2. With 384 vectors, the corresponding results are 67.4 and 61.6. This provides practical evidence that the model can accommodate different storage budgets.
-
Support for the proposed mechanism. Controlled ablations of importance and novelty, together with comparisons of nested supervision, support the design. Removing novelty reduces the ViDoRe V2 result from 61.6 to 58.1. Together with the gradient analysis under fixed assignments, this evidence supports the trainable compressor as a credible contribution beyond simply selecting fewer tokens.
Critical weaknesses
-
The incremental contribution has not been sufficiently isolated and validated. MURE already combines multi-resolution encoding, resolution-level Matryoshka learning, and post-hoc clustering [1]. MetaEmbed learns nested multi-vector representations [2]. ToMe and Light-ColPali already support using merged or compressed embeddings during training [3,4], while MRL predates these methods [5]. The learned retention priorities and residual novelty in Residual Homogeneity Compression (RHC) may constitute a valid distinction. However, Tables 1-3 do not isolate these factors through comparisons with compression baselines under matched conditions. MURE uses PaliGemma, MetaEmbed appears only in the ViDoRe V2 comparison, and Light-ColPali is discussed but not evaluated. This limits attribution of the gains, but does not establish that the method lacks novelty.
-
Efficiency and robustness are not sufficiently validated. RHC operates after multi-view MLLM contextualization, so fewer output vectors do not establish lower encoding cost. The paper reports token budgets but provides no measured encoding or scoring latency. The 37.5% figure implies a baseline of 1,024 vectors, whereas the public ColQwen2.5-v0.2 model card specifies a maximum of 768 image patches. Different evaluation configurations may explain this discrepancy. Appendix A reports using random seed 42 but provides no variability estimates. This limits confidence in gains of only 0.5-1.0 points over the strongest ViDoRe baselines.
[1] Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval. ICMR 2026.
[2] MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction. ICLR 2026.
[3] Token Merging: Your ViT But Faster. ICLR 2023.
[4] Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings. ACL 2025.
[5] Matryoshka Representation Learning. NeurIPS 2022.
Questions for authors
-
Could you clarify the training-data mixture and the retrieval budget used for MetaEmbed? A targeted comparison against similarity-only merging or a Light-ColPali-style compressor would help isolate the contribution of RHC. This comparison should use the same backbone, data, scoring rule, and output budget. Why is MetaEmbed absent from the MMEB comparison?
-
Which ColQwen2.5 checkpoint and image-resolution settings were used to obtain the denominator in the 37.5% claim? Please report actual token-count distributions, embedding dimensionality, numerical precision, and encoding and scoring times under a unified setting, including the additional overhead of RHC.
5 comments
Dear Reviewer MX4m
We sincerely appreciate your time and thoughtful feedback on our work.
We would greatly appreciate any further feedback on our responses and would be happy to address any remaining questions or concerns. If you feel that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider revisiting your evaluation.
Thank you again for your time and consideration.
Best regards,
The Authors
please examine rebuttal response
Hello Reviewer MX4m,
As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal, comment, and decide if it changes your thoughts regarding your current evaluation.
Thanks.
Response to Reviewer MX4m
Thank you for your positive feedback. We address each concern with the updated analyses below.
Thank you for encouraging us to clarify RHC's contribution. We provide our replies in "General Response - R1: Attribution of the performance gains and the independent contribution of RHC".
Thank you for your helpful suggestions. We provide our replies in "General Response - R2: RHC beyond existing compression methods".
Thank you for requesting unified storage and latency measurements. We provide our replies in "General Response - R3: End-to-end efficiency, storage, and retrieval latency analysis"
Thank you for paying attention to details. ResComEmb is trained for 30,000 steps with a global batch size of 64 on eight NVIDIA A100 GPUs. Each training run takes approximately two weeks (2,688 GPU-hours). Evaluation across three training seeds would require two additional runs, totaling 5,376 GPU-hours, or four weeks on the same hardware, which exceeds our available computational budget. We therefore use a fixed random seed (42) consistently across all experiments. While a fixed-seed evaluation does not quantify run-to-run variability, the results across ViDoRe V1, ViDoRe V2, and MMEB, together with controlled ablations, provide broad empirical support for ResComEmb’s effectiveness and consistent evidence for the contributions of its individual components.
Thank you for paying attention to the implementation details. We would clarify that:
Training Data Mixture. Both MetaEmbed and our main model are trained on MMEB-train and ViDoRe-train, use the same subset-sampling and interleaving strategy, and include one explicit hard negative from MoCa (Chen et al., 2025a) per query.
Retrieval Budget (Table 1). MetaEmbed uses fixed budgets of 16 on the query side and 64 on the target side. For ResComEmb, any input containing an image uses a budget of 384, regardless of whether it appears on the query or target side. Table 1 summarizes these retrieval budgets and embedding dimensions. MetaEmbed's 2,048-dimensional embedding is much larger than ResComEmb's 128-dimensional projection.
Table 1: Query- and target-side retrieval budgets and embedding dimensions. For ResComEmb, 384 applies whenever the corresponding input contains an image.
We have added the comparison between MetaEmbed and our ResComEmb on all three benchmarks in Table 2 below.
Highlight the storage–effectiveness trade-off. MetaEmbed has a much higher dimension (i.e., 2048) of each visual token compared to our ResComEmb method (i.e., 128). At the same numerical precision, the visual embedding storage is proportional to the number of vectors multiplied by their dimension:
384×12864×2048=2.67
Thus, under the reported retrieval budgets, ResComEmb uses an estimated 2.67× less visual embedding storage.
Performance Comparison of MetaEmbed and ResComEmb. Table 2 shows that our ResComEmb outperforms MetaEmbed on both V1 and V2 by 1.7 points and 1.3 points, respectively, and demonstrates competitive performance on MMEB.
Table 2: Performance Comparison of MetaEmbed and ResComEmb on Multimodal Embedding and Visual Document Retrieval Benchmarks.