acceptodds
Under review as a conference paper at ICLR 2027

ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compression

Abstract

Multimodal large language models (MLLMs) have shown strong potential for universal multimodal representation learning. However, existing methods either compress each input into a single vector, limiting fine-grained expressiveness, or retain long sequences of visual-token vectors, incurring substantial storage and interaction costs. To resolve this trade-off, we propose ResComEmb, a trainable framework for effective and efficient universal multi-vector multimodal embedding. ResComEmb first encodes each input at native dynamic resolution into ordered global, intermediate, and fine-grained views. After MLLM contextualization and embedding projection, a trainable Residual Homogeneity Compression (RHC) module reduces within-granularity redundancy and cross-granularity repetition under explicit visual token budgets. Then, ResComEmb introduces a length-adaptive Bidirectional Late-Interaction Matching mechanism for robust query–document scoring, which averages the strongest token-level matches in each direction and combines the two scores using a weight based on how many valid tokens each side has. Extensive experiments on MMEB, ViDoRe V1, and ViDoRe V2 show that ResComEmb produces higher-quality universal multimodal embeddings than VLM2Vec-V2, and outperforms ColQwen2.5 in visual document retrieval using only 37.5% of its full visual token budget, demonstrating a favorable effectiveness–efficiency trade-off. The code is available at: https://anonymous.4open.science/r/ResComEmb-D8B4.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal embedding, multimodal retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MX4mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ResComEmb addresses the trade-off between embedding quality and storage cost in universal multimodal embeddings. It encodes global, intermediate-scale, and fine-grained image views, then compresses contextualized embeddings using learned importance signals and cross-stage novelty signals. Nested supervision supports coarse-to-fine prefix representations, while bidirectional Top-K mean matching scores query-candidate pairs. With 384 visual vectors, the paper reports Precision@1 of 67.4 on MMEB and NDCG@5 of 90.4/61.6 on ViDoRe V1/V2, respectively (Tables 1-3). These results demonstrate useful compact representations, but the broader efficiency claims and comparative conclusions require clarification.

Critical strengths
  1. Useful compact representations. The reported improvements span MMEB and both versions of ViDoRe. Using only a prefix comprising its first 128 vectors, the same trained model achieves 66.7 on MMEB and 60.8 on ViDoRe V2. With 384 vectors, the corresponding results are 67.4 and 61.6. This provides practical evidence that the model can accommodate different storage budgets.

  2. Support for the proposed mechanism. Controlled ablations of importance and novelty, together with comparisons of nested supervision, support the design. Removing novelty reduces the ViDoRe V2 result from 61.6 to 58.1. Together with the gradient analysis under fixed assignments, this evidence supports the trainable compressor as a credible contribution beyond simply selecting fewer tokens.

Critical weaknesses
  1. The incremental contribution has not been sufficiently isolated and validated. MURE already combines multi-resolution encoding, resolution-level Matryoshka learning, and post-hoc clustering [1]. MetaEmbed learns nested multi-vector representations [2]. ToMe and Light-ColPali already support using merged or compressed embeddings during training [3,4], while MRL predates these methods [5]. The learned retention priorities and residual novelty in Residual Homogeneity Compression (RHC) may constitute a valid distinction. However, Tables 1-3 do not isolate these factors through comparisons with compression baselines under matched conditions. MURE uses PaliGemma, MetaEmbed appears only in the ViDoRe V2 comparison, and Light-ColPali is discussed but not evaluated. This limits attribution of the gains, but does not establish that the method lacks novelty.

  2. Efficiency and robustness are not sufficiently validated. RHC operates after multi-view MLLM contextualization, so fewer output vectors do not establish lower encoding cost. The paper reports token budgets but provides no measured encoding or scoring latency. The 37.5% figure implies a baseline of 1,024 vectors, whereas the public ColQwen2.5-v0.2 model card specifies a maximum of 768 image patches. Different evaluation configurations may explain this discrepancy. Appendix A reports using random seed 42 but provides no variability estimates. This limits confidence in gains of only 0.5-1.0 points over the strongest ViDoRe baselines.

[1] Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval. ICMR 2026.

[2] MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction. ICLR 2026.

[3] Token Merging: Your ViT But Faster. ICLR 2023.

[4] Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings. ACL 2025.

[5] Matryoshka Representation Learning. NeurIPS 2022.

Questions for authors
  1. Could you clarify the training-data mixture and the retrieval budget used for MetaEmbed? A targeted comparison against similarity-only merging or a Light-ColPali-style compressor would help isolate the contribution of RHC. This comparison should use the same backbone, data, scoring rule, and output budget. Why is MetaEmbed absent from the MMEB comparison?

  2. Which ColQwen2.5 checkpoint and image-resolution settings were used to obtain the denominator in the 37.5% claim? Please report actual token-count distributions, embedding dimensionality, numerical precision, and encoding and scoring times under a unified setting, including the additional overhead of RHC.

5 comments
AuthorsOpenReview ↗

Response to Reviewer MX4m

Thank you for your positive feedback. We address each concern with the updated analyses below.

W1: Contribution isolation

Thank you for encouraging us to clarify RHC's contribution. We provide our replies in "General Response - R1: Attribution of the performance gains and the independent contribution of RHC".

Q1.2: Comparison with existing compression methods

Thank you for your helpful suggestions. We provide our replies in "General Response - R2: RHC beyond existing compression methods".

W2.1 & Q2: Efficiency, token-budget correction, and unified measurements

Thank you for requesting unified storage and latency measurements. We provide our replies in "General Response - R3: End-to-end efficiency, storage, and retrieval latency analysis"

W2.2: Random seeds and statistical stability

Thank you for paying attention to details. ResComEmb is trained for 30,000 steps with a global batch size of 64 on eight NVIDIA A100 GPUs. Each training run takes approximately two weeks (2,688 GPU-hours). Evaluation across three training seeds would require two additional runs, totaling 5,376 GPU-hours, or four weeks on the same hardware, which exceeds our available computational budget. We therefore use a fixed random seed (42) consistently across all experiments. While a fixed-seed evaluation does not quantify run-to-run variability, the results across ViDoRe V1, ViDoRe V2, and MMEB, together with controlled ablations, provide broad empirical support for ResComEmb’s effectiveness and consistent evidence for the contributions of its individual components.

Q1.1 & Q1.3: MetaEmbed training-data mixture and retrieval budget (Table 1) and absent from MMEB

Thank you for paying attention to the implementation details. We would clarify that:

Training Data Mixture. Both MetaEmbed and our main model are trained on MMEB-train and ViDoRe-train, use the same subset-sampling and interleaving strategy, and include one explicit hard negative from MoCa (Chen et al., 2025a) per query.

Retrieval Budget (Table 1). MetaEmbed uses fixed budgets of 16 on the query side and 64 on the target side. For ResComEmb, any input containing an image uses a budget of 384, regardless of whether it appears on the query or target side. Table 1 summarizes these retrieval budgets and embedding dimensions. MetaEmbed's 2,048-dimensional embedding is much larger than ResComEmb's 128-dimensional projection.

Table 1: Query- and target-side retrieval budgets and embedding dimensions. For ResComEmb, 384 applies whenever the corresponding input contains an image.

ModelQuery-side budgetTarget-side budgetEmbedding dim.
MetaEmbed16642048
ResComEmb384384128

We have added the comparison between MetaEmbed and our ResComEmb on all three benchmarks in Table 2 below.

Highlight the storage–effectiveness trade-off. MetaEmbed has a much higher dimension (i.e., 2048) of each visual token compared to our ResComEmb method (i.e., 128). At the same numerical precision, the visual embedding storage is proportional to the number of vectors multiplied by their dimension:

64×2048384×128=2.67\frac{64\times2048}{384\times128}=2.67

Thus, under the reported retrieval budgets, ResComEmb uses an estimated 2.67× less visual embedding storage.

Performance Comparison of MetaEmbed and ResComEmb. Table 2 shows that our ResComEmb outperforms MetaEmbed on both V1 and V2 by 1.7 points and 1.3 points, respectively, and demonstrates competitive performance on MMEB.

Table 2: Performance Comparison of MetaEmbed and ResComEmb on Multimodal Embedding and Visual Document Retrieval Benchmarks.

ModelMMEBViDoRe V1ViDoRe V2
MetaEmbed69.188.760.3
ResComEmb67.490.461.6

Dear Reviewer MX4m

We sincerely appreciate your time and thoughtful feedback on our work.

We would greatly appreciate any further feedback on our responses and would be happy to address any remaining questions or concerns. If you feel that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider revisiting your evaluation.

Thank you again for your time and consideration.

Best regards,

The Authors

Area_Chair_MspVOpenReview ↗

please examine rebuttal response

Hello Reviewer MX4m,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal, comment, and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Reviewer_pKbiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper combines multi-granularity visual encoding with a compression module that merges tokens based on similarity, importance, and novelty relative to coarser representations. It also uses nested representation learning and bidirectional matching. The method is evaluated on MMEB and ViDoRe V1/V2.

Critical strengths

The paper addresses a practical problem: reducing the storage and matching costs of multi-vector representations while retaining useful visual details. The evaluation covers diverse tasks, and the experimental results show that the method has certain advantages.

Critical weaknesses
  1. The contribution beyond existing compression methods needs to be better established.

The framework shares several components with prior work, such as multi-resolution encoding and nested training in MURE[1], similarity-based merging in ToMe[2], and importance-guided compression in AGC[3]. The distinctive part of RHC is its use of coarse-grained representations to estimate novelty when compressing finer-grained tokens. Therefore the current evidence does not clearly establish its advantage over existing compression strategies.

  1. The experimental comparisons do not sufficiently establish the advantage of the proposed method.

The MMEB benchmark omits MetaEmbed [4], which reports 69.1 with the same Qwen2.5-VL-3B backbone, compared with 67.4 for the proposed ResComEmb.

Considering the main design is the compression, the paper also lacks controlled comparisons between other compression methods. Removing importance or novelty shows that these components help within RHC, but does not show that RHC is better than existing alternatives. The authors should consider comparisons with MURE, ToMe, and AGC. The current MURE comparison uses a different backbone, making it difficult to attribute the improvement to compression.

[1] MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval. ICMR 2026

[2] Token Merging: Your ViT But Faster. ICLR 2023

[3] Multi-Vector Index Compression in Any Modality. SIGIR 2026

[4] MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction. ICLR 2026

Questions for authors

The current experiments leave some uncertainty about what RHC offers over existing compression methods. More comparisons would help clarify its contribution and could change my assessment:

Can the authors compare RHC with other compression methods, such as MURE, ToMe, and AGC under the same setting? Detailed analysis and results on both retrieval accuracy and efficiency would help establish its advantages.

5 comments
AuthorsOpenReview ↗

Response to Reviewer pKbi

Thank you for your careful assessment of the contribution and experimental comparisons and we address your concerns one by one below.

W1 & Q1: Contribution Analysis of RHC beyond existing compression methods

Thank you for your helpful suggestions. We provide our replies in "General Response - R2: RHC beyond existing compression methods".

W2: Missing MetaEmbed comparison on MMEB

Thank you for paying attention to the details. We have added the comparison between MetaEmbed and our ResComEmb on all three benchmarks in Table 1 below, with our main model following MetaEmbed's MMEB-train/ViDoRe-train mixture and sampling and interleaving strategy .

Highlight the storage–effectiveness trade-off. MetaEmbed has a much higher dimension (i.e., 2048) of each visual token compared to our ResComEmb method (i.e., 128). At the same numerical precision, the visual embedding storage is proportional to the number of vectors multiplied by their dimension:

64×2048384×128=2.67\frac{64\times2048}{384\times128}=2.67

Thus, under the reported retrieval budgets, ResComEmb uses an estimated 2.67× less visual embedding storage.

Performance Comparison of MetaEmbed and ResComEmb. Table 1 shows that our ResComEmb outperforms MetaEmbed on both V1 and V2 by 1.7 points and 1.3 points, respectively, and demonstrates competitive performance on MMEB while using much less 128-dimensional projections compared with MetaEmbed's 2,048 dimensions.

Table 1: Performance Comparison of MetaEmbed and ResComEmb on Multimodal Embedding and Visual Document Retrieval Benchmarks.

ModelMMEBViDoRe V1ViDoRe V2
MetaEmbed69.188.760.3
ResComEmb67.490.461.6

Dear Reviewer pKbi

We sincerely appreciate your time and thoughtful feedback on our work.

We would greatly appreciate any further feedback on our responses and would be happy to address any remaining questions or concerns. If you feel that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider revisiting your evaluation.

Thank you again for your time and consideration.

Best regards,

The Authors

Area_Chair_MspVOpenReview ↗

please examine rebuttal response

Hello Reviewer pKbi,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal, comment, and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Reviewer_E8w4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ResComEmb is a multi-vector multimodal embedding model that is initialized from base ColQwen2.5. Each image is subjected to 3 levels of encoding at hierarchical detail levels. RHC is applied next to compress the embeddings. RHC ranks tokens by a learned importance score plus a novelty term that measures how different a token is from the coarser stages. On top of that, a length-adaptive bidirectional top-K scorer replaces MaxSim. The numbers look promising 67.4 P@1 on MMEB and 90.4/61.6 NDCG@5 on ViDoRe V1/V2 at just 384 visual tokens. Some concerns about how RHC compares to simpler compression under the same setup neeeds addressing.

Critical strengths
  • This is a very well motivated paper, compressing of multimodal multi-vector embeddings greatly increases the feasibility of using these techniques in a production setting.
  • Each fine token's novelty is Ni=MinMax(1−max⁡a∈Akcos⁡(hi,a))N_i = \mathrm{MinMax}\big(1 - \max_{a \in \mathcal{A}_k} \cos(h_i, a)\big), where Ak\mathcal{A}_k represents the compressed outputs of the coarser stages. Number points towards, Novelty mattering more as removing novelty drops V1/V2 by 0.8%/5.7%. Removing importance drops V1/V2 by 0.3%/3.4%. On V2 the biggest impacts are on ESG Syn-Mul (~14%) and Eco Mul (~9%).
  • The 128/256/384-token prefixes all come from one model. V1 goes 89.2→89.6→90.489.2 \rightarrow 89.6 \rightarrow 90.4 and MMEB goes 66.7→66.9→67.466.7 \rightarrow 66.9 \rightarrow 67.4. The 128-token prefix (89.2/60.8) is close to a separately trained 192-token model, so seems like nesting costs very little. Looks like MRL is contributing significaly as without it MMEB drops by 4.2–4.5% and V2 drops by 9.3–12.5% at every prefix. MRL + RHC let one model serve multiple storage budgets which would be important for adoption.
Critical weaknesses
  • Main concern is that the gains over ColQwen2.5 mix together different training data, multi-view encoding, MRL, the new scorer and RHC. Without MRL the full 384-token model gets 89.1/55.9 on ViDoRe, which is below ColQwen2.5 . Switching to vanilla MaxSim drops MMEB by 34%, and it is not clear if that variant was retrained. So it is hard to tell how much RHC itself is adding.

  • The 37.5% efficiency claim only counts output visual tokens. It is hard to understand how ResCombEmb is cheaper than ColQwen2.5 as RHC runs after the backbone has already processed all 3 views (7 crops). Text tokens are also kept uncompressed, and the top-K bidirectional scorer is not the same cost as MaxSim.

Questions for authors
  • Why is RHC not compared against a simple baseline like token pooling or clustering at the same budget, or a separately trained 128-token model for the smallest prefix ?
  • Can you provide total e2e encoding latency, bytes per document and retrieval latency against ColQwen2.5 on the same hardware?
5 comments
AuthorsOpenReview ↗

Response to Reviewer E8w4

Thank you for your positive and constructive feedback. We address your concerns one by one below.

W1: Attribution of the performance gains and the independent contribution of RHC

Thank you for your suggestion. We provide our replies in "General Response - R1: Attribution of the performance gains and the independent contribution of RHC".

W2 & Q2: End-to-end efficiency, storage, and retrieval latency

Thank you for your query. We provide our replies in "General Response - R3: End-to-end efficiency, storage, and retrieval latency analysis".

Q1: Same-budget simple compression baselines

Thank you for your helpful suggestions. We provide our replies in "General Response - R2: RHC beyond existing compression methods".

Dear Reviewer E8w4

We sincerely appreciate your time and thoughtful feedback on our work.

We would greatly appreciate any further feedback on our responses and would be happy to address any remaining questions or concerns. If you feel that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider revisiting your evaluation.

Thank you again for your time and consideration.

Best regards,

The Authors

Area_Chair_MspVOpenReview ↗

please examine rebuttal response

Hello Reviewer E8w4,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal, comment, and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We appreciate the reviewers’ constructive and valuable feedback. We provide our replies to common questions below.

R1: Attribution of the performance gains and the independent contribution of RHC

We disentangle the effects of multi-granularity encoding, MRL, RHC, and the late-interaction scorer and provide the analysis results.

1. MRL ablation (Table 1). Both variants use the same 384-token RHC configuration, multi-granularity encoding, RHC module, scorer, and matched training settings, and are separately trained. Table 1 demonstrates the importance of MRL. Removing MRL reduces NDCG@5 by 1.3 points on ViDoRe V1 and 5.7 points on ViDoRe V2, and Precision@1 by 3.0 points on MMEB. These consistent gains indicate that MRL improves the learned representation quality in addition to enabling flexible retrieval across token budgets.

Table 1: Effect of MRL at the 384-token budget. ViDoRe reports NDCG@5 (%), and MMEB reports Precision@1 (%).

VariantViDoRe V1ViDoRe V2MMEB
With MRL90.461.667.4
Without MRL89.155.964.4

2. RHC signal ablations (Table 2). The w/o Importance and w/o Novelty variants are separately retrained with the same other configurations as the complete model. Table 2 demonstrates that both Importance and Novelty contribute to ResComEmb’s retrieval effectiveness. Removing Importance reduces NDCG@5 by 0.3 points on ViDoRe V1 and 2.1 points on V2, while removing Novelty leads to larger drops of 0.7 and 3.5 points, respectively. The full model performs best on both benchmarks, supporting the joint use of these signals, with Novelty making a particularly strong contribution on V2.

Table 2: Ablation of RHC scoring signals on ViDoRe V1 and V2 (NDCG@5 %).

VariantViDoRe V1ViDoRe V2
Full ResComEmb90.461.6
Without Importance90.159.5
Without Novelty89.758.1

3. Late-interaction scorer ablations (Table 3). The full model is trained with the proposed bidirectional scorer. The w/o Adaptive Weighting and w/o Bidirectionality variants are separately retrained under their respective scoring configurations. Vanilla MaxSim, w/o Mean, and w/o TopK reuse the trained full model and change only the scoring function at inference, without retraining. Table 3 shows that all alternative variants underperform our proposed ResComEmb method, demonstrating the effectiveness of its late-interaction mechanism.

Table 3: Late-interaction ablations by benchmark. MMEB reports Precision@1 (%), and ViDoRe reports NDCG@5 (%).

Scorer variantMMEBViDoRe V1ViDoRe V2
ResComEmb (Full bidirectional scorer)67.490.461.6
Vanilla MaxSim44.590.059.8
Without mean aggregation58.190.059.1
Without Top-K63.889.154.5
Without adaptive weighting66.989.054.0
Without bidirectionality65.490.059.7
AuthorsOpenReview ↗

General Response

R2: RHC beyond existing compression methods

Compared to existing compression methods, we clarified the distinction of RHC and added controlled compression evidence.

Comparison of RHC with existing compression methods. MURE uses multi-resolution encoding and post-hoc non-trainable clustering; ToMe merges tokens by similarity; AGC learns importance scores to select representative tokens. Our RHC is a novel trainable multi-granularity token compression method by combining retrieval-supervised importance with cross-granularity novelty to prioritize details not yet covered by coarse representations.

Retrieval effectiveness and document-encoding latency comparison. We further report the retrieval effectiveness and document-encoding latency of our RHC and other compression methods, including HAC in the Qwen2.5-VL implementation of MURE, at 384 visual tokens and 128 dimensions under the official ViDoRe training-data setting, with all trainable methods retrained on ViDoRe only. The results are shown in Table 4. ResComEmb achieves the highest NDCG@5 on both benchmarks, exceeding K-Means by 1.5 points on V1 and 2.6 points on V2 at comparable encoding latency. Its gains over AGC and ToMe come with higher offline encoding cost, showing an effectiveness–encoding-cost trade-off.

Table 4: Retrieval effectiveness and document-encoding latency of compression methods.

MethodV1 NDCG@5V2 NDCG@5Document-encoding Latency (ms/page)
1D pooling85.352.1885.2
K-Means89.259.01028.2
HAC88.756.21962.8
ToMe86.856.7291.0
AGC87.456.8167.1
ResComEmb90.761.61035.8

R3: End-to-end efficiency, storage, and retrieval latency analysis

We measure all costs on one NVIDIA A100 80GB GPU using 128-dimensional embeddings and BF16 storage. The baseline is ColQwen2.5-v0.2 with its official 768-token visual cap and standard MaxSim. Its measured average output is 744 visual vectors per page.

Table 5 reports the V2 retrieval results together with the corresponding storage, offline encoding, and online retrieval costs. The 128-token prefix reduces visual embedding storage by 82.8% while exceeding ColQwen2.5-v0.2 by 0.2 NDCG@5 points on V2. At 384 tokens, the gain increases to 1.0 point while visual embedding storage remains 48.4% lower. Online retrieval takes 18.1--18.3 ms/query, compared with 18.0 ms/query for the baseline. Offline document encoding is more expensive, mainly because of the seven sampled views, and is performed during index construction. These results demonstrate improved retrieval effectiveness with substantially lower visual embedding storage and comparable online latency.

Table 5: Retrieval effectiveness and efficiency across visual-token budgets. Storage refers to visual embeddings. ViDoRe V2 reports NDCG@5 (%).

SettingVisual tokens/pageStorage (KiB/page)Offline encoding (ms/page)Online encoding (ms/query)Scoring (ms/query)Total (ms/query)V2 NDCG@5
ColQwen2.5-v0.2744186.0155.716.11.218.060.6
ResComEmb, 128-token prefix12832.01035.816.41.118.160.8
ResComEmb, 256-token prefix25664.01035.816.41.118.261.4
ResComEmb, 384-token prefix38496.01035.816.41.218.361.6

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.