acceptodds
Withdrawn from ICLR 2027

Look Across Layers, Learn Through Choices: Enhancing Multimodal Retrieval with Unified Representation and Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models have recently made remarkable progress in embedding-based retrieval tasks. However, research on the joint optimization of multimodal representation and generation within a single architecture remains underexplored. Existing unified approaches exhibit a trade-off: they tend to substantially compromise representational capacity to preserve generative performance, or fail to establish a synergistic alignment between the two capabilities. To address these limitations, we propose CLUE, a cross-layer unified representation and generation framework designed for holistic multimodal retrieval and comprehension. Specifically, CLUE introduces a Cross-Layer Embedding mechanism that integrates hierarchical semantic representations across Transformer layers, alleviating the information bottleneck of single-layer pooling. Furthermore, CLUE incorporates an Embedding Candidate Selection paradigm that leverages multimodal embeddings for candidate-level relevance discrimination through generative selection, establishing a more direct interaction between representation learning and generation. Extensive experiments on multimodal retrieval and comprehension benchmarks demonstrate that CLUE achieves state-of-the-art performance among the compared methods, substantially outperforming existing unified models on MMEB while effectively retaining robust generative capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal retrieval, representation learning, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wKPKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes two main components, Cross-Layer Embedding (CLE) and Embedding Candidate Selection (ECS), within the CLUE framework for unified multimodal retrieval and generation. CLE uses learnable queries to aggregate last-token representations across Transformer layers, rather than relying only on the final layer. ECS uses generative candidate selection to distinguish a positive match from hard negatives using both original multimodal inputs and projected embedding tokens. With joint contrastive and language-modeling training, CLUE achieves competitive retrieval performance on MMEB datasets while largely retaining the original backbones capabilities on multimodal comprehension benchmarks.

Critical strengths
  • CLUE achieves strong retrieval performance while largely retaining the original backbones' generation capabilities. Maintaining both capabilities in one model is a useful result.

  • The training approach is relatively straightforward. The ablation studies are well organized and show how the proposed components affect performance.

Critical weaknesses
  • Much of ECS's core idea is already present in ReMatch, which feeds projected embeddings back into the MLLM and trains them through generative matching. Moving from pairwise judgments to joint candidate selection is a reasonable extension, but the reported gain is only 0.4 points.

  • Although CLE differs in implementation, the broader idea of combining representations across layers was already established in ELMo (Peters et al., 2018). The paper does not discuss this connection or clearly explain what is conceptually new in CLE beyond adapting this established idea

  • The claimed retrieval and generation trade-off is not directly demonstrated against CAFe and CREM. Generation performance is compared only with the original backbones. This shows that CLUE retains generation capabilities, but not that it preserves them better than existing unified methods.

  • The same-backbone comparison with CREM does not show how much each proposed component adds. Table 3 uses Qwen3-VL-2B, whereas the comparison with CREM uses Qwen2-VL-2B Without the Ret + Gen with baseline on Qwen2-VL-2B, it is unclear whether this baseline already exceeds CREM's score of 66.7 and how much of the remaining improvement comes from CLE and ECS.

  • Natural answer lanaguge data come from Qwen3-VL-8B-Instruct and could be important to both retrieval and generation performance. Without an ablation of this supervision, it is unclear how much of the overall improvement and generation retention comes from the teacher-generated answers.

[1] Peters et al. Deep contextualized word representations. NAACL-HLT, 2018.

Questions for authors
  • Why is generation performance compared only with the original backbones? Can you also evaluate relevant unified methods, such as CAFe and CREM, on the same generation benchmarks?

  • Starting from base Qwen2-VL-2B, how much does adding CLE and then ECS improve performance? Can you compare these ablation with the relative models?

  • How much do the answers generated by Qwen3-VL-8B-Instruct contribute to retrieval and generation performance? Can you compare training with the original targets versus generated answers using the same examples and a comparable training budget?

  • Why is prior layer-aggregation work such as ELMo not discussed? How does CLE advance this established idea beyond its different implementation and multimodal setting?

  • Can you compare ECS with contrastive training using the same positive and hard-negative candidate sets and a matched training budget? Contrastive learning uses one mined hard negative per query, whereas ECS uses three. This would help distinguish the benefit of generative candidate selection from additional hard-negative supervision.

  • Since generative supervision of projected embeddings is already present in ReMatch, what additional benefit comes from joint candidate selection when candidate exposure and training budgets are matched? Could you show repeated runs to show consistent gains?

2 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer wKPK (1/2)

We sincerely appreciate your constructive feedback. Below, we address each concern and provide additional experimental results and analyses.

Why is generation performance compared only with the original backbones? Can you also evaluate relevant unified methods, such as CAFe and CREM, on the same generation benchmarks?

The CAFe and CREM papers mainly compare their generation capabilities with those of their original backbones, possibly because some model weights are not publicly available and it is difficult to standardize evaluation protocols across methods. Since we currently cannot obtain CAFe and CREM model weights suitable for a unified local evaluation, it is difficult to directly compare them with CLUE locally. However, CREM reports evaluation results based on Qwen2-VL-2B, which we compare with our CLUE results below:

ModelMMBMMVetAI2DHallusionMMMUMMStarAVG
Qwen2-VL-2B71.750.074.039.540.746.653.8
CREM72.545.172.841.241.445.553.1
CLUE71.551.971.439.140.746.753.6

As shown in the table, CLUE exhibits smaller performance decreases than CREM on several generation benchmarks. We believe this is due to our careful analysis of how representation learning affects generation capabilities and the targeted training-data processing strategies designed based on these findings.

Starting from base Qwen2-VL-2B, how much does adding CLE and then ECS improve performance? Can you compare these ablation with the relative models?

We have already reported detailed ablation results based on Qwen3-VL-2B in Table 3 of the paper. Below, we provide additional ablation experiments based on Qwen2-VL-2B, and the conclusions remain consistent with those in the original paper:

Ret.Gen.CLEECSGenerationMMEB
✓✓53.568.5
✓✓✓53.469.0
✓✓✓✓53.669.8

How much do the answers generated by Qwen3-VL-8B-Instruct contribute to retrieval and generation performance? Can you compare training with the original targets versus generated answers using the same examples and a comparable training budget?

As we observed in the paper, contrastive learning can affect the model's generation capabilities, causing it to favor short answers and exhibit Yes Bias on binary yes/no questions. Since the original LM Targets are mostly short answers, we use natural responses generated by Qwen3-VL-8B-Instruct as language modeling supervision targets to alleviate this shift in generation behavior. The following table compares Original Targets with Generated Answers:

ModelLM targetMMStarMMMUMMBenchAI2DHalluBenchMM-VetGenerationMMEB
CLUE w/ original targetsdataset GT56.242.373.175.844.739.355.271.2
CLUE w/ generated answersQwen3-VL-8B natural answers56.148.974.676.151.360.261.271.2

The results show that using the original short Targets leads to a clear decline in generation performance. Introducing Generated Answers substantially restores the model's generation capabilities while having almost no effect on representation performance.

Why is prior layer-aggregation work such as ELMo not discussed? How does CLE advance this established idea beyond its different implementation and multimodal setting?

We acknowledge that earlier works such as ELMo have already explored cross-layer information aggregation, and we will add the relevant discussion to the revised manuscript.

The novelty of CLE is not that it is the first to propose cross-layer aggregation, but that it designs a more flexible aggregation mechanism for multimodal representation learning. Unlike ELMo, which uses fixed task-level layer weights for linear combination, CLE employs multiple Learnable Queries and Cross-Attention to dynamically select complementary information from different layers based on the input content, followed by further fusion through nonlinear transformations. At the same time, CLE uses a residual connection to preserve the final-layer semantic representation, avoiding direct replacement of the original features with cross-layer aggregation. Furthermore, CLUE combines CLE and ECS to jointly optimize the objectives within a unified framework, allowing aggregated representations not only to be used for retrieval but also to participate in generative candidate discrimination, thereby strengthening the connection between representation learning and generation.

AuthorsOpenReview ↗

Authors' Response to Reviewer wKPK (2/2)

Can you compare ECS with contrastive training using the same positive and hard-negative candidate sets and a matched training budget? Contrastive learning uses one mined hard negative per query, whereas ECS uses three. This would help distinguish the benefit of generative candidate selection from additional hard-negative supervision.

To distinguish whether the gains from ECS come from generative candidate selection or additional hard-negative supervision, we conducted two supplementary ablation experiments. First, we replaced the three mined hard negatives among the ECS candidates with one mined hard negative and two negatives randomly sampled from the batch, while keeping the contrastive learning component unchanged. This experiment examines whether the benefits of ECS mainly depend on additional hard negatives. Second, we removed ECS and increased the number of hard negatives used for each positive example in contrastive learning from one to three, using exactly the same positive example and three mined hard negatives as in the original ECS setting while keeping the training budget matched.

SettingContrastive Hard NegativesECS CandidatesGenerationMMEB
CLUE w/o ECS1 Hard—61.270.4
CLUE w/o ECS3 Hard—61.470.3
CLUE w/ ECS1 Hard1 Hard + 2 Random61.371.0
CLUE w/ ECS1 Hard3 Hard61.271.2

By separately controlling the number of hard negatives in contrastive learning and ECS, we observe that the main gains from ECS come from the generative candidate selection mechanism rather than simply increasing the number of hard negatives.

Since generative supervision of projected embeddings is already present in ReMatch, what additional benefit comes from joint candidate selection when candidate exposure and training budgets are matched? Could you show repeated runs to show consistent gains?

We acknowledge that ReMatch has already used generative supervision to optimize projected Embeddings. The additional contribution of ECS is to transform independent relevance judgments into joint multi-candidate comparisons, enabling the model to directly distinguish positive examples from multiple hard negatives within the same generative context.

In Table 4 of the paper, we have already demonstrated experimentally that ECS achieves better representation performance than ReMatch-style. We have also added Attention and t-SNE visualization experiments to further analyze attention interactions among candidates and the learned representation distributions. The corresponding results have been added to the appendix of the revised paper.

To verify the consistency of the improvements, we repeated the experiments with two random seeds, as shown below:

Random SeedMethodMMEBGeneration
42Baseline (Ret. + Gen.)69.961.3
42CLUE71.261.2
43Baseline (Ret. + Gen.)70.161.5
43CLUE71.361.5

These results show that our method achieves consistent performance improvements across different random seeds.

Reviewer_eTCJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CLUE, a unified framework for holistic multimodal retrieval and comprehension. It introduces two key components: Cross-Layer Embedding (CLE), which aggregates representations across Transformer layers, and Embedding Candidate Selection (ECS), which leverages generative multiple-choice supervision to improve embedding discrimination. Experiments demonstrate competitive retrieval performance while largely preserving the generative capabilities of the underlying MLLMs.

Critical strengths
  1. Meaningful research problem. The paper addresses the important challenge of unifying multimodal representation learning and generation within a single model.
  2. Interesting framework design. ECS provides an intuitive approach to connecting generative supervision with embedding learning, while CLE leverages complementary information across Transformer layers.
  3. Solid experimental results. CLUE achieves competitive performance on MMEB across different backbones and model scales, with relatively limited degradation in generative capabilities.
Critical weaknesses
  1. Limited methodological novelty. CLE largely follows existing cross-layer aggregation approaches. The paper lacks comparisons with simpler alternatives, such as learnable layer-wise weighted averaging or concatenation followed by an MLP.
  2. Insufficient analysis of ECS. The improvements from ECS may primarily result from additional hard-negative discrimination rather than generative supervision itself. Comparisons with stronger contrastive or ranking-based objectives using the same negatives would strengthen the claims.
  3. Modest performance improvements. The gains from CLE and ECS are relatively small. Results across multiple random seeds would help establish their statistical reliability.
  4. Incomplete discussion of related work. The paper lacks a sufficiently comprehensive review and analysis of closely related studies, such as [1, 2, 3].

[1] Liu Y, Zhang Y, Cai J, et al. Lamra: Large multimodal model as your advanced retrieval assistant[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 4015-4025.

[2] Lin S C, Lee C, Shoeybi M, et al. Mm-embed: Universal multimodal retrieval with multimodal llms[C]//International Conference on Learning Representations. 2025, 2025: 44215-44234.

[3] Li Q, Zhao Y, Zhou Y, et al. Magic-mm-embedding: Towards visual-token-efficient universal multimodal embedding with mllms[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 592-612.

Questions for authors
  1. How does CLE compare with simpler layer aggregation methods, such as weighted averaging or concatenation?
  2. Can the authors demonstrate that ECS provides benefits beyond stronger hard-negative contrastive or ranking-based supervision?
  3. Are the reported improvements consistent across multiple random seeds?
  4. Can the authors provide additional evaluations of open-ended generation and analyze the computational overhead introduced by CLE and ECS?
  5. Can the authors expand the discussion of closely related work and clarify the key methodological differences between CLUE and existing approaches?
3 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer eTCJ (1/3)

We sincerely appreciate your constructive feedback. Below, we address each concern and provide additional experimental results and analyses.

How does CLE compare with simpler layer aggregation methods, such as weighted averaging or concatenation?

Regarding the aggregation of representation vectors, CLUE is not the first method to consider leveraging information from different layers to enrich representation semantics. For example, the earlier ELMo [1] learns weights for the hidden states of different layers and constructs representations through weighted averaging. The recent WAVE [2] concatenates hidden states from different layers and uses an MLP for feature fusion. Inspired by Q-Former [3], CLUE employs Learnable Queries and Cross-Attention to aggregate hidden states from different layers. To avoid the influence of other variables, we compare different layer aggregation methods using only the contrastive learning objective. The backbone is Qwen3-VL-2B, and the results are as follows:

Layer Aggregation MethodMMEB
Concat+MLP68.8
Weighted Averaging70.4
Final-layer Last-token70.7
CLE (Ours)71.1

The results show that CLE outperforms Weighted Averaging and Concat+MLP in layer aggregation. Notably, the MLP performs relatively poorly in our experiments, which differs from the results reported by WAVE. We believe this may be due to differences in the experimental settings. WAVE conducts joint training on video and audio-visual tasks based on Qwen2.5-Omni-7B, jointly optimizing the visual alignment module and LoRA during its main training stage, and additionally introduces a BEATs audio branch with a separately pretrained aligner. In contrast, our experiments are based on Qwen3-VL-2B and use contrastive learning only for static-image retrieval. Therefore, the advantages of MLP fusion observed in WAVE may not directly generalize to our training setting. We believe CLE's advantage mainly comes from its more targeted cross-layer aggregation mechanism. Compared with Weighted Averaging, which uses globally fixed layer weights, and Concat+MLP, which implicitly learns feature fusion through dense transformations, CLE uses multiple Learnable Queries and Cross-Attention to explicitly select and aggregate information from different layers based on the input content. This allows it to learn multiple complementary cross-layer aggregation patterns and more effectively exploit semantic information across layers.

We will include this discussion in the revised manuscript.

Can the authors demonstrate that ECS provides benefits beyond stronger hard-negative contrastive or ranking-based supervision?

To distinguish whether the gains from ECS come from generative candidate selection or additional hard-negative supervision, we conducted two supplementary ablation experiments. First, we replaced the three mined hard negatives among the ECS candidates with one mined hard negative and two negatives randomly sampled from the batch, while keeping the contrastive learning component unchanged. This experiment examines whether the benefits of ECS mainly depend on additional hard negatives. Second, we removed ECS and increased the number of hard negatives used for each positive example in contrastive learning from one to three, using exactly the same positive example and three mined hard negatives as in the original ECS setting while keeping the training budget matched.

SettingContrastive Hard NegativesECS CandidatesGenerationMMEB
CLUE w/o ECS1 Hard—61.270.4
CLUE w/o ECS3 Hard—61.470.3
CLUE w/ ECS1 Hard1 Hard + 2 Random61.371.0
CLUE w/ ECS1 Hard3 Hard61.271.2

By separately controlling the number of hard negatives in contrastive learning and ECS, we observe that the main gains from ECS come from the generative candidate selection mechanism rather than simply increasing the number of hard negatives.

Are the reported improvements consistent across multiple random seeds?

Repeating experiments with multiple random seeds helps assess whether the improvements are consistent rather than merely reflecting random fluctuations. We conducted experiments with two random seeds, and the results are as follows:

Random SeedMethodMMEBGeneration
42Baseline (Ret. + Gen.)69.961.3
42CLUE71.261.2
43Baseline (Ret. + Gen.)70.161.5
43CLUE71.361.5

The results show that CLUE achieves improved retrieval performance over the baseline under both random seeds while maintaining comparable generation capabilities. This indicates that CLUE's performance gains are consistent and are not merely due to fluctuations in a single run.

AuthorsOpenReview ↗

Authors' Response to Reviewer eTCJ (2/3)

Can the authors provide additional evaluations of open-ended generation and analyze the computational overhead introduced by CLE and ECS?

First, to further evaluate CLUE's generation capabilities after joint training, we conducted evaluations on four additional visual question-answering benchmarks: ChartQA [4], InfoVQA [5], DocVQA [6], and RealWorldQA [7]. These datasets cover chart reasoning, infographic understanding, document understanding, and spatial understanding in real-world scenarios, respectively. As shown in the table, CLUE largely preserves the performance of the original backbone.

ModelChartQAInfoVQADocVQARealWorldQAAVG
Qwen2-VL-2B73.2461.3988.2557.3470.05
CLUE74.2860.7486.0155.0569.02
Qwen3-VL-2B79.2470.6692.4765.8377.05
CLUE77.4469.7891.2265.1475.90

Regarding the computational overhead of the two components, CLE introduces only a lightweight cross-layer branch, which uses a small number of Learnable Queries to perform Cross-Attention over the last-token hidden states from different layers and fuses the result with the final-layer Embedding. Since this branch processes only a small number of cross-layer features, its computational cost is negligible compared with that of the backbone. During training, the branch is computed only once per forward pass, and the additional runtime is within normal fluctuations. As shown in the table below, overall, CLE achieves cross-layer information aggregation with small computational and memory overhead, and its relative inference overhead can be further amortized as the Batch Size increases. Therefore, the additional overhead of this branch is negligible in practical retrieval settings.

MethodBatch = 1 (ms)Batch = 8 (ms)Batch = 32 (ms)
Qwen3-VL-2B46.1126.4483.2
CLUE49.5129.6486.4
Per-sample Overhead+7.4%+2.5%+0.7%
MethodResident Weights (GB)Peak Activation (GB)
Qwen3-VL-2B4.264.74
CLUE4.414.74

ECS (generative candidate selection supervision) incurs relatively greater overhead, but only during training. It projects the Embedding produced by CLE into a Soft Token in the MLLM hidden space and uses it as the input for a Query or Candidate, requiring the MLLM to autoregressively select the matching candidate. This introduces an additional LM forward pass and a chunked backward pass at each training step, increasing iteration time and training memory usage. Crucially, ECS is not used during inference: the deployed forward pass contains only the CLE branch, so inference latency, throughput, and memory overhead correspond to the results reported above. We believe this additional one-time training cost is acceptable given the accuracy gains it provides, and we will add this analysis to the revised manuscript.

AuthorsOpenReview ↗

Authors' Response to Reviewer eTCJ (3/3)

Can the authors expand the discussion of closely related work and clarify the key methodological differences between CLUE and existing approaches?

We clarify the main differences between CLUE and the following three works:

  1. LamRA: LamRA trains retrieval Embeddings through contrastive learning and separately trains a reranking model using Pointwise and Listwise objectives. In contrast, CLUE directly introduces generative candidate selection into Embedding training, allowing comparative supervision to improve retrieval representations rather than being used only for subsequent reranking.
  2. MM-Embed: MM-Embed mainly improves multimodal retrieval performance through modality-aware hard-negative mining and contrastive learning. In contrast, CLUE introduces cross-layer representation aggregation (CLE) and generative candidate selection (ECS) to jointly optimize representation learning and language generation.
  3. Magic-MM-Embedding: Magic-MM-Embedding mainly focuses on visual Token compression and multi-stage training, and separately trains a reranking model. In contrast, CLUE jointly optimizes contrastive learning, language modeling, and Embedding Candidate Selection within a shared MLLM, allowing generative supervision to directly participate in Embedding learning.

Furthermore, regarding WAVE [2], another work with a similar cross-layer aggregation idea, our supplementary experiments show that, under our current training setting, simply using MLP or weighted averaging does not achieve the same representation performance as CLE and may even degrade representation quality. Please refer to our response to this reviewer's first question for details.

We will expand the related work discussion in the revised manuscript to further clarify these methodological differences.

[1] Peters M E, Neumann M, Iyyer M, et al. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2018: 2227-2237.

[2] Tang C, Xiao Q, Mei K, et al. WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM[C]//International Conference on Learning Representations, 2026.

[3] Li J, Li D, Savarese S, et al. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International Conference on Machine Learning. PMLR, 2023: 19730-19742.

[4] Masry A, Tan J Q, Joty S, et al. ChartQA: A benchmark for question answering about charts with visual and logical reasoning[C]//Findings of the Association for Computational Linguistics: ACL 2022. 2022: 2263-2279.

[5] Mathew M, Bagal V, Tito R, et al. InfographicVQA[C]//2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2022: 2582-2591.

[6] Mathew M, Karatzas D, Jawahar C V. DocVQA: A dataset for VQA on document images[C]//2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021: 2199-2208.

[7] xAI. Grok-1.5 Vision Preview. 2024. https://x.ai/blog/grok-1.5v.

Reviewer_WfMgOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims at the joint optimization of multi-modal representation and generation within a single MLLM-based architecture. The main techinical contributions are Cross-Layer Embedding(CLE) and Embedding Candidate Selection(ECS). CLE uses cross-attention to gather token information from multiple layers of the backbone. ECS leverages multi-modal embeddings for candidate-level relevance discrimination. The method is evaluated on MMEB and six multi-modal understanding datasets (Most of these datasets consist of short answers.).

Critical strengths

Improving multimodal retrieval while retaining the capabilities of generation is useful. The design of the proposed method is reasonable, and the current experiments demonstrate that it can achieve a certain improvement on retrieval tasks. Compared with the original Qwen backbones, it can also preserve text generation capability to a certain extent.

Critical weaknesses
  1. The methodological contribution of the paper appears limited.

The core of the CLE is multi-layer feature aggregation, which is widely studied in computer vision, representation-learning and retrieval settings. For instance, ReT[1] and WAVE[2] also use multi-layer features for multimodal embedding. MLR[3] and TransVPR[4] share a similar motivation for document and image retrieval. The design is reasonable, but a comparison solely against the final-layer representation methods fails to demonstrate its distinct advantages or novelty. Besides, the design of CLE is simple(aggregating feature by cross-attention), and no comparisons with other design choices are conducted. A stronger contribution would require either a clear advantage over established aggregation strategies or evidence that the design addresses a specific representation property of multimodal retrieval.

ECS appears to be an incremental extension of ReMatch-style generative matching. ReMatch already feeds raw query/document inputs or projected retrieval embeddings into the MLLM and trains the model to generate a relevance label. ECS extends this pointwise binary judgment (yes or no) to joint multiple-choice selection among several candidates (A, B, C, D, or None of the above). This design is reasonable, but it appears to be only an incremental improvement.

  1. The unified-framework framing is broader than the demonstrated comparison.

The paper compares CLUE's comprehension scores mainly with the original Qwen backbones. Table 3 indicates that the main recovery of generation performance comes from the standard language-modeling objective. Considering many related works (such as CAFE[5], CREM[6], UME-R1[7]) use this training strategy, it is reasonable that other works may also recover the generation ability to some degree. In the authors' abstract, it is mentioned that "Existing unified approaches exhibit a trade-off: they tend to substantially compromise representational capacity to preserve generative performance, or fail to establish a synergistic alignment between the two capabilities." Since the experiments do not compare generation capability with more previous works, the advantage of this work in this aspect is currently unknown. Besides, Table 4 shows that the generation capability of the proposed ECS is very close to that of ReMatch.

[1] Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval. CVPR 2025

[2] WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM. ICLR 2026

[3] Investigating Multi-layer Representations for Dense Passage Retrieval. Findings EMNLP 2025

[4] TransVPR: Transformer-Based Place Recognition With Multi-Level Attention Aggregation. CVPR 2022

[5] CAFE: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning. ICCV 2025 workshop

[6] CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension. CVPR2026 Findings

[7] UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings. ICLR 2026

Questions for authors

My main concerns are the limited novelty of the proposed designs and the difficulty of identifying their advantages from the current experiments. My score could increase if the authors address the following questions with supporting evidence:

(1) Can the authors compare CLE experimentally with other multi-layer aggregation methods and analyze which challenges CLE addresses or what advantages it provides?

(2) The generation capability should be compared with more methods to show what advantages the proposed method offers over existing methods.

4 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer WfMg (1/2)

We sincerely appreciate your constructive feedback. Below, we address each concern and provide additional experimental results and analyses.

(1) Can the authors compare CLE experimentally with other multi-layer aggregation methods and analyze which challenges CLE addresses or what advantages it provides?

Thank you for your question. Regarding the aggregation of representation vectors, CLUE is not the first method to consider leveraging information from different layers to enrich representation semantics. For example, the earlier ELMo [1] learns weights for the hidden states of different layers and constructs representations through weighted averaging. The recent WAVE [2] concatenates hidden states from different layers and uses an MLP for feature fusion. Inspired by Q-Former [3], CLUE employs Learnable Queries and Cross-Attention to aggregate hidden states from different layers. To avoid the influence of other variables, we compare different layer aggregation methods using only the contrastive learning objective. The backbone is Qwen3-VL-2B, and the results are as follows:

Layer Aggregation MethodMMEB
Concat+MLP68.8
Weighted Averaging70.4
Final-layer Last-token70.7
CLE (Ours)71.1

The results show that CLE outperforms Weighted Averaging and Concat+MLP in layer aggregation. Notably, the MLP performs relatively poorly in our experiments, which differs from the results reported by WAVE. We believe this may be due to differences in the experimental settings. WAVE conducts joint training on video and audio-visual tasks based on Qwen2.5-Omni-7B, jointly optimizing the visual alignment module and LoRA during its main training stage, and additionally introduces a BEATs audio branch with a separately pretrained aligner. In contrast, our experiments are based on Qwen3-VL-2B and use contrastive learning only for static-image retrieval. Therefore, the advantages of MLP fusion observed in WAVE may not directly generalize to our training setting. Overall, we believe CLE's advantage mainly comes from its more targeted cross-layer aggregation mechanism. Compared with Weighted Averaging, which uses globally fixed layer weights, and Concat+MLP, which implicitly learns feature fusion through dense transformations, CLE uses multiple Learnable Queries and Cross-Attention to explicitly select and aggregate information from different layers based on the input content. This allows it to learn multiple complementary cross-layer aggregation patterns and more effectively exploit semantic information across layers.

We will include this discussion in the revised manuscript.

AuthorsOpenReview ↗

Authors' Response to Reviewer WfMg (2/2)

(2) The generation capability should be compared with more methods to show what advantages the proposed method offers over existing methods.

Thank you for your suggestion. The CAFe and CREM papers mainly compare their generation capabilities with those of their original backbones, possibly because some model weights are not publicly available and it is difficult to standardize evaluation protocols across methods. Since we currently cannot obtain CAFe and CREM model weights suitable for a unified local evaluation, it is difficult to directly compare them with CLUE locally. However, CREM reports evaluation results based on Qwen2-VL-2B. We compare these with our CLUE results below. Note that the two works use their respective evaluation settings:

ModelMMBMMVetAI2DHallusionMMMUMMStarAVG
Qwen2-VL-2B71.750.074.039.540.746.653.8
CREM72.545.172.841.241.445.553.1
CLUE71.551.971.439.140.746.753.6

As shown in the table, relative to the original backbone, CLUE exhibits a smaller decrease in average generation performance than CREM. We believe this advantage stems from our careful analysis of how representation learning affects generation capabilities and the targeted training-data processing strategies designed based on these findings. Specifically, we first observed that retrieval-oriented training may shift the generation behavior of the shared multimodal backbone: the model gradually tends to produce short answers rather than the complete responses, including necessary explanations and reasoning, typically generated by the original model. In addition, on binary yes/no questions, the model exhibits a clear Yes Bias and fails to make judgments sufficiently grounded in the question and image content.

To alleviate this issue, we introduce Natural Answer Supervision. Specifically, we use Qwen3-VL-8B-Instruct to generate more natural and complete language responses for training samples and use these teacher-generated responses as an additional language modeling supervision signal, jointly optimized with the retrieval objective. In this way, we aim to alleviate the degradation in generation behavior while learning high-quality retrieval representations, preserving the natural-language response capability of the original backbone as much as possible. The following table shows the effect of different LM Targets on generation performance:

ModelLM targetMMStarMMMUMMBenchAI2DHalluBenchMM-VetGenerationMMEB
CLUE w/ original targetsdataset GT56.242.373.175.844.739.355.271.2
CLUE w/ generated answersQwen3-VL-8B natural answers56.148.974.676.151.360.261.271.2

We would also like to clarify the relationship and key differences between CLUE and two closely related works, CAFe and CREM. CAFe balances the two capabilities by adjusting the relative weights of the contrastive learning and autoregressive language modeling losses, but increasing the weight of the generation loss sacrifices some retrieval performance. Moreover, after joint training, an additional contrastive-only fine-tuning stage is required to improve retrieval capabilities, reflecting the trade-off between the two. CREM further introduces learnable Chorus Tokens and Compression-aware Attention to promote interactions between generation and representation tasks through shared compressed representations. However, its generative supervision mainly acts through language modeling over compressed representations, imposing relatively indirect constraints on representation learning. In contrast, CLUE not only better aggregates cross-layer representations through CLE but also feeds projected Embeddings back into the MLLM through ECS, using generative multi-candidate selection to directly provide candidate-level discriminative supervision for retrieval representations, thereby leveraging generation capabilities to improve representation learning.

Thank you for your suggestion. We will add this discussion to the revised manuscript.

[1] Peters M E, Neumann M, Iyyer M, et al. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2018: 2227-2237.

[2] Tang C, Xiao Q, Mei K, et al. WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM[C]//International Conference on Learning Representations, 2026.

[3] Li J, Li D, Savarese S, et al. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International Conference on Machine Learning. PMLR, 2023: 19730-19742.

Thank you for the detailed response and additional experiments. The new comparisons provide useful support for the proposed design and partially address my concerns, so I am raising my score to 2.

However, I still lean toward a weak reject. While CLE, ECS, and Natural Answer Supervision are reasonable engineering choices, their novelty remains limited, and the overall technical contribution appears incremental. Specifically, the gains from Natural Answer Supervision come with additional teacher-generated supervision and do not clearly establish a distinctive methodological advantage. I appreciate the clarification of how ECS differs from CREM and CAFe, but its design remains closely related to ReMatch.

The experiments and accompanying analysis offer limited deep insight into model design and training for multimodal retrieval. Considering both the original paper’s tables and the additional experiments in the rebuttal, the gains over existing methods still appear modest, particularly in generative capabilities. Differences in experimental settings further complicate the assessment of these advantages.

AuthorsOpenReview ↗

Authors' Response to Reviewer WfMg

We sincerely thank you for carefully reassessing our work and providing further constructive feedback. We would like to further clarify the methodological contributions of CLE and ECS, as well as outline our plans for more in-depth investigation and improvement.

Regarding CLE, its contribution lies not only in leveraging information from different Transformer layers, but also in how these complementary features are effectively selected and integrated. Inspired by Q-Former's learnable-query mechanism, CLE employs multiple learnable queries and input-dependent cross-attention to adaptively extract and aggregate semantic information from different layers. Compared with weighted averaging using globally shared weights or implicit fusion through Concat+MLP, CLE can dynamically adjust its feature aggregation strategy for different samples, enabling selective information extraction and compact semantic aggregation, while preserving the original final-layer representation through residual connections. Our additional comparative experiments support the effectiveness of this design. In future work, we will conduct more systematic exploratory experiments to investigate input-adaptive aggregation mechanisms and the interactions between different layers and token positions, and further optimize CLE to better exploit the complementarity of multi-level semantic information.

Regarding ECS, its key distinction from ReMatch's MVQDM lies in the supervision mechanism. MVQDM makes independent binary relevance judgments for each query–candidate pair, whereas ECS allows the model to simultaneously observe and compare positive and negative candidates within a single forward pass, explicitly learning the relative semantic relationships among candidates through joint discrimination. Specifically, independent binary supervision requires the model to predict Yes for positive samples and No for negative samples, without directly distinguishing the relative degrees of relevance among different candidates. When negative samples are relatively easy, such supervision may lack sufficient discriminative difficulty; when negative samples are highly similar to positive samples, strict binary labels may overlook the continuous semantic similarities among samples, potentially creating optimization conflicts with the relative relationships emphasized by contrastive learning. In contrast, ECS compares multiple candidates within a shared generative context, enabling the model to directly learn the relative semantic differences between positive and negative candidates, thereby providing more targeted comparative supervision for representation learning. Our comparative experiments also demonstrate that ECS achieves better retrieval representation performance. This is what we consider a key advantage of ECS over MVQDM.

Furthermore, NOTA enhances the discriminative capability of ECS. Conventional multiple-choice tasks implicitly assume that the candidate set must contain a correct answer, whereas real-world retrieval scenarios do not always guarantee the presence of a relevant candidate. By introducing candidate sets without positive samples, NOTA mitigates the answer bias induced by this artificial prior and reduces the shortcut of forcing a selection based solely on the relatively best candidate. The model must not only identify the most relevant candidate in the set but also determine whether it has sufficient absolute relevance, thereby encouraging the formation of an implicit rejection criterion analogous to a relevance threshold. Our ablation experiments further validate the effectiveness of NOTA.

Our experimental results demonstrate that the current joint training strategy can largely restore and preserve the generative capabilities of the original MLLMs. Building on this, we further analyze phenomena such as response shortening and answer bias arising during joint training, and mitigate them through natural-answer supervision. These findings and corresponding improvements complement prior work such as CAFe and CREM in terms of generative behavior analysis and generative capability preservation.

We sincerely thank you again for your valuable suggestions. In future work, we will conduct more comprehensive exploratory experiments to deepen our understanding of the underlying mechanisms and further refine the proposed methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks to All Reviewers

We sincerely thank all reviewers for their careful evaluations and constructive feedback. We are encouraged that reviewers recognize the importance of unifying multimodal representation learning and generation (Reviewer eTCJ), and consider our framework design interesting (Reviewer eTCJ) and reasonable (Reviewer WfMg). We also appreciate their recognition of CLUE's strong retrieval performance while largely preserving generative capabilities (Reviewers WfMg, eTCJ, and wKPK). Furthermore, reviewers acknowledge the solid experimental results (Reviewer eTCJ), well-organized ablation studies (Reviewer wKPK), and clear presentation (Reviewers WfMg and wKPK).

We would like to gently remind you that the author-reviewer discussion period ends on November 18, 2026 (AoE). We would greatly appreciate your further feedback and look forward to continuing the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.