- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes two main components, Cross-Layer Embedding (CLE) and Embedding Candidate Selection (ECS), within the CLUE framework for unified multimodal retrieval and generation. CLE uses learnable queries to aggregate last-token representations across Transformer layers, rather than relying only on the final layer. ECS uses generative candidate selection to distinguish a positive match from hard negatives using both original multimodal inputs and projected embedding tokens. With joint contrastive and language-modeling training, CLUE achieves competitive retrieval performance on MMEB datasets while largely retaining the original backbones capabilities on multimodal comprehension benchmarks.
Critical strengths
-
CLUE achieves strong retrieval performance while largely retaining the original backbones' generation capabilities. Maintaining both capabilities in one model is a useful result.
-
The training approach is relatively straightforward. The ablation studies are well organized and show how the proposed components affect performance.
Critical weaknesses
-
Much of ECS's core idea is already present in ReMatch, which feeds projected embeddings back into the MLLM and trains them through generative matching. Moving from pairwise judgments to joint candidate selection is a reasonable extension, but the reported gain is only 0.4 points.
-
Although CLE differs in implementation, the broader idea of combining representations across layers was already established in ELMo (Peters et al., 2018). The paper does not discuss this connection or clearly explain what is conceptually new in CLE beyond adapting this established idea
-
The claimed retrieval and generation trade-off is not directly demonstrated against CAFe and CREM. Generation performance is compared only with the original backbones. This shows that CLUE retains generation capabilities, but not that it preserves them better than existing unified methods.
-
The same-backbone comparison with CREM does not show how much each proposed component adds. Table 3 uses Qwen3-VL-2B, whereas the comparison with CREM uses Qwen2-VL-2B Without the Ret + Gen with baseline on Qwen2-VL-2B, it is unclear whether this baseline already exceeds CREM's score of 66.7 and how much of the remaining improvement comes from CLE and ECS.
-
Natural answer lanaguge data come from Qwen3-VL-8B-Instruct and could be important to both retrieval and generation performance. Without an ablation of this supervision, it is unclear how much of the overall improvement and generation retention comes from the teacher-generated answers.
[1] Peters et al. Deep contextualized word representations. NAACL-HLT, 2018.
Questions for authors
-
Why is generation performance compared only with the original backbones? Can you also evaluate relevant unified methods, such as CAFe and CREM, on the same generation benchmarks?
-
Starting from base Qwen2-VL-2B, how much does adding CLE and then ECS improve performance? Can you compare these ablation with the relative models?
-
How much do the answers generated by Qwen3-VL-8B-Instruct contribute to retrieval and generation performance? Can you compare training with the original targets versus generated answers using the same examples and a comparable training budget?
-
Why is prior layer-aggregation work such as ELMo not discussed? How does CLE advance this established idea beyond its different implementation and multimodal setting?
-
Can you compare ECS with contrastive training using the same positive and hard-negative candidate sets and a matched training budget? Contrastive learning uses one mined hard negative per query, whereas ECS uses three. This would help distinguish the benefit of generative candidate selection from additional hard-negative supervision.
-
Since generative supervision of projected embeddings is already present in ReMatch, what additional benefit comes from joint candidate selection when candidate exposure and training budgets are matched? Could you show repeated runs to show consistent gains?
2 comments
Authors' Response to Reviewer wKPK (2/2)
Can you compare ECS with contrastive training using the same positive and hard-negative candidate sets and a matched training budget? Contrastive learning uses one mined hard negative per query, whereas ECS uses three. This would help distinguish the benefit of generative candidate selection from additional hard-negative supervision.
To distinguish whether the gains from ECS come from generative candidate selection or additional hard-negative supervision, we conducted two supplementary ablation experiments. First, we replaced the three mined hard negatives among the ECS candidates with one mined hard negative and two negatives randomly sampled from the batch, while keeping the contrastive learning component unchanged. This experiment examines whether the benefits of ECS mainly depend on additional hard negatives. Second, we removed ECS and increased the number of hard negatives used for each positive example in contrastive learning from one to three, using exactly the same positive example and three mined hard negatives as in the original ECS setting while keeping the training budget matched.
| Setting | Contrastive Hard Negatives | ECS Candidates | Generation | MMEB |
|---|---|---|---|---|
| CLUE w/o ECS | 1 Hard | — | 61.2 | 70.4 |
| CLUE w/o ECS | 3 Hard | — | 61.4 | 70.3 |
| CLUE w/ ECS | 1 Hard | 1 Hard + 2 Random | 61.3 | 71.0 |
| CLUE w/ ECS | 1 Hard | 3 Hard | 61.2 | 71.2 |
By separately controlling the number of hard negatives in contrastive learning and ECS, we observe that the main gains from ECS come from the generative candidate selection mechanism rather than simply increasing the number of hard negatives.
Since generative supervision of projected embeddings is already present in ReMatch, what additional benefit comes from joint candidate selection when candidate exposure and training budgets are matched? Could you show repeated runs to show consistent gains?
We acknowledge that ReMatch has already used generative supervision to optimize projected Embeddings. The additional contribution of ECS is to transform independent relevance judgments into joint multi-candidate comparisons, enabling the model to directly distinguish positive examples from multiple hard negatives within the same generative context.
In Table 4 of the paper, we have already demonstrated experimentally that ECS achieves better representation performance than ReMatch-style. We have also added Attention and t-SNE visualization experiments to further analyze attention interactions among candidates and the learned representation distributions. The corresponding results have been added to the appendix of the revised paper.
To verify the consistency of the improvements, we repeated the experiments with two random seeds, as shown below:
| Random Seed | Method | MMEB | Generation |
|---|---|---|---|
| 42 | Baseline (Ret. + Gen.) | 69.9 | 61.3 |
| 42 | CLUE | 71.2 | 61.2 |
| 43 | Baseline (Ret. + Gen.) | 70.1 | 61.5 |
| 43 | CLUE | 71.3 | 61.5 |
These results show that our method achieves consistent performance improvements across different random seeds.
Authors' Response to Reviewer wKPK (1/2)
We sincerely appreciate your constructive feedback. Below, we address each concern and provide additional experimental results and analyses.
The CAFe and CREM papers mainly compare their generation capabilities with those of their original backbones, possibly because some model weights are not publicly available and it is difficult to standardize evaluation protocols across methods. Since we currently cannot obtain CAFe and CREM model weights suitable for a unified local evaluation, it is difficult to directly compare them with CLUE locally. However, CREM reports evaluation results based on Qwen2-VL-2B, which we compare with our CLUE results below:
As shown in the table, CLUE exhibits smaller performance decreases than CREM on several generation benchmarks. We believe this is due to our careful analysis of how representation learning affects generation capabilities and the targeted training-data processing strategies designed based on these findings.
We have already reported detailed ablation results based on Qwen3-VL-2B in Table 3 of the paper. Below, we provide additional ablation experiments based on Qwen2-VL-2B, and the conclusions remain consistent with those in the original paper:
As we observed in the paper, contrastive learning can affect the model's generation capabilities, causing it to favor short answers and exhibit Yes Bias on binary yes/no questions. Since the original LM Targets are mostly short answers, we use natural responses generated by Qwen3-VL-8B-Instruct as language modeling supervision targets to alleviate this shift in generation behavior. The following table compares Original Targets with Generated Answers:
The results show that using the original short Targets leads to a clear decline in generation performance. Introducing Generated Answers substantially restores the model's generation capabilities while having almost no effect on representation performance.
We acknowledge that earlier works such as ELMo have already explored cross-layer information aggregation, and we will add the relevant discussion to the revised manuscript.
The novelty of CLE is not that it is the first to propose cross-layer aggregation, but that it designs a more flexible aggregation mechanism for multimodal representation learning. Unlike ELMo, which uses fixed task-level layer weights for linear combination, CLE employs multiple Learnable Queries and Cross-Attention to dynamically select complementary information from different layers based on the input content, followed by further fusion through nonlinear transformations. At the same time, CLE uses a residual connection to preserve the final-layer semantic representation, avoiding direct replacement of the original features with cross-layer aggregation. Furthermore, CLUE combines CLE and ECS to jointly optimize the objectives within a unified framework, allowing aggregated representations not only to be used for retrieval but also to participate in generative candidate discrimination, thereby strengthening the connection between representation learning and generation.