acceptodds
Withdrawn from ICLR 2027

Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal embeddings serve as a bridge for aligning vision and language, with the two primary implementations—CLIP-based and MLLM-based embedding models—both limited to capturing only global semantic information. Although numerous studies have focused on fine-grained understanding, we observe that complex scenarios currently targeted by MLLM embeddings often involve a hybrid perceptual pattern of both global and fine-grained elements, thus necessitating a compatible fusion mechanism. In this paper, we propose Adaptive Global and Fine-grained perceptual Fusion for MLLM Embeddings (AGFF-Embed), a method that prompts the MLLM to generate multiple embeddings focusing on different dimensions of semantic information, which are then adaptively and smoothly aggregated. Furthermore, we adapt AGFF-Embed with the Explicit Gradient Amplification (EGA) technique to achieve in-batch hard negatives enhancement without requiring fine-grained editing of the dataset. Evaluation on the MMEB and MMVP-VLM benchmarks shows that AGFF-Embed comprehensively achieves state-of-the-art performance in both general and fine-grained understanding compared to other multimodal embedding models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
MLLM, multimodal perceptual fusion, hard negative enhancement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WCTAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AGFF-EMBED, a framework that combines global and fine-grained embeddings through multiple similarity patterns and LogSumExp aggregation, while incorporating explicit hard-negative gradient amplification (EGA). Although the method achieves promising results on MMEB and MMVP-VLM, the motivation and methodological contributions are not sufficiently justified. The introduction presents multi-granularity perception and hard-negative enhancement as separate motivations without establishing a coherent research objective. Moreover, the paper does not clearly demonstrate why existing MLLM embedding approaches are inadequate or why the proposed architectural choices are necessary. Consequently, the novelty and necessity of the proposed framework remain insufficiently established.

Critical strengths
  1. The method is evaluated on MMEB and MMVP-VLM, with ablation studies examining individual components and additional analyses of computational overhead.

  2. The paper addresses the challenge of matching multimodal inputs that require different levels of semantic granularity, which is relevant to general-purpose multimodal embedding.

Critical weaknesses
  1. Unfocused motivation. The introduction first motivates multi-granularity perception and then shifts to hard-negative enhancement, combining them without establishing a coherent research objective. This makes the work appear to be a combination of two loosely related techniques rather than a unified methodological contribution.

  2. Insufficiently justified problem formulation. The authors treat the diversity of matching tasks as evidence of limitations in existing MLLM embeddings, without demonstrating that current methods cannot adequately address these requirements. It remains unclear whether the limitations stem from single-vector representations, contrastive learning objectives, or the inadequacy of existing multi-vector and similarity interaction mechanisms.

  3. unclear necessity of individual components. The method combines learnable prompt tokens, multiple embeddings, similarity aggregation, and EGA-based hard-negative gradient amplification. Improvements over the baseline do not establish the necessity of these individual components or justify why this particular combination is required.

Questions for authors

What is the central research problem that necessitates combining multi-granularity perception with hard-negative gradient amplification? Can the authors provide empirical evidence that existing MLLM embedding methods are insufficient for this problem and clarify why the proposed combination of learnable prompts, multiple embeddings, LogSumExp aggregation, and EGA is necessary, rather than a combination of loosely related techniques?

Reviewer_zKeiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AGFF-Embed, a multimodal embedding method that combines global and fine-grained representations. Starting from a pretrained QQMM model, it uses learnable prompt tokens to produce fine-grained embeddings in addition to the global embedding. It computes similarities for four interaction patterns. These similarities are combined through logsumexp aggregation, and the method incorporates EGA to emphasize in-batch hard negatives. The authors fine-tune the model on MMEB-train for 40 steps and evaluate it on MMEB and MMVP-VLM. They report an overall MMEB score of 74.9 and an average MMVP-VLM score of 61.5, together with ablations of the interaction patterns, aggregation methods, EGA, and hyperparameters.

Critical strengths
  1. The motivation is clear: multimodal retrieval requires different levels of semantic detail on the query and target sides. Modeling four global/fine-grained interaction patterns provides a coherent way to address this variation.

  2. The reported MMEB results are promising: 74.9 overall, compared with 72.5 for the original QQMM and 73.7 for ReMatch. With EGA disabled, the proposed aggregation achieves 74.6 versus 72.8 for mean-max, supporting the aggregation choice within the tested setting.

Critical weaknesses
  1. The MMVP-VLM scoring needs further explanation.

The official MMVP-VLM evaluation scores image pairs, with 15 pairs per category; a pair is correct only when both matching decisions are correct. Table 2 reports category scores such as 43.3, 83.3, and 50.0, which are not directly explained by a single evaluation under this rule. Could the authors explain exactly how these scores were calculated, including the counting unit and any averaging procedure?

  1. Table 6 needs clarification.

AGFF-Embed is fine-tuned from QQMM for 40 steps, yet the M = N = 0 results in Table 6 match the original QQMM. Table 8 reports training time but does not clarify this baseline's fine-tuning status. Was it also fine-tuned for 40 steps under the same settings? If not, please add this control to distinguish gains from the proposed method from those of additional fine-tuning.

  1. Validation is limited to one initialization and model size.

The method is evaluated only on QQMM-7B; varying module or prompt-token counts does not test transfer across models. QQMM is a reasonable starting point, but experiments on another initialization, or a smaller model (like 2B), would help establish broader applicability.

  1. Inconsistent Grounding scores.

Grounding is 90.0 in Tables 1, 3, and 6 but 89.6 in Appendix Table 5. Please reconcile these values and confirm that the tables use the same checkpoint.

Questions for authors

See the Weakness.

Reviewer_dpU4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AGFF-EMBED, a multimodal embedding method that combines global and fine-grained information. It generates one global embedding and multiple fine-grained embeddings, computes four types of query–target similarity, and aggregates them with logsumexp. Training also applies explicit gradient amplification to hard negatives. The reported overall score is 74.9 on MMEB, compared with 72.5 for the starting QQMM model. On MMVP-VLM, the reported zero-shot average is 61.5, compared with 54.1 for QQMM. Ablations generally support the value of the additional similarity paths and the proposed aggregation method.

Critical strengths
  1. The paper addresses a meaningful limitation of multimodal embeddings: different query–target pairs may require different levels of semantic detail. Its four similarity paths offer an explicit way to model this variation.
  2. The method improves aggregate results on both MMEB and MMVP-VLM relative to the reported QQMM baseline. The ablations provide further evidence that the similarity paths and aggregation choice matter, although the method does not lead on every individual task.
Critical weaknesses
  1. The main comparison lacks a matched training-budget control. AGFF-EMBED starts from a trained QQMM model and receives 40 additional training steps, while the headline QQMM baseline is evaluated before that continued training. The 74.9-versus-72.5 MMEB gain therefore cannot be fully attributed to the proposed architecture. Continuing QQMM with the same data, steps, and optimization budget would substantially strengthen the claim.
  2. The evidence for “adaptive fine-grained perception” is indirect. Removing similarity paths reduces performance, but the paper does not show that different embeddings capture distinct attributes or that aggregation weights change appropriately across examples. Increased representation capacity could explain part of the gain. Comparisons with max and mean-max aggregation are also complicated by differences in the gradient-amplification procedure.
Questions for authors
  1. The authors should continue training QQMM with the same data, 40 steps, and optimization budget, and report the resulting MMEB and MMVP-VLM scores. This control would materially affect the interpretation of the main gains.
  2. The authors should analyze similarity among the fine-grained embeddings and provide examples of the information each captures. They should also report how the aggregation weights for the four similarity types vary across tasks and examples.
  3. The authors should clarify the differences in training and gradient amplification among the logsumexp, max, and mean-max experiments, and provide the closest feasible comparison under matched conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.