- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies multimodal retrieval with frozen generative models by appending a short sequence of trainable input embeddings shared by queries and candidate items. Only these embeddings are optimized with a contrastive loss, while the backbone and causal attention remain unchanged. Experiments show substantial improvements over unadapted and fixed-prompt representations, including cross-dataset transfer and strong results with a single trainable embedding. The paper also finds that independently trained guiding sequences can produce compatible query and item representations.
Critical strengths
-
Retrieval improves substantially over fixed prompting with very few trainable parameters. On Qwen3-VL-4B, tuning 25,600 parameters increases MMDocRAG MRR@20 from 52.09 to 70.79.
-
The cross-dataset transfer results are strong. MMDocRAG-trained embeddings improve ViDoSeek MRR@20 from 48.04 to 76.20 without target-dataset training or checkpoint selection.
-
The inclusion of recent evaluation datasets, such as MMDocRAG and ViDoSeek, is a strength of the experimental evaluation.
Critical weaknesses
-
The core method is standard suffix prompt tuning. Zhao et al. (2025) already train appended soft embeddings in a frozen causal model for contrastive retrieval, including single-token configurations. This paper cites that work, but does not establish a substantive methodological difference beyond applying it to multimodal inputs.
-
The retriever comparisons do not isolate the benefit of the adaptation method. The proposed method uses Qwen3-VL-4B, whereas the Nomic and ColQwen baselines use Qwen2.5-VL backbones of different sizes. The improvements over fixed prompting are clear, but the relative results cannot separate the effect of prompt tuning from differences in the pretrained models. A controlled comparison of prompt tuning and LoRA on the same backbone is essential, including testing the proposed method on Qwen2.5-VL.
-
Training-data comparability is not established. The proposed embeddings are trained on the MMDocRAG and SciMMIR training splits, but it is unclear whether the specialized retrievers are evaluated as released or additionally trained on these same splits. Table 1 acknowledges different training histories without specifying each baseline's adaptation data and budget. Without this information and a data-matched comparison, differences in supervision remain a confounder.
-
The baseline coverage is insufficient to establish how competitive the method is with current retrievers. VLM2Vec-V2 and GME are relevant omissions for ViDoSeek, where they have already been evaluated in the VLM2Vec-V2 paper. Recent public retrievers such as Qwen3-VL-Embedding are also useful comparison candidates. These models should be evaluated on the same candidate pools and metrics rather than compared through scores from different protocols.
-
The practical trade-off is evaluated mainly through trainable parameter counts. Appended soft tokens still require computation during encoding, whereas LoRA updates can be merged into the backbone without adding input positions. The paper appropriately distinguishes parameter count from computational cost, but does not measure encoding latency or throughput against a same-backbone merged-LoRA model. The size of any runtime overhead is therefore unknown.
-
Generalization across diverse document types, domains, and modality combinations is not sufficiently established. Broader evaluation on benchmarks such as ViDoRe (Faysse et al., 2025), MMEB (Jiang et al., 2025), and UMRB (Zhang et al., 2025) would provide stronger evidence.
[1] Faysse et al. (2025). ColPali: Efficient Document Retrieval with Vision Language Models. ICLR.
[2] Jiang et al. (2025). VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. ICLR.
[3] Zhang et al. (2025). Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models. CVPR.
Questions for authors
-
What distinguishes the method from Zhao et al.'s SPT beyond its application to multimodal inputs?
-
How do prompt tuning and LoRA compare on the same Qwen2.5-VL and Qwen3-VL backbones under matched training data, objectives, negative sampling, and training budgets?
-
Were the specialized retrievers fine-tuned on the MMDocRAG and SciMMIR training splits, or evaluated as released? Please clarify the training and validation data used for each model.
-
How does the method compare with strong recent retrievers such as VLM2Vec and GME on ViDoSeek using the same candidate pools and evaluation metrics?
-
Please report encoding latency, throughput, and peak training memory against same-backbone LoRA on identical hardware, with LoRA merged for inference.
-
Could you broaden the evaluation across document types, domains, and modality combinations by including additional benchmarks such as ViDoRe, MMEB, and UMRB?