acceptodds
Under review as a conference paper at ICLR 2027

Unlocking and Understanding the Retrieval Potential of Frozen Generative Models

Abstract

Generative models encode rich information about their inputs, yet their default hidden-state representations are often poorly suited for retrieval. We investigate the retrieval potential of frozen generative models and show that optimizing a few continuous guiding embeddings, with under 50K trainable parameters, yields effective retrieval representations without modifying backbone weights or causal attention. Experiments on three retrieval datasets demonstrate substantial improvements over frozen baselines, with performance approaching or exceeding specialized retrievers. Even a single guiding embedding, requiring only a few thousand parameters in total, retains competitive performance. The learned guiding embeddings also generalize well across datasets without further training. Beyond retrieval quality, our representation analysis reveals that distinct guiding embeddings learned in separate runs steer the same frozen backbone toward increasingly similar input-dependent representations. Queries and candidate answers encoded using guiding embeddings from different runs can be matched with little average performance loss, demonstrating functional compatibility beyond comparable retrieval scores. Together, these findings reveal substantial retrieval capability accessible through remarkably small adaptations to frozen generative models. This capability is not tied to a unique set of guiding embeddings. Distinct learned input parameterizations can induce closely aligned representations and compatible retrieval behavior within the same frozen backbone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Parameter-Efficient Fine-Tuning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qobfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multimodal retrieval with frozen generative models by appending a short sequence of trainable input embeddings shared by queries and candidate items. Only these embeddings are optimized with a contrastive loss, while the backbone and causal attention remain unchanged. Experiments show substantial improvements over unadapted and fixed-prompt representations, including cross-dataset transfer and strong results with a single trainable embedding. The paper also finds that independently trained guiding sequences can produce compatible query and item representations.

Critical strengths
  • Retrieval improves substantially over fixed prompting with very few trainable parameters. On Qwen3-VL-4B, tuning 25,600 parameters increases MMDocRAG MRR@20 from 52.09 to 70.79.

  • The cross-dataset transfer results are strong. MMDocRAG-trained embeddings improve ViDoSeek MRR@20 from 48.04 to 76.20 without target-dataset training or checkpoint selection.

  • The inclusion of recent evaluation datasets, such as MMDocRAG and ViDoSeek, is a strength of the experimental evaluation.

Critical weaknesses
  • The core method is standard suffix prompt tuning. Zhao et al. (2025) already train appended soft embeddings in a frozen causal model for contrastive retrieval, including single-token configurations. This paper cites that work, but does not establish a substantive methodological difference beyond applying it to multimodal inputs.

  • The retriever comparisons do not isolate the benefit of the adaptation method. The proposed method uses Qwen3-VL-4B, whereas the Nomic and ColQwen baselines use Qwen2.5-VL backbones of different sizes. The improvements over fixed prompting are clear, but the relative results cannot separate the effect of prompt tuning from differences in the pretrained models. A controlled comparison of prompt tuning and LoRA on the same backbone is essential, including testing the proposed method on Qwen2.5-VL.

  • Training-data comparability is not established. The proposed embeddings are trained on the MMDocRAG and SciMMIR training splits, but it is unclear whether the specialized retrievers are evaluated as released or additionally trained on these same splits. Table 1 acknowledges different training histories without specifying each baseline's adaptation data and budget. Without this information and a data-matched comparison, differences in supervision remain a confounder.

  • The baseline coverage is insufficient to establish how competitive the method is with current retrievers. VLM2Vec-V2 and GME are relevant omissions for ViDoSeek, where they have already been evaluated in the VLM2Vec-V2 paper. Recent public retrievers such as Qwen3-VL-Embedding are also useful comparison candidates. These models should be evaluated on the same candidate pools and metrics rather than compared through scores from different protocols.

  • The practical trade-off is evaluated mainly through trainable parameter counts. Appended soft tokens still require computation during encoding, whereas LoRA updates can be merged into the backbone without adding input positions. The paper appropriately distinguishes parameter count from computational cost, but does not measure encoding latency or throughput against a same-backbone merged-LoRA model. The size of any runtime overhead is therefore unknown.

  • Generalization across diverse document types, domains, and modality combinations is not sufficiently established. Broader evaluation on benchmarks such as ViDoRe (Faysse et al., 2025), MMEB (Jiang et al., 2025), and UMRB (Zhang et al., 2025) would provide stronger evidence.

[1] Faysse et al. (2025). ColPali: Efficient Document Retrieval with Vision Language Models. ICLR.

[2] Jiang et al. (2025). VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. ICLR.

[3] Zhang et al. (2025). Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models. CVPR.

Questions for authors
  • What distinguishes the method from Zhao et al.'s SPT beyond its application to multimodal inputs?

  • How do prompt tuning and LoRA compare on the same Qwen2.5-VL and Qwen3-VL backbones under matched training data, objectives, negative sampling, and training budgets?

  • Were the specialized retrievers fine-tuned on the MMDocRAG and SciMMIR training splits, or evaluated as released? Please clarify the training and validation data used for each model.

  • How does the method compare with strong recent retrievers such as VLM2Vec and GME on ViDoSeek using the same candidate pools and evaluation metrics?

  • Please report encoding latency, throughput, and peak training memory against same-backbone LoRA on identical hardware, with LoRA merged for inference.

  • Could you broaden the evaluation across document types, domains, and modality combinations by including additional benchmarks such as ViDoRe, MMEB, and UMRB?

Reviewer_6mExOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies retrieval from frozen multimodal generative models by learning a short sequence of continuous input embeddings. These embeddings are appended to queries and candidates, with the same sequence used for both. The model uses the final hidden state of the last token as the retrieval representation. Only the added embeddings are trained, using a contrastive objective. On Qwen3-VL-4B, ten embeddings (25,600 parameters) improve MRR@20 over the fixed summarization prompt from 52.09 to 70.79 on MMDocRAG and from 25.55 to 51.20 on SciMMIR. Gains over frozen controls appear across five backbones and three initialization strategies. A single embedding also gives strong results. The paper reports transfer to ViDoSeek and transfer in both directions between MMDocRAG and SciMMIR. The analysis examines agreement between representations produced by separately trained guiding sequences. At the selected checkpoints, using different runs for queries and candidates gives an average MRR@20 of 71.23, compared with 71.65 when both use the same run. The paper also reports similarities in representation structure across four backbones.

Critical strengths
  1. The improvements over fixed prompting are substantial and consistent across the tested backbones and initialization strategies. Qwen3.5-9B is a useful example: its R@1 barely changes with the fixed prompt, from 4.71 to 4.88, but exceeds 45 after input tuning with each initialization. The single-embedding results in Appendix F further show that useful adaptation is possible with only 2,560 trainable parameters. Together, these results give good evidence that fixed-prompt performance can underestimate a frozen model's retrieval potential.
  2. The cross-run matching experiment is a useful test of compatibility. Queries encoded with one run's guiding embeddings can be matched against candidates encoded with another's, without learning an alignment. The average MRR loss is only 0.42 points at the selected checkpoints, although individual pairings can lose more: V/V scores 73.21, while V/G scores 70.05. This tests whether the representations work together under the actual retrieval score, beyond simply showing that separate runs reach similar performance.
Critical weaknesses
  1. The comparison with specialized retrievers does not isolate the effect of the adaptation method. As the Table 1 caption acknowledges, the models differ in training history and visual budget. Their scores provide useful reference points, but the paper lacks a comparison with LoRA on the same backbone and training data, or with a small learned transformation of frozen fixed-prompt features. This matters because suffix prompt tuning with last-token readout for frozen causal text encoders was already studied by Zhao et al. (2025). I see the contribution here as mainly empirical. A controlled comparison would help establish what input tuning offers in this multimodal setting relative to these alternatives.
  2. The evaluation protocol leaves important details unresolved. Section 4 states that MMDocRAG uses the full dev_15 split for training and evaluation_15 for testing, while checkpoints are selected by validation performance. Appendix A.2 gives the selection rule and evaluation frequency, but does not identify the validation split. The paper also mentions two or three seeds for selected configurations without reporting their variation or how the displayed results were aggregated. These omissions matter especially for the claim of outperforming every specialized retriever: the best R@1 in Table 2 is 49.01, compared with 48.78 for the strongest baseline. The large gains over fixed prompting are clear in the reported results, but the smaller differences need a documented selection protocol and uncertainty estimates.
Questions for authors

Q1. Which data were used for checkpoint selection on MMDocRAG and SciMMIR? Please identify the validation splits and confirm whether they are disjoint from both training and test data. Q2. Could you compare input tuning with LoRA using the same backbone and training data? A learned per-dimension scaling of the frozen fixed-prompt representation would also be a useful control, since it can match the 2,560-parameter budget of the single-embedding setting. Q3. Which configurations were run with multiple seeds, and how were the reported scores obtained from those runs? Please report the mean and standard deviation for the main Qwen3-VL-4B results and the configuration supporting the claim of the best MMDocRAG performance. Q4. Please specify the MMDocRAG candidate pools, including whether they are shared or query-specific, their sizes, and the types of items included. How many SciMMIR training examples were actually used?

Reviewer_3oyeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper explored soft suffix-prompt tuning applied to multimodal retrieval task: appending tunable embeddings to the input queries/docs and fine-tune them with contrastive loss.

  • Significant retrieval performance improvement compared to frozen baselines. Experimented across three datasets and multiple LLM backbones.

Critical strengths
  • Parameter efficient: Interesting that a few added embeddings with a few thousands parameters could significantly improve the model performance.

  • Writing is clear and easy to follow. Experiments are conducted across multiple datasets and multiple backbones.

Critical weaknesses
  • Lack of novelty. The general idea of soft-prompt tuning has been extensively explored previously. The application of soft-prompt tuning on retrieval has been investigated by Zhao et al. 2025 (https://aclanthology.org/2025.repl4nlp-1.3/).

  • The claim is broader than experiments. Paper claims for general retrieval while experiments focused mainly on text-image tasks.

  • Comparison is not fair. Baselines like Colqwen, Colnomic, Nomic are not trained on the reported datasets, while the authors train their methods on the train set of the datasets.

  • Missing baselines with natural language prompt tuning approach (e.g., PromptEOL, DSPy, GEPA).

  • Datasets are very small (2000 - 16000 images), not sufficient to fully evaluate retrieval models which often have to deal with very large indexes in real world (million-scale).

Questions for authors
  1. How does the method differ from Zhao et al. (2025) beyond using vision-language backbones?

  2. Does the method work for text-only retrieval or page-level document benchmarks such as ViDoRe?

  3. How does it compare with baselines trained on the same data, e.g., LoRA on the same backbone?

  4. How does it compare with optimized natural-language prompts, e.g., PromptEOL, DSPy, GEPA?

  5. Does performance hold on million-scale indexes (e.g., 2M, 5M, 10M)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.