acceptodds
Under review as a conference paper at ICLR 2027

Self-Bootstrapped Multimodal Embeddings for Robust Text-to-Multimodal Retrieval

Abstract

Text-to-multimodal retrieval aims to retrieve documents containing both images and text in response to textual queries. Existing multimodal embedding models typically rely on large-scale heterogeneous training corpora, external teachers, or auxiliary data-generation models, while task-specific text-to-multimodal training data remain scarce. We present Self-Bootstrapped Multimodal Embeddings, a framework that exploits the backbone MLLM’s own generation, reasoning, and retrieval capabilities to improve multimodal representations without external teachers or auxiliary annotation models during retrieval training. First, the backbone generates detailed descriptions for document images, self-annotating existing text-to-image corpora and converting them into text-enriched multimodal retrieval data. Second, reasoning instruction conditioning during contrastive training elicits and internalizes the model’s native reasoning-oriented computation into single-vector embeddings, requiring neither explicit reasoning traces nor additional reasoning tokens at inference. Finally, the trained retriever uses its own retrieval capability to identify highly ranked non-relevant documents and adopts them as self-mined hard negatives for targeted representation refinement. The resulting framework improves the average performance of both Qwen3-VL-Instruct and VLM2Vec-V2 on text-to-multimodal and text-to-video retrieval. With only 300K training pairs, our VLM2Vec-V2-based model achieves an average Recall@1 of 81.3% across six text-to-multimodal benchmarks—the best among models fine-tuned with no more than 300K task-specific pairs—and reaches 94.1% on EDIS. We further introduce EDIS-Noisy and WebQA-Noisy to simulate realistic retrieval scenarios. Without noise-specific training, our model achieves average Recall@1 scores of 91.4% and 80.4%, respectively, outperforming both Qwen3-VL-Embedding and WeMM-Embedding. These results demonstrate that self-bootstrapping the backbone’s existing capabilities provides an effective approach to data-efficient and robust multimodal retrieval.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal embedding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8XnDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Bootstrapped Multimodal Embeddings, a framework combining self-generated document descriptions, reasoning-instruction-conditioned contrastive training, and self-mined hard-negative refinement. It aims to improve retrieval without external teachers during retrieval training or explicit reasoning-token generation at inference. The paper evaluates image/multimodal and video retrieval, and introduces EDIS-Noisy and WebQA-Noisy to assess robustness to input perturbations.

Critical strengths
  1. Practical framework. The method reuses the backbone’s capabilities while retaining single-vector retrieval without explicit reasoning generation at inference.
  2. Strong robustness results. Substantial gains on EDIS-Noisy and WebQA-Noisy demonstrate effectiveness under the evaluated perturbations.
  3. Informative ablations. Experiments examine training versus inference instructions, description quality, and annotation noise, providing useful insights into the design choices.
Critical weaknesses
  1. Insufficient evidence for internalized reasoning. Prompt-based improvements do not establish reasoning-specific computation. A Gen + Ret without Rea ablation is particularly important, since Rea alone reduces robustness in Table 9.
  2. Ambiguous robustness mechanism. Document-side perturbations modify only text, so gains may reflect reduced reliance on text rather than better multimodal integration. Image-only and text-only controls are needed.
  3. Uneven improvements. In the 300K VLM2Vec-V2 setting, gains are concentrated on EDIS, while several text-to-image benchmarks decline. Multiple-seed results would help assess reliability.
Questions for authors
  1. What evidence distinguishes internalized reasoning from general instruction conditioning? How does Gen + Ret without Rea perform in the 300K VLM2Vec-V2 setting?
  2. How do image-only and text-only baselines perform on the noisy benchmarks, and how are additional valid matches handled after query shortening?
  3. What are the total annotation, mining, and training costs, and do the gains persist against a compute-matched baseline?
  4. Are the improvements statistically stable across random seeds, particularly on benchmarks other than EDIS?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.

W1 & Q1: Evidence for internalized reasoning and Gen + Ret ablation

(1) Behavioral evidence for internalized reasoning
At retrieval time, our multimodal model performs a single forward pass and uses the final hidden state as the embedding. This process yields no explicit chain-of-thought trace. To assess reasoning ability through retrieval performance, we conducted additional experiments on MR²-Bench [1], a benchmark focused on reasoning-intensive retrieval.

ModelEconomics R@1Math R@1Nature R@1Avg. R@1
VLM2Vec-V2 (Baseline)33.333.4913.5816.80
Ours35.716.9813.8318.84

Our model outperforms VLM2Vec-V2 in all three domains. These results provide behavioral evidence that our embeddings better support reasoning-intensive retrieval.

(2) Gen + Ret without Rea
To extend the ablation in Tab. 9, we evaluated Gen + Ret in the 300K VLM2Vec-V2 setting.

ModelEDISEDIS-NoisyWebQAWebQA-Noisy
Baseline86.077.789.172.9
Gen + Ret (Ours)94.589.589.278.1
Rea + Gen + Ret (Ours Full)94.191.490.280.4

The full model outperforms Gen + Ret on EDIS-Noisy, WebQA, and WebQA-Noisy, showing that Rea contributes to retrieval performance. The result on clean EDIS may reflect a lower need for reasoning when retrieving clean multimodal documents. On EDIS, Rea provides a gain when the multimodal documents are noisy, suggesting that reasoning helps the model handle noisy data.

W2: Robustness to image noise and image-only and text-only controls

(1) Retrieval with noisy document images
Following the image-noise retrieval experiment in [2], we applied four noise types (Gaussian, Shot, Impulse, and Speckle) to document images in EDIS, each at five severity levels.

Image NoiseSeverityVLM2Vec-V2 (Baseline)Ours
Gaussian176.889.8
272.384.6
364.174.8
455.460.5
548.050.3
Shot176.489.0
271.984.5
364.874.6
455.058.7
549.952.0
Impulse175.788.7
271.483.4
368.678.6
458.464.4
550.852.1
Speckle178.991.2
277.289.7
369.881.1
466.177.5
560.267.0

As noise severity increases, retrieval performance declines for both models. Our full model consistently outperforms the baseline across all configurations, demonstrating robustness to image noise.

(2) Image-only and text-only inference on clean benchmarks
We also report retrieval results for the image-only and text-only inference variants on the clean benchmarks.

ModelEDISEDIS-ImageonlyEDIS-textonlyWebQAWebQA-ImageonlyWebQA-textonly
VLM2Vec-V2 (Baseline)86.090.139.989.163.083.6
Ours94.190.442.190.268.284.1

Our model outperforms the baseline in every corresponding modality setting on the clean benchmarks. Our full model also outperforms both of its single-modality variants, indicating that it makes effective use of multimodal data.

W3: Performance across training and evaluation settings

In the 300K VLM2Vec-V2 setting, the baseline is strong because both training and evaluation use Text-to-Image (T2I) retrieval. Our method targets Text-to-Multimodal (T2M) retrieval, so evaluating it on T2I introduces a mismatch between training and inference. Our model performs slightly below the strong baseline in this T2I setting and shows substantial gains on T2M, where training and inference are aligned. In the other three settings, where the trained baselines are weaker, our method substantially outperforms them on T2I, showing consistent gains across those settings. We report results across training seeds in our response to Q4 below.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

Q2: Noisy retrieval controls and query-shortening evaluation

(1) Image-only and text-only inference on WebQA-Noisy
We evaluated the image-only and text-only inference variants on WebQA-Noisy.

ModelSQNQSynQPDNDCDSDAvg.
VLM2Vec-V2 (Baseline)-imageonly58.763.064.864.464.464.464.354.2
VLM2Vec-V2 (Baseline)-textonly82.183.783.878.981.044.3NA75.6*
VLM2Vec-V2 (Baseline)87.989.089.187.087.060.110.172.9
Ours-imageonly62.267.667.068.268.268.268.267.1
Ours-textonly82.784.684.979.780.844.1NA76.1*
Ours88.089.789.087.588.074.046.680.4

(Note: All document text is shuffled in the shuffled-text subset, making text-only inference uninformative. We report these results as NA. * denotes the average of the first six scores.)

Our model outperforms the baseline on average in both single-modality settings. Full multimodal inference also outperforms both single-modality variants on average, indicating that our model can use image and text information together effectively.

(2) Evaluation protocol for query shortening
For the query-shortening subset, we retain the ground-truth document associated with each original query. We aim to simulate real-world situations where users provide incomplete queries and to assess whether the model can still retrieve the original target document. A shortened query may also be relevant to other documents. Because the evaluation focuses on recovering the original target, we retain the original relevance labels and count only the original ground-truth document as a hit.

Q3: Annotation, mining, and training costs

We used eight H20 GPUs. The description generation rate was 0.249 samples/s per process. Annotating 300k images with 32 parallel processes took 10 h 27 min. We mined negatives separately for each training dataset, taking 9 min for VisualNews, 16 min for MSCOCO, 23 min for VisDial, and 4 min for WebQA, for a total of 52 min. We trained with a batch size of 24 × 8 at 4.16 s/step. Stage 1 took approximately 3.5 h for 3k steps, and stage 2 took 1.2 h for 1k steps, for a total training time of 4.7 h.

Some other methods use more training time. For example, GME in Tab. 1 uses 8M training samples. At the caption-free baseline training speed of 1.83 s/step, a single pass over 8M samples would take approximately 21.1 h, excluding data annotation. This exceeds our combined annotation and training time, and our method achieves better performance.

Q4: Stability across training seeds and inference reproducibility

We trained our model with different random seeds and obtained the following results.

Train settingVisualNewsMSCOCOVisDialWikiEDISWebQAAvg
VLM2Vec-V2 (Baseline)77.576.781.569.486.089.180.0
Ours (seed=42)77.276.579.270.794.190.281.3
Ours (seed=2026)77.275.780.070.693.590.281.2

Across the two training seeds, our model performs consistently on the first four text-to-image benchmarks and maintains its advantage over the baseline on the last two text-to-multimodal benchmarks. Its average performance is also stable.

Our inference framework is based on ReMatch [3], and we do not fix an inference random seed. We repeated the evaluation several times on different machines and obtained identical results. The evaluation extracts embeddings and ranks results by similarity without stochastic sampling, with dropout disabled at inference. Recall@1 remained unchanged across repeated runs, indicating that any numerical variation did not affect the top-ranked retrieval results.

References:
[1] MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval. 2025.09
[2] Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift. DMLR 2024.
[3] ReMatch: Boosting Representation through Matching for Multimodal Retrieval. CVPR 2026.

Area_Chair_z27GOpenReview ↗

please examine rebuttal response

Hello Reviewer 8XnD,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal, comment, and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Reviewer_QKpiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a self-bootstrapping framework for text-to-multimodal retrieval. It enriches training documents with MLLM-generated image descriptions, uses reasoning instructions during contrastive training, and mines hard negatives from the retriever’s own results. Experiments across text-to-multimodal, text-to-video, and noisy retrieval benchmarks show gains over the tested baselines.

Critical strengths
  • Uses pretrained MLLM capabilities to reduce reliance on external supervision.
  • Evaluates two backbones across multiple retrieval tasks.
  • Ablations clarify the contributions of the three components.
  • Noisy benchmarks extend evaluation beyond clean inputs.
Critical weaknesses
  • Reasoning-instruction training accounts for most of the gains; the added value of the full framework is less clear.
  • The evidence that the model internalizes reasoning is indirect.
  • Generated descriptions and self-mined negatives may introduce errors or false-negative supervision.
  • The synthetic noise settings may not reflect real-world retrieval noise.
  • The paper does not analyze inference speed. Since the method relies on MLLM-generated descriptions and reasoning-oriented training, reporting query and document encoding latency would help assess its practical retrieval cost.
Questions for authors
  • What evidence distinguishes reasoning gains from prompt-specific adaptation?
  • How often do generated descriptions or mined negatives introduce incorrect supervision?
  • What are the additional compute costs of description generation and negative mining?
  • How representative are the noisy benchmarks of naturally occurring retrieval noise?
  • Can the authors clarify when each component helps or hurts?
  • Please report query and document encoding latency, including whether reasoning-oriented instructions add inference overhead. This would clarify the method’s practical cost relative to the retrieval gains.
6 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.

W1: Contribution of each component

The ablation study in Tab. 6 shows that Rea is highly effective in both IID and OOD settings. The framework's Gen and Ret components also contribute substantially in the OOD setting. In the OOD evaluation in Tab. 6, training with Rea alone raises the score from 50.4 to 62.6. Adding Gen raises it from 62.6 to 65.2, and then adding Ret raises it from 65.2 to 67.1. These gains illustrate the effectiveness of each component.

W2 & Q1: Behavioral Evidence for Internalized Reasoning

At retrieval time, our multimodal model performs a single forward pass and uses the final hidden state as the embedding. This process yields no explicit chain-of-thought trace. To assess reasoning ability through retrieval performance, we conducted additional experiments on MR²-Bench [1], a benchmark focused on reasoning-intensive retrieval.

ModelEconomics R@1Math R@1Nature R@1Avg. R@1
VLM2Vec-V2 (Baseline)33.333.4913.5816.80
Ours35.716.9813.8318.84

Our model outperforms VLM2Vec-V2 in all three domains. These results provide behavioral evidence that our embeddings better support reasoning-intensive retrieval.

W3 & Q2: Caption errors and clarification on false-negative supervision

(1) Generated captions may contain minor errors. During training, the model jointly encodes images and text, so it can tolerate such errors by drawing on both inputs. Tab. 8 shows the effect of injecting additional noise into the generated captions used for training. Our method still clearly outperforms the baseline with this additional noise.

(2) Our method does not introduce false-negative supervision. We use the original annotated training data, in which each query is paired with a single document. For each query in a batch, we treat its paired document as the sole positive and all other documents as negatives. The mined hard examples therefore come from the negatives under this labeling scheme.

W4 & Q4: Connection to real-world retrieval noise

We provide examples from prior work that uses real-world noise similar to the perturbations in our paper. PC² [2] and NCR [3] retain image–text mismatches that naturally arise during web data collection, which are similar to our conflicting and shuffled documents. OHRBench [4] retains OCR errors and the loss of document structure encountered in real-world settings, which are similar to our noisy and partial documents. These similarities suggest that our noisy benchmark captures some forms of real-world retrieval noise.

W5 & Q6: Training speed and encoding latency

Reasoning-oriented instructions add no inference overhead. The standard and reasoning-oriented instructions produce text inputs of similar token lengths. To encode a query or document, the model performs one forward pass and uses the final hidden state as the embedding. Inference latency is therefore comparable to that of the baseline. The baseline and our method trained without descriptions both take approximately 1.83 s/step, while our method trained with descriptions takes approximately 4.16 s/step.

We also report encoding latency. Encoding takes 20.9 ms for a text query, 20.4 ms for a text-only document, 180.0 ms for an image-only document, and 180.3 ms for a document containing both an image and text. Our model and the baseline share the same architecture and both use one forward pass at inference. Their average encoding latencies are comparable.

Q3: Data annotation cost and mining time

To annotate the data before training, a single process generates descriptions at 0.249 samples/s. Annotating 300k images with 32 parallel processes took 10 h 27 min. We mined negatives separately for each training dataset, taking 9 min for VisualNews, 16 min for MSCOCO, 23 min for VisDial, and 4 min for WebQA, for a total of 52 min.

Q5: When each component helps or hurts

As shown in the ablation study in Tab. 6, Rea and Gen each improve both IID and OOD performance when used alone. Using Ret alone slightly reduces IID performance by 0.1%. This may be because the IID training and test distributions are similar. The baseline already performs well on this distribution, so additional hard-negative mining may offer limited gains. Combining Rea and Gen further improves both IID and OOD performance. Adding Ret to obtain the full method further improves OOD performance, while IID performance decreases slightly by 0.1%. As with Ret alone, the limited benefit of additional hard-negative mining in the IID setting may help explain this decrease.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

References:
[1] MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval. 2025.09
[2] PC²: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval, ACMMM 2024.
[3] Learning with Noisy Correspondence for Cross-modal Matching. NIPS 2021.
[4] OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation. ICCV 2025.

Area_Chair_z27GOpenReview ↗

please examine rebuttal response

Hello Reviewer QKpi,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Reviewer_jrpUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a training recipe that combines image descriptions generated by the backbone, reasoning-oriented training instructions, and hard negatives mined by the trained retriever. It produces single-vector embeddings without generating reasoning text at inference. Experiments cover six text-to-multimodal and five text-to-video benchmarks using two backbones. The paper also introduces noisy versions of EDIS and WebQA. The strongest model reaches 81.3 average Recall@1 on the six main benchmarks with about 300K task-specific training pairs.

Critical strengths
  1. The recipe has practical value and avoids extra reasoning generation at inference. On Qwen3-VL-Instruct with 100K training pairs, average Recall@1 improves from 52.7 to 67.4. The stronger VLM2Vec-V2 setting improves from 80.0 to 81.3 with 300K pairs. The gain is not uniform across tasks, but the combination offers a useful way to improve retrieval using existing backbone capabilities.

  2. The ablations support several concrete design choices. Table 7 separates training-time and inference-time prompting; Table 8 tests short captions, detailed descriptions, and corrupted descriptions; Table 9 shows that description enrichment, rather than the reasoning prompt alone, drives most robustness gains. These experiments help explain which parts of the recipe are useful

Critical weaknesses
  1. The proposed reasoning mechanism is not demonstrated. Better retrieval margins and gains from step-by-step prompts show that the training instruction matters, but do not establish that reasoning is activated and internalized. The alternative prompts in Table 7 and Appendix A are useful controls, yet they still measure retrieval outcomes rather than the claimed mechanism. This matters for novelty: image recaptioning and model-based hard-negative mining are established ideas, so the distinct contribution is mainly the effective combination and training instruction. Relevant precedents include Recap-DataComp and ANCE, although their training settings differ from this paper.

  2. The noisy benchmarks do not fully establish robust use of both modalities. Conflicting and shuffled document text are evaluated with the original relevance labels. A model that relies more heavily on images could perform well under these changes without handling useful textual evidence better. The manual checks and component ablations are helpful, but image-only and text-only inference controls are needed to distinguish these explanations. This is important because robustness is a central contribution.

Questions for authors
  1. Which results distinguish internalized reasoning from a beneficial change in the training prompt? Would the authors narrow this claim to prompt-conditioned representation learning if no direct evidence is available?
  2. How does the proposed model compare with an image-only inference variant on the clean, conflicting-text, and shuffled-text subsets? Does it still use text effectively when the query requires information unavailable in the image?
4 comments

We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.

W1 & Q1: Reasoning-intensive retrieval and related work

(1) Evidence from MR²-Bench
At retrieval time, our multimodal model performs a single forward pass and uses the final hidden state as the embedding. This process yields no explicit chain-of-thought trace. To assess reasoning ability through retrieval performance, we conducted additional experiments on MR²-Bench [1], a benchmark focused on reasoning-intensive retrieval.

ModelEconomics R@1Math R@1Nature R@1Avg. R@1
VLM2Vec-V2 (Baseline)33.333.4913.5816.80
Ours35.716.9813.8318.84

Our model outperforms VLM2Vec-V2 in all three domains. These results provide behavioral evidence that our embeddings better support reasoning-intensive retrieval.

(2) Comparison with Recap-DataComp [2] and ANCE [3]

Our method differs substantially from Recap-DataComp. Recap-DataComp trains a separate 8B captioning model to recaption DataComp-1B, then compares CLIP's retrieval performance using the original and regenerated captions. Our method does not require an additional captioning model. We use the MLLM itself to expand text-to-image training data into text-to-multimodal training data. We then train the same MLLM on the resulting data. We investigate how to use the MLLM's own capabilities and what kinds of text it should generate to better support multimodal retrieval. ANCE focuses on hard-negative mining for text alone, while our method considers both text and images when mining hard negatives. Our method is a unified, self-bootstrapped framework for multimodal retrieval that can be applied to any MLLM without relying on external models.

W2 & Q2: Image-only and text-only inference controls

(1) Clean and noisy retrieval results

We report additional retrieval results for the image-only and text-only inference variants on the clean benchmarks (EDIS and WebQA).

ModelEDISEDIS-ImageonlyEDIS-textonlyWebQAWebQA-ImageonlyWebQA-textonly
VLM2Vec-V2 (Baseline)86.090.139.989.163.083.6
Ours94.190.442.190.268.284.1

On the clean benchmarks, our model outperforms the baseline in each corresponding modality setting.

We also evaluate the image-only and text-only inference variants on WebQA-Noisy.

ModelSQNQSynQPDNDCDSDAvg.
VLM2Vec-V2 (Baseline)-imageonly58.763.064.864.464.464.464.354.2
VLM2Vec-V2 (Baseline)-textonly82.183.783.878.981.044.3NA75.6*
VLM2Vec-V2 (Baseline)87.989.089.187.087.060.110.172.9
Ours-imageonly62.267.667.068.268.268.268.267.1
Ours-textonly82.784.684.979.780.844.1NA76.1*
Ours88.089.789.087.588.074.046.680.4

(Note: All document text is shuffled in the shuffled-text subset, making text-only inference meaningless. We therefore report the text-only results as NA. * denotes the average of the first six scores.)

Our model outperforms the baseline on average in both the image-only and text-only settings. Multimodal inference also outperforms both single-modality variants on average, indicating that our method can benefit from using image and text information together.

(2) Effect of document text on retrieval

Our model outperforms its image-only inference variant on the clean and conflicting-text subsets. On the shuffled-text subset, it scores below the image-only variant because shuffling disrupts the correspondence between the document text and the image. With both models evaluated in the same multimodal setting, our model still outperforms the baseline on the shuffled-text subset, indicating greater robustness to this perturbation.

To address the question of "Does it still use text effectively when the query requires information unavailable in the image?", we compare our model with its image-only variant. Our model outperforms this variant on both EDIS and WebQA, indicating that it can use information from the supplementary text to improve retrieval.

References:
[1] MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval. 2025.09
[2] What If We Recaption Billions of Web Images with LLaMA-3? 2024.06
[3] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. 2020.10

Area_Chair_z27GOpenReview ↗

please examine rebuttal response

Hello Reviewer jrpU,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Thank you for the additional experiments and for addressing most of my concerns in your response. I am keeping my rating at 3.

We sincerely thank you for your continued support and thoughtful feedback. Your constructive comments have helped us strengthen our experiments and improve the clarity of our paper.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their careful evaluation and constructive feedback, as well as their recognition of the paper’s strengths. Below, we briefly summarize this feedback and present the key experiments and clarifications added during the rebuttal.

Positive feedback

  1. Practical, self-bootstrapped framework. Reviewers recognized the value of using the backbone’s own capabilities to improve retrieval without external models or explicit reasoning generation at inference.
  2. Broad evaluation and strong robustness. Reviewers acknowledged our evaluation across multiple retrieval tasks and two backbones, highlighting substantial gains on EDIS-Noisy and WebQA-Noisy.
  3. Informative ablations. Reviewers appreciated the experiments examining training versus inference instructions, description quality, and annotation noise, which help explain the framework’s design choices.

Key additional experiments

  1. Reasoning-intensive retrieval. On MR²-Bench, our model outperforms VLM2Vec-V2 across Economics, Math, and Nature, improving average Recall@1 from 16.80 to 18.84. This provides behavioral evidence for improved reasoning-intensive retrieval.
  2. Image-only and text-only controls. We added single-modality inference controls on EDIS, WebQA, and WebQA-Noisy. Full multimodal inference outperforms both single-modality variants on the clean benchmarks, supporting the model’s ability to benefit from both image and text information.
  3. Robustness to image noise. We evaluated four image-noise types at five severity levels on EDIS. Our model consistently outperforms the baseline across all 20 configurations, extending the robustness evaluation beyond textual perturbations.
  4. Additional component ablation. Compared with Gen + Ret, the full framework improves Recall@1 on EDIS-Noisy from 89.5 to 91.4, WebQA from 89.2 to 90.2, and WebQA-Noisy from 78.1 to 80.4.

Thank you for the opportunity to strengthen our work. We sincerely appreciate the valuable time and effort of the Area Chair and all reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.