- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Bootstrapped Multimodal Embeddings, a framework combining self-generated document descriptions, reasoning-instruction-conditioned contrastive training, and self-mined hard-negative refinement. It aims to improve retrieval without external teachers during retrieval training or explicit reasoning-token generation at inference. The paper evaluates image/multimodal and video retrieval, and introduces EDIS-Noisy and WebQA-Noisy to assess robustness to input perturbations.
Critical strengths
- Practical framework. The method reuses the backbone’s capabilities while retaining single-vector retrieval without explicit reasoning generation at inference.
- Strong robustness results. Substantial gains on EDIS-Noisy and WebQA-Noisy demonstrate effectiveness under the evaluated perturbations.
- Informative ablations. Experiments examine training versus inference instructions, description quality, and annotation noise, providing useful insights into the design choices.
Critical weaknesses
- Insufficient evidence for internalized reasoning. Prompt-based improvements do not establish reasoning-specific computation. A Gen + Ret without Rea ablation is particularly important, since Rea alone reduces robustness in Table 9.
- Ambiguous robustness mechanism. Document-side perturbations modify only text, so gains may reflect reduced reliance on text rather than better multimodal integration. Image-only and text-only controls are needed.
- Uneven improvements. In the 300K VLM2Vec-V2 setting, gains are concentrated on EDIS, while several text-to-image benchmarks decline. Multiple-seed results would help assess reliability.
Questions for authors
- What evidence distinguishes internalized reasoning from general instruction conditioning? How does Gen + Ret without Rea perform in the 300K VLM2Vec-V2 setting?
- How do image-only and text-only baselines perform on the noisy benchmarks, and how are additional valid matches handled after query shortening?
- What are the total annotation, mining, and training costs, and do the gains persist against a compute-matched baseline?
- Are the improvements statistically stable across random seeds, particularly on benchmarks other than EDIS?
5 comments
Official Comment by Authors (Part 2/2)
Q2: Noisy retrieval controls and query-shortening evaluation
(1) Image-only and text-only inference on WebQA-Noisy
We evaluated the image-only and text-only inference variants on WebQA-Noisy.
| Model | SQ | NQ | SynQ | PD | ND | CD | SD | Avg. |
|---|---|---|---|---|---|---|---|---|
| VLM2Vec-V2 (Baseline)-imageonly | 58.7 | 63.0 | 64.8 | 64.4 | 64.4 | 64.4 | 64.3 | 54.2 |
| VLM2Vec-V2 (Baseline)-textonly | 82.1 | 83.7 | 83.8 | 78.9 | 81.0 | 44.3 | NA | 75.6* |
| VLM2Vec-V2 (Baseline) | 87.9 | 89.0 | 89.1 | 87.0 | 87.0 | 60.1 | 10.1 | 72.9 |
| Ours-imageonly | 62.2 | 67.6 | 67.0 | 68.2 | 68.2 | 68.2 | 68.2 | 67.1 |
| Ours-textonly | 82.7 | 84.6 | 84.9 | 79.7 | 80.8 | 44.1 | NA | 76.1* |
| Ours | 88.0 | 89.7 | 89.0 | 87.5 | 88.0 | 74.0 | 46.6 | 80.4 |
(Note: All document text is shuffled in the shuffled-text subset, making text-only inference uninformative. We report these results as NA. * denotes the average of the first six scores.)
Our model outperforms the baseline on average in both single-modality settings. Full multimodal inference also outperforms both single-modality variants on average, indicating that our model can use image and text information together effectively.
(2) Evaluation protocol for query shortening
For the query-shortening subset, we retain the ground-truth document associated with each original query. We aim to simulate real-world situations where users provide incomplete queries and to assess whether the model can still retrieve the original target document. A shortened query may also be relevant to other documents. Because the evaluation focuses on recovering the original target, we retain the original relevance labels and count only the original ground-truth document as a hit.
Q3: Annotation, mining, and training costs
We used eight H20 GPUs. The description generation rate was 0.249 samples/s per process. Annotating 300k images with 32 parallel processes took 10 h 27 min. We mined negatives separately for each training dataset, taking 9 min for VisualNews, 16 min for MSCOCO, 23 min for VisDial, and 4 min for WebQA, for a total of 52 min. We trained with a batch size of 24 × 8 at 4.16 s/step. Stage 1 took approximately 3.5 h for 3k steps, and stage 2 took 1.2 h for 1k steps, for a total training time of 4.7 h.
Some other methods use more training time. For example, GME in Tab. 1 uses 8M training samples. At the caption-free baseline training speed of 1.83 s/step, a single pass over 8M samples would take approximately 21.1 h, excluding data annotation. This exceeds our combined annotation and training time, and our method achieves better performance.
Q4: Stability across training seeds and inference reproducibility
We trained our model with different random seeds and obtained the following results.
| Train setting | VisualNews | MSCOCO | VisDial | Wiki | EDIS | WebQA | Avg |
|---|---|---|---|---|---|---|---|
| VLM2Vec-V2 (Baseline) | 77.5 | 76.7 | 81.5 | 69.4 | 86.0 | 89.1 | 80.0 |
| Ours (seed=42) | 77.2 | 76.5 | 79.2 | 70.7 | 94.1 | 90.2 | 81.3 |
| Ours (seed=2026) | 77.2 | 75.7 | 80.0 | 70.6 | 93.5 | 90.2 | 81.2 |
Across the two training seeds, our model performs consistently on the first four text-to-image benchmarks and maintains its advantage over the baseline on the last two text-to-multimodal benchmarks. Its average performance is also stable.
Our inference framework is based on ReMatch [3], and we do not fix an inference random seed. We repeated the evaluation several times on different machines and obtained identical results. The evaluation extracts embeddings and ranks results by similarity without stochastic sampling, with dropout disabled at inference. Recall@1 remained unchanged across repeated runs, indicating that any numerical variation did not affect the top-ranked retrieval results.
References:
[1] MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval. 2025.09
[2] Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift. DMLR 2024.
[3] ReMatch: Boosting Representation through Matching for Multimodal Retrieval. CVPR 2026.
please examine rebuttal response
Hello Reviewer 8XnD,
As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal, comment, and decide if it changes your thoughts regarding your current evaluation.
Thanks.
Official Comment by Authors (Part 1/2)
We sincerely thank you for your time and valuable feedback and suggestions. Please find our responses below.
W1 & Q1: Evidence for internalized reasoning and Gen + Ret ablation
(1) Behavioral evidence for internalized reasoning
At retrieval time, our multimodal model performs a single forward pass and uses the final hidden state as the embedding. This process yields no explicit chain-of-thought trace. To assess reasoning ability through retrieval performance, we conducted additional experiments on MR²-Bench [1], a benchmark focused on reasoning-intensive retrieval.
Our model outperforms VLM2Vec-V2 in all three domains. These results provide behavioral evidence that our embeddings better support reasoning-intensive retrieval.
(2) Gen + Ret without Rea
To extend the ablation in Tab. 9, we evaluated Gen + Ret in the 300K VLM2Vec-V2 setting.
The full model outperforms Gen + Ret on EDIS-Noisy, WebQA, and WebQA-Noisy, showing that Rea contributes to retrieval performance. The result on clean EDIS may reflect a lower need for reasoning when retrieving clean multimodal documents. On EDIS, Rea provides a gain when the multimodal documents are noisy, suggesting that reasoning helps the model handle noisy data.
W2: Robustness to image noise and image-only and text-only controls
(1) Retrieval with noisy document images
Following the image-noise retrieval experiment in [2], we applied four noise types (Gaussian, Shot, Impulse, and Speckle) to document images in EDIS, each at five severity levels.
As noise severity increases, retrieval performance declines for both models. Our full model consistently outperforms the baseline across all configurations, demonstrating robustness to image noise.
(2) Image-only and text-only inference on clean benchmarks
We also report retrieval results for the image-only and text-only inference variants on the clean benchmarks.
Our model outperforms the baseline in every corresponding modality setting on the clean benchmarks. Our full model also outperforms both of its single-modality variants, indicating that it makes effective use of multimodal data.
W3: Performance across training and evaluation settings
In the 300K VLM2Vec-V2 setting, the baseline is strong because both training and evaluation use Text-to-Image (T2I) retrieval. Our method targets Text-to-Multimodal (T2M) retrieval, so evaluating it on T2I introduces a mismatch between training and inference. Our model performs slightly below the strong baseline in this T2I setting and shows substantial gains on T2M, where training and inference are aligned. In the other three settings, where the trained baselines are weaker, our method substantially outperforms them on T2I, showing consistent gains across those settings. We report results across training seeds in our response to Q4 below.