- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to equip retrieval-oriented vision-language models with generative capabilities while preserving existing multimodal retrieval indices and minimizing retrieval representation drift caused by generative fine-tuning. The authors propose CALLOSUM, a three-stage framework for integrating retrieval and generation capabilities. First, it constructs a retrieval anchor surrogate within the generative model through representation distillation. Second, it uses singular value decomposition (SVD) to decompose generative parameter updates into an anchor-aligned shared component and a relatively independent private component, thereby mitigating interference between the two tasks. Finally, an input-adaptive, layer-wise gating mechanism dynamically adjusts the fusion weights of retrieval and generative updates. The experiments cover image-text retrieval, visual question answering, hallucination evaluation, and retrieval stability analysis. The results show that CALLOSUM improves generation performance while maintaining the original retrieval model's aggregate Recall@1, and reduces GPU memory usage and inference overhead compared with deploying two full models.
Critical strengths
-
The paper investigates the impact of generative fine-tuning on existing retrieval systems, rather than focusing solely on retrieval and generation performance. The authors point out that stable aggregate Recall@1 does not necessarily imply unchanged retrieval results. Accordingly, they further evaluate ranking correlation, Top-K candidate overlap, and changes in threshold-based decisions. This system-compatibility-oriented perspective is practically meaningful.
-
The authors compare various parameter-efficient fine-tuning and model merging methods, and conduct ablation studies to examine the contributions of individual components. CALLOSUM improves visual question answering performance while maintaining the original retrieval model's aggregate retrieval performance. Furthermore, the paper includes comparisons with FR-Head and Two-Path, reporting GPU memory usage, latency, and end-to-end task performance. These evaluations extend beyond isolated retrieval or generation metrics.
Critical weaknesses
-
I have some reservations about the motivation for adopting a unified model. I do not fully understand why retrieval and generation must be integrated into a single model, rather than simply using two independent models for the respective tasks. I am not entirely convinced that there is a compelling need for a unified model in real-world applications. In particular, considering the advantages of a two-model approach, such as task independence, easier maintenance, and avoiding interference with the existing retrieval system, I believe that the resource savings offered by model unification may not be sufficient to justify the additional complexity and loss of retrieval stability.
-
Second, the paper argues that generative fine-tuning changes retrieval representations and that rebuilding a large-scale index is costly, thereby motivating the integration of retrieval and generation capabilities while preserving the existing index. However, the authors do not adequately quantify the actual cost of re-encoding the corpus and rebuilding the index, nor do they evaluate retrieval performance when index rebuilding is allowed. For many retrieval systems, re-encoding the entire corpus can be performed as an offline batch process and may not necessarily incur prohibitive costs. More importantly, a model after generative fine-tuning may simply become incompatible with the original embedding space, while its actual retrieval capability may not have deteriorated and could even have improved. Without further analysis, the paper does not sufficiently justify why a complex anchor-preservation mechanism is necessary instead of directly updating the model and rebuilding the index.
-
Even if we accept the assumption that the existing retrieval index must remain unchanged, the paper still does not adequately explain why retrieval and generation capabilities need to be integrated through such a complex fusion mechanism. Since retrieval and generation tasks can usually be explicitly distinguished during inference, a natural and simpler alternative would be to maintain separate retrieval and generation LoRA adapters on a shared backbone and switch between them according to the current task. This approach could also avoid deploying two full models while reducing interference between task-specific updates. In particular, CALLOSUM's gating network uses prompt-format and modality information, while retrieval and visual question answering tasks in the experiments have clearly distinguishable input formats. Therefore, the gating mechanism may primarily be learning to identify task types rather than achieving deeper dynamic collaboration between the two capabilities. Without comparison against this simple yet competitive alternative, it is difficult to determine how much additional value the proposed complex fusion mechanism actually provides.
Questions for authors
Please refer to the concerns raised in the Critical Weaknesses section. I would particularly appreciate clarification on the practical necessity of a unified model, the justification for preserving the existing retrieval index, and the advantages of the proposed fusion mechanism over simpler task-specific LoRA switching.
2 comments
We hope the above addresses the concerns, and thus, the reviewer would consider raising their score. We are happy to answer any further question!
Response to jDck
We thank the reviewer for the comments. We follow the three points in your Critical Weaknesses section.
Weakness 1. Why use one model rather than separate retrieval and generation models?
From early vision-language foundation models to today's LLMs and VLMs, researchers have repeatedly joined retrieval with generation. CoCa (Yu et al., TMLR 2022) jointly trains retrieval embeddings and captioning, and BLIP (Li et al., ICML 2022) supports retrieval, captioning and VQA in one vision-language pretraining framework. GritLM (Muennighoff et al., 2024) brings the same idea to instruction-conditioned LLM embeddings and generation for text RAG. Recent VLM work now studies this as one serving workflow: SV-RAG (Chen et al., ICLR 2025) retrieves evidence from multi-page documents for question answering, URaG (Shi et al., AAAI 2026 Oral) uses early and later Qwen2.5-VL layers for retrieval and answering, and Hydra (Georgiou, 2026) combines ColBERT retrieval with autoregressive generation in one Qwen3.5 VLM. In Hydra's 4B setting, one-model serving reduces peak memory from 28.85 GB to 10.77 GB. This progression establishes a concrete need: visual RAG requires evidence retrieval and answer generation in the same request, and a unified model avoids keeping a second full model resident. Callosum adds generation to an existing service while retaining its corpus embeddings.
Our joint request searches an image collection, answers using the evidence and searches again using the interaction. In 2,000 such requests, Callosum reaches 79.9% answer accuracy and 62.9% complete-request success. Shuffling or removing the evidence lowers answer accuracy to 52.3% or 47.1% (supplementary Table 39).
Against Two-Path, we improve VQAv2 while using 48.4% less memory and reducing linked latency by 27.0%. Against FR-Head, we gain 1.5 VQAv2 points and raise complete-request success from 61.4% to 62.9% at an 8 ms latency increase.
Weakness 2. Why keep the existing index, and what happens if we rebuild it?
We test rebuilt galleries directly. Callosum retains its retrieval advantage.
Let A be the deployed anchor, M ordinary generation fine-tuning, C Callosum and S task-switched LoRA.
Rebuilding raises M from 60.0 to 62.5 and S from 62.8 to 63.5. Thus changing the embedding space explains part of the original loss. Callosum reaches 64.2 using the existing gallery, exceeding both rebuilt alternatives. Keeping the existing gallery delivers a quality benefit as well as avoiding corpus re-encoding.
The retrieval-only reference at 66.7 uses its own encoder for both queries and gallery. It defines a different index from the deployed anchor at 64.2. Callosum keeps the existing corpus vectors and reaches 85.6 VQAv2 at 64.2 COCO R@1.
Re-encoding cost. We will add the rebuilt-gallery results, the cited unified-model research and a corpus re-encoding cost breakdown to the camera-ready version. The cost breakdown will state corpus size, hardware, encoding time, index-building time and storage, and distinguish measured values from scale estimates.
Weakness 3. What do we gain over switching between task-specific LoRA adapters?
Callosum improves both task quality and retrieval stability over task switching.
We run the proposed two-LoRA alternative on one backbone. We use the same initialization, data, training steps, number of trainable parameters, cache access and validation selection rule. An explicit task label chooses the adapter.
Thus Callosum outperforms task switching on every reported measure.
We also compare the branch designs under the same gate. With task-specific layer weights, our shared and private branches add 1.0 COCO point and 0.5 VQAv2 points over standard adapters. With input features and a task label, they add 1.1 and 0.4 points. Input-dependent weights separately add 0.4 or 0.5 VQAv2 points, with COCO changes of -0.2 or -0.3 points. The branch gain holds under both routing choices. We will add these comparisons and settings to the camera-ready version.