MERIT: TOWARD RELIABLE GENERALIST– SPECIALIST COLLABORATION FOR MEDICAL VISUAL QUESTION ANSWERING

Reliable integration of specialist evidence is important for extending medical vision–language models (VLMs) beyond the capabilities of any single receiver, yet the heterogeneous and sometimes conflicting outputs of classifiers, segmenters, retrievers, and domain-specific generators make this difficult. Existing methods often mix multiple sources in a shared context, entangling supportive and contradictory signals before their reliability can be assessed. We introduce MERIT, a training-free framework toward reliable incorporation of specialist evidence into a frozen medical VLM at inference time. Rather than directly fusing expert outputs, MERIT preserves each specialist's native observation and feeds it to a separate source-isolated branch of the same receiver, together with a no-expert anchor branch. Because all branches share the same committed answer prefix, heterogeneous evidence is compared through its effect on the receiver's next-token distribution. Adaptive BARD forms a candidate revision from centered expert-induced residuals and accepts it only when sufficient isolated source groups prefer the candidate to the anchor; the chosen token is then synchronized across branches. This design separates evidence acquisition, influence estimation, and answer-time commitment without learning cross-expert projections or updating model parameters. We evaluate MERIT across medical VQA, multidisciplinary multimodal reasoning, and report-generation benchmarks using two representative frozen medical VLMs (i.e., HuatuoGPT-Vision and LLaVA-Med). MERIT consistently improves greedy decoding across all evaluated receiver–benchmark pairs. Ablation studies further indicate that source-isolated, receiver-mediated integration provides a general strategy for reusing heterogeneous medical expertise without additional training.