acceptodds
Under review as a conference paper at ICLR 2027

MERIT: TOWARD RELIABLE GENERALIST– SPECIALIST COLLABORATION FOR MEDICAL VISUAL QUESTION ANSWERING

Abstract

Reliable integration of specialist evidence is important for extending medical vision–language models (VLMs) beyond the capabilities of any single receiver, yet the heterogeneous and sometimes conflicting outputs of classifiers, segmenters, retrievers, and domain-specific generators make this difficult. Existing methods often mix multiple sources in a shared context, entangling supportive and contradictory signals before their reliability can be assessed. We introduce MERIT, a training-free framework toward reliable incorporation of specialist evidence into a frozen medical VLM at inference time. Rather than directly fusing expert outputs, MERIT preserves each specialist's native observation and feeds it to a separate source-isolated branch of the same receiver, together with a no-expert anchor branch. Because all branches share the same committed answer prefix, heterogeneous evidence is compared through its effect on the receiver's next-token distribution. Adaptive BARD forms a candidate revision from centered expert-induced residuals and accepts it only when sufficient isolated source groups prefer the candidate to the anchor; the chosen token is then synchronized across branches. This design separates evidence acquisition, influence estimation, and answer-time commitment without learning cross-expert projections or updating model parameters. We evaluate MERIT across medical VQA, multidisciplinary multimodal reasoning, and report-generation benchmarks using two representative frozen medical VLMs (i.e., HuatuoGPT-Vision and LLaVA-Med). MERIT consistently improves greedy decoding across all evaluated receiver–benchmark pairs. Ablation studies further indicate that source-isolated, receiver-mediated integration provides a general strategy for reusing heterogeneous medical expertise without additional training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Vision-Language Models, Generalist-Specialist Collaboration, Training-Free Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NgAHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

MERIT is a training-free framework that conditions isolated branches of a frozen medical VLM on specialist outputs alongside a no-expert anchor. Adaptive BARD proposes tokens via the geometric median of centered log-probability residuals, accepting revisions only when sufficient branch support is present. MERIT improves greedy decoding with two receivers on five benchmarks, as well as on MIMIC-CXR.

Critical strengths
  1. Comparing heterogeneous specialist evidence through its effect on a shared frozen receiver avoids learned projections, making the method training-free and directly applicable to any receiver.

  2. MERIT never underperforms greedy decoding across ten settings, with notable gains on LLaVA-Med. The ablation further reveals that naive multi-expert fusion substantially degrades medical VLMs.

Critical weaknesses
  1. The paper never specifies how segmentation masks are converted into text, so whether location, area, or tubular structures such as vessels are faithfully conveyed remains unclear.

  2. Both receivers are limited by current standards; effectiveness on stronger models such as Lingshu-7B or Hulu-Med-7B, and superiority over Table 1 baselines, remain untested.

  3. Report generation is evaluated solely with NLG metrics; clinically oriented metrics such as GREEN or RaTEScore are absent, leaving clinical quality unverified.

  4. It is unclear how fixed-label classifiers and segmenters apply to VQA-RAD and MMMU, what should be segmented, and whether TorchXRayVision and BiomedParse actually contribute.

Questions for authors
  1. The paper does not explain how masks produced by the segmentation model are converted into textual descriptions. Can information such as the location and area of a mask be accurately conveyed in text? How would tubular structures such as blood vessels be described? This is important for whether the receiver model can correctly interpret the output of the segmentation expert.

  2. Both LLaVA-Med and HuatuoGPT-Vision used in the experiments have relatively limited capabilities. Would the method remain effective on more advanced models such as Lingshu-7B and Hulu-Med-7B? Would it still outperform the comparison methods in Table 1?

  3. For the report generation task, only natural language generation metrics are used for evaluation. Have the authors considered metrics designed specifically for report generation, such as GREEN and RaTEScore, which assess the quality of generated reports more reliably?

  4. How are the classification and segmentation models in the expert pool applied across different downstream tasks? For example, the questions and imaging modalities in VQA-RAD and MMMU differ entirely. Can TorchXRayVision be applied to MMMU at all? What should the segmentation model segment in this context? Do these two expert models actually contribute on these benchmarks?

5 comments
AuthorsOpenReview ↗

Response to Reviewer NgAH (Part 1 of 3)

Thank you for focusing on mask transport and contemporary receivers. We clarify the dual-channel segmentation interface and report Lingshu/Hulu-Med evaluations.

[W1/Q1] How Are Segmentation Masks Represented?

Q1: The paper does not explain how masks produced by the segmentation model are converted into textual descriptions. Can information such as the location and area of a mask be accurately conveyed in text? How would tubular structures such as blood vessels be described? This is important for whether the receiver model can correctly interpret the output of the segmentation expert.

A1. We have added a dedicated segmentation schematic in the revised manuscript: Appendix C, Figure S1, "Segmentation-conditioned evidence branch." Together with Section 3.1 and Eq. (S1), it explains how predicted masks reach the frozen receiver. Segmentation is not reduced to text: a compatible receiver can use semantic and spatial evidence simultaneously within a single isolated source branch.

Figure S1: From BiomedParse to Adaptive BARD. BiomedParse predicts modality-compatible soft masks for question-relevant anatomical targets. The figure traces two complementary paths from the same masks:

Table N1. Semantic and spatial segmentation channels (see Figure S1).

ChannelEvidence transportReceiver input
SemanticPredicted label, source, and mask-derived geometry are serializedStructured text in the specialist prompt
SpatialSoft-mask coverage is aligned to the receiver's padded visual-patch gridPooled and reinjected projected visual-patch features

Mask-aware visual transport. As illustrated by the lower panel of Figure S1, the spatial path uses the receiver's original square-padding transform and averages soft-mask coverage at each visual patch, without binarizing the mask.

Let mkjm_{kj} denote region kk's coverage of patch jj, and let hjh_j denote the receiver's existing projected patch feature. The parameter-free update in Eq. (S1) is:

rkj=mkj∑ℓmkℓr_{kj} = \frac{m_{kj}}{\sum_{\ell} m_{k\ell}}

ek=∑jrkjhje_k = \sum_j r_{kj}h_j

hj′=hj+∑kwkmkjek1+∑kwkmkjh_{j}^{\prime} = \frac{h_j + \sum_k w_k m_{kj}e_k}{1 + \sum_k w_k m_{kj}}

The mask selects and pools regional information already encoded by the frozen receiver. Zero-coverage patches remain unchanged, the original image remains available, and no learned cross-model projection is introduced.

Location, area, and tubular structures. The semantic path exposes labels and coarse geometry such as location and mask coverage, while the spatial path retains shape-conditioned regional features at visual-patch resolution. This provides a more informative interface for elongated structures than a text-only bounding box, although detailed vessel branching is still limited by patch resolution.

Source grouping and decoding. Both paths from one BiomedParse observation—including multiple anatomical masks—are grouped as one specialist source for BARD. Their effect is evaluated by an isolated branch of the same frozen receiver under the shared answer prefix. Adaptive BARD then checks source support before accepting a candidate token.

Figure S1 makes the native-evidence interface concrete: MERIT preserves both the semantic meaning and spatial support of segmentation, translates them into receiver-compatible observations without training, and integrates them through its general source-isolated, anchor-protected decoding mechanism.

AuthorsOpenReview ↗

Response to Reviewer NgAH (Part 2 of 3)

[W2/Q2] Stronger Medical Receivers and Specialist Capacity

Q2:Both LLaVA-Med and HuatuoGPT-Vision used in the experiments have relatively limited capabilities. Would the method remain effective on more advanced models such as Lingshu-7B and Hulu-Med-7B? Would it still outperform the comparison methods in Table 1?

A2. We extended the receiver evaluation to Lingshu-7B and Hulu-Med-7B on the complete SLAKE and VQA-RAD cohorts.

Table N2. Stronger medical VLM comparisons.

MethodLingshu SLAKELingshu VQA-RADHulu-Med SLAKEHulu-Med VQA-RAD
Greedy82.2561.1464.1468.32
VCD81.9459.9563.2567.85
ICD81.8760.3862.9966.30
DoLa78.8050.8155.5651.43
MedRAG65.6944.0752.7755.18
MERIT82.6963.9470.6070.21
Δ vs Greedy (pp)+0.44+2.80+6.46+1.89

MERIT achieves the highest reported score in each of the four settings. This demonstrates positive transfer to two stronger receivers without training the receiver or a new fusion module.

Mechanistic explanation: Each frozen receiver constructs its own anchor and source-conditioned residuals under a synchronized prefix. The same MERIT inference rule can therefore operate with stronger receivers, making the collaboration framework naturally modular.

Additional retrieval-specialist capacity comparison. We evaluated MERIT with MedEmbed-Small and MedEmbed-Base using LLaVA-Med on SLAKE.

Table N3. MedEmbed retrieval encoder ablation.

MERIT retrieval encoderSLAKE (%)
MedEmbed-Small36.15
MedEmbed-Base36.34
Δ (pp)+0.19

Replacing MedEmbed-Small with MedEmbed-Base improves MERIT by +0.19 pp on SLAKE. These results show that upgrading the retrieval specialist can benefit the existing receiver and BARD decoder without retraining the framework.

The Lingshu/Hulu-Med experiments establish gains with stronger receiver backbones, while MedEmbed demonstrates a second, complementary path: improving the specialist itself also raises MERIT performance. Both results support the framework's modular extensibility.

[W3/Q3] Clinical Factuality Metrics for MIMIC-CXR

Q3:For the report generation task, only natural language generation metrics are used for evaluation. Have the authors considered metrics designed specifically for report generation, such as GREEN and RaTEScore, which assess the quality of generated reports more reliably?

A3. We expanded the original MIMIC-CXR comparison with clinical-content metrics: RadGraph Simple-F1, RadGraph Partial-F1, RadGraph Complete-F1, RaTEScore, and GREEN alongside BLEU-4, ROUGE-L, and METEOR.

Table N4. Clinical report-generation metrics (%, ↑ higher is better).

B4: BLEU-4; RL: ROUGE-L; MET: METEOR; RG-S/P/C: RadGraph Simple/Partial/Complete; RaTE: RaTEScore; GREEN reported ×100.

MethodB4↑RL↑MET↑RG-S↑RG-P↑RG-C↑RaTE↑GREEN↑
Greedy1.0212.879.906.015.464.4243.093.48
VCD1.0212.479.525.564.903.3639.822.17
ICD1.2113.0310.375.974.983.6941.593.97
DoLa1.0112.839.845.935.374.3543.053.44
PAI1.0612.859.645.595.073.8342.162.76
AvisC1.4813.0913.266.585.954.4943.382.88
VISTA1.0013.169.476.506.044.9842.943.01
AGLA1.1813.6910.896.986.445.3243.573.24
OPERA1.4113.8012.735.765.384.4541.051.55
MERIT2.5317.1114.8217.6615.8911.7950.9018.29

MERIT has the highest reported score in all eight columns. Relative to the strongest non-MERIT row for each clinical metric, its descriptive advantages are +10.68 pp (RadGraph Simple), +9.45 pp (Partial), +6.47 pp (Complete), and +7.33 pp (RaTEScore). GREEN also increases to 18.29 (×100), a gain of 14.32 points over the best comparator.

RadGraph evaluates medically meaningful entities and relations, RaTEScore measures clinical report-content alignment, and GREEN adds a report-specific assessment. The consistent improvement across both metric families directly addresses the reviewer's request for clinically oriented evaluation.

AuthorsOpenReview ↗

Response to Reviewer NgAH (Part 3 of 3)

We expand radiology-report evaluation with clinical metrics and clarify how classification and segmentation experts apply to different imaging tasks.

[W4/Q4] Applicability and Contribution of Classification/Segmentation Experts

Q4:How are the classification and segmentation models in the expert pool applied across different downstream tasks? For example, the questions and imaging modalities in VQA-RAD and MMMU differ entirely. Can TorchXRayVision be applied to MMMU at all? What should the segmentation model segment in this context? Do these two expert models actually contribute on these benchmarks?

A4. MERIT selects classification and segmentation specialists according to the modality, task, question, and anatomical context of each medical input. This capability-aware routing enables expert collaboration on the five-subject, 1,752-question MMMU medical subset.

Classification: TorchXRayVision provides findings on supported radiographs. In the MMMU medical subset, TorchXRayVision Findings is selected for 177 questions.

Segmentation: BiomedParse selects question-relevant labels from registered modality–anatomy target groups. For example, a spleen question can use the spleen target within the CT-Abdomen group. The revised §3.1 and Appendix C, Figure S1 show the resulting soft masks entering two paths: structured labels and geometry in the semantic prompt, and mask-aligned projected visual features in the parameter-free spatial interface. Both paths condition one source-isolated receiver branch, whose candidate support is evaluated by Adaptive BARD.

Actual specialist invocations across five evaluated datasets (MMMU denotes the medical subset).

Table N5. Expert invocations across datasets.

SpecialistPMC-VQAPathVQASLAKEVQA-RADMMMU-Medical
BiomedParse33,3995,8022,0944511,730
CheXagent1,457248740174234
CONCH3,6753,50300348
TorchXRayVision Findings1,300227637141177
MedCPT33,4306,7192,0944511,752

MMMU-Medical: specialist selection and accuracy improvements.

Table N6. Accuracy improvements among selected MMMU-Medical questions.

Selected specialistSelected questionsAccuracy improvement (pp)
BiomedParse1,730+10.35
CheXagent234+4.27
CONCH348+10.06
TorchXRayVision Findings177+5.08
MedCPT1,752+10.50

Accuracy increases on the selected-question cohorts are +10.35 pp for BiomedParse, +10.06 pp for CONCH, +4.27 pp for CheXagent, +5.08 pp for TorchXRayVision Findings, and +10.50 pp for MedCPT. These results demonstrate how heterogeneous specialist capabilities are associated with improvements under MERIT's receiver-mediated collaboration mechanism.

VQA-RAD and MMMU-Medical source-removal controls.

Table N7. Specialist-removal results on VQA-RAD and MMMU-Medical.

ConfigurationVQA-RAD (%)MERIT gain (pp)MMMU-Medical (%)MERIT gain (pp)
MERIT53.96—22.73—
Remove BiomedParse53.14+0.8222.50+0.23
Remove XRV Anatomy53.63+0.3322.67+0.06

Removing BiomedParse lowers the full-model scores by 0.82 pp on VQA-RAD and 0.23 pp on MMMU-Medical; removing XRV Anatomy lowers them by 0.33 pp and 0.06 pp, respectively. The consistent positive contributions across both benchmarks reinforce the value of anatomical segmentation and radiographic anatomy within MERIT's source-isolated, anchored collaboration framework.

We hope these responses and the revised analyses address the main concerns and welcome further discussion before the decision.

Reviewer_FtVTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MERIT, a training-free framework for integrating heterogeneous specialist evidence into frozen medical vision-language models. MERIT uses isolated receiver branches to measure how each specialist influences the generalist's next-token distribution. It further introduces Adaptive BARD, which aggregates expert-induced residuals using a geometric median and applies an anchor-based acceptance rule to determine whether to revise the generalist's prediction. Experiments across multiple medical VQA benchmarks and report generation tasks demonstrate improvements over greedy decoding and baselines.

Critical strengths
  1. This paper proposes an important and practical problem: integrating multiple specialist evidence into medical VLMs, where additional evidence does not necessarily improve performance.

  2. The proposed framework is novel and interesting. It provides a unified way to incorporate different types of specialist evidence to the generalist models.

  3. The experiments cover multiple benchmarks and two medical VLM backbones. The results show consistent improvements over greedy decoding, suggesting the potential effectiveness of the proposed approach.

Critical weaknesses
  1. The paper does not report the performance of the generalist conditioned on each individual specialist. It is therefore unclear whether the improvements come from effective multi-expert collaboration or primarily from one particularly strong specialist.

  2. The paper does not clearly report which specialists are used or how many are activated for each benchmark. Since the method depends on the number of expert groups, these details are important for reproducibility and understanding its practical applicability.

  3. The baseline comparisons may not be entirely fair regarding computational cost. MERIT requires multiple expert-conditioned decoding branches and additional specialist inference, introducing higher inference-time costs than the baselines. However, it only brings very small improvement on some datasets compared to baselines (e.g. PMC-VQA or PathVQA).

  4. Different medical VLMs may interpret the same specialist evidence differently, potentially affecting the reliability of expert-induced residuals. Although the paper evaluates two generalists, it does not sufficiently investigate their consistency in interpreting the same evidence.

Questions for authors
  1. If a single specialist already provides most of the gains, whether there is necessity of the proposed multi-expert framework?

  2. How are specialists selected for different benchmarks? Could the authors report the specific specialists used and the number of activated expert groups for each task? How would expert selection work in real-world deployment?

  3. Could the authors provide inference latency or computational cost analyses? More importantly, how does MERIT compare with compute-matched test-time scaling methods, such as self-consistency with majority voting (using some temperature > 0)? Would its advantages remain under comparable inference budgets?

  4. How consistently do different medical VLMs interpret the same specialist evidence? Could the authors investigate cross-model consistency and sensitivity to different evidence representations?

6 comments
AuthorsOpenReview ↗

Response to Reviewer FtVT (Part 1 of 2)

[W1/Q1] Is Multi-Expert Collaboration Necessary?

Q1:If a single specialist already provides most of the gains, whether there is necessity of the proposed multi-expert framework?

A1. We compare LLaVA-Med-7B with individual specialists on SLAKE and VQA-RAD.

Single-specialist results on SLAKE and VQA-RAD (%).

Table F1. Single-specialist comparison.

MethodSLAKEVQA-RAD
Generalist34.6848.47
MERIT39.4953.96
BiomedCLIP Anatomy only34.5448.25
BiomedParse only35.5046.68
CheXagent only39.0853.03
TorchXRayVision Anatomy only35.2048.87
TorchXRayVision Findings only37.5746.48
MedCPT only27.3833.84

CheXagent is the strongest single specialist (39.08%/53.03%), +4.40/+4.56 pp over Generalist. MERIT further gains +0.41/+0.93 pp, achieving the highest scores.

The diverse single-specialist results motivate selective use of complementary expertise through a frozen receiver and anchored support rule.

MERIT combines task-native evidence, source-isolated frozen-receiver branches, and Adaptive BARD commitment into a reusable, training-free collaboration framework.

Its central contribution is generalist–specialist collaboration, not a larger expert ensemble.

[W2/Q2] How Are Experts Selected and Activated?

How are specialists selected for different benchmarks? Could the authors report the specific specialists used and the number of activated expert groups for each task? How would expert selection work in real-world deployment?

A2. MERIT follows a capability-aware routing pipeline. Routing determines which expertise is applicable; Adaptive BARD determines how that evidence can improve the Generalist's prediction.

Input-dependent routing pipeline.

  1. Modality: Obtain an image type from acquisition metadata or frozen-model image-type inference.
  2. Capability matching: Filter the registry by supported modality, task, question type, anatomical scope, and required inputs.
  3. Native execution: Invoke compatible classifiers, segmenters, image–text encoders, radiology generators, and retrievers.
  4. Evidence preparation: Serialize retained observations, group related capabilities by source dependence, and condition separate branches of the same receiver. The revised Appendix C, Figure S1 illustrates how semantic and spatial outputs from one segmenter remain in one BARD source group.

Actual expert invocations across five evaluated datasets (MMMU denotes the medical subset).

Table F2. Expert invocations across datasets.

SpecialistPMC-VQAPathVQASLAKEVQA-RADMMMU-Medical
BiomedParse33,3995,8022,0944511,730
CheXagent1,457248740174234
CONCH3,6753,50300348
TorchXRayVision Findings1,300227637141177
MedCPT33,4306,7192,0944511,752

Huatuo active specialist groups. The no-expert anchor is excluded from the counts.

Table F3. Huatuo active specialist groups.

Dataset1 group2 groups3 groups4 groups5 groups≥2 groups≥3 groups
SLAKE1,182172426908912740
VQA-RAD2185921720233174

For Huatuo, 43.55% of SLAKE and 51.66% of VQA-RAD questions activate at least two specialist groups, highlighting MERIT's use of heterogeneous evidence.

Deployment: The capability registry accepts imaging metadata and enforces an expert-call budget, with no reliance on reference diagnoses. When only one group is available, the default BARD rule keeps the anchor; with multiple groups it commits revisions supported by the isolated receiver branches.

Together, the invocation and activation statistics support the routing → isolation → anchored commitment pipeline.

AuthorsOpenReview ↗

Response to Reviewer FtVT (Part 2 of 2)

We quantify the inference cost, compare against Self-Consistency under a receiver-time budget, and investigate how different receivers interpret the same evidence.

[W3/Q3] Inference Cost and Self-Consistency

Could the authors provide inference latency or computational cost analyses? More importantly, how does MERIT compare with compute-matched test-time scaling methods, such as self-consistency with majority voting (using some temperature > 0)? Would its advantages remain under comparable inference budgets?

A3. We provide complete inference timings and a positive-temperature Self-Consistency comparison to assess how MERIT uses test-time computation.

HuatuoGPT-Vision-7B on an RTX 4090.

Table F4. Inference cost.

MethodSeconds/sample ↓Relative cost ↓
Greedy0.1401.00×
VCD0.3072.19×
ICD0.2882.05×
DoLa0.1761.25×
PAI0.2131.52×
AvisC0.3072.19×
VISTA0.4823.43×
MERIT1.43910.25×

Self-Consistency diagnostic subset. Self-Consistency uses temperature 0.7, top-p 0.9, and majority voting. For each applicable question/repetition, the largest feasible number of samples from {1, 2, 4, 8} is chosen under the cached-evidence BARD receiver-time budget.

Table F5. Budget-constrained Self-Consistency diagnostic.

ReceiverSelf-ConsistencyBARDΔ (pp)
HuatuoGPT-Vision52.2455.52+3.28
LLaVA-Med52.9356.28+3.35

BARD achieves +3.28 pp on HuatuoGPT-Vision and +3.35 pp on LLaVA-Med over Self-Consistency within the diagnostic receiver-time budget.

Self-Consistency spends test-time computation on repeated stochastic answers, whereas MERIT spends it on independent expert-conditioned token distributions and anchor verification. The diagnostic results favor BARD and demonstrate a competitive evidence-centered approach to test-time scaling.

[W4/Q4] Cross-Model Agreement and Evidence Interfaces

How consistently do different medical VLMs interpret the same specialist evidence? Could the authors investigate cross-model consistency and sensitivity to different evidence representations?

A4. We directly examine the response of two frozen medical VLMs to the same specialist evidence. The resulting differences motivate MERIT's receiver-specific evaluation rule.

Table F6. Cross-receiver evidence agreement.

Cross-receiver diagnosticAgreementCohen's κ
Evidence-conditioned Yes/No preference50.89%0.202
Evidence-induced preference shift50.89%0.114

The limited agreement shows that identical observations need not induce identical token preferences. Hence MERIT measures each expert's contribution inside the current receiver, relative to its own no-expert branch:

Δk,t=zk,t−z0,t\Delta_{k,t} = z_{k,t} - z_{0,t}

rk,t(v)=Δk,t(v)−∑u∈Vp0,t(u)Δk,t(u)r_{k,t}(v) = \Delta_{k,t}(v) - \sum_{u \in V} p_{0,t}(u)\Delta_{k,t}(u)

Evidence representation (revised Section 3.1; Appendix C, Figure S1). The new diagram shows the two-channel segmentation interface: labels, provenance, and mask geometry enter the semantic prompt, while soft-mask coverage can condition the receiver's projected visual patches through a parameter-free spatial operator. Both channels remain one specialist source group, not two votes. Classification outputs retain native semantics and retrieval passages retain provenance.

The cross-model results explain why specialist evidence must be interpreted through the current receiver. MERIT converts heterogeneous outputs into comparable receiver-side effects without learning a common confidence projection.

We hope these responses and the revised analyses address the main concerns and welcome further discussion before the decision.

Thanks for the authors' response and the additional experiments. After reviewing the response to W3/Q3, I still have two concerns:

  1. Limited performance gains relative to computational cost. I don't think the authors have directly addressed my original concern that MERIT brings only marginal improvements on certain datasets (e.g., PMC-VQA and PathVQA). Given that MERIT requires 10.25× the inference time of greedy decoding and is substantially more expensive than other baselines, it remains unclear whether such small improvements justify the additional computational cost. In other words, if MERIT does not provide meaningful performance gains on these datasets, what motivates its use over simpler and more efficient alternatives?

  2. Limited improvements on pathology tasks. On PathVQA, MERIT underperforms the strongest baseline for both model backbones. Does this suggest potential limitations in its applicability to pathology tasks? I understand that a method may not generalize equally well across all medical domains, but I believe the authors should acknowledge potential domain-specific limitations when making broader generalizability claims.

Reviewer_cFXFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MERIT, a training-free framework for integrating heterogeneous specialist evidence into frozen medical vision-language models (VLMs). Instead of combining specialist outputs directly, MERIT processes evidence from classifiers, segmenters, retrievers, and generators through separate branches of the same receiver model. It introduces Adaptive BARD, a decoding strategy that aggregates expert-induced residuals using a geometric median and accepts token revisions only when sufficient independent source groups support them over the no-expert prediction. The authors evaluate MERIT using HuatuoGPT-Vision and LLaVA-Med across medical VQA, multimodal reasoning, and chest X-ray report generation benchmarks. The results show improvements over greedy decoding across all evaluated settings, with ablation studies highlighting the importance of source isolation and conservative token commitment.

Critical strengths
  1. A useful formulation for combining heterogeneous medical expertise. MERIT compares specialist evidence through separate branches of the same frozen receiver, using a shared answer prefix. This accommodates classifiers, segmenters, retrievers, and generators without retraining them or requiring compatible output representations. The authors also appropriately acknowledge that agreement between sources does not guarantee medical correctness.

  2. Encouraging initial results and meaningful integration ablations. MERIT numerically improves over greedy decoding in all ten receiver–benchmark combinations. The shared-context and isolated-aggregation comparisons in Table 2 provide useful evidence that conservative token commitment matters beyond simply supplying more specialist information. The report-generation experiment suggests potential applicability beyond short answers. These observations are based on the submitted manuscript.

Critical weaknesses
  1. The incremental contribution and relevance to current models are insufficiently established. RobustRAG already uses isolated next-token predictions, a shared generated prefix, and fallback to an unassisted model [1]; CCD incorporates medical specialist signals during decoding [2]. MERIT acknowledges these works, but comparisons against an adapted RobustRAG baseline and CCD would clarify its added value. Furthermore, Table 2 lacks an arithmetic-mean proposal with the same acceptance gate, leaving the benefit of geometric-median aggregation unresolved. Evaluation also relies on two older receivers. Testing MERIT on at least one contemporary receiver [3–7], under matched evaluation protocols, would better establish its significance and generality.

  2. The experimental reporting does not yet adequately support reliability and clinical faithfulness. Important details are missing, including dataset splits and sample counts, precise VQA scoring, prompts, baseline hyperparameters, specialist-generation settings, and retrieval leakage controls. Consequently, reproducibility and comparison fairness are difficult to assess. Tables 1 and 2 provide no uncertainty estimates, while the error bars in Figure 1 are undefined. This matters particularly for small HuatuoGPT-Vision gains, such as 0.07 percentage points on PMC-VQA and 0.24 on VQA-RAD. Finally, there are no controlled tests of corrupted or correlated specialists, and report generation is evaluated only through BLEU, ROUGE, and METEOR. These results do not establish that clinically important factual errors or hallucinations are reduced.

[1] Xiang, C., et al. Certifiably Robust RAG against Retrieval Corruption. SaTML, 2026.

[2] Zhang, X., et al. CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding. Findings of ACL, 2026.

[3] Ossowski, T., et al. OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning. CVPR, 2026.

[4] Dai, W., et al. QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training. NeurIPS, 2025.

[5] Dai, D., et al. QoQ-Med3: A Multimodal Reasoning Foundation Model for Clinical Analysis. npj Digital Medicine, 2026.

[6] Huang, X., et al. MedVLThinker: Simple Baselines for Multimodal Medical Reasoning. ML4H, 2025.

[7] Huang, X., et al. Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs. ICLR, 2026.

[8] Chen, J., et al. HuatuoGPT-3: RL-Only Domain Adaptation from Base Models. arXiv, 2026.

Questions for authors
  1. Which component produces the gains? Does geometric-median aggregation outperform arithmetic averaging when both use the identical acceptance gate and specialist evidence? How do results and inference costs compare with relevant prior methods?
  2. Can the evaluation protocol be made reproducible? Please provide splits, sample counts, answer extraction/scoring, checkpoint versions, prompts, baseline settings, and retrieval leakage controls. MERIT explicitly uses argmax token selection; do specialist generators or any baselines sample, and, if so, what temperatures, top-p values, and seeds are used?
  3. How robust are the reported improvements? Please define Figure 1's error bars and provide paired confidence intervals for the main comparisons. What fraction of examples has at least two active source groups? Does "MMMU" mean the full benchmark or a medical subset, and which specialists account for its substantial improvement?
  4. Does the method improve factual reliability? How often does MERIT correct an incorrect answer versus change a correct answer into an incorrect one? What happens with misleading or correlated specialist outputs? Do report-generation improvements persist under clinical factuality metrics, such as RadGraph-F1, or expert assessment?
  5. How should Figure 4's source labels be interpreted? The figure labels whole sources as "Rejected" or "Adopted", whereas the algorithm accepts or rejects candidate tokens. Could the authors provide the active source groups and token-level support margins that justify these labels?
7 comments
AuthorsOpenReview ↗

Response to Reviewer cFXF (Part 1 of 5)

Thank you for highlighting the need to establish MERIT's incremental contribution and reproducibility. We address the matched-gate ablation, existing baselines, and evaluation details below

[W1/Q1] Component Contributions and Related Methods

Q1:Which component produces the gains? Does geometric-median aggregation outperform arithmetic averaging when both use the identical acceptance gate and specialist evidence? How do results and inference costs compare with relevant prior methods?

A1. MERIT combines source-isolated evidence evaluation, geometric-median proposals, and anchored token commitment. We analyze each module under matched specialist evidence and directly compare against related methods

Component ablation (%). H/L denote HuatuoGPT-Vision/LLaVA-Med

Table C1. Component ablation

MethodH-SLAKEH-VQA-RADL-SLAKEL-VQA-RAD
Generalist54.6962.8234.6848.47
Joint-all49.3144.7932.2845.44
Isolated Mean48.0847.2332.9343.52
Isolated Geo48.0047.9533.1243.74
Mean + Support57.0561.9536.1150.33
Adaptive BARD57.1663.0639.4953.96

The contrast between direct/shared-context fusion and the support-enabled variants demonstrates the importance of receiver-mediated evidence isolation and anchored commitment. With identical specialist evidence and gate, Adaptive BARD improves over Mean + Support by 0.11, 1.11, 3.38, and 3.63 pp. The geometric median strengthens proposal construction; anchored verification governs when a well-supported expert-induced revision is committed

RobustRAG comparison (%), matching manuscript Appendix A.2.

Table C2. RobustRAG comparison

ReceiverDatasetGeneralistRobustRAGAdaptive BARDΔ vs RobustRAG (pp)
HuatuoSLAKE54.6948.4757.16+8.69
HuatuoVQA-RAD62.8247.6163.06+15.45
LLaVA-MedSLAKE34.6833.1639.49+6.33
LLaVA-MedVQA-RAD48.4744.5053.96+9.46

RobustRAG isolates contexts before aggregation. MERIT instead compares centered next-token residuals through the same frozen multimodal receiver, then tests candidate–anchor support before committing a revision

Comparison with CCD on chest-X-ray questions (%). We follow CCD's intended radiographic scope and evaluate the same chest-X-ray portions of SLAKE and VQA-RAD

Table C3.Chest-X-ray CCD comparison

ReceiverDatasetGeneralistCCDMERIT
HuatuoGPT-Vision-7BSLAKE45.9840.4055.74
HuatuoGPT-Vision-7BVQA-RAD59.3858.9860.09
LLaVA-Med-7BSLAKE31.0134.5644.37
LLaVA-Med-7BVQA-RAD54.6053.5168.52

MERIT ranks first in all four chest-X-ray comparisons, exceeding CCD by 15.34, 1.11, 9.81, and 15.01 pp. The results demonstrate the value of MERIT's general heterogeneous-expert interface for radiology image understanding

Contemporary receiver check (full SLAKE and VQA-RAD test sets)

Table C4.Contemporary receiver comparison

MethodLingshu SLAKELingshu VQA-RADHulu-Med SLAKEHulu-Med VQA-RAD
Greedy82.2561.1464.1468.32
MERIT82.6963.9470.6070.21
Δ (pp)+0.44+2.80+6.46+1.89

The positive gains with Lingshu-7B and Hulu-Med-7B additionally address the applicability of MERIT to more capable, contemporary receivers

Inference cost: HuatuoGPT-Vision-7B, one RTX 4090

Table C5. Inference cost

MethodSeconds/sample ↓Relative cost ↓
Greedy0.1401.00×
VCD0.3072.19×
ICD0.2882.05×
DoLa0.1761.25×
PAI0.2131.52×
AvisC0.3072.19×
VISTA0.4823.43×
MERIT1.43910.25×

The additional compute evaluates separate expert-conditioned decisions under a shared prefix. MERIT uses inference computation to verify evidence-supported revisions, yielding the accuracy gains reported above

AuthorsOpenReview ↗

Response to Reviewer cFXF (Part 2 of 5)

[W2/Q2] Reproducibility and Sampling

Q2:Can the evaluation protocol be made reproducible? Please provide splits, sample counts, answer extraction/scoring, checkpoint versions, prompts, baseline settings, and retrieval leakage controls. MERIT explicitly uses argmax token selection; do specialist generators or any baselines sample, and, if so, what temperatures, top-p values, and seeds are used?

A2. We provide the exact evaluation scopes, scoring rules, receiver checkpoints, generation settings, and retrieval procedure used in MERIT.

Evaluation manifests.

Table C6. Evaluation manifests.

DatasetEvaluation scopeQuestions
PMC-VQATEST, multiple choice33,430
PathVQATEST, short VQA6,719
SLAKEOriginal TEST, English and Chinese2,094
VQA-RADComplete frozen TEST451
MMMU (medical subset)All five medical subjects1,752

Closed-ended VQA uses normalized strict answer correctness; open-ended VQA uses reference-token recall. The overall score is the sample mean of per-question results; multiple-choice extraction follows the task-specific evaluator.

The frozen receivers are FreedomIntelligence/HuatuoGPT-Vision-7B and microsoft/llava-med-v1.5-mistral-7b. The short-answer protocol prompts closed questions with “Please answer Yes or No.” and open questions with “Give only the short answer. Do not explain.” The revised §4.2 and Appendix D specify receiver checkpoints, preprocessing, prompt configurations, retrieval controls, and Figure 1(a) bootstrap settings.

Table C7. Deterministic and specialist decoding.

ComponentInference ruleAutoregressive sampling
MERIT receiver / Adaptive BARDArgmax with anchor support checkingDisabled
CheXagent observationGreedy, one beam, at most 64 tokensDisabled
Classification and segmentationNative deterministic predictionsN/A
MedCPTCandidate retrieval and rerankingN/A

MERIT and CheXagent use greedy/argmax decoding with sampling disabled. Adaptive BARD uses at most 12 stabilized Weiszfeld iterations (tolerance 10−610^{-6}, distance floor 10−1210^{-12}). The additional Self-Consistency experiment uses temperature 0.7, top-p 0.9, and three seeded repetitions.

The frozen MedCPT-PubMed store contains 851,839 documents. Retrieval obtains 32 candidates, reranks them, and retains three passages. Reference answers do not enter inference queries. Question-only lexical checks found no complete-question matches among the eligible 975 SLAKE and 132 VQA-RAD questions.

The protocol specifies every stage of training-free collaboration, from native specialist acquisition to synchronized receiver decoding. Freezing the receiver and specialists, recording prompts and execution configurations, and keeping reference answers out of retrieval make the evaluation transparent and reproducible.

AuthorsOpenReview ↗

Response to Reviewer cFXF (Part 3 of 5)

We directly address your statistical, clinical-reliability, and Figure 4 questions using paired analysis, source records, and token-level evidence.

[W2/Q3] Statistical Robustness, Source Coverage, and MMMU

Q3:How robust are the reported improvements? Please define Figure 1's error bars and provide paired confidence intervals for the main comparisons. What fraction of examples has at least two active source groups? Does "MMMU" mean the full benchmark or a medical subset, and which specialists account for its substantial improvement?

A3. We directly quantify Figure 1(a)'s shared-context uncertainty, provide paired confidence intervals for MERIT, and examine how specialist capability and source coverage relate to the improvements.

Figure 1(a): error bars and shared-context uncertainty (HuatuoGPT-Vision). The error bars denote paired image-cluster bootstrap 95% percentile confidence intervals (2,500 resamples within each dataset, followed by equal-weight averaging over SLAKE/VQA-RAD gains) for the performance change Joint-all − Generalist, grouped by the number of source groups in the shared context.

Table C8. Figure 1(a) shared-context uncertainty.

Source groups in shared contextJoint-all − Generalist (pp)95% CI (pp)
1−20.26[−24.29, −16.21]
2−22.08[−31.13, −12.13]
≥3+3.37[−1.12, +7.98]

Figure 1(a) reveals substantial sensitivity of shared-context fusion to the number and combination of specialist sources. This directly motivates MERIT: each source is evaluated in an isolated branch of the same frozen receiver, and Adaptive BARD verifies support relative to the no-expert anchor before changing a token. The error bars make the source-composition effect explicit rather than inferring reliability from aggregate accuracy alone.

Paired 95% confidence intervals for MERIT's improvements.

Across both frozen receivers and all five medical VQA benchmarks, MERIT improves the reported point estimates in all ten settings. The gains are particularly pronounced on HuatuoGPT-Vision/MMMU-Medical and LLaVA-Med/SLAKE and VQA-RAD. The following paired confidence intervals for LLaVA-Med provide direct statistical support for the latter two improvements. The variation in gain size is also consistent with differences in question distribution, modality coverage, and the capabilities of the available specialist pool. MERIT's central contribution is the general, training-free collaboration mechanism that allows heterogeneous experts to benefit the same frozen receiver through source-isolated evaluation and anchored token commitment.

MERIT versus Generalist: paired 95% confidence intervals on LLaVA-Med.

Table C9. LLaVA-Med paired confidence intervals.

DatasetGeneralistMERITΔ (pp)Paired 95% CI (pp)
SLAKE34.6839.49+4.81[3.10, 6.47]
VQA-RAD48.4753.96+5.49[1.99, 9.20]

Both paired intervals exclude zero, supporting improvements on SLAKE and VQA-RAD. We also evaluate expert applicability and answer changes on the medical MMMU subset below.

Specialist capability: upgrading MERIT’s retrieval encoder. With LLaVA-Med as the receiver, we compare MedEmbed-Small and MedEmbed-Base within the retrieval-encoder experiment.

Table C10. MedEmbed retrieval encoder ablation.

MERIT retrieval encoderSLAKE (%)
MedEmbed-Small36.15
MedEmbed-Base36.34
Δ (pp)+0.19

Upgrading the retrieval specialist from MedEmbed-Small to MedEmbed-Base improves MERIT by 0.19 pp on SLAKE, with the receiver and Adaptive BARD unchanged. This provides concrete evidence of a key strength of MERIT: stronger specialist capabilities can be incorporated as modular upgrades to the same generalist–specialist collaboration framework, without retraining its decoding mechanism.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and Area Chairs

We thank the three reviewers and Area Chairs for their careful feedback. Reviewer cFXF recognizes the usefulness of comparing heterogeneous medical expertise through one frozen receiver; Reviewer FtVT finds the generalist–specialist integration problem important and the unified framework interesting; and Reviewer NgAH highlights the training-free nature of receiver-mediated evidence comparison. The concerns focus on whether MERIT offers a meaningful improvement over isolated aggregation, whether specialist collaboration improves reliability, and whether it generalizes beyond the initial receivers. Our revised analyses address these points through four concrete findings.

(1) A distinct mechanism, rather than another expert ensemble [cFXF, FtVT]. MERIT separates evidence acquisition, receiver-conditioned influence estimation, and token commitment. Under identical evidence and support gate, Adaptive BARD exceeds Mean + Support by 0.11/1.11/3.38/3.63 pp across the four ablation settings. RobustRAG and CCD comparisons additionally favor MERIT. The contribution is not the geometric median or isolation alone, but their integration with a no-expert anchor and source-aware token verification.

(2) Reliability supported by paired and clinical evidence [cFXF, NgAH]. Across the paper's ten receiver–benchmark comparisons, MERIT achieves higher reported scores than Greedy. Figure 1(a) quantifies how shared-context behavior varies with source composition, motivating source isolation and anchored commitment. Paired 95% confidence intervals on LLaVA-Med support the gains on SLAKE and VQA-RAD; paired correction counts and MERIT's leading RadGraph/RaTEScore/GREEN results add evidence at the answer and clinical-content levels.

(3) Generality across stronger receivers and heterogeneous specialists [All reviewers]. With Lingshu-7B and Hulu-Med-7B, MERIT improves over Greedy by +0.44/+2.80/+6.46/+1.89 pp across SLAKE/VQA-RAD. In the MedEmbed retrieval comparison, upgrading Small to Base improves MERIT on SLAKE without retraining. Single-specialist controls and MMMU-Medical results clarify expert complementarity. Adding the pretrained pathology specialist Patho-R1 raises HuatuoGPT-Vision/PathVQA from 38.86% to 42.89% (+4.03 pp), exceeding the strongest original baseline (39.65%) by 3.24 pp without retraining the receiver or Adaptive BARD (Appendix E, Table 9). The revised Figure S1 (Appendix C) now traces BiomedParse masks through the semantic and spatial channels to one source-isolated BARD branch, making the training-free evidence interface explicit.

(4) Explicit inference cost and a compute-constrained comparison [cFXF, FtVT]. MERIT takes 1.439 s/sample (10.25× Greedy) in the previously reported RTX 4090 timing setting. The new PathVQA profile reports 3.244 s/question (3.47× Greedy; Appendix E, Table 8), consistent with fewer applicable pathology specialists and task-dependent branch execution; these are distinct profiling workloads rather than a matched speedup comparison. In the separate receiver-time-capped Self-Consistency diagnostic, BARD has +3.28/+3.35 pp higher point estimates for Huatuo/LLaVA-Med. This clarifies how MERIT allocates computation to relevant expert evidence and selective answer revision.

Taken together, these results provide a more complete account of what MERIT contributes, when expert evidence helps, and why a training-free, dynamically extensible, receiver-mediated collaboration rule is valuable. We hope the new analyses address the central concerns and invite the reviewers to reassess the submission in light of the added evidence.

AuthorsOpenReview ↗

Summary of Changes in the Revised PDF and Additional Rebuttal Evidence

The table maps each main reviewer concern to the corresponding manuscript section or supplementary rebuttal analysis. Results already present in the revised PDF are distinguished from additional analyses supplied in these responses.

Table S1. Changes and added evidence.

LocationChange or additional evidenceRaised by
§3.1–3.4; Figures 2–3Clarify native evidence, isolated receiver branches, the shared committed prefix, and Adaptive BARD commitment.All reviewers
§3.1; Appendix C, Figure S1 and Eq. (S1)Add a segmentation-flow diagram: BiomedParse soft masks feed semantic serialization and parameter-free spatial patch pooling within one BARD source group.NgAH, FtVT
§4.4, Table 2Compare Mean + Support with Adaptive BARD under identical evidence and support rules.cFXF
Appendix A.2, Tables 5–6Compare with RobustRAG and with CCD on matched chest-X-ray inputs.cFXF
Appendix A.1, Table 4Evaluate Lingshu-7B and Hulu-Med-7B against Greedy, VCD, ICD, DoLa, and MedRAG.cFXF, NgAH
§4.5, Table 3; rebuttal Tables C15/N4Paper Table 3 reports RadGraph, RaTEScore, and GREEN; rebuttal Tables C15/N4 provide additional RadGraph variants and the expanded clinical-metric comparison.cFXF, NgAH
§4.2; Appendix DClarify frozen checkpoints, evaluation prompts, scoring, expert decoding, and Figure 1(a) bootstrap.cFXF
Appendix A.3, Table 7; rebuttal Table F5Report per-sample latency and positive-temperature Self-Consistency under a receiver-time budget.cFXF, FtVT
Appendix E, Tables 8–9; rebuttal Tables F7–F8/C17–C18Add PathVQA-specific inference latency and Patho-R1 expert-pool expansion, demonstrating task-dependent cost and modular specialist integration without retraining.FtVT, cFXF
Rebuttal analysesFigure 1 error-bar CIs, paired improvements, Huatuo active source-group coverage, and MMMU-Medical specialist-selection diagnostics.cFXF, FtVT, NgAH
Rebuttal analysesSingle-specialist controls, cross-receiver agreement, MedEmbed encoder upgrade, and Figure 4 diagnostic-token margins.cFXF, FtVT, NgAH

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.