- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
MERIT is a training-free framework that conditions isolated branches of a frozen medical VLM on specialist outputs alongside a no-expert anchor. Adaptive BARD proposes tokens via the geometric median of centered log-probability residuals, accepting revisions only when sufficient branch support is present. MERIT improves greedy decoding with two receivers on five benchmarks, as well as on MIMIC-CXR.
Critical strengths
-
Comparing heterogeneous specialist evidence through its effect on a shared frozen receiver avoids learned projections, making the method training-free and directly applicable to any receiver.
-
MERIT never underperforms greedy decoding across ten settings, with notable gains on LLaVA-Med. The ablation further reveals that naive multi-expert fusion substantially degrades medical VLMs.
Critical weaknesses
-
The paper never specifies how segmentation masks are converted into text, so whether location, area, or tubular structures such as vessels are faithfully conveyed remains unclear.
-
Both receivers are limited by current standards; effectiveness on stronger models such as Lingshu-7B or Hulu-Med-7B, and superiority over Table 1 baselines, remain untested.
-
Report generation is evaluated solely with NLG metrics; clinically oriented metrics such as GREEN or RaTEScore are absent, leaving clinical quality unverified.
-
It is unclear how fixed-label classifiers and segmenters apply to VQA-RAD and MMMU, what should be segmented, and whether TorchXRayVision and BiomedParse actually contribute.
Questions for authors
-
The paper does not explain how masks produced by the segmentation model are converted into textual descriptions. Can information such as the location and area of a mask be accurately conveyed in text? How would tubular structures such as blood vessels be described? This is important for whether the receiver model can correctly interpret the output of the segmentation expert.
-
Both LLaVA-Med and HuatuoGPT-Vision used in the experiments have relatively limited capabilities. Would the method remain effective on more advanced models such as Lingshu-7B and Hulu-Med-7B? Would it still outperform the comparison methods in Table 1?
-
For the report generation task, only natural language generation metrics are used for evaluation. Have the authors considered metrics designed specifically for report generation, such as GREEN and RaTEScore, which assess the quality of generated reports more reliably?
-
How are the classification and segmentation models in the expert pool applied across different downstream tasks? For example, the questions and imaging modalities in VQA-RAD and MMMU differ entirely. Can TorchXRayVision be applied to MMMU at all? What should the segmentation model segment in this context? Do these two expert models actually contribute on these benchmarks?
5 comments
Response to Reviewer NgAH (Part 2 of 3)
[W2/Q2] Stronger Medical Receivers and Specialist Capacity
Q2:Both LLaVA-Med and HuatuoGPT-Vision used in the experiments have relatively limited capabilities. Would the method remain effective on more advanced models such as Lingshu-7B and Hulu-Med-7B? Would it still outperform the comparison methods in Table 1?
A2. We extended the receiver evaluation to Lingshu-7B and Hulu-Med-7B on the complete SLAKE and VQA-RAD cohorts.
Table N2. Stronger medical VLM comparisons.
| Method | Lingshu SLAKE | Lingshu VQA-RAD | Hulu-Med SLAKE | Hulu-Med VQA-RAD |
|---|---|---|---|---|
| Greedy | 82.25 | 61.14 | 64.14 | 68.32 |
| VCD | 81.94 | 59.95 | 63.25 | 67.85 |
| ICD | 81.87 | 60.38 | 62.99 | 66.30 |
| DoLa | 78.80 | 50.81 | 55.56 | 51.43 |
| MedRAG | 65.69 | 44.07 | 52.77 | 55.18 |
| MERIT | 82.69 | 63.94 | 70.60 | 70.21 |
| Δ vs Greedy (pp) | +0.44 | +2.80 | +6.46 | +1.89 |
MERIT achieves the highest reported score in each of the four settings. This demonstrates positive transfer to two stronger receivers without training the receiver or a new fusion module.
Mechanistic explanation: Each frozen receiver constructs its own anchor and source-conditioned residuals under a synchronized prefix. The same MERIT inference rule can therefore operate with stronger receivers, making the collaboration framework naturally modular.
Additional retrieval-specialist capacity comparison. We evaluated MERIT with MedEmbed-Small and MedEmbed-Base using LLaVA-Med on SLAKE.
Table N3. MedEmbed retrieval encoder ablation.
| MERIT retrieval encoder | SLAKE (%) |
|---|---|
| MedEmbed-Small | 36.15 |
| MedEmbed-Base | 36.34 |
| Δ (pp) | +0.19 |
Replacing MedEmbed-Small with MedEmbed-Base improves MERIT by +0.19 pp on SLAKE. These results show that upgrading the retrieval specialist can benefit the existing receiver and BARD decoder without retraining the framework.
The Lingshu/Hulu-Med experiments establish gains with stronger receiver backbones, while MedEmbed demonstrates a second, complementary path: improving the specialist itself also raises MERIT performance. Both results support the framework's modular extensibility.
[W3/Q3] Clinical Factuality Metrics for MIMIC-CXR
Q3:For the report generation task, only natural language generation metrics are used for evaluation. Have the authors considered metrics designed specifically for report generation, such as GREEN and RaTEScore, which assess the quality of generated reports more reliably?
A3. We expanded the original MIMIC-CXR comparison with clinical-content metrics: RadGraph Simple-F1, RadGraph Partial-F1, RadGraph Complete-F1, RaTEScore, and GREEN alongside BLEU-4, ROUGE-L, and METEOR.
Table N4. Clinical report-generation metrics (%, ↑ higher is better).
B4: BLEU-4; RL: ROUGE-L; MET: METEOR; RG-S/P/C: RadGraph Simple/Partial/Complete; RaTE: RaTEScore; GREEN reported ×100.
| Method | B4↑ | RL↑ | MET↑ | RG-S↑ | RG-P↑ | RG-C↑ | RaTE↑ | GREEN↑ |
|---|---|---|---|---|---|---|---|---|
| Greedy | 1.02 | 12.87 | 9.90 | 6.01 | 5.46 | 4.42 | 43.09 | 3.48 |
| VCD | 1.02 | 12.47 | 9.52 | 5.56 | 4.90 | 3.36 | 39.82 | 2.17 |
| ICD | 1.21 | 13.03 | 10.37 | 5.97 | 4.98 | 3.69 | 41.59 | 3.97 |
| DoLa | 1.01 | 12.83 | 9.84 | 5.93 | 5.37 | 4.35 | 43.05 | 3.44 |
| PAI | 1.06 | 12.85 | 9.64 | 5.59 | 5.07 | 3.83 | 42.16 | 2.76 |
| AvisC | 1.48 | 13.09 | 13.26 | 6.58 | 5.95 | 4.49 | 43.38 | 2.88 |
| VISTA | 1.00 | 13.16 | 9.47 | 6.50 | 6.04 | 4.98 | 42.94 | 3.01 |
| AGLA | 1.18 | 13.69 | 10.89 | 6.98 | 6.44 | 5.32 | 43.57 | 3.24 |
| OPERA | 1.41 | 13.80 | 12.73 | 5.76 | 5.38 | 4.45 | 41.05 | 1.55 |
| MERIT | 2.53 | 17.11 | 14.82 | 17.66 | 15.89 | 11.79 | 50.90 | 18.29 |
MERIT has the highest reported score in all eight columns. Relative to the strongest non-MERIT row for each clinical metric, its descriptive advantages are +10.68 pp (RadGraph Simple), +9.45 pp (Partial), +6.47 pp (Complete), and +7.33 pp (RaTEScore). GREEN also increases to 18.29 (×100), a gain of 14.32 points over the best comparator.
RadGraph evaluates medically meaningful entities and relations, RaTEScore measures clinical report-content alignment, and GREEN adds a report-specific assessment. The consistent improvement across both metric families directly addresses the reviewer's request for clinically oriented evaluation.
Response to Reviewer NgAH (Part 3 of 3)
We expand radiology-report evaluation with clinical metrics and clarify how classification and segmentation experts apply to different imaging tasks.
[W4/Q4] Applicability and Contribution of Classification/Segmentation Experts
Q4:How are the classification and segmentation models in the expert pool applied across different downstream tasks? For example, the questions and imaging modalities in VQA-RAD and MMMU differ entirely. Can TorchXRayVision be applied to MMMU at all? What should the segmentation model segment in this context? Do these two expert models actually contribute on these benchmarks?
A4. MERIT selects classification and segmentation specialists according to the modality, task, question, and anatomical context of each medical input. This capability-aware routing enables expert collaboration on the five-subject, 1,752-question MMMU medical subset.
Classification: TorchXRayVision provides findings on supported radiographs. In the MMMU medical subset, TorchXRayVision Findings is selected for 177 questions.
Segmentation: BiomedParse selects question-relevant labels from registered modality–anatomy target groups. For example, a spleen question can use the spleen target within the CT-Abdomen group. The revised §3.1 and Appendix C, Figure S1 show the resulting soft masks entering two paths: structured labels and geometry in the semantic prompt, and mask-aligned projected visual features in the parameter-free spatial interface. Both paths condition one source-isolated receiver branch, whose candidate support is evaluated by Adaptive BARD.
Actual specialist invocations across five evaluated datasets (MMMU denotes the medical subset).
Table N5. Expert invocations across datasets.
| Specialist | PMC-VQA | PathVQA | SLAKE | VQA-RAD | MMMU-Medical |
|---|---|---|---|---|---|
| BiomedParse | 33,399 | 5,802 | 2,094 | 451 | 1,730 |
| CheXagent | 1,457 | 248 | 740 | 174 | 234 |
| CONCH | 3,675 | 3,503 | 0 | 0 | 348 |
| TorchXRayVision Findings | 1,300 | 227 | 637 | 141 | 177 |
| MedCPT | 33,430 | 6,719 | 2,094 | 451 | 1,752 |
MMMU-Medical: specialist selection and accuracy improvements.
Table N6. Accuracy improvements among selected MMMU-Medical questions.
| Selected specialist | Selected questions | Accuracy improvement (pp) |
|---|---|---|
| BiomedParse | 1,730 | +10.35 |
| CheXagent | 234 | +4.27 |
| CONCH | 348 | +10.06 |
| TorchXRayVision Findings | 177 | +5.08 |
| MedCPT | 1,752 | +10.50 |
Accuracy increases on the selected-question cohorts are +10.35 pp for BiomedParse, +10.06 pp for CONCH, +4.27 pp for CheXagent, +5.08 pp for TorchXRayVision Findings, and +10.50 pp for MedCPT. These results demonstrate how heterogeneous specialist capabilities are associated with improvements under MERIT's receiver-mediated collaboration mechanism.
VQA-RAD and MMMU-Medical source-removal controls.
Table N7. Specialist-removal results on VQA-RAD and MMMU-Medical.
| Configuration | VQA-RAD (%) | MERIT gain (pp) | MMMU-Medical (%) | MERIT gain (pp) |
|---|---|---|---|---|
| MERIT | 53.96 | — | 22.73 | — |
| Remove BiomedParse | 53.14 | +0.82 | 22.50 | +0.23 |
| Remove XRV Anatomy | 53.63 | +0.33 | 22.67 | +0.06 |
Removing BiomedParse lowers the full-model scores by 0.82 pp on VQA-RAD and 0.23 pp on MMMU-Medical; removing XRV Anatomy lowers them by 0.33 pp and 0.06 pp, respectively. The consistent positive contributions across both benchmarks reinforce the value of anatomical segmentation and radiographic anatomy within MERIT's source-isolated, anchored collaboration framework.
We hope these responses and the revised analyses address the main concerns and welcome further discussion before the decision.
Response to Reviewer NgAH (Part 1 of 3)
Thank you for focusing on mask transport and contemporary receivers. We clarify the dual-channel segmentation interface and report Lingshu/Hulu-Med evaluations.
[W1/Q1] How Are Segmentation Masks Represented?
A1. We have added a dedicated segmentation schematic in the revised manuscript: Appendix C, Figure S1, "Segmentation-conditioned evidence branch." Together with Section 3.1 and Eq. (S1), it explains how predicted masks reach the frozen receiver. Segmentation is not reduced to text: a compatible receiver can use semantic and spatial evidence simultaneously within a single isolated source branch.
Figure S1: From BiomedParse to Adaptive BARD. BiomedParse predicts modality-compatible soft masks for question-relevant anatomical targets. The figure traces two complementary paths from the same masks:
Table N1. Semantic and spatial segmentation channels (see Figure S1).
Mask-aware visual transport. As illustrated by the lower panel of Figure S1, the spatial path uses the receiver's original square-padding transform and averages soft-mask coverage at each visual patch, without binarizing the mask.
Let mkj denote region k's coverage of patch j, and let hj denote the receiver's existing projected patch feature. The parameter-free update in Eq. (S1) is:
rkj=∑ℓmkℓmkj
ek=∑jrkjhj
hj′=1+∑kwkmkjhj+∑kwkmkjek
The mask selects and pools regional information already encoded by the frozen receiver. Zero-coverage patches remain unchanged, the original image remains available, and no learned cross-model projection is introduced.
Location, area, and tubular structures. The semantic path exposes labels and coarse geometry such as location and mask coverage, while the spatial path retains shape-conditioned regional features at visual-patch resolution. This provides a more informative interface for elongated structures than a text-only bounding box, although detailed vessel branching is still limited by patch resolution.
Source grouping and decoding. Both paths from one BiomedParse observation—including multiple anatomical masks—are grouped as one specialist source for BARD. Their effect is evaluated by an isolated branch of the same frozen receiver under the shared answer prefix. Adaptive BARD then checks source support before accepting a candidate token.
Figure S1 makes the native-evidence interface concrete: MERIT preserves both the semantic meaning and spatial support of segmentation, translates them into receiver-compatible observations without training, and integrates them through its general source-isolated, anchor-protected decoding mechanism.