- Rating
- 2(−1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a medical vision-language model that handles many imaging modalities by routing each image to a modality-specific expert. They show that it beats baselines on zero-shot classification, linear probing, and retrieval, and that new modalities can be added without hurting the old ones.
Critical strengths
The premise is strong and the use case is clear: using a pretrained model and avoiding loss of performance on general tasks while keeping the flexibility of adding different tasks.
The ability to leverage unlabeled data is also a strength.
The evaluation seems thorough and the relevant baselines seem to be included.
Critical weaknesses
I would have liked to see more insight and ablations into the method. What about using 1 multimodal expert (does it help performance at the cost of flexibility to add new modalities?). It would be good to show the limits of the method like that.
Also, what if the input modality is wrong (a CT scan as input but labeled as a fundus image). What is the performance? Does it degrade gracefully? Can it be detected and can the model output some sort of uncertainty signal?
There is no “limits” section.
It is unclear to me whether the contribution is an application of an existing method to a new domain or something more novel.
Questions for authors
-
Wrong modality at inference. What happens when the input modality is mislabeled, such as a CT scan routed as a fundus image? Does performance degrade gracefully, and does it beat a baseline with no knowledge of the modality?
-
Stage 1 loss. Why train stage 1 on the sum of the two losses if they are independent?
10 comments
Part 2/3: Incorrect routing and uncertainty
Response to Critical Weakness 2 and Question 1 (Routing Performance)
Correct routing achieves higher mean balanced accuracy than mean incorrect routing on five of six datasets (Table 22).
-
The separate stress test uses saved 70k checkpoints from three training seeds across MRI, pathology, dermatology, X-ray, CT and ultrasound. Each image passes through all seven expert routes. Within each seed, we average the six incorrect-route scores; the table reports mean ± SD on each full evaluation dataset.
-
Probe heads trained on correct-route features remain fixed during incorrect-route inference.
Table 22. Correct versus mean incorrect routing. Balanced accuracy (%), mean ± SD across three training seeds. ↑ indicates higher is better.
| Task | Correct | Mean incorrect |
|---|---|---|
| BrainTumor probe | 96.45 ± 0.52 | 86.60 ± 2.34 |
| BreaKHis | 54.96 ± 2.91 | 52.72 ± 2.14 |
| HAM10000 | 46.99 ± 2.84 | 37.30 ± 3.05 |
| Pneumonia | 65.94 ± 4.85 | 62.74 ± 2.14 |
| SARS-CoV-2 CT | 62.63 ± 3.42 | 54.40 ± 5.00 |
| Thyroid | 52.64 ± 0.73 | 54.57 ± 1.12 |
-
Table 22 in Appendix I reports all six datasets. Correct routing achieves balanced accuracy of 96.45% versus 86.60% on BrainTumor, 46.99% versus 37.30% on HAM10000 and 62.63% versus 54.40% on SARS-CoV-2 CT, compared with mean incorrect routing. BreaKHis and Pneumonia also improve; Thyroid is the exception. These results support the benefit of modality-informed routing across the evaluated tasks.
-
The modality-agnostic comparison is reported in Table 5: MediVerse achieves 50.437% versus 48.945% in mean zero-shot balanced accuracy, 85.282% versus 84.971% in probing and 5.780% versus 5.359% in retrieval for the shared-extra-FFN control. Table 22 quantifies the effect of incorrect routing within the routed model; Table 5 compares the architectures under their standard inference settings.
-
MediVerse assumes known modality identity at inference. Table 22 measures sensitivity to incorrect assignments, whereas Table 5 compares architectures under correct routing.
Response to Critical Weakness 2 (Detection and Uncertainty)
Entropy provides a useful routing-error signal on HAM10000 and BrainTumor, with mean detection AUROC of 0.650 and 0.619, respectively (Table 23, Appendix I).
- The analysis compares entropy, confidence and prediction margin across six datasets and three training seeds. Its saved protocol uses a fixed 25%/75% calibration/evaluation split grouped by image identity. Temperature is fitted using correct-route calibration predictions and applied to every route; AUROC is computed on held-out predictions.
Table 23. Held-out routing-error detection AUROC: mean ± SD across three training seeds. Higher AUROC indicates better detection for every column.
| Task | Entropy | Low confidence | Low margin |
|---|---|---|---|
| BrainTumor probe | 0.619 ± 0.016 | 0.618 ± 0.016 | 0.617 ± 0.016 |
| BreaKHis | 0.525 ± 0.030 | 0.525 ± 0.030 | 0.525 ± 0.030 |
| HAM10000 | 0.650 ± 0.029 | 0.633 ± 0.045 | 0.600 ± 0.061 |
| Pneumonia | 0.557 ± 0.056 | 0.557 ± 0.056 | 0.557 ± 0.056 |
| SARS-CoV-2 CT | 0.258 ± 0.199 | 0.258 ± 0.199 | 0.258 ± 0.199 |
| Thyroid | 0.555 ± 0.116 | 0.555 ± 0.116 | 0.555 ± 0.116 |
-
Mean entropy-detection AUROC exceeds 0.5 on five of six datasets. Entropy has the highest mean AUROC among the three evaluated signals on HAM10000 and BrainTumor (Table 23), supporting its use as a routing-error signal on dermatology and MRI in this evaluation.
-
Effectiveness varies by task: SARS-CoV-2 CT yields AUROC 0.258, so the same high-entropy criterion does not transfer uniformly across modalities. These AUROCs measure routing-error discrimination; the SDs in Tables 5 and 22 measure variation across training seeds.
Part 3/3: Operating conditions, contribution and Stage-1 losses
Response to Critical Weaknesses 3 and 4
MediVerse supports specialization, new-modality acquisition and preservation of existing embedding functions under the stated operating conditions.
-
Section 4.8 clarifies these conditions: known modality IDs at inference, increasing stored capacity as experts are added, and unchanged existing paths for exact preservation. It also distinguishes the repeated-seed ViT-B controls from the original full-scale results, which do not quantify training-seed variability.
-
MediVerse combines shared visual computation, modality-specific FFN specialization and modular acquisition within a fixed medical embedding space. Figure 2 makes this design explicit. The matched controls in Table 5 support specialization beyond added shared capacity, and the full ViT-L comparison in Table 6 shows higher mean new-modality retrieval with our experts than with independent adapters or LoRA.
-
Proposition 1 (Section 3.3, p. 6) formalizes preservation of existing embedding functions when the original parameters, persistent state, routing and inference operations remain unchanged. Together, these results connect the architectural design to specialization, acquisition and preservation.
Response to Question 2
We sum the two losses to train the image and text pathways under one Stage-1 objective; this does not require paired image–text data.
The image loss predicts masked visual tokens, while the text loss predicts masked words from independently sampled text. Each loss trains its corresponding pathway. Stage 2 then uses paired images and text to align their representations.
-
Section 3.2 specifies an equally weighted sum of mean losses. MIM predicts discrete tokenizer codes from masked images, while MLM predicts masked tokens from independently sampled text. The sum jointly schedules the two reconstruction objectives, each updating its respective pathway.
-
The visual FFN is shared across imaging modalities, rather than between the image and text pathways.
- The objectives therefore have distinct roles: masked reconstruction develops the separate image and text representations, and subsequent contrastive alignment learns their cross-modal correspondence from paired data.
Part 1/3: Shared-expert comparison and expansion
We thank the reviewer for recognizing the motivation, extensibility and use of unpaired data. The revision adds direct evidence for modality-informed specialization and new-modality acquisition, together with routing and uncertainty analyses. We address the shared-expert ablation below, followed by Q1 on inference routing and Q2 on the Stage-1 objective.
Response to Critical Weakness 1
Modality-specific experts achieve higher mean performance than one additional expert shared across imaging modalities, with matched additional active FFN capacity (Table 5).
Revised manuscript Figure 2. Architectural distinction and capacity control. MediVerse uses S(x)+E_m(x); the shared-capacity control uses S(x)+A(x).
Proposition 1 — Preservation of existing embedding functions
If expansion preserves all original parameters, persistent state, routing and inference operations, the image embedding function for every existing modality and the text embedding function remain unchanged. This is the conditional preservation statement in Section 3.3.
Table 5 (Appendix Table 17). Capacity controls: mean ± SD over three training seeds. Macros cover nine zero-shot tasks, six probes and four retrieval tasks. Scores are percentages. ↑ indicates higher is better.
The shared-extra-FFN control S(x)+A(x) implements one additional expert for all image modalities: S is the existing shared visual FFN, and A has independent parameters. MediVerse instead uses S(x)+E_m(x), selecting a dedicated expert through the known modality ID; the text pathway remains separate (Figure 2, Section 3.1, p. 5).
All variants use ViT-B, identical 70k pairs, public initialization, tokenizer, masking and optimization, 3,000 masked-pretraining updates and ten alignment epochs. Scores are mean ± SD over three training seeds, covering nine zero-shot tasks, six probes and four retrieval tasks. Modality routing also exceeds random routing in all three task-family means.
The shared-extra-FFN control and MediVerse each use 14.186M additional active FFN parameters per image across three routed layers. Their stored capacities differ: 236.645M for the shared control and 321.759M for MediVerse. Random routing retains MediVerse’s seven-expert bank and matches both active and stored capacity, while assigning routes independently of modality.
The two controls therefore address active capacity and modality-informed assignment separately.
The single shared expert uses less stored capacity, while dedicated experts support modular expansion: a new expert can be trained with existing pathways frozen. Updating the common expert would modify a pathway used by existing modalities.
Table 6 separately compares new-modality acquisition using experts, adapters and LoRA on the same frozen ViT-L backbone; MediVerse experts achieve higher mean fundus and OCT retrieval, with greater added capacity. Continual expansion by updating the single shared expert was not evaluated in this comparison.
Table 6 (Appendix Table 20). Independent modules on the same frozen ViT-L backbone and matched fundus/OCT data. ↑ indicates higher is better.