acceptodds
Under review as a conference paper at ICLR 2027

MediVerse: A Unified Medical Vision–Language Foundation Model with Zero-Forgetting Continual Modality Expansion

Abstract

A unified medical foundation model should support many imaging modalities, provide a language interface, and expand without losing existing capabilities. Current models usually satisfy only part of this goal. Vision-only models support several modalities but cannot perform text-prompted zero-shot classification or image–text retrieval. Vision–language models provide these abilities but often require large paired corpora and may forget earlier modalities when adapted to new ones. We introduce MediVerse, a pair-efficient and extensible medical vision– language model. MediVerse uses a Multiway Transformer with a Domain-Aware Mixture-of-Experts. A deterministic router sends each image to its modalityspecific expert, while a separate text pathway interacts with the image pathway through contrastive alignment. We first pretrain on 19.4M unpaired images and then align the model using 3.2M image–text pairs. With a ViT-Large backbone, MediVerse wins 10 of 11 zero-shot benchmarks (mean 0.674 vs. 0.594 for the strongest baseline), is best or tied-best on all 10 linear probes (0.856 vs. 0.824), and wins all 4 cross-modal retrieval benchmarks (20.70 vs. 12.09) across nine imaging modalities. To add a new modality, we train a new routed expert while freezing the existing computation paths. This design preserves old-modality embeddings when the existing computation paths and inference conditions remain unchanged. Across fundus and OCT expansion, adapted baselines lose 2.97–6.11 points of pathology retrieval, whereas MediVerse shows no change on the evaluated pathology task.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical vision-language models, mixture-of-experts, continual learning, catastrophic forgetting, multimodal foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7A4BOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a medical vision-language model that handles many imaging modalities by routing each image to a modality-specific expert. They show that it beats baselines on zero-shot classification, linear probing, and retrieval, and that new modalities can be added without hurting the old ones.

Critical strengths

The premise is strong and the use case is clear: using a pretrained model and avoiding loss of performance on general tasks while keeping the flexibility of adding different tasks.

The ability to leverage unlabeled data is also a strength.

The evaluation seems thorough and the relevant baselines seem to be included.

Critical weaknesses

I would have liked to see more insight and ablations into the method. What about using 1 multimodal expert (does it help performance at the cost of flexibility to add new modalities?). It would be good to show the limits of the method like that.

Also, what if the input modality is wrong (a CT scan as input but labeled as a fundus image). What is the performance? Does it degrade gracefully? Can it be detected and can the model output some sort of uncertainty signal?

There is no “limits” section.

It is unclear to me whether the contribution is an application of an existing method to a new domain or something more novel.

Questions for authors
  1. Wrong modality at inference. What happens when the input modality is mislabeled, such as a CT scan routed as a fundus image? Does performance degrade gracefully, and does it beat a baseline with no knowledge of the modality?

  2. Stage 1 loss. Why train stage 1 on the sum of the two losses if they are independent?

10 comments
AuthorsOpenReview ↗

Part 1/3: Shared-expert comparison and expansion

We thank the reviewer for recognizing the motivation, extensibility and use of unpaired data. The revision adds direct evidence for modality-informed specialization and new-modality acquisition, together with routing and uncertainty analyses. We address the shared-expert ablation below, followed by Q1 on inference routing and Q2 on the Stage-1 objective.

Response to Critical Weakness 1

Modality-specific experts achieve higher mean performance than one additional expert shared across imaging modalities, with matched additional active FFN capacity (Table 5).

  • Mean zero-shot balanced accuracy is 50.437% versus 48.945%, probe balanced accuracy is 85.282% versus 84.971%, and retrieval is 5.780% versus 5.359%. The shared alternative uses less stored capacity; dedicated experts allow new modalities to be added with existing computation paths frozen.

Revised manuscript Figure 2. Architectural distinction and capacity control. MediVerse uses S(x)+E_m(x); the shared-capacity control uses S(x)+A(x).

Proposition 1 — Preservation of existing embedding functions

If expansion preserves all original parameters, persistent state, routing and inference operations, the image embedding function for every existing modality and the text embedding function remain unchanged. This is the conditional preservation statement in Section 3.3.

Table 5 (Appendix Table 17). Capacity controls: mean ± SD over three training seeds. Macros cover nine zero-shot tasks, six probes and four retrieval tasks. Scores are percentages. ↑ indicates higher is better.

ArchitectureZero-shot BA (%) ↑Probe BA (%) ↑Retrieval (%) ↑
Plain49.277 ± 2.34385.063 ± 0.5855.220 ± 0.173
Shared extra FFN48.945 ± 0.63584.971 ± 0.4445.359 ± 0.145
Random routing47.128 ± 0.73283.894 ± 0.2225.168 ± 0.165
Modality routing50.437 ± 0.92385.282 ± 0.2065.780 ± 0.159
  • The shared-extra-FFN control S(x)+A(x) implements one additional expert for all image modalities: S is the existing shared visual FFN, and A has independent parameters. MediVerse instead uses S(x)+E_m(x), selecting a dedicated expert through the known modality ID; the text pathway remains separate (Figure 2, Section 3.1, p. 5).

  • All variants use ViT-B, identical 70k pairs, public initialization, tokenizer, masking and optimization, 3,000 masked-pretraining updates and ten alignment epochs. Scores are mean ± SD over three training seeds, covering nine zero-shot tasks, six probes and four retrieval tasks. Modality routing also exceeds random routing in all three task-family means.

  • The shared-extra-FFN control and MediVerse each use 14.186M additional active FFN parameters per image across three routed layers. Their stored capacities differ: 236.645M for the shared control and 321.759M for MediVerse. Random routing retains MediVerse’s seven-expert bank and matches both active and stored capacity, while assigning routes independently of modality.

  • The two controls therefore address active capacity and modality-informed assignment separately.

  • The single shared expert uses less stored capacity, while dedicated experts support modular expansion: a new expert can be trained with existing pathways frozen. Updating the common expert would modify a pathway used by existing modalities.

  • Table 6 separately compares new-modality acquisition using experts, adapters and LoRA on the same frozen ViT-L backbone; MediVerse experts achieve higher mean fundus and OCT retrieval, with greater added capacity. Continual expansion by updating the single shared expert was not evaluated in this comparison.

Table 6 (Appendix Table 20). Independent modules on the same frozen ViT-L backbone and matched fundus/OCT data. ↑ indicates higher is better.

ModuleFundus retrieval (%) ↑OCT retrieval (%) ↑Added M/modality
Adapter16.596 ± 0.32712.662 ± 0.3883.153
FFN-output LoRA13.858 ± 0.08512.926 ± 0.0810.492
Expert21.171 ± 0.38913.444 ± 0.21850.412
AuthorsOpenReview ↗

Part 2/3: Incorrect routing and uncertainty

Response to Critical Weakness 2 and Question 1 (Routing Performance)

Correct routing achieves higher mean balanced accuracy than mean incorrect routing on five of six datasets (Table 22).

  • The separate stress test uses saved 70k checkpoints from three training seeds across MRI, pathology, dermatology, X-ray, CT and ultrasound. Each image passes through all seven expert routes. Within each seed, we average the six incorrect-route scores; the table reports mean ± SD on each full evaluation dataset.

  • Probe heads trained on correct-route features remain fixed during incorrect-route inference.

Table 22. Correct versus mean incorrect routing. Balanced accuracy (%), mean ± SD across three training seeds. ↑ indicates higher is better.

TaskCorrectMean incorrect
BrainTumor probe96.45 ± 0.5286.60 ± 2.34
BreaKHis54.96 ± 2.9152.72 ± 2.14
HAM1000046.99 ± 2.8437.30 ± 3.05
Pneumonia65.94 ± 4.8562.74 ± 2.14
SARS-CoV-2 CT62.63 ± 3.4254.40 ± 5.00
Thyroid52.64 ± 0.7354.57 ± 1.12
  • Table 22 in Appendix I reports all six datasets. Correct routing achieves balanced accuracy of 96.45% versus 86.60% on BrainTumor, 46.99% versus 37.30% on HAM10000 and 62.63% versus 54.40% on SARS-CoV-2 CT, compared with mean incorrect routing. BreaKHis and Pneumonia also improve; Thyroid is the exception. These results support the benefit of modality-informed routing across the evaluated tasks.

  • The modality-agnostic comparison is reported in Table 5: MediVerse achieves 50.437% versus 48.945% in mean zero-shot balanced accuracy, 85.282% versus 84.971% in probing and 5.780% versus 5.359% in retrieval for the shared-extra-FFN control. Table 22 quantifies the effect of incorrect routing within the routed model; Table 5 compares the architectures under their standard inference settings.

  • MediVerse assumes known modality identity at inference. Table 22 measures sensitivity to incorrect assignments, whereas Table 5 compares architectures under correct routing.

Response to Critical Weakness 2 (Detection and Uncertainty)

Entropy provides a useful routing-error signal on HAM10000 and BrainTumor, with mean detection AUROC of 0.650 and 0.619, respectively (Table 23, Appendix I).

  • The analysis compares entropy, confidence and prediction margin across six datasets and three training seeds. Its saved protocol uses a fixed 25%/75% calibration/evaluation split grouped by image identity. Temperature is fitted using correct-route calibration predictions and applied to every route; AUROC is computed on held-out predictions.

Table 23. Held-out routing-error detection AUROC: mean ± SD across three training seeds. Higher AUROC indicates better detection for every column.

TaskEntropyLow confidenceLow margin
BrainTumor probe0.619 ± 0.0160.618 ± 0.0160.617 ± 0.016
BreaKHis0.525 ± 0.0300.525 ± 0.0300.525 ± 0.030
HAM100000.650 ± 0.0290.633 ± 0.0450.600 ± 0.061
Pneumonia0.557 ± 0.0560.557 ± 0.0560.557 ± 0.056
SARS-CoV-2 CT0.258 ± 0.1990.258 ± 0.1990.258 ± 0.199
Thyroid0.555 ± 0.1160.555 ± 0.1160.555 ± 0.116
  • Mean entropy-detection AUROC exceeds 0.5 on five of six datasets. Entropy has the highest mean AUROC among the three evaluated signals on HAM10000 and BrainTumor (Table 23), supporting its use as a routing-error signal on dermatology and MRI in this evaluation.

  • Effectiveness varies by task: SARS-CoV-2 CT yields AUROC 0.258, so the same high-entropy criterion does not transfer uniformly across modalities. These AUROCs measure routing-error discrimination; the SDs in Tables 5 and 22 measure variation across training seeds.

AuthorsOpenReview ↗

Part 3/3: Operating conditions, contribution and Stage-1 losses

Response to Critical Weaknesses 3 and 4

MediVerse supports specialization, new-modality acquisition and preservation of existing embedding functions under the stated operating conditions.

  • Section 4.8 clarifies these conditions: known modality IDs at inference, increasing stored capacity as experts are added, and unchanged existing paths for exact preservation. It also distinguishes the repeated-seed ViT-B controls from the original full-scale results, which do not quantify training-seed variability.

  • MediVerse combines shared visual computation, modality-specific FFN specialization and modular acquisition within a fixed medical embedding space. Figure 2 makes this design explicit. The matched controls in Table 5 support specialization beyond added shared capacity, and the full ViT-L comparison in Table 6 shows higher mean new-modality retrieval with our experts than with independent adapters or LoRA.

  • Proposition 1 (Section 3.3, p. 6) formalizes preservation of existing embedding functions when the original parameters, persistent state, routing and inference operations remain unchanged. Together, these results connect the architectural design to specialization, acquisition and preservation.

Response to Question 2

We sum the two losses to train the image and text pathways under one Stage-1 objective; this does not require paired image–text data.

The image loss predicts masked visual tokens, while the text loss predicts masked words from independently sampled text. Each loss trains its corresponding pathway. Stage 2 then uses paired images and text to align their representations.

  • Section 3.2 specifies an equally weighted sum of mean losses. MIM predicts discrete tokenizer codes from masked images, while MLM predicts masked tokens from independently sampled text. The sum jointly schedules the two reconstruction objectives, each updating its respective pathway.

  • The visual FFN is shared across imaging modalities, rather than between the image and text pathways.

Lstage1=LMIM+LMLM.L_{\mathrm{stage1}} = L_{\mathrm{MIM}} + L_{\mathrm{MLM}}.

  • The objectives therefore have distinct roles: masked reconstruction develops the separate image and text representations, and subsequent contrastive alignment learns their cross-modal correspondence from paired data.
Reviewer_RNBNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

(1) The system uses a visual transformer with shared and modality-specific feed-forward branches, together with a separate text encoder. A teacher-trained discrete tokenizer supplies targets for masked image pretraining before image-text alignment. (2) New modality branches are trained while the existing embedding paths remain fixed. Evaluation examines classification and retrieval across nine imaging modalities, including fundus and OCT extensions.

Critical strengths

(1) Evaluation spans several imaging modalities and task families. Competitive performance with fewer image-text pairs than some existing models is a practical strength. (2) Preserving existing computation paths gives a clear mechanism for maintaining old embeddings during expansion. The paper also evaluates whether the added branches acquire useful capabilities.

Critical weaknesses

(1) The central scientific advance is not sufficiently established. The pretraining route follows MUSK (Xiang et al., 2025), and medical learning from unpaired images and text is also represented by MedCLIP (Wang et al., 2022). Preservation by freezing old paths follows the principle of progressive networks (Rusu et al., 2016). Proposition 1 does not distinguish DA-MoE from independently routed adapters on a frozen backbone. The paper needs to show what its particular combination contributes beyond these existing principles.

(2) The main experiments do not adequately isolate this contribution. Removing experts in Section 4.7 also removes active capacity, so the ablation cannot identify the benefit of modality specialization over an equally large shared branch. The pathology evaluations omit MUSK and CONCH, which are especially relevant competitors (Lu et al., 2024). In addition, the reported UniMed and MerMED aggregate probe scores do not match the listed task scores. These issues weaken the empirical case for the specific architecture.

References

Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., Parwani, A. V., Zhang, A., & Mahmood, F. (2024). A visual-language foundation model for computational pathology. Nature Medicine, 30(3), 863–874. https://doi.org/10.1038/s41591-024-02856-4

Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016). Progressive neural networks. arXiv. https://doi.org/10.48550/arXiv.1606.04671

Wang, Z., Wu, Z., Agarwal, D., & Sun, J. (2022). MedCLIP: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 3876–3887). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.emnlp-main.256

Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y., Li, Y., Bergstrom, C., Gopaulchan, M., Kim, T., Yu, K.-H., Willens, S., Olguin, F. M., Nirschl, J. J., Neal, J., Diehn, M., Yang, S., & Li, R. (2025). A vision–language foundation model for precision oncology. Nature, 638(8051), 769–778. https://doi.org/10.1038/s41586-024-08378-w

Questions for authors

(1) Is there a control that matches the active capacity of the expert model using a shared or ordinary residual feed-forward branch? What does it show? (2) How does the expansion method compare with a frozen backbone and independent adapters routed by the same known modality ID, and with MUSK/CONCH on applicable pathology tasks? (3) Please reconcile Table 2 and Section 4.3: the listed UniMed scores average 0.7894, rather than 0.793, and the seven MerMED scores average approximately 0.86543, rather than 0.867. These differences exceed the stated precision's rounding allowance.

4 comments
AuthorsOpenReview ↗

Response to W1/W2/Q1

W1/Q1. Methodological Novelty & W2. Evidence for the architecture

MediVerse contributes (1) a shared visual encoder with modality-specific specialization across medical imaging modalities and (2) modular expansion within the existing Transformer. Controlled experiments demonstrate the benefits of modality-aware capacity allocation, multi-teacher supervision, and new-modality acquisition.

1. Shared Visual Encoder with Modality-Specific Capacity

1.1 Contribution beyond MUSK and MedCLIP

  • We build on MUSK's masked-pretraining/alignment strategy and acknowledge MedCLIP's unpaired-learning precedent.
  • Our key distinction is specialization within vision: DA-MoE combines shared visual computation with dedicated FFN experts for CT, MRI, pathology, and other imaging modalities.

1.2 Performance gains from modality-specific FFNs

  • With identical data, initialization, tokenizer supervision, and training schedules, modality routing achieves higher mean performance than both capacity controls across all three task families (Table 1).

Table 1. Capacity comparison. Mean ± sample SD across three training seeds (%).

ArchitectureZero-shot BA (%) ↑Probe BA (%) ↑Retrieval (%) ↑
Plain49.277 ± 2.34385.063 ± 0.5855.220 ± 0.173
Shared extra FFN48.945 ± 0.63584.971 ± 0.4445.359 ± 0.145
Random routing47.128 ± 0.73283.894 ± 0.2225.168 ± 0.165
Modality routing (ours)50.437 ± 0.92385.282 ± 0.2065.780 ± 0.159

1.3 Performance gains from multi-teacher supervision

  • A separate three-seed ablation fixes the routed architecture and training budget. Six visual teachers achieve higher mean performance than a single teacher across all three task families (Table 2).

Table 2. Teacher comparison. Mean ± sample SD across three training seeds (%).

Teacher supervisionZero-shot BA (%) ↑Probe BA (%) ↑Retrieval (%) ↑
Single teacher48.322 ± 1.27885.066 ± 0.2995.176 ± 0.169
Six teachers48.674 ± 1.08285.479 ± 0.1105.528 ± 0.125

2. Modular Expansion

2.1 Contribution beyond progressive networks

  • Unlike progressive networks, which add task-specific columns with lateral connections, MediVerse adds modality-specific FFN experts within the existing Transformer, aligning new modalities to the frozen text embedding space.

2.2 Controlled comparison with independent adapters and LoRA

  • Using the same frozen ViT-L backbone, adaptation data (20k fundus / 16k OCT pairs), and evaluation protocol, MediVerse achieves higher mean retrieval on both modalities (Table 3), albeit with greater added parameter capacity.

Table 3. Matched expansion comparison. Mean ± sample SD across three training seeds; added parameters per modality.

Added moduleFundus retrieval (%) ↑OCT retrieval (%) ↑Added parameters (M)
Independent adapter16.596 ± 0.32712.662 ± 0.3883.153
Independent FFN-output LoRA13.858 ± 0.08512.926 ± 0.0810.492
MediVerse new expert21.171 ± 0.38913.444 ± 0.21850.412

2.3 Scope of Proposition 1

  • We acknowledge that independently routed adapters can satisfy the same preservation condition. Proposition 1 formalizes preservation of existing embedding functions, while our architectural contribution is supported by the controlled specialization and expansion experiments above.

Revision: These comparisons and supporting details have been incorporated into the revised manuscript.

AuthorsOpenReview ↗

Response to W2/Q2/Q3

W2/Q2/Q3. Pathology Comparison and Score Corrections

1. Pathology specialist comparison

  • We added direct comparisons with MUSK and CONCH using the same evaluation protocols.
  • MediVerse performs best on PanNuke and HEST probing and remains competitive on PathGen-600 retrieval; MUSK/CONCH perform better on BreaKHis and LC25000.
  • We therefore do not claim uniform superiority. Rather, these results show that a nine-modality generalist remains competitive with strong pathology-specific models while outperforming them on selected tasks.

Table. Pathology specialist comparison (%). Classification and probing use balanced accuracy; PathGen-600 uses mean bidirectional Recall@1/10/50.

TaskEvaluationMUSKCONCHMediVerse
BreaKHisZero-shot BA71.7774.8766.55
LC25000Zero-shot BA89.6091.2088.08
PanNukeZero-shot BA20.9518.8424.56
PathGen-600Retrieval26.6126.0625.75
HESTLinear-probe BA96.9796.6799.40

2. Aggregate-score corrections

  • We thank the reviewer for identifying the inconsistencies in the reported UniMed-CLIP and MerMED-FM aggregate probe scores.
  • We correct UniMed’s mean to 0.789 and MerMED’s common-seven mean to 0.865, preserving all original per-task scores.

Revision: These comparisons and corrections have been incorporated into the revised manuscript.

I am raising my score from 1 to 2. The new ViT-B ablations show that modality routing improves mean performance at matched active capacity. I still lean toward rejection because independently routed adapters can provide the same preservation guarantee, and it remains unclear whether the experts’ retrieval gains would persist at an equal parameter budget.

Thank you for recognizing the matched-capacity routing results and highlighting the remaining parameter-budget question.

MediVerse’s retrieval gains persist on both added modalities when adapter and expert parameter counts are matched.

We use the same independently routed adapter as in our earlier expansion comparison, increasing its hidden width from 256 to 4,100 to match the expert’s 50.412M added parameters per modality. Both methods use the same frozen ViT-L backbone, adaptation data and eight-epoch training budget.

Added moduleParameters/modality (M)Fundus retrieval (%)OCT retrieval (%)
Parameter-matched adapter50.41218.778 ± 0.66111.354 ± 0.320
MediVerse new expert50.41220.598 ± 0.33812.905 ± 0.440

Retrieval is mean bidirectional Recall@1/10/50; values are mean ± sample SD across three runs.

Preservation is shared by both approaches; the measured distinction is new-modality acquisition at matched parameter capacity. These results show that the retrieval advantage persists when the adapter’s parameter count is increased to match the expert’s.

Reviewer_ZZBNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MediVerse, a medical vision–language model with modality-specific experts. The proposed MediVerse combines masked pretraining on 19.4M images with alignment on 3.2M image–text pairs. New experts are trained while existing paths remain frozen. Empirical experiments across nine modalities demonstrate strong classification and retrieval performance, with unchanged old-modality representations after expansion.

Critical strengths
  1. This work is motivated-well. Specifically, learning from image-only data and adding new modalities without changing existing representations is a meaningful in medical foundation model.
  2. The experiments cover 25 settings across nine modalities, and the proposed MediVerse consistently outperforms the strongest baselines in zero-shot classification and retrieval.
Critical weaknesses
  1. The methodological novelty is limited. Technically, MUSK [1] already combines unpaired masked pretraining with image–text alignment, and progressive networks [2] preserve earlier computations through frozen modules and added capacity. The main contribution of this work is deterministic modality-specific FFNs. Proposition 1 correctly formalizes preservation by construction, but adds little methodological novelty.
  2. The source of the performance gains is not fully clear. MediVerse uses 1.28B stored parameters, six visual teachers, and 19.4M pretraining images. The only five-epoch ViT-B ablation cannot clearly separate the effect of routing from the benefit of having more model capacity.
  3. The comparison for model expansion is not fully fair. Table 4 compares independently trained full-data experts with LoRA adaptation using smaller subsets. The paper mentions this difference and refers to Appendix E, which was not included in the provided PDF. A stronger baseline would be per-modality LoRA or adapters on the same frozen backbone, with comparisons of new-modality performance and additional parameter cost.
  4. Several important experimental details are only provided in appendices that were not included in the submitted PDF. Moreover, the main tables also do not provide variance, even for very small differences such as 0.838 versus 0.837 on OCTDL. This makes some of the reported improvements difficult to assess.

[1] Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y., ... & Li, R. (2025). A vision–language foundation model for precision oncology. Nature, 638(8051), 769-778. [2] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., ... & Hadsell, R. (2016). Progressive neural networks. arXiv preprint arXiv:1606.04671.

Questions for authors

See strengths and weaknesses, please.

4 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the thoughtful feedback, which helped us clarify MediVerse’s contribution and strengthen the experimental comparisons.

W1. Methodological novelty

MEDIVERSE's novelty lies in (1) using multiple modality-specific visual FFNs, unlike MUSK, and (2) expansion through new FFN experts within the existing Transformer, rather than separate task-specific networks as in progressive networks.

1. Modality-specific visual FFNs

  • Our main architectural contribution is modality-specific visual FFNs. MUSK uses one visual FFN across image inputs; MediVerse combines a shared visual FFN with dedicated multiple FFNs for CT, MRI, pathology and other imaging modalities. At each routed layer, the shared and selected expert outputs are added.

  • Higher performance under matched capacity. This modality routing outperforms the shared-extra-FFN control across all three task-family means (Table 1).

  • Data, initialization, tokenizer and training schedule are identical across variants. (The schematics are added in the revision)

table 1. Capacity comparison mean ± sample SD across three training seeds (%)

ArchitectureZero-shot BA (%) ↑Probe BA (%) ↑Retrieval (%) ↑
Plain49.277 ± 2.34385.063 ± 0.5855.220 ± 0.173
Shared extra FFN48.945 ± 0.63584.971 ± 0.4445.359 ± 0.145
Random routing47.128 ± 0.73283.894 ± 0.2225.168 ± 0.165
Modality routing (ours)50.437 ± 0.92385.282 ± 0.2065.780 ± 0.159

2. Modality-aware expansion

  • Progressive networks add a separate task-specific network connected to earlier frozen networks; MediVerse expands the existing Transformer through modality-specific FFNs. New experts reuse the frozen backbone and align images with the unchanged text embedding space. Using the same frozen ViT-L backbone and adaptation data, they achieve higher mean fundus and OCT retrieval than independent adapters and LoRA (Table 2), with greater added capacity.

Table 2. Matched expansion mean ± sample SD across three training seeds (%)

Added moduleFundus retrieval (%)OCT retrieval (%)Added parameters (M)
Independent adapter16.596 ± 0.32712.662 ± 0.3883.153
Independent FFN-output LoRA13.858 ± 0.08512.926 ± 0.0810.492
MediVerse new expert21.171 ± 0.38913.444 ± 0.21850.412
  • Interpretable clinical integration. Modality-aware expansion simplifies clinical deployment by allowing new modalities to be added without affecting existing ones, while improving explainability through modality-specific routing.

Revision: The above discussion and tables have been incorporated into the revised manuscript.

AuthorsOpenReview ↗

Response to W2

W2. Source of the performance gains

Our controlled experiments show that (1) modality-specific routing improves performance beyond simply increasing model capacity, and (2) using six visual teachers instead of one further improves performance across all three task families.

(1) Routing versus capacity. The performance improvements from modality-specific routing have already been demonstrated in W1.

(2) Multiple visual supervision. A separate three-seed comparison fixes the routed architecture and training budgets. Six teachers achieve higher mean performance than a single teacher across all three task families.

Table. Teacher comparison mean ± sample SD across three training seeds (%)

Teacher supervisionZero-shot BA (%)Probe BA (%)Retrieval (%)
Single teacher48.322 ± 1.27885.066 ± 0.2995.176 ± 0.169
Six teachers48.674 ± 1.08285.479 ± 0.1105.528 ± 0.125

Revision: The above discussion and tables have been incorporated into the revised manuscript.

AuthorsOpenReview ↗

Response to W3

W3. Fair comparison for model expansion

We added the requested comparison: independent experts, adapters and per-modality LoRA on the same frozen ViT-L backbone, using identical adaptation data and reporting new-modality performance and added parameters.

  • Matched training. All methods use the same 20k fundus and 16k OCT pairs, validation/test splits, three training seeds and validation-only checkpoint selection. Each modality receives its own module; the backbone and previously trained modules remain frozen.

Table. Expansion comparison mean ± sample SD (percent); added parameters are per modality.

Added moduleFundus retrieval (%) ↑OCT retrieval (%) ↑Added parameters/modality (M)
Independent adapter16.596 ± 0.32712.662 ± 0.3883.153
Independent FFN-output LoRA13.858 ± 0.08512.926 ± 0.0810.492
MediVerse new expert21.171 ± 0.38913.444 ± 0.21850.412
  • Performance and parameter cost. MediVerse achieves the highest mean retrieval on both modalities, with 50.412M additional parameters per modality, compared with 3.153M for adapters and 0.492M for FFN-output LoRA. These results quantify the performance–parameter trade-off under the same data and backbone conditions.
  • Revised PDF. We have corrected the appendix reference and included the main paper, complete supporting appendices, and this comparison in the revised manuscript.
AuthorsOpenReview ↗

Response to W4

W4. Experimental accessibility and uncertainty

The revised PDF includes the supporting appendices, and the new controlled comparisons report variability across three training seeds. We interpret the near-equal OCTDL scores as competitive performance without claiming statistical superiority.

  • Accessible experimental details. We have combined the main paper and supporting appendices into one PDF and corrected the appendix reference. The experimental protocols are provided in Appendix.
  • Scope of uncertainty estimates. The new comparisons discussed in W1–W3 report mean ± sample SD across three training seeds. These estimates quantify variability in those experiments; they do not provide uncertainty estimates for the original main-table results. Section 4.3 explicitly states this limitation.
  • OCTDL interpretation. MediVerse scores 0.838 versus 0.837 for ConceptCLIP. We describe these results as numerically comparable and do not interpret the 0.001 difference as evidence of statistical superiority. The result demonstrates competitive OCT performance after adding a modality-specific expert.

Revision: The above discussion has been incorporated into the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers for their constructive feedback. We have revised the manuscript, with changes highlighted in blue. The main revisions are summarized below.

  • Architectural contribution (RNBN, ZZBN, 7A4B). Clarified the differences from MUSK and progressive networks, the preservation conditions, and the independent Stage-1 objectives (Sections 2–3; Figures 2 and 13).

  • Capacity and teacher controls (RNBN, ZZBN, 7A4B). Added shared-FFN, random-routing and teacher comparisons. Modality routing and six-teacher supervision each improve all three task-family means in their respective comparisons (Tables 5 and 7).

  • Learned routing (7A4B). Added a comparison using the same expert bank. Modality routing improves mean zero-shot and retrieval scores, with similar probing performance and measured computation (Tables 18 and 22).

  • Expansion and retention (RNBN, ZZBN, 7A4B). Added matched comparisons with adapters, LoRA and learned routing, including parameter-matched wide adapters. Reported acquisition, retention and parameter costs (Tables 6, 19–20 and 23).

  • Routing errors and limitations (7A4B). Added correct/incorrect/plain comparisons and discussed the Thyroid exception. Clarified dependence on modality labels, expansion cost and unresolved automatic error detection (Table 21; Section 4.8).

  • Pathology specialists (RNBN). Added MUSK and CONCH comparisons and clarified pretraining differences. MediVerse leads on PanNuke and HEST, with competitive PathGen retrieval (Table 8).

  • Consistency and reporting (RNBN, ZZBN). Corrected derived averages while preserving original per-task scores, included complete appendices, and clarified the near-equal OCTDL results. New repeated-training comparisons report mean ± sample SD across 3 seeds.

Our individual responses provide the supporting evidence and manuscript locations for each concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.