acceptodds
Under review as a conference paper at ICLR 2027

A Pathology Generative Vision-Language Model with Native-Preserving Expert Injection

Abstract

Pathology vision-language reasoning requires fine-grained morphological perception together with flexible multimodal reasoning and generation. General-purpose vision-language models (VLMs) provide broad reasoning capabilities, but their visual representations may miss pathology-specific evidence. Pathology foundation models (PFMs) capture detailed histopathological features but typically lack open-ended reasoning and generation. We introduce PathTIDE (Pathology Token Injection of Domain Experts), an architecture that retains the native VLM pathway while adding a pathology foundation model as a Pathology Expert. PathTIDE aligns Pathology Expert representations with a language-aware feature space and injects spatially structured expert tokens through gated residual fusion. We evaluate PathTIDE on closed-ended pathology VQA, including multiple-choice and Yes/No questions. Across four closed-ended VQA evaluation sets—PathMMU, SlideBench-VQA-TCGA, SlideBench-VQA-CPTAC, and the closed-ended portion of WSI-Bench—PathTIDE (PFM-Full) achieves accuracies of 0.869, 0.793, 0.812, and 0.869, respectively. Architecture ablations show that preserving the native General Expert and using gated expert injection outperforms both General Expert replacement and ungated residual addition. These results support adding pathology-specific evidence to the native visual branch of a general-purpose VLM.

Primary area:
generative models
Keywords:
computational pathology, vision-language models, pathology foundation models, multimodal learning, expert injection, visual question answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zAc9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PathTIDE couples a pretrained VLM (Qwen3.5-9B) with a pathology foundation model (MUSK) while preserving the VLM’s native visual pathway. Native visual tokens query projected MUSK patch tokens via cross-attention, and a learned gated residual injects pathology evidence without changing the native token layout. Training first aligns MUSK to a frozen Qwen3-VL-Embedding-2B teacher on PathGen-1.6M, then applies SFT and GRPO with a 9:1 expert-active/expert-bypassed schedule. Evaluation on PathMMU, SlideBench-VQA-TCGA, SlideBench-VQA-CPTAC, and closed-ended WSI-Bench reports accuracy with 1,000-resample bootstrap CIs, plus ablation and linear-probe results.

Critical strengths

PathTIDE addresses a clearly motivated integration problem: it keeps the pretrained VLM’s native visual-token layout and visual-language interface while adding a separately encoded pathology foundation model as a complementary expert. This is consequential because it avoids rebuilding multimodal coupling or replacing the native pathway, and makes the pathology branch modular and interchangeable in principle.

The matched post-training ablation is a substantive strength. Comparing the base model, matched pathology post-training, replacement of the General Expert Branch, ungated residual addition, and full gated fusion under the same backbone and evaluation protocol goes beyond reporting a single high-scoring system and attempts to attribute gains to native-pathway preservation and learned gating. The PFM-Free/PFM-Full comparison using the same final checkpoint further helps separate the complete expert-enabled route from the residual update alone, strengthening the paper’s internal argument.

Critical weaknesses

The PathMMU result is implausibly high relative to all published frontier systems and human experts, and the paper provides no per-question evidence to verify it. PathTIDE reports 0.869 on PathMMU, whereas the benchmark’s original human-expert baseline is 71.8% and its zero-shot baseline is 49.8%; recent independent literature reports the strongest frontier systems at 76.8%, with other reported systems between 63.97% and 72.3%. A 9B-parameter open model outperforming all reported frontier systems and human experts by this margin requires direct verification. Without sample-level predictions (question ID, gold answer, model answer), the paper cannot rule out that a substantial fraction of correct answers is driven by language priors, option patterns, or benchmark noise rather than visual pathology evidence, and this matters because the paper’s central empirical claim rests on this result.

The TCGA-derived evaluation benchmarks overlap with the TCGA-derived training corpus at the patient level, and the paper’s decontamination does not address this. PathGen-1.6M, used in both training stages, is TCGA-derived, and independent audits report case-level overlap of 83.1% between PathGen-1.6M and SlideBench-VQA-TCGA and 90.7% between PathGen-1.6M and WSI-Bench-test. The paper’s decontamination is slide-ID-level only; it does not exclude different slides from the same patient, and it cannot be applied to patch-level PathGen data, which lacks slide identifiers. The reported gains on SlideBench-VQA-TCGA and WSI-Bench may therefore be inflated by patient-level leakage rather than pathology reasoning ability, and this matters because two of the four main benchmarks are TCGA-derived, so the headline WSI claims are not supported by patient-disjoint evaluation.

Questions for authors

PathTIDE reports 0.869 on PathMMU, while the benchmark’s human-expert baseline is 71.8% and the strongest published frontier systems do not exceed 76.8%. Please provide sample-level prediction files (question ID, gold answer, model answer) for PathMMU and report accuracy on a visual-evidence-necessary subset. Without this, a reader cannot determine whether the result reflects pathology reasoning or reliance on language/option priors.

PathGen-1.6M is TCGA-derived, and independent audits report case-level overlap of 83.1% with SlideBench-VQA-TCGA and 90.7% with WSI-Bench-test. Please specify how decontamination is performed at the patient and TSS levels, and how PathGen-1.6M patches from overlapping TCGA cases are excluded given that patch-level data lack slide identifiers. A patient-disjoint evaluation is needed to support the SlideBench and WSI-Bench claims.

Reviewer_cw5aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PathTIDE, which adds pathology-specific visual features to a general-purpose vision-language model while retaining its native visual pathway. Native visual tokens query pathology expert features through cross-attention, and a token-wise gate controls the residual update without increasing the number of visual tokens passed to the language model. Training consists of expert alignment, supervised fine-tuning, and reinforcement learning. PathTIDE is evaluated on several pathology benchmarks, with ablations covering fusion designs, training stages, and visual representations. It also supports a Free mode that bypasses the expert branch using the same final checkpoint.

Critical strengths
  1. The experiments are comprehensive. The paper compares general-purpose, medical, pathology-specific, and WSI-supporting models, and analyzes native pathway preservation, expert fusion, the training phase, and visual representations. Ablation experiments provide evidence for the effectiveness of incorporating pathology expert information and preserving two visual branches.

  2. Incorporating expert information while preserving the native visual pathway—without increasing the number of final visual tokens—has practical value. A single model checkpoint supports both Full and Free inference modes. The model achieved significant improvements across multiple pathology tasks.

Critical weaknesses
  1. The relationship to existing expert fusion methods is insufficiently discussed. MoVA uses base visual features to query expert features through cross-attention, residual connections, and dynamic gating. In the medical domain, EyExIn retains a general-purpose visual encoder and uses token-wise gating and residual injection to incorporate domain expertise. Although these methods differ from PathTIDE in routing and injection design, both are relevant to its architectural contribution. Neither is cited or discussed, leaving the methodological advance unclear.

  2. The ablations do not fully distinguish the benefit of token-wise gating from that of reducing the overall strength of expert injection. The Add baseline fixes the gate to 1, so its comparison with PathTIDE does not establish an advantage over a tuned fixed coefficient or a learned global scalar. In the WSI experiments, the benefit of pathology-specific representations is also not clearly separated from access to finer-grained tissue patches.

Questions for authors
  1. MoVA (https://arxiv.org/abs/2404.13046) uses base visual features to query expert features through cross-attention, residual fusion, and dynamic gating. EyExIn (https://arxiv.org/abs/2603.07131) also retains a general visual encoder, introduces a domain expert, and uses token-wise gating and residual injection. Could you discuss how PathTIDE differs from these methods and clarify its main methodological contribution? A comparison with a closely related fusion design, using the same backbone, pathology expert, and training budget, would help establish this contribution.

  2. In Table 2, Add fixes all gate values to one, whereas PathTIDE learns a separate injection strength for each token. Do the gains come primarily from token-wise adaptation, or from reducing the overall magnitude of the expert update? Could you compare against a tuned fixed injection coefficient or a learned global scalar to determine whether token-wise gating provides an additional benefit?

  3. For WSI tasks, the pathology expert encodes tissue patches, whereas conventional image-interface VLM baselines receive a whole image or thumbnail. How much of the improvement comes from the pathology expert itself rather than finer-grained visual input? Could the native general-purpose visual encoder process the same patches through the same fusion mechanism and be compared with MUSK? This would help determine whether a pathology expert is necessary, or whether clearer local views could yield similar gains.

  4. PathTIDE is presented as a PFM-agnostic architecture, but all experiments use MUSK. Are the gains dependent on the particular combination of MUSK and the alignment procedure? Could you demonstrate the effectiveness of the same injection mechanism with other pathology foundation models to support its applicability beyond this specific model combination?

Reviewer_3wFyOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces PathTIDE, which augments a pretrained Qwen vision-language model with a MUSK pathology encoder while retaining the native visual branch. Pathology features are first adapted through contrastive and relational alignment, then incorporated through gated cross-attention. The model undergoes supervised fine-tuning followed by GRPO with answer, format, and evidence rewards. Evaluation covers four closed-ended pathology VQA settings and frozen-feature classification. The reported improvements are substantial, and comparisons with encoder replacement and ungated fusion support the usefulness of combining the two visual branches. That said, the exclusive choice of MUSK as the pathology encoder is insufficiently justified, particularly for a method presented as PFM-agnostic. Independent evaluations provide good reason to consider alternatives. In the independent PathBench evaluation (https://arxiv.org/pdf/2505.20202), MUSK ranked last among all 19 evaluated models across the 10 lung-cancer tasks, with an average rank of 18.8, while another benchmark PathoCellBench reports less favorable performance and scaling than DINOv2-based encoders. These findings do not necessarily predict performance after integration into a generative VLM, but they make the absence of any controlled encoder comparison consequential. The authors should explain, backed by extensive verifiable experimentation, why MUSK was selected and test whether the proposed fusion provides consistent improvements with alternative encoders such as Virchow 2, UNI2h, CONCH1.5, or Hoptimus. As presented, the experiments demonstrate one MUSK Qwen configuration and provide limited evidence that the architectural conclusions generalize across pathology encoders.

Critical strengths

A. The problem is well motivated. General VLMs and pathology encoders have complementary capabilities , and incorporating specialist features into an existing multimodal interface is a useful direction. The ability to run the same checkpoint with or without the pathology branch is also practically interesting.

B. The empirical gains are large enough to warrant attention. For example, Path MMU accuracy increases from 0.755 for Qwen 3.5-9B + PT to 0.869 for the full model. The paper includes matched post-training comparisons and a reward ablation, rather than relying entirely on comparisons with unadapted models. I also appreciate the explicit reporting of training-slide overlap and the steps taken to remove it.

Critical weaknesses

A. The evaluation leaves important questions about data independence and visual grounding unresolved which signficantly dampens my enthusiasm about the paper. Appendix A.2 describes filtering SlideInstruct and WSI-LLaVA training records against evaluation slide identifiers. However I could not find an equivalent audit for PathGen 1.6M, which is used in Stage1 and is also listed among the Stage 2 sources. This matters because PathGen derives its patches from TCGA slides. Excluding a slide from instruction tuning would not remove exposure through patches used to adapt MUSK. The audit also establishes slide level rather than patient level separation, as the authors also themselves acknowledge. Please extend the provenance analysis across both training stages and report results on cases confirmed to be independent wherever identifiers permit. The current documentation leaves the extent of residual overlap unknown.

B. The evidence reward raises a separate concern. GPT5.6 Luna scores rationales after being given the correct answer, but no independent validation establishes that these scores reliably distinguish supported observations from plausible explanations of that answer. Table 3 shows that the reward improves answer accuracy it does not establish that the resulting reasoning is more faithful to the image. Similarly, an increase in the optimized judge score is not independent evidence of better grounding. A blinded pathologist assessment of a manageable sample, together with image-removal or image-swapping controls, would be much more informative. Figure 5 makes this concern concrete. The bladder example treats an apparent extravesical mass at low magnification as evidence for PT3B. That category concerns macroscopic perivesical extension low magnification on a histological section does not by itself establish the gross finding. The authors should clarify whether supporting gross information was available and have the rationale independently reviewed. Agreement with the reference answer is insufficient to label the explanation as correct visual reasoning.

C. The ablations support the full system, but do not establish the claimed architectural contribution convincingly. Table 2 is useful, particularly the comparison with Add+PT. However, Add fixes the gate to one and omits the gate regularizer. Its comparison with PathTIDE therefore does not distinguish the value of token-dependent gating from the value of controlling residual strength more generally. A learned scalar gate would be a straightforward and informative control. There is also no downstream comparison using unadapted MUSK, or removing the Stage 1 alignment terms, so the necessity of that stage remains unclear. The optimization curves in Figure 6 do not answer this question. The external comparisons also need careful interpretation. For slide tasks, PathTIDE accesses retained tissue patches through MUSK, whereas conventional image-interface baselines receive a thumbnail. These comparisons demonstrate the benefit of the complete system, but cannot separate specialist representations from access to additional image detail. The Add comparison partly addresses this concern; a general visual encoder processing the same patches would help establish what pathology specialization contributes. More fundamentally, retaining the native interface is not evidence that its original capabilities are preserved. Both the native visual pathway and language backbone receive LoRA updates, and all reported VQA evaluations are within pathology. PFM-Free is another inference route through the adapted checkpoint, so its pathology performance does not establish retention of general capabilities. A small before/after evaluation outside pathology is needed to support that motivation. Likewise, the PFM-agnostic claim is plausible from the interface design but is demonstrated with only one encoder pairing.

D. MUSK ranked last among 19 models across the 10 lung-cancer tasks in the independent PathBench evaluation. Yet the paper uses MUSK exclusively, without a clear rationale or matched comparisons with alternatives such as Virchow2, Uni2h, conch1.5 or Hoptimus. This leaves the claimed generality across pathology encoders untested.

Questions for authors

Were evaluation slides & related patient cases excluded from the PathGen data used in both stages? Can the authors provide the corresponding counts and results on a verified independent subset?

What parameters are trained during SFT for Qwen3.5 9B+PT? Supplenent B.4 describes a warm start that freezes the native model and trains only injection modules, which this baseline does not contain. Clarifying how the baseline receives equivalent adaptation would materially affect my interpretation of Tab 2.

What visual input does the evidence judge receive for whole-slide examples, and is there any independent assessment of its ability to evaluate fine-grained claims? Please also explain how the Fig5 rationales were generated sinc B.6 specifies answer-only benchmark responses.

What is the bootstrap sampling unit? With multiple questions associated with a slide, uncertainty should account for clustering by slide or patient. Reporting unique slide counts alongside question counts would help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.