- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper systematically investigates whether pathology multimodal large language models (PMLLMs) effectively utilize slide-specific visual evidence for whole-slide image (WSI) understanding. The authors evaluate seven representative models through counterfactual visual interventions, cross-model behavioral comparisons, spatial evidence analysis, and controlled training experiments. They also introduce VAPath, a benchmark containing 3,605 WSI question-answering samples with 19,956 pathology-annotated visual anchors. Experimental results reveal substantial discrepancies between benchmark accuracy and visual evidence utilization. Model predictions often remain insensitive to visual perturbations, exhibit weak alignment with diagnostically relevant regions, and show limited improvements in visual dependence following multimodal instruction tuning.
Critical strengths
-
Important research question and systematic evaluation. The paper addresses a critical gap in pathology MLLM evaluation by distinguishing benchmark accuracy from genuine visual evidence utilization. Its comprehensive analysis, combining counterfactual interventions, matched slide-pair comparisons, spatial attribution, and training experiments, provides complementary evidence of insufficient reliance on slide-specific information.
-
Valuable benchmark contribution. The proposed VAPath benchmark integrates 3,605 WSI question-answering samples with 19,956 pathology-annotated visual anchors across diverse diagnostic tasks. By enabling quantitative assessment of spatial evidence alignment, it provides a useful resource for evaluating visual grounding beyond conventional answer accuracy, potentially benefiting future pathology MLLM research.
Critical weaknesses
The comparison between external CONCH retrieval and model-derived attention or causal signals lacks methodological consistency, as these approaches operate at different representation levels. Moreover, retrieval does not verify diagnostic correctness. Consequently, the experiments cannot conclusively establish whether visual information is lost during decision-making.
The instruction-tuning analysis relies on a single partially frozen architecture, two training datasets, and one training epoch. The observed reduction in visual contribution may reflect optimization limitations or generalization issues rather than a fundamental weakness of multimodal instruction tuning.
Questions for authors
Given that CONCH retrieval and model-derived signals operate at different representation levels, how can the authors establish that diagnostically relevant information remains accessible to the PMLLM? Can additional controlled comparisons distinguish information loss during visual compression from insufficient downstream utilization?
To what extent do the observed reductions in visual contribution depend on the specific SlideChat initialization, frozen components, and one-epoch training protocol?
3 comments
Response to Reviewer ZgHT (2/2)
W2 & Q2: Instruction tuning across model architectures
Thank you for the insightful question. To examine this, we extended our instruction-tuning analysis to two additional pathology MLLMs, WSI-LLaVA and HistoSelect, which have different architectures and component configurations from SlideChat.
- Instruction tuning reduces mean visual gain on the test split across all three models and both training corpora, indicating that this pattern extends beyond SlideChat. We summarize these results in Table 1 of the revised manuscript.
- We detail the training settings for SlideChat, WSI-LLaVA, and HistoSelect in Appendix G.3, including the initialization checkpoints, trainable and frozen components, optimization settings, training schedules, and random seeds.
| Model | SFT data | Eval. set | Before NLL (w/o WSI) | Before NLL (w/ WSI) | After NLL (w/o WSI) | After NLL (w/ WSI) | Before | After | (w/o WSI) | (w/ WSI) |
|---|---|---|---|---|---|---|---|---|---|---|
| SlideChat | SlideInstruction | Train | 4.873 | 4.596 | 0.249 ± 0.028 | 0.248 ± 0.033 | +0.277 | +0.001 ± 0.029 | 4.624 ± 0.028 | 4.348 ± 0.033 |
| SlideChat | SlideInstruction | Test | 8.874 | 4.801 | 4.469 ± 0.042 | 4.532 ± 0.052 | +4.074 | -0.062 ± 0.087 | 4.405 ± 0.042 | 0.269 ± 0.052 |
| SlideChat | WSI-BenchTrain | Train | 5.968 | 5.997 | 0.822 ± 0.008 | 0.785 ± 0.004 | -0.029 | +0.037 ± 0.010 | 5.146 ± 0.008 | 5.212 ± 0.004 |
| SlideChat | WSI-BenchTrain | Test | 8.874 | 4.801 | 4.695 ± 0.544 | 4.937 ± 0.387 | +4.074 | -0.242 ± 0.456 | 4.179 ± 0.544 | -0.137 ± 0.387 |
| WSI-LLaVA | SlideInstruction | Train | 2.325 | 1.799 | 0.505 ± 0.018 | 0.451 ± 0.002 | +0.526 | +0.055 ± 0.017 | 1.819 ± 0.018 | 1.348 ± 0.002 |
| WSI-LLaVA | SlideInstruction | Test | 4.399 | 3.273 | 2.897 ± 0.090 | 2.976 ± 0.090 | +1.127 | -0.079 ± 0.097 | 1.502 ± 0.090 | 0.297 ± 0.090 |
| WSI-LLaVA | WSI-BenchTrain | Train | 3.563 | 3.068 | 0.724 ± 0.002 | 0.688 ± 0.006 | +0.495 | +0.037 ± 0.005 | 2.839 ± 0.002 | 2.381 ± 0.006 |
| WSI-LLaVA | WSI-BenchTrain | Test | 4.399 | 3.273 | 3.140 ± 0.198 | 3.029 ± 0.268 | +1.127 | +0.111 ± 0.082 | 1.260 ± 0.198 | 0.244 ± 0.268 |
| HistoSelect | SlideInstruction | Train | 4.899 | 4.739 | 0.243 ± 0.002 | 0.234 ± 0.004 | +0.160 | +0.009 ± 0.003 | 4.657 ± 0.002 | 4.505 ± 0.004 |
| HistoSelect | SlideInstruction | Test | 8.865 | 4.734 | 4.058 ± 0.129 | 4.134 ± 0.110 | +4.132 | -0.077 ± 0.078 | 4.808 ± 0.129 | 0.600 ± 0.110 |
| HistoSelect | WSI-BenchTrain | Train | 5.969 | 6.058 | 0.858 ± 0.008 | 0.844 ± 0.007 | -0.089 | +0.014 ± 0.002 | 5.111 ± 0.008 | 5.213 ± 0.007 |
| HistoSelect | WSI-BenchTrain | Test | 8.865 | 4.734 | 4.583 ± 0.283 | 4.674 ± 0.310 | +4.132 | -0.091 ± 0.101 | 4.283 ± 0.283 | 0.060 ± 0.310 |
Note: After-training values and training improvements are reported as mean ± sample standard deviation. reports the mean NLL reduction relative to the fixed before-training baseline. SFT data denotes the instruction-tuning dataset, and Eval. set denotes the evaluation subset (Train/Test).
Dear Reviewer,
Thank you once again for your time and constructive comments on our paper. As the discussion period is drawing to a close, we would like to gently remind you of our response and the revised manuscript. We sincerely hope that our rebuttal has addressed your concerns, and we would greatly appreciate it if you could take them into consideration in your final assessment. Thank you very much!
Response to Reviewer ZgHT (1/2)
Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.
W1 & Q1: Spatial alignment and diagnostic information
Thank you for raising these concerns. We would like to clarify that we compare the three signals specifically in terms of their spatial alignment with pathologist-annotated ROIs, using APnorm as a common evaluation criterion. This comparison concerns their ROI localization performance, rather than implying that they operate at the same representation level or measure equivalent aspects of visual information utilization. To examine whether diagnostic information remains recoverable and is reflected in native decisions, we conducted stage-wise probing and matched-slide analyses.
Stage-wise probing examines diagnostic recoverability:
Matched-slide analysis examines whether recoverable information is reflected in native decisions:
Together, these analyses show that diagnostic information remains recoverable within visual representations but is not reliably translated into native diagnostic decisions. This insufficient translation of retained visual information into final decisions complements the behavioral and training evidence in §7, providing an account of the gap between visual information availability and utilization. Full implementation details are provided in Appendix F.7.
[1] Gurnee and Tegmark, Language Models Represent Space and Time (2024).
[2] Wiher et al., On Decoding Strategies for Neural Text Generators (2022).
[3] Brown et al., Language Models are Few-Shot Learners (2020).