acceptodds
Under review as a conference paper at ICLR 2027

Blind at Gigapixel Scale: Visual Information Utilization in MLLMs for Whole-Slide Images

Abstract

Pathology multimodal large language models (PMLLMs) have attracted increasing attention for their ability to process whole-slide images (WSIs) and support tasks such as diagnosis assistance, question answering, and report generation. Despite strong performance on existing benchmarks, whether these models effectively utilize slide-specific visual evidence remains unclear. Unlike normal images, WSIs require highly compressed visual representations due to their gigapixel scale, raising questions about whether diagnostically relevant information can be accessed and incorporated into model decisions. We systematically analyze visual evidence utilization in PMLLMs by examining their inputs, representations, and predictions. We evaluate representative models across different input paradigms under controlled visual interventions and introduce VAPath, a WSI–QA–ROI benchmark with pathologist-annotated visual anchors for assessing whether model-derived visual signals align with pathology-relevant regions. Our analysis reveals a gap between visual information availability and decision-level utilization: visual interventions often have limited effects on model predictions, while an independent pathology retrieval procedure identifies stronger ROI-associated patterns in WSI patch features than model-derived signals. We further show that multimodal instruction tuning can improve answer fitting without increasing the measured marginal contribution of WSI visual information under our evaluation protocol. These findings demonstrate that benchmark accuracy alone provides an incomplete measure of WSI understanding and motivate evaluations that directly characterize how visual evidence contributes to PMLLM decisions.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Models, Whole-Slide Images, Visual Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZgHTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically investigates whether pathology multimodal large language models (PMLLMs) effectively utilize slide-specific visual evidence for whole-slide image (WSI) understanding. The authors evaluate seven representative models through counterfactual visual interventions, cross-model behavioral comparisons, spatial evidence analysis, and controlled training experiments. They also introduce VAPath, a benchmark containing 3,605 WSI question-answering samples with 19,956 pathology-annotated visual anchors. Experimental results reveal substantial discrepancies between benchmark accuracy and visual evidence utilization. Model predictions often remain insensitive to visual perturbations, exhibit weak alignment with diagnostically relevant regions, and show limited improvements in visual dependence following multimodal instruction tuning.

Critical strengths
  1. Important research question and systematic evaluation. The paper addresses a critical gap in pathology MLLM evaluation by distinguishing benchmark accuracy from genuine visual evidence utilization. Its comprehensive analysis, combining counterfactual interventions, matched slide-pair comparisons, spatial attribution, and training experiments, provides complementary evidence of insufficient reliance on slide-specific information.

  2. Valuable benchmark contribution. The proposed VAPath benchmark integrates 3,605 WSI question-answering samples with 19,956 pathology-annotated visual anchors across diverse diagnostic tasks. By enabling quantitative assessment of spatial evidence alignment, it provides a useful resource for evaluating visual grounding beyond conventional answer accuracy, potentially benefiting future pathology MLLM research.

Critical weaknesses

The comparison between external CONCH retrieval and model-derived attention or causal signals lacks methodological consistency, as these approaches operate at different representation levels. Moreover, retrieval does not verify diagnostic correctness. Consequently, the experiments cannot conclusively establish whether visual information is lost during decision-making.

The instruction-tuning analysis relies on a single partially frozen architecture, two training datasets, and one training epoch. The observed reduction in visual contribution may reflect optimization limitations or generalization issues rather than a fundamental weakness of multimodal instruction tuning.

Questions for authors

Given that CONCH retrieval and model-derived signals operate at different representation levels, how can the authors establish that diagnostically relevant information remains accessible to the PMLLM? Can additional controlled comparisons distinguish information loss during visual compression from insufficient downstream utilization?

To what extent do the observed reductions in visual contribution depend on the specific SlideChat initialization, frozen components, and one-epoch training protocol?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZgHT (1/2)

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.


W1 & Q1: Spatial alignment and diagnostic information

Thank you for raising these concerns. We would like to clarify that we compare the three signals specifically in terms of their spatial alignment with pathologist-annotated ROIs, using APnorm as a common evaluation criterion. This comparison concerns their ROI localization performance, rather than implying that they operate at the same representation level or measure equivalent aspects of visual information utilization. To examine whether diagnostic information remains recoverable and is reflected in native decisions, we conducted stage-wise probing and matched-slide analyses.

Stage-wise probing examines diagnostic recoverability:

  • Since SlideChat uses CONCH patch features as visual input, it provides a natural setting to connect external retrieval of diagnostic regions with the recoverability of diagnostic information throughout the model's visual pathway.
  • Following prior work on decoding structured information from model representations [1], we trained supervised linear probes on representations mean-pooled only over visual-token positions at five stages of SlideChat's visual pathway (CONCH, LongNet, Projector, L28, and final RMSNorm). The probes predict the same diagnostic labels using shared group-disjoint five-fold outer splits.
RepresentationBalanced AccuracyAccuracyF1
CONCH72.778.769.3
LongNet72.777.368.9
Projector72.876.368.5
L2866.173.863.5
Final RMSNorm65.773.863.2
  • Diagnostic information remains linearly recoverable throughout SlideChat's visual pathway, including its late-stage representations.

Matched-slide analysis examines whether recoverable information is reflected in native decisions:

  • We further examined 839 pairs of slides with different ground-truth diagnoses but identical diagnostic prompts and option sets to establish whether it is reflected in the model's predictions. Both slides in each pair were held out from the same stage-specific probe. For each stage, we retained pairs in which the corresponding probe correctly classified both slides and evaluated native decisions using greedy decoding [2] and full-option mean-NLL selection [3].
Probe correctness conditionBoth slides correctly decodedGreedy same-answer rate (%)NLL same-answer rate (%)
Projector456100.091.7
L28381100.091.6
Final RMSNorm385100.091.2
All five stages correct257100.090.3
  • Even when both slides' different diagnoses are correctly recoverable from internal representations, SlideChat produces identical greedy answers for every observed pair and identical NLL choices for approximately 90–92% of pairs.

Together, these analyses show that diagnostic information remains recoverable within visual representations but is not reliably translated into native diagnostic decisions. This insufficient translation of retained visual information into final decisions complements the behavioral and training evidence in §7, providing an account of the gap between visual information availability and utilization. Full implementation details are provided in Appendix F.7.

[1] Gurnee and Tegmark, Language Models Represent Space and Time (2024).
[2] Wiher et al., On Decoding Strategies for Neural Text Generators (2022).
[3] Brown et al., Language Models are Few-Shot Learners (2020).

AuthorsOpenReview ↗

Response to Reviewer ZgHT (2/2)

W2 & Q2: Instruction tuning across model architectures

Thank you for the insightful question. To examine this, we extended our instruction-tuning analysis to two additional pathology MLLMs, WSI-LLaVA and HistoSelect, which have different architectures and component configurations from SlideChat.

  • Instruction tuning reduces mean visual gain on the test split across all three models and both training corpora, indicating that this pattern extends beyond SlideChat. We summarize these results in Table 1 of the revised manuscript.
  • We detail the training settings for SlideChat, WSI-LLaVA, and HistoSelect in Appendix G.3, including the initialization checkpoints, trainable and frozen components, optimization settings, training schedules, and random seeds.
ModelSFT dataEval. setBefore NLL (w/o WSI)Before NLL (w/ WSI)After NLL (w/o WSI)After NLL (w/ WSI)GvisG_{\mathrm{vis}} BeforeGvisG_{\mathrm{vis}} AfterΔNLL\Delta\mathrm{NLL} (w/o WSI)ΔNLL\Delta\mathrm{NLL} (w/ WSI)
SlideChatSlideInstructionTrain4.8734.5960.249 ± 0.0280.248 ± 0.033+0.277+0.001 ± 0.0294.624 ± 0.0284.348 ± 0.033
SlideChatSlideInstructionTest8.8744.8014.469 ± 0.0424.532 ± 0.052+4.074-0.062 ± 0.0874.405 ± 0.0420.269 ± 0.052
SlideChatWSI-BenchTrainTrain5.9685.9970.822 ± 0.0080.785 ± 0.004-0.029+0.037 ± 0.0105.146 ± 0.0085.212 ± 0.004
SlideChatWSI-BenchTrainTest8.8744.8014.695 ± 0.5444.937 ± 0.387+4.074-0.242 ± 0.4564.179 ± 0.544-0.137 ± 0.387
WSI-LLaVASlideInstructionTrain2.3251.7990.505 ± 0.0180.451 ± 0.002+0.526+0.055 ± 0.0171.819 ± 0.0181.348 ± 0.002
WSI-LLaVASlideInstructionTest4.3993.2732.897 ± 0.0902.976 ± 0.090+1.127-0.079 ± 0.0971.502 ± 0.0900.297 ± 0.090
WSI-LLaVAWSI-BenchTrainTrain3.5633.0680.724 ± 0.0020.688 ± 0.006+0.495+0.037 ± 0.0052.839 ± 0.0022.381 ± 0.006
WSI-LLaVAWSI-BenchTrainTest4.3993.2733.140 ± 0.1983.029 ± 0.268+1.127+0.111 ± 0.0821.260 ± 0.1980.244 ± 0.268
HistoSelectSlideInstructionTrain4.8994.7390.243 ± 0.0020.234 ± 0.004+0.160+0.009 ± 0.0034.657 ± 0.0024.505 ± 0.004
HistoSelectSlideInstructionTest8.8654.7344.058 ± 0.1294.134 ± 0.110+4.132-0.077 ± 0.0784.808 ± 0.1290.600 ± 0.110
HistoSelectWSI-BenchTrainTrain5.9696.0580.858 ± 0.0080.844 ± 0.007-0.089+0.014 ± 0.0025.111 ± 0.0085.213 ± 0.007
HistoSelectWSI-BenchTrainTest8.8654.7344.583 ± 0.2834.674 ± 0.310+4.132-0.091 ± 0.1014.283 ± 0.2830.060 ± 0.310

Note: After-training values and training improvements are reported as mean ± sample standard deviation. ΔNLL=NLLBefore−NLLAfter\Delta\mathrm{NLL}=\mathrm{NLL}_{\mathrm{Before}}-\mathrm{NLL}_{\mathrm{After}} reports the mean NLL reduction relative to the fixed before-training baseline. SFT data denotes the instruction-tuning dataset, and Eval. set denotes the evaluation subset (Train/Test).

Dear Reviewer,

Thank you once again for your time and constructive comments on our paper. As the discussion period is drawing to a close, we would like to gently remind you of our response and the revised manuscript. We sincerely hope that our rebuttal has addressed your concerns, and we would greatly appreciate it if you could take them into consideration in your final assessment. Thank you very much!

Reviewer_76i3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a fundamental question in pathology: multimodal large language models (PMLLMs), whether models operating on gigapixel whole-slide images (WSIs) genuinely utilize slide-specific visual evidence when answering pathology questions, or whether their strong benchmark performance is driven primarily by language priors, dataset biases, and compressed representations.

The authors develop a systematic evaluation framework organized around four complementary analyses:

(1) Counterfactual visual interventions, including text-only, blank, and replaced conditions, evaluated using Counterfactual Accuracy Retention (CAR) and Answer Consistency under Input Replacement (ACIR); (2) Cross-model behavioral analyses separating sensitivity to visual inputs from effective evidence utilization; (3) Spatial auditing of attention, causal influence, and independent retrieval signals against pathologist-annotated regions of interest (ROIs); and (4) Slide-swap and instruction-tuning analyses using paired visual preference and visual-gain measures.

A central contribution is VAPath, a WSI–QA–ROI benchmark containing 3,605 samples and 19,956 pathologist-annotated visual anchors drawn from multiple public datasets and a private sarcoma cohort. Seven representative models spanning WSI-input and general-image-input paradigms are evaluated. The experiments consistently suggest that model predictions are surprisingly insensitive to removal or replacement of slide-specific information. For example, replaced-slide CAR remains very high on average, while model-derived attention and causal signals show limited alignment with pathologist-identified ROIs.

An independent CONCH-based retrieval analysis demonstrates comparatively stronger localization of relevant tissue patterns. The instruction-tuning experiments further suggest that multimodal fine-tuning can substantially improve answer fitting without correspondingly increasing and, in some cases, decreasing the marginal contribution of visual information.

Overall, the paper identifies an important gap between the availability of diagnostic visual information and its effective use in downstream PMLLM decisions. This is an important issue for evaluating pathology MLLMs, where high QA accuracy alone may provide a misleading indication of genuine WSI understanding.

Critical strengths
  1. Comprehensive and convergent evaluation framework. The paper goes substantially beyond standard QA accuracy by combining counterfactual intervention, behavioral sensitivity, spatial localization, causal analysis, retrieval, matched slide swapping, and training dynamics. The convergence of these analyses makes the central empirical observation considerably more convincing than a simple image-removal ablation. The 3,605 WSI–QA–ROI samples and 19,956 pathologist-annotated visual anchors provide an important basis for directly assessing whether model-derived signals correspond to pathology-relevant regions.

  2. The matched slide-swap protocol is particularly strong. Holding the question and answer space fixed while swapping patient-disjoint slides provides a clean test of slide-specific dependence. The option-order reversal control further helps distinguish genuine visual insensitivity from generic prediction stability or positional bias. Showing that multimodal training can dramatically improve answer fitting without increasing marginal visual contribution has direct implications for dataset construction and training objectives in pathology MLLMs. Organizing the analysis around explicit research questions and formally defining CAR, ACIR, APnorm, (G_{\text{pair}}), and (G_{\text{vis}}) makes a technically dense paper relatively easy to follow.

Critical weaknesses
  1. Chance-level performance may confound the interpretation of high CAR and visual non-reliance. A major concern is that some model–dataset combinations appear to have full-input accuracy only modestly above chance. In such settings, CAR values close to or even above 100% and small Full-versus-counterfactual accuracy differences may arise partly because the model is already weak on the task, rather than because a competent model has learned to solve the task without using visual information. This distinction is important for the central claim. “The model performs the task successfully without relying on the slide” is substantially stronger than “the model performs poorly and remains similarly poor when the slide is removed or replaced.” CAR values exceeding 100% in some experiments further motivate a more careful statistical treatment and investigation of the affected samples. The authors should therefore report the appropriate chance or majority-class baseline for each dataset/task and provide uncertainty estimates for the principal CAR/accuracy comparisons. More importantly, the counterfactual analysis should be repeated or stratified for (i) model–dataset pairs performing significantly above chance and (ii) samples answered correctly under the Full condition. It would also be useful to analyze the question types and answer distributions associated with CAR >100% to determine whether replacement removes distracting visual information, exposes dataset priors, or simply reflects statistical variability.

  2. The spatial comparison between attention, causal influence, and retrieval is not fully apples-to-apples. The ROI-based spatial analysis is conceptually valuable, but the three compared signals have substantially different semantics and spatial granularity. Retrieval is computed at the patch level using an external representation; attention operates over model-specific visual tokens; and causal influence is estimated through cell-wise interventions on a coarse spatial grid. Their normalized AP values therefore should not necessarily be interpreted as directly comparable measurements of the same phenomenon. The CONCH retrieval baseline is especially important to the paper's “information is available but not used” interpretation, yet the retrieval procedure may identify broadly pathology-relevant or tumor-versus-background regions without demonstrating that those regions contain answer-discriminative information for the specific question. If retrieval uses the maximum similarity over candidate answer descriptions, this concern becomes stronger. Moreover, CONCH features are not necessarily the features actually consumed by each PMLLM. The authors should test retrieval using the correct-answer description alone and, where possible, contrast correct-answer retrieval against distractor-answer retrieval. This would better establish whether the retrieved regions contain question- and answer-specific evidence. The manuscript should also explicitly distinguish “visual information recoverable by an external representation” from “visual information preserved within the evaluated model's own visual pathway.” Relatedly, the relatively modest absolute APnorm values of the retrieval baseline should be discussed carefully. The results support stronger recoverability than the model-derived signals, but the current evidence may not justify an overly categorical conclusion that diagnostic information is fully identifiable upstream and simply ignored downstream.

Questions for authors
  1. What are the chance/majority-class accuracies for each evaluated dataset, and do the principal CAR/ACIR conclusions remain when analysis is restricted to models significantly above chance or to samples correctly answered under Full input?
  2. What explains CAR values above 100%? Are these cases concentrated in particular datasets, question categories, or answer distributions?
  3. Does CONCH retrieval using only the correct-answer description—and especially correct-answer versus distractor contrast—still outperform attention and causal influence under APnorm?
  4. How do the causal results change under multi-cell or complete ROI occlusion compared with single-cell interventions?
  5. How exactly are Blank inputs/features constructed, and to what extent might they represent an out-of-distribution intervention?
  6. What input resolution and preprocessing are used for the general-image-input models, and how much diagnostically relevant information can reasonably survive this representation?
  7. During the instruction-tuning experiments, which components of the multimodal architecture are updated? Please also clarify the shared “before” values, patient-disjointness of the relevant splits, and variation across random seeds.
  8. How does the contribution differ technically and empirically from the closest prior pathology visual-grounding study, particularly Zhang et al. (2026a)?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 76i3 (1/4)

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.


W1 & Q1 & Q2: Chance baselines and CAR above 100%

Thank you for raising these concerns and encouraging further examination of the counterfactual results. Our additional analyses show that the observed visual non-reliance cannot be explained simply by poor task performance. We would like to clarify the intended role of CAR: it quantifies the extent of performance change under counterfactual interventions to assess models' sensitivity to WSI inputs. Explaining the specific causes of occasional accuracy increases is a related but distinct question from our central research question of whether model decisions depend on WSI visual information. Nevertheless, we examine whether cases with CAR above 100% exhibit systematic patterns.

The observed visual non-reliance cannot be explained simply by poor task performance:

  • We report both random-guessing and majority-answer-option baselines for all five datasets. Across all 40 model–dataset combinations, the 95% CIs for full-input accuracy lie entirely above the corresponding random-guessing baselines, and 38/40 combinations also lie entirely above their majority-option baselines.
  • We further analyzed the counterfactual results for samples correctly answered under Full input. Answer consistency under slide replacement ranges from 72.0% to 99.1% across all 40 model–dataset combinations. For SlideChat and both HistoSelect variants, it remains 94.4%–99.1%, with all 95% CI lower bounds above 91%, supporting persistent answer stability even among initially correct predictions. We provide Fig. 1 and the full results with 95% CIs in Appendix E, and add 95% CIs for paired accuracy differences to Appendix Table 6.
DatasetMajority-option baselineRandom-guessing baseline
SlideBench-VQA-TCGA41.37%25.00%
SlideBench-VQA-BCNB42.65%38.18%
WSI-Bench43.63%25.00%
WSI-VQA38.72%25.13%
VAPath35.28%35.28%

Transcription Correction: Cross-checking the original Table 5 (Table 6 in the revised manuscript) against the original per-question predictions identified four Patho-R1 counterfactual entries transcribed under incorrect datasets. We have corrected these entries and rechecked all affected downstream statistics, confirming that the paper's main conclusions hold.

Interpretation of CAR values above 100%:

  • A CAR value above 100% indicates that counterfactual accuracy exceeds full-input accuracy. Across all 40 model–dataset configurations, only 11 of the 120 counterfactual comparisons have bootstrap 95% CIs with lower bounds strictly above 100%, indicating that statistically supported accuracy increases occur in only a minority of comparisons.
  • To further investigate CAR values above 100%, we examined correctness transitions in the 11 model–dataset–intervention comparisons whose bootstrap 95% CI lower bounds exceeded 100% and examined their distributions across datasets, question categories, and ground-truth options.
DimensionCategoryIncorrect → CorrectCorrect → Incorrect
DatasetTCGA23.37%25.87%
BCNB52.59%47.49%
WSI-Bench2.44%2.85%
WSI-VQA0.00%0.00%
VAPath21.60%23.79%
Question categoryMicroscopy10.67%11.67%
Diagnosis52.95%58.97%
Clinical36.39%29.36%
Ground-truth optionA31.01%37.72%
B43.59%35.18%
C21.94%22.54%
D3.08%3.92%
E0.37%0.52%
F0.02%0.12%
G0.00%0.00%

Note: Transitions are from Full to counterfactual conditions. Percentages describe the distribution within each transition direction. Questions from all five datasets are categorized using SlideChat's taxonomy.

  • The two transition directions show broadly similar distributions, with no pronounced concentration in a particular dataset or question category. Overall, these analyses reveal no consistent concentration of correctness changes in particular question categories or ground-truth answer options, or confinement to a single dataset.
AuthorsOpenReview ↗

Response to Reviewer 76i3 (2/4)

W2 & Q3: Spatial signal comparison and answer-specific retrieval

Thank you for pointing this out. We would like to clarify that we compare the three signals specifically in terms of their spatial alignment with pathologist-annotated ROIs, using APnorm as a common evaluation criterion. This comparison concerns ROI localization performance and does not imply that the signals operate at the same representation level or measure equivalent aspects of visual information utilization. We further examine answer specificity, internal diagnostic information, and native decisions through answer-conditioned retrieval comparisons, stage-wise probing, and matched-slide analysis.

Answer-conditioned retrieval shows answer-related spatial localization:

  • We now compute retrieval similarity using only the correct-answer description and report the results below. We compared correct-answer retrieval against the mean scores across all answer options and incorrect options using paired bootstrap resampling.
MethodAPnorm
Mean over incorrect answer options0.1262
Mean over all answer options0.1486
Correct-answer retrieval0.1649
  • Correct-answer retrieval achieves higher APnorm than both answer-option controls, supporting answer-specific spatial localization. The highest overall mean APnorm values for attention and causal influence are 0.0700 and 0.0150, respectively. Full implementation details and confidence intervals are provided in Appendix F.4.

Stage-wise probing examines diagnostic recoverability:

  • Since SlideChat uses CONCH patch features as visual input, it provides a natural setting to connect external retrieval of diagnostic regions with the recoverability of diagnostic information throughout the model's visual pathway.
  • Following prior work on decoding structured information from model representations [1], we trained supervised linear probes on mean-pooled representations only over visual-token positions from five stages of SlideChat's visual pathway (CONCH, LongNet, Projector, L28, and final RMSNorm). The probes predict the same diagnostic labels using shared group-disjoint five-fold outer splits.
RepresentationBalanced AccuracyAccuracyF1
CONCH72.778.769.3
LongNet72.777.368.9
Projector72.876.368.5
L2866.173.863.5
Final RMSNorm65.773.863.2
  • Diagnostic information remains linearly recoverable throughout SlideChat's visual pathway, including its late-stage representations.

Matched-slide analysis reveals a gap between diagnostic recoverability and native decisions:

  • We further examined 839 pairs of slides with different ground-truth diagnoses but identical diagnostic prompts and option sets to establish whether it is reflected in the model's predictions. Both slides in each pair were held out from the same stage-specific probe. For each stage, we retained pairs in which the corresponding probe correctly classified both slides and evaluated native decisions using greedy decoding [2] and full-option mean-NLL selection [3].
Probe correctness conditionBoth slides correctly decodedGreedy same-answer rate (%)NLL same-answer rate (%)
Projector456100.091.7
L28381100.091.6
Final RMSNorm385100.091.2
All five stages correct257100.090.3
  • Even when both slides' different diagnoses are correctly recoverable from internal representations, SlideChat produces identical greedy answers for every observed pair and identical NLL choices for approximately 90–92% of pairs. Full implementation details and confidence intervals are provided in Appendix F.7.

Together, these analyses support answer-specific spatial localization and show that diagnostic information remains recoverable within visual representations but is not reliably translated into native diagnostic decisions. This insufficient translation of retained visual information into final decisions complements the behavioral and training evidence in §7, providing an account of the gap between visual information availability and utilization.

[1] Gurnee and Tegmark, Language Models Represent Space and Time (2024).
[2] Wiher et al., On Decoding Strategies for Neural Text Generators (2022).
[3] Brown et al., Language Models are Few-Shot Learners (2020).

AuthorsOpenReview ↗

Response to Reviewer 76i3 (3/4)

Q4: Multi-cell and ROI occlusion

Thank you for suggesting stronger spatial interventions. We extended the SlideChat analysis to multi-cell ablations and ROI and non-ROI occlusions. We report mean absolute changes in the correct-answer logit margin and the correct option's softmax probability between full and occluded inputs. Probability changes are expressed as percentages.

Multi-cell interventions:

  • We jointly blanked contiguous 2×2 and 4×4 regions on the same 8×8 grid. We evaluated these interventions alongside single-cell occlusion across all five datasets and report results with 95% confidence intervals.
InterventionMean absolute logit-margin changeMean absolute probability change (%)Prediction-change rate
Single cell0.0401 [0.0396, 0.0405]0.410% [0.401%, 0.418%]0.564% [0.482%, 0.652%]
2×2 region (4 cells)0.0503 [0.0496, 0.0510]0.508% [0.497%, 0.518%]0.721% [0.624%, 0.824%]
4×4 region (16 cells)0.0860 [0.0843, 0.0878]0.849% [0.828%, 0.869%]1.368% [1.218%, 1.526%]

ROI/non-ROI-union occlusion:

  • We additionally evaluated two complementary interventions on VAPath pairs: blanking all grid cells overlapping annotated ROI boxes (ROI occlusion) and blanking all remaining cells while retaining the ROI-overlapping cells (non-ROI occlusion). The single-cell results below were recalculated on the same subset for a matched comparison (95% CI).
InterventionMean absolute logit-margin changeMean absolute probability change (%)Prediction-change rate
Single cell0.0407 [0.0399, 0.0415]0.434% [0.416%, 0.453%]0.308% [0.204%, 0.426%]
ROI occlusion0.1210 [0.1164, 0.1256]1.425% [1.347%, 1.506%]1.131% [0.791%, 1.495%]
Non-ROI occlusion0.1306 [0.1253, 0.1361]1.176% [1.110%, 1.246%]1.329% [0.969%, 1.715%]

Overall, larger spatial interventions produce greater changes in model outputs, but their effects on final predictions remain limited. Even when 16 cells (25% of the spatial grid) are jointly ablated, the mean absolute correct-answer probability change is only 0.849%, and 98.63% of predictions remain unchanged. On the matched VAPath subset, jointly occluding all ROI-overlapping cells leaves 98.87% of predictions unchanged. Occluding non-ROI cells likewise leaves 98.67% unchanged. These interventions provide complementary evidence of limited decision-level sensitivity beyond single-cell occlusion.


Q5 & Q6: Blank inputs and whole-slide thumbnails

Thank you for these questions about the visual inputs.

Blank inputs and feature-space validation:

  • Blank inputs are visually empty images passed through each model's native image encoder and feature aggregation pipeline, rather than zeroed latent features, to represent tissue-free background regions naturally occurring in WSIs.
  • To assess their feature-space correspondence, we compared the CONCH blank embedding against 67.36 million patch embeddings from 6,167 real WSIs. Its nearest neighbor was confirmed to be a naturally occurring tissue-free background patch, with an almost zero cosine distance (1.63 × 10⁻¹³).

Thumbnail resolution and diagnostic detail:

  • Following the preprocessing approach used in SlideChat, WSI-LLaVA, and HistoSelect, we resized whole-slide thumbnails to 2,048 pixels on the shorter side, consistent with the native input format of the general-image-input models. Thumbnails preserve global tissue architecture and coarse morphological patterns but lose fine-grained cellular details and small focal structures. This limitation motivates pathology-specific MLLMs (e.g., SlideChat and WSI-LLaVA), which employ specialized visual encoders and feature aggregation to process high-resolution tissue regions.
Reviewer_bNN1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper answer an important question of whether pathology MLLMs use whole-slide images when answering questions. It combines image removal and replacement, spatial analysis against pathology annotations, answer-changing slide pairs, and a controlled instruction-tuning comparison. It also introduces VAPath, containing 3,605 question–slide–ROI samples.

I would say the behavioral evidence is useful and insightful. Several models retain their answers even when the slide changes, and the matched-pair experiments make this observation more convincing than image removal alone. My main reservation concerns the mechanistic interpretation that is the experiments establish weak visual dependence more clearly than they identify where information is lost inside the model.

Critical strengths

The paper does more than compare full-input and text-only accuracy. Its matched slide pairs hold the question and answer choices fixed while requiring different correct answers. This directly tests whether predictions respond to slide-specific evidence. VAPath provides a useful resource with pathology-region annotations that more than a simple text-only ablation.

Critical weaknesses

W1. The spatial analysis does not identify the claimed bottleneck

CONCH retrieval is computed from upstream patch features, and the score takes the maximum over all candidate descriptions, including incorrect answers. Finding tissue associated with any candidate is not the same as finding evidence that distinguishes the correct answer.

Analyse where is the evidence lost is important. Does the current experiment locate that failure, or only show that an external model retrieves some annotated regions while the evaluated model’s signals align poorly with them?

W2. The training result is difficult to reconcile with the implementation description

The appendix says the language model is frozen, but its text-only NLL changes substantially after training. Perhaps adapters are being updated, or the text-only input still passes through trained components. The current description does not say.

What is trained, and why does it change predictions when no image is provided?

W3. The main figures leave too much for the reader to guess

Figure 6 shows the matched-pair result without confidence intervals. Table 1 averages three runs without showing their spread. Its labels also look like different models, even though both conditions use SlideChat trained on different corpus.

W4. Test explainations.

An unchanged answer after random slide replacement can be correct if both slides have the same diagnosis. Removing one region may leave the same evidence elsewhere. A signed average close to zero can also hide positive and negative effects.

Do the conclusions hold when the replacement requires a different answer, when relevant regions are removed together, and when absolute effects are reported?

W5. Figure clearness.

In my view, the overall quality of the figures does not yet meet ICLR presentation standards. The figures appear overly simplistic and lack a consistent visual style, making them difficult to interpret and less polished. I encourage the authors to improve the clarity, consistency, and overall visual quality of the figures to better communicate the proposed methodology and results. Though it will not affect my rating of the paper, I would suggest improve them.

Questions for authors

Q1. Which exact checkpoints and visual inputs were used? What do HistoSelect-S and HistoSelect-W denote, and what representation is replaced for WSI-LLaVA?

Q2. Which releases supply the public ROI contours, particularly TCGA? How is each contour linked to its question? Q3. What are the ethics approval or waiver and access conditions for SARC? Q4. Eq 6 uses sequence log probability, while the appendix describes token-averaged NLL. Which definition produces the reported results? Q5. Please correct “with a average,” “provide a evaluation,” and “(Figure 5).Let,” as well as the malformed Qwen author field and inconsistent TCGA publication year.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bNN1 (1/2)

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.


W1: Spatial analysis and diagnostic information

Thank you for this important insight into the relationship between visual evidence and diagnostic decisions. It motivated us to examine how answer-related visual information is reflected across upstream retrieval, internal representations, and native decisions. We conducted three complementary analyses: answer-conditioned retrieval, stage-wise diagnostic probing, and matched-slide comparisons.

Answer-conditioned retrieval shows correct-answer-related spatial localization:

  • We now compute retrieval similarity using only the correct-answer description and report the results below. We compared correct-answer retrieval against the mean scores across all answer options and incorrect options using paired bootstrap resampling. The results show that correct-answer retrieval achieves higher APnorm than both controls and the evaluated attention and causal signals, supporting the presence of spatially localized information associated with the correct diagnosis in upstream patch features.
MethodAPnorm
Attention¹0.0700
Causal influence¹0.0150
Mean over incorrect answer options0.1262
Mean over all answer options0.1486
Correct-answer retrieval0.1649

¹ The attention and causal rows are the configurations with the highest overall average within their respective signal types.

Stage-wise probing shows that diagnostic information remains readable:

  • Since SlideChat uses CONCH patch features as visual input, it provides a natural setting to connect external retrieval of diagnostic regions with the recoverability of diagnostic information throughout the model's visual pathway.
  • Following prior work on decoding structured information from model representations [1], we trained supervised linear probes on mean-pooled representations only over visual-token positions from five stages of SlideChat's visual pathway (CONCH, LongNet, Projector, L28, and final RMSNorm). The probes predict the same diagnostic labels using shared group-disjoint five-fold outer splits.
RepresentationBalanced AccuracyAccuracyF1
CONCH72.778.769.3
LongNet72.777.368.9
Projector72.876.368.5
L2866.173.863.5
Final RMSNorm65.773.863.2
  • The results show that diagnostic information remains substantially linearly decodable even in late-stage representations, rather than being completely lost during visual processing.

Matched-slide analysis reveals that readable diagnostic distinctions are not expressed in native decisions:

  • Linear probing establishes diagnostic information recoverability but does not establish whether the model itself uses this information in its predictions. We therefore identified 839 pairs of slides with different ground-truth diagnoses but identical diagnostic prompts and option sets, with both slides held out from the same stage-specific probe. For each stage, we retained pairs in which the probe correctly classified both slides and evaluated the model's native decisions using greedy decoding [2] and full-option mean-NLL selection [3].
  • The results show that even when both slides' different diagnoses are correctly recoverable from internal representations, the evaluated model produces identical greedy answers for every observed pair and identical NLL choices for approximately 90–92% of pairs.
Probe correctness conditionBoth slides correctly decodedGreedy same-answer rate (%)NLL same-answer rate (%)
Projector456100.091.7
L28381100.091.6
Final RMSNorm385100.091.2
All five stages correct257100.090.3

Together, these analyses support answer-specific spatial localization and show that diagnostic information remains recoverable within visual representations but is not reliably translated into native diagnostic decisions. This insufficient translation of retained visual information into final decisions complements the behavioral and training evidence in §7, providing an account of the gap between visual information availability and utilization. Implementation details and confidence intervals for the probing and matched-slide analyses are provided in Appendix F.7.

[1] Gurnee and Tegmark, Language Models Represent Space and Time (2024).
[2] Wiher et al., On Decoding Strategies for Neural Text Generators (2022).
[3] Brown et al., Language Models are Few-Shot Learners (2020).

AuthorsOpenReview ↗

Response to Reviewer bNN1 (2/2)

W2 & W3: SlideChat training and presentation of results

Thank you for raising these points.

  • The original manuscript mistakenly used the Stage-1 description, in which the LLM is frozen. Our Stage-2 instruction tuning freezes CONCH and updates LongNet, the Projector, and the LLM. Text-only prompts also pass through the updated LLM, explaining the change in NLL. We correct the training description in Appendix G.3.
  • Table 1 separates models and training corpora, includes results from additional models, and reports standard deviations across runs. Figure 6 includes confidence intervals.

W4: Matched-slide replacement and ROI interventions

Thank you for raising these points.

Matched-slide replacement requires different answers:

  • This concern is addressed by our controlled slide-pair analysis in §7.1. We construct patient-disjoint slide pairs with identical questions and option sets but different ground-truth diagnoses, ensuring that the correct answer must change after slide swapping.
  • In the revised manuscript, we extend this analysis to eight model configurations across seven sources. Among pairs with positive paired visual preference (Gpair>0G_{\mathrm{pair}} > 0), we compare scores for the two matched answers. Across configurations, 82.90%–98.02% still prefer the same candidate on both slides, while only 1.98%–17.10% prefer the respective correct answer for both slides. These conditional rates are computed within each source and then averaged equally, with ties retained in the denominator.

ROI removal:

  • Predictions remained largely stable after joint ROI removal on VAPath.
  • Patch-feature interventions. Removing patch features that overlapped annotated ROIs yielded CAR of 98.34%–99.80% and ACIR of 97.63%–99.05% across SlideChat and HistoSelect. Retaining only ROI patches yielded CAR of 99.73%–100.27% and ACIR of 97.63%–99.14%.
  • Grid-cell occlusion. We tested ROI and non-ROI occlusion on SlideChat. Mean absolute changes in the correct-answer logit margin were 0.1210 [0.1164, 0.1256] and 0.1306 [0.1253, 0.1361], respectively (95% CIs). ROI and non-ROI occlusion changed 1.131% and 1.329% of predictions, respectively. Appendix F.3 reports the detailed results.

Absolute intervention effects:

  • The mean absolute change ∣Δk∣\lvert\Delta_k\rvert is 0.0534±0.08130.0534 \pm 0.0813 across cell-wise interventions, indicating that the interventions produced small changes in the prediction margin on average.

W5 & Q5: Figure, text, and reference revisions

Thank you for your careful reading and feedback. In the revised manuscript, we standardise model colours, fonts, panel labels, and units across figures, correct grammatical errors, and verify the references.


Q1: Model checkpoints and visual inputs

Thank you for the question. We clarify the checkpoint sources, the HistoSelect labels, and the WSI-LLaVA intervention interface in Table 3 of Appendix D.1. HistoSelect-S and HistoSelect-W denote HistoSelect trained on the SlideChat and WSI-LLaVA training corpora, respectively. Both WSI-LLaVA interventions replace the aggregated Gigapath slide representation fed to the LLM. For Blank, we encode blank images into patch features and aggregate them using the official Gigapath pipeline. For Replaced, we use the representation extracted from the replacement WSI.


Q2: ROI sources and question linkage

Thank you for the question. Appendix D.4 now clarifies the ROI sources. For public datasets, we retain ROI annotations from the original releases or separately published resources. We check candidate samples for reliable visual evidence and task consistency. A sample identifier links each QA record to its WSI and ROI.


Q3: SARC ethics approval and data access

Thank you for raising this important question. The SARC cohort was used under a research protocol formally approved by the responsible institutional ethics committee, which granted a waiver of informed consent. All WSIs and associated data were de-identified before analysis. The approval details are withheld during double-blind review to preserve anonymity and will be provided in the final version. Regarding data access, both the original WSIs and derived resources, including ROI annotations and QA pairs, are available through controlled access, subject to applicable institutional approval procedures. We have clarified these arrangements in the revised manuscript.


Q4: Answer-token-averaged NLL in Eq. (6)

Thank you for the question. For Table 1, we average NLL over scored answer tokens within each example, then across examples, as described in Appendix G.3. The revised Eq. (6) makes this normalization explicit.

=\frac{1}{T_i}\left[ \log p(y_i^{\ast}\mid q_i,I_i)-\log p(y_i^{\ast}\mid q_i) \right].$$ Here, $T_i$ is the number of scored answer tokens.
Reviewer_bNN1OpenReview ↗

Thanks

Thank you for the authors' feedback. My concerns have been addressed. Based on my evaluation, I believe a score of 3 would be fair.

Good Luck.

Thank you for your constructive comments. We are glad that we could address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their constructive feedback. Our main revisions are:

  • New and expanded experiments: We added counterfactual confidence intervals and analyses restricted to samples correctly answered under full input (Appendix E, Tables 6–9); multi-cell and ROI/non-ROI occlusions (Appendix F.3, Tables 10–11); correct-answer retrieval controls (Appendix F.4, Table 12); and stage-wise diagnostic probes with matched-slide decision analyses (Appendix F.7, Figure 14 and Table 14). We also expanded matched-slide evaluation (§7.1, Figure 6) and instruction-tuning experiments to three models, reporting variability across three runs (§7.2, Table 1; Appendix G.3).
  • Key clarifications: We clarified checkpoints and visual inputs (Appendix D.1), trainable components and initialization (Appendix G.3), and token-averaged visual gain (Eq. 6). We distinguish spatial localization, diagnostic recoverability, and native decision-making, and clarify the scope of the resulting conclusions. An Ethics Statement now specifies SARC approval, consent waiver, de-identification, and controlled access.
  • Corrections: We corrected wording, reference inconsistencies, training-component descriptions, and transcription errors in four Patho-R1 intervention results in the original Table 5 (revised Table 6). Full-input results were unaffected. The corrected results do not change our central empirical conclusions.
  • Presentation: We improved figure styling, labels, legends, and uncertainty displays, and reorganized tables to distinguish models, training corpora, and evaluation splits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.