- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper finds that most pathology VLMs respond the same way whether given a blank image, the whole slide, or just the lesion. It introduces a TCGA benchmark that uses patient- and site-disjoint splits, question audits, and extra evaluation steps to filter out questions that a text-only model could answer. The authors also present Pair-DPO, a method that compares two real slides from the same cancer type and site but with opposite findings for the same answer, and uses this approach to train a slide-level VLM.
Critical strengths
- the blank vs slide vs lesion test is simple and interesitng
- site and patient disjoint splits, blind scores next to every accuracy
- using real same-site slide pairs is neat
- good controls, step-matched ablations and SlideChat same on the same data
- lambda picked on validation
Critical weaknesses
- fixing the answer and using contrasting images within DPO is called V-DPO (Findings of EMNLP 2024, pp. 13258-73). What does your approach add beyond using real slides?
- MediConfusion (ICLR 2025) asks the same question on two images with opposite answers. You might consider reporting pair accuracy as they do.
- PathBench (TMI 2025;44(10):4087-97) already showed that pathology LMMs can answer MCQs without looking at the image. You list this in Figure 5 but do not discuss it.
- for site-disjoint splits, you should cite Howard et al. (Nat Commun 2021;12:4423).
- The Layer-C filter uses blind runs of SlideChat and Patho-R1, and then you evaluate those same models on it. SlideChat scores below chance. This introduces bias.
- the
15.29% gain from image evidencein the abstract refers to the EWA score, not an actual gain. The raw gain is much lower. - Table 2, the EWA confidence intervals for Pair-DPO and CE overlap, and only Sens separates them. You mention using three seeds, but where is the spread shown?
- Differences between the slides, such as tumour fraction, size, and stage, still correlate with the finding and separate the pair.
- for the pathologist review, how many reviewers were involved and what was their agreement rate? Only abou 80% of cases were kept?
- 25% option shortcut is chance performance for four options.
Xie Y, Li G, Xu X, Kan MY. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. InFindings of the Association for Computational Linguistics: EMNLP 2024 2024 Nov (pp. 13258-13273).
Sepehri MS, Fabian Z, Soltanolkotabi M, Soltanolkotabi M. Mediconfusion: Can you trust your ai radiologist? probing the reliability of multimodal medical foundation models. InInternational Conference on Learning Representations 2025 May 1 (Vol. 2025, pp. 72145-72169).
Sun Y, Wu H, Zhu C, Si Y, Chen Q, Zhang Y, Zhang K, Li J, Cai J, Wang Y, Sun L. Pathbench: Advancing the benchmark of large multimodal models for pathology image understanding at patch and whole slide level. IEEE transactions on medical imaging. 2025 Jul 2;44(10):4087-97.
Howard FM, Dolezal J, Kochanny S, Schulte J, Chen H, Heij L, Huo D, Nanda R, Olopade OI, Kather JN, Cipriani N. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nature communications. 2021 Jul 20;12(1):4423.
Questions for authors
see above
3 comments
W6. 15.29% gain in the abstract
Thank you for pointing this out.15.29 is the evidence-weighted accuracy (EWA) of PairSlide. The raw gain, Layer-B accuracy with the slide minus the blind score, is 9.59 points for PairSlide (67.35 vs. 57.76) and 3.71 points for SlideChat (44.72 vs. 41.01). We revise the abstract and Sec. 6.
W7. Seed spread for Table 2
The intervals in Table 2 are patient-level bootstrap intervals for one seed of each arm. The EWA of the three seeds is:
| EWA | CE control | Pair-DPO |
|---|---|---|
| Seed 0 | 12.55 | 15.29 |
| Seed 1 | 12.20 | 15.45 |
| Seed 2 | 13.31 | 15.22 |
Every Pair-DPO seed is above every CE seed, and the difference in means is 2.63. We have added this table to the revised paper.
W8. Residual differences between paired slides
Pair-DPO cancels what the two slides of a pair share, but the two slides can still differ in finer-grained properties that correlate with the finding, such as tumour fraction, size, and stage. So we ran a masking experiment to test whether the answer depends on the region that carries the finding. If it does, removing that region should change the answer, while removing the same number of other tiles should not.
Here, we use CAMELYON16, a public set of breast-cancer sentinel lymph-node slides in which pathologists outlined every metastasis. Every slide is asked one four-option question, “Does this sentinel lymph node section contain metastatic breast carcinoma?”, with “Yes” correct for metastatic slides and “No” for normal ones. We trained two models from our SFT checkpoint, one with CE and one with Pair-DPO, where a pair is one metastatic and one normal slide asked the same question. We then compare four interventions on the test slides: removing the expert-annotated lesion, removing the same number of random non-lesion tiles, removing the same number of non-lesion tiles that CONCH ranks as most tumour-like, and keeping only the annotated lesion.
| CAMELYON16, accuracy (%) | CE control | Pair-DPO |
|---|---|---|
| Full slide | 45.12 | 48.35 |
| Annotated lesion removed (expert-annotated) | 40.57 (−4.55) | 41.64 (−6.71) |
| Only the annotated lesion kept | 64.61 (+19.50) | 77.50 (+29.15) |
| Same number of random non-lesion tiles removed | 45.66 (+0.54) | 48.92 (+0.57) |
| Same number of most tumour-like non-lesion tiles removed (CONCH) | 45.54 (+0.42) | 48.60 (+0.25) |
Removing the annotated lesion lowers the accuracy of Pair-DPO by 6.71 points, whereas removing the same number of non-lesion tiles changes it by at most 0.57 points, even when these are the most tumour-like tiles. The effect is also larger for Pair-DPO than for the CE control, both when the lesion is removed (−6.71 vs. −4.55) and when only the lesion is kept (+29.15 vs. +19.50). On this task, the answer therefore responds to the target region and not to other tissue of equal area.
W9. Pathologist review
Two pathologists audited a stratified sample of 80 questions against their slides. Both rated every question independently, and they agree for 86.25% of questions. Table 5 shows where it comes from. The text-level criteria are at or near 100%: no stem answers away, the wording is sound, and the distractors are plausible for 96% of questions. The losses are on clinical validity, from two sources. About 10% of questions can be answered from the text alone, which is the prior that CleanSlide measures with the blind score. About 8% cannot be answered from the slide, and these concentrate in staging, molecular subtype, and treatment, whose answers come from the clinical record and not from the H&E slide. We also correct and specify the scope of the review in the revised paper.
W10. Option-shortcut rate
The reviewer is right that 25% is the chance for four options, and this is what the value means. Under our definition, 25% is the lowest possible option-shortcut rate, reached when the options carry no cue to the answer. Under the same definition, WSI-Bench scores 49% and SlideBench 35%. We give a clear definition in Sec. 3.1 in the revised paper.
We hope these responses address your concerns and would be grateful if you could reconsider your score. We are happy to provide further clarification.
Thanks for the rebuttal
Thanks for the rebuttal and the new experiments. Many points were resolved. I keep my score for the following reasons:
- Layer-B is screened with the same blind models that are then evaluated on it. Re Table 1, three of them have larger raw gains than PairSlide
- The pathologist review covers 80 imported questions. The generated tier, which supplies the Pair-DPO labels, was not reviewed.
- The effect of the loss is small.
- The CAMELYON masking test is on a yes/no finding, an the accuracy is below of the majority-class baseline. It does not address the tumour-fraction parts in the TCGA pairs for diffuse findings.
Thank you for your detailed comments. Before the point-by-point responses, we would like to briefly clarify why pathology differs from natural images. Methods that reduce reliance on language priors (e.g., mDPO, V-DPO) assume that (1) the image can be edited to create a counterfactual, (2) the prior is a nuisance to remove, and (3) the question concerns the presence of an object. None of these holds in pathology: a finding such as necrosis or a higher grade cannot be faithfully added or flipped, clinical priors reflect real disease epidemiology, and questions concern graded findings such as invasion, necrosis, and grade. Our paper therefore measures the prior with a blind score, and trains on real, report-verified counterfactual slide pairs.
W1. Relation to V-DPO
V-DPO (Xie et al., 2024) and mDPO (Wang et al., 2024a) both keep the response fixed and contrast two images, and Sec. 4.2 states that our loss does so “as in mDPO.” We now cite V-DPO next to it. The difference is where the contrasting image comes from. V-DPO builds its image-contrast pairs by editing images with Stable Diffusion, and mDPO by randomly cropping the image. However, this is not available for reporting findings in pathology because they cannot be faithfully inserted or removed by editing a slide. Pair-DPO therefore uses a real slide from another patient with the same cancer type and tissue source site and an opposite, report-verified finding. In Table 2, the synthetic variant (mDPO, which removes pathology-related tiles) reaches EWA 13.94 against 15.29 for real pairs.
W2. MediConfusion and pair accuracy
We now cite MediConfusion in Sec. 2 and report its metrics on our counterfactual pairs. We randomly sampled 306 pairs from the CleanSlide paired test. The two slides of a pair come from different patients with the same cancer type and tissue source site, and have opposite answers for one report-verified finding. Following MediConfusion, set accuracy is the share of pairs with both slides answered correctly, and individual accuracy is the share of single slides answered correctly. Each answer is the option letter with the highest next-token probability.
The published SlideChat answers both slides of a pair correctly in only 2.94% of pairs. This pattern is consistent with MediConfusion’s observation that models tend to give the same answer to both images of a pair. Both models trained on CleanSlide perform better, reaching a set accuracy of 20.59% with the CE control and 22.55% with Pair-DPO. Among the three models, Pair-DPO achieves the highest set and individual accuracy.
W3 & W4. PathBench and Howard et al.
Thank you for pointing out these two missing references. We now cite and discuss both in Sec. 2, and both findings motivate the design of CleanSlide. PathBench showed that pathology LMMs can answer multiple-choice questions without the image. CleanSlide builds on this observation: it reports a blind score next to every accuracy, so that the prior and the image contribution can be separated for each model, and it adds a paired counterfactual test. Howard et al. (2021) showed that site-specific signatures in TCGA slides bias model accuracy. This is the reason CleanSlide splits by tissue source site as well as by patient.
W5. Layer-C
We rebuilt Layer-C from the three members that Table 1 does not evaluate: the text-only LLMs Qwen3-32B, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-7B. The revised subset keeps 10,465 questions. Qwen2.5-7B-Instruct is the language model of PairSlide, so this screen, if anything, works against our model.
On the revised Layer-C, PairSlide still has the highest accuracy. The revised screen contains no pathology model, so the revised Layer-C removes only the questions that a text-only LLM can answer. In the revision, Table 1 reports the revised Layer-C for every model, and we give the new definition of Layer-C accordingly.