- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses a meaningful problem by introducing a concept-based benchmark for interpreting pathology foundation models. The evaluation across 16 encoders is fairly comprehensive, but the reliability of the benchmark annotations and the limited novelty of the findings weaken the overall contribution.
Critical strengths
The paper addresses a meaningful problem, as conventional performance benchmarks provide limited insight into what pathology foundation models actually learn. The evaluation across 16 encoders is relatively comprehensive, and the distinction between factor accessibility and downstream sensitivity offers an interesting perspective on model interpretability.
Critical weaknesses
The reliability of the proposed benchmark is a major concern. The morphology labels are derived from figure captions without independent image-level validation by pathologists. The reported 99.1% agreement only reflects consistency between LLM-based caption annotations rather than the visual validity of the concepts.
The methodological novelty and significance of the findings are also somewhat limited. The framework mainly relies on existing linear probing and concept erasure techniques, while the conclusions are largely descriptive. Furthermore, the downstream sensitivity analysis is restricted to a single melanoma cohort, and the observed performance changes after concept removal are generally small.
Questions for authors
Could the authors provide image-level validation by pathologists to establish the reliability of the morphology annotations? It would also be helpful to validate the downstream sensitivity analysis on additional cancer types to demonstrate whether the findings generalize beyond melanoma.
6 comments
Response 2/2
Methodological novelty and significance
We see the contribution as the construction and reuse of a shared, externally defined morphology space rather than a new probing or erasure operator: the same 23 factors support cross-encoder comparison, post-hoc diagnosis of trained predictors, and a predictor built from the factors themselves.
On the comparison of encoders, we find that models with similar aggregate scores differ at the factor level. UNI2 and Virchow2 have polarity macro AUCs of 0.702 and 0.694, but UNI2 is stronger on invasive component (0.785 versus 0.651) and Virchow2 on mitotic activity (0.820 versus 0.761). Encoder–factor interactions explain 31.7% of explicit-polarity AUC variation (Section 4.3, Tables 8–9). Among the five added vision–language models, CONCH v1.5 has the highest broad and polarity accessibility of all 21 encoders (0.860 and 0.776; Section 4.1, Table 2).
On the concern that the conclusions are largely descriptive, we added a 23-factor concept bottleneck (FCB). It maps patch embeddings to named morphology scores, pools them with gated attention, and predicts only from that factor representation, so each logit is a bias plus one contribution per morphology score (Section 5.3, Figure 6). In the UNI2 COAD lymphatic-invasion model, for example, mitotic activity has the largest mean absolute contribution (0.356 logit units), and the profile changes for nodal-status prediction. A pathologist can inspect the tissue behind a contribution and question the interpretation. Predicting from only 23 named scores, the FCB reaches mean AUCs of 0.767, 0.845 and 0.881 across the 21 encoders on the three melanoma tasks (Table 11).
Additional cancer types and the size of the effects
Following your suggestion, we expanded the downstream study from melanoma to TCGA-COAD (405 slides) and TCGA-READ (141 slides), evaluated separately and pooled (546 slides). Lymphatic-invasion and nodal-status prediction add six colorectal settings to the three VisioMel tasks, giving nine settings across 21 encoders and three training seeds (Section 5.1; Appendix C.2).
With 21 encoders and nine settings, the effects are larger than in the original table. Targeted erasure reduces AUC by more than one percentage point in 291 encoder–factor–setting combinations, and the largest reductions are 4–8 points. Five random-direction erasures per encoder and setting change AUC by at most 0.624 points (Appendix C.3).
The effects also differ between predictors with nearly the same accuracy. Phikon and Phikon-v2 reach melanoma relapse AUCs of 0.791 and 0.794, but removing the mitotic-activity direction reduces them by 0.56 and 4.26 points, respectively (Section 5.1, Figure 7; Table 13). Mitotic rate is an established prognostic factor in melanoma (Thompson et al., Prognostic significance of mitotic rate in localized primary cutaneous melanoma, J. Clin. Oncol. 29(16):2199–2205, 2011), and mitotic activity has the largest encoder-averaged erasure loss of the eight factors on relapse prediction, 1.03 points compared with 0.10–0.70 for the others (Table 13).
Accessibility and sensitivity also remain largely decoupled in the new settings, as in the original submission, with a median correlation of +0.096 across all nine (Appendix C.8, Table 12).
Thank you again for the constructive feedback.
Thank you to the authors for the response. These newly added contents indeed largely address my concern, but regarding the pathologist validation, I understand that it is unrealistic to have pathologists verify such a large volume of data. However, I suggest that a small subset could be sampled for pathologists to conduct an evaluation, to see whether the overall setting is reasonable and what the overall accuracy of the captions is. However, this may not necessarily need to be completed in this rebuttal.
Additionally, this benchmark has a certain inherent bias: vision-language models will inevitably outperform vision-only models. Therefore, there is a certain gap between the conclusions drawn from this concept-based benchmark and the application to specific downstream tasks. So what does the author think is the practical significance of this benchmark?
Response 1/2
Thank you for the careful review, and for recognizing the value of comparing pathology foundation models through interpretable morphology and of the distinction between factor accessibility and downstream sensitivity. The revised paper adds a human audit of benchmark construction, evaluates 21 encoders, including five vision–language models, across nine melanoma and colorectal settings, and introduces a factor concept bottleneck (FCB) predictor.
Image-level validation and benchmark reliability
We agree that expert image-level judgement would greatly enhance the value of the dataset. The 99.1% agreement was reported as a check on label extraction, and the revision complements it with a human audit of every construction step. Pathologist annotation at this scale was not available to us in this study. What we propose is a scalable way to build a partially verified dataset, so that later expert effort can be more targeted.
We divide the verification work into three parts. The first is machine-verifiable: automated extraction and consistency checks identify panels, assign caption spans, and label asserted presence or absence. The second is human-verifiable: a reader without pathology training checks that these steps reflect the source publication. The third is expert-verifiable: whether the named morphology is visually correct requires a pathologist. The first two parts rely on public reporting standards and published figures and captions, and we have completed both.
We conducted a 4,000-judgment human audit of the human-verifiable steps, which found 98.4% agreement for blinded histopathology-panel identification, 95.2% for cropping, 95.0% for caption decomposition, 97.0% for panel–caption binding, and 97.2% for blinded polarity labels (Section 3.3; Appendix A.2–A.3, Tables 5–6). Every example retains its source panel and caption, so a pathologist can begin from a focused, traceable set instead of unfiltered images. Figures 2–3 show the construction process with asserted and denied examples. The labels also originate from experts: the factors are defined by ICCR and RCPath reporting standards, and each caption is the publishing authors' own description of the panel.