- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Perturb-Path, a benchmark for pathology foundation models based on in vivo genetic perturbations. The dataset includes 84 genetic perturbations across 32 mice, with 19,018 tumor regions and 116,383 H&E tiles. The authors evaluate 11 pathology foundation models and 3 ImageNet baselines. The results show that pathology foundation models generally outperform ImageNet baselines, although their ability to capture biological signals varies across tasks.
Critical strengths
The dataset has a useful biological design, linking H&E images with genetic perturbations and tumor microenvironment information.
The benchmark covers different tasks, including perturbation retrieval, immune cell fraction prediction, and robustness.
The paper reports not only model strengths but also limitations in predicting specific biological effects.
Critical weaknesses
- The 84 genetic perturbations are distributed across two libraries, each associated with a separate animal cohort. The authors also report that mouse-specific signals remain after F8 normalization and PCA. It is therefore unclear to what extent the learned perturbation-related signals can be distinguished from experimental differences.
- The authors normalize embeddings using F8 control tumors from the same mouse. While this helps reduce batch effects, it also makes the evaluation dependent on a specific internal control. It remains unclear whether the evaluation would still be applicable when such controls are unavailable.
- In the T-cell fraction comparison between B2m and Socs1 perturbations, only 9 of the 15 models predicted the correct direction, and none predicted a significant difference. This suggests that good retrieval or prediction performance does not necessarily translate into reliable identification of specific biological effects. The unique scientific value of the benchmark therefore needs to be established more clearly.
Questions for authors
The robustness analysis shows that mouse-specific signals remain after F8-normalization and PCA-50. Given that the 84 perturbations are distributed across two libraries, how do the authors disentangle perturbation-associated biological signals from batch effects in the cross-mouse retrieval task?
Response to Reviewer ETkg
We thank the reviewer for their feedback and comments.
Weaknesses
The reviewer raises an important question here. Each perturbation (excluding positive controls) is associated with one of two CRISPR libraries, and each library is pooled independently. Yet when we compute the mAP metric we compare each perturbation against all other perturbations, including those in the other library. How then can we be confident that above-chance mAP results are not driven entirely by library batch effects where tumor embeddings from the same library cluster more closely than embeddings from different libraries?
To answer this, we recalculated the mAP metric for each model but restricted the comparisons to perturbations in the same library. The results are in Figure 13 in Appendix Section A.6: the difference between unrestricted
mAPand within-librarymAPis negligible. mAP / null is still greater than 1 for all models, and for any given model the gap between unrestricted and within-library is much smaller than the overall differences in mAP across models. In fact, the ranking of models by mAP remains nearly unchanged (Spearman = 0.996) between unrestricted and within-librarymAP.The reviewer accurately identifies one of the strengths of our dataset: engineering an internal control per animal allows us to normalize embeddings to reduce batch effects. This fixes a limitation found in most datasets, and we agree with the reviewer that this has restricted the use of other datasets as efficient benchmark datasets for PFMs. As part of this publication, we are releasing the Perturb-Path dataset so that others can either develop their own sets of benchmarks or evaluate their own models against the tasks developed here. We have now emphasized in the abstract that this dataset will be made publicly available upon acceptance.
The review raises an important concern: unlike for the other tasks, none of the PFMs achieve statistically significant performance for
tme-effecttask, which brings into question the scientific validity of the task. However, we believe the underwhelming performance from this suite of PFMs does not invalidate thetme-effecttask; rather, it is a justification for its inclusion in the Perturb-Path dataset.By design, we evaluated the models against a series of increasingly complex tasks. The purpose of the
tme-effecttask is to be challenging so it will be a hill to climb for future generations of PFMs. (We intentionally removed B2m and Socs1 tumors from the training data to make the task harder.) As seen at the top of Figure 7(a), the measured mIF aligns with the expected effect described in the literature, and we know from the results onmiFtask that PFMs are capable of predicting T cell infiltration from H&E. Moreover, the models that perform best at this task—UNI2-h, Gigapath, and Gigapath-Flash—also perform well on the other tasks. Collectively these results suggest thetme-effecttask is difficult for the current generation of PFMs but potentially solvable by the next generation. Such tasks can be an especially valuable contribution to the field given the plateauing on PFM benchmarks in recent years, as demonstrated by Veldhuizen et al. (2025).Questions:
This question is answered in our response to the first weakness above.