acceptodds
Under review as a conference paper at ICLR 2027

Perturb-Path: Benchmarking H&E Pathology Foundation Models with in vivo Genetic Perturbations in a Lung Cancer Model

Abstract

Pathology foundation models (PFMs) trained on datasets of histopathology images can learn robust patient representations that transfer to downstream disease modeling and drug response prediction tasks. While PFMs are trained on large amounts of unlabeled histopathology data, the scarcity of meaningful clinical labels forces these models to be evaluated on much smaller cohorts. Moreover, the labels in these evaluation cohorts are often noisy and corrupted by hard-to-resolve confounded factors. To mitigate these difficulties, we sought to create a PFM evaluation dataset with rich biological diversity, carefully-measured labels, and biological replicates. We leveraged Perturb-map to generate hundreds of genetically diverse tumors in a mouse model of lung cancer with paired hematoxylin & eosin (H&E) and multiplex immunofluorescence (mIF) imaging readouts. The resulting Perturb-Path dataset includes 84 knockout perturbations across 32 mice, resulting in 19,018 identified tumors with 116,383 H&E patches. We release a new PFM benchmark suite around this dataset and assessed eleven publicly-available PFMs and three ImageNet-based baselines. The benchmark suite evaluates the models on their ability to represent a range of phenotypic factors, including retrieval of genotypic perturbation labels and prediction of tumor microenvironment composition derived from the paired mIF readouts. This is, to our knowledge, the first publicly-available pathology benchmark dataset involving gene perturbations. We find that PFMs varied in their ability to retrieve specific perturbations, suggesting the presence of a genetically-associated phenotype that is captured by some models but not others. Further, we identify differences in ability to predict macrophage fraction versus T cell fraction across models, highlighting specific areas of improvement for model development. Overall, we found the PFMs outperform the ImageNet baselines, indicating that training on histopathology data of human origin translates to improvement on tasks in a mouse model.

Primary area:
datasets and benchmarks
Keywords:
pathology foundation models, benchmarks, evals, datasets, cancer, multimodal, biology

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ETkgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Perturb-Path, a benchmark for pathology foundation models based on in vivo genetic perturbations. The dataset includes 84 genetic perturbations across 32 mice, with 19,018 tumor regions and 116,383 H&E tiles. The authors evaluate 11 pathology foundation models and 3 ImageNet baselines. The results show that pathology foundation models generally outperform ImageNet baselines, although their ability to capture biological signals varies across tasks.

Critical strengths

The dataset has a useful biological design, linking H&E images with genetic perturbations and tumor microenvironment information.

The benchmark covers different tasks, including perturbation retrieval, immune cell fraction prediction, and robustness.

The paper reports not only model strengths but also limitations in predicting specific biological effects.

Critical weaknesses
  1. The 84 genetic perturbations are distributed across two libraries, each associated with a separate animal cohort. The authors also report that mouse-specific signals remain after F8 normalization and PCA. It is therefore unclear to what extent the learned perturbation-related signals can be distinguished from experimental differences.
  2. The authors normalize embeddings using F8 control tumors from the same mouse. While this helps reduce batch effects, it also makes the evaluation dependent on a specific internal control. It remains unclear whether the evaluation would still be applicable when such controls are unavailable.
  3. In the T-cell fraction comparison between B2m and Socs1 perturbations, only 9 of the 15 models predicted the correct direction, and none predicted a significant difference. This suggests that good retrieval or prediction performance does not necessarily translate into reliable identification of specific biological effects. The unique scientific value of the benchmark therefore needs to be established more clearly.
Questions for authors

The robustness analysis shows that mouse-specific signals remain after F8-normalization and PCA-50. Given that the 84 perturbations are distributed across two libraries, how do the authors disentangle perturbation-associated biological signals from batch effects in the cross-mouse retrieval task?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ETkg

We thank the reviewer for their feedback and comments.

Weaknesses

  1. The 84 genetic perturbations are distributed across two libraries, each associated with a separate animal cohort. The authors also report that mouse-specific signals remain after F8 normalization and PCA. It is therefore unclear to what extent the learned perturbation-related signals can be distinguished from experimental differences.

The reviewer raises an important question here. Each perturbation (excluding positive controls) is associated with one of two CRISPR libraries, and each library is pooled independently. Yet when we compute the mAP metric we compare each perturbation against all other perturbations, including those in the other library. How then can we be confident that above-chance mAP results are not driven entirely by library batch effects where tumor embeddings from the same library cluster more closely than embeddings from different libraries?

To answer this, we recalculated the mAP metric for each model but restricted the comparisons to perturbations in the same library. The results are in Figure 13 in Appendix Section A.6: the difference between unrestricted mAP and within-library mAP is negligible. mAP / null is still greater than 1 for all models, and for any given model the gap between unrestricted and within-library is much smaller than the overall differences in mAP across models. In fact, the ranking of models by mAP remains nearly unchanged (Spearman = 0.996) between unrestricted and within-library mAP.

  1. The authors normalize embeddings using F8 control tumors from the same mouse. While this helps reduce batch effects, it also makes the evaluation dependent on a specific internal control. It remains unclear whether the evaluation would still be applicable when such controls are unavailable.

The reviewer accurately identifies one of the strengths of our dataset: engineering an internal control per animal allows us to normalize embeddings to reduce batch effects. This fixes a limitation found in most datasets, and we agree with the reviewer that this has restricted the use of other datasets as efficient benchmark datasets for PFMs. As part of this publication, we are releasing the Perturb-Path dataset so that others can either develop their own sets of benchmarks or evaluate their own models against the tasks developed here. We have now emphasized in the abstract that this dataset will be made publicly available upon acceptance.

  1. In the T-cell fraction comparison between B2m and Socs1 perturbations, only 9 of the 15 models predicted the correct direction, and none predicted a significant difference. This suggests that good retrieval or prediction performance does not necessarily translate into reliable identification of specific biological effects. The unique scientific value of the benchmark therefore needs to be established more clearly.

The review raises an important concern: unlike for the other tasks, none of the PFMs achieve statistically significant performance for tme-effect task, which brings into question the scientific validity of the task. However, we believe the underwhelming performance from this suite of PFMs does not invalidate the tme-effect task; rather, it is a justification for its inclusion in the Perturb-Path dataset.

By design, we evaluated the models against a series of increasingly complex tasks. The purpose of the tme-effect task is to be challenging so it will be a hill to climb for future generations of PFMs. (We intentionally removed B2m and Socs1 tumors from the training data to make the task harder.) As seen at the top of Figure 7(a), the measured mIF aligns with the expected effect described in the literature, and we know from the results on miF task that PFMs are capable of predicting T cell infiltration from H&E. Moreover, the models that perform best at this task—UNI2-h, Gigapath, and Gigapath-Flash—also perform well on the other tasks. Collectively these results suggest the tme-effect task is difficult for the current generation of PFMs but potentially solvable by the next generation. Such tasks can be an especially valuable contribution to the field given the plateauing on PFM benchmarks in recent years, as demonstrated by Veldhuizen et al. (2025).

Questions:

  1. The robustness analysis shows that mouse-specific signals remain after F8-normalization and PCA-50. Given that the 84 perturbations are distributed across two libraries, how do the authors disentangle perturbation-associated biological signals from batch effects in the cross-mouse retrieval task?

This question is answered in our response to the first weakness above.

Reviewer_CmGSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new benchmark for pathology foundation models (PFMs), which is based on H&E image patches of 84 knockout perturbations across 32 mice from an in vivo mouse model of lung cancer. From this dataset, multiple tasks are constructed, including the measurement of robustness, retrieval accuracy, predicting immune cell fractions obtained from mIF measurements, and TME relationships. The robustness is measured by an adapted robustness index (PathoROB) and quantifies the relative dominance of the perturbation label vs. the mouse donor ID as the batch variable in the embedding space. Generally, a low robustness index is observed, highlighting strong organization by the mouse ID in the embedding space. Based on these findings, the embeddings are normalized by the statistics of the embeddings from patches obtained by knocking out the F8 gene, which counteracts the observed mouse ID bias. Retrieval performance of embeddings with the same knockout perturbation is measured as a second task, and predicting the immune cell fractions in the tumor regions via a ridge regression is measured as a third task. As the fourth task, the models need to predict a known TME relationship between the difference of Socs1 and B2m perturbations and T cell fractions. In total, the authors evaluated 11 PFMs on these four tasks and three ImageNet baselines. The results show that the PFMs clearly outperform the ImageNet baselines, and UNI2-h performed consistently near or at the top of the ranking. Correlating the model ranking with the ranking on other PFM benchmarks yields a quite similar global ranking with a few clear outliers.

Critical strengths
  • The paper is written with a lot of clarity. The motivation, experimental setup, and results are easy to follow.

  • The paper contributes a well-curated novel dataset, which includes expensive biological measurements (mIF) and a wide range of evaluation tasks. Pathology foundation model development needs further challenging datasets. This paper makes a valuable contribution for the community in this regard and gives further insights about the predictability of knockout perturbations and mIF derived targets from H&E embeddings.

  • The data and prediction tasks are approached from multiple angles (robustness, retrieval, regression prediction, tme-task), giving a comprehensive view on probing the embedding space for the perturbation information.

Critical weaknesses
  • To get more trust whether the perturbation information can even be properly predicted from the H&E representations, or we are primarily measuring batch effects, it would be valuable to further link relative accuracy per perturbation to biological background information of the individual perturbations, and then discuss if these findings make biological sense.

  • As robustness, is a key evaluation factor in section 4.1, It would be interesting to add recent PFMs that significantly improve on the cited PathoROB robustness benchmark such as RudolfV 2 [1] and Mascaret [2]. Further, also any image-text models (e.g. CONCH [3] or MUSK [4]) would be a valuable addition to the analysis to assess if such a different objective changes the conclusions. The discussion explicitly states that the small differences between models might stem from the fact that they have been trained with the same objective (DINOv2).

  • The authors only test pretrained PFMs trained on human data. Any analysis or discussion whether the models transfer well to mouse data is missing. It would be interesting to measure whether the patch representations are OOD in embedding space compared to human data or even have a small self-supervised finetuning phase with mouse data, and see if that makes a significant difference in the task performance.

[1] https://huggingface.co/collections/Aignostics/rudolfv-2

[2] https://huggingface.co/wearewaiv/mascaret

[3] https://huggingface.co/MahmoodLab/conchv1_5

[4] https://huggingface.co/xiangjx/musk

Questions for authors
  • Will the benchmark be released publicly?

  • Did you analyze whether the results from this benchmark correlate with the results from HEST? As HEST is about predicting gene expressions, I could imagine that this better correlates with the results from this benchmark than PathBench.

3 comments
AuthorsOpenReview ↗

Response to Reviewer CmGS (Part 1/2)

Thank you for taking the time to review our work; we appreciate your feedback. We have addressed several concerns in our response to all reviewers; we address your specific weaknesses and questions below.

Weaknesses

  1. To get more trust whether the perturbation information can even be properly predicted from the H&E representations, or we are primarily measuring batch effects, it would be valuable to further link relative accuracy per perturbation to biological background information of the individual perturbations, and then discuss if these findings make biological sense.

We acknowledge the reviewer’s concern whether we are measuring the ability of the PFMs to predict biology from the H&E representations, or merely predicting batch effects. We would like to highlight that the ability to measure perturbations despite batch effects is a unique strength of this dataset compared to the other histopathology benchmarks. This is because A. The protein barcodes are a directly measurable phenotype in the midst of complex biology. B. The experimental design enables pools and replicates, and C. We also account for the batch effects in our benchmark tasks. We go into detail below:

  • A: Perturbations can affect the tumor H&E in many ways, including changes in cellular morphology, colocalization, density, and infiltration. Perturb-map works around this complexity, by using an engineered protein barcode to determine the perturbation in the tumor. Thus, our benchmark dataset can capture the presence of a perturbation without attribution to a specific explainable H&E outcome.

  • B: The controlled-experiment nature of this dataset is unlike what is found in most human H&E studies. The main batch effects are known a priori, consisting of inter-lobular, inter-library, and inter-mouse effects. Replicates of perturbations are numerous, and fall within and across batches. Finally, the pooled-measurements enable controlled comparisons.

  • C: The benchmark tasks themselves can account for the batch effects. By measuring the perturbational signal in the tasks relative to the nuisance variables, we can determine whether the signal from the observed perturbation is significant. Please note, we have added additional controls for batch effects as described in the global response.

The embeddings from the PFMs are inherently uninterpretable, which makes it complicated to evaluate which biological features from the H&E are contributing to successful perturbation retrieval. However, we highlight two examples of perturbations that have a high retrieval score across all PFMs, Socs1 KO and Ctnnd1 KO. In our mIF dataset, we were able to link these perturbations to changes in the immune fractions in tumors (CD8+ T cells and macrophages, respectively), which can be identified from H&E (see Figure 17). Both findings are validated in independent datasets found in the literature. Loss of Ctnnd1 has been linked in a mouse breast cancer model with increased macrophage infiltration and impact on the epithelial/mesenchymal state of tumors (Kurley et al, 2021). Socs1 KO were included in our dataset as a control, after samples generated independently with Perturb-map described an increased CD8 T cell infiltration in lung tumors carrying a Socs1 KO (Dhainaut, Rose et al, 2021).

We strive to work towards identifying an interpretable layer to the H&E outcomes of the perturbations in this benchmark dataset. We see the following adjustments as a path forward to address the reviewer comments in the future:

  • In the current version of Perturb-Path, the dataset contains 84 KOs that cover broad gene families across diverse phenotypes. This is by design as we didn’t want to bias the benchmark towards the identification of a specific phenotype. In the future, larger datasets could be used to enrich for gene families that have known convergent functions enabling testing of effects across specific gene families.
  • The Perturb-Path tasks are not limited to the evaluation of PFMs, but could also be used for foundation models trained on multimodal datasets, which can generate embeddings that have a rich biological layer for interpretability (gene expression, etc). We expect these models to be able to better decode the biology underlying the different perturbations.

Response continued in next comment.

AuthorsOpenReview ↗

Response to Reviewer CmGS (Part 2/2)

  1. As robustness is a key evaluation factor in section 4.1, It would be interesting to add recent PFMs that significantly improve on the cited PathoROB robustness benchmark such as RudolfV 2 [1] and Mascaret [2]. Further, also any image-text models (e.g. CONCH [3] or MUSK [4]) would be a valuable addition to the analysis to assess if such a different objective changes the conclusions. The discussion explicitly states that the small differences between models might stem from the fact that they have been trained with the same objective (DINOv2).

We agree with the reviewer about enlarging the pool of PFMs in Perturb-Path. We added three vision-language PFMs to the benchmark: MUSK, CONCH v1, and CONCH v1.5. We find that the Perturb-Path results for these PFMs agree with the results from the THUNDER and PathBench leaderboards: all three fall in the middle of the rankings, and CONCH v1 outperforms CONCH v1.5. Unfortunately, RudolfV 2 and Mascaret cannot be included in our submission. Both models have been released under custom licensing that is more restrictive than typical Apache 2.0, MIT, or CC-BY-NC-ND 4.0 licenses, which conflicts with our goal to only include models in our benchmark that have permissive licensing.

  1. The authors only test pretrained PFMs trained on human data. Any analysis or discussion whether the models transfer well to mouse data is missing. It would be interesting to measure whether the patch representations are OOD in embedding space compared to human data or even have a small self-supervised fine-tuning phase with mouse data, and see if that makes a significant difference in the task performance.

Some preliminary analyses showed that these mouse embeddings tend to be out of distribution compared to the images in the human data in TCGA. However, this is hard to disambiguate from differences in staining, scanner, or image acquisition. This, however, highlights a central point of the paper: despite the differences between mouse and human, the PFMs are still able to capture signal pertaining to both knockout identifiability and TME. We have future studies planned to more closely align the mouse data with the human models together, but we’re considering it out of scope for this study.

Questions:

Will the benchmark be released publicly? Yes, the benchmark dataset and source code will be released publicly. Please see our statement at the beginning of our global response for all reviewers.

Did you analyze whether the results from this benchmark correlate with the results from HEST? As HEST is about predicting gene expressions, I could imagine that this better correlates with the results from this benchmark than PathBench.

We added a comparison between Perturb-Path and HEST to the manuscript. As the reviewer expected, there is a positive correlation between Perturb-Path and HEST performance, although the correlation is lower than for the other two benchmark comparisons: the Spearman correlations with the rankings for THUNDER, PathBench, and HEST are 0.40, 0.55, and 0.21, respectively. However, Perturb-Path still replicates several key aspects of the HEST leaderboard: UNI2-h and H-optimus-0 perform very close to the top, and ResNet-50 is below all the PFMs.

Thank you for addressing the raised points. I think the analysis in Figure 17 linking knockouts to morphological patterns which can be seen in H&E is an important sanity check which helped to improve the manuscript. I encourage the authors to add more discussion around this also in the final version of the manuscript in the main text. It is great that the benchmark will be released publicly and I see that also as the main contribution of the paper. Thank you for adding the results for the image/text models. I would still be really inclined to see some results with the more robust models but this can also be done in the public benchmark. Overall, for me the paper is now definitely above the acceptance threshold and I will raise my score.

Reviewer_vDswOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a benchmarking dataset for H&E pathology foundation models (FMs), and benchmarks 11 pathology FMs and three ImageNet baseline models using the dataset. The authors have created the dataset using lung cancer models grown in mice to enable controlling gene knockouts (KO) common in non-small cell lung cancer (84 different KOs). The created dataset, Perturb-Path, is generated in 32 mice and includes 116,383 tiles (224x224 px, 0.5 um pixel size). The paper is clearly written.

The paper evaluates FMs using four tasks: 1) robustness, 2) mAP, 3) mIF, and 4) tme-effect. 1) The robustness measures whether the same perturbation in different experimental batches is more similar than different perturbations in the same experimental batch. Thus, it measures if batch effect or biological perturbation is giving stronger signal in representations extracted with evaluated FMs. The results show that all models perform worse than random sampling. 2) The mAP measures retrieval task similarity for perturbations. All FMs were able to enrich retrieval of similar perturbations whereas some KOs showed no recognizable changes in extracted representations. 3) The mIF task evaluates prediction of three immune cell types (CD8+ T cells, CD4+ T cells, and macrophages) from FM extracted features. The ground-truth for the task was prepared using separate multiplexed IF staining of the cells. The evaluation was done using 5-fold CV. 4) Finally, the tme-effect task evaluates T cell fraction predicted by the FMs between two different KOs that are known to have opposite effects in T cell fraction in the tumor microenvironment. The FMs were not able to find a similar difference between KOs as observed in mIF data.

Critical strengths
  • The Perturb-Path dataset includes a large number of genetic perturbations where KOs are controlled in the in vivo experiment stage. This control is not possible in typically used human H&E datasets that use diagnostic slides.
  • The dataset is of reasonable size considering the required wet-lab mouse experiments.
  • 11 pathology FMs and 3 ImageNet trained baseline models are benchmarked which is an impressive number of models benchmarked.
Critical weaknesses
  • The most significant contribution of this work is the dataset. However, the paper focuses mostly on benchmarking existing FMs using the dataset instead of thoroughly studying also the contents of the dataset.
  • The authors use mean aggregation of features extracted from tiles. In the mAP task, the authors write "We hypothesized that aggregation by mean-pooling across a lobe or mouse would further enhance the signal by reducing the inter-tumor variability", however, this is not evaluated. It is possible that mean aggregation also hides signals that are present only in rare tiles.
Questions for authors

Please address the listed weak points.

Related to the first weak point. Could the similarity inside perturbation be evaluated? The robustness task compares the same perturbation in different batches to different perturbations in the same batch. How similar are the same perturbations in the same batch vs. different batches? Thus, evaluation and QC related to the dataset would be an important addition to understand whether the dataset is of good quality for benchmarking tasks.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vDsw

We thank the reviewer for their feedback on our work. We appreciate that the reviewer recognized that these types of controls are not possible in typical human H&E datasets, and that the reviewer found the number of models benchmarked to be impressive. We have addressed several concerns in our response to all reviewers; we address your specific weaknesses and questions below.

Weaknesses

  1. The most significant contribution of this work is the dataset. However, the paper focuses mostly on benchmarking existing FMs using the dataset instead of thoroughly studying also the contents of the dataset. We appreciate the reviewer’s recognition of the contribution of our unique dataset. On the reviewer’s suggestion, we are now providing additional details on the content of the dataset in the Appendix, Figure 9. Figure 9 includes the distribution of tumor regions across libraries, the distribution of H&E tiles per tumors, and split half correlation of T cell and macrophage abundance, further reinforcing the robustness of our dataset.

We further validated the dataset with examples of known biology underlying the perturbations. Perturbations may affect the underlying tumor through complex biology including tumor morphology and immune-cell infiltration. The strength of our dataset is that the perturbations are identified by a protein barcode, which can identify a perturbation among a diverse range of biological phenotypes and without the need of extensive orthogonal validation of the phenotypes. Nevertheless, we added two selected examples showing that the PFMs can identify different aspects of biology that are cross verified in the literature (Appendix, Figure 17). PFMs were able to retrieve the Socs1 perturbation and its association with increased T cell infiltration (Dhainaut, Rose et al, 2021). They also identified the Ctnnd1 KO tumors and their association with increased macrophage infiltration, a relationship previously identified in a mouse model for invasive breast cancer (Kurley et al, 2021).

  1. The authors use mean aggregation of features extracted from tiles. In the mAP task, the authors write "We hypothesized that aggregation by mean-pooling across a lobe or mouse would further enhance the signal by reducing the inter-tumor variability", however, this is not evaluated. It is possible that mean aggregation also hides signals that are present only in rare tiles.

We did actually test the aggregation at the lobe-level to improve signal, with results in Figure 5b. We apologize if our manuscript was unclear; we have now updated the writing in the manuscript to make this more clear. “We hypothesized…” has been changed to “We next evaluated whether…”

Questions:

  1. Related to the first weak point. Could the similarity inside perturbation be evaluated? The robustness task compares the same perturbation in different batches to different perturbations in the same batch. How similar are the same perturbations in the same batch vs. different batches? Thus, evaluation and QC related to the dataset would be an important addition to understand whether the dataset is of good quality for benchmarking tasks.

The reviewer asks an important question about similarity within perturbations, which captures aspects orthogonal to the robustness index. We have performed this analysis by measuring the cosine similarity between pairs of tumors (after F8 normalization within each mouse), shown in Figure 12 in the Appendix. We define different sets of pairs depending on whether the tumors occur in 1) the same mouse and the same lobe 2) the same mouse and a different lobe and 3) different mice. As expected due to the residual batch effects, the cosine similarity decreases from 1) to 2) to 3). Encouragingly, the mean similarity between pairs of tumors arising from the same perturbation is higher than the mean similarity between pairs of tumors arising from different perturbations across all three sets.

We would like to emphasize that the unique structure within our dataset was crucial to enable this analysis. We further have added additional analyses to account for batch effects as discussed in the global response.

I would like to thank the authors for adding new batch effect analysis presented in Figure 12 (note wrong color in figure legend) and additional description of the dataset presented in Figure 9 that improve the critical weaknesses and answer partially to my questions. The cosine similarity seems fairly low specifically for the same perturbation in the same mouse and the same lobe samples. Could the authors elaborate in discussion whether that is expected taking into account the effect of perturbations seeable visually in histological sections or are the following steps in sample preparation (sectioning, staining etc.) causing this variation?

Regarding my comment about mean pooling for aggregation, I meant that mean aggregation is simplistic approach and more advanced methods such as attention pooling could be tested.

Since the authors have included new analyses that respond to my concerns, I will increase my rating for the paper.

We thank the reviewer for recognizing the value of our new analyses, and for increasing their rating to reflect this. We also thank the reviewer for flagging the issue with the legend for Figure 12; we’ve fixed that issue in the current version of the manuscript.

To address the reviewer’s specific questions and suggestions:

  • Aggregation methods: We quickly did a re-analysis today on UNI2-h embeddings with attention pooling aggregation, and found a benefit of 1.44% in the mAP.  Since this is just a preliminary look, we have left it out of the manuscript, but can include a more complete and verified analysis in the final version if the manuscript is accepted.
  • Low cosine similarity between perturbations: Note that the low mean cosine similarity are expected given the complexity of the dataset. Beyond variability in perturbation efficiency (which varies per tumor), the tumor size and tumor local environment can also create heterogeneity. As the reviewer suggests, we can elaborate this in more detail in the Discussion for our final version.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response to Reviewers (Part 1/2)

We would like to thank all the reviewers for their constructive feedback. We were particularly encouraged by the facts that:

  • Reviewers found our multimodal in vivo dataset of gene knockout tumors to be novel, well-curated, and sufficiently distinguished from existing datasets in this field by having a robust experimental design not present in human histopathology datasets.
  • Reviewers acknowledged our significant efforts to benchmark many open-source models in a comprehensive manner across a wide range of biologically meaningful tasks.

A point that we would like to clarify to all reviewers is that our entire dataset and benchmarking code will be open-sourced upon acceptance to ensure reproducibility of the benchmark. We have updated the abstract of our paper to make this point more clear. We believe that open sourcing the Perturb-Path benchmark will be a significant contribution to the entire research community as a novel, challenging, and biologically relevant benchmark for pathology foundation models.

The reviewers’ concerns are centered around two main questions below. Addressing their feedback (summarized below) has strengthened our manuscript and reinforced our conclusions.

Common question 1: Have we sufficiently controlled for batch effects of various kinds?

We wish to clarify that there are three levels of potential batch effects in our dataset:

  • lobe-level batch effects (variance in measurement from lobe to lobe within the same mouse)
  • mouse-level batch effects (variance in measurement from mouse to mouse within the same library)
  • library-level batch effects (variance in measurement across libraries)

We designed our benchmark tasks so that lobe-level and mouse-level batch effects cannot inflate benchmarks. In the mIF task, predictions are evaluated on held-out mice so a model cannot improve its benchmark performance by picking up on these batch effects. In within-lobe mAP, only tumors on the same lobe are compared to each other, and neither lobe-level nor mouse-level batch effects can affect these comparisons. And in cross-mouse mAP we only compute similarities between perturbations in different mice, so mouse-specific batch effects can only hurt retrieval. In other words: batch effects can only decrease model performance. Yet despite this, we see statistically significant performance on mIF and both mAP tasks, suggesting that there is real biological signal in the Perturb-Path dataset that overcomes the lobe-level and mouse-level batch effects.

However, the reviewers are correct to raise concerns around library-level batch effects. The perturbations are split between two separate CRISPR libraries. We never properly assessed whether the same-library similarities were stronger than cross-library similarities. If this library-level batch effect were sufficiently strong, it could inflate the results on the cross-mouse mAP task (but not the within-mouse mAP task) by making some perturbations appear to be artificially retrievable, which in turn would result in the PFMs being ranked by their ability to detect library batch effects rather than their ability to capture perturbation biology. We now demonstrate in Appendix Section A.6 that this is not the case. While there are library-level batch effects, the effect is not strong enough to drastically impact the mAP / null measurements (Appendix, Figure 13). In fact, the ranking of PFMs barely changes (Spearman correlation = 0.996) when we switch from unrestricted comparisons to only within-library comparisons.

In theory library-level batch effects should not affect mIF-related tasks, which are perturbation-agnostic and randomly sample train/test mice independent of library. Nonetheless, as a sanity check we reran the mIF task in a controlled cross-library vs. within-library comparison; there is still a positive correlation for all PFMs in the cross-library setting, suggesting that biological signal from mIF is still conserved across libraries (Appendix, Figure 14).

In summary:

  • Lobe-level batch effects cannot inflate benchmark performance due to the design of our benchmark tasks.
  • Mouse-level batch effects similarly cannot inflate benchmark performance.
  • Library-level batch effects do exist, but there is still sufficient alignment in biological signal from H&E and mIF across libraries to empower our benchmark tasks.

Global response continued in the thread below.

AuthorsOpenReview ↗

Global Response to Reviewers (Part 2/2)

Common question 2: Can we further explore the content of the dataset and show that it contains diverse biological phenotypes that can be retrieved from H&E? Is there a real (model-independent) biological signal in the mouse data that correlates with the human data?

To answer this question we dug deeper into the dataset itself with some additional analyses. In the Appendix (Figure 9), we have added a heatmap depicting the perturbations included in our benchmark dataset (9a) and the number of H&E tiles that contribute to each tumor region (9b). To describe the consistency of cell type effect, we have added split-half correlations for the measured CD8+ T cells, CD4+ T cells, and macrophages across perturbations. Encouragingly, despite not all perturbations expected to have a cell-type effect, we see a strong correlation in these split-half correlations.

Second, as an additional positive control, we confirmed whether known biological effects of specific perturbations are captured in this dataset. We selected two example perturbations, Socs1 and Ctnnd1, with high identifiability (by mAP) in our benchmark. Tumors with these Socs1 and Ctnnd1 KOs have been associated in the literature to have increased CD8+ T cell abundance and macrophage abundance, respectively (Dhainaut, Rose et al, 2021, Kurley et al, 2021). Encouragingly, we also observe this in our dataset, as shown in the Appendix (Appendix, Figure 17), further adding evidence that the PFMs are capturing biological effects from the H&E.

We have additionally addressed the questions that are specific to each review in the corresponding threads below.

In addressing reviewer’s concerns and questions, we have made several additions to the manuscript:

  • Three new PFMs: CONCH v1, CONCH v1.5, and MUSK. Unlike the original suite of PFMs we benchmarked, all three of the new models are vision-language PFMs. This brings the total number of PFMs benchmarked to 14, plus the 3 ImageNet baseline models.
  • Comparison with HEST benchmark alongside THUNDER and PathBench.
  • New analyses in the Appendix on batch effects and reproducibility of known biology.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.