acceptodds
Under review as a conference paper at ICLR 2027

HiLO: Hierarchical Label-Aware Optimal Transport for Cross-Domain Alignment of Pathology Foundation Model Embeddings

Abstract

Pathology foundation models produce strong patch embeddings for H&E whole-slide images (WSIs). However, their embeddings remain highly predictive of non-biological factors such as source centre, scanner, and staining protocol. Consequently, downstream models may fail to rely on biologically meaningful features, which limits their clinical applicability. Optimal transport (OT) offers a principled way to align feature distributions across domains. However, standard OT algorithms exhibit quadratic memory and time complexity in the number of points, making them highly computationally expensive for WSIs. We propose HiLO, a hierarchical label-aware optimal transport framework for cross-domain feature alignment that can scale efficiently to large datasets. Building on hierarchical refinement, we compute globally optimal one-to-one correspondences between foundation-model patch embeddings from different source domains in linear memory. We embed label information into the transport cost by augmenting features with a label embedding. This preserves the low-rank cost factorization required for scalability and suppresses cross-class matches under label shift. The resulting one-to-one correspondences define a Monge map that transports patch embeddings from each source domain to the target domain, on which any downstream model can be trained. We evaluate our method on two publicly available pathology WSI datasets AGGC2022 and BEETLE. All experiments follow a leave-one-domain-out protocol, where each held-out domain is a scanner or centre depending on the dataset, and we report both in-domain (ID) and out-of-domain (OOD) performance. Compared with state-of-the-art OT, stain normalisation, and domain generalisation methods, HiLO achieves higher performance, while also attaining lower total transport cost than the OT baselines. In conclusion, HiLO is a scalable and flexible framework for cross-domain feature alignment that is applicable to large-scale pathology WSIs and can be extended to a wide range of downstream applications.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Optimal Transport, Computational Pathology, Pathology Foundation Models, Domain Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EU7WOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper targets an important practical limitation of Pathology Foundation Models (PFMs), whose embeddings encode non-biologically relevant factors, e.g. scanner, source center. To overcome these batch effects, they propose a three-fold approach (HiLO) that leverages a label-aware optimal transport formulation. Specifically, HiLO, first creates two pools of patch embeddings with equal class counts and number of patches per pool. Secondly, they perform the label aware matching of the two sets. Lastly, the method uses the learned map to train an alignment framework jointly with a linear classifier. With this, they aim to overcome the DG objectives' limitations that perform alignment on a distribution rather than a sample level and the label-agnostic nature of the current OT techniques used.

Critical strengths
  • The lable-aware extension of the current OT approaches seems well-motivated
  • The plug-and-plug nature of the approach that allows it to be combined with other methods, seems flexible and beneficial. The method tackles a really important problem in histopathology and offers an easy deployment solution as it operates on pre-extracted embeddings.
Critical weaknesses
  • Methodologically, the extension to label-aware OT feels a bit incremental. Also, the gain from label-awareness over λ_y = 0 is small, which questions the motivation.
  • The single proposition provided, while it is true, the proof provided in the appendix does not support the claim, but rather a much weaker version of it.
  • The experimental section gives some mixed signals. While the methodology prescribes that λy2>2\lambda^2_y > 2, most experiments use λy=0.5\lambda_y = 0.5. There is no std provided in any of the presented Tables, and given that the method is close to the baselines this is critical. Moreover, the class accuracy drop (Table 9, Appendix) seems large, as scanner accuracy vs class accuracy is not an inherent trade-off of the biological problem but rather the method.
Questions for authors
  • While I understand the value of incorporating class labels to the OT formulation, what is the motivation of solving the problem across all tasks C and not per class? As for λy>2\lambda_y > 2 it holds:

min⁡σ∑i=1Nci,σ(i)=∑c=1Cmin⁡σc:Ac⟶Bc∑i∈Ac∥ziA−zσc(i)B∥22,\min_{\sigma}\sum_{i=1}^{N}c_{i,\sigma(i)}=\sum_{c=1}^{C}\min_{\sigma_c:A_c{\longrightarrow}B_c}\sum_{i\in A_c}\|z_i^A-z_{\sigma_c(i)}^B\|_2^2,

where Ac,BcA_c, B_c the subsets that contain the samples of the c class of A,BA, B, respectively and σc\sigma_c is bijective by definition of the problem. This would enforce the class constraint directly, without the need for λy\lambda_y and can be computationally more efficient as 1) it requires less memory per class and 2) can be easily parallelized.

  • Re the proof: What the authors prove is that if λy2>2N\lambda^2_y > 2N then the minimizer contains no cross-class match. Is this a typo in proposition's statements? For the stronger version you should have in the spirit of: suppose there is an optimal coupling σ\sigma that minimizes (4) with cost CC and contains k>0k>0 cross-class matches, then C≥2λy2k>4kC \ge 2\lambda^2_yk > 4k. Then if I rematch these kk pairs to have no cross-class matches, I will have C′≤C−2λy2k+∑k∣z~i−z~i′∣22C' \le C - 2\lambda^2_yk + \sum_k|\tilde{z}_i - \tilde{z}_i'|^2_2, where the sum is over the new pairs. Since these pairs have matching labels and the feature term in Eq. (3) lies in [0,4][0,4], we have ∑k∣z~i−z~i′∣22−2λy2k≤4k−2λy2k<0\sum_k|\tilde{z}_i - \tilde{z}_i'|^2_2 - 2\lambda^2_yk \le 4k - 2\lambda^2_yk < 0 (strictly negative) for λy2>2\lambda^2_y > 2, and so C′<CC' < C (contradiction). Then, you combine it with the existence of the minimizer with no cross-matches.

  • Why in practice λy\lambda_y is set to 0.5?

  • Retaining 87% of the class accuracy indicates that this alignment approach, while beneficial for removing scanner accuracy, results in loosing biologically relevant information. How do then other baselines behave in this trade-off, as I understand that Table 9 only concerns HiLO?

  • The method emphasizes scalability but does not report any relevant experiments. What are the wall-clock time and memory costs compared with the other OT baselines?

1 comment

The authors are grateful for the time and effort spent by the reviewer in assessing our submission, and especially for the corrected argument for Proposition 1. We have addressed all comments by correcting the proof, reporting error bars, and conducting new experiments. Detailed actions and responses to the reviews are as below.

1. The extension to label-aware OT feels a bit incremental. Also, the gain from label-awareness over λy=0\lambda_y = 0 is small…

We propose our method HiLO based on the pathological significance and computational cost. Since HiLO allows the cost to stay as a squared Euclidean distance, we can still use the low-rank factorisation of HiRef and scale to WSI datasets. We agree that the gain over λy=0\lambda_y = 0 is small on cleaner labels. However, it is important to note that computational pathology often involves noisy, heterogeneous labels with complex patterns. In our experiments with noisy labels, the gain reaches 0.013 when 30% of the training labels are flipped (Appendix E).

2. The proof provided in the appendix does not support the claim, but rather a much weaker version of it.

We thank the reviewer for pointing out this mistake. The proof only showed the weaker condition (λy2>2N\lambda_y^2 > 2N). We have corrected it with the argument outlined by the reviewer (Appendix A) and acknowledged this in the revised manuscript.

3. While the methodology prescribes that λy2>2\lambda_y^2 > 2, most experiments use λy=0.5\lambda_y = 0.5.

We apologise for the unclear description. Proposition 1 only gives a sufficient condition for the exact solution, and it was not meant as a rule for choosing λy\lambda_y. We have now swept λy\lambda_y from 0 to 2.0 on CONCH v1.5 (Appendix D). OOD macro-F1 is highest at λy=0.5\lambda_y = 0.5, where the cross-class matches already drop from 18.8% to 0.8%.

4. There is no std provided in any of the presented Tables…

We acknowledge that the error bars are required to make a fair comparison. All OOD results are now reported with mean ±\pm std over all combinations of fold and seed, and the claims that HiLO is better on every FM have been removed.

5. The class accuracy drop (Table 9, Appendix) seems large…

We computed the same probe on ERM features, which come from the same network without alignment (Appendix G). On CONCH v1.5, class accuracy is 0.921 for ERM and 0.879 for HiLO. Section 4.2 is revised accordingly.

Q1. What is the motivation of solving the problem across all tasks C and not per class? As for λy>2\lambda_y > 2 it holds that the minimum over σ\sigma decomposes into C independent per-class problems…

We thank the reviewer for this suggestion. We agree that for λy2>2\lambda_y^2 > 2 the problem can be solved per class. We have now added this baseline (Appendix D), which reaches an OOD macro-F1 of 0.811 on CONCH v1.5, compared with 0.817 for HiLO with λy=0.5\lambda_y = 0.5.

Q2. Is this a typo in proposition's statements?

This was a mistake in the proof and not a typo in the statement. Please refer to our response to point 2.

Q3. Why in practice λy\lambda_y is set to 0.5?

Please refer to our response to point 3.

Q4. How do then other baselines behave in this trade-off, as I understand that Table 9 only concerns HiLO?

We have run the same probe for ERM, low-rank OT and Fish on CONCH v1.5 (Appendix G). HiLO gives the lowest scanner accuracy (0.276) and a higher class accuracy than low-rank OT (0.879 against 0.871).

Q5. What are the wall-clock time and memory costs compared with the other OT baselines?

We have added this experiment in Appendix F. At N=32,768N = 32{,}768, HiLO uses 365 MB of GPU memory, compared with 34.7 GB for Sinkhorn and 17.4 GB for low-rank OT and FRLC. These methods run out of memory on a 40 GB A100 at N=65,536N = 65{,}536, while HiLO runs up to N=262,144N = 262{,}144.

Once more, we would like to thank the reviewer for the constructive comments, which have strengthened our paper significantly.

Reviewer_4rJdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HiLO, a hierarchical label-aware optimal transport framework for aligning pathology foundation model embeddings across different domains. The method incorporates tissue class labels into the OT cost to reduce incorrect cross-class matching, and uses a hierarchical refinement algorithm to efficiently obtain patch-level correspondences. These matched embeddings are then used to train an alignment network for improved domain generalization.

Critical strengths

Scanner and center-related variations can significantly affect pathology foundation model embeddings, and improving their robustness is valuable for downstream applications.

The proposed method is relatively simple and can be applied to embeddings from different pretrained foundation models without modifying the original encoders.

Critical weaknesses

The proposed matching prevents patches from different tissue classes being aligned, but considerable biological heterogeneity may still exist within the same class. For example, tumor patches can differ in morphology, differentiation, and other features potentially relevant to prognosis. HiLO may still align such patches even when they are biologically different. The authors mainly use tissue classification performance to show that biological information is preserved.

Most of the experiments focus on scanner-related domain shifts, which is certainly a relevant problem. However, generalization across scanners is not necessarily equivalent to generalization across independent hospitals or patient populations. In particular, AGGC2022 includes the same specimens scanned by multiple scanners. It is not fully clear from the main paper how specimen-level independence is handled between training and held-out test domains. If different scans of the same specimen can appear across these splits, the setting would mainly evaluate scanner robustness rather than generalization to completely unseen specimens.

The main idea in this paper is to incorporate one-hot class labels into patch embeddings and penalize cross-class matches during OT. However, the method assumes that the two sampled domains have the same number of patches for each class. Under this setting, when the label penalty is sufficiently large, the optimal matching contains only same-class pairs, as also shown in Proposition 1. This makes the approach quite similar to simply performing OT separately within each tissue class.

Questions for authors

How does HiLO compare with performing class-wise OT using HiRef directly, without label-augmented embeddings?

How robust is the proposed matching when tissue labels contain errors or are not consistently annotated across domains?

1 comment

We sincerely thank the reviewer for the thoughtful and constructive comments, which we found very helpful in improving our work. In response to the comments and comments by other reviewers, we have carried out a comprehensive revision of the manuscript, carefully addressing each of the points raised. We believe these changes have strengthened the clarity, rigor, and overall quality of the paper, and we are grateful for the reviewer's time and valuable feedback.

1. Within-class biological heterogeneity.

We agree that the current implementation of HiLO matches tiles only at the tissue-class level and may align tiles that differ in finer morphology within a class. To quantify the effect, we compared class and scanner probe accuracy against ERM and other baselines (Appendix G). On CONCH v1.5, HiLO retains most of the class information (0.879 vs. 0.921 for ERM) while giving the lowest scanner accuracy.

2. Scanner shift and specimen-level generalization.

We agree that scanner robustness is not equivalent to generalization across hospitals or patient populations. However, working with public data restricts our data resources, and the current work provides a strong foundation for future research in this area.

3. Similarity to class-wise OT.

We agree that for sufficiently large λy\lambda_y the problem decomposes into per-class problems, and we have corrected the proof of Proposition 1 accordingly (Appendix A). We have added the class-wise baseline using HiRef directly, which reaches an OOD macro-F1 of 0.811 on CONCH v1.5, compared with 0.817 for HiLO (Appendix D). We have also conducted experiments on changes to λy\lambda_y and show that the best performance occurs at λy=0.5\lambda_y = 0.5 (Appendix D).

Q1. Comparison with class-wise HiRef.

See our response to point 3 and Appendix D.

Q2. Robustness to label errors.

We ran experiments with noisy labels. With 30% of training labels flipped, the gain over λy=0\lambda_y = 0 reaches 0.013 (see Appendix E).

Once more, we thank the reviewer for the insightful comments, which have helped us improve the clarity and rigor of the paper.

Reviewer_FMBvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce an optimal transport based domain alignment approach for various foundation models in the specific sub-field of domain generalisation. While the topic is important, and strictly inside the scope of ICLR, there are several glaring issues in the manuscript that needs revision. The lack of error bars also does not extend confidence to the results in general.

Critical strengths

1.⁠ ⁠The problems addressed by the authors is real and concerning. Foundation models tend to have domain generalization issues which limits their applicability to out-of-distribution tasks. 2.⁠ ⁠The authors introduce a clever Optimal Transport based approach in order to address the domain generalization issue. 3.⁠ ⁠While the quality of the results can be discussed further, the authors have done a thorough evaluation of their approach against a few publicly available datasets, methods, and many foundation models. The extent to which this has been done is greatly appreciated.

Critical weaknesses

At the outset, vulnerabilities of the results section needs to be pointed out. The presented results have no error bars, therefore a comparison taking into account the spread of the predictions is impossible from this version of the manuscript draft, and consequently, a judgement of the merits of the presented approach is not possible.

Even after discounting the lack of spread information, several sentences by the authors in the concluding remarks can be challenged from the presentation of their own results. I would like to point out 2 examples:

Item 1: “HiLO improves OOD macro-F1 over stain normalisation, other OT methods, and DG methods on all FMs”, second last sentence of the conclusions section. 1.⁠ ⁠Table 1, all stain normalisation methods are better for PLIP. 2.⁠ ⁠Table 1, HOptimus-1, with low-rank OT, is better. 3.⁠ ⁠Table 1, Midnight, with mini-batch Sinkhorn, is better. 4.⁠ ⁠Table 1, MUSK, with low-rank OT and FRLC , is better. 5.⁠ ⁠Table 2, Paired CORAL is better for PLIP. 6.⁠ ⁠Table 2, Fish for CONCH v1.5 is better, same extends for Midnight 7.⁠ ⁠Table 2, both CORAL and IRM are better for MUSK.

Item 2: “OT methods that derive matches from mini-batch or low-rank couplings yield lower OOD macro-F1 than HiLO on all FMs”, Discussion section, second paragraph, second-last sentence. 1.⁠ ⁠low-rank OT 0.794 > 0.789 (H-optimus-1) 2.⁠ ⁠0.764 > 0.755 (Midnight) 3.⁠ ⁠FRLC 0.658 and low-rank OT 0.651 > 0.649 (MUSK)

Furthermore, this following sentence incorrectly references the authors’ own results, conflating two results from different evals altogether.

“HiLO alone also outperforms both Mixup and GroupDRO on all 8 FMs and the largest gain occurs on Virchow, where the combination with GroupDRO reaches 0.763, compared with 0.740 for HiLO and 0.696 for GroupDRO.” Last sentence right before Discussions. If we consult table 3, the numbers for Virchow cited in this sentence correspond to the AGGC2022 dataset, whereas the numbers for HiLO correspond to the BEETLE dataset.

Furthermore, there is one additional claim in the abstract: “we compute globally optimal one-to-one correspondences between foundation-model patch embeddings from different source domains in linear memory”. I expect that, for such a claim, global optimality needs to be strictly motivated. Such a claim remains unsubstantiated in the manuscript.

Furthermore, there seems to be an unnecessary confusion brought about by the use of histopathology terminology (tiles that are cut out from whole slide images are usually called patches), which leaks into discussion about vision transformer representations. A vision transformer produces a CLS-token and a patch-token, so the authors use of the patch-embeddings to signify CLS-token is very confusing.

Presumably DG means domain generalization but it is never defined in the manuscript.

Questions for authors

If you substantially clean up these results, there is good work underlying this presentation.

Can you explain how you reach some of your claims: "we compute globally optimal one-to-one correspondences between foundation-model patch embeddings from different source domains in linear memory." "HiLO alone also outperforms both Mixup and GroupDRO on all 8 FMs and the largest gain occurs on Virchow, where the combination with GroupDRO reaches 0.763, compared with 0.740 for HiLO and 0.696 for GroupDRO." etc.

1 comment

We are grateful to the reviewers for taking the time to evaluate our manuscript and for their valuable feedback. Having carefully considered all of the comments, we have made a number of revisions to the text, which we detail point by point. Every result-related statement has been checked against the tables, and all OOD results now include error bars. Detailed actions and responses to the reviews are as below.

1. The presented results have no error bars…

We acknowledge that the error bars are required to make a fair comparison. All OOD results are now reported with mean ±\pm std over all combinations of fold and seed. Furthermore, we confirm that our method consistently outperforms ERM on all 10 FMs, by 0.036±0.0070.036 \pm 0.007 in OOD macro-F1 (mean ±\pm std across FMs).

2. Several sentences by the authors in the concluding remarks can be challenged… (Item 1 and Item 2)

We apologise for the inaccurate wording. Several experiments were added close to the deadline and the text was not fully updated to match the tables. We have now carefully checked every result-related sentence against the tables and corrected it. In conclusion, our results show that HiLO improves OOD macro-F1 and is the best or second-best method on most FMs. All changes related to this are listed below and highlighted in red in the revised manuscript.

  1. Abstract (Item 1). Revised to "Compared with state-of-the-art OT, stain normalisation, and domain generalisation methods, HiLO achieves higher performance on most FMs."
  2. Introduction, last paragraph (Item 1). Revised to "On BEETLE, HiLO alone also outperforms Mixup and GroupDRO on all FMs" and "HiLO outperforms stain normalisation and existing OT methods on most FMs".
  3. Section 4.2, first paragraph (Item 1). Revised to "HiLO achieves the highest OOD macro-F1 among stain normalisation and OT methods on 6 of the 10 FMs ... On the remaining 4 FMs, HiLO is outperformed by low-rank OT on H-optimus-1 (0.794 vs. 0.789), mini-batch Sinkhorn on Midnight (0.764 vs. 0.755), FRLC on MUSK (0.658 vs. 0.649), and Vahadane on PLIP (0.620 vs. 0.577)."
  4. Section 4.3, second paragraph (Item 1). Revised to "HiLO exceeds every DG method on 6 of 10 FMs ... On the remaining FMs it trails the best DG method by at most 0.008, and it performs comparably to Paired CORAL on PLIP, at 0.577 against 0.578."
  5. Discussion, second paragraph, second-last sentence (Item 2). Revised to "OT methods that derive matches from mini-batch or low-rank couplings yield lower OOD macro-F1 than HiLO on most FMs."
  6. Conclusion, second-last sentence (Item 1). Revised to "On AGGC2022, HiLO improves OOD macro-F1 over stain normalisation, other OT methods, and DG methods on most FMs, and on both AGGC2022 and BEETLE it outperforms Mixup and GroupDRO on every FM."

3. This following sentence incorrectly references the authors' own results…

We apologise for this mistake. The concluding statement (Section 4.4) is clarified now as "Adding HiLO to a DG method improves OOD macro-F1 over that method alone on 19 of 20 settings on AGGC2022 and on all 16 settings on BEETLE, by 0.042 and 0.022 on average. The combination also exceeds HiLO alone on most FMs, and HiLO alone outperforms both Mixup and GroupDRO on every FM of both datasets."

4. Global optimality needs to be strictly motivated…

We acknowledge that global optimality may not be guaranteed by our method, since HiLO builds the one-to-one matching by hierarchical refinement. We have therefore removed "globally optimal" from the abstract and the manuscript. The abstract is corrected accordingly.

The linear-memory claim is also now supported by a new scaling experiment (Appendix F). At our working point of N=32,768N = 32{,}768, HiLO uses 365 MB of GPU memory, which is 95×95\times less than Sinkhorn and 48×48\times less than low-rank OT and FRLC. These three methods run out of memory on a 40 GB GPU at around N=65N = 65k, while HiLO scales up to N=262N = 262k.

5. …the authors use of the patch-embeddings to signify CLS-token is very confusing.

All mentions of image regions extracted from whole slide images are renamed as tiles. In accordance with this, all mentions of 'patch embeddings' are renamed to 'tile embeddings'. The term 'patch' is now used only for vision transformer patch tokens.

6. Presumably DG means domain generalization but it is never defined…

The abbreviation is defined at the first mention in the Introduction.

Questions. Can you explain how you reach some of your claims…

We thank the reviewer for the encouraging comment. Please refer to our responses to points 3 and 4.

Once more, we would like to thank the reviewer for the constructive comments. The corrections change how the results are described, while the original proposed method and experiments remain unchanged.

Comments on the paper

1 comment

The authors are grateful for the time and effort spent by the reviewers in assessing our submission. We would like to sincerely acknowledge the shortcomings of our writing and presentation caused by a rushed final revision of the paper before the deadline. We have conducted numerous experiments just before the deadline and therefore the accompanying text was not fully aligned with the final results.

We are encouraged that the reviewers recognised the importance of the problem, identified the strengths of our work, acknowledged the comprehensiveness of our experiments. More importantly, we are very pleased to receive constructive comments that have strengthened our paper significantly.

We have corrected all shortcomings pointed out by the reviewers. We have also answered the comments of each reviewer individually.

We would be grateful to request your reconsideration of our revised manuscript for evaluation and publication, and we thank you for taking another look at our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.