acceptodds
Under review as a conference paper at ICLR 2027

Optimal Transport Meets t-SNE: Integrated Visualization without Batch Effects in OT-tSNE

Abstract

Visual exploration is an integral step in understanding the structure of datasets. However, systematic variation between datasets collected under different conditions, known as batch effects, can obscure shared structure and complicate their joint visualization. Existing methods often rely on cross-batch nearest-neighbor relations to integrate the batches. Yet, these local correspondences can be unreliable for challenging cross-batch geometries. We propose optimal-transport t-SNE (OT-tSNE), which replaces nearest-neighbor-based matching with globally coupled affinities derived from entropic optimal transport. Controlled synthetic experiments demonstrated that nearest-neighbor-based methods can deteriorate while optimal-transport affinities and OT-tSNE embeddings remain robust. Across seven real single-cell datasets, we benchmarked OT-tSNE against a broad range of integration and visualization methods and found that it achieves a Pareto-efficient trade-off between batch mixing and biological conservation, and integrates batches particularly well.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
visualization, data integration, batch effects, low-dimensional embedding, dimensionality reduction, entropic optimal transport

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WXuzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes optimal-transport t-SNE (OT-tSNE). Datasets collected under different conditions are affected by batch effects, which make joint visualization of the data difficult. To address this problem, the proposed method provides a methodology that generates well-integrated visualizations across batches while preserving the structure of each individual batch. The authors evaluate the visualization of seven datasets containing batch effects in two dimensions and demonstrate the effectiveness of the proposed method.

Critical strengths

As stated in the contributions of this paper, the primary strength of this work lies in proposing OT-tSNE, the first combination of optimal transport and neighbor embedding methods for the integrated visualization of data with batch effects. The paper also shows that while conventional nearest-neighbor-based methods can degrade even when the batch effect is orthogonal to the data, OT-tSNE is not affected by it. Furthermore, through a large-scale benchmark for visualizing data with batch effects, this paper demonstrates that OT-tSNE achieves high performance, particularly in batch mixing.

Critical weaknesses

A more in-depth discussion of the validity of the parameters used in the proposed method is desirable. For example, the rationale for choosing the values of the parameters described in the Implementation details, including setting the number of neighbors to 15, is not sufficiently explained. It would also be desirable to clarify how sensitively these parameters affect the results of the proposed method.

A theoretical comparison of the computational complexity of the proposed method with that of existing methods is desirable. Although the paper discusses runtime in the Limitations and Future Work section, runtime depends on the implementation and the computing environment. The reviewer considers it necessary to base the discussion on a comparison of theoretical computational complexity, rather than on whether the method can be accelerated through implementation.

The reviewer would like the authors to present the algorithm of the proposed method explicitly, for example in the form of pseudocode. This would clarify how the processing pipeline differs from that of existing methods and which steps, if removed, would reduce the method to existing work. As a result, it would also become easier to understand how the newly added steps contribute to the effectiveness of the method. In addition, it would be desirable for the description of the proposed method to include a theoretical explanation, through comparison with existing work, of why the proposed method can solve the problem.

OT-tSNE is currently formulated for two batches. However, since real-world data may involve three or more batches, the reviewer believes that the authors should present a concrete method for extending it to multiple batches and conduct experiments that reflect realistic problem settings. Note that this point is listed here as a limitation of the proposed method and does not lower the reviewer’s evaluation of this paper.

Questions for authors

The quantitative evaluation is based on rankings, and this paper states that this follows established practice in prior work. For the proposed method, the results sufficiently confirm its effectiveness. On the other hand, the reviewer would like the authors to explain whether the rankings among the compared existing methods can be considered valid from the perspective of the proposal in this paper.

How sensitively do the parameters of the proposed method affect accuracy? The reviewer would like to see results that clarify this. In addition, the reviewer would like the authors to clarify, in terms of the number of parameters, how much more complex the proposed method is compared with existing methods.

The paper states that OT-tSNE relies on balanced optimal transport, which forces every observation in each batch to be equally associated with the other batch. The reviewer would like the authors to clarify how much this constraint affects the results when applied to real-world data.

6 comments

Dear reviewer WXuz,

Thank you very much for the time invested in reading our paper and the detailed feedback you provided!

Validity of the chosen hyperparameters and sensitivity analysis

OT-tSNE has two main hyperparameters for affinity construction. The neighborhood size kk controls graph sparsity. A small kk can omit local connections, while a large kk retains weaker affinities that can blur local structure. We chose k=15k=15, following the neighborhood-size trade-off discussed for UMAP (McInnes et al., 2018). Overall, it is a standard default for neighborhood graphs in visualization, used in UMAP and NCVis (McInnes et al., 2018, Artemenkov & Panov, 2020).

For entropic regularization, we follow Landa et al. (2024) and set ϵ=rmedian⁡(C)\epsilon=r \operatorname{median}(C), where CC is the matrix of squared Euclidean distances between batches. Setting r=1r=1 makes the regularization strength adapt to the cost scale of each dataset, which ''has been found robust against high-dimensional noise and theoretically appealing'' (Landa et al., 2024, Ding & Ma, 2023).

The sensitivity analysis we now added to Appendix E.1 supports our choice of defaults. When averaged across the seven datasets, the overall score improved by less than 0.010.01 relative to the default (k,r)=(15,1)(k,r)=(15,1). We discuss how varying kk and rr affected batch mixing and biological preservation in our response to Reviewer WXuz.

Pseudocode

We added the pseudocode in the new Appendix C.1 and refer to it in line 187 of the revised manuscript. Thank you for suggesting this addition!

[...] to include a theoretical explanation, through comparison with existing work, of why the proposed method can solve the problem.

We fully agree that it is important to understand why OT-tSNE can help with batch integration. This was our aim in Section 5.1, where we compared OT-based affinities with the shared and mutual kNN graphs used by MultiMAP and Scanorama, respectively. We focused on perturbations orthogonal to the biological signal because this assumption also motivates other batch correction methods (e.g., Haghverdi et al., 2018). Eqs. 8 and 9 give a theoretical explanation for the advantage we observed for OT-tSNE in this setting. Specifically, the invariance of the OT plan to these perturbations helps explain why OT-tSNE's cross-batch affinities remained stable even under strong batch effects, while the quality of the cross-batch connections in the kNN graphs declined.

In the Introduction, we also explain our choice of OT by examining where cross-batch kNN graphs can fail. Fig. 1b illustrates a complementary case in which a constant shift along the signal direction changes cross-batch nearest neighbors, leading to a ''kissing problem'' in the visualization. The balanced OT plan remains invariant at fixed ϵ\epsilon because the shift adds only row and column terms to the squared-distance matrix. The figure illustrates this effect in two dimensions, but the argument holds in any dimension. We then examined this case on MNIST by comparing visualizations from OT-tSNE and kNN-based methods. We plan to provide Jupyter notebooks illustrating both constant and heterogeneous batch effects, with perturbations either parallel or orthogonal to the signal subspace.

Interpretation of the rankings

The rankings summarize overall performance, batch mixing, and biological conservation across the seven datasets. The overall ordering reflects the trade-off between batch mixing and biological conservation evaluated in this paper, while the two category rankings show how methods compare on each objective separately. For example, LIGER 50D ranked ahead of MultiMAP on batch mixing, while MultiMAP ranked ahead on biological conservation. With both objectives weighted equally, MultiMAP placed higher overall (Fig. S4).

We revised the main text to clarify how we compute the ranks and added guidance on their interpretation in Appendix D.1. Since a rank difference does not convey the size of a performance difference, we also report the metric scores themselves. The pooled plots show how methods compare on both batch mixing and biological conservation (Fig. 4), and the per-dataset plots show how that comparison varies across datasets (Figs. S10, S11).

Effect of marginal constraints when applied to real data

Thank you for this question! We believe our initial submission did not sufficiently emphasize that the cell-type compositions in the seven benchmarked datasets are not balanced (Figs. S1, S2). Despite the imbalances, our real data benchmark showed that OT-tSNE can achieve strong batch mixing and competitive cell-type preservation even with balanced marginal constraints. On Mouse Atlas, for example, B cells accounted for roughly 29% of one batch and 8% of the other (Fig. S1c). We identified Mouse Atlas as the dataset with the second largest compositional imbalance in the benchmark. Yet OT-tSNE had the strongest balance in batch mixing and bio preservation compared to other methods on Mouse Atlas (Fig. S5). We now extended our Section 5.2 and Conclusion to state these results more explicitly.

Extension to multiple batches

We are currently working on extending OT-tSNE for multibatch integration. For each pair of batches, we compute an OT plan and use it to construct the corresponding cross-batch affinity block. Within each batch, we compute affinities using either OT or kNN, or set the corresponding block to zero. We combine these blocks into a single affinity graph and sparsify it as in the two-batch method. A single t-SNE optimization on this graph then yields a joint embedding of all batches.

Preliminary results on two real single-cell datasets settings are promising. A broader benchmark is underway to assess how consistently the extension preserves cell types while mixing batches across multiple batches.

Computational complexity and hyperparameter complexity comparison

We are currently comparing OT-tSNE with the benchmarked methods in terms of computational complexity and the number of hyperparameters that require tuning. We plan to share this comparison during the discussion period.

Reviewer_WXuzOpenReview ↗

Re: Official Comment by Authors

Thank you for submitting your comments. The reviewer has responded favorably to many of the revisions.

However, regarding the validity of the parameters, while the authors mention that they are using values commonly employed in other methods, it does not seem that they have gone into detail regarding why those values are appropriate for OT-tSNE. The reviewer would like to learn more about this point.

The reviewer also looks forward to the authors’ comments regarding the comparison of computational complexity and the complexity of the number of parameters.

Reviewer_ukDmOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper solves an interesting problem of visualizing datasets with batch effects using optimal transport (OT) and t-SNE. Its contributions are as follows:

(1) First to use OT and tSNE for visualization: The paper uses OT to obtain a coupling matrix, then combines it with kNN information to create an affinity matrix that is used with t-SNE for visualization.

(2) The results show that OT-tSNE outperforms other competing methods for batch-effected visualization, with extensive benchmarking on different metrics combining within-batch and between-batch matching.

Critical strengths

(1) To my knowledge, this is the first attempt to use OT with tSNE for visualization

(2) The results show promising results that the proposed method performs reasonably well in many experiments

Critical weaknesses

(1) t-SNE, as a visualization tool, is not stable and does not reliably represent the actual distances of real data. Furthermore, it can be manipulated with different seeds. The paper does not address these points.

(2) The real-world use case of the proposed method is limited because the dataset needs to be balanced; only applied to 2 batches. Whereas in a real-world setting, one cannot ensure balanced datasets that are limited to 2 batches

(3) Benchmarking on simulation needs to add a metric for 1-1 correspondence matching (like Fraction of Samples Closer than the True Match score) to utilize ground truth rather than metrics just for batch mixing, as those metrics may not reliably capture the ground-truth correspondences.

Questions for authors

How are hyperparameters tuned with OT and tSNE in these datasets? And how sensitive are the results to them?

3 comments

Dear Reviewer ukDm,

Thank you for the time and effort invested in reviewing our work!

W1a: t-SNE, as a visualization tool, [...] does not reliably represent the actual distances of real data.

First, we appreciate the concern that t-SNE can introduce distortions. Therefore, we added cautionary references in lines 102-104 to prevent future readers from interpreting distances and global relationships in OT-tSNE plots. Specifically, we state that "Such embeddings distort distances between observations and fail to capture global data geometry (Chari & Pachter, 2023)". Nevertheless, t-SNE is a very popular visualization tool in exploratory data analysis, especially of biological datasets (Kobak & Berens, 2019). Since biological data is often subject to batch effects, we believe that it is still important to adapt this popular tool for integrated visualization.

W1b: t-SNE, as a visualization tool, is not stable [..]. Furthermore, it can be manipulated with different seeds.

In our analysis, we used three random seeds for every method. The stability of OT-tSNE across seeds is reflected in the low standard deviations in the biological conservation metrics for OT-tSNE embeddings (see Figs. S10, S11).

Besides the stable metrics, we understand that your concern could be about variability in the global layouts in t-SNE outputs for different seeds. We now added Fig. S9 that depicts OT-tSNE visualizations across seeds for different initializations. There, we see that the global layout of the clusters was stable for deterministic initializations, i.e., spectral, EOT, and PCA. The global layout was changing from seed to seed only for random initialization. Following your and reviewer JxZS' comments, we have switched to the EOT initialization in the revision. This setting makes OT-tSNE fully deterministic, so that it cannot be manipulated by different random seeds.

We would appreciate it if you could let us know whether the updated submission clarifies the issues related to t-SNE.

W2a: The real-world use case of the proposed method is limited because the dataset needs to be balanced;

To ensure clarity, we restate here that in OT-tSNE, balanced refers to the marginal constraints on the transport plan. Batches may therefore differ in both size and cell-type composition. When a cell type is more abundant in one batch than in the other, some of its mass must be transported to other cell types. We now state this more explicitly in Sections 5.2 and 6.

Therefore, our method is best suited for batches with matching distributions, i.e the proportions of clusters should be roughly the same across the batches. However, please note that our benchmark includes seven real single-cell datasets with varying degrees of compositional imbalance (Figs. S1, S2), where OT-tSNE still achieved highly competitive results.

Across the datasets with shared cell types, OT-tSNE achieved very strong batch mixing and remains competitive in preserving biological structure. Mouse Atlas provides a direct example. Cell-type proportions differ substantially between its two batches (Fig. S1c), yet OT-tSNE has the best trade-off in batch mixing and biological preservation on this dataset compared to other methods (Fig. S11).

The Hematopoietic dataset illustrates a highly unbalanced case, where some cell types occur in only one batch (Fig. S1b). However, all methods we evaluated, not only OT-tSNE, struggle to achieve good batch mixing and biological preservation.

Thus, we have extended the Results and Conclusion sections to explicitly state that OT-tSNE is fairly robust even when datasets with shared classes have unbalanced distributions.

W2b: The real-world use case of the proposed method is limited because [...] only applied to 2 batches. Whereas in a real-world setting, one cannot ensure balanced datasets that are limited to 2 batches

Arguably, only two batches need to be integrated in many real-world situations, for instance, in the seven datasets in our benchmark. Nevertheless, we agree that being able to integrate more than two batches is desirable. We are currently working on extending OT-tSNE to jointly visualize more than two batches, following the outline given in Section 6. Please see our response to reviewer WXuz (Extension to multiple batches) for concrete details on how we extend OT-tSNE for multibatch integration.

We are currently evaluating this extension on two single-cell datasets. Preliminary experiments show promising results, but the rebuttal is too brief to run a comprehensive benchmark as for the two-batch case.

W3: Benchmarking on simulation needs to add a metric for 1-1 correspondence matching (like Fraction of Samples Closer than the True Match score) to utilize ground truth rather than metrics just for batch mixing, as those metrics may not reliably capture the ground-truth correspondences.

In our submitted MNIST experiment, we evaluated cross-batch alignment using ground-truth digit labels alongside the batch-mixing metrics (see the neighbor purity metric in Fig. 3b). For each image, we measured the fraction of its neighbors in the opposite batch that shared its digit label.

The suggested FOSCTTM metric requires a known one-to-one correspondence between individual images. Our two batches contain disjoint sets of MNIST images, so there are no ground-truth image pairs to rank. FOSCTTM is therefore undefined for this simulation.

We have revised Section 5.1 to make these experimental assumptions explicit.

Q1: How are hyperparameters tuned with OT and tSNE in these datasets? And how sensitive are the results to them?

For the t-SNE part of OT-tSNE, we use default openTSNE hyperparameters where possible (iteration steps, exaggeration, learning rate, etc.).

OT-tSNE itself has two main hyperparameters, both chosen based on defaults in the literature and not tuned by us. First, for the OT part, entropic optimal transport requires only one hyperparameter for the regularization strength rr: ϵ=r∗median(C)\epsilon = r*\text{median}(C). The main paper shows results for r=1r=1, following Landa et al. 2024. Second, our affinity construction introduces a hyperparameter kk for the number of chosen neighbors per sample. In our benchmark, k=15k=15, which we motivate as a default choice in UMAP and NCVis (McInnes et al., 2018, Artemenkov & Panov, 2020).

We now ablated kk and rr across 25 combinations of k∈{5,10,15,30,60}k\in\lbrace5,10,15,30,60\rbrace and r∈{0.1,0.5,1,5,10}r\in\lbrace0.1,0.5,1,5,10\rbrace (see new Figs. S5, S6). At fixed kk, lower rr generally favored batch mixing at the expense of biological preservation. At fixed rr, larger kk tended to favor biological preservation. The overall score varied little near the default (k,r)=(15,1)(k,r)=(15,1), with larger trade-offs at the grid extremes. Tuning kk and rr based on this sensitivity analysis would yield minor improvements. But we prefer to stick to our defaults to avoid double dipping and to ensure a fair comparison to competing methods, which we also ran with their default hyperparameters without tuning.

A similar concern was expressed by the reviewer JxZS for initialization. Our method’s performance for real datasets was not sensitive to the choice of initialization. Please see our answer for more details.

Reviewer_ukDmOpenReview ↗

Acknowleding authors' response

Thank you to the authors for thoroughly addressing the concerns. I will raise my score.

Reviewer_JxZSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OT-tSNE for jointly visualizing datasets that are affected by batch effects. The method replaces nearest-neighbor-based matching with globally coupled affinities derived from entropic optimal transport. It combines these cross-batch affinities with within-batch affinities obtained through a spectral construction. The resulting graph is sparsified, binarized and used to generate t-SNE embedding. The paper evaluates their method using synthetic controlled experiments and a benchmark of 23 method configurations across seven single-cell datasets. The method achieves competitive biological conservation with a strong batch mixing and obtains the best overall average rank in the reported benchmark.

Critical strengths
  1. Paper makes a useful methodological contribution by combining globally coupled cross-batch affinities from entropic optimal transport with a spectral construction for within-batch affinities.
  2. Evaluation provides controlled synthetic experiments and a benchmark across seven single-cell datasets.
  3. Reports combination of competitive biological conservation and strong batch mixing along with the best overall average rank.
Critical weaknesses
  1. Eqs. 8 & 9 indicate that the transportation plan remains unchanged with the defined orthogonal transformations if the regularization parameter is constant. In contrast, Section 3 defines the regularization parameter as the median cross-batch cost, which varies with the same orthogonal transformations.
  2. As indicated in Appendix C, OT-tSNE is performed using random initialization and 200 iterations, while the baselines use PCA initialization and 450 iterations. Comparison under matched settings would be better.
Questions for authors
  1. How regularization is set fixed across corruption strengths in Figure 3? And how does this link to the theory of invariance?
  2. How are the negative intermediate entries in the spectral construction treated?
  3. Comparison under matching initialization and optimization would highlight the effect of the proposed affinity construction. Please provide them.
1 comment

Dear reviewer JxZS,

Thank you for the time and effort invested in reviewing our submission and for your positive feedback!

W1, Q1a: How regularization is set fixed across corruption strengths in Figure 3?

We do not set a fixed regularization across the corruption strengths. You are right that the regularization ϵ\epsilon varies with the cross-batch cost, and therefore with the corruption strength λ\lambda, which we ablate in the synthetic experiment. In preliminary experiments, we held ϵ\epsilon fixed and observed similar results. We now revised Section 5.1 to improve the clarity of our experimental assumptions. Thank you for spotting this detail!

W1, Q1b: And how does this link to the theory of invariance?

The exact invariance of the OT plan to orthogonal batch effects holds for a fixed regularisation and fixed marginals. The OT plan can be different under different regularizations. We now include the new Fig. S3 in the Appendix B to show the synthetic experiment with a fixed regularization ϵ=median(C0)\epsilon = \text{median}(C_0), where C0C_0 is the cost at λ=0\lambda=0. We found that the sparsified transport plans in the original setting and the one with fixed regularization differ little both in overlap of the edge sets and in total variation. Thus, the adaptive regularization strength barely affects the results. In the adaptive setting, digit purity is affected only at the third decimal for increasing λ\lambda, while in the fixed setting, it remains constant, due to the constant plan.

Q2: How are the negative intermediate entries in the spectral construction treated?

Negative entries are explicitly set to zero. At the default regularization, they are very rare and extremely small: all datasets except nCoV do not have negative entries, while nCoV has fewer than 30, with the largest −2.1⋅10−8-2.1 \cdot 10^{-8}.

Q3: Comparison under matching initialization and optimization would highlight the effect of the proposed affinity construction. Please provide them.

We fully agree with you that comparison under matched settings would be better. First, we ablated different initializations for OT-tSNE. In particular, we 1) reimplemented the default openTSNE spectral initialization for our custom affinities, 2) added PCA initialization, 3) checked EOT initialization, based on the second and third eigenvectors of the SVD of the optimal transport plan. Compared to the random initialization baseline, the performance with other initialization modes changed only within seed noise (see new Appendix E.2). We conceptually prefer EOT initialization as default over PCA. Despite little quantitative performance difference, PCA can produce noticeable batch effects that the OT-tSNE optimization then needs to overcome (please note that the 50D --> t-SNE comparisons initialize with PCA on the already integrated 50D space).

Second, we set openTSNE defaults for the iteration schedule, such that all methods that rely on the t-SNE embedding step have 250 early-exaggeration and 500 post-exaggeration iterations (the methods that directly modify t-SNE for batch corrections, such as BC/RC/ref t-SNE, use their own defaults). We updated Figures 1, 4, 5, and Table 1 in the main paper and updated implementation details in Appendix C. Overall, in the matched settings, the OT-tSNE results even slightly improved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.