Optimal t-kernel shape in t-SNE

Neighbor-embedding methods have become an established tool for visualizing complex high-dimensional datasets in various scientific disciplines. The celebrated $t$-SNE algorithm uses the Cauchy kernel to define embedding similarity between points, often strongly outperforming the original SNE that uses the Gaussian kernel. However, a theoretical justification and an empirical investigation of the optimal kernel shape have been lacking. Here we address this gap by studying the optimal kernel shape ($t$-distribution degree of freedom) in $t$-SNE. We use a simplified uniform ansatz to derive the optimal kernel shape and show that the tail thickness increases with intrinsic dimensionality. Implementing a fast kernel-shape optimization in both the Barnes–Hut and the FFT-based approximations of $t$-SNE, we qualitatively confirm theoretical predictions on simulated datasets. Finally, we conduct a large-scale empirical investigation of the learned kernel shapes across various datasets, and show that optimizing the kernel can reduce t-SNE's tendency to over-fragment unstructured, low-dimensional data and can lead to more accurate visualizations of single-cell RNA-seq data.