acceptodds
Desk rejected at ICLR 2027

Wasserstein Singular Vectors and Metric Dowker Duality for Interpretable Feature Geometry

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

The Wasserstein singular vector (WSV) problem asks for a metric on the samples and a metric on the features of a data matrix such that each reproduces the other through optimal transport. We prove existence for the general, unregularized WSV problem, for almost every matrix, and match degenerate solutions to lumpings of the matrix. While they determine each other, the WSV sample and feature metrics do not generally agree. We show that their Gromov–Hausdorff and Gromov–Wasserstein distance is bounded by the spread of the features over the samples. When this spread is small, every feature has a location among the samples and the features together form a metric space that mirrors the sample geometry. This is a global and label-free notion of an interpretable feature geometry. All our mathematical results are machine-verified in Lean 4. Spread is also a quantity a model can be trained on. We regularize WSV spread in autoencoders on geometric toy data and on Gemma-2-2b residual-stream activations along a prompt manifold. As the theory predicts, this allows us to recover the sample geometry on the feature side. We show that interpretable feature geometry induced by regularization can also be explained by low WSV spread.

Primary area:
interpretability and explainable AI
Keywords:
Wasserstein singular vectors, metric Dowker duality, optimal transport, ground metric learning, interpretable feature geometry, mechanistic interpretability, sparse autoencoders, Gromov–Wasserstein distance

Related papers

Discussion (0)

Sign in to comment.