Mind Your Marginals: What Governs Fidelity in Linearized Hierarchical Optimal Transport?

Linearized optimal transport (LOT) replaces the n(n-1)/2 transport solves needed to compare n measures with n offline embeddings into a tangent space at a fixed reference. We study LOT for hierarchical transport, where the ground metric between support atoms is itself a Wasserstein distance over a learned, coarsened support, and instantiate this construction (HiLOT) in text, graphs and images from one shared implementation. Our central finding is that fidelity is governed less by modality than by four construction-level choices, the first of which has been overlooked. (i) Orientation: a transport plan between measures on a shared atom set admits two barycentric projections, indexed by the measure or by the reference, and only the latter lies in a tangent space shared across objects. Across 24 arms in two modalities the benefit of correcting the orientation is proportional to the linearization gap it repairs (Spearman ρ = -0.946, p = 3×10^-12; ρ = -0.947 and -0.806 within graphs and images), reducing test error by up to 0.235 on a single benchmark. (ii) Plan density: an exact plan is a polytope vertex with at most n+m-1 nonzeros, so the embedding degenerates into a quantizer; entropic regularization removes this with a provably interior optimum. Orientation and regularization together close the gap to exact hierarchical transport on curated retrieval (0.1123 against 0.1121 NDCG@10, t = -0.04) at  100 queries/s against 1.5. (iii) Coarsening acts as capacity control, with a truncation sweep isolating coarsening rather than support truncation as the mechanism behind its one clear failure case. (iv) The component model is exchangeable and matters about as much as coarsening, an order of magnitude less than orientation. Across six text benchmarks the linearization preserves accuracy relative to exact hierarchical transport at 12 to 182 times its throughput.