Recovering Rank Axes in Embedding Spaces via Differentiable Ranking

Recent studies have shown that visual embeddings contain latent directions along which projections preserve ordinal attributes such as age, crowd count, and aesthetic quality. These *rank axes* provide an interpretable way to uncover ordinal structure in pretrained representations, yet existing estimation methods do not directly target ordinal agreement. In this paper, we study rank axis estimation in a fixed embedding space through differentiable ranking and ask when minimizing a rank-based objective recovers the underlying ordinal direction. We consider two estimators based on quadratically regularized ranking over the permutahedron, using a Spearman-type squared rank loss and a Fenchel–Young loss. We establish consistency of the Spearman-based estimator under Gaussian noise and of the Fenchel–Young estimator in the noise-free setting. Notably, their behavior under noise differs, showing that closely related differentiable ranking objectives can have different recovery properties. We also establish convergence to first-order stationarity for both optimization procedures. Experiments on synthetic and real-world image datasets demonstrate accurate rank-axis recovery and improved overall and top-$k$ ranking performance over linear regression and learning-to-rank baselines. Our results show that differentiable ranking provides an effective framework for recovering ordinal structure in pretrained embeddings, while highlighting the statistical consequences of the choice of ranking objective.