COMPASS: COvariance-Mapped Prototypes with Anisotropic Spherical Similarity

Case-based reasoning networks like ProtoPNet ground their predictions in visual similarities between image patches and learned concepts that can be visually audited by users, offering an alternative to post-hoc explanation methods. Recent advances represent prototypes as mixtures of Gaussian distributions to capture richer similarity structures, but the nature of these representations breaks the canonical projection onto image patches that makes prototypes interpretable. Separately, architectures using cosine similarity over Euclidean distance have demonstrated improved accuracy on fine-grained recognition, but their probabilistic relaxation remains restricted to isotropic von Mises-Fisher distributions. In this work, we propose a family of anisotropic spherical distributions – including the Kent, Anisotropic Spherical Gaussian (ASG), Angular Central Gaussian (ACG), and Symmetric Mahalanobis Cosine (SMC) distributions – as prototype similarity measures for cosine-based prototype networks, alongside principled volume regularizers that prevent geometric collapse during training. Across fine-grained image classification benchmarks, our anisotropic models consistently outperform isotropic baselines, improving accuracy by up to 4.7% on Stanford Dogs, while preserving the interpretability of prototype-based reasoning. Ultimately, our findings demonstrate that expressive covariance-aware metrics are crucial for closing the gap between interpretable-by-design and black-box models.