acceptodds
Under review as a conference paper at ICLR 2027

COMPASS: COvariance-Mapped Prototypes with Anisotropic Spherical Similarity

16 views

Abstract

Case-based reasoning networks like ProtoPNet ground their predictions in visual similarities between image patches and learned concepts that can be visually audited by users, offering an alternative to post-hoc explanation methods. Recent advances represent prototypes as mixtures of Gaussian distributions to capture richer similarity structures, but the nature of these representations breaks the canonical projection onto image patches that makes prototypes interpretable. Separately, architectures using cosine similarity over Euclidean distance have demonstrated improved accuracy on fine-grained recognition, but their probabilistic relaxation remains restricted to isotropic von Mises-Fisher distributions. In this work, we propose a family of anisotropic spherical distributions – including the Kent, Anisotropic Spherical Gaussian (ASG), Angular Central Gaussian (ACG), and Symmetric Mahalanobis Cosine (SMC) distributions – as prototype similarity measures for cosine-based prototype networks, alongside principled volume regularizers that prevent geometric collapse during training. Across fine-grained image classification benchmarks, our anisotropic models consistently outperform isotropic baselines, improving accuracy by up to 4.7% on Stanford Dogs, while preserving the interpretability of prototype-based reasoning. Ultimately, our findings demonstrate that expressive covariance-aware metrics are crucial for closing the gap between interpretable-by-design and black-box models.

Primary area:
interpretability and explainable AI
Keywords:
explainability, interpretability, geometry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CLerOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work extends prototypical part networks like ProtoPNet from Chen et al. to enable probabilistic prototypes in a hyperspherical space with anisotropic covariances. This means, that the prototypes are based on a cosine similarity function, but also model probability distributions with non isotropic covariances, in contrast to existing formulations like the von Mises-Fisher distribution.

The work defines four desired properties, which existing methods do not fulfill, and propose four alternative formulations with different variance definitions. These are trained with common cholesky decomposition for numerical stability, and a new KL-Divergence based regularization loss.

All for formulations are evaluated on three datasets with ResNet50 backbones on single experimental seeds with reported gains of up to 1.9% to prior vMF formulations.

Critical strengths

The work is clearly written and easy to understand. The theoretical foundation and desired properties for the proposed methods are well defined. The research direction of using probabilistic prototype formulations for case based reasoning remains an important field of research.

Critical weaknesses
  1. The experimental evaluation and procedure seem insufficient to me:
  • Unclear procedure: In my understanding, TesNet not only replaces ProtoPNet's similarity measure with a cosine similarity, but also introduces additional loss objectives for shaping the representation. Thus, it is unclear, which training regime the evaluation follows, and how it compares to the "original" ProtoPNet / TesNet formulations.
  • Only one experimental seed, so no statistical power / confidence intervals are reported, making comparison difficult. This is even more important, as several proposed formulations show training collapse on Stanford Cars. Highlighting the best gains only further skews the perceived results Other tested CNN backbones are only mentioned in the Appendix, with even smaller best gains. In my personal opinion, I would prefer to see a transformer based backbone than continue testing different CNN backbones in different sizes.
  1. Missing qualitative examples, demonstrating that all proposed formulations still cover human-readable concepts. I also recommend making Fig 1. more abstract. Together with the caption this reads like a demonstrated fact. Whereas from my reading this is an illustrative example for motivating this work, which is not further verified in experiments. Rather, it is left as future work.

  2. **Limited insights. ** The work proposes four formulations with 3 variants each, so 12 total. But for me as a reader it is left open which one might be most advantageous to use. I did not find any verification, that the proposed methods actually produce anisotropic prototypes, or if the Clustering and Separation losses from PPN together enforce a (near) isotropic formulation anyway.

Questions for authors
  • How do the methods perform with 3 random seeds, especially compared to isotropic vMF baseline? Up to 1.9% gains can quickly vanish over more seeds.
  • Do the proposed formulations actually produce anisotropic prototypes? Maybe an ablation over different values of λKL\lambda_{KL} with scatter plots of prototypes and their KL divergence to an isotropic formulation could provide additional insights.
4 comments
AuthorsOpenReview ↗

Response to CLer

We thank you for your careful suggestions and for taking the time to dig into our training recipes.

W1.a. Your understanding of TesNet is absolutely correct, and we do indeed run TesNet (and all anisotropic variants) models with two extra diversity losses that are introduced in the TesNet paper, and we reuse the loss coefficients originally used by the authors. This information is now fully provided in Appendix A.1. Furthermore, we argue that the lack of extra diversity loss terms should not be a problem for ProtoPNet, since we evaluate with/without anisotropy on equal terms (i.e. all ProtoPNet models do not have these extra coefficients, so any gain cannot come from adding/removing these terms). Keeping the diversity loss terms for TesNet and not ProtoPNet is for fair comparison with prior work found in the literature.

Q1.W1.b. We (currently) have only one seed for our Table 2 (and other comparable tables throughout Appendix A) but we have ongoing experiments to reproduce the results with different seeds, and we would like to reiterate that we used Hyperband to select hyperparameters: the results do carry statistical evidence from this selection.

W2. We hope that our additional work on illustrative examples sufficiently answers your justified call for more qualitative examples of how human-readable the concepts are, and what they correspond to in practice. Furthermore, akin to our answer to reviewer xbuH, we argue that Figure 1 was indeed for motivation, and does not represent what we finalized, or can currently visualize.

Q2.W3. We agree that it is not clear from our experimental analysis which anisotropic distribution is the absolute best in all situations, though we argue in Section 5.2 that diagonal anisotropy generally has the best trade-off in terms of compute and accuracy. Furthermore, we show in Table 8 that the proposed methods do indeed produce anisotropic prototypes, with a high ratio between the maximum eigenvalue and the minimum eigenvalue of the precision matrices. We further show in Table in Table 12 how anisotropy and KL to Identity evolve with Kl regularization for Kent models trained on the RIVAL-10 dataset.

We hope that your concerns have been addressed, and we stay open to more discussions on our proposed method.

Reviewer_CLerOpenReview ↗

Random Seeds

I thank the authors for the response. Especially the newly added illustrative figures and evaluations are highly appreciated and strengthen the claims.

However, I might be misunderstanding hyperband. My understanding is, that it is an alternative to gridsearch for finding best-performing hyperparameters. However, it does not test or demonstrate stability of the method / settings across parameter initialization, training data order or other numerical effects.

Stanford Cars - Gated - isotropic often collapses to single digit accuracy. But there should be nothing special about this configuration, which raises the concern that the proposed methods are all inherently unstable and most other results got a lucky seed.

Therefore I don't feel confident in increasing my score without seeing Tab 2 with at least 3 random seeds and reported mean and sd for each cell.

AuthorsOpenReview ↗

Robust evaluation

We thank the reviewer for reading through the improved version, and we appreciate that the extra efforts that went into providing clearer illustrations are approved.

Table 2 is now fully completed with three seeds for each cell, with reported mean and std. The Best Δ\Delta column has been replaced with Avg Δ\Delta for more appropriate reading. We further thank the reviewer for pointing out the special case with gated Stanford Cars: it happens to be difficult training both baseline and anisotropic models on this setting, with a ResNet backbone. We thus remove this column when computing average gains. We hope that the reviewer will appreciate Appendix E which motivates further how and why we decide to fairly exclude this column.

Concerning Hyperband selection, it works as follows in our hyperparameter search: 16 random configurations are generated (no duplicates) and run for 25% of the training, then only half of them are kept for training until 50%, etc. We run this procedure twice, one for the baselines, and one for the anisotropic models' additional parameters. This hyperparameter search is done in favor of the baselines (since the best backbone learning rate for the baseline is not necessarily the best learning rate for a COMPASS model).

We sincerely thank the reviewer for helping improve the paper.

I thank the authors for the new results and clarification. I believe they strengthen the claims and make the reading of performance gains more reliable. The openly communicated handling of Stanford Cars with a reference to Appendix E in the main text is highly appreciated.

For a camera-ready version I recommend trying to further improve the presentation of the newly added experiments, and especially Table 2.

But for now I am happy to increase my rating and recommend acceptance.

Reviewer_FWugOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

COMPASS introduces four spherical prototype similarities that learn anisotropic geometry while keeping similarity maxima anchored to projectable training examples, using tangent-space precision matrices and Gaussian KL regularization. Experiments on three fine-grained classification datasets and three principal backbones report gains up to 4.7 percentage points over TesNet, although interpretability is established structurally rather than evaluated empirically.

Critical strengths
  1. Restricting the precision matrix to the prototype’s tangent space provides a clear mechanism for learning directional sensitivity without displacing the similarity maximum.
  2. The experiments distinguish learned concentration from anisotropy through isotropic, diagonal, and full-matrix variants, alongside an isotropic vMF baseline. Inheriting the baseline architecture settings helps isolate the effect of changing the similarity, and the parameter-count analysis provides useful evidence that full covariance often offers little benefit over diagonal precision.
Critical weaknesses
  1. Appendix A.2 reports that seven of nine ResNet-50 baseline settings use 1×1 pooling, making their prototypes whole-image representations despite the paper’s emphasis on localized visual parts. Evaluation with retained spatial grids or actual prototype visualizations is needed to establish that the proposed geometry preserves useful visual explanations beyond exact latent anchoring.
  2. The claim of closing the gap to black-box models lacks matched black-box comparisons, while the probabilistic prototype methods motivating COMPASS, such as MGProto, are not evaluated directly.
Questions for authors
  1. Why is Kent’s κ\kappa fixed to 1 when Table 1 states that cosine-ranking reduction requires κ\kappa≥2?
  2. What exact criterion selects the training example during projection: the learned anisotropic similarity or the original architecture’s distance? Please also clarify whether all reported accuracies are measured after the final projection and subsequent classifier fine-tuning.
2 comments
AuthorsOpenReview ↗

Response to FWug

We thank reviewer FWug for paying close attention to key details in our proposed anisotropic similarities.

W1. Indeed, Appendix A.2 faithfully reports that claim. However, a 1×11\times 1 grid does not make prototypes non-local: the similarity is computed on one vector per image, but PRP attributions show it depends on localized regions (Figures 4–8). Spatial grids do not guarantee locality either, since receptive fields span most of the image (Xu-Darme et al., 2023: Wolf et al., 2024). We have clarified Appendix A.2 accordingly.

W2. We do not include MGProto in our evaluation specifically because it lacks proper projection rules (as detailed in Section 3.1). Furthermore, because our central argument is that using anisotropic similarity is better than isotropic similarity, our evaluation focuses on this aspect (rather than white-box vs black-box). We apologize for the overclaim and the revised version reflects this adjustment.

Q1. We thank the reviewer for pointing this out, we initially set κ=1\kappa=1 to avoid parameter redundancy and training instability, and only later recognized that it fails D4. As such, experiments in Table 2, Table 8, and Table 12 use models with κ=2\kappa=2 to correct this. (do note that other tables like our results on DenseNet-161 and ConvNeXt-S are still using κ=1\kappa=1)

Q2. All projection steps are done using the learned similarity metric to choose which patch is the best to push onto. All reported accuracies are done on the test set and measured after the final projection step and subsequent fine-tuning steps. Section 5.1 is updated for more transparency on this subject.

We are open to more discussions on our work and we once again thank you for your time.

Thank you for the thoughtful response. I appreciate the authors’ discussion on the remaining issues of their work. The clarifications are helpful, and I will keep my positive rating.

Reviewer_xbuHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes a family of anisotropic spherical similarity functions (including SMC, ACG, ASG, and Kent) for prototype-based models to address the inability of Gaussian-mixture prototypes to project onto a single image patch and of anisotropic vMF to preserve a maximum at the prototype itself. The central idea is to apply learned precision matrices to directions orthogonal to each prototype. Evaluations show accuracy gains in several settings across a few backbones and different datasets.

Critical strengths
  • Sound theoretical framework and analysis: The mathematical analysis of peak displacement in a naive anisotropic vMF extension and the structural critique of Gaussian-mixture projections provide a useful foundation for introducing the method. The proposed method provides a principled way to increase geometric expressivity and retain exact projection of the prototype center onto a training example.
  • Broad and transparent empirical evaluation: Evaluation includes learnable isotropic controls, diagonal and general precision matrices, gating variants, and multiple backbones, which helps distinguish the effect of anisotropy from that of a learnable concentration. Reporting is transparent (including divergent runs) which makes the findings and comparisons checkable.
Critical weaknesses
  • Lack of interpretability analysis: The main weakness of this paper for me is that it does not establish that the resulting similarity corresponds to understandable visual features. Even Figure 1 is conceptual rather than evidence about learned invariances. An evaluation of explanation quality from human assessment and VLM scoring proxies from prototype evidence, and qualitative analysis of the learned precision axes are necessary from my point of view. Assessing with newer prototype-based architectures such as ProtoViT, in addition to ProtoPNet is also needed. The experiments now do not establish the central promise of improved accuracy while preserving meaningful localized prototype explanations.
  • Training Instability: There are many failed runs or divergent ones, although KL regularization is proposed to prevent collapse (some of them still with KL regularization active). Many failures occur in the gated variants, including configurations selected through hyperparameter search, which suggests that the method can be sensitive to hyperparameter choices and gating in practice. This needs more analysis in my opinion.
Questions for authors

Most questions and suggestions come from the above weaknesses, including qualitative visualization of a learned prototype's principal axes, analysis of conditions for the failed/divergent runs and sensitivity to the KL regularizer, and human-in-the-loop (or VLM Proxy) evaluations of the learned anisotropic covariance axes.

3 comments
AuthorsOpenReview ↗

Response to xbuH

We thank reviewer xbuH for their helpful comments and feedback. Here are our answers to weaknesses and questions raised.

W1. We provide additional interpretability analysis in Section 5.3 and Appendix C to further strengthen the case that using anisotropic prototypes is useful both in terms of (i) accuracy and (ii) interpretability:

  • Table 2 (in Section 5.2) shows that they are more accurate on traditional fine-grained image classification datasets.
  • Figures 4 and 5 illustrate that we can gain understanding of what is important for a prototype (and what is not).

Figure 4 shows what happens for prototypes after perturbations of their corresponding training sample, both in terms of angular distance and anisotropic similarity. Figure 5 starts from a source image, showing how other (test) images that are equally distant (in terms of angles) have different similarity scores (but same cosine similarity) and what it means for the prototypes' attribution

W2. We fixed instability issues which rose from floating point errors (concerning ASG, Section 4.2), and observe that gating does indeed lead to poor learning dynamics in some cases. We respectfully argue that this is a finding of the paper rather than a limitation.

Limitations. We'd like to particularly address the remaining weaknesses raised.

  • Comparing to ProtoViT is not feasible because the architecture uses a different way of computing similarity, using subprototypes, which is a behaviour we identify as uninterpretable in our paper (see Section 3.1)
  • We are aware of recent works using VLM proxies as judges in complementary work like [1], it is a promising avenue of automated evaluation, but we argue that their methodology is not mature enough right now.
  • Showing something like Figure 1 but with a real model would mean being able to generate an image (or image patch) from a latent representation, something which is not reliable enough and heavily depends on the generator used, although we agree that this would represent the natural next step of this work. Furthermore, we tried to convey that Figure 1 was purely conceptual and motivational, and does not reflect what we can do (but instead, what motivated us to push this line of work forward).

We hope we cleared any confusion and we are open to more discussions.

[1] "ProtoTTA: Prototype-Guided Test-Time Adaptation" (Abootorabi et al., 2026)

Thank you for adding the new illustrative figures. They make the effect of anisotropic similarity much clearer and more intuitive.

The additions strengthen the paper and address some of my concerns. However, I believe my current score reflects the contributions and remaining limitations of the work, so I will maintain it. I raised the "Presentation" score since it is now easier to follow.

AuthorsOpenReview ↗

Further explanation

For reviewer xbuH:

We thank the reviewer for the follow-up and for raising the presentation score; we are glad that the work that went into the illustrations is appreciated.

We however felt that we did not show to what extent we tried to answer each of the points brought up, and that we perhaps insisted too much on the limitations (which have changed since the beginning of the rebuttal period); we now highlight the overall improvements that came after our first response.

Concerning ProtoViT, we further considered how we could incorporate it into our study. We initially answered that using sub-prototypes is one of the ways prototype-based networks lose interpretability, which we still claim; COMPASS is nonetheless architecturally compatible with it. ProtoViT compares tokens to sub-prototypes with cosine similarity, so any of our anisotropic spherical similarities could be substituted, with a precision matrix learned per prototype or per sub-prototype. We compare against the cosine baseline because Moffett et al. [1] show that a cosine ProtoPNet matches or outperforms more complex architectures. Section 2 of the paper is slightly modified to clarify our positioning.

Concerning prototype evaluation, during the rebuttal period, we added (in Appendix D) a quantitative prototype evaluation on the metric Part Consistency [2] which is an established metric for prototype quality. It however relies on part annotations, which are present in CUB-200. We invite the reviewer to take a look at Table 15, where we show that part consistency improves for gated anisotropic models when the attribution method is SmoothGrad, and stays close to the baselines when the attribution method is PRP. This directly addresses the concerns around the lack of interpretability evaluations raised by the reviewer.

Concerning VLM-based evaluation, we thank the reviewer for suggesting the use of VLMs as judges of prototype explanations. The idea is quite exciting, but it is itself an open research problem (concurrent work [3] uses VLM ratings of prototype activations as one diagnostic for test-time adaptation), rather than a standardized benchmark tool. We therefore preferred to rely on reproducible metrics --- Part Consistency [2] and class AUPRC (Table 15) --- complemented by qualitative visualizations (Figures 4–5 and 7–9).

We'd like to thank the reviewer for pointing out the training instability issues encountered, and we hope that our rebuttal work answers the reviewer's concerns:

  • Three seeds for every model of Table 2 and the analyses of these models (Tables 8, 14 and 15), and six for the failure analysis (Table 16).
  • A KL regularization weight sweep (Appendix A.5, Table 12): λKL\lambda_{KL} controls the degree of anisotropy, and accuracy drops only when the precisions are pulled near the identity.
  • A fix for ASG’s numerical failure: in bf16, the gradient of was arccos infinite, producing NaNs. ASG is now computed in fp32 in every table.
  • An analysis of the training instability (Appendix E). The TesNet baseline itself fails to learn in 3 of 6 seeds on gated Stanford Cars, and the median configuration of its search reaches 5.5% validation accuracy, against 64.6–81.2% elsewhere (Table 16, Figure 10). Across Tables 2, 9 and 10, 22 of the 23 COMPASS cells with a failed run use the hard gate with ACG, ASG or Kent, which are positive everywhere, so the gate cuts them off discontinuously at the equator; the exception is Kent with κ = 2 on ConvNeXt-S, which reuses hyperparameters tuned for κ = 1. SMC, whose gate is a continuous ReLU, fails in 1 of 6 seeds with isotropic and diagonal precision (Table 16). A continuous gate is not enough on its own: with the configuration tuned for the hard gate, it removes every ASG failure on gated Stanford Cars, but only one of five ACG failures, and Kent fails more often with it (Table 17).

We hope that these issues are clearer and we sincerely thank the reviewer for engaging with our work, and we would be delighted to further exchange with you on how the currently revised version meets your expectations and how it could be improved.

[1] Moffett, Luke, et al. "Cosine Similarity is Almost All You Need (for Prototypical-Part Models)." 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2026.

[2] Huang, Qihan, et al. "Evaluation and improvement of interpretability for self-explainable part-prototype networks." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[3] Abootorabi, Mohammad Mahdi, et al. "ProtoTTA: Prototype-Guided Test-Time Adaptation." arXiv preprint arXiv:2604.15494 (2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment for Reviewers

COMPASS Rebuttal

We thank the reviewers for their thoughtful remarks and constructive feedback. Over the past few days, we have done our best to address the weaknesses identified by the reviewers and accurately answer their questions. To put it briefly, the revised version is improved over two axes, illustrative and qualitative analysis, and more robust experiments. Changes include the following:

  • Kent models relaunched with κ=2\kappa=2 so that desideratum 4 is met (see Table 1 of Section 3.2)
  • ASG models relaunched with floating point errors reduced, using fp32 instead of bf16. (Section 4.2)
  • Training the models of Table 2 with two extra random seeds to have more representative results (still ongoing).
  • Illustrating the usefulness of anisotropic models in two distinct ways: (i) Figure 4 showing what happens for prototypes after perturbations of their corresponding training sample, both in terms of angular distance and anisotropic similarity, (ii) Figure 5 starting from a source image, showing how other (test) images that are equally distant (in terms of angles) have different similarity scores (but same cosine similarity) and what it means for the prototypes' attribution
  • Showing the effect of regularization term λKL\lambda_{\mathrm{KL}} on the model's anisotropy, measured by the (distribution of the) condition number λmax⁡/λmin⁡\lambda_{\max}/\lambda_{\min} of its prototypes. (see Table 12 of Appendix A.5)
  • Showing what happens when replacing a hard gate with a smooth gate, like in the original paper that introduces ASG: 1(zTμj≥0)⟶max⁡(zTμj,0)\mathbf{1}(z^T\mu_j \geq 0) \longrightarrow \max(z^T\mu_j, 0) (see Table 13 Appendix A.6)

We appreciate the reviewers for taking the time to see what has changed and answer them further individually.

Edited text is in dark red\textcolor{red}{\text{dark red}} for better legibility.

* some numbers in Table 2 have changed due to a more fair method of selection: the best models are selected on validation accuracy, but test accuracy is reported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.