- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work extends prototypical part networks like ProtoPNet from Chen et al. to enable probabilistic prototypes in a hyperspherical space with anisotropic covariances. This means, that the prototypes are based on a cosine similarity function, but also model probability distributions with non isotropic covariances, in contrast to existing formulations like the von Mises-Fisher distribution.
The work defines four desired properties, which existing methods do not fulfill, and propose four alternative formulations with different variance definitions. These are trained with common cholesky decomposition for numerical stability, and a new KL-Divergence based regularization loss.
All for formulations are evaluated on three datasets with ResNet50 backbones on single experimental seeds with reported gains of up to 1.9% to prior vMF formulations.
Critical strengths
The work is clearly written and easy to understand. The theoretical foundation and desired properties for the proposed methods are well defined. The research direction of using probabilistic prototype formulations for case based reasoning remains an important field of research.
Critical weaknesses
- The experimental evaluation and procedure seem insufficient to me:
- Unclear procedure: In my understanding, TesNet not only replaces ProtoPNet's similarity measure with a cosine similarity, but also introduces additional loss objectives for shaping the representation. Thus, it is unclear, which training regime the evaluation follows, and how it compares to the "original" ProtoPNet / TesNet formulations.
- Only one experimental seed, so no statistical power / confidence intervals are reported, making comparison difficult. This is even more important, as several proposed formulations show training collapse on Stanford Cars. Highlighting the best gains only further skews the perceived results Other tested CNN backbones are only mentioned in the Appendix, with even smaller best gains. In my personal opinion, I would prefer to see a transformer based backbone than continue testing different CNN backbones in different sizes.
-
Missing qualitative examples, demonstrating that all proposed formulations still cover human-readable concepts. I also recommend making Fig 1. more abstract. Together with the caption this reads like a demonstrated fact. Whereas from my reading this is an illustrative example for motivating this work, which is not further verified in experiments. Rather, it is left as future work.
-
**Limited insights. ** The work proposes four formulations with 3 variants each, so 12 total. But for me as a reader it is left open which one might be most advantageous to use. I did not find any verification, that the proposed methods actually produce anisotropic prototypes, or if the Clustering and Separation losses from PPN together enforce a (near) isotropic formulation anyway.
Questions for authors
- How do the methods perform with 3 random seeds, especially compared to isotropic vMF baseline? Up to 1.9% gains can quickly vanish over more seeds.
- Do the proposed formulations actually produce anisotropic prototypes? Maybe an ablation over different values of with scatter plots of prototypes and their KL divergence to an isotropic formulation could provide additional insights.
4 comments
Random Seeds
I thank the authors for the response. Especially the newly added illustrative figures and evaluations are highly appreciated and strengthen the claims.
However, I might be misunderstanding hyperband. My understanding is, that it is an alternative to gridsearch for finding best-performing hyperparameters. However, it does not test or demonstrate stability of the method / settings across parameter initialization, training data order or other numerical effects.
Stanford Cars - Gated - isotropic often collapses to single digit accuracy. But there should be nothing special about this configuration, which raises the concern that the proposed methods are all inherently unstable and most other results got a lucky seed.
Therefore I don't feel confident in increasing my score without seeing Tab 2 with at least 3 random seeds and reported mean and sd for each cell.
Robust evaluation
We thank the reviewer for reading through the improved version, and we appreciate that the extra efforts that went into providing clearer illustrations are approved.
Table 2 is now fully completed with three seeds for each cell, with reported mean and std. The Best column has been replaced with Avg for more appropriate reading. We further thank the reviewer for pointing out the special case with gated Stanford Cars: it happens to be difficult training both baseline and anisotropic models on this setting, with a ResNet backbone. We thus remove this column when computing average gains. We hope that the reviewer will appreciate Appendix E which motivates further how and why we decide to fairly exclude this column.
Concerning Hyperband selection, it works as follows in our hyperparameter search: 16 random configurations are generated (no duplicates) and run for 25% of the training, then only half of them are kept for training until 50%, etc. We run this procedure twice, one for the baselines, and one for the anisotropic models' additional parameters. This hyperparameter search is done in favor of the baselines (since the best backbone learning rate for the baseline is not necessarily the best learning rate for a COMPASS model).
We sincerely thank the reviewer for helping improve the paper.
I thank the authors for the new results and clarification. I believe they strengthen the claims and make the reading of performance gains more reliable. The openly communicated handling of Stanford Cars with a reference to Appendix E in the main text is highly appreciated.
For a camera-ready version I recommend trying to further improve the presentation of the newly added experiments, and especially Table 2.
But for now I am happy to increase my rating and recommend acceptance.
Response to CLer
We thank you for your careful suggestions and for taking the time to dig into our training recipes.
W1.a. Your understanding of TesNet is absolutely correct, and we do indeed run TesNet (and all anisotropic variants) models with two extra diversity losses that are introduced in the TesNet paper, and we reuse the loss coefficients originally used by the authors. This information is now fully provided in Appendix A.1. Furthermore, we argue that the lack of extra diversity loss terms should not be a problem for ProtoPNet, since we evaluate with/without anisotropy on equal terms (i.e. all ProtoPNet models do not have these extra coefficients, so any gain cannot come from adding/removing these terms). Keeping the diversity loss terms for TesNet and not ProtoPNet is for fair comparison with prior work found in the literature.
Q1.W1.b. We (currently) have only one seed for our Table 2 (and other comparable tables throughout Appendix A) but we have ongoing experiments to reproduce the results with different seeds, and we would like to reiterate that we used Hyperband to select hyperparameters: the results do carry statistical evidence from this selection.
W2. We hope that our additional work on illustrative examples sufficiently answers your justified call for more qualitative examples of how human-readable the concepts are, and what they correspond to in practice. Furthermore, akin to our answer to reviewer xbuH, we argue that Figure 1 was indeed for motivation, and does not represent what we finalized, or can currently visualize.
Q2.W3. We agree that it is not clear from our experimental analysis which anisotropic distribution is the absolute best in all situations, though we argue in Section 5.2 that diagonal anisotropy generally has the best trade-off in terms of compute and accuracy. Furthermore, we show in Table 8 that the proposed methods do indeed produce anisotropic prototypes, with a high ratio between the maximum eigenvalue and the minimum eigenvalue of the precision matrices. We further show in Table in Table 12 how anisotropy and KL to Identity evolve with Kl regularization for Kent models trained on the RIVAL-10 dataset.
We hope that your concerns have been addressed, and we stay open to more discussions on our proposed method.