- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method called LAVA to give model agnostic, prototype-based explanations for latent embeddings generated by unsupervised learning methods. The authors tested their method on MNIST, KPMP and BOW datasets. Their evaluations contain the stability of LAVA explanations, showing LAVA properties such as "revealing globally reoccurring local patterns", a qualitative comparison with LFXAI, and a case study relating a KPMP module to chronic kidney disease.
Critical strengths
This paper proposes a novel method for generating explanations for embeddings produced by unsupervised learning methods, which applies to non-parametric embeddings that existing methods cannot explain. The authors show a practical use case on scientific data, e.g., KPMP single-cell kidney gene expression, where a module is related to chronic kidney disease. The method design and the experimental designs are careful.
Critical weaknesses
-
The current comparison lacks baselines. The authors compare to NMF in Appendix only, and to LFXAI in qualitative comparisons. No other explanation method is benchmarked with LAVA. Moreover, the quantitative evaluation is mainly about stability and internal consistency, which is not strong enough to show that LAVA gives a "good" explanation.
-
The paper does not show that the explanations depend on the embedding being explained. The authors state that LAVA "detects feature covariation irrespective of feature usage by the model". But explanations should also give users the opportunity to recognize a "bad" learned embedding space, which LAVA cannot do if its output is embedding-independent. Or the authors should frame that LAVA is a "data-centric" explanation.
Questions for authors
-
Is there more related work on similar methods beyond those mentioned in the Introduction? Benchmarking against these baselines is needed in the experimental results.
-
Is it possible to evaluate the correctness of LAVA explanations with "ground-truth" explanations? The authors should consider datasets that contain human labels on feature attributions, or synthetic data with correlation patterns.
-
Is it possible to conduct human evaluations and show practical uses, such as scientific discovery of "new concepts" or debugging a 'bad' embedding space with LAVA?
3 comments
Thank you for your reply.
I thank the authors for their reply and additional experiments. However, the current reply does not address all my concerns. Regarding the baselines, the authors argue that the current feature-attributions methods are not comparable, but they do not include other options. Moreover, the human evaluation would be an interesting experiment which is not discussed either. Moreover, it would be great if the revised text in the pdf are highlighted in another color for a better view.
Addressing remaining comments
We thank the reviewer for their reply and apologize for not fully answering their concerns related to baseline comparisons and human evaluation more directly. We do so below.
Comparisons
We argue that LAVA addresses a fundamentally different question than existing explainability methods (Sec. 1), from “why” an embedding is generated, to what patterns in the data are captured by the latent structure. To quote: "explaining black-box embeddings requires new strategies for revealing what patterns in the data are captured by a learned multivariate latent structure". Existing methods for unsupervised embeddings yield “single-sample or dataset-summary explanations that are generally too fine-grained or reductive to describe the latent structure”. This holds for XAI methods in general, including feature attribution and example-centric XAI.
The methods we added to the revised paper (FADEx, ClusterShapley, Corbugy et al.) produce single-sample, cluster-level, or dataset-level feature attributions. To our knowledge, no existing method extracts locally reoccurring (correlation) patterns across embeddings, making direct comparison difficult. Repurposing methods towards this goal would require new methodology. Given these constraints, we compare LAVA at different levels for which alternatives exist:
- XAI: LAVA vs. LFXAI (Sec. 3.4): for the same MNIST embedding, LFXAI aggregates feature importances vs. LAVA reveals reoccurring patterns, offering complementary views.
- XAI: LAVA vs. neighborhood retention XAI (Sec. 3.3): a new embedding-awareness experiment compares LAVA explanations between original data and 3 embeddings, showing that LAVA adds a new view of covariance structure preservation beyond existing neighborhood retention XAI (Han et al., 2022).
- Module extraction: AMF vs. NMF (Sec 3.2, App. B.11): replacing AMF with NMF, a standard factorization, leads to 50% overestimation error ratio and denser components, representing average correlations vs. LAVA’s desired reoccurring patterns.
- Ground truth (Sec. 3.1): a new experiment evaluates LAVA on 5 distinct synthetic embeddings with ground-truth, including cases of non-uniqueness.
Beyond feature attribution, other XAI also answers different questions than LAVA. For instance, example importance identifies individual examples that influence an embedding, whereas LAVA focuses on local neighborhoods (sets of examples) and describing patterns that reoccur across embedding neighborhoods. Similarly, prototype XAI selects representative samples (e.g., of a class or cluster) for parts of an embedding space, unlike LAVA modules which reoccur across the embeddings. Furthermore, concept-based methods such as TCAV (Kim et al., 2018) require predefined concepts, whereas LAVA discovers patterns without prior knowledge. Finally, counterfactuals and anchors are defined with respect to a predicted target, which unsupervised methods lack (Introduction, lines 46-49).
If the reviewer has a specific method in mind, we would be happy to incorporate further discussion on how it relates to LAVA.
Human evaluation
We agree with the reviewer that a human evaluation would be interesting. Unfortunately, a user study was not feasible within the revision period. Instead, we aimed to show the utility of LAVA through an extensive range of applications across domains and settings:
- LAVA on MNIST digits (Sec. 3.2–3.4): LAVA reveals correlation patterns that reoccur across digits (e.g., shape similarities of digits 3, 5, and 8), which are not apparent from the clustering or from LFXAI explanations.
- LAVA on Bag-of-words (Sec. 3.2): LAVA discovers topics shared across documents.
- LAVA on KPMP kidney single-cell data (Sec. 3.5): without supervision, LAVA finds a module related to chronic kidney disease specific to epithelial cells, pointing to candidate gene markers for further investigation.
- LAVA-enabled diagnosis of embeddings (Sec. 3.3): comparing LAVA explanations shows that a badly parameterized UMAP fails to preserve the correlation structure of the original data and quantifies differences of correlation structure between different embeddings more generally.
- LAVA recovery of ground truth (Sec. 3.1): correctness against ground truth, and the effect of non-uniqueness, showing LAVA results on 5 different synthetic embedding datasets.
We see LAVA as a proof of concept for a different view of explainability for unsupervised ML, one that explains the structure of embeddings rather than individual examples or their summary (Sec. 4). We believe we demonstrate that LAVA brings added value to the field of XAI, and think that our contribution will be of interest to researchers and future work in XAI. We hope that this can be recognized.
In the new version of the PDF, we now also highlight all added text and figures in yellow, as requested. We thank the reviewer once more for their time and effort.
We thank the reviewer and answer in turn.
Baseline, correctness, ground-truth, quantitative evaluation
We add new experiments on synthetic data (Section 3.1). These highlight properties and performance of the LAVA method, as well as loss behavior of the AMF method, based on 5 synthetic embedding datasets showing a simple starting example and 4 more complex cases of non-uniqueness and low-rank approximation. For detail on methodology and results, we kindly refer the reviewer to the added Section 3.1.
Comparison to related work, including feature attribution
We found three new references, which we added to the Introduction: ClusterShapley (Marcilio-Jr & Eler, 2021), a method adapting Shapley explanations for feature attribution to explain manually annotated clusters of manifold learning methods; FADEx (Meneses et al., 2026), a single-sample (local) feature attribution method for manifold learning; and a gradient-based feature attribution method for t-SNE (Corbugy et al., 2024). All three fall into previously defined categories of methods adapting feature importance to unsupervised embeddings, either single-sample or dataset-wide, or require manual annotation. We argue that LAVA presents a fundamentally different view of explainability, shifting the focus from “why” some structure is uncovered to “what” the structure uncovered, well suited for the unsupervised setting (Section 1). Specifically, LAVA explanations represent local patterns reoccurring globally across latent embeddings. This makes direct comparison difficult, as feature attribution answers a different question and it is not designed or easily extendable to show commonality or reoccurrence of patterns across the entire structure of a latent space like LAVA (Section 3.4). We further note that feature attribution ground truth cannot be used to assess LAVA explanations, as LAVA examines features that covary together in neighborhoods of embeddings, rather than why an embedding method groups those samples into neighborhoods. Generally, we think repurposing existing XAI methods towards a fundamentally different goal, to enable direct or quantitative comparison with LAVA, is non-trivial and beyond the scope (as it would require substantial methodology contributions). We also note that there is no widely accepted consensus on the definition of an explanation, where distinct types of XAI can offer reasonable and complementary explanations from different perspectives on the same model. We believe that the comparison of LAVA to the LFXAI method, as a representative of feature-attribution XAI adapted for unsupervised embedding methods, illustrates these challenges well and also makes a general case about the differences and complementarity between LAVA and existing methods (Section 3.4).
LAVA embedding-dependence and diagnosing embedding “quality”
Conceptually, LAVA defines locality neighborhoods based on the embeddings, which inform pattern discovery across localities. As a result, LAVA explanations are influenced by the structure of the embeddings and do not merely recapitulate original data. We now include an embedding-awareness experiment in Section 3.3, assessing neighborhood and LAVA explanation (dis)similarity based on the original data and each of 3 different embeddings of MNIST: 2 differently parameterized UMAPs and a convolutional autoencoder. Briefly, we see differences in neighborhoods and in LAVA explanations, showing LAVA is indeed embedding-dependent. We also confirm that correlation patterns of the original space are poorly preserved when using a badly parametrized UMAP. Practically, comparison of LAVA explanations can be used complementarily to neighborhood retention (Han et al., 2022), to assess preservation of covariance structure between embeddings and the original data, and to diagnose “bad” embeddings (Section 3.3).
LAVA “detects feature covariation irrespective of feature usage by the model”
Clarification: explanations generated by LAVA depend on the embedding, since they capture patterns based on localities defined as neighborhoods in the latent space. At the same time, localities are summarized based on correlations of the original features, enabling LAVA to recover correlations that support the embedding structure rather than only those that might have been used by the method. In that sense, LAVA explanations are both data-oriented and embedding dependent.
Scientific discovery
Our results show LAVA extracts reoccurring correlation structure across MNIST digits, topics in documents of the BOW dataset, and kidney disease signal across cell types (Section 3), which are not apparent in the clustering (in the case of MNIST, also not apparent using LFXAI feature attribution; Section 3.4). For KPMP, we show LAVA discovers signal related to chronic kidney disease (CKD) specific to epithelial cells without supervision, identifying new genes for further investigation as potential disease biomarkers or therapy targets.