- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Conceptual Archetype Decomposition (CAD), an archetypal-analysis-based method that represents neural activations as convex combinations of concepts derived from training features. By retaining the decomposition coefficients, CAD enables tracing reconstructed test activations back to specific training locations
Critical strengths
- The method provides an intuitive and mathematically well-defined framework for tracing reconstructed activations to their training sources.
- The approach operates on frozen pretrained models and provides explicit provenance information without modifying the original predictor.
Critical weaknesses
-
Limited empirical evidence: The evaluation relies heavily on qualitative concept visualizations and reconstruction metrics. The central test-to-training provenance mechanism is not demonstrated on held-out examples.
-
Lack of interpretability evaluation: Although the learned representations are referred to as concepts, there is little quantitative evidence that they capture coherent, human-interpretable semantics. Reconstruction quality and traceability do not necessarily imply interpretability.
-
Nonnegativity and sparsity: While the decomposition coefficients are nonnegative, the concept vectors themselves are not necessarily nonnegative. Furthermore, the softmax parameterization produces dense coefficients without explicit sparsity constraints, potentially making the concepts and their supporting evidence difficult to interpret.
-
Limited methodological novelty: The core formulation closely follows classical archetypal analysis, and the proposed provenance mechanism is a direct consequence of composing the decomposition matrices. It is unclear what the method contributes beyond applying existing archetypal analysis to neural activations while retaining training-source identities.
-
Limited faithfulness to the original model: The reconstructed activations achieve only 12.9% agreement with the original predictions and 10.3% top-1 accuracy in the global CUB ablation. This raises concerns about whether the extracted concepts faithfully capture the information used by the original model, and consequently whether they provide meaningful explanations of its behavior.
Questions for authors
-
Sparsity: Since the method relies on softmax-normalized coefficients, the learned decompositions are generally dense. Have the authors investigated whether introducing sparsity constraints on the concept coefficients or activations could improve interpretability and yield more concentrated, meaningful training supports?
-
Quantitative interpretability comparisons: Could the authors evaluate the semantic interpretability of the discovered concepts and compare against methods such as CRAFT and A-SAE, as well as general SAE methods and linear extraction methods, such as PCA, NMF, and clustering approaches? Several established metrics could be considered:
- Concept clarity: Dreyer et al. (2025), Mechanistic Understanding and Validation of Large AI Models with SemanticLens,
- Monosemanticity (MS): Pach et al. (2025), Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models,
- Text-concept alignment: measuring how well concepts could be described with text, for example using the CLIP-Disscet scores: Oikarinen and Weng (2023), CLIP-Dissect.
- Concept/class detection ability: Bykov et al. (2023), Labeling Neural Representations with Inverse Recognition (INVERT), which evaluates how well representations discriminate semantic concepts using AUC
Such comparisons would help establish whether CAD discovers genuinely interpretable concepts rather than simply providing traceable reconstruction components