acceptodds
Withdrawn from ICLR 2027

Conceptual Archetype Decomposition for Tracing Reconstructed Neural Activations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Concept decompositions of neural activations usually expose a dictionary and reconstruction weights, but inspecting how a reconstructed test feature draws on particular training locations requires an explicit provenance map. We present Conceptual Archetype Decomposition (CAD), a post-hoc explanatory interface derived from classical archetypal analysis. Concepts are convex combinations of full-image training activations, and test features are reconstructed as convex combinations of these fixed concepts. Composing the two coefficient matrices gives a normalized map from each test location through concepts to training locations. This map is exact for the learned decomposition, remains available before optimizer convergence, and its reconstructed features satisfy a perturbation bound when the coefficients are fixed. Existing controlled CUB results describe reconstruction sensitivity to the implementation choices; we distinguish these results from a secondary pairwise logit diagnostic that does not establish prediction fidelity. CAD supplies inspectable training evidence while leaving the original predictor unchanged. The core implementation code is available in the supplementary material.

Primary area:
interpretability and explainable AI
Keywords:
concept decomposition, interpretable machine learning, concept bottleneck models, sample-grounded explanations, archetypal decomposition, vision interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_zVvrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Conceptual Archetype Decomposition (CAD), an archetypal-analysis-based method that represents neural activations as convex combinations of concepts derived from training features. By retaining the decomposition coefficients, CAD enables tracing reconstructed test activations back to specific training locations

Critical strengths
  1. The method provides an intuitive and mathematically well-defined framework for tracing reconstructed activations to their training sources.
  2. The approach operates on frozen pretrained models and provides explicit provenance information without modifying the original predictor.
Critical weaknesses
  1. Limited empirical evidence: The evaluation relies heavily on qualitative concept visualizations and reconstruction metrics. The central test-to-training provenance mechanism is not demonstrated on held-out examples.

  2. Lack of interpretability evaluation: Although the learned representations are referred to as concepts, there is little quantitative evidence that they capture coherent, human-interpretable semantics. Reconstruction quality and traceability do not necessarily imply interpretability.

  3. Nonnegativity and sparsity: While the decomposition coefficients are nonnegative, the concept vectors themselves are not necessarily nonnegative. Furthermore, the softmax parameterization produces dense coefficients without explicit sparsity constraints, potentially making the concepts and their supporting evidence difficult to interpret.

  4. Limited methodological novelty: The core formulation closely follows classical archetypal analysis, and the proposed provenance mechanism is a direct consequence of composing the decomposition matrices. It is unclear what the method contributes beyond applying existing archetypal analysis to neural activations while retaining training-source identities.

  5. Limited faithfulness to the original model: The reconstructed activations achieve only 12.9% agreement with the original predictions and 10.3% top-1 accuracy in the global CUB ablation. This raises concerns about whether the extracted concepts faithfully capture the information used by the original model, and consequently whether they provide meaningful explanations of its behavior.

Questions for authors
  1. Sparsity: Since the method relies on softmax-normalized coefficients, the learned decompositions are generally dense. Have the authors investigated whether introducing sparsity constraints on the concept coefficients or activations could improve interpretability and yield more concentrated, meaningful training supports?

  2. Quantitative interpretability comparisons: Could the authors evaluate the semantic interpretability of the discovered concepts and compare against methods such as CRAFT and A-SAE, as well as general SAE methods and linear extraction methods, such as PCA, NMF, and clustering approaches? Several established metrics could be considered:

  • Concept clarity: Dreyer et al. (2025), Mechanistic Understanding and Validation of Large AI Models with SemanticLens,
  • Monosemanticity (MS): Pach et al. (2025), Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models,
  • Text-concept alignment: measuring how well concepts could be described with text, for example using the CLIP-Disscet scores: Oikarinen and Weng (2023), CLIP-Dissect.
  • Concept/class detection ability: Bykov et al. (2023), Labeling Neural Representations with Inverse Recognition (INVERT), which evaluates how well representations discriminate semantic concepts using AUC

Such comparisons would help establish whether CAD discovers genuinely interpretable concepts rather than simply providing traceable reconstruction components

Reviewer_hZEiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to encode each concept as a weighted-sum of training samples rather than arbitrary directions or points in the space. So a concept can be decomposed into a set of crops from a training set. There is a qualitative sample Fig 2 of how these representative crops would look like. The paper also shows downstream accuracy of model with concepts represented this way (not sure about this table). There is also Table 2 showing reconstruction , downstream agreement to original model, and downstream accuracy with different ways of representing concepts.

Critical strengths

S1 Restricting the way concepts are represented in the space (archetypes) is a sensible direction to me, which follows Archetypal SAEs (Fel et al.)

S2 Doing downstream logit evaluation and not just focusing on the level, is a good way to evaluate.

Critical weaknesses

W1) I am not sure how the manuscript was exactly produced, but I hardly followed anything. It seems heavily AI-edited (if not generated), which seems to be acceptable as per conference policy, so I’ll try to not let this fact in itself affect my score. However, when it affects the understanding of the paper, it can be interpreted as a weakness.

To be more concrete:

  • Line 32 in Introduction “how do those regions enter a reconstruction of a new image?” This sentence in the first paragraph of introduction is probably not understandable for most future readers. It is not pointing to a concrete problem statement and what is the target.

  • Similarly, the second paragraph of introduction jumping to the method and BtestB_test throws the reader off. What even is B? What is a “complete normalized provenance map”? Again in the second paragraph of the introduction this should not be included in such a way.

  • Figure 1 (teaser/method figure?): There is Z, C, A, B, and B$_test$ written in the caption of the figure, which is nowhere to be found in the figure itself. In fact, I did not see any relevance of the figure and its caption. Also in the figure, what is “attribute concept?”

  • In the related work section, the paper should ideally connect how the related work “relates” to this work. In Sec 2.1, we currently have a few attribution methods listed, ending with what concept explanations are.

  • Further, in Sec 2.2 of related work, lines 074-077 just give a summary of the method again rather than connecting the work to existing literature.

  • The paper does not need to cite Goodfellow et al. for the word “intermediate activations” when describing the method. It is better suited for the beginning of an introduction or related work section.

  • I don’t see the point of mentioning which seed number was used in the caption of Figure 2, instead of explaining what we should observe in the figure.

  • I also don’t see why the file path should be mentioned in implementation details in the main paper (lines 357-358)

I understand that this submission must have taken significant effort and appreciate the authors for their submission. I also understand that the deadline rush may cause formatting errors in the manuscript. However, every future reader/reviewer will have a certain amount of patience and if on every other paragraph they somehow get confused, it is likely that the paper would lose that reader/reviewer. I hope my comment and samples appear helpful in putting the manuscript in a better shape!

W2) Aside from formatting concerns which significantly hinder my understanding of the paper, the results seem quite limited. There is one qualitative figure of how the archetype for a particular concept looks like, which if I have understood the paper correctly, is showing which crops are used to reconstruct a particular concept embedding. I think a single qualitative figure (or even a few more) would not justify using this method.

I also did not understand Table 1, what Gold-over-original is. Are these prediction agreements with the model? Why is it that here CAD is at 99% agreement with the model? Also why is the other baseline PCA?

Additionally, in Table 2, Top-1 accuracy on CUB for the best row is at 0.10. As far as I remember for CUB on ResNet-50 the numbers would be at 80% and above. Could the authors explain this?

W3) There is an abundant literature on prototypical networks (ProtoPNet, ProtoPPool, etc.), that also try to ground each concept (i.e prototype) to real training patches and samples. These are also not discussed in the paper and not included in any comparisons.

Questions for authors

A few under W2 above.

Reviewer_4AMjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents Conceptual Archetype Decomposition (CAD) for decomposing and tracing intermediate activations of frozen vision models. Following the nested convex-combination structure of classical archetypal analysis, CAD represents concepts as convex combinations of training-location activations and reconstructs test activations using the fixed concepts. Composing the two coefficient matrices yields normalized provenance weights from reconstructed test locations to training locations. The paper analyzes the composed provenance map, fixed-coefficient perturbation bounds, and truncation error for displayed supports. Experiments on CUB and ImageNet100 include a class-conditioned comparison with PCA and a global ablation of the composition constraint and activation support.

Critical strengths
  1. The method retains training-image and location identities and connects concept construction coefficients with test reconstruction coefficients. This makes the training sources and mixture weights of a selected reconstruction directly queryable. Unlike retrieving similar images solely as concept examples, it explicitly records the sources participating in the reconstruction.
  2. The paper carefully states the scope of its theoretical results, including the assumptions of the fixed-coefficient perturbation bound and the distinction between idealized exact block minimization and the implemented Adam solver. It also explicitly distinguishes provenance weights from semantic relevance and classifier effects.
Critical weaknesses
  1. The contribution beyond existing methods is not sufficiently established. CAD’s objective and nested convex-combination structure come from classical archetypal analysis, while the nonnegativity and normalization of the composed provenance map follow directly from these constraints. A-SAE also constructs dictionaries using convex combinations of data. CAD emphasizes retained location identities, fixed-dictionary test reconstruction, and provenance queries, but it is unclear what additional capability these choices provide over standard AA with the same source information retained. The absence of matched comparisons with AA and A-SAE further limits the assessment of its practical advantages.
  2. The experiments do not directly validate the utility of the complete provenance interface. Figure 2 and Appendix Figure 3 show only highly weighted training supports for individual concepts, without a complete held-out test-location-to-concept-to-training-location example, retained-mass measurements, or evidence of inspection benefits. Table 1 uses true-class information to select the decomposition, and its pairwise score automatically counts originally correct predictions as successes; near-perfect scores therefore do not establish prediction fidelity. In Table 2, full CAD achieves only 12.9% mean agreement with the original predictions, limiting the value of these reconstructions for explaining the original predictor’s behavior. Moreover, the large error increase after removing the composition normalization may reflect numerical optimization effects rather than establish the advantage of the constraint itself.
Questions for authors
  1. What additional capability does CAD provide over standard AA when training-location identities are retained and test coefficients are fitted with a fixed dictionary? Could you provide matched comparisons with AA and A-SAE?
  2. Could you provide a complete held-out provenance example, including concept weights, composed training-source weights, retained mass, and reconstruction error, and explain what concrete insight this analysis provides?
  3. What is the original model’s accuracy in Table 2, and what explains CAD’s 12.9% prediction agreement? Could you report results separately by concept count to show whether this limitation persists?
  4. For Table 1, could you report the original accuracy, original error counts, and pairwise success rates restricted to those errors, and clarify the tie-breaking rule?
  5. Does the large error increase without the B-simplex constraint arise from optimization difficulties? Could you provide convergence curves and test whether suitable initialization and learning-rate choices change the conclusion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.