acceptodds
Withdrawn from ICLR 2027

FLACE: Explaining Frozen Classifiers through Signed Feature-Space Bottlenecks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Explaining a frozen classifier through intermediate coordinates requires an explicit link between those coordinates and its prediction. FLACE builds this link by learning a signed dictionary from the classifier's own features with classical semi-NMF, inferring nonnegative reconstruction coordinates, and reusing the original linear head. Each coordinate has an exact signed contribution to the reconstructed class logit, equal to the logit change on deleting that coordinate and its zero-baseline integrated gradient. This gives a direct interface for inspecting and editing reconstructed predictions without concept labels or an external text dictionary. On CIFAR-100, CUB, and ImageNet-100 with EfficientNet-B3, the evaluated native prediction paths show lower within-basis correlations and higher insertion AUC for FLACE than for the compared CBMs, with mixed deletion results. The coordinates weight numerical feature directions; optional names aid visual inspection without defining their meaning. Core implementation code is provided in the supplementary material.

Primary area:
interpretability and explainable AI
Keywords:
concept bottleneck models, label-free concepts, semi-NMF, feature-space decomposition, interpretable machine learning, vision models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UnEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

semi-NMF on features + frozen linear head -> exact closed form attribution/ablation on recon logits.idea is neat, table3 fidelity ok on archived bundles.but results not really reproducible and baselines arent fully controlled. weak reject for now

Critical strengths

attribution = signed logit drop when ablating (also zero baseline IG),so its tied to the reconstructed forward pass not just picked patches. nice

tab 3: nonneg coords keep top1 almost same acc, no retrain. seems usable

Critical weaknesses

code doesnt match paper

main.py hardcodes n_components=200 but sec 5.3 says k=600/700/900 for main table. so the released driver doesnt even run the reported config

eq 9 uses mean of max(α,0) but insert_deletion.py line33 just averages raw signed vals?? in principle → ranking may flip, so ins/del curves may change

experiment.py ranks + evaluates on same test split. not held-out

tab2 solver not said, tab3 from a separate archive. incomplete run-linked code/logs so cant check the main numbers

baselines

attribution identity is kinda free once u have linear decoder+affine head,so everything rides on beating simpler methods. tab2 mixes diff heads/dims/budgets.. app K has PCA,TopK-SAE but no same-protocol FLACE row. so does it beat plain PCA? not clear

Questions for authors

tab2 made w/ eq9 or the unclamped mean in repo? holds on seperate calib split?

one clean rerun for tab2 (solver cfg,recon fidelity,logit residuals)?

FLACE vs PCA vs topk-SAE in app K setup, same feats/head/budget?

Reviewer_QPrWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a technique to explain the concept contribution in classifier models without reliance on external tools. The paper describes the pipeline to do this, including the decomposition method (semi-NMF, signed dictionary with non negative coordinates), normalization and attribution to the logit. Evaluation is done on three datasets with CBMs as baselines.

Critical strengths

None

Critical weaknesses

1)The contribution is narrow as most of it already present in very similar contexts in the prior work. [1,3] already does pipeline similar to FLACE which is NMF style decomposition on frozen features, reconstruct and pass them through original head. Additionally the large account of how coordinates compose each reconstructed logit in this context is shown in [2].

2)The experimental design, and presentation of the paper has significant room for improvement:

a)The paper motivates based on not using external tools for explainability but only focuses on CBMs and doesn't compare against unsupervised decomposition techniques like ACE, ICE, or CRAFT which addresses this issue.

b)The paper does not explicitly state any hypothesis regarding the motivating point or does not present clear reasoning to the experimental design. This makes the experiment read like descriptive measurement rather an tests or evaluation.

c)The presentation is lacking structure and coherence. Also, sentences jump from one thing to another or end abruptly. Some abbreviations are used without defining them. It reads more like a technical report than a paper.

[1] Zhang, Ruihan, et al. "Invertible concept-based explanations for cnn models with non-negative concept activation vectors." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 13. 2021. [2] Fel, Thomas, et al. "A holistic approach to unifying automatic concept extraction and concept importance estimation." Advances in Neural Information Processing Systems 36 (2023): 54805-54818. [3] Fel, Thomas, et al. "Craft: Concept recursive activation factorization for explainability." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

Questions for authors

None

Reviewer_JkKjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a CBM-like method named FLACE, which uses semi-NMF to decompose the penultimate layer representations of an image classifier into a set of feature directions. For each image, FLACE approximates its original representation as a combination of these feature directions and feeds the reconstructed representation into the classifier's original prediction head. Since both the reconstruction and classification head are linear, it is possible to compute a contribution of each of these feature directions to the reconstructed class logits. Experiments are performed using EfficientNet-B3 as the only architecture and CIFAR-100, CUB, and ImageNet-100 as datasets. The paper shows that reconstructed representations largely recover the original classifier prediction, with only a small loss in accuracy. The paper also investigates whether the learned directions are redundant and how manipulating them affects classification confidence. To evaluate whether these directions are actually interpretable, qualitative visualizations of top-activating images with post-hoc textual descriptions are provided in the Appendix.

Critical strengths

None

Critical weaknesses
  1. Novelty is limited, and there is substantial overlap with existing methods that are not mentioned in the paper. The claimed contribution of discovering concepts directly from a model's representations without the need for predefined vocabularies has already been investigated in several existing papers such as UCBM [1], DN-CBM [2], and M-CBM [3]. UCBM uses NMF similarly to this paper, while M-CBM and DN-CBM use SAEs to decompose learned representations. Since these works are not discussed, the novelty and contribution of FLACE relative to these works remain uncertain.

  2. I think that using CBMs as baselines is inadequate for this work and post-hoc concept discovery and attribution methods would be more appropriate. The reason is that the purpose of a concept bottleneck is to constrain predictions to depend on something inherently interpretable, while trying to avoid (ideally) that uninterpretable information reaches the classifier. However, FLACE allows the classifier to use feature directions without establishing what they represent, so in this sense it is more similar to post-hoc concept discovery and importance methods (e.g., [4]). Furthermore, the interpretability of these directions is investigated mainly qualitatively, showing top activating examples for 5 features in the appendix. It is also not clear how the labels were assigned to these features. Appendix K pretty explicitly says: "The original figure-caption naming rubric remains unrecorded". Furthermore, maybe there were more qualitative examples, but they were omitted. Appendix H.1 says: "The two previously displayed top-16 grids are omitted from empirical evidence because the available record does not link their images to dataset/split/sample identifiers and activation ordering. Their image assets are preserved in the source package;no dataset identity or naming validity is inferred from their appearance".

There are also more problems than these two. For instance, concerning reproducibility, the paper says: "The optimizer used to produce the main table is not established by the available code record" in page 6 of main paper, and it says in appendix A: “The available July 25 implementation uses projected Adam as specified below. It is distinct from the alternating least-squares and multiplicative updates described in an earlier manuscript, and its linkage to the main-table runs has not been established”.

[1] Schrodi et al., Selective Concept Bottleneck Models Without Predefined Concepts, TMLR 2025.

[2] Rao et al., Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery, ECCV 2024.

[3] De Santis et al., Learning Concept Bottleneck Models from Mechanistic Explanations, ICLR 2026.

[4] Fel et al., A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation, NeurIPS 2023.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.