acceptodds
Under review as a conference paper at ICLR 2027

Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs

Abstract

Extracting human-interpretable concepts from LLMs is essential to understand the mechanisms underlying their capabilities. Sparse autoencoders (SAEs) approach this task by decomposing LLM internal representations into a sparse dictionary, with the hope that the extracted features align with human-interpretable concepts. Despite their empirical progress, SAEs suffer from a fundamental theoretical ambiguity: the relationship between LLM representations and human-interpretable concepts remains unclear. This lack of theoretical grounding gives rise to several methodological challenges, especially how sparsity should be imposed to facilitate concept extraction. In this work, we extend recent theoretical analyses of LLM representations and show that LLM representations can be approximated as a linear mixture of the marginal log-posteriors of individual latent concepts conditioned on the input context. Importantly, this characterization extends beyond the final layer to intermediate layers of LLMs. These findings motivate CONcept Component Analysis (ConCa), a principled formulation of concept extraction as a linear unmixing problem for recovering the log-posterior coordinates of individual concepts. We develop a sparse instantiation of ConCa, termed Sparse ConCa, and demonstrate its ability to extract meaningful concepts across multiple LLMs, offering theory-guided advantages over existing SAEs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Language Language Models, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ec1jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends Liu et al. (2026), who approximate final-layer LLM representations with a linear mixture of joint concept log-posteriors. The paper replaces the joint log-posteriors with marginal ones plus a bounded residual and extends this to intermediate layers. This motivates Sparse ConCa: a linear encoder with a regularization module, a linear decoder, and sparsity on the exponential of the latent space (to match the fact that the latent space is the log-posterior. Experiments on five LLMs show marginally better posterior alignment than SAE baselines.

Critical strengths
  • Deriving where sparsity should go from a rigorous generative model is a useful idea. A log-posterior of 0 means the concept is fully active, so sparsity belongs on exp⁡(z^)\exp(\hat z).
Critical weaknesses
  1. The error is not controlled relative to the signal. Only Ar(x)Ar(x) is bounded, but e~L(x)\tilde e_L(x) also contains eL(x)=−(L^⊤)−1hye_L(x) = -(\hat L^\top)^{-1}h_y. There is no discussion on the effect of this term on the quality of the approximation.
  2. The architecture is only partly derived from the theory, and the experiments do not isolate the part that is. R\mathcal{R} is justified only by generic overfitting, and LayerNorm makes the encoder nonlinear, against Insight 1.
Questions for authors
  1. Can you bound ∥e~L(x)∥\|\tilde e_L(x)\| relative to the log-posterior term in Eq. (4), or measure the total residual e~L(x)\tilde e_L(x) on E2E?
  2. Can you add ablations with α=0\alpha = 0, R\mathcal{R} = identity, and the sparsity term in Eq. (10) applied as S(z^)\mathcal{S}(\hat z) instead of S(exp⁡(z^))\mathcal{S}(\exp(\hat z))?
  3. MCC correlates z^\hat z with probe outputs log⁡σ(w⊤f(x)+a)\log\sigma(w^\top f(x) + a), a monotone function of a linear score. Does this favour a near-linear encoder over Top-kk/ReLU encoders? A PCA or ICA baseline would help.
1 comment

Q1: Full approximation residual

To address this concern, we have added Appendix O (Fig. 11), evaluating the full representation-level fitted residual on E2E. Using 24 joint concept configurations, we estimate marginal log-posteriors under a posterior-concentration approximation and evaluate the linear model on held-out configurations.

The results show that marginal log-posteriors explain 74.85%, 84.12%, and 75.78% of representation variance across three LLMs, corresponding to relative residual RMS values of 0.50, 0.40, and 0.49. These results quantify the approximation error relative to the representation signal under the stated posterior approximation. The remaining variance may also contain information beyond the annotated concepts.

Q2: Ablation studies

To examine these design choices, we have added Appendix L (Table 8), evaluating Posterior Alignment (MCC), Posterior Sensitivity, and Posterior Recovery (PCC²). Full ConCa achieves 0.725, 0.227, and 0.999, respectively. Removing sparsity (α=0\alpha=0), applying sparsity directly to z^\hat{\mathbf z} rather than exp⁡(z^)\exp(\hat{\mathbf z}), and removing LayerNorm (R=identity\mathcal R=\mathrm{identity}) reduce MCC to 0.703, 0.711, and 0.703, and Sensitivity to 0.211, 0.219, and 0.216, respectively. PCC² remains close to 0.999 for the first two ablations but decreases to 0.953 without LayerNorm.

Q3: Evaluation bias and linear baselines

We thank the reviewer for this suggestion. To examine whether Posterior Alignment favors near-linear encoders, we have added PCA and FastICA as strictly linear unsupervised baselines in Appendix R (Figs. 14–16).

Across four datasets, ConCa achieves an average Posterior Alignment MCC of 0.62, compared with 0.51 for PCA and 0.32 for ICA. The corresponding Posterior Sensitivity scores are 0.27, 0.18, and 0.09, respectively.

Interestingly, PCA and ICA achieve nearly perfect Posterior Recovery but substantially lower Alignment and Sensitivity. This shows that preserving posterior-related information through linear transformations does not necessarily produce individually aligned and concept-selective components. Thus, ConCa's advantage is not simply a consequence of using a near-linear encoder.


We thank the reviewer for the constructive feedback. We believe that our clarifications, additional theoretical analysis, and new experiments address the concerns raised, and hope the reviewer will reconsider their assessment of our work.

Reviewer_ui6COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Concept Component Analysis (ConCa), a theory-motivated framework for extracting interpretable concepts from LLM representations. Building on prior work showing that final-layer representations approximately linearly encode joint latent-concept log-posteriors, the paper extends this characterization to marginal concept posteriors and to intermediate layers. Based on this view, concept extraction is formulated as a linear unmixing problem, leading to Sparse ConCa, an autoencoder-style method that imposes sparsity in the probability domain via exp⁡(z^)\exp(\hat z). Empirical validations of ConCa are justified across multiple LLMs.

Critical strengths
  • The paper is overall well-written.

  • The paper extends the existing theory on the relationship between LLM representations and human-interpretable concepts.

  • The proposed method is motivated theoretically and validated empirically.

Critical weaknesses
  • Several important theoretical related works are missing, e.g., On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy (ICLR26), Toward Identifiable Sparse Autoencoders (ICML26), and On the Theoretical Foundation of Sparse Dictionary Learning in Mechanistic Interpretability (arxiv).

  • There seem to be a missing link between the theory and methodology. The two corollaries establish the existence of a global linear mixing of the true posterior coordinates, whereas the ConCa objective only minimizes expected reconstruction with a sparsity regularizer. It is unclear whether the population optimum of this objective identifies the theoretical mixing matrix and the true marginal log-posterior coordinates.

  • An empirical comparison on the pareto frontier is missing. Particularly, since different methods impose sparsity in substantially different ways, it would be necessary to report reconstruction and concept-quality metrics at matched sparsity levels, and provide the corresponding Pareto frontiers.

  • The baseline coverage appears to be a bit overclaimed. Some modern SAE variants are missing, e.g., JumpReLU, Gated SAE, Matryoshka SAE, etc. Also, the theoretical motivated WSAE would also contribute as a proper baseline.

  • Empirical validations on some key components of the theoretical bound is missing, including e~L(x)\tilde{e}_L(x) and ∣G−1∣2\\|G^{-1}\\|_2.

Questions for authors
  • Does the proposed ConCa lie around the SAE pareto frontiers? How does it compare with baseline SAE methods on the pareto frontier?

  • How does ConCa perform compared with a broader SAE baselines (e.g., JumpReLU, Gated SAE, Matryoshka SAE, WSAE, etc.)?

  • The empirical validation establishes that the additional interaction residual Ar(x)Ar(x) induced by replacing joint posteriors with marginal posteriors is relatively small, but does not directly quantify the full approximation residual e~L(x)=Ar(x)+eL(x)\tilde{e}_L(x)=Ar(x)+e_L(x). Since the main interpretation of ConCa relies on the total residual being small, could the authors report the full representation-level approximation error relative to the representation signal?

  • Similar as the previous point, how large is ∣G−1∣2\\|G^{-1}\\|_2 empirically?

2 comments

Related works

We thank the reviewer for pointing out these relevant works. These studies primarily investigate concept recovery under the linear representation hypothesis underlying SAEs, i.e., f(x)≈Az\mathbf f(\mathbf x)\approx\mathbf A\mathbf z. As highlighted in the Introduction, our central contribution is to establish Eq. 4 (Corollary 3.1). ConCa is developed based on this posterior-based formulation. We have added the references.

Connection between theory and method

Before response, we would like to clarify that concept extraction involves two distinct and nontrivial problems: (i) establishing the theoretical relationship between LLM representations and latent concepts, and (ii) developing algorithms with guarantees for recovering these concepts.

Our primary theoretical contribution addresses (i), as emphasized in the Introduction and Sec. 3, i.e., we characterize the underlying components as marginal concept log-posteriors, providing a principled formulation of what should be extracted. Building on this, we propose ConCa. Its structure follows Eq. (4), while sparsity serves as an inductive bias motivated by sparse dictionary learning. To further justify this choice, we have added Appendix M, extending classical sparse dictionary learning arguments to the effective posterior log-odds components of Eq. (4). Under additional assumptions, we establish identifiability up to permutation and scaling. Regarding (ii), guaranteeing identifiability for a practical learning objective is itself a substantial challenge. Such guarantees require additional assumptions on sparsity, dictionary structure, and optimization. Indeed, Toward Identifiable Sparse Autoencoders, as mentioned, specifically investigates this challenge under the conventional SAE formulation.

In summary, our contribution is to establish the posterior-based formulation and develop a theory-guided method. Providing guarantees for practical method remain an important future direction.

Pareto frontier

We thank the reviewer for this suggestion. ConCa aims to extract the posterior-related components characterized in Eq. (4), rather than maximize sparsity. Sparsity is an inductive bias, while reconstruction measures information preservation, neither alone determines concept quality.

Nevertheless, matched sparsity provides a useful control. We have added Appendix N (Figs. 7–10), sweeping eight sparsity coefficients for ConCa, PAnneal, and Structured SAE. We measure achieved effective decoder support to compare their reconstruction and concept-quality trade-offs. At comparable effective supports (within 6%), ConCa achieves reconstruction MSE 2.62, compared with 2.90 for PAnneal and 23.09 for Structured SAE. ConCa also achieves the highest Posterior Alignment on all four datasets, with average MCC improvements of 0.0303 and 0.0404, respectively. Please see Appendix N for more details.

Additional baselines

We have added Appendix Q (Fig. 13), comparing ConCa with JumpReLU, Gated SAE, Matryoshka SAE, and WSAE. ConCa achieves reconstruction cosine similarity 0.99 and MSE 2.76, compared with the best additional-baseline results of 0.88 and 34.08. It also achieves the highest Posterior Alignment and Sensitivity across all four concept datasets. Its Alignment MCC ranges from 0.5232 to 0.7254, compared with 0.3937 to 0.6089 for the strongest additional baselines. Posterior Recovery also remains high (0.9967–0.9996). These results further support ConCa's advantages. Thanks again!

Full approximation residual

We thank the reviewer for pointing out this distinction. We have added Appendix O (Fig. 11) to evaluate the full representation-level fitted residual. Using 24 joint concept configurations from E2E and nested cross-validation over held-out configurations, we fit the marginal-posterior linear model under a posterior-concentration approximation. The model explains 74.85%, 84.12%, and 75.78% of the representation variance for Pythia-1.4B, Gemma-3-1B, and Qwen3-1.7B, respectively. The corresponding relative residual RMS values are 0.50, 0.40, and 0.49. Please see Appendix O for more details.

The inherited residual

We have added Appendix P (Fig. 12) to evaluate the inherited term. Using the same E2E setting, we measure both the operator-norm bound and its realized directional counterpart. The realized normalized residuals are 0.46, 0.37, and 0.44 for Pythia-1.4B, Gemma-3-1B, and Qwen3-1.7B, respectively, compared with worst-case operator-norm bounds of 2.34, 681.36, and 41.05. These results show that the operator-norm bound can be highly conservative, particularly for RMSNorm models. See Appendix P for more details.


We sincerely thank the reviewer for these constructive comments. We hope that the above clarifications adequately address the concerns and will be taken into consideration when reassessing our work.

Thank you for the detailed rebuttal. It addressed most of my empirical concerns.

However, two main theoretical concerns remain (see below), and I maintain my original rating of 2.

W2. The additional identifiability analysis is helpful, but it does not establish recovery guarantees for the practical Sparse ConCa objective. The gap between the theoretical characterization and the proposed method remains.

W5. Although the new experiments quantify the approximation errors, the reported relative residuals remain substantial (0.40–0.50), and the theoretical upper bounds are very loose. This weakens the empirical support for the proposed posterior-based characterization.

Reviewer_NVbAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a probabilistic account of concept extraction, characterizing LLM representations as approximate linear mixtures of marginal concept log-posteriors and extending this analysis to intermediate layers. It introduces Sparse ConCa, an autoencoder that applies sparsity regularization to exponentiated latent features. Experiments across Pythia, Gemma3, and Qwen3 compare ConCa with SAE variants using reconstruction, probe-based concept metrics, and out-of-distribution tasks.

Critical strengths
  1. The paper distinguishes preserving concept information from organizing it into individual features. Its separate alignment, sensitivity, and recovery evaluations provide a useful perspective beyond reconstruction quality alone.
  2. The experiments cover five LLM backbones, multiple concept datasets, and several SAE baselines. The reported improvements in alignment and sensitivity across these settings suggest that the empirical benefits extend beyond a single model or benchmark.
Critical weaknesses
  1. The posterior interpretation is not established. The paper interprets exponentiated features as marginal posterior probabilities, but exponentiation only guarantees positivity. The stated architecture does not enforce normalization over the possible values of each concept, and the evaluation does not establish that these features are calibrated probabilities. The paper should explain what connects its learned coordinates to the posterior quantities in Eq. (4).
  2. Incompatible assumption in the posterior-readout argument. Corollary E.1 assumes that the mixing matrix is left-invertible. However, its definition following Eq. (3) implies zero row sums, so it annihilates the all-ones vector and cannot have a left inverse. This makes the stated readout justification inapplicable to the matrix defined in the paper.
  3. Unreported effective sparsity. The paper does not report ConCa’s achieved sparsity or compare methods at matched sparsity levels. Its reconstruction gains may therefore reflect weaker sparsity constraints rather than better concept extraction
Questions for authors
  1. What connects the learned ConCa coordinates to the marginal posteriors in Eq. (4), and what evidence supports this probability interpretation beyond correlation?

  2. How do you reconcile the left-invertibility assumption in Corollary E.1 with the zero row sums of the matrix defined after Eq. (3)? Summing over all joint configurations gives

=\sum_i p(z=z_i\mid y_j)-\sum_i p(z=z_i\mid y_0) =1-1=0.$$ Therefore, $$A\mathbf{1}=(\widehat L^\top)^{-1}L\mathbf{1}=0,$$ so $A$ has a nontrivial null space and cannot have a left inverse. 3. Is there evidence that its reconstruction gains persist at comparable effective sparsity, rather than arising from weaker sparsity constraints?
2 comments

Q1: Posterior interpretation, normalization, and empirical validation

We thank the reviewer for raising this question. The posterior interpretation of ConCa is supported from three perspectives: theoretical, methodological, and empirical.

1. Theoretical perspective. We establish that, under the stated conditions, LLM representations approximately encode a linear mixture of marginal concept log-posteriors (Corollary 3.1, Eq. (4)). This provides a theoretical motivation for extracting posterior-related components through linear unmixing.

2. Methodological perspective. We agree that exponentiation guarantees positivity but not probability normalization. However, this does not invalidate the posterior-based interpretation, given the inherent scaling ambiguity of linear inverse problems.

For example, given

fL(x)=Aℓmarg(x)+b,\mathbf f_L(\mathbf x)=\mathbf A\boldsymbol\ell_{\mathrm{marg}}(\mathbf x)+\mathbf b,

where ℓmarg(x)=[log⁡p(zk=a∣x)]k,a\boldsymbol\ell_{\mathrm{marg}}(\mathbf x)=[\log p(z_k=a\mid\mathbf x)]_{k,a}, we always have

fL(x)=(AD)(D−1ℓmarg(x))+b\mathbf f_L(\mathbf x)=(\mathbf A\mathbf D)(\mathbf D^{-1}\boldsymbol\ell_{\mathrm{marg}}(\mathbf x))+\mathbf b

for any invertible diagonal matrix D\mathbf D. Thus, component scales cannot be uniquely determined from reconstruction alone. Resolving this ambiguity requires additional constraints, such as unit-variance normalization in ICA or unit-norm dictionaries in sparse coding. ConCa also normalizes decoder columns, but this does not guarantee the true posterior scale.

Enforcing probability normalization would require knowing which coordinates belong to each concept and the number of possible values, generally unavailable in unsupervised extraction. Moreover, scaling ambiguity is acceptable in applications such as concept steering, where rescaling a direction can be compensated for by its steering coefficient.

Therefore, ConCa aims to extract posterior-related components up to scaling and permutation, rather than exact calibrated posteriors. This also motivates our use of MCC, which is insensitive to positive coordinate rescaling.

3. Empirical perspective. Posterior Alignment examines whether ConCa extracts the components predicted by our theory. As established in Appendix F, under the stated conditions, a linear probe trained with binary cross-entropy estimates marginal posterior log-odds. Applying sigmoid and taking the logarithm yields the reference marginal log-posteriors.

We use Hungarian assignment for global one-to-one matching between learned features and reference coordinates, followed by MCC evaluation. This accounts for permutation and scaling ambiguities while preventing one feature from being matched to multiple concepts. We further evaluate Counterfactual Sensitivity to measure concept-specific responses and Posterior Recovery to assess information preservation. Together, these evaluations support the posterior-based interpretation without claiming exact probability calibration.

Q2: Left-invertibility assumption

We thank the reviewer for pointing out this issue. We agree that the original left-invertibility assumption is incompatible with the mixing matrix since A1=0\mathbf A\mathbf 1=\mathbf 0. We have corrected Corollary E.1 and its proof.

Specifically, we consider a subset of latent concepts and use the Moore–Penrose pseudoinverse under rank⁡(AR)=∣ZS∣−1\operatorname{rank}(\mathbf A\mathbf R)=|\mathcal Z_S|-1. This recovers the selected joint log-posteriors up to a shared additive constant, which is eliminated by softmax. The revised corollary provides the complete derivation.

Q3: Sparsity vs. Reconstruction

As emphasized in the Introduction, our goal is to establish the relationship between LLM representations and latent concepts and develop a method for concept extraction. Sparsity serves as an inductive bias for concept separation, rather than an objective itself. Thus, greater sparsity does not necessarily imply better concept extraction.

Nevertheless, comparing methods at similar achieved sparsity helps determine whether reconstruction gains arise from weaker constraints. We have added Appendix N, varying the sparsity coefficient for ConCa, PAnneal, and Structured SAE. Since these methods impose sparsity differently, we compare their achieved effective decoder support, reconstruction, and concept quality (Figs. 8 and 10).

At the closest operating points, their effective supports are 3025, 2902, and 3068, differing by less than 6%. The corresponding reconstruction MSE values are 2.62, 2.90, and 23.09. ConCa also achieves the highest Posterior Alignment across all four datasets, outperforming PAnneal and Structured SAE by an average MCC of 0.0303 and 0.0404, respectively.


We hope these clarifications adequately address your concerns and will be taken into consideration during the reassessment of our work. Thank you for your time and for reconsidering.

Thank you for the detailed response, the corrected Corollary E.1, and the new experiments. However, the response weakens the link between theory and method. If z^\hat{z} recovers log-posteriors only up to scaling, then exp⁡(z^)\exp(\hat{z}) is no longer a posterior, which removes the stated reason for imposing sparsity on exp⁡(z^)\exp(\hat{z}). The ablation in Appendix L agrees: sparsity on exp⁡(z^)\exp(\hat{z}) improves MCC by only 0.01–0.02 over no sparsity or sparsity on z^\hat{z}. At matched sparsity, the gain over PAnneal is also small. I will therefore keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.