acceptodds
Under review as a conference paper at ICLR 2027

Concept Attribution Analysis: A Principled Approach to Attributing Changes in LLMs

Abstract

Large language models (LLMs) encode rich semantic information in their internal representations, motivating growing interest in extracting human-interpretable concepts for model understanding and analysis. Existing approaches, including linear probing, sparse autoencoders, and concept component analysis, primarily adopt a single-context perspective, asking what concepts are encoded in the representation associated with a single input context. Such a perspective is inherently underdetermined: multiple concept-level explanations may provide equally plausible interpretations of the same representation, making it difficult to determine which one faithfully reflects the underlying concept structure. In this work, we move from single contexts to pairs of related contexts and exploit the structured variation in their corresponding LLM representations. Building on the recent theoretical characterization of LLM representations through concept log-posteriors, we formulate the representation change between paired contexts as a sum of concept-specific change contributions. We then establish conditions under which these contributions are identifiable up to a global permutation, even when multiple concepts change simultaneously within a given context pair. Guided by this analysis, we develop a practical method, termed Concept Attribution Analysis, that attributes an observed representation change to individual concept changes from paired LLM representations. Experiments on controlled simulations validate the predicted identifiability behavior and demonstrate robustness under departures from the idealized assumptions, while experiments on real LLM representations demonstrate the practical utility of concept-level change attribution.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Concept Unmixing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HsG9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Concept Attribution Analysis. It is a method for decomposing attribute changes in LLM representation into split concept contributions. The authors study each concept can be identified when multiple concepts change together. They build the identifiability results under three assumptions: Local Directness, Conditional Richness, and Support Intersection. Based on the analysis, they develop a method that combining graph-based subspace initialization with block-sparse reconstruction to recover concept-specific contribution from paired representations. The experiments include synthetic Gaussian concept mixtures and real LLM evaluation. The results show improved contribution recover, semantic readout, reconstruction and intervention.

Critical strengths
  1. this paper studies an important problem in LLM interpretability. Understanding how indivisual concept contributes to attribute changes in representation space is meaningful.
  2. The theoretical analysis is detailed. This paper explicit conditions/assumptions when individual concept can be identified.
  3. The synthetic experiment has useful robustness analysis. And the real-world LLM experiment is meaningful on evaluating whether recovered contributions can actually influence model predictions
Critical weaknesses
  1. Insufficient ablation studies to isolate the main contributions. This paper introduce several components: graph-based initialization, subspace clustering, and multi-dimensional block-sparse reconstruction. However, there is no experiment and ablation study to isolate the contribution of each component. In synthetic experiment, there is no experiment to compare the proposed method with SSAE.
  2. Limited real-LLM evaluation. This paper studies Pythia-70M and Qwen3-1.7B. But it is unclear whether the results generalize to larger LLMs.
Questions for authors
  1. The proposed method combines graph-based initialization, subspace clustering, and multidimensional block-sparse reconstruction. Can the authors provide an ablation study to isolate the contribution of each component?
  2. Since the synthetic experiments provide ground-truth concept contributions, can authors include SSAE as a baseline in these experiments? A direct comparison would help demonstrate whether ConAn offers an advantage over existing methods in recovering individual concept contributions.
  3. In the synthetic experiments, graph-based and random initialization use different decoder-freezing schedules. can the authors compare them under the same training schedule?
  4. The real-LLM experiments are limited to Pythia-70M and Qwen3-1.7B. Do the authors have plan to test on larger LLMs to show the proposed method remains effective?
2 comments

Thank you for your positive assessment of our theoretical analysis and experiments, and for suggesting the additional comparisons.

1. Component ablations and a shared training schedule

"Can the authors provide an ablation study to isolate the contribution of each component?"

"can the authors compare them under the same training schedule?"

Following your suggestion, we compared all four initialization variants under a shared training schedule of 5,000 steps with the dictionary frozen and only the encoder trained, followed by 5,000 steps of joint training. Full initialization retains its advantage in contribution recovery under this schedule, achieving a contribution FVE of 0.9914 compared with 0.0877 for a random dictionary.

We originally compared full and random initialization to evaluate the procedure as a whole, because the graph's sample-membership estimates and the subspace clustering jointly determine the initial concept spaces. We used the initial frozen-dictionary phase so the encoder could learn to allocate contributions within those estimated spaces. Your suggested comparison lets us examine the two stages more closely by adding one variant that constructs a dictionary from graph-derived directions without subspace clustering or subsequent refinement, and another that replaces the graph-based memberships with random sample groups while retaining clustering. We train all four variants with the same multidimensional block-sparse objective at d=256,m=32,r=6d=256,m=32,r=6, with 2–4 concepts changing in each pair.

InitializationSubspace overlap (before→after training) ↑Contribution FVE after training ↑Reconstruction FVE after training ↑
Full initialization0.9622→1.00000.99140.9922
Random dictionaryN/A→0.47020.08770.9758
Graph directions without clustering0.3079→0.68550.46170.9817
Random sample grouping0.3259→0.69640.48400.9828

In this table, contribution FVE measures recovery of individual concept vectors, whereas reconstruction FVE measures recovery of their sum. Full initialization already captures the concept spaces well before training (0.9622 subspace overlap) and reaches 1.0000 as block-sparse training learns to allocate contributions. While both stage-replacement variants also improve during training, their lower contribution FVE confirms the benefit of retaining both initialization stages. The random dictionary's high reconstruction FVE alongside low contribution FVE shows that the advantage of full initialization lies in recovering the individual contributions even when the total change is reconstructed well.

2. Synthetic SSAE comparisons

"can authors include SSAE as a baseline in these experiments?"

In the added synthetic comparison, SSAE recovers contribution vectors very accurately when the true dictionary has full column rank, while ConAn has a substantial advantage in the overcomplete setting. Although we originally compared the methods through semantic tasks on real models because SSAE's scalar atoms and ConAn's multidimensional blocks represent different units, expressing both methods' outputs as concept vectors provides the direct comparison with ground-truth contributions that you suggested.

Both methods use the same training and validation data and the same initial dictionary of 32 groups of six directions, with two to four concepts changing in each pair (the true dictionary has full column rank at d=256d=256 and is 1.5× overcomplete at d=128d=128). Because SSAE treats the 192 columns as scalar atoms under its native signed affine model and constrained objective, we evaluate its outputs under the same concept-vector metric as ConAn by summing the six atom contributions within each group fixed before training, preserving the learned directions and coefficients without rotation or regrouping.

DimensionMethodContribution FVE ↑Reconstruction FVE ↑Support precision ↑
256ConAn0.99140.9922100.00%
256SSAE0.99741.00009.57%
128ConAn0.98250.987085.43%
128SSAE0.33330.96879.36%

The table reports the best tested SSAE contribution recovery at each dimension.

At d=256d=256, SSAE achieves slightly higher contribution FVE than ConAn, though it predicts small nonzero contributions for many concepts that did not change. Under the shared contribution-norm detection threshold, these false positives contribute little to the vector error but lower support precision (the fraction of detected concepts that actually changed) to 9.57%. At d=128d=128, by contrast, the best tested SSAE setting reaches a contribution FVE of only 0.3333 compared with ConAn's 0.9825, showing a clear advantage for ConAn when the dictionary is overcomplete.

3. Interventions on a larger model

"Do the authors have plan to test on larger LLMs to show the proposed method remains effective?"

On Qwen3-4B, graph-initialized ConAn achieves higher contribution-transfer success than SSAE and Top-K SAE both at the predicted magnitude and after norm matching. While we initially used smaller backbones, as is common in SAE research, to facilitate controlled semantic evaluation, we followed your suggestion and extended this experiment to Qwen3-4B to test whether contribution transfer remains effective on a larger model.

Using the original texts and data splits, we extract Qwen3-4B representations and transfer a donor contribution to another context on the same test samples, either applying the predicted contribution directly or rescaling it to match the norm of the donor's full representation difference.

MethodTransfer success at original magnitudeTransfer success with norm matching
Graph-initialized ConAn63.41%94.52%
Randomly initialized ConAn51.75%94.61%
SSAE29.64%84.56%
Top-K SAE51.45%79.06%

Graph initialization improves transfer when the predicted magnitude is retained. Although Top-K SAE has lower reconstruction error than graph-initialized ConAn (0.0329 versus 0.0447), its transfer success is lower under both evaluations.

The tables report means, with full statistics and protocols provided in Appendix H.2 (ablations), H.3 (SSAE), and H.5 (Qwen3-4B).

Reviewer_v9joOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a change-centric view of LLM representations to examine how concept-level information changes across pairs of related contexts. Unlike SAEs, the authors claim that this view can provide more stable concepts across different contexts than “single-context” analysis. The authors develop identifiability theory and corresponding methods.

Critical strengths
  1. The presentation of the paper is clear.
  2. Identifiability of the change-centric LLM representations is established. The theory and the devised methods are connected.
  3. Experiments are designed to test violations of the assumptions.
Critical weaknesses
  1. There has already been a study on the identifiability of interpretable concepts in LLMs [1], published before the Liu et al., 2026a work you cite. You should discuss the relationship between that work and yours, including the theoretical conditions (sparsity, sufficient variability, etc.) and what is identifiable under each theory. In particular, both works assume linear models and make use of sparsity. Furthermore, [1] considers relationships between concepts.
  2. Please incorporate some examples in the introduction to clarify the distinction between “what concepts are represented” and “how concept information changes.”
  3. Why do you want to identify the change in the k-th latent in a pair of contexts, i.e., Gk(x,x′)G_k(x,x^\prime), instead of z itself? It’s also interesting that, in your proposed method, you try to fit the change by reconstructing it with a sparsity loss, as shown in Eq. 5. This makes me even more curious about my question in W1.
  4. In Eq. 3, it seems that the concepts should be independent for you to obtain this decomposition, but in the synthetic experiments, you construct dependent Gaussian concepts. For linear Gaussian settings, can you confirm that all your assumptions hold? This is known to be a hard case because the Gaussian density has only constant cross-second-order derivatives, which might affect the validity of assumptions involving “sufficient variability.” Please explain.
  5. Without examples, I am not sure how to interpret the conclusion from your real-world experiments in Figure 2: “Using counterfactual text pairs with changes in object identity, color, material, or size, we evaluate reconstruction error and whether adding a selected contribution shifts the model’s prediction toward the target answer.”
Questions for authors
  1. Your “single-context analysis” is not very clear to me. Even though results such as those for SAEs are presented as case studies and shown case by case, the SAE model itself is trained on a large amount of data. Therefore, does that mean the interpretations are not limited to a single context? Also, regarding your framing of “Compared with concept extraction from a single context, which primarily reveals what concepts are represented, the paired-context perspective in this work provides a complementary view of how concept information changes,” isn’t it the case that previous work tries to understand each dimension of the recovered latent representations by looking at a wide range of contexts to make sure the meaning is somehow stable?
  2. [Minor] Please briefly introduce the term “latent concept posterior” before its first mention on line 55 and again before its mention on line 100. In particular, it’s hard to see why you adopt CAA starting on line 55.
  3. Can you explain more about the assumption that “An observed input context x and its next token y are assumed to be generated conditionally on the latent concept configuration z” on lines 109–110? SAE assumes that the activations (the outputs of LLMs) are generated by z. Here, however, the context should be the input to the LLM. How could it be generated by some interpretable concepts of the LLM? Or do you mean that, here, “context” refers to the same input as in SAE: the LLM-processed input tokens—the activations?
  4. What’s the significance of lines 118–126?
  5. Why are only ConAn and SSAE compared in Figure 2(b)–(d)?
  6. For the real-world experiments, can you also test on SAEBench [2]?
  7. How do you construct the pairs? Are there any hard requirements?
9 comments

Thank you for your positive assessment of our presentation and the connection between our theory and method.

1. Relationship to reference [1]

“You should discuss the relationship between that work and yours”

Thank you for pointing us to reference [1]. Because the review does not include its bibliographic details, we would be grateful if you could provide the title or a link so we can discuss the relationship directly.

2. Why analyze paired changes?

“Why do you want to identify the change in the k-th latent in a pair of contexts, i.e., G_k(x,x'), instead of z itself?”

We focus on GkG_k because we want to extract a concept's change contribution from one context and test whether it produces the same semantic change in another. For example, when “The cup on the table is red” becomes “The cup on the table is blue,” zkz_k describes the color state, whereas GkG_k gives the color contribution to the representation difference that we can use in this intervention. Contributions that are equal at the two endpoints cancel, allowing us to study changes across different objects and backgrounds while retaining the variation in each contribution's direction and magnitude.

As you point out, SAEs are trained on many contexts and their features are interpreted across contexts to establish stable semantic meanings. We use “single-context” to describe the input being decomposed: a conventional SAE decomposes each endpoint representation whereas our method decomposes their difference. Pairing supplies additional information about which contributions change together across related contexts, which we use to identify the individual change contributions.

Sparsity is useful here because each endpoint can contain many concepts even when only a few contributions change within a pair. In Eq. (5), the reconstruction term fits the total representation difference while the sum of contribution norms encourages only a few concept blocks to be active, with each block containing multiple directions so the same concept can change in different ways across pairs. Theorem 3.7 justifies this group penalty at the population level, showing that when the true decomposition satisfies Assumptions 3.2–3.4 and integrability, it uniquely minimizes the expected sum of contribution norms, up to permutation, among candidate decompositions that reconstruct exactly and satisfy local directness and conditional richness.

3. Connecting text, posteriors, and activations

“What’s the significance of lines 118–126?”

Lines 118–126 explain how the generative model of text leads to the activation decomposition in Eq. (3). Here, xx is the input text, f(x)f(x) is its LLM activation, and zz describes the semantic configuration underlying the text and next-token distributions (for example, an object's identity and color, which influence both a scene description and the words likely to follow it). We write this generative model as p(x,y)=∑zp(x∣z)p(y∣z)p(z)p(x,y)=\sum_zp(x\mid z)p(y\mid z)p(z), where x⊥y∣zx\perp y\mid z assumes that the complete semantic configuration accounts for their dependence, and the joint prior p(z)p(z) allows dependencies among concepts and can be non-Gaussian.

The connection to activations goes through the concept posterior p(zk∣x)p(z_k\mid x), which gives the probabilities of the concept's possible states given the text and thus records uncertainty about its state. Writing ℓk(x)=[log⁡p(zk∣x)]zk\ell_k(x)=[\log p(z_k\mid x)]_{z_k}, Liu et al.'s Theorem 2.2 gives f(x)≈A[ℓ1(x);…;ℓK(x)]+bf(x)\approx A[\ell_1(x);\ldots;\ell_K(x)]+b under its output-embedding diversity, information sufficiency, and regularity conditions. We then partition AA by concept and subtract the endpoint representations to cancel the shared bias and obtain Eq. (3) with Gk(x,x′)=Ak[ℓk(x)−ℓk(x′)]G_k(x,x')=A_k[\ell_k(x)-\ell_k(x')], allowing us to interpret an activation difference in terms of changes in concept information.

We have introduced “concept posterior” before its first use in both the introduction and preliminaries. In addition, the reference on line 55 is to existing Concept Component Analysis (ConCA), which extracts posterior components, whereas our Concept Attribution Analysis (ConAn) uses the posterior–representation relationship to learn change contributions.

4. Do dependent Gaussians satisfy the assumptions?

“For linear Gaussian settings, can you confirm that all your assumptions hold?”

Yes, the true generative decomposition satisfies local directness, conditional richness, support intersection, and integrability across all four main synthetic settings. The Gaussian directed acyclic graph (DAG) generates continuous contribution coefficients uu, whose role differs from the semantic variable zz in the text model, and although these coefficients are dependent, neither Eq. (3) nor our identifiability conditions require independent concepts.

We sample a change set SS independently of the coefficients, keep the other coordinates fixed, and jointly resample the selected coordinates from their conditional Gaussian distribution, making the vectors of selected coordinates at the two endpoints conditionally independent given the retained coordinates even though the coefficients within either endpoint remain correlated. For precision matrix Λ≻0\Lambda\succ0, the change obeys ΔuS∣S∼N(0,2ΛSS−1)\Delta u_S\mid S\sim\mathcal N(0,2\Lambda_{SS}^{-1}) and ΔuSc=0\Delta u_{S^c}=0, giving d=BS⋆ΔuSd=B_S^\star\Delta u_S.

Because this covariance is positive definite and BS⋆B_S^\star has full column rank, dd has a density on the full combined change space and assigns probability zero to any finite union of its proper subspaces, establishing conditional richness through the directions covered by the changes even though the Gaussian log density has a constant Hessian.

Proposition E.1 in Appendix E.1 gives the dimension bounds and complete argument for the remaining conditions. Under those bounds, the random dictionary almost surely has full column rank on every union of two maximal supports, giving local directness. Because every support of the maximum size has positive probability and that size is smaller than the concept count, support intersections can isolate every concept, while the Gaussian distribution's finite moments ensure integrability.

5. Pair construction and requirements

“How do you construct the pairs? Are there any hard requirements?”

We construct pairs by editing sentences or matching related contexts, preserving a shared background while allowing several concepts to change. In the real-text experiments, StylePTB pairs original and edited sentences, the controlled object experiment varies identity, color, material, or size, and RAVEL pairs different entities rendered with the same template, whereas our synthetic pairs use the joint conditional resampling described above (Appendix H.1).

Both endpoint representations must use the same coordinate system, which we ensure by using the same frozen backbone, layer, token-position extraction rule, and training-set normalization. Within this common input space, the decomposition is trained on unlabeled pairs in which several unknown concepts can change together, while semantic labels enter only in the separate probes, calibration-based selection, RAVEL attribute masks, and evaluation.

The identifiability requirements apply to the collection of pairs as a whole, requiring that changes explore the relevant spaces and occur in sufficiently diverse combinations to separate the concepts, as guaranteed by our synthetic construction. When conditional richness and local directness hold, for instance, color–material and color–size changes reveal the color space through the intersection of their combined spaces, whereas if color and material always changed together, their separate contributions could not be identified from that pairing pattern.

6. Baseline selection and interpretation of Figure 2

“Why are only ConAn and SSAE compared in Figure 2(b)–(d)?”

We originally chose SSAE for Figure 2(c–d) because, like ConAn, it takes a representation difference as input and directly decomposes the change. A conventional SAE instead decomposes each endpoint state and requires subtracting the two endpoint codes to compare changes, and we have now added an endpoint Top-K SAE baseline to test this alternative directly.

Panel (b) gives a category breakdown for ConAn and SSAE, while the full StylePTB comparison in panel (a) includes both methods together with L1 SAE, Top-K SAE, and ConCA (complete comparison in H.4).

In the cross-context intervention of Figure 2(d), for example, we add a yellow-to-red color contribution recovered from one object to another object's representation and test whether the model then answers “red” about the recipient's color while preserving its answers about the other attributes. With norm-matched contributions, graph-initialized ConAn achieves 93.70% target success and 99.75% preservation, with full preservation and joint-success results reported in H.4, Table 12.

We also compare against endpoint Top-K SAE on the larger Qwen3-4B backbone in H.5, where graph-initialized ConAn and Top-K achieve transfer success rates of 63.41% and 51.45% at the original contribution magnitude, respectively, rising to 94.52% and 79.06% after norm matching.

7. Additional SAEBench-RAVEL evaluation

“can you also test on SAEBench [2]?”

We have added an evaluation adapted from SAEBench's RAVEL subtask in H.6. We chose RAVEL because its source and recipient contexts provide paired inputs for our method while its interventions test whether a target attribute transfers and other attributes are preserved across six tasks asking about attributes of cities and Nobel laureates.

On Qwen3-4B, we train decomposers without attribute labels, freeze them, and learn feature-selection masks under the same supervised objective for every method. At test time, we add the donor–recipient change represented by the selected features to an intermediate-layer representation and let the model continue generating its answer. We then measure whether the target answer changes to the donor value (Cause) and whether, with the same mask, the recipient's answers to separately sampled questions about other attributes are preserved (Isolation), using their arithmetic mean as the combined score. The table reports means with equal weighting across the six tasks on a 0–100 scale, with standard deviations in H.6.

MethodCauseIsolationCombined score
Graph-initialized ConAn36.0874.8055.44
Randomly initialized ConAn35.3375.1455.24
SSAE22.2681.1251.69
Top-K SAE14.5285.3449.93

On these six RAVEL tasks, both ConAn initializations achieve higher target-change rates and combined scores, while Top-K has higher Isolation.


We sincerely thank the reviewer for these constructive comments. We hope that the above clarifications adequately address the concerns and will be taken into consideration when reassessing our work.

Reviewer_YRPqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies pairs of related contexts instead of single contexts. Under the ConCA model, the difference between two LLM representations is a sum of per-concept change contributions. Theorem 3.5 shows that these contributions are identifiable up to a permutation under three assumptions. The key mechanism is that intersections of combined change spaces isolate each concept, so no single-concept pair is needed. Theorem 3.7 shows that group sparsity selects the true decomposition. The method, ConAn, is a block-sparse autoencoder on representation differences with a graph-based initialization. Synthetic experiments confirm the theory. Experiments on Pythia-70M and Qwen3-1.7B show gains over SAE and SSAE baselines.

Critical strengths

The identifiability result is correct and fills a real gap. The proof sketch of Theorem 3.5 is sound, and the argument is elementary. The result removes the need for single-concept pairs.

The synthetic study validates the theory Each assumption is broken in isolation, and the recovery degrades in the way the theory predicts. In particular, bound concept pairs lose individual recovery while their sum stays recoverable, which is exactly what the support intersection condition implies. This design is rare in the SAE literature and makes the theory testable.

Critical weaknesses

The evidence on real LLMs is too thin for the claim of practical utility. The semantic readout is a supervised probe trained per label on the recovered components, which is the type of evidence the introduction describes as unreliable for single-context methods. The intervention study uses one baseline, SSAE, and one model. The models are small, and the results appear only as bars in Figure 2 with no numbers. The abstract claims practical utility, and these experiments support a demonstration at most.

The recovery comes from the initializer, and the theory leaves the initializer uncovered. Figure 1(a) shows that graph and random initialization reconstruct the aggregate change equally well, while only graph initialization recovers the individual contributions. So the objective in Equation 5, which Theorem 3.7 justifies, is insufficient on its own. The initializer is a chain of graph construction, factorization, and subspace clustering with many fixed choices, and Section 4 describes it in one paragraph. The paper presents a principled method, but the part that makes it work is a heuristic. The main text also leaves open how the number and rank of the candidate blocks are chosen when the true values are unknown.

Questions for authors

1.How are the number of candidate blocks and their rank chosen in the synthetic experiments?

2.The candidate decomposition must satisfy local directness and conditional richness. The estimator in Section 4 imposes block orthonormality and group sparsity only. Does the guarantee apply to the output of the estimator?

4 comments

Thank you for your positive assessment of our theoretical results and synthetic experiment design.

1. Motivation and theoretical basis for initialization

“The paper presents a principled method, but the part that makes it work is a heuristic.”

The distinction you draw between identifiability and recovery in practice motivates our initialization. While the theory establishes that a decomposition satisfying its conditions is unique, training still has to find that decomposition through nonconvex optimization of an encoder and dictionary. We describe the method as principled because its heuristic initializer is guided by the combined subspaces and their intersections in the identifiability proof.

We estimate the concept spaces before joint training because early assignment errors in the encoder can distort which subspaces the dictionary learns. For an input difference dd and residual e=∑jBjzj−de=\sum_jB_jz_j-d, a dictionary block has reconstruction gradient 2E[ezj⊤]2\mathbb E[ez_j^\top]. If the encoder assigns a concept change to the wrong block, subsequent updates can therefore shift that block's directions to compensate. In Appendix G.1, we give an explicit example in which a gradient update lowers the training objective while moving a correctly initialized direction away from its true subspace, as errors across different blocks cancel in the total reconstruction. To avoid this drift, we first estimate the concept spaces, fit the encoder with the dictionary fixed, and only then refine the encoder and dictionary jointly.

The initializer exploits concept spaces shared across different combinations of changes. Let UkU_k denote the concept spaces, SS the set of changing concepts, and VT=∑k∈TUkV_T=\sum_{k\in T}U_k a combined space. For any maximal change pattern TT of positive probability, conditional richness and finite second moments give range⁡E[dd⊤∣S=T]=VT\operatorname{range}\mathbb E[dd^\top\mid S=T]=V_T, and local directness together with support intersection further gives

⋂T∈GkVT=Uk.\bigcap_{T\in\mathcal G_k}V_T=U_k. Intersecting the spaces for changes in {1,2}\lbrace 1,2\rbraceand {1,3}\lbrace 1,3\rbrace, for example, isolates the space of concept 1. We therefore construct a similarity graph over whitened directions and apply nonnegative matrix factorization with overlapping assignments so each observation can involve multiple concepts. From these assignments, we estimate candidate directions by moment regression and apply subspace clustering to group directions for the same concept into multidimensional blocks. Finally, we select the dictionary on held-out data using the group-sparse coding objective, following the norm-minimization principle in Theorem 3.7 (Appendices G.2 and D.2 explain how these steps follow from the theory and analyze their estimation errors).

2. What the theory guarantees about the estimator output

“The estimator in Section 4 imposes block orthonormality and group sparsity only. Does the guarantee apply to the output of the estimator?”

Under Theorem 3.5, if the true and learned decompositions both satisfy Assumptions 3.2–3.4 and the output exactly reconstructs the observed representation differences almost surely, the two decompositions have the same number of nonzero concept contributions and agree up to a single global permutation σ\sigma:

K^=K,G^σ(k)=Gkalmost surely.\widehat K=K,\qquad \widehat G_{\sigma(k)}=G_k\quad\text{almost surely}. Each output contribution then corresponds to a true contribution. Theorem 3.7 extends this recovery guarantee to a broader class of candidates that need not satisfy the support intersection condition, showing that when the true decomposition satisfies the assumptions above and has integrable contributions, it can be recovered by minimizing the expected sum of contribution norms over candidates in this class that reconstruct exactly. Our estimator follows this selection principle through the group penalty, while orthonormality within each block gives ∥Bjzj∥2=∥zj∥2\Vert{}B_jz_j\Vert{}_2=\Vert{}z_j\Vert{}_2 and provides a numerically stable basis for its subspace.

Conditional richness. This condition requires enough directional variation under each maximal combination of changes to span its entire combined space, meaning formally that the conditional distribution cannot be concentrated on a finite union of proper subspaces. Candidate decompositions must also satisfy this condition when observations are grouped by which candidate blocks are active. Because the condition is a property of the data distribution rather than individual samples, we use it to estimate combined spaces from observations instead of adding it as a regularizer to the training loss.

Local directness. This condition requires the concept spaces in the relevant change combinations to be linearly independent so their contributions cannot cancel each other. Enforcing it directly in the loss would require checking linear independence across many block combinations at every update. For 32 blocks, enumerating all eight-block combinations alone gives over ten million cases, making these repeated checks intractable at scale. Instead, we estimate the concept spaces during initialization and derive sufficient conditions for preserving local directness during training in Appendix G.3.

Specifically, let BAB_A concatenate the dictionary blocks involved in two relevant maximal change patterns. When the initial matrix BA(0)B_A^{(0)} has full column rank,

σmin⁡(BA(t))≥σmin⁡(BA(0))−∥BA(t)−BA(0)∥op,\sigma_{\min}(B_A^{(t)})\ge\sigma_{\min}(B_A^{(0)})-\Vert{}B_A^{(t)}-B_A^{(0)}\Vert{}_{\mathrm{op}}, so these spaces remain linearly independent as long as the right-hand side stays positive. Holding the dictionary fixed preserves this bound throughout the initial training stage, and local directness is maintained during joint training provided the dictionary updates keep the bound positive for every pair of maximal change patterns used by the output. We give the full derivation in Appendix G.3.

3. Choosing the block count and ranks

“How are the number of candidate blocks and their rank chosen in the synthetic experiments?”

“The main text also leaves open how the number and rank of the candidate blocks are chosen when the true values are unknown.”

In the synthetic experiments, we set the block count and ranks to match the data-generating model. The main experiments use 32 rank-six blocks (total width 192) across representation dimensions 64, 128, and 256 to test 3-fold and 1.5-fold overcomplete settings as well as an undercomplete setting, while reference settings use six rank-two blocks at dimension 10 and 32 rank-three blocks at dimension 64 (Appendix H.1).

When the true block count and ranks are unknown, our theory suggests estimating the combined spaces from the leading eigenvectors of the corresponding second-moment matrices and intersecting them to identify the concepts shared across combinations. Under local directness and support intersection, the inclusion-minimal nonzero intersections are exactly the concept spaces, so their number gives the block count and their dimensions give the block ranks (Lemma C.4). Intuitively, this procedure recovers the minimal subspaces that recur across combinations of changes and cannot be separated further using those combinations.

In noisy data, these intersections must be approximated, and Appendix G.4 describes how to estimate ranks via eigenvalue separation, approximate the intersections, and merge duplicate spaces. Because concepts in real data may admit different interpretations and vary widely in change magnitude, these ambiguities and scale differences, together with the initialization parameters, affect the raw estimates. We therefore suggest using the estimated capacity as a starting point and comparing nearby settings based on their held-out fit, their group-sparse coding objective, and whether they recover similar spaces across subsamples. When held-out fits are comparable, a smaller capacity with more stable spaces is preferable.

4. Additional evidence on real language models

“The evidence on real LLMs is too thin for the claim of practical utility.”

Thank you for this constructive criticism and your suggestions. In response, we have added experiments with a larger language model, another baseline, and interventions followed by generation, and we report the numerical values and standard deviations for Figure 2 in Appendix H.4. Together with our original results, these findings show that the extracted contributions can reliably modify answers about a target attribute and transfer the same semantic change across contexts on real LLMs.

We apply the same supervised probing protocol to our method and all baselines to assess what semantic information can be read from the components, and use interventions to test whether those components causally change the model's answers. In the norm-matched cross-context experiment of Figure 2(d), for instance, graph-initialized ConAn achieves 93.70% target success while preserving 99.75% of unchanged-attribute answers (Appendix H.4, Table 12).

Larger model and additional baseline. We extended cross-context interventions to Qwen3-4B and added a Top-K SAE baseline that encodes the two endpoints separately (Appendix H.5). Graph-initialized ConAn outperforms Top-K under both magnitude choices on the larger model, achieving transfer success rates of 63.41% versus 51.45% at the original magnitude and 94.52% versus 79.06% after scaling the contributions to the norm of the donor's full representation difference.

Generative interventions. We also evaluated our method on RAVEL from SAEBench, whose paired donor and recipient contexts allow us to decompose their representation difference, add the selected contributions at an intermediate layer of Qwen3-4B, and continue generating answers across six attributes of cities and Nobel laureates. Cause measures whether the target answer changes to the donor value, Isolation measures whether answers about other attributes are preserved, and the combined score is their mean.

MethodCause (%)Isolation (%)Combined score
Graph-initialized ConAn36.0874.8055.44
Randomly initialized ConAn35.3375.1455.24
SSAE22.2681.1251.69
Top-K SAE14.5285.3449.93

ConAn achieves higher target-change rates and combined scores than the baselines, whereas Top-K preserves more of the other attribute answers. Full settings and statistics are in Appendix H.6.

Thank the authors for the detailed response. They have addressed my concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive feedback. We appreciate their positive assessments of the following aspects of our work.

Contribution:

“The identifiability result is correct and fills a real gap.” - Reviewer YRPq

“this paper studies an important problem in LLM interpretability.” - Reviewer HsG9

“Identifiability of the change-centric LLM representations is established. The theory and the devised methods are connected.” - Reviewer v9jo

Writing:

“The presentation of the paper is clear.” - Reviewer v9jo

Experiments:

“The synthetic study validates the theory.” “This design is rare in the SAE literature and makes the theory testable.” - Reviewer YRPq

“The synthetic experiment has useful robustness analysis.” - Reviewer HsG9

“Experiments are designed to test violations of the assumptions.” - Reviewer v9jo

In response to the concerns and suggestions raised by each reviewer:

For Reviewer YRPq, we have clarified how the theory guides initialization and added real-LLM experiments (Appendices G and H).

For Reviewer HsG9, we have added the requested component ablations and experiments on a larger LLM (Appendices H.2, H.3, and H.5).

For Reviewer v9jo, we have clarified the paired-context formulation with examples, explained pair construction, and added an endpoint Top-K SAE comparison and an evaluation adapted from SAEBench’s RAVEL task.

Please see our individual responses for the detailed explanations and additional results. We thank all reviewers again for their suggestions and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.