acceptodds
Under review as a conference paper at ICLR 2027

Concept Geometry in Language Models: Cones, Edits, and Compositional Paths

Abstract

A language model is commonly thought to represent a concept as a direction in its hidden states, found by averaging how the state changes when one word of a sentence is swapped. The average hides whether the individual changes agree, and it says nothing about several words changing at once. We keep every change. Repeating a counterfactual edit, such as a change of tense or a synonym for an adjective, across contexts and across the words that carry it gives a cloud of vectors shaped like a cone, with an axis and a width; comparing the axes of different words for the same edit with those of random words in the same position tells whether the model treats the edit as one change or as several. Because an edit's axis barely depends on the sentence it acts on, the sentences reachable by several edits form a lattice, and their combined effect divides among the edits by the Shapley value, each edit's contribution averaged over the orders in which the edits can be applied. The lattice grows exponentially with the number of edits, yet a single path that applies the edits one at a time already estimates this division: apart from sampling noise, its error is the interaction between edits, which averages out over a few paths. Across five models from three families, synonyms of an adjective scatter more than random adjectives in all combinations of model and sentence, while verbs of destruction align more than random verbs in . Shares vary little with the order of the edits (standard deviation at most over all orders of six edits). On a lattice of sentences, at most eight paths, visiting of the sentences, recover every share to within , and the shares predict which edit's removal most weakens a combined steering vector (Spearman ).

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Concept geometry, Linear representation hypothesis, Counterfactual, Cones, Shapley value, Residual stream

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YixNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method for estimating the geometry of Language Models by modeling edits from high-dimensional cones and compositional paths. Using counterfactual edits, the authors build up a lattice of edits that would map to an underlying common geometry learned by the LLM. This geometry could be useful for interpretability applications and steering model behavior.

Critical strengths

The paper evaluates the methodology on 5 different models with a considerable set of examples on a large set of metrics, demonstrating a strong set of results that can be considered trustworthy. The approach for measuring the large-dimensional lattices with the walker formulation is also novel, providing a new way to compute approximate Shapley values in a sensible manner.

Critical weaknesses

There are theoretical errors to be addressed. In Def 2.3, the geometric interpretation of $ s=\mathbb E[\cos(\Delta,\hat{\gamma})]$ as the cos of the cone opening angle is false in general because you this is not a hard bound on how far individual effects can lie from the axis, it is an average directional concentration. Similarly, a general convex cone (Def 2.1) doesn't require one circular axis and opening angle. I think to get a cone one should define a genuine enclosing, like the smallest angle containing 95% of directions.

Also, the sentence "Two sentences that differ in one edit share all other cones, so their difference cancels them and isolates the edit" needs more careful analysis. This sort of behavior is valid for strong additive model, so every concept would contribute exactly the same vector before and after the edit, but this is not necessarily true for neural networks and LLMs even if polysemanticity or superposition hypothesis is disregarded simply because a single word can have different meaning in different contexts, so it would lead to different concepts and one edit could entirely change the understanding of a sentence and "activate" another meaning for a word that was not edited. For example, "The musician examined the organ" and "The surgeon examined the organ"; "The programmer found a bug" and "The entomologist found a bug", etc.

Then, there is a lack of proper comparison and discussion with prior literature. [1], [2], [3] also discuss textual edits as Shapley players and have several intersections with this work. [4], [5], [6] provide the basic tools of analyzing cones and concept geometry beyond individual directions, so it could be relevant for comparison and discussion.

  • [1] Goto, Vasselli, and Watanabe, Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction, ACL 2025
  • [2] Fern and Pope, Text Counterfactuals via Latent Optimization and Shapley-Guided Search, EMNLP 2021
  • [3] Singhvi et al., Using Shapley Interactions to Understand How Models Use Structure, ACL 2025
  • [4] Wollschläger et al., The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence, ICML 2025
  • [5] Banerjee, Dhillon, Ghosh, and Sra, Clustering on the Unit Hypersphere Using von Mises-Fisher Distributions, JMLR 2005
  • [6] Valois et al., Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation, TACL 2025

Finally, the proof of proposition 3.3 seems to be wrong. The usage of the Hanson-Wright inequality does not directly apply to arbitrary jointly sub-Gaussian step vectors. As far as I know, it requires independent components for correlated vectors, but independent samples of a sub-Gaussian vector do not mean independent coordinates. Also, the proof is organized in a way that is not so clear even though it is in the appendices where one would expect the space could be used more freely, so the steps needs a bit more clarity. Please check this issue carefully.

Questions for authors

The interpretation of the results as the defined geometry and cones is weak.

  1. What evidence shows the proposed semantic decomposition is different from a simple projection onto the empirical mean and its orthogonal complement?
  2. Why is context-related variation always orthogonal to the shared direction?
  3. Could the authors compare the suggested cone representation with a low-rank linear or affine model? Singular value spectra (principle angle stability) or angular error could clarify the nature of the variability (directional, low dimensional, etc).
3 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you for recognizing the empirical measurements and the walker evaluation. We response with explicit arguments, additional analyses, and comparisons with the suggested papers.

[W1] Concentration and the angle containing the effects. We will state the geometric quantities separately. For nonzero measured effects with a nonzero mean, let

u_i=\frac{\Delta_i}{\|\Delta_i\|},\qquad s=\frac1n\sum_i\langle u_i,\widehat\gamma\rangle.$$ Here, $s$ measures average directional agreement. We will additionally report $$\theta_{.95}=\inf\\{t:\frac1n\sum_i\mathbf1[\arccos\langle u_i,\hat\gamma\rangle\le t]\ge.95\\}$$ This is the smallest angle **about the reported axis** containing at least 95% of the measured directions. This states both the observed geometry and its behavior on new text. The circular axis-and-angle description will be distinguished from the definition of a general convex cone. The distinction has an established statistical basis: [Banerjee et al. (2005)](https://www.jmlr.org/papers/volume6/banerjee05a/banerjee05a.pdf) treat mean direction and concentration as separate quantities for directional data. We use directly measured edit vectors rather than requiring a fitted directional distribution. **[W2] What does subtracting two representations measure?** We define $$\Delta_v(c)=e(\delta_v x,c)-e(x,c)$$ as the **total representation response to the specified text edit**. This includes changes in the contexts. The organ and bug examples therefore describe interactions that the measurement includes. Our experiments also measure interaction explicitly: the submitted six-edit experiments report a relative interaction magnitude of 0.29–0.54 and mean cross-node direction cosines of 0.87–0.95. Thus directional agreement and interaction magnitude are measured separately. This distinction allows edits to have related directions while their magnitudes depend on preceding edits. **[Q1, Q2] What does the projection describe?** Projection separates each edit effect into a component along its mean direction and a perpendicular remainder. While this decomposition is mathematical, the distribution of effects across contexts and word replacements is measured empirically. Effects can have the same mean direction but differ in their consistency across contexts or replacements. Our experiments test this distinction using same-position, same-part-of-speech word controls (Table 2). Across 15 model–sentence combinations, adjective synonyms show lower replacement agreement than the reference (mean difference −0.12), while action verbs show higher agreement (+0.07). These differences are empirical observations, not consequences of orthogonal projection. Both the parallel and perpendicular components can vary with context. Orthogonality defines the decomposition, but does not determine the observed variation. **Table 2. Agreement between replacements compared with same-position word controls** | Replacement family | Above reference range | Within reference range | Below reference range | Mean difference from reference | | --- | --- | --- | --- | --- | | Adjective synonyms | 0 | 0 | 15 | −0.12 | | Object nouns | 1 | 3 | 11 | −0.07 | | Human → machine subjects | 5 | 9 | 1 | +0.03 | | Action verbs | 14 | 1 | 0 | +0.07 |
AuthorsOpenReview ↗

Author Response (2/3)

We will also make the context interpretation explicit:

a_v(c)=\langle\Delta_v(c),\widehat\gamma_v\rangle,\qquad \langle r_v(c),\widehat\gamma_v\rangle=0.$$ **Both** $a_v(c)$ and $r_v(c)$ can depend on context. Orthogonality defines the two components; their variation is measured from the data. This separation of geometry from functional interpretation is consistent with [Wollschläger et al. (2025)](https://proceedings.mlr.press/v267/wollschlager25a.html), who explicitly distinguish orthogonality from independence under intervention. **[W2] How are interactions handled?** Fix the measured sentences for the allowed edit combinations, let $m_S$ be their mean representations, and let $\mu=m_N-m_\varnothing\ne0$. The game in section 3.2 is $$v(S)=\frac{\langle m_S-m_\varnothing,\mu\rangle}{\|\mu\|^2}.$$ For any allowed order, adding adjacent differences gives $m_N-m_\varnothing$ exactly. Therefore, the path marginals sum to $v(N)-v(\varnothing)=1$, and averaging each edit's marginal over allowed orders gives its Shapley allocation. These are algebraic identities for the measured game and hold when edits interact. The treatment of required edit order follows the framework of [Faigle and Kern (1992)](https://doi.org/10.1007/BF01258278). For example, the interaction between two edits can be measured as $$m_{S\cup\{i,j\}}-m_{S\cup\{i\}}-m_{S\cup\{j\}}+m_S.$$ The Shapley calculation averages marginal contributions in the presence of this interaction. Directional consistency helps explain the observed accuracy of a few paths; it is separate from the identities that define the allocation. This is also complementary to the pairwise linguistic interactions studied by [Singhvi et al. (2025)](https://aclanthology.org/2025.acl-long.1011/). **[W4] Revised proof of Proposition 3.3.** We can address coordinate dependence using the distinct-context estimator described in Proposition 3.3 and Appendix A. For one step $A_i$ and the total effect $B_i$ in context $i$, define $$U_N=\frac1{T(T-1)}\sum_{i\ne j}\langle A_i,B_j\rangle,\qquad U_D=\frac1{T(T-1)}\sum_{i\ne j}\langle B_i,B_j\rangle, \qquad \widehat w_U=U_N/U_D.$$ Independent contexts are sufficient; the step and total effect within one context, and their coordinates, may be dependent. Write $\nu=\mathbb EA$, $\mu=\mathbb EB$, $n=\|\mu\|>0$, and $w=\langle\nu,\mu\rangle/n^2$. With centered variables $a=A-\nu$, $b=B-\mu$, define $z=a-wb$ and $L=\langle\mu,a\rangle+\langle\nu-2w\mu,b\rangle$. We prove $$\operatorname{Var}(U_D)= \frac{4\mu^\top\Sigma_b\mu}{T}+ \frac{2\operatorname{tr}(\Sigma_b^2)}{T(T-1)}=:V_D,$$ $$\operatorname{Var}(U_N-wU_D)\le \frac{\operatorname{Var}(L)}{T}+ \frac{2\operatorname{tr}(\Sigma_z\Sigma_b)}{T(T-1)}=:V_R,$$ where $\Sigma$ denotes covariance. These expressions use finite second moments and allow arbitrary coordinate dependence. They follow by separating the linear sample-mean term from centered products of different contexts, using the order-two U-statistic decomposition of [Hoeffding (1948)](https://doi.org/10.1214/aoms/1177730196). For failure probability $\alpha$, if $d_\alpha=\sqrt{2V_D/\alpha}<n^2$, Chebyshev's inequality and a union bound give, with probability at least $1-\alpha$, $$|\widehat w_U-w|\le \frac{\sqrt{2V_R/\alpha}}{n^2-d_\alpha}.$$ This supplies an explicit bound for the ratio. With covariance operator norms bounded by $\sigma^2$, its terms have orders $1/\sqrt T$ and $\sqrt{\operatorname{tr}\Sigma_b}/T$, with the mean-effect lengths and denominator condition stated in the full proof. The argument uses independence across contexts rather than the independent-coordinate assumption in Theorem 1.1 of [Rudelson and Vershynin (2013)](https://arxiv.org/abs/1306.2872). Across eight experiments, 360 orders, and six shares, the maximum difference from the original readout is only 0.000395 (Table 3), and the shares sum to one numerically. **Table 3. Effect of using inner products from different contexts** | Model | Experiment | Largest change in one path's share | Largest change after averaging all orders | | --- | --- | --- | --- | | pythia-410m | Judicial, natural contexts | 0.000299 | 0.000235 | | pythia-410m | Judicial | 0.000287 | 0.000262 | | pythia-410m | Literary | 0.000145 | 0.000137 | | pythia-410m | Medical | 0.000212 | 0.000143 | | Qwen2.5-1.5B | Judicial, natural contexts | 0.000395 | 0.000305 | | Qwen2.5-1.5B | Judicial | 0.000258 | 0.000196 | | Qwen2.5-1.5B | Literary | 0.000258 | 0.000212 | | Qwen2.5-1.5B | Medical | 0.000358 | 0.000263 | We compared the original method with the distinct-context estimator using saved activations from eight experiments, each covering 48 contexts and all 360 allowed edit orders. The results are nearly identical, with a maximum difference below 0.0004. The estimated contributions also sum to one. This confirms that the choice of estimator has little effect on our experimental results.
AuthorsOpenReview ↗

Author Response (3/3)

[Q3] Comparison with low-rank linear and affine models.

Our angular enclosure describes the spread of edit directions across contexts, while low-rank models examine whether this variation can be approximated by a few directions.

We compared these descriptions using 27 fixed word pairs and 331 test vectors per model, varying only the natural context. All texts from each test source were excluded from fitting.

Table 4. Angular reconstruction error across contexts (degrees; lower is better).

ModelEdit typeMean axis ↓Linear 1 ↓Affine 1 ↓Linear 4 ↓Affine 4 ↓
pythia-410mAdjectives41.843.841.639.139.6
pythia-410mPhrases39.041.937.136.035.9
pythia-410mTense30.331.930.628.529.1
pythia-410mVerbs44.746.644.241.440.5
Qwen2.5-1.5BAdjectives42.845.343.541.742.0
Qwen2.5-1.5BPhrases37.740.736.135.635.4
Qwen2.5-1.5BTense37.039.337.635.335.8
Qwen2.5-1.5BVerbs43.847.543.741.841.2

Mean axis uses the average training effect; linear models fit directions through the origin, while affine models add an offset. Results use layer 6 for Pythia and layer 7 for Qwen. The results show structured variation beyond the mean direction, partially captured by low-rank models.

We further examine this variation through singular value spectra after removing each word pair's mean effect.

Table 5. Structure of context-dependent variation after removing the mean effect.

ModelEdit typeFirst 2 directionsFirst 4 directionsEffective rank
pythia-410mAdjectives68.4%89.7%3.51
pythia-410mPhrases63.3%84.8%4.03
pythia-410mTense69.5%79.6%2.63
pythia-410mVerbs47.1%67.5%6.38
Qwen2.5-1.5BAdjectives68.9%90.2%3.47
Qwen2.5-1.5BPhrases65.8%86.2%3.76
Qwen2.5-1.5BTense66.3%77.7%3.44
Qwen2.5-1.5BVerbs53.4%71.4%5.54

The first four directions explain 67.5%–90.2% of the observed residual variance, revealing structured context-dependent variation. These values describe the observed samples rather than an intrinsic dimensionality. Together, the experiments show that angular enclosures and low-rank approximations offer complementary descriptions of edit-effect geometry.

[W3] Related work and presentation.

We thank the reviewer for the six suggested references and compare them in Table 6. Our contributions are as follows. First, we measure text-edit geometry across contexts and word replacements, distinguishing context concentration from replacement coherence using same-position controls. Second, we study how edits combine, measuring vector interactions and assigning Shapley contributions through a concept network. We validate efficient path-based estimation against an exhaustively evaluated network of 65,536 sentence variants.

Table 6. Connections to the six suggested papers.

ReferenceMain focusDistinction from our work
[1] Goto et al. (2025)Shapley attribution of grammatical edits to sentence quality.We attribute representation changes and study their composition.
[2] Fern and Pope (2021)Shapley-guided counterfactual text search.We measure edit effects and allocate contributions on a sentence network.
[3] Singhvi et al. (2025)Linguistic interactions between input components.We measure vector interactions and compositional attribution across edits.
[4] Wollschläger et al. (2025)Intervention-based refusal directions and concept cones.We analyze context-dependent text-edit effect clouds and their composition.
[5] Banerjee et al. (2005)Directional distributions and concentration.We measure concentration and coherence of text-edit effects empirically.
[6] Valois et al. (2025)Multi-token representation frames and concept-guided generation.We analyze variation across contexts, replacements, and edit combinations.

We will incorporate these comparisons into the related-work section and clarify the distinction between established tools, theoretical assumptions, and empirical findings.

Reviewer_D7tCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how counterfactual edits change internal representations in language models and how individual edits contribute to the overall change. The paper introduces Effect Clouds to analyze representation changes across contexts and word replacements. It also uses Shapley Values to measure the contributions of multiple edits. Experiments show that edit effects are relatively stable across contexts and edit orders. The results also show some correlation between contribution scores and changes in model predictions.

Critical strengths
  1. The paper provides a systematic analysis framework. The paper uses Effect Clouds to analyze how single counterfactual edits change model representations across different contexts and word replacements. It also uses Shapley Values to assign contributions to individual edits when multiple edits are combined. This provides a relatively complete framework for studying concept representations and their composition in language models.

  2. The experiments show that Effect Clouds can capture representation changes under different editing conditions. The experiments show that the same word replacement produces relatively stable directions of representation change across different contexts. Different edit types also show clear differences in their directions. Meanwhile, word replacements within the same edit type show different levels of directional consistency. For example, action verb replacements tend to have more consistent directions, while adjective synonym replacements are more scattered. These findings suggest that Effect Clouds can provide a more detailed view of how counterfactual edits affect internal model representations.

Critical weaknesses
  1. The experiments have limited text diversity and edit complexity. The main experiments use only three base sentences from the judicial, medical, and literary domains. The paper tests different contexts and word replacements across five models. It also uses one additional long sentence for an experiment with 16 combined edits. However, the experiments still focus on a small number of manually designed sentences and local word replacements. There is limited evaluation on more natural texts and complex edits. In addition, the contexts are manually written rather than sampled from real-world corpora. Therefore, the current results do not fully show whether the method can generalize to more diverse language expressions, complex sentence structures, or edits involving multiple words or phrases.

  2. The paper lacks sufficient evaluation of relative errors for low-contribution edits. The paper evaluates the stability of the Walker method using small standard deviations across edit orders. It also compares Walker estimates with exact Shapley Values. However, the experiments mainly focus on absolute standard deviations and absolute estimation errors. There is little analysis of relative errors. According to Table 2, the standard deviation of some low-contribution edits reaches 40%–50% of their mean contribution scores. This indicates relatively large variation across different edit orders. Therefore, although a small number of Walkers can achieve low absolute errors, it remains unclear whether their relative estimation accuracy is also reliable for low-contribution edits. Further evaluation of relative errors under different numbers of Walkers would provide a more complete analysis of the method's stability.

  3. The paper lacks convincing evidence of its practical interpretability value. The paper uses Activation Steering experiments to study the relationship between Shapley contribution scores and changes in the model's next-token prediction distribution after removing individual edit vectors. However, the main evaluation uses Spearman correlation to measure whether the two quantities have similar rankings. In other words, it tests whether edits with larger contribution scores also tend to cause larger drops in steering effectiveness when removed. This evaluation does not fully show whether the contribution scores can explain probability changes for specific tokens. It also does not show how these scores can help us understand model decisions in real tasks. In addition, the correlation between Shapley Shares and steering effects is 0.78, while a simple vector-norm baseline already achieves 0.74. The improvement is relatively small. More convincing applications are needed, such as explaining changes in model answers in question-answering tasks or analyzing how different edits affect the final generated text. These evaluations would help show whether the proposed method provides more useful explanations than simple baselines.

Questions for authors

The paper could further clarify how edit vectors of different types and their Shapley contribution scores can be used together to explain specific model predictions. It would also be helpful to explain what additional insights or practical applications these results can provide beyond ranking the importance of different edits.

2 comments
AuthorsOpenReview ↗

Author Response (1/2)

[W1] Text diversity. We tested natural contexts from encyclopedia articles, news, and movie reviews. With the same judicial sentence and edits, the mean cosine between directions at different edit combinations remained about 0.92, and every share changed by less than 0.01 on both models. We also tested 220 edits within natural sentences, including phrase replacements. Tables 1 and 2 answer two complementary questions: whether the same edits transfer across contexts, and whether their directions agree across natural sentences.

Table 1. Changing the surrounding context to natural text

ModelDirection agreement: original contextsDirection agreement: natural contextsLargest change in a share
pythia-410m0.92620.92260.0091
Qwen2.5-1.5B0.92340.92690.0088

We keep the judicial sentence and its six edits fixed. We place 48 natural texts before it: 16 from encyclopedia articles, 16 from news, and 16 from movie reviews. We evaluate all 48 allowed sentence versions and all 360 allowed edit orders. Direction agreement is the average cosine between the same edit's directions at different sentence versions; a value closer to 1 means closer agreement. The last column is the largest absolute change among the six Shapley shares. This test changes the context while retaining the original target sentence.

Table 2. Do edit directions agree across natural sentences?

Edit typeOriginal sentences testedEdited sentences testedPythia mean cosine [95% confidence interval]Qwen mean cosine [95% confidence interval]
Tense changes59590.857 [0.845, 0.870]0.792 [0.775, 0.806]
Adjective replacements551650.711 [0.676, 0.742]0.710 [0.684, 0.736]
Verb replacements481410.673 [0.635, 0.708]0.688 [0.657, 0.717]
Phrase replacements50500.773 [0.752, 0.791]0.781 [0.764, 0.797]

A cosine closer to 1 means closer agreement. The average direction excludes texts from the same recorded source as the sentence being tested. We use layer 6 of 24 for Pythia and layer 7 of 28 for Qwen. Of 220 selected edits, 212 had enough other texts for this comparison. For example, 55 sentences with three adjective replacements each give 165 edited sentences.

To estimate uncertainty, we recalculate the mean cosine 1,000 times. Each time, we randomly select articles or text entries, allowing the same source to be selected more than once. All results from a selected source stay together. The brackets contain the middle 95% of these means. Sources are articles for encyclopedia text and separate dataset entries for news and reviews. Replacements follow fixed rules supplied with the data; author review of grammar and meaning is the next validation step.

[W2] Accuracy for small contributions. We added a direct evaluation of relative accuracy. We rebuilt the three six-edit experiments on two models and calculated the exact shares from all 360 allowed edit orders. We then sampled paths at random and compared their estimates with those exact values.

The new evaluation includes shares as small as 0.027. With 64 paths, all six experiments achieve a maximum relative error below 10% in at least 95% of the 2,000 sampling trials (Table 3). The full curves show how accuracy changes with the number of paths. This gives a practical way to select the number of paths for a requested absolute or relative accuracy.

Table 3. Absolute and relative accuracy with 64 random paths

ModelSentence typeSmallest exact shareMaximum absolute errorMaximum relative error
pythia-410mjudicial0.0860.004744.1%
pythia-410mmedical0.0600.009984.0%
pythia-410mliterary0.0640.005934.0%
Qwen2.5-1.5Bjudicial0.0590.003604.0%
Qwen2.5-1.5Bmedical0.0270.006356.2%
Qwen2.5-1.5Bliterary0.0460.005829.3%

For each experiment, we sample 64 allowed paths at random and average their estimates. We repeat this 2,000 times, recording the largest error among the six edits in each trial. The table reports the value below which 95% of trials fall. Relative error is absolute error divided by the exact share. Exact values use all 360 allowed orders and the same 48 contexts. The full calculation record includes all tested path counts from 1 to 128. The table concerns relative accuracy in these six-edit experiments; the submitted sixteen-edit result uses its stated absolute-error target.

AuthorsOpenReview ↗

Author Response (2/2)

[W3, Q2] What do the shares add beyond vector length? The additional information is direction relative to the combined change. Two edit vectors can have the same length while pointing in different directions; their shares distinguish these cases. Shares can also describe an edit that opposes the combined change. The full set of edit effects measures consistency across contexts and word choices.

The submitted steering experiment provides a direct comparison with vector length (Table 4). It also reports the relationship after controlling for the other predictor, making the contribution of directional information explicit.

Table 4. Steering comparison reported in the submission

MeasurementShapley shareVector length
Rank correlation with the effect of removing an edit0.780.74
Partial rank correlation after controlling for the other predictor0.400.01

Source: section 4.4 of the submitted paper, using 90 edits across 15 experiments. The first row compares each predictor with the effect of removing an edit vector from the combined steering vector. The second row measures the relationship after controlling for the other predictor. These are the submitted steering results; the additional complete-answer calculations have their own explicitly defined score.

[W3, Q1, Q2] How can the calculation explain a specific answer? We apply the same averaging over edit orders to the score log p(B) − log p(A), where A is the answer before editing and B is the answer supported by the edited passage. We use the probabilities of complete answers and calculate each edit's Shapley contribution to this score. These answer-score contributions and the representation shares have distinct, explicitly defined targets.

We implemented this calculation on 64 questions from 39 SQuAD articles, using four edits and all 16 combinations per question. Table 5 illustrates the result. Changing the named owner accounts for 40.661 of the measured 40.749 change in the answer score, while the three tense edits have contributions close to zero. The generated answer changes from NDS to RVT, matching the hypothetical name replacement in the passage. This explains the magnitude and sign of the contribution to a specific answer, in addition to its rank.

Table 5. An example explaining the probability of a specific answer

EditReplacementContribution to the answer score
1NDS → RVT+40.6607
2are → were-0.0306
3has → had+0.0334
4are → were+0.0852
TotalAll four edits+40.7487

Illustrative Qwen example: “Who is VideoGuard owned by?” The original passage says NDS; the experiment makes the hypothetical name replacement NDS → RVT. The model generates NDS before editing and RVT after all edits. The score is log p(RVT) − log p(NDS), using complete answer probabilities. Values are exact Shapley contributions to this answer score, calculated from all 16 combinations of four edits. The two are → were edits occur at different positions. The contributions sum to the measured score change. This table illustrates the answer-score calculation; representation shares use the separate representation-change target. The full set of 64 question examples is saved with the measurements.

Reviewer_JQoqOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper, “Concept Geometry in Language Models: Cones, Edits, and Compositional Paths,” investigates how counterfactual lexical edits are represented in the residual streams of language models and how the effects of multiple edits combine. Rather than representing an edit solely by its average activation difference, the authors characterize its distribution of effects across contexts and lexical realizations as a geometric cloud, summarized by a directional axis, concentration, and coherence. The empirical results suggest that different lexical edit families exhibit distinct geometric properties, that edit effects are approximately directionally consistent across contexts and lattice nodes, and that a small number of paths can approximate Shapley shares in a much larger lattice. The reported correlation between these shares and the behavioral impact of removing individual steering vectors provides an additional validation of the proposed attribution method. The paper offers a coherent framework for connecting representation geometry, counterfactual interventions, and compositional attribution. However, its conclusions about the generality of the observed geometry and the interpretation of Shapley shares as meaningful concept contributions require more careful qualification, given the restricted experimental coverage and the assumptions underlying the proposed analysis.

Critical strengths
  1. A useful unification of representation geometry and compositional attribution. The paper goes beyond treating a concept as a single mean activation direction. Its distinction between context concentration and coherence across lexical realizations is meaningful: two sets of edits may each produce consistent effects across contexts while differing in whether their individual realizations share a common direction.

  2. The walker estimator is evaluated against exhaustive calculations, including a large lattice. The empirical validation is stronger than simply showing that an approximation produces plausible shares. For six edits, the authors enumerate all 48 well-formed lattice nodes and evaluate all 360 admissible orderings. For sixteen edits, they evaluate all 65,536 sentence variants and compare the walker estimates against exact Shapley shares. The paper reports that no more than eight walkers are required to recover every share within 0.02 on the largest lattice for the most demanding of the three evaluated models.

Critical weaknesses
  1. The paper's central interpretation depends on the claim that an edit's direction is approximately invariant across contexts and lattice nodes. The reported cross-node cosine values of 0.87–0.95 are encouraging, but they are measured on a small, deliberately constructed set of sentences. The authors themselves acknowledge that the contexts were written for the study rather than sampled from a corpus. Consequently, it remains unclear whether the observed consistency is a general property of model representations or a property of the chosen templates and edit families.

  2. The interpretation of Shapley shares as meaningful concept contributions depends on a specific projection and a restricted behavioral validation. The steering experiment provides some supporting evidence, but its scope is limited. It evaluates changes in next-token log-probabilities at a selected model layer rather than effects on generated text, task performance, or downstream behavior. A stronger distinction between representation-level attribution and behavioral importance is needed. Without broader validation, the paper risks overinterpreting a useful geometric decomposition as a general account of how concepts contribute to model outputs.

Questions for authors
  1. How well do the proposed geometric properties generalize beyond the constructed templates?

  2. What exactly do the Shapley shares explain about model behavior?

  3. How sensitive are the results to the choice of representation layer and the construction of the edit families?

3 comments
AuthorsOpenReview ↗

Author Response (1/2)

[W1, Q1] Do the results extend beyond the written examples? We first kept the judicial sentence and its six edits fixed, but replaced the surrounding context with text from encyclopedia articles, news, and movie reviews. The directions remained consistent across the different combinations of edits. Every Shapley share changed by less than 0.01 on both models (Table 1).

Table 1. Changing the surrounding context to natural text

ModelDirection agreement: original contextsDirection agreement: natural contextsLargest change in a share
pythia-410m0.92620.92260.0091
Qwen2.5-1.5B0.92340.92690.0088

We keep the judicial sentence and its six edits fixed. We place 48 natural texts before it: 16 from encyclopedia articles, 16 from news, and 16 from movie reviews. We evaluate all 48 allowed sentence versions and all 360 allowed edit orders. Direction agreement is the average cosine between the same edit's directions at different sentence versions; a value closer to 1 means closer agreement. The last column is the largest absolute change among the six Shapley shares. This test changes the context while retaining the original target sentence.

We then tested edits within natural sentences. For each word replacement, we estimated its average direction from other texts and compared it with the direction in the sentence being tested. Table 2 gives mean cosines of 0.673–0.857 across four edit types and two models. These measurements extend the analysis to natural sentences and phrase edits, and show how consistency depends on the edit being studied.

Table 2. Do edit directions agree across natural sentences?

Edit typeOriginal sentences testedEdited sentences testedPythia mean cosine [95% confidence interval]Qwen mean cosine [95% confidence interval]
Tense changes59590.857 [0.845, 0.870]0.792 [0.775, 0.806]
Adjective replacements551650.711 [0.676, 0.742]0.710 [0.684, 0.736]
Verb replacements481410.673 [0.635, 0.708]0.688 [0.657, 0.717]
Phrase replacements50500.773 [0.752, 0.791]0.781 [0.764, 0.797]

A cosine closer to 1 means closer agreement. The average direction excludes texts from the same recorded source as the sentence being tested. We use layer 6 of 24 for Pythia and layer 7 of 28 for Qwen. Of 220 selected edits, 212 had enough other texts for this comparison. For example, 55 sentences with three adjective replacements each give 165 edited sentences.

To estimate uncertainty, we recalculate the mean cosine 1,000 times. Each time, we randomly select articles or text entries, allowing the same source to be selected more than once. All results from a selected source stay together. The brackets contain the middle 95% of these means. Sources are articles for encyclopedia text and separate dataset entries for news and reviews. Replacements follow fixed rules supplied with the data; author review of grammar and meaning is the next validation step.

[W2, Q2] What do the shares explain about behavior? Our shares divide the change in a sentence's representation among its edits. They measure each edit's contribution along the average direction from the original sentence to the fully edited sentence. A large share identifies an edit that contributes strongly to this change; a negative share identifies an edit that acts against it.

The submitted steering experiment connects these shares to the effect of removing an edit vector. Table 3 gives the comparison with vector length, including the relationship after controlling for the other predictor.

AuthorsOpenReview ↗

Author Response (2/2)

Table 3. Steering comparison reported in the submission

MeasurementShapley shareVector length
Rank correlation with the effect of removing an edit0.780.74
Partial rank correlation after controlling for the other predictor0.400.01

Source: section 4.4 of the submitted paper, using 90 edits across 15 experiments. The first row compares each predictor with the effect of removing an edit vector from the combined steering vector. The second row measures the relationship after controlling for the other predictor. These are the submitted steering results; the additional complete-answer calculations have their own explicitly defined score.

For a specific answer, the same averaging over edit orders can be applied directly to the score log p(B) − log p(A), where A is the original answer and B is the answer supported by the edited passage. We use the probabilities of the complete answers. We implemented this calculation on 64 questions, evaluating all 16 combinations of four edits. Table 4 gives an example: a hypothetical name replacement changes the generated answer from NDS to RVT, and its contribution accounts for 40.661 of the measured 40.749 change in this answer score. Representation shares and answer-score contributions each have an explicitly defined target.

Table 4. An example explaining the probability of a specific answer

EditReplacementContribution to the answer score
1NDS → RVT+40.6607
2are → were-0.0306
3has → had+0.0334
4are → were+0.0852
TotalAll four edits+40.7487

Illustrative Qwen example: “Who is VideoGuard owned by?” The original passage says NDS; the experiment makes the hypothetical name replacement NDS → RVT. The model generates NDS before editing and RVT after all edits. The score is log p(RVT) − log p(NDS), using complete answer probabilities. Values are exact Shapley contributions to this answer score, calculated from all 16 combinations of four edits. The two are → were edits occur at different positions. The contributions sum to the measured score change. This table illustrates the answer-score calculation; representation shares use the separate representation-change target. The full set of 64 question examples is saved with the measurements.

[Q3] How do layer choice and word replacements affect the results? Appendix C of the submitted paper reports results at every layer. The layer sweeps identify the early layers where edit directions agree most closely across sentences. We use layer 6 of 24 for Pythia and layer 7 of 28 for Qwen in Table 2, giving a concrete and reproducible layer choice. For the word replacements, we report the words in each edit type, measure agreement between their directions, and compare them with unrelated words placed in the same position. Appendix D of the paper also checks replacements that keep the same number of tokens. The layer sweeps and these replacement checks make both choices explicit.

These results strengthen the evidence for transfer across contexts. The method measures how reliable an edit direction is and how edits contribute to a specified change in the model's representation.

Thanks for the rebuttal . I clears some of my doubts and I increase my confidence score and I will keep my score of 3 .

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.