- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses a relevant problem: linear concept representations may not capture all the structures present in LLM activations. Exploring nonlinear representations is worthwhile. Methodology: They first use UMAP to reduce activation vectors to a 50-dimensional space, followed by HDBSCAN to discover clusters, and fuzzy memberships to describe how strongly each token belongs to each concept. Next, they introduce Concept-Based Alignment (CBA), a score comparing the relationships among concepts discovered independently in different representations. Authors conduct experiments and present results answering many questions such as: 1) Can nonlinear concepts be reliably discovered in LLM activations? 2) Can CBA compare representations more informatively than linear alignment methods? 3) How does concept structure change across transformer layers? 4) How does the composition of concepts change with layer depth? 5) Do different languages share the same internal concepts? 6) Do models of different sizes or model families share concepts? 7) How does fine-tuning change concept organization?
Post Rebuttal: - Authors have made useful improvements in clarifying the geometric definition of concepts and distinguishing density-based clustering quality from semantic validity. However, the central concern that UMAP–HDBSCAN may produce clustering artifacts rather than intrinsically nonlinear concept manifolds remains unresolved, as no comparison with linear concept representations or validation in the original activation space is provided.
- The additional robustness experiments are encouraging but insufficient to establish general stability: the three-seed analysis reports a mean same-layer CBA of 0.8571, while the threshold-sensitivity experiments themselves demonstrate substantial dependence on clustering parameters, and robustness to alternative UMAP neighborhood settings and extraction corpora remains unestablished.
- The new semantic validation using two LLM judges is a positive addition, but it does not provide independent human validation, and the reported AUROC of 0.86 for the normalized match margin is particularly concerning because this measure was introduced post hoc after observing the limitations of the original matching score. Therefore, the evidence does not yet establish that CBA reliably identifies semantically equivalent concepts.
- In my opinion, the second annotator experiment exposes an important weakness rather than fully resolving it. Cohen's kappa of 0.622 for concept type is moderate, whereas the value of 0.130 for orientation indicates very poor agreement, making the claims concerning the evolution of input/output orientation across layers particularly uncertain.
- The additional examples of previously unclustered tokens recovering into smaller clusters are useful, but the authors still do not establish the semantic coherence of the discovered concepts or quantify how many noise assignments correspond to meaningful rare concepts, leaving the interpretation of coverage and concept quality limited.
- Several broader conclusions remain stronger than the experimental evidence supports, particularly interpreting alignment blocks as meaningful representational organization, attributing multilingual differences to training rather than confounded model-family differences, and inferring that post-training largely preserves pretrained concepts from high CBA scores without corresponding behavioural or functional validation.
Overall, I acknowledge the authors' efforts but do not consider the fundamental concerns about the validity of nonlinear concept discovery and the semantic interpretation of CBA adequately resolved. Hence, I would keep my original score.
Critical strengths
Strengths are listed below:
-
The paper addresses a relevant problem: linear concept representations may not capture all the structures present in LLM activations. Exploring nonlinear representations is worthwhile.
-
Authors provide a pipeline using UMAP, HDBSCAN, fuzzy concept memberships, and CBA. They report important implementation details, including the use of 100,000 token positions, 50-dimensional UMAP embeddings, and fixed clustering parameters.
-
The paper evaluates several aspects of representation structure: layer depth, concept composition, multilingual sharing, model scale, cross-family alignment, and training-stage changes.
-
Result 0 informs median coverage to be 0.4469 and median DBCV to be 0.2743. It states that the embedding reconstruction error decreases with layer depth. These measurements help in evaluating the behavior of the pipeline, although they do not establish semantic concept quality, in my opinion.
-
Result 1 compares CBA against PCA, linear CKA, and RBF CKA.
-
Result 2 identifies nonuniform patterns in layer-alignment matrices, including diagonal bands and blocks.
-
Result 3 attempts to characterize discovered clusters instead of treating all clusters as meaningful without inspection.
-
Result 4 includes shuffled English–Mandarin sentence pairs to explore whether different languages share the same internal concepts.
-
Result 5 compares models at different scales within the Qwen family and includes a cross-family comparison with Llama.
-
Result 6 examines several checkpoints from one training pipeline, allowing a detailed comparison of SFT, DPO, and RLVR.
Critical weaknesses
Weaknesses:
-
Result 0 does not establish that the discovered clusters are genuine concepts. The reported median coverage of 0.4469 means that about 44.69% of tokens are assigned non-noise memberships under the chosen procedure. It does not mean that 44.69% of tokens have been assigned to valid semantic concepts. Likewise, DBCV measures density-based separation, not conceptual coherence.
-
The use of HDBSCAN's noise assignments is a major concern. Sparse clusters need not be noise: they may correspond to rare concepts, infrequent linguistic constructions, named entities, or specialized knowledge. Conversely, a dense cluster can group tokens for reasons that are not semantically meaningful. Without qualitative inspection of both clustered and unclustered tokens, the reported coverage is an incomplete measure of concept discovery quality.
-
The fixed minimum cluster size and minimum-samples threshold of 50 may systematically exclude rare concepts. The same settings across layers and models improve procedural consistency, but do not guarantee that the method is equally suitable for representations with different density structures. Result 0 therefore does not justify the broad claim that the discovery step is technically valid across all the evaluated layers.
-
The paper's central claim concerns nonlinear concepts, but the use of nonlinear dimensionality reduction followed by density-based clustering does not establish that the underlying concepts are intrinsically nonlinear. UMAP can create or distort local geometric structure, and HDBSCAN can cluster that structure. The resulting clusters could reflect nonlinear concepts, ordinary clusters made easier to separate by the embedding, or artifacts of the embedding and density thresholds. The paper needs evidence that the discovered concepts cannot be represented adequately by linear directions or subspaces.
-
The reconstruction quality score in Result 0 does not resolve this issue. Lower NRMSE in later layers is evidence about the reduced embedding's preservation of pairwise distances under the chosen metric. It is not evidence that the clusters are semantically valid or that the underlying concepts are nonlinear. The relationship between the reported reconstruction error, the original activation geometry, and the validity of the extracted concepts needs to be made clearer.
-
The fixed UMAP seed and clustering profile do not establish robustness. The paper does not show whether the discovered concepts, coverage, DBCV, and CBA scores remain stable across random seeds, neighborhood sizes, minimum cluster sizes, sampling choices, and extraction corpora. A single reproducible run is not the same as a robust result.
-
Result 1 does not demonstrate that CBA measures concept alignment more accurately than the baselines. The neighboring-layer test reports a Neighbor z-score of 2.515 for CBA, compared with 1.714 for RBF CKA, 1.867 for PCA, and 1.287 for linear CKA. This establishes a stronger neighboring-layer signal under the authors' tests. However, the assumption that adjacent layers should be most similar is a plausibility check, not ground truth for concept correspondence. A method that creates higher alignment patterns is not necessarily a more accurate method.
-
The nearest-neighbor scores in Result 1: CBA achieves 0.997, RBF CKA 1.000, PCA 0.983, and linear CKA 0.978. All methods perform well on this test. The differences in Neighbor z-score are interesting, but the paper does not show that the additional structure revealed by CBA corresponds to correct or useful concept matches.
-
CBA measures similarity between pairwise membership relations, not semantic identity between concepts. A score close to one means that two clustering methods induce similar pairwise membership similarities on the evaluated observations. It does not establish that the concepts have the same meaning. Different semantic concepts can organize the sampled tokens similarly, while the same underlying concept can receive different cluster boundaries and membership strengths. This is a central limitation of the interpretation of every later result.
-
Result 2 treats diagonal bands and alignment blocks as evidence of recurring representational organization. These patterns are descriptive, but they do not establish that the blocks correspond to distinct computational stages or functional modules. They could partly reflect smooth changes between neighboring layers, the clustering procedure, or common properties of the input observations. The paper needs stronger null models and tests of whether the blocks are stable and functionally meaningful.
-
Result 3 relies on LLM-generated concept labels. The reported aggregate annotations show that semantic and syntactic concepts are roughly balanced in Llama, at 45.7% and 45.1%, while Qwen has 39.8% semantic and 53.2% syntactic concepts. The authors also report that input-oriented concepts dominate both models. These are results of the annotation procedure, but the paper does not establish the accuracy of the labels through independent human annotation, inter-annotator agreement, or a systematic analysis of labeling errors. The interpretation of concept composition therefore remains uncertain.
-
Result 3 also requires careful interpretation of output orientation. The paper reports no purely output-oriented concepts in the two models summarized in Table 5, while concepts classified as having both input and output evidence increase in later layers. This supports a possible increase in joint input/output evidence under the labeling procedure, not the stronger claim that purely output-oriented concepts emerge toward the final layers.
-
Result 4 does not establish that the measured multilingual sharing is semantic sharing. The mean excess shared-concept mass is 0.0009 for GPT-2, 0.0242 for Llama, and 0.0392–0.0487 for Qwen. The shuffled-pair control is useful, but excess overlap can still reflect shared syntax, punctuation, sentence structure, token frequency, or other surface regularities. The paper itself notes that much of the excess remains unlabeled. The conclusion that multilingual concept sharing is training-dependent is plausible, but the cause and semantic content of that sharing are not isolated.
-
Result 4 also confounds training with model family and other differences. GPT-2, Llama, and Qwen differ in architecture, tokenizer, training data, and training procedures. The observed differences cannot be attributed exclusively to multilingual training. The study of English and Mandarin is informative, but insufficient to establish a general claim about multilingual representations.
-
Result 5 uses only 1,024 aligned observations per cross-model comparison. Character-span alignment is a reasonable way to handle different tokenizers, but mean-pooling memberships across source tokens can change the representation being compared. The reported stronger alignment among Qwen models than between Qwen and Llama may reflect shared training lineage, tokenization, architecture, or data, rather than shared semantic concepts. The quantitative patterns do not establish that models from the same family encode the same concepts.
-
Result 5 also does not demonstrate the general claim that corresponding blocks are functionally similar across model families. A weak diagonal or middle-layer block in a CBA matrix is evidence of structural correspondence under the metric, not proof of a shared computational role. The conclusion should remain descriptive unless independently validated.
-
Result 6 reports DPO–RLVR CBA scores around 0.85–0.86 across most layers and alignment above 0.8 across training-stage comparisons. These are high structural similarity scores under CBA, but they do not show that most concepts were formed during pretraining or that post-training only moderately changes the model's behavior. Substantial behavioral changes can occur without large changes in aggregate representation geometry, and conversely geometric changes need not imply behavioral changes.
-
Result 6 studies a single training pipeline. The larger shift between the base model and SFT, and the higher alignment between DPO and RLVR, are useful observations for Tulu-3, but do not establish a general pattern across fine-tuning methods or models. The interpretation of the layer-14-to-15 drop as especially important for post-training behavior also goes beyond the evidence unless linked to a specific behavioral or functional test.
Overall, the paper's broad scientific conclusions depend on three unverified assumptions: 1) that the non-linear concepts are extracted using the proposed pipeline, 2) that the extracted clusters are (semantically) meaningful concepts, and 3) that CBA scores reliably reflect semantic concept correspondence. The current experiments do not adequately validate these assumptions. This weakens the interpretation of results and claims presented in the paper.
Questions for authors
Questions for the authors:
-
In Result 0, how do the authors distinguish genuine noise from rare but meaningful concepts? Can they provide qualitative examples of low-density or small clusters and of tokens classified as noise, and evaluate whether the latter contain coherent concepts?
-
Can the authors evaluate concept coverage using more than the non-noise token fraction? For example, can they report human-judged concept coherence, concept diversity, and the proportion of noise tokens that appear to belong to meaningful rare concepts?
-
Given the fixed HDBSCAN minimum cluster size and minimum-samples threshold of 50, how sensitive are the number and quality of discovered concepts to these thresholds? Do the main findings persist when rare concepts are allowed?
-
What evidence establishes that the discovered concepts are intrinsically nonlinear rather than artifacts of nonlinear dimensionality reduction and density-based clustering?
-
Can the authors show that the clusters found after UMAP are stable in the original activation space? For example, do the clusters remain coherent under alternative embeddings or direct neighborhood and density analyses in the original space?
-
How much do the discovered concepts and CBA scores change across UMAP seeds, neighborhood sizes, HDBSCAN thresholds, and token samples? Can the authors report repeated-run variability for the principal results?
-
In Result 1, why should a stronger neighboring-layer signal be interpreted as better concept alignment? Can the authors evaluate CBA on synthetic or controlled representations with known correspondences, including cases where concepts are preserved, permuted, split, merged, or corrupted?
-
Can the authors demonstrate that CBA is more accurate than RBF CKA on a task with known ground truth, rather than showing only that CBA produces more pronounced alignment structure? How do they determine that the additional structures revealed by CBA are correct?
-
How should a reader interpret a high CBA score when the method compares pairwise membership relations rather than concept meanings? Can the authors provide a direct evaluation of concept pairs with high CBA-based correspondence to determine whether independent annotators judge them to represent the same or similar concepts?
-
In Result 2, how do the authors distinguish genuine functional blocks from visual patterns caused by smooth layer-to-layer changes, clustering artifacts, or the selected input corpus? Do the blocks persist across seeds, datasets, and reasonable clustering settings?
-
Can the authors connect the alignment blocks to independent measures of function, such as task-specific behavior, controlled interventions, or activation ablations? If not, should the blocks be described as structural patterns rather than functional stages?
-
In Result 3, what is the accuracy of the LLM-generated concept labels? Can the authors validate a representative sample with independent human annotators, report inter-annotator agreement, and quantify agreement between the human and LLM labels?
-
In Result 4, how much of the excess shared-concept mass reflects semantic correspondence rather than syntax, punctuation, token frequency, or sentence structure? Can the authors use controls that preserve these properties while disrupting semantic correspondence?
-
Since the multilingual comparison includes only English and Mandarin and contrasts different model families, how do the authors separate the effect of multilingual training from architecture, tokenizer, data, and other training differences? Can the finding be replicated across additional language pairs and models with more controlled training comparisons? Choosing only two languages does not establish the generalized claim.
-
Can the authors demonstrate that the high-alignment regions between Qwen models correspond to independently validated shared concepts, rather than simply similar clustering patterns? What evidence supports the claim that the blocks represent corresponding functions across model families?
-
In Result 6, what justifies interpreting CBA values around 0.85–0.86 as evidence that RLVR largely preserves concepts established during DPO? Can the authors test whether these scores track independently measured behavioral changes across the training stages?
2 comments
Reply 2
HDBSCAN threshold and rare concepts: We added a controlled threshold-sensitivity analysis using fixed paper-run embeddings and token reservoirs, covering early, middle and final layers of all five main checkpoints. Lowering minimum cluster size from 50 to 25 gives mean CBA 0.920 against the original memberships. Lowering minimum samples from 50 to 20 gives 0.874. Lowering both gives 0.800 and increases the mean concept count from 341 to 741. All 45 perturbed comparisons exceed their corresponding row-shuffled controls. We report all configurations, showing both a sensitivity to the selected parameters but also preserved structural agreement between the different runs. We fully share the reviewer‘s assessment that tokens currently designated as noise (given the assumption about the geometric structure of the feature space is correct) would be assigned to concepts discovered in concept extraction runs on larger corpora. Therefore, we do not see a distinction between genuine noise tokens and rare but undiscovered concept tokens.
Meaning of noise assignments: We agree that density-based noise can contain meaningful structures. With threshold 25/20, an average of 24.4% of originally unclustered observations become clustered. For example, a recovered 42-member Qwen-32B cluster contains fossil/anatomy terms and dinosaur-fossil contexts. All of its member tokens were previously noice. We added this example and clarify that coverage measures density-based assignment, rather than the proportion of observations belonging to semantically valid concepts.
Annotation sensitivity: We repeated R3 with GPT-6 Luna using identical prompts and evidence. Pooled Cohen’s kappa is 0.622 for concept type and 0.130 for orientation. The broad depth-dependent trends persist, while individual orientation assignments are sensitive to the annotator. The new results and plots are included in the appendix.
Reply
We thank the reviewer for the detailed comments. Due to the large number of identified weaknesses and related questions, we organized them into categories but will refrain from explicitly referencing every single issue/question. Several concerns concern the distinction between geometric structure, semantic interpretation, and functional significance. We have clarified these distinctions and added robustness analyses, annotation-sensitivity results, and further cross-model comparisons. We share the current preliminary version of the draft to allow the reviewer to inspect the new results already at this stage.
Concept definition and interpretation: We added an explicit geometric concept definition and clarified the interpretation of R0. Clustering and embedding diagnostics characterize the extraction procedure, while semantic consistency is evaluated separately. The manifold formulation is a modeling assumption and does not mean that we expect every recovered cluster to be intrinsically nonlinear.
NN benchmark: To address whether stronger neighboring-layer discriminations corresponds to meaningful concept relationships, we added the matched-versus-rnadom semantic evaluation described below. This provides evidence beyond the neighboring-layer sanity check.
Semantic consistency of concept matches: We added a new Result 7 that evaluates Hungarian-matched concept pairs from R5/R6, together with randomly re-paired controls, using two LLM judges (Mistral-Small-3.1-24B-Instruct, Gemma-3-27B-it). The judges see only the raw evidence behind each cluster (top tokens, predicted next tokens, example contexts), never generated concept names, model names, layers, or CBA scores, and rate each pair as equivalent, related, or unrelated (agreement 83%, κ = 0.74). Across all pairs (matched and random), pair-level CBA separates pairs judged equivalent from those judged unrelated (AUROC 0.77), and so does whether a pair was matched at all, which is determined by the CBA-based assignment (AUROC 0.93). At the level of comparisons, the fraction of matches judged equivalent tracks mean layer-pair CBA closely (Spearman ρ = 0.98 across comparisons, 0.88 across layer pairs; 34-79% equivalent vs. <1% for random pairs). Restricting to matched pairs alone is a harder task on which raw pair-level CBA is uninformative, because small concepts incur low cost to any partner, whereas a normalized match margin, introduced post hoc after observing this effect, recovers discrimination (AUROC 0.86); we report both. We conclude that CBA-based correspondences reflect agreement in cluster content and not only similar partition geometry. We state in the paper that this does not establish semantic identity beyond the sampled tokens and that the judges are LLMs rather than human annotators (see Appendix A.11).
Repeated-run robustness: We added three-seed discovery experiments for the five main checkpoints, holding the original activation reservoirs fixed. Across all 180 residual layers, mean same-layer CBA against the original run is 0.8571. Changing the 5,000-token evaluation sample produces a mean within-comparison range of 0.00328. These results provide evidence of stability under discovery randomness and evaluation sampling. The protocol and per-model results are reported in the appendix.