acceptodds
Under review as a conference paper at ICLR 2027

Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models

Abstract

Understanding information processing in large language models (LLMs) requires dissecting the geometric organization of their internal token representations. While existing mechanistic interpretability (MI) methods seek to extract human-understandable concepts, they are constrained by a strong linearity assumption—an assumption increasingly challenged by evidence of non-linear feature manifolds. In this work, we move beyond linear concepts by adapting Non-Linear Multi-Dimensional Concept Discovery (NLMCD) from computer vision to token-level LLM activations, modeling concepts as low-dimensional manifolds. To enable quantitative comparison of these independently discovered concept manifolds across layers and models, we introduce a concept-based alignment (CBA) score, a generalized Rand index that measures geometric proximity without requiring explicit feature matching. Our analysis yields six key findings: (i) a neighboring-layer sanity check shows CBA is more sensitive than PCA- or CKA-based linear baselines; (ii) layer×layer alignment matrices reveal two characteristic block structures concentrated in intermediate and late layers, consistent across models and obscured by linear metrics; (iii) concept composition remains syntax-dominated through most of the network before giving way to increasingly mixed syntactic–semantic concepts in later layers, with increasing output-orientation toward the final layers; (iv) multilingual concept sharing between English and Mandarin is training-dependent rather than universal — strongest in Qwen, weaker in Llama, and absent in GPT-2; (v) inter-model alignment mirrors this structure: strong correspondence between same-family Qwen models of different scale, consistent with the block structures in (ii), but weak alignment across model families; and (vi) tracing representational alignment across the training stages of the Tulu-3 pipeline shows alignment is highest between adjacent stages, validating CBA, with the largest shift occurring between the base model and the SFT stage, while subsequent preference-alignment stages (DPO, RLVR) leave early layers largely unchanged and RLVR mostly preserves DPO's concepts in late layers.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, concept discovery, LLMs, representation learning, XAI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4NpoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses a relevant problem: linear concept representations may not capture all the structures present in LLM activations. Exploring nonlinear representations is worthwhile. Methodology: They first use UMAP to reduce activation vectors to a 50-dimensional space, followed by HDBSCAN to discover clusters, and fuzzy memberships to describe how strongly each token belongs to each concept. Next, they introduce Concept-Based Alignment (CBA), a score comparing the relationships among concepts discovered independently in different representations. Authors conduct experiments and present results answering many questions such as: 1) Can nonlinear concepts be reliably discovered in LLM activations? 2) Can CBA compare representations more informatively than linear alignment methods? 3) How does concept structure change across transformer layers? 4) How does the composition of concepts change with layer depth? 5) Do different languages share the same internal concepts? 6) Do models of different sizes or model families share concepts? 7) How does fine-tuning change concept organization?

Post Rebuttal: - Authors have made useful improvements in clarifying the geometric definition of concepts and distinguishing density-based clustering quality from semantic validity. However, the central concern that UMAP–HDBSCAN may produce clustering artifacts rather than intrinsically nonlinear concept manifolds remains unresolved, as no comparison with linear concept representations or validation in the original activation space is provided.

- The additional robustness experiments are encouraging but insufficient to establish general stability: the three-seed analysis reports a mean same-layer CBA of 0.8571, while the threshold-sensitivity experiments themselves demonstrate substantial dependence on clustering parameters, and robustness to alternative UMAP neighborhood settings and extraction corpora remains unestablished.

- The new semantic validation using two LLM judges is a positive addition, but it does not provide independent human validation, and the reported AUROC of 0.86 for the normalized match margin is particularly concerning because this measure was introduced post hoc after observing the limitations of the original matching score. Therefore, the evidence does not yet establish that CBA reliably identifies semantically equivalent concepts.

- In my opinion, the second annotator experiment exposes an important weakness rather than fully resolving it. Cohen's kappa of 0.622 for concept type is moderate, whereas the value of 0.130 for orientation indicates very poor agreement, making the claims concerning the evolution of input/output orientation across layers particularly uncertain.

- The additional examples of previously unclustered tokens recovering into smaller clusters are useful, but the authors still do not establish the semantic coherence of the discovered concepts or quantify how many noise assignments correspond to meaningful rare concepts, leaving the interpretation of coverage and concept quality limited.

- Several broader conclusions remain stronger than the experimental evidence supports, particularly interpreting alignment blocks as meaningful representational organization, attributing multilingual differences to training rather than confounded model-family differences, and inferring that post-training largely preserves pretrained concepts from high CBA scores without corresponding behavioural or functional validation.

Overall, I acknowledge the authors' efforts but do not consider the fundamental concerns about the validity of nonlinear concept discovery and the semantic interpretation of CBA adequately resolved. Hence, I would keep my original score.

Critical strengths

Strengths are listed below:

  1. The paper addresses a relevant problem: linear concept representations may not capture all the structures present in LLM activations. Exploring nonlinear representations is worthwhile.

  2. Authors provide a pipeline using UMAP, HDBSCAN, fuzzy concept memberships, and CBA. They report important implementation details, including the use of 100,000 token positions, 50-dimensional UMAP embeddings, and fixed clustering parameters.

  3. The paper evaluates several aspects of representation structure: layer depth, concept composition, multilingual sharing, model scale, cross-family alignment, and training-stage changes.

  4. Result 0 informs median coverage to be 0.4469 and median DBCV to be 0.2743. It states that the embedding reconstruction error decreases with layer depth. These measurements help in evaluating the behavior of the pipeline, although they do not establish semantic concept quality, in my opinion.

  5. Result 1 compares CBA against PCA, linear CKA, and RBF CKA.

  6. Result 2 identifies nonuniform patterns in layer-alignment matrices, including diagonal bands and blocks.

  7. Result 3 attempts to characterize discovered clusters instead of treating all clusters as meaningful without inspection.

  8. Result 4 includes shuffled English–Mandarin sentence pairs to explore whether different languages share the same internal concepts.

  9. Result 5 compares models at different scales within the Qwen family and includes a cross-family comparison with Llama.

  10. Result 6 examines several checkpoints from one training pipeline, allowing a detailed comparison of SFT, DPO, and RLVR.

Critical weaknesses

Weaknesses:

  1. Result 0 does not establish that the discovered clusters are genuine concepts. The reported median coverage of 0.4469 means that about 44.69% of tokens are assigned non-noise memberships under the chosen procedure. It does not mean that 44.69% of tokens have been assigned to valid semantic concepts. Likewise, DBCV measures density-based separation, not conceptual coherence.

  2. The use of HDBSCAN's noise assignments is a major concern. Sparse clusters need not be noise: they may correspond to rare concepts, infrequent linguistic constructions, named entities, or specialized knowledge. Conversely, a dense cluster can group tokens for reasons that are not semantically meaningful. Without qualitative inspection of both clustered and unclustered tokens, the reported coverage is an incomplete measure of concept discovery quality.

  3. The fixed minimum cluster size and minimum-samples threshold of 50 may systematically exclude rare concepts. The same settings across layers and models improve procedural consistency, but do not guarantee that the method is equally suitable for representations with different density structures. Result 0 therefore does not justify the broad claim that the discovery step is technically valid across all the evaluated layers.

  4. The paper's central claim concerns nonlinear concepts, but the use of nonlinear dimensionality reduction followed by density-based clustering does not establish that the underlying concepts are intrinsically nonlinear. UMAP can create or distort local geometric structure, and HDBSCAN can cluster that structure. The resulting clusters could reflect nonlinear concepts, ordinary clusters made easier to separate by the embedding, or artifacts of the embedding and density thresholds. The paper needs evidence that the discovered concepts cannot be represented adequately by linear directions or subspaces.

  5. The reconstruction quality score in Result 0 does not resolve this issue. Lower NRMSE in later layers is evidence about the reduced embedding's preservation of pairwise distances under the chosen metric. It is not evidence that the clusters are semantically valid or that the underlying concepts are nonlinear. The relationship between the reported reconstruction error, the original activation geometry, and the validity of the extracted concepts needs to be made clearer.

  6. The fixed UMAP seed and clustering profile do not establish robustness. The paper does not show whether the discovered concepts, coverage, DBCV, and CBA scores remain stable across random seeds, neighborhood sizes, minimum cluster sizes, sampling choices, and extraction corpora. A single reproducible run is not the same as a robust result.

  7. Result 1 does not demonstrate that CBA measures concept alignment more accurately than the baselines. The neighboring-layer test reports a Neighbor z-score of 2.515 for CBA, compared with 1.714 for RBF CKA, 1.867 for PCA, and 1.287 for linear CKA. This establishes a stronger neighboring-layer signal under the authors' tests. However, the assumption that adjacent layers should be most similar is a plausibility check, not ground truth for concept correspondence. A method that creates higher alignment patterns is not necessarily a more accurate method.

  8. The nearest-neighbor scores in Result 1: CBA achieves 0.997, RBF CKA 1.000, PCA 0.983, and linear CKA 0.978. All methods perform well on this test. The differences in Neighbor z-score are interesting, but the paper does not show that the additional structure revealed by CBA corresponds to correct or useful concept matches.

  9. CBA measures similarity between pairwise membership relations, not semantic identity between concepts. A score close to one means that two clustering methods induce similar pairwise membership similarities on the evaluated observations. It does not establish that the concepts have the same meaning. Different semantic concepts can organize the sampled tokens similarly, while the same underlying concept can receive different cluster boundaries and membership strengths. This is a central limitation of the interpretation of every later result.

  10. Result 2 treats diagonal bands and alignment blocks as evidence of recurring representational organization. These patterns are descriptive, but they do not establish that the blocks correspond to distinct computational stages or functional modules. They could partly reflect smooth changes between neighboring layers, the clustering procedure, or common properties of the input observations. The paper needs stronger null models and tests of whether the blocks are stable and functionally meaningful.

  11. Result 3 relies on LLM-generated concept labels. The reported aggregate annotations show that semantic and syntactic concepts are roughly balanced in Llama, at 45.7% and 45.1%, while Qwen has 39.8% semantic and 53.2% syntactic concepts. The authors also report that input-oriented concepts dominate both models. These are results of the annotation procedure, but the paper does not establish the accuracy of the labels through independent human annotation, inter-annotator agreement, or a systematic analysis of labeling errors. The interpretation of concept composition therefore remains uncertain.

  12. Result 3 also requires careful interpretation of output orientation. The paper reports no purely output-oriented concepts in the two models summarized in Table 5, while concepts classified as having both input and output evidence increase in later layers. This supports a possible increase in joint input/output evidence under the labeling procedure, not the stronger claim that purely output-oriented concepts emerge toward the final layers.

  13. Result 4 does not establish that the measured multilingual sharing is semantic sharing. The mean excess shared-concept mass is 0.0009 for GPT-2, 0.0242 for Llama, and 0.0392–0.0487 for Qwen. The shuffled-pair control is useful, but excess overlap can still reflect shared syntax, punctuation, sentence structure, token frequency, or other surface regularities. The paper itself notes that much of the excess remains unlabeled. The conclusion that multilingual concept sharing is training-dependent is plausible, but the cause and semantic content of that sharing are not isolated.

  14. Result 4 also confounds training with model family and other differences. GPT-2, Llama, and Qwen differ in architecture, tokenizer, training data, and training procedures. The observed differences cannot be attributed exclusively to multilingual training. The study of English and Mandarin is informative, but insufficient to establish a general claim about multilingual representations.

  15. Result 5 uses only 1,024 aligned observations per cross-model comparison. Character-span alignment is a reasonable way to handle different tokenizers, but mean-pooling memberships across source tokens can change the representation being compared. The reported stronger alignment among Qwen models than between Qwen and Llama may reflect shared training lineage, tokenization, architecture, or data, rather than shared semantic concepts. The quantitative patterns do not establish that models from the same family encode the same concepts.

  16. Result 5 also does not demonstrate the general claim that corresponding blocks are functionally similar across model families. A weak diagonal or middle-layer block in a CBA matrix is evidence of structural correspondence under the metric, not proof of a shared computational role. The conclusion should remain descriptive unless independently validated.

  17. Result 6 reports DPO–RLVR CBA scores around 0.85–0.86 across most layers and alignment above 0.8 across training-stage comparisons. These are high structural similarity scores under CBA, but they do not show that most concepts were formed during pretraining or that post-training only moderately changes the model's behavior. Substantial behavioral changes can occur without large changes in aggregate representation geometry, and conversely geometric changes need not imply behavioral changes.

  18. Result 6 studies a single training pipeline. The larger shift between the base model and SFT, and the higher alignment between DPO and RLVR, are useful observations for Tulu-3, but do not establish a general pattern across fine-tuning methods or models. The interpretation of the layer-14-to-15 drop as especially important for post-training behavior also goes beyond the evidence unless linked to a specific behavioral or functional test.

Overall, the paper's broad scientific conclusions depend on three unverified assumptions: 1) that the non-linear concepts are extracted using the proposed pipeline, 2) that the extracted clusters are (semantically) meaningful concepts, and 3) that CBA scores reliably reflect semantic concept correspondence. The current experiments do not adequately validate these assumptions. This weakens the interpretation of results and claims presented in the paper.

Questions for authors

Questions for the authors:

  1. In Result 0, how do the authors distinguish genuine noise from rare but meaningful concepts? Can they provide qualitative examples of low-density or small clusters and of tokens classified as noise, and evaluate whether the latter contain coherent concepts?

  2. Can the authors evaluate concept coverage using more than the non-noise token fraction? For example, can they report human-judged concept coherence, concept diversity, and the proportion of noise tokens that appear to belong to meaningful rare concepts?

  3. Given the fixed HDBSCAN minimum cluster size and minimum-samples threshold of 50, how sensitive are the number and quality of discovered concepts to these thresholds? Do the main findings persist when rare concepts are allowed?

  4. What evidence establishes that the discovered concepts are intrinsically nonlinear rather than artifacts of nonlinear dimensionality reduction and density-based clustering?

  5. Can the authors show that the clusters found after UMAP are stable in the original activation space? For example, do the clusters remain coherent under alternative embeddings or direct neighborhood and density analyses in the original space?

  6. How much do the discovered concepts and CBA scores change across UMAP seeds, neighborhood sizes, HDBSCAN thresholds, and token samples? Can the authors report repeated-run variability for the principal results?

  7. In Result 1, why should a stronger neighboring-layer signal be interpreted as better concept alignment? Can the authors evaluate CBA on synthetic or controlled representations with known correspondences, including cases where concepts are preserved, permuted, split, merged, or corrupted?

  8. Can the authors demonstrate that CBA is more accurate than RBF CKA on a task with known ground truth, rather than showing only that CBA produces more pronounced alignment structure? How do they determine that the additional structures revealed by CBA are correct?

  9. How should a reader interpret a high CBA score when the method compares pairwise membership relations rather than concept meanings? Can the authors provide a direct evaluation of concept pairs with high CBA-based correspondence to determine whether independent annotators judge them to represent the same or similar concepts?

  10. In Result 2, how do the authors distinguish genuine functional blocks from visual patterns caused by smooth layer-to-layer changes, clustering artifacts, or the selected input corpus? Do the blocks persist across seeds, datasets, and reasonable clustering settings?

  11. Can the authors connect the alignment blocks to independent measures of function, such as task-specific behavior, controlled interventions, or activation ablations? If not, should the blocks be described as structural patterns rather than functional stages?

  12. In Result 3, what is the accuracy of the LLM-generated concept labels? Can the authors validate a representative sample with independent human annotators, report inter-annotator agreement, and quantify agreement between the human and LLM labels?

  13. In Result 4, how much of the excess shared-concept mass reflects semantic correspondence rather than syntax, punctuation, token frequency, or sentence structure? Can the authors use controls that preserve these properties while disrupting semantic correspondence?

  14. Since the multilingual comparison includes only English and Mandarin and contrasts different model families, how do the authors separate the effect of multilingual training from architecture, tokenizer, data, and other training differences? Can the finding be replicated across additional language pairs and models with more controlled training comparisons? Choosing only two languages does not establish the generalized claim.

  15. Can the authors demonstrate that the high-alignment regions between Qwen models correspond to independently validated shared concepts, rather than simply similar clustering patterns? What evidence supports the claim that the blocks represent corresponding functions across model families?

  16. In Result 6, what justifies interpreting CBA values around 0.85–0.86 as evidence that RLVR largely preserves concepts established during DPO? Can the authors test whether these scores track independently measured behavioral changes across the training stages?

2 comments

We thank the reviewer for the detailed comments. Due to the large number of identified weaknesses and related questions, we organized them into categories but will refrain from explicitly referencing every single issue/question. Several concerns concern the distinction between geometric structure, semantic interpretation, and functional significance. We have clarified these distinctions and added robustness analyses, annotation-sensitivity results, and further cross-model comparisons. We share the current preliminary version of the draft to allow the reviewer to inspect the new results already at this stage.

Concept definition and interpretation: We added an explicit geometric concept definition and clarified the interpretation of R0. Clustering and embedding diagnostics characterize the extraction procedure, while semantic consistency is evaluated separately. The manifold formulation is a modeling assumption and does not mean that we expect every recovered cluster to be intrinsically nonlinear.

NN benchmark: To address whether stronger neighboring-layer discriminations corresponds to meaningful concept relationships, we added the matched-versus-rnadom semantic evaluation described below. This provides evidence beyond the neighboring-layer sanity check.

Semantic consistency of concept matches: We added a new Result 7 that evaluates Hungarian-matched concept pairs from R5/R6, together with randomly re-paired controls, using two LLM judges (Mistral-Small-3.1-24B-Instruct, Gemma-3-27B-it). The judges see only the raw evidence behind each cluster (top tokens, predicted next tokens, example contexts), never generated concept names, model names, layers, or CBA scores, and rate each pair as equivalent, related, or unrelated (agreement 83%, κ = 0.74). Across all pairs (matched and random), pair-level CBA separates pairs judged equivalent from those judged unrelated (AUROC 0.77), and so does whether a pair was matched at all, which is determined by the CBA-based assignment (AUROC 0.93). At the level of comparisons, the fraction of matches judged equivalent tracks mean layer-pair CBA closely (Spearman ρ = 0.98 across comparisons, 0.88 across layer pairs; 34-79% equivalent vs. <1% for random pairs). Restricting to matched pairs alone is a harder task on which raw pair-level CBA is uninformative, because small concepts incur low cost to any partner, whereas a normalized match margin, introduced post hoc after observing this effect, recovers discrimination (AUROC 0.86); we report both. We conclude that CBA-based correspondences reflect agreement in cluster content and not only similar partition geometry. We state in the paper that this does not establish semantic identity beyond the sampled tokens and that the judges are LLMs rather than human annotators (see Appendix A.11).

Repeated-run robustness: We added three-seed discovery experiments for the five main checkpoints, holding the original activation reservoirs fixed. Across all 180 residual layers, mean same-layer CBA against the original run is 0.8571. Changing the 5,000-token evaluation sample produces a mean within-comparison range of 0.00328. These results provide evidence of stability under discovery randomness and evaluation sampling. The protocol and per-model results are reported in the appendix.

HDBSCAN threshold and rare concepts: We added a controlled threshold-sensitivity analysis using fixed paper-run embeddings and token reservoirs, covering early, middle and final layers of all five main checkpoints. Lowering minimum cluster size from 50 to 25 gives mean CBA 0.920 against the original memberships. Lowering minimum samples from 50 to 20 gives 0.874. Lowering both gives 0.800 and increases the mean concept count from 341 to 741. All 45 perturbed comparisons exceed their corresponding row-shuffled controls. We report all configurations, showing both a sensitivity to the selected parameters but also preserved structural agreement between the different runs. We fully share the reviewer‘s assessment that tokens currently designated as noise (given the assumption about the geometric structure of the feature space is correct) would be assigned to concepts discovered in concept extraction runs on larger corpora. Therefore, we do not see a distinction between genuine noise tokens and rare but undiscovered concept tokens.

Meaning of noise assignments: We agree that density-based noise can contain meaningful structures. With threshold 25/20, an average of 24.4% of originally unclustered observations become clustered. For example, a recovered 42-member Qwen-32B cluster contains fossil/anatomy terms and dinosaur-fossil contexts. All of its member tokens were previously noice. We added this example and clarify that coverage measures density-based assignment, rather than the proportion of observations belonging to semantically valid concepts.

Annotation sensitivity: We repeated R3 with GPT-6 Luna using identical prompts and evidence. Pooled Cohen’s kappa is 0.622 for concept type and 0.130 for orientation. The broad depth-dependent trends persist, while individual orientation assignments are sensitive to the annotator. The new results and plots are included in the appendix.

Reviewer_uzPyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a method to extract non-linear concepts from model-internal LLM representations. The method called NLMCD was originally proposed in computer vision and is here applied to NLP (LLM models) (with the help of sparse autoencoders) and hierarchical clustering (HBSCAN) to compare models within and across LLM families using a non-linear hypothesis. The results examine a) to what degree non-noise concepts are recoverable across layer, b) a sanity check that tests the assumption that closest layers need to have the largest similarity c) the alignment across layers within and across models and d) an experiment in which parallel English-Chinese sentences are examined. The study uses three model families (Llama, Qwen and gpt2). The paper finds evidence for structure that is not visible when a linear representation is used.

Critical strengths
  • The paper is overall well written (I have though some questions for clarity, as some parts of the setup and evaluation remain unclear), and the study is motivated with a clear hypothesis. Results confirm the hypothesis and intuitively do make sense.

  • Overall, going beyond the assumption of linear representations is a valid and a timely contribution.

  • Code base is provided and looks good from a quick look at the anonymous code repository.

Critical weaknesses
  • While being an interesting paper, I do miss some discussions on the concepts themselves. First, they are based on sparse autoencoders and are not further interpreted while the analysis rests on LLM-labeled categories. So how reliable are the concepts remains a question. The quality of the LLM labels should be checked manually on a sample, I think adding such a quality check and providing actual examples of concepts would strengthen the analysis. Second, I miss a description on how concept tokens split into subtokens are treated.

  • The paper could be more self-contained, particularly the NLMCD from computer vision could be described in more detail than the short introduction in lines 170-171. For example, by elaborating on what adaptations were necessary.

  • The paper evaluates coverage by a token-level analysis based on coverage. Coverage is only defined very briefly (line 158) as "the non-noise token fraction", what is missing is what determines a noise vs non-noise token. As such, the metric is only partially defined.

Questions for authors
  • For the direct neighbor analysis in Table 1, did you consider neighbors in both direction equally? Figure 3 suggests both are similar impactful, though I was wondering it the upper layer (towards the output) has higher similarity than the layer below the current layer (Figure 3 is a bit difficult to see, I wonder how the results in Table 1 are changing if analyzed by upper/lower layer)

  • Have you considered an alternative to HBDBSCAN? How would it affect the findings?

  • Please introduce N and C in line 119.

  • Result 0: it seems the argument assumes models are comparable, but the non-noise ratio will depend on the vocabulary size of the LLM. So why is the results comparable, especially for the cross-model comparison where models have different vocabs? and related to above, what are non-noise tokens? I also wonder how you can conclude technical validity in line 214 ("Taken together, these results support the technical validity"- why?) Moreover, regarding different vocabs and tokenizers, you later mention "Because Qwen and Llama use different tokenizers, we align their tokens using character spans within the same sentence" - can you elaborate, how does this work exactly and why does it solve the comparability issue? (what if few too few tokens are being aligned?)

2 comments

W1 (Concept definition): We thank the reviewer for pointing out where our approach was unclear. Our method does not use sparse autoencoders. We apply UMAP and HDBSCAN directly to contextual token activations. We added an explicit definition of concepts as density-supported manifolds represented by activation point clouds, and defined N and C. Each sampled tokenizer unit, including subwords, contributes its own contextual activation. LLM annotations characterize the discovered concepts afterward.

W1 (Concept annotation): Descriptive concept names do not enter discovery, CBA, or the category classifiers. Category annotations affect the descriptive summaries in R3/R4. To assess annotation sensitivity, we repeated R3 with GPT-6 Luna as an annotation model. Pooled Cohen‘s kappa is 0.622 for concept type and 0.130 for orientation and the broad trends for R3 stay the same across annotation models. Results were added to the appendix.

Q1 (One-sided prediction for direct neighbour analysis): We thank the reviewer for suggesting this addition to our evaluation metrics. We have added separate previous- and next-layer diagnostics to the appendix, excluding boundary layers for a matched comparison. For CBA, one-sided NN scores are 0.991 toward previous layers and 0.997 toward next layers, with corresponding neighbor z-scores of 2.292 and 2.312. The next neighbor is more similar in 55.7% of interior-layer comparisons, averaged equally across checkpoints. Thus, CBA shows a modest aggregate preference toward the next layer while retaining strong adjacent-layer correspondence in both directions. CBA also retains the highest neighbor z-score in each direction.

Q2 (Alternative to HDBSCAN): We retain UMAP/HDBSCAN because it matches our density-based concept definition and has empirical precedents in other domains (Allaoui et al., 2020; Pealat et al., 2021). Changing the clustering algorithm could alter the concept inventory and alignment scores, though investigating (computationally more efficient) alternatives to HDBSCAN represents a promising direction for future work.

Q3+W2 (Self-containedness and missing details): We rewrote the description of the NLMCD algorithm in Methods to make it more self-contained, including definitions of all relevant quantities.

Q4+W3 (Noise tokens and CBA robustness): We thank the reviewer for highlighting these ambiguities. „Noise“ denotes token occurrences assigned HDBSCAN label -1, rather than vocabulary entries or tokens excluded from the input. Coverage is the fraction of sampled occurrences assigned to clusters. The denominator is the sampled token occurrences, not vocabulary size. Our added three-seed robustness analysis across the five main checkpoints provides complementary evidence of stability, with mean same-layer CBA of 0.8571.

Q4 (Cross-tokenizer comparisons): For cross-tokenizer comparisons, we match complete character spans within identical sentences and mean-pool source-piece memberships when necessary. We exclude incomplete matches. The saved runs contain 17,939-19,667 eligible matches, from which 1,024 are sampled consistently across layer pairs. We have added the procedure and retention counts to the appendix.

I thank the authors for the updated pdf and response. I find the added concept definition and elaboration useful, thus happy to see my review was useful. I am leaning positive for this paper. Thanks

Reviewer_K4UdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors propose a methodology to explore the geometry of concepts as manifolds in LLM representations in order to use new alignment measures that could better capture the multidimensionality of the data beyond linear approaches. Across several experiments, the authors show that this methodology reveals more fine-grained representational structures compared with the baselines, allowing them to explore different aspects of LLM representations, such as within- and between-model similarity, multilingual alignment, and the effects of fine-tuning.

Critical strengths

This is a well-written paper with a sound methodology, and the authors' proposal is compelling, since their results show that non-linear approaches can be more sensitive in detecting differences in representational alignment, allowing for the exploration of layer-by-layer representations and a better understanding of different aspects of models, even when they have different sizes and training procedures. Their results are also interesting, and they provide strong evidence that their approach is useful for representational alignment research.

Critical weaknesses

I think the manuscript would benefit from additional experiments, including more model comparisons, especially for R5, and from further developing the interpretations of its findings, particularly regarding the implications of this methodology for representational alignment research and the Platonic Representation Hypothesis. These extensions would strengthen the claims of the paper.

Questions for authors
  1. I think it may be important to perform the annotation procedure of R3 with different LLMs to assess cross-model robustness.

  2. Is there a specific reason why the authors didn’t include all the models/families for the experiments in R5? I think including them all may allow for a stronger comparison across model sizes, families, and the differences among alignment within and between models.

  3. I was wondering why the authors did not include the Qwen3-4B in Figure 6, given that this was the one with the higher excess mass.

  4. I find the results of R4 that earlier layers concentrate the most shared representational structure between Mandarin-English interesting. Why do the authors think a particular activation pattern emerges toward the final layers for Qwen3-32B (Fig. 11)?

  5. I find the affirmation of the results being against the Platonic Representation Hypothesis interesting. Could the authors deep into how their results may challenge it?

  6. How do the authors visualize how their results affect research in representational alignment? Could the authors deep into it in the discussion section?

*Minor: I think the authors may check the order of Tables and Figures. On page 6, for example, it is not specified that Table 5 is in the appendix. I think this would ease the readability of the paper (which is already well-written, by the way).

1 comment

Q1 (Annotation procedure): We thank the reviewer for suggesting this experiment. We repeated R3 with a second annotator (GPT 6 Luna) using identical prompts. The late-layer increase in mixed concepts and output orientation persists. Agreement is substantially stronger for concept type than orientation: pooled Cohen’s kappa is 0.622 and 0.130, respectively. The broad trends therefore persist while the annotators differ on individual orientation assignments. Q2 (Additional models/families for R5): We have included additional model comparisons for R5 in the appendix involving GPT-2, Llama-3.1-8B base, and Tulu-3-8B-DPO. The GPT-2 comparisons show relatively diffuse cross-family alignment, whereas the Llama-derived checkpoints retain pronounced corresponding-layer alignment. Q3 (Qwen4B Figure): We thank the reviewer for this suggestion. We replaced Qwen3-8B with Qwen3-4B in Figure 6, as Qwen3-4B exhibits the highest mean excess shared mass. The Qwen3-8B panel is retained in the appendix. Q4 (R4 late-layer peak for Qwen32B): We have investigated the late-layer peak for Qwen32B in greater detail. At layer 60, at least 67.9% of shared mass comes from concepts outside both sentences’ top-ten lists, and shuffled pairings retain 98.5% of observed mass. Together with the inspected examples, this suggests diffuse background overlap with possible positional or continuation-related components. In the case of layer 59 a concept labeled ``short imperative or exclamatory phrases,'' contributes 26.2% of shared mass. At layer 60, concept 24 includes examples such as „Exactly!“ and „Don't cheat me!“. Its strongest recorded token is sentence-initial in 4/5 English and 5/5 Mandarin selected high-scoring examples. Q5 (Platonic Representation Hypothesis): We revised the PRH discussion to clarify that our results qualify claims of automatic convergence at the evaluated scales, rather than refuting the hypothesis. Q6 (Implications for LLM alignment research): Our expanded discussion explains the implications for alignment-metric selection, independent validation, concept-level inspection beyond scalar similarity scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.