- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes the Gärdenforsian Representation Hypothesis (GRH), a layered view of conceptual geometry in LLM representations. The framework organizes conceptual structure into five linked levels: quality dimensions as linear directions, semantic domains as subspaces, properties as convex regions within domains, concepts as regions spanning multiple domains, and prototypes as geometric centers that predict graded typicality. The authors progressively reconstruct these structures from LLM representations and evaluate the five hypotheses using human semantic annotations from THINGSplus and Binder Feature Norms. The analysis is repeated across Llama-3.1-8B, Gemma-7B, and Mistral-7B, with broadly consistent results across models.
Critical strengths
-
The paper provides a coherent and genuinely informative unification of several previously separate geometric views of concepts. I found the progression from quality directions to domains, property regions, multi-domain concepts, and prototypes particularly elegant. Rather than treating these geometric objects as unrelated observations, the paper organizes them into a single layered account and tests each level empirically. This makes the work substantially more interesting to me than an incremental representation-analysis paper.
-
The empirical program is broad and reasonably systematic. The authors do not rely on a single model or a single geometric statistic. The full reconstruction pipeline is repeated independently across three LLM backbones, and several stages include meaningful controls such as unseen-category generalization, wrong-domain or random-subspace comparisons, and random domain decompositions. I especially appreciate that the paper attempts to validate the semantic meaning of the recovered geometry rather than only visualizing it.
Critical weaknesses
The main limitation, in my view, is that the evidence is not fully independent of the human conceptual structure that the paper aims to recover. Human ratings are used to identify quality directions, human-defined domains are used as an evaluation target, thresholded human ratings define several properties, and category membership is used to estimate concept regions and prototypes. As a result, I view the experiments as strong evidence that LLM representations are compatible with and can support a Gärdenforsian organization, but somewhat weaker evidence that this complete conceptual organization emerges naturally or independently inside the model. This distinction matters for the strongest interpretation of the paper, although it does not overturn my overall positive assessment.
Questions for authors
-
How do the authors distinguish “recoverability under human supervision” from “naturally emergent conceptual geometry”? Several stages of A1–A5 are constructed using human semantic ratings or category membership. I would be interested in the authors’ view on which parts of GRH are genuinely discovered from the representation geometry, and which parts should instead be interpreted as showing that the geometry can support a human-specified conceptual organization. A more explicit discussion of this distinction would strengthen the central claim.
-
How essential is the higher-level GRH geometry relative to simpler alternatives? For A3, the single-direction half-space slightly outperforms the domain-level convex region, which is understandable because the properties are defined by thresholding scalar ratings. For A5, the prototype-typicality correlations are consistently positive but modest, and the raw hidden-space centroid is stronger than the proposed quality-space prototype for Llama. I would therefore appreciate a clearer discussion of which parts of the higher-level geometry are empirically necessary, rather than merely compatible with the data.
General Response and Summary of Revisions
We thank the reviewers for their feedback. We uploaded a revised manuscript with new experiments added in Sections 4.3 and 4.7.
Summary of Common Reviewer Questions & Core Clarifications
Intrinsic/Discovered vs. Imposed Geometry (Reviewers w3kL & Pb8z). Both reviewers asked whether the recovered conceptual geometry is intrinsic to the LLM representations or imposed by the construction procedure. We study intrinsic or emergent geometry by following the readout approach used by Park et al. in the Linear Representation Hypothesis [1] and by Xiong in the Lattice Representation Hypothesis [2]. We keep the model representration geometry fixed and use human ratings and category labels to only estimate quality directions, concept regions, and prototypes. We added two additional controls to test whether the same procedure gives comparable results when we change the annotation assignments or the representations. Our controlled analysis further confirms that the recovered geometry is driven by the learned LLM representations instead of the construction procedure itself.
Simpler Baselines vs. Controlled Baselines (Reviewers w3kL & Pb8z). Simpler alternatives show what is sufficient for a given task. A single-direction half-space is convex, and a raw hidden-space centroid is a prototype, so their strong performance is consistent with A3 and A5. Controlled baselines test what the specific recovered geometry contributes. As shown in Table 11, replacing the recovered domains with same-rank random subspaces lowers held-out concept-membership AUROC from 0.874–0.879 to 0.631–0.746 across the three backbones. This demonstrates that our recovered domain structure provides essential empirical leverage that cannot be replicated by arbitrary projections.
Additional Controlled Experiments in the Revised Manuscript
Selected results from Table 5. Shuffled annotations and input embedding controls. Shuffled scores are means ± standard deviations over 199 permutations. A2 uses Binder. The other rows use THINGSplus.
Selected results from Fig. 6. Betweenness compares the share of high-rated cases using the same number of held-out cases per property in each domain. The hull rows report rating proportions inside the recovered-domain hulls.
References
[1] Park, K., Choe, Y. J., and Veitch, V. (2024). The Linear Representation Hypothesis and the Geometry of Large Language Models. ICML 2024.
[2] Xiong, B. (2026). The Lattice Representation Hypothesis of Large Language Models. ICLR 2026.