acceptodds
Under review as a conference paper at ICLR 2027

On the Geometry of Hierarchy: An Empirical Study in Vision and Language Transformers

Abstract

Does hierarchy in the data become hierarchy in the representation, and if so, in what geometric form? The taxonomies that label vision and language datasets are often hierarchical, but a network trained on them is never told that a beagle is a dog, an animal, or a living thing; so whether a model's representations carry the taxonomic tree, and in what shape, must be tested empirically. We define two measurements: whether a child concept's subspace is contained in its parent's, and whether the parent's feature fires concurrently with the child's. In small transformers trained on synthetic data, tree-structured training data gives rise to nested subspaces, whereas data without hierarchical structure does not. Similarly, across all families of pretrained large vision, language, and vision-language models (VLMs) we tested - examining the image encoders of the latter - hierarchical nesting is present, aligns with the ground-truth semantic tree, and varies little with model size. At the same time, the readouts across models differ depending on the training objective. VLMs (e.g., CLIP) preserve the parent firing on the child at the output, ImageNet training with class labels (e.g., ViT) preserve a portion of it, and self-supervised models (e.g., DINO) preserve almost none of it. Our results indicate that semantic hierarchies in the training data are naturally encoded in nested model subspaces rather than in single directions; however, how much of this hierarchy stays readable differs with the training objective.

Primary area:
interpretability and explainable AI
Keywords:
geometry, interpretability, hierarchy, vision models, language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_JZjUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how semantic hierarchies are encoded in vision and language transformers. It measures whether child-concept subspaces lie within their parents’ subspaces and whether parent detectors also activate on child concepts. Experiments include synthetic transformers and pretrained models and find hierarchy-specific subspace nesting across model families, while the extent to which parent concepts remain detectable varies with the training objective.

Critical strengths
  1. The controls are well designed. The authors exclude the evaluated child when estimating parent representations and use shuffled pairings and firing-rate-matched controls.

  2. Distinguishing containment from subsumption is useful. The experiments show that child subspaces can remain aligned with parent subspaces even when parent detectors respond less often.

Critical weaknesses
  1. The language-model evaluation only includes models released in 2025 or earlier. Whether the findings still hold for the newer SoTA LLMs and larger models is unclear.

  2. The claim that nesting 'is present at every depth' is broader than the evidence supports. Sec. 4 reports no tree-specific containment in the early and middle layers of GPT-2 and Qwen2.5.

  3. The authors acknowledge that the role of the training objective remains a hypothesis, since the models also differ in size and training data. However, the Introduction states that the objective 'determines' hierarchy readout, which these comparisons do not establish.

Questions for authors
  1. Do encoders with higher containment or subsumption scores also perform better on hierarchical classification tasks?

  2. Have the authors examined the vision encoders of MLLMs after multimodal training? How do their containment and subsumption scores compare with those of the original pre-trained encoders?

Reviewer_rTBEOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the geometric form of semantic hierarchy in learned representations. Its central hypothesis is that the variation subspace of a child concept is contained in that of its parent. The paper separates this geometric notion, measured by containment, from whether the hierarchy remains directly readable through feature activation, measured by subsumption. The authors test these two properties across controlled toy models, vision encoders, vision-language models, and language models, using several matched controls to rule out trivial overlap and other confounds. The appendices further compare alternative geometric pictures of hierarchy and investigate the nonlinear shape of concept clouds within their linear spans.

Critical strengths
  1. The empirical methodology is unusually careful and convincing. I particularly appreciate the separation between the geometry of hierarchy (containment) and its readout (subsumption). The paper then treats the obvious confounds seriously, using held-out parents and matched controls rather than interpreting a single positive statistic as evidence of hierarchy. This decomposition makes the empirical conclusions substantially more convincing.

  2. The paper goes beyond validating one proposed metric and genuinely investigates the phenomenon. The appendices compare several alternative geometric pictures of hierarchy, test whether the concept clouds themselves are curved, and explore how the resulting geometric insight could inform feature learning. I found this broader empirical investigation especially valuable. More generally, I appreciate the research attitude of the paper: the authors appear willing to reject explanations that do not survive stronger controls rather than simply accumulate evidence for their initial hypothesis.

Critical weaknesses

None.

Questions for authors

My main remaining question concerns the reliance on PCA-based linear subspaces for the central characterization. Although the sensitivity analyses and the nonlinear tests in the appendix alleviate this concern, to what extent do the authors expect the main nesting result to depend on PCA as the particular estimator of concept variation? Have the authors considered whether an alternative subspace estimator would lead to the same qualitative conclusions?

Reviewer_wXasOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether models learn the hierarchy between concepts, and what that hierarchy looks like in their representations. It measures two things separately: whether a child concept’s variation lies in its parent’s PCA subspace, and whether a parent detector fires when a child detector does. Experiments on synthetic transformers and pretrained vision and language models find overlap that follows the semantic taxonomy. They also find that this geometry and its readability can behave differently: language-supervised vision models retain stronger parent activation than DINOv2. Validation includes held-out children, matched controls, sensitivity checks, and experiments on recovering the taxonomy.

Critical strengths
  1. I appreciated the care taken with the controls. Excluding the child from the parent estimate addresses an obvious source of artificial containment, and the shuffled-parent comparison catches generic overlap. The appendix also reports claims that failed these checks. This makes the surviving results more credible.

  2. The separation between geometry and readout is interesting, and the coverage across model families gives it weight. A representation can retain taxonomy-related variation while a simple parent detector becomes less reliable. That is a useful qualification for interpreting probes.

Critical weaknesses
  1. I am not sure the measurements distinguish hierarchical nesting from shared variation among semantically similar concepts. Containment is often well below one, and its directional asymmetry depends on subspace dimensions. Comparing against unrelated parents establishes taxonomy-related overlap, but a comparison matched for semantic similarity, rank, and sample count would make the stronger interpretation more convincing.
  2. The tree-recovery claim is difficult to assess from the description. I could not find a sufficiently explicit account of how candidate relations are selected, oriented, and assembled into a tree. This matters because recovering a taxonomy from activations alone is stronger evidence than scoring known parent–child pairs.
Questions for authors

These are clarification requests that could change my assessment:

  1. Does the containment advantage survive unrelated-parent controls matched for subspace rank, sample count, and semantic similarity?
  2. What is the exact tree-recovery algorithm, and what information from WordNet is available to it?
  3. For language models, does the result hold when parent subspaces use only occurrences of the parent’s own words, rather than pooled descendant occurrences, and when ambiguous/shared lemmas are excluded?
1 comment

Thank you for the detailed response! I think the rebuttal addresses all the concerns I have for the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We are very grateful to all three reviewers and to the AC for the careful, constructive and encouraging feedback, which made the paper clearer and more rigorous. The revised manuscript is uploaded and contains every analysis and rewording promised in our responses.

  • Hierarchy and similarity (wXas). The control matched on semantic similarity, rank and sample count, graded containment read against matched nulls, the exact tree-recovery algorithm (reported as descriptive), and language-model parents built from their own words (Sections 2 and 4, Appendix A).
  • Beyond PCA (rTBE). Five alternative subspace estimators on eight networks give the same nesting (Section 2, Appendix A).
  • Newer models and wording (JZjU). Seven language models released in 2026, the rewording of "present at every depth" and "determines", the comparison with and without captions, hierarchical classification, and the vision encoders of four multimodal models (Introduction, Section 4, Appendix A).

While revising, we also spotted and fixed a labelling error in the comparison with the biological (NCBI) taxonomy and two duplicate language-model entries. Once corrected, NCBI nests about as strongly as WordNet at the family and order ranks, in line with the similarity control (Sections 4 and 5, Appendix B).

Thank you again for your time and for helping us improve this work. We remain available for any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.