- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how semantic hierarchies are encoded in vision and language transformers. It measures whether child-concept subspaces lie within their parents’ subspaces and whether parent detectors also activate on child concepts. Experiments include synthetic transformers and pretrained models and find hierarchy-specific subspace nesting across model families, while the extent to which parent concepts remain detectable varies with the training objective.
Critical strengths
-
The controls are well designed. The authors exclude the evaluated child when estimating parent representations and use shuffled pairings and firing-rate-matched controls.
-
Distinguishing containment from subsumption is useful. The experiments show that child subspaces can remain aligned with parent subspaces even when parent detectors respond less often.
Critical weaknesses
-
The language-model evaluation only includes models released in 2025 or earlier. Whether the findings still hold for the newer SoTA LLMs and larger models is unclear.
-
The claim that nesting 'is present at every depth' is broader than the evidence supports. Sec. 4 reports no tree-specific containment in the early and middle layers of GPT-2 and Qwen2.5.
-
The authors acknowledge that the role of the training objective remains a hypothesis, since the models also differ in size and training data. However, the Introduction states that the objective 'determines' hierarchy readout, which these comparisons do not establish.
Questions for authors
-
Do encoders with higher containment or subsumption scores also perform better on hierarchical classification tasks?
-
Have the authors examined the vision encoders of MLLMs after multimodal training? How do their containment and subsumption scores compare with those of the original pre-trained encoders?
Thank you for the detailed response! I think the rebuttal addresses all the concerns I have for the paper.