- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks whether LLM embeddings contain the hierarchy between concepts, such as knowing that Bus is a kind of Vehicle. Using SNOMED CT, it studies whether concept attributes can be recovered from embeddings, whether one common direction can separate child concepts from their parents, and whether this direction together with concept similarity can predict subsumption relations. It also tests the same idea across other LLMs and WordNet.
Critical strengths
-
The hierarchy-direction result is empirically interesting and is observed via fairly strong tests. A low-capacity principal component (PC) subspace correctly orients 94.2% of held-out child-parent edges, and still reaches 90.3% when entire branches are held out. Shuffled labels fall to chance, while a random direction and PC1 perform much worse. Similar results are obtained with the other tested encoders, and the direction also transfers between SNOMED and WordNet without retraining on the target ontology. This is useful evidence that frozen LLM embeddings contain a shared directional signal related to hierarchy .... i.e., the result is not just memorization of individual edges. I think this is a real empirical finding, even though I do not think it is sufficient to establish an ontology or lattice embedding.
-
The paper gives a useful diagnosis of why flat attribute inclusion fails on SNOMED. The ExactFlat experiment is particularly informative. Under the flat attribute representation, 65.3% of true hierarchy edges appear incomparable. Once the hierarchy among destination values is restored, this falls to 0% and oracle F1 rises to 0.996. This clearly shows that the hierarchy of attribute values matters and that a flat FCA representation can lose important subsumption information. I found this diagnostic useful even though it also raises concerns about using the flat FCA construction to make broader claims about the ontology.
Critical weaknesses
-
What I could get from the paper is that the strongest result is a supervised child-parent prediction. This is not the same as what the paper claims -- i.e., an evidence for a hierarchy structure inside the LLM. PCA first finds the main alignment patterns across the sampled concept embeddings. For a child-parent pair, the difference vector then tells us how that change aligns with each of these PCA patterns. Logistic regression uses known child-parent labels to decide how much each PCA pattern should count, and combines them into the final direction . In other words, is not itself a direction discovered directly from the LLM geometry. It is a supervised mapping that makes child-parent differences point in the desired direction. It can downweight a dominant PCA pattern and strongly amplify a weaker one if that helps prediction. The PCA basis itself also depends on which concept embeddings are used to estimate it, and the paper does not test what happens to when only that concept set is perturbed.
Even if I give the paper the most favorable case, that is, suppose the PCA space is stable, closely follows the natural embedding geometry, and the child-parent scores have a healthy range rather than tiny margins, the main claim still does not follow. Consider Vehicle with two children, Bus and Car. A scalar can give , , and . Both true child-parent relations are perfectly oriented, but the scalar has also invented an order between Bus and Car although they are incomparable. Therefore, even a perfect pairwise subsumption prediction would show that hierarchy information can be read from the embeddings. It would not show that the ontology, or its lattice structure, is actually embedded in the LLM.
-
Before testing whether the LLM contains a lattice (which itself is restricted to only two LLM families of not much consequential difference with unclear justification of selection), the paper first replaces the SNOMED ontology (best represented via a Description Logics fragment) by a much simpler FCA object. The paper does not establish that this replacement preserves the structure being studied. For example, suppose a parent has
causative agent = dustand its child hascausative agent = inorganic dust. SNOMED knows that inorganic dust is a specialization of dust. In the flat FCA representation, these initially become two different(relation type, destination)attributes. The paper also ignores role-group identity. The paper's own results show how much this representation choice matters. With the flat attributes, 65.3% of true hierarchy edges appear incomparable. After the hierarchy among destination values is added back, the incomparable fraction falls to 0% and oracle F1 rises to 0.996. So a large part of the apparent lattice failure can be created or removed simply by changing how the Description-Logic ontology is converted into the FCA table. This makes the starting point of the structural claim unclear. Before asking whether the LLM represents the SNOMED lattice, the paper needs to show that the FCA object being tested is a faithful representation of the relevant SNOMED semantics.
Questions for authors
Clarification requests
-
How stable is the learned hierarchy direction to the particular concepts used to fit PCA? The paper reports stability across folds, but those folds change both the PCA population and the hierarchy-supervision data. Could you keep the hierarchy-training edges fixed, perturb only the concept embeddings used to fit PCA, and report how much the resulting PCA subspace and change? A stable under this test would make the interpretation of as a property of the frozen representation.
-
How much does the learned rely on the dominant PCA directions vs. weak PCA directions? Please report how the weight of is distributed over PCs together with their eigenvalues.
-
After destination-value closure, how much of the original SNOMED semantics is still lost by the FCA construction? In particular, role-group identity is ignored. Can you quantify whether preserving the omitted SNOMED structure changes the containment/comparability results? This would affect how much I can interpret the FCA results as statements about the SNOMED ontology itself.
Suggestions for future work
- Testing more genuinely different model families and scales would help establish how general the observed phenomenon is.
- A full order-recovery evaluation, including incomparability, branching, multiple inheritance, and meet/join structure, would be useful if the goal is to argue for an ontology or lattice embedding rather than pairwise subsumption decodability.
11 comments
W2, Q3: The attribute construction
You were right that the construction matters, and it matters on the embedding side too. The submission showed this for the full-attribute oracle: destination-value closure reduced the proportion of direct edges with incomparable attribute profiles from 65.3% to 0% and raised oracle macro-F1 to 0.996. We have now run the corresponding test on the embeddings. We used the same embeddings, training concepts, and released-code Incl rule in both conditions. We used the same 104 high-support attribute columns, excluding one of the original 105 because all training concepts became positive for it after closure. We fitted attribute directions separately with exact-value and value-closed labels; the latter also mark in-vocabulary ancestors of a recorded value as positive within the same relation type. The 7,153 matched direct edges are the subset of the 7,352 internal direct edges for which both sibling and unrelated candidates were available.
| Embedding-based inclusion score | Exact-value labels | Value-closed labels |
|---|---|---|
| Orientation accuracy, edges whose child was held out from attribute fitting (747) | 55.76% | 77.91% |
| Orientation accuracy, all matched edges (7,153; descriptive) | 59.46% | 82.04% |
| Forward-versus-reverse AUROC, all matched edges (descriptive) | 0.556 | 0.808 |
The weak orientation reported in the submission depended on how the attribute labels were constructed: incorporating the hierarchy of destination values improved orientation while keeping the attribute columns and scoring rule fixed. It should not be treated as a general limitation of LRH inclusion. The improvement was not limited to orienting known edges: on the same 747 held-out-child cases, value closure increased the rate at which Incl ranked the true parent above a matched sibling from 62.58% to 79.52%, counting ties as half.
What we will change. The revision will report both constructions side by side on the oracle and embedding sides, and will withdraw the general statement that inclusion scores cannot orient subsumption. The first half of the paper will be rewritten as a diagnosis, which is the reading your Strength 2 already gives it: exact matching omits coverage implied by the destination hierarchy. Restoring that coverage reduced incomparability in the oracle (65.3% → 0%) and improved orientation in the embeddings (55.8% → 77.9%).
What does not change. The grade does not use the attribute table in any form. It is fitted from embeddings and is-a edges alone, so none of the results in the next two sections depend on how attributes are constructed. We refit the grade on the same concept split, using only direct is-a edges whose endpoints were both in the training set, and evaluated all three readouts on the same 747 direct edges with a held-out child.
| Readout | What it requires | Paired orientation accuracy (747 edges) |
|---|---|---|
| Inclusion score, exact-value labels | 104 attribute directions | 55.76% |
| Inclusion score, value-closed labels | 104 attribute directions and the value hierarchy | 77.91% |
| Grade | one direction learned from embeddings with is-a supervision | 94.38% |
Value closure substantially improved Incl, yet the single linear grade was 16.47 percentage points more accurate at orienting these known parent–child pairs, without requiring an attribute table.
Two readouts of hierarchical direction. We read the two results as convergent. In the full-attribute oracle, value closure made the child's attribute set a superset of its parent's on every tested direct edge. The grade recovers an ordering of the same kind directly from the embeddings, without being given attributes or their hierarchy. On this reading the lattice and the grade are not rival accounts. The contribution of the grade is that hierarchical direction is available as a single linear coordinate, and transfers across branches, disease domains and ontologies.
The algebraic decomposition is unchanged by value closure wherever the attribute gate is nonempty. What changes empirically is how well the resulting inclusion scores distinguish the two orientations of a known edge.
Why a binary context. LRH is stated over a binary object–attribute formal context. To evaluate it with human-curated relations, we represented each recorded non-is-a relation type and destination as a binary attribute. Treating these pairs as atomic and flattening role groups were choices in our construction, not requirements of LRH, and a flat binary table is not a lossless rendering of SNOMED CT's description-logic semantics. We will say so explicitly.
Q3: Role groups after value closure. To preserve role grouping, we required each parent role group to be covered by a single child role group, rather than allowing its relations to be matched across groups. The cohort is the respiratory hierarchy: 7,352 direct edges and 41,532 sibling pairs that are incomparable in the reference graph.
| Attribute containment after value closure | Flattened role groups | Preserved role groups |
|---|---|---|
| Direct edges: forward containment (including bidirectional) | 7,352 / 7,352 | 7,352 / 7,352 |
| Direct edges: bidirectional containment | 596 | 589 |
| Sibling pairs: containment in at least one direction (out of 41,532) | 5,784 (13.93%) | 5,732 (13.80%) |
Preserving role groups retained forward containment on all 7,352 direct edges and removed reverse containment on seven. It removed attribute comparability for 52 of the 5,784 sibling pairs deemed comparable after value closure (0.90%). Even with groups preserved, 13.80% of sibling pairs that are incomparable in the reference graph still satisfy attribute containment in at least one direction.
W1 (a): The Vehicle / Bus / Car example
We agree with the example: Bus and Car can receive different grades without either subsuming the other. An ideal monotone grade satisfies when , but the converse does not hold. The learned grade approximates this ordering on true edges; a grade difference alone does not establish comparability.
Siblings are included as hard negatives in our relation-prediction evaluations, which is the Bus/Car case. The table reports mean AUROC across five held-out SNOMED CT disease domains. Both rows use the same positives: direct edges and sampled two-hop ancestor pairs.
| Test negatives | Cosine | Grade difference | Cosine + grade |
|---|---|---|---|
| All negative types | 0.630 | 0.851 | 0.905 |
| Siblings only | 0.450 | 0.858 | 0.844 |
Cosine alone does not reliably rank the positives above siblings in this comparison. Grade differences tend to be larger for the positives, but this statistical separation does not decide comparability for an individual pair. Under strict branch holdout, the Llama-3.1-8B combined scorer also reaches 0.843 AUROC against sibling negatives.
W1 (b), Q1, Q2: Supervised readout and PCA controls
We estimate with a supervised linear probe trained on known child–parent edges, and we will say so wherever it is introduced. Three controls bound what the supervision contributes. With shuffled training labels, the same pipeline falls to chance. The learned direction reaches 90.3% mean orientation accuracy under complete-branch holdout, and a direction fitted on either ontology retains an orientation signal on the other without refitting. And a direction with no reweighting at all, the mean child-minus-parent vector in the same PCA-64 subspace, already reaches 83.79% orientation accuracy and 0.898 AUROC on held-out branches, against 90.34% and 0.964 for logistic regression (cosine between the two directions: 0.661). The signal is present in the mean displacement; logistic regression refines it.
Q1: Perturbing only the PCA fit set. We refitted PCA on different concept sets, keeping the supervised training edges, test edges, regularization and scaling fixed, and then refitted the probe in each perturbed subspace. The first row is the full-fit baseline; the 80% and 50% rows use five draws of training-edge endpoints per held-out branch. The last row uses three draws per branch from a pooled set of four other disease domains after excluding all respiratory concepts. Values are means over branches and draws, compared with the full-fit baseline of the same branch.
| PCA fit set | PCA-64 subspace overlap | Cosine of to full-fit (original space) | Orientation accuracy | Orientation AUROC |
|---|---|---|---|---|
| 100% of training endpoints | 1.000 | 1.000 | 90.34% | 0.9640 |
| 80% | 0.973 | 0.985 (range 0.972–0.993) | 90.35% | 0.9633 |
| 50% | 0.921 | 0.960 (range 0.931–0.977) | 90.35% | 0.9632 |
| Out-of-domain concepts | 0.479 | 0.681 | 90.50% | 0.9645 |
Subspace overlap is the mean squared cosine of the principal angles. Under subsampling, the subspace and the direction are both stable. Under a change of population, the retained subspace changes substantially and the fitted vector moves with it, yet mean held-out performance remains close to the baseline after refitting. The orientation signal is therefore robust to the tested changes in the PCA fit set, while the particular vector is specific to the subspace it is fitted in. We will describe as one representative readout and not as a unique axis.
Thank you for the detailed and technically precise review. Your W2 prompted an additional experiment whose outcome changes part of our paper, and we report it in full below together with the three checks you requested. In summary:
What our results establish
Our results establish a supervised linear readout of hierarchical direction from frozen concept representations. They do not establish an embedding of the full SNOMED CT ontology or its lattice, and we agree that even perfect pairwise orientation would not establish recovery of the full partial order. The submitted wording blurred this distinction. We will revise the abstract, introduction, and Section 2.4 to state the empirical claim as approximate order preservation along true edges and to distinguish it from an order embedding.