acceptodds
Under review as a conference paper at ICLR 2027

Ordinal Axes in the Geometry of Ordered Categorical Knowledge in LLMs

Abstract

Nothing in the word viscount says that it outranks baron, yet language models answer most questions about such ordered categories correctly, even though their order is a convention that cannot be read off the words. Current accounts of concept geometry describe continuous, cyclic and unordered categorical concepts, but not how such orders are represented. We find that ordered categories are organised along an ordinal axis in the residual stream. The axis can be recovered from the activations of the two endpoint categories alone, and it places the held-out intermediate categories in their correct order across several language models and many domains, from ranks and titles to physical scales. We then show that a model uses this axis when it compares two categories. Exchanging their coordinates along this single direction, and nothing else, flips about a third of the decisions it gets right, whereas random directions of the same size flip far fewer. Finally, we show why a trained causal direction outperforms the ordinal axis. A one-dimensional direction trained to flip the comparison combines the ordinal representation with the gradient of the answer logits. Adding this gradient to the ordinal axis recovers much of the trained direction's effect. These results identify a geometric representation of ordered categorical knowledge, read and used by the model, and add ordered categories to the account of concept geometry.

Primary area:
interpretability and explainable AI
Keywords:
Large language models, concept geometry, residual stream, representation geometry, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_toQsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows that models represent the order of concepts along a direction. The authors first show that they are able to find this direction using difference of mean activations. They then show that by combining this direction with the gradient of the answer logits, they can identify a causal direction that enables controlled manipulation of the model.

Critical strengths

The main strength of this paper is that they were able to "build" a steering vector without any training. They did it by adding to the difference of means vector the gradient of the answer logits.

Critical weaknesses

I don't find the results novel or exciting enough. I think there are already a few papers that indicates that models uses an ordinal direction in the residual stream [1,2,3]. In addition most of the analysis techniques, like difference of means [4] or DAS , are also not novel.

References: [1] https://arxiv.org/pdf/2607.04167 [2] https://arxiv.org/pdf/2409.05448 - see ordering IDs [3] https://aclanthology.org/2023.findings-acl.383.pdf [4] https://arxiv.org/abs/2406.11717

Questions for authors

I don't have question that will materially affect my assessment. I found the paper clear.

4 comments
AuthorsOpenReview ↗

Reviewer toQs reply (Part 1/2).

We thank the reviewer for their review and for pointing us to these relevant works. However, we respectfully disagree that [1–4] anticipate our results. They provide important precedents for studying representation geometry and causal interventions, but they address different questions. [1] and [2] mostly study structural positions and order of mention, defined by the input, [3] studies representations of numerical magnitude, and [4] identifies a direction that mediates refusal. To our knowledge, none of them looks at how LLMs represent ordered categorical knowledge acquired in pretraining, or at whether an axis built only from the endpoint categories recovers the order of the intermediate ones and is used by the model when it compares them. In addition, [1] and [3] make no causal claim about the model's behaviour. Changes related to this review are highlighted in purple.

  • W1. "I don't find the results novel or exciting enough. I think there are already a few papers that indicates that models uses an ordinal direction in the residual stream [1,2,3]."

These papers do find directions related to order in the residual stream, and we have now included them in our paper. However, the orders they study are of a different kind, as they are given by the prompt ([1], [2]) or by the number itself ([3]). We study orders between categories that the model learned in pretraining, and we show that an axis built from the two endpoint categories places the remaining ones and is used by the model when it compares them. We go through each paper below.

[1] (Bassi & Tomar, 2026) is the closest to our work in name. It studies four ordinal variables: bracket nesting depth, Python indentation level, position in a markdown table and numeric magnitude. In the first three, the value is determined by the input sequence (e.g. the nesting depth of a token is the number of brackets still open at that point in the prompt), and in the fourth the number itself carries its magnitude. Our ladders, such as noble titles, military ranks or physical scales, are not stated in the prompt, so the order must come from what the model has learned. The methods also differ. [1] fits probes and PCA on all values, while our axis is built from the two endpoints, with the intermediate rungs held out. Finally, [1] states that its patching "tests informational concentration in the subspace, not behavioral causality", while we test whether the model uses the axis when it decides which of two rungs ranks higher.

[2] (Dai et al., 2024) does make a causal claim, but of a different kind. It studies the "ordering ID" of entities in prompts such as "The coffee is in Box Z, the stone is in Box M, the map is in Box H... Box Z contains the ___". The ordering ID is the position at which an item is mentioned: "coffee" has index 0 because it comes first, and would have another index if the sentence were reordered. In contrast, a viscount outranks a baron, and Saturn is farther from the Sun than Mercury, whatever the order in which they appear in the prompt. [2] finds its direction with PCA over all positions and intervenes by adding steps of a chosen size along it, which moves the retrieved object down the list. We build the axis from the two endpoints alone, exchange the coordinates of the two compared concepts, so the size of the intervention is set by the data, and test whether the model's judgment is reversed, against norm-matched random directions.

[3] (Shah et al., 2023) asks whether the embeddings of the numbers 1–9 in pretrained models (BERT, GPT-2, T5, ...) show the distance, size and ratio effects known from human number comparison, by testing whether the cosine similarity between two number tokens varies with those quantities. Their variable is numerical magnitude, where the symbol itself denotes the quantity, and each number is embedded on its own, outside any prompt. Apart from our two number ladders, our ladders have no numerical metric, so nothing in the word reveals its rank. [3] does not construct a direction or intervene on the model. Its aim is the cognitive plausibility of number representations, while ours is the geometry of learned categorical orders and its use in comparisons.

AuthorsOpenReview ↗

Reviewer toQs reply (Part 2/2).

In summary, earlier work has described the geometry of continuous concepts such as space and time (Gurnee & Tegmark, 2024), cyclic concepts such as days and months (Engels et al., 2025), and categorical and hierarchical concepts (Park et al., 2025). Ordered categorical concepts were missing from this picture. We show across 26 ladders and four models that they are arranged along an axis fixed by their two endpoints, and that the model uses this axis when it compares them. We also explain why a trained direction is more effective, and build one without training by combining the endpoint axis with the answer-logit gradient, which the reviewer highlights as the main strength of the paper. Thus, we believe these results add a missing piece to the account of concept geometry rather than reapplying known methods.

  • W2. "In addition most of the analysis techniques, like difference of means [4] or DAS, are also not novel."

[4] shows that a difference-in-means direction can causally mediate a behaviour, and we agree that our tools, difference-in-means, interventions and DAS, are standard. We do not claim them as a contribution. What we claim is what they reveal about ordered knowledge. The simplicity of the method is also what makes it valuable. The axis is built from the two endpoints alone, so the intermediate rungs play no part in it, and finding them in order along the axis is a held-out result.

All in all, we agree that these works are relevant and should have been discussed. We have added them to the related work of the revised version, [1] and [2] as studies of orders defined by the input (structural position and order of mention), [3] as evidence of magnitude structure in number embeddings, and [4], together with Belrose (2023), among the applications and foundations of difference-in-means. They do not, however, cover the core contributions of our paper: the endpoint axis for orders learned in pretraining, its causal role in comparisons, and the construction of an effective direction without training.

We hope that these clarifications and the additions to the related work address the reviewer's concerns, and that the reviewer will consider revising their assessment.

References:

[1] Bassi, S. and Tomar, S. Geometry of ordinal representations in language models. arXiv:2607.04167, 2026.

[2] Dai, Q., Heinzerling, B., and Inui, K. Representational analysis of binding in language models. arXiv:2409.05448, 2024.

[3] Shah, R. S., Marupudi, V., Koenen, R., Bhardwaj, K., and Varma, S. Numeric magnitude comparison effects in large language models. Findings of ACL, 2023.

[4] Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction. NeurIPS, 2024.

Belrose, N. Diff-in-means concept editing is worst-case optimal: Explaining a result by Sam Marks and Max Tegmark. EleutherAI Blog, 2023. https://blog.eleuther.ai/diff-in-means/

Engels, J., Michaud, E., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are one-dimensionally linear. ICLR, 2025.

Gurnee, W. and Tegmark, M. Language models represent space and time. ICLR, 2024.

Park, K., Choe, Y. J., Jiang, Y., and Veitch, V. The geometry of categorical and hierarchical concepts in large language models. ICLR, 2025.

Thank you for your response and for the further discussion. I will update my score.

Thank you for taking the time to consider our rebuttal. We are glad that our response addressed your concerns and appreciate your further assessment of our work.

Reviewer_M86nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether ordered categorical knowledge is organized along an ordinal direction in LLM residual streams. An axis built only from endpoint categories orders held-out intermediate categories, and swapping option coordinates along that axis changes comparison behavior. The paper further decomposes a trained DAS direction into a natural ordinal and an out-of-span component aligned with answer sensitivity. I regard this decomposition, rather than the existence of ordinal geometry itself, as the paper's main contribution.

Critical strengths
  1. Clean construction and validation. Building the axis only from the endpoints makes the ordering of intermediate categories meaningful, and the paper includes careful intervention controls.

  2. Useful extension of the read-write distinction. The strongest contribution is the empirical decomposition of the optimized DAS direction into a component describing natural ordinal variation and another providing downstream steering leverage. The explicit read/write interventions and gradient-based approximation make this a useful methodological result.

Critical weaknesses
  1. The largest concern is uncontrolled pretraining exposure. Many of these orders may appear explicitly in text as pairwise comparisons, ordered lists, or tables. The intermediate categories are held out only from fitting the axis, not from pretraining. The experiments therefore cannot distinguish an emergent representation of global order from a representation built from memorized or strongly hinted ordering facts. This materially limits the claim: the paper establishes how already learned order is represented and used, but not how it is acquired. A decisive experiment would train a smaller model on a controlled synthetic corpus with nonce labels and known relational exposure, in the spirit of Physics of Language Models: Part 3.1/3.2. For example, one could provide only adjacent relations or systematically varied hints, then test unseen comparisons and the emergence of the axis.

  2. The remaining contribution is somewhat marginal relative to prior work. Ordinal and monotonic representation geometry is anticipated by work on spatial, temporal, numerical, categorical, and ordinal variables. Moreover, Makelov et al. (ICLR 2024) already showed that an optimized patching direction can combine a naturally discriminative component with a dormant but causally powerful component. The present paper advances this idea through a broad empirical decomposition and a gradient-based account, but it should state this delta explicitly.

These concerns leave me borderline

Questions for authors
  1. Have the authors performed any controlled-training experiment in which the order and every exposure to it are known? If not, would the endpoint axis still emerge for nonce categories when the model sees only adjacent relations and is tested on unseen distant pairs?

  2. What is the precise advance over Makelov et al.'s disconnected-plus-dormant construction?

  3. How does the endpoint axis compare with one-dimensional random directions sampled within the rung-centroid span and matched for projection variance or exchange magnitude?

2 comments
AuthorsOpenReview ↗

Reviewer M86n reply (Part 1/2).

We thank the reviewer for their suggestions and insightful comments.

The changes discussed below are already changed in the new submitted version. Where possible, we have highlighted the changes related to your review in orange.

We will answer related Weaknesses and questions together.

  • W1. "The largest concern is uncontrolled pretraining exposure. Many of these orders may appear explicitly in text as pairwise comparisons, ordered lists, or tables. The intermediate categories are held out only from fitting the axis, not from pretraining. The experiments therefore cannot distinguish an emergent representation of global order from a representation built from memorized or strongly hinted ordering facts."

  • Q1. "Have the authors performed any controlled-training experiment in which the order and every exposure to it are known? If not, would the endpoint axis still emerge for nonce categories when the model sees only adjacent relations and is tested on unseen distant pairs?"

We agree with this observation. To address this limitation, and a related concern raised by Reviewer nY1i, we have added two experiments with invented hierarchies (SM Section C). These allow us to examine whether the same ordinal structure appears for orders that the models could not have encountered during pretraining.

In Section C.1, following Reviewer nY1i's suggestion, we introduce a new hierarchy through a paragraph in the prompt. In Section C.2, closer to your suggestion, we fine-tune models on comparisons between invented words, with full control over which relations are presented during training. The ranks are assigned at random, and some pairs, including the endpoint pair, are never shown during training. We note that the two settings are conceptually different as in the first the order is held only in the activations of that prompt, while in the second it is stored in the weights, as with orders learned in pretraining.

In both settings, when the model correctly answers comparisons about the new hierarchy, the intermediate ranks are ordered along the endpoint-defined direction, and exchanging coordinates along it reverses the model's comparisons, as in the main experiments. These results show that the ordinal geometry is not limited to familiar hierarchies encountered during pretraining. However, our focus remains on how learned orders are represented and used, so understanding how these representations develop during learning is beyond the scope of this work. We discuss these experiments in Sections 4.1, 4.2 and 5.

We also tested fine-tuning using only adjacent pairs, as you suggested. The models generalise to the unseen distant pairs less often in this setting (7 of 12 cases on Llama-3.1-8B and 1 of 9 on Qwen2.5-32B). When they do, the endpoint axis emerges and the exchange reverses comparisons, although the order along the axis is weaker than when non-adjacent pairs are also included in training. For this reason the paper reports the balanced setting.

  • W2. "Moreover, Makelov et al. (ICLR 2024) already showed that an optimized patching direction can combine a naturally discriminative component with a dormant but causally powerful component. The present paper advances this idea through a broad empirical decomposition and a gradient-based account, but it should state this delta explicitly."

  • Q2. "What is the precise advance over Makelov et al.'s disconnected-plus-dormant construction?"

Thank you for raising this point. As the reviewer notes, our decomposition of the trained DAS direction matches the one that Makelov et al. describe for directions found by optimisation. The in-span component (u∥u_\parallel) plays the role of their component that distinguishes the inputs, and the out-of-span component (u⊥u_\perp) that of their dormant component, along which the activations barely vary but to which the output is sensitive.

However, the two differ in the first component. In theirs, it is causally disconnected from the output. In our case, the centroid space that contains u∥u_\parallel is causally connected to it, as exchanging coordinates along the endpoint axis (v^\hat v), which lies in this space and involves no optimisation, reverses comparisons (Section 4.2 and SM Section B.3). In addition, we show that the dormant component can be approximated by the part of the answer-logit gradient outside the centroid space, so an effective direction can be built by combining the endpoint axis with this gradient, without DAS training. We now state this relation to Makelov et al. explicitly in Section 4.3.

AuthorsOpenReview ↗

Reviewer M86n reply (Part 2/2).

  • Q3. "How does the endpoint axis compare with one-dimensional random directions sampled within the rung-centroid span and matched for projection variance or exchange magnitude?"

We had not tested this in the original submission. We have now added in-span random-direction controls matched for either exchange magnitude or projection variance (SM Section B.3, Table 8). The endpoint axis has a stronger causal effect than the average of these controls, showing that its effect is not simply due to intervening within the rung-centroid span. Nonetheless, the rank direction fitted to all rungs reverses more comparisons than the endpoint axis, so we observe that the endpoint axis is not the most effective direction in the span.

Reviewer_nY1iOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

In this work, the authors assess whether language models have ordinal representations in line with the literature regarding high-level concepts in the activation space by studying the model geometry. They performed experiments in which an ordinal axis was explored using forced-choice questions between different rungs of a ladder, computing centroid representations of the “first” and the “last” concept of an ordinal structure of concepts (for example, from first to last), and then to explore whether intermediate concepts can be positioned in an ordered way along the axis between first and last. Then, the authors exchange the projected coordinates of the compared categories of the rungs and test whether exchanging these coordinates reverses previously correct model decisions. Their experiments recovered the ordinal ordering of held-out intermediate categories along the axis constructed from the first and last categories, and their causal intervention showed results that are better than for random directions in activation space. For the authors, this implies that ordered categories are arranged along a direction in the ordinal axis.

Critical strengths

This paper represents clear and well organized work, with an organized presentation of the ideas, providing important evidence for the internal geometry of ordinal categories along the residual stream of large language models, and of causal effects in the internal representations of the model. The experiments are sound, and the results are very interesting in terms of interpretability of AI. It favors a deeper comprehension of the model's internal geometry and is a good starting point in terms of understanding ordered categorical knowledge representations in LLMs, with results going in the direction of ordinal categories, but with potential applications for other of the kind such as magnitude, number and cardinal representations.

Critical weaknesses

To strengthen the claims and the results of the paper, I think cross-linguistic generalization may be pivotal. Furthermore, inclusion of experiments with contextualized unfamiliar hierarchies introduced within a short paragraph may shed light on the role of ordinal geometry outside explicit comparison tasks, and familiar learned context. Finally, understanding how ordinal representations work in human cognition would enrich the author’s theoretical construction of their geometrical representation and their interpretation regarding the base of the acquisition of ordinal representations in LLM models.

Questions for authors
  1. The authors discussed geometry of LLM representations, in terms of high-level concepts. Could the authors elaborate on how their results can be interpreted in terms of what is known regarding concept geometry? What new interpretations do these results support beyond existing frameworks?
  2. Given the differences in how intermediate rungs are distributed along the ordinal axes for different concept categories, do the authors think that these results support a shared ordinal representation across categories, or might different categories have different ordinal representations?
  3. I think it would be useful to give a look at how processing of ordinal categories is structured in human cognition. Do you think that theoretical models of human ordinal cognition offer competing explanations regarding the observed structure of LLM ordinal internal representations?
  4. I think a highly informative extension of the work could be to analyze ordinal representations in whole paragraphs, with information that is introduced within the paragraph itself. For example, if we have a paragraph talking about a newly invented martial art and their different ranks, could we extract ordinal representations of these previously unfamiliar categories? Furthermore, could the authors' causal intervention influence the model's subsequent judgments about the newly introduced hierarchy?
  5. Do the authors think that the number of rungs would influence the results of their experiments? Could the number of rungs affect how ordinal relationships are represented or recovered?
  6. Does semantic distance between different rungs influence the results? Are rungs with lower semantic distance easier or harder to build ordinal representations on?

If not feasible, may be interesting for future work:

  1. Given that linguistic representations may have different ordinal organization structure (for example, the difference between grammatical organization of English vs. Chinese languages may cause differences in their ordinal representation), I think it would be important to have information regarding cross-linguistic generalization. I know this is something the authors acknowledge in the limitations, but I think that such additional experiments may further strengthen the points raised by the authors. This may also allow for accounting language specific shifts into the activation space.
4 comments
AuthorsOpenReview ↗

Reviewer nY1i reply (Part 1/2).

We thank the reviewer for their thoughtful and helpful comments!

The changes discussed below are already changed in the new submitted version. Where possible, we have highlighted the changes related to your review in green.

We will answer related Weaknesses and questions together.

  • W1. "To strengthen the claims and the results of the paper, I think cross-linguistic generalization may be pivotal."

  • FW1. "Given that linguistic representations may have different ordinal organization structure (for example, the difference between grammatical organization of English vs. Chinese languages may cause differences in their ordinal representation), I think it would be important to have information regarding cross-linguistic generalization. I know this is something the authors acknowledge in the limitations, but I think that such additional experiments may further strengthen the points raised by the authors. This may also allow for accounting language specific shifts into the activation space."

We agree that cross-linguistic generalisation is an important question. Following your suggestion, we have added experiments with Chinese prompts, using four ladders translated from English and two hierarchies that exist only in Chinese (SM Section F). We find that exchanging coordinates along the axis built from the Chinese prompts changes the models' comparisons on most ladders, showing that the causal effect is not limited to English. Interestingly, the English axis also affects comparisons in Chinese for some ladders but not for others, suggesting that ordinal representations may contain both shared and language-specific components. We have also added a brief discussion of these findings to the limitations in the main paper.

  • W2. "Furthermore, inclusion of experiments with contextualized unfamiliar hierarchies introduced within a short paragraph may shed light on the role of ordinal geometry outside explicit comparison tasks, and familiar learned context."

  • Q4. "I think a highly informative extension of the work could be to analyze ordinal representations in whole paragraphs, with information that is introduced within the paragraph itself. For example, if we have a paragraph talking about a newly invented martial art and their different ranks, could we extract ordinal representations of these previously unfamiliar categories? Furthermore, could the authors' causal intervention influence the model's subsequent judgments about the newly introduced hierarchy?"

Thank you for this suggestion. We have added an experiment in which the models are given a paragraph describing an invented hierarchy, with randomly assigned category names and relations presented out of order (SM Section C.1).

We find that, once the models can compare the new ranks, the intermediate ranks are ordered along a direction constructed from the endpoints, as in our main experiments. Importantly, exchanging coordinates along this direction also reverses some of the model's correct comparisons, showing that the representation has a causal role even when the hierarchy is introduced only through the prompt. We have also added a complementary experiment, addressing a related concern raised by Reviewer M86n, in which models are fine-tuned on a controlled subset of comparisons between invented categories (SM Section C.2). We observe similar geometric and causal effects in this setting. These results are now discussed in Sections 4.1 and 4.2 of the revised manuscript.

  • W3. "Finally, understanding how ordinal representations work in human cognition would enrich the author's theoretical construction of their geometrical representation and their interpretation regarding the base of the acquisition of ordinal representations in LLM models."

  • Q3. "I think it would be useful to give a look at how processing of ordinal categories is structured in human cognition. Do you think that theoretical models of human ordinal cognition offer competing explanations regarding the observed structure of LLM ordinal internal representations?"

Good question! We think that some findings in human ordinal cognition suggest parallels with the representations we observe in language models, particularly the use of one-dimensional structures to represent learned orders. However, we cannot conclude that the underlying mechanisms are the same, and a detailed comparison with theories of human ordinal cognition is beyond the scope of this paper. We have added a paragraph to the Discussion addressing these connections and their limitations, as well as the possibility of exploring them in future work.

AuthorsOpenReview ↗

Reviewer nY1i reply (Part 2/2).

  • Q1. "The authors discussed geometry of LLM representations, in terms of high-level concepts. Could the authors elaborate on how their results can be interpreted in terms of what is known regarding concept geometry? What new interpretations do these results support beyond existing frameworks?"

We see our work as another piece in the emerging picture of concept geometry in language models. Previous work has shown that different types of concepts exhibit different geometric structures: continuous concepts such as space and time (Gurnee & Tegmark, 2024), cyclic concepts such as days and months (Engels et al., 2025), and categorical and hierarchical concepts (Park et al., 2025).

Our work complements these findings by studying ordered categorical concepts. Our concept elements are discrete categories, without necessarily having an intrinsic continuous scale, but with an underlying ordinal relationship. Interestingly, this structure appears across semantically very different domains. Our results suggest that, despite their semantic differences, ordered categorical concepts are represented through a similar geometric mechanism. This extends the current picture of concept geometry by showing that such a mechanism also applies to discrete categories whose ordering is relational rather than inherently numerical.

  • Q2. "Given the differences in how intermediate rungs are distributed along the ordinal axes for different concept categories, do the authors think that these results support a shared ordinal representation across categories, or might different categories have different ordinal representations?"

Very interesting question! We have added an analysis to investigate it (SM Section D.1). We build a direction by averaging the endpoint axes of 25 ladders and test it on the remaining one. This shared direction recovers part of the order of the held-out categories, although less well than the ladder's own axis. Exchanging coordinates along it also changes comparisons in some models. This suggests that ordinal representations have both a shared and a category-specific component. In addition, in the experiments with Chinese prompts we find that, for some ladders, the axes obtained in the two languages are similar, which suggests that part of the ordinal representation is also shared across languages (SM Section F). We would like to explore both questions further in future work. The geometric result is now also reported in Section 4.1.

  • Q5. "Do the authors think that the number of rungs would influence the results of their experiments? Could the number of rungs affect how ordinal relationships are represented or recovered?"

We have added an analysis of this question in SM Section D.2. Across our 26 ladders, we find no significant relationship between the number of rungs and how well the intermediate categories are ordered along the ordinal axis. The causal effect of exchanging coordinates, however, tends to decrease with the number of rungs, although this relationship is significant for only one model. Given the limited number of ladders, we cannot rule out a dependence on ladder length, but our results suggest that the geometric ordering is relatively stable across the lengths considered.

  • Q6. "Does semantic distance between different rungs influence the results? Are rungs with lower semantic distance easier or harder to build ordinal representations on?"

This is an interesting question. We have added an analysis of how the causal effect varies with the distance between compared ranks (SM Section D.2). We find that exchanging the ordinal coordinate has a stronger effect when the two categories are farther apart in the hierarchy. However, this measures ordinal distance rather than semantic similarity between the categories. We agree that semantic similarity could influence how easily ordinal representations are recovered, but we have not tested this directly. We think this would be an interesting direction for future work.

I would like to thank the authors for their response and their work during this rebuttal session. I appreciate the new experiments and clarifications, which have substantially addressed my concerns. I will increase my score and recommend accepting the manuscript.

Thank you for taking the time to review our work. Your input has been very valuable in strengthening the manuscript, and we truly believe the paper is better as a result.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Comment to all reviewers regarding the revision.

We thank all three reviewers for their careful reading of our paper and their constructive feedback. Their comments have made the paper considerably stronger. In the revised version, the changes are highlighted in a different colour for each reviewer, and we address the comments and questions individually below.

We have also highlighted in blue a few general changes relative to the original submission, independent of the reviewers' specific requests. These include correcting the token positions used for the military-rank and education ladders, rerunning the experiments with fixed random seeds (which led to minor changes in the reported values), extending the Qwen2.5-32B analysis in Section 4.3 from 8 to 25 ladders, and refining the layer-selection procedure in Section 4.3.

We would be happy to answer any further questions that may arise from these changes during the discussion period.

AuthorsOpenReview ↗

Final Remarks

As the discussion period comes to a close, we would like to thank all reviewers for their valuable feedback and constructive suggestions, which have helped us improve the manuscript. We sincerely appreciate the time and effort devoted to evaluating our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.