acceptodds
Under review as a conference paper at ICLR 2027

Representations in Motion: Layer-wise Rotation and Trajectory Divergence in LLMs

Abstract

Recent work suggests that activation dynamics–how representations change during inference–might be more revealing about language model behavior than static readouts from individual layers. In this work, we investigate the layer-wise evolution of representations of factuality using true and false statements when the model is prompted to either lie or to be honest. We report a geometric phenomenon in which the factuality directions extracted from the lying and honest conditions appear to rotate away from each other across layers. We find that such layer-wise rotation also occurs in a task that requires the model to take different visual perspectives in the 3D world, suggesting that the rotation phenomenon might be indicative of manipulating complex conceptual spaces in general, rather than unique to factuality. We then investigate the "real time" transformation of conceptual representations on a trial-to-trial basis. We show that it is possible to make strong predictions about the model's target task and behavior from trajectories alone. In addition, diverging points in single-trial trajectories correspond to the onset of rotation in aggregated-trial analysis. Finally, we show that both rotation and divergent trajectories occur in more naturalistic settings in which models are not overtly prompted to "lie" but rather report false statements in the course of completing a different task.

Primary area:
interpretability and explainable AI
Keywords:
Representation, dynamics, geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AvQaOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studied how the internal representations in LLMs changes across layers when instructed with lie, force agree with incorrect opinions, or reason from a different perspective. I am not an expert in this field. I think the authors observe a phenomenon they call representations rotation, where the directions representing concepts such as true/false change across layers. They also show that internal activating trajectory can help distinguish hones and dishonest answers.

Critical strengths
  1. An interesting perspective on LLM interpretability. The paper looks at the changes in representations across layers.
  2. Good experiment coverage. The authors test the phenomenon across different models, tasks and prompts. This helps demonstrate the observation in many different settings.
Critical weaknesses

I am not convinced about the discussion that observed rotation reveals a new mechanism for how LLMs manipulate information. When a model is instructed to lie, it needs to change its answer from true to false. Similarly, the spatial reasoning task involves changing left to right. Could the observed rotation simply reflect these changes in the expected answers? The activation patching experiments are interesting, but I don't think this establishes that rotation itself causes the behaviour.

Questions for authors
  1. How can you distinguish the proposed rotation mechanism from the simpler explanation that the model is just changing its expected answer? Would a basic answer-reversal task produce similar results?
  2. Could you explain, see Table2, why sycophancy is not significant while hallucination is? How does this affect the main conclusions?
1 comment

Thank you for your review. We provide a detailed response below.

W1, Q1: Is rotation just the model changing its answer?.

  • We agree that changing the expected answer could be an interpretation of the observed rotation phenomenon. However, we believe that changing the expected answer is still a phenomenon worth studying. It means that the model has a notion of the "expected answer", but outputs something different. Our analyses support this: under the lying instruction, the truth can still be read out in late layers (probe accuracy of at least 0.96), even though the model outputs the opposite. Regarding a basic answer reversal task, we expect it to show a similar pattern, since it also requires the model to output the opposite of an answer it represents.

Q2: Significance of rotation metric for sycophancy and hallucination.

"…why sycophancy is not significant while hallucination is? How does this affect the main conclusions?"

  • The rotation metric r is meant to summarize the cosine similarity across layers in a single number. We acknowledge that a single number can hide details, and the counterintuitive results between hallucination and sycophancy reflects limitations of r. For sycophancy, the directions are only moderately aligned to begin with, so even a clear rotation to opposite gives a drop no larger than what shuffled labels produce. For hallucination, as noted in Section 3.2, the representations are not well captured by a single linear direction, so their cosine similarity fluctuates across layers. Because r takes the largest drop over all early/late layer pairs, these fluctuations alone yield a small but significant value (0.88, the lowest of all tasks), even though the heatmap (Fig. 14) shows no consistent transition from aligned to opposite.

  • To check that our conclusions do not depend on these limitations, we recomputed rotation with two metrics that account for where the directions start:

    • Relative rotation, (c_init − c_final) / max(|c_init|, |c_final|), measures the change in cosine similarity relative to its starting or ending value, whichever is larger in magnitude. It ranges from −2 to 2, where 2 means a full rotation from aligned to opposite.
    • Normalized angular displacement measures the change in angle between the two directions, as a fraction of the largest change possible from the starting angle. It ranges from −1 to 1, where 1 means the directions end up fully opposite.
  • While the original metric indicates significant rotation in hallucination but not in sycophancy, both new metrics indicate significant rotation in lying, sycophancy, and spatial reasoning, but not in hallucination. This pattern is more consistent with the phenomenon observed in the raw data and highlights the importance of accounting for initial alignment (Appendices B2 and C3).

TaskEarly to late drop(paper)┃Relativerotation┃Normalized angulardisplacement
r (max 2)p┃r (max 2)p┃r (max 1)p
Lying1.45<10⁻⁴┃1.47<10⁻⁴┃0.60<10⁻⁴
Sycophancy1.160.632┃1.60<10⁻⁴┃0.49<10⁻⁴
Spatial reasoning1.40<10⁻⁴┃1.42<10⁻⁴┃0.57<10⁻⁴
Hallucination0.883.0×10⁻⁴┃0.181.00┃0.130.99
Reviewer_LosWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the activation dynamics in LLMs, i.e. how internal representations of factuality evolve across layers. Given true or false statements, a model is instructed either to stay honest or to lie. The internal representations rotates and eventually in opposite direction. Further experiments are conducted on sycophantic and hallucination and spatial relation scenarios. Analyses on interpretability and trajectories of representations are performed next. An experiment on more natural settings is performed to verify the findings in a setting without explicit lying instruction.

Critical strengths
  • The manuscript is well written, easy to follow and understand.

  • The representation dynamics follows the pattern that starts by formulating clusters in early layers and rotating in late layers. The rotation also exists when using paraphrased prompts, different task settings with the same purpose i.e. sycophantic and task with implicit purpose.

  • The rotation property can also be observed in a spatial reasoning task.

  • Trajectories analyses confirms the property of formulating clusters in early layers and rotating in late layers.

Critical weaknesses
  • The claim of 'activation space rotation is a general geometric property of tasks requiring active representation transformation, rather than a feature unique to factuality or “true-false” dimensions' is too strong, only spatial reasoning task is studied and it is not enough to say 'a general geometric property'.

  • The metric cosine similarity/r for measuring rotation is not robust and intuitive enough. For instance, r for Sycophancy and Hallucination are both close to 1 but their internal representation and rotation are quite different.

Questions for authors
  • In the spatial reasoning task, what if there are two objects? Will the rotation be observed same as the one object case? Moreover, what if Bob is not sitting on the opposite of Alice but sitting on the side of table. The object is thus in front of Bob, the direction is then orthogonal. Will the rotation also stop at orthogonal rather than opposite? Experiment on these settings will be interesting, it examines how rotation behaviour in a more complicated scene and whether the rotation of internal representation correlates with rotation in observation space.
1 comment

Thank you for your review. We provide a detailed response below.

W1: Generality claim. We do not claim that rotation is a universal mechanism; rather, our goal is to document and characterize an interesting phenomenon that merits further study. Our extension to spatial reasoning tasks provides preliminary evidence of the phenomenon's generalizability, while a more comprehensive characterization remains an open question. In particular, extending the investigation to non-binary and more complex tasks is an important direction for future work. We acknowledge this limitation in the Limitations section.

W2: Rotation metric is not intuitive enough.

  • The rotation metric r is meant to summarize the cosine similarity across layers in a single number. We agree that a single number can hide details, and the reviewer's example (similar values for sycophancy and hallucination) reflects limitations of r. For sycophancy, the directions are only moderately aligned to begin with, so even a clear rotation to opposite gives a drop no larger than what shuffled labels produce. For hallucination, as noted in Section 3.2, the representations are not well captured by a single linear direction, so their cosine similarity fluctuates across layers. Because r takes the largest drop over all early/late layer pairs, these fluctuations alone yield a value (0.88) not far from that of sycophancy (1.16), even though the heatmap (Fig. 14) shows no consistent transition from aligned to opposite.

  • To address this, we recomputed rotation with two metrics that account for where the directions start.

    • Relative rotation, (c_init − c_final) / max(|c_init|, |c_final|), measures the change in cosine similarity relative to its starting or ending value, whichever is larger in magnitude. It ranges from −2 to 2, where 2 means a full rotation from aligned to opposite.
    • Normalized angular displacement measures the change in angle between the two directions, as a fraction of the largest change possible from the starting angle. It ranges from −1 to 1, where 1 means the directions end up fully opposite.
  • While the original metric indicates significant rotation in hallucination but not in sycophancy, both new metrics indicate significant rotation in lying, sycophancy, and spatial reasoning, but not in hallucination. This pattern is more consistent with the phenomenon observed in the raw data and highlights the importance of accounting for initial alignment (Appendices B2 and C3).

TaskEarly to late drop(paper)┃Relativerotation┃Normalized angulardisplacement
r (max 2)p┃r (max 2)p┃r (max 1)p
Lying1.45<10⁻⁴┃1.47<10⁻⁴┃0.60<10⁻⁴
Sycophancy1.160.632┃1.60<10⁻⁴┃0.49<10⁻⁴
Spatial reasoning1.40<10⁻⁴┃1.42<10⁻⁴┃0.57<10⁻⁴
Hallucination0.883.0×10⁻⁴┃0.181.00┃0.130.99

Q1: Extending the spatial reasoning task.

  • Thank you for these suggestions. We implement both task settings and report the behavioral results below. Overall, we find the models are able to do the two object task but not the 90° rotation task.
    • Two objects: one on each side of the table, with the question naming one of them.
    • 90 degree rotation: person B sitting at the side of the table, with the object at one of the four corners.
SettingModelPerson APerson B
Two objectsLlama-3.1-8B1.000.50
Two objectsLlama-3.3-70B1.001.00
90° rotationLlama-3.1-8B0.530.54
90° rotationLlama-3.3-70B0.460.51
  • Given that the only setup with non-chance performance is Llama-3.3-70B on the two objects task, we study the rotation metrics r of this task below.
SettingModelEarly-to-late drop (paper)Relative rotationAngular displacement
One objectLlama-3.1-8B1.521.420.57
Two objectsLlama-3.3-70B1.651.750.68
  • We think it would be interesting to study model behavior (or its failure) specifically on these spatial reasoning tasks, which could involve constructing cognitive maps, and we leave this for future work.
Reviewer_ER4QOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study how the true/false direction inside an LLM changes across layers when the model is told to lie. They call "factuality direction" the difference between the mean activations for true and for false statements. The authors report that the directions for honest and lying prompts agree in early layers and become opposite in late layers, which they call "rotation". They report the same for sycophancy and for a left/right perspective-taking task, but not for hallucination. They observe that patching a few mid-layer attention heads from lying into honest runs shifts both the answer and the rotation. Projecting single trials onto three directions (true/false, honest/lie, lie success/failure) often separates honest, successful-lie and failed-lie trials.

Critical strengths
  • Following a concept direction across all layers, rather than reading it at one layer, is a worthwhile perspective to take. Crossing honest/lying prompts with true/false statements, with PCA views at every layer (Fig. 1, App. B), makes the transformation easy to see.

  • The pattern is tested across many prompt phrasings (156 for lying) and three model families, and cases where the model fails to follow the instruction serve as a built-in control (Table 4).

  • Extending the analysis to perspective taking and to goal-driven lying in role play is a good test of generality.

Critical weaknesses
  1. The "rotation" may be built into the measurement. If I am not mistaken, the direction is defined by whether each statement is actually true, and only trials where the model obeyed are kept, so under the lying prompt the model always answers the opposite of the truth. At the last prompt token, the final layers must encode the answer the model is about to give. The lying direction therefore has to end up opposite the honest one, whatever happens inside. A model that writes its answer into a new direction while the fact fades would produce the same curve, with nothing rotating. For the same reason, the curve must appear in any task where the answer is the opposite of the label, so finding it in sycophancy and perspective taking does not show a shared mechanism. To show a genuine rotation, the authors could check two things. Does the distance between the true and false averages stay constant through the middle layers, as a rotation predicts, or shrink, as a mixture predicts? And can the truth still be read out in late layers? The one result that could argue against this account, prompts on which the model fails to lie but which still show rotation (Table 4), rests on only 6 prompts.

  2. The closest prior work is not cited. Please take a look at Campbell et al. (2023, "Localizing Lying in Llama") -- they told Llama-2 to lie on statements from the same Azaria & Mitchell dataset. If I remember correctly, they found that the honest and lying directions agree in early layers and become opposite later, and by patching they located a few attention heads that largely restore honest answers. The paper should cite this work and explain which aspects of their work are the contribution.

  3. The paper's own statistics do not back some claims. Under the authors' permutation test (Table 2), rotation for sycophancy is not significant (p = 0.63): shuffled labels give similar values. Yet the text reports it as an observation. The hallucination control, presented as showing no rotation, is significant. A simple baseline is also missing: how much "rotation" appears between two different honest prompts?

  4. The causal claim is overstated. The introduction promises evidence of a causal link between rotation and behavior, but Section 4 concludes that it does not establish one (l. 323). In addition, the behavioural measure, as described, uses the absolute logit difference, under which a complete flip of the answer would show as no change.

  5. The trajectory classifier lacks a baseline. Lying and honest runs differ in their system prompt, so separating them is easy. The paper suggests that monitoring should use the whole path through the layers, but never compares this with a probe at a single layer. It would also help to show from which layer a failed lie can be predicted.

Questions for authors
  1. What does the paper add beyond Campbell et al. (2023)?

  2. Does the distance between the true and false averages stay constant across the middle layers? Can the truth still be read out in late layers? Do directions defined by the model's answer also rotate?

  3. How much rotation appears between two different honest prompts?

  4. What fraction of answers flip after patching the top heads?

  5. How does the trajectory classifier compare with a single-layer probe?

3 comments
AuthorsOpenReview ↗

Response 1/3

Thank you for your review. We provide a detailed response below.

Several points raised in the review (Weaknesses 1 and 3, and Question 3) concern the rotation metric r. We therefore clarify the motivation for and improvements to the rotation metrics. In summary, we introduced two complementary metrics that provide a more robust characterization of the phenomenon. Both metrics support our main findings: rotation is observed across multiple tasks, including lying, sycophancy, and spatial reasoning, but not hallucination (the negative control).

Below, we first explain the motivation for and limitations of the original metric. We then introduce the two new metrics and compare their results with those of the original metric.

The original rotation metric, r, summarizes the change across layers with a single number: the largest decrease in cosine similarity between an early and a late layer. This single-number summary has a limitation: it captures the absolute change in cosine similarity but does not account for the initial alignment of the directions. For example, in sycophancy, the mean cosine similarity decreases from +0.60 to −0.56, a substantial change of 1.16. However, because the directions are only weakly aligned initially, this change is no larger than that produced by shuffled labels in the permutation test (as mentioned in Weakness 3).

To assess whether our conclusions depend on the choice of metric, we computed rotation using two additional metrics, both of which account for initial alignment:

  • Relative rotation: (c_init − c_final) / max(|c_init|, |c_final|). This measures the change in cosine similarity relative to the larger-magnitude of the initial and final values. It ranges from −2 to 2, with 2 corresponding to a full rotation from aligned to opposite directions.

  • Normalized angular displacement: This measures the change in angle between the two directions as a fraction of the largest angular change possible from the initial angle. It ranges from −1 to 1, with 1 corresponding to the directions ending up fully opposite.

While the original metric indicates significant rotation in hallucination but not in sycophancy, both new metrics indicate significant rotation in lying, sycophancy, and spatial reasoning, but not in hallucination. This pattern is more consistent with the phenomenon observed in the raw data and highlights the importance of accounting for initial alignment (Appendices B2 and C3).

TaskEarly to late drop(paper)┃Relativerotation┃Normalized angulardisplacement
r (max 2)p┃r (max 2)p┃r (max 1)p
Lying1.45<10⁻⁴┃1.47<10⁻⁴┃0.60<10⁻⁴
Sycophancy1.160.632┃1.60<10⁻⁴┃0.49<10⁻⁴
Spatial reasoning1.40<10⁻⁴┃1.42<10⁻⁴┃0.57<10⁻⁴
Hallucination0.883.0×10⁻⁴┃0.181.00┃0.130.99
AuthorsOpenReview ↗

Response 2/3

W1, Q2: “Rotation may be built into the measurement.”

  • We would like to clarify that the behavioral observation (i.e., the model gives the opposite answer under a lying prompt) does not by itself determine how the model represents or processes the ground truth internally. This is why we track the representation of the ground truth across layers, rather than simply the model's final answer. Although the model is expected to produce the opposite of the ground truth when prompted to lie, this behavioral constraint does not prescribe how the ground truth is represented along the way. For example, the ground truth direction under lying could point opposite to the honest direction from start to finish (i.e., no rotation), or the ground truth might not be encoded at all. The rotation we observe shows that the ground truth direction under the lying prompt changes across layers. This suggests that the model retains and transforms information about the ground truth, rather than simply encoding the opposite answer.

  • “Does the distance between the true and false averages stay constant?”: We find that the true and false centroids remain distinct, with their Euclidean distance increasing across layers. The table below reports the Euclidean distances between the true and false centroids for each condition, sampled every five layers. However, we are not convinced that a genuine rotation requires constant centroid distance. Could the reviewer elaborate on the assumptions underlying the claim that rotation should preserve the centroid distance? | Layer | 0 | 5 | 10 | 15 | 20 | 25 | 30 | 31 | |---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | Honest Condition | 0.01 | 0.11 | 0.43 | 3.39 | 7.14 | 11.92 | 19.97 | 23.80 | | Lying Condition | 0.01 | 0.11 | 0.34 | 2.51 | 4.73 | 7.19 | 12.34 | 16.53 |

  • “Can the truth still be read out in late layers?”: Yes, the truth can be read out by a probe trained on lying examples. It has an accuracy of at least 0.96 from layer 12 onward.

  • Failure cases: We initially defined failure cases as system prompts with fewer than 100 successful lying examples. Only 6 of the 156 system prompts met this criterion. However, we agree that the small sample size could make the results noisy. To obtain more reliable estimates, we instead analyze system prompts for which the model produces at least 100 failure examples (50 per true/false class). This gives 79 system prompts. We recompute the rotation metrics on these failure cases and compare them with the successful cases in the table below. Overall, although some rotation is observed when the model fails to lie, its magnitude is small and not statistically significant.

Early to late drop(paper)┃Relativerotation┃Normalized angulardisplacement
r (max 2)p┃r (max 2)p┃r (max 1)p
Success1.45 ± 0.26<10⁻⁴┃1.45 ± 0.28<10⁻⁴┃0.60 ± 0.10<10⁻⁴
Failure0.66 ± 0.190.002┃0.60 ± 0.591.00┃0.13 ± 0.111.00
  • “Do directions defined by the model's answer also rotate?”: If we define the direction by the model's output, the direction under lying is simply the opposite of our original setup, since the answer is the opposite of the truth. Could you clarify what you mean here?

W2, Q1: Campbell et al. paper

  • Thank you for pointing us to this work. We will cite it in the revision. Cambpell et al. identify a small set of attention heads important for explicit lying. We extend it in three ways:
    • Generalization: We extend the investigation beyond explicit lying to implicit lying, sycophancy, and perspective taking across three model families.
    • Representational dynamics: Beyond externally trained linear probes, we characterize lying through raw internal activations across layers, revealing the geometric phenomenon of rotation.
    • Single-trial analysis: A key contribution of our work is to characterize activation trajectories at the single-trial level, going beyond the aggregated-trial analyses that are more commonly used in prior work. Aggregating across trials can obscure meaningful trial-specific information. More importantly, the model processes individual inputs, not averages across trials, making single-trial analysis the more direct way to characterize how conceptual representations evolve across layers on each forward pass. We further connect the two levels of analysis, showing that divergence points in single-trial trajectories correspond to the onset of rotation in aggregated representations. Finally, these trajectories are not merely descriptive: a trajectory-level classifier achieves substantially better out-of-distribution (OOD) prediction than single-layer probing baselines.
AuthorsOpenReview ↗

Response 3/3

Q3: How much rotation appears between two different honest prompts?

  • We computed the rotation between our honest prompt and 10 paraphrases of that system prompt, with the same metrics. The honest prompts stay aligned at every layer (cosine above 0.8). We report the mean of the three rotation metrics r below.
ComparisonEarly-to-late drop (paper)Relative rotationAngular displacement
Honest vs. lying1.461.470.60
Honest vs. honest0.180.110.11

W4, Q4: “Causal claim is overstated”

  • We are careful to avoid overclaiming, so we explicitly define what we mean by a “causal link”: there are shared model components that causally drive both lying behavior and the rotation phenomenon. We explicitly clarify in the paper that our results establish this causal link, and it does not imply a direct causal effect of rotation on behavior.

  • Establishing such a direct causal effect is challenging with current methods, as rotation is defined over class centroids across many trials, which has no distinct location in a single forward pass that can be intervened upon. Therefore, we test whether the components that causally drive the behavior also produce the rotation. They do, while the same number of random components do not (Fig. 4).

  • We use logit difference to measure the patching effect, as it is commonly used in the interpretability community [1, 2]. It is more fine-grained than flip rate because it identifies components that have an effect even though it might not be strong enough to flip the output.

  • For completeness, we also computed flip rates and report them here; we will also add them to the appendix in the revision. The table shows the fraction of answers that flip after patching the top-k heads, compared with k random heads:

Number of heads patched135102030
Top-ranked heads2%11%19%24%32%29%
Random heads0%0%0%0%0%0%

W5, Q5: “Trajectory classifier lacks a baseline”

  • As a baseline, we evaluate single-layer probes by training a separate classifier at each layer and selecting the best-performing one based on 5-fold cross-validation accuracy. For Llama-3.1-8B-Instruct (32 layers), the selected layers are layer 31 for explicit lying and layer 30 for implicit lying. While these probes achieve high within-setting test accuracy, their cross-setting transfer accuracy (between explicit and implicit lying) is near chance (approximately 33%). In contrast, our trajectory-level classifier achieves substantially better cross-setting transfer performance, demonstrating that trajectory dynamics capture shared representations that static single-layer features miss.
Train → TestTrajectory classifier (Table 1 of paper)Single-layer probe (new)
In-distribution
Explicit → Explicit0.900.91
Implicit → Implicit0.820.90
Transfer
Explicit → Implicit0.680.37
Implicit → Explicit0.760.39

[1] Zhang and Nanda, Towards Best Practices of Activation Patching in Language Models: Metrics and Methods, https://arxiv.org/abs/2309.16042.

[2] Mueller et al., MIB: A Mechanistic Interpretability Benchmark, https://arxiv.org/abs/2504.13151.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their feedback. We first summarize the main points addressed in our discussions, followed by a more detailed discussion of the rotation metrics, an issue raised by all reviewers.

Summary of discussion:

  • We made several additions to strengthen the empirical support for our claims and address concerns raised across the reviews:

    • We introduced two complementary rotation metrics to provide a more robust characterization of the phenomenon.
    • We added or expanded two negative controls, showing that distinct honest prompts produce almost no rotation and that rotation is significantly weaker when the model fails to lie.
    • We supplemented the logit-level analysis of our causal interventions with output-level metrics, reporting answer-flip rates in the activation-patching experiments.
    • We strengthened the evaluation of our single-trial analysis by adding a single-layer probe baseline and showing that our trajectory-level classifier achieves substantially higher out-of-distribution (OOD) transfer accuracy.
  • We appreciate the reviewers' positive feedback on the perspective of tracking representations across layers worthwhile, and that our experiments are thorough across prompts, tasks, and model families. Reviewers also found the extensions to perspective taking and implicit lying a good test of generality, and the paper clear and easy to follow.

Rotation metric:

All reviewers raised concerns about the intuitiveness and robustness of the original rotation metrics. In response, we introduced two complementary metrics that provide a more robust characterization of the phenomenon. The results from both metrics support our main findings: rotation is observed across multiple tasks, including lying, sycophancy, and spatial reasoning, but not hallucination (the negative control).

Below, we first explain the motivation and limitations of the original metrics. We then introduce the two new metrics and compare their results with those of the original metric.

The original rotation metric, r, summarizes the change across layers with a single number: the largest decrease in cosine similarity between an early and a late layer. This single-number summary has limitations: it captures the absolute change in cosine similarity but does not account for the initial alignment of the directions. For example, in sycophancy, the mean cosine similarity decreases from +0.60 to −0.56, representing a substantial change of 1.16. However, because the directions are only weakly aligned initially, this change is no larger than that produced by shuffled labels in the permutation test.

To check that our conclusions do not depend on the choice of metric, we computed rotation using two additional metrics, both of which account for the initial alignment:

  • Relative rotation: (c_init − c_final) / max(|c_init|, |c_final|). This measures the change in cosine similarity relative to its starting or ending value, whichever is larger in magnitude. It ranges from −2 to 2, with 2 corresponding to a full rotation from aligned to opposite directions.
  • Normalized angular displacement: This measures the change in angle between the two directions, as a fraction of the largest change possible from the initial angle. It ranges from −1 to 1, where 1 means the directions end up fully opposite.

While the original metric indicates significant rotation in hallucination but not in sycophancy, the new metrics show significant rotation in lying, sycophancy, and spatial reasoning, but not in hallucination. This pattern is more consistent with the phenomenon observed in the raw data and highlights the importance of accounting for initial alignment (Appendices B2 and C3).

TaskEarly to late drop(paper)┃Relativerotation┃Normalized angulardisplacement
r (max 2)p┃r (max 2)p┃r (max 1)p
Lying1.45<10⁻⁴┃1.47<10⁻⁴┃0.60<10⁻⁴
Sycophancy1.160.632┃1.60<10⁻⁴┃0.49<10⁻⁴
Spatial reasoning1.40<10⁻⁴┃1.42<10⁻⁴┃0.57<10⁻⁴
Hallucination0.883.0×10⁻⁴┃0.181.00┃0.130.99

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.