- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studied how the internal representations in LLMs changes across layers when instructed with lie, force agree with incorrect opinions, or reason from a different perspective. I am not an expert in this field. I think the authors observe a phenomenon they call representations rotation, where the directions representing concepts such as true/false change across layers. They also show that internal activating trajectory can help distinguish hones and dishonest answers.
Critical strengths
- An interesting perspective on LLM interpretability. The paper looks at the changes in representations across layers.
- Good experiment coverage. The authors test the phenomenon across different models, tasks and prompts. This helps demonstrate the observation in many different settings.
Critical weaknesses
I am not convinced about the discussion that observed rotation reveals a new mechanism for how LLMs manipulate information. When a model is instructed to lie, it needs to change its answer from true to false. Similarly, the spatial reasoning task involves changing left to right. Could the observed rotation simply reflect these changes in the expected answers? The activation patching experiments are interesting, but I don't think this establishes that rotation itself causes the behaviour.
Questions for authors
- How can you distinguish the proposed rotation mechanism from the simpler explanation that the model is just changing its expected answer? Would a basic answer-reversal task produce similar results?
- Could you explain, see Table2, why sycophancy is not significant while hallucination is? How does this affect the main conclusions?
Thank you for your review. We provide a detailed response below.
W1, Q1: Is rotation just the model changing its answer?.
Q2: Significance of rotation metric for sycophancy and hallucination.
The rotation metric r is meant to summarize the cosine similarity across layers in a single number. We acknowledge that a single number can hide details, and the counterintuitive results between hallucination and sycophancy reflects limitations of r. For sycophancy, the directions are only moderately aligned to begin with, so even a clear rotation to opposite gives a drop no larger than what shuffled labels produce. For hallucination, as noted in Section 3.2, the representations are not well captured by a single linear direction, so their cosine similarity fluctuates across layers. Because r takes the largest drop over all early/late layer pairs, these fluctuations alone yield a small but significant value (0.88, the lowest of all tasks), even though the heatmap (Fig. 14) shows no consistent transition from aligned to opposite.
To check that our conclusions do not depend on these limitations, we recomputed rotation with two metrics that account for where the directions start:
While the original metric indicates significant rotation in hallucination but not in sycophancy, both new metrics indicate significant rotation in lying, sycophancy, and spatial reasoning, but not in hallucination. This pattern is more consistent with the phenomenon observed in the raw data and highlights the importance of accounting for initial alignment (Appendices B2 and C3).