- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that LLMs don't represent harmfulness linearly in activation space; instead, the representation lies along a curved and roughly one-dimensional manifold. The authors then show that modeling this curve predicts and controls model behavior (Llama-3.1-8B, Qwen3.5-9B and Gemma-3-4B) better than linear methods do.
Critical strengths
To my understanding, curved geometry hasn't yet been applied much to safety representations. The authors implemented three complementary tests, one prediction, one intervention, and one test that assumes no particular shape. The evidence, together, examined the hypothesis reliably.
Critical weaknesses
-
I am slightly confused by your linear baselines. You compared latent t (a non-linear function of two principle components) with 1-dim projection onto the first principle component and the difference-of-means direction, both of which are not trained to predict the output at all. To have a fair comparison, the standard approach should be training a supervised linear probe on the full activation.
-
You describe that binary classification templates squeeze model reaction to be 1-dim. In the same spirit, a simpler hypothesis is: the linear-to-curve change is about binary vs. multiple choices. That is to say, you should do the same tests on a non-safety binary task for control. If it also gives you a horseshoe, then your claim about harmfulness is actually about answer format. A related construct-validity issue is that the model may learn three totally different things from the prompts: (1) the request's content is dangerous, (2) the requester has malicious intent, or (3) the overall "this is a harmful request" judgement. I think you should be explicit about this limitation of the design, or adding new analysis to pinpoint which is truly represented by the curve.
-
For 5.3, the optimal direction changing with vector length is necessary but not sufficient evidence of a curve. Saturation or normalization nonlinearities downstream can also produce it. Optimizer learning to cancel parts of the original activation at large lengths can also produce this.
-
My last question concerns about the novelty of this study. It reads to me as an incremental step over Wurgaft et al. and Raval et al. Section 5.3 also explicitly follows Wurgaft. The paper needs a sharper statement of what's new.
Questions for authors
-
How does a supervised linear probe trained to predict the logit difference compare to the fitted curve's latent t, at the same layers as Fig. 2?
-
Could you run the same multiple-choice setup on a non-safety binary task (e.g., sentiment) and report whether a similar curve appears there?
4 comments
Response to Reviewer CHm7 - Part 2
Q2: Non-safety binary task control
A2: Following the reviewer's suggestion, we ran the same analysis on a non-safety binary task (sentiment classification; Appendix G), which also yields a curved geometry. This indicates that curvature is a general property of how models encode graded binary judgements, rather than specific to harmfulness. However, we would like to clarify that our claim is not that harmfulness, in particular, forms curved representations, but that this curve, revealed through the multiple-choice template, is the underlying geometry of arbitrary harmfulness judgments.
Importantly, we extract and analyse the instances of curves specifically for harmfulness in Section 5.2. Two results show that these curves reflect the harmfulness judgement rather than the answer format:
- Independence of the answer token. At middle layers, all four template configurations share a single curve, with harmful activations in the same region regardless of which letter or position denotes "harmful" (Table 5, Appendix C). Steering at these layers changes harmfulness decisions regardless of answer token (Table 3, Section 5.2).
- Transfers beyond the template. Steering along the curve fitted with the template changes how the model characterizes the unmodified prompts in open-ended generation without any template (Table 2, Section 5.2).
We agree that our design does not separate the three aspects of harmfulness. As a first study of the curved geometry of harmfulness, we focus on the overall harmfulness judgement. We have added this to the limitations.
Q3: The optimal direction changing with vector length is necessary but not sufficient evidence of a curve
A3: We agree that the optimal direction changing with vector length by itself does not sufficiently support the existence of an underlying curved geometry. As you noted, we consider this one of many necessary conditions for the underlying geometry being curved. We present this evidence alongside the experiments in steering along the curve and predicting model output using the curve, and argue that together they support our hypothesis.
To briefly summarise our evidence for a curved harmfulness manifold, we have shown that positions along the proposed curve closely trace LLMs’ judgements on whether an unseen and natural prompt is harmful. Steering along and beyond the curve, with different assumptions on how much the training set covers the true underlying geometry, tend to outperform linear steering with the same steering strength. Finding the optimal steering direction at varying steering strengths also reveals a path following the proposed curve and away from the difference-of-means vector particularly at larger strengths. These results combine to support our hypothesis that LLMs represent the concept of harmfulness along a curved manifold, rather than in linear directions.
Q4: Novelty of this work compared to Wurgaft et al. and Raval et al.
A4: We would like to clarify that this work does not directly build on Raval et al. or Wurgaft et al.’s modelling of curved concept geometries. The similarities are mainly in the types of experiments used to demonstrate the existence of a curved representation geometry, which we consider as standard practices in the field. Due to the different concepts we study, the methods used to capture and model the curved geometries are completely different. Wurgaft et al. fit a cubic spline through centroids of varying concept values of cyclic or ordered concepts (e.g. weekdays or numbers). Raval et al. treat behavioural attributes as distinct clusters in the activation space, joined by paths that are straight in kernel space but curved in activation space. Our work focuses on the binary decision on whether a prompt is harmful, which neither of these prior methods can be directly applied to. We instead treat the geometry of harmfulness judgements as a continuous curve, on which representations with varying degrees of harmfulness lie.
In addition, our work provides novelty in two further aspects. First, in modelling: our sinusoidal parametric form is grounded in the theory of the horseshoe effect (Diaconis et al., 2008) and can be extended beyond the observed data (Eq. 5), where steering along it remains effective while linear steering loses its effectiveness (Table 3). This provides a new way to steer model behaviour beyond the range of the observed data, which neither linear directions nor spline-based methods support. Second, in safety-specific findings: harmfulness is encoded at middle layers independently of the answer token (Table 5), and steering along the curve changes how the model characterises a request without changing its refusal (Figure 5). We have refined these distinctions in Section 2.
We hope this response addresses the reviewer’s concerns, and we would be glad to provide any additional details. Thank you again for your time and thoughtful assessment.
I thank the authors for their detailed responses. Many of my concerns have being properly addressed. I would love to increase my score accordingly.
We are glad that our response has addressed your concerns, and we sincerely appreciate your updated assessment. Thank you again for your thoughtful review, which has helped us improve the paper.
Response to Reviewer CHm7 - Part 1
Thank you very much for your detailed comments and suggestions. We greatly appreciate your recognition of the reliability of our evidence. Please find our responses to your questions below.
Q1: Linear baselines for predicting output logits
A1: Thank you for raising this point. We would like to clarify the purpose of the experiment in Section 5.1. The goal of predicting logit differences using linear projections compared to the latent t is to show the relationship between the geometrical structure of the activations and the model output. As you have pointed out, neither the curved or linear modelling are trained to predict the logit difference directly, and they simply describe the activation geometry itself. We hypothesise that the observed curve, formed naturally through how the LLM arranges inputs in the activation space, and modelled by the proposed parametric function, corresponds to the LLM’s harmfulness decision.
To clarify our findings, we have revised Section 5.1 to instead use the spearman correlation between positions along each geometric modelling and the output logits. We further add two supervised linear baselines to contextualise these results, one constrained in the PC1-PC2 subspace, and one in the full activation space as suggested.
With the comparisons to the PC1 and difference-of-means (DoM) directions unchanged, positions along the proposed curve also correlate more with model outputs compared to a supervised linear direction in the same subspace (i.e. PC1-PC2). This shows that the curve’s correspondence with output logits does not simply stem from a linear direction in the same subspace.
We note that a supervised linear probe can still be trained in the full activation space to more accurately predict output logits. However, we show that this trained direction lies outside the subspace in which the activations predominantly vary, accounting for only ~0-5% of variance in the training set activations, and nearly orthogonal to the top principal components and difference-of-means direction, as seen in Table R1 below. This suggests that while the output can be linearly predicted from a direction in the full activation space, this direction does not sufficiently describe how the activations are distributed. In contrast, our proposed curved model links how LLMs arrange activations to their decisions.
Table R1. Absolute cosine similarity between the probe direction and PC1, PC2 and the difference-of-means direction (DoM), and the fraction of the variance of the training activations along the probe direction, by layer.
We further add this probe direction in the full space as a baseline in the steering experiment in Section 5.2. Steering along the probe direction, at the same norms as the other methods, is less effective at changing the model's decision than even the difference-of-means methods (Table R2). This suggests that despite strongly correlating with the model’s harmfulness decisions, the direction found with the linear probe is not the geometry underlying this decision.
Table R2. Percentage of harmful multiple-choice decisions flipped to the harmless option. Best results in bold.