acceptodds
Under review as a conference paper at ICLR 2027

Harmfulness in Large Language Models Lies on Curved Manifolds

Abstract

Existing studies on how LLMs recognise harmfulness in textual inputs often rely on the linear representation hypothesis, which assumes that concepts are encoded as global linear directions in activation space. We show that this assumption does not fully capture the geometry underlying harmfulness judgements and can be unreliable for steering model behaviour. Using principal component analysis (PCA), we find that activations associated with harmfulness judgements lie along a curved, low-dimensional manifold rather than a fixed linear direction. This geometry resembles the horseshoe effect, where a single latent variable induces sinusoidal structure across principal components. Building on this connection, we characterise such manifolds in LLM activation spaces using sinusoidal parametric functions. Across diverse datasets, the latent variable recovered from the fitted manifold predicts model harmfulness decisions more accurately than projections onto linear directions. Steering along the manifold reliably shifts model judgements and outperforms linear steering, including beyond the region covered by the observed data. These effects generalise to arbitrary, unstructured instructions. Our findings highlight the value of geometry-aware modelling for understanding and evaluating how instruction-tuned models internally represent harmfulness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
mechanistic interpretability, representation geometry, activation steering, safety alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CHm7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that LLMs don't represent harmfulness linearly in activation space; instead, the representation lies along a curved and roughly one-dimensional manifold. The authors then show that modeling this curve predicts and controls model behavior (Llama-3.1-8B, Qwen3.5-9B and Gemma-3-4B) better than linear methods do.

Critical strengths

To my understanding, curved geometry hasn't yet been applied much to safety representations. The authors implemented three complementary tests, one prediction, one intervention, and one test that assumes no particular shape. The evidence, together, examined the hypothesis reliably.

Critical weaknesses
  1. I am slightly confused by your linear baselines. You compared latent t (a non-linear function of two principle components) with 1-dim projection onto the first principle component and the difference-of-means direction, both of which are not trained to predict the output at all. To have a fair comparison, the standard approach should be training a supervised linear probe on the full activation.

  2. You describe that binary classification templates squeeze model reaction to be 1-dim. In the same spirit, a simpler hypothesis is: the linear-to-curve change is about binary vs. multiple choices. That is to say, you should do the same tests on a non-safety binary task for control. If it also gives you a horseshoe, then your claim about harmfulness is actually about answer format. A related construct-validity issue is that the model may learn three totally different things from the prompts: (1) the request's content is dangerous, (2) the requester has malicious intent, or (3) the overall "this is a harmful request" judgement. I think you should be explicit about this limitation of the design, or adding new analysis to pinpoint which is truly represented by the curve.

  3. For 5.3, the optimal direction changing with vector length is necessary but not sufficient evidence of a curve. Saturation or normalization nonlinearities downstream can also produce it. Optimizer learning to cancel parts of the original activation at large lengths can also produce this.

  4. My last question concerns about the novelty of this study. It reads to me as an incremental step over Wurgaft et al. and Raval et al. Section 5.3 also explicitly follows Wurgaft. The paper needs a sharper statement of what's new.

Questions for authors
  1. How does a supervised linear probe trained to predict the logit difference compare to the fitted curve's latent t, at the same layers as Fig. 2?

  2. Could you run the same multiple-choice setup on a non-safety binary task (e.g., sentiment) and report whether a similar curve appears there?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CHm7 - Part 1

Thank you very much for your detailed comments and suggestions. We greatly appreciate your recognition of the reliability of our evidence. Please find our responses to your questions below.


Q1: Linear baselines for predicting output logits

A1: Thank you for raising this point. We would like to clarify the purpose of the experiment in Section 5.1. The goal of predicting logit differences using linear projections compared to the latent t is to show the relationship between the geometrical structure of the activations and the model output. As you have pointed out, neither the curved or linear modelling are trained to predict the logit difference directly, and they simply describe the activation geometry itself. We hypothesise that the observed curve, formed naturally through how the LLM arranges inputs in the activation space, and modelled by the proposed parametric function, corresponds to the LLM’s harmfulness decision.

To clarify our findings, we have revised Section 5.1 to instead use the spearman correlation between positions along each geometric modelling and the output logits. We further add two supervised linear baselines to contextualise these results, one constrained in the PC1-PC2 subspace, and one in the full activation space as suggested.

With the comparisons to the PC1 and difference-of-means (DoM) directions unchanged, positions along the proposed curve also correlate more with model outputs compared to a supervised linear direction in the same subspace (i.e. PC1-PC2). This shows that the curve’s correspondence with output logits does not simply stem from a linear direction in the same subspace.

We note that a supervised linear probe can still be trained in the full activation space to more accurately predict output logits. However, we show that this trained direction lies outside the subspace in which the activations predominantly vary, accounting for only ~0-5% of variance in the training set activations, and nearly orthogonal to the top principal components and difference-of-means direction, as seen in Table R1 below. This suggests that while the output can be linearly predicted from a direction in the full activation space, this direction does not sufficiently describe how the activations are distributed. In contrast, our proposed curved model links how LLMs arrange activations to their decisions.

Table R1. Absolute cosine similarity between the probe direction and PC1, PC2 and the difference-of-means direction (DoM), and the fraction of the variance of the training activations along the probe direction, by layer.

ModelLayercos(probe, PC1)cos(probe, PC2)cos(probe, DoM)Variance explained
Llama-3.1-8B130.150.010.161.1%
Llama-3.1-8B190.100.000.100.7%
Llama-3.1-8B250.130.040.131.2%
Llama-3.1-8B310.160.080.162.1%
Qwen3.5-9B130.010.090.140.2%
Qwen3.5-9B190.170.020.181.2%
Qwen3.5-9B250.220.020.223.2%
Qwen3.5-9B310.260.030.274.7%
Gemma-3-4B130.000.010.120.0%
Gemma-3-4B190.040.020.040.1%
Gemma-3-4B250.030.010.040.1%
Gemma-3-4B310.050.010.050.2%

We further add this probe direction in the full space as a baseline in the steering experiment in Section 5.2. Steering along the probe direction, at the same norms as the other methods, is less effective at changing the model's decision than even the difference-of-means methods (Table R2). This suggests that despite strongly correlating with the model’s harmfulness decisions, the direction found with the linear probe is not the geometry underlying this decision.

Table R2. Percentage of harmful multiple-choice decisions flipped to the harmless option. Best results in bold.

ModelMethod0.511.522.533.544.55
Llama-3.1-8BLinear (DoM)1.53.34.35.77.710.919.630.235.333.0
Linear (probe)1.62.33.44.96.48.710.714.418.825.6
Curve2.14.96.88.311.315.424.737.142.643.7
Qwen3.5-9BLinear (DoM)1.02.54.76.77.78.48.87.36.66.3
Linear (probe)1.01.42.53.34.35.96.26.66.96.9
Curve1.33.67.412.817.928.234.435.636.439.0
Gemma-3-4BLinear (DoM)8.928.963.899.1100.099.899.191.177.067.6
Linear (probe)1.84.27.28.88.26.27.611.421.134.0
Curve10.741.177.598.996.594.593.492.391.689.8
AuthorsOpenReview ↗

Response to Reviewer CHm7 - Part 2

Q2: Non-safety binary task control

A2: Following the reviewer's suggestion, we ran the same analysis on a non-safety binary task (sentiment classification; Appendix G), which also yields a curved geometry. This indicates that curvature is a general property of how models encode graded binary judgements, rather than specific to harmfulness. However, we would like to clarify that our claim is not that harmfulness, in particular, forms curved representations, but that this curve, revealed through the multiple-choice template, is the underlying geometry of arbitrary harmfulness judgments.

Importantly, we extract and analyse the instances of curves specifically for harmfulness in Section 5.2. Two results show that these curves reflect the harmfulness judgement rather than the answer format:

  • Independence of the answer token. At middle layers, all four template configurations share a single curve, with harmful activations in the same region regardless of which letter or position denotes "harmful" (Table 5, Appendix C). Steering at these layers changes harmfulness decisions regardless of answer token (Table 3, Section 5.2).
  • Transfers beyond the template. Steering along the curve fitted with the template changes how the model characterizes the unmodified prompts in open-ended generation without any template (Table 2, Section 5.2).

We agree that our design does not separate the three aspects of harmfulness. As a first study of the curved geometry of harmfulness, we focus on the overall harmfulness judgement. We have added this to the limitations.


Q3: The optimal direction changing with vector length is necessary but not sufficient evidence of a curve

A3: We agree that the optimal direction changing with vector length by itself does not sufficiently support the existence of an underlying curved geometry. As you noted, we consider this one of many necessary conditions for the underlying geometry being curved. We present this evidence alongside the experiments in steering along the curve and predicting model output using the curve, and argue that together they support our hypothesis.

To briefly summarise our evidence for a curved harmfulness manifold, we have shown that positions along the proposed curve closely trace LLMs’ judgements on whether an unseen and natural prompt is harmful. Steering along and beyond the curve, with different assumptions on how much the training set covers the true underlying geometry, tend to outperform linear steering with the same steering strength. Finding the optimal steering direction at varying steering strengths also reveals a path following the proposed curve and away from the difference-of-means vector particularly at larger strengths. These results combine to support our hypothesis that LLMs represent the concept of harmfulness along a curved manifold, rather than in linear directions.


Q4: Novelty of this work compared to Wurgaft et al. and Raval et al.

A4: We would like to clarify that this work does not directly build on Raval et al. or Wurgaft et al.’s modelling of curved concept geometries. The similarities are mainly in the types of experiments used to demonstrate the existence of a curved representation geometry, which we consider as standard practices in the field. Due to the different concepts we study, the methods used to capture and model the curved geometries are completely different. Wurgaft et al. fit a cubic spline through centroids of varying concept values of cyclic or ordered concepts (e.g. weekdays or numbers). Raval et al. treat behavioural attributes as distinct clusters in the activation space, joined by paths that are straight in kernel space but curved in activation space. Our work focuses on the binary decision on whether a prompt is harmful, which neither of these prior methods can be directly applied to. We instead treat the geometry of harmfulness judgements as a continuous curve, on which representations with varying degrees of harmfulness lie.

In addition, our work provides novelty in two further aspects. First, in modelling: our sinusoidal parametric form is grounded in the theory of the horseshoe effect (Diaconis et al., 2008) and can be extended beyond the observed data (Eq. 5), where steering along it remains effective while linear steering loses its effectiveness (Table 3). This provides a new way to steer model behaviour beyond the range of the observed data, which neither linear directions nor spline-based methods support. Second, in safety-specific findings: harmfulness is encoded at middle layers independently of the answer token (Table 5), and steering along the curve changes how the model characterises a request without changing its refusal (Figure 5). We have refined these distinctions in Section 2.


We hope this response addresses the reviewer’s concerns, and we would be glad to provide any additional details. Thank you again for your time and thoughtful assessment.

I thank the authors for their detailed responses. Many of my concerns have being properly addressed. I would love to increase my score accordingly.

We are glad that our response has addressed your concerns, and we sincerely appreciate your updated assessment. Thank you again for your thoughtful review, which has helped us improve the paper.

Reviewer_gayrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLMs represent harmfulness on a curved, low-dimensional manifold rather than along a linear direction. The authors construct minimal pairs by replacing the harmful span of jailbreak-benchmark prompts with a benign phrase. In this setting, final-token activations form a horseshoe-shaped arc in the top two principal components. The authors model it as a sinusoidal parametric curve. They report three results on Llama-3.1-8B, Qwen3.5-9B, and Gemma-3-4B.

Critical strengths
  • The observation that steering along the curve changes how the model characterizes a request while leaving its refusal decision intact is interesting. It adds evidence for the separation of harmfulness and refusal reported by Zhao et al. (2025).
  • The experimental design that separates the harmfulness judgement from the answer token is clean. Permuting letter assignment and option order into four configurations, and intervening at the layer where the four configurations’ geometries align, makes it more convincing that the intervention targets the judgement rather than a preference for a specific token.
Critical weaknesses

W1. The arc may reflect a linear decision variable plus an uncertainty signal, rather than curved harmfulness.

In Fig. 1b and Figs. 8–10, most activations sit in two dense clusters at the ends of the arc, while the arc itself is sparsely populated. This pattern is also consistent with a simpler structure: PC1 encodes a linear harmful–harmless decision variable, and PC2 encodes the model’s uncertainty (e.g., a signal that peaks when the logit difference is near zero). PC2 would then be roughly a quadratic function of PC1, which produces exactly a horseshoe without harmfulness itself being curved. This alternative also explains why the arc appears only under the classification template and not for the original prompts (Fig. 1a), since uncertainty about the answer arises only when the model must make a decision. I ask the authors to test this directly, for example by reporting the correlation between PC2 and the absolute logit difference or the answer entropy, and by checking whether a linear harmfulness direction plus an uncertainty term explains the logit difference.

W2. Constructing harmless counterparts by replacing only the minimal harmful span may introduce bias.

The harmless prompts are produced by swapping the shortest harmful span for a benign phrase and leaving the rest of the request intact. This ignores harmful intent carried by the surrounding context. Phrases such as “and get away with it,” “without being detected,” or the framing of the request can remain harmful-sounding after the swap. The examples in Appendix A illustrate this: “Design a gift that can be used to impress a high-profile individual” is grammatical but unnatural, and requests rewritten this way may not resemble the harmless prompts a model normally encounters. This raises three concerns.

W3. Missing baselines and limited model scale.

The steering comparison is not fair. The curve intervention is source-adaptive: it projects each activation onto the curve and moves it toward a target point on the harmless side. The linear intervention adds the same delta to every activation regardless of where it starts. The natural baseline is an adaptive linear steer that moves each activation along delta only until it reaches a target point (e.g., the harmless centroid’s projection), which would test whether the advantage comes from curvature or from adaptivity. Similarly, the curve intervention is confined to the top-two PC plane, so steering along PC1 would be an informative in-plane linear baseline. Finally, all three models are 9B or smaller. Whether the curved geometry, and the advantage of steering along it, persist at larger scale is untested.

Questions for authors

Questions are in the weakness section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gayr - Part 1

We thank the reviewer for providing valuable feedback and suggestions. We sincerely appreciate your expressed interest in our findings, and your recognition of our experimental design. Please find our responses to your questions below.


Q1: The arc may reflect a linear decision variable plus an uncertainty signal, rather than curved harmfulness.

A1: We thank the reviewer for this insightful alternative explanation. The spearman correlation between PC2 and |logit(A) − logit(B)| can be seen below:

Table R1: Spearman |ρ| between PC2 and |logit(A) − logit(B)| by layer.

ModelL13L16L19L22L25L28L31
Llama-3.1-8B0.460.870.880.850.850.830.81
Qwen3.5-9B0.530.560.800.920.940.940.96
Gemma-3-4B0.190.080.060.120.130.160.31

For Llama and Qwen, PC2 correlates strongly with the absolute logit difference in middle and late layers. For Gemma-3-4B, however, the correlation stays low across layers (|ρ| ≤ 0.31), so the arc cannot be explained by decision confidence alone. To check whether PC2 reflects general output uncertainty, we correlate it with the A/B logit margin over all other tokens, m = (logit(A) + logit(B))/2 − logsumexp(other). If PC2 tracked general uncertainty, it should correlate with m in addition to correlating with |logit(A) − logit(B)|. However, the correlations are weak (Table R2), so PC2 does not reflect general uncertainty.

Table R2: Spearman |ρ| between PC2 and (logit(A) + logit(B)) / 2 - logsumexp(other) by layer.

ModelL13L16L19L22L25L28L31
Llama-3.1-8B0.170.370.340.340.380.310.29
Qwen3.5-9B0.070.000.230.290.300.280.31
Gemma-3-4B0.340.380.060.140.160.070.09

Our revised experiments in Section 5.2 also address this concern by further adding PC1 as a linear baseline for steering. Steering along the proposed curve tends to change more of the model’s decision compared to steering along the PC1 direction at the same steering strength, which shows that the model’s harmfulness decision is not encoded along PC1 alone. The updated results can be found in the Table 3 of the updated manuscript, as well as in Table R3 in A3 below.


Q2: Constructing harmless counterparts by replacing only the minimal harmful span may introduce bias.

A2: Replacing only the minimal harmful span is a deliberate design: each pair differs in harmfulness while keeping topic, length and phrasing fixed, which controls for these confounds. To verify that the fitted geometry does not depend on this construction, we evaluate it only on unmodified data:

  • Natural prompts. Activation positions along curves fitted on the constructed prompts correlate strongly with the model's output logits on unseen natural harmless instructions from Alpaca [1] (revised Section 5.1).
  • Causal tests. Steering experiments (Sections 5.2–5.3) are applied to unmodified harmful prompts and do not use the constructed data.

The geometry therefore generalises beyond the constructed prompts.

[1] Taori, Rohan, et al. “Stanford Alpaca: An Instruction-following LLaMA Model.” GitHub repository, 2023.

AuthorsOpenReview ↗

Response to Reviewer gayr - Part 2

Q3: Missing baselines and limited model scale.

A3: We would like to clarify that the proposed curve steering is not source adaptive in strength and steers the activation by the same amount compared to its linear counterpart. The steering methods only differ in the direction of the steering, for each fixed strength ∥δ∥\lVert \boldsymbol{\delta} \rVert. Linear steering, by definition, requires that each activation is steered towards the same direction. Curve steering instead steers the activation along the curve, until it reaches a point where its distance from the starting point reaches the target strength ∥δ∥\lVert \boldsymbol{\delta} \rVert. The fixed direction of the linear modeling regardless of steering strength or relative position is exactly the limitation we aim to reveal and address through this study.

For steering, we add another linear baseline steering in only the PC1 direction as suggested. The results are shown in Table R3 below. Steering along the PC1 direction performs similarly to the difference-of-mean (DoM) direction, and does not change any of our conclusions.

Table R3. Percentage of harmful multiple-choice decisions flipped to the harmless option, pooled over the three configurations the geometries were not fitted in. Linear (DoM) steers along the difference-of-means direction and Linear (PC1) along the first principal component, at the same norms as the curve. Best results in bold.

ModelMethod0.511.522.533.544.55
Llama-3.1-8BLinear (DoM)1.53.34.35.77.710.919.630.235.333.0
Linear (PC1)1.73.84.96.18.311.520.832.536.433.9
Curve2.14.96.88.311.315.424.737.142.643.7
Qwen3.5-9BLinear (DoM)1.02.54.76.77.78.48.87.36.66.3
Linear (PC1)1.23.25.98.311.012.712.911.910.810.4
Curve1.33.67.412.817.928.234.435.636.439.0
Gemma-3-4BLinear (DoM)8.928.963.899.1100.099.899.191.177.067.6
Linear (PC1)8.025.158.295.1100.0100.099.798.689.574.0
Curve10.741.177.598.996.594.593.492.391.689.8

To address the concern about generalisation to larger model scales, we further include experiments on Gemma-3-27B. A similar curve can be observed in the top principal components of the activations towards the middle to late layers, and modelled with our proposed parametric function. Table R4 shows that positions along the curve continue to closely track with the model’s harmfulness decision, and Table R5 shows that steering along the curve remains more effective compared to linear steering. The full results can be seen in Appendix I of the revised manuscript. These results are consistent with our main findings and suggest that the proposed curved model is not specific to the model sizes in the main experiments.

Table R4. Curve fit and correlation with the model's decision for Gemma-3-27B. Curve R²: fit of the parametric function to the training activations. Remaining columns: Spearman |ρ| between the logit difference and the coordinate on each direction or along the curve (t), on held-out test prompts. Best results in bold, second best in italics.

LayerCurve R²tPC1DoMProbe (PC1–PC2)Probe (full space)
350.9530.9620.6960.7040.7560.971
400.9670.9620.7250.7270.7110.986
450.9870.9540.7500.7450.7780.990
500.9860.9560.7590.7520.7770.990
550.9920.9620.7530.7420.7940.992
610.9820.9720.7370.7360.8110.995

Table R5. Percentage of harmful multiple-choice decisions flipped to the harmless option for Gemma-3-27B, steering at layer 35, pooled over the three configurations the geometries were not fitted in. Columns give the steering strength ‖δ‖ as multiples of the centroid gap ‖Δμ‖. Best results in bold.

Method0.511.522.533.544.55
Linear (DoM)2.46.128.037.839.734.216.46.24.14.0
Curve (f=1/5)5.643.982.288.690.590.990.191.491.385.3

We hope the provided clarifications sufficiently address the reviewer’s concerns, and we remain available to elaborate further if needed. We are grateful for the reviewer’s constructive feedback on our work.

Thank you again for the thought you have put into reviewing our paper and for your constructive feedback. This has allowed us to improve our work, including testing alternative explanations for the observed curvature, clarifying the role and influence of our prompt construction method, and expanding our experiments to include an additional baseline and a larger model scale.

With the discussion period nearing its end, we would be grateful for any feedback you may have on our responses. We remain available to clarify any points that are still unclear. If our responses have sufficiently addressed your concerns, we would greatly appreciate it if you would consider updating your assessment of our submission.

Reviewer_nFxUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that harmfulness is represented on a curved low-dimensional manifold rather than along a linear direction in the internal representations of LLMs. Specifically, the authors analyze activations for harmful and harmless instructions using PCA and model the observed curved structure with a sinusoidal parametric function. They also demonstrate that latent positions on the manifold can predict models' harmfulness judgments more accurately than conventional linear projections. Furthermore, the authors conduct activation steering experiments using three instruction-tuned LLMs and report that steering along the proposed curved manifold is more effective in modifying models' harmfulness judgments than linear steering. In addition, they provide further experimental evidence supporting the curved geometry of harmfulness representations by showing that the optimal steering vectors point in different directions depending on the steering strength.

Critical strengths
  • This paper investigates harmfulness representations from a different perspective, namely a curved manifold. Also, it is interesting observation that a curved structure emerges even for a binary concept such as harmfulness.
  • It examines their hypothesis from multiple perspectives, including modeling with a sinusoidal parametric function, predicting harmfulness judgments using latent positions, and performing causal interventions through activation steering. These experiments provide multifaceted evidence supporting the effectiveness of the proposed curved manifold.
  • The authors conduct experiments using three different model families and consider the effects of answer tokens and option ordering in multiple-choice classification by evaluating four different configurations. This partially rules out the possibility that the observed geometry depends on specific answer tokens.
Critical weaknesses
  1. It is unclear whether the curved manifold is specific to harmfulness.
    • This paper argues that harmfulness representations form a curved manifold. However, Figure 1 shows different activation distributions between original prompts and multiple-choice classification prompts, making it unclear whether the observed curved structure is specific to harmfulness or arises from the binary classification task setting. For example, similar structures may also be observed in other binary classification tasks, such as sentiment classification.
    • Therefore, I would like you to experimentally demonstrate that the observed curved structure is specific to harmfulness. Alternatively, if the findings can be generalized to other tasks, the authors should revise their conclusions accordingly.
  2. Insufficient evidence supporting the curved manifold hypothesis.
    • This paper demonstrates that the proposed latent position can predict harmfulness judgments more accurately than linear projections. However, the comparison is limited to simple univariate linear regression, and comparisons with alternative methods, such as nonlinear regression, are missing. Therefore, it remains unclear whether the performance improvement supports the geometric hypothesis of a curved manifold or simply results from nonlinear modeling.
    • Similarly, the authors extrapolate the manifold beyond the range of observed activations and perform steering along the extrapolated curve. However, there is no evidence that the extrapolated curve reflects the actual activation geometry, and improved steering performance alone does not establish its validity. Furthermore, the observation that optimal steering directions vary with steering strength could also be explained by LLMs' nonlinear computations.
    • I would like you to provide additional experimental evidence supporting the curved manifold hypothesis or include further discussion of these alternative explanations.
  3. Please provide justification and validation of synthetic data construction.
    • This paper constructs harmless counterparts by rewriting harmful prompts using an LLM. However, although this paper mentions the need to minimize distributional shifts, it does not sufficiently explain why directly using existing harmful and harmless datasets would be problematic.
    • Furthermore, the necessity of constructing such task-specific synthetic data remains unclear, raising questions about whether similar findings would hold for real-world prompts. In addition, the validity and quality control of the LLM-generated harmless counterparts require more detailed verification. In its current form, the experimental setting appears overly controlled, and its necessity and practical effectiveness are not sufficiently justified.
    • Therefore, I suggest including comparisons with existing datasets without synthetic data construction, or a quality assessment of the generated data. Alternatively, further discussion is needed to justify these experimental design choices.
Questions for authors
  1. Please address the three major weaknesses raised above (W1-W3). I have included specific suggestions for improvement in each bottom item.
  2. It is better to explicitly discuss the limitations of this study. Clarifying the scope and limitations of the findings would also provide useful implications and directions for future research.
  3. The research questions should be reconsidered. In particular, RQ3 in L60 appears to depend on the assumptions underlying RQ1 and RQ2, rather than representing an independent research question. This makes the current formulation problematic from a hypothesis-testing perspective, as the RQs appear to have been formulated retrospectively to organize the experimental findings. If possible, I recommend reformulating the RQs so that each can be investigated independently. I also suggest explicitly clarifying which experiments address each RQ. Alternatively, if the RQs cannot be formulated independently, it may be preferable to present the study without explicitly enumerating them.
5 comments
AuthorsOpenReview ↗

Response to Reviewer nFxU - Part 1

We sincerely thank you for taking the time to review our paper and for providing thoughtful and valuable comments. We greatly appreciate your recognition of the novelty of our work and the value of our supporting evidence. Please find our responses to your questions and comments below.


Q1: It is unclear whether the curved manifold is specific to harmfulness.

A1: Thank you for this insightful comment. In short, the observed phenomenon of activations arranging along a curve is not specific to harmfulness, but the specific instances of this curve that we extract are specific to harmfulness. We tested the sentiment classification setting as suggested using the CAD dataset [1], modified with the multiple-choice prompt template. The activations similarly lie along curves towards the later layers and can similarly be modelled by our proposed parametric function. The full results can be seen in the updated Appendix G. This is in line with our hypothesis that the curvature in the activations is the result of the activations varying along a single latent variable between the binary concept values. This example shows that our method in theory could be applied to other binary concepts to reveal their underlying activation geometry. However, we focus specifically on LLMs’ judgement of harmfulness because this is often crucial in real-world applications (e.g. content moderation, LLM-as-judge), particularly in the AI safety field. Prior work in the mechanistic interpretability of harmfulness or other similar binary decisions has not considered our proposed curved model. We leave extending our hypothesis to other concepts as future work and have included a limitation section in the updated manuscript to discuss this.

Importantly, we extract and analyse the instances of curves specifically for harmfulness in Section 5.2. Two results show that these curves reflect the harmfulness judgement rather than the binary classification:

  • Independence of the answer token. At middle layers, all four template configurations share a single curve, with harmful activations in the same region regardless of which letter or position denotes "harmful" (Table 5, Appendix C). Steering at these layers changes harmfulness decisions regardless of answer token (Table 3, Section 5.2).
  • Transfers beyond the template. Steering along the curve fitted with the template changes how the model characterizes the unmodified prompts in open-ended generation without any template (Table 2, Section 5.2).

Our claim is not that harmfulness, in particular, arranges activations along a curve, but that the curve revealed through the multiple-choice template and extracted with our proposed method, is the underlying geometry of harmfulness judgements.

[1] Kaushik, Divyansh, Eduard Hovy, and Zachary C. Lipton. "Learning the difference that makes a difference with counterfactually-augmented data." ICLR, 2020

AuthorsOpenReview ↗

Response to Reviewer nFxU - Part 2

Q2: Insufficient evidence supporting the curved manifold hypothesis.

A2: Thank you for your detailed examination of our experiments and evidence. We would first like to clarify that the experiment to predict harmfulness judgement is to show the relationship between the geometrical structure of the activations and the model output. To better illustrate this point, and to remove any dependence on the choice of the regression model, we compute Spearman’s correlation of the positions with the output logits, and have added stronger linear baselines using supervised methods. We have highlighted the relevant changes in the manuscript in Section 5.1. From the layer at which the curve forms, position along the curve orders the prompts more closely to the model's decision than position along PC1, the difference-of-means direction, or a supervised linear direction in the same PC1–PC2 subspace, in all three models. This revised experiment continues to support our hypothesis that each activation’s position in our proposed curve closely reflects the degree of harmfulness the model assigns to the corresponding prompt, linking how LLMs arrange activations to their decisions.

Regarding your concern about the extrapolated curve, we further support our claim through extrapolating the curve with varying values of the coverage parameter ff, as shown below:

Table R1. Percentage of harmful multiple-choice decisions flipped to the harmless option, by steering strength ‖δ‖ (× centroid gap) and coverage f of the extended curve. Best in bold, second best in italics; -- marks strengths not reachable on the curve.

ModelMethod0.511.522.533.544.55
Llama-3.1-8BLinear (DoM)1.53.34.35.77.710.919.630.235.333.0
Curve (f=1/10)2.34.76.58.411.515.224.536.442.543.3
Curve (f=1/5)2.14.96.88.311.315.424.737.142.643.7
Curve (f=1/3)2.35.06.88.611.515.023.935.641.9--
Qwen3.5-9BLinear (DoM)1.02.54.76.77.78.48.87.36.66.3
Curve (f=1/10)1.23.47.312.818.128.134.735.336.138.2
Curve (f=1/5)1.33.67.412.817.928.234.435.636.439.0
Curve (f=1/3)1.13.57.312.518.228.336.838.139.641.5
Gemma-3-4BLinear (DoM)8.928.963.899.1100.099.899.191.177.067.6
Curve (f=1/10)10.841.476.998.996.694.793.693.092.090.0
Curve (f=1/5)10.741.177.598.996.594.593.492.391.689.8
Curve (f=1/3)10.841.076.398.194.891.088.683.6----

This shows that the improved performance does not depend on a particular extrapolation, and that the strongest intervention requires steering beyond the observed curve, further supporting the idea that the underlying geometry of harmfulness decisions continues beyond the observed range. We have updated Table 3 and Section 5.2 accordingly in the revised manuscript.

Regarding the steering experiments, we agree that more effective steering beyond the curve, or optimal steering directions varying with strength alone does not necessarily point to an underlying curved geometry. However, when examined together, the overlap of these pieces of evidence is consistent with our claim.

To briefly summarise our evidence for a curved harmfulness manifold, we have shown that positions along the proposed curve closely trace LLMs’ judgements on whether an unseen and natural prompt is harmful. Steering along and beyond the curve, with different assumptions on how much the training set covers the true underlying geometry, tend to outperform linear steering with the same steering strength. Finding the optimal steering direction at varying steering strengths also reveals a path following the proposed curve and away from the difference-of-means vector particularly at larger strengths. These results combine to support our hypothesis that LLMs represent the concept of harmfulness along a curved manifold, rather than in linear directions.

AuthorsOpenReview ↗

Response to Reviewer nFxU - Part 3

Q3: Justification and validation of synthetic data construction.

A3: We would like to clarify that our experimental results are validated on non-synthetic data. The data construction of deriving harmless prompts from harmful ones only applies to the training set, which the linear baselines also use. The quality of the synthesised data therefore mainly affects our estimation of the geometry, and not our evaluation of this geometry. The steering experiments in Sections 5.2 and 5.3 steer the unmodified harmful prompts towards the harmless ends of both geometrical models, and do not involve the synthesised data. The revised Section 5.1 also uses natural benign instructions from the Alpaca dataset [2] to correlate activation positions with harmless output logits.

We further illustrate why using natural data alone to fit the proposed parametric function could be challenging in Appendix H of the revised manuscript. Natural prompt datasets mainly contain clearly harmful and clearly harmless prompts, with few examples of intermediate harmfulness. Their activations therefore concentrate near the two ends of the curve, leaving the region in between unobserved. This reflects a gap in data coverage rather than an absence of curvature. We use synthetic prompts spanning graded levels of harmfulness as a controlled probe to fill this gap and constrain the full shape of the curve.

[2] Taori, Rohan, et al. “Stanford Alpaca: An Instruction-following LLaMA Model.” GitHub repository, 2023.


Q4: Limitations of this study.

A4: Thank you for this suggestion. We have added a limitation and future works section to the revised manuscript. Our work has several limitations. In short: (1) we study a limited set of LLMs, from three model families and 4B to 27B parameters, so the curved manifolds may not be universal, particularly for other architectures; (2) we model the geometry as a one-dimensional curve in a two-dimensional plane, which approximates the structure well but may be refined by higher-dimensional modelling; (3) our harmless training counterparts are generated by an LLM and may contain imperfect substitutions, although all evaluations use only natural prompts from existing datasets; and (4) we focus on the general harmfulness judgement, leaving its subconcepts, as well as other binary concepts such as sentiment that show a similar curve, to future work.


Q5: Formulation of research questions.

A5: Thank you for your suggestion. We agree that the research questions did not map well onto our experiments, and have removed them from the revised manuscript. The experiments instead test the hypothesis stated in Section 4, as described in the introduction of Section 5.


We hope our response fully addresses the reviewer’s concerns and welcome any further enquiries. We thank the reviewer again for their time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revision

We thank all reviewers for their constructive and insightful comments. In response to the reviews, we have made the following modifications to our submission:

  1. Revision of Section 5.1 (raised by reviewers nFxU, gayr, and CHm7): We have revised Section 5.1 to remove any dependence on the choice of the regression model, and have further included two stronger linear baselines obtained from supervised probes.

  2. Additional baselines for steering (raised by reviewers nFxU, gayr, and CHm7): We have included the PC1 direction and the direction obtained from the supervised linear probe in the full activation space as baselines in Table 3.

  3. Extrapolating the curve with varying values of the coverage parameter ff (raised by reviewer nFxU ): We include results from different assumptions of the extent of underlying geometry beyond the observed data in Table 3, showing that the proposed curve steering does not depend on a specific extrapolation.

  4. Experiments on larger scale models (raised by reviewer gayr): We repeat the experiments on Gemma-3-27B to show that the curved manifold is not specific to the model sizes used in the main experiments. Results are shown in Appendix I.

  5. Non-safety binary tasks (raised by reviewers nFxU and CHm7): We have added discussion and results of the activation geometry in the sentiment classification setting in Appendix G.

  6. Natural harmless data (raised by reviewers nFxU and gayr): We have added discussion and results of the activation geometry using natural harmless prompts in Appendix H.

  7. Clarification of novelty (raised by reviewer CHm7): We refined our novelty and contribution of our work compared to previous work in curved activation geometries in Section 2.

  8. Limitations and future work (raised by reviewer nFxU): We discuss the limitations and future work of our paper in Section 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.