acceptodds
Under review as a conference paper at ICLR 2027

Form Before Harm: Diagnosing the Semantic Influence of Jailbreak Formats with Activation Steering

Abstract

Large language model safety defenses often organize harmful inputs by topic without accounting for the semantic shifts induced by jailbreak formats. We show that these shifts can alter this organization: attack format has a substantially greater impact on the distribution of LLM semantic representations than harmful topic. To study this effect in activation space, we introduce Context-Aware Steering (COS), an adaptive activation-steering method that learns a routable steering space from mixed jailbreak data and achieves strong defense performance with limited benign-side effects. Across both the model's natural representations and the intervention structure learned by COS, attack format consistently plays a larger organizing role than harmful topic; COS further adapts its shared and specialized steering components to the internal structure of each LLM rather than relying on predefined topic categories. We further identify a clear out-of-distribution generalization gap across jailbreak formats. Increasing the diversity of attack formats during training alleviates this gap but does not eliminate it. Together, these findings show that jailbreak format is a major determinant of LLM semantic representations and should be considered when organizing safety detection and defense.

Primary area:
interpretability and explainable AI
Keywords:
Representation Analysis, Activation Steering, Jailbreak Attacks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xSBbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how jailbreak formats shape LLM representations, finding that attack format has a greater influence than harmful topic. It proposes Context-Aware Steering (COS), an adaptive activation-steering method that improves defense performance with limited effects on benign inputs. The paper also identifies a generalization gap across unseen attack formats, which greater training diversity mitigates but does not eliminate.

Critical strengths

None

Critical weaknesses
  1. Insufficient positioning, unclear novelty, and weak motivation. The related-work section only covers studies from 2025 or earlier and omits relevant work from 2026, particularly the closely related study. Without an explicit technical comparison, it is difficult to determine how COS differs from existing approaches and what substantive contributions it makes. Moreover, the introduction states “Motivated by these observations,” but the preceding discussion appears to describe existing technical approaches rather than empirical observations or concrete insights that motivate COS. The authors should clarify both the novelty of the proposed method and the rationale connecting the limitations of existing approaches [1] to its specific design choices.

  2. Limited experimental evidence for general applicability. The evaluation is restricted to 8B-scale, open-source Qwen and LLaMA models. This narrow experimental scope provides insufficient evidence to establish the broad effectiveness of COS across model scales and families. Evaluation on larger models and a more diverse set of model families would be needed to substantiate claims of general applicability.

[1] Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models. arXiv:2602.04896.

Questions for authors
  1. Could the authors clarify how COS differs technically from the closely related work “Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models” and what substantive contributions it makes beyond existing approaches? The related-work section omits relevant studies from 2026, making the novelty difficult to assess. Additionally, what specific observations or insights motivate COS and its design choices? The phrase “Motivated by these observations” appears to refer to existing technical approaches rather than empirical findings or concrete insights, leaving the connection between the stated motivation and the proposed design unclear.

  2. Could the authors provide evidence that COS remains effective beyond the 8B-scale, open-source Qwen and LLaMA models evaluated in the paper? In particular, does its effectiveness generalize to larger models and other model families? If broader evaluation is not feasible, could the authors clarify the scope of their claims and discuss the limitations imposed by the current experimental coverage?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xSBb

Dear Reviewer xSBb,

Thank you for your contribution to reviewing for ICLR 2027 and for your comments on related work and experimental coverage. We respond point by point below.

Weakness 1 and Question 1

Weakness 1: Insufficient positioning, unclear novelty, and weak motivation. The related-work section only covers studies from 2025 or earlier and omits relevant work from 2026, particularly the closely related study. Without an explicit technical comparison, it is difficult to determine how COS differs from existing approaches and what substantive contributions it makes. Moreover, the introduction states “Motivated by these observations,” but the preceding discussion appears to describe existing technical approaches rather than empirical observations or concrete insights that motivate COS. The authors should clarify both the novelty of the proposed method and the rationale connecting the limitations of existing approaches [1] to its specific design choices.

Question 1: Could the authors clarify how COS differs technically from the closely related work “Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models” and what substantive contributions it makes beyond existing approaches? The related-work section omits relevant studies from 2026, making the novelty difficult to assess. Additionally, what specific observations or insights motivate COS and its design choices? The phrase “Motivated by these observations” appears to refer to existing technical approaches rather than empirical findings or concrete insights, leaving the connection between the stated motivation and the proposed design unclear.

Thank you for helping us further clarify the related work. To maintain the coherence of the main-text discussion, we cited and discussed several studies from 2026 in Appendix G of the original manuscript (page 22, lines 1154–1159 and 1164–1165). These citations are retained in the revised manuscript.

Xiong et al. study how steering for benign capabilities can increase jailbreak risk. Our paper studies how attack formats affect representations and safety boundaries, using input-dependent refusal steering for analysis. We have added this comparison to Appendix G (revised manuscript, page 30, lines 1613–1615).

Regarding the motivation for COS, attack formats may disrupt the representational structure of predefined harmful-topic categories. We therefore use adaptive interventions without predefined harmful-topic categories to observe which groupings and interventions are effective. We have added this connection to the Introduction (page 1, lines 51–53; page 2, lines 72–80). The supporting observations are in Figure 1 (page 2), Section 5.2 (pages 6–7), and Section 5.4 (page 8).

Weakness 2 and Question 2

Weakness 2: Limited experimental evidence for general applicability. The evaluation is restricted to 8B-scale, open-source Qwen and LLaMA models. This narrow experimental scope provides insufficient evidence to establish the broad effectiveness of COS across model scales and families. Evaluation on larger models and a more diverse set of model families would be needed to substantiate claims of general applicability.

Question 2: Could the authors provide evidence that COS remains effective beyond the 8B-scale, open-source Qwen and LLaMA models evaluated in the paper? In particular, does its effectiveness generalize to larger models and other model families? If broader evaluation is not feasible, could the authors clarify the scope of their claims and discuss the limitations imposed by the current experimental coverage?

Thank you for suggesting broader model coverage. We additionally evaluated Qwen3-14B, Gemma 3 4B-IT, and Gemma 3 12B-IT. The table below reports ASR on PromptSecurity-Eval and ORR on XSTest. The settings and results have been added to Appendix F.6 (page 30, Table 19). The two Gemma rows use a three-step intervention limit.

ModelBase ASR ↓COS ASR ↓Base ORR ↓COS ORR ↓
Qwen3-14B30.37%2.11%3.20%3.20%
Gemma 3 4B-IT36.00%1.33%0.00%0.40%
Gemma 3 12B-IT24.11%0.78%0.00%0.00%

Thank you again for your feedback. If you have any further questions or suggestions, we would be glad to hear and respond to them.

Reviewer_pDJBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that organizing large language model safety defenses by harmful topic is the wrong abstraction, and that the format in which a harmful request is presented matters more for how the model represents it. The argument has three parts.

First, a diagnostic analysis. Linear probes on hidden states recover attack-method identity far more accurately than harmful-topic identity (Table 6: 95.64% versus 13.28% at layer 0 on Qwen3-8B, rising to 100.00% versus 13.88% at layer 4), and a variance decomposition attributes far more representation variance to attack-method labels and to nine explicit format features than to topic labels (Table 7: 67.51-91.74% and 54.06-78.79% versus 0.87-4.61%).

Second, a method. COS is a four-stage adaptive activation-steering defense: harmful-versus-benign probes select a contiguous band of intervention layers (17-22 for Qwen3-8B, 12-16 for Llama-3.1-8B-Instruct); harmful requests are clustered and a source steering vector is trained per cluster under a refusal-prefix objective; an autoencoder compresses this library into a routable steering space with an input-conditioned router; and a calibrated scalar threshold gates intervention at inference. On PromptSecurity-Eval, COS reduces LlamaGuard ASR from 37.2% to 0.7% on Qwen3-8B and from 48.2% to 1.4% on Llama-3.1-8B-Instruct, against 22.7% and 6.9% for the strongest baseline (AdaSteer), while XSTest over-refusal moves from 4.00% to 5.60% and from 6.00% to 6.00% respectively, at gate activation rates of 3.20% and 0.00% on benign inputs (Tables 1 and 2).

Third, a transfer analysis. Safety boundaries fitted under one attack format retain high ranking quality when applied to another (mean AUROC around 99.5%) but lose a great deal of fixed-threshold balanced accuracy (76.41% for Qwen3-8B and 80.26% for Llama-3.1-8B-Instruct at layer 24, with the worst pairs at 50.00%). Broader format coverage during training reduces but does not close the gap, and an activation-conditioned correction of the predicted threshold shift recovers part of it (Figures 5 and 6).

Critical strengths
  1. Separating ranking quality from threshold calibration in cross-format transfer.

Section 5.4 establishes that a safety boundary can remain almost perfectly rank-separating across attack formats (AUROC 95.90-99.74% for Qwen3-8B and 96.53-99.55% for Llama-3.1-8B-Instruct, Table 5) while its fixed operating point degrades toward chance. These two properties are routinely conflated in the activation-steering and probing literature, where AUROC is reported as though it settled the question of transfer. The decomposition matters for my assessment because it reframes format robustness as a calibration problem rather than a discrimination problem, which is a different and more tractable target; the correction experiments in Section 5.5, exploratory as they are, follow directly from it. This is the most useful and least obvious result in the paper, and I would want it in the literature even if the rest of the submission were set aside.

  1. Reporting results that work against the authors' own method.

The paper states plainly that broader format coverage reduces but does not eliminate the generalization gap; that the nonlinear corrector underperforms the linear one on Qwen3-8B (84.07% versus 89.66%, Figure 6c); that the reported R-squared values describe association on fitted observations rather than held-out prediction (Appendix D.2); and that the layer-19 routing association “does not establish that bypassing layer 19 causes jailbreak success” (line 383). Appendix F.1 goes further and shows that a baseline's apparently excellent 0.00% string-match over-refusal rate on Llama-3.1-8B-Instruct conceals a 93.60% safe-prompt failure rate dominated by degenerate responses — a correction that works against the paper's own headline metric. This is uncommon and it raises my confidence in the numbers that are reported.

Critical weaknesses
  1. The central format-versus-topic comparison is confounded with surface lexical form, and the paper's own evidence points to the confound.

The paper's organizing claim is that attack format plays a greater role than harmful topic in how a request is represented. I do not think the experiments support that claim as stated, because the two factors are not measured on comparable footing.

Appendix C describes fourteen of the attack transforms as operating by “inserting method-specific prefixes and suffixes.” For those methods, attack identity is close to a deterministic function of the surface string: a prompt carrying the AIM preamble is an AIM prompt by construction. That description is given for the boundary-transfer subset, but Table 6 establishes the same point independently for the probe set, and more directly. Attack-method identity is 95.64% decodable on Qwen3-8B and 95.20% on Llama-3.1-8B-Instruct at layer 0 — the embedding layer, before any contextual computation has occurred. A property that is recoverable from the raw embedding of the input is, by definition, a lexical property. This is not a peripheral detail; it is the layer at which the paper's headline asymmetry is already fully established.

Table 7 reinforces the point. Nine shallow text statistics — character count, word count, punctuation fraction, digit fraction and the like — explain 54.06-78.79% of the representation variance, against 67.51-91.74% for attack-method labels. The main text interprets the fact that the two trajectories track each other across layers as evidence “of an association between method-level organization and explicit format.” The more economical reading is that both quantities are measuring the same underlying artifact: templated boilerplate occupying much of the prompt. In that reading, the comparison between format and topic contrasts a near-perfectly identified lexical label against a semantic label, and the result is close to guaranteed in advance.

The inference drawn from persistence into upper layers does not close this gap. The paper writes that persistence shows the signal “is not limited to shallow lexical differences, but remains strongly represented throughout semantic processing.” The transformer residual stream carries token-level information forward by default, so persistence is the expected behavior whether or not any semantic reorganization has occurred; it is not diagnostic of one.

A second mismatch is that the labels differ in reliability. Attack-method identity is exact and known by construction. Harmful-topic labels are partly produced by a Qwen2.5-7B-Instruct classifier (Appendix B), so they carry annotation noise whose magnitude is nowhere estimated. A probe accuracy of 13.28% against a 10% chance rate for a noisily labeled task is not directly comparable to 95.64% for an exact one. Without a noise ceiling for the topic labels, the size of the asymmetry cannot be interpreted.

Finally, the headline numbers are in-sample. As Appendix D.2 acknowledges, the R-squared values are descriptive, computed on fitted observations across 50 PCA dimensions, and not cross-validated or corrected for degrees of freedom. With 88 prompts, seven topic classes, and 50 retained components, in-sample R-squared is optimistic by a margin the paper does not quantify. That caveat appears in the appendix but not in the abstract or the contribution list, where the claim is stated at full strength.

Taken together, the abstract's formulation — that formats “impact the distribution of LLM semantic representations” — is a causal claim about semantic reorganization, supported here by a correlational and in-sample analysis of lexically templated text. I would accept that attack formats are highly decodable from hidden states. I am not persuaded that this demonstrates format outranks topic as an organizing principle of semantic representation, and I think it is fair to ask for the controls that would distinguish the two readings before the claim is stated in a title.

What would resolve it:

Report topic decodability and variance explained on untransformed Original prompts, alongside the attack-transformed versions, so that the transformation itself is isolated.

Add a template-matched control: benign content wrapped in the same attack templates, tested for whether the template alone produces the measured shift.

Match the two tasks for label noise — for instance by hand-verifying a subset of topic labels — and report a noise ceiling for topic classification.

Report variance explained under cross-validation, with degrees-of-freedom correction, rather than in-sample.

Add a layer-0-only baseline to the variance decomposition, and state how much of the upper-layer signal survives when layer-0 information is partialled out.

  1. COS is evaluated only on attack methods it was trained on, so the headline defense result does not test the paper's own thesis.

Appendix B states that calibration and evaluation data are separated “by the identity of the underlying prompt,” and Table 1 reports all twenty attack methods. The separation therefore holds out seed prompts, not attack formats: COS saw every attack method during training, and the evaluation set contains no format it has not been fitted to. Table 1 reports no held-out-attack-method result for COS at all. The only unseen-method measurements in the paper (Figures 6a and 6b) belong to the separate linear safety boundary of Section 5.4, not to COS.

This matters because of what the paper itself argues. If format is the dominant organizing factor, and if safety boundaries learned under one format fail to transfer to another (Section 5.4), then the 0.7% and 1.4% ASR figures are measured at precisely the point where the method is expected to look strongest. Read alongside Weakness 1, the diagnostic result and the defense result may be a single fact observed twice: attack templates are near-perfectly decodable (100.00% by layer 4, Table 6), and a router trained on those templates reaches 0.0% ASR on fourteen of the twenty methods in Table 1. Neither observation is independent evidence for the paper's thesis, and the second is close to what one would expect from template memorization.

Four further limitations of the evaluation compound this:

Single seed. Table 9 records seed 42 for both models, and no error bars appear in Tables 1, 2, 8, or 12. The pipeline involves clustering into 70 and 72 clusters, an autoencoder trained for 200 epochs, and alternating optimization over four cycles; variability across initializations is plausible and is not measured. The interquartile ranges in Figures 6(c) and 6(d) are over method splits, not seeds.

No operating-point sweep. The threshold is a continuous knob, and one point per method is reported. Without sweeping the threshold for COS and the corresponding strength parameter for CAA, BiPO, CAST, and AdaSteer, Table 1 cannot show that COS dominates the baselines rather than occupying a different position on a shared ASR/over-refusal frontier. Given that Section 5.4 is about threshold miscalibration, this is a conspicuous omission.

Benign-side evaluation is too thin to support “limited benign-side effects.” The evidence is 250 XSTest prompts and nothing else: there is no capability, reasoning, or generation-quality measurement, even though the intervention is applied during generated-token decoding (Table 11). The Qwen3-8B over-refusal change from 4.00% to 5.60% is four prompts out of 250, reported without an interval, and the paper does not supply a second benign set that would indicate whether this is signal or sampling variation. For Llama-3.1-8B-Instruct the reported intervention rate is 0.00%: the gate never opens on those prompts, so the unchanged 6.00% over-refusal rate follows by construction and carries no information about behavior under intervention.

The metric and the training objective are not aligned, and the paper's own diagnostic tool is not applied to its own method. COS is trained to emit a refusal prefix, and ASR is then scored by LlamaGuard. A judge keyed to refusal language may credit a refusal prefix regardless of what follows it. Appendix F.1 shows that string-match over-refusal scores substantially understate degeneration for the CAST baseline, judged at 93.60% failure; the same Qwen2.5 judge is then not applied to COS. Applying it would cost little and would directly address the most obvious failure mode of prefix-level steering.

  1. Model coverage is narrow, and the two models already disagree qualitatively.

As the authors note, the main experiments cover Qwen3-8B and Llama-3.1-8B-Instruct. The concern is not only that two models is few, but that the two behave differently in ways that cut against a general account. Their learned steering spaces differ sharply in sharing: 25 of Qwen's 62 active components reach at least 90% coverage of harmful inputs, against 0 of Llama's 63 (Figure 8). Their correction behavior differs in kind, with a global offset carrying most of the gain on Qwen3-8B and activation-conditioned prediction carrying it on Llama-3.1-8B-Instruct (Section 5.5). Their layer profiles are also model-specific (17-22 versus 12-16). With two samples, and with qualitative disagreement between them, the generalization of the conclusions to other architectures and scales remains open. A third model, or a second scale of the same family, would materially change what can be claimed.

  1. The mechanistic account associates rather than explains, and its central layer claim rests on an untested split.

The paper is careful here — line 383 states the association does not establish causation — and I credit that. The difficulty is that the surrounding narrative does not always maintain the same distance. The routing analysis shows that fifteen of the twenty attack methods peak at layer 19 on Qwen3-8B, that these have a mean ASR of 29.8% against 59.1% for the five that peak later, and that Qwen's safety routing “concentrates at layer 19.” A 15-versus-5 split of twenty methods receives no statistical test, and the underlying quantity is a component logit averaged over a small number of training inputs. Patching or ablating layer-19 routing and observing the effect on jailbreak success would convert this from a correlation into evidence. Absent that, the causal language in the section title and the figure caption outruns the analysis.

  1. Cross-format generalization remains unsolved, and the paper's own results show the mitigations are partial.

This is the authors' own conclusion, and I agree with it as far as it goes: broader training coverage reduces the gap without removing it (Figures 6a and 6b), and score correction reduces threshold miscalibration without, as the nonlinear result on Qwen3-8B shows, reliably improving transfer. My concern is that the paper presents correction as a promising direction while the evidence for it is preliminary: the analysis is labeled exploratory, runs at a fixed layer, uses 100 training and 100 test prompts, and applies a fairly heavy modeling apparatus — ridge regression on an SVD span, RBF kernel ridge regression — to a small problem. As a direction for future work this is reasonable; as a contribution it is underdeveloped, and the negative nonlinear result on one of the two models is left unexplained.

Minor points that did not affect the score

The following are presentation and clarity issues. I list them separately because none of them influenced my rating, and I would expect them to be straightforward to address.

The acronym does not match its expansion: “Context-Aware Steering (COS)” (abstract, and throughout). This reads as a residue of an earlier name and should be reconciled.

Figure 1 carries the opening argument of the paper, but t-SNE geometry does not license claims about distance or cluster structure. Table 3 supplies the right quantitative companion; the silhouette values are, however, modest in absolute terms (0.142-0.312 for transformation, with the layer-19 topic silhouette at -0.005), and the caption should say so rather than leaving the visual impression to stand on its own.

Figures 5(a) and 5(b) use a color scale spanning -1 to 1 for data that appear to be entirely positive, which discards half the available dynamic range and flattens the contrast the panels are meant to show.

Figure 4(c) and Figure 5 use type sizes that are difficult to read at print scale, and the dashed and solid series in Figure 2(c) are hard to separate. Table 1 would read better with per-method rows moved to an appendix and aggregates with dispersion in the main text.

The benchmark name is inconsistent: “PromptSecurity” in the caption of Table 1 against “PromptSecurity-Eval” in the text.

The selected threshold for Qwen3-8B is 0.145, close to the upper limit of the calibration grid [-0.05, 0.15] (Table 11). Please state whether the grid was wide enough, or whether the selected operating point sits at a boundary.

Section 5.5 and Appendix H apply the heaviest machinery in the paper to its least developed analysis. A short statement of why ridge regression on an SVD span and RBF kernel ridge regression were chosen, and what the nonlinear regression on Qwen3-8B indicates, would help the reader calibrate how much weight the section carries.

Code is promised “upon acceptance.” Anonymized code at submission time would have let reviewers check the data-separation protocol directly, which is central to Weakness 2.

Questions for authors

What is COS's attack success rate on attack methods excluded from its training entirely? Holding out, say, five methods, retraining the full pipeline on the remaining fifteen, and reporting ASR on the held-out five would directly test the claim that COS generalizes, and would speak to Weakness 2 more than any other single experiment.

Can you report harmful-topic decodability and variance explained on untransformed Original prompts, and on a template-matched control in which benign content is wrapped in the same attack templates? What is the estimated label-noise ceiling for the topic labels produced by the Qwen2.5-7B-Instruct annotator?

Do the R-squared values in Table 7 hold under cross-validation with degrees-of-freedom correction, rather than in-sample on 50 PCA dimensions? If the gap between attack method and harmful topic narrows under correction, by how much?

Does the Qwen2.5 judge used in Appendix F.1 reproduce the ASR reductions that LlamaGuard reports for COS? Specifically, in cases scored as successful refusals, does harmful content follow the refusal prefix?

Can you provide ASR against over-refusal trade-off curves, sweeping the threshold for COS and the corresponding strength parameter for CAA, BiPO, CAST, and AdaSteer? Without them it is difficult to tell whether COS dominates the baselines or occupies a different operating point.

Can the main tables be reported over at least three seeds, with dispersion? Given the clustering, autoencoder training, and alternating optimization in the pipeline, single-seed results are difficult to interpret, particularly for differences of one or two percentage points.

Is there any measurement of benign capability under active intervention — a general-purpose benchmark, or a generation-quality evaluation on prompts where the gate actually fires? Because the XSTest intervention rate for Llama-3.1-8B-Instruct is 0.00%, the current benign evidence for that model does not exercise the intervention path at all.

For the layer-19 claim: is there a causal test, such as patching or ablating layer-19 routing and measuring the effect on jailbreak success? And does the 15-versus-5 split of attack methods survive a statistical test, given that the comparison rests on twenty observations?

(Future work, not blocking.) Would you consider adding a third model and a second scale? The two current models already disagree qualitatively — 25 of 62 against 0 of 63 components at 90% coverage, and global-offset against activation-conditioned correction — which makes it hard to tell which findings are general and which are model-specific.

8 comments
AuthorsOpenReview ↗

Response to Reviewer pDJB Part 1

Dear Reviewer pDJB,

Thank you for your contribution to the ICLR 2027 review process. We greatly value your suggestions. Several of your comments pointed out blind spots in our previous work and are important for improving it. Below, we respond to your comments and questions one by one.

Weakness 1

The central format-versus-topic comparison is confounded with surface lexical form, and the paper's own evidence points to the confound.

1.1 Surface lexical form, templates, and Original controls

Concern summary Fixed templates and suffixes provide direct lexical cues to attack-method identity, making it easy to decode from the embedding layer onward. Harmful topics must instead be inferred from the request content. The reviewer therefore considers the decodability gap foreseeable and argues that this gap alone makes it difficult to determine how the model organizes semantics.

Response

Thank you for pointing out this asymmetry. We agree that attack methods and harmful topics provide different cues, and attack-method signals can appear in very early layers (Layer 0 in our LLM layer indexing is the first decoder block, not the embedding layer). This comparison is still necessary because surface lexical form is itself part of an attack method. These features enter the model through token embeddings and can substantially affect subsequent representations and generation. For example, GCG suffixes can change a model's refusal behavior for the same request even when they lack coherent semantics for human readers. Research on prompt formatting also shows that changing surface format can substantially affect model outputs while preserving the meaning of the request (Zou et al., 2023; Sclar et al., 2024).

In the activations we measure, human-defined harmful topics explain much less variation than these format features. Adding templates also reduces the linear readability of topics. Even if this gap is foreseeable, it still reveals a consequence with practical relevance for defense. Under the impact of attack formats, requests sharing the same harmful topic need not remain close in hidden space. This weakens the basis for routing by predefined topics. For example, CAST's topic-conditioned configuration needs to identify predefined topics reliably from activations to trigger the corresponding interventions (Lee et al., 2025).

References

  • Zou et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models.
  • Sclar et al. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design…
  • Lee et al. (2025). Programming Refusal with Conditional Activation Steering.

1.2 Source and reliability of topic labels

Concern summary The reviewer believes that some harmful-topic labels in Section 5.2 come from automatic annotation by Qwen2.5 and is therefore concerned that annotation noise affects the topic-classification results.

Response

Thank you for reminding us to clarify the source of the labels. The harmful-topic labels in Section 5.2 and Tables 6 and 7 directly use JailbreakBench's ten official categories. The Qwen2.5 annotation described in Appendix B is used for a separate seven-category PromptSecurity analysis. Therefore, manual verification of automatically annotated labels does not apply to the comparison in Section 5.2.

Following the settings in Table 6, we compare topic classification on untransformed prompts (Original) and prompts with attack formats, using the same 50 JBB seeds, official topic labels, and data splits. The table below reports linear classification accuracy at layer 24. After adding attack formats, topic readability decreases in both models even when a classifier is retrained separately for each format.

ModelOriginalWith attack formats (retrained per format)
Qwen3-8B45.60%20.56%
Llama 3.1 8B62.40%32.20%

We further examine categories defined directly by model representations. We use five-fold evaluation on the 88 PromptSecurity seeds. In each fold, the Original activations of the training seeds are clustered into seven clusters, matching the number of topic categories. Each test seed's cluster is determined from its Original activation, and its attack variants retain that cluster label. The following table reports layer-24 linear classification accuracy for identifying the Original clusters.

ModelOriginalWith attack formats (retrained per format)
Qwen3-8B72.73%31.10%
Llama 3.1 8B95.45%56.16%

This shows that even categories defined by proximity in the Original activation space become harder to distinguish linearly after adding attack formats. It further shows that attack methods and their formats substantially affect the readability of existing categories in model representations. We have added both sets of results to Appendix D.3 of the revised manuscript (page 18, Table 8). Thank you again for this insight.

AuthorsOpenReview ↗

Response to Reviewer pDJB Part 2

1.3 In-sample variance, cross-validation, and degrees-of-freedom correction

Concern summary The reviewer is concerned that in-sample fitting exaggerates the R² gap between attack methods and harmful topics in Table 7, and asks whether the gap narrows after degrees-of-freedom correction and held-out validation.

Response

Thank you for pointing this out. Each group in Table 7 contains variants of 50 JBB seeds under 10 attack methods, giving 500 inputs. The 50 PCA coordinates are regression outputs. The attack-method and harmful-topic regressions each use nine category indicators and an intercept, with the same degrees of freedom.

We performed in-sample degrees-of-freedom correction and five-fold validation grouped by seed. During cross-validation, PCA and regression are fitted only on the training fold, and R² is calculated on held-out seeds. The layer-24 results are as follows.

ModelIn-sample R² gap (original → corrected, percentage points)Held-out R² (attack method / harmful topic)
Qwen3-8B81.10 → 82.5986.23% / −0.43%
Llama 3.1 8B62.90 → 64.0583.20% / −1.38%

The in-sample gap does not narrow after degrees-of-freedom correction. On held-out seeds, attack-method R² remains substantially higher than harmful-topic R². With held-out validation directly on full-dimensional activations, the gaps remain 76.94 and 56.47 percentage points for the two models, respectively. We have added the settings and results of both checks to Appendix D.4 of the revised manuscript (page 18, Table 9).

AuthorsOpenReview ↗

Response to Reviewer pDJB Part 3

1.4 Computational updates in higher layers

Concern summary The reviewer argues that attack formats remaining decodable in higher layers can be explained by residual connections preserving early lexical cues. This observation alone is therefore insufficient to support the paper's claim about attack formats reorganizing semantic representations.

Response

We believe that the paper's claim that “formats impact the distribution of LLM semantic representations” holds. As discussed in 1.1, hidden representations carry the information processed by an LLM. They contain both information entering through input embeddings and information reorganized through the model's contextual computation. Subsequent model layers directly read the full hidden states, and activation-steering methods also read or modify activations in this space.

To further examine whether the effects of attack formats also appear in updates produced by the model's layer-by-layer computation, we calculate the difference between the output and input residual states of each decoder block. The input to Layer 0 is the embedding. We use the linear probes from Table 6 to measure the linear separability of attack methods and harmful topics in each layer's net update. The following table lists the complete results for layers 20–25. Layer numbering follows the paper, and accuracies are averaged over five attack-method groups.

LayerQwen3-8B attack methodQwen3-8B harmful topicLlama 3.1 8B attack methodLlama 3.1 8B harmful topic
2099.72%19.24%99.64%33.16%
2199.88%19.84%99.84%34.76%
2299.88%21.80%99.64%32.76%
2399.88%22.80%99.68%33.76%
2499.84%25.84%99.96%32.24%
2599.80%24.76%99.96%33.76%

In every layer's net update in both models, attack methods are easier to decode linearly than harmful topics. This shows that format-related information also appears in the updates written into the residual stream by the model's layer-by-layer computation. We have added the experimental settings and complete results for layers 20–25 to Appendix D.5 of the revised manuscript (page 19, Table 10), with the new content highlighted in light blue.

We also provide the following results for the other checks you suggested. Their settings and values are in Appendix D.6 of the revised manuscript (page 19).

For the same-template benign-input control, we apply the same 14 templates to harmful and benign requests. At layer 24, the median relative displacement norms for benign/harmful inputs are 0.414/0.362 for Qwen and 0.650/0.661 for Llama. The median cosine similarities between the two classes' mean displacement directions under the same template are 0.800/0.832, respectively. Thus, the same templates also produce substantial representation shifts in benign requests, with similar directions.

For topic explained variance on Original prompts, held-out R² in the full hidden space at layer 24 under Original, independently fitted individual templates, and mixed templates is −5.36%/−13.19%/−0.49% for Qwen and −8.31%/−13.63%/−1.11% for Llama, respectively. Negative values mean that topic centroids predict held-out representations with higher error than the training-fold mean baseline. Each condition uses its own target variance. The classification experiments in 1.2 directly compare topic readability before and after adding templates to the same seeds.

For the Layer 0 control, we linearly predict layer-24 activations from the first decoder block's representations. Held-out R² is 75.94% for Qwen and 57.01% for Llama. Adding attack-method labels explains an additional 1.18 and 1.43 percentage points of total variance, respectively. This check quantifies the increment beyond the early-layer linear model, while the layer-by-layer experiment in 1.4 measures format-signal readability in subsequent computational updates.

Reviewer_2GauOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the influence of jailbreak attack formats on the internal semantic representations of large language models. It shows that different attack formats are more distinguishable in activation space than attack topics. This finding challenges existing safety defenses that rely on predefined topic-dependent steering strategies. To address this limitation, the paper proposes Context-Aware Steering (COS), an adaptive activation-steering method that learns input-dependent steering vectors across diverse jailbreak attack formats and harmful topics. Furthermore, the paper demonstrates that safety boundaries generalize poorly across attack formats and explores activation-based score correction methods to mitigate this issue.

Critical strengths
  1. The observation that jailbreak attack formats have a stronger influence on LLM internal representations than harmful topics is important and provides a valuable warning for LLM safety.

  2. The joint learning of harmful-input detection and adaptive steering interventions is a potentially novel and interesting architectural design.

  3. The experimental evaluation is comprehensive, and COS achieves substantial improvements over existing baselines, demonstrating strong practical effectiveness.

Critical weaknesses
  1. The motivation connecting the main observation to the proposed method is insufficiently explained. Specifically, it is unclear why the dominance of attack format over harmful topic in the representation space necessarily motivates learning heterogeneous steering vectors, rather than alternative approaches that explicitly account for attack formats.
  2. The analysis of cross-format safety-boundary generalization (third contribution) appears somewhat disconnected from the main observation and the proposed COS method (first and second contribution). Also, it is unclear whether the cross-format generalization issue identified in the third contribution is explicitly considered or addressed by COS.
Questions for authors

The paper demonstrates that attack-format identity is substantially more distinguishable than harmful-topic identity in activation space. How does harmfulness itself (i.e., harmful versus benign inputs) compare in terms of representation-space separability? Is harmfulness also less distinguishable than attack-format identity? And how does this issue motivates the method design?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2Gau Part 1

Dear Reviewer 2Gau,

Thank you for your contribution to the ICLR 2027 review process. Your comments are helpful for improving the logical coherence of our paper. We respond to your comments and questions below.

Weakness 1

The motivation connecting the main observation to the proposed method is insufficiently explained. Specifically, it is unclear why the dominance of attack format over harmful topic in the representation space necessarily motivates learning heterogeneous steering vectors, rather than alternative approaches that explicitly account for attack formats.

Thank you for pointing this out. This question helps us clarify the connection between the representation observations and the method design. Defense methods with topic-based routing define categories in advance and assign interventions to each category. However, attack methods and their formats can significantly change request representations, making predefined categories difficult to use as a reliable basis for intervention. We therefore designed COS to adaptively learn how to group inputs and how to intervene from the training data. This can reduce the restrictions that unsuitable predefined categories impose on learning, while allowing us to observe which groupings and interventions are effective from the results learned by COS. We have added this connection to the Introduction (revised manuscript, page 1, lines 51–53, and page 2, lines 72–73), highlighted with a light-blue background.

Weakness 2

The analysis of cross-format safety-boundary generalization (third contribution) appears somewhat disconnected from the main observation and the proposed COS method (first and second contribution). Also, it is unclear whether the cross-format generalization issue identified in the third contribution is explicitly considered or addressed by COS.

The question of “when intervention is needed,” which you precisely identify, is an aspect that can be overlooked when organizing defenses by topic. Lines 13–16 of the original abstract state that organizing harmful inputs by topic may overlook representation changes induced by jailbreak formats. In defense, directly applying a threshold calibrated on some input formats to other formats implicitly assumes that the intervention decision remains applicable. Section 5.4 shows that this assumption can fail.

This is also part of what COS learns adaptively. COS learns both when and how to intervene, learning intervention decisions from training and calibration data with mixed formats. This can address a substantial part of the format differences covered by training. Section 5.4 further uses linear classifiers independent of COS to separately measure whether scores still rank harmful and benign inputs correctly and whether the original threshold remains applicable. This allows us to examine more directly why intervention decisions that work on one set of formats may fail on new formats.

The table below reports supplementary JBB evaluation results on three models. The evaluation contains 989 inputs covering 23 attack methods not seen during COS training.

ModelBase ASR ↓COS ASR ↓
Qwen3-8B50.96%26.59%
Qwen3-14B45.70%26.90%
Llama 3.1 8B36.60%24.77%

We have clarified the above connection in the overview paragraph of the Introduction (page 2, lines 74–80) and the opening of Section 5.4 (page 8, lines 381–383). The out-of-distribution results for COS have been added to Section 5.1 (page 6, lines 280–281) and Appendix F.2 (page 26, Table 16). Section 5.5 retains more robust cross-format intervention as a direction for future work (page 9, lines 458–463).

AuthorsOpenReview ↗

Response to Reviewer 2Gau Part 2

Question 1

The paper demonstrates that attack-format identity is substantially more distinguishable than harmful-topic identity in activation space. How does harmfulness itself (i.e., harmful versus benign inputs) compare in terms of representation-space separability? Is harmfulness also less distinguishable than attack-format identity? And how does this issue motivates the method design?

Thank you for raising these two questions, which concern the central point of our paper. When stable decisions across formats are required, distinguishing harmful from benign inputs is harder than identifying attack formats. Section 5.4 places harmful and benign requests in the same attack formats and retains the harmfulness labels of the original requests; thus, benign requests with attack formats are still labeled benign. A safety boundary learned on one format often does not transfer well to another format when its original threshold is retained (page 8, Figure 5(c,d)). The out-of-distribution generalization gap remains even when the attack formats covered by training are increased (page 9, Figure 6(a,b)). Meanwhile, Figure 3 on page 6 shows that attack-method identity is easy to distinguish in representation space.

For the design of harmfulness detection, attack format is an important feature that significantly affects model representations and safety boundaries. As stated in the Introduction, page 1, lines 43–47, changing the wrapping can significantly change the position of even the same harmful behavior in representation space. The assumption that samples humans consider semantically similar will also be close in LLM representation space should not serve as a design premise for harmfulness detection methods. We hope that future methods can disentangle harmfulness signals that remain stable under format changes from model representations.

Thank you again for your feedback. Your comments are helpful for improving the clarity and logical coherence of our paper. If you have any further questions or suggestions, we would be happy to hear and respond to them.

Thanks for the clarification. I still have two concerns:

  1. Different representations do not necessarily need different steering directions. Different attack formats may move inputs to different places in activation space, but the same direction might still steer them toward refusal. The results show that format matters. However, why does this call for multiple learned directions, rather than one shared direction with an input-dependent strength or threshold? Is there an existing comparison that separates these choices?
  2. The new JBB results use a different version of COS. Appendix F.2 adds a logistic gate and applies steering to both prompt and generated tokens. The main method uses the router’s scores for gating and steers only generated tokens. The new results are useful, but they somehow lead to confusion about whether the original COS design handles unseen formats well. Are results available for the original configuration on the same test? Please make clear which conclusions apply to the original method and which apply to the modified version.

Thank you for your further suggestions.

Follow-up Question 1: Shared and input-dependent steering directions

Thank you for highlighting this distinction. Activation steering involves two important questions: when intervention is needed, and what intervention is appropriate. Our paper primarily focuses on the first under attack-format variation, while your question draws attention to the second. We allowed COS to learn shared or specialized interventions so that adaptive steering had the freedom to reveal the structure learned from the training data, without prescribing either organization.

We therefore examined the final predicted steering vectors from the frozen original COS on 899 Qwen and 900 Llama test inputs. We normalized each vector to separate direction from strength and calculated the following statistics separately within each layer, then averaged across layers. The common direction is the normalized mean of unit vectors within that layer.

ModelMean pairwise direction cosineDirections with cosine ≥0.90 to the common direction
Qwen3-8B0.92090.23%
Llama 3.1 8B0.4670.60%

Qwen's directions are predominantly aligned, whereas Llama's directions are more diverse. Seed-blocked permutation tests also found significant attack-method and harmful-topic effects in both models (9,999 permutations; Holm-adjusted p=0.0004 for all four omnibus tests). After controlling for seed identity, attack methods additionally explain 90.96% and 80.51% of total centered directional variation in Qwen and Llama; after controlling for attack method, harmful topics explain 0.96% and 1.40%, respectively. In Qwen, systematic method-related differences therefore coexist with strong alignment around a common direction. These observations characterize the interventions COS adaptively learns. This analysis has been added to Appendix E.6 (page 25, Table 15).

Follow-up Question 2: Original and revised gating configurations

Thank you for requesting this clarification. We have supplemented the results of the original configurations (Appendix F.2, page 27, Table 17):

ModelBase ASR ↓Original router gate ASR ↓Revised gate ASR ↓
Qwen3-8B50.96%18.91%26.59%
Llama 3.1 8B36.60%33.37%24.77%

Thank you again for your feedback. Your comments are helpful for improving the clarity and logical coherence of our paper. If you have any further questions or suggestions, we would be happy to hear and respond to them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.