- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This study seeks to identify why adding adversarial suffixes to harmful queries causes models to make incorrect safety judgments. Specifically, the authors compare hidden activations for harmful text with and without adversarial suffixes, then test whether transplanting attack-time neuron activations reproduces the incorrect judgment. They identify neurons shared by many circuits and associate these circuits with themes such as games, cooking, poetry, and horror fiction.
The proposed method, RAMEN, begins with a text and a version of that text containing an adversarial suffix. Both are fed into the model, and their activations are transplanted to evaluate faithfulness (whether the selected circuit alone can reproduce the attack) and completeness (whether the attack cannot be reproduced using activations outside the selected circuit).
Critical strengths
Harmful requests vary widely in topic and wording, making it difficult to construct standardized examples or compare them token by token. By pairing a clean text with the same text plus an adversarial suffix, this study enables token-level comparison. This is a useful research direction. The authors also identify relatively small circuits.
Critical weaknesses
The circuits identified for each input are input-dependent. For understanding model safety, the circuits shared across inputs and the insights they provide therefore matter more than any individual circuit. From this perspective, I would like the authors to clarify the following points.
-
By examining neurons repeatedly activated across many circuits, the authors identify textual patterns involving games, cooking, poetry, and horror fiction that appear likely to induce successful attacks. However, the paper does not describe in detail how these patterns were identified. My understanding is that neurons activated by adversarial suffixes were also activated by clean texts without those suffixes. How was the correlation between clean text and neuron activation measured, and what was the sample size? Given the many possible text categories, do games, cooking, poetry, and horror fiction activate these neurons significantly more than other categories? Do attacks incorporating these categories actually activate the identified neurons? The paper lacks explanations or experiments addressing these questions.
-
The authors propose a new attack prompt with the following structure: “a fixed preamble about poetry and cooking” -> “a fixed transition asking the model to ghostwrite” -> “20 tokens optimized with GCG” -> “a harmful question reframed as a horror fiction scenario.” This appears intended to validate the categories identified above. Could the attack succeed simply because it uses GCG? If the identified categories are useful for attacks, would the prompt succeed without the GCG-optimized tokens? Even when GCG is used, the authors should test whether replacing the identified categories with other categories significantly reduces the attack success rate. The authors may intend the experiment using I-GCG alone to address this concern, but its prompt has a different overall structure and therefore does not constitute a valid ablation study.
Questions for authors
The experimental framework and the method for identifying small circuits appear novel, but the resulting insights and conclusions remain open to question. I would appreciate it if the authors could provide further clarification and address the concerns raised above.
3 comments
Part 2
Q4: The authors propose a new attack prompt with the following structure: “a fixed preamble about poetry and cooking” -> “a fixed transition asking the model to ghostwrite” -> “20 tokens optimized with GCG” -> “a harmful question reframed as a horror fiction scenario.” This appears intended to validate the categories identified above. Could the attack succeed simply because it uses GCG? If the identified categories are useful for attacks, would the prompt succeed without the GCG-optimized tokens? Even when GCG is used, the authors should test whether replacing the identified categories with other categories significantly reduces the attack success rate. The authors may intend the experiment using I-GCG alone to address this concern, but its prompt has a different overall structure and therefore does not constitute a valid ablation study.
R4. To isolate which parts of the attack matter, we provide ablations of the full attack against LAT and RR (n = 50, 95% CI). All rows include I-GCG.
| Attack variant | LAT ASR % (95% CI) | RR ASR % (95% CI) |
|---|---|---|
| I-GCG only | 16.0 (8.3–28.5) | 14.0 (7.0–26.2) |
| Full attack (horror+poetry+cooking) | 54.0 (40.4–67.0) | 70.0 (56.2–80.9) |
| Horror only | 50.0 (36.6–63.4) | 40.0 (27.6–53.8) |
| Fiction only | 4.0 (1.1–13.5) | 0.0 (0.0–7.1) |
I-GCG alone reaches far lower ASR than the full attack. The horror framing accounts for most of the gain on LAT and over half on RR, where the poetry and cooking levers add a further 30 points. Replacing horror with a non-horror, fairytale fiction framing drops ASR below even the I-GCG baseline. This shows that the effect comes from the specific categories surfaced by the recurring neurons, not a generic creative writing framing.
Our attacks on WildGuard do not use GCG and are prompt-universal.
I thank the authors for their response. However, several points remain unclear to me.
First, I now understand that the categories of interest were identified through human visual inspection. The authors have now reported the percentage associated with the identified category for each neuron. However, they should provide a ranking of categories rather than reporting the percentage for a single category alone. In particular, when the percentage does not exceed 50%, it is unclear whether the identified category is the most prevalent (that is, whether it ranks first).
Furthermore, the analysis covers only four neurons. What are the results for the other neurons? These results should be reported for all of the top ten neurons. This analysis might also reveal the presence of other categories. The sum-z metric may be relevant here, but unless the categories are ranked by this metric, it remains unclear how strongly the statistics favor the categories identified through visual inspection. The additional evidence provided by the authors supports their visual observations but takes those observations as its starting point. It would be preferable to identify categories using a well-defined statistical method from the outset.
(I did not raise the following point in my initial review, so it does not affect the score.) As other reviewers have also noted, the manuscript is generally dense and contains many passages that are difficult to understand. Sections 3, 4, and 5, in particular, require substantial improvement. For example, the number of mathematical symbols should be reduced considerably. Many readers will find it very difficult to keep track of the meaning of each of the numerous symbols introduced throughout these sections. I recommend resubmitting the manuscript after clarifying the statistical issues discussed above and restructuring the paper. I find some aspects of this study’s methods and overall approach interesting.
We thank the reviewer for their constructive feedback and provide detailed responses below.
Part 1
R1. We inspect the top activating clean examples across a corpus of training samples. Our corpus consists of 2000 benign samples and 2000 harmful samples randomly drawn from the safety fine tuning dataset for each respective model.
R2. The initial categories were observed by human inspection. For quantitative data, we annotate the 4000 samples with Qwen3-30B-A3B-Instruct and then compare how often the top 10 (16 for WildGuard due to ties) most frequent F+C neurons’ top activating examples consist of samples corresponding to its observed category. Sum-z refers to the examples that maximize the sum of z scores of all the top 10/16 most frequent neurons.
WildGuard
Llama RT
R3. For the identified neurons, we report below the difference between its maximum activation across positions under attack and its maximum activation without the attack. Since the activation scales can vary across neurons, we include in parentheses the percentile of that difference across that of 256 randomly selected neurons (higher is better).
LAT
RR
WildGuard
For LAT and RR, L10#4789 and L16#9106 appear to be selective for their category. L10#4789 sees its largest shifts in the two rows when horror is present, full and horror. L16#9106, which responded to recipes, sees its largest shift in full, the only row where a recipe is present.
For WildGuard, the attacks do activate the identified neurons but they seem to track how successful the attack is more than their own identified category.
A plausible explanation for why the attacks succeed is as follows: neurons that are consistently causally relevant across adversarial examples have a large role in the model’s processing of the safety decision. When such a neuron also responds strongly to an unrelated concept, the model has intertwined its representation of that concept with its representation of harm or safety. Polysemanticity of this kind can persist if the two features rarely came into conflict during training, since harmful requests involving the concept had been common enough for the overlap to cause errors, training would have separated the two representations. Such a neuron can signal a gap in the training distribution and a potential non-robust association, even if that single neuron does not carry the entire causal weight of its category.