acceptodds
Under review as a conference paper at ICLR 2027

Scaling Practical Circuit Discovery for Adversarial Attacks on Language Models

Abstract

Circuit level interpretability has developed almost entirely on templated tasks, which limits its applicability to real world settings. We scale it to safety classification and refusal over diverse, unstructured prompts by using adversarial suffixes as contrastive pairs, and search for circuits directly in the vast MLP neuron basis. Gradient scoring methods such as attribution patching and integrated gradients fail at this granularity because they score every neuron once, at the empty circuit, as if each acted alone. We instead re-score along the search trajectory, proposing block additions and removals from orderings evaluated at the current circuit and accepting only moves that causal evaluation confirms. This recovers circuits far smaller than existing baselines while satisfying standard circuit metrics.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, circuit discovery, explainable AI, non-robust features

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sPWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This study seeks to identify why adding adversarial suffixes to harmful queries causes models to make incorrect safety judgments. Specifically, the authors compare hidden activations for harmful text with and without adversarial suffixes, then test whether transplanting attack-time neuron activations reproduces the incorrect judgment. They identify neurons shared by many circuits and associate these circuits with themes such as games, cooking, poetry, and horror fiction.

The proposed method, RAMEN, begins with a text and a version of that text containing an adversarial suffix. Both are fed into the model, and their activations are transplanted to evaluate faithfulness (whether the selected circuit alone can reproduce the attack) and completeness (whether the attack cannot be reproduced using activations outside the selected circuit).

Critical strengths

Harmful requests vary widely in topic and wording, making it difficult to construct standardized examples or compare them token by token. By pairing a clean text with the same text plus an adversarial suffix, this study enables token-level comparison. This is a useful research direction. The authors also identify relatively small circuits.

Critical weaknesses

The circuits identified for each input are input-dependent. For understanding model safety, the circuits shared across inputs and the insights they provide therefore matter more than any individual circuit. From this perspective, I would like the authors to clarify the following points.

  1. By examining neurons repeatedly activated across many circuits, the authors identify textual patterns involving games, cooking, poetry, and horror fiction that appear likely to induce successful attacks. However, the paper does not describe in detail how these patterns were identified. My understanding is that neurons activated by adversarial suffixes were also activated by clean texts without those suffixes. How was the correlation between clean text and neuron activation measured, and what was the sample size? Given the many possible text categories, do games, cooking, poetry, and horror fiction activate these neurons significantly more than other categories? Do attacks incorporating these categories actually activate the identified neurons? The paper lacks explanations or experiments addressing these questions.

  2. The authors propose a new attack prompt with the following structure: “a fixed preamble about poetry and cooking” -> “a fixed transition asking the model to ghostwrite” -> “20 tokens optimized with GCG” -> “a harmful question reframed as a horror fiction scenario.” This appears intended to validate the categories identified above. Could the attack succeed simply because it uses GCG? If the identified categories are useful for attacks, would the prompt succeed without the GCG-optimized tokens? Even when GCG is used, the authors should test whether replacing the identified categories with other categories significantly reduces the attack success rate. The authors may intend the experiment using I-GCG alone to address this concern, but its prompt has a different overall structure and therefore does not constitute a valid ablation study.

Questions for authors

The experimental framework and the method for identifying small circuits appear novel, but the resulting insights and conclusions remain open to question. I would appreciate it if the authors could provide further clarification and address the concerns raised above.

3 comments

We thank the reviewer for their constructive feedback and provide detailed responses below.

Part 1

Q1. My understanding is that neurons activated by adversarial suffixes were also activated by clean texts without those suffixes. How was the correlation between clean text and neuron activation measured, and what was the sample size?

R1. We inspect the top activating clean examples across a corpus of training samples. Our corpus consists of 2000 benign samples and 2000 harmful samples randomly drawn from the safety fine tuning dataset for each respective model.

Q2. Given the many possible text categories, do games, cooking, poetry, and horror fiction activate these neurons significantly more than other categories?

R2. The initial categories were observed by human inspection. For quantitative data, we annotate the 4000 samples with Qwen3-30B-A3B-Instruct and then compare how often the top 10 (16 for WildGuard due to ties) most frequent F+C neurons’ top activating examples consist of samples corresponding to its observed category. Sum-z refers to the examples that maximize the sum of z scores of all the top 10/16 most frequent neurons.

WildGuard

  • L20#14286: 53% of top 30 activating examples are nonsense discrimination, vs. corpus base rate of 6.8%.
  • L19#2556: 43% of top 30 activating examples are homonyms, vs. corpus base rate of 4.0%.
  • sum-z 27% of top 30 examples are videogame related, vs. corpus base rate of 6.5%.

Llama RT

  • L10#4789: 50% of top 30 activating examples are horror or dark writing, vs. corpus base rate of 2.85%.
  • L16#9106: 73% of top 30 activating examples are cooking related, vs. corpus base rate of 5.2%.
  • sum-z 13% of top 30 examples are poetry related, vs. corpus base rate of 2.6%.

Q3. Do attacks incorporating these categories actually activate the identified neurons?

R3. For the identified neurons, we report below the difference between its maximum activation across positions under attack and its maximum activation without the attack. Since the activation scales can vary across neurons, we include in parentheses the percentile of that difference across that of 256 randomly selected neurons (higher is better).

LAT

AttackL10#4789L16#9106
I-GCG only+0.0324 (71.5th)+0.0384 (78.5th)
Full+0.1947 (86.3rd)+3.3179 (99.6th)
Horror+0.3584 (98.0th)+0.0377 (43.0th)
Fiction+0.0314 (48.0th)+0.0410 (59.8th)

RR

AttackL10#4789L16#9106
I-GCG only+0.0288 (69.9th)+0.0512 (90.6th)
Full+0.2580 (91.0th)+3.0302 (99.6th)
Horror+0.4672 (98.4th)+0.0506 (54.3th)
Fiction+0.0291 (41.8th)+0.0511 (68.8th)

WildGuard

AttackL20#14286L19#2556
Combo A+0.6332 (95.7th)+0.0887 (79.3th)
Combo B+1.1027 (97.7th)+0.5973 (94.5th)
Combo B + wrap+4.3589 (99.2th)+2.4261 (98.8th)

For LAT and RR, L10#4789 and L16#9106 appear to be selective for their category. L10#4789 sees its largest shifts in the two rows when horror is present, full and horror. L16#9106, which responded to recipes, sees its largest shift in full, the only row where a recipe is present.

For WildGuard, the attacks do activate the identified neurons but they seem to track how successful the attack is more than their own identified category.

A plausible explanation for why the attacks succeed is as follows: neurons that are consistently causally relevant across adversarial examples have a large role in the model’s processing of the safety decision. When such a neuron also responds strongly to an unrelated concept, the model has intertwined its representation of that concept with its representation of harm or safety. Polysemanticity of this kind can persist if the two features rarely came into conflict during training, since harmful requests involving the concept had been common enough for the overlap to cause errors, training would have separated the two representations. Such a neuron can signal a gap in the training distribution and a potential non-robust association, even if that single neuron does not carry the entire causal weight of its category.

Part 2

Q4: The authors propose a new attack prompt with the following structure: “a fixed preamble about poetry and cooking” -> “a fixed transition asking the model to ghostwrite” -> “20 tokens optimized with GCG” -> “a harmful question reframed as a horror fiction scenario.” This appears intended to validate the categories identified above. Could the attack succeed simply because it uses GCG? If the identified categories are useful for attacks, would the prompt succeed without the GCG-optimized tokens? Even when GCG is used, the authors should test whether replacing the identified categories with other categories significantly reduces the attack success rate. The authors may intend the experiment using I-GCG alone to address this concern, but its prompt has a different overall structure and therefore does not constitute a valid ablation study.

R4. To isolate which parts of the attack matter, we provide ablations of the full attack against LAT and RR (n = 50, 95% CI). All rows include I-GCG.

Attack variantLAT ASR % (95% CI)RR ASR % (95% CI)
I-GCG only16.0 (8.3–28.5)14.0 (7.0–26.2)
Full attack (horror+poetry+cooking)54.0 (40.4–67.0)70.0 (56.2–80.9)
Horror only50.0 (36.6–63.4)40.0 (27.6–53.8)
Fiction only4.0 (1.1–13.5)0.0 (0.0–7.1)

I-GCG alone reaches far lower ASR than the full attack. The horror framing accounts for most of the gain on LAT and over half on RR, where the poetry and cooking levers add a further 30 points. Replacing horror with a non-horror, fairytale fiction framing drops ASR below even the I-GCG baseline. This shows that the effect comes from the specific categories surfaced by the recurring neurons, not a generic creative writing framing.

Our attacks on WildGuard do not use GCG and are prompt-universal.

I thank the authors for their response. However, several points remain unclear to me.

First, I now understand that the categories of interest were identified through human visual inspection. The authors have now reported the percentage associated with the identified category for each neuron. However, they should provide a ranking of categories rather than reporting the percentage for a single category alone. In particular, when the percentage does not exceed 50%, it is unclear whether the identified category is the most prevalent (that is, whether it ranks first).

Furthermore, the analysis covers only four neurons. What are the results for the other neurons? These results should be reported for all of the top ten neurons. This analysis might also reveal the presence of other categories. The sum-z metric may be relevant here, but unless the categories are ranked by this metric, it remains unclear how strongly the statistics favor the categories identified through visual inspection. The additional evidence provided by the authors supports their visual observations but takes those observations as its starting point. It would be preferable to identify categories using a well-defined statistical method from the outset.

(I did not raise the following point in my initial review, so it does not affect the score.) As other reviewers have also noted, the manuscript is generally dense and contains many passages that are difficult to understand. Sections 3, 4, and 5, in particular, require substantial improvement. For example, the number of mathematical symbols should be reduced considerably. Many readers will find it very difficult to keep track of the meaning of each of the numerous symbols introduced throughout these sections. I recommend resubmitting the manuscript after clarifying the statistical issues discussed above and restructuring the paper. I find some aspects of this study’s methods and overall approach interesting.

Reviewer_KMDmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RAMEN addresses the problem of finding small neuron-level circuits for safety classification and refusal on prompts that do not follow a fixed template. Re-scoring neurons as the circuit changes is a well-motivated design choice. The reductions in circuit size reported in Section 6 are substantial, and the ablations in Appendix F support the main design choices. The search results are more convincing than the mechanistic interpretation of the resulting circuits. The paper finds compact sets of effective intervention sites, but further validation is needed to support the accompanying conclusions about the attack mechanism and the roles of recurring neurons. The behavioral evaluation and computational comparison also leave important questions open. I recommend rejection in its current form for the following reasons.

Critical strengths

The topic is good. Re-scoring neurons as the circuit changes is a well-motivated design choice.

Critical weaknesses
  1. The node-patching results establish causal intervention effects, but do not fully resolve how the attack changes the model's computation. The paper explicitly studies node-level circuits; the concern is therefore not that the absence of edge-level discovery makes its interventions non-causal. However, the selected neurons still operate within the full network. As described in Section 3, unselected downstream neurons respond to the patched activations and may carry an essential part of the effect. When selected neurons in several layers are patched together, the later neurons are assigned their attacked values directly; those values do not have to be produced by the selected upstream neurons in that run. This leaves a gap between showing where to intervene and explaining how the original attack works. A neuron may be unnecessary to patch while still being important to the computation. Resolving this concern does not require mapping every edge in the model, but targeted cross-layer tests are needed. For example, do the later selected neurons recover their attack-related activity when only earlier neurons are patched and the later ones compute normally? Which unselected components are needed to transmit the effect? These tests would give the proposed mechanism a firmer basis.

  2. The interpretation of recurring neurons needs more direct evidence. Different prompts may involve different circuits, and the paper does not claim that every per-prompt circuit transfers to every attack. The stronger statement appears in Section 2, where recurring neurons are treated as "generalizable, non-redundant mechanisms." Their repeated selection is informative, but Appendix A evaluates each circuit on the same contrastive pair used to find it. This does not establish whether a fixed set retains its causal role when the suffix or request changes. Repeated retention by an approximate search also does not establish that each neuron is individually non-redundant. The shared AtP result in Appendix B is relevant, but it concerns necessity rather than the RAMEN sufficiency circuits. Likewise, the semantic-template attacks test new input strategies, not whether the same neurons continue to mediate their effects. A direct test would fix the recurring set using discovery data and then evaluate it on held-out pairs, using the corresponding activations without searching for new neurons. Testing several successful suffixes for the same request, repeating the search with different seeds, and checking individual removals would help separate shared roles from attack-specific choices and search variability.

  3. It is not clear how much the conclusions depend on the chosen search space. Section 2 gives a reasonable justification for using post-nonlinearity MLP neurons. What remains unclear is how strongly the findings depend on excluding other components and restricting the layer range. Appendix A limits the pool to layer 20 and below, and also removes Gemma's first layer because it otherwise contributes disproportionately many neurons. These choices may avoid uninformative solutions, but the paper does not show that they reliably separate computation that forms a decision from computation that merely encodes it.

  4. The computational cost is difficult to judge from the current comparison. Using the same candidate pool and thresholds makes the reported circuit sizes meaningfully comparable. However, RAMEN repeatedly re-scores neurons and tests candidate exchanges, whereas the main ranking baselines keep their orderings fixed. Appendix D gives pass counts for the scoring operations, but does not report end-to-end timings, peak-memory measurements, or circuit size as a function of the compute budget. The additional computation may be justified by the smaller circuits, but the current results do not quantify that trade-off. Appendix E does include an iterative competitor, DiscoGP, but the comparison is explicitly preliminary, limited to Mistral, and acknowledged to be sensitive to further tuning. A more thoroughly tuned comparison would make the advantage more convincing. The ablations support RAMEN's design, but they do not replace a comparison of quality at similar budgets. Reporting circuit-search costs separately from suffix construction would also help readers judge how practical the method is to use.

  5. The semantic-template experiments are interesting because the extracted features can be composed into stronger attacks. However, improving attack success with a semantic template does not establish that the original GCG suffix exploited those same semantic features. Changing the input in this way can simultaneously alter many other neurons and attention patterns. The paper itself acknowledges the polysemantic nature of these neurons and the possibility that GCG may use different meanings to drive them. It would therefore be useful to connect the semantic intervention back to the discovered neurons more directly, for example by testing whether restoring or ablating the corresponding neurons removes the semantic attack advantage.

  6. A faithfulness threshold of 0.95 is a useful continuous criterion, but it is not equivalent to reproducing the actual model decision. For the safety classifiers, the relevant behavior depends on whether the unsafe-versus-safe margin crosses the decision boundary. A circuit can recover most of the margin difference while still failing to flip the model's prediction. The same issue is even more apparent for the generative models, where the metric is based on a target-completion logit margin rather than directly checking whether free generation returns the attacked behavior. Reporting behavioral recovery in addition to the continuous metrics would make the circuit evaluation much more convincing.

  7. Using an adversarial suffix as the contrastive pair may itself complicate the interpretation. The suffix leaves the original harmful request unchanged, which is useful for constructing a practical contrastive pair. At the same time, an optimized suffix can substantially alter attention patterns and internal representations throughout the prompt. Consequently, the discovered circuit may be capturing the internal consequences of adding this particular suffix rather than the mechanism that specifically causes the safety failure. In other words, the current setup establishes which activations are sufficient to reproduce the difference between the clean and attacked runs, but not necessarily why the suffix produces the behavioral change. This distinction deserves more attention, especially when different suffixes can induce the same final behavior through different internal routes.

  8. A low completeness score does not mean that the remaining neurons are causally irrelevant. This is particularly important for a node-level circuit. Consider a simple pathway A → B → Y, where A is selected but B is not. Patching A can change B and ultimately reproduce the output effect, while patching B alone may do almost nothing. Under the paper's completeness test, B could therefore appear irrelevant even though it is an essential mediator of the effect. The current criterion establishes that the complement is insufficient on its own; it does not establish that the complement does not participate in the causal pathway. This is another reason why node-level sufficiency and pathway-level causal explanation should be distinguished more carefully

Questions for authors

Pls check the weakness.

5 comments

We thank the reviewer for their constructive feedback and provide detailed responses below

Part 1

Before addressing individual weaknesses, we wish to clarify the scope of our work.

Our primary contribution is algorithmic: we show that existing automated circuit finding tools fail to find sparse structure in complex tasks, isolate why they fail, and overcome those limitations with a well-motivated algorithmic solution. RAMEN pushes circuit discovery beyond the simple tasks that existing methods are limited to, opening an entirely new class of complex tasks to study at circuit-level and neuron-level granularity.

Our secondary contribution is that even a cursory view of the neurons our circuits consistently contain can be used to search for other vulnerabilities, which has downstream implications beyond the analysis our circuits.

Our goal is to make progress on understanding why individual attacks succeed, so we study circuits per contrastive pair. Separately optimized attacks, even on the same prompt, can exploit different pathways, so an aggregated circuit across attacks would miss the sparsity of each attack’s mechanism and the structure specific to that attack. All of our baselines are evaluated in the same per-attack setting.

The immediate output of RAMEN is not meant to produce a complete map of how an adversarial attack works. Along with the complexity of the task, node-level circuits have inherent limitations that are not specific to our work. As Reviewer KMDm notes, neurons outside a complete circuit can still be relevant, and a node-level circuit can miss important neurons that are left unpatched but are driven by upstream changes. Recovering deeper mechanisms may require combining several RAMEN runs under different layer and unit constraints even for a single attack. Tracing how the resulting components interact and figuring out the extent that mechanisms can be aggregated across attacks and prompts would require nontrivial methodology of its own that deserve separate scrutiny. RAMEN is a necessary first step toward this goal but does not aim to give all of these answers. We leave this exploration to future work.

We acknowledge that the paper could state this scope more clearly, and we will clarify it in the revision.

Part 2

W1. A neuron may be unnecessary to patch while still being important to the computation. Resolving this concern does not require mapping every edge in the model, but targeted cross-layer tests are needed. For example, do the later selected neurons recover their attack-related activity when only earlier neurons are patched and the later ones compute normally?

R1. Under faithfulness and minimality, RAMEN should not select neurons whose recomputed activations already match their attacked values, since that patch would not increase faithfulness while increasing the size of the circuit.

Nevertheless, it is meaningful to ask the question: under the patched circuit, do margin-relevant recomputed components outside of the circuit take on similar roles compared to the real attacked run? Otherwise, the circuit would be exploiting pathways that are not representative of those of the original attack.

We score neurons and attention heads outside the circuit by direct logit attribution [1], which measures a component's effect through its direct write at the output position. For a component cc, let WcW_c map its activation to the residual stream (a neuron's output weights or a head's WOW_O). Following the convention described by [2], we apply the final normalization with its scale cached from the clean run, Ncached(v)=γ⊙v/RMS(r(xclean))\mathrm{N}_\text{cached}(v) = \gamma \odot v / \mathrm{RMS}(r(x_\text{clean})), which makes it linear. So we have DLAc(S)=⟨WU[unsafe]−WU[safe], Ncached(Wc (ac,Spatch(xclean)−ac(xclean)))⟩IE(M,m,xclean,xadv)\mathrm{DLA}_c(S) = \frac{\big\langle W_U[\text{unsafe}] - W_U[\text{safe}],\ \mathrm{N}_\text{cached}\big(W_c\ (a_{c,S_\text{patch}}(x_\text{clean}) - a_c(x_\text{clean}))\big) \big\rangle}{\mathrm{IE}(M, m, x_\text{clean}, x_\text{adv})} for the fraction of the attack's effect that cc carries directly to the output when SS is patched.

Write uc=DLAc(M)u_c = \mathrm{DLA}_c(M) for how much component cc directly affects the margin under the attack and vc=DLAc(S)v_c = \mathrm{DLA}_c(S) for how much cc does after recomputed by patching circuit SS. We define the delivered share D(S)D(S) as the part of the total that patching SS reproduces, with each component counted in the direction it pushes under the attack: D(S)=1D(S) = 1 when every component pushes exactly as under the attack, and D(S)=0D(S) = 0 when none moves. The mismatch E(S)E(S) is the summed per-component error on the same scale: E(S)=0E(S) = 0 when every component is exact, and E(S)=1E(S) = 1 when the run is as far off as leaving every component at its clean value.

\begin{equation} D(S) = \frac{\sum_{c \in K} \operatorname{sign}(u_c), v_c}{\sum_{c \in K} |u_c|}, \qquad E(S) = \frac{\sum_{c \in K} |v_c - u_c|}{\sum_{c \in K} |u_c|}. \end{equation}

Table: Delivered share DD (higher is better) and mismatch EE (lower is better) of the top components outside the circuit by ∣DLAc(M)∣|\mathrm{DLA}_c(M)|, median over the 100 prompts of WildGuard.

Components KD(Θ)D(\Theta)D(OF)D(O_\text{F})D(OF+C)D(O_\text{F+C})E(Θ)E(\Theta)E(OF)E(O_\text{F})E(OF+C)E(O_\text{F+C})
Neurons, top 10100%98%98%1%5%4%
Neurons, top 100100%98%98%2%8%6%
Neurons, top 1,000100%98%98%2%10%7%
Heads, top 1099%99%100%1%5%4%
Heads, top 10098%90%94%4%17%14%

Overall, under the searched circuit, downstream margin relevant components outside the circuit contribute similarly to their attacked version. The largest gap is in the top 100 heads at 15%, which is expected since the pool can only consist of MLP neurons.

[1] https://transformer-circuits.pub/2021/framework/index.html [2] https://arxiv.org/pdf/2507.02559

W2. Different prompts may involve different circuits, and the paper does not claim that every per-prompt circuit transfers to every attack. The stronger statement appears in Section 2, where recurring neurons are treated as "generalizable, non-redundant mechanisms."

Repeated retention by an approximate search also does not establish that each neuron is individually non-redundant.

R2. We thank the reviewer for the careful reading. In Section 2, “generalizable” and “non-redundant” are referring to recurrence across independent per-prompt searches. “Generalizable” means the neuron is selected across many independently optimized attacks. “Non-redundant” means nearly every search retains that neuron, even under separate optimization pressures across different prompts. Under node-level faithfulness and granularity, neurons can be removed entirely from the set as long as other upstream neurons produce its effect or if its effect can otherwise be made up for by other selected neurons. However, we note that some neurons can frequently be selected due to their contribution to completeness which does not imply non-redundancy, so we will remove “non-redundant” in the revision.

Part 3

W3. Appendix A limits the pool to layer 20 and below, and also removes Gemma's first layer because it otherwise contributes disproportionately many neurons. These choices may avoid uninformative solutions, but the paper does not show that they reliably separate computation that forms a decision from computation that merely encodes it.

R3. An early experiment patched the attacked residual stream at the last position into the unattacked forward pass and found that faithfulness rose from 88.7, 92.0, 96.4, 98.5 from layers 18 to 21 on Mistral Wildguard, averaged over 100 GCG attacks. In other words, by layer 21, nearly all of the verdict relevant computation is stored in the last position, so studying layers 21-32 would not provide a path towards understanding how the adversarial sequence influences the earlier process of aggregating information across token positions. Of course, there may still be computation worth studying from layers 21-32 or other ranges of layers, but a layer cap of layer 20 is a reasonable starting point to demonstrate the utility of our algorithm.

We use the same constraint for the other three models we study, and in practice, this constraint does not prevent the algorithm from finding sparse circuits.

Prior work [1,2] has shown that the first MLP layer of transformers can act as an additional embedding layer. Therefore, circuits that contain too many MLP layer 0 neurons may simply be deleting token level information, which is why we decide to remove Gemma’s first layer from the pool.

[1] https://www.lesswrong.com/posts/3tqJ65kuTkBh8wrRH/fact-finding-simplifying-the-circuit-post-2

[2] https://www.neelnanda.io/mechanistic-interpretability/attribution-patching

W7. The current setup establishes which activations are sufficient to reproduce the difference between the clean and attacked runs, but not necessarily why the suffix produces the behavioral change. This distinction deserves more attention, especially when different suffixes can induce the same final behavior through different internal routes.

R7. In the standard causal framing adopted in our work and much of the mechanistic interpretability literature [1,2,3,4], a circuit is defined relative to a contrastive pair of inputs: it identifies the components through which the input difference gives rise to the observed output difference. Under this framing, explaining how the suffix induces the safety failure amounts to identifying the causal components and pathways through which the suffix changes the model's behavior. Our work focuses on building a method to identify these components while explicitly leaving the pathways and how components interact to be studied by future work.

We acknowledge that this does not provide a complete algorithmic account of the underlying computation. However, this is not unique to our setting: it applies for any circuit level explanation beyond small toy models whose underlying algorithms have been fully reverse engineered, such as one layer modular addition [5].

[1] https://arxiv.org/pdf/2211.00593

[2] https://arxiv.org/pdf/2202.05262

[3] https://arxiv.org/pdf/2403.19647

[4] https://arxiv.org/pdf/2601.22594

[5] https://arxiv.org/pdf/2301.05217

Reviewer_8vgqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces RAMEN, an algorithm that scales circuit-level interpretability to real-world safety tasks. By iteratively refining circuits through causal interventions, RAMEN overcomes the limitations of gradient-based methods and identifies compact circuits underlying adversarial attacks. RAMEN reveals exploitable non-robust features that can bypass state-of-the-art safety defenses like WildGuard and Circuit Breakers.

Critical strengths
  • The authors provide a clear summary and analysis of the limitations in existing methods within Section 4, which effectively motivates the design principles of RAMEN.
  • The approach of leveraging the discovered circuits to construct adversarial suffixes is interesting.
Critical weaknesses
  • The manuscript is difficult to follow, and many parts lack clear logical connections to the surrounding content.
  • I recommend adding a Background section to introduce the preliminary concepts of circuits. The content of Sections 4 and 8 could be condensed and integrated into this new section.
  • It is unclear how this work relates to security. The core Sections 3-5 do not explicitly link circuit discovery with adversarial attacks, making the combination of these two fields feel somewhat disjointed.
  • If the paper's primary focus is on addressing a security problem, I recommend adding a Threat Model section to clarify the target scenarios and the knowledge and capabilities available to both attackers and defenders.
  • Could the authors provide an anonymized link to the source code to facilitate reproducibility?
  • Section 4 introduces several existing methods, but the evaluation includes only one baseline, as shown in Table 1. The experimental results presented in the main text are also quite limited.
  • The references largely lack proper publication venue details, often containing only authors, titles, and URLs.
Questions for authors

Refer to the weaknesses mentioned above.

2 comments

We thank the reviewer for their feedback on presentation and we will incorporate it in the next revision.

Section 4 introduces several existing methods, but the evaluation includes only one baseline, as shown in Table 1.

As for technical soundness, we included comparisons to other baseline methods in Figure 1 as well as Appendix E. We have additionally included the comparison to DiscoGP under our response R4 to Reviewer KMDm. These results demonstrate that RAMEN finds substantially smaller circuits compared to baselines while satisfying our faithfulness and completeness criteria.

We provide our anonymized code at https://anonymous.4open.science/r/circuit_search-EAE0/.

Thanks for the author's response; however, the current content is not enough to convince me, so I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.