acceptodds
Under review as a conference paper at ICLR 2027

ProbeFool: Evading LLM Jailbreak Activation Monitors with Probe-Agnostic Attention Routing

Abstract

Activation probes are increasingly proposed as lightweight monitors for harmful language-model requests. We ask whether attention at the monitored token is a usable proxy for the probe's own signal: whether the representation a monitor reads can be moved without ever optimizing against it. We introduce , a continuous-embedding attack that places a fixed benign distractor before an unchanged jailbreak prompt and optimizes a trainable steerer after it from frozen base-model signals alone, raising the monitored token's attention to the distractor while preserving the original response distribution. The probe is a decision-only black box: its decision would tell a deployed attacker when to stop, but it never enters the update. Against logistic-regression and MLP probes on Llama-3.2-3B-Instruct, Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.2, we conservatively report terminal-step results averaged over every transformer-layer probe. With the best of three distractor–steerer initializations selected on each model–probe track by joint success, the attack achieves 77–93% evasion with 52–77% joint success, meaning evasion and a still-harmful response on the same prompt, and it transfers across broad layer ranges and to unseen prompts. Controlled interventions locate the mechanism. Transplanting the attack's monitored-token attention onto an unoptimized wrapper recovers most of its evasion, while the optimized embedding under clean attention evades almost nothing. At fixed mass, which heads carry it decides the outcome.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Attacks, Mechanistic Interpretability, Jailbreaking, LLMs, Oversight, Monitoring

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mne3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes PROBEFOOL, an attack against activation-based jailbreak monitors. The attacker has white box access to the underlying model but not to the monitor, and optimizes a continuous embedding wrapper to preserve the original harmful response while increasing attention from the monitored token to a benign distractor. The attack achieves fairly high probe evasion on three models and two probe types, although joint success is considerably lower. The paper also includes extensive intervention experiments aimed at showing that attention routing is causally responsible for the observed probe evasion.

Critical strengths
  • I found the mechanistic analysis to be the strongest part of the paper. In particular, the attention transplantation experiments make a reasonably convincing case that the change in attention routing is what drives probe evasion, rather than simply the optimized embedding moving the representation in some arbitrary way.

  • I also liked the experiment showing that, at fixed total attention mass, which heads carry the mass makes a very large difference. To me, this is a more interesting finding than the attack performance itself and gives some useful insight into how fragile these activation monitors can be.

Critical weaknesses
  • My main concern is the threat model. The attacker has white-box access to the base model, can backpropagate through its attention patterns, knows the position read by the monitor, and can directly optimize continuous embeddings. At the same time, only the monitor is inaccessible. I am not sure what realistic deployment has this particular combination of capabilities. With this much access to the underlying model, I also do not find it very surprising that one can substantially alter the representation seen by a simple probe.

  • This concern becomes stronger when looking at the discrete-token results. The continuous attack has reasonable evasion, but the practical token-space version does not achieve meaningful joint success: the strongest evading run loses the harmful response entirely, and the paper reports only 10.7% best joint success across these experiments. Thus, I currently see this more as a mechanistic stress test / upper bound than a realistic attack.

  • My second concern is novelty and effectiveness. Attention-based objectives for jailbreak optimization already exist, including AttnGCG and Attention Eclipse. The new aspect here seems to be using attention from the monitor's readout token toward benign content as a proxy for the unknown probe signal, rather than the attention objective itself. I think this distinction should be made more clearly.

  • Finally, the joint success rates are not especially strong given the amount of attacker access. The headline evasion rates look much stronger than the actual probability of both evading the monitor and retaining the harmful behavior, particularly on held-out prompts.

Questions for authors
  • What concrete deployment scenario do the authors envision where an attacker has full model gradients and continuous-embedding access, but the activation monitor remains the relevant hidden security boundary?

  • How does the attack perform if the defender selects a robust monitor layer rather than evaluating by averaging uniformly across all layers? Relatedly, what happens for monitors that aggregate information over several token positions rather than reading one known token?

  • Do the authors believe the discrete-token results are sufficient to support a practical attack claim? If not, I think the paper would benefit from more clearly framing the continuous embedding results as a mechanistic or upper bound study.

2 comments

Thank you. We agree that the mechanistic contribution and practical attack scope should be distinguished more clearly.

Questions 1 and 3: threat model and interpretation

For example, a cloud platform can offer API/chat access to an open-weight model while adding a private harmfulness probe as a guardrail. An attacker can compute gradients on a local copy of the public backbone while the hosted monitor remains private. Our continuous study additionally assumes embedding-level input control; API/chat access alone does not provide that capability. The readout position is also assumed known and lies outside the writable span.

Continuous embedding access relaxes the vocabulary constraint, so we regard it as an upper-bound setting for studying the evasion mechanism. Better token-space optimizers may eventually recover more of this effect, although matching the continuous results remains an open question.

Our current discrete results do not establish a practical token-space bypass. In the small 3B study (five prompts), the strongest evading run reaches 64.3% ER but 0% response success; the best JSR across the tested runs is 10.7%. We agree that the paper would benefit from more emphasis on its upper-bound scope and mechanistic contribution, and will make this framing explicit in the abstract and introduction.

Question 2: robust layers and multiple positions

We agree that layer placement can provide protection. Our uniform-layer averages summarize performance across placements; they do not guarantee success against a defender-selected robust layer.

Clean probe validation and adversarial robustness are different criteria. For example, selecting 8B/LR's highest-validation-AUROC layer picks readout 17, with validation AUROC 1.0. Nevertheless, all three original pairs achieve 100% held-out ER there (46–60% JSR; 50 prompts per pair). Clean validation performance alone therefore does not establish resistance to this attack.

Resistance also depends on initialization. On held-out 3B/MLP prompts, readout 2 has 0% ER for pair 3 but 100% for pair 2; readout 6 has 0% ER for pair 2 but 64% for pair 3. Some resistant readouts are shared across pairs, however: 8B readouts 1 and 4 resist all three. Changing pairs may help, but is not guaranteed to bypass a robust layer choice.

Additionally, we trained LR/MLP probes on the mean states of the last 1/2/5 input tokens and extended the attack objective over the corresponding queries. Held-out JSR (%), averaged over layers, with matching attack/defender widths: 50 prompts per Llama, 255 for Mistral. We use the paper's in-sample-selected pairs, fixed across widths: 8B pair 1, Mistral pair 3, 3B LR pair 2 and MLP pair 3; no averaging over pairs.

Model/probe1 token2 tokens5 tokens
8B/LR45.6245.9437.44
8B/MLP46.3142.0034.12
Mistral/LR68.1668.2751.96
Mistral/MLP71.1070.8862.13
3B/LR47.0036.7135.79
3B/MLP42.0049.7950.36

One-position attacks use 500 updates; two/five positions add 1,000 updates to that checkpoint (1,500 total). On 8B pair 1, held-out SR is 60/60/50% at widths 1/2/5. The Llamas' last five input positions are outside the optimized span; Mistral's five-token window includes one steerer token. These are known input-position monitors. Pooling remains vulnerable in these tests, while the robust-layer results limit any claim of uniform success across defenses.

Novelty and joint success

We agree that attention-guided jailbreak optimization predates this work (AttnGCG, Attention Eclipse). Our contribution is using attention from the monitor's readout toward benign content as a proxy for an inaccessible probe, while preserving an already harmful response. The transplantation and fixed-mass head-allocation experiments test why that proxy works; the novelty claim is not attention manipulation itself.

These results also show why ER should not stand in for joint success: response preservation is a substantial limitation, especially on held-out prompts. Our added Bailey comparison also finds stronger surrogate attacks on 8B/Mistral, with conditional advantages for ProbeFool in some settings. We therefore frame the contribution around a monitor-independent objective and its causal analysis, without claiming general attack superiority or a demonstrated practical text-only bypass.

Thank you for the detailed response and for the added experiments.

I appreciate that you accept the main concerns: the continuous attack is an upper bound, the discrete token results do not establish a practical bypass (best joint success 10.7%), and the novelty lies in the readout attention proxy rather than attention manipulation itself. However, the new results show that robustness depends on initialization and that stronger surrogate attacks exist on 8B and Mistral. My assessment is therefore unchanged. I do consider the mechanistic analysis the paper’s strongest contribution.

Reviewer_BcG2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProbeFool, an attention-routing attack against activation-based LLM safety monitors. By optimizing a continuous embedding wrapper using only base-model signals, the attack evades safety probes without accessing their parameters or gradients, while preserving harmful responses. The paper further investigates the underlying mechanism through attention transplantation and head-level interventions.

Critical strengths
  1. The paper addresses an interesting problem: evading activation-based safety monitors without direct access to the monitors.

  2. The proposed attack is simple and intuitive, combining attention routing with response preservation. And the extensive ablation and causal intervention experiments provide useful insights into how attention routing affects safety monitoring.

Critical weaknesses
  1. Restrictive assumptions about the safety monitor. Although the attack does not require access to the probe's parameters or gradients, it assumes knowledge of the monitored token position. Its effectiveness against monitors with unknown readout positions, multiple-token readouts, or pooled representations remains unclear, potentially limiting the generality of the Probe-Agnostic claim.

  2. Limited effectiveness against alternative defenses. The paper shows that a simple Mahalanobis-based OOD detector successfully identifies all evaluated attacks. This raises concerns about whether the proposed method reveals a general vulnerability in activation-based safety monitoring or primarily exploits the decision boundaries of specific discriminative probes.

  3. Insufficient comparison with prior attacks. Although the paper identifies Obfuscated Activations as the closest related work, it lacks a direct empirical comparison under a unified evaluation setup. Such a comparison would help quantify the trade-off between reduced monitor access and attack effectiveness.

Questions for authors
  1. Can the authors provide a direct comparison with Obfuscated Activations under the same experimental setup to better demonstrate the advantages and trade-offs of the proposed probe-agnostic approach?

  2. How would ProbeFool perform when the monitored token position is unknown or when the monitor uses multiple token positions or pooled representations?

  3. Given the substantial performance variation across distractor–steerer initializations, how can an attacker efficiently identify effective initializations without extensive trial and error?

  4. Would ProbeFool remain effective against adversarially trained probes or monitors that aggregate hidden states across multiple layers?

1 comment

Thank you. We evaluated Bailey et al. under our protocol and tested monitors that pool across input positions or layers.

1. Comparison with Bailey et al.

We ran Bailey's released loss and optimizer on the same backbones and cohorts, with greedy 512-token generation and StrongREJECT. A is the defender and B is a separate grouped-bootstrap fit on the same training pool. B→A measures surrogate transfer. A→A uses the defender directly.

Held-out JSR (%), averaged over transformer-layer probes, for single-position attacks at 500 updates. Batch sizes and compute differ. Both tables use 50 held-out prompts per Llama and 255 for Mistral. ProbeFool (PF) uses the listed pairs, chosen by in-sample JSR in the paper and fixed across comparisons.

Model/probePF pairPFBailey B→ABailey A→A
8B/LR145.6257.8862.00
8B/MLP146.3152.0068.31
Mistral/LR368.1678.3378.04
Mistral/MLP371.1081.9678.04
3B/LR247.0029.9340.00
3B/MLP342.0026.0038.00

Bailey leads on 8B and Mistral, while PF leads on 3B in this comparison. The ordering can change with the surrogate's training data. On Mistral/MLP, Bailey gets 49.13% held-out JSR against the full-data defender using a surrogate trained without jailbreak examples, versus 71.10% for PF pair 3 (paired 95% bootstrap interval for the gap of 15.82–27.91 points).

On 8B, if Bailey uses an MLP surrogate trained with DeepInception and GPTFuzzer as its only jailbreak sources. Against the MLP defender trained on all five methods, held-out JSR is 44.81% versus 46.31% for PF pair 1. Against an MLP defender trained with DeepInception and ReNeLLM, the same Bailey suffix reaches 50.25% versus PF's 56.31%. Bailey reaches 68.00% with direct access to that defender. PF's optimization uses the frozen backbone alone.

2. Multiple input positions

We trained LR/MLP probes on the mean hidden state of the last two or five input tokens. Across the six tracks, the unchanged single-position wrappers give 17.69–41.18% held-out JSR against mean-2 probes and 2.33–18.71% against mean-5 probes. We then continued optimization for 1,000 updates from the original 500-update checkpoints, with the objective covering the same two or five positions that the probe pools.

The following entries are layer-averaged evasion rate (ER) / JSR (%).

Model/probeMean-2, matchedMean-5, matched
8B/LR75.25 / 45.9476.19 / 37.44
8B/MLP69.81 / 42.0068.88 / 34.12
Mistral/LR89.78 / 68.2773.30 / 51.96
Mistral/MLP93.74 / 70.8888.04 / 62.13
3B/LR66.21 / 36.7175.43 / 35.79
3B/MLP88.57 / 49.7989.14 / 50.36

These experiments assume a known input pooling window. We will clarify that “probe-agnostic” refers to the optimization's independence from probe weights and scores. Evasion with a hidden readout position remains open.

3. Initialization

On 8B, the 49 optimization prompts rank the 20 pairs consistently by LR evasion (Kendall's W=0.855). This motivates screening on fewer prompts, although these rankings were measured after optimization.

Random ordering without replacement gives these in-sample, layer-averaged JSR results:

JSR targetLR pairsMean triesMLP pairsMean tries
≥40%171.17141.40
≥50%63.0053.50

Pair counts are out of 20. Each try is a full 500-update run, including the first. Held-out selection efficiency remains open.

4. Multi-layer monitors and OOD detection

We tested two monitors trained on unmodified inputs and calibrated to a nominal 1% benign validation FPR. One averages probe scores before thresholding. The other fits LR/MLP on states concatenated from four evenly spaced depths. With the original step-500 wrappers and the same held-out cohorts and pairs as above, mean-score aggregation gives 80–100% ER and 44.00–75.69% JSR. Its results vary across initializations. Concatenation gives 100% ER on all six paper-selected tracks and 50.00–75.69% JSR, with 99.35–99.92% clean test harmful recall and 0.66–1.33% clean test FPR.

Adversarial training can improve robustness. Without probe access, PF cannot optimize the adapted defender's loss directly. Regularization or targeting different distractor tokens (n, n−1, n−2, …) might diversify its attention patterns. Whether this evades adapted probes, or training on these variants raises false positives on benign inputs, remains untested.

We acknowledge that the benign-only Mahalanobis detector catches all original PF wrappers at every evaluated layer across all three models, an effective defense in our tests. As discussed in the paper, harmless inputs from new domains may also lie far from the fitted benign distribution and be rejected. The benign acceptance was measured in distribution; false positives under benign distribution shift remain untested here.

Reviewer_z641OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a continuous embedding attack against a set of harmfulness probes that are allowed to look at only one token position. They do this by optimizing an embedding suffix to route attention away from harmful tokens in the prompt to a distractor prefix, while retaining the final output distribution.

They also show that certain heads matter more than others by patching them individually.

Critical strengths
  • The paper proposes a probe obfuscation attack without requiring the need to get a signal from the probe itself
  • The paper shows that, just by tuning an embedding suffix, you can control how much attention is given to what part of the prompt to some extent.
Critical weaknesses
  • The paper does not consider probes that pool signals across token positions. Removing direct attention to the prompt at multiple generation positions should directly harm performance significantly, and this makes me much more skeptical of this method.
  • Bailey et al. already demonstrate transfer between monitors, including from an OOD detector to input-based probes. An attacker with white-box access to the model could therefore optimize against an independently trained surrogate model and transfer the attack without accessing the deployed probe’s scores or gradients. The paper does not compare against this baseline. Its results establish that attention is a usable proxy, but leave unclear how much it improves upon existing routes to evading an inaccessible monitor.
Questions for authors
  • Maybe the authors can try probes pooled across multiple token positions by selecting a particular layer or a subset of layers? Or alternatively, if the authors can show such obfuscation is possible at all layers and multiple generation tokens, without affecting the performance, that would change my score to strong accept.
  • The authors present a confusing result under Section 6.1 that claims, if you remove all attention to tokens after the distractor at the final input token position, the evasion rate is less wrt if you remove less than all attention. But mathematically, there should be no information regarding harmfulness at this token position? This signal should either be noise, or the harmful signal is leaking at the token position somehow?
1 comment

Thank you. We address the three concerns below.

1. Multiple input positions

We trained LR/MLP probes on mean states of the last 1/2/5 input tokens and extended our attention objective over those queries.

Held-out ER/JSR (%), matching attack/defender widths and averaged over layers: 50 prompts per Llama, 255 for Mistral. Each row uses the listed pair, selected by in-sample JSR in the paper and fixed across widths; no averaging over pairs.

Model/probePair1 token2 tokens5 tokens
8B/LR175.94/45.6275.25/45.9476.19/37.44
8B/MLP177.56/46.3169.81/42.0068.88/34.12
Mistral/LR389.09/68.1689.78/68.2773.30/51.96
Mistral/MLP393.11/71.1093.74/70.8888.04/62.13
3B/LR283.50/47.0066.21/36.7175.43/35.79
3B/MLP383.71/42.0088.57/49.7989.14/50.36

The one-position attack has 500 updates; two/five positions continue for 1,000 more (1,500 total). On 8B pair 1, held-out response success (SR) is 60% for one/two positions and 50% for five.

Generated-token monitors remain out of scope: an attack could instead target benign content at selected monitored output positions while preserving a harmful continuation, requiring a different objective.

2. Section 6.1: full distractor attention

At ρ=1\rho = 1, the query reads only distractor values, but its state need not resemble a normal benign example. A harmful probe prediction does not establish dependence on harmful content.

The LR/OOD sweeps use all 30 unoptimized pairs and in-sample prompts (3B/8B/Mistral: 50/49/50). Increasing ρ\rho can first move the state toward LR's benign side, then reverse that projection. OOD scores increase, but OOD magnitude alone does not explain the downturn.

We scored intermediate operations using both the paper's probes and operation-specific probes. Acceptance was often much lower after attention/MLP residual addition, especially on 3B. These descriptive comparisons motivated residual swaps.

The 3B pilot uses two in-sample prompts crossed with sweep pairs 0/10/20 (short/long codeword and task-anchor). Donor vectors come from runs with ρ=0.6\rho = 0.6 or 1 at all blocks. At LR readout 23, full routing gives 37.8% mean benign probability. Replacing only the skip-connection vector at block 23's attention residual addition with its moderate-ρ\rho donor raises this to 57.7%; the attention computation and output remain from full routing. Conversely, keeping the full-routing residual and replacing only the attention output gives 44.7%. Both swaps recompute the MLP from the mixed sum and use the same destination LR probe.

Excess routing can therefore move the accumulated state toward the probe's harmful region, depending on model and depth. Our per-layer attention objective lacks an explicit constraint on residual accumulation, which may contribute. This remains a hypothesis, compatible with the fixed-mass head-allocation result.

3. Bailey et al.

We agree that an attacker can transfer from a surrogate. We ran Bailey's released loss/optimizer on our data/backbones, averaging per-layer scores, with greedy 512-token generation and StrongREJECT.

Held-out JSR (%), single-position attacks at 500 updates, averaged over layers. Cohorts and PF pairs are as above. A is the defender; B is a separate grouped-bootstrap fit on the same training pool. B→A is surrogate transfer; A→A uses the actual defender. Bailey has one suffix per model/source probe.

Model/probePF pairPFBailey B→ABailey A→A
8B/LR145.6257.8862.00
8B/MLP146.3152.0068.31
Mistral/LR368.1678.3378.04
Mistral/MLP371.1081.9678.04
3B/LR247.0029.9340.00
3B/MLP342.0026.0038.00

Bailey leads on 8B/Mistral; PF leads this 3B comparison through better response preservation despite lower evasion.

Against Mistral's full-data MLP defender, a Bailey surrogate trained without jailbreak examples gives 49.13% held-out JSR versus 71.10% for PF pair 3: a 21.97-point difference (paired request-group 95% interval [15.82, 27.91]).

The two 8B OOD sources achieve 98.4–98.6% held-out ER against LR/MLP, but 4–12% SR limits JSR to 3.94–12.00%, versus PF pair 1's 45.62–46.31%. Evasion transfers well; utility limits joint success. The tested OOD defenders still detect the original 8B PF attacks.

PF's advantage is conditional: strong surrogates often win. PF avoids fitting or optimizing against a surrogate and assuming its training distribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.