- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes PROBEFOOL, an attack against activation-based jailbreak monitors. The attacker has white box access to the underlying model but not to the monitor, and optimizes a continuous embedding wrapper to preserve the original harmful response while increasing attention from the monitored token to a benign distractor. The attack achieves fairly high probe evasion on three models and two probe types, although joint success is considerably lower. The paper also includes extensive intervention experiments aimed at showing that attention routing is causally responsible for the observed probe evasion.
Critical strengths
-
I found the mechanistic analysis to be the strongest part of the paper. In particular, the attention transplantation experiments make a reasonably convincing case that the change in attention routing is what drives probe evasion, rather than simply the optimized embedding moving the representation in some arbitrary way.
-
I also liked the experiment showing that, at fixed total attention mass, which heads carry the mass makes a very large difference. To me, this is a more interesting finding than the attack performance itself and gives some useful insight into how fragile these activation monitors can be.
Critical weaknesses
-
My main concern is the threat model. The attacker has white-box access to the base model, can backpropagate through its attention patterns, knows the position read by the monitor, and can directly optimize continuous embeddings. At the same time, only the monitor is inaccessible. I am not sure what realistic deployment has this particular combination of capabilities. With this much access to the underlying model, I also do not find it very surprising that one can substantially alter the representation seen by a simple probe.
-
This concern becomes stronger when looking at the discrete-token results. The continuous attack has reasonable evasion, but the practical token-space version does not achieve meaningful joint success: the strongest evading run loses the harmful response entirely, and the paper reports only 10.7% best joint success across these experiments. Thus, I currently see this more as a mechanistic stress test / upper bound than a realistic attack.
-
My second concern is novelty and effectiveness. Attention-based objectives for jailbreak optimization already exist, including AttnGCG and Attention Eclipse. The new aspect here seems to be using attention from the monitor's readout token toward benign content as a proxy for the unknown probe signal, rather than the attention objective itself. I think this distinction should be made more clearly.
-
Finally, the joint success rates are not especially strong given the amount of attacker access. The headline evasion rates look much stronger than the actual probability of both evading the monitor and retaining the harmful behavior, particularly on held-out prompts.
Questions for authors
-
What concrete deployment scenario do the authors envision where an attacker has full model gradients and continuous-embedding access, but the activation monitor remains the relevant hidden security boundary?
-
How does the attack perform if the defender selects a robust monitor layer rather than evaluating by averaging uniformly across all layers? Relatedly, what happens for monitors that aggregate information over several token positions rather than reading one known token?
-
Do the authors believe the discrete-token results are sufficient to support a practical attack claim? If not, I think the paper would benefit from more clearly framing the continuous embedding results as a mechanistic or upper bound study.
2 comments
Thank you for the detailed response and for the added experiments.
I appreciate that you accept the main concerns: the continuous attack is an upper bound, the discrete token results do not establish a practical bypass (best joint success 10.7%), and the novelty lies in the readout attention proxy rather than attention manipulation itself. However, the new results show that robustness depends on initialization and that stronger surrogate attacks exist on 8B and Mistral. My assessment is therefore unchanged. I do consider the mechanistic analysis the paper’s strongest contribution.
Thank you. We agree that the mechanistic contribution and practical attack scope should be distinguished more clearly.
Questions 1 and 3: threat model and interpretation
For example, a cloud platform can offer API/chat access to an open-weight model while adding a private harmfulness probe as a guardrail. An attacker can compute gradients on a local copy of the public backbone while the hosted monitor remains private. Our continuous study additionally assumes embedding-level input control; API/chat access alone does not provide that capability. The readout position is also assumed known and lies outside the writable span.
Continuous embedding access relaxes the vocabulary constraint, so we regard it as an upper-bound setting for studying the evasion mechanism. Better token-space optimizers may eventually recover more of this effect, although matching the continuous results remains an open question.
Our current discrete results do not establish a practical token-space bypass. In the small 3B study (five prompts), the strongest evading run reaches 64.3% ER but 0% response success; the best JSR across the tested runs is 10.7%. We agree that the paper would benefit from more emphasis on its upper-bound scope and mechanistic contribution, and will make this framing explicit in the abstract and introduction.
Question 2: robust layers and multiple positions
We agree that layer placement can provide protection. Our uniform-layer averages summarize performance across placements; they do not guarantee success against a defender-selected robust layer.
Clean probe validation and adversarial robustness are different criteria. For example, selecting 8B/LR's highest-validation-AUROC layer picks readout 17, with validation AUROC 1.0. Nevertheless, all three original pairs achieve 100% held-out ER there (46–60% JSR; 50 prompts per pair). Clean validation performance alone therefore does not establish resistance to this attack.
Resistance also depends on initialization. On held-out 3B/MLP prompts, readout 2 has 0% ER for pair 3 but 100% for pair 2; readout 6 has 0% ER for pair 2 but 64% for pair 3. Some resistant readouts are shared across pairs, however: 8B readouts 1 and 4 resist all three. Changing pairs may help, but is not guaranteed to bypass a robust layer choice.
Additionally, we trained LR/MLP probes on the mean states of the last 1/2/5 input tokens and extended the attack objective over the corresponding queries. Held-out JSR (%), averaged over layers, with matching attack/defender widths: 50 prompts per Llama, 255 for Mistral. We use the paper's in-sample-selected pairs, fixed across widths: 8B pair 1, Mistral pair 3, 3B LR pair 2 and MLP pair 3; no averaging over pairs.
One-position attacks use 500 updates; two/five positions add 1,000 updates to that checkpoint (1,500 total). On 8B pair 1, held-out SR is 60/60/50% at widths 1/2/5. The Llamas' last five input positions are outside the optimized span; Mistral's five-token window includes one steerer token. These are known input-position monitors. Pooling remains vulnerable in these tests, while the robust-layer results limit any claim of uniform success across defenses.
Novelty and joint success
We agree that attention-guided jailbreak optimization predates this work (AttnGCG, Attention Eclipse). Our contribution is using attention from the monitor's readout toward benign content as a proxy for an inaccessible probe, while preserving an already harmful response. The transplantation and fixed-mass head-allocation experiments test why that proxy works; the novelty claim is not attention manipulation itself.
These results also show why ER should not stand in for joint success: response preservation is a substantial limitation, especially on held-out prompts. Our added Bailey comparison also finds stronger surrogate attacks on 8B/Mistral, with conditional advantages for ProbeFool in some settings. We therefore frame the contribution around a monitor-independent objective and its causal analysis, without claiming general attack superiority or a demonstrated practical text-only bypass.