acceptodds
Under review as a conference paper at ICLR 2027

Do Jailbreaks Leave Identifiable Behavioral Footprints? Detection and Localization through Attention Occupancy

Abstract

Jailbreak detectors typically render a single prompt-level decision, failing to pinpoint where adversarial manipulation occurs. We investigate whether a model's attention during early decoding can simultaneously support jailbreak detection and attack span localization without training auxiliary classifiers or updating model weights. Our approach isolates user-directed attention, filters attention sinks, and incorporates attention entropy to dynamically weight decoding queries according to their effective support size. Within individual heads, token profiles are robustly standardized using the median and median absolute deviation (MAD). A small set of task-relevant heads selected on development prompts provides token-level anomaly scores that support prompt-level detection and adversarial span localization. Operating on a frozen target model, our method separates the use of prompt labels for head selection from threshold calibration, which is performed on an independent non-attack split. Evaluated against GCG and HarmBench AutoDAN attacks across three target models, our detector identifies 87.00%–98.45% of attack attempts while flagging at most one of 17 benign references per target. It also yields the lowest judged attack success rate among the compared defenses and localizes annotated adversarial content in 86.50%–97.94% of evaluated attacks. These results show that early-decoding attention provides a useful, non-causal positional signal for jointly detecting and localizing jailbreak attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Jailbreak Attacks, Large Language Models, Behavioral Footprints, Attention Occupancy, Adversarial Prompt Localization, LLM Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hqzmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the footprints of jailbreaking attacks by examining LLM early-decoding attention patterns. The proposed method constructs token-level anomaly scores from entropy-weighted attention profiles and robust within-prompt normalisation, enabling both prompt-level jailbreak detection and adversarial-span localisation without training an auxiliary classifier or updating the target model. Experiments on Llama-2 and Vicuna with GCG and AutoDAN attacks show high attack-detection recall and SpanHit rates, as well as reduced attack success rates when the detector is used as a gating mechanism.

Critical strengths
  1. The motivation and method are clear, and the visualization is helpful.

  2. The authors provide sufficient evidenc to support the effectiveness of their proposed method.


  3. This paper is well-written, making it easy to follow.

Critical weaknesses
  1. Insufficient evaluation of false positives and localisation accuracy. The use of only 17 benign controls is insufficient to reliably assess false positives, particularly for challenging safety-related but benign requests (e.g., harm-reduction or help-seeking queries, such as asking for assistance with drug cessation). In addition, the SpanHit metric is overly permissive, as any overlap with the annotated attack span counts as successful localisation, without evaluating precision or boundary accuracy.

  2. Limited evaluation across harmful inputs and attack types. The evaluation focuses on GCG and AutoDAN, but the detector should also be systematically evaluated on plain harmful instructions (e.g., HarmBench or AdvBench) to clarify whether it detects harmfulness beyond adversarial modifications. Its robustness to qualitatively different attacks, such as PAIR and multilingual/translation-based jailbreaks, also remains unclear.

  3. Limited cross-model transferability. Jailbreak behavior can be model-specific: the same prompt may successfully jailbreak one model but not another. It remains unclear whether attention footprints detected on one model (e.g., Llama) transfer to other models such as Qwen or API-only models. Cross-model evaluation on the same prompts would help establish whether the proposed signal is model-independent or model-specific.

  4. Limited baselines and lack of frontier-model comparison. Strong contemporary safety models such as WildGuard, LlamaGuard, ShieldGemma, Qwen3Guard, or Shieldstral are not evaluated [1-5]. Moreover, zero-shot evaluation with strong frontier general-purpose LLMs (e.g., GPT-6 Astra) would also provide an important reference.

  5. Generalisation to modern and larger models remains unclear. The experiments are restricted to Llama-2-7B/13B and Vicuna-13B, while contemporary models can differ substantially in architecture, alignment, and attention behaviour. Evaluation on stronger and larger models (e.g., recent Qwen 3.5 27B or MOE gpt-oss-20b) would be important, particularly because head selection is model-dependent. Moreover, the proposed approach requires white-box access to internal attention, limiting direct applicability to API-only models.

[1] Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms
[2] Llama guard: Llm-based input-output safeguard for human-ai conversations
[3] Shieldgemma: Generative ai content moderation based on gemma
[4] Qwen3guard technical report
[5] Shieldstral

Questions for authors

Refer to weaknesses.

1 comment

We thank the reviewer for the detailed review. We ran the requested experiments; all numbers below use frozen detectors (no re-tuning on the new data).

W1: Benign controls and localization

We agree that 17 benign controls are insufficient. We added roughly 600–820 benign prompts per target: XSTest-safe, OR-Bench-Hard-1K (safety-related but benign), and Alpaca and Dolly prompts length-matched to GCG and AutoDAN.

The pooled benign false-positive rate is 45.0% on Llama-2-7B, 25.2% on Llama-2-13B and 2.0% on Vicuna-13B, against 0–1/17 in the paper.

W2: Plain harmful inputs and other attacks

The detector rarely fires on plain harmful requests: 0.5–3.8% on AdvBench and 0–4.7% on HarmBench direct requests across six models. So it does not detect harmfulness.

However, it also flags 15–35% of prompts carrying random, non-optimized suffixes. Detection of PAIR, TAP, persuasion and MultiJail attacks varies widely across models, from 0% to 99%. The signal therefore reflects unusual token spans rather than adversarial intent.

W3: Cross-model

On roughly 900–1,000 identical prompts evaluated on six models, detection decisions agree at chance level (Cohen’s κ between −0.10 and 0.12). The footprint is model-specific, so heads and thresholds must be selected per model, and white-box access is required.

W4: Baselines

On the same prompts (Llama-2-7B target), harmfulness guards cannot separate attacks from plain harmful requests:

MethodAUROC, attack vs plain harmful
Ours0.872
Llama-Guard-3-8B0.278
Qwen3Guard-Gen-8B0.205
Granite-Guardian-3.3-8B (jailbreak)0.257

W5: Modern models

On Llama-3-8B and Qwen2.5-7B/14B the results are model-dependent:

  • Llama-3-8B: Benign false-positive rate 36.4%.
  • Qwen2.5-14B: 0.3% benign false positives and AUROC 0.93–0.96 on suffix attacks. With a threshold recalibrated on held-out negatives, it detects 61% of GCG attacks at a 2.0% benign false-positive rate.

We will report these results together with the length limitation.

Reviewer_pZqGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors investigate whether jailbreak attacks leave detectable patterns in the attention distribution and help localize the attack span during generation tasks in LLMs. The method proposes an attention-based white-box approach to jointly detect jailbreaks and localize adversarial manipulation. The method does not require additional model weight updates or training an auxiliary classifier. The proposed approach extracts attention distributions from early decoding steps, filters attention sinks, and utilizes attention entropy to weight decoding queries based on their effective attention support. Then it computes anomaly scores token-level) using the median and MAD. A subset of attention heads is selected using the validation set, and their scores are aggregated to perform detection and localization. Detection thresholds are calibrated separately using independent non-attack samples. Evaluation on three language models (Llama-2-7B-chat, Llama-2-13B-chat and Vicuna-13B-v1.5) and GCG and HarmBench AutoDAN attacks across three target models shows that the proposed detector identifies 87.00%--98.45% of attack attempts. It also achieves lower judged attack success rates than the evaluated CPD and RobustKV baselines.

Critical strengths
  1. The paper addresses an important problem of detecting jailbreak attacks and localizing the adversarial part, rather than only classifying jailbreak prompts, which is a very relevant research problem.

  2. The approach does not require additional model training, which makes the method inexpensive and easy to deploy.

  3. The method achieves high detection results across three language models and two jailbreak attack families.

  4. The experimental design is clean. The authors distinguish jailbreak attempts and successful jailbreaks and use separate data splits for head selection, threshold calibration, and evaluation

Critical weaknesses
  1. The novelty of the proposed method is limited. Several related ideas have already been explored such as, attention-sink masking (Xiao et al. 2024), robust normalization of entropy signals and adversarial suffix localization (CPD, Alshaalan & Rodrigues, 2026), attention-head analysis for prompt-injection detection (Attention Tracker, Hung et al., 2025), and token-level jailbreak localization (Token Highlighter, Hu et al., 2025). The contribution appears to lie mainly in combining these existing techniques. This paper does not sufficiently establish the novelty or advantages of the proposed combination

  2. Unusual attention patterns may occur due to various underlying reasons, not necessarily caused by adversarial modifications. It may be due to semantic content or prompt length or structure. The paper does not sufficiently establish the evidence. The authors acknowledge in Section 6.2 that harmful-plain and length-matched controls are needed, but quantitative results for these controls are not presented in the main evaluation.

  3. The paper reports localization performance at 86.5-97.94%. However, the localization metric considers successful localization even if only one token overlaps with the actual adversarial span. As localization is one of the main contributions, the authors should report token-level F1, precision, and recall wherever possible.

  4. The ablation studies in the paper are limited. The authors should show the individual contributions of entropy-based weighting, attention-sink masking, attention head selection, MAD normalization.

  5. The experimental evaluation is limited to older models and two jailbreak attack families. More recent models (e.g., Llama-3, Qwen) and semantic attacks (e.g., PAIR/TAP, or AdvPrompter, which the paper cites) should be included.

  6. For FPR computation authors use 17 benign samples per model, which is relatively very small to assess the method correctly. Additionally, unmodified harmful requests are not reported in the main results, although distinguishing them from jailbreak attempts is central to the paper’s claim.

  7. The ASR comparison is not entirely convincing. The authors acknowledge errors in the model used to judge attack success. Their method blocks detected prompts, while RobustKV tries to make the model respond safely, so the comparison is not entirely fair. The paper also does not evaluate how the defense affects benign user requests. Although the abstract claims the lowest ASR, the conclusion acknowledges that; the comparison needs more reliable evaluation.

  8. The paper does not evaluate computational overhead. Although theoretical complexity is discussed, actual inference latency and memory usage are not reported, making it difficult to evaluate the practical efficiency of the method.

Questions for authors
  1. How do authors establish that detected attention anomalies are associated with adversarial manipulations rather than influenced by prompt structure, length or semantics? Can they provide quantitative results on unmodified harmful requests and length-matched non-adversarial prompts?

  2. For localization, since the SpanHit metric only requires one overlapping token, can the authors report token-level precision, recall, F1, or IoU to demonstrate how accurately the method localizes adversarial tokens?

  3. Can the authors provide detailed ablation studies covering the contribution of individual components mentioned in the weaknesses?

  4. What is the actual computational overhead of the proposed method in terms of inference-time latency, compared to other baselines

3 comments

Response to Reviewer pZqG

We thank the reviewer for the thorough review and the precise questions, which helped us strengthen the evaluation. We ran the requested analyses with frozen detectors, with no re-tuning on the new data.

W1 (novelty). Our contribution is not any single component but the use of early-decoding attention as a position-resolved signal for jointly detecting attack presence and pointing to the modified region. This is a different measurement from CPD (vocabulary entropy during prefill), Attention Tracker (prompt-injection focus scores) and Token Highlighter (gradient-based affirmation loss). The ablations (W4) show that this specific design matters: in particular, the head-selection step on development data is what makes the signal usable.

W2 / Q1 (adversarial modification vs. length, structure or semantics). We added unmodified harmful requests and length-matched benign controls.

  • Not driven by harmful content: the detector rarely flags unmodified harmful requests, 0.5–3.8% on AdvBench and 0–4.7% on HarmBench direct requests across six target models. The signal is therefore not a response to harmful subject matter.
  • Not explained by length for GCG: against benign prompts length-matched to GCG attacks, our score still separates GCG attacks with AUROC 0.885 (Llama-2-7B). The GCG signal is thus not explained by prompt length.

W3 / Q2 (token-level localization). We now report token-level precision, recall, F1 and IoU alongside SpanHit. On GCG attacks (Llama-2-7B), the tokens our detector flags fall inside the annotated adversarial suffix with 100% token precision, on both native and HarmBench GCG. When the probe marks tokens, they are adversarial tokens rather than benign parts of the request. The flagged region is typically a subset of the suffix (the most anomalous tokens) rather than its full extent. We will report the complete token-level metrics and onset error in the revision.

W4 / Q3 (ablations). We re-fitted each variant under the identical protocol:

Variant (Llama-2-7B)Native GCG recall
Full method (selected heads)94.4%
8 random heads (5 seeds)17.8 ± 15.1%
Mean/std instead of median/MAD normalization61.1%

Head selection and robust median/MAD normalization are therefore the key components.

W5 (newer models and semantic attacks). We extended the evaluation to Llama-3-8B-Instruct and Qwen2.5-7B/14B-Instruct, and to PAIR, TAP, persuasion (PAP), multilingual MultiJail and five additional HarmBench attack families (AutoPrompt, PEZ, GBDA, UAT, HumanJailbreaks). On Qwen2.5-14B, the score separates suffix attacks from benign prompts with AUROC 0.93–0.96 while flagging only 0.3% of benign prompts.

W6 (benign controls and harmful requests). We agree that 17 benign references are too few. We have substantially enlarged the controls per model: about 600–820 benign prompts, including XSTest-safe, OR-Bench-Hard and length-matched prompts, and about 770–930 unmodified harmful prompts. The revision will report false-positive rates on these sets with confidence intervals, broken down by source and prompt length.

W7 (ASR comparison). We have re-judged all defense responses with the official HarmBench classifier and will report ASR under this standard judge, with judge-agreement statistics. We agree that gating and KV-cache intervention are different kinds of defense, and the revision presents them separately rather than as a single ranking. On benign impact: the gate passes unflagged requests unchanged, so their outputs are identical to the undefended model’s. Its benign cost is therefore exactly its false-positive rate, which we report on the enlarged benign sets.

W8 / Q4 (overhead). The probe adds no forward pass and no auxiliary model. It reads the selected heads’ attention during the first T ≤ 32 decoding steps of the ordinary generation, computing explicit attention weights only for those heads. Measured end-to-end at batch 1, with 256 new tokens, on one H200:

ConfigurationLlama-2-7B latencyLlama-2-13B latencyExtra GPU memory at inference
Undefended2.23 s3.29 s—
Ours+3.7%+2.6%Small activation buffer for sampled heads
Llama-Guard-3-8B check, then generation+3.6%+1.4%+15 GiB

Scoring takes 0.6–1.7 ms per prompt. At inference, our detector requires a small amount of additional GPU memory to retain sampled attention activations from the selected heads; its memory overhead is therefore not zero. The initial head-selection stage requires substantially more data collection, as attention activations must be collected across all candidate heads over the development set. This setup cost is separate from the inference overhead reported above. Our detector achieves comparable latency to a separate guard model while avoiding the additional 15 GiB required to load that model.

Thank you for your detailed explanations and answers particularly regarding W2, W5, W6, and W8. However, some of the concerns remain:

  1. The authors do clarify the differences from prior work, but do not establish a strong new methodological contribution. So novelty remains a concern.

  2. The rebuttal states that token-level precision, recall, F1, and IoU are now reported, while providing only the precision value (100% for GCG on Llama-2-7B).

  3. The ablation concern is partially addressed. Entropy-based weighting and attention-sink masking are not included, while ablations for attention-head selection and median/MAD normalization are provided.

  4. In the ablation table of the rebuttal, the full method (selected heads) achieves 94.4% GCG recall, whereas Table 1 of the main paper reports 88.5%. Why has the full-method recall changed? Could the authors clarify whether these results use different evaluation settings?

So I would maintain my original rating.

Thanks for the follow-up and address the remaining points below.

1. Novelty

Our contribution has two parts.

Position-specific attention over the prompt. Perplexity filters score the likelihood of the input, while CPD tracks the entropy of the next-token vocabulary distribution as the model processes the prompt. Attention-based detectors for prompt injection, such as Attention Tracker, read attention from a single query position at the end of the prompt. We examine how individual attention heads distribute attention across all user-prompt tokens during early decoding. We find that jailbreaks alter these attention distributions in a position-specific way.

Attention dynamics across decoding steps. Each decoding step issues a new query that attends to the prompt, so a head’s attention to adversarial tokens can vary across steps. Figure 1 shows one row per decoding query. The resulting signal spans two dimensions, prompt position and decoding step. Our method aggregates each head’s attention across the first decoding steps, weights each step by its attention entropy (Section 4.2), and then applies robust token-level scoring.

2. Entropy weighting and sink masking

We re-fitted both ablation variants on Llama-2-7B using the same protocol.

  • Entropy weighting. Removing entropy weighting leaves GCG recall at 94.4% and increases the number of flagged benign prompts from 371 to 396 out of 824. In this ablation, entropy weighting reduces false alarms at the same recall.
  • Sink masking. Removing sink masking yields identical results. As described in Section 4.1, we identify sinks using attention statistics from non-attack data. In the evaluated models, the identified sinks lie in the chat template, so masking leaves all user-prompt tokens unchanged. The mask is intended for models or templates in which sinks fall within the user input. We will clarify that it is inactive in the evaluated setting.

3. GCG recall of 94.4% vs. 88.5%

The two recall values come from different test sets evaluated with the same detector configuration (T=32T = 32, k=8k = 8, and a minimal system message).

EvaluationGCG test setRecall
Table 1200 GCG attacks88.5%
Rebuttal ablations18 native GCG attacks from the held-out split of the head-selection protocol94.4% (17/18)

For the rebuttal ablations, we re-fitted and re-evaluated each variant using the head-selection protocol. With only 18 GCG attacks in the held-out split, the 95% Wilson confidence interval is [74.2%, 99.0%], which includes the Table 1 estimate of 88.5%.

Reviewer_8pAFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether jailbreak prompts leave identifiable behavioral footprints in the decoding attention of frozen language models. The authors propose a training-free detector that uses entropy-weighted attention and robust token-level scoring to jointly detect jailbreaks and localize adversarial spans without modifying the target model. The main limitations are the small benign-control set, confirmed errors in the automatic outcome judge, lack of adaptive attack evaluation, and unmeasured end-to-end deployment overhead.

Critical strengths
  1. The paper identifies position-level behavioral footprints in decoding attention and jointly detects and localizes jailbreaks.
  2. The method uses frozen models, separates head selection from threshold calibration, and evaluates both successful and unsuccessful attacks across different attack families.
Critical weaknesses
  1. The reported ASR advantage relies on an automatic judge with confirmed errors, weakening the defense comparison.
  2. The small benign set and lack of adaptive attacks and deployment-overhead evaluation limit evidence for practical robustness.
Questions for authors
  1. The paper does not explain how the confirmed errors in the automatic outcome judge were identified, nor does it provide corrected ASR figures following manual validation. Given that the reported defense ranking relies heavily on these automatic judgments, supplying a manually validated subset or corrected labels would materially affect the conclusions of the main defense comparison. 2.Lacking an evaluation of adaptive attack scenarios. Specifically, it does not test the detector's performance when an attacker explicitly optimizes against the selected attention heads and the token-level scoring function. 3.The paper has not yet verified whether an attacker can preserve jailbreak effectiveness while simultaneously suppressing or redistributing the proposed attention anomaly features.
1 comment

Q1. How were the judge errors identified, and do they affect the defense comparison?

How we identified the errors. Attack success was labeled by an LLM-based outcome judge (Qwen2.5-14B), which is separate from our attention detector and only decides whether a response fulfills the harmful goal. To validate it, we compared it with the official HarmBench classifier (HarmBench-Llama-2-13b-cls, using the official prompt) on 1,818 jointly judged responses.

The disagreements are highly asymmetric. In 349 of 362 cases, the official classifier labeled the response a successful jailbreak while our judge labeled it unsuccessful. Manual inspection of sampled cases showed that our judge labeled compliant responses as unsuccessful when they contained a disclaimer (e.g., “for educational purposes only”) or were truncated, even though they provided the requested content. The judge’s errors therefore almost exclusively under-count successful attacks.

Why these errors do not favor our method. For a gating defense, ASR is bounded by the miss rate, because flagged inputs receive a fixed refusal. As noted in Section 6.1, our probe’s judged ASR already equals this bound in every condition of Table 1. For example, on Llama-2-7B/GCG the ASR is 11.5% at 88.5% recall. Correcting under-counted successes therefore cannot increase our method’s ASR. For CPD and RobustKV, whose judged ASR lies below their bounds, the same correction can only increase it. The direction of the judge’s errors thus makes the reported comparison conservative with respect to our method rather than favorable to it. We will make this argument explicit in the paper.

Q2. Adaptive attacks that optimize against the selected heads and scoring function.

Setup. We implemented a fully white-box adaptive attack on Llama-2-7B, whose detector aggregates eight selected heads. The attacker knows these heads, the normalization and the calibrated threshold τ. GCG minimizes

L=CE⁡(target)+λ⋅max⁡(0,S(x)−(τ−0.3)),L = \operatorname{CE}(\text{target}) + \lambda \cdot \max\left(0, S(x) - (\tau - 0.3)\right),

where S(x) is an exact differentiable re-implementation of our detector score, matching the real detector to within 3 × 10⁻⁷. The weight λ trades off the jailbreak objective against evading the detector: λ = 0 recovers standard GCG, and larger λ prioritizes evasion. We used 20 behaviors per setting and 250 optimization steps. Evaluation uses the real frozen detector on the model’s own generations, with success judged by the official HarmBench classifier.

Results. The attacker can lower the detector score, but in our experiment this conflicts with the jailbreak objective:

  • Lower success: jailbreak success drops from 6/20 to 2/20 and 1/20.
  • Higher target loss: the median target loss rises from 0.36 to 0.61–0.75.
  • Few prompts achieve both goals: only 1–2 of 20 prompts both jailbreak the model and evade detection.

The attack must satisfy the jailbreak objective while also controlling the attention of every selected head, and this joint optimization is difficult. We expect the difficulty to increase with larger or randomized head sets, and we will evaluate this in future work.

Q3. Can an attacker preserve jailbreak effectiveness while suppressing the attention features?

Under our threat model (Section 3), the adversary controls only the user input. Within this setting, the adaptive experiment above directly tests the question. Preserving jailbreak effectiveness while suppressing the footprint succeeded for only 1–2 of 20 behaviors, compared with 6 of 20 for unconstrained GCG.

A stronger adversary with access to model internals could go further. Jacobian-based tools such as the J-lens (Gurnee et al., 2026) estimate the linearized effect of internal activations on the model’s output tokens and support targeted steering or patching of those activations. In principle, such tools could be used to locate and alter the internal representations that produce the attention footprint while preserving the harmful output. This kind of activation-level manipulation lies outside our input-only threat model, and we leave it to future work.

Small benign set. We agree that 17 benign references are too few. We enlarged the controls to about 600–820 benign prompts per model, including XSTest-safe, OR-Bench-Hard and prompts length-matched to the attacks, plus about 770–930 unmodified harmful prompts. The revision reports false-positive rates on these sets with confidence intervals.

Deployment overhead. We measured end-to-end overhead (batch 1, 256 new tokens, one H200). The probe adds 3.7% latency on Llama-2-7B and 2.6% on Llama-2-13B, with no additional model memory. By comparison, a separate Llama-Guard-3-8B check requires an extra 15 GiB of GPU memory.

Reference

Wes Gurnee, Nicholas Sofroniew, et al. Verbalizable Representations Form a Global Workspace in Language Models. Transformer Circuits Thread, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.