acceptodds
Under review as a conference paper at ICLR 2027

Inference-Time Attention Manipulation

Abstract

Open-weight language models expose their internal computation at inference time, creating a safety surface that is not captured by robustness to adversarial prompts alone. We introduce a novel white-box threat model based on **region-level attention manipulation**, which tests refusal behavior by steering or cutting attention between semantic regions of a chat. Under this threat model, each manipulation modifies inference-time attention while holding model parameters fixed, requiring neither training nor adversarial prompt optimization. *Attention cutting* from the decision region to the harmful request can bypass refusal in several models without modifying the prompt or forcing a prefill, while some models resist this manipulation. For models that resist, attention manipulation during continuation can still bypass refusal, suggesting that attention dynamically maintains refusal beyond the response opening. We further show that *attention steering* toward a supplied harmful prefill increases attack success across most evaluated model families, although stronger interventions do not consistently produce larger effects. Together, these results show that refusal depends not only on what appears in the context but also on where and when the model allocates attention during generation. Safety evaluations of open-weight models should therefore test attention-level interventions both at the decision region and during continuation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, AI Safety, Jailbreaking, Attention Manipulation, Mechanistic Interpretability, Refusal

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BZigOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for manipulating region-level attention during inference to investigate the refusal behaviors and safety vulnerabilities of large language models by cutting off or adjusting attention between different semantic regions. The authors conducted experiments on 12 open-weight models and three security benchmarks, and found that some models, even if they performed robustly at the beginning of the response, may still be bypassed and rejected in subsequent generation.

Critical strengths
  1. The topic holds research value, offering a new perspective on the safety evaluation of open-weight models.
  2. The paper demonstrates sound overall logic and a clear structure, with the research problem and key contributions explicitly articulated.
  3. The experiments encompass multiple model families and safety benchmarks, and the main findings are supported by robust experimental evidence.
Critical weaknesses
  1. Some experimental setups differ from standard usage practices for the models; the scope of applicability of the conclusions requires further clarification.
  2. Comparisons with existing methods are insufficient, and the relative advantages of the proposed method remain unclear.
  3. The analysis of performance differences among the models lacks depth, and the explanation of underlying mechanisms needs refinement.
  4. Some figures and tables lack visual appeal; there is room for improvement regarding layout and the clarity of annotations.
Questions for authors
  1. If the full reasoning process of the reasoning-oriented model is retained, do the main conclusions still hold? Are the results robust across different chat templates and generation settings?
  2. Provide a direct comparison with existing internal intervention methods and demonstrate the advantages of the proposed method under comparable experimental conditions.
  3. Analyze the differences in sensitivity to attention intervention among various models and provide evidence to support the explanation of the underlying mechanisms.
  4. Optimize the layout, color scheme, and labeling of certain figures to make the differences between experimental conditions more intuitive.
11 comments

We thank the reviewer for the constructive feedback and for recognizing the research value of our work and the strength of our experimental evidence. We address each concern below.

1. Reasoning modes, chat templates, and generation settings.

Appendix D.2 describes the original generation modes: decision-link and continuation experiments skip reasoning for Qwen3 and R1-Distill, while gpt-oss uses an empty analysis channel. Prefill experiments for Qwen3 and R1-Distill instead continue within the reasoning stage.

We have now repeated the decision-cut and continuation experiments with full reasoning enabled and evaluated refusal ablation [1] under the same reasoning modes. ClearHarm ASR is reported in percent, with each entry showing reasoning off → on:

ModelDecision cutContinuationRefusal ablation
gpt-oss-20B59.2 → 4.582.1 → 0.611.7 → 1.1
gpt-oss-120B46.9 → 1.155.9 → 0.019.6 → 0.0
R1-Distill-7B30.2 → 24.072.6 → 38.515.1 → 1.7
R1-Distill-14B8.4 → 21.876.5 → 19.05.0 → 1.7
R1-Distill-32B10.1 → 24.069.3 → 21.80.0 → 0.0
Qwen3-8B3.9 → 18.439.1 → 6.713.4 → 7.8
Qwen3-14B2.2 → 12.835.2 → 4.50.0 → 0.0
Qwen3-32B1.1 → 29.115.1 → 5.00.0 → 0.0

Reasoning strongly suppresses both attacks on gpt-oss and lowers continuation ASR across all tested models. Decision-cut ASR increases on Qwen3 and the larger R1-Distill models, while R1-Distill-7B retains substantial ASR; Reasoning also weakens refusal ablation baseline. We will report these differences explicitly when stating the scope of our conclusions.

We also tested greedy decoding and temperature 1 alongside each model's stored defaults. Decision-cut ASR gains relative to the matched baseline are reported in percentage points:

ModelDefault decodingGreedyTemperature 1
Llama-3.2-3B+43.6+19.6+31.3
R1-Distill-7B+24.0+17.3+27.9
gpt-oss-20B+53.6+3.9+26.8
Qwen3-8B−1.7+0.0+0.6

Llama and R1-Distill retain positive gains across these settings; gpt-oss is much less sensitive under greedy decoding, and Qwen3 remains largely unaffected.

We use each model's native chat template and identify semantic regions from its rendered template (Appendix C), covering four model families. We have not systematically substituted different templates within the same model. The decoding comparisons above test generation settings while retaining the native template.

2. Comparison with existing internal interventions.

We have completed comparisons with refusal-direction ablation (Arditi et al., 2024) [1] and contrastive activation addition (CAA; Rimsky et al., 2024) [2] across 12 models and three datasets, using matched models, harmful requests, decoding settings, and evaluators. We tuned the parameters for refusal ablation and selected CAA's best result across strengths 1, 2, and 4. We also tested strengths 6 and 8, where ASR decreased because the responses became nonsensical.

Both baselines require activation collection and a model-specific direction. Our decision cut requires no activation collection, offline calibration, gradients, model-weight changes, fine-tuning, or adversarial prompt optimization. It changes only the specified attention connections while keeping the model parameters and evaluated prompt unchanged.

ClearHarm, ASR (%):

ModelDecision cutContinuationRefusal ablationCAA
Llama-3.2-3B48.656.44.534.1
Llama-3.1-8B34.611.21.733.0
Llama-3.3-70B19.617.958.128.5
Qwen3-8B3.939.113.41.1
Qwen3-14B2.235.20.01.1
Qwen3-32B1.115.10.01.7
Qwen3-2507-30B0.635.20.02.2
R1-Distill-7B30.272.615.16.1
R1-Distill-14B8.476.55.01.7
R1-Distill-32B10.169.30.00.0
gpt-oss-20B59.282.111.748.0
gpt-oss-120B46.955.919.63.4

HarmBench, ASR (%):

ModelDecision cutContinuationRefusal ablationCAA
Llama-3.2-3B43.556.050.033.5
Llama-3.1-8B32.555.058.544.5
Llama-3.3-70B20.027.054.537.0
Qwen3-8B4.036.559.52.5
Qwen3-14B2.528.56.02.5
Qwen3-32B1.025.50.50.5
Qwen3-2507-30B6.027.056.01.5
R1-Distill-7B20.064.515.57.5
R1-Distill-14B4.567.51.50.5
R1-Distill-32B12.066.00.02.5
gpt-oss-20B34.559.544.029.5
gpt-oss-120B29.552.029.09.5

JailbreakBench, ASR (%):

ModelDecision cutContinuationRefusal ablationCAA
Llama-3.2-3B30.056.038.039.0
Llama-3.1-8B26.052.048.038.0
Llama-3.3-70B13.015.050.025.0
Qwen3-8B1.039.058.00.0
Qwen3-14B1.018.00.01.0
Qwen3-32B0.015.00.00.0
Qwen3-2507-30B0.021.048.00.0
R1-Distill-7B14.071.06.02.0
R1-Distill-14B3.073.01.00.0
R1-Distill-32B3.065.00.00.0
gpt-oss-20B35.065.044.021.0
gpt-oss-120B25.053.025.03.0

CAA strength sweep on ClearHarm, ASR (%):

Modelα=1α=2α=4
Llama-3.2-3B12.334.10.0
Llama-3.1-8B7.333.03.4
Llama-3.3-70B11.228.519.0
Qwen3-8B0.60.61.1
Qwen3-14B1.10.61.1
Qwen3-32B1.11.71.1
Qwen3-2507-30B2.21.71.7
R1-Distill-7B5.66.15.0
R1-Distill-14B0.60.61.7
R1-Distill-32B0.00.00.0
gpt-oss-20B2.87.348.0
gpt-oss-120B2.83.42.2

Refusal ablation remains stronger on Llama-3.3-70B across all three datasets and on some Qwen3 models on HarmBench and JailbreakBench. The comparison therefore shows different patterns of susceptibility across model families, alongside the broader effectiveness of continuation manipulation.

These comparisons establish relative effectiveness and attack requirements. The value of our work also lies in identifying which attention connections contribute to refusal, when they matter, and how these dependencies differ across models. These controlled tests extend prompt-based safety evaluation to internal attention. Other internal attacks examine refusal directions, components, or output distributions. We believe a higher ASR from another method does not invalidate the attention-level vulnerability we identify; different weaknesses can require different defenses.

3. Model differences and underlying mechanisms.

Our explanation is that models differ in how refusal is established at the assistant boundary and response opening, and how it is maintained during continuation. The existing ablations and new measurements provide evidence for these distinct stages.

Decision-region processing.

Appendix F.2 separates attention from the assistant boundary and response opening to the harmful request. ClearHarm mean ASR gains across the 12 models are reported in percentage points (mean ± SEM):

Region from which attention is cutMean ASR gain
Complete decision region18.25 ± 5.71
Assistant boundary12.75 ± 5.95
Response opening4.56 ± 3.24

The assistant-boundary effect places substantial sensitivity before the first token is sampled. Refusal is therefore affected by prompt processing as well as the computation at the response opening.

We additionally measured baseline attention to the harmful request and the projection of the decision-position hidden state onto the refusal direction. Relative projection drop is defined as

Pbaseline−Pcut∣Pbaseline∣\frac{P_{\mathrm{baseline}} - P_{\mathrm{cut}}}{|P_{\mathrm{baseline}}|}

where values above 1 indicate that a positive baseline projection crosses zero. ASR gains are reported in percentage points relative to the matched baseline.

ModelBaseline attention to requestRelative refusal-signal dropDecision-link ASR gain (pp)Continuation ASR gain (pp)
gpt-oss-20B0.0890.77654.779.9
Llama-3.2-3B0.0940.76543.650.8
Llama-3.1-8B0.1000.93441.33.9
R1-Distill-7B0.3520.83726.370.4
R1-Distill-32B0.1322.8439.569.3
Qwen3-2507-30B0.1301.0318.434.6
Qwen3-14B0.1360.6527.334.1
R1-Distill-14B0.3970.8583.474.9
Qwen3-32B0.10413.6063.415.1
Qwen3-8B0.1441.6042.238.0

The decision cut lowers the measured refusal signal in every model in this analysis, including models with little behavioral change. Reducing that projection alone therefore does not explain the differences in ASR.

Associations with decision-link ASR gain across these models are:

MeasurementPearson r (p)Spearman ρ (p)
Baseline decision-region attention to the request−0.298 (0.40)−0.675 (0.032)
Relative refusal-projection drop−0.350 (0.32)−0.486 (0.15)

Models that allocate more baseline decision-region attention to the request tend to have smaller decision-link gains in the rank comparison. The projection-drop magnitude does not show a statistically significant association with gain. These measurements distinguish changing a refusal-related representation from changing the model's generated behavior.

Reviewer_d6P4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an attention-based attack mechanism for white-box models called region-level attention manipulation. The author proposes two approaches, including attention cutting and attention steering, to test the vulnerability of open-weight models' capabilities on common harmful intents. The author shows on multiple jailbreak datasets that both attention cutting and steering increase the attack success rate (ASR) and allow more frequent bypass of safety guardrails. This method does not require additional training or adversarial prompt optimization, instead it only operates on inference-time attention while model parameters frozen.

Critical strengths
  1. The paper is well structured with clear presentation and well-organized experimental design. The overall paragraph has clear motivation, methodology, and experimental design. In particular, Figure 1 effectively illustrates the proposed region-level attention manipulation framework, making it easy to understand how attention cutting and steering are applied to different semantic regions of a conversation.
Critical weaknesses
  1. My first concern is the limited generalizability of the paper's experimental setup. Specifically, the proposed attention manipulation assumes that harmful and benign request regions are known in advance. However, even in a white-box setting, access to model internals does not necessarily provide this label. In realistic conversations, benign and harmful content may be interleaved. I suggest that the authors clarify how these regions could be identified in more general settings and whether the approach can generalize beyond manually constructed prompts.

  2. My second concern is regarding the limited experimental comparisons and response quality evaluation. The experiments primarily compare the proposed method against unmodified inference, without comparisons to existing white-box jailbreak methods. Such comparisons would help readers to understand the effectiveness and novelty of the proposed approach. Additionally, although utility results are provided in Appendix G, I suggest including a brief summary in the main paper and providing qualitative response examples or semantic alignment evaluations (between response and harmful request) to examine whether attention manipulation affects general instruction-following and response quality.

Questions for authors

Please refer to the Weaknesses section for my questions.

7 comments

We thank the reviewer for the detailed feedback and for recognizing the clarity of our framework and experimental design. We address each concern below.

1. Identification of harmful and benign regions.

Our method uses boundaries available from the chat template or the attacker's own inputs. In the original decision-link setting, we cut attention to the harmful-request tokens. However, decision-link cutting can also target the entire user turn, including both benign and harmful content, without requiring the attacker to identify the harmful span. In the continuation setting, the attacker supplies the benign request before the evaluated query and therefore knows the boundary between them. Specifically, the attacker knows the number of tokens preceding the user input, the tokens of the supplied benign request, and the chat template tokens following the user input, allowing them to identify the boundaries of the user input without inspecting its content. Similarly, in the prefill setting, the attacker knows the supplied prefill and its token positions. Appendix C explains how these text boundaries are mapped to token positions using chat templates.

We added an experiment across all 12 models in which a benign request precedes each harmful request in the same user turn. For this experiment, we apply decision-link manipulation to the entire user turn (both benign and harmful requests), rather than targeting only the harmful-request span as in the original setting. We report results for three benign preambles and one sample each. ASR is reported in percent and changes in percentage points:

ModelBaseline (%)Decision Manipulation (%)Change (pp)
Llama-3.2-3B6.526.1+19.7
Llama-3.1-8B5.620.1+14.5
Llama-3.3-70B2.97.5+4.6
Qwen3-8B1.52.2+0.7
Qwen3-14B1.11.5+0.4
Qwen3-32B1.71.9+0.2
Qwen3-2507-30B0.31.0+0.7
R1-Distill-7B5.717.8+12.1
R1-Distill-14B2.55.2+2.7
R1-Distill-32B0.73.0+2.3
gpt-oss-20B2.627.6+25.0
gpt-oss-120B2.913.3+10.4

This directly tests mixed benign and harmful input without requiring labels for harmful sub-spans. Qwen3 remains less sensitive to decision cutting, consistent with the original results. The decision cut requires neither a benign request nor a supplied prefill; continuation uses a two-part prompt whose boundary the attacker already knows.

2. Comparison with two white-box attacks.

We have completed comparisons with refusal-direction ablation (Arditi et al., 2024) [1] and contrastive activation addition (CAA; Rimsky et al., 2024) [2] across 12 models on ClearHarm, HarmBench, and JailbreakBench, using matched models, harmful requests, decoding settings, and evaluators. We tuned the parameters for refusal ablation and selected CAA's best result across strengths 1, 2, and 4. We also tested strengths 6 and 8, where ASR decreased because the responses became nonsensical.

Both baselines require collecting activations on a harmful and a benign dataset and extracting a model-specific direction. Our decision cut requires no activation collection, model-specific calibration, gradients, fine-tuning, or adversarial prompt optimization. Our work isolates the role of attention by keeping the evaluated prompt and model parameters fixed and changing only attention relations. The value of our work is to expose systematic vulnerabilities in how refusal depends on attention between chat regions, including vulnerabilities missed by these direction-based interventions.

ClearHarm, ASR (%):

ModelDecision cutContinuationRefusal ablationCAA
Llama-3.2-3B48.656.44.534.1
Llama-3.1-8B34.611.21.733.0
Llama-3.3-70B19.617.958.128.5
Qwen3-8B3.939.113.41.1
Qwen3-14B2.235.20.01.1
Qwen3-32B1.115.10.01.7
Qwen3-2507-30B0.635.20.02.2
R1-Distill-7B30.272.615.16.1
R1-Distill-14B8.476.55.01.7
R1-Distill-32B10.169.30.00.0
gpt-oss-20B59.282.111.748.0
gpt-oss-120B46.955.919.63.4

HarmBench, ASR (%):

ModelDecision cutContinuationRefusal ablationCAA
Llama-3.2-3B43.556.050.033.5
Llama-3.1-8B32.555.058.544.5
Llama-3.3-70B20.027.054.537.0
Qwen3-8B4.036.559.52.5
Qwen3-14B2.528.56.02.5
Qwen3-32B1.025.50.50.5
Qwen3-2507-30B6.027.056.01.5
R1-Distill-7B20.064.515.57.5
R1-Distill-14B4.567.51.50.5
R1-Distill-32B12.066.00.02.5
gpt-oss-20B34.559.544.029.5
gpt-oss-120B29.552.029.09.5

JailbreakBench, ASR (%):

ModelDecision cutContinuationRefusal ablationCAA
Llama-3.2-3B30.056.038.039.0
Llama-3.1-8B26.052.048.038.0
Llama-3.3-70B13.015.050.025.0
Qwen3-8B1.039.058.00.0
Qwen3-14B1.018.00.01.0
Qwen3-32B0.015.00.00.0
Qwen3-2507-30B0.021.048.00.0
R1-Distill-7B14.071.06.02.0
R1-Distill-14B3.073.01.00.0
R1-Distill-32B3.065.00.00.0
gpt-oss-20B35.065.044.021.0
gpt-oss-120B25.053.025.03.0

CAA strength sweep on ClearHarm, ASR (%):

Modelα=1α=2α=4
Llama-3.2-3B12.334.10.0
Llama-3.1-8B7.333.03.4
Llama-3.3-70B11.228.519.0
Qwen3-8B0.60.61.1
Qwen3-14B1.10.61.1
Qwen3-32B1.11.71.1
Qwen3-2507-30B2.21.71.7
R1-Distill-7B5.66.15.0
R1-Distill-14B0.60.61.7
R1-Distill-32B0.00.00.0
gpt-oss-20B2.87.348.0
gpt-oss-120B2.83.42.2

Refusal ablation achieves stronger results on Llama-3.3-70B across all three datasets and on some other models on HarmBench and JailbreakBench.

3. Benign utility.

We have added a discussion of the results in Appendix G to Section 5.6. Appendix G evaluates MATH-500 and GPQA Diamond across all 12 models, using matched baselines with the same prompt or supplied prefill. The mean changes in Pass@1, in percentage points (mean ± SEM), are:

BenchmarkDecision-linkContinuationPrefill
MATH-500−2.05 ± 1.35+0.25 ± 1.76−1.13 ± 0.57
GPQA Diamond+2.57 ± 1.41+6.65 ± 1.29−1.18 ± 1.19

We attribute the improvement under continuation manipulation to increased attention to the user request, which can help the model follow it more precisely. These results show that harmful compliance can increase without large average losses on either utility benchmark.

4. Response quality and semantic alignment.

GPT-OSS-Safeguard prompt [3] (Appendix H) instructs it to assign scores of 4 or 5 only when the response provides actionable harmful assistance aligned with the request. We count a success only if the response receives one of these scores and Qwen3Guard classifies it as unsafe and non-refusal. Generic, repetitive, or off-topic text does not qualify.

We added a semantic-alignment analysis on ClearHarm across all 12 models, measuring cosine similarity between sentence embeddings of the generated response and the harmful request:

ManipulationMean baseline similarityMean manipulated similarityModels with increased similarity
Decision cut0.3670.4818/12
Continuation0.2180.48512/12
Prefill0.6420.66810/12

Continuation increases relevance on every model; the prefill baseline already uses the same supplied prefill and starts with higher similarity. These measurements complement the guards' assessment of whether a response meaningfully answers the request.

We also applied an additional filter excluding incoherent or repetitive responses, including those that merely repeat the supplied prefill. ClearHarm ASR before → after filtering is reported in percent:

ModelDecision cutContinuationPrefillRefusal ablationCAA
Llama-3.2-3B48.6→47.556.4→55.362.6→59.24.5→3.434.1→22.3 (−12pp)
Llama-3.1-8B34.6→33.511.2→8.954.2→49.71.7→1.733.0→30.7
Llama-3.3-70B19.6→19.017.9→17.951.4→48.058.1→58.128.5→28.5
Qwen3-32B1.1→1.115.1→15.146.9→43.00→01.7→1.7
Qwen3-2507-30B0.6→0.635.2→35.258.7→54.70→02.2→2.2
R1-Distill-7B30.2→30.272.6→72.645.8→43.015.1→15.16.1→6.1
R1-Distill-14B8.4→8.476.5→76.560.9→59.85.0→5.01.7→1.7
gpt-oss-20B59.2→59.282.1→82.117.3→16.811.7→11.748.0→48.0
gpt-oss-120B46.9→46.955.9→55.935.8→35.819.6→19.63.4→3.4

The strongest continuation results remain after filtering.

Appendix E already validates the automated criterion against three independent human annotators on 100 responses:

EvaluationSuccessful attemptsFalse positivesFalse negatives
Human annotations46N/AN/A
Automated criterion (helpfulness ≥ 4)30723

Agreement with human annotations is reported below (percentages except Cohen's κ).

EvaluatorTaskAgreementκPrecisionRecallFalse positivesFalse negatives
Qwen3GuardHarmful compliance92.00.8198.590.317
GPT-OSS-SafeguardHarmful compliance96.00.9095.998.631
GPT-OSS-SafeguardHelpfulness ≥ 472.00.4376.556.5820
GPT-OSS-SafeguardHelpfulness ≥ 380.00.6074.187.0146
Both guardsAttempt success, helpfulness ≥ 470.00.3876.750.0723
Both guardsAttempt success, helpfulness ≥ 378.00.5675.078.31210

We retain helpfulness ≥ 4 in the main evaluation to favor fewer false positives at the cost of missing successes, which undercounts successful attempts for the samples.

Reviewer_91y1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore the topic of model jailbreaking. Specifically, they focus on open-weight language models during inference time. The authors jailbreak the model by steering or cutting attention between semantic regions of the chat during inference time, which can make the model bypass refusal and follow harmful requests.

Critical strengths

The authors introduce a new threat model that involves steering or cutting attention between semantic regions in a chat.

Critical weaknesses

Is this scenario realistic? The authors claim that "Open-weight language models expose their internal computation at inference time, creating a safety surface that is not captured by robustness to adversarial prompts alone." However, this is debatable in terms of deployment. For instance, OpenRouter hosts many open-weight language models, but can the authors manipulate their internal computation at the model inference time? Without clear, widely adopted attacking scenarios, unfortunately, I cannot see the significance of this work.

Questions for authors

See weaknesses.

9 comments
AuthorsOpenReview ↗

Deployment Scenarios

We thank the reviewer for raising this concern. We address two points: the deployment scenarios covered by our threat model and the broader scientific value of studying white-box attacks.

Deployment Scenarios

First, we agree that standard hosted APIs, such as OpenRouter, expose only model inputs and outputs and do not provide the internal access required by our attack. An attacker limited to such an API cannot apply our method to the provider's hosted deployment. However, for models with publicly released weights, the attacker is not limited to the hosted API. The same model can be downloaded and run in an environment where the attacker controls its forward pass. Thus, API restrictions protect the provider's specific deployment but do not prevent the attack from being applied to other deployments of the same model.

More broadly, our threat model covers several deployment scenarios beyond an attacker running an open-weight model locally. We distinguish between permission to modify model weights, control over the software executing the forward pass, and permission to submit API requests. Our attack requires control over inference computation, but does not require modifying weights, fine-tuning the model, or computing gradients. This distinction applies in the following scenarios:

  1. Malicious inference operators: When model training and serving are managed separately, an employee controlling the inference software can modify attention without permission to change model checkpoints or perform fine-tuning. For example, vLLM [1] supports registering and overriding custom attention backends , while NVIDIA Triton [2] supports custom inference backends maintained separately from model repositories .

  2. Compromised inference software: An attacker who compromises an inference backend, attention kernel, or serving dependency can manipulate attention without changing model checkpoints. MITRE ATLAS [3] explicitly recognizes AI inference engines and their dependencies as targets of software supply-chain attacks. Microsoft [4] also identifies malicious code in LLM libraries as a threat capable of altering model decisions. Such an intervention would not appear in the user-visible input.

  3. Third-party model hosting: Services such as Hugging Face Inference Endpoints [5] support custom inference containers, where the serving code is supplied separately from the model weights. A malicious or compromised operator controlling such a container can modify attention independently of the model checkpoint.

  4. Restricted model deployments: Amazon SageMaker [6] supports custom inference containers where model artifacts are provided through a read-only directory. An operator controlling the inference code can therefore manipulate attention without modifying the protected checkpoint files.

Our manipulation settings require no fine-tuning, gradients, or adversarial prompt optimization. It changes only selected attention relations, leaving model parameters and the evaluated prompt unchanged. Prompt inspection and checkpoint verification alone would therefore not reveal the intervention. Unlike refusal-direction attacks [7], our method does not require collecting hidden activations or extracting and selecting a model-specific direction. It directly changes predefined attention connections, so it also applies when an attacker can modify attention but cannot inspect other hidden states.

The decision manipulation has no strength parameter and requires no training data or offline optimization. Our vLLM throughput measurements are:

ModelFamilyBaseline (tokens/s)Decision Manipulation (tokens/s)Throughput reduction
llama-3.1-8bllama (dense)2953.42772.26.1%
qwen3-8bqwen3 (dense)2239.72103.66.1%
deepseek-r1-7bdeepseek (dense)2791.02630.75.7%
qwen3-32bqwen3 (dense)1179.71097.37.0%
gpt-oss-20bgpt-oss (MoE)2325.61969.615.3%

Applying the intervention therefore requires only a moderate inference-time cost, without a separate training or calibration stage.

AuthorsOpenReview ↗

White-box Attacks

White-box Attacks

Second, white-box attacks are an established research direction in LLM safety, with many papers accepted at top conferences. Their scientific value extends beyond whether they can be applied through a commercial API: they help explain how safety mechanisms work and where they fail which can help model publishers improve safety in black-box deployment.

Recent examples include Adaptive Probe-based Steering (ICML 2026) [8] and Command-V (ICLR 2026) [9], which intervene on internal activations, including to facilitate jailbreaking. SABER (EMNLP 2025) [10] modifies cross-layer residual connections, while Weak-to-Strong Jailbreaking (ICML 2025) [11] modifies decoding probabilities during inference. TAO-Attack (ICLR 2026) [12] develops gradient-based jailbreak optimization. These are five recent examples; many more papers on white-box attacks have been accepted at top conferences. They use different forms of access and intervention, showing that safety research extends beyond attacks available to ordinary API users.

Our work examines how directed attention between semantic regions of a chat affects refusal at different stages of generation. Beyond attack success rates, our experiments provide evidence about refusal mechanisms and differences across model families. In particular, we find models that resist decision-region manipulation but remain vulnerable during continuation. This shows why refusal at the response opening is insufficient to assess robustness throughout generation. The value of our work does not depend on whether commercial APIs expose the same internal controls.

Manuscript Revisions. We have updated Section 3 to clarify the inference-access requirements and added Appendix D.4 with deployment scenarios and measured inference costs. All revisions addressing your comments are highlighted in orange\color{#A34500}{\text{orange}}.

References

[1] vLLM Documentation. Attention Backend Registry https://docs.vllm.ai/en/stable/api/vllm/v1/attention/backends/registry/

[2] NVIDIA Documentation. Python-based Backends https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/backend/docs/python_based_backends.html

[3] MITRE ATLAS. Supply Chain Compromise: Software https://atlas.mitre.org/techniques/AML.T0010.001

[4] Microsoft Security. Supply Chain Vulnerabilities (OSS) https://learn.microsoft.com/en-us/security/zero-trust/catalog-ai-attack-techniques/supply-chain-vulnerabilities

[5] Hugging Face Documentation. Inference Endpoints https://huggingface.co/docs/inference-endpoints/engines/custom_container

[6] Amazon SageMaker Documentation. Custom Inference Code with Hosting Services https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html

[7] Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024.

[8] Chen et al. Adaptive Probe-based Steering for Robust LLM Jailbreaking. ICML 2026.

[9] Wang et al. Command-V: Training-Free Representation Finetuning Transfer. ICLR 2026.

[10] Joshi et al. SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection. EMNLP 2025.

[11] Zhao et al. Weak-to-Strong Jailbreaking on Large Language Models. ICML 2025.

[12] Xu et al. TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models. ICLR 2026.

Reviewer_91y1OpenReview ↗

Response

I thank the authors for their response,

The same model can be downloaded and run in an environment where the attacker controls its forward pass. Thus, API restrictions protect the provider's specific deployment but do not prevent the attack from being applied to other deployments of the same model.

In this case, first, the users can check if the weight is the same as the original model weight. Even if we accept such a scenario, the authors shall also compare the attack with the most obvious baseline, such as directly fine-tuning the model on harmful request-response pairs.

Comments on the paper

2 comments

We thank all reviewers for their time and constructive feedback, which have helped us strengthen the paper. The revised manuscript includes additional experiments, analyses, and clarifications.

To make the revisions easier to follow, We have highlighted the revisions in different colors for each reviewer:

  • Reviewer 91y1: orange\color{#A34500}{\text{orange}}
  • Reviewer d6P4: blue\color{#005A9C}{\text{blue}}
  • Reviewer BZig: purple\color{#70309A}{\text{purple}}
  • Changes addressing multiple reviewers: teal\color{#006B63}{\text{teal}}

We welcome further questions or suggestions and would be happy to clarify our responses or any aspect of the revised manuscript during the discussion period.

AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear ACs, SACs, and Reviewers,

We sincerely thank all reviewers, ACs, and SACs for their time and constructive feedback. We appreciate the reviewers' recognition of the new attention-based threat model (Reviewer 91y1), the clear framework and experimental design (Reviewer d6P4), and the research value and strong experimental evidence across multiple model families and benchmarks (Reviewer BZig).

We have posted detailed responses to all reviews and follow-up questions. Below, we summarize the main concerns and how we addressed them:

  1. Threat model and practical relevance (Reviewer 91y1). We clarified the inference-access requirements, identified four deployment scenarios, and measured inference overhead. We also explained why checkpoint verification would not detect our attack and how attention manipulation exposed vulnerabilities distinct from other white-box methods.

  2. Comparison with existing attacks (Reviewers 91y1, d6P4, BZig). We compared against refusal-direction ablation and CAA across 12 models and three benchmarks. Continuation manipulation achieved higher ASR than both in 27/36 comparisons and exceeded the existing prefill-only baseline on 9/12 models.

  3. Region identification and generalization (Reviewer d6P4). We clarified how region boundaries were identified and tested mixed benign and harmful inputs by cutting attention to entire user turns without identifying harmful spans. ASR increased by up to 25.0 percentage points, demonstrating that the attack did not require identifying harmful spans.

  4. Reasoning and generation settings (Reviewer BZig). We conducted additional experiments with full reasoning enabled and alternative decoding settings. Full reasoning reduced continuation ASR but increased decision-link ASR on Qwen3 models, while positive attack gains persisted for most models across different decoding settings. These results revealed different vulnerabilities across models and generation settings.

  5. Refusal mechanisms and model differences (Reviewers 91y1, BZig). We extended the analysis with new attention and refusal-direction measurements. Decision-link manipulation reduced refusal-direction projections across ten models, but these reductions did not predict ASR gains. Lower baseline attention to harmful requests was associated with larger gains (Spearman ρ\rho = −0.675), providing further insight into why models responded differently to attention manipulation.

  6. Response quality and benign utility (Reviewer d6P4). We added semantic-alignment analyses and response-quality filtering, showing that continuation responses became more relevant to harmful requests and that our attack results remained robust after filtering. We also clarified the existing human evaluation and benign-utility results on MATH-500 and GPQA Diamond.

The additional results and clarifications were incorporated into the revised manuscript. We believe these revisions strengthened both the experimental evidence and our understanding of region-level attention manipulation and refusal mechanisms.

We would be happy to answer any further questions. We sincerely thank all reviewers, ACs, and SACs again for their time and constructive feedback.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.