acceptodds
Under review as a conference paper at ICLR 2027

PrisonBreak: Jailbreaking LLMs with at Most Twenty-Five Targeted Bit-flips

Abstract

We present a new vulnerability in aligned large language models (LLMs): their alignment can be bypassed by flipping only a few bits in their parameter values. We exploit this vulnerability with an attack that jailbreaks billion-parameter LLMs with just 5–25 bit-flips—up to 40× fewer than prior bit-flip attacks. To this end, we present an efficient bit-selection algorithm that identifies critical bits for jailbreaks up to 20× faster than exhaustive strategies. Unlike prompt-based jailbreaks, our method directly alters model behavior in memory at inference time, enabling harmful outputs without any input-level modifications. We evaluate our attack on 10 LLMs, achieving attack success rates (ASRs) of 80–98% with negligible impact on model utility. We further demonstrate an end-to-end exploit via Rowhammer-based fault injection, reliably jailbreaking five models (69–91% ASR) on a GDDR6 GPU by exploiting just two vulnerable physical bit locations. Moreover, bit-flip locations identified in one fine-tuned model transfer to others that share the same pre-trained LLMs, demonstrating the black-box exploitability in standard pretraining-then-finetuning paradigms. We evaluate potential countermeasures and find that our attack remains effective against defenses at various stages of the inference pipeline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, jailbreak attacks, bit-flip attacks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pgFxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a bit-flip attack framework to demonstrate that the behavior of the models can be altered with alterations in the model weights, with the methodology being input agnostic. The paper demonstrates that 5-25 bit-flips can be used for a white-box and black-box attack.

Critical strengths
  1. The paper presents an input-agnostic approach for jailbreaking LLMs with an end-to-end validation using a Rowhammer-based fault injection.
Critical weaknesses
  1. The results suggest that the bit locations corresponding to the MSB are more critical for jailbreaking LLMs. However, as shown in ResCiM (TCASAI 2026) and Unicorn-CiM (ISLPED 2026), bit-flips in the exponents of the operands can lead to triggering the compute result to be NaN or Inf. Thus, the jailbreaking could be easily detected. How is the algorithm ensuring that this kind of compute failure is avoided during the jailbreak? Can the authors quantify the rate of occurrence of such failures during inference or candidate selection?
  2. How generalizable is the robustness observed for INT8 in Appendix C? Is the robustness observed due to the attack strategy, or is it due to the representation itself?
  3. Can you please explain how is the current version of the paper better in terms of recent jailbreaking attacks such as SilentStriker (NeurIPS 2025)? Particularly, across different types of LLM tasks, how does the behavior of the model change in terms of reasoning capabilities or language modeling?
Questions for authors

N/A

4 comments

We thank the reviewer for recognizing our contribution to demonstrating input-agnostic jailbreaking through real-world Rowhammer exploitation. We address the questions and concerns below.

(W1/Q1) Bit-flips in the exponents of the operands can lead to triggering the compute result to be NaN or Inf. Thus, the jailbreaking could be easily detected. How is the algorithm ensuring that this kind of compute failure is avoided during the jailbreak? Can the authors quantify the rate of occurrence of such failures during inference or candidate selection?

During our search, we explicitly reject candidate bit-flips that would produce non-finite weight values (NaN or Inf). We additionally verified that none of the selected bit-flip sequences produced in our evaluation introduce NaN or Inf values.

To further examine whether finite-weight perturbations could still cause numerical failures during inference, we re-ran our HarmBench evaluation on four models (Vicuna-7B-v1.5, Llama3-8B, Tulu3-8B, and Qwen2-7B). We counted non-finite activations both without attack and under PrisonBreak. Across all four models, we observe zero non-finite activations in either condition. Thus, PrisonBreak successfully compromises model alignment without producing non-finite weights or activations, meaning NaN/Inf-based checks alone would not detect our attack.

(Q2) How generalizable is the robustness observed for INT8 in Appendix C? Is the robustness observed due to the attack strategy, or is it due to the representation itself?

To assess the generalizability of our INT8 results, we evaluated an additional model, Llama3-8B, and increased the attack budget from 25 to 50 bit flips. Below, we report ASR for Llama3-8B and Vicuna-7B-v1.3 across bit-flip budgets.

Number of FlipsVicuna-7b-v1.3Llama3-8b
014.476.92
1023.9010.69
2030.8217.61
3034.5914.47
4039.6216.98
5045.9118.24

INT8 models remain less susceptible to PrisonBreak in these experiments. For Llama3-8B, ASR reaches only 18.24% with 50 bit-flips, compared to 86.79% for FP16. However, ASR generally continues to rise with the bit-flip budget. This indicates that PrisonBreak is not hitting a search barrier; rather, because INT8 bit flips produce smaller weight perturbations, a larger bit budget is needed to induce comparable behavioral changes. This interpretation is consistent with prior work [1,2]. We will clarify this distinction in our final version.

[1] Yao et al., “DeepHammer: Depleting the Intelligence of Deep Neural Networks through Targeted Chain of Bit Flips,” USENIX (2020).

[2] Hong et al., “Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks,” USENIX (2019).

AuthorsOpenReview ↗

Rebuttal (Continued)

(Q3) Can you please explain how is the current version of the paper better in terms of recent jailbreaking attacks such as SilentStriker? Particularly, across different types of LLM tasks, how does the behavior of the model change in terms of reasoning capabilities or language modeling?

We first clarify that SilentStriker and PrisonBreak address distinct security objectives. SilentStriker targets utility degradation while preserving output naturalness, rather than jailbreaking. Notably, SilentStriker cites PrisonBreak as prior work and explicitly distinguishes these objectives [1].

Our main evaluation (Table 3) shows an average accuracy loss of only 2.26 percentage points across models and tasks.

We also ran additional experiments to show that these drops can be substantially reduced through utility-aware bit-flip selection. PrisonBreak prioritizes the bit-flips (25) that maximize ASR, but this time, we instead select the number of bit-flips that maximizes accuracy while keeping ASR within 5 percentage points of its maximum. The table below summarizes our results.

Model# Bit-FlipsASRAcc.GSM8K Acc.
Llama2-13B1885.53-4.17-12.03
1381.13-3.15-7.71
Vicuna-13B-v1.52090.57-4.09-4.60
1787.42-2.26-1.15
Qwen2-1.5B2580.50-4.09-11.44
976.73-1.41-2.43

PrisonBreak can substantially improve utility preservation with only modest reductions in ASR. Across these models, this selection criterion improves Acc. by 1.02–2.68 percentage points and reduces the GSM8K accuracy loss by 1.56–4.71$\times$, while reducing ASR by only 3.15–4.40 percentage points. Thus, the utility degradation observed in certain settings is not a limitation of PrisonBreak, but can be mitigated through a utility-aware selection criterion.

We further analyzed all 1,319 GSM8K examples for three models to characterize output quality, both without attack and under PrisonBreak. We measure (1) GSM8K accuracy, (2) average output length in tokens, and (3) seq-rep-4 from Welleck et al. [2], which measures the proportion of repeated 4-grams within each response, averaged across responses:

ModelAttackGSM8K Acc.Output Tokensseq-rep-4 (%)
Llama2-13BNo Attack35.21313.2
PrisonBreak25.61284.4
Vicuna-13B-v1.5No Attack30.91324.2
PrisonBreak24.91295.6
Qwen2-1.5BNo Attack59.71284.4
PrisonBreak44.11214.8

Despite accuracy reductions of 6.0–15.6 percentage points, average output length decreases by only 3–7 tokens, and seq-rep-4 increases by only 0.4–1.4 percentage points. Qualitative inspection shows that the additional failures often involve incorrect intermediate calculations or reasoning steps, rather than malformed or repetitive outputs. These findings suggest that utility degradation does not necessarily produce obvious output degeneration (e.g., degradation of reasoning capabilities or language modeling) that would make PrisonBreak easy to detect, even when some task-specific accuracy loss occurs.

[1] Xu et al., “SilentStriker: Toward Stealthy Bit-Flip Attacks on Large Language Models,” NeurIPS (2025).

[2] Welleck et al., “Neural Text Generation With Unlikelihood Training,” ICLR (2020).


We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.

Reviewer_pgFxOpenReview ↗

Score change

Thanks for addressing my comments. I have updated my score.

AuthorsOpenReview ↗

Thank You!

We thank the reviewer again for taking the time to carefully consider our rebuttal and for updating the assessment. We appreciate the constructive feedback and will incorporate our additional analyses and clarifications into the revised paper.

Reviewer_o9ypOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents PrisonBreak, an attack designed to bypass LLM safety alignment by flipping a small number of bits in model weights. It employs a progressive search strategy with gradient-based weight selection. The authors also incorporate a utility score to constrain bit flipping and preserve model performance. Across ten models, the authors report that PrisonBreak achieves an attack success rate (ASR) of over 80% with no more than 25 bit flips. Additionally, the authors provide an end-to-end exploitation framework to demonstrate the effectiveness of the attack on real GPU hardware.

Critical strengths
  • Reveals significant threats to LLM safety alignment by demonstrating that flipping a small number of bits can bypass safety mechanisms.
  • Conducts extensive experiments across multiple models to demonstrate the attack success rate (ASR) of PrisonBreak.
  • Provides a systematic analysis of bit locations and layer types, offering insights into the vulnerabilities of LLMs.
  • Develops an end-to-end exploitation framework to demonstrate the feasibility and effectiveness of the attack on real GPU hardware.
Critical weaknesses

Weaknesses

  • Practicality concerns: The practicality of the attack on GPUs with ECC disabled remains questionable. Additionally, frequently transferring models between CPU and GPU memory raises concerns about the feasibility of the attack for an unprivileged user.
  • Lack of comparison with runtime defenses: The paper lacks an evaluation of runtime protection mechanisms, such as simple checksum-based integrity checks, which could potentially detect bit flips.
  • Model performance degradation: The attack causes noticeable performance degradation in some cases, particularly for models such as LLaMA-2-13B.
Questions for authors
  1. Could the authors evaluate whether simple runtime protection mechanisms, such as checksum-based integrity verification, can detect the bit flips introduced by PrisonBreak?
  2. Could the authors demonstrate whether the required CPU–GPU model transfers can be performed by an unprivileged user? This would help clarify the practicality of the attack in realistic deployment scenarios.
  3. Could the authors extend the evaluation to more recent model families, such as Qwen3, Qwen3.5, and Gemma 4, to investigate whether the attack remains effective against newer LLMs?
5 comments

We thank the reviewer for their time and their positive assessment of PrisonBreak's key contributions (its characterization of a new threat to LLM safety alignment, extensive experimental evaluation, and end-to-end exploitation on real GPUs). Below we answer the questions and concerns.

(W1) The practicality of the attack on GPUs with ECC disabled remains questionable.

We appreciate the opportunity to address this concern.

(1) Our assumption that ECC is disabled reflects configurations found in real-world deployments. For example, during preliminary investigations, we observed several Runpod A6000 instances with ECC disabled and confirmed that unprivileged users cannot enable it. Moreover, recent work demonstrates Rowhammer on the consumer-grade RTX 3060 [1], which lacks configurable system-level ECC.

(2) PrisonBreak can remain effective with ECC enabled. To demonstrate this, we use GPUThor [2], a recent GPU Rowhammer attack capable of bypassing ECC, to profile vulnerable memory cells on our A6000. During profiling, we identify a four-bit flip pattern that ECC does not detect. We then run our hardware-aware search with the constraint that each iteration applies the entire observed pattern, requiring all four target bits to transition from 0→10 \rightarrow 1. Because patterns that ECC cannot detect are scarce, we set k=10,000k=10{,}000 and increase the maximum bit budget to 35. On Vicuna-7B-v1.3, PrisonBreak achieves 72.33% ASR with a Δ\DeltaAcc. of -2.94, demonstrating that enabling ECC does not necessarily mitigate PrisonBreak. We will include these results in the final version.

[1] Wan et al., “GeForge: Hammering GDDR Memory to Forge GPU Page Tables for Fun and Profit,” IEEE S&P (2026).

[2] Lin et al., “GPUThor: Amplifying Rowhammer Attacks via Non-Uniform Patterns to Exploit ECC-Protected GPUs,” ACM CCS (2026).

(W2/Q2) Frequently transferring models between CPU and GPU memory raises concerns about the feasibility of the attack for an unprivileged user.

We clarify that the CPU–GPU model transfers assumed by our threat model are performed by the platform, rather than directly by the attacker. In our threat model (Sec. 3), we assume a platform with memory swapping enabled, such as Run:AI, which automatically offloads GPU memory to CPU memory under memory oversubscription. Our attacker triggers this mechanism by allocating GPU memory from their own unprivileged process, without directly controlling the victim’s allocations or transfers.

We also clarify that the role of memory swapping is to enable the reuse of Rowhammer-vulnerable memory cells, as our profiling of the A5000 and A6000 GPUs identified only 3–10 such cells. More recent attacks, such as GPUThor [1], identify orders-of-magnitude more vulnerable cells, potentially reducing the attack’s dependence on this reuse. We will clarify the platform assumptions and the role of memory swapping in our final version.

[1] Lin et al., “GPUThor: Amplifying Rowhammer Attacks via Non-Uniform Patterns to Exploit ECC-Protected GPUs,” ACM CCS (2026).

AuthorsOpenReview ↗

Rebuttal (Continued)

(W3/Q1) The paper lacks an evaluation of runtime protection mechanisms, such as simple checksum-based integrity checks, which could potentially detect bit flips. Could the authors evaluate whether simple runtime protection mechanisms, such as checksum-based integrity verification, can detect the bit flips introduced by PrisonBreak?

To address this concern, we evaluate PrisonBreak against two simple checksum-based integrity checks: (1) additive checksums and (2) XOR checksums. For each weight matrix, both checks compute a separate checksum for every row and column, following the row-and-column organization of classical ABFT [1]. We compute these checksums over integer encodings of the weights’ raw bit patterns, enabling exact comparisons without floating-point rounding.

Both checksums detect all bit-flips made by PrisonBreak in our main evaluation. PrisonBreak tends to choose bit-flips in distinct rows/columns and with the same flip direction (0$\rightarrow$1), making both additive and XOR checksums highly effective.

However, PrisonBreak can bypass both checksums through adaptive search. We modify its search to, for each candidate bit-flip, also identify three bit-flips forming a rectangle within the weight matrix, such that each affected row and column contains two flips of the same bit, one 0→1 and one 1→0. The paired bit-flips therefore cancel in both the additive checksum (+2k2^k − 2k2^k = 0) and the XOR checksum (the bit is toggled twice). We run this adaptive search on Vicuna-7B-v1.3 for 25 iterations, achieving 88.68 ASR with +0.12 Δ\DeltaAcc., demonstrating that PrisonBreak can still compromise model alignment under these defenses.

More broadly, these results, along with our evaluation of potential countermeasures in Sec. 8, suggest that the quantity of alignment-degrading bit-flips is large enough such that attackers can simply choose bit-flips not covered by a particular defense. We will include these results and this discussion in our final version.

[1] Huang and Abraham, “Algorithm-Based Fault Tolerance for Matrix Operations,” IEEE Transactions on Computers (1984).

AuthorsOpenReview ↗

Rebuttal (Continued)

(W4) The attack causes noticeable performance degradation in some cases, particularly for models such as LLaMA-2-13B.

We acknowledge that PrisonBreak incurs accuracy drops in some settings, particularly on certain model-task combinations. However, our additional experiments show that these drops can be substantially reduced through utility-aware bit-flip selection while maintaining high attack success rates.

In particular, we modify our bit-flip selection procedure to select the number of bit-flips that maximizes Acc. while keeping ASR within 5% of its maximum. The table below reports results for the three models with the largest aggregate accuracy drops in our main results.

Model# Bit-FlipsASRAcc.GSM8K Acc.
Llama2-13B1885.53-4.17-12.03
1381.13-3.15-7.71
Vicuna-13B-v1.52090.57-4.09-4.60
1787.42-2.26-1.15
Qwen2-1.5B2580.50-4.09-11.44
976.73-1.41-2.43

These results show that PrisonBreak can substantially improve utility preservation with only modest reductions in ASR. Across these models, this selection criterion improves Acc. by 1.02–2.68 percentage points and reduces the GSM8K accuracy loss by 1.56–4.71$\times$, while reducing ASR by only 3.15–4.40 percentage points. Thus, the utility degradation observed in certain settings is not a limitation of PrisonBreak, but can be mitigated through a utility-aware selection criterion.

We further examine whether utility degradation manifests as broader output degeneration. For the three models with the largest aggregate accuracy drops in our main evaluation, we generate responses to the entire GSM8K test set (1,319 examples), both without attack and under PrisonBreak. We measure (1) GSM8K accuracy, (2) average output length in tokens, and (3) seq-rep-4 from Welleck et al. [1], which measures the proportion of repeated 4-grams within each response, averaged across responses:

ModelAttackGSM8K Acc.Output Tokensseq-rep-4 (%)
Llama2-13BNo Attack35.21313.2
PrisonBreak25.61284.4
Vicuna-13B-v1.5No Attack30.91324.2
PrisonBreak24.91295.6
Qwen2-1.5BNo Attack59.71284.4
PrisonBreak44.11214.8

Despite accuracy reductions of 6.0–15.6 percentage points, average output length decreases by only 3–7 tokens, and seq-rep-4 increases by only 0.4–1.4 percentage points. Qualitative inspection shows that the additional failures often involve incorrect intermediate calculations or reasoning steps, rather than malformed or repetitive outputs. These findings suggest that utility degradation does not necessarily produce obvious output degeneration that would make PrisonBreak easy to detect, even when some task-specific accuracy loss occurs.

[1] Welleck et al., “Neural Text Generation With Unlikelihood Training,” ICLR (2020).

(Q3) Could the authors extend the evaluation to more recent model families, such as Qwen3, Qwen3.5, and Gemma 4?

We thank the reviewer for the suggestion to broaden our evaluation. We evaluated Qwen3-8B and Gemma-4-12B(-it). Results are shown in the table below.

ModelAttackASRAcc.
Qwen3-8BNo Attack13.2160.20
PrisonBreak86.7960.83
Gemma-4-12BNo Attack4.4070.48
PrisonBreak91.8270.35

PrisonBreak remains effective against recent models: we achieve ASRs of 86.79–91.82, comparable to the models evaluated in Table 3, while reducing Acc. by at most 0.13 percentage points. We will include these results in our final version.


We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.

Reviewer_gSDvOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

PrisonBreak bypass the safety alignment of LLMs by flipping a small number of targeted bits in the model parameters. The method combines a gradient guided search, a harmful-response objective, utility filtering and hardware-aware constraints. Across ten models it reaches 80.50–98.11% attack success with only 5–25 flips in the hardware-agnostic setting. In the physical Rowhammer experiments on an A6000 GPU, the attack gets 69.18–91.19% success across five models. The authors also study transferabilty, defenses, alternative judges and an A5000 GPU.

Critical strengths

The paper actually demonstrate Rowhammer attacks on a physical GPU instead of relying only on simulated corruption. Table 5 reports 69.18–91.19% ASR across five models, and Appendix M.4 validates the attack on a second GPU.

Across the ten models, PrisonBreak achieves 89.56% average ASR vs 67.26% for Baseline BFA, while reducing the average accuracy loss from 15.48 to 2.26 percentage points.

The evaluation goes beyond the main benchmark and judge. On 313 StrongREJECT prompts PrisonBreak gets 76.68% average ASR, and it still has the highest average ASR under two additional judges across three models.

Critical weaknesses

Calling the utility loss negligible overlooks some substantial task specific degradation. Table 8 shows GSM8K accuracy drops of 12.03 percentage points for Llama 2-13B and 11.44 for Qwen 2-1.5B, which weakens the claim that the corruption preserve normal model behavior.

In Section 7.2 the authors conclude that PrisonBreak works through a fundamentally different mechanism, based on refusal-direction similarity and UMAP visualizations. But these are observational analysis and they dont establish that refusal processing stays causally intact after the corruption.

The baselines used may not be state of the art as there are many recent state of the art methods (both blackbox and whitebox). Comparing these recent approaches may make the paper more strong.

Questions for authors

How do you reconcile the claim of negligible utility loss with the double digit GSM8K accuracy drops in Table 8?

What evidence, beyond cosine similarity and UMAP, supports the conclusion that PrisonBreak preserve refusal processing while still inducing harmful responses?

Given that Ortho is included as an internal-intervention baseline, could the authors discuss how HMNS [1] relates to their approach? It would be helpful to understand whether a comparison using matched models, prompts, and judges would be informative, and how differences in access requirements might affect the feasibility and interpretation of such a comparison.

[1] Vishal Pramanik, Maisha Maliha, Susmit Jha, and Sumit Kumar Jha. Jailbreaking the Matrix: Nullspace Steering for Controlled Model Subversion. ICLR, 2026.

3 comments

We thank the reviewer for recognizing the novelty and practical significance of PrisonBreak, particularly its real-world GPU demonstrations and substantial improvements over prior bit-flip attacks. Below we address the concerns and questions.

(W1/Q1) Calling the utility loss negligible overlooks some substantial task specific degradation. How do you reconcile the claim of negligible utility loss with the double digit GSM8K accuracy drops in Table 8?

We acknowledge that “negligible utility loss” overstates our findings for certain model-task combinations. We will revise this claim to acknowledge task-specific degradation.

Importantly, the observed utility loss partly reflects our bit-flip selection criterion: we prioritize the number of bit-flips (25) that maximize ASR, then report the corresponding accuracy. We can adjust to prioritize utility. To demonstrate, we instead select the number of bit-flips that maximizes Acc. while keeping ASR within 5 percentage points of its maximum. The table below reports results for the three models with the largest aggregate accuracy drops in our main results.

Model# Bit-FlipsASRAcc.GSM8K Acc.
Llama2-13B1885.53-4.17-12.03
1381.13-3.15-7.71
Vicuna-13B-v1.52090.57-4.09-4.60
1787.42-2.26-1.15
Qwen2-1.5B2580.50-4.09-11.44
976.73-1.41-2.43

These results show that PrisonBreak can substantially improve utility preservation with only modest reductions in ASR. Across these models, this selection criterion improves Acc. by 1.02–2.68 percentage points and reduces the GSM8K accuracy loss by 1.56–4.71$\times$, while reducing ASR by only 3.15–4.40 percentage points. Thus, the utility degradation observed in certain settings is not a limitation of PrisonBreak, but can be mitigated through a utility-aware selection criterion.

We further examine whether utility degradation manifests as broader output degeneration. For the three models with the largest aggregate accuracy drops in our main evaluation, we generate responses to the entire GSM8K test set (1,319 examples), both without attack and under PrisonBreak. We measure (1) GSM8K accuracy, (2) average output length in tokens, and (3) seq-rep-4 from Welleck et al. [1], which measures the proportion of repeated 4-grams within each response, averaged across responses. Results are shown below.

ModelAttackGSM8K Acc.Output Tokensseq-rep-4 (%)
Llama2-13BNo Attack35.21313.2
PrisonBreak25.61284.4
Vicuna-13B-v1.5No Attack30.91324.2
PrisonBreak24.91295.6
Qwen2-1.5BNo Attack59.71284.4
PrisonBreak44.11214.8

Despite accuracy reductions of 6.0–15.6 percentage points, average output length decreases by only 3–7 tokens, and seq-rep-4 increases by only 0.4–1.4 percentage points. Qualitative inspection shows that the additional failures often involve incorrect intermediate calculations or reasoning steps, rather than malformed or repetitive outputs. These findings suggest that utility degradation does not necessarily produce obvious output degeneration that would make PrisonBreak easy to detect, even when some task-specific accuracy loss occurs.

[1] Welleck et al., “Neural Text Generation With Unlikelihood Training,” ICLR (2020).

AuthorsOpenReview ↗

Rebuttal (Continued)

(W2/Q2) The cosine similarity and UMAP experiments in Section 7.2 are observational and don't establish that refusal processing stays causally intact after the corruption. What additional evidence supports the conclusion that PrisonBreak preserve refusal processing while still inducing harmful responses?

To address the concern, we conducted two activation intervention experiments to test whether PrisonBreak relies on refusal-feature suppression or a distinct mechanism. We used Llama3-8B and all 159 HarmBench prompts from our main evaluation; each experiment intervenes at every Transformer layer and every token position, matching Ortho's layerwise removal of the refusal feature.

(1) PrisonBreak does not rely on refusal-feature suppression. At each generation step, we recomputed a clean-model forward pass on the identical (attacked) prefix and restored only the attacked model's projection onto the refusal direction to the clean model's projection, leaving all orthogonal activation components unchanged. This substantially reduced Ortho's ASR (90.57%→5.66%) but had little effect on PrisonBreak's (72.96%→74.21%). A matched random-direction control, patched identically, leaves both attacks' ASR largely unaffected (Ortho: 92.45%, PrisonBreak: 72.96%). Thus, Ortho's success is causally mediated by refusal-feature suppression, whereas PrisonBreak's is not.

(2) PrisonBreak amplifies activations associated with harmful-answer continuation. Using 41 held-out HarmBench prompts, we constructed a per-layer direction contrasting harmful-answer and refusal continuations and orthogonalized it against the refusal direction. Restoring only this coordinate to its clean-model value reduced PrisonBreak's ASR (72.96%→28.93%) but had a much smaller effect on Ortho's (90.57%→81.13%); a matched random orthogonal-direction control had little effect (Ortho: 93.71%, PrisonBreak: 74.84%). Since this direction is orthogonal to the refusal direction by construction, the refusal coordinate is unaffected. This is consistent with our hypothesis that PrisonBreak’s effect is mediated by activations associated with continuing a harmful answer, rather than suppressing the model's representation of harmful intent.

We will include these results and clarify the scope of our mechanistic claims in our final version.

(W3/Q3) The baselines used may not be state of the art. Could the authors discuss how HMNS relates to their approach?

We note that recent white-box and black-box jailbreak methods do not operate under our threat model and therefore serve as contextual comparisons rather than baselines under the same constraints. To our knowledge, PrisonBreak is the first bit-flip attack on LLMs targeting jailbreaking. We therefore adapted prior bit-flip attack methods as our primary baseline. Our inclusion of GCG and Ortho serves to provide additional context through comparisons with popular prompt-based and parameter-based jailbreaking techniques, rather than to imply they are practical under our threat model. We will clarify these distinctions in our final version.

We thank the reviewer for pointing out HMNS. While a comparison could help contextualize attack success, HMNS requires interventions unavailable to our attacker (i.e., attention-head output masking and activation steering). Thus, the comparison would not evaluate jailbreaking effectiveness under our assumed threat model. We will discuss HMNS’s capabilities and threat-model differences in our related work.


We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer gSDv,

Thank you again for your time and thoughtful review. As the discussion period is coming to a close, we would like to kindly check whether our response has addressed your concerns. If you have any further questions or comments, we would be happy to address them.

Best regards,

The Authors of Submission 35246

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.