- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a bit-flip attack framework to demonstrate that the behavior of the models can be altered with alterations in the model weights, with the methodology being input agnostic. The paper demonstrates that 5-25 bit-flips can be used for a white-box and black-box attack.
Critical strengths
- The paper presents an input-agnostic approach for jailbreaking LLMs with an end-to-end validation using a Rowhammer-based fault injection.
Critical weaknesses
- The results suggest that the bit locations corresponding to the MSB are more critical for jailbreaking LLMs. However, as shown in ResCiM (TCASAI 2026) and Unicorn-CiM (ISLPED 2026), bit-flips in the exponents of the operands can lead to triggering the compute result to be NaN or Inf. Thus, the jailbreaking could be easily detected. How is the algorithm ensuring that this kind of compute failure is avoided during the jailbreak? Can the authors quantify the rate of occurrence of such failures during inference or candidate selection?
- How generalizable is the robustness observed for INT8 in Appendix C? Is the robustness observed due to the attack strategy, or is it due to the representation itself?
- Can you please explain how is the current version of the paper better in terms of recent jailbreaking attacks such as SilentStriker (NeurIPS 2025)? Particularly, across different types of LLM tasks, how does the behavior of the model change in terms of reasoning capabilities or language modeling?
Questions for authors
N/A
4 comments
Rebuttal (Continued)
(Q3) Can you please explain how is the current version of the paper better in terms of recent jailbreaking attacks such as SilentStriker? Particularly, across different types of LLM tasks, how does the behavior of the model change in terms of reasoning capabilities or language modeling?
We first clarify that SilentStriker and PrisonBreak address distinct security objectives. SilentStriker targets utility degradation while preserving output naturalness, rather than jailbreaking. Notably, SilentStriker cites PrisonBreak as prior work and explicitly distinguishes these objectives [1].
Our main evaluation (Table 3) shows an average accuracy loss of only 2.26 percentage points across models and tasks.
We also ran additional experiments to show that these drops can be substantially reduced through utility-aware bit-flip selection. PrisonBreak prioritizes the bit-flips (25) that maximize ASR, but this time, we instead select the number of bit-flips that maximizes accuracy while keeping ASR within 5 percentage points of its maximum. The table below summarizes our results.
| Model | # Bit-Flips | ASR | Acc. | GSM8K Acc. |
|---|---|---|---|---|
| Llama2-13B | 18 | 85.53 | -4.17 | -12.03 |
| 13 | 81.13 | -3.15 | -7.71 | |
| Vicuna-13B-v1.5 | 20 | 90.57 | -4.09 | -4.60 |
| 17 | 87.42 | -2.26 | -1.15 | |
| Qwen2-1.5B | 25 | 80.50 | -4.09 | -11.44 |
| 9 | 76.73 | -1.41 | -2.43 |
PrisonBreak can substantially improve utility preservation with only modest reductions in ASR. Across these models, this selection criterion improves Acc. by 1.02–2.68 percentage points and reduces the GSM8K accuracy loss by 1.56–4.71$\times$, while reducing ASR by only 3.15–4.40 percentage points. Thus, the utility degradation observed in certain settings is not a limitation of PrisonBreak, but can be mitigated through a utility-aware selection criterion.
We further analyzed all 1,319 GSM8K examples for three models to characterize output quality, both without attack and under PrisonBreak. We measure (1) GSM8K accuracy, (2) average output length in tokens, and (3) seq-rep-4 from Welleck et al. [2], which measures the proportion of repeated 4-grams within each response, averaged across responses:
| Model | Attack | GSM8K Acc. | Output Tokens | seq-rep-4 (%) |
|---|---|---|---|---|
| Llama2-13B | No Attack | 35.2 | 131 | 3.2 |
| PrisonBreak | 25.6 | 128 | 4.4 | |
| Vicuna-13B-v1.5 | No Attack | 30.9 | 132 | 4.2 |
| PrisonBreak | 24.9 | 129 | 5.6 | |
| Qwen2-1.5B | No Attack | 59.7 | 128 | 4.4 |
| PrisonBreak | 44.1 | 121 | 4.8 |
Despite accuracy reductions of 6.0–15.6 percentage points, average output length decreases by only 3–7 tokens, and seq-rep-4 increases by only 0.4–1.4 percentage points. Qualitative inspection shows that the additional failures often involve incorrect intermediate calculations or reasoning steps, rather than malformed or repetitive outputs. These findings suggest that utility degradation does not necessarily produce obvious output degeneration (e.g., degradation of reasoning capabilities or language modeling) that would make PrisonBreak easy to detect, even when some task-specific accuracy loss occurs.
[1] Xu et al., “SilentStriker: Toward Stealthy Bit-Flip Attacks on Large Language Models,” NeurIPS (2025).
[2] Welleck et al., “Neural Text Generation With Unlikelihood Training,” ICLR (2020).
We hope our responses address the reviewer's questions and concerns. We would be happy to provide further clarification if needed.
Score change
Thanks for addressing my comments. I have updated my score.
Thank You!
We thank the reviewer again for taking the time to carefully consider our rebuttal and for updating the assessment. We appreciate the constructive feedback and will incorporate our additional analyses and clarifications into the revised paper.
Rebuttal
We thank the reviewer for recognizing our contribution to demonstrating input-agnostic jailbreaking through real-world Rowhammer exploitation. We address the questions and concerns below.
During our search, we explicitly reject candidate bit-flips that would produce non-finite weight values (NaN or Inf). We additionally verified that none of the selected bit-flip sequences produced in our evaluation introduce NaN or Inf values.
To further examine whether finite-weight perturbations could still cause numerical failures during inference, we re-ran our HarmBench evaluation on four models (Vicuna-7B-v1.5, Llama3-8B, Tulu3-8B, and Qwen2-7B). We counted non-finite activations both without attack and under PrisonBreak. Across all four models, we observe zero non-finite activations in either condition. Thus, PrisonBreak successfully compromises model alignment without producing non-finite weights or activations, meaning NaN/Inf-based checks alone would not detect our attack.
To assess the generalizability of our INT8 results, we evaluated an additional model, Llama3-8B, and increased the attack budget from 25 to 50 bit flips. Below, we report ASR for Llama3-8B and Vicuna-7B-v1.3 across bit-flip budgets.
INT8 models remain less susceptible to PrisonBreak in these experiments. For Llama3-8B, ASR reaches only 18.24% with 50 bit-flips, compared to 86.79% for FP16. However, ASR generally continues to rise with the bit-flip budget. This indicates that PrisonBreak is not hitting a search barrier; rather, because INT8 bit flips produce smaller weight perturbations, a larger bit budget is needed to induce comparable behavioral changes. This interpretation is consistent with prior work [1,2]. We will clarify this distinction in our final version.
[1] Yao et al., “DeepHammer: Depleting the Intelligence of Deep Neural Networks through Targeted Chain of Bit Flips,” USENIX (2020).
[2] Hong et al., “Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks,” USENIX (2019).