- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes SimJIR, a simple black-box method for jailbreaking aligned LLMs without external attacker models or curated harmful demonstrations. It uses the model's previous refusals as negative examples and provides reward feedback across multiple turns, achieving competitive results at a relatively low cost. However, I have some concerns about the novelty, experimental fairness, generalizability, and overall figure quality. I would be willing to raise my score if the authors could substantially improve the figures.
Critical strengths
I) The method is simple and only uses fixed prompts and the target model’s conversation history, without needing extra attacker models or harmful examples. II) Demonstrating attack effectiveness across multiple model families, this efficiency result holds practical significance for black-box, multi-turn attacks.
Critical weaknesses
I) The novelty and underlying mechanism are not fully convincing. SimJIR shares ideas with existing methods, and Table 4 even shows higher ASR without history on some models. II) The experimental comparison may not be entirely fair. SimJIR uses the same judge for attack termination and evaluation, while the cost analysis only considers successful attacks. III) The generalization claims seem too broad. Only two defenses are tested, one without an official implementation, and the multimodal evaluation is limited to a small TYPO subset. IV) The figures and overall layout need more work. Several figures are crowded, visually inconsistent, and show noticeable AI-generated styling, which hurts the paper's presentation quality.
Questions for authors
I) In Table 4, removing history actually improves ASR for some models. Could you explain why this happens and how it supports the claimed mechanism of SimJIR? II) Could you clarify whether judge reuse, success-only cost statistics, and disabling PANDAS Adaptive Sampling affect the fairness of the comparison? III) Given the limited defense and multimodal experiments, could you clarify how broadly the findings can be generalized? IV) Could you improve Figures 1, 2, and 4 by making the visual style more consistent and reducing the noticeable AI-generated and template-like elements?
5 comments
Response to Reviewer (2 of 3)
Comment 2. How do (1) reusing the same judge for attack termination and evaluation, (2) cost statistics restricted to successful attacks, and (3) disabling PANDAS Adaptive Sampling affect the fairness of the comparison?
Response. To directly address this question, we present additional results which indicate that (1) an independent evaluation judge leaves the ASR-cost tradeoff unchanged and (3) enabling Adaptive Sampling leaves the ranking of PANDAS relative to SimJIR unchanged, while (2) success-only cost is the measure that remains comparable across attacks.
First, for (1), SimJIR uses Llama Guard 3 as its inner judge, Crescendo [3] uses GPT-4o mini, and the ASRs of all methods are evaluated with Llama Guard 3 for a consistent comparison. This means SimJIR's inner judge is the same as the evaluation judge. To check whether this favors SimJIR, the revised Appendix A.10 re-evaluates the ASRs of SimJIR and Crescendo with an independent evaluation judge, Qwen3Guard-Gen-8B [5], while each method keeps its inner judge. The ASRs of both methods drop, which indicates that Qwen3Guard is a stricter judge than Llama Guard 3. Nevertheless, Fig. 10 demonstrates that the ASR-cost tradeoff stays the same: SimJIR reaches ASR comparable to Crescendo on most models while needing far fewer turns to jailbreak. Using different inner and evaluation judges therefore does not affect our conclusions.
Second, for (2), we would like to kindly remind the reviewer that Fig. 2 and Section 4.2.2 report mean input tokens and turns over the successfully jailbroken queries only. Unsuccessful queries are excluded since (i) their cost is already reflected in lower ASRs, and (ii) the number of tokens and turns they consume is fixed by the budget set by the attacker, which follows each method's default for faithful reproduction and is not comparable across attacks.
Third, for (3), our PANDAS baseline disables Adaptive Sampling (AS) since AS costs many GPU hours and the PANDAS authors [2] report marginal ASR gains from it. To check its effect, we reran PANDAS with AS on AdvBench50 with one seed. Table R1 below reports the ASRs (%), where the "PANDAS (no AS)" and SimJIR columns come from Table 1 of the original submission.
Table R1. ASR (%) of PANDAS with and without AS on AdvBench50.
| Model | PANDAS (no AS) | PANDAS (with AS) | SimJIR |
|---|---|---|---|
| Llama-3.1-8B-Instruct | 54.0 | 60.0 | 94.0 |
| Qwen2.5-7B-Instruct | 4.0 | 10.0 | 88.0 |
| Qwen3-8B | 8.7 | 12.0 | 70.7 |
| GLM-4-9B-Chat | 46.0 | 46.0 | 16.0 |
| GLM-4.1V-9B-Thinking | 20.7 | 20.0 | 78.0 |
| GLM-4.6V-Flash | 1.3 | 0.0 | 100.0 |
| Gemma-3-4B-IT | 12.0 | 4.0 | 90.0 |
| Gemma-4-E4B-IT | 0.7 | 0.0 | 66.7 |
AS changes the ASR of PANDAS by between −8 and +6 points and does not change its ranking relative to SimJIR on any model. Disabling AS therefore does not affect the conclusions of our comparison.
Response to Reviewer (3 of 3)
Comment 3. (1) How broadly do the defense findings generalize, given that two defenses are tested and one of them lacks an official implementation? (2) How broadly do the multimodal findings generalize beyond the TYPO subset?
Response. We would like to bring to the attention of the reviewer that (1) Section 3.4 of the original submission identifies two parts of the SimJIR prompt that guide edits against a defense, and the two tested defenses demonstrate one of each, and (2) the revised multimodal evaluation adds an image-text benchmark and an audio benchmark, on which SimJIR raises ASR for every model.
First, for (1), the two parts are the isolated harmful query and the reward-maximization task, and Table 2 demonstrates an edit to each. Our NBF-LLM [6] bypass modifies only the harmful query, and our SafeEnd [7] bypass extends the reward task so that the safety demonstration of SafeEnd becomes part of the task. Both are recent defenses designed against multi-turn and many-shot jailbreaks, and neither had a published bypass. Since SimJIR is a multi-turn attack whose refusal history acts as demonstrations, these two defenses are the most directly relevant, and each bypass edits one part of the prompt. For NBF-LLM, we use the official code and pretrained weights. The official repository of SafeEnd contains no source code, and its authors did not respond to our inquiry. We therefore reimplemented SafeEnd and document the reproduction in Appendix A.8.
Second, for (2), Section 4.2.4 and Table 3 of the revised PDF add two benchmarks: HADES-200, with 200 harmful queries paired with generated images rather than typographic text, and JALMBench, with 246 spoken harmful queries. JALMBench is evaluated on Gemma-4-E4B-IT, the only model in our list that accepts audio input. The following table reproduces the revised Table 3 together with the original TYPO results.
Table 3 (revised PDF). ASR (%) with and without SimJIR on the multimodal benchmarks.
| Benchmark | Model | No Attack ASR (%) | SimJIR ASR (%) |
|---|---|---|---|
| TYPO | GLM-4.1V-9B-Thinking | 63.9 | 81.6 |
| TYPO | GLM-4.6V-Flash | 59.2 | 81.6 |
| TYPO | Gemma-3-4B-IT | 30.6 | 80.0 |
| TYPO | Gemma-4-E4B-IT | 5.9 | 71.8 |
| HADES-200 | GLM-4.1V-9B-Thinking | 64.5 | 90.0 |
| HADES-200 | GLM-4.6V-Flash | 52.0 | 85.5 |
| HADES-200 | Gemma-3-4B-IT | 30.0 | 87.5 |
| HADES-200 | Gemma-4-E4B-IT | 10.0 | 69.5 |
| JALMBench | Gemma-4-E4B-IT | 2.0 | 53.7 |
SimJIR raises ASR on every model and benchmark without any algorithm change, with the image or audio passed in the first turn. Its effectiveness therefore holds across typographic images, generated images, and speech.
Comment 4. Could Figs. 1, 2, and 4 be made less crowded and more consistent in visual style, with fewer AI-generated and template-like elements?
Response. The revised PDF redraws Figs. 1, 2, and 4 with simpler annotations and a reorganized cost presentation. Specifically, Fig. 1 replaces the pictorial badges with plain-text labels and uses one color, matching the Qwen logo, for all target-model turns. Fig. 2 replaces the dual-axis chart with a semi-log scatter plot of input tokens against ASR, and the component-level token breakdown moves to Appendix A.5. Fig. 4 moves the legend from the caption into the figure, and its caption now states the takeaway. The revised layout also increases the spacing between Figs. 1 and 2.
We hope that these revisions and results answer the questions of the reviewer.
[1] Anil et al. Many-shot jailbreaking. NeurIPS, 2024.
[2] Ma et al. PANDAS: Improving many-shot jailbreaking via positive affirmation, negative demonstration, and adaptive sampling. ICML, 2025.
[3] Russinovich et al. Great, now write an article about that: The Crescendo Multi-Turn LLM Jailbreak Attack. USENIX Security, 2025.
[4] TheBurninator99. Presenting DAN 6.0. Reddit, 2023.
[5] Qwen Team. Qwen3Guard Technical Report. arXiv:2510.14276, 2025.
[6] Hu et al. Steering dialogue dynamics for robustness against multi-turn jailbreaking attacks. TMLR, 2026.
[7] Chen et al. Mitigating many-shot jailbreak attacks with one single demonstration. arXiv:2605.08277, 2026.
Response to Reviewer (1 of 3)
We thank the reviewer for the feedback on SimJIR.
Response. (1) To our knowledge, SimJIR is the first attack that keeps the target's own refusals in context as on-policy negative demonstrations and the first to use in-context rewards as a jailbreak signal, and (2) removing history raises ASR only on the two reasoning models, which infer the past refusal from the SimJIR prompt itself and therefore still act on the same mechanism.
First, for (1), the Introduction and Related Work of the original submission separate SimJIR from three groups of related methods. MSJ [1] and PANDAS [2] rely on curated, off-policy harmful demonstrations. Crescendo [3] requires an attacker LLM and removes refusals from the target's context. The token system of DAN 6.0 [4] resembles a reward in spirit, although Appendix A.7 finds no evidence that it raises ASR. In contrast, SimJIR uses only a fixed prompt and the target's own conversation history.
Second, for (2), Table 4 reports that removing history lowers ASR on AdvBench50 for three non-reasoning models (Llama-3.1-8B-Instruct: 94.0 → 12.0, Qwen2.5-7B-Instruct: 88.0 → 4.0, Gemma-3-4B-IT: 90.0 → 40.0) and raises it for two reasoning models (Gemma-4-E4B-IT: 66.7 → 85.3, GLM-4.1V-9B-Thinking: 78.0 → 86.7). The SimJIR prompt works as feedback on a past refusal. Without history, the non-reasoning models see no past refusal to improve on, so they treat the prompt as a standalone harmful request and refuse. In contrast, the reasoning models appear to infer the past refusal from the prompt itself. For example, a reasoning trace of GLM-4.1V-9B-Thinking reads: "my previous response was refusing, which got -1. Now I need to provide a response that's better". Since the past refusal is implied by the prompt rather than present in context, these models still try to improve on an implied negative demonstration, which is a core mechanism of SimJIR.