- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies four approaches for jailbreak attack strategy scheduling from a fixed attack library, namely ASR-Sort, LLM-Guided Sort, Thompson Sampling, and LLM-Guided Online. These strategies are then evaluated in turn against 3 target models and 3 target defenses in order to determine their relative strengths. The paper argues that the results reveal complementary roles for the three evaluated signals, improving budget-limited ASR.
Critical strengths
- The problem the paper aims to address is indeed important. Autonomous red-teaming strategies do increasingly rely on sampling from attack libraries, and a comprehensive study of which ordering strategies work best may beneficial to the community.
- The paper is generally clear and well-presented
Critical weaknesses
- The current experimentation is not sufficient to enable an evaluation of the proposed methods. More benchmarks, samples, target models (particularly stronger models), and reruns are needed, as current cross-method differences cannot be confidently differentiated from noise, and it is impossible to identify key trends.
- The paper should more clearly insert itself into the existing literature. Comparison of the proposed methods against the existing baseline in AutoRedTeamer and an experiment that inserts the proposed methods into existing frameworks that rely on selection from attack strategy pools would increase the significance of the results.
- Some key technical details are missing from the paper. For example, how are each of the attacks implemented? Is one attempt one attempted strategy, and if so are budget limits placed on queries within each individual strategy?
Questions for authors
See the weaknesses.
4 comments
Response to Reviewer 33za (continued)
Isolating and evaluating AutoRedTeamer's scheduling policy. To enable a fair comparison, we reproduce AutoRedTeamer's scheduling policy based on its published methodology and evaluate it under our unified setting. Since its prompt-level composition mechanism cannot accommodate all attacks, we adopt a selection-only adaptation, preserving each attack's original execution procedure in Jailbreak Foundry.
Additional comparison (Success@5, %).
| Scheduling Policy | DeepSeek-V4-Pro | Gemma-4-26B | GPT-5.2 | Backtranslation | SmoothLLM | LlamaGuard |
|---|---|---|---|---|---|---|
| ASR-Sort | 99.0 | 90.0 | 77.0 | 62.0 | 51.0 | 19.0 |
| LLM-Guided Sort | 99.0 | 90.0 | 79.0 | 67.0 | 54.0 | 28.0 |
| Thompson Sampling | 99.0 | 87.0 | 77.6 | 62.2 | 54.8 | 25.4 |
| LLM-Guided Online | 99.0 | 90.0 | 79.8 | 67.0 | 55.0 | 34.6 |
| AutoRedTeamer (Adapted) | 99.0 | 90.0 | 77.0 | 62.0 | 51.0 | 26.0 |
The three defense columns correspond to GPT-5.2.
Analysis. Both AutoRedTeamer and LLM-Guided Online leverage historical feedback for adaptive attack selection. However, our approach additionally incorporates explicit semantic guidance to improve strategy selection, particularly when historical feedback is limited or unreliable. The adapted AutoRedTeamer achieves performance comparable to ASR-Sort in most settings, while LLM-Guided Online consistently matches or outperforms it, with notable improvements on GPT-5.2 (77.0% → 79.8%) and LlamaGuard (26.0% → 34.6%). These results suggest that historical feedback alone provides limited additional benefits, whereas combining semantic guidance with online adaptation enables more effective budget-constrained attack scheduling.
3. On Implementation and Budget Details.
Some key technical details are missing from the paper. For example, how are each of the attacks implemented? Is one attempt one attempted strategy, and if so are budget limits placed on queries within each individual strategy?
We thank the reviewer for highlighting the need for implementation details. We adopt Jailbreak Foundry (line 273 in the main text), a standardized framework that faithfully reproduces 30 jailbreak attacks, achieving a mean ASR deviation of only +0.26 percentage points from their originally reported results. Importantly, each attack follows its original methodology and internal execution procedure, rather than being simplified for our scheduling framework. Each attempt corresponds to executing one complete attack strategy, with its internal query configuration preserved. The budget (K) limits the number of strategies attempted per query, rather than the internal queries required by each strategy. We will clarify these details in the revision.
Reference
[1] Zhou et al., AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, NIPS 2025
[2] Fang et al., Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking, ICML 2026 Oral
I thank the authors for their detailed response. However, my main concerns regarding the current experimentation have not been alleviated. Accordingly, I will maintain my original score.
Request for Further Clarification on the Remaining Concerns
Thank you for your response. We respect your decision, but we would appreciate more specific clarification regarding which aspects of our revisions remain insufficient.
In our previous response, we directly addressed each of the three concerns raised in your original review:
-
Experimental reliability: We clarified the rationale behind our victim model selection, expanded the experimental analysis across different defense conditions, and reported results across five matched query-order permutations to examine sensitivity to evaluation order.
-
Comparison with prior work: We implemented an adaptation of AutoRedTeamer's scheduling policy under our unified evaluation setting and provided additional quantitative comparisons across all three victim models and three defense mechanisms.
-
Implementation details: We clarified how the 30 attacks are executed, what constitutes an individual attempt, and how the overall query budget relates to the internal queries of each attack.
We fully acknowledge that additional experiments can always strengthen an empirical study. However, your latest response does not identify which of these concerns remains unresolved, why the additional evidence is insufficient, or what specific aspects of the experimental design undermine the validity of our findings.
Could you please clarify which particular experimental limitations still concern you? For example, is the concern primarily about the number of victim models, the statistical robustness of the results, the validity of the AutoRedTeamer comparison, or another aspect of the experimental methodology?
We would genuinely welcome more concrete feedback, as it would help us understand your assessment and make meaningful improvements to the work. We hope our revisions can be evaluated based on the additional evidence provided, rather than leaving the original concerns unresolved without further clarification. Thank you.
Clarifying the Experimental Design and Comparison with AutoRedTeamer
Thank you for your thoughtful and constructive feedback. We believe that several concerns can be addressed through further clarification and supporting evidence, which we provide below. We sincerely hope our responses will resolve your concerns and encourage you to reconsider your evaluation more favorably.
1. On Experimental Reliability.
We thank the reviewer for this important concern. Our experimental setup is deliberately designed to examine systematic trends across varying attack difficulties, rather than relying on arbitrary model selection.
Experimental coverage. We select three victim models, DeepSeek-V4-Pro, Gemma-4-26B, and GPT-5.2, spanning progressively stronger resistance to jailbreak attacks, as measured by their attack success rates. We adopt 30 high-fidelity reproduced attacks from Jailbreak Foundry (ICML 2026 Oral) [2] and evaluate them on JailbreakBench-100 under native safeguards and three representative defense mechanisms. This controlled setup enables us to systematically examine how scheduling strategies behave across different victim strengths and defense conditions.
Consistent empirical trends. Our results reveal clear patterns rather than isolated improvements: (1) historical priors substantially improve initialization across all three victim models; (2) semantic guidance provides additional benefits when historical priors are unreliable; and (3) online feedback becomes increasingly valuable under distribution shifts.
Robustness to randomness. We further evaluate online strategies across five matched query-order permutations, demonstrating consistent trends across different evaluation orders. These results support the reliability of our main findings, which concern the complementary roles of different scheduling signals rather than marginal performance differences between individual methods.
In summary, we emphasize that representative coverage and consistent empirical trends matter more than simply increasing the number of victim models. Our controlled evaluation across different attack difficulties and defense conditions reveals systematic patterns that support our central findings.
2. On Comparisons with AutoRedTeamer.
Reproducibility issues. To the best of our knowledge, AutoRedTeamer [1] has not released its official implementation. Although a third-party reproduction is available, it lacks substantial coverage of attack strategies and omits critical implementation details, making faithful reproduction and rigorous comparison challenging. We believe that reproducibility is a fundamental requirement for rigorous and reliable LLM red-teaming research.
Methodological differences. AutoRedTeamer and our work differ fundamentally in their problem formulations and scheduling mechanisms. AutoRedTeamer focuses on memory-guided attack selection and prompt-level composition, whereas modern jailbreak attacks extend beyond prompt transformations, often involving iterative optimization, multi-turn interactions, and complex execution procedures. Consequently, its composition mechanism cannot be directly applied to our heterogeneous attack library without additional adaptations. Moreover, the authors acknowledge limitations in LLM-based attack implementation and strategy proposal (Sec. 5). In contrast, our work focuses on budget-constrained scheduling of complete, independently executable attack strategies, preserving their original execution procedures. Thus, the two frameworks address related but distinct problems and are not directly interchangeable.