acceptodds
Under review as a conference paper at ICLR 2027

Budgeted Attack Scheduling for LLM Red-Teaming: The Roles of Static Priors, Semantic Guidance, and Online Feedback

Abstract

As jailbreak attacks against large language models (LLMs) become increasingly diverse, red-teaming has come to rely on large libraries of jailbreak prompts and attack strategies to evaluate the robustness of victim models and their defense settings. However, attack effectiveness is highly condition-dependent: an attack that succeeds under one victim-defense setting may fail under another, while exhaustively evaluating the full attack library for each setting incurs substantial token and monetary costs. This motivates studying how jailbreak attacks should be prioritized under a limited attempt budget for efficient robustness evaluation. In this work, we formalize this problem as budgeted attack scheduling over a fixed attack library and investigate three complementary signals under a fixed victim–defense condition: (1) static historical attack success rate (ASR) priors; (2) semantic guidance from LLMs; and (3) online feedback from earlier queries. Building on these signals, we design four scheduling strategies and evaluate them across three victim models and three defense settings on GPT-5.2. Extensive experiments clearly characterize the conditions under which each signal is most effective. Historical ASR provides a strong initialization, but its reliability degrades under defense-induced distribution shifts. Semantic guidance consistently improves or matches static prior-based scheduling in the cold-start regime, while online feedback becomes particularly valuable when transferred priors become stale. Semantic guidance and online feedback can also partially compensate for missing historical priors. Taken together, these findings characterize budgeted red-teaming as an attack scheduling problem in which historical priors, semantic guidance, and online feedback play complementary roles.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, red-teaming, jailbreak attacks, budgeted attack scheduling, online learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_33zaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies four approaches for jailbreak attack strategy scheduling from a fixed attack library, namely ASR-Sort, LLM-Guided Sort, Thompson Sampling, and LLM-Guided Online. These strategies are then evaluated in turn against 3 target models and 3 target defenses in order to determine their relative strengths. The paper argues that the results reveal complementary roles for the three evaluated signals, improving budget-limited ASR.

Critical strengths
  • The problem the paper aims to address is indeed important. Autonomous red-teaming strategies do increasingly rely on sampling from attack libraries, and a comprehensive study of which ordering strategies work best may beneficial to the community.
  • The paper is generally clear and well-presented
Critical weaknesses
  • The current experimentation is not sufficient to enable an evaluation of the proposed methods. More benchmarks, samples, target models (particularly stronger models), and reruns are needed, as current cross-method differences cannot be confidently differentiated from noise, and it is impossible to identify key trends.
  • The paper should more clearly insert itself into the existing literature. Comparison of the proposed methods against the existing baseline in AutoRedTeamer and an experiment that inserts the proposed methods into existing frameworks that rely on selection from attack strategy pools would increase the significance of the results.
  • Some key technical details are missing from the paper. For example, how are each of the attacks implemented? Is one attempt one attempted strategy, and if so are budget limits placed on queries within each individual strategy?
Questions for authors

See the weaknesses.

4 comments
AuthorsOpenReview ↗

Clarifying the Experimental Design and Comparison with AutoRedTeamer

Thank you for your thoughtful and constructive feedback. We believe that several concerns can be addressed through further clarification and supporting evidence, which we provide below. We sincerely hope our responses will resolve your concerns and encourage you to reconsider your evaluation more favorably.

1. On Experimental Reliability.

The current experimentation is not sufficient to enable an evaluation of the proposed methods. More benchmarks, samples, target models (particularly stronger models), and reruns are needed, as current cross-method differences cannot be confidently differentiated from noise, and it is impossible to identify key trends.

We thank the reviewer for this important concern. Our experimental setup is deliberately designed to examine systematic trends across varying attack difficulties, rather than relying on arbitrary model selection.

Experimental coverage. We select three victim models, DeepSeek-V4-Pro, Gemma-4-26B, and GPT-5.2, spanning progressively stronger resistance to jailbreak attacks, as measured by their attack success rates. We adopt 30 high-fidelity reproduced attacks from Jailbreak Foundry (ICML 2026 Oral) [2] and evaluate them on JailbreakBench-100 under native safeguards and three representative defense mechanisms. This controlled setup enables us to systematically examine how scheduling strategies behave across different victim strengths and defense conditions.

Consistent empirical trends. Our results reveal clear patterns rather than isolated improvements: (1) historical priors substantially improve initialization across all three victim models; (2) semantic guidance provides additional benefits when historical priors are unreliable; and (3) online feedback becomes increasingly valuable under distribution shifts.

Robustness to randomness. We further evaluate online strategies across five matched query-order permutations, demonstrating consistent trends across different evaluation orders. These results support the reliability of our main findings, which concern the complementary roles of different scheduling signals rather than marginal performance differences between individual methods.

In summary, we emphasize that representative coverage and consistent empirical trends matter more than simply increasing the number of victim models. Our controlled evaluation across different attack difficulties and defense conditions reveals systematic patterns that support our central findings.

2. On Comparisons with AutoRedTeamer.

The paper should more clearly insert itself into the existing literature. Comparison of the proposed methods against the existing baseline in AutoRedTeamer and an experiment that inserts the proposed methods into existing frameworks that rely on selection from attack strategy pools would increase the significance of the results.

Reproducibility issues. To the best of our knowledge, AutoRedTeamer [1] has not released its official implementation. Although a third-party reproduction is available, it lacks substantial coverage of attack strategies and omits critical implementation details, making faithful reproduction and rigorous comparison challenging. We believe that reproducibility is a fundamental requirement for rigorous and reliable LLM red-teaming research.

Methodological differences. AutoRedTeamer and our work differ fundamentally in their problem formulations and scheduling mechanisms. AutoRedTeamer focuses on memory-guided attack selection and prompt-level composition, whereas modern jailbreak attacks extend beyond prompt transformations, often involving iterative optimization, multi-turn interactions, and complex execution procedures. Consequently, its composition mechanism cannot be directly applied to our heterogeneous attack library without additional adaptations. Moreover, the authors acknowledge limitations in LLM-based attack implementation and strategy proposal (Sec. 5). In contrast, our work focuses on budget-constrained scheduling of complete, independently executable attack strategies, preserving their original execution procedures. Thus, the two frameworks address related but distinct problems and are not directly interchangeable.

AuthorsOpenReview ↗

Response to Reviewer 33za (continued)

Isolating and evaluating AutoRedTeamer's scheduling policy. To enable a fair comparison, we reproduce AutoRedTeamer's scheduling policy based on its published methodology and evaluate it under our unified setting. Since its prompt-level composition mechanism cannot accommodate all attacks, we adopt a selection-only adaptation, preserving each attack's original execution procedure in Jailbreak Foundry.

Additional comparison (Success@5, %).

Scheduling PolicyDeepSeek-V4-ProGemma-4-26BGPT-5.2BacktranslationSmoothLLMLlamaGuard
ASR-Sort99.090.077.062.051.019.0
LLM-Guided Sort99.090.079.067.054.028.0
Thompson Sampling99.087.077.662.254.825.4
LLM-Guided Online99.090.079.867.055.034.6
AutoRedTeamer (Adapted)99.090.077.062.051.026.0

The three defense columns correspond to GPT-5.2.

Analysis. Both AutoRedTeamer and LLM-Guided Online leverage historical feedback for adaptive attack selection. However, our approach additionally incorporates explicit semantic guidance to improve strategy selection, particularly when historical feedback is limited or unreliable. The adapted AutoRedTeamer achieves performance comparable to ASR-Sort in most settings, while LLM-Guided Online consistently matches or outperforms it, with notable improvements on GPT-5.2 (77.0% → 79.8%) and LlamaGuard (26.0% → 34.6%). These results suggest that historical feedback alone provides limited additional benefits, whereas combining semantic guidance with online adaptation enables more effective budget-constrained attack scheduling.

3. On Implementation and Budget Details.

Some key technical details are missing from the paper. For example, how are each of the attacks implemented? Is one attempt one attempted strategy, and if so are budget limits placed on queries within each individual strategy?

We thank the reviewer for highlighting the need for implementation details. We adopt Jailbreak Foundry (line 273 in the main text), a standardized framework that faithfully reproduces 30 jailbreak attacks, achieving a mean ASR deviation of only +0.26 percentage points from their originally reported results. Importantly, each attack follows its original methodology and internal execution procedure, rather than being simplified for our scheduling framework. Each attempt corresponds to executing one complete attack strategy, with its internal query configuration preserved. The budget (K) limits the number of strategies attempted per query, rather than the internal queries required by each strategy. We will clarify these details in the revision.

Reference

[1] Zhou et al., AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, NIPS 2025

[2] Fang et al., Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking, ICML 2026 Oral

I thank the authors for their detailed response. However, my main concerns regarding the current experimentation have not been alleviated. Accordingly, I will maintain my original score.

AuthorsOpenReview ↗

Request for Further Clarification on the Remaining Concerns

Thank you for your response. We respect your decision, but we would appreciate more specific clarification regarding which aspects of our revisions remain insufficient.

In our previous response, we directly addressed each of the three concerns raised in your original review:

  1. Experimental reliability: We clarified the rationale behind our victim model selection, expanded the experimental analysis across different defense conditions, and reported results across five matched query-order permutations to examine sensitivity to evaluation order.

  2. Comparison with prior work: We implemented an adaptation of AutoRedTeamer's scheduling policy under our unified evaluation setting and provided additional quantitative comparisons across all three victim models and three defense mechanisms.

  3. Implementation details: We clarified how the 30 attacks are executed, what constitutes an individual attempt, and how the overall query budget relates to the internal queries of each attack.

We fully acknowledge that additional experiments can always strengthen an empirical study. However, your latest response does not identify which of these concerns remains unresolved, why the additional evidence is insufficient, or what specific aspects of the experimental design undermine the validity of our findings.

Could you please clarify which particular experimental limitations still concern you? For example, is the concern primarily about the number of victim models, the statistical robustness of the results, the validity of the AutoRedTeamer comparison, or another aspect of the experimental methodology?

We would genuinely welcome more concrete feedback, as it would help us understand your assessment and make meaningful improvements to the work. We hope our revisions can be evaluated based on the additional evidence provided, rather than leaving the original concerns unresolved without further clarification. Thank you.

Reviewer_K2y8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an attack library optimization and scheduling framework called Budgeted Attack Scheduling and evaluates its performance. The proposed approach considers three factors for attack prioritization: (1) historical attack success rates (ASRs), (2) semantic guidance, and (3) online feedback from previous attack attempts. Based on these factors, the authors investigate four scheduling methods: ASR Sort, LLM-guided Sort, Thompson Sampling, and LLM-guided Online. The experiments use 30 attacks from Jailbreak Foundry, AdvBench50 to estimate historical attack success rates, and JBB-Behaviors for evaluation

Critical strengths
  1. Under LlamaGuard, LLM-guided Online achieves 34.6% Success@5, compared with 19.0% for ASR Sort and 28.0% for LLM-guided Sort. This shows a practical gain under the same attempt budget.
  2. The paper uses different queries to estimate past success rates and evaluate performance. It compares the online methods using the same query orders.
Critical weaknesses
  1. Limited Novelty and Technical Contribution The novelty of the paper appears rather limited. For semantic guidance, the authors use GPT-4o to rank candidate attacks based on descriptions of the attacks and the victim model, the current query, and historical success rates. The online method further incorporates a score derived from the number of successful and failed attempts. However, the idea of using LLMs and feedback to guide attack selection has already been extensively explored in AutoRedTeamer (Zhou et al., 2025), although the authors argue that their specific mechanisms and use of feedback differ. While using a fixed set of attacks is useful for controlled comparisons, real-world attackers are not restricted to predefined attack templates. Therefore, the paper does not sufficiently demonstrate whether the proposed method can generalize to dynamically evolving attack environments where new attack strategies continuously emerge. Overall, the experimental results mainly confirm that historical success rates may become less reliable when defenses change and that incorporating feedback can improve attack selection. While these findings are useful, the underlying insights are relatively intuitive, and the technical contribution appears somewhat incremental.

  2. Experimental Design and Evaluation Methodology Thompson Sampling and LLM-guided Online use different selection and update rules. Therefore, their comparison does not clearly isolate the contribution of semantic guidance from the effects of these differences. Furthermore, the paper does not clearly specify which judge models and evaluation criteria are used to determine attack success. These details have become increasingly important in recent jailbreak research and are particularly critical for this paper, as the resulting success labels directly affect key performance measures and algorithmic decisions, including attack success rates, early stopping, and online updates.

  3. Novelty The reported improvements in attack success rates are useful, but the paper does not appear to offer sufficiently novel ideas or findings beyond prior work on adaptive attack selection. Overall, the proposed approach appears to be an incremental engineering extension of existing methods, with several components combined in a somewhat ad hoc manner. As a result, its technical contribution and novelty remain limited.

Questions for authors

Which judge model, version, and criteria are used to decide whether an attack succeeds? Is the same procedure used for all attacks and defense conditions?

3 comments
AuthorsOpenReview ↗

Clarifying Our Contributions and Strengthening the Experimental Evidence

Thank you for your insightful comments and for specifying the questions that could lead to a more favorable assessment. We appreciate your concerns regarding novelty, the isolation of scheduling signals, and evaluation reproducibility, and address them below.

1. On Novelty and Empirical Contributions.

However, the idea of using LLMs and feedback to guide attack selection has already been extensively explored in AutoRedTeamer (Zhou et al., 2025), although the authors argue that their specific mechanisms and use of feedback differ.

Distinct research objectives. While AutoRedTeamer [1] focuses on autonomous attack discovery, integration, and refinement, we systematically investigate when and why different scheduling strategies succeed or fail under strict per-query attempt budgets (K∈1,3,5K \in {1,3,5}). Rather than improving individual attacks, we isolate the roles of historical priors, semantic guidance, and online feedback in attack prioritization. Our work thus addresses a complementary research question that is not systematically examined in AutoRedTeamer.

Therefore, the paper does not sufficiently demonstrate whether the proposed method can generalize to dynamically evolving attack environments where new attack strategies continuously emerge.

A fixed but extensible attack library. We build on Jailbreak Foundry [2], which provides 30 standardized, fidelity-evaluated reproductions of published attacks, rather than unverified attack candidates. Importantly, fixed does not mean closed: Jailbreak Foundry supports automated integration of newly published attacks. We deliberately freeze the pool during evaluation to isolate scheduling effects from changes in attack quality, while retaining extensibility for future attacks. This provides a high-quality, reproducible foundation for studying attack scheduling.

While these findings are useful, the underlying insights are relatively intuitive, and the technical contribution appears somewhat incremental.

Beyond intuitive observations. Our contribution goes beyond confirming that priors and feedback are useful; we identify when they suffice, when they fail, and when adaptation becomes necessary. Specifically, (1) static rankings can approach hindsight performance when priors transfer reliably, but rank agreement drops to τb=0.39\tau_b=0.39 under LlamaGuard; (2) without priors, ASR Sort's Success@5 drops from 77% to 19% on GPT-5.2 Native, while semantic guidance achieves 55%; and (3) under LlamaGuard, online adaptation further improves Success@5 from 28.0% to 34.6% over LLM-guided Sort. These findings characterize the applicability and limitations of different scheduling paradigms, rather than merely demonstrating that combining them improves performance.

2. On Isolating the Contributions of Scheduling Signals.

Therefore, their comparison does not clearly isolate the contribution of semantic guidance from the effects of these differences.

We apologize for not making this distinction sufficiently clear. Our experiments explicitly isolate the contribution of semantic guidance through controlled comparisons between ASR Sort and LLM-guided Sort. Without historical priors, ASR Sort reduces to random ranking, whereas LLM-guided Sort relies solely on semantic guidance. With historical priors, both methods use the same prior information, while LLM-guided Sort additionally incorporates semantic signals.

To further validate this distinction, we conducted an additional ablation of LLM-guided Sort, retaining the current query and historical ASR priors while removing the victim-model and attack-method descriptions, with all ranking instructions unchanged. As shown below, its performance becomes identical to ASR Sort across all six evaluation settings and all three success metrics, providing rigorously controlled evidence for the contribution of semantic guidance.

Victim / DefenseASR SortLLM Sort w/o SemanticsLLM Sort
DeepSeek-V4-Pro92 / 98 / 9992 / 98 / 9992 / 99 / 99
Gemma-4-26B59 / 84 / 9059 / 84 / 9059 / 85 / 90
GPT-5.2 Native36 / 69 / 7736 / 69 / 7739 / 74 / 79
Backtranslation12 / 50 / 6212 / 50 / 6231 / 66 / 67
SmoothLLM27 / 41 / 5127 / 41 / 5127 / 45 / 54
LlamaGuard3 / 19 / 193 / 19 / 195 / 24 / 28

Each cell reports Success@1 / Success@3 / Success@5 (%). The three defense settings are evaluated on GPT-5.2.

AuthorsOpenReview ↗

Response to Reviewer K2y8 (continued)

A similar relationship holds for Thompson Sampling. As the prior strength κ\kappa increases, the posterior distributions concentrate around the historical ASR priors, reducing both stochastic exploration and the influence of online feedback. Consequently, Thompson Sampling approaches ASR Sort as κ→∞\kappa \to \infty (up to tie-breaking).

In summary, we isolate the contribution of semantic guidance by comparing ASR Sort with LLM-guided Sort, and assess the benefits of online feedback by comparing ASR Sort with Thompson Sampling and LLM-guided Sort with LLM-guided Online (lines 362–368). We do not use the comparison between Thompson Sampling and LLM-guided Online to attribute performance gains to semantic guidance, as this comparison reflects differences in their overall scheduling policies rather than the isolated contribution of a single signal.

Furthermore, the paper does not clearly specify which judge models and evaluation criteria are used to determine attack success.

We follow the standardized evaluation protocol of Jailbreak Foundry [2], which integrates 30 reproduced attacks under a unified framework (line 272). Specifically, we adopt its fixed GPT-4o judge and standardized success rubric, which assesses whether the victim's response fulfills the original harmful request and produces a binary success label. The same judging criteria are consistently applied across all attacks and defense settings, with the resulting labels used for performance evaluation, early stopping, and online updates. All attacks retain their original-paper-matched configurations, including internal search and retry budgets, while the three defenses use their respective default settings. Thank you.

3. On Technical Novelty and Empirical Insights.

the proposed approach appears to be an incremental engineering extension of existing methods, with several components combined in a somewhat ad hoc manner. As a result, its technical contribution and novelty remain limited.

As clarified in Response 1, our work addresses a distinct research question from AutoRedTeamer, with a focus on systematically understanding budget-constrained attack scheduling. We further emphasize that our contribution is not an ad hoc combination of existing techniques, but a systematic empirical investigation of scheduling paradigms. We deliberately adopt established scheduling methods to examine the distinct roles of historical priors, semantic guidance, and online feedback under a unified evaluation setting.

Importantly, our findings go beyond the intuitive observation that feedback improves attack selection. We identify conditions under which static scheduling approaches hindsight performance, quantify its limitations under defense-induced shifts (Kendall's τb\tau_b drops from 0.75–0.83 to 0.39), and reveal the order sensitivity of online adaptation. These findings demonstrate that online adaptation is not universally necessary or superior, characterizing when different scheduling paradigms are effective or insufficient.

We therefore emphasize that our novelty lies in the systematic characterization of scheduling paradigms and their applicability boundaries, rather than introducing new ranking algorithms. Thank you for your time and careful consideration! We hope our clarifications provide a clearer understanding of our work.

Reference

[1] Zhou et al., AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, NIPS 2025

[2] Fang et al., Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking, ICML 2026 Oral

The authors explain that the paper differs in that it focuses on a complementary research question that is not examined in AutoRedTeamer. However, AutoRedTeamer also evaluated its approach in terms of cost and, more importantly, selects attacks based on performance metrics stored in its Attack Memory. Thus, the reviewer still think that the methods therefore overlap at a broader level. The response explains that the attack library can be extended, but it does not show that the proposed scheduling methods remain effective when it is extended. Although I appreciate the authors’ clarifications and additional experimental results, my main concerns remain insufficiently addressed. I will maintain my original score.

Reviewer_3fcPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies budgeted attack scheduling for LLM red-teaming, where attacks from a fixed library are prioritized under a limited attempt budget. It compares four scheduling policies that use different combinations of historical attack success rates, LLM-based semantic guidance, and online feedback. The evaluation covers three target models, with additional experiments on three defense mechanisms applied to GPT-5.2. The reported results suggest that historical priors provide a useful initialization, while semantic guidance and online feedback can improve scheduling when these priors are unavailable or become less reliable.

Critical strengths
  • The paper addresses an important problem in LLM safety: improving the efficiency of red-teaming under limited evaluation budgets.
Critical weaknesses
  • Limited methodological novelty and insufficiently established empirical contribution. The proposed schedulers largely combine standard ASR-based ranking, Beta–Bernoulli Thompson sampling, and LLM ranking with a heuristic feedback correction. Since prior works such as AutoRedTeamer [1] already use semantic information, historical effectiveness, and accumulated feedback for attack selection, the contribution is on clearly empirical insights, but the current comparisons and ablations do not establish these insights convincingly.

  • Critical evaluation details are missing. Currently, the draft does not identify the judge for attack success or its configuration, although its labels determine both the reported metrics and the online updates. Attack-internal budgets and concrete defense settings are also missing. These details are necessary to reproduce the results and assess whether the observed improvements are robust to the evaluation configuration.

  • The practical efficiency gains are not established under matched total cost. On GPT-5.2, LLM-guided Online improves Success@5 over Thompson Sampling by 2.2% while adding 11,885 scheduler tokens at nearly identical attack-side token usage. Cache-hit rates and attack-side savings alone do not establish a favorable end-to-end trade-off. The paper should report success under matched total cost and clarify how attack-internal calls, defenses, judging, and selector overhead are accounted for.

[1] Zhou et al., AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, NeurIPS 2025

Questions for authors

Please address the concerns above.

4 comments
AuthorsOpenReview ↗

Clarifications on Novelty, Evaluation Protocol, and Cost Trade-offs

Thank you for your insightful comments. We believe that some concerns may stem from misunderstandings, which we aim to clarify in detail below. We hope that our response can clarify the misunderstandings and you can consider our work more favorably.

1. About novelty and empirical contribution.

Since prior works such as AutoRedTeamer already use semantic information, historical effectiveness, and accumulated feedback for attack selection, the contribution is on clearly empirical insights, but the current comparisons and ablations do not establish these insights convincingly.

We thank the reviewer for highlighting the connection to AutoRedTeamer [1]. We would like to clarify the distinct research objectives and empirical contributions of our work.

Distinct Research Objectives. AutoRedTeamer aims to improve attack effectiveness through autonomous attack discovery, composition, and iterative refinement, expanding its library from 4 to 12 strategies. In contrast, we study how to select and order attacks from a fixed library of 30 reproduced methods under strict per-query budgets (K∈{1,3,5}K \in \{1,3,5\}). Rather than improving individual attacks, our objective is to maximize vulnerability discovery through effective attack prioritization. This controlled setting isolates the value of scheduling decisions from changes in attack generation or quality.

Motivation for a Fixed but Extensible Attack Library. Our attack pool is built upon Jailbreak Foundry [2], which provides standardized, fidelity-evaluated reproductions of published attacks rather than an ad hoc collection of unverified LLM-generated candidates. Importantly, the pool is not inherently closed: Jailbreak Foundry supports automated integration of newly published attacks, enabling continuous expansion. We deliberately keep the pool fixed within each evaluation to isolate scheduling effects from changes in attack availability or quality. Thus, we provide a high-quality, reproducible, and extensible attack library for systematically studying attack scheduling under controlled evaluation settings.

Distinction in Empirical Contributions. We apologize for any confusion. While AutoRedTeamer demonstrates the effectiveness of memory-guided attack selection within an autonomous red-teaming framework, our work investigates a fundamentally different empirical question: when and why do different scheduling signals succeed or fail under constrained attack budgets?

Our experiments provide three distinct findings:

  1. Historical priors are condition-dependent. Static rankings can be highly effective when priors are reliable, but become less reliable under defense-induced shifts.
  2. Semantic guidance and online feedback play complementary roles. Through controlled comparisons, we disentangle their contributions and show how they compensate for missing or stale historical priors.
  3. Online adaptation introduces order sensitivity. We quantify how query order affects scheduling performance and demonstrate that semantic guidance can improve stability.

Unlike AutoRedTeamer's system-level evaluation, our experiments characterize the effectiveness, limitations, and complementary roles of scheduling signals over a fixed attack library. We therefore believe that our empirical contributions are distinct and complementary rather than redundant.

2. About evaluation details.

Currently, the draft does not identify the judge for attack success or its configuration, although its labels determine both the reported metrics and the online updates. Attack-internal budgets and concrete defense settings are also missing.

We adopt Jailbreak Foundry [2], a standardized framework for reproducible jailbreak evaluation that integrates 30 reproduced attacks under a unified execution and evaluation protocol (line 272 of the main text). Following its evaluation protocol, we use GPT-4o as a fixed judge with consistent success criteria for both performance evaluation and online feedback. All 30 attacks retain their original-paper-matched configurations, including attack-specific search and retry budgets, while the three defenses follow their respective default settings. Notably, our scheduling budget KK limits the number of attack methods attempted per query and is independent of the attack-internal budgets.

AuthorsOpenReview ↗

Response to Reviewer 3fcP (continued)

3. On the performance gains and costs of LLM-Guided Scheduling.

On GPT-5.2, LLM-guided Online improves Success@5 over Thompson Sampling by 2.2% while adding 11,885 scheduler tokens at nearly identical attack-side token usage. Cache-hit rates and attack-side savings alone do not establish a favorable end-to-end trade-off. The paper should report success under matched total cost and clarify how attack-internal calls, defenses, judging, and selector overhead are accounted for.

We thank the reviewer for raising this important distinction. As explicitly defined in Sections 3.1 and 4.2, our objective is to maximize attack success under a fixed attempt budget KK, rather than a fixed token budget. This design directly reflects our research question: how to prioritize attacks when only KK attempts are available. Since the 30 attacks have heterogeneous internal costs, enforcing equal token budgets would introduce an additional resource-allocation problem, confounding the effects of historical priors, semantic guidance, and online feedback on attack prioritization. We therefore control KK and report token consumption separately.

We acknowledge that LLM-guided scheduling incurs additional overhead and do not claim universal end-to-end cost superiority. However, its gains are setting-dependent: while the Success@5 improvement over Thompson Sampling is 2.2 pp on GPT-5.2 Native, it reaches 9.2 pp under LlamaGuard (25.4% → 34.6%), highlighting the value of semantic guidance in challenging environments. While evaluating performance under matched total costs is valuable, it addresses a distinct research question beyond the scope of our fixed-attempt scheduling formulation, which we leave for future work.

Reference

[1] Zhou et al., AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, NIPS 2025

[2] Fang et al., Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking, ICML 2026 Oral

Dear authors,

Thank you for the detailed response. Indeed, there was no misunderstanding for my initial reviews:

  • About novelty and empirical contribution: I clearly recognized that the goal of this work and AutoRedTeamer is different. While I also agree that the aimed problem of exploring f when and why different scheduling works in constrained attack budget is interesting research goal, what I want to point is that many aspects, including the search space and diagnosis tools, are mainly from the previous work.
  • Evaluation details: In line 272 that you pointed out, there was no exact mention about configuration or evaluation protocol, but just mention what you use ("We use a frozen library of 30 reproduced attack instances from Jailbreak Foundry"). Also, each paper should be self-contained, hence all the configurations and defense settings should be included in the manuscript; reviewers are hard to spend time for checking other papers to search those details. In addition, which decoding hyperparameters are used for GPT-4o judge? If it is not using temperature=0, the results are stochastic.
AuthorsOpenReview ↗

Our Appreciation and Responses to Reviewer 3fcP

We sincerely thank you for your careful review and the time you devoted to helping us improve our work. Below, we provide clarifications and revisions in response to your insightful comments.

1. Novelty and empirical contribution. We really appreciate the reviewer’s clarification and agree that several components of our study, including the attack search space and diagnosis tools, build upon prior work. Our contribution, however, lies in a fundamentally different research objective from AutoRedTeamer.

Specifically, AutoRedTeamer focuses on autonomous attack discovery, composition, and refinement, whereas we investigate how to prioritize a fixed library of complete, independently executable jailbreak attacks under a constrained query budget. Rather than generating or improving individual attacks, we isolate the effect of scheduling decisions while preserving the original execution procedures of existing attacks. Our attack library is built upon Jailbreak Foundry, which provides fidelity-evaluated implementations of published attacks.

Beyond comparing existing scheduling strategies, our empirical study addresses when and why different scheduling signals succeed or fail. Our findings reveal that (1) historical priors are effective when reliable but can become less useful under distribution shifts; (2) semantic guidance and online feedback play complementary roles in compensating for missing or stale priors; and (3) online adaptation is sensitive to query order and is not universally superior to static scheduling. We believe these findings provide a systematic characterization of the effectiveness, limitations, and complementary roles of different scheduling paradigms, distinguishing our work from prior system-level autonomous red-teaming approaches.

2. Evaluation details and reproducibility.

Thank you! We sincerely appreciate your thorough and constructive review, which has helped us improve the quality of our paper. We have revised the manuscript to provide the following implementation and evaluation details:

  • Attack configurations: We provide a complete list of the 30 attack instances adopted from Jailbreak Foundry, along with their key attack-specific configurations, including strategy variants, hyperparameters, and model choices where applicable (Lines 972–996).
  • Defense configurations: We follow the configurations specified in the original papers for all defense mechanisms (Lines 287–288).
  • Evaluation protocol: We adopt the standard evaluation protocol provided by JailbreakBench (JBB) (Lines 300–307).
  • GPT-4o judge: The GPT-4o judge uses a temperature of 0.

We appreciate the reviewer's suggestions. These revisions will make the experimental methodology more transparent and the manuscript more self-contained.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Fundamental Differences Between Budgeted Attack Scheduling and AutoRedTeamer

Dear Reviewers,

We sincerely thank all three reviewers for their thoughtful and constructive feedback! We noticed that all three reviewers highlighted AutoRedTeamer [1], an excellent prior work on autonomous red-teaming. We appreciate the reviewers for recognizing this important connection. However, we would like to emphasize that our work differs fundamentally from AutoRedTeamer in three key aspects:

(1) Research Objectives. AutoRedTeamer focuses on memory-guided attack selection and prompt-level composition, whereas modern jailbreak attacks often involve iterative optimization, multi-turn interactions, and complex execution procedures beyond prompt transformations. Its composition mechanism therefore cannot be directly applied to heterogeneous attack methods without additional adaptations. In contrast, our work systematically investigates budget-constrained scheduling of complete, independently executable jailbreak attacks, preserving their original execution procedures. Rather than generating or refining attacks, we study how to effectively prioritize existing attack strategies under strict attempt budgets.

(2) Reproducibility. To the best of our knowledge, AutoRedTeamer has not released its official implementation, while the available third-party reproduction lacks substantial attack coverage and critical implementation details, making faithful reproduction and rigorous comparison challenging. In contrast, our work builds upon Jailbreak Foundry [2] (ICML 2026 Oral), a standardized framework that faithfully reproduces published jailbreak attacks as executable implementations, leveraging 30 reproduced attacks to establish a rigorous and reproducible evaluation foundation. Furthermore, we reproduce an adapted version of AutoRedTeamer's scheduling policy under the same protocol, achieving improvements from 26.0% to 34.6% Success@5 under LlamaGuard.

(3) Empirical Findings. While AutoRedTeamer demonstrates the effectiveness of memory-guided attack selection, it does not systematically investigate when static scheduling is sufficient, why historical priors fail under distribution shifts, or when online adaptation becomes necessary. Our work addresses these questions through controlled experiments, revealing the complementary roles of historical priors, semantic guidance, and online feedback. Importantly, we demonstrate that online adaptation is not universally superior to static scheduling and identify its sensitivity to query order, establishing a systematic understanding of the applicability and limitations of different scheduling paradigms.

We hope these distinctions, together with our additional experiments and detailed responses, clarify the independent research contributions and empirical significance of our work and encourage the reviewers to reconsider their assessments.

Reference

[1] Zhou et al., AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration, NIPS 2025

[2] Fang et al., Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking, ICML 2026 Oral

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.