acceptodds
Under review as a conference paper at ICLR 2027

Discovering jailbreaks via curriculum of models

Abstract

Automated jailbreak discovery aims to find prompts that elicit prohibited behaviors from safety-trained language models, helping developers identify weaknesses before deployment. One class of approaches trains an investigator model to query a target model, using reinforcement learning to reward prompts that successfully elicit a specified behavior. However, training directly against a safety-trained target is difficult because most prompts are refused, leaving the investigator with a sparse reward signal. Prior work addresses this problem with the propensity bound (PRBO) (Chowdhury et al., 2025), a proxy reward estimated by sampling responses from a modified target that exhibits the behavior more readily, and then adjusting for how likely those responses would have been under the original target. We propose Curriculum of Models (CoM), an alternative method that trains the investigator against a sequence of modified versions of the target model, from which the behavior is progressively harder to elicit. The curriculum begins with a version that rarely refuses harmful requests; once the investigator effectively elicits the behavior from the current version, training advances to the next, harder version, eventually reaching the original target model. We evaluate CoM on single- and multi-behavior elicitation tasks involving CBRN materials, explosives, and illicit drugs using two open-weight targets, gemma-4-31b-it and gpt-oss-120b, and find that CoM successfully discovers jailbreaks and achieves performance competitive with PRBO. Our results suggest that a simple curriculum over target models offers a promising alternative for addressing sparse rewards in rare-behavior elicitation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
jailbreak, reinforcement learning, automated red teaming

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_saVaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies automated jailbreak discovery when successful target responses are too rare to provide useful training feedback. It proposes Curriculum of Models (CoM), which trains an investigator model against a sequence of target variants that become progressively harder to elicit the desired behavior from. The paper evaluates the approach in single- and multi-behavior settings, compares it with direct training and propensity-bound rewards, and examines generalization to held-out behaviors.

Critical strengths

S1: Addresses a clear training bottleneck

Section 2 formulates rare-behavior discovery as expected-reward maximization for the investigator policy and uses adjustable target difficulty to provide denser early feedback. Stage transitions and recoveries in Figure 1, together with failures of overly coarse curricula in Figure 3, connect the motivation to observed behavior. Avoiding likelihood corrections between response distributions is a clear design distinction.

S2: Reports results that constrain its own conclusions

Section 3.2 explicitly states that PRBO outperforms CoM on training and held-out behaviors for both targets; direct training also slightly exceeds CoM on gpt-oss. Table 1 reports seed ranges rather than means alone, and Section 6 discusses judge self-preference. These findings help define applicability instead of presenting only favorable cases.

S3: Evaluation at several levels and cost breakdowns are useful

The paper covers single-behavior, multi-behavior, and held-out-behavior evaluation. Appendix D reports pass@1 and pass@32 at different content thresholds, while Appendix E separates target inference, investigator training, and judge costs. This is more informative for a new training procedure than final success rates alone.

S4: Examines reward quality beyond a single judge

Appendix H re-evaluates 1,024 outputs using models from several developers and repeats calls to the primary judge. It explicitly distinguishes judge agreement from correctness. Although this cannot replace expert assessment, it identifies important measurement risks.

Critical weaknesses

W1: Curriculum gains are not sufficiently separated from suppressing investigator refusals

In Section 2.4 and Appendix C.5, the direct-training baseline starts from an investigator given anti-refusal SFT, whereas CoM and PRBO start from the original investigator. The authors intend this to make the comparison conservative, which deserves credit. However, Section 3.1 also attributes PRBO's failure to the investigator's unwillingness to generate prompts. Table 1 therefore combines two issues: whether the investigator proposes requests at all, and whether those requests yield useful target-side learning signals.

The most informative additions would use a shared investigator initialization across all methods and a two-stage control that trains against the easy target before switching directly to the original target. Broad task expansion is unnecessary; representative difficult settings already selected would suffice. If a simple warm start achieves the same outcome, the independent contribution of progressive staging would shrink substantially.

W2: The PRBO comparison reflects a particular implementation and training scale

The PRBO hyperparameter sweep in Appendix C.1 is a strength. Nevertheless, the implementation changes the proposal distribution, group/batch sizes, natural-reward threshold, and likelihood truncation relative to prior work. Single- and multi-behavior experiments also change both task diversity and total rollouts. Consequently, failure to learn here cannot be generalized into an inherent limitation of PRBO.

In one single-behavior setting, I recommend PRBO with the same warm start and matched total rollouts per optimization step, reporting refusal rates, nonzero-reward rates, and normalized-advantage variance. Learning curves against actual cost would also support the practical claim of a competitive alternative more directly than matched step counts.

W3: Independent evaluation of technical correctness and method rankings remains insufficient

Section 2.4 requires a score of 100 for success, but cross-judge pass/fail agreement in Appendix H is 77.7%–89.6%, and repeated primary-judge agreement is only 92.9%. At a strict threshold, a small number of errors can substantially affect rare-success estimates. When gpt-oss is both target and judge, other judges are systematically stricter.

Moreover, the re-evaluated samples all come from single-behavior CoM, so they cannot establish whether its ranking relative to PRBO and direct training is robust. A balanced, blinded sample across the three methods should be evaluated, ideally with a small number of qualified experts assessing correctness and actionability under controlled conditions. Aggregate agreement and error categories would suffice; dangerous procedures need not be published. Current results should be described as satisfying model-judge criteria, rather than independently verified real-world feasibility.

W4: Task selection and statistical aggregation limit external validity

Section 3.1 and Appendix B screen for four behaviors that are difficult for the original target but elicitable from the weakened target. This is appropriate for mechanism testing, but selects conditions favorable to curriculum applicability. Multi-behavior evaluation uses only one random split, with held-out behaviors from related domains, and some single-behavior means obscure seed ranges spanning zero to one.

The screening should be framed as an explicitly conditional experiment, with screening costs and candidate counts documented. The main text should show pass@1 alongside pass@32 to avoid saturation hiding differences. Hierarchical intervals over behaviors and training seeds would better characterize generalization than three-seed means alone. A second split would be a useful, lower-priority robustness addition.

W5: Simple curriculum-design controls and monotonicity checks are missing

Figure 5 supports difficulty ordering through aggregate refusal rates, but does not establish monotonic difficulty for each behavior under the current investigator policy. Advancement based on one group's mean is sensitive to judge noise. Figure 3 primarily tunes settings on one additional behavior and does not compare success-based advancement against fixed-time progression or a two-stage switch. An inexpensive control, accompanied by final-stage reach rates, stage durations, and failure rates for each behavior, would help establish when the curriculum is effective.

Questions for authors
  1. How much of the Gemma single-behavior advantage remains when all methods use the same warm-started investigator?
  2. What are the gains over easy-target training followed by a single switch, and over fixed-time progression?
  3. Do method rankings and pass@1 differences persist under independent judging of balanced samples from all three methods?
  4. Does single-behavior PRBO still fail to overcome investigator refusals when given the total rollout count used in multi-behavior training?
2 comments
AuthorsOpenReview ↗

Response to Reviewer saVa

We thank the reviewer for their thoughtful review and constructive suggestions. We appreciate their recognition of the sparse-reward bottleneck, the simplicity of our approach, and the value of candidly reporting comparisons and evaluation limitations.

We have revised the manuscript, with additions and revised text highlighted in green. We thank the reviewer for the suggestions that helped strengthen the work. Below, we address each weakness, followed by the questions.

Addressing weaknesses

W1: Curriculum gains versus investigator refusal suppression

We ran PRBO with an anti-refusal SFT warm start on a difficult GPT-OSS behavior. This eliminates investigator refusals but does not reliably enable successful elicitation, whereas CoM succeeds in the same setting without a warm start (Table 14, Appendix F.2). Direct RL already uses a warm start, yet fails in our Gemma single-behavior experiments. Thus, refusal suppression alone does not reproduce CoM’s results.

W2: PRBO implementation and training scale

We agree that our results concern the evaluated implementation and training settings, rather than an inherent limitation of PRBO. Appendix C.1 documents the differences from [1] and our hyperparameter sweep. We also tested both an anti-refusal warm start and increasing rollouts per step from 64 to 256 in separate ablations. Neither reliably resolves the failure; refusal rates and elicitation results are reported in Figure 12 and Table 14 (Appendix F).

W3: Independent evaluation of correctness and rankings

Our success criterion requires satisfying every criterion in the strict model-judge rubric. The cross-judge analysis provides some reassurance: pass/fail agreement is 77.7%–89.6%, compared with 92.9% between repeated primary-judge calls (Appendix J). We acknowledge these measurement limitations, including potential self-preference, in Section 6.1.

W4: Task selection and statistical aggregation

We deliberately select rare behaviors because sparse-reward elicitation is precisely the setting this paper studies; readily elicitable behaviors do not meaningfully test this problem. Appendix B documents screening across 48 candidate behaviors per target before investigator RL. We have also switched the main-text results to pass@1 to avoid saturation, while retaining pass@32 in Appendix D.

W5: Curriculum controls and behavior-specific difficulty

We ran the requested two-stage control, K=1K=1: all three seeds achieve a final score of zero. This agrees with the existing K=2K=2 failures, whereas K=5K=5 and K=10K=10 succeed in this setting (Figure 3, Section 4.1).

A fixed schedule could work if appropriately chosen. However, advancing too quickly can remove the learning signal, while advancing too slowly spends unnecessary computation on easier stages. Appropriate stage durations may differ across behaviors; performance-based advancement avoids specifying them in advance.

Addressing questions

Q1: What happens with a shared warm-started investigator?

Our warm-started PRBO control is on GPT-OSS, where CoM still performs better without warm-starting (Table 14). The Gemma Direct RL baseline already uses a warm start.

Q2: What are the gains over a single switch and fixed-time progression?

The single-switch control is our new K=1K=1 experiment: all three seeds fail, whereas finer curricula succeed (Figure 3).

Q3: Do rankings persist under balanced independent judging?

We have checked sensitivity to judge choice by independently re-scoring 1,024 CoM rollouts with five other judge models. Pass/fail agreement is 77.7%–89.6%, compared with 92.9% between repeated primary-judge calls (Appendix J). Together with requiring every criterion in the rubric for success, this provides an empirical check beyond relying on a single judge. We acknowledge the remaining evaluation uncertainty in Section 6.1.

Q4: Does increasing PRBO’s rollout budget eliminate investigator refusals?

Increasing rollouts per step to 256 does not: all three seeds retain high refusal rates and achieve zero pass@1 (Table 14). This matches the multi-behavior per-step rollout count; the corresponding training lengths are reported in Appendix F.2.

References

[1] Chowdhury et al. Automatically Jailbreaking Frontier Language Models with Investigator Agents. 2025. https://transluce.org/jailbreaking-frontier-models

Thank you for the detailed rebuttal, which has addressed my concerns. I maintain my score for this paper

Reviewer_oYByOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The work proposes Curriculum of Models (CoM), which trains an attacker model to generate queries which jailbreak a target model. The method leverages the target model with anti-refusal LoRA applied at decreasing strengths to avoid the sparse-reward problem. Results are reported comparing against PRBO on single and multi-behavior elicitation tasks, where CoM performs well on single tasks while PRBO performs and generalizes better on multi-behavior tasks.

Critical strengths
  1. Method is clear and well motivated. The curriculum is a linear weight interpolation of an anti-refusal LoRA, emperically justified by the increasing refusal rates as anti-refusal weight decreases.

  2. Ablations for each hyperparameter help show key settings required for the method to work.

Critical weaknesses
  1. Weak evaluations. The primary measures rely on just one investigator model, two target models, CBRN tasks, and there's very high variance among seeds. Additionally, the proposed CoM performs worse than PRBO in the more-relevant multi-behavior setting, and in the single-behavior setting black-box baseline performs similar on GPT-OSS-120b

  2. Limited novelty, lacking analysis. Easy to hard curriculum [1] and RL-based jailbreak models [2] have been explored (including works cited in the paper), but the proposed method does not clearly distinguish itself from these works. Additionally there is not have much deeper analysis on why PRBO fails in the single-behavior setting, or quantified analysis on whether CoM finds diverse distinct jailbreaks versus collapsing to one-role templates.

[1] Liu, Yanjiang, et al. "Auto-rt: Automatic jailbreak strategy exploration for red-teaming large language models." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Paulus, Anselm, et al. "Advprompter: Fast adaptive adversarial prompting for llms." arXiv preprint arXiv:2404.16873 (2024).

Questions for authors
  1. How much compute (in gpu-hours rather than dollars) is required to train and evaluate the method? Evaluating on more smaller models should be reasonable, especially if PRBO is getting very low scores for medium-sized safe models.
  2. Are controlled experiments possible for analyzing why PRBO fails in single-behavior but succeeds in multi-behavior setting? Such as matching rollouts-per-step in the single-behavior setting.
4 comments
AuthorsOpenReview ↗

Response to Reviewer oYBy [1/2]

We thank the reviewer for their thoughtful review and constructive suggestions. We appreciate their positive assessment of the clarity and motivation, the empirical support for our curriculum, and the usefulness of the hyperparameter ablations.

We have revised the manuscript, with additions and revised text highlighted in green. We believe these revisions have strengthened the manuscript, and we thank the reviewer for the suggestions that helped us improve our work. Below, we address each weakness in turn, followed by the reviewer’s questions.

W1. Evaluation scope and comparison with baselines

Our experimental coverage was constrained by a limited compute budget. We would have liked to evaluate more training seeds, investigator models, target models, and behaviors. However, these dimensions multiply the number of runs, with each configuration also requiring comparisons against multiple baselines. Within our budget, we sought to balance experimental breadth, repeated runs, and comparisons against strong baselines. For context, a representative single-behavior Gemma CoM run costs approximately $45 even after sharing target inference across concurrent runs (see Appendix G for the detailed cost breakdown). We acknowledge the resulting limits on coverage and statistical precision in Section 6.1.

Our original single-behavior evaluation reused behaviors selected for Gemma. However, some of these behaviors could already be elicited from GPT-OSS before investigator RL training (Figure 4b), helping explain Direct RL’s strong performance; these behaviors therefore did not adequately test the sparse-reward setting we intended to study. We corrected this by applying the same behavior-selection procedure separately to GPT-OSS: selecting behaviors with no observed successful elicitations from the original target before RL. On these difficult GPT-OSS behaviors, CoM outperforms both baselines (Section 3.1). PRBO does indeed achieve higher success rates in the multi-behavior setting, which we now explicitly highlight in the abstract and introduction. CoM nevertheless achieves substantial success on both training and held-out behaviors. Although CoM does not outperform existing methods in every setting, we hope that introducing this alternative and candidly reporting its strengths and limitations relative to other methods constitutes a useful contribution.

W2. Relationship to prior work and additional analysis

Relationship to prior work. We agree that RL-based jailbreak discovery is established, and we discuss and cite this literature in Section 5. Our contribution is an easy-to-hard curriculum constructed from modified versions of the same target model, with training advancing toward the original target as investigator performance improves.

Auto-RT [1] shares the motivation of addressing sparse rewards using a refusal-weakened model. However, it uses this model to shape the reward: its safety-feedback component assigns 0 when the weakened model does not produce harmful content, 1 when only the weakened model does, and 2 when both the weakened and original models do. It selects a single weakened variant for this purpose, rather than progressively training against a sequence of increasingly difficult targets. CoM explicitly constructs and traverses such a sequence. AdvPrompter [2] instead learns instruction-specific adversarial suffixes through alternating search and supervised fine-tuning. We have added citations and discussion of both papers.

We also distinguish CoM from other uses of curricula. [3] constructs a curriculum over progressively restored request text while keeping the safeguard classifier fixed. In concurrent work, [4] study a closely related curriculum over distinct target models for prompt-injection discovery—for example, GPT-5-nano → GPT-5.6-Luna → GPT-5.6-Terra. CoM studies jailbreak discovery and constructs its curriculum by varying a refusal-weakening adapter on a single target model. We have added a citation to this concurrent work and discuss these distinctions in Section 5.

PRBO’s single-behavior failures. We have added controlled experiments testing an anti-refusal warm start and increased rollout group size (Section 3.1 and Appendix F). Neither intervention reliably resolves the failure. These results narrow the possible explanations, although they do not establish why PRBO succeeds in multi-behavior training. One hypothesis, originally proposed by [5], is that behaviors of varying difficulty provide a natural curriculum.

Prompt diversity. We have added a prompt-diversity analysis in Appendix E. In the evaluated runs, successful prompts show substantial convergence on shared academic-persona framings, both within and across runs. We acknowledge that CoM (as well as PRBO and Direct RL) yields limited diversity: the objective rewards successful elicitation without explicitly rewarding distinct strategies, and a single effective strategy can therefore yield high reward.

AuthorsOpenReview ↗

Response to Reviewer oYBy [2/2]

Q1: Compute requirements and smaller target models

There are three main cost components. For a representative single-behavior Gemma CoM run , their approximate contributions are:

  • Investigator inference and training: hosted via Tinker; $27 per run, or 60% of the total.
  • Target inference: self-hosted via vLLM on rented GPUs; $12 per run, or 27%.
  • Judge inference: hosted via OpenRouter; $6 per run, or 13%.

For target inference, a representative run -- 400 training steps with 64 rollouts per step -- takes approximately 11 hours on two A100 GPUs. In practice, we run three training runs concurrently against the same server without a noticeable increase in latency. Amortized across these runs, target inference therefore requires approximately 2×11/3≈7.32 \times 11 / 3 \approx 7.3 A100 GPU-hours per run. These GPU-hours cover target inference only; we report investigator and judge API usage in tokens and dollars rather than GPU-hours. Appendix G provides the detailed breakdown and evaluation costs.

Evaluating smaller models could have allowed broader coverage at somewhat lower cost. However, target inference accounts for only about a quarter of this representative run’s cost; smaller targets would still incur the same investigator and judging costs, limiting the overall savings. We prioritized Gemma 4 and GPT-OSS as practically relevant open-weight targets with substantial safety training, documented in their respective model reports.

Q2: Investigating single-behavior PRBO failures

We investigated the single-behavior PRBO failures and found that, in 40 of 48 runs, the investigator did not learn to stop refusing to generate red-teaming prompts; all of these runs achieved zero pass@1. In contrast, all multi-behavior PRBO runs learned to suppress investigator refusals.

We therefore tested two interventions on one GPT-OSS behavior, with three seeds per condition:

  • Matching rollouts per step: We increased the rollout group size from 64 to 256, matching the multi-behavior setting. All three seeds still achieved zero pass@1.
  • Removing investigator refusals directly: We initialized the investigator with an anti-refusal SFT warm start. This eliminated investigator refusals, but still did not lead to reliable jailbreak discovery.

These analyses are summarized in Section 3.1 and detailed in Appendix F. As originally hypothesized by Chowdhury et al. [5], jointly training on behaviors of different difficulties may provide a natural curriculum, allowing learning on easier behaviors to support harder ones. CoM explicitly constructs a curriculum when such easier behaviors are unavailable.

References

[1] Liu et al. Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models. 2026. https://arxiv.org/abs/2501.01830

[2] Paulus et al. AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs. 2025. https://arxiv.org/abs/2404.16873

[3] Davies et al. Boundary Point Jailbreaking of Black-Box LLMs. 2026. https://arxiv.org/abs/2602.15001

[4] Yin et al. Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning. 2026. https://arxiv.org/abs/2609.33628

[5] Chowdhury et al. Automatically Jailbreaking Frontier Language Models with Investigator Agents. 2025. https://transluce.org/jailbreaking-frontier-models

Dear reviewer oYBy,

Thank you for your thoughtful review. We have incorporated your feedback into the new revision, and we feel that the manuscript has been improved. Thank you for helping us improve our work!

Given our revisions in response to your thoughtful feedback, we kindly ask that you reconsider your score.

Thank you for your detailed response and paper revisions.

I still have some concerns around novelty and limited evaluation scope, but recognize compute budget limitations. My primary concerns have been mostly addressed so I raise my score.

Reviewer_H9XsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Curriculum of Models (CoM), a reinforcement learning-based approach for automated jailbreak discovery that addresses the sparse-reward problem. Instead of training an investigator directly against a safety-trained target model, CoM progressively increases the difficulty by gradually reducing the influence of an anti-refusal LoRA adapter. The authors evaluate CoM on single- and multi-behavior jailbreak tasks using Gemma-4-31B and GPT-OSS-120B, comparing it against black-box training and PRBO. Experimental results demonstrate strong performance in single-behavior settings, particularly on Gemma-4-31B, and generalization to unseen behaviors. However, PRBO outperforms CoM in multi-behavior experiments on both target models.

Critical strengths
  1. The paper is well written and clearly organized. The motivation and proposed methodology are easy to follow.

  2. I like the idea of an anti-refusal LoRA to construct a curriculum of target models with progressively increasing difficulty. By adjusting the LoRA strength based on the investigator's performance, the authors provide an interesting approach to addressing sparse rewards in reinforcement learning.

Critical weaknesses
  1. Although CoM achieves strong results in single-behavior experiments, PRBO consistently outperforms it in multi-behavior experiments. Furthermore, CoM slightly underperforms the black-box baseline on GPT-OSS-120B. These results raise questions about when the proposed approach would be preferable to existing methods. The authors also do not sufficiently explain why PRBO performs better in multi-behavior training.

  2. The authors do not adequately demonstrate whether the proposed curriculum design is necessary. For example, they could compare CoM against a simple two-stage approach that first trains against the fully refusal-weakened target and then against the original target. Another useful comparison would be gradually reducing the LoRA strength according to a fixed schedule rather than using reward-based advancement. Although the paper includes curriculum-length and threshold ablations, these do not fully isolate the contributions of the intermediate stages and adaptive advancement mechanism.

  3. The authors evaluate generalization on 39 held-out behaviors drawn from the same dataset as the training behaviors. Therefore, the training and held-out behaviors likely follow similar data distributions. This evaluation does not provide strong evidence of out-of-domain generalization. Evaluating the proposed method on an independent benchmark would provide a more convincing assessment of its generalizability.

Questions for authors

The proposed curriculum assumes that reducing the anti-refusal LoRA strength progressively increases the difficulty of eliciting prohibited behaviors. Although Figure 5 shows this trend in overall refusal rates, does the same pattern hold for individual behaviors? Have the authors investigated whether decreasing the LoRA strength consistently makes each behavior harder to elicit?

3 comments
AuthorsOpenReview ↗

Response to Reviewer H9Xs [1/2]

We thank the reviewer for their thoughtful review and constructive suggestions. We appreciate their positive assessment of the manuscript’s clarity and organization, the motivation for our approach, and the use of an anti-refusal adapter to construct a curriculum of target models.

We have revised the manuscript, with additions and revised text highlighted in green. We believe these revisions have strengthened the manuscript, and we thank the reviewer for the suggestions that helped us improve our work. Below, we address each weakness in turn, followed by the reviewer’s question.

Addressing weaknesses

W1: CoM performance, compared to baselines

We acknowledge that PRBO achieves higher success rates in our multi-behavior experiments, and we now explicitly highlight this in the abstract and introduction. CoM nevertheless achieves substantial success on both training and held-out behaviors in this setting. Although CoM does not outperform existing methods in every setting, we hope that introducing this alternative and candidly reporting its strengths and limitations relative to other methods constitutes a useful contribution.

Our original single-behavior evaluation reused behaviors selected for Gemma. However, some of these behaviors could already be elicited from GPT-OSS before investigator RL training (Figure 4b), helping explain Direct RL’s strong performance; these behaviors therefore did not adequately test the sparse-reward setting we intended to study. We corrected this by applying the same behavior-selection procedure separately to GPT-OSS: selecting behaviors with no observed successful elicitations from the original target before RL. On these difficult GPT-OSS behaviors, CoM outperforms both baselines (Section 3.1).

Finally, we added experiments investigating why PRBO performs poorly in single-behavior training despite its strong multi-behavior performance (see Section 3.1 and Appendix F). Neither an anti-refusal warm start nor increasing the rollout group size reliably resolves the failure. As originally hypothesized by [1], jointly training on behaviors of different difficulties may provide a natural curriculum, allowing learning on easier behaviors to support harder ones. CoM explicitly constructs a curriculum when such easier behaviors are unavailable.

W2: Is curriculum necessary?

Section 4.1 examines curriculum lengths K=2,5,10K=2,5,10. The K=2K=2 curriculum starts with the fully refusal-weakened target, moves to an intermediate target, and then the original target. All three seeds fail to make progress against the original target, whereas K=5K=5 and K=10K=10 reliably succeed in this setting. Given that failure even with an intermediate stage, we expected the two-stage K=1K=1 curriculum to fail as well. We validated this empirically during the rebuttal period -- all three seeds likewise achieve a final score of zero. We have added these results to Figure 3 and the discussion in Section 4.1.

A fixed schedule is a reasonable alternative, provided the schedule is appropriately chosen. However, choosing stage durations introduces a trade-off: a slow schedule may allow sufficient learning on difficult stages while spending unnecessary computation and potentially overfitting on easy ones; a fast schedule may advance before the investigator is ready, losing the reward signal needed for further learning. The appropriate schedule may therefore differ across behaviors, whose difficulties vary and are not known in advance. This actually motivates CoM’s threshold-based advancement: easier behaviors can progress quickly, while harder behaviors receive more training before advancing.

W3: Generalization

We agree that held-out behaviors from the same dataset do not establish out-of-domain generalization. Our evaluation studies generalization within the substance-elicitation task family, following [1]. We have made this limitation more explicit in Section 6.1.

We note, however, that even generalization to unseen substances within this domain is nontrivial. Figure 7 in Appendix D shows that an investigator trained on a single substance generalizes less well across the broader behavior suite than an investigator trained on multiple behaviors. Thus, finding a successful strategy for one substance does not imply that the investigator can successfully elicit the other substances.

AuthorsOpenReview ↗

Response to Reviewer H9Xs [2/2]

Addressing questions

Q1: How does elicitation difficulty scale with alpha?

We agree that this is an important consideration, as refusal rate is not a direct measure of elicitation difficulty. We have run an additional experiment checking this more directly, evaluating six behaviors of varying difficulty on Gemma. For each behavior, we evaluated a fixed collection of investigator prompts across six adapter strengths, α∈{0,0.2,0.4,0.6,0.8,1.0}\alpha \in \{0, 0.2, 0.4, 0.6, 0.8, 1.0\}. Mean judge scores generally increase with alpha, although the curves differ across behaviors. We have included these results in the revised manuscript (Appendix C.5, Figure 6).

References

[1] Chowdhury et al. Automatically Jailbreaking Frontier Language Models with Investigator Agents. 2025. https://transluce.org/jailbreaking-frontier-models

Thank you to the authors for the rebuttal and revised manuscript. I appreciate the additional experiments and clarifications.

W1: Partially addressed. The new experiments strengthen the single-behavior results, but PRBO's superior multi-behavior performance remains insufficiently explained.

W2: Largely addressed. The new two-stage curriculum experiment is helpful, although a comparison with fixed-schedule advancement is still missing.

W3: The limitation remains. While I appreciate the authors explicitly acknowledging it, I still consider the lack of out-of-domain evaluation a major weakness of this paper.

Q1: Adequately addressed. The new behavior-level analysis provides useful evidence supporting the curriculum's difficulty progression.

Overall, the revisions have addressed several of my concerns. I have increased my score to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript and responses to reviewers

We thank the reviewers for their thoughtful feedback and constructive suggestions. We have uploaded a revised manuscript and posted responses to all three reviews. In the revised manuscript, additions are shown in green. We believe these revisions have strengthened the manuscript and appreciate the reviewers’ contributions to improving our work.

Dear reviewers,

Thank you for your thoughtful reviews. We revised the manuscript in response to your feedback and replied to each review in detail. Your suggestions helped us strengthen the paper.

With the discussion period closing soon (November 18 AoE), we'd appreciate it if you would please reconsider your scores in light of the revisions and our responses. If anything is still unclear or unresolved, we'd be happy to clarify.

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.