- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies automated jailbreak discovery when successful target responses are too rare to provide useful training feedback. It proposes Curriculum of Models (CoM), which trains an investigator model against a sequence of target variants that become progressively harder to elicit the desired behavior from. The paper evaluates the approach in single- and multi-behavior settings, compares it with direct training and propensity-bound rewards, and examines generalization to held-out behaviors.
Critical strengths
S1: Addresses a clear training bottleneck
Section 2 formulates rare-behavior discovery as expected-reward maximization for the investigator policy and uses adjustable target difficulty to provide denser early feedback. Stage transitions and recoveries in Figure 1, together with failures of overly coarse curricula in Figure 3, connect the motivation to observed behavior. Avoiding likelihood corrections between response distributions is a clear design distinction.
S2: Reports results that constrain its own conclusions
Section 3.2 explicitly states that PRBO outperforms CoM on training and held-out behaviors for both targets; direct training also slightly exceeds CoM on gpt-oss. Table 1 reports seed ranges rather than means alone, and Section 6 discusses judge self-preference. These findings help define applicability instead of presenting only favorable cases.
S3: Evaluation at several levels and cost breakdowns are useful
The paper covers single-behavior, multi-behavior, and held-out-behavior evaluation. Appendix D reports pass@1 and pass@32 at different content thresholds, while Appendix E separates target inference, investigator training, and judge costs. This is more informative for a new training procedure than final success rates alone.
S4: Examines reward quality beyond a single judge
Appendix H re-evaluates 1,024 outputs using models from several developers and repeats calls to the primary judge. It explicitly distinguishes judge agreement from correctness. Although this cannot replace expert assessment, it identifies important measurement risks.
Critical weaknesses
W1: Curriculum gains are not sufficiently separated from suppressing investigator refusals
In Section 2.4 and Appendix C.5, the direct-training baseline starts from an investigator given anti-refusal SFT, whereas CoM and PRBO start from the original investigator. The authors intend this to make the comparison conservative, which deserves credit. However, Section 3.1 also attributes PRBO's failure to the investigator's unwillingness to generate prompts. Table 1 therefore combines two issues: whether the investigator proposes requests at all, and whether those requests yield useful target-side learning signals.
The most informative additions would use a shared investigator initialization across all methods and a two-stage control that trains against the easy target before switching directly to the original target. Broad task expansion is unnecessary; representative difficult settings already selected would suffice. If a simple warm start achieves the same outcome, the independent contribution of progressive staging would shrink substantially.
W2: The PRBO comparison reflects a particular implementation and training scale
The PRBO hyperparameter sweep in Appendix C.1 is a strength. Nevertheless, the implementation changes the proposal distribution, group/batch sizes, natural-reward threshold, and likelihood truncation relative to prior work. Single- and multi-behavior experiments also change both task diversity and total rollouts. Consequently, failure to learn here cannot be generalized into an inherent limitation of PRBO.
In one single-behavior setting, I recommend PRBO with the same warm start and matched total rollouts per optimization step, reporting refusal rates, nonzero-reward rates, and normalized-advantage variance. Learning curves against actual cost would also support the practical claim of a competitive alternative more directly than matched step counts.
W3: Independent evaluation of technical correctness and method rankings remains insufficient
Section 2.4 requires a score of 100 for success, but cross-judge pass/fail agreement in Appendix H is 77.7%–89.6%, and repeated primary-judge agreement is only 92.9%. At a strict threshold, a small number of errors can substantially affect rare-success estimates. When gpt-oss is both target and judge, other judges are systematically stricter.
Moreover, the re-evaluated samples all come from single-behavior CoM, so they cannot establish whether its ranking relative to PRBO and direct training is robust. A balanced, blinded sample across the three methods should be evaluated, ideally with a small number of qualified experts assessing correctness and actionability under controlled conditions. Aggregate agreement and error categories would suffice; dangerous procedures need not be published. Current results should be described as satisfying model-judge criteria, rather than independently verified real-world feasibility.
W4: Task selection and statistical aggregation limit external validity
Section 3.1 and Appendix B screen for four behaviors that are difficult for the original target but elicitable from the weakened target. This is appropriate for mechanism testing, but selects conditions favorable to curriculum applicability. Multi-behavior evaluation uses only one random split, with held-out behaviors from related domains, and some single-behavior means obscure seed ranges spanning zero to one.
The screening should be framed as an explicitly conditional experiment, with screening costs and candidate counts documented. The main text should show pass@1 alongside pass@32 to avoid saturation hiding differences. Hierarchical intervals over behaviors and training seeds would better characterize generalization than three-seed means alone. A second split would be a useful, lower-priority robustness addition.
W5: Simple curriculum-design controls and monotonicity checks are missing
Figure 5 supports difficulty ordering through aggregate refusal rates, but does not establish monotonic difficulty for each behavior under the current investigator policy. Advancement based on one group's mean is sensitive to judge noise. Figure 3 primarily tunes settings on one additional behavior and does not compare success-based advancement against fixed-time progression or a two-stage switch. An inexpensive control, accompanied by final-stage reach rates, stage durations, and failure rates for each behavior, would help establish when the curriculum is effective.
Questions for authors
- How much of the Gemma single-behavior advantage remains when all methods use the same warm-started investigator?
- What are the gains over easy-target training followed by a single switch, and over fixed-time progression?
- Do method rankings and pass@1 differences persist under independent judging of balanced samples from all three methods?
- Does single-behavior PRBO still fail to overcome investigator refusals when given the total rollout count used in multi-behavior training?
2 comments
Thank you for the detailed rebuttal, which has addressed my concerns. I maintain my score for this paper
Response to Reviewer saVa
We thank the reviewer for their thoughtful review and constructive suggestions. We appreciate their recognition of the sparse-reward bottleneck, the simplicity of our approach, and the value of candidly reporting comparisons and evaluation limitations.
We have revised the manuscript, with additions and revised text highlighted in green. We thank the reviewer for the suggestions that helped strengthen the work. Below, we address each weakness, followed by the questions.
Addressing weaknesses
W1: Curriculum gains versus investigator refusal suppression
We ran PRBO with an anti-refusal SFT warm start on a difficult GPT-OSS behavior. This eliminates investigator refusals but does not reliably enable successful elicitation, whereas CoM succeeds in the same setting without a warm start (Table 14, Appendix F.2). Direct RL already uses a warm start, yet fails in our Gemma single-behavior experiments. Thus, refusal suppression alone does not reproduce CoM’s results.
W2: PRBO implementation and training scale
We agree that our results concern the evaluated implementation and training settings, rather than an inherent limitation of PRBO. Appendix C.1 documents the differences from [1] and our hyperparameter sweep. We also tested both an anti-refusal warm start and increasing rollouts per step from 64 to 256 in separate ablations. Neither reliably resolves the failure; refusal rates and elicitation results are reported in Figure 12 and Table 14 (Appendix F).
W3: Independent evaluation of correctness and rankings
Our success criterion requires satisfying every criterion in the strict model-judge rubric. The cross-judge analysis provides some reassurance: pass/fail agreement is 77.7%–89.6%, compared with 92.9% between repeated primary-judge calls (Appendix J). We acknowledge these measurement limitations, including potential self-preference, in Section 6.1.
W4: Task selection and statistical aggregation
We deliberately select rare behaviors because sparse-reward elicitation is precisely the setting this paper studies; readily elicitable behaviors do not meaningfully test this problem. Appendix B documents screening across 48 candidate behaviors per target before investigator RL. We have also switched the main-text results to pass@1 to avoid saturation, while retaining pass@32 in Appendix D.
W5: Curriculum controls and behavior-specific difficulty
We ran the requested two-stage control, K=1: all three seeds achieve a final score of zero. This agrees with the existing K=2 failures, whereas K=5 and K=10 succeed in this setting (Figure 3, Section 4.1).
A fixed schedule could work if appropriately chosen. However, advancing too quickly can remove the learning signal, while advancing too slowly spends unnecessary computation on easier stages. Appropriate stage durations may differ across behaviors; performance-based advancement avoids specifying them in advance.
Addressing questions
Q1: What happens with a shared warm-started investigator?
Our warm-started PRBO control is on GPT-OSS, where CoM still performs better without warm-starting (Table 14). The Gemma Direct RL baseline already uses a warm start.
Q2: What are the gains over a single switch and fixed-time progression?
The single-switch control is our new K=1 experiment: all three seeds fail, whereas finer curricula succeed (Figure 3).
Q3: Do rankings persist under balanced independent judging?
We have checked sensitivity to judge choice by independently re-scoring 1,024 CoM rollouts with five other judge models. Pass/fail agreement is 77.7%–89.6%, compared with 92.9% between repeated primary-judge calls (Appendix J). Together with requiring every criterion in the rubric for success, this provides an empirical check beyond relying on a single judge. We acknowledge the remaining evaluation uncertainty in Section 6.1.
Q4: Does increasing PRBO’s rollout budget eliminate investigator refusals?
Increasing rollouts per step to 256 does not: all three seeds retain high refusal rates and achieve zero pass@1 (Table 14). This matches the multi-behavior per-step rollout count; the corresponding training lengths are reported in Appendix F.2.
References
[1] Chowdhury et al. Automatically Jailbreaking Frontier Language Models with Investigator Agents. 2025. https://transluce.org/jailbreaking-frontier-models