- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the cold-start problem in RL-based jailbreaks against robust target models and proposes training an attacker through a curriculum. The problem is practically relevant, and the experiments demonstrate substantial attack success and some generalization across models and benchmarks.
However, the method primarily combines the existing PISmith algorithm with a target-model curriculum, so its methodological novelty is limited. Limited novelty alone might be fine, as long as a paper proves strong empirical value. Hence, I was mainly looking for convincing evidence of effectiveness, reproducible curriculum-design principles, and benefits under comparable resource budgets. At present, incomplete budget reporting and missing controlled comparisons make it difficult to identify the source of the gains and assess the method’s practical value. I therefore lean toward borderline reject.
Critical strengths
-
Practically relevant problem. Extremely sparse binary rewards can prevent an RL attacker from learning against robust target models. The paper addresses this obstacle with a clear and intuitive approach.
-
Strong results for the selected curriculum. The Nano → Luna → Terra curriculum substantially improves attack success against Luna and Terra relative to direct training and the reported RL-Hammer baseline. The intermediate-stage ablations and pre-stage ASR measurements also support the explanation that a useful initialization must provide some successful attacks to reinforce.
-
Meaningful transfer results. The trained attacker succeeds against several targets not used during training and transfers from AgentDyn to AgentDojo. These results provide evidence of generalization. Table 1 also shows that further training against Terra improves, rather than degrades, the reported attack success against Luna.
Critical weaknesses
-
Limited methodological novelty and insufficient isolation of the contribution. The approach retains PISmith and primarily changes the training targets and their order. Although the results establish that the selected curriculum works, they do not sufficiently distinguish the benefits of sequential training from those of better intermediate-target selection or additional training and interaction budgets.
-
Insufficient training-cost reporting. The paper does not report cumulative training queries, update counts per stage, target-model token usage, or monetary costs for the compared methods. Curriculum selection and pre-stage testing costs are also not fully accounted for. Matching the number of evaluation attempts does not establish comparable training budgets. Consequently, the current results do not establish superiority under comparable resource constraints.
-
Missing comparisons directly relevant to the central claim.
- AutoInject [1] (or similar other approaches) that provide dense feedback are relevant alternatives for addressing reward sparsity and cold start, but they are not discussed or compared. An informative comparison would be to run AutoInject under the same querying budget as curriculum learning, and compare the ASR.
- This concern is particularly important for RL-Hammer, since joint training against easier targets is a closely related alternative to the proposed sequential curriculum. However, the reported RL-Hammer baseline jointly trains against Llama-3.1-8B-Instruct and the final target, whereas the proposed method trains sequentially against Nano, Luna, and Terra. The comparison therefore changes both the target-model selection and the training schedule, with the cumulative budgets left unspecified. The reported failure of this RL-Hammer configuration does not establish that joint training would fail when given access to the same intermediate targets and comparable resources.
An informative comparison with any baseline would evaluate joint or mixed training using the same Nano, Luna, and Terra targets under matched cumulative query budgets, alongside token and monetary-cost reporting. However, the paper does not conduct such comparisons and does not report cost information for all the methods. Without these comparisons, it remains unclear how much of the improvement comes from sequential curriculum learning, better intermediate-target selection, or differences in training expenditure.
- Incomplete evaluation of capability retention after continued adaptation. The Luna results in Table 1 provide positive evidence of retention, but the paper does not systematically evaluate each checkpoint against all previously encountered targets. In particular, after adapting to Muse, the attacker is not re-evaluated against earlier targets. The evidence therefore remains incomplete for the practical claim that the attacker can be repeatedly reused and extended while retaining its existing capabilities.
[1] Chen, X., Zhang, J., & Tramèr, F. (2026). Learning to inject: Automated prompt injection via reinforcement learning.
Questions for authors
-
Are the training budgets matched? Please report cumulative training attack attempts, actual target-model calls, and training updates for the curriculum, PISmith, and RL-Hammer experiments. Do these counts include intermediate stages, auxiliary targets, curriculum selection, and pre-stage testing? Do the main performance gaps persist under matched cumulative query budgets? Given the different costs of the target models, please also provide token usage or monetary-cost comparisons.
-
Do the gains come from target selection or curriculum order? Using the same Nano, Luna, and Terra targets, training data, and comparable total budgets, does sequential curriculum training outperform mixed or joint training? Can longer training without introducing a new intermediate target achieve similar gains?
-
How does the method compare with alternative solutions to cold start? Could the authors include AutoInject or another dense-feedback approach in a representative setting, with clearly specified access assumptions and comparable total budgets? This would help establish the advantages of a target-model curriculum over alternative approaches to the same problem.
-
Does continued adaptation preserve existing capabilities? Could the authors provide a checkpoint-by-target evaluation matrix covering previously encountered targets, including Nano, and re-evaluate the Muse-adapted attacker against Luna and Terra? Table 1 already provides encouraging retention results for Luna after Terra training, but it remains unclear whether this behavior persists through further adaptation.
-
How reproducible is curriculum construction? How were candidate targets, robustness rankings, stage transitions, and stopping conditions determined? How many candidate probes or training runs were required to identify the reported successful curriculum? These details are important for assessing both applicability to new targets and end-to-end cost.