acceptodds
Under review as a conference paper at ICLR 2027

Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning

Abstract

Prompt injection is a leading security risk for LLMs and LLM-based applications such as agents. State-of-the-art red-teaming methods for prompt injection leverage reinforcement learning (RL) to train an attacker LLM to generate effective injected prompts. However, when targeting frontier LLMs such as GPT-6-Luna, a major challenge is the cold-start problem: every attack attempt by the attacker LLM fails and thus receives zero reward, providing no signal for learning. In this work, we propose a curriculum learning-based method to address the cold-start problem. In particular, we propose to train the attacker LLM against a sequence of increasingly robust target LLMs, with each stage warm-starting from the attacker LLM obtained in the previous one. However, simply training against a weak target (e.g., GPT-4o-mini) may not sufficiently prepare the attacker LLM to obtain useful learning signals against a frontier LLM (e.g., GPT-5.6-Terra). Instead, we find that the design of the curriculum is critical: after each stage, the attacker LLM needs to partially succeed against the next target LLM such that it can learn from successful attempts to attack the new target. Our extensive evaluation shows that our method can effectively red-team frontier LLMs, achieving an attack success rate (ASR@10) of 93.8% and 45.0% against GPT-5.6-Luna and GPT-5.6-Terra on AgentDyn, whereas state-of-the-art RL methods such as RL-Hammer and PISmith achieve 0% ASR under the same setting. Moreover, we find that the attacker LLM transfers across targets, e.g., an attacker LLM trained to defeat one strong LLM (GPT-5.6-Terra) also succeeds against six other frontier LLMs (e.g., GPT-6-Luna) it was never trained on. Our code is available at https://anonymous.4open.science/r/Climb_Hill_Prompt_Injection-80CE.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, prompt injection, red teaming, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Bx6dOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the cold-start problem in RL-based jailbreaks against robust target models and proposes training an attacker through a curriculum. The problem is practically relevant, and the experiments demonstrate substantial attack success and some generalization across models and benchmarks.

However, the method primarily combines the existing PISmith algorithm with a target-model curriculum, so its methodological novelty is limited. Limited novelty alone might be fine, as long as a paper proves strong empirical value. Hence, I was mainly looking for convincing evidence of effectiveness, reproducible curriculum-design principles, and benefits under comparable resource budgets. At present, incomplete budget reporting and missing controlled comparisons make it difficult to identify the source of the gains and assess the method’s practical value. I therefore lean toward borderline reject.

Critical strengths
  1. Practically relevant problem. Extremely sparse binary rewards can prevent an RL attacker from learning against robust target models. The paper addresses this obstacle with a clear and intuitive approach.

  2. Strong results for the selected curriculum. The Nano → Luna → Terra curriculum substantially improves attack success against Luna and Terra relative to direct training and the reported RL-Hammer baseline. The intermediate-stage ablations and pre-stage ASR measurements also support the explanation that a useful initialization must provide some successful attacks to reinforce.

  3. Meaningful transfer results. The trained attacker succeeds against several targets not used during training and transfers from AgentDyn to AgentDojo. These results provide evidence of generalization. Table 1 also shows that further training against Terra improves, rather than degrades, the reported attack success against Luna.

Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of the contribution. The approach retains PISmith and primarily changes the training targets and their order. Although the results establish that the selected curriculum works, they do not sufficiently distinguish the benefits of sequential training from those of better intermediate-target selection or additional training and interaction budgets.

  2. Insufficient training-cost reporting. The paper does not report cumulative training queries, update counts per stage, target-model token usage, or monetary costs for the compared methods. Curriculum selection and pre-stage testing costs are also not fully accounted for. Matching the number of evaluation attempts does not establish comparable training budgets. Consequently, the current results do not establish superiority under comparable resource constraints.

  3. Missing comparisons directly relevant to the central claim.

  • AutoInject [1] (or similar other approaches) that provide dense feedback are relevant alternatives for addressing reward sparsity and cold start, but they are not discussed or compared. An informative comparison would be to run AutoInject under the same querying budget as curriculum learning, and compare the ASR.
  • This concern is particularly important for RL-Hammer, since joint training against easier targets is a closely related alternative to the proposed sequential curriculum. However, the reported RL-Hammer baseline jointly trains against Llama-3.1-8B-Instruct and the final target, whereas the proposed method trains sequentially against Nano, Luna, and Terra. The comparison therefore changes both the target-model selection and the training schedule, with the cumulative budgets left unspecified. The reported failure of this RL-Hammer configuration does not establish that joint training would fail when given access to the same intermediate targets and comparable resources.

An informative comparison with any baseline would evaluate joint or mixed training using the same Nano, Luna, and Terra targets under matched cumulative query budgets, alongside token and monetary-cost reporting. However, the paper does not conduct such comparisons and does not report cost information for all the methods. Without these comparisons, it remains unclear how much of the improvement comes from sequential curriculum learning, better intermediate-target selection, or differences in training expenditure.

  1. Incomplete evaluation of capability retention after continued adaptation. The Luna results in Table 1 provide positive evidence of retention, but the paper does not systematically evaluate each checkpoint against all previously encountered targets. In particular, after adapting to Muse, the attacker is not re-evaluated against earlier targets. The evidence therefore remains incomplete for the practical claim that the attacker can be repeatedly reused and extended while retaining its existing capabilities.

[1] Chen, X., Zhang, J., & Tramèr, F. (2026). Learning to inject: Automated prompt injection via reinforcement learning.

Questions for authors
  1. Are the training budgets matched? Please report cumulative training attack attempts, actual target-model calls, and training updates for the curriculum, PISmith, and RL-Hammer experiments. Do these counts include intermediate stages, auxiliary targets, curriculum selection, and pre-stage testing? Do the main performance gaps persist under matched cumulative query budgets? Given the different costs of the target models, please also provide token usage or monetary-cost comparisons.

  2. Do the gains come from target selection or curriculum order? Using the same Nano, Luna, and Terra targets, training data, and comparable total budgets, does sequential curriculum training outperform mixed or joint training? Can longer training without introducing a new intermediate target achieve similar gains?

  3. How does the method compare with alternative solutions to cold start? Could the authors include AutoInject or another dense-feedback approach in a representative setting, with clearly specified access assumptions and comparable total budgets? This would help establish the advantages of a target-model curriculum over alternative approaches to the same problem.

  4. Does continued adaptation preserve existing capabilities? Could the authors provide a checkpoint-by-target evaluation matrix covering previously encountered targets, including Nano, and re-evaluate the Muse-adapted attacker against Luna and Terra? Table 1 already provides encouraging retention results for Luna after Terra training, but it remains unclear whether this behavior persists through further adaptation.

  5. How reproducible is curriculum construction? How were candidate targets, robustness rankings, stage transitions, and stopping conditions determined? How many candidate probes or training runs were required to identify the reported successful curriculum? These details are important for assessing both applicability to new targets and end-to-end cost.

Reviewer_HxhgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies RL-based prompt injection red-teaming against frontier LLMs. The authors identify a cold-start problem: against a robust target, every attack from the initial attacker fails, so RL receives no reward signal. They propose training the attacker (Qwen3-4B, with PISmith as the RL algorithm) against a curriculum of increasingly robust targets (GPT-5-nano -> GPT-5.6-Luna -> GPT-5.6-Terra). Each next target is chosen so that the current attacker already succeeds against it occasionally. Although the attacker is trained only against GPT-5.6-Terra on AgentDyn, it transfers to six other target LLMs and to another benchmark, AgentDojo.

Critical strengths
  1. The problem is timely, and the transfer results are interesting. Prompt injection against agents is a practical security concern, and how to do RL-based red-teaming against frontier models (where the attacker almost never succeeds at the beginning) is a question many people currently care about. I think the most informative part of the paper is not the main comparison in Table 1 but the transfer results (Tables 2–4). An attacker trained against one frontier model achieves nontrivial ASR on six other frontier models and on another benchmark without further training, which suggests it learns something more general than target-specific exploits. The case study in Appendix D also helps. It shows the attacker moving from "overriding the user’s task" to "rewriting the task as a workflow that includes the injected step", which gives some intuition about why frontier models fail.
  2. The narrative and the guiding principle are clear and easy to reason about. The informative-group analysis (Eq. 2) and the pre-stage ASR test give a simple, cheap criterion for whether a curriculum stage is trainable. The paper is well written and the story is easy to follow.
Critical weaknesses
  1. The baselines are closer to ablations than to comparable baselines. PISmith and RL-Hammer in Table 1 are essentially the same RL pipeline without a curriculum. Their 0% ASR follows from how the cold-start problem is defined rather than being an experimental finding. When every compared method is near zero, the comparison no longer tells us much. Reasonable alternatives are missing: search-based or adaptive attacks (e.g., TAP, PAIR, Nasr et al. 2026, which shows adaptive attacks can bypass strong defenses), other fixes for cold start (SFT warm-start, a stronger attacker), and an equal-budget comparison, since the curriculum runs several RL stages and the baselines run one.
  2. The cold-start problem may partly come from the setup itself, and the curriculum may not generalize. The base attacker gets 0% on every target, including weaker open models, and is even worse than a single static template on SecOPD. The attacker template (Appendix B) asks the attacker to override the task and sound authoritative and urgent, which is the opposite of the strategy that eventually works (Appendix D). A better template might remove much of the cold start. In addition, the successful curriculum stays within one model family and seems to have been found post hoc (GPT-4o-mini fails, GPT-5-nano works), so it is unclear how the method works when no graded family of models is available.
Questions for authors
  1. How sensitive is the cold-start problem to the attacker prompt template? If the Appendix B template is replaced with a neutral one (e.g., without "override" and "authoritative and urgent"), or with one that hints at the workflow-rewriting strategy, does direct RL against GPT-5.6-Luna/Terra still receive zero reward?
  2. What are the total query and training budgets of the curriculum compared with the baselines? Do the baselines still get 0% ASR under the same total budget?
  3. How do search-based or adaptive attacks (e.g., TAP, PAIR, or the adaptive attacks in Nasr et al. 2026) perform against GPT-5.6-Luna/Terra on AgentDyn? Even if they are more expensive at inference time, this would show whether the frontier targets are hard in general or only hard for direct RL.
  4. Does a simple SFT warm-start (e.g., on successful injections from a search-based attack or from the GPT-5-nano stage) solve the cold-start problem as well as the curriculum does?
  5. How many candidate targets and stages were tried before the reported curriculum was found, and what is the actual cost of running Algorithm 1? Does the method work with a cross-family curriculum (e.g., when the frontier target has no weaker sibling models)?
  6. Are the results from a single training run? I would like to see variance reported across seeds if possible.
Reviewer_tpJsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission proposes a curriculum learning method to solve the cold-start problem in RL. They identify the reward sparsity problem and try to train the model against a series of models, from weaker models to stronger models. They provide many extensive experiments to demonstrate their methods and findings.

Critical strengths
  1. Very readable for readers. The main content is easy to follow.

  2. The experiments show a very clear improvement. The ablation study also demonstrates that the choice of middle models is critical for curriculum learning.

  3. In section 4.1-4.3, it shows that the trained model can attack some targets and perform very in some benchmarks without seeing them during training.

Critical weaknesses
  1. The novelty is limited. This method relies heavily on standard curriculum learning without proposing a new optimization algorithm. The authors should claim more clear about what the methodological novelty is beyond ordering target models.

  2. Can the improvement be gained by additional training budget? Or by any warm-start methods like SFT? I'm not sure whether purely cold-start RL is common in this field.

  3. This submission provides the design of selecting targets, but does not systematically evaluate the proposed curriculum selection algorithm among different target sets. How is the nonzero-ASR criterion implemented in practice?

Questions for authors

Please refer to the Weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.