acceptodds
Under review as a conference paper at ICLR 2027

For Research Purpose Only - Jailbreaking Large Language Models with Jailbreaking Research

Abstract

We introduce a simple yet powerful jailbreak framework for LLMs: Jailbreaking with Alignment Research Context (JARC). JARC interacts with the victim LLM through a two-stage process. The first stage establishes the context of research on LLM alignment and then requests for answering questions in a neutralized template. The second stage asks the victim model to fill in the template blanks, framing the task as a cloze test puzzle via blank-filling function calls. As a static attack strategy that does not rely on an auxiliary red-teaming model, JARC demonstrates high attack effectiveness with significantly lower token usage compared to existing attacks. Specifically, JARC achieves on average 80.7% Attack Success Rate (ASR) when attacking heavily safely-aligned frontier LLMs: GPT-5.1-Codex-Max, Gemini 3.1 Pro, and Claude Sonnet 4.5, yielding a 40%+ ASR improvement compared to state-of-the-art baseline attacks. We further encode the design pattern of JARC as a lightweight skill-based workflow and utilize Claude Code to instantiate the framework, demonstrating that agentic harnesses can autonomously generate diverse realizations of the attack while retaining strong effectiveness. Beyond attack effectiveness, we systematically investigate the robustness, contributing factors, and execution stability of JARC. Our study reveals the following two key messages: (1) *The scaling of agentic capabilities may not result in proportionally enhanced safety alignment.* Safety mechanisms that resist direct harmful requests may become less reliable when the same generation task is decomposed across stages, transferred across models or sessions, or expressed through agentic interfaces. As a result, agentic capabilities such as tool calling and multi-stage task execution, along with disparities in alignment efforts across LLMs can translate to new attack surfaces. (2) *Agentic harnesses can enable the "scaling" of attacks themselves.* These "dual-use" harnesses can be easily re-engineered to generate automated, diverse, highly effective jailbreaks at low cost.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Trustworthy, LLM safety, Jailbreaking, Agentic LLMs, Tool-calling, Long-context, Red teaming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dGN5OpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JARC, a two-stage jailbreak framework. It first generates a response template with sensitive details replaced by placeholders under a safety-research context. A separate session or model then supplies the missing values through dialogue or function calling, and the final response is assembled locally. The paper evaluates self-generated and auxiliary-generated templates, compares interaction interfaces, and tests five implementations produced through a guided coding-agent workflow. Experiments on AdvBench and HarmBench report substantial attack-success and token-efficiency advantages over the evaluated baselines.

Critical strengths
  1. The proposed attack achieves substantial gains across the evaluated models. AUXGEN reaches an average ASR of 80.7% on the three commercial targets, compared with 37.5% for MANY-SHOT, while using fewer counted tokens. The full HarmBench evaluation, additional PAIR budget checks, and API-based token accounting provide useful support for the effectiveness and efficiency claims.

  2. The paper investigates the attack beyond aggregate success rates, examining context, template generation, interaction interfaces, and stage-wise refusal. Five independently generated implementations show that the approach is not limited to one handcrafted realization. The follow-up evaluation on a newer model also transparently reports reduced but remaining effectiveness.

Critical weaknesses
  1. The comparisons do not sufficiently establish the advantage over the closest methodological alternatives. The baselines mainly cover long-context prompting and iterative refinement, while the function-calling attack discussed in Appendix B is not evaluated. Related masking and decomposition approaches, such as SATA and DrAttack, are also absent from the comparison. The internal ablations are useful, but a direct comparison would better establish the additional benefit of JARC’s stage-separated response construction.

  2. Some mechanism interpretations need stronger controls and clearer correspondence between experiments. The single-turn versus two-stage comparison does not isolate session reset, while victim-side paraphrasing changes template content and wording as well as its generation history. The relationship between refusal attribution and final ASR also needs clarification: GPT-5.1 AUXGEN with dialogue filling achieves 90% ASR in Table 5, whereas the separate fixed-template experiment in Table 23 reports 21/50 Stage-2 refusals. These figures' relationship matters for explaining the main results.

Questions for authors
  1. Could the authors compare JARC with JailbreakFunction and at least one closely related masking or decomposition method, such as SATA or DrAttack, using the same target versions, evaluators, and comparable attack budgets?

  2. Could the authors isolate session reset by holding the template and Stage-2 request fixed while varying whether Stage-1 history is retained? For the paraphrasing experiment, how much harmful detail and placeholder structure is preserved after rewriting?

  3. How do the runs, templates, and refusal definitions in Table 23 correspond to Tables 5 and 22? Could the authors provide the per-example relationship between stage refusals and final success labels, including whether a refusal can still supply enough content for successful assembly?

1 comment

Thank you for adding the decomposition/masking baselines and the session-reset control. The correction from 90% to 56% explains the discrepancy and shows overstated performance for Codex-Max. And as other reviewer noted, the auxiliary template alone often passes the judge. The five agent-generated implementations cannot support broader claim about attack diversity. Therefore, I am revising my score from 3 to 2.

Reviewer_EjR2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes JARC, a method for getting LLMs to produce harmful content by splitting the task into two separate stages. In the first stage, the attacker has a conversation with a model framed as legitimate research on AI safety and jailbreaking, then asks it to write a detailed step-by-step template for a harmful task where all the sensitive words are replaced with blank placeholders. In the second stage, the attacker starts a fresh session, sometimes with a different model, and asks it to fill in those blanks one by one, through a tool call or dialogue call. The attacker then merges the template and the filled-in values on their own machine to assemble the final harmful response. The key idea is that each stage looks fairly harmless by itself, so safety training that would block a direct request fails to catch the attack once it is broken apart across different stages, sessions, and interfaces. The experiments show high success rates against several models including GPT-5.1-Codex-Max, Gemini 3.1 Pro, and Claude Sonnet 4.5, while using far fewer tokens than existing attacks like PAIR or many-shot prompting. The paper also shows the whole process can be turned into an automated workflow, where a coding agent like Claude Code builds working versions of the attack on its own.

Critical strengths
  • There is an interesting contribution that safety behavior changes when generation is split across intermediate artifacts, models, sessions, and interfaces.
  • The empricial results of the proposed JARC is strong against other baselines, such as PAIR, MANY-SHOTS. Moreover, the paper measures the ASR under different judge systems (Llama-Guard, GPT-4 and Harmbench Judges) to mitigate the judging bias problem.
  • The static-versus-agentic comparison includes five independent runs, standard deviations, and confidence intervals, rather than reporting only a successful implementation.
Critical weaknesses
  • My biggest concern is that it is not clear how much of the attack's success should actually be credited to the victim model in the JARC-AUXGEN setting. Here an auxiliary model writes the template, the victim only fills in the blanks, and the attacker stitches the two outputs together at the end. So the final harmful answer is not something the victim produced on its own. Measuring how harmful that final answer is tells you the whole pipeline worked, but it does not tell you how much of the unsafe content came from the victim versus the auxiliary model. The template from the auxiliary model might already be quite harmful on its own, or already be enough to pass the benchmark judge before the victim ever fills anything in. This does not mean cross-model jailbreaking is not real, since supplying the missing harmful details can count as a genuine failure by the victim. But the paper should separate that failure from harm that was really inherited from the auxiliary model.
  • There is a specific inconsistency worth raising with the paper. Table 5 reports 90% ASR under both judges for GPT-5.1-Codex-Max with AuxGen and dialogue fill-in, but Table 23 reports 21 out of 50 Stage-2 dialogue refusals for what appears to be the same victim and variant, with no Stage-1 refusals. If these describe the same outputs and refusals count as failures, the highest possible success rate would be (50 minus 21) over 50, or 58%, not 90%. I would ask to clarify explicit run identifiers, template provenance, prompt and configuration versions, and a clear definition of refusal for these tables, which matters especially because the stage-wise breakdown is a central part of their explanation for why the method works.
  • The paper should justify why its selected victims are representative of the frontier at the stated evaluation date, particularly why Sonnet 4.5 is used as the main Claude victim when Sonnet 4.6 was already available and is used elsewhere in the study.
  • Table 1 reports approximately 0% SelfGen ASR on Claude Sonnet 4.5, while Table 23 reports Stage-1-led refusal on 41/50 SelfGen examples. These results demonstrate an important limitation of the proposed research-framing and placeholder-generation mechanism on that victim. AuxGen bypasses this bottleneck by obtaining the template from GPT-4o, but its success does not establish that Claude’s Stage-1 defenses were overcome. Moreover, because the final evaluated response combines auxiliary-generated structure with target-generated slot values, reconstructed-output ASR does not by itself establish the target’s unsafe contribution. The paper should separately report pipeline success and victim-attributed unsafe assistance, supported by template-only controls and contextual evaluation of the target’s actual completions.
  • The paper's explanations for why the attack works are not always isolated cleanly by its experiments. First, the claim that resetting the session between stages is what helps is not tested on its own. The experiment in Table 21 compares the full two-stage procedure against a single-turn version that does everything at once, but that changes several things at the same time, including the number of model calls, how the text is generated, and how much context the model sees. So you cannot tell whether the session reset specifically mattered or one of the other changes did. A cleaner test would keep everything the same, the same template and the same two separate calls, and flip only one thing: whether the model remembers the earlier conversation or starts fresh. Second, the paper cannot fully separate who wrote the template from what the template actually says. The paraphrasing experiment is reasonable, since it starts from the same template and only has the victim rewrite it, and ASR drops from 86% to 52%, but rewriting a template can also change its content, such as how specific the harmful details are, how the steps are organized, where the blanks sit, or how much safety language it includes. So the experiment really shows that having the victim rewrite the template changes the outcome, not the sharper claim the paper wants, which is that the origin of the template matters even when the content is held constant. These are different things, and the test does not cleanly establish the second one.
Questions for authors
  1. In the JARC-AUXGEN setting, how much of the harmful content in the final response is already present in the auxiliary model's template before the victim fills in any slot values? Can you report how often the template alone passes the benchmark judge?

  2. Can you separate pipeline success from the victim's own unsafe contribution, for example by scoring the templates on their own, adding controls where the blanks are never filled in, and judging the victim's slot-value outputs in their original context?

  3. How do you reconcile the 90% ASR in Table 5 for GPT-5.1-Codex-Max (AuxGen, dialogue fill-in) with the 21/50 Stage-2 dialogue refusals reported in Table 23 for the same victim and variant? Do these tables use the same outputs, prompts, model snapshots, and definition of refusal?

  4. Why is Claude Sonnet 4.5 used as the main Claude victim when Sonnet 4.6 was already available and is used elsewhere in the study? More broadly, why are the chosen victims representative of the frontier at your stated evaluation date?

  5. Given that SelfGen ASR is near 0% on Claude Sonnet 4.5 with 41/50 Stage-1 refusals, does AuxGen's success show that Claude's Stage-1 defenses were actually overcome, or only that the refused template can be sourced from a more compliant model?

  6. For the session-reset claim, can you provide a controlled experiment that holds the template and the two-call structure fixed and varies only whether the earlier conversation is retained or reset?

  7. For the template-provenance claim, how do you rule out that victim-side paraphrasing changed the template's content (its specificity, structure, placeholder placement, or safety language) rather than only its origin?

Reviewer_JLtyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a jailbreak framework called JARC (Jailbreaking with Alignment Research Context). Specifically, it consists of two stages: (1) first framing an alignment research context and asking an LLM to explain the (harmful) topic in a neutralized template with sensitive keywords masked, and (2) asking the victim model to instantiate the masked slot by calling a blank-filling tool function. The proposed attack framework is fully black-box, efficient in token consumption, and effective in jailbreaking state-of-the-art LLMs.

Critical strengths
  • The paper has a nice structure and is easy to follow. The two-stage design of the JARC jailbreak framework is well explained.

  • The experimental results are comprehensive, particularly covering various frontier commercial and open-sourced LLMs. The improvement of attack performance (ASR, token consumption) over baselines is convincing and consistent across victim models.

  • The authors followed the responsible disclosure practices to notify OpenAI and Anthropic in advance, and coordinate mitigation efforts

Critical weaknesses
  • My main concern is that JARC-SelfGen (only prompts the victim LLM) does not attain strong attack performance on frontier commercial models, especially weak on HarmBench-80. These results seem to suggest that the safeguard mechanism of frontier models is effective against academic research framing (stage 1 of JARC). While the attack performance of JARC-AuxGen is impressive, it leverages the auxiliary (less-aligned) GPT-4o for the first stage, which is not a typical setup for LLM jailbreaks. Note that if an auxiliary LLM is allowed, an adversary can potentially completely resort to this LLM (or even earlier models) to generate harmful responses (no need to query the victim LLM).

  • The evaluated defense strategies are fairly limited, considering the existing literature on jailbreak defenses. I would expect a broader discussion on potentially effective countermeasures against JARC.

  • Most compared jailbreak baselines are static. Only PAIR is considered as a dynamic baseline. According to the introduction, the paper could be strengthened by including a few more competitive dynamic baselines.

Questions for authors
  1. Can the attack success rate of JARC-SelfGen be improved on Frontier commercial models? Can you explain why there is a large drop in ASR attained by JARC-SelfGen from AdvBench-50 to HarmBench-80 (e.g., on Gemini 3.1 Pro)?

Comments on the paper

1 comment

Dear Reviewers,

With the discussion period closing on November 18, please take a moment to read the authors’ responses. Please also indicate whether any further clarification or follow-up is needed.

Thank you for your time and contributions!
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.