- Rating
- 2(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces JARC, a two-stage jailbreak framework. It first generates a response template with sensitive details replaced by placeholders under a safety-research context. A separate session or model then supplies the missing values through dialogue or function calling, and the final response is assembled locally. The paper evaluates self-generated and auxiliary-generated templates, compares interaction interfaces, and tests five implementations produced through a guided coding-agent workflow. Experiments on AdvBench and HarmBench report substantial attack-success and token-efficiency advantages over the evaluated baselines.
Critical strengths
-
The proposed attack achieves substantial gains across the evaluated models. AUXGEN reaches an average ASR of 80.7% on the three commercial targets, compared with 37.5% for MANY-SHOT, while using fewer counted tokens. The full HarmBench evaluation, additional PAIR budget checks, and API-based token accounting provide useful support for the effectiveness and efficiency claims.
-
The paper investigates the attack beyond aggregate success rates, examining context, template generation, interaction interfaces, and stage-wise refusal. Five independently generated implementations show that the approach is not limited to one handcrafted realization. The follow-up evaluation on a newer model also transparently reports reduced but remaining effectiveness.
Critical weaknesses
-
The comparisons do not sufficiently establish the advantage over the closest methodological alternatives. The baselines mainly cover long-context prompting and iterative refinement, while the function-calling attack discussed in Appendix B is not evaluated. Related masking and decomposition approaches, such as SATA and DrAttack, are also absent from the comparison. The internal ablations are useful, but a direct comparison would better establish the additional benefit of JARC’s stage-separated response construction.
-
Some mechanism interpretations need stronger controls and clearer correspondence between experiments. The single-turn versus two-stage comparison does not isolate session reset, while victim-side paraphrasing changes template content and wording as well as its generation history. The relationship between refusal attribution and final ASR also needs clarification: GPT-5.1 AUXGEN with dialogue filling achieves 90% ASR in Table 5, whereas the separate fixed-template experiment in Table 23 reports 21/50 Stage-2 refusals. These figures' relationship matters for explaining the main results.
Questions for authors
-
Could the authors compare JARC with JailbreakFunction and at least one closely related masking or decomposition method, such as SATA or DrAttack, using the same target versions, evaluators, and comparable attack budgets?
-
Could the authors isolate session reset by holding the template and Stage-2 request fixed while varying whether Stage-1 history is retained? For the paraphrasing experiment, how much harmful detail and placeholder structure is preserved after rewriting?
-
How do the runs, templates, and refusal definitions in Table 23 correspond to Tables 5 and 22? Could the authors provide the per-example relationship between stage refusals and final success labels, including whether a refusal can still supply enough content for successful assembly?
Thank you for adding the decomposition/masking baselines and the session-reset control. The correction from 90% to 56% explains the discrepancy and shows overstated performance for Codex-Max. And as other reviewer noted, the auxiliary template alone often passes the judge. The five agent-generated implementations cannot support broader claim about attack diversity. Therefore, I am revising my score from 3 to 2.