- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a new jailbreak attack called CARD, which uses category-specific refusal directions to improve attack performance. It extracts refusal directions from harmful and harmless prompts for different harm categories and uses these directions to guide adversarial suffix optimization. When the category is unknown, a linear classifier is used to select the corresponding direction. Experiments on three models and three datasets show that CARD outperforms most baselines. Ablation studies show that category-specific directions are more effective than a single global direction.
Critical strengths
- The experiments show strong attack performance across different models and datasets. The ablation studies also demonstrate the benefits of using category-specific directions instead of a global direction.
Critical weaknesses
-
The experimental setup in Section 4.4 is not sufficiently clear. Specifically, the authors evaluate CARD on AdvBench and HarmBench, but it is unclear whether they attack all harmful requests in these datasets or only a subset. The paper does not seem to provide this information, even in the supplmentary material. If the authors only use a subset, I suggest evaluating on the 50 harmful requests from AdvBench used in I-GCG [1], which are publicly available. This would make the results more convincing and easier to compare with prior work. In addition, the choice of baselines in Table 2 is somewhat confusing. The authors compare CARD with GCG, PAIR, TAP, and AutoDAN-HGA. However, based on the results in Table 1, I believe that including stronger baselines such as TAO-Attack and PIRAS/SOM would provide a more meaningful comparison.
-
The authors evaluate CARD on Qwen2-1.5B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3. Although these are commonly used models, Llama-2-7B-Chat is also a standard model in jailbreak research. It has relatively strong safety alignment and is generally more difficult to jailbreak than Llama-3.1-8B-Instruct. I suggest that the authors include Llama-2-7B-Chat or other models with stronger safety alignment to better evaluate the effectiveness of CARD against more challenging targets.
[1] Improved Techniques for Optimization-Based Jailbreaking on Large Language Models
Questions for authors
Please refer to the Critical Weaknesses.
Response to Reviewer X4nT
Response to Reviewer
We are grateful for the detailed comments. Here are our reponses to your questions.
Weakness 1
CARD is evaluated on all 520 AdvBench and all 400 HarmBench prompts, not a subset. This avoids subset-induced bias and tests effectiveness across the full prompt distribution. Section 4.1 specifies these benchmark sizes.
Comparison with I-GCG
We compare CARD with I-GCG on the same 50 AdvBench and 50 HarmBench prompts, selected with seed 42. I-GCG performs better overall. Running I-GCG under its official configuration on the full 520/400 benchmarks across three models would require considerable GPU time, so this is a reference comparison rather than a full reproduction.
TAO-Attack and PIRAS/SOM on the full benchmarks
CARD, TAO-Attack, and PIRAS/SOM are evaluated on all 520 AdvBench and 400 HarmBench prompts. We use their official-code parameters: TAO-Attack uses 500 steps, batch size 256, top-k 256, and suffix length 20; PIRAS/SOM uses 10,000 SOM iterations and a 500×500 grid. Full configurations are in Appendix I.1.
CARD generally achieves lower ASR than these near-saturated SOTA baselines. For example, TAO-Attack reaches 100% on AdvBench for Llama-2-7B-chat, Vicuna-7B-v1.5, and Mistral-7B-Instruct-0.2 [1]. Our goal is cross-benchmark generalization, not surpassing near-ceiling benchmark-specific scores. On BeaverTails, TAO-Attack is consistently lower than CARD; PIRAS/SOM is lower on Qwen2 and Mistral but higher on Llama. On AdvBench and HarmBench, CARD remains competitive.
Weakness 2
We thank the reviewer for this suggestion. Following recent LLM safety studies [2, 3, 4], we use Llama-3.1-8B-Instruct as the Llama-family representative. We have not established a direct comparison of their safety alignment under our protocol. Given the limited compute and rebuttal timeline, we cannot run Llama-2-7B-Chat with CARD and all baselines on all benchmarks under the same evaluation and manual-adjudication protocol. We will evaluate Llama-2-7B-Chat and more models in future work.
References
[1] Zhi Xu, Jiaqi Li, Xiaotong Zhang, Hong Yu, and Han Liu. TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models. ICLR 2026.
[2] Hamin Koo, Minseon Kim, and Jaehyung Kim. Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges. ICLR 2026.
[3] Hongye Cao, Sijia Jing, Yanming Wang, Ziyue Peng, Zhixin Bai, Zhe Cao, Meng Fang, Fan Feng, Jiaheng Liu, Boyan Wang, Tianpei Yang, Jing Huo, Yang Gao, Fanyu Meng, Xi Yang, Chao Deng, and Junlan Feng. SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks. ICLR 2026.
[4] Seungwon Jeong, Jiwoo Jeong, Hyeonjin Kim, Yunseok Lee, and Woojin Lee. SlotGCG: Exploiting the Positional Vulnerability in LLMs for Jailbreak Attacks. ICLR 2026.