- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CICD-Guard is a black-box jailbreak defense that rewrites user prompts, scores the original and rewritten versions with multiple LLMs, and uses absolute risk and risk changes to decide whether to block or modify the request.
Critical strengths
The authors evaluate two target models with different safety behaviors and provide ablations and sensitivity analyses.
Critical weaknesses
(1) Deployment feasibility is not demonstrated. The default configuration generates five rewrites and scores all six representations with four models, requiring 24 scoring evaluations per request. The computational overhead is tremendous. Despite using several large auxiliary models, the paper reports no end-to-end latency, token consumption, throughput, or cost comparison. This overhead applies to benign requests as well, which is in fact the majority of user request, making the trade-off particularly important for ordinary conversational workloads.
(2) The central robustness assumption remains unvalidated. The method assumes that rewriting reliably exposes harmful intent while preserving legitimate constraints. However, increased risk after rewriting may reflect semantic distortion rather than previously concealed intent. Moreover, an attacker could target the rewriter or scorers, including through combined contextual and adversarial-suffix attacks.
The paper tests three attacks, but does not clearly establish generalization to held-out attack families or robustness against attacks optimized against the complete defense. This is a concern about the mechanism’s security assumptions, not merely a request for more benchmarks. Adaptive evaluation has established precedent in adversarial ML [1, 2], and jailbreak defense papers like Circuit Breakers evaluate diverse LLM attack familys [3].
(3) The ensemble configuration is insufficiently justified beyond one setting. Table 2 studies scorer and rewrite counts only under CIPHER on Llama-3-8B-Uncensored. Adding a fifth scorer reduces DSR from 85% to 66% and increases ORR from 8% to 32%, suggesting substantial sensitivity to model composition. The paper should clarify model-subset selection, score calibration, and whether configuration selection was separated from final evaluation.
(4) Over-refusal does not capture benign task quality. Rewriting could remove constraints or change query meaning. Downstream answer quality and rewriting fidelity remain unmeasured. Favorable benign-task results would address this concern.
(5) Presentation needs improvement. Figure 3’s caption provides little explanation, uncertainty reporting is unclear, and the related-work discussion needs a sharper comparison with closely related defenses.
[1] Athalye, Carlini, and Wagner. Obfuscated Gradients Give a False Sense of Security. ICML, 2018.
[2] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. ICLR, 2025
[3] Zou et al. Improving Alignment and Robustness with Circuit Breakers. NeurIPS, 2024.
Questions for authors
-
How robust is the system to held-out attack families and attacks targeting rewriting or scoring?
-
What are the measured latency, token usage, and cost relative to the baselines? Does the selected configuration remain preferable when cost is considered?