- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Faster-GCG, an efficient variant of the Greedy Coordinate Gradient (GCG) jailbreak attack for aligned large language models. The authors identify three major sources of inefficiency in GCG: inaccurate gradient-based token ranking in the discrete token space, inefficient uniform sampling among candidate tokens, and redundant evaluation of previously explored adversarial suffixes. To address these issues, Faster-GCG introduces distance-regularized candidate selection to improve the reliability of gradient estimates, temperature-controlled sampling to prioritize promising candidates while maintaining exploration, and a visited-suffix marking mechanism to avoid repeated evaluations. Experiments on multiple open-source LLMs and jailbreak benchmarks show that Faster-GCG achieves comparable or better attack success rates than GCG while reducing the number of candidate evaluations by approximately 4–8×. A stronger Faster-GCG++ configuration further achieves an average ASR of 78.1% across five models, demonstrating the effectiveness of the proposed efficiency improvements for discrete optimization-based jailbreak attacks.
Critical strengths
- The paper provides a systematic analysis of GCG and identifies three concrete sources of inefficiency: inaccurate gradient-based candidate ranking, inefficient uniform sampling, and redundant evaluations of previously explored suffixes. This clear problem diagnosis provides strong motivation for the proposed improvements.
- Faster-GCG introduces three lightweight modifications that directly address the identified limitations. The overall framework remains close to the original GCG, making the method easy to understand, implement, and integrate with existing optimization-based jailbreak approaches.
- In particular, the distance regularization is motivated through a Taylor expansion showing that the error of the first-order gradient approximation is related to the embedding distance between tokens. The empirical improvement in ranking correlation further supports this motivation.
Critical weaknesses
- Similar to GCG, Faster-GCG requires access to model gradients. The authors acknowledge that transfer attacks to proprietary models are possible, but also note that finding suitable surrogate models remains challenging, which limits the practical impact on closed-source LLMs.
- For example, the visited-suffix marking mechanism removes repeated evaluations, but the paper reports that only about 6% of suffixes are revisited. This suggests that this component alone contributes relatively limited algorithmic novelty and efficiency gain.
- Faster-GCG++ changes not only the optimization method but also the attack surface by increasing the suffix length to 40 tokens and changing the suffix placement. Therefore, its stronger ASR cannot be attributed solely to the proposed Faster-GCG techniques, and the comparison with other methods is somewhat less controlled.
- The Taylor-expansion analysis motivates why embedding distance may correlate with approximation error, but the specific regularizer is still heuristic. The paper does not show that this form is theoretically optimal, nor does it deeply analyze when embedding distance is a reliable proxy for higher-order approximation error.
Questions for authors
Please see weakness
3 comments
response to rebuttal
The rebuttal partially solved my concern. Weakness 1,3,4 are not good enough. I will maintain the score
Thank you for your careful reading of our rebuttal. We provide further evidence and elaboration.
W1 (white-box access). Although the white-box requirement remains a limitation, we showed in our response to Reviewer Hkcw that white-box optimization provides new understanding of each model's token-space vulnerabilities. We repeat that analysis below.
(From our response to Reviewer Hkcw, Q1: What new understanding is provided?) Suffix analysis reveals distinct model-dependent patterns that reflect each model's token-space vulnerabilities, and Faster-GCG's improved optimization provides stronger evidence of these patterns. In matched base-setting comparisons (GCG → Faster-GCG):
- Llama2 favors punctuation and delimiter structures; punctuation token mass rises from 39.2% to 43.8%.
- Llama-3.1 exhibits extremely low-norm embeddings; token mass with norm < 0.01 rises from 3.65% to 4.85%.
- Qwen3 exhibits language-switching/translation patterns; suffix coverage rises from 40% to 48%.
- Qwen3.5 exhibits vision/image boundary markers; coverage rises from 11% to 28%.
- Gemma3 exhibits identifiers and unused markers that share an embedding direction; prototype alignment increases slightly (0.1105 → 0.1260).
- Vicuna favors common language fragments; reference-seen fragment mass rises from 80.73% to 91.50%.
We will add these findings to the revised manuscript.
W3 (Faster-GCG++ attribution). Below, we clarify the threat model, the optimization space and the controlled comparisons. The threat model (Appendix B) allows the attacker to construct any user input, of arbitrary length. This threat model is compatible with both white-box and black-box attacks. For the GCG family, the optimization space is restricted to a 20-token adversarial suffix. This defines a fixed white-box discrete optimization problem in which all methods (except I-GCG) share the same initialization, search budget and search space, so Faster-GCG, TAO, GCG-PS and GCG are directly comparable in this setting. We added the 40-token prepended setting to obtain stronger performance under the stated threat model; within that setting, Faster-GCG++ and GCG++ are directly comparable.
We will revise the abstract and introduction to report the controlled ASR comparison between Faster-GCG and GCG as the main claim. We will also restructure Table 1 so that it contains only fair comparisons:
- The first block will compare GCG, GCG-PS, TAO and Faster-GCG (with AutoDAN and I-GCG removed).
- The second block will compare GCG++ and Faster-GCG++.
- The full comparison under the any user input threat model is already provided in Table A3.
In both controlled comparisons, Faster-GCG (and its ++ variant) achieves the best average loss and ASR.
W4 (heuristic regularizer). Under a locally isotropic curvature approximation , the second-order Taylor term becomes , which motivates as a simple alternative that ignores directional curvature. This approximation replaces directional curvature with a single scalar; it does not compute the Hessian or assume that second-order terms dominate for finite token replacements. We compared (the original setting) with at fixed and .
| Model | ||
|---|---|---|
| Gemma3-4B | 0.4528 | 0.4512 |
| Qwen3-8B (reasoning) | 1.1160 | 1.1209 |
We detect no difference on either model ( and ). This is consistent with our argument in the first-round rebuttal that the distance term expresses a soft locality preference: using either the norm or the squared norm as the regularizer yields similar performance in the tested settings.
Thank you for recognizing the systematic diagnosis and ease of integration. We number the four weaknesses in their original order.
W1. White-box access and limited transfer. We agree that gradient access and weak transfer limit applicability to proprietary services. The main contribution is more efficient white-box evaluation. GCG-like optimization also serves defense construction: PAT [1] and RPO [2] optimize defensive prompts and evaluate white-box and black-box attacks.
W2. Limited standalone contribution of marking. The 5.9% revisit rate was measured under vanilla GCG. With distance regularization and temperature sampling, but marking disabled, the revisit rate rises to 29.3% (the number will be added to the revised manuscript). This supports marking's greater importance in the combined configuration, consistent with the 42.2% versus 58.2% ablation. Marking itself is standard; the interaction of the three components is the contribution of this paper.
W3. Confounded Faster-GCG++ comparison. We agree that length and placement must be controlled. Under the same 40-token prepended setting, GCG++ achieves an average ASR of 76.3%, compared with 78.1% for Faster-GCG++. Separately, the 20-token appended comparison is 65.8% versus 69.5%.
W4. Heuristic distance regularizer. The distance term expresses a soft locality preference: discrete replacements have finite embedding displacements that cannot generally be scaled down while remaining valid tokens. A local gradient can therefore be extrapolated beyond its reliable range. We use λ∥δ∥ to balance predicted improvement against displacement, not to estimate the Taylor remainder, since we cannot determine whether the first-, second-, or third-order term dominates at this scale. Its linear form is empirical; we provide the Taylor expansion as an informal justification. This will be clarified in the revised manuscript.
References:
[1] Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Fight Back Against Jailbreaking via Prompt Adversarial Tuning. NeurIPS, 2024.
[2] Andy Zhou, Bo Li, and Haohan Wang. Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks. NeurIPS, 2024.