acceptodds
Under review as a conference paper at ICLR 2027

Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Models

Abstract

Aligned Large Language Models (LLMs) have attracted significant attention for their safety, particularly in the context of jailbreak attacks that attempt to bypass guardrails via adversarial prompts. Among existing approaches, the Greedy Coordinate Gradient (GCG) attack pioneered automated jailbreaks through discrete token optimization; however, its low sample efficiency limits practical applicability. In particular, GCG requires approximately 256K evaluations per harmful behavior to achieve a satisfactory jailbreak success rate, due to the inherent difficulty of the underlying discrete optimization problem. In this work, we identify three key factors that limit the sample efficiency of GCG: inaccurate gradient-based estimation, inefficient uniform sampling, and repeated evaluation of previously explored suffixes. To address these issues, we propose Faster-GCG, a streamlined variant of GCG that incorporates distance-based regularization for improved estimation, temperature-controlled sampling for more effective exploration, and a visited-suffix marking mechanism to avoid redundant evaluations. Faster-GCG reduced the required evaluations to 32K, achieving up to 8x improvement in sampling efficiency compared to GCG, without significant degradation on performance. A separate Faster-GCG++ setting uses a 40-token suffix; it attained an average jailbreak success rate of 78.1% across five aligned LLMs, and achieved 88.7% against Qwen3.5-4B, outperforming state-of-the-art white-box jailbreak methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM jailbreak attack, LLM safety, discrete optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hACKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Faster-GCG, an efficient variant of the Greedy Coordinate Gradient (GCG) jailbreak attack for aligned large language models. The authors identify three major sources of inefficiency in GCG: inaccurate gradient-based token ranking in the discrete token space, inefficient uniform sampling among candidate tokens, and redundant evaluation of previously explored adversarial suffixes. To address these issues, Faster-GCG introduces distance-regularized candidate selection to improve the reliability of gradient estimates, temperature-controlled sampling to prioritize promising candidates while maintaining exploration, and a visited-suffix marking mechanism to avoid repeated evaluations. Experiments on multiple open-source LLMs and jailbreak benchmarks show that Faster-GCG achieves comparable or better attack success rates than GCG while reducing the number of candidate evaluations by approximately 4–8×. A stronger Faster-GCG++ configuration further achieves an average ASR of 78.1% across five models, demonstrating the effectiveness of the proposed efficiency improvements for discrete optimization-based jailbreak attacks.

Critical strengths
  1. The paper provides a systematic analysis of GCG and identifies three concrete sources of inefficiency: inaccurate gradient-based candidate ranking, inefficient uniform sampling, and redundant evaluations of previously explored suffixes. This clear problem diagnosis provides strong motivation for the proposed improvements.
  2. Faster-GCG introduces three lightweight modifications that directly address the identified limitations. The overall framework remains close to the original GCG, making the method easy to understand, implement, and integrate with existing optimization-based jailbreak approaches.
  3. In particular, the distance regularization is motivated through a Taylor expansion showing that the error of the first-order gradient approximation is related to the embedding distance between tokens. The empirical improvement in ranking correlation further supports this motivation.
Critical weaknesses
  1. Similar to GCG, Faster-GCG requires access to model gradients. The authors acknowledge that transfer attacks to proprietary models are possible, but also note that finding suitable surrogate models remains challenging, which limits the practical impact on closed-source LLMs.
  2. For example, the visited-suffix marking mechanism removes repeated evaluations, but the paper reports that only about 6% of suffixes are revisited. This suggests that this component alone contributes relatively limited algorithmic novelty and efficiency gain.
  3. Faster-GCG++ changes not only the optimization method but also the attack surface by increasing the suffix length to 40 tokens and changing the suffix placement. Therefore, its stronger ASR cannot be attributed solely to the proposed Faster-GCG techniques, and the comparison with other methods is somewhat less controlled.
  4. The Taylor-expansion analysis motivates why embedding distance may correlate with approximation error, but the specific regularizer is still heuristic. The paper does not show that this form is theoretically optimal, nor does it deeply analyze when embedding distance is a reliable proxy for higher-order approximation error.
Questions for authors

Please see weakness

3 comments

Thank you for recognizing the systematic diagnosis and ease of integration. We number the four weaknesses in their original order.

W1. White-box access and limited transfer. We agree that gradient access and weak transfer limit applicability to proprietary services. The main contribution is more efficient white-box evaluation. GCG-like optimization also serves defense construction: PAT [1] and RPO [2] optimize defensive prompts and evaluate white-box and black-box attacks.

W2. Limited standalone contribution of marking. The 5.9% revisit rate was measured under vanilla GCG. With distance regularization and temperature sampling, but marking disabled, the revisit rate rises to 29.3% (the number will be added to the revised manuscript). This supports marking's greater importance in the combined configuration, consistent with the 42.2% versus 58.2% ablation. Marking itself is standard; the interaction of the three components is the contribution of this paper.

W3. Confounded Faster-GCG++ comparison. We agree that length and placement must be controlled. Under the same 40-token prepended setting, GCG++ achieves an average ASR of 76.3%, compared with 78.1% for Faster-GCG++. Separately, the 20-token appended comparison is 65.8% versus 69.5%.

W4. Heuristic distance regularizer. The distance term expresses a soft locality preference: discrete replacements have finite embedding displacements that cannot generally be scaled down while remaining valid tokens. A local gradient can therefore be extrapolated beyond its reliable range. We use λ∥δ∥\lambda\lVert\delta\rVert to balance predicted improvement against displacement, not to estimate the Taylor remainder, since we cannot determine whether the first-, second-, or third-order term dominates at this scale. Its linear form is empirical; we provide the Taylor expansion as an informal justification. This will be clarified in the revised manuscript.

References:

[1] Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Fight Back Against Jailbreaking via Prompt Adversarial Tuning. NeurIPS, 2024.

[2] Andy Zhou, Bo Li, and Haohan Wang. Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks. NeurIPS, 2024.

Reviewer_hACKOpenReview ↗

response to rebuttal

The rebuttal partially solved my concern. Weakness 1,3,4 are not good enough. I will maintain the score

Thank you for your careful reading of our rebuttal. We provide further evidence and elaboration.

W1 (white-box access). Although the white-box requirement remains a limitation, we showed in our response to Reviewer Hkcw that white-box optimization provides new understanding of each model's token-space vulnerabilities. We repeat that analysis below.

(From our response to Reviewer Hkcw, Q1: What new understanding is provided?) Suffix analysis reveals distinct model-dependent patterns that reflect each model's token-space vulnerabilities, and Faster-GCG's improved optimization provides stronger evidence of these patterns. In matched base-setting comparisons (GCG → Faster-GCG):

  • Llama2 favors punctuation and delimiter structures; punctuation token mass rises from 39.2% to 43.8%.
  • Llama-3.1 exhibits extremely low-norm embeddings; token mass with norm < 0.01 rises from 3.65% to 4.85%.
  • Qwen3 exhibits language-switching/translation patterns; suffix coverage rises from 40% to 48%.
  • Qwen3.5 exhibits vision/image boundary markers; coverage rises from 11% to 28%.
  • Gemma3 exhibits identifiers and unused markers that share an embedding direction; prototype alignment increases slightly (0.1105 → 0.1260).
  • Vicuna favors common language fragments; reference-seen fragment mass rises from 80.73% to 91.50%.

We will add these findings to the revised manuscript.

W3 (Faster-GCG++ attribution). Below, we clarify the threat model, the optimization space and the controlled comparisons. The threat model (Appendix B) allows the attacker to construct any user input, of arbitrary length. This threat model is compatible with both white-box and black-box attacks. For the GCG family, the optimization space is restricted to a 20-token adversarial suffix. This defines a fixed white-box discrete optimization problem in which all methods (except I-GCG) share the same initialization, search budget and search space, so Faster-GCG, TAO, GCG-PS and GCG are directly comparable in this setting. We added the 40-token prepended setting to obtain stronger performance under the stated threat model; within that setting, Faster-GCG++ and GCG++ are directly comparable.

We will revise the abstract and introduction to report the controlled ASR comparison between Faster-GCG and GCG as the main claim. We will also restructure Table 1 so that it contains only fair comparisons:

  • The first block will compare GCG, GCG-PS, TAO and Faster-GCG (with AutoDAN and I-GCG removed).
  • The second block will compare GCG++ and Faster-GCG++.
  • The full comparison under the any user input threat model is already provided in Table A3.

In both controlled comparisons, Faster-GCG (and its ++ variant) achieves the best average loss and ASR.

W4 (heuristic regularizer). Under a locally isotropic curvature approximation H≈cIH \approx cI, the second-order Taylor term becomes c2∥δ∥2\frac{c}{2}\lVert\delta\rVert^2, which motivates d2d^2 as a simple alternative that ignores directional curvature. This approximation replaces directional curvature with a single scalar; it does not compute the Hessian or assume that second-order terms dominate for finite token replacements. We compared dd (the original setting) with d2d^2 at fixed λ=10\lambda=10 and τ=0.1\tau=0.1.

Modelddd2d^2
Gemma3-4B0.45280.4512
Qwen3-8B (reasoning)1.11601.1209

We detect no difference on either model (p=0.95p = 0.95 and 0.860.86). This is consistent with our argument in the first-round rebuttal that the distance term expresses a soft locality preference: using either the norm or the squared norm as the regularizer yields similar performance in the tested settings.

Reviewer_CrfGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Faster-GCG, an improved version of Greedy Coordinate Gradient (GCG) for generating adversarial jailbreak suffixes against aligned LLMs. It identifies three inefficiencies in conventional GCG and introduces corresponding improvements:

  1. Distance-regularized candidate selection: Penalizes token replacements with large embedding distances to improve gradient-based ranking accuracy.
  2. Temperature-controlled sampling: Prioritizes promising replacement candidates rather than sampling uniformly.
  3. Visited-suffix marking: Avoids redundant evaluations of previously explored adversarial suffixes. Experiments across five open-source LLMs demonstrate 4-8x fewer candidate evaluations with broadly comparable attack success rates (ASRs). At equal budgets, Faster-GCG improves average ASR from 65.8% to 69.5% on JailbreakBench. An extended Faster-GCG++ configuration achieves 78.1% average ASR.
Critical strengths

S1. Well-motivated improvements to a widely used optimization method

The paper systematically identifies concrete inefficiencies in GCG and proposes relatively simple, interpretable solutions. The distance-regularization approach is motivated by a Taylor expansion, while empirical diagnostics show that the concordance correlation coefficient (CCC) between predicted and actual candidate rankings improves from 0.044 to 0.285. This provides useful evidence connecting the proposed modifications to observed optimization behavior.

S2. Meaningful efficiency improvements supported by extensive evaluations

The reported 4-8x reduction in candidate evaluations is practically significant, especially given the computational expense of white-box jailbreak optimization. The experiments cover multiple model families, two benchmarks, several competing methods, and component ablations. The inclusion of wall-clock measurements further strengthens the efficiency claim.

Critical weaknesses

W1. Limited methodological novelty and incomplete theoretical justification

The three modifications are relatively straightforward combinations of existing optimization heuristics: distance regularization, biased sampling, and duplicate elimination. More importantly, the Taylor expansion motivates the relevance of embedding distance but does not establish that an additive distance penalty is the appropriate approximation to higher-order error. The regularization coefficient is empirically selected, and the resulting ranking correlation (0.285) remains relatively weak. Stronger theoretical or empirical justification would help distinguish the method from heuristic engineering.

W2. Limited understanding of the interactions between the proposed components

Following W1, although the paper provides ablation experiments, the results reveal surprising interactions that are not adequately explained. In particular, Table 4 shows that combining distance regularization and temperature-controlled sampling without visited-suffix marking reduces ASR to 42.2%, substantially below the GCG baseline of 58.2%. This is unexpected given that redundant evaluations account for only approximately 6% of the original attack budget. The lack of analysis makes it difficult to understand why the components work well together and whether their effectiveness generalizes across optimization settings.

Questions for authors
  1. Distance regularization (W1): Why is embedding distance an appropriate proxy for Taylor approximation error? Have you compared it against second-order approximations, normalized distance metrics, or alternative regularizers?
  2. Ablation interpretation (W2): In Table 4, combining distance regularization and temperature sampling without visited-suffix marking substantially degrades ASR (42.2% versus 58.2% for GCG). Why is the interaction so severe, given that redundant evaluations reportedly account for only approximately 6% of the baseline budget?
  3. Statistical reliability: Can the authors provide confidence intervals, multiple-seed results, and non-inferiority tests to substantiate the claim of negligible performance degradation at 4-8x lower evaluation budgets?
  4. Generalization: How sensitive are the improvements to the fixed hyperparameters λ=10\lambda=10 and τ=0.1\tau=0.1 across architectures, embedding scales, and larger reasoning models?
4 comments

Thank you for the detailed questions and for recognizing the efficiency gains.

W1. Novelty and theoretical justification. Our contribution is the diagnosis and reduction of interacting optimization bottlenecks, rather than individually new primitives. The distance term expresses a soft locality preference: discrete replacements have finite embedding displacements that cannot generally be scaled down while remaining valid tokens. A local gradient can therefore be extrapolated beyond its reliable range. We use λ∥δ∥\lambda\lVert\delta\rVert to balance predicted improvement against displacement, not to estimate the Taylor remainder, since we cannot determine whether the first-, second-, or third-order term dominates at this scale. Its linear form is empirical; CCC improves from 0.044 to 0.285 in the reported single-token diagnostic, but substantial prediction error remains. This is why temperature-controlled random search and a batch size of 64 are still needed. We will clarify this informal justification in the revised manuscript.

W2. Unexpected component interaction. The 5.9% revisit rate was measured under vanilla GCG. With distance regularization and temperature-controlled sampling but visited-suffix marking disabled, the revisit rate rises to 29.3% (the number will be added to the revised manuscript). Thus, vanilla-GCG redundancy substantially understates the combined configuration's redundancy. This supports the interaction in the 42.2% versus 58.2% ASR ablation: distance regularization and temperature-controlled sampling make the candidate distribution more concentrated, and visited-suffix marking is thus an essential component.

Q1. Why distance, and have alternatives been compared? The rationale is given in W1. Please note that this differs from first- and second-order approximations in continuous optimization: discrete replacements have finite embedding displacements that cannot generally be scaled down while remaining valid tokens. Importantly, squared distance is not itself a second-order approximation: 12δ⊤Hδ\frac{1}{2}\delta^\top H \delta also contains directional curvature HH. A local Taylor expansion does not establish which term dominates for a finite replacement; a quadratic error bound requires additional pathwise smoothness assumptions. The choice of a linear form is largely empirical.

Q2. Ablation interpretation. Please see the detailed response in W2.

Q3. Confidence intervals, seeds, and non-inferiority. Table A4 in the appendix reports pooled loss differences of −0.001±0.038−0.001 \pm 0.038 at 4x fewer candidate evaluations and +0.039±0.049+0.039 \pm 0.049 at 8x (95% CIs; Faster-GCG minus GCG). Separately, the equal-budget three-seed results (Table A9) show a loss difference of −0.050±0.040-0.050 \pm 0.040 on Llama2.

Q4. Hyperparameters and generalization. We fix λ=10\lambda=10 and τ=0.1\tau=0.1 across the tested models; the sensitivity sweep is on Llama2. Since λ\lambda operates on a normalized distance, it is applicable to different models. τ\tau is related to top-kk, which is fixed in our experiments.

Clarifications on W1, W2 (Q1, Q2) are sound. I would maintain my score due to incomplete ablation studies (Q4).

Thank you for your careful reading of our rebuttal. To answer Q4 in detail, we provide further evidence from an extended ablation study.

Setup. Using the same setup as the ablation study in the paper, we added two models: Gemma3-4B (a different architecture, with a 262K-token vocabulary) and Qwen3-8B with reasoning enabled. For Qwen, we retain the existing target format, which begins with an empty <think></think> block. The table below lists the mean loss over 100 behaviors for Faster-GCG under different combinations of λ\lambda and τ\tau.

Modeldefault (λ=10,τ=0.1\lambda=10, \tau=0.1)λ=3,τ=0.1\lambda=3, \tau=0.1λ=30,τ=0.1\lambda=30, \tau=0.1λ=10,τ=0.05\lambda=10, \tau=0.05λ=10,τ=0.2\lambda=10, \tau=0.2
Gemma3-4B0.4530.4640.4450.4420.443
Qwen3-8B (reasoning)1.1161.1181.1641.1511.070

The results show that a parameter sweep can find a combination that is better than the default, but overall the results are insensitive to both hyper-parameters.

  • Architecture (Gemma3-4B). We detect no difference between any two settings. The largest loss gap is 0.022, and all ten pairwise comparisons have p>0.28p > 0.28 (paired t-tests over 100 behaviors).
  • Larger reasoning model (Qwen3-8B with reasoning enabled). Relative to the default, neither λ=3\lambda=3 nor λ=30\lambda=30 is statistically different. τ\tau matters more on this model: τ=0.2\tau=0.2 gives lower loss than τ=0.05\tau=0.05 (p=0.013p = 0.013). The default τ=0.1\tau=0.1 lies between them, and its differences from both are not statistically significant.
  • Embedding scale. The raw embedding geometries differ across the three tested models (Llama2-7B, Gemma3-4B and Qwen3-8B). Median token-embedding norms are 1.09 for Llama2, 1.00 for Gemma3 and 1.45 for Qwen3, and median random-pair distances are 1.52, 1.39 and 2.02, respectively. Qwen3-8B's norms are also much more spread out: their 10th–90th percentile range is 0.97–1.68, compared with 0.98–1.02 for Gemma3. These statistics are computed from 8,192 sampled vocabulary rows and 4,096 random token pairs per model, rather than from search-eligible tokens only. The implementation L2-normalizes the gradient and distance vectors at each position, so λ\lambda acts on normalized quantities. This is consistent with λ\lambda transferring across models. However, the models differ in more than embedding scale, so this comparison does not isolate the effect of scale.

In the revision, we will report the extended ablation results and recommend checking alternative hyper-parameter values on new models, rather than presenting the defaults as universally optimal.

Form of the distance term (related to Q1). Under a locally isotropic curvature approximation H≈cIH \approx cI, the second-order Taylor term becomes c2∥δ∥2\frac{c}{2}\lVert\delta\rVert^2, which motivates d2d^2 as a simple alternative that ignores directional curvature. This approximation replaces directional curvature with a single scalar; it does not compute the Hessian or assume that second-order terms dominate for finite token replacements. We compared dd (the original setting) with d2d^2 at fixed λ=10\lambda=10 and τ=0.1\tau=0.1.

Modelddd2d^2
Gemma3-4B0.45280.4512
Qwen3-8B (reasoning)1.11601.1209

We detect no difference on either model (p=0.95p = 0.95 and 0.860.86). This is consistent with our argument in the first-round rebuttal that the distance term expresses a soft locality preference: using either the norm or the squared norm as the regularizer yields similar performance in the tested settings.

Reviewer_HkcwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Faster‑GCG, a variant of the classic GCG white‑box jailbreak attack for aligned large language models. It identifies three in‑source inefficiencies of vanilla GCG: inaccurate gradient‑based candidate estimation, uniform candidate sampling, and redundant re‑evaluation of already‑explored adversarial suffixes. Three tweaks are proposed: distance‑regularized token ranking, temperature‑weighted sampling, and visited‑suffix deduplication. The authors report 4×‑8× sampling efficiency reduction of model evaluations, and Faster‑GCG++ yields 78.1% average ASR across five open‑source LLMs.

Critical strengths
  1. The paper clearly dissects three concrete internal in‑efficiencies within the original GCG pipeline, supported by quantitative statistics including CCC correlation and redundant‑evaluation ratio.

  2. The experimental pipeline is relatively complete, covering multiple open‑source LLMs, two standard jailbreak benchmarks JBB‑Behaviors and AdvBench, plus comparisons against several existing white‑box baselines.

  3. Ablation studies verify each proposed component brings measurable optimization‑loss benefits; hyper‑parameter sensitivity analysis is provided.

Critical weaknesses
  1. Severely outdated research paradigm and limited practical relevance. White‑box gradient‑based suffix‑optimization jailbreak has been extensively studied since 2023. Real‑world adversaries almost never have full white‑box gradient access to production LLMs. Even if attackers obtain gradients, modern aligned LLMs deploy diverse safety defenses against token‑suffix adversarial attacks. Many newer jailbreak paradigms have shown stronger practical impact. This submission only incrementally optimizes the old GCG machinery and fails to situate itself against these shifting community priorities.

  2. Incremental, narrow technical novelty. The three modifications are lightweight engineering tweaks rather than conceptual advances. Distance‑regularized ranking, temperature‑driven sampling and visited‑set deduplication are well‑known general‑optimization heuristics; they are merely ported onto GCG. No new fundamental insight about LLM safety or jailbreak mechanism is delivered.

  3. Poor transfer‑attack practicality. Transfer ASR on closed‑source proprietary models remains extremely low. The method yields almost no usable real‑world attack capability, and the paper provides no actionable path to bridge this gap.

Questions for authors
  1. All three core techniques are known general‑optimization heuristics. What new understanding about LLM adversarial vulnerability, rather than discrete‑optimization engineering tricks, does this paper contribute?

  2. Transfer‑attack performance on proprietary models stays very poor. What concrete modifications could make gradient‑optimized adversarial suffixes transfer effectively across different model families? If no feasible fix exists, what is the practical value of further optimizing white‑box‑only jailbreak pipelines?

3 comments

Thank you for recognizing the quantitative diagnosis and experimental coverage. We address each weakness and question in order.

W1. Practical relevance of white-box optimization. White-box token search also serves defense development. Prompt Adversarial Tuning (PAT) [1] uses GCG-like discrete search to optimize a guard prefix. In its evaluation, ASR decreases from 92% to 1% for GCG, 79% to 1% for PAIR, and 55% to 2% for TAP (Table 1 of the paper). Robust Prompt Optimization (RPO) [2] independently uses AutoPrompt/GCG-style search for defensive suffixes and evaluates white-box and black-box attacks. These demonstrate uses beyond proprietary-model attacks. The techniques introduced by Faster-GCG improve discrete search and are relevant to these valuable applications.

W2. Incremental novelty. Our contribution is a systematic diagnosis of discrete-search bottlenecks in GCG and their interactions, together with measured efficiency gains. For example, the revisit rate rises from 5.9% under vanilla GCG to 29.3% with distance regularization and temperature sampling, explaining why marking matters more in the combined configuration. New insights into LLM adversarial vulnerability are detailed under Q1.

W3. Poor proprietary-model transfer. We agree that low absolute transfer ASR limits this application. Our contribution concerns efficient white-box evaluation; it does not establish practical attacks on proprietary services or resolve surrogate selection. We discuss these limitations in the Limitations section.

Q1. What new understanding is provided? Suffix analysis reveals distinct model-dependent patterns that demonstrate each model's token-space vulnerabilities. Faster-GCG's improved optimization provides stronger evidence of these patterns. In matched base-setting comparisons (GCG → Faster-GCG):

  • Llama2 favors punctuation and delimiter structures; punctuation token mass rises from 39.2% to 43.8%.
  • Llama-3.1 exhibits extremely low-norm embeddings; token mass with norm < 0.01 rises from 3.65% to 4.85%.
  • Qwen3 exhibits language-switching/translation patterns; suffix coverage rises from 40% to 48%.
  • Qwen3.5 exhibits vision/image boundary markers; coverage rises from 11% to 28%.
  • Gemma3 exhibits identifiers and unused markers sharing an embedding direction. Prototype alignment increases slightly (0.1105 → 0.1260).
  • Vicuna favors common language fragments; reference-seen fragment mass rises from 80.73% to 91.50%.

These findings will be added to the revised manuscript.

Q2. How can transfer be improved, and what is the value without it? When transferred to proprietary models, the performance of GCG and Faster-GCG remains inferior to black-box attack methods that perform query-based optimization on these models. As explained in W1 and W3, defense construction and evaluation of a developer's own model remain useful applications of gradient access.

References:

[1] Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Fight Back Against Jailbreaking via Prompt Adversarial Tuning. NeurIPS, 2024.

[2] Andy Zhou, Bo Li, and Haohan Wang. Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks. NeurIPS, 2024.

I appreciate the authors' detailed rebuttal, which partially addresses my original concerns. Nevertheless, my core reservations regarding novelty (W2) and real‑world practical value (W3) remain insufficiently resolved. I will maintain the score.

Thank you for your follow-up. We would like to offer a new perspective on your remaining concerns.

Novelty (W2). Our contribution is a measured account of how the three primitives interact in discrete search, together with a lightweight way to reduce the search cost in the GCG setting. To derive this solution, we provide a non-trivial analysis of the inefficiencies of GCG. Regarding new fundamental insight into LLM safety and jailbreak mechanisms, we added an analysis of the optimized adversarial suffixes. It shows that different models expose different vulnerable tokens, and that Faster-GCG's improved optimization provides stronger evidence of these patterns. These vulnerabilities differ from those revealed by black-box attacks (see our first-round response to Q1).

Practical value (W3). We understand your concern to be that a jailbreak method is practical only if it is strong against proprietary models. We would like to offer a different view of where the practical value lies. In safety research, attacks are developed mainly to find and fix safety-alignment failures, and the party that fixes them is the model developer, who has full white-box access to its own model. This also holds for proprietary models: their developers can run gradient-based red-teaming and apply fixes internally. As detailed in our response to Q1, white-box attacks discover vulnerabilities that differ from those found by black-box attacks. From this perspective, the practical bottleneck is the cost of the attack rather than its transferability, and this cost is exactly what Faster-GCG reduces relative to GCG. For example, HarmBench's R2D2 adversarial training [1] could afford only 5 GCG steps per iteration because GCG takes about 20 minutes per test case on a 7B model, where we could reduce the cost by about 4x.

We would be grateful if you could let us know whether this perspective addresses your concerns.

References:

[1] Mazeika, Mantas, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. ICML, 2024.

Comments on the paper

1 comment

We thank all reviewers for their time and constructive feedback. The reviewers agreed that the paper gives a clear, quantitative diagnosis of three inefficiencies in GCG (all three reviewers), that the 4–8x reduction in candidate evaluations is practically significant (CrfG), that the modifications are lightweight and easy to integrate (hACK), and that the evaluation covers several models, two benchmarks and multiple baselines (Hkcw, CrfG). Below, we summarize how the rebuttal and discussion address the shared and key concerns.

1. Practical value of white-box optimization (Hkcw W1/W3/Q2, hACK W1).

  • GCG-like optimization also serves defense construction: PAT and RPO optimize defensive prompts and evaluate white-box and black-box attacks.
  • Attacks in safety research are developed mainly to find and fix alignment failures, and the party that fixes them, the model developer, has full white-box access to its own model, including proprietary ones. The practical bottleneck is therefore the cost of the attack rather than its transfer to other parties' closed models. For example, HarmBench's R2D2 adversarial training could afford only 5 GCG steps per iteration because of GCG's cost, which Faster-GCG reduces by about 4x.

2. Novelty and new understanding (Hkcw W2/Q1, CrfG W1, hACK W1).

  • Our contribution is a measured account of how the three primitives interact in discrete search, together with a lightweight way to reduce the search cost in the GCG setting. To derive this solution, we provide a non-trivial analysis of the inefficiencies of GCG.
  • With suffix analysis, we show that different models expose different vulnerable token patterns, and that Faster-GCG's stronger optimization surfaces them more clearly. These vulnerabilities differ from those revealed by black-box attacks.

3. Justification of the distance regularizer (CrfG W1/Q1, hACK W4). The penalty expresses a soft locality preference: a discrete token replacement has a finite embedding displacement that cannot be scaled down while remaining a valid token. It is not an estimate of the Taylor remainder, and we do not claim optimality. A squared-distance penalty, motivated by an isotropic second-order approximation, gives no detectable difference from the linear form on Gemma3-4B or Qwen3-8B, which is consistent with the soft-locality interpretation.

4. Component interaction and the role of marking (CrfG W2/Q2, hACK W2). The revisit rate rises from 5.9% under vanilla GCG to 29.3% once distance regularization and temperature sampling are combined.

5. Hyperparameter generalization (CrfG Q4). We extended the λ\lambda and τ\tau sweep to Gemma3-4B and Qwen3-8B (reasoning). Across Llama-2, Gemma3-4B and Qwen3-8B, the default values were not detectably worse than any other tested value; the hyper-parameters generalize. We also report the embedding-scale statistics of the three models.

6. Fair comparison and Faster-GCG++ (hACK W3). Faster-GCG, TAO, GCG-PS and GCG are directly comparable. We added the 40-token prepended setting to obtain stronger performance under the stated threat model; within that setting, Faster-GCG++ and GCG++ are directly comparable. In both controlled comparisons, Faster-GCG (and its ++ variant) achieves the best average loss and ASR. We will restructure the table to clearly reflect this comparison.

7. Statistical reliability (CrfG Q3). Provided in Table A4 and A9.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.