acceptodds
Withdrawn from ICLR 2027

Accelerating Suffix-Based Jailbreak Evaluation with Prefix-Shared KV Cache

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Suffix jailbreak attacks are a systematic tool for red-teaming large language models (LLMs), but they require evaluating many candidate suffixes before finding an effective jailbreak. This paper presents Prefix-Shared KV Cache (PSKV), a plug-and-play inference optimization tailored to suffix jailbreak evaluation. PSKV exploits the structure that many candidate prompts share the same harmful instruction prefix while differing only in the candidate suffix. Instead of redundantly recomputing or physically duplicating the prefix cache, PSKV stores one compact prefix KV cache and expands it lazily at each layer during candidate evaluation. This design preserves PyTorch autograd compatibility and enables larger batched searches with lower memory overhead. Across six suffix attacks and five open-source LLMs, PSKV reduces inference time by about 40% and peak memory by about 50% while preserving attack effectiveness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
KV cache, Adversarial Suffix Attacks, Safety, Efficiency ML

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yEpoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Prefix-Shared KV Cache (PSKV), a lightweight optimization for suffix-based jailbreak evaluation. Its central observation is that candidate suffixes share an unchanged instruction prefix, making repeated prefix computation and persistent cache replication unnecessary. PSKV stores one prefix cache per instruction and expands it lazily at each attention layer while retaining compatibility with PyTorch autograd. A suffix-centric alignment strategy additionally enables batched evaluation of variable-length instructions and targets. The main evaluation covers six attacks and five open-source target models, comparing PSKV with uncached execution and explicitly replicated Standard KV caching. Supplementary experiments examine larger models, larger evaluation sets, longer sequences, universal suffixes, and ASR stability. The results suggest practical runtime savings over uncached execution and memory savings over explicit cache replication. However, stronger matched baselines and more consistent interpretation of the measurements are needed to establish the distinct contribution.

Critical strengths
  1. Practical, differentiable optimization. PSKV targets genuine computational redundancy while preserving gradient access, making it useful for attack pipelines that cannot directly adopt forward-only serving engines.
  2. Broad experimental coverage. The evaluation spans multiple attacks and architectures, with supplementary checks on larger workloads, longer sequences, and multi-seed ASR stability.
Critical weaknesses
  1. Insufficiently controlled baselines. Standard KV replicates prefixes across all candidates. A matched baseline that replicates only the current scoring microbatch is needed to separate PSKV’s contribution from microbatching and implementation choices.
  2. Some efficiency conclusions need further qualification. The reported results are encouraging, but the interpretation of runtime and memory improvements could be more consistent across the text, tables, and supplementary analysis. Clearer measurement definitions and more carefully scoped claims would strengthen the evidence.
Questions for authors
  1. Can you compare PSKV against a Standard KV implementation that replicates prefixes only for the current scoring microbatch, holding the attack code, precision, attention backend, and microbatch sizes fixed? Please include phase-wise memory measurements and suffix-gradient consistency tests to clarify both the distinct benefit and the peak-memory claim.
  2. Could you clarify the measurement scope and aggregation used for the reported runtime and memory improvements, and explain how these results support the overall efficiency claims?
Reviewer_YSw5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prefix-Shared KV Cache (PSKV), a method for accelerating suffix-based jailbreak attacks. It reduces redundant computation and memory usage by sharing the prefix KV cache across candidate suffixes and expanding it at each layer when needed. The authors evaluate PSKV across multiple jailbreak methods and LLMs, comparing its efficiency and attack effectiveness with existing caching approaches.

Critical strengths
  1. The proposed method reduces redundant prefix KV cache storage while maintaining compatibility with gradient-based jailbreak attacks.

  2. The paper evaluates PSKV across multiple jailbreak methods and LLMs. The results show improvements in memory efficiency and execution time while largely preserving attack effectiveness.

Critical weaknesses
  1. The novelty of PSKV over existing prefix caching implementations is unclear. Methods such as nanoGCG already support prefix KV cache reuse. The paper does not sufficiently compare PSKV with these optimized implementations, making it difficult to assess the contribution of layer-wise lazy expansion.

  2. Although preserving gradient computation is a key motivation of PSKV, the paper mainly validates numerical equivalence through output logits and attack success rates. It does not directly compare the gradients produced by PSKV and standard implementations. Such comparisons are important for establishing the correctness of gradient-based jailbreak optimization.

Questions for authors
  1. Could the authors compare PSKV with existing optimized prefix caching implementations, such as nanoGCG? How much of the efficiency improvement comes specifically from layer-wise lazy expansion?

  2. Could the authors directly compare the gradients computed by PSKV and standard implementations, for example using gradient cosine similarity or top-k token candidate overlap?

Reviewer_UXfkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Prefix-Shared KV Cache (PSKV) for repeated suffix-candidate evaluation. It stores one KV cache per shared input prefix, expands it layer by layer during scoring, and aligns variable-length inputs for batching. The implementation supports PyTorch autograd. Experiments across six reimplemented attacks and five target models report runtime savings over uncached execution and memory savings over eagerly replicated caching, with broadly comparable attack success rates.

Critical strengths
  • Memory savings against the replicated-cache baseline. For example, Vicuna-7B GCG uses 27 versus 100 GB, with similar runtime, 146 versus 147 minutes (Tables 4-5). This supports usefulness relative to that implementation.
  • Repeated-run effectiveness checks. Within the six-attack, five-model benchmark, the repeated-run ASR checks support broadly preserved attack effectiveness.
Critical weaknesses
  • Limited demonstrated research novelty. Prefix sharing, on-demand layer-wise expansion, and padded-batch alignment appear to be straightforward engineering choices (Sections 4.2-4.3). Prefix sharing is established in the systems discussed in Section 2, and autograd support appears to follow from standard PyTorch operations. The paper does not establish a sufficiently nontrivial advance beyond integrating these ingredients for suffix search.
  • The principal memory advantage is measured against avoidable duplication. Standard KV persistently replicates identical prefix KVs across candidates (Section 5.1). Storing each distinct prefix once and mapping samples to that cache already removes this resident-cache redundancy. Table 7 compares serving engines for BEAST, but does not isolate PSKV's incremental benefit over a simple autograd-compatible PyTorch shared cache. Remaining differences in temporary workspace and execution cost need matched measurement.
Questions for authors
  • Novelty: What nontrivial technical obstacle does PSKV overcome beyond standard prefix sharing, layer-wise tensor assembly, and batch alignment? Does autograd support require a new mechanism?
  • Deduplicated baseline: Under matched attack settings, what runtime and peak-memory advantage remains over an autograd-compatible PyTorch cache that stores each distinct prefix once and assembles the current layer's batch on demand? Please distinguish resident storage from temporary and backward-retained tensors.
  • GCQ's editable input: Does your GCQ implementation restrict optimization to a designated suffix of the user-input jailbreak prompt, as in GCG, rather than allowing changes throughout that prompt? If so, please clarify this adaptation and whether the reported cache savings extend to the original unrestricted setting, in which the entire user-input jailbreak prompt is editable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.