- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes Prefix-Shared KV Cache (PSKV), a lightweight optimization for suffix-based jailbreak evaluation. Its central observation is that candidate suffixes share an unchanged instruction prefix, making repeated prefix computation and persistent cache replication unnecessary. PSKV stores one prefix cache per instruction and expands it lazily at each attention layer while retaining compatibility with PyTorch autograd. A suffix-centric alignment strategy additionally enables batched evaluation of variable-length instructions and targets. The main evaluation covers six attacks and five open-source target models, comparing PSKV with uncached execution and explicitly replicated Standard KV caching. Supplementary experiments examine larger models, larger evaluation sets, longer sequences, universal suffixes, and ASR stability. The results suggest practical runtime savings over uncached execution and memory savings over explicit cache replication. However, stronger matched baselines and more consistent interpretation of the measurements are needed to establish the distinct contribution.
Critical strengths
- Practical, differentiable optimization. PSKV targets genuine computational redundancy while preserving gradient access, making it useful for attack pipelines that cannot directly adopt forward-only serving engines.
- Broad experimental coverage. The evaluation spans multiple attacks and architectures, with supplementary checks on larger workloads, longer sequences, and multi-seed ASR stability.
Critical weaknesses
- Insufficiently controlled baselines. Standard KV replicates prefixes across all candidates. A matched baseline that replicates only the current scoring microbatch is needed to separate PSKV’s contribution from microbatching and implementation choices.
- Some efficiency conclusions need further qualification. The reported results are encouraging, but the interpretation of runtime and memory improvements could be more consistent across the text, tables, and supplementary analysis. Clearer measurement definitions and more carefully scoped claims would strengthen the evidence.
Questions for authors
- Can you compare PSKV against a Standard KV implementation that replicates prefixes only for the current scoring microbatch, holding the attack code, precision, attention backend, and microbatch sizes fixed? Please include phase-wise memory measurements and suffix-gradient consistency tests to clarify both the distinct benefit and the peak-memory claim.
- Could you clarify the measurement scope and aggregation used for the reported runtime and memory improvements, and explain how these results support the overall efficiency claims?