- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SPACE, a jailbreak attack method that adaptively selects where to insert adversarial tokens into LLM prompts. The method uses optimization feedback and Bayesian inference to estimate effective token positions and gradually allocates attack tokens. The authors evaluate SPACE across multiple LLM families, compare it with GCG and SlotGCG, and investigate whether learned positional information can transfer across models.
Critical strengths
-
The paper proposes an adaptive approach to selecting attack token positions based on optimization feedback, rather than relying on fixed positional heuristics.
-
The paper evaluates the method across multiple LLM families and shows that learned positional information can be transferred to improve attacks on unseen models, including gradient-free attacks.
Critical weaknesses
-
SPACE incrementally inserts attack tokens while preserving previously selected positions. However, newly inserted tokens may interfere with previously optimized attack tokens by changing the input context. It is unclear how much this affects the optimization loss and whether subsequent optimization can recover the previous performance. The paper lacks an analysis of this effect across insertion rounds and different insertion schedules.
-
It is unclear how much Bayesian inference contributes to the performance improvement. Since SPACE explores multiple token positions during optimization, the improvement may come from positional exploration rather than Bayesian inference itself. The paper does not provide sufficient comparisons with simpler strategies, such as random position exploration or loss-based position selection without Bayesian updates. Furthermore, SPACE requires approximately 33% more wall-clock time than SlotGCG, making comparisons under equal computational budgets important.
Questions for authors
-
When new attack tokens are inserted in later rounds, they may interfere with previously optimized tokens and increase the loss. Could the authors analyze the loss before and after each insertion round and examine whether subsequent optimization recovers the previous performance?
-
Could the authors compare SPACE with simpler positional exploration strategies without Bayesian updates? This would help clarify whether the improvement comes from Bayesian inference or simply exploring different token positions.
-
How does SPACE perform under the same wall-clock time budget as GCG and SlotGCG? Also, could the authors clarify how newly inserted attack tokens are initialized and how optimization steps are distributed across insertion rounds?
3 comments
Follow-up from the authors
Dear Reviewer dFjY,
We thank you again for your time and for your positive assessment of our work. As the end of the discussion period is fast approaching, we were wondering whether our responses were sufficient to address the points you raised: 1.) interference from newly inserted tokens, 2.) Bayesian inference versus positional exploration and 3.) the equal wall-clock comparison. We highlight the following facts that may aid clarity:
1.) Insertion does raise the loss, as you anticipated, but it returns to or below its pre-insertion value within the same round in 95.1% of cases.
2.) At an identical query budget, SPACE-Uniform improves on random positional exploration without any Bayesian update by 3.5 points (95% CI [+0.8, +6.2]; ahead on 10 of 11 models).
3.) Running GCG and SlotGCG to SPACE's wall-clock budget does not change the conclusion.
We would be happy to engage in any further discussion before the end of the period, and are more than willing to provide any further evidence or amendments that may enhance your confidence in championing our paper.
I am satisfied with the authors’ response and will therefore maintain my original score.
Author Rebuttal
We would like to thank Reviewer dFjY for their time and for their positive assessment of our work. The questions raised are clear and actionable, and we have added new analysis and new experiments to address each of them directly.
W1 / Q1: Interference from newly inserted tokens
We thank the reviewer for their suggestion as it does clarify an important aspect of our inference framework. To answer this question, we measured the attack loss immediately before and after every insertion round over all 1,200 main-configuration runs (12 models, two seeds):
The reviewer's intuition is correct: inserting new (unoptimized) tokens does increase the loss over the previously optimized ones. However, subsequent optimization recovers quickly. Across all 4,800 later-round insertions, the loss returns to or below its pre-insertion value within the same round in 95.1% of cases, typically within a small fraction of the round's steps, and every round ends at a lower loss than the previous one. Recovery also matters for the outcome: runs in which every round recovered reach 56.4% ASR, against 45.3% for runs with an unrecovered round. In future versions we will experiment with various insertion schedules and different inference approaches.
W2 / Q2: Bayesian inference versus positional exploration
We agree that this is an interesting positional baseline and have added a random-position control to the black-box experiments. It runs the exact same algorithm, schedule and query budget, but explores randomly drawn position allocations without any Bayesian update (two independent draws, 11 models, two seeds):
Random positional exploration is itself a strong baseline: it bests both suffix (by 7.1 points, p < 0.001) and VSS-Pooled (by 3.7 points, p = 0.02). Part of the improvement therefore does come from exploring positions, as the reviewer suggests. On top of this, however, Bayesian inference adds a further improvement at identical compute: SPACE-Uniform improves on random allocation by +3.5 points (95% CI [+0.8, +6.2]; p = 0.023 and 0.037 against the two draws) and is ahead on 10 of 11 models. Moreover, inference provides re-usable positional information that the random baseline cannot.
Q3: Equal wall-clock budget, initialisation and step distribution
Equal wall-clock: Measuring attack time only (our submitted figure included the time spent writing diagnostic logs), SPACE takes 1.4x the wall-clock time of the baselines on average over the twelve models (median 1.3x). For three small models we allowed GCG and SlotGCG to continue for additional steps until they had used the same wall-clock time as SPACE. This does not substantially change the conclusion (ASR out of 50 behaviours, 500 steps → matched budget; bold indicates best at the matched budget):
Pooled over the three models, SPACE succeeds on 99 of 150 behaviours against 94 for GCG and 97 for SlotGCG at their matched budgets. Though the small models yield a more modest comparison, we highlight that the extra compute led to a net change of -1 for GCG and +8 for SlotGCG out of 150. This is now added to the appendix and referenced in the main text. The black-box results are already compute matched, since every method uses the same number of target queries.
Initialisation and schedule: SPACE inserts 4 tokens per round over 5 rounds (20 tokens in total). Newly inserted tokens are initialised to "!", as in GCG. The 500 optimization steps are distributed over the rounds as 33 / 67 / 100 / 133 / 167 (17 / 33 / 50 / 67 / 83 for the three models run at 250 steps), so that later rounds, which optimize more tokens, receive proportionally more steps. We have added this description to Appendix A.2 of the revised manuscript, referenced from Section 5.