acceptodds
Under review as a conference paper at ICLR 2027

Positional Inference for Stronger Attacks on Language Models

Abstract

Optimization-based adversarial attacks on large language models (LLMs) perform coordinate descent over a fixed set of token positions. While seminal works focus exclusively on suffix-based attacks, recent studies suggest that inserting tokens into alternate positions (prefix and infix) can achieve higher attack success rates (ASR). Although methods that move beyond suffixes often improve ASR, their position selection usually relies on architecture-dependent heuristics. We cast position selection as Bayesian inference: intermediate optimizer evaluations that existing attacks discard serve as observations about which positions are worth attacking, and existing heuristics can be formulated as prior beliefs about impactful positions. Because the resulting posteriors are Gaussian, beliefs inferred from individual attacks can be merged in closed form, so computation spent attacking one model becomes a positional prior for attacking another. Across 12 LLMs ranging from 0.6B to 32B parameters, and despite the exploration cost of jointly inferring positions and optimizing tokens, our attack improves mean ASR by 11.1 percentage points over suffix-based GCG and 3.6 points over attention-guided SlotGCG. The inferred posteriors reveal that vulnerability concentrates at both ends of the prompt, reconciling conflicting reports that favor suffixes or prefixes, yet correlates only weakly with the attention signal current heuristics rely on. Most interestingly, we find that positional knowledge transfers: in leave-one-model-out experiments, using posteriors merged from the other 11 models as a prior improves a gradient-free attack on the held-out model by 9–11 points over suffix placement and 6–7 points over transferred attention heuristics, matching suffix-only ASR with roughly a quarter of the query budget. Our results suggest that where to attack is learnable, transferable, and enables an important form of adaptive attacks for language models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Attacks, Bayesian Inference, Natural Language Processing

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_dFjYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPACE, a jailbreak attack method that adaptively selects where to insert adversarial tokens into LLM prompts. The method uses optimization feedback and Bayesian inference to estimate effective token positions and gradually allocates attack tokens. The authors evaluate SPACE across multiple LLM families, compare it with GCG and SlotGCG, and investigate whether learned positional information can transfer across models.

Critical strengths
  1. The paper proposes an adaptive approach to selecting attack token positions based on optimization feedback, rather than relying on fixed positional heuristics.

  2. The paper evaluates the method across multiple LLM families and shows that learned positional information can be transferred to improve attacks on unseen models, including gradient-free attacks.

Critical weaknesses
  1. SPACE incrementally inserts attack tokens while preserving previously selected positions. However, newly inserted tokens may interfere with previously optimized attack tokens by changing the input context. It is unclear how much this affects the optimization loss and whether subsequent optimization can recover the previous performance. The paper lacks an analysis of this effect across insertion rounds and different insertion schedules.

  2. It is unclear how much Bayesian inference contributes to the performance improvement. Since SPACE explores multiple token positions during optimization, the improvement may come from positional exploration rather than Bayesian inference itself. The paper does not provide sufficient comparisons with simpler strategies, such as random position exploration or loss-based position selection without Bayesian updates. Furthermore, SPACE requires approximately 33% more wall-clock time than SlotGCG, making comparisons under equal computational budgets important.

Questions for authors
  1. When new attack tokens are inserted in later rounds, they may interfere with previously optimized tokens and increase the loss. Could the authors analyze the loss before and after each insertion round and examine whether subsequent optimization recovers the previous performance?

  2. Could the authors compare SPACE with simpler positional exploration strategies without Bayesian updates? This would help clarify whether the improvement comes from Bayesian inference or simply exploring different token positions.

  3. How does SPACE perform under the same wall-clock time budget as GCG and SlotGCG? Also, could the authors clarify how newly inserted attack tokens are initialized and how optimization steps are distributed across insertion rounds?

3 comments
AuthorsOpenReview ↗

Author Rebuttal

We would like to thank Reviewer dFjY for their time and for their positive assessment of our work. The questions raised are clear and actionable, and we have added new analysis and new experiments to address each of them directly.

W1 / Q1: Interference from newly inserted tokens

We thank the reviewer for their suggestion as it does clarify an important aspect of our inference framework. To answer this question, we measured the attack loss immediately before and after every insertion round over all 1,200 main-configuration runs (12 models, two seeds):

RoundLoss before insertionLoss after insertionRecovered within the roundMedian steps to recoverLoss at end of round
1 (clean prompt)1.961.93--1.06
21.061.34 (x1.20)99.8%20.60
30.601.02 (x1.63)94.5%70.44
40.440.85 (x1.84)93.5%120.33
50.330.71 (x2.09)92.7%180.25

The reviewer's intuition is correct: inserting new (unoptimized) tokens does increase the loss over the previously optimized ones. However, subsequent optimization recovers quickly. Across all 4,800 later-round insertions, the loss returns to or below its pre-insertion value within the same round in 95.1% of cases, typically within a small fraction of the round's steps, and every round ends at a lower loss than the previous one. Recovery also matters for the outcome: runs in which every round recovered reach 56.4% ASR, against 45.3% for runs with an unrecovered round. In future versions we will experiment with various insertion schedules and different inference approaches.

W2 / Q2: Bayesian inference versus positional exploration

We agree that this is an interesting positional baseline and have added a random-position control to the black-box experiments. It runs the exact same algorithm, schedule and query budget, but explores randomly drawn position allocations without any Bayesian update (two independent draws, 11 models, two seeds):

SuffixVSS-PooledSPACE-PooledSPACE-UniformSPACE + PriorRandom: mean (draw 1, draw 2)
HarmBench / GPT-4.122.926.330.233.532.530.0 (29.9, 30.1)
StrongREJECT / GPT-4.127.932.138.041.840.839.1 (39.1, 39.1)

Random positional exploration is itself a strong baseline: it bests both suffix (by 7.1 points, p < 0.001) and VSS-Pooled (by 3.7 points, p = 0.02). Part of the improvement therefore does come from exploring positions, as the reviewer suggests. On top of this, however, Bayesian inference adds a further improvement at identical compute: SPACE-Uniform improves on random allocation by +3.5 points (95% CI [+0.8, +6.2]; p = 0.023 and 0.037 against the two draws) and is ahead on 10 of 11 models. Moreover, inference provides re-usable positional information that the random baseline cannot.

Q3: Equal wall-clock budget, initialisation and step distribution

Equal wall-clock: Measuring attack time only (our submitted figure included the time spent writing diagnostic logs), SPACE takes 1.4x the wall-clock time of the baselines on average over the twelve models (median 1.3x). For three small models we allowed GCG and SlotGCG to continue for additional steps until they had used the same wall-clock time as SPACE. This does not substantially change the conclusion (ASR out of 50 behaviours, 500 steps → matched budget; bold indicates best at the matched budget):

ModelMatched budgetSPACE (500 steps)GCGSlotGCG
Phi-4-Mini1.5x4040 → 3735 → 39
Qwen3-0.6B2.0x2825 → 2522 → 28
Llama-3.2-1B1.5x3130 → 3232 → 30

Pooled over the three models, SPACE succeeds on 99 of 150 behaviours against 94 for GCG and 97 for SlotGCG at their matched budgets. Though the small models yield a more modest comparison, we highlight that the extra compute led to a net change of -1 for GCG and +8 for SlotGCG out of 150. This is now added to the appendix and referenced in the main text. The black-box results are already compute matched, since every method uses the same number of target queries.

Initialisation and schedule: SPACE inserts 4 tokens per round over 5 rounds (20 tokens in total). Newly inserted tokens are initialised to "!", as in GCG. The 500 optimization steps are distributed over the rounds as 33 / 67 / 100 / 133 / 167 (17 / 33 / 50 / 67 / 83 for the three models run at 250 steps), so that later rounds, which optimize more tokens, receive proportionally more steps. We have added this description to Appendix A.2 of the revised manuscript, referenced from Section 5.

AuthorsOpenReview ↗

Follow-up from the authors

Dear Reviewer dFjY,

We thank you again for your time and for your positive assessment of our work. As the end of the discussion period is fast approaching, we were wondering whether our responses were sufficient to address the points you raised: 1.) interference from newly inserted tokens, 2.) Bayesian inference versus positional exploration and 3.) the equal wall-clock comparison. We highlight the following facts that may aid clarity:

1.) Insertion does raise the loss, as you anticipated, but it returns to or below its pre-insertion value within the same round in 95.1% of cases.

2.) At an identical query budget, SPACE-Uniform improves on random positional exploration without any Bayesian update by 3.5 points (95% CI [+0.8, +6.2]; ahead on 10 of 11 models).

3.) Running GCG and SlotGCG to SPACE's wall-clock budget does not change the conclusion.

We would be happy to engage in any further discussion before the end of the period, and are more than willing to provide any further evidence or amendments that may enhance your confidence in championing our paper.

I am satisfied with the authors’ response and will therefore maintain my original score.

Reviewer_3MqwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper’s central idea is that where adversarial tokens are inserted into a prompt should be learned during an attack. Existing methods usually append tokens at the end or choose positions using a fixed attention-based heuristic. The authors propose SPACE, which adapts token placement using evidence gathered during optimization.

SPACE maintains a probability distribution over how useful each insertion position is. It samples candidate token allocations, optimizes the inserted tokens, and uses the resulting loss reductions to update its beliefs. Evaluations that ordinary attacks would discard therefore become evidence about promising positions.

The authors evaluate SPACE against vanilla GCG and SlotGCG baselines, on 12 instruction-tuned models using 50 harmful behaviors from AdvBench and a budget of 20 inserted tokens. GPT-4.1 judges whether the generated responses constitute successful jailbreaks. The attack success rate of SPACE outperforms baselines on 7/12 models. On average SPACE yields 11 and 4 points higher attack success rate than GCG and slotGCG respectively.

Critical strengths
  1. SPACE turns positional selection into a learnable component of token-based attacks, using intermediate optimizer evaluations that would otherwise be discarded. Closed-form posterior updates and an exact allocation rule for the surrogate objective make the framework tractable and compatible with both gradient-based and gradient-free optimizers. This is consequential because it provides a general alternative to fixed placement heuristics, with reusable positional knowledge as an additional methodological benefit.

  2. Evidence of practical gains and useful transfer across models. Across twelve models, SPACE improves reported mean attack success by 11.1 percentage points over GCG and 3.6 points over SlotGCG, while solving 62 model–behavior pairs neither baseline solves. More compellingly, transferred positional beliefs improve gradient-free attacks on held-out models by approximately 9–11 points over suffix placement at matched target-query budgets. These results support acceptance by showing that learned placement can both expand vulnerability coverage and reduce the target-model queries needed for successful attacks.

Critical weaknesses
  1. The claims in ln370, "allow us to mimic – and potentially improve – heuristics" and in ln 377, "SPACE performs no worse than their reference heuristic attack." are unsupported by the results in which SPACE underperforms the baselines on 4/12 of the evaluated models in Table 1. Although, in most cases, the difference is small but for Qwen3-32B it is 6pt, which needs to be explained. Even where the difference is small I would like to see some statistical analysis that show that SPACE's ASR is within the margin of random error of the baseline approaches. Meanwhile, Table 2 reports mean improvements of only 0.3 and 0.5 percentage points, alongside substantial individual regressions—for example, VSS-initialized SPACE decreases ASR from 68% to 58% on Llama-3.1-8B. Perhaps, paired confidence intervals and a non-inferiority analysis with a justified margin would better support the “performs no worse” claim; a non-significant difference alone would not establish equivalence. This matters because the paper does not adequately characterize when adaptive placement helps or harms attack performance.

  2. ASR evaluation via GPT4.1 judge is not well motivated and justified. Firstly, there are no metrics presented for the accuracy of judge and how well do its judgements align with human judgements. In the absence of these we can not be sure how reliable is the judge. While Appendix A.2 describes manual review of ambiguous judge responses, but this does not establish accuracy on cases the judge classifies confidently. I would like to see validation against an adequately sized human-labeled sample covering different attacks and target models, with multiple independent annotators, inter-annotator agreement, and judge precision and recall. The analysis should assess whether judging errors differ across attack methods, since such differences could affect the reported 3.6-point mean advantage over SlotGCG. A sensitivity analysis using another independently validated judge would further strengthen confidence in the comparisons. Because ASR is the primary outcome, uncertainty in its measurement directly affects the paper’s central empirical conclusions. Secondly, GPT-4.1 is an old model and AI model capabilities have since improved significantly. It would be interesting to see how the results hold up when judged by the latest models.

Questions for authors

The following could materially change my assessment:

  1. a statistical justification that SPACE “performs no worse” than the baselines.
  2. Data and statistics indicating that the GPT judge is reliable.
3 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1)

We would like to thank Reviewer 3Mqw for their time and for a clear statement of what would change their assessment. We have worked to address both items directly: a statistical justification for our claims, and data on the reliability of the judge.

W1 / Q1: Statistical justification that SPACE "performs no worse"

We thank the reviewer for raising this concern as we should have been more precise in our statement here. Most importantly, the statement intends to reference the on-average performance across models and we have now amended it to make this clear. Secondarily, we want to point out that the statements referred to in the weakness refer to Table 2 and not Table 1. Regardless, we do perform a paired statistical analysis for both tables:

Table 1, paired analysis of SPACE versus the baselines (12 models, two seeds, average over the six judges):

ContrastMean difference95% CIp (exact permutation)Models ahead / behind
SPACE − SlotGCG+3.8[+0.9, +7.0]0.0379 / 3
SPACE − GCG+14.0[+8.8, +19.3]< 0.00112 / 0

SPACE is superior to both baselines on the judge average. Per judge, SPACE is ahead of SlotGCG under all six, and is non-inferior at a 2-point margin under every judge individually. Below, p is the one-sided p-value of the null hypothesis that SPACE is worse than SlotGCG by at least the margin (a small p establishes non-inferiority):

Judge (prompt / model)SPACE − SlotGCG95% CIp (2-point margin)p (4-point margin)
HarmBench / GPT-4.1 (submission)+2.3[-0.7, +5.7]0.002< 0.001
HarmBench / GPT-6.1 Sol+1.8[-0.7, +4.3]0.001< 0.001
HarmBench / HarmBench classifier+2.6[-1.9, +7.1]0.0210.002
StrongREJECT / GPT-4.1+5.9[+1.8, +10.3]< 0.001< 0.001
StrongREJECT / GPT-4o-mini+5.5[+1.0, +10.2]< 0.001< 0.001
StrongREJECT / GPT-6.1 Sol+4.7[+2.2, +7.2]< 0.001< 0.001
Average+3.8[+0.9, +7.0]< 0.001< 0.001

Qwen3-32B: the 6-point deficit in the submitted table came from a single seed. With two seeds, SPACE is at 38% on Qwen3-32B, against 34% for GCG and 36% for SlotGCG (seed 0: 36 vs 42 and 40; seed 1: 40 vs 26 and 32) under the judge used in our original submission numbers in Appendix C.3. The deficit was seed variance, which is exactly the concern the reviewer raises, and it is resolved by the second seed.

Table 2 (ln 370 / ln 377), non-inferiority of SPACE seeded with another method's prior: we follow the reviewer's suggestion of paired confidence intervals and a non-inferiority analysis with a justified margin. We report margins of 2 ASR points (one behaviour out of 50) and 4 ASR points; for reference, the median seed-to-seed movement of a single ASR entry in our evaluation is 6 points. Paired over the six models of Table 2, two seeds each:

ContrastJudgeMean difference95% CIp (2-point margin)p (4-point margin)
SPACE (suffix prior) − GCGHarmBench / GPT-4.1 (submission)+0.3[-3.2, +4.2]0.130.007
SPACE (suffix prior) − GCGHarmBench classifier+1.8[-3.2, +6.3]0.0650.012
SPACE (VSS prior) − SlotGCGHarmBench / GPT-4.1 (submission)+0.7[-4.5, +5.0]0.150.037
SPACE (VSS prior) − SlotGCGHarmBench classifier-2.5[-9.0, +2.0]0.590.29

Under the judge used in our submission (HarmBench / GPT-4.1), the mean difference is positive for both priors and SPACE is non-inferior to its reference method at a 4-point margin (p = 0.007 and p = 0.037). Under the HarmBench classifier this also holds for the suffix prior (p = 0.012). For the VSS prior under the HarmBench classifier non-inferiority is not established; this is driven by a single model (Llama-3.1-8B, -18 points), with the other five models within 3 points of SlotGCG. Under the judge of our submission, SPACE seeded with a heuristic's prior is therefore no worse than the heuristic itself, which is the intended claim of ln 377.

Ethics flag

We appreciate the concern raised by the reviewer and take the matter seriously. Our work studies the robustness of safety-aligned models with the aim of making evaluation stronger: fixed-position attacks underestimate vulnerability, and defences evaluated only against them give a false sense of security. All behaviours are from the public AdvBench benchmark, all target models are openly released, and the paper reports no harmful completions. The revised manuscript includes an ethics statement to this effect.

AuthorsOpenReview ↗

Author Rebuttal (Part 2)

We continue to address the reviewers concern in this comment due to the 5000 character limit per comment.

W2 / Q2: Reliability of the judge

We agree that ASR is the primary outcome and that its measurement deserved further ablations. Our submission used GPT-4.1 with the standard HarmBench judging template. We have now conducted an ablation along two axes, the judging template (HarmBench, StrongREJECT) and the judge model (GPT-4.1, the HarmBench classifier, GPT-4o-mini as released with StrongREJECT, and the current-generation GPT-6.1 Sol), giving six judge configurations over all 3,600 white-box completions (best in bold):

Judge (prompt / model)GCGSlotGCGSPACE
HarmBench / GPT-4.1 (submission)42.852.154.4
HarmBench / GPT-6.1 Sol37.245.847.6
HarmBench / HarmBench classifier45.856.058.6
StrongREJECT / GPT-4.147.359.265.1
StrongREJECT / GPT-4o-mini49.861.466.9
StrongREJECT / GPT-6.1 Sol31.941.346.0
Average42.452.656.4

The ordering of the three methods is the same under every judge, including two independently validated judges (the HarmBench classifier and StrongREJECT) and the latest model, and the margin over SlotGCG is largest under StrongREJECT.

Do judging errors differ across attack methods?

The reviewer's concern is that a method-dependent judging error could produce the margin over SlotGCG. Agreement between GPT-4.1 and GPT-6.1 Sol under the same prompt is similar for each method:

PromptOverall agreement (Cohen's kappa)GCGSlotGCGSPACE
HarmBench91.6% (0.83)91.8%92.0%91.0%
StrongREJECT82.2% (0.66)84.1%82.0%80.6%

On the black-box experiments (7,546 completions), the HarmBench/GPT-4.1 and StrongREJECT/GPT-4.1 judges agree on 91.5% of completions (kappa 0.81), and all conclusions hold under both.

Human validation

We have validated the judge against human labels in two ways.

First, two of our six judge configurations have published human validation. StrongREJECT with GPT-4o-mini reaches a Spearman correlation of 0.846 with the median of five human labellers on 1,361 prompt-response pairs (Souly et al., 2024, Table 1), and the HarmBench classifier agrees with human labels on 93.2% of its validation set (Mazeika et al., 2024, Table 3; Spearman 0.819 in Souly et al.). SPACE is the strongest method under both of these judges (table above). On our 3,600 white-box completions, the judge of our submission agrees with the HarmBench classifier on 91.1% of completions (kappa 0.82) and with StrongREJECT / GPT-4o-mini on 85.1% (kappa 0.70).

Second, we have collected our own human labels. One author labelled a stratified random sample drawn from all 5,840 final-state completions of our white-box and black-box experiments. The annotator saw only the behaviour and the generation, exactly as the judge does; the attack method, the target model and every judge's verdict were hidden, and the order was shuffled. Of the 157 labelled without ambiguity (estimates reweighted to the full set of completions) we have:

Judge of our submission vs humanEstimate95% CI
Precision95.3%[90.7, 99.1]
Recall84.7%[76.1, 93.5]
Accuracy91.7%[86.8, 96.0]
Cohen's kappa0.83[0.73, 0.92]

The judge’s decisions are conservative relative to the human annotator: its errors are 4 false positives against 21 false negatives, and the ASR under human labels is 42.6% against 37.8% under the judge. Our reported ASR values are therefore, if anything, underestimates.

Does the judging error differ across attack methods?

On the white-box completions of the three methods of Table 1:

MethodLabelledJudge false positives / false negativesJudge error rateASR, human / judge
GCG220 / 21.7%44.7 / 43.0
SlotGCG312 / 24.6%48.9 / 51.0
SPACE320 / 59.8%64.9 / 55.2

The error rate does not differ significantly between SlotGCG and SPACE (stratified permutation test, p = 0.42). More importantly, the direction of the errors is the opposite of the reviewer's concern: the judge never credits SPACE with a success that the human rejects, and it misses SPACE successes more often than SlotGCG successes. On these labelled completions the margin of SPACE over SlotGCG is +16.0 points under human labels (95% CI [+6.8, +27.0]) against +4.2 under the judge. A method-dependent judging error therefore does not produce our margin.

We note the limits of this check: it is a single annotator, who is an author (though blind to method, model and verdict), and the per-method rows rest on 22 to 32 labels each.

AuthorsOpenReview ↗

Follow-up from the authors

Dear Reviewer 3Mqw,

We thank you again for your time and for stating so clearly what would change your assessment. As the end of the discussion period is fast approaching, we were wondering whether our responses were sufficient to address the two items you raised: 1.) the statistical justification for our claims and 2.) the reliability of the judge. We highlight the following facts that may aid clarity:

1.) With two seeds on all twelve models and a paired analysis, SPACE improves on SlotGCG by 3.8 points on the judge average (95% CI [+0.9, +7.0], p = 0.037) and is non-inferior at a 2-point margin under every judge. The Qwen3-32B deficit was a single-seed effect and is resolved by the second seed.

2.) The ordering of the three methods is the same under six judge configurations, two of which have published human validation. Against blind human labels the judge of our submission has 91.7% accuracy (kappa 0.83) and never credits SPACE with a success that the human rejects.

We would be happy to engage in any further discussion before the end of the period, and hope our answers are detailed enough to convince you to consider upgrading your score if you believe it is merited.

Reviewer_id7fOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPACE treats adversarial token placement as a latent variable. It fits a linear-Gaussian model of per-slot utility, updates it in closed form from the candidate losses that GCG normally discards, and chooses allocations by Thompson sampling plus an exact greedy rule. Existing heuristics (suffix, prefix, SlotGCG) become priors, and posteriors can be merged across prompts and models. On 12 models × 50 AdvBench behaviors, SPACE reaches 54.9% mean ASR, vs. 43.8% for GCG and 51.3% for SlotGCG. The paper also reports U-shaped posteriors, weak correlation with attention (ρ = 0.34), and a leave-one-model-out black-box transfer experiment with BEAST.

Critical strengths
  1. A principled, unifying formulation. Existing placement heuristics can be encoded as priors, and sufficiently concentrated priors can recover them. Posterior merging also lets attack compute be reused on new targets.
  2. Broad evaluation (12 models, 6 families), with one clearly positive result: in the black-box setting, online positional inference outperforms both fixed allocations and the suffix.
Critical weaknesses
  1. The white-box gain (+3.6pp over SlotGCG) cannot be cleanly attributed to positional inference, and the comparison is not compute-matched. SPACE changes three things at once: multi-slot placement, a multi-round insertion schedule, and Bayesian updating. Key controls are missing: random allocation, suffix GCG with the same schedule, and a fixed prefix+suffix split. This matters because SlotGCG (Sec. 3.4) already found that random multi-slot insertion lowers the loss faster than the suffix, and here informed priors add only +0.3/+0.5pp (Table 2). Equal steps are not equal compute: SPACE needs a forward+backward pass per distinct allocation. The unweighted mean of the per-model wall-clock ratios is approximately 1.67×, reaching 2.8×, whereas the reported 33% overhead is the ratio of mean runtimes. The robustness of this margin is difficult to assess given the experimental variability. With 50 behaviors per model, one behavior = 2pp. Half of the models use a single seed, SPACE loses to SlotGCG on 4/12, and about half of the gain comes from two Gemma models run at only 250 steps.

  2. Fig. 5 supports a narrower transfer claim than the abstract suggests. Without any transfer, SPACE-Uniform nearly matches SPACE+Prior (30.6 vs. 31.0; 38.4 vs. 40.5), so most of the gain comes from online inference, not transfer. The “quarter of the query budget” result also holds without transfer. The clearest like-for-like evidence for transfer is posterior-pooled vs. VSS-pooled: +4.9pp and then +1.2pp at the larger budget, with no uncertainty reported for this contrast. Cross-model, cross-behavior posterior similarity is at the shuffled null (Fig. 4c: 0.598 vs. 0.602), consistent with only a generic U-shape transferring. Source and target also share the same 50 behaviors, leaving behavior-disjoint transfer untested.

  3. (Minor) Writing Problems and Self-Contradictions: (1) Fig. 2 reports 76% coverage vs. 74.0% in the text; (2) Table 1's "remaining 5" vs. 6 single-seed models; (3) Duplicate citation entries (Ben-Tov and Eddoubi).

Questions for authors
  1. Can you add the three controls (random, multi-round suffix, prefix+suffix) under the same schedule and budget? If SPACE is close to random multi-slot allocation, does the thesis survive as "breadth of placement matters"?

  2. Can you provide a compute-matched comparison (equal wall-clock or forward+backward count), with per-model wall-clock ratios rather than a ratio of means?

  3. For Fig. 5: what is the paired SPACE+Prior − Uniform difference with uncertainty? Are posteriors merged per behavior or pooled? Can you test behavior-disjoint transfer?

  4. Your baselines are well below those reported by SlotGCG (e.g., Llama-2-7B: 22/46 vs. 52/80). SlotGCG uses early stopping plus manual verification. Can you also report any-step ASR?

5 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1)

We would like to thank Reviewer id7f for a detailed review and for providing us with clear actionable feedback that allows us to clarify our contribution. We believe that the additional ablations address each of the issues brought up, but are happy to engage with any further points the reviewer may have.

W1a / Q1: Missing controls

We agree that the random-positioning baseline would isolate the effect of our inference relative to simply randomly selecting positions. We have added a random-position control to the black-box experiments. It runs the exact same algorithm, insertion schedule and query budget, but on randomly drawn position allocations (two independent draws, 11 models, two seeds):

SuffixVSS-PooledSPACE-PooledSPACE-UniformSPACE + PriorRandom: mean (draw 1, draw 2)
HarmBench / GPT-4.122.926.330.233.532.530.0 (29.9, 30.1)
StrongREJECT / GPT-4.127.932.138.041.840.839.1 (39.1, 39.1)

Regarding the question of whether the thesis survives as "breadth of placement matters", this random position baseline shows that poorly selected breadth can lead to worse performance than random (VSS underperforms random); however, placement informed by posterior inference outperforms both VSS and random. So while breadth does matter to an extent (VSS, Random, and SPACE beat suffix) the current thesis survives as is: intelligent position selection leads to stronger attacks. We reference this new result from the main text and in Appendix C.8.

W1b / Q2: Compute-matched comparison and per-model wall-clock ratios

This is indeed a point that needed further clarification. We first note that the wall-clock figures in the submission included the time SPACE spends writing diagnostic logs after the attack has finished. Measuring attack time only, the per-model ratios (SPACE / mean of the two baselines; seed 0 / seed 1) are:

ModelRatioModelRatio
Nemotron-3.5-30B1.04 / 0.99Qwen3-8B1.33 / 1.31
Phi-4-14B1.17 / 1.16Gemma-4-12B1.33 / 1.50
Llama-3.1-8B1.24 / 1.22Phi-4-Mini1.53 / 1.52
Mistral-7B1.25 / 1.21Llama-3.2-1B1.62 / 1.56
Qwen3-32B1.28 / 1.29Gemma-4-26B1.73 / 2.20
Llama-2-7B1.30 / 1.27Qwen3-0.6B2.00 / 1.92

The unweighted mean of the per-model ratios is 1.40 (seed 0) and 1.43 (seed 1), with a median of 1.3. We now report the per-model ratios rather than a ratio of means.

For three small models we allowed GCG and SlotGCG to continue until they had used the same wall-clock time as SPACE (ASR out of 50 behaviours, 500 steps → matched budget; bold indicates best at the matched budget):

ModelMatched budgetSPACE (500 steps)GCGSlotGCG
Phi-4-Mini1.5x4040 → 3735 → 39
Qwen3-0.6B2.0x2825 → 2522 → 28
Llama-3.2-1B1.5x3130 → 3232 → 30

Pooled, SPACE succeeds on 99 of 150 behaviours against 94 for GCG and 97 for SlotGCG at matched budgets. Net, GCG has changed by -1 point and SlotGCG +8. We have updated our white-box discussion to point to this result in the appendix. However, we stress that the black-box results are already compute matched, since every method uses the same number of target queries.

W1c: Robustness of the white-box margin

We have completed the table so that all twelve models now use two seeds, and we have re-judged every completion under six judge configurations (global response). SPACE is the strongest method under every judge. Averaged across judges, SPACE improves on SlotGCG by +3.8 points (95% CI [+0.9, +7.0], exact permutation p = 0.037) and is ahead on 9 of 12 models, and improves on GCG by +14.0 points (95% CI [+8.8, +19.3]), ahead on 12 of 12.

On the Gemma models: these are the models on which the baselines are weakest, and it is where adaptive placement helps the most, which arguably strengthens our argument. Even if we exclude both, the margin over SlotGCG is +0.4 (95% CI [-2.0, +2.9]) under the submission judge and +3.3 under StrongREJECT (95% CI [+0.3, +6.6] with GPT-4.1; [+1.1, +5.4] with GPT-6.1 Sol). We also note that SPACE is the most stable of the three methods across seeds: the mean absolute difference between seeds per model is 4.8 points for SPACE, against 7.2 for GCG and 9.5 for SlotGCG.

AuthorsOpenReview ↗

Author Rebuttal (Part 2)

We continue to address the reviewers concern in this comment due to the 5000 character limit per comment.

W2a / Q3: Transfer versus online inference in Fig. 5

We agree with this reading and have revised the abstract and main body text to attribute the black-box gain primarily to online inference, especially in light of the random baseline findings.

W2b: Uncertainty for posterior-pooled versus VSS-pooled

SPACE-Pooled − VSS-Pooled is +4.8 points at 4,801 queries over two seeds (95% CI [+1.5, +8.2], p < 0.001; +6.0 on seed 0 and +3.7 on seed 1) and +1.2 at 18,601 queries (95% CI [-5.6, +7.1], seed 0). These intervals are now reported in the paper.

W2c: Posterior similarity and the shuffled null (Fig. 4c)

We have re-analysed posterior agreement with a rank statistic over position bins. A model's own profile agrees strongly with the pool of the other eleven models (mean Spearman +0.85, positive for 12 of 12 models), far above a permuted null (0.00). The reviewer is correct that most of this agreement is a shape shared across behaviours (the behaviour-specific component adds +0.10, 95% CI [+0.08, +0.13], small but clearly distinguishable from zero).

We must stress that we see the emergence of this shared shape as one of our main empirical findings rather than a limitation: across twelve models from six families, positional vulnerability follows a consistent U-shaped profile that is only weakly related to attention and that transfers to held-out models and to unseen behaviours. This profile could not have been observed with fixed-position attacks. It is a finding that is the direct result of applying our method, and it is an insight that our method can measure for future models, whose profile need not be the same.

W2d / Q3: Behaviour-disjoint transfer

We have run this experiment. In the submission, the profile used to attack a behaviour is pooled from the other eleven models' runs on that same behaviour. To make sources and targets behaviour-disjoint, we split the behaviours once into two fixed halves and pool each behaviour's profile from the other eleven models' runs on the opposite half only, so that no source run shares a behaviour or a model with its target. Everything else is unchanged (4,801 queries, 49 behaviours, two seeds). Due to time constraints this is currently on three target models (Qwen3-0.6B, Llama-3.2-1B, Phi-4-Mini); ASR pooled over 294 behaviour-runs:

Same-behaviour profileBehaviour-disjoint profileDifference (95% CI)Disjoint − Suffix (95% CI)
VSS-Pooled46.646.3-0.3 [-6.1, +5.1]+11.2 [+4.8, +18.0]
SPACE-Pooled51.752.0+0.3 [-5.4, +6.1]+17.0 [+10.9, +23.1]
SPACE + Prior50.750.0-0.7 [-6.5, +5.1]+15.0 [+8.2, +21.8]

The transferred profiles recover the same ASR when no source model has seen the target behaviour. Every arm changes by less than one point, and the improvement over the suffix baseline (35.0% on these models) is preserved in full. This strengthens the finding of W2c: the vulnerability profile recovered by SPACE is a property of the models rather than of the individual behaviour, which is what makes it reusable. We have added this experiment to the appendix.

W3: Writing problems

We thank the reviewer for catching these. (1) The coverage is 74.0% (444 of 600); Fig. 2 has been corrected. (2) Six models had a single seed at submission; this is moot now that all twelve have two. (3) The duplicate citation entries have been removed.

Q4: Baseline levels and any-step ASR

Our protocol scores the completion generated from the final state of the attack, once, with a fixed judge and no early stopping or manual verification; this is stricter than the SlotGCG protocol but we also feel it is more general. We also note that across seeds SlotGCG has the highest seed-to-seed variation of the attacks tested (a mean absolute difference between seeds of 9.5 points per model, nearly twice that of SPACE at 4.8). For reference, scoring the best-loss state, or counting a success at either state (seed 0, 12 models):

Final state (headline)Best-loss stateEither
GCG43.045.851.3
SlotGCG51.052.858.2
SPACE55.255.556.8
AuthorsOpenReview ↗

An Additional Experiment Requested by the Reviewer

Addendum to W1a / Q1: suffix under the multi-round schedule

We have now run the remaining control requested in W1a: GCG's suffix placement under SPACE's multi-round insertion schedule. The staged-suffix arm runs the SPACE code with the same schedule, token count and step budget, but every token is forced into the suffix, so the only difference from SPACE-Uniform is where tokens may be placed. All three arms were attacked fresh in one launch (five models, two seeds, 50 behaviours; 1,500 runs). ASR (%) is the average of the six judges of our revised main table:

ModelGCGStaged suffixSPACE-UniformSPACE − staged [95% CI]
Qwen3-0.6B48.855.053.5−1.5 [−7.8, +4.8]
Llama-3.2-1B55.739.063.3+24.3 [+13.5, +35.5]
Phi-4-Mini74.766.575.3+8.8 [+0.7, +17.3]
Mistral-7B82.285.286.0+0.8 [−6.2, +8.5]
Qwen3-8B66.760.870.5+9.7 [−0.7, +20.0]
All five65.661.369.7+8.4 [+4.6, +12.4]

Two findings follow.

The schedule alone does not improve on GCG. Staged suffix is 4.3 points below GCG pooled (95% CI [−8.0, −0.6], p = 0.032). Its effect varies in sign across models (from −16.7 on Llama-3.2-1B to +6.2 on Qwen3-0.6B) and it is not significantly above GCG on any of the five.

With the schedule held fixed, positional placement adds 8.4 points (p < 0.001, paired permutation test over behaviours). The difference is positive under each of the six judges individually (+7.2 to +9.6 points, p ≤ 0.019 each). It is largest where the schedule alone hurts most, and close to zero on Qwen3-0.6B and on Mistral-7B, where all three arms are within four points of each other at 82–86%.

Link to the random-position control. Together, we have run the two controls suggested by the reviewer. We thank them for these suggestions as both controls help us better understand our method, and experimentally, both reinforce the message of our work that our positional inference is the reason for the improved attack performance.

In particular, this staged-suffix control holds the schedule fixed and shows the schedule alone is not the source of the gain. The random-position control in our black-box experiments (reported above) holds the schedule, query budget and breadth of placement fixed and removes only the posterior update: there SPACE-Uniform improves on random allocation by 3.5 points (HarmBench prompt, GPT-4.1 judge: 33.5 vs 30.0 (29.9, 30.1); 95% CI [+0.8, +6.2]; p = 0.023 and 0.037 against the two draws; ahead on 10 of 11 models). Neither the schedule nor breadth of placement accounts for the improvement; taken together, the two controls are evidence that the Bayesian update over positions is the underlying reason for the improvement.

We will add the staged-suffix results to the appendix alongside the random-position control.

Reviewer_hP7ROpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces SPACE, a framework for adaptive allocation of adversarial tokens in language-model attacks. The method uses intermediate optimization losses to update a Bayesian model of positional utility and applies Thompson sampling to select subsequent token allocations. The resulting Gaussian posteriors can be aggregated to initialize attacks on other models. Experiments on 50 AdvBench behaviors across 12 LLMs report average ASR improvements of 11.1 percentage points over GCG and 3.6 percentage points over SlotGCG. The paper also evaluates cross-model transfer through gradient-free BEAST attacks and analyzes the relationship between learned positional preferences and attention-based scores.

Critical strengths
  1. Adaptive allocation based on optimization feedback. SPACE incorporates position selection into the attack optimization process, rather than relying on a fixed suffix or a predetermined attention-based allocation. The use of intermediate losses as observations, together with tractable posterior updates, provides a coherent framework for learning positional preferences. Its integration with both GCG and BEAST demonstrates applicability to gradient-based and gradient-free optimization.

  2. Comprehensive experiments with broad attack coverage across multiple models. The evaluation covers 12 LLMs and reports higher average ASR, with substantial improvements on the two Gemma models. In the seed-0 analysis, SPACE successfully attacks 62 model-behavior pairs that neither baseline solves. This additional coverage is relevant to robustness evaluation. The pooled-allocation experiments further provide evidence that positional information obtained from source models can improve attacks without access to target-model gradients.

Critical weaknesses
  1. The contribution of Bayesian position learning is not sufficiently isolated. SPACE takes approximately 33% longer than the baselines despite matched iteration counts. Comparisons at matched compute, together with simple positional controls, are needed to distinguish posterior learning from additional computation and broader positional search.

  2. The additional benefit of transferred priors appears modest. In Figure 5, SPACE + Prior exceeds SPACE Uniform by only about 0.4 and 2.1 percentage points. Although fixed pooled allocations demonstrate transfer potential, paired uncertainty estimates and per-model results are needed to establish how consistently transferred priors improve online adaptation.

Questions for authors
  1. Does SPACE retain its advantage at matched actual compute and against simple positional controls using the same insertion schedule?

  2. How consistent is the advantage of SPACE + Prior over SPACE Uniform across models and random seeds? Please provide paired uncertainty estimates for this comparison.

3 comments
AuthorsOpenReview ↗

Author Rebuttal

We would like to thank Reviewer hP7R for their time and for their positive assessment of our work. We are glad the reviewer found the framework coherent and the experimental coverage comprehensive. Both weaknesses raised are clear and actionable, and we have run additional experiments to address each one directly.

W1 / Q1: Isolating Bayesian position learning (matched compute and positional controls)

We agree that the submission did not separate posterior learning from additional computation and from broader positional search. We have addressed both parts (full details are in the global response).

Matched compute

Measuring attack time only (our submitted figure included the time spent writing diagnostic logs), SPACE takes 1.4x the wall-clock time of the baselines on average over the twelve models (median 1.3x). For three small models we allowed GCG and SlotGCG to continue for additional steps until they had used the same wall-clock time as SPACE. This does not substantially change the conclusion (ASR out of 50 behaviours, 500 steps → matched budget; bold indicates best at the matched budget):

ModelMatched budgetSPACE (500 steps)GCGSlotGCG
Phi-4-Mini1.5x4040 → 3735 → 39
Qwen3-0.6B2.0x2825 → 2522 → 28
Llama-3.2-1B1.5x3130 → 3232 → 30

Pooled over the three models, SPACE succeeds on 99 of 150 behaviours against 94 for GCG and 97 for SlotGCG at their matched budgets. We also stress that our black-box results are already compute matched: every method uses the same number of target queries.

Positional controls

We have added a random-position control to the black-box experiments. It runs the exact same algorithm, schedule and query budget, but on randomly drawn position allocations (two independent draws, 11 models, two seeds):

SuffixVSS-PooledSPACE-PooledSPACE-UniformSPACE + PriorRandom: mean (draw 1, draw 2)
HarmBench / GPT-4.122.926.330.233.532.530.0 (29.9, 30.1)
StrongREJECT / GPT-4.127.932.138.041.840.839.1 (39.1, 39.1)

Random multi-position allocation

Random allocation bests both suffix (by 7.1 points, p < 0.001) and VSS-Pooled (by 3.7 points, p = 0.02), which confirms that broader positional search matters. On top of this, online inference adds a further improvement: SPACE-Uniform improves on random allocation by +3.5 points (95% CI [+0.8, +6.2]; p = 0.023 and 0.037 against the two draws) and is ahead on 10 of 11 models. This is the contribution of posterior learning at identical compute.

W2 / Q2: Consistency of SPACE + Prior over SPACE-Uniform

We thank the reviewer for this request and now report the paired analysis in the revised manuscript. On the submitted black-box runs (12 targets, 49 behaviours):

Contrast4,801 queries, two seedsSeed 0Seed 118,601 queries, seed 0
SPACE + Prior − SPACE-Uniform+0.3, 95% CI [-2.5, +3.1] (p = 0.86)+2.2-1.5+2.0, 95% CI [-1.9, +6.5]
SPACE-Pooled − VSS-Pooled+4.8, 95% CI [+1.5, +8.2] (p < 0.001)+6.0+3.7+1.2, 95% CI [-5.6, +7.1]

The transferred prior is ahead of the uniform prior on 7 of 12 targets and behind on 5 thus putting the two methods statistically on par with one another. We agree with the reviewer's reading and have revised the text accordingly: once the attack performs online inference, it recovers within the query budget most of what the transferred prior provides. The value of transfer is clearest where no target queries are spent on positions at all: the fixed allocation obtained from pooled SPACE posteriors improves on the attention-based pooled allocation by +4.8 points (95% CI [+1.5, +8.2]). The per-target differences are included in the Appendix of the revised manuscript.

AuthorsOpenReview ↗

Follow-up from the authors

Dear Reviewer hP7R,

We thank you again for your time and for your positive assessment of our work. As the end of the discussion period is fast approaching, we were wondering whether our responses were sufficient to address the two points you raised: 1.) isolating Bayesian position learning from additional compute and broader positional search, and 2.) the consistency of SPACE + Prior over SPACE-Uniform. We highlight the following facts that may aid clarity:

1.) At an identical query budget and breadth of search, SPACE-Uniform improves on random position allocation by 3.5 points (95% CI [+0.8, +6.2]; ahead on 10 of 11 models). Running GCG and SlotGCG to SPACE's wall-clock budget does not change the conclusion.

2.) The paired analysis you requested shows the transferred and uniform priors are statistically on par, and we have revised the text accordingly.

We would be happy to engage in any further discussion before the end of the period.

Thank you for your detailed response. I particularly appreciate the matched compute results, which greatly addressed my concerns. I'm keeping my positive assessment with higher confidence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Global Summary of Rebuttal Responses

We would like to thank all of the reviewers for their time and careful consideration of our work. We are grateful that reviewers have given clear and actionable feedback. We believe the result of the additional ablations we have completed strengthens our work considerably. In this global response we provide an overview of our responses to shared issues raised by the reviewers.

All edits to our submission PDF have been made in blue text to aid reviewers in pinpointing the updates.

Additional Seed Results

At submission time, some of the more expensive models to attack were presented with only a single seed ASR (pointed out by Reviewers id7f-W1c, 3Mqw-W1). We have completed the table such that all twelve models reported now use two seeds.

Judge Ablations and Revised Main Table

Reviewer 3Mqw asked us to further justify and explore the robustness of the judging underlying our ASR. Our initial submission included ASR computed by GPT-4.1 using the standard HarmBench judging template. To make this more robust we have now added StrongREJECT as a judging template and use the HarmBench classifier, GPT-4o-mini (as released with StrongREJECT) and GPT-6.1 Sol as judge models and have carried out a systematic, blind human check of the judge's verdicts.

We report the per-judge means below (12 models, two seeds; best in bold) and include the full tables in the appendix of our revised submission:

Judge (prompt / model)GCGSlotGCGSPACE
HarmBench / GPT-4.1 (submission)42.852.154.4
HarmBench / GPT-6.1 Sol37.245.847.6
HarmBench / HarmBench classifier45.856.058.6
StrongREJECT / GPT-4.147.359.265.1
StrongREJECT / GPT-4o-mini49.861.466.9
StrongREJECT / GPT-6.1 Sol31.941.346.0
Average42.452.656.4

SPACE is the strongest method under every judge. Averaged across judges we maintain a +3.8 ASR improvement over SlotGCG and extend our improvement over GCG to +14.0 ASR. Judge consistency results are provided in our response to Reviewer 3Mqw and are included in the revised manuscript.

Matched Compute Baselines

Reviewers hP7R-W1, id7f-W1b, dFjY-W2 and dFjY-Q3 pointed out the need for matched-compute baselines to contextualize our results. We agree and we have included a new short discussion in the experimental section of the main body and a more extensive discussion and results in the Appendix. We summarize the discussion below:

  • Black-box: We first stress that the black-box results that are in the paper are already compute matched.
  • White-box: This is indeed a point that needed further clarification in our work. Measuring attack time only (excluding the time spent writing diagnostic logs, which was included in the figure in our initial submission), SPACE takes 1.4x the wall-clock time of the baselines on average over the twelve models. For three small models, we take their wall-clock ratio with SPACE and allow GCG and SlotGCG to run for the corresponding number of additional steps, continuing from their 500-step state. For each model we find that this does not substantially change the conclusion.

Random Position Baselines

Reviewers hP7R-W1, id7f-W1a and dFjY-W2 all point out the need to understand our results versus a random position baseline. To this end, we have augmented our black-box results with attacks that run the exact same algorithm on two randomly drawn position allocations. These are currently stated for 11 out of the 12 models (missing Nemotron-3.5-30B due to computational expense). The re-run of all methods is as follows (mean ASR over 11 models, two seeds):

SuffixVSS-PooledSPACE-PooledSPACE-UniformSPACE + PriorRandom: mean (draw 1, draw 2)
Original submission21.226.030.831.431.6N/A
Re-run (hardware differences)22.926.330.233.532.530.0 (29.9, 30.1)
Re-run, StrongREJECT (GPT-4.1)27.932.138.041.840.839.1 (39.1, 39.1)

In summary, we find that random allocation bests both suffix (by 7.1 points, p < 0.001) and VSS-Pooled (by 3.7 points, p = 0.02). It is competitive with SPACE-Pooled, but as soon as we use inference our method is stronger:

  • SPACE-Uniform improves on random allocation by +3.5 points (95% CI [+0.8, +6.2]; p = 0.023 and 0.037 against the two random draws; ahead on 10 of 11 models).
  • SPACE + Prior improves on it by +2.5 points (95% CI [-0.2, +5.2]; p = 0.11 and 0.15).

Statistical Analyses for Claims

Reviewer 3Mqw asked for further statistical analysis of our results. We give the full analysis for Tables 1 and 2 in our response to the reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.