acceptodds
Withdrawn from ICLR 2027

ICO: Enhancing Semantic-Shift Jailbreaks via Iterative Context Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Foundation models have achieved remarkable success across diverse tasks, but they remain vulnerable. To investigate such vulnerabilities, semantic-shift jailbreaks have recently emerged as a promising attack paradigm. They bypass explicit safety mechanisms by replacing harmful terms in original harmful questions with benign alternatives and leveraging contextual information to induce the target model to reinterpret these alternatives as their corresponding harmful concepts. However, existing semantic-shift jailbreaks often achieve limited effectiveness. In this work, we reveal that this limitation arises from overlooking the semantic-shift capability of contexts. Through systematic analysis, we find that contexts exhibit substantially different abilities in inducing semantic shifts: contexts with stronger semantic-shift capabilities are more likely to guide models toward recovering harmful meanings and achieving successful jailbreaks. Based on this finding, we systematically identify and distill the characteristics of effective contexts and propose a black-box context-aware semantic-shift jailbreak framework with Iterative Context Optimization (ICO). In each iteration, ICO leverages these characteristics and feedback from the target model to optimize contexts. Extensive experiments on three datasets and eight target foundation models demonstrate that ICO consistently outperforms eight state-of-the-art baselines, achieving an average attack success rate of 74.6%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language model safety, jailbreak attacks, semantic shift, black-box attacks, iterative context optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SL6fOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies semantic-shift jailbreaks, which replace explicit harmful terms with benign substitutes or placeholders and use context to convey the original intent. It identifies an association between contextual semantic recovery and attack success, and proposes Iterative Context Optimization (ICO): a black-box procedure that uses target-model responses, judge feedback, and distilled guidance to refine placeholder contexts.

The main evaluation covers 200 HarmBench behaviors across five text-only and five multimodal settings involving eight distinct target models, with additional evaluations on subsets of AdvBench and StrongREJECT. The reported average Full ASR is 86.0% for text-only settings and 63.2% for multimodal settings, yielding 74.6% across the ten model–modality settings. Ablations examine feedback-based iteration, guidance, and replacement semantics.

Critical strengths
  1. The controlled ablations provide meaningful evidence for the value of adaptive context refinement.

The comparison with 25 independently generated contexts under the same target-query budget is particularly useful: replacing feedback-based optimization with independent generation reduces Full ASR by 14.5–19.0 percentage points across the five text-only targets (Table 4). Removing the guidance while retaining iteration also reduces performance by 7.5–14.5 points. These results support a contribution from adaptive refinement and tailored guidance beyond simply trying more candidates, and are the strongest evidence in favor of the method.

  1. The evaluation addresses an important failure mode with useful distinctions in the outcome metric.

The paper evaluates multiple text and vision-language targets and distinguishes Partial Success from Full Success. This distinction is consequential for semantic-shift attacks: a response may contain related harmful information while still misunderstanding the substituted concept or failing to satisfy the original request. The reported Full ASR improvements, together with the replacement-strategy ablation and additional dataset evaluations, make the approach a potentially useful addition to black-box safety testing.

Critical weaknesses
  1. The comparative claims, particularly for multimodal attacks, exceed what the current controls and implementation details establish.

ICO receives up to 25 target-model queries, whereas Doublespeak, FlipAttack, and all multimodal baselines receive one query (Appendix B). The matched-budget comparisons with PAIR and AutoDAN-Turbo, and the independent-context ablation in Table 4, provide useful evidence for the text-only method. However, they do not establish that the large multimodal margins persist under comparable search budgets. Moreover, Section 4 and Algorithm 1 formalize textual context optimization, while Appendix D.2 provides an illustrative image-binding example without specifying the benchmark-wide image-selection and binding procedure. The same benign image is retained while the text changes its intended interpretation. Without a complete protocol and same-model controls isolating the visual input, the multimodal advantage is difficult to reproduce and attribute.

  1. The principal success metric needs more clearly independent and quantified validation.

GPT-4o is the default judge, and judge feedback and decisions guide optimization and stopping (Section 4.3.3). The manuscript does not clearly distinguish this judge from the evaluator used to compute final ASR. If the same evaluator is used throughout, adaptive search could select its errors; the current evidence does not establish whether this affects the results. Appendix B does report a manual sanity check with no observed misclassifications, but omits the sample size, composition, and annotation details. Consequently, the strength of this validation cannot be assessed. An independent re-evaluation of fixed outputs, supported by a disclosed human audit, is important for establishing that the reported gains reflect fulfillment of the original harmful requests.

Questions for authors
  1. Multimodal protocol and comparable budgets. Please specify how images are selected or generated, assigned to placeholders, and handled during optimization. Do the gains over the strongest multimodal baselines persist under comparable target-query budgets? On the same vision-language targets and at the same budget, how does ICO perform with the image removed or replaced? These results would clarify both the comparative advantage and the contribution of visual conditioning.

  2. Independent validation of attack success. Is the evaluator used for final ASR computation the same judge used for feedback and stopping? Please report the human-validation sample size, sampling strategy, coverage across methods and targets, and annotation procedure. Would the main conclusions hold when fixed outputs from ICO and the strongest baselines are assessed by an independent evaluator or a blinded human audit?

  3. Separation of guidance development and evaluation. Which examples were used to construct the Guidance Prompt, and were they excluded from the reported evaluation sets? Appendix B states that the rules were fixed before the main evaluation, but does not establish example-level separation. If there is overlap, do the conclusions hold on a strictly held-out subset, including the additional dataset evaluations?

4 comments
AuthorsOpenReview ↗

Author Response to SL6f (Part 1/2)

Response to Reviewer SL6f

We sincerely thank the reviewer for the constructive comments and suggestions! Below we respond to the comments in Weakness(W) and Questions (Q).


W1 — The comparative claims, particularly for multimodal attacks, exceed what the current controls and implementation details establish. ICO receives up to 25 target-model queries, whereas Doublespeak, FlipAttack, and all multimodal baselines receive one query (Appendix B). The matched-budget comparisons with PAIR and AutoDAN-Turbo, and the independent-context ablation in Table 4, provide useful evidence for the text-only method. However, they do not establish that the large multimodal margins persist under comparable search budgets. Moreover, Section 4 and Algorithm 1 formalize textual context optimization, while Appendix D.2 provides an illustrative image-binding example without specifying the benchmark-wide image-selection and binding procedure. The same benign image is retained while the text changes its intended interpretation. Without a complete protocol and same-model controls isolating the visual input, the multimodal advantage is difficult to reproduce and attribute.

Q1 - Multimodal protocol and comparable budgets. Please specify how images are selected or generated, assigned to placeholders, and handled during optimization. Do the gains over the strongest multimodal baselines persist under comparable target-query budgets? On the same vision-language targets and at the same budget, how does ICO perform with the image removed or replaced? These results would clarify both the comparative advantage and the contribution of visual conditioning.

(1) To address unequal query budgets, we extended Doublespeak and FlipAttack to Best-of-N evaluations (N = 5, 10, 25) in the text-only setting. Here, @N denotes the percentage of cases achieving at least one successful attack within N target-model queries. The results report mean Full ASR across five target models. ICO achieves the highest mean ASR at @5, @10, and @25, reaching 86.0% at @25 compared with 67.8% for FlipAttack and 42.4% for Doublespeak.

Method (Mean ASR)@5@10@25
Doublespeak40.5%41.4%42.4%
FlipAttack60.2%63.0%67.8%
ICO71.1%78.9%86.0%

For multimodal attacks, we similarly extended three representative baselines (Visual Object Replacement, Visual Text Replacement, and HADES) to Best-of-N evaluations (N = 5, 10, 25). The results report mean ASR across four LVLMs: GPT-5.4 Nano, Gemini 3.1 Flash-Lite, Qwen3-VL-32B, and Qwen3-VL-8B. ICO achieves the highest mean ASR at @5, @10, and @25, reaching 70.9% at @25 compared with 36.5% for the strongest baseline.

Method (ASR)@5@10@25
Visual Object Replacement27.5%31.5%36.5%
Visual Text Replacement21.8%25.8%31.9%
HADES7.9%8.8%10.5%
ICO55.0%63.9%70.9%

(2) To clarify the multimodal protocol, for each harmful term, we select a benign visual concept matching its semantic type and generate a corresponding image. Each image is explicitly bound to its placeholder through an instruction (e.g., "P1 refers to the concept depicted in Image 1"). The generated images, placeholder bindings, and replaced question remain fixed throughout optimization; only the textual contexts are iteratively revised.

(3) To isolate the contribution of image content, we replaced the generated benign images with blank white images while retaining the same textual context optimization procedure. The results are shown below. Replacing benign images with blank white images reduces mean ASR from 70.9% to 58.0%, with consistent decreases across all four LVLMs. These results suggest that cross-modal semantic recovery contributes to ICO's effectiveness.

Image (ASR)GPT-5.4Gemini 3.1Qwen 32BQwen 8BAvg.
Benign image22.0%88.0%83.5%90.0%70.9%
White image14.5%75.0%65.0%77.5%58.0%

Paper Revision (green/magenta). Multimodal image protocol (green): Section 4.3.2 (p. 6, lines 315–317) and Appendix B.1 (pp. 15–16, lines 805–827). Image-content ablation (magenta): Section 5.4 (p. 10, lines 510–516) and Table 10 (p. 10, lines 495–500). Matched query budgets (magenta): Section 5.4 (p. 10, lines 517–524), Tables 11–12 (p. 10, lines 495–515), and Appendix C.1/Table 13 (p. 16, lines 810–822 and 856–861) and C.2/Table 14 (p. 17, lines 864–875 and 886–889).


AuthorsOpenReview ↗

Author Response to SL6f (Part 2/2)

Response to Reviewer SL6f (Part 2)

W2 — The principal success metric needs more clearly independent and quantified validation. GPT-4o is the default judge, and judge feedback and decisions guide optimization and stopping (Section 4.3.3). The manuscript does not clearly distinguish this judge from the evaluator used to compute final ASR. If the same evaluator is used throughout, adaptive search could select its errors; the current evidence does not establish whether this affects the results. Appendix B does report a manual sanity check with no observed misclassifications, but omits the sample size, composition, and annotation details. Consequently, the strength of this validation cannot be assessed. An independent re-evaluation of fixed outputs, supported by a disclosed human audit, is important for establishing that the reported gains reflect fulfillment of the original harmful requests.

Q2 - Independent validation of attack success. Is the evaluator used for final ASR computation the same judge used for feedback and stopping? Please report the human-validation sample size, sampling strategy, coverage across methods and targets, and annotation procedure. Would the main conclusions hold when fixed outputs from ICO and the strongest baselines are assessed by an independent evaluator or a blinded human audit?

(1) Yes. GPT-4o was used for revision feedback and stop judgments, and original final ASR evaluation.

(2) To address potential judge-specific bias, we conducted two additional experiments. First, replacing GPT-4o with DeepSeek-V4.1-Flash for revision feedback while retaining GPT-4o as the in-loop and final judge yields 84.0% mean ASR, compared with the original 86.0%. Second, we independently re-evaluated the fixed final outputs of ICO and FlipAttack using DeepSeek-V4.1-Flash as the final judge, while retaining GPT-4o for ICO's in-loop judgments and revision feedback. ICO achieves 81.9% mean ASR, compared with 86.0%, while FlipAttack achieves 52.3%, compared with 52.7%. These results suggest that using GPT-4o for both feedback and evaluation does not introduce substantial judge-specific optimization bias.

Final Judge ModelICO Feedback ModelICO Mean ASRFlipAttack Mean ASR
GPT-4oGPT-4o86.0%52.7%
GPT-4oDeepSeekV4.1-flash84.0%52.7%
DeepSeekV4.1-flashGPT-4o81.9%52.3%

(3) To validate the judge decisions, we conducted a blinded human audit of 150 final ICO responses, randomly sampling 30 responses from each of the five text-only target models. One annotator, blinded to GPT-4o judgments, assessed Full Success using the criteria in Section 3. The human annotations achieved 92.67% agreement (139/150) with GPT-4o, supporting the reliability of the automated evaluation.

Paper Revision (magenta). Independent final-judge and feedback checks: Section 5.4 (p. 10, lines 503–508) and Table 9 (p. 10, lines 486–493). Human annotation procedure and agreement: Appendix C.5 (p. 17, lines 911–916).


Q3 — Separation of guidance development and evaluation. Which examples were used to construct the Guidance Prompt, and were they excluded from the reported evaluation sets? Appendix B states that the rules were fixed before the main evaluation, but does not establish example-level separation. If there is overlap, do the conclusions hold on a strictly held-out subset, including the additional dataset evaluations?

(1) The Guidance Prompt was distilled from the 120 examples used in the Motivation Study, sampled from HarmBench, AdvBench, and StrongREJECT. These examples were not excluded from the original evaluation sets.

(2) To assess the impact of this overlap, we excluded the overlapping behaviors and recomputed ASR on the remaining held-out subsets across five target models. The ASR changes are small across all three held-out datasets, indicating that ICO's effectiveness remains stable after excluding overlapping development examples.

DatasetOriginal SamplesHeld-out SamplesOriginal Mean ASRHeld-out Mean ASR
HarmBench20016686.0%86.4%
AdvBench1009288.2%88.3%
StrongREJECT1008778.8%77.2%

Paper Revision (magenta). Held-out evaluation: Appendix C.3 (p. 17, lines 891–902) and Table 15 (p. 17, lines 878–883).

Thank you for your response. I understand your point to some extent, but your reply bears clear hallmarks of being AI-generated. This raises some doubts for me, so I will stand by my original assessment.

Thank you for taking the time to reply. We used AI only to polish the English phrasing of our rebuttal for clarity. All new experiments and analyses were conducted and verified by us. Could you please specify which parts or results raise concerns, so we can address them directly?

Reviewer_xzpXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Iterative Context Optimization (ICO), a black-box approach to semantic-shift jailbreaks. The method replaces sensitive terms with placeholders, constructs contextual clues intended to recover their meanings, and iteratively revises those contexts using target-model responses and feedback from a judge model.

Critical strengths

Substantial reported empirical gains. ICO achieves higher Full ASR than the evaluated baselines across all main text-only and multimodal target configurations.

Critical weaknesses
  1. Several headline comparisons use unequal target-query budgets. ICO receives up to 25 target queries, whereas Doublespeak, FlipAttack, and all multimodal baselines receive one.
  2. The central evaluation relies on insufficiently validated judge decisions. The same judge appears to guide optimization and determine success, without a clearly independent final assessment.
  3. The novelty boundary needs sharper justification. As described in the manuscript, ICO combines semantic substitution with an iterative attacker–target–judge loop. The paper needs to demonstrate more precisely what is distinctive about its context-specific feedback and guidance beyond applying a general response-guided optimization procedure to this representation.
Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Author Response to xzpX

Response to Reviewer xzpX

We sincerely thank the reviewer for the constructive comments and suggestions! Below we respond to the comments in Weakness(W).


W1 — Several headline comparisons use unequal target-query budgets. ICO receives up to 25 target queries, whereas Doublespeak, FlipAttack, and all multimodal baselines receive one.

(1) For text-only attacks, we extended Doublespeak and FlipAttack to Best-of-N evaluations (N = 5, 10, 25) under matched query budgets. Here, @N denotes the percentage of cases achieving at least one successful attack within N target-model queries. The results below report mean Full ASR across five target models. Under matched query budgets, ICO achieves the highest mean ASR at @5, @10, and @25, reaching 86.0% at @25 compared with 67.8% for FlipAttack and 42.4% for Doublespeak.

Method (Mean ASR)@5@10@25
Doublespeak40.5%41.4%42.4%
FlipAttack60.2%63.0%67.8%
ICO71.1%78.9%86.0%

(2) For multimodal attacks, we extended three representative baselines (Visual Object Replacement, Visual Text Replacement, and HADES) to Best-of-N evaluations (N = 5, 10, 25). Here, @N denotes the percentage of cases achieving at least one successful attack within N target-model queries. The results below report mean ASR across four LVLMs: GPT-5.4 Nano, Gemini 3.1 Flash-Lite, Qwen3-VL-32B, and Qwen3-VL-8B. ICO maintains its performance advantage at @5, @10, and @25, achieving 70.9% mean ASR at @25 compared with 36.5% for the strongest baseline.

Method (ASR)@5@10@25
Visual Object Replacement27.5%31.5%36.5%
Visual Text Replacement21.8%25.8%31.9%
HADES7.9%8.8%10.5%
ICO55.0%63.9%70.9%

Paper Revision (magenta). Text-only matched-budget comparison: Section 5.4 (p. 10, lines 517–524), Table 11 (p. 10, lines 495–500), and Appendix C.1/Table 13 (p. 16, lines 810–822 and 856–861). Multimodal matched-budget comparison: Section 5.4 (p. 10, lines 517–524), Table 12 (p. 10, lines 510–515), and Appendix C.2/Table 14 (p. 17, lines 864–875 and 886–889).


W2 — The central evaluation relies on insufficiently validated judge decisions. The same judge appears to guide optimization and determine success, without a clearly independent final assessment.

To address concerns about evaluator independence, we conducted two additional experiments. First, replacing GPT-4o with DeepSeek-V4.1-Flash as the revision feedback model while retaining GPT-4o final judge yields 84.0% ASR, compared with the original 86.0%. Second, using DeepSeek-V4.1-Flash as an independent final judge to re-evaluate the fixed final responses yields 81.9% ASR. These results suggest that using GPT-4o for both feedback and evaluation does not introduce substantial judge-specific optimization bias.

Final Judge ModelFeedback ModelASR
GPT-4oGPT-4o86.0%
GPT-4oDeepSeekV4.1-flash84.0%
DeepSeekV4.1-flashGPT-4o81.9%

Paper Revision (magenta). Judge/feedback sensitivity: Section 5.4 (p. 10, lines 503–508) and Table 9 (p. 10, lines 486–493); see also Appendix C.5 (p. 17, lines 911–916) for the human audit.


W3 — The novelty boundary needs sharper justification. As described in the manuscript, ICO combines semantic substitution with an iterative attacker–target–judge loop. The paper needs to demonstrate more precisely what is distinctive about its context-specific feedback and guidance beyond applying a general response-guided optimization procedure to this representation.

Feedback-driven prompt optimization is an established technique (e.g., PAIR and AutoDAN-Turbo), and we do not claim it as our contribution. We build on this technique by introducing a Guidance Prompt distilled from effective contexts, which serves as an empirical prior for iterative context optimization alongside target-model feedback. To our knowledge, such context-specific prior guidance has not been explored for semantic-shift jailbreaks. Unlike PAIR and AutoDAN-Turbo, which optimize the full attack prompt, ICO keeps the replaced question fixed and optimizes only placeholder-specific contexts. Removing the Guidance Prompt while retaining iterative feedback reduces ASR by 7.5–14.5% across five target models (Table 4), demonstrating the added value of this guidance.

Paper Revision (magenta). Contribution clarification: Section 1 (p. 2, lines 100–107). Existing supporting evidence: Section 3 (pp. 3–5) and Table 4 (p. 8; unchanged/black).

Reviewer_RdJtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how context quality affects semantic-shift jailbreaks and proposes Iterative Context Optimization (ICO), a black-box framework that replaces harmful terms with placeholders and iteratively refines their surrounding contexts using target-model feedback. The authors analyze the relationship between semantic recovery and attack success, and evaluate ICO on three safety benchmarks and eight foundation models, reporting substantial improvements over existing text-only and multimodal jailbreak baselines.

Critical strengths

1、The paper addresses a meaningful and practically relevant problem in LLM safety, as understanding how contextual information enables models to recover hidden harmful semantics is important for identifying vulnerabilities that may not be captured by conventional input-level safety mechanisms. 2、The motivation study provides useful empirical evidence by examining different levels of semantic recovery across four target models and analyzing their relationship with attack success rates, which establishes a reasonable basis for the proposed context optimization strategy. 3、The proposed framework is straightforward and well structured, combining harmful-term replacement, initial context generation, and feedback-driven iterative refinement into a unified black-box approach that does not require access to model parameters, gradients, or internal representations. 4、The experiments cover multiple language models and vision-language models, including both open-source and proprietary systems, and the reported results show substantial improvements over the selected baselines, while the ablation studies provide supporting evidence for the effectiveness of guidance-based refinement and iterative optimization.

Critical weaknesses

1、The methodological novelty appears somewhat limited, as Doublespeak has already explored context-based semantic shifts, while methods such as PAIR and AutoDAN-Turbo have studied feedback-driven prompt optimization; moreover, the paper does not clearly establish whether ICO's improvements come primarily from better semantic recovery or from reducing safety refusals. 2、The text-only baseline comparison is relatively limited, particularly on GPT-5.4 Nano, where the strongest evaluated baseline achieves only 7.5% Full ASR compared with ICO's 57.5%; including at least one more recent and competitive adaptive black-box jailbreak method under the same query budget and evaluation protocol would make the reported performance gains more convincing. 3、The multimodal comparison is not fully balanced, as ICO is allowed up to 25 target-model queries while all multimodal baselines use only a single query, and the absence of image-removal or text-only control experiments makes it difficult to determine whether the improvements arise from cross-modal semantic recovery or primarily from textual context optimization with a larger search budget. 4、GPT-4o is used both to guide context revisions and to evaluate attack success, which raises concerns about potential judge-specific optimization bias; although Appendix B mentions manual validation, it does not report the number of evaluated samples or the annotation procedure, making independent evaluation particularly important given the near-perfect Full ASRs reported for several target models.

Questions for authors

Could you provide a quantitative or qualitative breakdown of ICO’s failure cases? Specifically, when ICO fails to achieve Full Success, is it primarily because the model still refuses the request (safety mechanism triggered), or because the model successfully engages but fails to recover the correct harmful semantics (semantic recovery failure)?

Could you clarify if the baseline implementations (especially PAIR and AutoDAN-Turbo) were tuned specifically for GPT-5.4 Nano, or if they used default hyperparameters/prompts that might be suboptimal for this model?

To isolate the source of the improvement, could you provide a control experiment where the image input is removed (or replaced with a blank/neutral image), and ICO optimizes the text-only context against the LVLM? This would clarify whether the gains are truly due to cross-modal semantic recovery or just superior textual context optimization afforded by the higher query budget.

2 comments
AuthorsOpenReview ↗

Author Response to RdJt (Part 1/2)

Response to Reviewer RdJt

We sincerely thank the reviewer for the constructive comments and suggestions! Below we respond to the comments in Weakness(W) and Questions (Q).


W1 — ICO's novelty is unclear given Doublespeak's semantic shifts and PAIR/AutoDAN-Turbo's feedback-driven optimization. It is also unclear whether ICO primarily improves semantic recovery or reduces safety refusals.

Q1 — When ICO fails to achieve Full Success, how often is the cause a safety refusal versus an incorrect recovery of harmful semantics? Please provide a quantitative or qualitative breakdown.

(1) Semantic-shift jailbreaks are a recent paradigm, and the factors governing their effectiveness have received little systematic study. To our knowledge, we are the first to systematically investigate the role of context quality in semantic-shift jailbreaks, identifying it as a critical factor governing semantic recovery and ASR. We also examine the replacement strategy, finding ASR differences of 5.5–18.0% between placeholders and benign terms (Table 7). With the placeholders and replaced question fixed, Full Recovery contexts achieve up to 22.50% higher Full ASR than Low Recovery contexts across four target models (Figure 2).

Feedback-driven prompt optimization is an established technique (e.g., PAIR and AutoDAN-Turbo), and we do not claim it as our contribution. We build on this technique by introducing a Guidance Prompt distilled from effective contexts, which serves as an empirical prior for iterative context optimization alongside target-model feedback. To our knowledge, such context-specific prior guidance has not been explored for semantic-shift jailbreaks. Unlike PAIR and AutoDAN-Turbo, which optimize the full attack prompt, ICO keeps the replaced question fixed and optimizes only placeholder-specific contexts. Removing the Guidance Prompt while retaining iterative feedback reduces ASR by 7.5–14.5% across five target models (Table 4), demonstrating the added value of this guidance.

(2) To distinguish semantic recovery from refusal reduction and address Q1, we analyzed the initial failure modes before optimization and their subsequent conversion to Full Success. The results are summarized in the table below. Among the 565 newly successful cases, 508 (89.9%) originated from "semantic recovery failure" (off-target) responses, compared with 57 (10.1%) from "safety mechanism triggered" (refusal) responses. The substantially higher conversion rate for semantic recovery failure cases (86.4% vs. 48.7%) indicates that ICO's improvements primarily stem from resolving semantic recovery failure rather than merely bypassing safety mechanisms.

Initial failure modeInitial failuresConverted to successNot convertedConversion rate
Semantic recovery failure5885088086.4%
Safety mechanism triggered117576048.7%

Paper Revision (magenta). Novelty and contribution: Section 1 (p. 2, lines 85–89 and 100–107). Failure-mode analysis: Appendix C.4 (p. 17, lines 904–909) and Table 16 (p. 18, lines 918–937). Existing supporting evidence: Section 3 (pp. 3–5, lines 145–246) and Table 4 (p. 8, lines 417–428; unchanged/black).


W2 — The text-only baseline comparison is relatively limited, particularly on GPT-5.4 Nano, where the strongest evaluated baseline achieves only 7.5% Full ASR compared with ICO's 57.5%; including at least one more recent and competitive adaptive black-box jailbreak method under the same query budget and evaluation protocol would make the reported performance gains more convincing.

We thank the reviewer for this suggestion. We have added MAJIC (AAAI 2026) [1], a recent adaptive black-box jailbreak method, as an additional baseline under the same query cap and evaluation protocol. The results are summarized below. Under the same 25-query cap, ICO achieves substantially higher Full ASR on GPT-5.4 Nano and across all five target models, while requiring fewer queries on average.

MethodMax. queriesMean queriesGPT-5.4Gemini-3.1DeepSeek-V3.2DeepSeek-V4Llama-3.3Avg.
MAJIC (AAAI 2026)2512.3210.5%24.0%18.0%19.5%24.0%19.2%
ICO257.1057.5%97.0%99.0%99.0%77.5%86.0%

[1] MAJIC: Markovian Adaptive Jailbreaking via Iterative Composition of Diverse Innovative Strategies. AAAI 2026.

Paper Revision (orange). Added MAJIC results: Table 2 (p. 7, line 332), Table 5 (p. 9, line 439). Related baseline description: Section 5.1 (p. 7, lines 366–377; black, not color-highlighted).


AuthorsOpenReview ↗

Author Response to RdJt (Part 2/2)

Response to Reviewer RdJt (Part 2)

W3 — The multimodal comparison is not fully balanced, as ICO is allowed up to 25 target-model queries while all multimodal baselines use only a single query, and the absence of image-removal or text-only control experiments makes it difficult to determine whether the improvements arise from cross-modal semantic recovery or primarily from textual context optimization with a larger search budget.

Q3 — To isolate the source of the improvement, could you provide a control experiment where the image input is removed (or replaced with a blank/neutral image), and ICO optimizes the text-only context against the LVLM? This would clarify whether the gains are truly due to cross-modal semantic recovery or just superior textual context optimization afforded by the higher query budget.

(1) To address the unequal query budgets, we extended the multimodal baselines to Best-of-N evaluations (N = 5, 10, 25) under matched query budgets. Here, @N denotes the percentage of cases achieving at least one successful attack within N target-model queries. The results below report mean ASR across four LVLMs: GPT-5.4 Nano, Gemini 3.1 Flash-Lite, Qwen3-VL-32B, and Qwen3-VL-8B. Under matched query budgets, ICO maintains its performance advantage, achieving 70.9% mean ASR at @25 compared with 36.5% for the strongest baseline.

Method (ASR)@5@10@25
Visual Object Replacement27.5%31.5%36.5%
Visual Text Replacement21.8%25.8%31.9%
HADES7.9%8.8%10.5%
ICO55.0%63.9%70.9%

(2) To isolate the contribution of image content and address Q3, we replaced the generated benign images with blank white images while retaining the same textual context optimization procedure. The results are shown below. Replacing benign images with blank white images reduces mean ASR from 70.9% to 58.0%, with consistent decreases across all four LVLMs. These results suggest that cross-modal semantic recovery contributes to ICO's effectiveness.

Image (ASR)GPT-5.4Gemini 3.1Qwen 32BQwen 8BAvg.
Benign image22.0%88.0%83.5%90.0%70.9%
White image14.5%75.0%65.0%77.5%58.0%

Paper Revision (magenta). White-image control: Section 5.4 (p. 10, lines 510–516) and Table 10 (p. 10, lines 495–500). Multimodal matched-budget comparison: Section 5.4 (p. 10, lines 517–524), Table 12 (p. 10, lines 510–515), and Appendix C.2/Table 14 (p. 17, lines 864–875 and 886–889).


W4 — GPT-4o is used both to guide context revisions and to evaluate attack success, which raises concerns about potential judge-specific optimization bias; although Appendix B mentions manual validation, it does not report the number of evaluated samples or the annotation procedure, making independent evaluation particularly important given the near-perfect Full ASRs reported for several target models.

(1) To address potential judge-specific bias, we conducted two additional experiments. First, replacing GPT-4o with DeepSeek-V4.1-Flash as the revision feedback model while retaining GPT-4o final judge yields 84.0% ASR, compared with the original 86.0%. Second, using DeepSeek-V4.1-Flash as an independent final judge to re-evaluate the fixed final responses yields 81.9% ASR. These results suggest that using GPT-4o for both feedback and evaluation does not introduce substantial judge-specific optimization bias.

Final Judge ModelFeedback ModelASR
GPT-4oGPT-4o86.0%
GPT-4oDeepSeekV4.1-flash84.0%
DeepSeekV4.1-flashGPT-4o81.9%

(2) To address concerns about sample size, we manually evaluated 150 samples and found 92.67% agreement (139/150) between human annotations and model judgments.

Paper Revision (magenta). Judge/feedback sensitivity: Section 5.4 (p. 10, lines 503–508) and Table 9 (p. 10, lines 486–493). Human audit: Appendix C.5 (p. 17, lines 911–916).


Q2 — Could you clarify if the baseline implementations (especially PAIR and AutoDAN-Turbo) were tuned specifically for GPT-5.4 Nano, or if they used default hyperparameters/prompts that might be suboptimal for this model?

All evaluated baselines, including PAIR and AutoDAN-Turbo, follow their official implementations with default hyperparameters and prompts. Neither method was specifically tuned for GPT-5.4 Nano.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.