acceptodds
Under review as a conference paper at ICLR 2027

FAME: FAILURE-AWARE AND MODEL-ADAPTIVE EXPLORATION FOR JAILBREAKING MLLMS

Abstract

Existing jailbreak attacks against multimodal large language models (MLLMs) make limited use of feedback from failed attempts and generally apply the same exploration strategy across target models. Consequently, they may repeatedly explore regions similar to those associated with previous failures while overlooking model-specific vulnerability patterns. To address these limitations, we propose a Failure-Aware and Model-adaptive Exploration framework (FAME), which reformulates image-side jailbreaking as closed-loop candidate selection over a pregenerated image pool. Within each attack, FAME uses previous failures to suppress candidates in low-success regions through local neighborhood avoidance and global energy attenuation. Across attacks, it trains a lightweight proxy discriminator on historical outcomes to estimate target-specific attack potential and prioritize promising candidates for each MLLM. Experiments on four benchmarks and eight open-source and commercial MLLMs show that FAME consistently improves attack success rate over state-of-the-art baselines, achieving 9.1% higher average attack success rate.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, MLLM Jailbreaking, Black-Box Attacks, Failure-Aware Exploration, Model-Adaptive Exploration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MReAOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FAME is a black-box multimodal jailbreak method that selects images from a pre-generated pool. It combines two within-query filters—a neighborhood constraint around failed candidates and an energy-based constraint aggregating previous failures—with a classifier trained on historical attack outcomes to rank candidates for a target model. The selected image is combined with a typographic rendering of the query. The paper evaluates four benchmarks and eight MLLMs, reports an average ASR of 67.4%, compared with 58.3% for the strongest baseline in terms of average ASR, and provides component-wise ablations and a six-round comparison with HADES on Qwen-VL-Chat

Critical strengths
  1. The paper addresses a practical problem: allocating limited target-model queries more effectively during multimodal safety evaluation. The association between visual proximity and attack outcomes reported in Figure 3 provides an empirical motivation for avoiding similar candidates. Using actual target-response outcomes also differs meaningfully from optimizing an image-harmfulness proxy.

  2. The reported gains are substantial and span several datasets and model families. Table 1 reports average ASRs of 67.4% for FAME and 58.3% for VRP, although VRP remains stronger on Gemini across all four benchmarks. On Qwen-VL-Chat/HADES, Table 2 reports 52.9% for random selection, 82.5% for neighborhood filtering, and 88.9% for the full method; Figure 4 also shows an advantage over HADES across multiple rounds. These results suggest that the selection heuristic can be useful.

Critical weaknesses
  1. The methodological increment and its attribution remain insufficiently established. HADES (Li et al., 2024; https://arxiv.org/html/2403.09792v3#S3.SS2) already uses feedback-driven image refinement, although its image-generation feedback concerns a harmfulness proxy rather than the victim outcomes used here. FAME's distinct contribution is therefore the particular failure-based filtering and historical scoring scheme. Under the stated fixed-pool, binary-feedback, stop-on-success protocol, with a fixed classifier and fixed selection randomness, every continuing history consists only of failures. Assuming no response-text feedback or within-query classifier updates, its candidate order can be computed in advance along this all-failure history. This is a property of the stated protocol, not a proof that the heuristic is ineffective; a good fixed ordering can itself be useful. However, the current experiments do not establish what is gained beyond diversity-aware candidate selection and additional historical supervision. Table 2 lacks a matched diversity-selection control and target-specific versus pooled or cross-model classifier comparisons. It also does not isolate the energy constraint's contribution when both neighborhood filtering and the classifier are present. Given the limited methodological novelty, the available attribution evidence is insufficient to establish a compelling contribution.

  2. The evaluation protocol does not adequately substantiate model adaptation or performance under comparable resources. Section 3.2.2 describes a discriminator trained for each target using that target's historical outcomes, whereas Section 4.1 describes one open-weight surrogate and transfer evaluation on the remaining models. Their relationship needs clarification: which model supplies the historical labels and the online feedback for each result? The historical dataset's size, source, separation from test queries, and label-acquisition cost are unspecified. These details determine whether the results demonstrate target adaptation, transfer, or benefits from additional supervision. Figure 4 provides useful six-round evidence for one model/dataset, but does not establish matched target-query budgets and stopping rules for all Table 1 baselines; the random-selection protocol in Table 2 is also insufficiently described. Without these details, I cannot confidently attribute the reported gains to the proposed search strategy under comparable resources. These are unresolved protocol questions, not evidence of leakage or proven budget inequality.

  3. Given the dual-use nature of the proposed jailbreak method, the paper would benefit from an explicit discussion of potential misuse and responsible release of code, attack examples, and historical attack data. The current limitations discussion primarily addresses technical constraints.

Questions for authors
  1. For each evaluated model, which model supplies the proxy-training labels and which supplies feedback during search? Please reconcile Sections 3.2.2 and 4.1, and report the historical training-set size, source, query-level separation from evaluation, and acquisition cost.

  2. Do the main baselines and random selection receive the same target-query budget and stop-on-success rule? For the random control, are candidates sampled without replacement from the same image pool? Please distinguish online target queries from historical-data acquisition and image-generation costs.

  3. What evidence distinguishes failure-aware selection from a fixed diversity-based ordering under the same pool and query budget, and the target-specific classifier from one trained on pooled or other-model outcomes? The neighborhood-plus-classifier ablation without the energy constraint would additionally clarify the contribution of that component.

3 comments
AuthorsOpenReview ↗

Response to Reviewer MReA Comments

W1: We thank the reviewer for the feedback. Unlike HADES, which relies on image-harmfulness feedback, FAME exploits victim-model failures and historical outcomes for candidate selection. We clarify that the proxy discriminator is not fixed: to mitigate cold-start issues, we first pretrain a shared discriminator, assign an independent copy to each target model, and update each copy every 50 attack rounds using target-specific feedback. These updates enable adaptation across queries. To further validate our contributions, we provide matched-budget comparisons against diversity-based selection and classifiers trained on different model outcomes (Table 1), together with an additional ablation isolating the energy constraint (Table 2).

Table 1. Diversity selection and classifier adaptation (Qwen-VL-Chat / HADES, ASR %).ICC: Inter-Class Constraint.

MethodASR (%)
Diversity selection Farthest-point84.7
Cross-model ICC65.6
Target-specific ICC**88.9

Table 2. Contribution of energy attenuation (Qwen-VL-Chat / HADES, ASR %).ICC: Inter-Class Constraint. NC: Neighborhood Constraint. EAC: Energy Attenuation Constraint.

MethodASR (%)
NC + ICC (w/o EAC)84.2
FAME (NC + EAC + ICC)88.9

W2: To mitigate cold starts, we pretrain a shared discriminator DθD_\theta on 200 labeled Do-Not-Answer samples, disjoint from the four evaluation benchmarks, requiring 200 offline surrogate queries. Each target MLLM maintains an independent copy, updated every 50 attack rounds using only its prior feedback. Thus, target-specific updates enable model adaptation without future-label leakage. We will clarify this in Sections 3.2.2 and 4.1.

We clarify that FigStep, MMSB, VRP, and BAP are evaluated under their original single-shot protocols, while HADES and FAME allow up to six attempts. To ensure fair comparisons, we provide equal-budget ASR@1–6 results under a consistent stop-on-success rule, including Best-of-N, HIMRD, and MIRAGE, as shown in Table 3. Random selection samples without replacement from the same 20-image pool under an identical query budget and stopping rule. We will clarify these protocols and separately report offline costs.

Table 3.

MethodASR@1ASR@2ASR@3ASR@4ASR@5ASR@6
FigStep70.871.171.171.571.772.3
MMSB58.158.458.459.459.459.4
VRP81.581.781.782.182.182.1
BAP5.35.35.55.55.85.8
HADES31.739.247.552.557.962.9
Best-of-N75.275.576.176.176.976.9
HIMRD80.180.180.981.281.281.7
MIRAGE83.183.183.784.084.084.0
FAME31.950.764.474.783.188.9

W3: We acknowledge FAME's dual-use risks and will expand the limitations to address potential misuse. Our responsible release strategy will share aggregate results and sanitized examples while restricting sensitive attack data, harmful outputs, and reusable attack materials to support reproducibility while minimizing misuse.

Q1: To avoid cold starts, we pretrain a shared discriminator DθD_\theta on 200 labeled Do-Not-Answer samples, disjoint from all four evaluation benchmarks, using 200 offline queries. Each target MLLM maintains an independent copy, updated every 50 attack rounds using only its prior feedback. Thus, shared pretraining and target-specific updates enable model adaptation. We will clarify this in Sections 3.2.2 and 4.1.

Q2: We clarify that FigStep, MMSB, VRP, and BAP were originally evaluated under their single-shot protocols, while HADES and FAME allow up to six attempts. Random selection uses the same 20-image pool, samples without replacement, and follows the same query budget and stop-on-success rule as FAME. Following the reviewer's suggestion, we additionally provide equal-budget ASR@1–6 comparisons under a consistent stop-on-success rule, as shown in Table 3.

Q3: To isolate the contribution of failure-aware selection, we compare FAME with fixed farthest-point diversity ordering under identical candidate pools and query budgets(Table 4). We further compare target-specific, and cross-model classifiers under matched training budgets (Table 4). Additionally, we remove the energy constraint while retaining neighborhood filtering and the classifier to quantify its independent contribution (Table 5).

Table 4. Diversity selection and model-specific adaptation (Qwen-VL-Chat / HADES, ASR %). ICC: Inter-Class Constraint.

MethodASR (%)
Diversity selection Farthest-point (Fixed Ordering)80.3
Farthest-point + ICC84.7
FAME + Cross-model ICC65.6
FAME + Target-specific ICC88.9

Table 5. Contribution of energy attenuation (Qwen-VL-Chat / HADES, ASR %).

MethodASR (%)
NC + ICC (w/o EAC)84.2
**FAME **88.9

Thank you for your response, but since you do not have an updated version of the paper, I am maintaining my score.

AuthorsOpenReview ↗

Updated Manuscript with Revisions Highlighted in Blue

Thank you for your feedback. We apologize that the revised manuscript was not available earlier, as we were still incorporating the revisions. We have now uploaded the updated version, with all changes highlighted in blue for clarity. We would greatly appreciate it if you could review the revised manuscript and reconsider your assessment. Thank you for your time and consideration.

Reviewer_572QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study black-box jailbreak attacks against multimodal large language models and argue that existing methods do not sufficiently exploit failed attempts or model-specific attack history. They propose FAME, which leverages failed samples to avoid low-success regions and trains a lightweight proxy discriminator to rank candidate images for each target model. The attack searches over a pre-generated image pool and combines neighborhood avoidance, energy attenuation, and model-adaptive candidate selection. Experiments on four benchmarks and eight MLLMs demonstrate that FAME improves the average attack success rate over the evaluated baselines.

Critical strengths
  • The proposed method is simple and practical for black-box MLLMs, and it combines within-attack failure feedback with model-specific historical feedback.

  • The experiments cover four harmful-request benchmarks and eight open-source and commercial MLLMs, which provides broad empirical evaluation.

Critical weaknesses
  • The main failure-aware design assumes that candidates close to a failed image are also unlikely to succeed. However, the local consistency analysis is only shown in a limited setting, while FAME applies the same hard exclusion rule across different datasets and target models. A failure may also come from model randomness or an imperfect judge, so directly removing its neighborhood may discard useful candidates.

  • The model-adaptive discriminator requires historical attack outcomes from the target model. This introduces a cold-start cost and makes the practical query cost larger than what is reflected by the six online attack rounds. The paper should report how many historical target-model queries are needed and how performance changes when only a small amount of history is available.

Questions for authors

Please see the Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 572Q Comments

W1: We thank the reviewer for this concern. To validate local outcome consistency, we extend Figure 3 to multiple datasets and target models at a fixed similarity threshold of 0.8, using both Beaver-dam-7B and human evaluation (Table 1). These additional results provide broader evidence that candidates near failed samples consistently exhibit low attack success rates across different settings. We acknowledge that hard neighborhood exclusion may discard potentially successful candidates due to model stochasticity or judge errors. We therefore quantify the false-exclusion rate (FER) through human evaluation (Table 2). The low FER indicates that The low FER suggests that neighborhood exclusion provides useful benefits while introducing a relatively limited risk of discarding potentially successful candidates. Future work will explore uncertainty-aware soft constraints to further reduce false exclusions.

Table 1. Cross-model and cross-dataset evaluation of local outcome consistency (similarity ≥ 0.8, ASR %).

Model / DatasetBeaver Success-adj.Beaver Failure-adj.Human Success-adj.Human Failure-adj.
Qwen-VL-Chat / HADES90787.35.8
LLaVA-1.5-7B / HADES92.46.589.65.2
Qwen-VL-Chat / SafeBench88.78.486.26.7
InternVL2-4B / SafeBench91.37.288.55.9
LLaVA-1.5-7B / AdvBench93.15.790.44.6
Kimi-VL-A3B / AdvBench89.68.787.17.3
InternVL2-4B / HarmBench87.89.385.47.8
GLM-4.6V-Flash / HarmBench91.76.989.25.5

Table 2. Human-audited false-exclusion analysis of neighborhood filtering (δ = 0.8).

Model / DatasetFalse-Exclusion Rate (%)
Qwen-VL-Chat / HADES2.2
LLaVA-1.5-7B / HADES1.5
Qwen-VL-Chat / SafeBench2.8
InternVL2-4B / SafeBench1.7
LLaVA-1.5-7B / AdvBench1.9
Kimi-VL-A3B / AdvBench1.1
InternVL2-4B / HarmBench2.3
GLM-4.6V-Flash / HarmBench3.1

W2: We thank the reviewer for the valuable comments. To mitigate cold-start costs, we pretrain a shared discriminator on 200 labeled samples from Do-Not-Answer, requiring 200 offline surrogate-model queries. Each target MLLM receives an independent copy, updated every 50 attack rounds using its previously observed feedback, without additional target-model queries. We further evaluate performance with varying amounts of target-specific history (Table 3) to assess adaptation under limited feedback.

Table 3. Impact of target-specific historical data on LLaVA-1.5-7B / HADES (ASR, %).

Historical Target FeedbackASR (%)
0 (Pretrained only)78.4
50 samples82.3
100 samples85.6
200 samples87.5
Full FAME93.3
AuthorsOpenReview ↗

Kind Request for Your Consideration of Our Rebuttal

Dear Reviewer 572Q,

We sincerely appreciate your valuable comments and the time you have devoted to evaluating our work.

In response to your suggestions, we have provided detailed clarifications and additional results in our rebuttal. We have also submitted an updated manuscript, with revisions marked in blue for your convenience.

As the discussion period draws to a close, we would be grateful if you could take a moment to consider our responses and let us know if any concerns remain.

Thank you again for your thoughtful review and consideration.

Best regards, The Authors

Reviewer_13ubOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FAME is an vision-language-model jailbreak that takes the failure into consideration. Specifically, each query generates a pool of 20 related images from PixArts. Each image in the pool is concatenated with a typographic rendering of the harmful content and sent with a fixed benign prompt. Over up to six rounds, FAME would filter out candidates that are close to earlier failures in CLIP semantics, using a cosine-based distance rule with an energy attenuation rule; it would rerank the remaining candidates with an MLP trained on past success/failure outcomes. Across 4 benchmarks and 8 MLLMs, it reports higher ASR performance than baselines.

Critical strengths
  • The method is simple, failure-aware, and fully black-box, and using failures as feedbacks and per-target histories to guide selection is a practical idea.
  • The evaluation is comprehensive: 4 benchmarks × 8 targets, including 3 commercial models.
  • The paper includes detailed ablation and sensitivity analysis experiments.
Critical weaknesses
  1. FAME stops at the first response that the judge flags, out of up to 6 tries; ASR partly measures how often the judge can be triggered within 6 attempts, and judge false positives inflate it. Please add an experiment to evaluate with an independent judge that is not used during the search, such as the HarmBench classifier, LlamaGuard, etc. Please also add a human audit of a sample if possible.

  2. The budgets for FigStep, MMSB, VRP, and BAP are not stated. If those methods are single-shot, Table 1 compares ASR@6 with ASR@1. FAME's own random-selection variant reaches quite low on Qwen-VL-Chat/HADES, while VRP alone gets promising performance (Table 1). This suggests a best-of-6 VRP might match FAME. Please report ASR against the number of queries for all methods at equal budgets. Please also add Best-of-N jailbreaking (Hughes et al., 2024) and feedback-driven multimodal attacks such as HIMRD (Ma et al., 2025) and MIRAGE (You et al., 2025); both are cited but not compared.

  3. The proxy discriminator is under-specified and may leak test information; the threat model is unclear. The training data ("historical attack records across various queries and public jailbreak datasets") is not described: its size, its sources, how many target queries it cost to collect, and whether it overlaps with the evaluation queries or benchmarks, or whether it comes from the same benchmarks. §4.1 says the authors "use one open-weight model as the surrogate and evaluate transferability", yet §3.2.2 trains Dθ on each target's own outcomes. Nothing shows that Dθ captures model-specific patterns. For example, compare a Dθ trained on model A and tested on model B against one trained and tested on A.

  4. The energy constraint seems nearly redundant, and the attack payload follows FigStep, MMSB, and HADES. The contribution is a selection heuristic over 20 fixed images. With a single failure, τ(x) in Eq. (7) is a monotone function of ‖Φ(x) − Φ(x₁)‖, so the energy rule reduces to a second distance radius. g(x) depends on each candidate, so there is no single global direction. τ also scales with k, σ, and the feature norm, so the threshold 0.5 has no clear meaning. Please compare against standard alternatives: max-min diversity, and bandit or Bayesian-optimization selection, e.g. thompson sampling or GP-UCB over CLIP features.

  5. Seeds or confidence intervals are missing, given that generation, selection, and decoding are all stochastic; ablations and tuning of δ, η, and m use only Qwen-VL-Chat/HADES, and Table 3 reports results on the same set used to choose the defaults. Please report the pool's oracle ASR, i.e. the rate at which any of the 20 images succeeds, to separate pool quality from selection quality, and the cost per query: GPT-4o and PixArt generation, and target and judge calls.

  6. Every baseline is forced into the image+typography concatenation, which departs from the original methods. It might also destroy BAP's optimized perturbation: BAP reaches 36.5% ASR, e.g., 0.4% on InternVL2-4B. Please also report the baselines under their original protocols.

Questions for authors
  1. How many attempts, and what feedback, does each baseline get?
  2. About Dθ: how large is its training set and where does it come from? Does it overlap with the evaluation queries? How many target queries did it cost, and is it updated online?
  3. How is the first candidate chosen when F₀ = ∅? What happens when every candidate is filtered out?
  4. Is Φ applied to the candidate image alone or to the composite that includes the typographic query?
  5. Are the "adjacent" samples from the same query's pool or from other queries in Figure 3? If they come from the same query, the consistent outcomes may reflect query difficulty rather than image regions.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Comments

W1: Table 1.

Evaluation protocolFAME ASR (%)HADES ASR (%)
Beaver-dam-7B88.962.9
HarmBench classifier87.161.3
LlamaGuard-2-8B89.763.8
StrongREJECT-style rubric86.360.4
Human-audited ASR83.157.6

We agree with the concern. Beaver-dam-7B is used only as the online stopping signal, not as the final evaluation judge. Following the reviewer's suggestion, we evaluate attack success using the HarmBench classifier, LlamaGuard-2-8B, a StrongREJECT-style rubric and human-audited ASR, as shown in Table 1.

W2: Table 2

MethodASR@1ASR@2ASR@3ASR@4ASR@5ASR@6
FigStep70.871.171.171.571.772.3
MMSB58.158.458.459.459.459.4
VRP81.581.781.782.182.182.1
BAP5.35.35.55.55.85.8
HADES31.739.247.552.557.962.9
Best-of-N75.275.576.176.176.976.9
HIMRD80.180.180.981.281.281.7
MIRAGE83.183.183.784.084.084.0
FAME31.950.764.474.783.188.9

We agree the budget mismatch must be explicit. FigStep, MMSB and BAP are single-shot; VRP is single-shot in Table 1. We will state this in the caption and add Table 2 for all methods at equal budgets, including Best-of-N, HIMRD, and MIRAGE.

W3: To avoid cold starts, we pretrain a shared discriminator on 200 labeled Do-Not-Answer samples, disjoint from the four evaluation benchmarks, at a cost of 200 offline queries. Each target MLLM maintains a copy, updated every 50 attack rounds using its own feedback, enabling model-specific adaptation. Table 3 further demonstrates its effectiveness through cross-model discriminator comparisons.

Table 3. Cross-model discriminator evaluation.

Discriminator Update ModelQwen-VL-ChatInternVL2-4B
Qwen-VL-Chat88.925.8
InternVL2-4B30.292.5

W4: We acknowledge that the energy constraint reduces to distance filtering with a single failure. With multiple failures, it forms a potential field whose negative gradient guides steepest descent, providing a unified criterion rather than a shared direction. We will clarify the threshold's scale dependence. Table 4 compares FAME with farthest-point diversity, Thompson sampling, and GP-UCB under matched pools and query budgets, and isolates the energy constraint through ablation.

Table 4.

Selection StrategyASR (%)
Farthest-point Diversity78.1
Thompson Sampling69.2
GP-UCB74.4
FAME (Full)88.9

W5: We evaluated FAME over three seeds with 95% confidence intervals and held-out hyperparameter tuning (Table 5). FAME approaches pool-oracle ASR within six queries. Each query uses 20 GPT-4o captions, 20 PixArt images, and up to six target-model and judge calls. Shared pretraining requires 200 offline surrogate queries, with target-specific updates every 50 attack rounds.

Table 5

Model / DatasetFAME ASR@6Oracle ASR@20
Qwen-VL-Chat / HADES88.990.1
LLaVA-1.5-7B / HADES93.394.2
Qwen-VL-Chat / SafeBench61.462.7
InternVL2-4B / AdvBench95.596.4

W6: We clarify that image–text concatenation refers to the general attack paradigm, not baseline modifications. All baselines retain their original protocols, including BAP's optimized perturbations. We will revise Sec. 4.1 to clarify this.

Q1: All baselines follow their original protocols: HADES uses six feedback-guided attempts, while FigStep, MMSB, VRP, and BAP are single-shot without iterative target feedback. FAME uses up to six failure-guided attempts, stopping upon success.

Q2: To mitigate cold-start issues, we pretrain DθD_\theta on 200 labeled image-query samples from Do-Not-Answer, disjoint from all four evaluation benchmarks, requiring 200 offline queries. Each target MLLM maintains an independent copy, updated every 50 attack rounds using only its prior outcomes.

Q3: When F0=∅F_0=\emptyset, the first candidate is randomly selected. If all remaining candidates are filtered out, the attack terminates as a failure. Such cases are rare due to the diversity of our image pool. We will clarify in the revision.

Q4: We clarify that Φ\Phi is applied only to the generated candidate image, not to the composite image containing the typographic query. We will clarify this in the revised paper.

Q5: To control for query difficulty, we select successful and failed anchors from the same query and evaluate equal numbers of neighboring images at identical similarity thresholds. Table 6 shows consistently higher ASR near successful anchors, supporting local visual consistency rather than query-difficulty effects.

Table 6.

SimilaritySuccess-adjacentFailure-adjacent
0.553.240.5
0.781.614.3
0.887.59.8
AuthorsOpenReview ↗

Kind Request for Your Feedback on Our Rebuttal and Revised Manuscript

Dear Reviewer 13ub,

Thank you for your time and effort in reviewing our paper.

We have carefully addressed your concerns in our rebuttal and uploaded a revised manuscript with changes highlighted in blue. As the discussion period is approaching its end, we would greatly appreciate it if you could kindly review our responses and share any further feedback.

Thank you very much for your time and consideration.

Best regards, The Authors

Thank you for the detailed response! Since my concerns have been addressed, I raise my score to a weak accept. Also, I realized that I have limited knowledge and background in this field, so I reduced my confidence as well. I am curious why the latest vision-language models, such as Qwen3 VL, are not included in the baselines.

We sincerely thank you for your thoughtful feedback and positive reassessment. We agree that evaluating FAME on the latest vision-language models is important. While our current experiments cover eight representative MLLMs, we plan to include additional evaluations on newer models, such as Qwen3-VL, InternVL3.5, and DeepSeek-VL, in the final version of our paper to further assess the generalizability of our method. We greatly appreciate your valuable suggestion and continued consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal and Manuscript Revisions

Dear ICLR 2027 AC, SAC, PC, and Reviewers,

We sincerely thank you and the three reviewers for your valuable feedback. We have carefully addressed the concerns raised during the rebuttal and uploaded our revised manuscript and supplementary material, with revisions highlighted in blue. We summarize our main updates below.

1. Independent Evaluation and Reliability. We separate the online stopping signal from final evaluation by introducing HarmBench, LlamaGuard-2-8B, a StrongREJECT-style rubric, and human auditing. FAME consistently outperforms HADES across these independent assessments (Table 6 in the revised paper).

2. Fair Baseline Comparisons. We clarify the original protocols of all baselines and provide matched-budget ASR@1–6 comparisons, including Best-of-N, HIMRD, and MIRAGE. Under six target calls, FAME achieves 88.9% ASR, outperforming MIRAGE (84.0%). We also preserve the original baseline implementations and clarify the stopping rules (Tables 1–2 in the revised paper).

3. Discriminator Training and Model Adaptation. We specify the historical data source, query-level separation, and initialization costs. Each target uses 200 Do-Not-Answer initialization samples and maintains an independently trained discriminator, updated every 50 rounds using target-specific feedback. Cross-model transfer and limited-history experiments further demonstrate the value of model-specific supervision (Tables 4–5 in the revised paper).

4. Validity of Failure-Aware Selection. We extend local outcome consistency analysis across eight model–benchmark settings, incorporating human evaluation. Same-query comparisons control for query difficulty, while human-audited false-exclusion rates (1.1%–3.1%) quantify the limitations of neighborhood filtering (Tables 8–9 in the revised paper).

5. Methodological Contributions and Ablations. We clarify the distinction from HADES and acknowledge the fixed-order property within individual queries. Matched-budget comparisons show that FAME improves ASR over fixed farthest-point ordering (88.9% vs. 80.3%). Additional ablations isolate the energy constraint (88.9% vs. 84.2% without EAC), while Thompson sampling, GP-UCB, and discriminator-transfer comparisons provide further controls. We also clarify the energy constraint's interpretation and threshold dependence (Table 3 and Appendix A).

6. Evaluation Transparency and Computational Costs. We provide pool-oracle comparisons, with FAME approaching oracle ASR within 0.9–1.3 percentage points across four settings (Table 7 in the revised paper). We further clarify offline initialization, image-generation, and online query costs, and report held-out hyperparameter selection and additional sensitivity analyses in Appendix A. The three-seed study currently reports point estimates.

7. Implementation and Protocol Clarifications. We specify random first-candidate selection, termination when all candidates are filtered, image-only feature extraction, and the random baseline's sampling without replacement. We also clarify the sources and timing of feedback, improving methodological transparency and reproducibility.

8. Limitations and Responsible Release. We have expanded the limitations discussion to acknowledge model stochasticity, judge disagreement, potential false exclusions, and evaluation constraints. We also outline a responsible release strategy that prioritizes aggregate results and sanitized examples while restricting sensitive attack data and reusable attack materials.

Overall, these revisions provide additional empirical evidence and methodological clarifications addressing the reviewers' main concerns regarding evaluation fairness, methodological contributions, model adaptation, reliability, and reproducibility.

We sincerely appreciate your time and consideration and hope these updates can be taken into account in the final assessment.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.