acceptodds
Withdrawn from ICLR 2027

Black-box Optimization of LLM Outputs by Asking for Directions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present a novel approach for attacking black-box large language models (LLMs) by exploiting their ability to express confidence in natural language. Existing black-box attacks require either access to continuous model outputs like logits or confidence scores (which are rarely available in practice), or rely on proxy signals from other models. Instead, we demonstrate how to prompt LLMs to express their internal confidence in a way that is sufficiently calibrated to enable effective adversarial optimization. We apply our general method to three attack scenarios: adversarial examples for vision-LLMs, jailbreaks and prompt injections. Our attacks successfully generate malicious inputs against systems that only expose textual outputs, thereby dramatically expanding the attack surface for deployed LLMs. We further find that better and larger models exhibit superior calibration when expressing confidence, creating a concerning security paradox where model capability improvements directly enhance vulnerability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Black-box Optimization, LLM attacks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zVC9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a query-based black-box attack that uses only the target LLM's textual outputs, without requiring logits, confidence scores, or auxiliary models. The key observation is that LLMs can provide useful binary comparisons between candidate inputs, even when their absolute confidence estimates are poorly calibrated. Based on this observation, the authors develop a hill-climbing approach and evaluate it on adversarial examples, prompt injections, and jailbreaks. The paper also investigates the effects of model capability, transfer-based initialization, and stronger search strategies.

Critical strengths
  1. The paper presents a simple and interesting approach to black-box optimization. Using binary comparisons to extract optimization signals directly from the target model is intuitive and requires neither numerical feedback nor auxiliary models.
  2. The experiments cover diverse attack scenarios and reveal interesting empirical findings. Beyond demonstrating effectiveness across multiple models and tasks, the cross-model experiments suggest a relationship between model capability, preference quality, and attack effectiveness, offering insights into potential vulnerabilities in stronger LLMs.
Critical weaknesses
  1. The comparison with existing black-box attacks is insufficient. The paper largely excludes established methods such as PAIR and TAP because they use auxiliary models, although these methods operate under the same target-access constraints. However, it is common and easy to use auxiliary models in practice, and evaluation under typical auxiliary models is more significant. Given that the authors also use surrogate CLIP models for transfer initialization, comparisons with stronger existing attacks under comparable query budgets would better establish the practical significance of the proposed method.
  2. Lack of straightforward baselines and incremental improvements in effectiveness. The paper lacks comparisons with simple response-only baselines, such as random search or success/failure-guided optimization, under matched query budgets. Moreover, improvements over Transfer-only are marginal on several vision-LLMs, making it unclear how much additional benefit the proposed feedback mechanism provides in these settings.
Questions for authors
  1. For vision-language adversarial examples, attack success is evaluated using a fixed binary classification prompt (e.g., "Does this image contain a dog?"). How sensitive are the reported ASRs to this evaluation prompt?
  2. How sensitive is the binary preference signal to query wording, candidate ordering, and sampling randomness?
  3. Defense implications: Given the observed capability–vulnerability relationship, what insights does the proposed method offer for designing effective defenses? In particular, can models be trained to resist providing attack-useful preference feedback without substantially compromising their general comparison and reasoning capabilities?
Reviewer_4wV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a black-box attack method against llm using model's textual output instead of logits or information from auxiliary models for optimization. The key insight is that LLMs can provide more reliable binary comparison between input than confidence score. Based on this observation, the authors then use the signal to guide a iterative optimization process. This method is evaluated with various experiment including adversarial attack, prompt injections, and jailbreak to show its effectiveness and generalizability.

Critical strengths

1.Novel optimization signal: Using the target LLM's binary preferences enables a simple and generalizable black-box optimization across different attack settings and achieved non-trivial result. 2.Interest findings: the findings on more capable models produces stronger signal/attack as well as the result on framing of an attacker's query matters for the success of optimization sounds particularly interesting to me.

Critical weaknesses
  1. Limited evaluation: the image experiments use only 57 samples with a relatively lenient perturbation budget of 32/255. More stricter budget of 8/255 and 16/255 are needed to demonstrate the reliability of the proposed optimization technique.
  2. Limited improvement on targeted setting: In figure 8, the improvement form transfer-only to transfer+ours is limited. Given the additional queries needed, it's unclear if the proposed method provides meaningful benefit to trasnfer-based attack in the more challenging and practical targeted setting.
  3. Insufficient support for capability–vulnerability paradox: the result in table 1 doesn't fully support the claim that "more capable models produce stronger attacks".
Questions for authors
  1. Can the authors conduct a small scale experiment under stricter perturbation budget (16/32 or 8/32) and show if the binary preference feedback still remain reliable?
  2. Can the author explain why the proposed method only has margin gains in targeted attack?
  3. For the jailbreak evaluation, how consistent are two LLM judges? Would the reported ASR changes if the attack is only counted as successful when both judge labels the response as harmful?
Reviewer_TtrMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes using the attacked model itself as a pairwise preference oracle for black-box adversarial search. Instead of requesting absolute confidence or accessing logits, it asks which of two candidate inputs is closer to an attack goal and uses that decision in a random-mutation hill-climbing procedure. Experiments cover image attacks on 57 ImageNet examples, prompt injection on 56 constructed task/attack combinations, and jailbreaks on 50 AdvBench requests. The paper compares with unoptimized inputs and log-probability-based optimization, and explores transfer initialization, alternative feedback models, and stronger search procedures.

Critical strengths
  1. The central idea is simple and practically relevant: a target that does not expose numerical scores may still answer comparative questions that provide useful search feedback. Algorithm 1 separates candidate generation from feedback, making the proposed contribution easy to understand. The distinction between absolute confidence and relative preference is a useful angle on the security implications of model self-evaluation, and the reported successes warrant attention even without state-of-the-art attack rates.

  2. The experiments explore the idea across three different attack settings and include useful attempts to separate its components. The transfer-only and hybrid results in Table 5 quantify complementary successes, while Table 2 separates the feedback provider from the attacked target. Table 7 also checks sensitivity to strong-template initialization on two jailbreak models. These experiments make the submission more informative than a single attack demonstration and suggest that the feedback mechanism could be a useful component in other search procedures.

Critical weaknesses
  1. The experiments do not isolate the incremental value of the proposed preference signal under a matched text-only query budget. The no-optimization baseline is a single starting point, while Algorithm 1 repeatedly generates candidates and directly checks success. A comparison against a logprob-based attack with more information does not replace a control that uses the same initialization, mutations, and total target-call budget but removes the informative pairwise preference, such as ordinary candidate testing or uninformative accept/reject decisions. This is especially important for jailbreaks initialized with a strong template: repeated attempts alone may account for part of the gain. Table 7 changes initialization, not the preference signal. In addition, it is unclear whether the query budget counts both the comparison call and direct success check in Algorithm 1, and whether Table 4 averages include unsuccessful runs. Without these controls and accounting, the ASRs establish that the overall procedure can succeed, but not how much the new signal contributes or its query efficiency.

  2. The capability–vulnerability paradox is stronger than the evidence supports. Holding a target fixed and improving the feedback model in Table 2 shows that a better oracle improves the attack; it does not establish that a more capable model is less robust when it is the target. The within-family target-size evidence is also limited: the Llama-11B/90B rates of 26.3% and 28.1% correspond to 15 versus 16 successes on the 57 images, with no reported paired uncertainty or repeated-search variability. Figure 3 supports useful ranking behavior for one Qwen model, but does not quantitatively connect comparative-feedback quality to vulnerability across the tested models. Since the paradox is presented as a principal finding, it needs stronger separation of oracle capability and target robustness, or a narrower interpretation of the existing evidence.

Questions for authors
  1. Under the same initialization, perturbations, and total target-call budget, how much does informative pairwise feedback improve over text-only search without that signal? Please clarify whether comparison calls, direct success checks, refusals, and unsuccessful runs are included in the budgets and mean-query figures.

  2. Can you provide paired/repeated-run uncertainty for the target-size comparisons and distinguish evidence that stronger feedback improves search from evidence that stronger targets are more vulnerable? Would the central conclusion change if the latter claim were restricted to the particular models and settings observed?

  3. Section 4.3 presents the RL case study as still relying on the binary preference signal, but Appendix C.1 describes a reward using feedback-model log probabilities blended with tool-call success, then concludes that the binary signal is sufficient. Is there a binary-only ablation supporting that conclusion, or should the case study be explicitly limited to the stronger access/reward setting actually used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.