acceptodds
Under review as a conference paper at ICLR 2027

CrossAnchor: Image-Anchored Text Optimization Exposes Blind Spots in Multi-line Defenses of Agentic Systems

Abstract

Red teaming of LLMs is still largely conducted by testing models in isolation. In reality, however, external guardrails typically stand as a first line of defense by screening and filtering harmful text inputs before they are passed to an LLM. In our work, we develop novel red-teaming attacks that test the multi-line defense as a whole. We build on the Greedy Coordinate Gradient (GCG) approach with two key differences. First, we swap the target-model log-likelihood loss of GCG for a cross-modal anchor, namely, the cosine distance to a non-text representation of the same intent. This is motivated by the asymmetry in a model's alignment across modalities. As the anchor we use a typographic render or a related image-only artifact, and we average the cosine distance across an ensemble of independently trained dual-encoders. This helps remove dependence of the optimization signal on any specific target model and produces transfer across architectures and modalities. The optimizer never sees the target model. Crucially, the anchor is needed only to construct the attack, not to apply it. Second, instead of appending a suffix, we search over a sparse, randomly selected subset of the prompt's own interior tokens, sparse enough to disrupt guardrail pattern-matching. The cross-modal loss recovers the steering signal that masking destroys. On prompts drawn from SALAD-Bench, evaluated across 8 sparsity levels and 6 frontier models, Qwen3Guard detection drops from 50.3% for unedited prompts to 1.2% for optimized prompts at 80% sparsity while optimized ASR at the same sparsity recovers from  4.3% for random-fill baselines to 14.3% to 68.3% across the six target models. Played as a sparsity cascade, the method succeeds end-to-end on 71.4% to 97.5% of the cohort across the six target models, including 71.4% and 97.5% on the two held-out models. This exposes a blind spot in typical evaluation scenarios and highlights the need for defenses that are robust to techniques with non-textual optimization signals. The optimized prompts and end-to-end success envelope form an adversarial corpus and residual-risk benchmark to harden input-side guardrails of agentic systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
adversarial robustness, agent safety, multimodal, embedding, guardrails, LLM, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7bD9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes CrossAnchor, which guides sparse edits within text prompts using image anchors and an ensemble of dual encoders. Optimization requires no target-model feedback, and the deployed input remains text-only. Evaluation covers six targets, two guardrails, and three judges. On a selected cohort of 161 prompts, optimized edits substantially outperform random replacements at high sparsity. A nine-attempt cascade achieves 71.4–97.5% success under Qwen3Guard and the primary GPT-4o judge.

Critical strengths
  1. A relevant evaluation of combined defenses. The paper evaluates both input-guardrail evasion and downstream model behavior, exposing failures that component-level evaluation can overlook. The separation between image-based optimization and text-only deployment is practically interesting.

  2. Useful paired controls and sensitivity analysis. Optimized and random-fill variants share prompts and editable positions. At 80% sparsity, optimization improves attack success by 14.3–64.0 percentage points across six targets under the primary judge. Additional guardrails, secondary judges, and targets excluded from cohort selection help establish the scope and robustness of the finding.

Critical weaknesses
  1. The image anchor’s contribution remains insufficiently isolated. A matched text-anchor baseline is essential to distinguish the benefit of cross-modal anchoring from semantic reconstruction after corruption. Table 4 also changes ensemble size, similarity scope, and search budget together, so its 1.1-point average improvement provides limited evidence about the individual design choices.

  2. The headline effectiveness is conditional on sample selection and retry budget. The cohort retains 161 of 1,000 candidates whose image anchors already succeed against all four selection targets. Original-text success is already 54.0–95.7% before guardrail gating (Tables 10–15). Furthermore, the nine-attempt cascade is compared primarily with a single original-text attempt. Evaluation on the original candidate pool, initially refused cases, and matched-budget baselines would clarify generality and the contribution beyond repeated attempts.

  3. Success-label validity needs stronger evidence. Appendix D counts factual information related to the harmful topic as potential success, which may include benign engagement with the topic. Across all evaluated responses, pooled success is 49.3% under GPT-4o and 31.2% under either secondary judge (Table 27). The shared rubric and substantial disagreement motivate a human audit focused on assistance toward the original harmful intent.

Questions for authors
  1. How does image anchoring compare with text anchoring when the ensemble, editable positions, scope, and optimization budget are held constant?

  2. Can you report results on the original candidate pool and initially refused prompts, together with random-fill and competitive text-attack baselines under equal attempt budgets?

  3. Can you provide a blinded human audit of representative successes and judge disagreements to validate intent-specific harmful assistance?

Reviewer_K2BMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents CrossAnchor, a method for generating text-only jailbreak attacks using image embeddings. Instead of optimizing prompts directly against a target language model, it uses an image with the same meaning as the original prompt to guide token optimization. Some tokens are replaced or modified to avoid input safety filters, while the image embedding helps preserve the original intent. Experiments on 161 prompts and six target models show that CrossAnchor can bypass input guardrails and still produce unsafe responses. Results also show some transfer to unseen models.

Critical strengths
  1. Evaluating the full defense pipeline is a useful direction. Considering both input guardrails and model safety alignment makes the setting more realistic.

  2. Using image embeddings to guide text-only attacks is an interesting idea. It also avoids the need to access target model gradients during optimization.

  3. Experiments cover several models, including two held-out targets. Testing different guardrails and using multiple judges also makes the evaluation more complete.

Critical weaknesses
  1. GCG, sparse token optimization, and embedding-based prompt rewriting have already been studied. Combining them with image embeddings is interesting, but the improvement over existing text-based optimization methods is still unclear.

  2. Random token replacement is a relatively weak baseline. A more useful comparison would be optimizing text against the original prompt's text embedding, rather than an image embedding. This would help show whether the cross-modal design is really necessary.

  3. Only 161 prompts are used, and they are selected because their image anchors already work against several target models. Such filtering may make the attacks easier and raises questions about generalization to other harmful requests.

  4. Table 4 shows only a small average improvement when moving from two to four encoders. Different search settings are also used, making it difficult to understand where the improvement comes from.

Questions for authors
  1. Could the authors compare CrossAnchor with a text-to-text embedding optimization baseline under the same search budget? This would help explain why an image anchor is needed.

  2. How well does the method work on randomly selected SALAD-Bench prompts without the initial image-anchor success filtering?

  3. Can the authors provide a fair comparison between one, two, and four encoders using the same search space and optimization budget?

  4. How effective is CrossAnchor when only one or two attack attempts are allowed? Could the authors highlight single-attempt results separately from cumulative success?

Reviewer_TbdHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrossAnchor, which optimizes text prompts toward image representations of the same harmful intent without querying the target model during optimization. The method edits randomly selected positions using a GCG-style search and scores candidates with an ensemble of dual-encoders. Experiments evaluate the resulting text against two input guardrails and six generative models on 161 selected SALAD-Bench prompts.

Critical strengths
  1. The image is used only during optimization, while the final attack remains text-only. This makes the construction relevant to text-only interfaces, and evaluating the input guardrail together with the target model addresses a practical security setting.

  2. The proposed method performs better than randomly replacing tokens at the same positions, showing that the optimization is useful. The authors also test two additional models, evaluate the attack against WildGuard, and use several models to assess attack success.

Critical weaknesses
  1. The reported success rate does not distinguish newly induced harmful compliance from filter evasion. The 161-prompt cohort is also selected for image-anchor success on four targets, while many original text prompts already count as successful under the primary judge without a guard.

  2. The benefit of image anchoring is not isolated from optimization and retrying. Random-fill establishes that optimization helps, but there is no text-anchor comparison with the same encoders, search budget and cascade attempts. Table 4 also changes the ensemble, similarity scope and budget together. Since the image anchor is the main methodological contribution, this comparison is important for establishing what it adds over a text-based objective.

Questions for authors
  1. After independently checking whether responses substantively assist the original harmful intent, what fraction of attacks turns an initially refused request into harmful compliance?

  2. Does image anchoring still outperform text anchoring when encoders, optimization budgets and cascade attempts are matched?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Detailed response to reviews have been provided to the reviewers.

We have provided a detailed response with explanations to all the reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.