acceptodds
Under review as a conference paper at ICLR 2027

IntentEdit: Intent-Level Lifelong Editing for VLM Jailbreak Defense

Abstract

Vision Language Models (VLMs) remain vulnerable to multimodal jailbreak attacks. Correcting these vulnerabilities requires safety edits that generalize across attacks sharing a malicious intent and remain effective through subsequent updates. However, learning a separate safety target for each attack does not enforce consistency across different expressions of the same intent. We propose IntentEdit, an intent-level lifelong editing framework that learns one shared textual target and one shared visual target from each pre-grouped intent batch while retaining instance-specific keys. The method jointly incorporates these mappings into a cumulative least-squares objective with constraints on changes along general-knowledge and benign multimodal directions. Recursive updates retain historical mapping constraints through accumulated statistics. We augment two multimodal jailbreak benchmarks with intent groupings to evaluate sequential safety editing. Across the evaluated VLMs, IntentEdit achieves a better safety–utility balance and lower editing runtime than baseline methods. Evaluations on attacks excluded from editing show within-intent transfer, and evaluations of earlier attacks support the safety retention of previous samples in later updates. Ablations support the contribution of shared targets beyond cumulative editing alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Lifelong Model Editing, Jailbreak Defense, Visual Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pk8aOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies lifelong safety editing of VLMs, where each editing session receives a cluster of jailbreak attacks sharing one malicious intent. The authors argue that per-instance targets do not enforce consistency within an intent and degrade over long editing sequences. IntentEdit extracts per-instance textual and visual keys, optimizes one shared refusal target per modality for each intent cluster, and writes these mappings into MLP layers through a cumulative ridge least-squares objective with general and benign preservation terms, solved recursively in the style of RLSEdit. On USB and JailBreakV augmented with LLM-induced intent clusters, 100 sequential edits on three VLMs yield better MeanRisk and F1 than EVA, VLGuard, and a system prompt. The paper also reports better retention of earlier edits, within-intent transfer on a held-out split, preserved general knowledge, and lower runtime.

Critical strengths

S1. A relevant problem formulation with an efficient, well-posed solver. Framing jailbreak repair as sequential, intent-level editing matches how vulnerabilities are discovered in deployment. The cumulative ridge objective with a block-RLS update is clean: it never revisits past sessions, its normal matrix is always well-conditioned, and it is substantially faster than EVA. The retention analysis on fixed earlier cohorts is a useful protocol. It shows that IntentEdit avoids the large forgetting EVA exhibits, for example +2.40 vs. +28.00 pp on USB/Qwen.

S2. Consistent aggregate gains over the compared baselines across models, benchmarks, and horizons. IntentEdit achieves the best Mean and F1 at every reported checkpoint on three VLMs and two benchmarks. This advantage holds up to 300 edits, and knowledge preservation is maintained on the WIT-based VQA set up to 450 edits. The evaluation reports both harmful compliance and benign refusal rather than ASR alone.

Critical weaknesses

My main worry is that the main results are in-sample and the “generalization” looks more like a general rise in refusals rather than intent-specific correction. Combined with missing editing baselines, single-run results, and several numbers that don’t reconcile, I don’t think the central claim holds. Details as follows:

W1. Generalization claim not supported. Table 1 evaluates on the clusters that were edited, so it is largely in-sample, and held-out results exist only for USB/LLaVA. In that held-out setting, edited and held-out samples have nearly identical ASR. Benign refusal also rises sharply, from 8% to 27% on LLaVA and from 13% to 43% on VILA. Random clustering performs close to intent clustering. Together these results suggest a global shift toward refusal rather than intent-specific correction; ASR on intents not yet edited would help distinguish the two.

W2. Limited novelty and missing baselines. The method mainly combines EVA’s key extraction with RLSEdit’s solver, adding shared per-cluster targets. The appendix theory consists of standard ridge-regression bounds, and its key coverage assumption is unrealistic for low-rank session keys. RLSEdit, AlphaEdit, MEMIT, and SAME are not compared. The speedup over EVA partly comes from filtering already-safe samples, which EVA could equally use.

W3. Weak rigor and numerical inconsistencies. All results come from a single run, while baselines appear highly unstable: EVA’s ARR is 1.67% with full data but 41% with half the data. Some cohorts are tiny (22 samples on JailBreakV), and the hyperparameter selection data are unstated. Several numbers do not reconcile. At step 100 on USB/LLaVA, the EVA and IntentEdit ASRs do not fit the 1,250-sample denominator the other methods use. The ARR of 88.90% is impossible with 300 benign prompts. Table 9’s step-200 Vanilla row duplicates Table 1’s step-3 row.

W4. Unrealistic assumptions and narrow metrics. The setting assumes attacks arrive already grouped by intent, and here the grouping is produced offline by an LLM without human validation. Utility is measured mainly by refusal rate, and standard benchmarks such as MM-Vet and MMBench are absent. The ASR judge is not specified. There is no cross-benchmark transfer and no evaluation against attacks adapted to the edited model.

Questions for authors

Please refer to the weakness.

Reviewer_dEi1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IntentEdit is a lifelong model-editing defense against multimodal jailbreaks in VLMs. Attacks are grouped into LLM-induced “malicious intent” clusters, and each cluster is edited in one session. The method learns one shared refusal target per modality for the cluster and writes it into MLP layers via cumulative ridge least squares with benign-preservation terms. Across LLaVA-1.5, Qwen2.5-VL and VILA on USB and JailBreakV, it reports a better safety–utility balance and more stability than EVA over 100–300 sequential edits, plus within-intent transfer and lower runtime.

Critical strengths

Long-horizon stability: EVA collapses on Qwen2.5-VL after about 50 edits, while IntentEdit holds its ASR/ARR operating point through 300 edits, and retention and knowledge-preservation results agree.

Simple, isolated contribution: Sharing one target per intent is easy to implement, and the controlled ablation shows a large drop without it (F1 79.28 → 50.86).

Practical efficiency: Block-RLS with accumulated statistics makes editing faster than EVA on all three models (e.g., 9.81 h → 2.00 h on Qwen).

Critical weaknesses

Evaluation largely on edited samples: Main ASR is computed on the growing set of edited attacks. The only held-out test is one within-cluster split on USB/LLaVA, where held-out items may be near-duplicates.

No cross-intent or out-of-distribution test: Transfer to unedited intents, new attack families, or other benchmarks is not evaluated. The intent grouping itself is an oracle built with whole-dataset access.

Modest gains over trivial baselines: On Qwen, Mean improves only slightly over Vanilla (15.23 vs 17.45), and a system prompt achieves much lower ASR. On LLaVA and VILA, ARR rises 3–4×, which equal-weight Mean/F1 hides.

Thin baselines and utility evaluation: EVA is the only editing baseline, and utility is measured on about 300 benign prompts plus a custom VQA metric rather than standard benchmarks.

No variance reporting: All results come from single runs with one cluster order, and the ablations cover only one setting.

More recent jailbreak works [1] are not discussed, which also unifies one simple entropy target to successfully break some pioneering models.

[1] High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models, WACV 2027

Questions for authors

Can you report ASR on held-out base queries/templates and on not-yet-edited intents for all model/dataset settings?

Why does the per-instance-target ablation (F1 50.86) underperform EVA (65.17), and were its hyperparameters re-tuned?

Is the ARR benign set disjoint from the preservation pool and VLGuard’s training data? Does the same judge both filter samples and score ASR?

Please verify two oddities: Vanilla LLaVA at step 200 (Table 9) is identical to step 3 (Table 1), and EVA’s per-sample runtime is constant across all checkpoints (Table 8).

Reviewer_jZnKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IntentEdit, a lifelong model-editing framework for defending vision-language models (VLMs) against multimodal jailbreaks. Attacks are first grouped offline into malicious-intent clusters using Qwen3.5-Plus, and each editing session then processes one cluster. For each attack, the method extracts a textual key and a visual key. The textual key is the MLP activations over the harmful keyword span, averaged across paraphrases generated by an LLM, with a benign “general content” direction subtracted. The visual key is the activation at the single image token that receives the most attention from the harmful span, after subtracting a benign attention baseline. The experiments show that IntentEdit works well on Llava-1.5-7B and Qwen2.5-VL-7B.

Critical strengths
  1. Many attacks ask for the same harmful thing in different ways, so the paper teaches the model one shared “refuse” signal for the whole group. This is a great improvement from the previous EVA method that can only address one signal for one sample at a time.

  2. It looks like the IntentEdit does not fall apart after many edits. As shown in Table 1 and Figure 3, EVA rejects more and more safe requests at higher steps and retain less knowledge. On the other hand, IntentEdit maintains most of the general knowledge and does not over-refuse with many edits. This makes the idea of the "lifelong" edit possible.

Critical weaknesses
  1. The evaluations were performed on very early models such as Llava-1.5-7b and qwen-2.5-VL, which were released 1.5 and 3 years ago. Most of the open-sourced models released in the past year such as llama-4, gemma-4 and qwen-3.5 and higher already have the innate ability to process image inputs and are presumably more robust against attacks. In fact, qwen-3.5 was used in the data preprocessing steps for this paper. So it makes much more sense to test on more recent models to understand the validity of IntentEdit and if it can still apply to the up-to-date models.

  2. Before editing starts, the authors use an LLM to sort the entire dataset into neat intent groups, so each round gets one clean group. In real life, new attacks arrive unlabeled and mixed, and you don’t know future intents in advance. To test “unseen” attacks, this paper holds out half of each group. Those held-out attacks are still close siblings of the ones they edited. They never test new kinds of intent.

Questions for authors
  1. Can you test at least one current model (for example Llama-4, Qwen3.5 or Gemma 4)? How often do these attacks succeed on it before editing? Does IntentEdit still help?
  2. How would the method work on those mixture-of-expert models?
  3. What is the judge model? And how does it decide whether an answer is harmful or a refusal?
  4. What happens if the attack groups are mixed or wrongly labeled, as they would be in practice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.