- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies lifelong safety editing of VLMs, where each editing session receives a cluster of jailbreak attacks sharing one malicious intent. The authors argue that per-instance targets do not enforce consistency within an intent and degrade over long editing sequences. IntentEdit extracts per-instance textual and visual keys, optimizes one shared refusal target per modality for each intent cluster, and writes these mappings into MLP layers through a cumulative ridge least-squares objective with general and benign preservation terms, solved recursively in the style of RLSEdit. On USB and JailBreakV augmented with LLM-induced intent clusters, 100 sequential edits on three VLMs yield better MeanRisk and F1 than EVA, VLGuard, and a system prompt. The paper also reports better retention of earlier edits, within-intent transfer on a held-out split, preserved general knowledge, and lower runtime.
Critical strengths
S1. A relevant problem formulation with an efficient, well-posed solver. Framing jailbreak repair as sequential, intent-level editing matches how vulnerabilities are discovered in deployment. The cumulative ridge objective with a block-RLS update is clean: it never revisits past sessions, its normal matrix is always well-conditioned, and it is substantially faster than EVA. The retention analysis on fixed earlier cohorts is a useful protocol. It shows that IntentEdit avoids the large forgetting EVA exhibits, for example +2.40 vs. +28.00 pp on USB/Qwen.
S2. Consistent aggregate gains over the compared baselines across models, benchmarks, and horizons. IntentEdit achieves the best Mean and F1 at every reported checkpoint on three VLMs and two benchmarks. This advantage holds up to 300 edits, and knowledge preservation is maintained on the WIT-based VQA set up to 450 edits. The evaluation reports both harmful compliance and benign refusal rather than ASR alone.
Critical weaknesses
My main worry is that the main results are in-sample and the “generalization” looks more like a general rise in refusals rather than intent-specific correction. Combined with missing editing baselines, single-run results, and several numbers that don’t reconcile, I don’t think the central claim holds. Details as follows:
W1. Generalization claim not supported. Table 1 evaluates on the clusters that were edited, so it is largely in-sample, and held-out results exist only for USB/LLaVA. In that held-out setting, edited and held-out samples have nearly identical ASR. Benign refusal also rises sharply, from 8% to 27% on LLaVA and from 13% to 43% on VILA. Random clustering performs close to intent clustering. Together these results suggest a global shift toward refusal rather than intent-specific correction; ASR on intents not yet edited would help distinguish the two.
W2. Limited novelty and missing baselines. The method mainly combines EVA’s key extraction with RLSEdit’s solver, adding shared per-cluster targets. The appendix theory consists of standard ridge-regression bounds, and its key coverage assumption is unrealistic for low-rank session keys. RLSEdit, AlphaEdit, MEMIT, and SAME are not compared. The speedup over EVA partly comes from filtering already-safe samples, which EVA could equally use.
W3. Weak rigor and numerical inconsistencies. All results come from a single run, while baselines appear highly unstable: EVA’s ARR is 1.67% with full data but 41% with half the data. Some cohorts are tiny (22 samples on JailBreakV), and the hyperparameter selection data are unstated. Several numbers do not reconcile. At step 100 on USB/LLaVA, the EVA and IntentEdit ASRs do not fit the 1,250-sample denominator the other methods use. The ARR of 88.90% is impossible with 300 benign prompts. Table 9’s step-200 Vanilla row duplicates Table 1’s step-3 row.
W4. Unrealistic assumptions and narrow metrics. The setting assumes attacks arrive already grouped by intent, and here the grouping is produced offline by an LLM without human validation. Utility is measured mainly by refusal rate, and standard benchmarks such as MM-Vet and MMBench are absent. The ASR judge is not specified. There is no cross-benchmark transfer and no evaluation against attacks adapted to the edited model.
Questions for authors
Please refer to the weakness.