- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VASE, a sparse editing method for selective visual unlearning in multimodal language models. It identifies visual FFN channels using normalized activations on forget samples and restricts parameter updates to the corresponding projection rows. The training objective combines bounded target-answer suppression with output-level and activation-level retention losses.
The authors evaluate VASE on MLLMU-Bench and CLEAR, primarily using Qwen2.5-VL-3B and InternVL3.5-8B. Additional results cover two other backbones and different forget ratios. The method achieves substantial reductions in Forget VQA accuracy while largely preserving retain-set and general performance.
Critical strengths
The main experimental results are strong. VASE achieves the lowest Forget VQA accuracy among the compared methods across the two main backbones and benchmarks. The improvements are substantial, especially considering that retain-set performance remains close to the original models. Although VASE does not outperform every baseline on every forgetting metric, its overall forgetting–retention trade-off is encouraging.
The method is simple and requires updating only a small fraction of the model parameters. The ablation study is reasonably thorough, including random, retain-based, and unrelated-image channel selection, matched Visual LoRA, and different forgetting and retention objectives. I also appreciate the analysis of selected-channel interventions on held-out examples.
Critical weaknesses
- The robustness of the unlearned behavior is not sufficiently evaluated.
The main evaluation measures forgetting through VQA accuracy and ROUGE-L. These results show that VASE successfully suppresses target information under the tested queries. However, they do not establish whether the information can still be recovered using alternative prompts or image inputs.
The examples in Figure 5 illustrate this distinction. VASE often produces an alternative answer rather than the original target answer. This is consistent with the paper's objective of suppressing visual access to target knowledge, and is not necessarily a failure. Nevertheless, it would be useful to test whether the original information remains recoverable through paraphrased questions, image transformations, or multimodal extraction attacks.
- The specificity of the activation-based channel selection could be better established.
Equations (3)–(6) select channels based on their normalized activation energy on forget samples, rather than explicitly measuring their selectivity for forget versus retain inputs.
The comparisons in Table 3 are useful. In particular, VASE substantially outperforms random channel selection. However, selecting channels using retain or unrelated images also produces considerable forgetting. On Qwen2.5-VL-3B, these alternatives reach 21.6% and 22.4% Forget VQA accuracy, compared with 17.6% for VASE.
VASE has a clearer advantage in generative forgetting, but the classification results suggest that selecting generally influential visual channels may already account for a substantial part of the improvement. A comparison with a contrastive Forget–Retain activation criterion would help clarify the role of forget-specific localization.
Questions for authors
-
Have the authors tested the edited models against alternative prompts, image transformations, or extraction attacks? Such experiments would help distinguish suppression on the benchmark queries from more robust visual unlearning.
-
Have the authors considered ranking channels using a contrastive score based on both forget and retain activations? How would this compare with the current forget-only criterion?
-
How stable are the results across different forget-identity splits or independent editing runs? The additional results on LLaVA-1.5-7B show a smaller reduction in Forget VQA accuracy than those on the main backbones. It would be helpful to understand whether this reflects architectural differences or sensitivity to the experimental setting.
2 comments
Response to Reviewer sWQu (2/2): Questions for Authors
Q1: Have the edited models been tested under alternative prompts, image transformations, or extraction attacks?
Response: Yes. W1 and W2 report tests on the final edited models: two paraphrases per question, two crop levels, two JPEG quality levels, unseen portraits, and two clue-based extraction prompts with blank-image controls. The results support forgetting under the tested paraphrases, crops, and compression levels. They also identify a limit of single-portrait editing: forgetting is weaker when the portrait is replaced. Using multiple views during editing improves transfer.
Q2: How does a contrastive Forget–Retain ranking compare with Forget-only ranking?
Response: We compared ranking by with ranking by , computing both energies with the same statistic and holding the editor, losses, parameter budget, and schedule fixed. On the official Qwen split, Forget-only ranking lowers mean Forget ROUGE-L from 0.413 to 0.263 across three editing seeds (paired Δ −0.150, 95% CI [−0.216, −0.093], resampling identities after averaging over the seeds). The Forget accuracy difference is −1.1 pp (95% CI [−4.8, +2.7] pp), and the interval includes zero.
Forget-only ranking also lowers both Forget metrics in all four alternate model–split comparisons, with Retain accuracy at most 1.7 pp lower. These results favor Forget-only ranking for open-ended forgetting in our setting. Table R12 in our response to Reviewer 1u4p gives the full comparison.
Q3: How stable are the results across independent edits and different Forget identities?
Response: On the official Qwen split, VASE obtains Forget VQA accuracies of 18.4%, 15.2%, and 19.2% across three editing seeds (17.6% ± 2.1 pp), with Retain accuracy of 53.8% ± 0.6 pp. Forget generation ROUGE-L varies more across runs (0.263 ± 0.146; Table R10).
We also sampled two disjoint groups of 25 identities from the official Retain set. In each alternate split, the sampled identities became the Forget set and the other 475 formed the Retain set. We kept the editing settings fixed within each backbone.
Table R16. VASE on alternate MLLMU-Bench splits (5% Forget). Each cell shows Vanilla → VASE.
| Backbone | Forget split | Forget VQA accuracy ↓ | Forget generation ROUGE-L ↓ | Retain VQA accuracy ↑ |
|---|---|---|---|---|
| Qwen2.5-VL-3B | Alternate 1 | 56.7% → 22.5% | 0.573 → 0.399 | 53.8% → 53.7% |
| Qwen2.5-VL-3B | Alternate 2 | 60.0% → 21.6% | 0.567 → 0.436 | 53.4% → 54.6% |
| InternVL3.5-8B | Alternate 1 | 71.7% → 32.5% | 0.592 → 0.459 | 64.9% → 64.8% |
| InternVL3.5-8B | Alternate 2 | 70.4% → 37.6% | 0.583 → 0.465 | 65.0% → 65.1% |
Forget accuracy drops by 32.8–39.2 pp on the alternate splits, compared with 42.4–44.8 pp on the official split. Retain accuracy changes by at most 1.2 pp, while Real-World accuracy drops by 1.2–3.9 pp. Thus, forgetting persists when the target identities change, although its magnitude and impact on general utility vary by split.
Q4: Why is the reduction in Forget accuracy smaller on LLaVA-1.5-7B?
Response: We compared predictions before and after editing on the same 125 Forget multiple-choice questions. LLaVA starts with lower accuracy, so fewer questions can switch from correct to incorrect. Even among the questions each Vanilla model answered correctly, VASE changes a smaller fraction to incorrect on LLaVA.
Table R17. Paired Forget VQA results on MLLMU-Bench (5% Forget).
| Backbone | Forget accuracy, Vanilla → VASE | Originally correct answers that became incorrect |
|---|---|---|
| Qwen2.5-VL-3B | 62.4% → 17.6% | 61/78 (78.2%) |
| InternVL3.5-8B | 66.4% → 24.0% | 60/83 (72.3%) |
| LLaVA-1.5-7B | 45.6% → 37.6% | 16/57 (28.1%) |
The lower starting accuracy therefore explains only part of the gap. LLaVA still shows a reduction in Forget generation ROUGE-L (0.411 → 0.251; Table 6), but the multiple-choice effect is weaker. These results establish that the effect varies by backbone under the tested settings; they do not isolate architecture from sensitivity to the editing configuration.
Response to Reviewer sWQu (1/2): Critical Weaknesses
Thank you for recognizing VASE’s forgetting–retention trade-off, small parameter updates, and ablation studies. We address each concern below and will add the key results and conclusions to the revised paper.
W1: It is unclear whether visual forgetting persists when the question wording or image changes.
Response: We evaluated the final edited models on paraphrased questions, cropped and compressed portraits, and unseen portraits of the same identities. The paraphrases preserved the original meaning without adding clues. For the image tests, we kept the questions and answer options fixed.
Table R14. Forget VQA accuracy / generation ROUGE-L (both ↓) on MLLMU-Bench (Qwen2.5-VL-3B, 5% Forget). The last row averages three unseen portraits per identity from the official Test Set.
VASE lowers both Forget metrics under paraphrasing, cropping, and compression. Transfer to unseen portraits is weaker when editing uses one portrait per identity, as in the paper. To test whether additional editing views improve transfer, we generated two poses from each original Forget portrait and edited with all three views using the same channels, losses, and optimization settings. The unseen portraits were used only for evaluation. These results show that VASE’s forgetting persists under the tested paraphrases, crops, and compression levels, and that multi-view editing improves transfer to unseen portraits. See Tables R2, R3, and R5 for the full comparisons.
W2: Producing a different answer does not establish whether the target fact is still recoverable.
Response: We tested recoverability directly. We gave the model a different fact about the same person as a clue, using two prompt phrasings. We counted semantic matches and correct guesses as recovery, and repeated both prompts with a blank image. We also scored the likelihood of the reference answer on the original image–question pairs.
Table R15. Reference-answer perplexity and target-fact recovery on MLLMU-Bench (5% Forget). Each cell shows Vanilla → VASE. In the clue tests, a fact counts as recovered if either phrasing elicits it.
These results show that VASE lowers the likelihood of the reference answer and suppresses semantic recovery through the original portrait under the tested prompts. Recovery with a blank image changes little, consistent with VASE targeting visual access to the fact. See Tables R1, R6, and R7 for the prompt-level results and Retain controls.
W3: It is unclear how much Forget-conditioned channel selection contributes beyond the editing objective.
Response: As stated in §4.4, VASE combines Forget-conditioned localization with constrained sparse editing. Table 2 shows that the editing objective contributes to both forgetting and retention. Table 3 isolates localization: Retain and Unrelated images are used only to select channels, while all sparse editing variants use the same Forget and Retain data, losses, schedule, and parameter budget during editing.
With the editor fixed, Forget-based selection lowers Forget accuracy by 4.0 and 4.8 pp relative to Retain and Unrelated selection. The gain is clearer in generation: Forget ROUGE-L is 0.369, compared with 0.485 and 0.468 for those controls, with comparable Retain accuracy. These results show that selecting channels from Forget images improves open-ended forgetting under the same objective. Q2 compares directly with a Forget–Retain score.