acceptodds
Under review as a conference paper at ICLR 2027

Visual Activation-Guided Sparse Editing for Multimodal Large Language Model Unlearning

13 views

Abstract

Multimodal large language models (MLLMs) may memorize private, copyrighted, or unsafe visual information, making selective unlearning essential for their responsible deployment. Existing methods either update broad parameter regions or remove target-related neurons, which can damage representations shared with retained knowledge. To address this challenge, we propose Visual Activation-Guided Sparse Editing (VASE), a novel method for selective visual unlearning in MLLMs. VASE locates a small set of visual feed-forward network channels using normalized activation energy on forget images and learns sparse residual updates only for these channels. A bounded forgetting objective prevents excessive suppression, while output- and activation-level constraints preserve retained behavior. The learned updates are merged into the visual backbone without additional inference overhead. Experiments across multiple benchmarks and MLLM backbones show that VASE achieves state-of-the-art forgetting while preserving retained knowledge and general multimodal capabilities. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Machine Unlearning, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sWQuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VASE, a sparse editing method for selective visual unlearning in multimodal language models. It identifies visual FFN channels using normalized activations on forget samples and restricts parameter updates to the corresponding projection rows. The training objective combines bounded target-answer suppression with output-level and activation-level retention losses.

The authors evaluate VASE on MLLMU-Bench and CLEAR, primarily using Qwen2.5-VL-3B and InternVL3.5-8B. Additional results cover two other backbones and different forget ratios. The method achieves substantial reductions in Forget VQA accuracy while largely preserving retain-set and general performance.

Critical strengths

The main experimental results are strong. VASE achieves the lowest Forget VQA accuracy among the compared methods across the two main backbones and benchmarks. The improvements are substantial, especially considering that retain-set performance remains close to the original models. Although VASE does not outperform every baseline on every forgetting metric, its overall forgetting–retention trade-off is encouraging.

The method is simple and requires updating only a small fraction of the model parameters. The ablation study is reasonably thorough, including random, retain-based, and unrelated-image channel selection, matched Visual LoRA, and different forgetting and retention objectives. I also appreciate the analysis of selected-channel interventions on held-out examples.

Critical weaknesses
  1. The robustness of the unlearned behavior is not sufficiently evaluated.

The main evaluation measures forgetting through VQA accuracy and ROUGE-L. These results show that VASE successfully suppresses target information under the tested queries. However, they do not establish whether the information can still be recovered using alternative prompts or image inputs.

The examples in Figure 5 illustrate this distinction. VASE often produces an alternative answer rather than the original target answer. This is consistent with the paper's objective of suppressing visual access to target knowledge, and is not necessarily a failure. Nevertheless, it would be useful to test whether the original information remains recoverable through paraphrased questions, image transformations, or multimodal extraction attacks.

  1. The specificity of the activation-based channel selection could be better established.

Equations (3)–(6) select channels based on their normalized activation energy on forget samples, rather than explicitly measuring their selectivity for forget versus retain inputs.

The comparisons in Table 3 are useful. In particular, VASE substantially outperforms random channel selection. However, selecting channels using retain or unrelated images also produces considerable forgetting. On Qwen2.5-VL-3B, these alternatives reach 21.6% and 22.4% Forget VQA accuracy, compared with 17.6% for VASE.

VASE has a clearer advantage in generative forgetting, but the classification results suggest that selecting generally influential visual channels may already account for a substantial part of the improvement. A comparison with a contrastive Forget–Retain activation criterion would help clarify the role of forget-specific localization.

Questions for authors
  1. Have the authors tested the edited models against alternative prompts, image transformations, or extraction attacks? Such experiments would help distinguish suppression on the benchmark queries from more robust visual unlearning.

  2. Have the authors considered ranking channels using a contrastive score based on both forget and retain activations? How would this compare with the current forget-only criterion?

  3. How stable are the results across different forget-identity splits or independent editing runs? The additional results on LLaVA-1.5-7B show a smaller reduction in Forget VQA accuracy than those on the main backbones. It would be helpful to understand whether this reflects architectural differences or sensitivity to the experimental setting.

2 comments
AuthorsOpenReview ↗

Response to Reviewer sWQu (1/2): Critical Weaknesses

Thank you for recognizing VASE’s forgetting–retention trade-off, small parameter updates, and ablation studies. We address each concern below and will add the key results and conclusions to the revised paper.

W1: It is unclear whether visual forgetting persists when the question wording or image changes.

Response: We evaluated the final edited models on paraphrased questions, cropped and compressed portraits, and unseen portraits of the same identities. The paraphrases preserved the original meaning without adding clues. For the image tests, we kept the questions and answer options fixed.

Table R14. Forget VQA accuracy / generation ROUGE-L (both ↓) on MLLMU-Bench (Qwen2.5-VL-3B, 5% Forget). The last row averages three unseen portraits per identity from the official Test Set.

Evaluation conditionVanillaVASE
Two paraphrases per question57.2% / 0.49623.2% / 0.329
Center crop, 90% of each side58.4% / 0.62932.8% / 0.510
Center crop, 75% of each side65.6% / 0.62836.8% / 0.518
JPEG quality 7560.0% / 0.62324.0% / 0.479
JPEG quality 3560.8% / 0.62226.4% / 0.473
Unseen portraits57.9% / 0.60453.6% / 0.585 (single-view); 18.7% / 0.450 (multi-view)

VASE lowers both Forget metrics under paraphrasing, cropping, and compression. Transfer to unseen portraits is weaker when editing uses one portrait per identity, as in the paper. To test whether additional editing views improve transfer, we generated two poses from each original Forget portrait and edited with all three views using the same channels, losses, and optimization settings. The unseen portraits were used only for evaluation. These results show that VASE’s forgetting persists under the tested paraphrases, crops, and compression levels, and that multi-view editing improves transfer to unseen portraits. See Tables R2, R3, and R5 for the full comparisons.

W2: Producing a different answer does not establish whether the target fact is still recoverable.

Response: We tested recoverability directly. We gave the model a different fact about the same person as a clue, using two prompt phrasings. We counted semantic matches and correct guesses as recovery, and repeated both prompts with a blank image. We also scored the likelihood of the reference answer on the original image–question pairs.

Table R15. Reference-answer perplexity and target-fact recovery on MLLMU-Bench (5% Forget). Each cell shows Vanilla → VASE. In the clue tests, a fact counts as recovered if either phrasing elicits it.

BackboneForget reference-answer perplexity ↑Recovery with portrait + clue ↓Recovery with blank image + clue
Qwen2.5-VL-3B3.04 → 13.3517/48 → 0/488/48 → 8/48
InternVL3.5-8B3.26 → 16.0021/48 → 0/4812/48 → 13/48

These results show that VASE lowers the likelihood of the reference answer and suppresses semantic recovery through the original portrait under the tested prompts. Recovery with a blank image changes little, consistent with VASE targeting visual access to the fact. See Tables R1, R6, and R7 for the prompt-level results and Retain controls.

W3: It is unclear how much Forget-conditioned channel selection contributes beyond the editing objective.

Response: As stated in §4.4, VASE combines Forget-conditioned localization with constrained sparse editing. Table 2 shows that the editing objective contributes to both forgetting and retention. Table 3 isolates localization: Retain and Unrelated images are used only to select channels, while all sparse editing variants use the same Forget and Retain data, losses, schedule, and parameter budget during editing.

With the editor fixed, Forget-based selection lowers Forget accuracy by 4.0 and 4.8 pp relative to Retain and Unrelated selection. The gain is clearer in generation: Forget ROUGE-L is 0.369, compared with 0.485 and 0.468 for those controls, with comparable Retain accuracy. These results show that selecting channels from Forget images improves open-ended forgetting under the same objective. Q2 compares directly with a Forget–Retain score.

AuthorsOpenReview ↗

Response to Reviewer sWQu (2/2): Questions for Authors

Q1: Have the edited models been tested under alternative prompts, image transformations, or extraction attacks?

Response: Yes. W1 and W2 report tests on the final edited models: two paraphrases per question, two crop levels, two JPEG quality levels, unseen portraits, and two clue-based extraction prompts with blank-image controls. The results support forgetting under the tested paraphrases, crops, and compression levels. They also identify a limit of single-portrait editing: forgetting is weaker when the portrait is replaced. Using multiple views during editing improves transfer.

Q2: How does a contrastive Forget–Retain ranking compare with Forget-only ranking?

Response: We compared ranking by AfA_f with ranking by Af−ArA_f-A_r, computing both energies with the same statistic and holding the editor, losses, parameter budget, and schedule fixed. On the official Qwen split, Forget-only ranking lowers mean Forget ROUGE-L from 0.413 to 0.263 across three editing seeds (paired Δ −0.150, 95% CI [−0.216, −0.093], resampling identities after averaging over the seeds). The Forget accuracy difference is −1.1 pp (95% CI [−4.8, +2.7] pp), and the interval includes zero.

Forget-only ranking also lowers both Forget metrics in all four alternate model–split comparisons, with Retain accuracy at most 1.7 pp lower. These results favor Forget-only ranking for open-ended forgetting in our setting. Table R12 in our response to Reviewer 1u4p gives the full comparison.

Q3: How stable are the results across independent edits and different Forget identities?

Response: On the official Qwen split, VASE obtains Forget VQA accuracies of 18.4%, 15.2%, and 19.2% across three editing seeds (17.6% ± 2.1 pp), with Retain accuracy of 53.8% ± 0.6 pp. Forget generation ROUGE-L varies more across runs (0.263 ± 0.146; Table R10).

We also sampled two disjoint groups of 25 identities from the official Retain set. In each alternate split, the sampled identities became the Forget set and the other 475 formed the Retain set. We kept the editing settings fixed within each backbone.

Table R16. VASE on alternate MLLMU-Bench splits (5% Forget). Each cell shows Vanilla → VASE.

BackboneForget splitForget VQA accuracy ↓Forget generation ROUGE-L ↓Retain VQA accuracy ↑
Qwen2.5-VL-3BAlternate 156.7% → 22.5%0.573 → 0.39953.8% → 53.7%
Qwen2.5-VL-3BAlternate 260.0% → 21.6%0.567 → 0.43653.4% → 54.6%
InternVL3.5-8BAlternate 171.7% → 32.5%0.592 → 0.45964.9% → 64.8%
InternVL3.5-8BAlternate 270.4% → 37.6%0.583 → 0.46565.0% → 65.1%

Forget accuracy drops by 32.8–39.2 pp on the alternate splits, compared with 42.4–44.8 pp on the official split. Retain accuracy changes by at most 1.2 pp, while Real-World accuracy drops by 1.2–3.9 pp. Thus, forgetting persists when the target identities change, although its magnitude and impact on general utility vary by split.

Q4: Why is the reduction in Forget accuracy smaller on LLaVA-1.5-7B?

Response: We compared predictions before and after editing on the same 125 Forget multiple-choice questions. LLaVA starts with lower accuracy, so fewer questions can switch from correct to incorrect. Even among the questions each Vanilla model answered correctly, VASE changes a smaller fraction to incorrect on LLaVA.

Table R17. Paired Forget VQA results on MLLMU-Bench (5% Forget).

BackboneForget accuracy, Vanilla → VASEOriginally correct answers that became incorrect
Qwen2.5-VL-3B62.4% → 17.6%61/78 (78.2%)
InternVL3.5-8B66.4% → 24.0%60/83 (72.3%)
LLaVA-1.5-7B45.6% → 37.6%16/57 (28.1%)

The lower starting accuracy therefore explains only part of the gap. LLaVA still shows a reduction in Forget generation ROUGE-L (0.411 → 0.251; Table 6), but the multiple-choice effect is weaker. These results establish that the effect varies by backbone under the tested settings; they do not isolate architecture from sensitivity to the editing configuration.

Reviewer_1u4pOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VASE, a sparse visual unlearning method for multimodal large language models. It identifies visual FFN channels based on their normalized activation energy on forget samples and learns residual updates restricted to the selected channels.

Critical strengths
  • The paper addresses an important problem in multimodal unlearning.
  • The experimental results are encouraging. VASE achieves good improvements in Forget performance across two datasets and two main backbones, while keeping Retain performance close to the original models.
  • The ablation study is relatively thorough.
Critical weaknesses
  • The evaluation does not fully establish whether the target knowledge has actually been unlearned. The main forgetting metrics are multiple-choice accuracy and ROUGE-L, both of which measure whether the model produces the original target answer. A lower score does not necessarily mean the underlying association has been removed. In fact, Figure 5 shows several cases where VASE replaces correct information with specific but incorrect attributes, such as changing someone's profession or place of origin. I find this concerning for a privacy-oriented unlearning method.

  • The forget-specificity of the proposed channel localization is not fully convincing. The channel selection score in Eq. (4) only considers activation energy on forget samples, although the motivation is based on differences between Forget and Retain activations. Table 3 shows that selecting channels using Retain or unrelated images also produces relatively strong forgetting. Forget-selected channels achieve 17.6% Forget accuracy, compared with 21.6% and 22.4% for Retain-selected and Unrelated-selected channels. These results suggest that the optimization objective may account for a substantial portion of the improvement, even when the selected channels are not specifically associated with the forget set.

  • The advantage over simpler parameter-efficient editing deserves further analysis. The parameter-matched Visual LoRA baseline in Table 3 achieves 19.2% Forget accuracy, compared with 17.6% for VASE, with nearly identical Retain accuracy. VASE does achieve a larger reduction in Forget ROUGE-L, which is encouraging, but the paper does not report uncertainty for these comparisons.

Questions for authors
  • Why wasn't Factuality Score being used as one of the evaluation metrics from MLLMU-Bench?
  • Why is Forget-only activation energy preferred over a score that explicitly compares Forget and Retain activations? Given the relatively strong results from Retain-selected and Unrelated-selected channels in Table 3, how much of the improvement can be attributed specifically to forget-aware localization?
  • Could the authors clarify the computational trade-off between sparse editing and the additional localization/optimization cost?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1u4p (1/3): Critical Weaknesses (W1–W2)

Thank you for recognizing the importance of the problem, the forgetting and retention results, and the ablations. We address each concern below and will add the key results and conclusions to the revised paper.

W1: Lower multiple-choice accuracy and ROUGE-L do not establish whether the edited model can still recover the target fact.

Response: We agree that these metrics alone cannot distinguish a changed output from a lower probability of the correct fact. We added two tests of the final edited models: the likelihood of the reference answer and semantic recovery under extraction prompts.

First, we teacher-forced the reference answer and measured perplexity on answer tokens only. For a fixed answer, perplexity is the reciprocal of the geometric-mean token probability in Eq. (11). The table pools answer tokens within each split. We include the original instruct checkpoint before benchmark fine-tuning as a reference.

Table R6. Reference-answer perplexity on MLLMU-Bench (5% Forget; 50 original image–question pairs per split). Higher Forget perplexity means lower reference-answer likelihood. VASE denotes the reruns used for the recovery experiments.

BackboneSplitBefore benchmark fine-tuningVanillaVASE
Qwen2.5-VL-3BForget8.833.0413.35
Qwen2.5-VL-3BRetain8.492.963.06
InternVL3.5-8BForget13.003.2616.00
InternVL3.5-8BRetain12.793.053.17

Second, we checked whether open-ended answers disclosed the target fact, including paraphrases and correct guesses. We also provided a different fact about the same person as a clue, with two prompt phrasings and matched blank-image controls. A question counts as recovered if either phrasing reveals the fact.

Table R7. Semantic recovery on MLLMU-Bench (5% Forget). The clue experiment uses 48 eligible questions from 24 identities.

BackboneVASE, portrait without clueVanilla → VASE, portrait with clueVanilla → VASE, blank image with clue
Qwen2.5-VL-3B0/5017/48 → 0/488/48 → 8/48
InternVL3.5-8B0/5021/48 → 0/4812/48 → 13/48

These results show that VASE lowers the probability of the reference answer and suppresses semantic recovery through the original portrait under the tested prompts. Retain perplexity changes little, and a text clue can still elicit some facts without the portrait. We will clarify the scope of Eq. (2) in the revision. Table R3 in our reply to 4GXF separately reports weak transfer to new photos for single-view editing and the gains from multi-view editing.

W2: The edited model can give specific but incorrect personal attributes (Fig. 5).

Response: We agree that abstaining is preferable to giving an incorrect personal attribute. Following prior work [1, 2, 3], we evaluate forgetting by comparing responses with reference answers, without requiring abstention. Eq. (13) suppresses the target answer up to a margin and leaves the replacement response unspecified. As shown in W1, VASE lowers the likelihood of the reference answer, and neither backbone recovered a target fact when given the original portrait and an additional clue. This provides evidence of reduced visual access beyond a change in the generated answer.

To test whether VASE can explicitly abstain, we added a cross-entropy loss targeting “I don’t know” (IDK) on open-ended Forget prompts, keeping channel selection and the optimization schedule fixed. We call this variant VASE+IDK.

Table R8. Qwen2.5-VL-3B on MLLMU-Bench (5% Forget). VASE accuracies are from Table 1. Abstentions are counted on the same 50 open-ended Forget questions for both models.

MethodOpen-ended abstentions ↑Forget multiple-choice accuracy ↓Retain VQA accuracy ↑Real-World VQA accuracy ↑
VASE (paper)4/5017.6%53.7%69.7%
VASE+IDK45/5027.2%54.2%67.5%

Adding the IDK loss increases abstentions from 4/50 to 45/50. Neither variant reveals a target fact in these open-ended responses. Explicit refusal training can change reference-based scores by changing how the model responds. We therefore report VASE+IDK separately and keep the original objective in Table 1, where the goal is to compare visual forgetting with prior methods under the standard evaluation protocol.

References

[1] Liu et al. Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench. NAACL, 2025.

[2] Liu et al. Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models. ACL, 2025.

[3] Huo et al. MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models. Findings of ACL, 2025.

AuthorsOpenReview ↗

Response to Reviewer 1u4p (2/3): Critical Weaknesses (W3–W4)

W3: It is unclear how much Forget-conditioned channel localization contributes to VASE’s forgetting gains.

Response: As stated in the Introduction and §4.4, VASE combines two contributions: Forget-conditioned localization determines where to edit, and constrained sparse optimization determines how to suppress target answers while preserving Retain behavior. Table 2 shows the contribution of the optimization objective: replacing our forgetting loss with gradient ascent raises Forget accuracy from 17.6% to 31.2% on Qwen2.5-VL-3B and from 24.0% to 36.0% on InternVL3.5-8B.

Table 3 isolates the contribution of localization. The sparse editing variants use the same training data, objective, schedule, and parameter budget; only the channel selection rule changes. Forget-based selection lowers Forget accuracy by 4.0 and 4.8 percentage points relative to Retain and Unrelated selection. The gain is clearer in open-ended generation: Forget ROUGE-L is 0.369, compared with 0.485 for Retain, 0.468 for Unrelated, and 0.538 for Random, while Retain accuracy remains comparable. These results show that selecting channels using Forget images improves open-ended forgetting even when the optimization is held fixed.

W4: The advantage over parameter-matched Visual LoRA is unclear given the small Forget accuracy gap and the lack of uncertainty estimates.

Response: We repeated the comparison on Qwen2.5-VL-3B with three editing seeds. Both methods used the same starting checkpoint, data, objective, and optimization schedule, with matched batch order within each seed. We also measured how many original visual weights each method changed after merging.

Table R10. VASE vs. parameter-matched Visual LoRA on MLLMU-Bench (5% Forget). Performance is reported as mean ± SD across three editing seeds. The paired 95% CIs resample identities after averaging over those seeds. Changed-weight counts are averaged across the three runs.

MetricVASEVisual LoRAVASE − LoRA [95% CI]
Forget generation ROUGE-L ↓0.263 ± 0.1460.422 ± 0.070−0.159 [−0.221, −0.100]
Forget VQA accuracy ↓17.6% ± 2.1 pp22.1% ± 1.2 pp−4.5 pp [−10.7, +1.3]
Retain generation ROUGE-L ↑0.555 ± 0.0040.559 ± 0.003−0.0045 [−0.0087, −0.0003]
Visual weights changed after merging ↓0.653M243.9M—

The clearest advantage is in open-ended generation: VASE lowers average Forget ROUGE-L by 0.159, with a paired interval below zero for these runs, although Visual LoRA performs better on one seed. The Forget accuracy interval includes zero, and Retain generation ROUGE-L is slightly lower for VASE.

At a matched trainable parameter budget, VASE changes only about 1/373 as many visual weights after merging. We count BF16 entries in the visual FFN input projections that differ from the common pre-edit checkpoint. VASE confines updates to selected rows, while merging a low-rank update can change weights throughout the projection matrices.

These results show that VASE achieves stronger average open-ended forgetting while changing far fewer original visual weights. We will add the multi-seed results and changed-weight counts to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 1u4p (3/3): Questions for Authors

Q1: Why did the paper omit MLLMU-Bench’s Factuality Score?

Response: Following prior work, we use multiple-choice accuracy and open-ended ROUGE-L as our main metrics. Both can be applied to all methods without an external judge. To assess factual agreement beyond text overlap, we also scored the open-ended answers with Qwen2-7B-Instruct on a 1–10 scale. The judge saw only the question, reference answer, and model response, and passed 12 checks covering correct answers, paraphrases, factual errors, and nonanswers. This evaluation uses a different judge from the official benchmark.

Table R11. Local factuality on MLLMU-Bench (5% Forget). Δ is VASE minus Vanilla, with paired 95% CIs over identities. InternVL VASE uses a separate run of the Table 1 configuration.

BackboneSplitVanillaVASEΔ [95% CI]
Qwen2.5-VL-3BForget (50) ↓5.442.56−2.88 [−3.96, −1.84]
Qwen2.5-VL-3BRetain (950) ↑5.265.14−0.12 [−0.241, +0.001]
InternVL3.5-8BForget (50) ↓5.262.76−2.50 [−3.56, −1.54]
InternVL3.5-8BRetain (950) ↑5.405.39−0.01 [−0.13, +0.11]

These results show that VASE reduces agreement with target facts on both backbones, while Retain scores change little. We will add this evaluation and specify the judge and rubric in the revision.

Q2: Why rank channels by Forget activation alone, and what does this choice add beyond the editing objective?

Response: Fig. 1 shows that ranking by Forget energy already captures part of the positive Forget–Retain activation gap. Subtracting Retain energy can downrank channels used by both sets, even when the editing objective can preserve their Retain behavior. To test whether subtraction helps, we compared ranking by AfA_f with ranking by Af−ArA_f-A_r, holding the editing data, losses, parameter budget, and training schedule fixed within each split.

To check whether the result depends on which identities are forgotten, we also created two alternate 5% Forget splits. We sampled two disjoint groups of 25 identities from the official Retain set. For each alternate split, the sampled group became the Forget set and the other 475 identities formed the Retain set.

Table R12. Forget-only / Forget–Retain ranking on MLLMU-Bench (5% Forget). The official Qwen row reports mean ± SD across three matched editing seeds. Each alternate row reports one run per ranking.

BackboneForget splitForget VQA accuracy ↓Forget generation ROUGE-L ↓Retain VQA accuracy ↑
Qwen2.5-VL-3BOfficial17.6% ± 2.1 pp / 18.7% ± 3.2 pp0.263 ± 0.146 / 0.413 ± 0.02653.8% ± 0.6 pp / 53.7% ± 0.5 pp
Qwen2.5-VL-3BAlternate 122.5% / 24.2%0.399 / 0.49353.7% / 54.5%
Qwen2.5-VL-3BAlternate 221.6% / 38.4%0.436 / 0.48554.6% / 56.3%
InternVL3.5-8BAlternate 132.5% / 36.7%0.459 / 0.46964.8% / 65.2%
InternVL3.5-8BAlternate 237.6% / 42.4%0.465 / 0.49165.1% / 65.3%

The clearest benefit is in open-ended generation. On the official Qwen split, Forget-only ranking lowers mean Forget ROUGE-L by 0.150 (paired 95% CI [−0.216, −0.093], resampling identities after averaging over the three seeds). The Forget accuracy difference is smaller, and its interval includes zero (−1.1 pp; 95% CI [−4.8, +2.7] pp). Forget-only ranking also lowers both Forget metrics in all four alternate comparisons, with Retain accuracy at most 1.7 pp lower. Together with W3, these results show that channel selection contributes under the same editing objective and support using Forget-only ranking for open-ended forgetting in our setting.

Q3: What is the computational trade-off of localization and sparse optimization?

Response: Localization adds little time; optimization accounts for almost all of the cost. We timed both stages on an RTX A6000 without a cached locator, excluding model loading, checkpoint saving, and evaluation.

Table R13. VASE cost on MLLMU-Bench (5% Forget). Extra peak memory is measured relative to the loaded model. These timings are from new runs on an RTX A6000, separate from the earlier RTX 4090 timing reported in Table 4.

BackboneEdited parametersLocalizationOptimizationTotalExtra peak memory
Qwen2.5-VL-3B0.655M (0.017%)27.9 s129.4 min129.9 min7.8 GiB
InternVL3.5-8B0.524M (0.006%)41.4 s235.6 min236.3 min13.0 GiB

Localization uses forward passes and takes less than 0.4% of total unlearning time on both backbones. During optimization, gradients still pass through downstream layers, so updating fewer weights does not yield a proportional speedup. Once merged, the edit adds no inference operations. Table 4 of the paper compares VASE with MANU and MMUnlearner in parameter footprint, unlearning time, and forgetting effectiveness.

Reviewer_4GXFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VASE for selective visual unlearning in multimodal large language models. It uses activation statistics from forget images to select visual channels and learns sparse updates to their weights. A forgetting objective suppresses target answers, while preservation losses protect retained knowledge. The updates can be merged into the original model without adding inference modules. Experiments on MLLMU-Bench and CLEAR show a favorable balance between forgetting and knowledge retention.

Critical strengths
  • A promising balance between visual forgetting and retention. The main experiments show substantial suppression of target visual answers with generally small changes in retained visual performance. Although some utility measures decline, the overall results support the method’s practical value within the stated visual unlearning setting.

  • A clear design supported by useful controls. The method combines targeted channel editing with output and activation preservation. The ablation studies support the contribution of these components. Comparisons with alternative channel selections and a matched Visual LoRA baseline also help assess the design. These experiments make the proposed approach more convincing.

Critical weaknesses
  • Limited evidence of robust visual unlearning. Some qualitative examples replace the original facts with incorrect ones. This shows that the answers have changed, but does not establish whether the target information remains recoverable through different prompts, additional hints, or modified images. The localization probe uses unseen images and questions, but does not directly test the robustness of the final edited model. Targeted extraction tests would strengthen the evidence for reliable forgetting. I accept the visual-only unlearning setting and do not consider the preservation of text-only knowledge a weakness.
Questions for authors
  • Clarification on baseline selection. Why was the closely related null-space-constrained visual forgetting method not included in the experiments? Are there differences in the task or evaluation setup that prevent a direct comparison?

  • Evidence of reliable forgetting. Have you tested whether the target information remains suppressed when the questions are rephrased? Such results would help clarify whether the forgetting extends beyond the standard evaluation prompts.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4GXF (1/2): Critical Weaknesses

Thank you for recognizing VASE’s balance between visual forgetting and retention and the value of our ablations and controls. We address each concern below and will add the key experimental results and conclusions to the revised paper.

W1: The edited model may still recover a target fact when given an additional clue.

Response: We gave the model a different fact about the same person. One prompt presented it as background; the other asked the model to use the portrait and clue to identify the person. We repeated both with a blank image. Two blinded language-model judges counted equivalent wording and correct guesses as recovery and agreed on 767 of 768 responses.

Table R1. Target-fact recovery on MLLMU-Bench (5% Forget; 48 questions from 24 identities). Each cell shows Vanilla → VASE.

BackbonePortrait + background cluePortrait + direct clueBlank + background clueBlank + direct clue
Qwen2.5-VL-3B31.3% → 0.0%35.4% → 0.0%14.6% → 14.6%16.7% → 16.7%
InternVL3.5-8B41.7% → 0.0%41.7% → 0.0%14.6% → 16.7%22.9% → 22.9%

The portrait increased recovery for Vanilla. With VASE, neither prompt recovered a target fact with the portrait, while recovery with a blank image stayed close to Vanilla. These results show that VASE suppresses access to target facts through the original portrait even when the model is given a clue.

W2: It is unclear whether visual forgetting holds when the original image is cropped or compressed.

Response: To test whether visual forgetting persists after image transformations, we ran additional experiments with two center crops and two JPEG quality levels. We compare our VASE method with CVF+NCU [1]. We kept the questions, options, and per-question decoding seeds fixed across methods and image conditions.

Table R2. Forget VQA accuracy / generation ROUGE-L (both ↓) under center crops and JPEG compression of MLLMU-Bench Forget portraits (Qwen2.5-VL-3B, 5% Forget). Original uses the Table 1 protocol. Δ is edited accuracy minus Vanilla in the same row (pp).

Image conditionVanillaCVF+NCUVASEΔ Forget VQA vs. Vanilla (pp), CVF+NCU / VASE
Original62.4% / 0.63151.2% / 0.54317.6% / 0.369−11.2 / −44.8
Center crop, 90% of each side58.4% / 0.62960.0% / 0.62332.8% / 0.510+1.6 / −25.6
Center crop, 75% of each side65.6% / 0.62864.0% / 0.63636.8% / 0.518−1.6 / −28.8
JPEG quality 7560.0% / 0.62356.8% / 0.57024.0% / 0.479−3.2 / −36.0
JPEG quality 3560.8% / 0.62255.2% / 0.57726.4% / 0.473−5.6 / −34.4

Perturbations weakened VASE’s forgetting, but it still lowered Forget accuracy by at least 25.6 pp versus Vanilla and reduced ROUGE-L in every condition. CVF+NCU was close to Vanilla after cropping and showed modest suppression after JPEG compression. VASE’s accuracy was 27.2 pp below CVF+NCU’s at both crop levels. These results show that VASE remains effective after cropping or compression.

W3: It is unclear whether visual forgetting transfers to unseen photos of the same person.

Response: The paper uses one portrait per Forget identity (Single-view VASE). To test whether more editing views improve transfer, we added Multi-view VASE. Using Arc2Face, as in MLLMU-Bench, we generated two poses from each original Forget portrait. We edited with all three views using the same localized channels, loss, hyperparameters, and number of steps. Held-out photos were used only for evaluation.

Table R3. Forget VQA accuracy / generation ROUGE-L (both ↓) on Qwen2.5-VL-3B. We pair the original questions for the 25 MLLMU-Bench Forget identities (5% split) with their three held-out portraits from the official Test Set.

Held-out photoVanillaCVF+NCUSingle-view VASEMulti-view VASE
View 156.0% / 0.59758.4% / 0.60350.4% / 0.59317.6% / 0.437
View 260.8% / 0.61361.6% / 0.62956.0% / 0.60020.0% / 0.455
View 356.8% / 0.60257.6% / 0.59754.4% / 0.56218.4% / 0.459
Mean57.9% / 0.60459.2% / 0.61053.6% / 0.58518.7% / 0.450

Single-view VASE’s forgetting weakened on unseen photos, but it still scored lower than CVF+NCU on both metrics for all three views. Adding two editing views lowered mean Forget accuracy from 53.6% to 18.7% and ROUGE-L from 0.585 to 0.450. These results show that the same editing procedure can maintain strong forgetting on unseen photos when given multiple views of the same person.


References

[1] Wang, Yuhang, et al. "Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearning." arXiv preprint arXiv:2605.05909 (2026).

AuthorsOpenReview ↗

Response to Reviewer 4GXF (2/2): Questions for Authors

Q1: Why was CVF+NCU omitted, and do differences in the task or evaluation prevent a direct comparison?

Response: Thank you for pointing out this omission. We agree that CVF+NCU is a relevant baseline and should have been included. We faithfully reproduced CVF+NCU as described in [1]. We evaluated it on our two main backbones, Qwen2.5-VL-3B and InternVL3.5-8B, under the same experimental protocol as VASE.

Table R4. Matched comparison on MLLMU-Bench and CLEAR (5% Forget). Scores are CVF+NCU / VASE.

BenchmarkBackboneForget VQA accuracy ↓Retain VQA accuracy ↑Real-World VQA accuracy ↑
MLLMU-BenchQwen2.5-VL-3B51.2% / 17.6%53.6% / 53.7%69.1% / 69.7%
MLLMU-BenchInternVL3.5-8B56.8% / 24.0%66.5% / 65.1%70.0% / 70.1%
CLEARQwen2.5-VL-3B22.9% / 32.4%65.1% / 64.8%58.0% / 57.8%
CLEARInternVL3.5-8B42.6% / 36.7%66.6% / 67.3%68.1% / 68.1%

VASE achieves lower Forget VQA accuracy in three of four settings. CVF+NCU achieves lower Forget VQA accuracy on Qwen2.5-VL-3B with CLEAR. Retain VQA accuracy differs by at most 1.4 percentage points, and Real-World VQA accuracy is similar across methods. We will add the full comparison, including all metrics, to Table 1.

Q2: Does visual forgetting hold when the questions are rephrased?

Response: Yes. VASE continues to suppress target answers when the questions are rephrased. To test whether forgetting persists when the wording changes, we evaluated two paraphrases per question on the Forget and Retain sets. The paraphrases preserved the original meaning and added no clues. We used the Qwen2.5-VL-3B checkpoints and per-question seeding protocol from W2 and kept the images, answer keys, and multiple-choice options fixed. We also measured how often either paraphrase elicited the correct choice.

Table R5. Forgetting and retention under paraphrasing on MLLMU-Bench (Qwen2.5-VL-3B, 5% Forget). Each split contains 125 multiple-choice and 50 open-ended questions from 25 identities before paraphrasing.

Metric under paraphrasingVanillaVASE
Forget VQA accuracy ↓57.2%23.2%
Forget questions answered correctly under either paraphrase ↓66.4%26.4%
Forget generation ROUGE-L ↓0.4960.329
Retain VQA accuracy ↑57.6%58.0%
Retain generation ROUGE-L ↑0.5110.510

As shown in Table R5, even when either paraphrase could elicit the correct answer, VASE reduced Forget accuracy by 40.0 percentage points. It also lowered Forget generation ROUGE-L, while both Retain metrics stayed close to Vanilla. These results show that VASE’s forgetting persists beyond the original question wording in both multiple-choice and open-ended evaluations.

Thank authors for the response. They basically addressed my concerns, so I will maintain my positive score.

Thank you for your positive assessment and helpful suggestions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses

Thank you for your careful review and constructive feedback. Below, we address three recurring questions raised by the reviewers.

  1. Can the edited model still recover the target facts? VASE lowered the likelihood of the reference answers. With the original portrait and the two tested clue prompts, target-fact recovery fell to 0/48 on both backbones, while recovery with a blank image changed little (Tables R6–R7).

  2. Does forgetting hold when the question or image changes? VASE continued to suppress target answers under the tested paraphrases, crops, and JPEG compression. Single-view editing transferred poorly to unseen portraits of the same identities. Adding two editing views lowered Forget VQA accuracy on those held-out portraits from 53.6% to 18.7% (Tables R2, R3, and R5).

  3. What does Forget-based channel selection add? With the editing objective and parameter budget held fixed, Forget-only ranking reduced mean Forget ROUGE-L from 0.413 to 0.263 compared with Forget–Retain ranking across three Qwen editing runs. The difference was clearest in open-ended generation; the confidence interval for the VQA accuracy difference included zero (Table R12).

The individual replies below provide the full results. We have incorporated these important findings and their limitations into the revised manuscript, with the changes highlighted in blue. We have also corrected the outdated title in the manuscript to match the title of our original OpenReview submission. We are happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.