acceptodds
Under review as a conference paper at ICLR 2027

Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress on vision-language tasks, but they may also memorize and expose sensitive or restricted knowledge, raising concerns about privacy and broader safety risks. Machine Unlearning (MU) provides a promising way to remove targeted undesirable knowledge from trained models without retraining from scratch while preserving general model utility. Nevertheless, effective unlearning in MLLMs remains particularly challenging. Existing training-based methods often struggle to balance unlearning effectiveness and model utility. In contrast, training-free methods such as in-context unlearning preserve model utility by avoiding parameter updates, but they do not remove memorized knowledge at the parameter level and may remain vulnerable to reverse-engineering attacks. More importantly, in-context unlearning is insufficient in multimodal settings, where visual inputs can provide strong conditioning signals and induce undesirable outputs. To address these challenges, we propose **Visual-Noise Guided In-Context Distillation (VGID)**, a distillation-based framework for MLLM unlearning. VGID dynamically constructs an unlearning-oriented teacher distribution from the frozen base model through dual-modal intervention that combines visual perturbation with textual in-context unlearning. The resulting intervention-induced distribution serves as a teacher signal for distillation, guiding the student model toward parameter-level unlearning without requiring external teacher models or explicit undesirable response annotations. Experimental results show that VGID achieves strong unlearning effectiveness while preserving competitive model utility, reducing forget set ROUGE-L by 0.371 with only a 0.055 drop in retain set ROUGE-L in a representative setting.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Multimodal LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_AXbqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission proposes a distillation-based framework for MLLM unlearning, leveraging a combination of visual perturbations and textual guidance. Specifically, the framework applies multimodal in-context unlearning to a teacher MLLM to construct an unlearned response distribution, which is subsequently distilled into a student MLLM via post-training. The proposed method is evaluated on three representative open-source MLLMs, demonstrating improved unlearning effectiveness compared to baseline methods.

Critical strengths
  1. The extension of in-context unlearning to MLLMs is beneficial, which addresses a bottleneck in MLLM unlearning.
  2. The overall design of the proposed framework is well-structured and easy to follow, supported by helpful illustrations.
Critical weaknesses
  1. Unclear novelty. The core contribution of the submission appears to rely on extending existing in-context unlearning techniques through pre-defined visual perturbations and textual guidance. Rather than introducing a fundamental unlearning paradigm, the methodology reads primarily as an incremental extension. Given that the authors explicitly acknowledge inspiration from Zhong et al. (2026), the technical originality and standalone novelty seem limited. Besides, the application of visual perturbations in multimodal unlearning is not entirely novel. Prior work, such as AUVIC [1], has already demonstrated the effectiveness of adversarial visual perturbations (generated via a learned CLIP model) for precise MLLM unlearning. The authors should explicitly differentiate their perturbation mechanism from existing learning-based perturbation approaches.

  2. Insufficient rationale and generalization analysis. In the submission, visual perturbations are primarily used to construct target response distributions for subsequent distillation. However, simpler and more direct strategies exist, such as using static refusal templates (e.g., Sorry, I cannot answer this), applying stricter prompt constraints, or completely omitting visual inputs. The authors should justify why visual perturbations are necessary or superior to these straightforward baselines. Besides, as in-context unlearning inherently supports black-box models, a compelling use case would be leveraging advanced frontier models as teachers to generate high-quality unlearned target distributions. The authors should investigate or discuss whether target distributions generated by a teacher model can successfully generalize to fine-tune student models with different architectures.

  3. Incomplete experimental validation. Effective MLLM unlearning should erase the privacy information of target identity while keeping general perception intact. Following the evaluation protocol in MMUnlearner, I recommend the authors evaluate general visual perception performance on the forget set to demonstrate the robustness of the proposed unlearning framework. Besides, the experiments on model utility and copyright unlearning (Table 13 and 14) are conducted only on part of the baseline methods (i.e., GD, PO, NPO), while other baseline methods (e.g., RMU, MANU, MMUnlearner) are omitted without explanation. The authors should either provide complete comparative results across all baselines or explicitly justify their exclusion.

[1] AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models

Questions for authors
  1. How does the pre-defined visual perturbation mechanism fundamentally differ from existing perturbation-based unlearning methods?
  2. Can the target unlearned distributions generated by the teacher MLLM effectively generalize to fine-tune student models with different architectures?
Reviewer_oFHdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Visual-Noise Guided In-Context Distillation (VGID) for multimodal large language model unlearning, aiming to balance targeted forgetting with utility preservation. VGID combines visual perturbations and textual unlearning instructions to induce an unlearning-oriented teacher distribution from a frozen base model. A student model learns this distribution on the original forget-set inputs through KL-divergence minimization, while matching the base model’s outputs on retain-set inputs to preserve existing capabilities. Experiments on MLLMU-Bench across multiple model backbones and forget ratios, supplemented by knowledge-recovery attacks, general multimodal benchmarks, and copyright-unlearning evaluations, suggest a competitive forgetting–utility trade-off and improved robustness under the tested settings. However, these behavioral results do not establish that the targeted knowledge has been completely removed from the model’s parameters.

Critical strengths
  1. VGID constructs teacher distributions from a frozen base model through visual and textual interventions, avoiding external teacher models and manually annotated refusal responses.

  2. Experiments across multiple model backbones and forget ratios show effective suppression of targeted information while preserving competitive retain-set performance.

  3. The paper includes component ablations, hyperparameter analysis, several knowledge-recovery attacks, and supplementary evaluations of general multimodal utility and copyright unlearning.

Critical weaknesses
  1. VGID closely resembles prior approaches such as [1-2]. Its main distinction appears to be the addition of visual perturbations when constructing the teacher distribution. The paper should clarify this contribution and compare against on-policy distillation to establish whether its distillation strategy offers additional benefits.

  2. Retain-set degradation remains substantial in some settings, particularly for Qwen2.5-VL-3B, while forgetting performance is not consistently stronger than competing methods. These results weaken the claim that VGID achieves a superior balance between effective forgetting and utility preservation. Comparisons at matched utility or forgetting levels would provide stronger evidence.

  3. The evaluation omits relevant recent approaches, such as ASRU [3]. Including these baselines under the same experimental protocol would help establish VGID’s competitiveness and better distinguish its contribution from existing multimodal unlearning techniques.

  4. The paper does not separately report results on the celebrity subset of MLLMU-Bench, leaving its impact on real-world celebrity knowledge insufficiently assessed. The paper does not compare training time or computational costs against competing methods. Since VGID requires teacher and student forward passes during distillation, reporting runtime and peak GPU memory would help assess its practical efficiency.

[1] Attention Smoothing Is All You Need For Unlearning.

[2] DUET: Distilled LLM unlearning from an efficiently contextualized teacher

[3] ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models.

Questions for authors

See weaknesses.

Reviewer_wZRZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VGID, which combines visual perturbations with textual unlearning instructions to construct a teacher distribution from the frozen original model. The student model is then trained to match this distribution on the original inputs. Meanwhile, distribution alignment on the retain set helps preserve other capabilities. The authors evaluate the method across three models, different unlearning ratios, and multiple knowledge-recovery attacks.

Critical strengths

The method combines visual noise with textual unlearning instructions to construct an unlearning-oriented teacher distribution from the frozen original model, without requiring additional teacher training.

Critical weaknesses

Utility preservation is less convincing on Qwen2.5-VL-3B. In Table 3, retain-set classification accuracy drops from 66.46% to 52.20%, substantially below several compared baselines. The paper does not sufficiently explain why retention degrades much more on this model than on LLaVA-1.5-7B.

The experiments use a fixed random seed without reporting statistics across multiple runs, making it difficult to assess the stability of the reported improvements.

Questions for authors

In Table 3, retain-set classification accuracy on Qwen2.5-VL-3B drops from 66.46% to 52.20%. How do the authors explain this substantial performance degradation?

Reviewer_2T6VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Vgid, a distillation-based approach to unlearn user data from trained MLLMs reliably. Starting from a baseline fine-tuned on identities drawn from the MLLMU benchmark (an unlearning benchmark), unlearning is performed by jointly optimizing two losses, which are both distillation objectives. The retain loss uses the frozen baseline weights as the teacher and the current, updating model as the student. Instead, for the forget loss, the method draws inspiration from in-context unlearning, proposing the baseline weights as the teacher with perturbed inputs to elicit refusal in the teacher's output distribution. To ensure proper refusal, perturbations are dual-modal: input noise on the visual input and refusal prompting on the textual input. Optimizing this dual objective shows state-of-the-art performance on the MLLMU benchmark, with great robustness against knowledge recovery attacks.

Critical strengths
  1. The paper comprehensively compares the proposed method against existing baselines under multiple unlearning ratios, different models, and under recovery attacks. Additionally, it provides a thorough ablation study of each design choice, experimenting with multiple input perturbations.
  2. The paper is well-polished, easy to read, and generally provides a high level of detail about the design choices.
Critical weaknesses
  1. Novelty. This paper is a direct translation of DUET [1] from the language domain to the multimodal domain. This raises issues regarding the novelty, which is limited to: (i) acknowledging that prompting is insufficient in the multimodal setting to ensure proper in-context unlearning, and (ii) exploring corruption measures for the visual input, which was already partly explored in prior work [2]. Additionally, one might argue that the proposed method directly relates to Preference Optimization [3] but with soft target labels rather than hard ones. Thus, I was expecting a proper discussion about why soft labels are a better candidate in this scenario.
  2. Experimental setting. I have serious concerns about the experimental integrity of these results. Every baseline shares the same hyperparameters, i.e., lr 2e-5, batch size 4, and 1 epoch. Considering that hyperparameter tuning is especially important for MU methods [4, 5], I find that this choice leads to an unfair comparison with the proposed method. To this extent, the authors did not report the hyperparameters used for the Vgid. A quick check on existing public works seems to confirm that baselines are likely under-tuned [6]. A better experimental setting would have used a held-out set for estimating the unlearning hyperparameters, with a shared computational budget for every method, guaranteeing fairness. Finally, experiments on just one benchmark are not enough for demonstrating that Vgid is better than existing methods.

[1] Zhong, Yisheng, Zhengbang Yang, and Zhuangdi Zhu. "DUET: Distilled LLM unlearning from an efficiently contextualized teacher." ICLR 2026.
[2] Caldarella, Simone, et al. "The phantom menace: unmasking privacy leakages in vision-language models." ECCVW 2024.
[3] Maini, Pratyush, et al. "Tofu: A task of fictitious unlearning for llms." arXiv preprint arXiv:2401.06121 (2024).
[4] Wang, Qizhou, et al. "Towards effective evaluations and comparisons for llm unlearning methods." ICLR 2025.
[5] Yao, Jin, et al. "Machine unlearning of pre-trained large language models." ACL 2024.
[6] Liu, Zheyuan, et al. "Modality-aware neuron pruning for unlearning in multimodal large language models." ACL 2025.

Questions for authors
  1. Could the authors compare their method against carefully tuned baselines under a shared unlearning budget?
  2. Could the authors extend the results to an additional benchmark?
  3. Could the authors explain the advantages of using soft instead of hard labels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.