acceptodds
Withdrawn from ICLR 2027

Rethinking Supervision Targets for Multimodal LLM Unlearning: Less Private Information, Reduced Distribution Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) can memorize and reproduce sensitive information from their training data, posing significant privacy risks. Their ability to encode knowledge across both textual and visual modalities further complicates the selective removal of such private information. Many existing methods guide model unlearning by training the model toward constructed labels or output distributions, which we refer to as supervision targets. However, these methods often suffer from a trade-off between forgetting effectiveness and model utility. We identify supervision target limitations as a key factor in this trade-off and empirically show that targets containing more private information lead to insufficient forgetting, while larger distribution shifts from the original output distribution are associated with greater utility degradation. Motivated by these findings, we propose FFN-Perturbed Self-Distillation (FPD), which starts from the teacher’s original output distribution as a zero-shift reference and selectively perturbs FFN activations associated with private information across textual and visual modalities. By precisely reducing private information from the teacher’s original output distribution, FPD constructs supervision targets with less private information and reduced distribution shift. Extensive experiments on S-MLLMUn Bench demonstrate that FPD achieves a better forgetting-utility trade-off than existing methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal LLMs, Knowledge unlearning, privacy safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r9JZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reexamines the forgetting–utility trade-off in multimodal LLM unlearning from the view of supervision targets. The authors find that targets with more private information may lead to insufficient forgetting, while larger distribution shifts from the original output distribution may harm model utility. To address these issues, the authors propose FFN-Perturbed Self-Distillation (FPD). It selectively perturbs FFN activations associated with private information in the teacher model to construct supervision targets with less private information and reduced distribution shift. The targets are then used to guide the student model through self-distillation. Experiments show that FPD achieves a better balance between forgetting effectiveness and model utility than existing methods.

Critical strengths

The paper is clear and easy to follow, and the motivation is natural. The paper also provides detailed ablation studies, and the results further support the authors' main claims.

Critical weaknesses
  1. Limited evaluation across different unlearning domains. The unlearning experiments are mainly conducted on S-MLLMUn Bench, which focuses on synthetic private information. The evaluation setting is relatively limited. Although the authors evaluate general model capabilities on MMBench and MMStar, they do not test FPD on other unlearning domains. I suggest adding one or two benchmarks from different domains, such as SafeEraser [1] and CoVUBench [2]. These experiments could help show whether FPD can generalize to other multimodal unlearning tasks.

  2. Strong dependence on entity-based knowledge limits the use of the method. FPD relies on entity tokens in the question and answer to select FFN activations for textual unlearning. However, unlearning targets are not always related to specific entities. They may also involve abstract concepts or procedural knowledge. For example, if we want a model to forget the steps of a dangerous procedure while keeping related general scientific knowledge, how should we select the entity tokens to perturb? For knowledge without clear entities, the current activation selection strategy may not work directly. Therefore, the authors should further discuss whether FPD can handle non-entity-based unlearning and what its limitations are.

[1] Safeeraser: Enhancing safety in multimodal large language models through multimodal machine unlearning

[2] Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models

Questions for authors
  1. FPD selects the top-k FFN activations to reduce private information. However, how can the authors make sure that these activations cover all representations related to the target knowledge? Also, the perturbation ratios are quite different across models. Could the authors explain how k is chosen and study how different k values and entity token lengths affect forgetting performance?

  2. FPD constructs unlearning targets through FFN activation perturbation. Could the authors consider simpler target construction methods, such as directly modifying the logits of private answer tokens? A comparison under similar levels of private information and distribution shift could help show the advantages of FFN perturbation.

  3. The paper uses logit restoration in textual unlearning to reduce distribution shift, but directly uses the perturbed teacher output in multimodal unlearning. Could the authors explain why logit restoration is not needed for multimodal unlearning?

Reviewer_mtvwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FFN-Perturbed Self-Distillation (FPD) for multimodal large language model unlearning. It identifies two properties of supervision targets that affect the forgetting–utility trade-off: residual private information and deviation from the original model’s output distribution. FPD constructs soft targets by selectively zeroing high-magnitude FFN activations at entity-token or image-token positions in a frozen teacher. For textual inputs, it additionally restores selected answer-entity logits to reduce unnecessary distribution shift. A student learns these targets through KL-divergence minimization while retaining knowledge through cross-entropy supervision on the retain set. Experiments on S-MLLMUn Bench show a favorable balance between targeted forgetting, visual understanding, and robustness to knowledge-recovery attacks.

Critical strengths
  1. The paper identifies residual private information and distribution shift as two important properties associated with the forgetting–utility trade-off.

  2. FPD constructs soft supervision targets through selective FFN activation suppression, with logit restoration to limit unnecessary changes to textual output distributions.

  3. Experiments cover multiple model backbones and forgetting ratios, complemented by component ablations, knowledge-recovery attacks, and general capability benchmarks.

Critical weaknesses
  1. The evaluation does not include the widely used MLLMU-Bench or LLaVA models. Broader benchmark coverage and additional model architectures would strengthen the evidence for generalizability and competitiveness.

  2. The textual intervention requires identifying entity-token boundaries, which may be ambiguous in other unlearning tasks. Moreover, high activation magnitude does not necessarily imply a strong association with private knowledge. More direct evidence is needed to establish the privacy specificity of the selected activations.

  3. Distillation-based unlearning has been extensively explored [1–2], as have neuron-level [3-4] interventions. FPD primarily combines activation suppression with self-distillation. The paper should more clearly establish its contribution beyond these existing paradigms.

  4. Suppressing high-magnitude activations at image-token positions may disrupt general visual processing alongside private knowledge retrieval. Although retain-set supervision and limited distribution shift may mitigate this effect, neither explicitly distinguishes privacy-related features from general visual features. The mechanism underlying the reported preservation of image understanding remains insufficiently established.

  5. The case studies suggest that FPD sometimes replaces private answers with fabricated information. Avoiding the original answer through hallucination does not establish reliable forgetting and introduces factuality concerns. The evaluation should distinguish hallucination from appropriate uncertainty or refusal.

[1] Attention Smoothing Is All You Need For Unlearning.

[2] DUET: Distilled LLM unlearning from an efficiently contextualized teacher.

[3] Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models.

[4] MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models.

Questions for authors

See weaknesses.

Reviewer_sXfbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies supervision-target construction for multimodal LLM unlearning. It associates residual private information in targets with insufficient forgetting, and target distribution shift with utility degradation. The proposed FFN-Perturbed Self-Distillation (FPD) zeros high-magnitude FFN activations at image-token or textual entity-token positions in a frozen teacher, restores selected answer-side logits for textual targets, and distills the resulting distributions into a student with retain-set supervision. Evaluation covers two models, three forget ratios, component ablations, and additional extraction-attack and general-capability tests.

Critical strengths

Table 1 evaluates two models and three forgetting ratios, with rephrased evaluation queries distinct from the training/unlearning queries (Appendix A.1). The evaluation measures forgotten and retained memory, image understanding, semantic correctness, and response quality.

Critical weaknesses

W1. The central claim of privacy-specific activation selection is not established by the current ablation.

Section 3.2 ranks activations solely by absolute magnitude at image/entity-token positions. This is a plausible heuristic, but magnitude alone does not distinguish private-memory features from features supporting general entity or visual processing. Figure 4 compares Top-k selection with random selection while matching token positions and the number of perturbed entries. However, zeroing the largest absolute entries removes more activation energy than zeroing the same number of random entries. The comparison therefore confounds the selected features with intervention strength.

This distinction is consequential to the claim that FPD identifies activations most strongly associated with private information. In particular, Table 4 sets the Qwen image-activation perturbation ratio to 0.50, so the intervention need not be a small, sharply localized change. An energy-matched or target-KL-matched randomized intervention, together with a private-versus-benign teacher evaluation, would test whether the ranking preferentially suppresses private recall beyond applying a stronger generic perturbation. The current ablation supports the usefulness of magnitude selection, but not yet its claimed privacy specificity.

The Section 2 analysis should likewise retain its observational interpretation: its “private information” score measures learnability by a particular distilled probe, and changing target strength changes both this score and KL simultaneously. These measurements do not independently isolate the proposed mechanisms.

W2. The empirical comparison does not yet establish the added value of FPD across comparable forgetting–utility operating points.

Table 1 compares selected configurations using, among other summaries, retained memory minus forgotten memory. FPD improves forgetting substantially, but also loses more retained knowledge than some distillation baselines. For example, on Gemma at 5%, retained image/text memory EM is 0.578/0.633 for FPD versus 0.765/0.744 for UNDIAL. This is a legitimate trade-off; the aggregate score alone does not establish that FPD preserves more utility at the same forgetting level.

The ASU and UNDIAL sensitivity analyses in Appendix B.2 are useful, but they are not accompanied by an FPD trade-off curve or a common held-out configuration-selection protocol. The perturbation ratios are model-specific, including image ratios of 0.50 for Qwen and 0.05 for Gemma. Comparisons at matched forgotten-memory accuracy or matched extraction risk would clarify whether the improvement comes from the target construction rather than the chosen operating points.

A particularly relevant missing target-construction comparison is UniLogit (Vasilev et al., Findings of ACL 2025), which is already cited and dynamically adjusts logits to give the targeted token a uniform probability without an additional suppression-strength hyperparameter. Including this close alternative would strengthen the evidence for the practical benefit of the proposed FFN intervention.

Questions for authors
  1. Privacy specificity: Does Top-k selection retain its advantage over randomized interventions matched in removed activation energy or induced target KL? How do these teacher interventions affect private-memory queries versus benign visual/entity queries?

  2. Trade-off comparison: How were FPD and baseline configurations selected? Do matched-forgetting or matched-risk comparisons, including a closely related target method such as UniLogit, support a better utility frontier?

  3. Implementation clarification: How are entity spans mapped to subword positions, and exactly which shifted logit rows are restored? Since a hidden state at token y_t predicts y_(t+1), please specify the perturbation/restoration masks and whether rankings are computed before or during successive layer interventions.

  4. Removal reference: How do forgotten-answer likelihoods compare with a model fine-tuned only on the retain profiles? The supplied base model lacks both forget and retain benchmark knowledge, so it is not that reference. Such a comparison would help calibrate the removal interpretation alongside the existing black-box attacks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.