- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper reexamines the forgetting–utility trade-off in multimodal LLM unlearning from the view of supervision targets. The authors find that targets with more private information may lead to insufficient forgetting, while larger distribution shifts from the original output distribution may harm model utility. To address these issues, the authors propose FFN-Perturbed Self-Distillation (FPD). It selectively perturbs FFN activations associated with private information in the teacher model to construct supervision targets with less private information and reduced distribution shift. The targets are then used to guide the student model through self-distillation. Experiments show that FPD achieves a better balance between forgetting effectiveness and model utility than existing methods.
Critical strengths
The paper is clear and easy to follow, and the motivation is natural. The paper also provides detailed ablation studies, and the results further support the authors' main claims.
Critical weaknesses
-
Limited evaluation across different unlearning domains. The unlearning experiments are mainly conducted on S-MLLMUn Bench, which focuses on synthetic private information. The evaluation setting is relatively limited. Although the authors evaluate general model capabilities on MMBench and MMStar, they do not test FPD on other unlearning domains. I suggest adding one or two benchmarks from different domains, such as SafeEraser [1] and CoVUBench [2]. These experiments could help show whether FPD can generalize to other multimodal unlearning tasks.
-
Strong dependence on entity-based knowledge limits the use of the method. FPD relies on entity tokens in the question and answer to select FFN activations for textual unlearning. However, unlearning targets are not always related to specific entities. They may also involve abstract concepts or procedural knowledge. For example, if we want a model to forget the steps of a dangerous procedure while keeping related general scientific knowledge, how should we select the entity tokens to perturb? For knowledge without clear entities, the current activation selection strategy may not work directly. Therefore, the authors should further discuss whether FPD can handle non-entity-based unlearning and what its limitations are.
[1] Safeeraser: Enhancing safety in multimodal large language models through multimodal machine unlearning
[2] Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models
Questions for authors
-
FPD selects the top-k FFN activations to reduce private information. However, how can the authors make sure that these activations cover all representations related to the target knowledge? Also, the perturbation ratios are quite different across models. Could the authors explain how k is chosen and study how different k values and entity token lengths affect forgetting performance?
-
FPD constructs unlearning targets through FFN activation perturbation. Could the authors consider simpler target construction methods, such as directly modifying the logits of private answer tokens? A comparison under similar levels of private information and distribution shift could help show the advantages of FFN perturbation.
-
The paper uses logit restoration in textual unlearning to reduce distribution shift, but directly uses the perturbed teacher output in multimodal unlearning. Could the authors explain why logit restoration is not needed for multimodal unlearning?