- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies personalized unlearning for MLLMs. The authors propose PPUBench, a benchmark based on real-world knowledge. PPUBench contains three fine-grained settings. The experiments are conducted on four MLLMs; the authors find that personalized unlearning poses substantial difficulties for current unlearning methods.
Critical strengths
-
This paper is generally well written, and the proposed benchmark is comprehensive, which is a real strength.
-
The experiments are well conducted. The authors employ two models and six unlearning methods.
Critical weaknesses
-
Since the benchmark is fully based on real-world entities. What if the original model does not possess this real-world knowledge? In my opinion, an unlearning problem is well-defined only when the original model possesses the knowledge to be unlearned. Because it's unreasonable for a model to unlearn knowledge that it didn't know. In Table 2, it seems that the original Qwen3-VL-8B does not master the knowledge well.
-
Since multimodal unlearning is an important and novel problem, I think the related work is not comprehensively discussed. I would suggest that the authors discuss more related work, which includes but is not limited to MLUBench [1].
[1] MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:66465-66500.
- The authors employ a relatively old metric (ROUGE-L recall) in open-ended generation and cloze-style tasks. However, maybe it's better to employ an LLM to judge the semantic difference between the generation and the ground truth.
Questions for authors
Please refer to the weakness part.
3 comments
Official Comment
Thanks for your detailed response; most of my concerns have been addressed. I have increased my score to 3.
Response to Reviewer ztJJ
We sincerely thank you for your thoughtful assessment and positive feedback. In response to your comments, we have expanded the Related Work section and clarified the distinctions between PPU-Bench and existing benchmarks to better establish our contributions and research positioning. We have also added an experiment on LLM Evaluation to further strengthen our evaluation. We greatly appreciate your constructive feedback, which has helped us improve the manuscript. Thank you again for your time and consideration.
Response to Reviewer ztJJ
We appreciate your thoughtful comments and address your concerns below.
W1: Pre-existing Knowledge in the Original Model.
We agree that unlearning is meaningful only when the target knowledge is accessible to the original model. First, our target facts are collected from Wikipedia biographies of public figures, a widely used source of public knowledge, and are therefore likely to be covered by the pre-training corpora of general-purpose MLLMs. More importantly, we employ Memory-based Filtering with Qwen3-VL-8B to exclude facts that the original model cannot sufficiently recover, ensuring that the selected forgetting targets are based on its pre-existing knowledge.
We further compare the original Qwen3-VL-8B on PPU-Bench with the LoRA-finetuned model on MLLMU-Bench, where synthetic profiles are used to inject knowledge through additional fine-tuning. As shown in Table 1, these results are in a comparable numerical range, suggesting that the target knowledge in PPU-Bench is already substantially accessible to the original model, without requiring benchmark-specific knowledge injection. Therefore, unlike synthetic knowledge injection settings, PPU-Bench aims to evaluate the removal of naturally acquired pre-trained knowledge.
Table 1: Comparison PPU-Bench and MLLMU-Bench on Qwen3-VL-8B.
W2: Insufficient Discussion of Related Work.
We expand Section 2.1 and 2.2 to incorporate MLUBench (Li et al., 2026), and update Table 1 to include this benchmark, providing a more comprehensive comparison with existing multimodal unlearning benchmarks.
In particular, we clarify the distinction between MLUBench and PPU-Bench. MLUBench focuses on lifelong unlearning, evaluating whether MLLMs can handle successive knowledge deletion requests while mitigating cumulative performance degradation. In contrast, PPU-Bench focuses on personalized partial unlearning, investigating whether an MLLM can forget different user-specified facts for different individuals while preserving other facts about the same individual. Through its Complete, Selective, and Personalized settings, PPU-Bench provides a systematic evaluation of the model's ability to achieve fine-grained, subject-specific knowledge forgetting.
We believe these revisions provide a more comprehensive overview of recent multimodal unlearning research and better clarify the positioning of PPU-Bench.
W3: LLM Evaluation.
We thank the reviewer for this valuable suggestion. Following the LLM-as-a-Judge evaluation paradigm adopted in MLUBench, we conduct additional evaluations using GPT-6-Luna to assess whether generated responses actually contain the target facts. Specifically, responses are scored on a 0–2 scale, where higher scores indicate that the target facts are more clearly and correctly conveyed.
As shown in Table 2, the LLM-as-a-Judge results exhibit consistent trends with ROUGE-L. NPO achieves the strongest suppression of target facts. However, this comes at the cost of substantial degradation in knowledge retention. In contrast, GA_diff achieves the best knowledge retention but exhibits limited forgetting effectiveness. These consistent findings across lexical-overlap and semantic evaluations further support the reliability of our experimental conclusions.
Table 2:Partial LLM-as-Judge evaluation results under Personalized Unlearning on Qwen3-VL-8B(VQA).