acceptodds
Under review as a conference paper at ICLR 2027

PPU-Bench: Real-World Multimodal Benchmark for Personalized Partial Unlearning

Abstract

Multimodal Large Language Models (MLLMs) may memorize person-related information that remains accessible through both visual and textual inputs. However, existing MLLM unlearning benchmarks often rely on synthetic knowledge injection or complete subject-level deletion, overlooking real-world knowledge and fine-grained deletion targets within the same subject. We introduce PPU-Bench, a real-world, knowledge-injection-free benchmark for personalized partial unlearning in MLLMs. PPU-Bench contains over 24K multimodal and unimodal samples derived from pre-existing knowledge of 500 public figures under three progressively fine-grained settings: Complete, Selective, and Personalized Unlearning. It evaluates six unlearning methods across four MLLM backbones using paired QA/VQA probes, multiple evaluation formats, general-utility tests, and robustness attacks. Our experiments reveal that Complete Unlearning often suppresses visual identity associations while leaving the underlying factual knowledge accessible through textual queries. Selective Unlearning better captures category-level factual forgetting but introduces pronounced modality and forget–retain trade-offs, whereas Personalized Unlearning exposes substantial difficulties in controlling subject-specific factual boundaries. Robustness analysis under cross-image and prompt-based attacks further reveals setting-dependent vulnerabilities.

Primary area:
datasets and benchmarks
Keywords:
Multimodal unlearning, Benchmark, Multimodal large language model, Machine unlearning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ztJJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies personalized unlearning for MLLMs. The authors propose PPUBench, a benchmark based on real-world knowledge. PPUBench contains three fine-grained settings. The experiments are conducted on four MLLMs; the authors find that personalized unlearning poses substantial difficulties for current unlearning methods.

Critical strengths
  1. This paper is generally well written, and the proposed benchmark is comprehensive, which is a real strength.

  2. The experiments are well conducted. The authors employ two models and six unlearning methods.

Critical weaknesses
  1. Since the benchmark is fully based on real-world entities. What if the original model does not possess this real-world knowledge? In my opinion, an unlearning problem is well-defined only when the original model possesses the knowledge to be unlearned. Because it's unreasonable for a model to unlearn knowledge that it didn't know. In Table 2, it seems that the original Qwen3-VL-8B does not master the knowledge well.

  2. Since multimodal unlearning is an important and novel problem, I think the related work is not comprehensively discussed. I would suggest that the authors discuss more related work, which includes but is not limited to MLUBench [1].

[1] MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:66465-66500.

  1. The authors employ a relatively old metric (ROUGE-L recall) in open-ended generation and cloze-style tasks. However, maybe it's better to employ an LLM to judge the semantic difference between the generation and the ground truth.
Questions for authors

Please refer to the weakness part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ztJJ

We appreciate your thoughtful comments and address your concerns below.

W1: Pre-existing Knowledge in the Original Model.

We agree that unlearning is meaningful only when the target knowledge is accessible to the original model. First, our target facts are collected from Wikipedia biographies of public figures, a widely used source of public knowledge, and are therefore likely to be covered by the pre-training corpora of general-purpose MLLMs. More importantly, we employ Memory-based Filtering with Qwen3-VL-8B to exclude facts that the original model cannot sufficiently recover, ensuring that the selected forgetting targets are based on its pre-existing knowledge.

We further compare the original Qwen3-VL-8B on PPU-Bench with the LoRA-finetuned model on MLLMU-Bench, where synthetic profiles are used to inject knowledge through additional fine-tuning. As shown in Table 1, these results are in a comparable numerical range, suggesting that the target knowledge in PPU-Bench is already substantially accessible to the original model, without requiring benchmark-specific knowledge injection. Therefore, unlike synthetic knowledge injection settings, PPU-Bench aims to evaluate the removal of naturally acquired pre-trained knowledge.

Table 1: Comparison PPU-Bench and MLLMU-Bench on Qwen3-VL-8B.

ModelBenchmarkForget Class.↓Forget Gen.↓Retain Class.↑Retain Gen.↑
Qwen3-VL-8BPPU-Bench (Before)55.910.70762.110.714
Qwen3-VL-8BMLLMU-Bench (LoRA)55.000.60051.590.570

W2: Insufficient Discussion of Related Work.

We expand Section 2.1 and 2.2 to incorporate MLUBench (Li et al., 2026), and update Table 1 to include this benchmark, providing a more comprehensive comparison with existing multimodal unlearning benchmarks.

In particular, we clarify the distinction between MLUBench and PPU-Bench. MLUBench focuses on lifelong unlearning, evaluating whether MLLMs can handle successive knowledge deletion requests while mitigating cumulative performance degradation. In contrast, PPU-Bench focuses on personalized partial unlearning, investigating whether an MLLM can forget different user-specified facts for different individuals while preserving other facts about the same individual. Through its Complete, Selective, and Personalized settings, PPU-Bench provides a systematic evaluation of the model's ability to achieve fine-grained, subject-specific knowledge forgetting.

We believe these revisions provide a more comprehensive overview of recent multimodal unlearning research and better clarify the positioning of PPU-Bench.

W3: LLM Evaluation.

We thank the reviewer for this valuable suggestion. Following the LLM-as-a-Judge evaluation paradigm adopted in MLUBench, we conduct additional evaluations using GPT-6-Luna to assess whether generated responses actually contain the target facts. Specifically, responses are scored on a 0–2 scale, where higher scores indicate that the target facts are more clearly and correctly conveyed.

As shown in Table 2, the LLM-as-a-Judge results exhibit consistent trends with ROUGE-L. NPO achieves the strongest suppression of target facts. However, this comes at the cost of substantial degradation in knowledge retention. In contrast, GA_diff achieves the best knowledge retention but exhibits limited forgetting effectiveness. These consistent findings across lexical-overlap and semantic evaluations further support the reliability of our experimental conclusions.

Table 2:Partial LLM-as-Judge evaluation results under Personalized Unlearning on Qwen3-VL-8B(VQA).

MethodForget-Cloze ↓Forget-Gen. ↓Retain-Cloze ↑Retain-Gen. ↑
Before1.2091.0291.1581.264
GA_diff1.2510.9781.2421.310
NPO0.4110.3700.4870.446
KL_Min1.1060.8681.1521.272
MANU0.9940.6800.7830.650
Reviewer_ztJJOpenReview ↗

Official Comment

Thanks for your detailed response; most of my concerns have been addressed. I have increased my score to 3.

AuthorsOpenReview ↗

Response to Reviewer ztJJ

We sincerely thank you for your thoughtful assessment and positive feedback. In response to your comments, we have expanded the Related Work section and clarified the distinctions between PPU-Bench and existing benchmarks to better establish our contributions and research positioning. We have also added an experiment on LLM Evaluation to further strengthen our evaluation. We greatly appreciate your constructive feedback, which has helped us improve the manuscript. Thank you again for your time and consideration.

Reviewer_n1ruOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PPU-Bench, a real-world personalized partial forgetting benchmark specifically designed for multimodal large models (MLLM). Unlike existing benchmarks that mainly rely on synthetic knowledge injection or focus on complete character forgetting, PPU-Bench is based on the existing knowledge of 500 real public figures, creating over 24K QA/VQA samples and designing three different granularity forgetting tasks: Complete, Selective, and Personalized Unlearning. The authors evaluated six existing Unlearning methods through various assessment forms, general ability tests, and robustness tests.

Critical strengths

The paper is well-written and easy to understand. Moreover, the author has provided sufficient experiments to support their viewpoints, which offers certain reference value for subsequent research.

Critical weaknesses
  1. The paper presents Personalized Unlearning as one of its main contributions. However, if I understand it correctly, the main difference between Personalized Unlearning and Selective Unlearning is how the Forget and Retain Sets are divided. Personalized Unlearning uses LLMs to simulate each person's deletion preferences and select facts to forget, while Selective Unlearning uses a fixed category of sensitive information. Both follow the same basic unlearning task setup. In this sense, Personalized Unlearning may be viewed as a form of selective unlearning. Although the authors report different results under these two settings, the differences may also come from the size of the Forget Set, the types of knowledge, or the difficulty of the facts. The current experiments do not fully rule out these factors.

  2. Although the authors compare PPU-Bench with existing MLLM unlearning benchmarks in Table 1 and Section 2.1, the comparison mainly focuses on differences in knowledge sources and task settings. Apart from Personalized Unlearning, it remains unclear how much the other designs of PPU-Bench address the limitations of existing benchmarks. In addition, some of the main findings are similar to those of previous studies. For example, the gap between VQA and QA after unlearning in Finding 1 has already been studied in UMU-Bench [1]. The difficulty of preserving non-target knowledge while forgetting target knowledge in Finding 2 is also similar to the forget–retain trade-off reported by OFFSIDE [2] under fine-grained unlearning.

[1] UMU-Bench: Closing the Modality Gap in Multimodal Unlearning Evaluation

[2] OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models

Questions for authors
  1. The paper emphasizes that Personalized Unlearning aims to forget different facts for different people. However, the current evaluation mainly reports the overall Forget and Retain performance separately. Have the authors considered adding more fine-grained metrics to evaluate whether a model can forget the target facts while keeping other facts about the same person?

  2. Could the authors explain in more detail how the data are divided for Selective Unlearning and Personalized Unlearning? I am concerned that the two settings may have different levels of difficulty. For example, Personalized Unlearning may contain more facts that are difficult to forget. If so, its lower performance may come from the data itself rather than from personalization.

  3. The paper uses Qwen3-VL-8B for Memory-based Filtering to select facts that the model already knows. However, different MLLM backbones may have different knowledge about the real world, and the main experiments only cover two backbones. How well can PPU-Bench generalize to other models? For models that do not know some of the target facts, should the authors check their existing knowledge first, or use SFT to teach them these facts before unlearning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer n1ru(W1&Q2, W2)

We appreciate your thoughtful comments and address your concerns below.

W1&Q2: Distinction between Selective and Personalized Unlearning

Although both settings involve partial unlearning, they differ fundamentally in how forgetting targets are defined. Selective Unlearning applies the same predefined knowledge category (i.e., sensitive information) across all subjects, while Personalized Unlearning selects facts according to individual preferences, potentially covering basic, normal, and sensitive information. Consequently, different subjects have different forgetting targets and knowledge compositions. Rather than applying a fixed category-level filter, Personalized Unlearning requires fine-grained, subject-specific boundary control within heterogeneous factual knowledge.

Regarding the Forget-Set size, we adopt a 30% forgetting ratio for Complete Unlearning in the main experiments to maintain comparable dataset scales. As shown in Table 1, the Forget Sets contain 3,779, 3,126, and 3,635 samples for Selective, Personalized, and Complete Unlearning, respectively, indicating comparable sizes.

Table 1: Forget-Set Sizes Across Different Unlearning Settings.

SettingsForget-Set Size
Selective3,779
Personalized3,126
Complete(30% Forget)3,635

Regarding knowledge types and difficulty, we provide a detailed analysis in Section 5. Specifically, basic information is generally harder to forget than normal and sensitive information. This further highlights the challenge of Personalized Unlearning, which involves heterogeneous knowledge types and subject-specific forgetting boundaries, rather than a single predefined category.

W2:Differences from OFFSIDE and UMU-Bench

(1) Difference from UMU-Bench.

UMU-Bench mainly investigates the impact of modality alignment between unlearning and evaluation. It compares unimodal, multimodal, and mixed-modal unlearning, and primarily considers individual-level knowledge removal, which essentially corresponds to Complete Unlearning.

In contrast, our Finding 1 further investigates how the VQA–QA discrepancy varies with the granularity of forgetting targets. Under Complete Unlearning, methods exhibit strong forgetting on VQA but retain substantial knowledge on the corresponding QA. However, this discrepancy is alleviated under Selective and Personalized Unlearning, where models must forget certain facts while retaining others about the same individual. This suggests that apparent VQA forgetting under Complete Unlearning may result from suppressing visual identity associations rather than removing the underlying facts. Our analysis therefore reveals how the granularity of forgetting targets affects modality-specific knowledge suppression.

(2) Difference from OFFSIDE.

OFFSIDE focuses on forgetting a predefined category of knowledge while preserving other information. In contrast, PPU-Bench introduces a more fine-grained Personalized Unlearning setting, where deletion preferences vary across individuals and can span multiple knowledge categories, including basic, normal, and sensitive information. This requires models to establish different forgetting boundaries for different subjects and selectively suppress specific facts while preserving other knowledge about the same individual, introducing additional challenges beyond category-level unlearning.

AuthorsOpenReview ↗

Response to Reviewer n1ru(Q1, Q3)

Q1: Fine-grained Evaluation of Personalized Unlearning

To provide a more fine-grained evaluation of personalized unlearning, we introduce Subject-level Net Forgetting Gain (SNFG), which measures the forgetting of target facts against the retention of non-target facts for each individual. For each subject ss, we calculate the average performance on the Forget and Retain sets before and after unlearning:

Gs,t=Fs,tbefore−Fs,tafter,G_{s,t}=F_{s,t}^{\mathrm{before}}-F_{s,t}^{\mathrm{after}},

Ls,t=Rs,tbefore−Rs,tafter.L_{s,t}=R_{s,t}^{\mathrm{before}}-R_{s,t}^{\mathrm{after}}.

where Gs,tG_{s,t} denotes the forgetting gain on target facts, and Ls,tL_{s,t} denotes the retention loss on non-target facts for subject ss under task tt.

The Subject-level Net Forgetting Gain (SNFG) is then defined as:

SNFGs,t=Gs,t−Ls,t.SNFG_{s,t}=G_{s,t}-L_{s,t}.

Finally, we calculate the macro-average SNFG across all subjects:

\sum_{s\in\mathcal{S}}SNFG_{s,t},$$ where $\mathcal{S}$ denotes the set of subjects. A higher SNFG indicates that the forgetting gain outweighs the loss of non-target knowledge. As shown in Table 2, the results reveal distinct method behaviors under Personalized Unlearning. For example, KL-Min achieves consistently positive net gains across all four evaluation settings, while MMUnlearner exhibits substantially different performance between classification and generation. These results provide a more fine-grained assessment of whether models can selectively forget target facts while preserving other knowledge about the same individual. Table 2: SNFG under Selective and Personalized Unlearning. |Setting|Method|Class-VQA↑|Class-QA↑|Gen-VQA↑|Gen-QA↑| |:---|:---|---:|---:|---:|---:| |Selective|NPO|0.59|1.35|-12.30|-2.34| ||KL-Min|28.77|14.55|29.14|4.50| ||MMUnlearner|31.22|28.97|28.04|18.10| ||MANU|-2.63|-0.31|-3.57|-3.69| |Personalized|NPO|7.90|-0.57|-2.17|-0.49| ||KL-Min|8.11|10.08|5.73|5.37| ||MMUnlearner|0.26|-4.88|30.21|27.26| ||MANU|0.28|0.17|-3.50|-2.85| **Q3: Generalization across MLLM Backbones and Model-specific Knowledge Availability.** To investigate the generalizability of PPU-Bench, we conduct additional experiments using two different MLLM backbones: Qwen2-VL-7B and LLaVA-OneVision-2-8B-Instruct, complementing our original experiments on Qwen3-VL-8B and Gemma3-12B. As shown in Table 3, before unlearning, Qwen2-VL-7B achieves Forget-Set ROUGE-L Recall scores of 0.4174–0.4995 across the three unlearning settings, while LLaVA-OneVision-2-8B-Instruct achieves 0.4989–0.5442. These results demonstrate that substantial portions of the real-world target knowledge in PPU-Bench are accessible to different MLLM backbones, rather than being specific to the model used for memory-based filtering. Table 3: Pre-unlearning Knowledge Accessibility across Different MLLM Backbones |Model|Setting|Forget-Set|Retain-Set| |:---|:---|---:|---:| |Qwen2-VL-7B|Complete|0.4174|0.4040| ||Selective|0.4995|0.3678| ||Personalized|0.4716|0.3869| |LLaVA-OneVision-2-8B|Complete|0.4989|0.4690| ||Selective|0.5442|0.4490| ||Personalized|0.5281|0.4611| We further compare the original Qwen3-VL-8B on PPU-Bench with the LoRA-finetuned model on MLLMU-Bench, where synthetic profiles are used to inject knowledge through additional fine-tuning. As shown in Table 4, these results are in a comparable numerical range, suggesting that the target knowledge in PPU-Bench is already substantially accessible to the original model, without requiring benchmark-specific knowledge injection. Table 4: Comparison PPU-Bench and MLLMU-Bench on Qwen3-VL-8B. |Model|Benchmark|Forget Class.|Forget Gen.|Retain Class.|Retain Gen.| |:---|:---|---:|---:|---:|---:| |Qwen3-VL-8B|PPU-Bench (Before)|55.91|0.707|62.11|0.714| |Qwen3-VL-8B|MLLMU-Bench (LoRA)|55.00|0.600|51.59|0.570| All in all, our target facts are collected from Wikipedia biographies of public figures, a widely used source of public knowledge, and are therefore likely to be covered by the pre-training corpora of general-purpose MLLMs. This is further supported by our additional experiments. We do not apply additional supervised fine-tuning (SFT) to inject target facts into models with insufficient initial knowledge, as this would fundamentally change the research objective from **unlearning naturally acquired pre-trained knowledge** to **unlearning artificially injected knowledge**. We will further clarify in the paper that evaluation results should be interpreted by taking into account differences in each model's ability to acquire and access specific target knowledge.

Thank you for the author's reply. I believe the author's response has addressed my main concerns, and I will raise my score.

AuthorsOpenReview ↗

Response to Reviewer n1ru

We sincerely thank you for your thoughtful assessment and positive feedback. We greatly appreciate your constructive comments, which have helped us improve the clarity and rigor of our work. We are glad that our clarifications and additional analyses have addressed your main concerns. Thank you again for your time and consideration.

Reviewer_gg5VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PPU-Bench, a multimodal benchmark for "personalized partial unlearning" in multimodal large language models (MLLMs). The dataset is constructed from Wikipedia biographies of 500 public figures, with their personal information categorized into three types: basic, standard, and sensitive. The resulting dataset comprises approximately 24K question-answering (QA) and visual question-answering (VQA) samples. Three eval scenarios are defined as follows: "Complete" (unlearning entire subjects), "Selective" (unlearning only sensitive information), and "Personalized" (unlearning specific facts chosen by the subjects themselves). The authors evaluate six methods (GA, GA_diff, KL_Min, NPO, MMUnlearner, and MANU) using the Qwen3-VL-8B and Gemma-3-12B open-weights models.

Critical strengths

PPU-Bench is a useful and timely contribution, and the motivation is sound. Existing benchmarks that inject synthetic PPI knowledge via fine-tuning do not represent real-world user-requests for information removal.

The evaluation covers classification, generation, and cloze-style tasks; it also includes retention assessment, general utility (MMBench), and various robustness tests (cross-image, random prefixes, paraphrasing, and jailbreak prompts).

Critical weaknesses

Not 100% real data

The personalized unlearning dataset is, in part, synthetic. The information removal requests are derived from LLMs that simulate the wishes of a subject. While the limitations section acknowledges this, the claim that this constitutes a realistic and "user-centric" evaluation is stronger than what the methodology actually delivers.

Weak aspects in the provided evaluation:

  • The ROUGE-L recall metric measures lexical overlap with the ground truth sentence, not whether the fact is present or not. The same problem affects the token-F1 memorization measure in Section 3.2 and Figure 5. In my opinion it would be better to mask or remove irrelevant words, specially if they are shared with the question, then compare only the answer-bearing tokens (exact match or F1 on the PPI entities), not the full answer sentences with GT sentences.

  • The results correspond to single runs, without variance analysis, confidence intervals, or statistical significance tests. Some of the differences presented in Table 2 are small, and a single run analysis might be inconclusive. Furthermore, different hyper-parameters are used for each method, which make comparisons weak. The criterion for hyper-parameter selection is described only as " the best achievable forgetting performance while ensuring that the model does not collapse or produce garbled output", which may favor configurations that are prone to aggressive unlearning. This aspect should be documented in more detail, ideally through hyper-parameter sweeps.

Questions for authors
  • In the context of the personalized unlearning dataset, what degree of consensus did the three LLMs reach before the majority-voting? How many items were discarded in the manual verification stage?

  • How was the hyper-parameter shown in Table 11 determined for each method? Was the same criterion, based on unlearning performance vs. collapse, applied to all of them? Would the ranking change if all methods used the same learning rate or the same number of epochs? IS there any "default" configuration in the literature for each method you evaluate?

  • Did you measure statistical significance of the differences presented in Table 2? Is it possible to indicate the observed variability considering (at least with three random seeds)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gg5V(W1,W2.1,W2.2,W2.3)

We appreciate your thoughtful comments and address your concerns below.

W1: Clarification of Real-World Data

The real-world nature of PPU-Bench primarily refers to the source of the target knowledge, rather than every component of the benchmark being directly collected from real users. Unlike existing benchmarks that inject synthetic knowledge through additional fine-tuning, PPU-Bench is built upon factual knowledge extracted from the real-world biographies of 500 public figures on Wikipedia. These facts undergo manual verification and are converted into diverse QA/VQA samples without additional knowledge injection. We further apply memory-based filtering using the original Qwen3-VL-8B. Figure 6 also quantifies the memorization of these facts in the original Qwen3-VL-8B and Gemma3-12B. Thus, our goal is to evaluate the unlearning of pre-existing, accessible factual knowledge rather than artificially injected knowledge.

Second, Personalized deletion preferences are LLM-simulated rather than directly collected from the individuals themselves. Our goal is to construct controllable, subject-specific deletion preferences to investigate fine-grained forgetting boundaries within the same individual. To mitigate potential annotation biases, we employ three independent LLMs and aggregate their decisions through majority voting. As shown in Table 10, different preference simulators affect task difficulty but do not substantially change the overall evaluation trends. We will clarify the distinction between real-world knowledge and LLM-simulated deletion preferences in the revised manuscript, while identifying direct collection of user preferences as a promising direction for future research.

W2.1 Rationale for ROUGE-L Recall.

Following established MLLM unlearning benchmarks (e.g., MLLMU-Bench), we adopt ROUGE-L Recall for open-ended generation, as models are allowed to produce unrestricted natural language rather than answers in a fixed format. Strict Exact Match (EM) may incorrectly penalize semantically correct responses due to minor wording differences or additional contextual information. For example:

  • Question: The individual founded ___ in 1975 for visual effects work.

    • Ground Truth: Industrial Light & Magic
    • Prediction: Industrial Light and Magic
  • Question: Carrie was published in ___.

    • Ground Truth: 1974
    • Prediction: Carrie was published in 1974.

Both predictions convey the correct target facts despite differences in surface form. ROUGE-L Recall is therefore more suitable for measuring reference information coverage in free-form generation. Notably, RWKU [1] also adopts ROUGE-L Recall for cloze-style knowledge probing.

W2.2 Additional Exact Match Evaluation.

To facilitate comparison across different metrics, we additionally report exact match score for the Cloze task under Personalized Unlearning. The results show a trend consistent with that observed using ROUGE-L: NPO achieves the strongest unlearning performance, while GA_diff provides the best knowledge-retention performance.

Table 1: Exact match score of the cloze task for personalized unlearning on Qwen3-VL-8B.

MethodForget-Set↓Retain-Set↑
Before8.5715.29
GA_diff7.8717.81
NPO3.906.79
KL_Min6.9414.79
MANU6.498.91

W2.3 Additional LLM-as-Judge Evaluation.

Furthermore, following the LLM-as-a-Judge evaluation paradigm adopted in MLUBench, we conduct additional evaluations using GPT-6-Luna to assess whether generated responses actually contain the target facts. Specifically, responses are scored on a 0–2 scale, where higher scores indicate that the target facts are more clearly and correctly conveyed.

As shown in Table 2, the results exhibit the same overall trends as ROUGE-L: NPO achieves stronger suppression of target facts but at the cost of substantial degradation in model utility, whereas GA_diff better preserves non-target knowledge but exhibits weaker forgetting effectiveness. This consistency across different evaluation metrics further supports the reliability of our evaluation findings.

Table 2:Partial LLM-as-Judge evaluation results under Personalized Unlearning on Qwen3-VL-8B(VQA).

MethodForget-Cloze ↓Forget-Gen.↓Retain-Cloze ↑Retain-Gen. ↑
Before1.2091.0291.1581.264
GA_diff1.2510.9781.2421.310
NPO0.4110.3700.4870.446
KL_Min1.1060.8681.1521.272
MANU0.9940.6800.7830.650
AuthorsOpenReview ↗

Response to Reviewer gg5V(W2.4, W3, Q1)

W2.4: Memorization Quantification

Regarding Token-F1 in Section 3.2 and Figure 6, it serves as a proxy for measuring the original models' memorization of target knowledge. As shown in Figure 6, we compare the memorization distributions of PPU-Bench on the original Qwen3-VL-8B and Gemma3-12B models with those of MLLMU-Bench, where target knowledge is injected through additional fine-tuning. PPU-Bench exhibits comparable memorization levels, with relatively concentrated distributions of high Token-F1 and moderate NLL, indicating that its target facts are already substantially embedded in the original models, without requiring additional knowledge injection.

To further verify whether target facts are genuinely present in model-generated responses beyond lexical overlap, we conduct an additional semantic evaluation using GPT-6-Luna, achieving an accuracy of 53.59%, comparable to the memorization performance of synthetic knowledge-injection benchmarks such as MLLMU-Bench. The memorization distributions and the additional semantic evaluation demonstrate that PPU-Bench targets pre-existing knowledge already memorized by pretrained MLLMs, rather than knowledge artificially acquired through additional fine-tuning.

Reference:

[1] Pengfei Cao,Chenhao Wang,et al. RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024.

W3: Hyperparameter Selection

We thank the reviewer for raising these concerns. As also discussed in OFFSIDE [1], the fine-tuning process is highly dependent on the specific method, training data, and forgetting targets. Applying identical learning rates or training epochs across different methods therefore does not necessarily ensure a fair comparison. Thus, we adopt method-specific hyperparameter configurations. Importantly, the configuration of each method remains fixed across all three unlearning settings, rather than being individually tuned for each setting. These configurations are summarized in Table 11 of the paper.

We follow a model-collapse-aware selection criterion, considering not only forgetting performance but also retained knowledge and general multimodal utility. Specifically, we exclude configurations that lead to model collapse or garbled outputs, and additionally evaluate the resulting models on MMBench to monitor general capability degradation. To further clarify the selection process, we conduct additional hyperparameter sweeps for NPO on Qwen3-VL-8B under Selective Unlearning, varying the learning rate and training epochs.

Table 3: Hyperparameter sensitivity of NPO on Qwen3-VL-8B under Selective Unlearning.

ConfigurationForget-VQA ↓Retain-VQA ↑MMBench ↑
Before0.6270.75090.07
NPO (2e-4, 4 epochs)0.3150.42288.84
NPO (2e-4, 2 epochs)0.3170.42188.80
NPO (2e-5, 4 epochs)0.6210.74489.49
NPO (2e-5, 2 epochs)0.6220.74689.44

The results show that the learning rate substantially affects the forgetting–retention trade-off: a smaller learning rate (2e-5) preserves retained knowledge but yields limited forgetting, whereas a larger learning rate (2e-4) achieves substantially stronger forgetting, accompanied by some degradation in retention. Meanwhile, changing the training epochs from two to four has only a marginal effect at the same learning rate. We therefore select 2e-4 with four epochs as an effective operating point that achieves stronger forgetting while largely preserving general multimodal capability.

Reference:

[1] Hao Zheng, Zirui Pang, Ling Li, Zhijie Deng, Yuhan Pu, Zhaowei Zhu, Xiaobo Xia, Jiaheng Wei. OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models. Findings of ACL 2026.

Q1: Annotation Agreement and Manual Verification

Before majority voting, the three LLM annotators achieved a Fleiss' Kappa of 0.6758, indicating substantial inter-annotator agreement (Appendix D.1, Table 8). We further invited three human experts to independently validate the simulated deletion preferences and their rationales for 20 randomly sampled individuals. Their inter-annotator agreement reached 0.6433, with individual validation accuracies of 94.47%, 93.81%, and 96.02%, respectively (Table 9 in Paper). Regarding manual verification, we additionally inspected 100 sampled cases and found no explicit contradictions among the rationales provided by different LLMs. We deliberately retained these challenging cases to ensure that the benchmark includes non-trivial examples for evaluating fine-grained personalized unlearning. Furthermore, Table 10 in paper shows that different LLM simulators affect task difficulty but do not substantially change the overall evaluation trends, supporting the robustness of our majority-voting strategy.

AuthorsOpenReview ↗

Response to Reviewer gg5V(Q2, Q3)

Q2: Hyperparameter selection criterion.

We apply the same model-collapse-aware selection criterion to all methods, aiming to maximize forgetting while avoiding model collapse and excessive utility degradation. However, we adopt method-specific hyperparameters because different methods have distinct optimization objectives and sensitivities to learning rates and training epochs. For example, GA can cause model collapse even with a learning rate of 1e-6 and only one epoch (Appendix F.7).

To examine the effect of shared hyperparameters, we additionally evaluate all methods using the same learning rate (2e-5) and four epochs on Qwen3-VL-8B under Selective Unlearning. The results are shown in Table 4.

Table 4: Comparison under identical hyperparameter settings (2e-5, 4 epochs).

MethodForget-VQA↓Retain-VQA↑MMBench↑
Before0.6270.75090.07
GA0.0000.0000.00
GA_diff0.0290.76384.22
NPO0.6210.74489.49
KL_Min0.3000.73589.81
MMUnlearner0.0500.74085.31

The results reveal substantially different optimization behaviors under identical hyperparameters: GA suffers from complete model collapse, NPO achieves very limited forgetting, while GA_diff and MMUnlearner achieve stronger forgetting at the cost of greater degradation in general capabilities. This demonstrates that identical hyperparameters may lead to unfair comparisons, supporting the need for method-specific configurations.

Q3:Experimental Variance

To assess the variability of our results, we additionally repeat NPO training on Qwen3-VL-8B under Selective Unlearning using three random seeds (26, 42, and 123). We report the results of each run, together with the mean and standard deviation, in Table 5. The results show relatively limited variation across runs, supporting the stability of our experimental observations.

Table 5: NPO performance across different random seeds under Selective Unlearning.

ConfigurationForget-VQA↓Retain-VQA↑
Original run0.3150.422
Seed 420.3550.454
Seed 1230.4130.462
Seed 260.3600.450
Mean ± Std0.362 ± 0.0350.447 ± 0.015

Ethics Concerns

Our dataset construction follows established data collection practices adopted by prior real-world unlearning benchmarks [1], and we adhere to the relevant data usage policies and protocols of the respective sources. The images were collected from YouGov and Bing Image Search, while the associated factual information was derived from Wikipedia. The dataset is designed solely for academic research on privacy-oriented machine unlearning, rather than for inferring or disclosing private information about individuals. We will therefore add an explicit ethics statement to the revised manuscript, clarifying the data sources, collection procedures, and applicable usage protocols.

Reference:

[1] Pengfei Cao,Chenhao Wang,et al. RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

Dear ACs/SACs/PCs,

We hope this message finds you well. We would like to sincerely express our gratitude for your dedicated efforts in coordinating the review process for this conference. We also appreciate the constructive feedback and valuable suggestions from all reviewers. As the discussion phase comes to an end, we would like to provide a brief summary of our responses.

To address the reviewers' concerns, we provided the following additional experiments and clarifications in our responses:

Clarification of the real-world: We clarified the distinction between real-world factual knowledge and LLM-simulated deletion preferences. (Reviewer gg5V)

Additional semantic evaluation: We supplemented ROUGE-L-based evaluation with LLM-as-a-Judge evaluation to assess whether generated responses actually contain the target facts. (Reviewers gg5V, ztJJ)

Further validation of knowledge memorization: We provided additional analyses of model memorization and cross-backbone experiments to verify the accessibility of target knowledge in different pretrained MLLMs, without additional knowledge injection. (Reviewers gg5V, ztJJ)

Statistical and stability analysis: We conducted experiments with multiple random seeds and reported the corresponding mean and standard deviation. (Reviewer gg5V)

Hyperparameter analysis: We clarified the hyperparameter selection criteria and conducted supplementary experiments under identical learning rates and training epochs to facilitate fairer comparisons across methods. (Reviewer gg5V)

Further clarification of Personalized Unlearning: We clarified its distinction from Selective Unlearning in terms of subject-specific deletion preferences and heterogeneous forgetting targets, and provided additional comparisons of Forget-Set sizes. (Reviewer n1ru)

Introduction of a subject-level evaluation metric: We proposed Subject-level Net Forgetting Gain (SNFG) to quantify the trade-off between forgetting target facts and retaining non-target knowledge for individual subjects. (Reviewer n1ru)

Expanded discussion of related work: We incorporated MLUBench and further clarified the differences between PPU-Bench and existing benchmarks, particularly OFFSIDE and UMU-Bench, to better establish the distinct problem settings and contributions of PPU-Bench. (Reviewers ztJJ ,n1ru)

Clarification of data ethics and usage: We clarified that all data are derived from Wikipedia and other publicly available sources and are used solely for academic research, rather than for inferring or disclosing private information about individuals. We also confirmed that our data collection and usage follow the applicable policies and protocols of the respective sources, and added an explicit ethics statement to the revised manuscript. (Reviewer gg5V)

Overall, Reviewers n1ru and ztJJ acknowledged that their major concerns had been addressed and raised their ratings to 3. Although we did not have a further opportunity to discuss with Reviewer gg5V, we made additional efforts to address their concerns through supplementary experiments and clarifications.


We would also like to highlight the key strengths and contributions of our paper recognized by the reviewers:

Timely and meaningful research problem: The paper addresses an important yet underexplored problem of personalized partial unlearning in MLLMs, moving beyond synthetic knowledge injection and complete subject-level deletion. (as noted by Reviewers gg5V)

Comprehensive and fine-grained benchmark: PPU-Bench provides over 24K samples covering 500 public figures and three progressively fine-grained unlearning settings, together with diverse evaluation tasks and robustness tests. (as noted by Reviewers gg5V, n1ru and ztJJ)

Comprehensive experimental evaluation: The extensive experiments across multiple MLLM backbones, unlearning methods, evaluation formats, and robustness settings provide valuable insights into the challenges of multimodal unlearning. (as noted by Reviewers gg5V, n1ru and ztJJ)

Clear presentation and practical research value: The paper is well written and clearly structured, offering a useful benchmark and systematic evaluation framework for future research on multimodal unlearning. (as noted by Reviewers n1ru and ztJJ)

Overall, the reviewers acknowledged the timeliness of the research problem, the comprehensiveness of the benchmark, and the value of the experimental evaluation.

Finally, we believe that PPU-Bench makes a valuable contribution to multimodal unlearning by providing a systematic benchmark for evaluating real-world and personalized partial unlearning. We hope our additional experiments and clarifications have further strengthened the paper.

Thank you again for your time, effort, and consideration. We greatly appreciate your support.

Sincerely,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.