acceptodds
Withdrawn from ICLR 2027

PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have demonstrated capabilities in storing and recalling rich person-related knowledge, raising increasing concerns about reliable knowledge removal. However, existing machine unlearning approaches for MLLMs typically assume access to original forget and retain corpora, which are often unavailable in realistic deletion scenarios. To address this limitation, we introduce PRMU, a benchmark for evaluating corpus-free multimodal unlearning under realistic person-centric deletion requests. PRMU focuses on naturally acquired person-related knowledge and evaluates whether models can remove target knowledge while preserving related knowledge through diverse textual and visual probes, including adversarial evaluation and fine-grained locality analysis. To facilitate research in this setting, we introduce Similarity-Gated Projection Editing (SGPE), a lightweight corpus-free unlearning baseline with knowledge displacement, protected parameter-space editing, and locality-aware multimodal control. Extensive experiments on representative MLLMs reveal that existing unlearning methods often suffer from unfavorable forgetting-locality trade-offs, with significant locality degradation under aggressive forgetting settings, and remain vulnerable to multimodal knowledge reactivation. Meanwhile, SGPE provides a competitive trade-off between target forgetting, locality preservation, and general multimodal utility. We hope PRMU can facilitate future research toward realistic and scalable multimodal machine unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1SXTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PRMU, a benchmark for person-centric multimodal knowledge unlearning without access to the original training corpus. It contains 1,080 public figures and 91.9K textual and visual probes, evaluating target forgetting, preservation of knowledge about related and unrelated people, and general multimodal utility. The authors also propose Similarity-Gated Projection Editing (SGPE), combining model-generated proxy supervision, protected projection editing, and input-dependent activation.

Critical strengths

The paper tackles a timely and challenging problem: removing person-related knowledge from multimodal models without access to the original training data.

Critical weaknesses
  1. The proposed method SGPE leaves the original checkpoint unchanged and applies an external weight edit through a similarity gate. The results show that it suppresses target answers, but do not establish whether the knowledge remains accessible when the gate fails to activate. Incorrect factual substitutions also count as successful forgetting, making it difficult to distinguish knowledge removal from changes in output behavior.

  2. The paper does not test whether using only some of a person’s facts during unlearning also removes the remaining facts (selective forgetting). For example, if the method receives 50% of a person’s information, can it forget the other 50%? Without this experiment, comprehensive person-level forgetting is not established.

  3. Unlearning also harms knowledge about other people. For example, Qwen3-VL-8B batch unlearning, visual QA scores fall from .596 to .346 for related people and from .629 to .352 for unrelated people, even though MMBench accuracy remains unchanged.

Questions for authors
  1. What operational criterion distinguishes successful knowledge removal from gated response suppression in this work? Do you have evidence that target information remains inaccessible through inputs that fail to activate the similarity gate?

  2. Evidence needed to support the stated scope: are evaluation facts explicitly separated from proxy supervision and gate calibration at the underlying-fact level? If so, please report forgetting separately for supplied and withheld facts. A controlled experiment using only 50% (or other probability) of each person’s facts would affect my assessment of comprehensive person-level forgetting.

  3. How should the locality claim be reconciled with the substantial neighbor and unrelated-person degradation in Table 2, and for Table 6 where removing protected projection improves neighbor visual QA from .346 to .473?

Reviewer_YNZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PRMU studies unlearning personal knowledge when the original training corpus is unavailable, using public figure knowledge naturally acquired by the model as a reproducible proxy.

The benchmark includes textual and visual probes, adversarial questions, and knowledge retention tests for related individuals who share occupations, nationalities, similar ages, or visual resemblance. The accompanying method, SGPE, combines model-generated proxy data, protected parameter editing, and similarity-based gating to enable localized knowledge interventions.

Critical strengths

1.The problem setting is practically relevant. Real deletion requests often involve training data that cannot be located or accessed. The corpus-free setting better reflects this constraint than assuming access to the complete forget training set.

2.The benchmark uses naturally acquired knowledge. Compared with injecting artificial facts and then removing them, this setting more closely reflects the knowledge already present in pretrained models.

  1. The evaluation is relatively comprehensive. It combines multiple-choice questions, fill-in-the-blank questions, open-ended questions, and several forms of adversarial questioning with text and visual inputs, providing broader coverage than a single question-answering accuracy metric.
Critical weaknesses

1.The evaluation must distinguish between knowledge the model never possessed and knowledge it successfully forgot. Coverage of naturally acquired knowledge varies across models, levels of public recognition, and modalities. Failure to answer after unlearning may simply mean that the model did not know the answer beforehand.

The authors should report paired changes on instances that each model answered correctly before unlearning, alongside results on the full evaluation set. For multiple-choice questions, chance-level correct answers must also be considered.

Otherwise, less well-known targets may inherently receive lower Forget scores, confounding comparisons between methods.

2.ROUGE-L is insufficient for measuring unlearning. The open-ended, fill-in-the-blank, and adversarial evaluations rely heavily on ROUGE-L. Lower lexical similarity may result from paraphrasing, switching languages, or giving shorter answers, even when the target facts are still disclosed.

This is particularly concerning because SGPE uses alternative responses as displacement anchors. Merely changing the output may therefore improve this metric.

3.The related work review is insufficient. One of the paper’s core contributions, evaluating knowledge retention for neighboring individuals, substantially overlaps with the contribution of SGPVT: Self-Generated Proximal Visual Tokens for Mitigating Proximal Collateral Damage in MLLM Unlearning.

Questions for authors

See weakness.

Reviewer_C6PNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PRMU, a benchmark for person-centric multimodal unlearning without access to original training corpora. It provides model-generated proxy data and evaluates target forgetting, neighboring-person knowledge preservation, general utility, and adversarial robustness. The paper also proposes SGPE, a baseline combining knowledge displacement, protected projection editing, and similarity-based gating.

Critical strengths
  1. PRMU studies naturally acquired person-related knowledge without requiring access to original forget or retain corpora. This setting better reflects deletion requests where only the target identity is available and the model’s pretraining data cannot be retrieved.

  2. Neighbor-based locality tests assess whether forgetting a target person also damages knowledge about related individuals. Diverse textual, visual, and adversarial probes further examine residual knowledge that conventional direct-question evaluations may overlook.

  3. Experiments across three MLLMs investigate deletion scale, proxy-data sources and composition, and SGPE’s individual components. These analyses help characterize both the challenges of corpus-free unlearning and the trade-offs between target suppression, locality preservation, and general utility.

Critical weaknesses
  1. SGPE combines alternative-answer promotion, original-answer suppression, protected projection, and similarity-based gating. While the component ablations demonstrate their usefulness within SGPE, they do not sufficiently establish novelty over existing knowledge-editing methods or stronger DPO variants. Comparisons using the same proxy data, alternative answers, and comparable computational budgets would better isolate its contribution.

  2. Neighbor-person probes and MMBench do not directly establish whether ordinary visual understanding is preserved on images of the forgotten person. Questions about clothing color, visible objects, or scene attributes would help distinguish selective person-knowledge suppression from broader impairment of visual processing.

  3. NKS retains probes answered correctly by Qwen2.5-VL-32B, which does not guarantee that each evaluated model initially knows the corresponding facts. Although initial knowledge coverage is analyzed, the evaluation should explicitly report forgetting on probes each model answered correctly before unlearning. Otherwise, low post-unlearning scores may partly reflect pre-existing knowledge gaps rather than successful forgetting.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.