- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces OpenMMUnlearning, a unified framework for unlearning MLLMs. OpenMMUlearning follows a modular design, featuring data/model workers, trainer, and unified evaluator that can be readily extended to support new unlearning datasets/methods/metrics. Additionally, OpenMMUlearning provides meta-evaluation of unlearning metrics through tests of faithfulness (i.e., separability between models fine-tuned with and without data) and robustness (i.e., stability with slight modifications in model parameters). The current implementation consolidates 5 datasets, 4 models, 12 unlearning methods, and extensive metrics for forgetting effectiveness, utility preservation, and unlearning robustness.
Critical strengths
-
Establishing a unified benchmarking framework meaningfully contributes to progress in MLLM unlearning. Future unlearning methods can be fairly compared against each other by using the same evaluation suite. Additionally, a unified framework can mitigate implementation errors that can affect the results being reported across different works.
-
OpenMMUnlearning is thoughtfully and extensitively designed. Modular components make it easy to add new unlearning datasets, methods, and metrics. A wide variety of commonly used unlearning datasets, methods, and metrics in MLLM unlearning are available.
Critical weaknesses
-
OpenMMUnlearning fails to evaluate efficiency of unlearning methods. Efficiency is a crucial factor in machine unlearning, where the goal is to remove the influence of the data from a trained model using significantly fewer resources than retraining the model from scratch. Hence, efficiency should also be benchmarked for a fair, multifaceted comparison.
-
OpenMMUnlearning is designed for unlearning at the fine-tuning stage. It is unclear if it can be used to benchmark unlearning at the pre-training stage. Supporting unlearning at the pre-training stage is equally, if not more, important because pre-trained models are often trained on large-scale datasets from the internet, which have a high likelihood to contain contaminated/poisoned data, private or copyrighted data -- of which unlearning is useful for removal.
-
OpenMMUnlearning fails to evaluate the sensitivity of unlearning methods to hyperparameters. Hyperparameter sensitivity reflects how reliable an unlearning method is to be useful in practice. Some methods (e.g., GA, GDiff) are inherently sensitive to learning rates, number of epochs, and forgetting coefficient. Hence, they should be treated differently from more reliable methods.
-
OpenMMUnlearning should include more methods to select data to be unlearned. OpenMMUnlearning currently uses fixed forget sets for benchmarking. However, characteristics of the forget set can affect unlearning performance (e.g., forget set similar to retain set might cause more collateral damaging for unlearning). Hence, having diverse forget-retain splits of different characteristics could be useful.
I am open to increasing my score if the answers sufficiently address my concerns above.
Questions for authors
-
Can the authors provide insights into the efficiency and the hyperparameter sensitivity of the implemented unlearning methods in OpenMMUnlearning?
-
Different unlearning methods show different strengths in forgetting effectiveness and utility preservation in Table 2. This makes it difficult to compare one method against another. Do you think it is better to define a fixed threshold for utility and report the forgetting effectiveness at such a fixed utility threshold?
-
(Clarifying question) In Table 2, Retain-only doesn't achieve the highest scores despite being the gold standard in unlearning. Moreover, its performance is comparable to Finetune, which does not perform unlearning at all. Can the authors explain why Retain-only does not achieve the highest scores and why Finetune and Retain-only performance are similar?
-
(Clarifying question) It is unclear how representative the unlearning methods included in OpenMMUnlearning are. Can the authors elaborate on the differences among these methods and whether they represent different approaches to MMLU unlearning?
13 comments
W2. Support for unlearning knowledge acquired during pretraining
We thank the reviewer for highlighting the importance of unlearning knowledge acquired during pretraining. To examine this setting, we use MMUBench, introduced by Li et al. alongside Single Image Unlearning (SIU). MMUBench evaluates concept-level visual-recognition unlearning across 20 concepts, including real-world identities and artistic styles. Its construction screens concepts using the original MLLM's recognition ability, providing a benchmark for targeting knowledge already accessible in a released model. [1]
We additionally evaluated ten unlearning methods on MMUBench, selecting two of its 20 concepts (10%) as the forget set and using the remaining 18 concepts as the retain set.
| Method | Overall ↑ | Forget Quality ↑ | Model Utility ↑ | Robustness ↑ | Model ↑ | Input ↑ | Output ↑ |
|---|---|---|---|---|---|---|---|
| GA | 0.550 | 0.606 | 0.526 | 0.519 | 0.602 | 0.436 | - |
| GD | 0.462 | 0.459 | 0.535 | 0.392 | 0.504 | 0.281 | - |
| KL | 0.479 | 0.486 | 0.529 | 0.421 | 0.532 | 0.310 | - |
| RMU | 0.461 | 0.453 | 0.532 | 0.399 | 0.511 | 0.287 | - |
| NPO | 0.483 | 0.492 | 0.534 | 0.424 | 0.534 | 0.314 | - |
| UNDIAL | 0.696 | 0.682 | 0.521 | 0.884 | 0.767 | 1.000 | - |
| MMUnlearner | 0.653 | 0.623 | 0.535 | 0.800 | 0.768 | 0.833 | - |
| MANU | 0.509 | 0.483 | 0.530 | 0.515 | 0.571 | 0.460 | - |
| MIP-Editor | 0.488 | 0.481 | 0.511 | 0.473 | 0.623 | 0.323 | - |
| SMFA | 0.467 | 0.417 | 0.538 | 0.445 | 0.500 | 0.389 | - |
Table R1. Evaluation of ten representative MLLM unlearning methods on MMUBench in terms of Forget Quality, Model Utility, and Robustness. We also report an overall aggregate score and divide Robustness into model-, input-, and output-side evaluations. Higher scores are better. Bold and italicized values indicate the best and second-best results among the unlearning methods, respectively. Missing results are marked with "-".
In this pretrained-knowledge unlearning setting, we lack a matched held-out non-member set whose exclusion from the original model's training data can be verified. Output-side robustness is therefore marked with "-", as MIA ROC-AUC evaluation requires this control. Reported Robustness averages the model-side and input-side scores.
On MMUBench, UNDIAL achieves the highest overall score of 0.696, Forget Quality of 0.682, and Robustness of 0.884. MMUnlearner ranks second overall at 0.653, with the highest model-side robustness of 0.768 and Model Utility of 0.535. GA ranks third overall at 0.550, and MANU and MIP-Editor occupy intermediate positions. GD, KL, RMU, and NPO obtain lower Forget Quality than the two leading methods. SMFA achieves the highest Model Utility of 0.538 and the lowest Forget Quality of 0.417, illustrating the trade-off between utility preservation and forgetting effectiveness.
The leading methods differ from those in the MLLMU 10% fine-tuning setting, where GD and MIP-Editor share the highest reported aggregate score of 0.647, followed by UNDIAL at 0.640. The shift toward UNDIAL and MMUnlearner on MMUBench suggests that methods may have different strengths for synthetic profile knowledge introduced through fine-tuning and pre-existing visual identity knowledge.
Eight of ten methods have lower reported aggregates on MMUBench, with a ten-method mean of 0.525 compared with 0.619 on MLLMU. This pattern is consistent with additional difficulty in the pretrained-knowledge setting. Prior studies identify training exposure frequency as a factor in memorization and unlearning. Wei et al. use Hubble's standard and perturbed pretrained models to show that relative repetition frequency affects memorization strength. [2] Krishnan et al. find that more frequent pretraining knowledge is harder to unlearn. [3]
Pretrained knowledge may be encountered more frequently during large-scale training than the fictitious profile facts introduced through MLLMU fine-tuning. Following the frequency-based account in these studies, such repeated exposure may reinforce the learned associations and make the target knowledge more resistant to unlearning, helping explain the lower aggregate performance of most methods on MMUBench.
References
[1] Li, Jiaqi, et al. "Single image unlearning: Efficient machine unlearning in multimodal large language models." NeurIPS 37 (2024): 35414–35453.
[2] Wei, Johnny, et al. "Hubble: a model suite to advance the study of LLM memorization." International Conference on Learning Representations. 2026.
[3] Krishnan, Aravind, Siva Reddy, and Marius Mosbach. "Not all data are unlearned equally." arXiv preprint arXiv:2504.05058 (2025).
W3 & Q1. Hyperparameter sensitivity and practical reliability
We thank the reviewer for highlighting the importance of hyperparameter sensitivity for practical reliability. We analyzed 150 distinct hyperparameter configurations from the main MLLMU 10% experiment using LLaVA-1.5-7B. The specific hyperparameter combinations are provided in Appendix A.3.2.
| Method | Forget Quality | Model Utility | Robustness | Overall |
|---|---|---|---|---|
| GA | 0.757 ± 0.024 | 0.075 ± 0.020 | 0.874 ± 0.020 | 0.568 ± 0.008 |
| GD | 0.727 ± 0.049 | 0.297 ± 0.161 | 0.781 ± 0.011 | 0.601 ± 0.041 |
| KL | 0.725 ± 0.049 | 0.192 ± 0.103 | 0.782 ± 0.016 | 0.566 ± 0.027 |
| RMU | 0.719 ± 0.038 | 0.365 ± 0.103 | 0.784 ± 0.036 | 0.623 ± 0.018 |
| NPO | 0.752 ± 0.029 | 0.339 ± 0.069 | 0.786 ± 0.034 | 0.625 ± 0.005 |
| UNDIAL | 0.710 ± 0.019 | 0.430 ± 0.031 | 0.748 ± 0.014 | 0.629 ± 0.009 |
| MMUnlearner | 0.700 ± 0.027 | 0.418 ± 0.012 | 0.770 ± 0.013 | 0.629 ± 0.004 |
| MANU | 0.684 ± 0.004 | 0.436 ± 0.004 | 0.764 ± 0.018 | 0.628 ± 0.006 |
| MIP-Editor | 0.696 ± 0.020 | 0.368 ± 0.108 | 0.790 ± 0.018 | 0.618 ± 0.030 |
| SMFA | 0.703 ± 0.008 | 0.146 ± 0.008 | 0.777 ± 0.004 | 0.542 ± 0.004 |
Table R2. Hyperparameter sensitivity across the evaluated search spaces. Scores are reported as mean ± sample standard deviation across distinct configurations. Higher mean scores indicate better performance, and larger standard deviations indicate greater hyperparameter sensitivity. Bold and italicized entries mark the largest and second-largest reported standard deviations, respectively, with ties receiving the same formatting.
GD has the largest Overall standard deviation at 0.041, followed by MIP-Editor at 0.030 and KL at 0.027, indicating greater hyperparameter sensitivity of their aggregate performance. MMUnlearner and SMFA have the smallest Overall standard deviation at 0.004. The standard deviations are 0.005 for NPO, 0.006 for MANU, 0.008 for GA, and 0.009 for UNDIAL. RMU exhibits an intermediate level of variation, with a standard deviation of 0.018. Surprisingly, GA also exhibits relatively good stability, with an Overall standard deviation of 0.008. Examining the table suggests that this may partly arise from its consistently low Model Utility of 0.075 ± 0.020. Its forget-only updates may damage shared capabilities across learning rates, placing utility close to the lower bound and limiting further decline.
We thank the reviewer for highlighting the importance of efficiency evaluation.
We agree that efficiency should be evaluated jointly with forgetting effectiveness and model utility. We conducted additional measurements of the complete deletion cost for ten methods using LLaVA-1.5-7B on MLLMU with a 10% forget ratio and a single NVIDIA A800 80GB GPU.
We measured four phases covering model loading, method-specific preparation (including analysis and reference-model initialization), parameter updates, and export of the resulting model or artifacts. We synchronized CUDA at phase boundaries, recorded peak allocated and reserved GPU memory with PyTorch, and sampled GPU memory through NVML every 200 ms. All times are reported in minutes, with durations below 0.01 min shown as <0.01 and totals computed from unrounded phase durations.
In our measurements, the general unlearning methods have total deletion times ranging from 6.83 to 15.14 min. Most multimodal-specific methods take longer. SMFA has the longest total deletion time at 191.35 min, followed by MIP-Editor at 181.01 min. MANU is an exception, achieving the shortest total deletion time at 4.28 min.
MANU ranks and prunes target-related neurons using activation statistics from forget and retain data under multimodal and text-only inputs. The procedure consists mainly of forward-pass analysis and a lightweight pruning-mask application, accounting for its shortest total runtime in our measurements.
MMUnlearner estimates gradient-based parameter saliency on forget and preservation data, then uses their ratio to construct a mask for forget-gradient updates. Estimating these saliency maps requires additional forward and backward passes before masked training. This preparation overhead makes its total runtime longer than those of the general unlearning baselines in the table.
MIP-Editor's second-longest total runtime mainly arises from gradient-integration-based attribution for locating influential cross-modal neuron paths. Path discovery requires repeated gradient computations, making preparation the dominant cost. The subsequent localized parameter updates account for a smaller share of the runtime.
SMFA trains multimodal and text-only forgetting adapters and a retaining anchor, then uses the anchor to sculpt updates according to their direction and relative magnitude. Repeated optimization of these components concentrates its computational cost in parameter updates and produces the longest total runtime in our measurements.