acceptodds
Under review as a conference paper at ICLR 2027

Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning

17 views

Abstract

As multimodal large language models (MLLMs) become more capable and widely deployed, concerns about privacy and safety have become increasingly pressing. Machine unlearning offers one approach to addressing these concerns by removing designated information from trained models while preserving unrelated capabilities. However, fragmented implementations and evaluation protocols, incomplete robustness testing, and limited understanding of metric reliability make progress in MLLM unlearning difficult to assess systematically. We introduce **Open-MMUnlearning**, an open-source, extensible framework that integrates target-model preparation, multimodal data processing, unlearning, and evaluation through shared interfaces and structured configurations. The framework supports five benchmarks spanning privacy, safety, and copyright, eight MLLMs from four model families, and twelve unlearning methods. Its evaluation suite jointly assesses forgetting effectiveness, retained utility, and robustness to model interventions, adversarial inputs, and membership inference attacks. Using a common evaluation protocol, we compare ten representative unlearning methods. In this comparison, GD and MIP-Editor tie for the highest overall score: GD achieves the highest Forget Quality, while MIP-Editor preserves more Model Utility. We further introduce a metric meta-evaluation protocol that tests faithfulness using models with controlled exposure to target knowledge and robustness under quantization and relearning. Among the thirteen evaluated metrics, BLEU achieves the highest aggregate reliability score. KS-Test attains the highest faithfulness AUC but performs less well on robustness. Together, the framework and these findings support reproducible comparison of MLLM unlearning methods and systematic assessment of evaluation reliability. We release the code and model checkpoints at [this anonymous repository](https://anonymous.4open.science/r/Open-MMunlearning-4E20).

Primary area:
datasets and benchmarks
Keywords:
Machine Unlearning, Multimodal LLM

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_obY6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OpenMMUnlearning, a unified framework for unlearning MLLMs. OpenMMUlearning follows a modular design, featuring data/model workers, trainer, and unified evaluator that can be readily extended to support new unlearning datasets/methods/metrics. Additionally, OpenMMUlearning provides meta-evaluation of unlearning metrics through tests of faithfulness (i.e., separability between models fine-tuned with and without data) and robustness (i.e., stability with slight modifications in model parameters). The current implementation consolidates 5 datasets, 4 models, 12 unlearning methods, and extensive metrics for forgetting effectiveness, utility preservation, and unlearning robustness.

Critical strengths
  1. Establishing a unified benchmarking framework meaningfully contributes to progress in MLLM unlearning. Future unlearning methods can be fairly compared against each other by using the same evaluation suite. Additionally, a unified framework can mitigate implementation errors that can affect the results being reported across different works.

  2. OpenMMUnlearning is thoughtfully and extensitively designed. Modular components make it easy to add new unlearning datasets, methods, and metrics. A wide variety of commonly used unlearning datasets, methods, and metrics in MLLM unlearning are available.

Critical weaknesses
  1. OpenMMUnlearning fails to evaluate efficiency of unlearning methods. Efficiency is a crucial factor in machine unlearning, where the goal is to remove the influence of the data from a trained model using significantly fewer resources than retraining the model from scratch. Hence, efficiency should also be benchmarked for a fair, multifaceted comparison.

  2. OpenMMUnlearning is designed for unlearning at the fine-tuning stage. It is unclear if it can be used to benchmark unlearning at the pre-training stage. Supporting unlearning at the pre-training stage is equally, if not more, important because pre-trained models are often trained on large-scale datasets from the internet, which have a high likelihood to contain contaminated/poisoned data, private or copyrighted data -- of which unlearning is useful for removal.

  3. OpenMMUnlearning fails to evaluate the sensitivity of unlearning methods to hyperparameters. Hyperparameter sensitivity reflects how reliable an unlearning method is to be useful in practice. Some methods (e.g., GA, GDiff) are inherently sensitive to learning rates, number of epochs, and forgetting coefficient. Hence, they should be treated differently from more reliable methods.

  4. OpenMMUnlearning should include more methods to select data to be unlearned. OpenMMUnlearning currently uses fixed forget sets for benchmarking. However, characteristics of the forget set can affect unlearning performance (e.g., forget set similar to retain set might cause more collateral damaging for unlearning). Hence, having diverse forget-retain splits of different characteristics could be useful.

I am open to increasing my score if the answers sufficiently address my concerns above.

Questions for authors
  1. Can the authors provide insights into the efficiency and the hyperparameter sensitivity of the implemented unlearning methods in OpenMMUnlearning?

  2. Different unlearning methods show different strengths in forgetting effectiveness and utility preservation in Table 2. This makes it difficult to compare one method against another. Do you think it is better to define a fixed threshold for utility and report the forgetting effectiveness at such a fixed utility threshold?

  3. (Clarifying question) In Table 2, Retain-only doesn't achieve the highest scores despite being the gold standard in unlearning. Moreover, its performance is comparable to Finetune, which does not perform unlearning at all. Can the authors explain why Retain-only does not achieve the highest scores and why Finetune and Retain-only performance are similar?

  4. (Clarifying question) It is unclear how representative the unlearning methods included in OpenMMUnlearning are. Can the authors elaborate on the differences among these methods and whether they represent different approaches to MMLU unlearning?

13 comments

W1 & Q1. Missing efficiency evaluation of unlearning methods

We thank the reviewer for highlighting the importance of efficiency evaluation.

We agree that efficiency should be evaluated jointly with forgetting effectiveness and model utility. We conducted additional measurements of the complete deletion cost for ten methods using LLaVA-1.5-7B on MLLMU with a 10% forget ratio and a single NVIDIA A800 80GB GPU.

We measured four phases covering model loading, method-specific preparation (including analysis and reference-model initialization), parameter updates, and export of the resulting model or artifacts. We synchronized CUDA at phase boundaries, recorded peak allocated and reserved GPU memory with PyTorch, and sampled GPU memory through NVML every 200 ms. All times are reported in minutes, with durations below 0.01 min shown as <0.01 and totals computed from unrounded phase durations.

MethodLoading (min)Preparation (min)Updates (min)Export (min)Total (min)Peak allocated memory (GiB)
GA0.20<0.016.62<0.016.8314.66
GD0.19<0.0112.27<0.0112.4715.55
KL0.20<0.0114.06<0.0114.2729.74
RMU0.19<0.019.74<0.019.9529.54
NPO0.19<0.0114.94<0.0115.1428.93
UNDIAL0.19<0.018.03<0.018.2328.66
MMUnlearner0.1823.5812.34<0.0136.1115.07
MANU0.214.07<0.01<0.014.2813.20
MIP-Editor0.22175.455.33<0.01181.0162.31
SMFA0.300.42190.560.07191.3516.14

In our measurements, the general unlearning methods have total deletion times ranging from 6.83 to 15.14 min. Most multimodal-specific methods take longer. SMFA has the longest total deletion time at 191.35 min, followed by MIP-Editor at 181.01 min. MANU is an exception, achieving the shortest total deletion time at 4.28 min.

MANU ranks and prunes target-related neurons using activation statistics from forget and retain data under multimodal and text-only inputs. The procedure consists mainly of forward-pass analysis and a lightweight pruning-mask application, accounting for its shortest total runtime in our measurements.

MMUnlearner estimates gradient-based parameter saliency on forget and preservation data, then uses their ratio to construct a mask for forget-gradient updates. Estimating these saliency maps requires additional forward and backward passes before masked training. This preparation overhead makes its total runtime longer than those of the general unlearning baselines in the table.

MIP-Editor's second-longest total runtime mainly arises from gradient-integration-based attribution for locating influential cross-modal neuron paths. Path discovery requires repeated gradient computations, making preparation the dominant cost. The subsequent localized parameter updates account for a smaller share of the runtime.

SMFA trains multimodal and text-only forgetting adapters and a retaining anchor, then uses the anchor to sculpt updates according to their direction and relative magnitude. Repeated optimization of these components concentrates its computational cost in parameter updates and produces the longest total runtime in our measurements.

W2. Support for unlearning knowledge acquired during pretraining

We thank the reviewer for highlighting the importance of unlearning knowledge acquired during pretraining. To examine this setting, we use MMUBench, introduced by Li et al. alongside Single Image Unlearning (SIU). MMUBench evaluates concept-level visual-recognition unlearning across 20 concepts, including real-world identities and artistic styles. Its construction screens concepts using the original MLLM's recognition ability, providing a benchmark for targeting knowledge already accessible in a released model. [1]

We additionally evaluated ten unlearning methods on MMUBench, selecting two of its 20 concepts (10%) as the forget set and using the remaining 18 concepts as the retain set.

MethodOverall ↑Forget Quality ↑Model Utility ↑Robustness ↑Model ↑Input ↑Output ↑
GA0.5500.6060.5260.5190.6020.436-
GD0.4620.4590.5350.3920.5040.281-
KL0.4790.4860.5290.4210.5320.310-
RMU0.4610.4530.5320.3990.5110.287-
NPO0.4830.4920.5340.4240.5340.314-
UNDIAL0.6960.6820.5210.8840.7671.000-
MMUnlearner0.6530.6230.5350.8000.7680.833-
MANU0.5090.4830.5300.5150.5710.460-
MIP-Editor0.4880.4810.5110.4730.6230.323-
SMFA0.4670.4170.5380.4450.5000.389-

Table R1. Evaluation of ten representative MLLM unlearning methods on MMUBench in terms of Forget Quality, Model Utility, and Robustness. We also report an overall aggregate score and divide Robustness into model-, input-, and output-side evaluations. Higher scores are better. Bold and italicized values indicate the best and second-best results among the unlearning methods, respectively. Missing results are marked with "-".

In this pretrained-knowledge unlearning setting, we lack a matched held-out non-member set whose exclusion from the original model's training data can be verified. Output-side robustness is therefore marked with "-", as MIA ROC-AUC evaluation requires this control. Reported Robustness averages the model-side and input-side scores.

On MMUBench, UNDIAL achieves the highest overall score of 0.696, Forget Quality of 0.682, and Robustness of 0.884. MMUnlearner ranks second overall at 0.653, with the highest model-side robustness of 0.768 and Model Utility of 0.535. GA ranks third overall at 0.550, and MANU and MIP-Editor occupy intermediate positions. GD, KL, RMU, and NPO obtain lower Forget Quality than the two leading methods. SMFA achieves the highest Model Utility of 0.538 and the lowest Forget Quality of 0.417, illustrating the trade-off between utility preservation and forgetting effectiveness.

The leading methods differ from those in the MLLMU 10% fine-tuning setting, where GD and MIP-Editor share the highest reported aggregate score of 0.647, followed by UNDIAL at 0.640. The shift toward UNDIAL and MMUnlearner on MMUBench suggests that methods may have different strengths for synthetic profile knowledge introduced through fine-tuning and pre-existing visual identity knowledge.

Eight of ten methods have lower reported aggregates on MMUBench, with a ten-method mean of 0.525 compared with 0.619 on MLLMU. This pattern is consistent with additional difficulty in the pretrained-knowledge setting. Prior studies identify training exposure frequency as a factor in memorization and unlearning. Wei et al. use Hubble's standard and perturbed pretrained models to show that relative repetition frequency affects memorization strength. [2] Krishnan et al. find that more frequent pretraining knowledge is harder to unlearn. [3]

Pretrained knowledge may be encountered more frequently during large-scale training than the fictitious profile facts introduced through MLLMU fine-tuning. Following the frequency-based account in these studies, such repeated exposure may reinforce the learned associations and make the target knowledge more resistant to unlearning, helping explain the lower aggregate performance of most methods on MMUBench.

References

[1] Li, Jiaqi, et al. "Single image unlearning: Efficient machine unlearning in multimodal large language models." NeurIPS 37 (2024): 35414–35453.

[2] Wei, Johnny, et al. "Hubble: a model suite to advance the study of LLM memorization." International Conference on Learning Representations. 2026.

[3] Krishnan, Aravind, Siva Reddy, and Marius Mosbach. "Not all data are unlearned equally." arXiv preprint arXiv:2504.05058 (2025).

W3 & Q1. Hyperparameter sensitivity and practical reliability

We thank the reviewer for highlighting the importance of hyperparameter sensitivity for practical reliability. We analyzed 150 distinct hyperparameter configurations from the main MLLMU 10% experiment using LLaVA-1.5-7B. The specific hyperparameter combinations are provided in Appendix A.3.2.

MethodForget QualityModel UtilityRobustnessOverall
GA0.757 ± 0.0240.075 ± 0.0200.874 ± 0.0200.568 ± 0.008
GD0.727 ± 0.0490.297 ± 0.1610.781 ± 0.0110.601 ± 0.041
KL0.725 ± 0.0490.192 ± 0.1030.782 ± 0.0160.566 ± 0.027
RMU0.719 ± 0.0380.365 ± 0.1030.784 ± 0.0360.623 ± 0.018
NPO0.752 ± 0.0290.339 ± 0.0690.786 ± 0.0340.625 ± 0.005
UNDIAL0.710 ± 0.0190.430 ± 0.0310.748 ± 0.0140.629 ± 0.009
MMUnlearner0.700 ± 0.0270.418 ± 0.0120.770 ± 0.0130.629 ± 0.004
MANU0.684 ± 0.0040.436 ± 0.0040.764 ± 0.0180.628 ± 0.006
MIP-Editor0.696 ± 0.0200.368 ± 0.1080.790 ± 0.0180.618 ± 0.030
SMFA0.703 ± 0.0080.146 ± 0.0080.777 ± 0.0040.542 ± 0.004

Table R2. Hyperparameter sensitivity across the evaluated search spaces. Scores are reported as mean ± sample standard deviation across distinct configurations. Higher mean scores indicate better performance, and larger standard deviations indicate greater hyperparameter sensitivity. Bold and italicized entries mark the largest and second-largest reported standard deviations, respectively, with ties receiving the same formatting.

GD has the largest Overall standard deviation at 0.041, followed by MIP-Editor at 0.030 and KL at 0.027, indicating greater hyperparameter sensitivity of their aggregate performance. MMUnlearner and SMFA have the smallest Overall standard deviation at 0.004. The standard deviations are 0.005 for NPO, 0.006 for MANU, 0.008 for GA, and 0.009 for UNDIAL. RMU exhibits an intermediate level of variation, with a standard deviation of 0.018. Surprisingly, GA also exhibits relatively good stability, with an Overall standard deviation of 0.008. Examining the table suggests that this may partly arise from its consistently low Model Utility of 0.075 ± 0.020. Its forget-only updates may damage shared capabilities across learning rates, placing utility close to the lower bound and limiting further decline.

Reviewer_KrBJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Open-MMUnlearning, a unified framework for MLLM unlearning. It supports five benchmarks, eight MLLMs, and twelve unlearning methods. The authors further assess the reliability of thirteen unlearning evaluation metrics through controlled target-knowledge exposure, quantization, and relearning experiments. The results reveal trade-offs among forgetting effectiveness, model utility, and robustness across different unlearning methods, as well as substantial differences in the reliability of evaluation metrics.

Critical strengths

The main strength of this paper is the development of a modular and extensible framework for MLLM unlearning evaluation. The authors integrate both general unlearning and MLLM-specific unlearning methods, along with evaluations of Forget Quality, Model Utility, and various aspects of Robustness, into a unified experimental pipeline. Such an infrastructure facilitates more consistent comparisons across different methods and improves the reproducibility of experimental results, providing practical value to the research community.

Critical weaknesses

My primary concern is whether the definitions of Faithfulness and Robustness in the proposed metric meta-evaluation adequately capture genuine machine unlearning. The authors assess a metric's ability to identify the presence of target knowledge by comparing models trained with and without exposure to that knowledge. However, distinguishing between these two groups of models is not necessarily equivalent to determining whether target knowledge has been successfully removed through unlearning. For instance, unlearning may primarily alter a model's output behavior while leaving some of the underlying knowledge encoded in its internal representations. Conversely, target knowledge may remain recoverable through related concepts or knowledge acquired during pretraining. Therefore, a metric that successfully distinguishes models with and without exposure to target knowledge may not necessarily provide a reliable assessment of genuine unlearning effectiveness.

Questions for authors

N/A

3 comments

W1. Whether Faithfulness and Robustness adequately reflect successful unlearning

We thank the reviewer for raising this concern about the reliability of unlearning evaluation.

Faithfulness calibrates a metric's sensitivity to target-fact exposure using model pools whose fine-tuning data include or exclude the correct target facts (Appendix A.4.5). Robustness then examines the initial forgetting judgments on actual unlearned checkpoints judged successful by each metric, using quantization and matched targeted relearning (Section 5 and Appendix A.4.2).

Quantization changes the parameter representation with zero additional target examples and tests whether target answers become accessible again. Targeted relearning exposes the unlearned model and a Retain-Only reference to the same target examples under an identical training budget. Greater recovery in the unlearned model provides evidence that the initial metric judgment may have overstated forgetting. Table 3 illustrates the distinction between exposure discrimination and reliability under recovery interventions. KS-Test achieves a Faithfulness AUC of 0.991 and a relearning robustness score of 0.727, showing that strong exposure discrimination can coexist with weaker reliability under recovery interventions.

To examine residual knowledge in internal representations, the framework includes probing at layers 16 and 24 (Appendices A.2.1 and A.4.2). We freeze the unlearned backbone and train an auxiliary linear readout on its hidden states. Recovery above matched controls would indicate that target-related information remains decodable after suppression in standard outputs. Table 9 shows that the current probes give similar scores for Finetune and Retain-Only, suggesting limited discrimination in this setting.

From the perspective of unlearning pretrained knowledge, recovery through related concepts can be examined using multi-hop jailbreaks. MMUBench, introduced with SIU, provides questions about the factual knowledge of a concept depicted in an image. These questions probe whether the model can still access associations between visual identity and factual knowledge after direct identity responses have been suppressed. We will incorporate the released multi-hop questions into our MMUBench evaluation data to extend coverage of this recovery pathway. [1]

Our work provides a unified framework for implementing, evaluating, and comparing unlearning methods, with empirical reliability evidence under the tested interventions. We leave the development of more discriminative representation-level probes and the evaluation of broader recovery pathways to future work.

References

[1] Li, Jiaqi, et al. "Single image unlearning: Efficient machine unlearning in multimodal large language models." Advances in Neural Information Processing Systems 37 (2024): 35414–35453.

Reviewer_KrBJOpenReview ↗

Thanks for the feedback.

The authors addressed my questions; though still not fully solved, I appreciate the effort. I am keeping my original rating of weak acceptance. Thanks.

We sincerely thank you for your careful evaluation of our manuscript and your valuable comments. We would be pleased to address any further questions you may have and provide additional clarification as needed.

Reviewer_n1NYOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Open-MMUnlearning, a modular framework for multimodal large language model unlearning. Through shared interfaces and structured configurations, it integrates target-model preparation, multimodal data processing, unlearning methods, and evaluation. The framework supports twelve methods, eight models across four model families, and five benchmarks covering privacy, safety, and copyright.

The empirical study compares ten methods on the MLLMU 10% forget setting, jointly examining forgetting effectiveness, retained utility, and robustness across multiple attack surfaces. GD and MIP-Editor achieve the same highest aggregate score through different forgetting–utility tradeoffs. The paper additionally conducts a meta-evaluation of thirteen metrics using controlled target-knowledge exposure and interventions involving quantization and relearning. BLEU obtains the highest aggregate reliability score, whereas KS-Test achieves the highest faithfulness AUC. The contribution combines reusable research infrastructure with an empirical investigation of unlearning evaluation reliability.

Critical strengths
  1. A useful infrastructure contribution for reproducible multimodal unlearning research. The framework brings model preparation, heterogeneous unlearning methods, benchmark processing, and robustness audits into a common experimental pipeline. Integrating multimodal inputs, model-component adaptations, and different evaluation protocols has practical technical value. The shared interfaces and explicit configurations could reduce the effort needed to reproduce comparisons and evaluate new methods. The manuscript’s reported provision of code and over 400 checkpoints further strengthens its value as a community resource.

  2. An informative examination of both unlearning methods and their measurement tools. The paper demonstrates why forgetting and robustness scores must be interpreted alongside retained utility. For example, GA achieves high forgetting and robustness scores while its Model Utility falls to 0.066, illustrating how favorable scores can coexist with substantial capability degradation. The metric meta-evaluation also reveals meaningful differences between faithfulness and robustness: KS-Test achieves a faithfulness AUC of 0.991, while BLEU performs better in aggregate reliability. These observations offer practical guidance for metric selection and result interpretation, extending the contribution beyond software integration.

Critical weaknesses

There is room to broaden the empirical coverage. The framework supports multiple models and benchmarks, while the controlled method comparison primarily focuses on the MLLMU 10% forget setting, and the metric meta-evaluation uses LLaVA-1.5-7B in that setting. Consequently, the method rankings and BLEU’s aggregate advantage currently reflect these particular experimental conditions. The authors acknowledge this limitation in Appendix A.5; a small number of representative evaluations on additional models or tasks would further strengthen the generality of the findings.

Questions for authors
  1. Scope of the conclusions: Do the authors already have preliminary observations from other models or forget ratios? If so, could they briefly explain whether the main trends are consistent with the current results?

  2. Comparability of metrics: The robustness analysis filters checkpoints according to each metric’s assessment of forgetting. How many checkpoints remain eligible for each metric, and how much do these sets overlap? Does BLEU retain its aggregate advantage when metrics are compared on a common eligible checkpoint set?

4 comments

W1 & Q1. Broader empirical coverage and the scope of the conclusions

We thank the reviewer for highlighting the importance of broader empirical coverage. We additionally evaluated all ten methods on Qwen2.5-VL-7B under MLLMU 10% and on LLaVA-1.5-7B under MLLMU 5%.

MethodOverall ↑Forget Quality ↑Model Utility ↑Robustness ↑Model ↑Input ↑Output ↑
GA0.5110.7360.0050.7910.8870.9960.491
GD0.6320.7030.5680.6250.7080.7280.437
KL0.6120.6780.5740.5840.7200.7230.309
RMU0.6080.6930.5610.5710.7430.7220.247
NPO0.6420.6930.5580.6760.7260.7480.555
UNDIAL0.6400.7320.4060.7820.7690.7790.800
MMUnlearner0.5920.7490.1530.8760.9010.9560.770
MANU0.6120.7150.5460.5740.7370.7360.250
MIP-Editor0.6100.6880.5260.6160.7220.7270.398
SMFA0.5980.6720.5730.5480.7150.7040.225

Table R1. Evaluation of ten representative MLLM unlearning methods on Qwen2.5-VL-7B under the MLLMU 10% setting. Higher scores are better. Bold and italicized values indicate the highest and second-highest unrounded scores among the ten methods.

MethodOverall ↑Forget Quality ↑Model Utility ↑Robustness ↑Model ↑Input ↑Output ↑
GA0.6000.8220.0700.9070.9950.9640.763
GD0.6370.7460.4020.7620.8600.6500.776
KL0.6450.7450.4290.7600.8290.6320.819
RMU0.6420.7430.4160.7670.8530.6450.803
NPO0.6350.7330.4320.7410.8360.6670.721
UNDIAL0.6460.7500.4190.7680.8370.7250.741
MMUnlearner0.6460.7640.3950.7780.8900.7590.687
MANU0.6390.7260.4290.7620.8410.6290.816
MIP-Editor0.6230.7400.3570.7730.8080.7000.811
SMFA0.6340.7380.3850.7790.8740.6490.813

Table R2. Evaluation on LLaVA-1.5-7B under MLLMU 5%. Higher scores are better. Bold and italicized values mark the highest and second-highest reported scores. Overall ranking uses unrounded scores. KL and MANU share the second-highest Model Utility at the displayed precision.

The forgetting–utility trade-off persists across the additional settings. GA obtains Forget Quality of 0.736 with Model Utility of 0.005 on Qwen at 10%, and Forget Quality of 0.822 with Model Utility of 0.070 on LLaVA at 5%. On Qwen, MMUnlearner achieves the highest Forget Quality of 0.749 and Robustness of 0.876 with Model Utility of 0.153. These results reinforce the importance of evaluating retained capabilities alongside forgetting, consistent with the main LLaVA 10% experiment.

Method rankings change with the backbone and forget ratio. In the LLaVA 10% main experiment, GD and MIP-Editor share the highest reported Overall of 0.647. On Qwen at 10%, NPO leads at 0.642, followed by UNDIAL at 0.640 and GD at 0.632. On LLaVA at 5%, MMUnlearner ranks first at 0.6460 and UNDIAL second at 0.6456, followed by KL at 0.645. NPO achieves the highest 5% Model Utility of 0.432. These changes show that the leading method depends on the experimental setting, while the forgetting–retention trade-off remains consistent.

Q2. Comparability across metrics and checkpoint selection

We thank the reviewer for raising the issue of metric comparability and checkpoint selection. We agree that metric-specific eligibility can affect the comparison. Of the 162 unlearned LLaVA-1.5-7B checkpoints evaluated on the MLLMU 10% forget split, 118 satisfy the original utility-drop limit of 20%.

We therefore recomputed quantization and relearning robustness for all metrics on the same 118 utility-qualified checkpoints, using only the utility criterion and keeping the original faithfulness AUCs fixed. BLEU retains the highest aggregate score at 0.9500, followed by KS-Test at 0.9468 and ROUGE-L at 0.9229. We also compared BLEU with each other metric on their pairwise intersection, so both metrics had to deem every checkpoint in that comparison successfully unlearned. BLEU had the higher aggregate score in all 12 pairwise comparisons. On the 74 jointly eligible checkpoints for BLEU and ROUGE-L, their aggregate scores were 0.9490 and 0.9205, respectively. BLEU's lead persists on matched checkpoints in this setting, with a margin of 0.0031 over KS-Test on the shared utility-qualified set.

MetricEligible under the original rulesJointly eligible with BLEUAggregate on the same 118 utility-qualified checkpoints ↑
Cloze57430.7678
Classification90700.7425
Truth Ratio114860.8950
Probability63510.9185
KS-Test1290.9468
JS Distance40330.7837
ROUGE-L76740.9229
BLEU90900.9500
LOSS68540.8492
ZLIB54440.8520
GradNorm118900.8734
Min-K24230.8629
Min-K++980.7155

Table R1. Checkpoint eligibility, overlap with BLEU, and aggregate scores on the same 118 utility-qualified checkpoints. Bold and italicized values indicate the highest and second-highest aggregate scores, respectively.

Reviewer_n1NYOpenReview ↗

Thanks for authors' response

Thank you for the additional experiments. I hope the experiments and their findings will be included in the final version of the paper. I will maintain my current score.

We sincerely thank you for your careful evaluation of our manuscript and your valuable comments. We would be pleased to address any further questions you may have and provide additional clarification as needed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair,

We sincerely thank you, the SPC, and all reviewers for the time and effort devoted to evaluating our submission and for the constructive feedback throughout the review process.

Open-MMUnlearning addresses fragmented implementations, inconsistent evaluation protocols, and the limited systematic assessment of metric reliability in MLLM unlearning. Our work combines a unified, extensible research framework with systematic empirical analysis, supporting reproducible method comparisons, a better understanding of forgetting–utility trade-offs, and informed selection of evaluation metrics. The framework integrates 8 MLLMs from 4 model families, 5 unlearning benchmarks, 12 unlearning methods, and over 30 evaluation measures and robustness assessments. We further compare 10 representative methods and systematically meta-evaluate 13 metrics. By sharing reusable implementations, common interfaces, and reproducible experimental workflows, we provide research infrastructure for the open-source community, enabling researchers to build on existing components and contribute new models, methods, datasets, and metrics within a shared framework.

Our rebuttal provides additional experiments and clarifications addressing the following concerns.

  1. Broader empirical coverage and applicability. In response to Reviewer n1NY (W1, Q1), we evaluated all ten methods on Qwen2.5-VL-7B under MLLMU 10% and LLaVA-1.5-7B under MLLMU 5%. These results reinforce the importance of jointly assessing forgetting and retained capabilities and show that method rankings depend on the experimental setting. To address Reviewer obY6 (W2), we additionally evaluated all ten methods on MMUBench, demonstrating the framework’s applicability to unlearning knowledge already present in pretrained models.

  2. Meaning and scope of metric meta-evaluation. In response to Reviewer KrBJ (W1), we clarified the operational meaning and empirical scope of faithfulness and robustness. We explained how controlled knowledge exposure, quantization, and matched targeted relearning provide complementary evidence about metric reliability. We also discussed the role and discriminative capacity of the existing representation probes in examining residual knowledge.

  3. Comparability across evaluation metrics. To address Reviewer n1NY (Q2), we reported checkpoint eligibility counts and overlaps and recomputed quantization and relearning robustness for all metrics on the same 118 utility-qualified checkpoints. We further compared BLEU with every other metric on their jointly eligible checkpoints. Within the evaluated setting, BLEU retained the highest aggregate reliability score on the common set and achieved higher aggregate scores in all twelve pairwise comparisons.

  4. Efficiency and hyperparameter sensitivity. In response to Reviewer obY6 (W1, Q1), we measured end-to-end deletion time and peak GPU memory for all ten methods, covering model loading, method-specific preparation, parameter updates, and export. To address Reviewer obY6 (W3, Q1), we analyzed 150 hyperparameter configurations and reported performance means and variability, characterizing computational costs and tuning sensitivity.

  5. Forget-set selection and utility-constrained comparison. To address Reviewer obY6 (W4), we constructed high- and low-similarity forget–retain splits and evaluated all ten methods. Greater similarity was associated with lower average forgetting effectiveness and retained utility. In response to Reviewer obY6 (Q2), we also compared the highest observed forgetting effectiveness under a common utility floor of 80% of the Finetune baseline, providing an additional comparison for scenarios with utility-preservation requirements.

  6. Reference baselines and method representativeness. In response to Reviewer obY6 (Q3), we explained Retain-Only’s counterfactual role, the reasons for its similar performance to Finetune, and how aggregate scores reflect forgetting–utility trade-offs. To address Reviewer obY6 (Q4), we detailed the mechanisms of the implemented methods and their representative coverage across four unlearning operator families.

We respectfully hope that these additional results and clarifications will receive further consideration in the final assessment. Thank you again for your time and thoughtful consideration of our submission. We would be pleased to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.