acceptodds
Under review as a conference paper at ICLR 2027

ICU-Bench: Benchmarking Continual Unlearning in Multimodal Large Language Models

Abstract

Privacy deletion requests often arrive sequentially, creating a continual unlearning challenge for deployed multimodal large language models (MLLMs). However, existing benchmarks mainly focus on static or short-sequence settings, offering limited support for evaluating continual privacy deletion on privacy-critical documents. To bridge this gap, we introduce ICU-Bench, an Identity-centric Continual Unlearning benchmark (pronounced “I see you”) for privacy-critical multimodal documents. ICU-Bench contains 1,000 synthetic privacy-sensitive profiles from medical reports and labor contracts, comprising 9,500 document images, 16,000 question-answer pairs, and 100 sequential forget tasks. We further introduce history-aware evaluation protocols and sequence-aware metrics to assess current forgetting, historical forgetting preservation, retained utility, and stability throughout the unlearning sequence. Experiments with representative unlearning methods on two MLLMs reveal that methods effective on current targets often fail to preserve earlier forgetting or retained capabilities over long sequences. Some methods further obtain low forget accuracy through severe model degradation. These results expose long-horizon failure modes overlooked by conventional evaluations and highlight the need for multimodal unlearning methods explicitly designed for continual privacy deletion.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Continual Unlearning, Multimodal Large Language Models, Privacy, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a1g4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ICU-Bench, a benchmark for evaluating continual unlearning in MMLMs under sequential privacy-deletion requests. The benchmark uses synthetic medical reports and labor contracts, with multiple image and text views, and evaluates current forgetting, historical forgetting, retention, and general utility over long task sequences. It further proposes sequence-level metrics for forgetting rebound and retain stability. Experiments on two MLLMs show that existing unlearning methods often lose retained utility, fail to preserve earlier forgetting, or achieve low forget accuracy through broader model degradation.

Critical strengths
  1. The benchmark targets an evaluation gap. Separating current forgetting from historical forgetting exposes failures that a one-shot evaluation cannot detect. The upper-triangular evaluation design also provides a useful view of how each earlier task behaves after later unlearning updates.

  2. The evaluation distinguishes selective forgetting from model collapse. Combining forget accuracy with retained performance, external utility, and generation quality prevents severely degraded models from appearing successful merely because they can no longer answer forget-set questions.

Critical weaknesses
  1. The novelty over concurrent multimodal-unlearning benchmarks is limited. The paper frames long-horizon continual MLLM unlearning as an under-explored evaluation setting. However, to the best of my knowledge, MLUBench [1] also evaluates sequential MLLM unlearning, cumulative degradation, and utility preservation. The current paper needs a direct comparison to clarify whether ICU-Bench contributes more than benchmark packaging and longer task organization.

[1] MLUBench: ABenchmark for Lifelong Unlearning Evaluation in MLLMs, ICML 2026

  1. The sequence-aware metrics do not cleanly identify the phenomena they claim. RSR averages absolute differences between consecutive retain accuracies. A model that remains consistently poor also can score as “stable”. Also, the benchmark associates different retain sets with different batches, so changes across checkpoints may reflect retain-set composition rather than temporal degradation. FR measures increase in accuracy on the forget set. However, as new tasks expand that set, an increase may reflect changes in the new test samples rather than the model recovering forgotten information.

  2. The main empirical findings are not specific to multimodal continual unlearning. The experiments show that repeated gradient-based updates can damage retained knowledge, which is already explored in much continual-unlearning research. The paper does not isolate whether the observed failures arise from cross-modal alignment, document-layout representations, or task ordering. The paper provides limited insight into why this multimodal setting causes those failures.

Questions for authors
  1. How is ICU-Bench differentiated from MLUBench? Beyond the document domains and sequence length, what scientific question or failure mechanism can ICU-Bench evaluate that existing benchmarks cannot?

  2. Do RSR and FR compare performance on fixed evaluation settings across time? If the evaluated retain or forget set changes across checkpoints, how do the metrics separate temporal drift or rebound from changes in sample composition?

  3. What evidence shows that the failures are specific to multimodal unlearning rather than generic damage from repeated parameter updates? Is there a controlled comparison that isolates modality alignment or visual-document representations as the cause? (See W3 for details)

  4. Without a retrained model as a reference, how can lower forget-set accuracy show successful unlearning rather than merely forgot answers on the benchmark?

1 comment

Thanks for the rebuttal, some of my concerns are stressed, but the authors still didn't isolate where the observed failures arise from. I will remain my score.

Reviewer_p84sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on continual unlearning for MLLMs. The challenge is that existing multimodal unlearning benchmarks mostly examine static or short-sequence settings, leaving long-horizon, history-aware evaluation on privacy-critical documents limited. These settings cannot reveal the persistence of forgetting or the accumulation of retained knowledge. To address these limitations, this paper introduces ICU-Bench, which contains 1000 synthetic medical report and labor contract profiles presented in full-image, masked-image, and text-only views and divided into 100 sequential forgetting tasks. The authors also propose a history-aware protocol and sequence-level metrics. Experiments with unlearning methods on LLaVA-1.5-7B and Qwen2-VL-7B show that methods that are effective in the current round often fail to maintain earlier forgetting or retain-set performance over long sequences. Some methods achieve low forget accuracy only because the model collapses.

Critical strengths
  1. The continual unlearning problem is practical. In the real world, deletion requests often arrive sequentially. The sequence of 100 tasks is also much longer than those in current multimodal unlearning benchmarks.

  2. The document domain is a good choice. The masked-image design is interesting, as it can evaluate whether the model still remembers the text when it cannot read it directly.

Critical weaknesses
  1. The evaluated baselines are too simple and insufficient and the evaluation protocol is not comprehensive. As a new benchmark, it should include at least strong fine-tuning-based, localization-based, and inference-time methods, rather than just six older baselines. Although this paper focuses on continual unlearning, it does not evaluate any continual unlearning methods. Several important continual unlearning baselines should be evaluated [1, 2]. The evaluation only uses direct queries, so the benchmark is not comprehensive. Including attacks such as MIA, relearning attacks [3], or paraphrase attacks would help provide a better understanding of unlearning methods using the proposed benchmark.

[1] Gao, Chongyang, et al. “On large language model continual unlearning.” International Conference on Learning Representations. Vol. 2025. 2025.

[2] Zhao, Hongbo, et al. “Continual forgetting for pre-trained vision models.” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[3] Hu, Shengyuan, et al. “Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning.” International Conference on Learning Representations. Vol. 2025. 2025.

  1. The novelty and contribution relative to existing benchmarks is not very clear. Among MLLM unlearning benchmarks, MLLMU-Bench, CLEAR, and FIUBench [4] already include identity-level profiles. Sustainability over sequential unlearning tasks has also been studied in MUSE and PULSE [5]. The differences here appear to be the document domain and the longer sequence. The reported failure modes also seem similar to those already reported in MUSE. The differences between benchmarks and the novelty of this benchmark should be further discussed and clarified. It would be useful to show how the methods’ performance changes under the masked view or different document layouts.

[4] Ma, Yingzi, et al. “Benchmarking vision language model unlearning via fictitious facial identity dataset.” International Conference on Learning Representations. Vol. 2025. 2025.

[5] Kawakami, Tatsuki, et al. “Pulse: Practical evaluation scenarios for large multimodal model unlearning.” arXiv preprint arXiv:2507.01271 (2025).

Questions for authors
  1. There is no reference for what “good forgetting” should look like, as no results from a retrained model are reported. Many forget accuracy values are below 25%, which may indicate model collapse rather than genuine forgetting. It would therefore be useful to retrain the model on the retain sets after removing each forget set and use the retrained model as a gold-standard reference.

  2. I am concerned that some findings may result from fixed hyperparameters rather than the methods themselves. In the continual unlearning setting, none of the methods appear to use early stopping or hyperparameter tuning. The evaluated baselines, GA and NPO, already collapse after 10 tasks on Qwen, so their failure over longer sequences is expected. The paper should include stronger unlearning baselines rather than methods that can easily lead to model collapse. A small hyperparameter tuning study could also make the conclusions more convincing and comprehensive, providing stronger evidence for the benchmark’s value.

  3. No code or dataset has been released for what is presented as a benchmark paper.

  4. Some numbers do not match across tables. In Table 2, KL-Min on Qwen is reported as “–” at Task 100, but Table 4 reports 68.5 and Table 3 reports 1.84 at Task 100. For LLaVA, KL-Min has complete results elsewhere but is reported as “–” in Table 4. How is retain performance at Task 1 computed? Could the author explain these mismatch?

Some other questions:

  • How are the profiles and document images generated? How many templates are used, and how are the incorrect answer options sampled?

  • The “preliminary experiments” in Section 3 are simply the main results on the same benchmark (Figure 3), so the motivation reads as somewhat circular.

  • The two backbones are somewhat old. Including a newer model, such as Qwen2.5-VL, would be useful.

  • The paper misuses \citep{} and \citet{}; these commands should be used appropriately.

Reviewer_tbGpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ICU-Bench for continual multimodal unlearning, using 1,000 synthetic medical and employment profiles organized into 100 deletion requests. It evaluates six methods on two MLLMs and tracks current and historical forgetting, retention, and general utility.

Critical strengths
  • The paper addresses a useful problem, and the document setting provides a relevant test case for repeated privacy deletion.
  • Masked-image and text-only queries are a sensible way to test whether the model still remembers information that cannot be directly read from the image (App. A.1).
  • The historical trajectories, external VQA results, and generation quality measurements provide useful checks beyond current-task forgetting accuracy (Figs. 7–11; Table 4; App. B.4).
Critical weaknesses

Major Weaknesses

1.PULSE already studies sequential multimodal unlearning and reports cumulative retention and utility collapse [1]. ICU-Bench adds document views, a longer sequence, and explicit historical tracking, but provides no comparison with an existing benchmark under a matched continual schedule

  1. As written, FR compares accuracies on a historical forget set that grows across checkpoints (Sec. 4.3). The difference can therefore reflect changes in the evaluated samples rather than recovery of previously forgotten information, unless both terms use the same historical samples.

  2. Table 2 reports MMUnlearner’s retain VQA/QA as 42.7/46.5 for Qwen2-VL at T100, while the corresponding final cells in Fig. 10 are 76.2/86.0.

  3. The paper does not specify whether forget and retain identities remain disjoint across the full sequence, or how historical retain sets handle identities deleted later (Secs. 4.2–4.3). Since retain samples are also used for training, these rules are necessary to distinguish rebound from reintroduced information and retention loss from intentional deletion.

Minor Weaknesses

  1. Several baseline settings are missing from Table 6 and App. A.2, including the KL weight, NPO’s β and reference-model update policy, and MANU’s pruning budget and cumulative pruning policy.

  2. MANU’s first five GQ-F scores in Table 9 average to 0.7448, whereas Table 3 reports 0.776.

Questions for authors
  1. Are both terms in FR evaluated on the same historical samples? If not, do the rankings and rebound conclusions hold after recomputing the metric on a shared set?

  2. How should the retention results in Table 2 and Fig. 10 be reconciled, and which results support the main conclusions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.