- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ICU-Bench, a benchmark for evaluating continual unlearning in MMLMs under sequential privacy-deletion requests. The benchmark uses synthetic medical reports and labor contracts, with multiple image and text views, and evaluates current forgetting, historical forgetting, retention, and general utility over long task sequences. It further proposes sequence-level metrics for forgetting rebound and retain stability. Experiments on two MLLMs show that existing unlearning methods often lose retained utility, fail to preserve earlier forgetting, or achieve low forget accuracy through broader model degradation.
Critical strengths
-
The benchmark targets an evaluation gap. Separating current forgetting from historical forgetting exposes failures that a one-shot evaluation cannot detect. The upper-triangular evaluation design also provides a useful view of how each earlier task behaves after later unlearning updates.
-
The evaluation distinguishes selective forgetting from model collapse. Combining forget accuracy with retained performance, external utility, and generation quality prevents severely degraded models from appearing successful merely because they can no longer answer forget-set questions.
Critical weaknesses
- The novelty over concurrent multimodal-unlearning benchmarks is limited. The paper frames long-horizon continual MLLM unlearning as an under-explored evaluation setting. However, to the best of my knowledge, MLUBench [1] also evaluates sequential MLLM unlearning, cumulative degradation, and utility preservation. The current paper needs a direct comparison to clarify whether ICU-Bench contributes more than benchmark packaging and longer task organization.
[1] MLUBench: ABenchmark for Lifelong Unlearning Evaluation in MLLMs, ICML 2026
-
The sequence-aware metrics do not cleanly identify the phenomena they claim. RSR averages absolute differences between consecutive retain accuracies. A model that remains consistently poor also can score as “stable”. Also, the benchmark associates different retain sets with different batches, so changes across checkpoints may reflect retain-set composition rather than temporal degradation. FR measures increase in accuracy on the forget set. However, as new tasks expand that set, an increase may reflect changes in the new test samples rather than the model recovering forgotten information.
-
The main empirical findings are not specific to multimodal continual unlearning. The experiments show that repeated gradient-based updates can damage retained knowledge, which is already explored in much continual-unlearning research. The paper does not isolate whether the observed failures arise from cross-modal alignment, document-layout representations, or task ordering. The paper provides limited insight into why this multimodal setting causes those failures.
Questions for authors
-
How is ICU-Bench differentiated from MLUBench? Beyond the document domains and sequence length, what scientific question or failure mechanism can ICU-Bench evaluate that existing benchmarks cannot?
-
Do RSR and FR compare performance on fixed evaluation settings across time? If the evaluated retain or forget set changes across checkpoints, how do the metrics separate temporal drift or rebound from changes in sample composition?
-
What evidence shows that the failures are specific to multimodal unlearning rather than generic damage from repeated parameter updates? Is there a controlled comparison that isolates modality alignment or visual-document representations as the cause? (See W3 for details)
-
Without a retrained model as a reference, how can lower forget-set accuracy show successful unlearning rather than merely forgot answers on the benchmark?
Thanks for the rebuttal, some of my concerns are stressed, but the authors still didn't isolate where the observed failures arise from. I will remain my score.