- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies fair evaluation of multimodal large language model unlearning, where higher knowledge retention may simply come at the expense of weaker forgetting. The authors introduce iso-forget evaluation, which obtains checkpoints at different unlearning strengths and randomly selects between adjacent checkpoints to compare methods at the same expected forgetting level. The framework characterizes methods through forget–retain frontiers and decomposes retention gains into movement along a reference frontier and improvement relative to it. Experiments on six models from three families examine how identity similarity, acquisition frequency, portrait- and name-based queries, training configurations, and unlearning objectives affect these frontiers.
Critical strengths
-
Comparing retention at the same forgetting level is a clear and valuable idea. It directly addresses whether higher retention in conventional single-point comparisons merely reflects weaker forgetting. The paper also demonstrates how this evaluation changes practical judgments: format-weighted training ranks highly by retention at its default setting but drops to last under RMF@50, whereas SMFA remains competitive. This shows that the framework can distinguish genuine improvements from apparent gains.
-
Separating rehearsed retain knowledge from other knowledge absent during unlearning provides a useful observation. The latter begins to deteriorate before the former suffers substantial damage. This suggests that evaluating only the retain set used during training can overestimate preservation of other knowledge, and provides a useful design for future unlearning evaluations.
Critical weaknesses
-
The reliability of the scorer underlying the main conclusions has not been sufficiently validated. The main experiments assess forgetting by checking whether a response contains the complete reference answer. However, the paper's own appendix shows that this scorer misses paraphrased factual leakage and can even indicate the opposite direction of change from human judgments. The main frontiers and method rankings are not rechecked using semantic scoring. A parent model's ability to produce the complete reference answer does not ensure that the unlearned model will not continue leaking the fact in different words. This directly affects confidence in the core conclusions.
-
The evaluation idea is strong, but the paper does not sufficiently establish whether a general relationship governs forgetting and retention. A more fundamental question is whether changing the forget-loss weight, training steps, learning rate, trainable parameters, or training objective leads to approximately the same R whenever the resulting F is the same. If so, the frontier would have substantial predictive value, making this one of the most promising directions to explore more deeply. Although the authors examine some of these factors, the evidence does not yet adequately show which factors follow this relationship and which break it. Moreover, the main experiments focus on retrieving personal facts through portraits or names and use relatively old model families, further limiting the generality of these conclusions.
Questions for authors
-
Do the main frontier shapes and method rankings under RMF still hold when using semantic scoring that recognizes paraphrased answers? Given the appendix's finding that textual matching and human judgments can disagree in direction, please verify this on at least the main models and representative methods.
-
The paper finds that various training choices mainly move models along the frontier. Does this suggest a more general relationship: regardless of which factor changes unlearning strength, approximately the same R is obtained whenever the final F is the same? Drawing on the factors already studied and a broader range of training configurations, please clarify how broadly this relationship holds and which factors cause it to break down.
2 comments
Thank you for the response and additional experiments. They address some of my concerns, and I will keep my original rating.
Response to Reviewer Dav5
We thank the reviewer for the review, and we apologize for the late response: we ran additional experiments and analyzed them carefully.
W1 and Q1: Do frontiers and rankings hold under semantic scoring?
Yes. Under two LLM judges that credit paraphrased answers, the frontiers keep their shape and the ranking of methods is unchanged. We re-scored all runs with Qwen3-8B and Mistral-Nemo, which rate agreement with the reference from 0 to 10 and give faithful paraphrases full credit, so paraphrased leakage counts as leakage. Retention at 50% forgetting on the answer-only frontiers:
Both frontiers keep their knee under every scorer: at the weakest strength, forgetting has already dropped while retention stays close to the parent. We also swept four more objectives on InternVL2.5-8B (gradient ascent, gradient difference, KL minimization and NPO). Their ranking together with answer-only at 50% forgetting is identical under occurrence and both judges.
W2 and Q2: Is there a general relationship between F and R?
No single relationship holds across objectives: the same forgetting level can come with very different retention. On InternVL2.5-8B, retention at 50% forgetting differs widely across objectives:
This is why one checkpoint per method is not enough: a good unlearning evaluation should trace each method's frontier and compare methods along it.
W3: Generality.
The evaluation protocol is independent of the model, and its conclusions hold across model families, scorers and objectives. Iso-forget evaluation needs only a strength parameter and scores that average over a fixed set of queries, which every model and benchmark provides. Its extra cost is a few strength settings per method, and matching is done afterwards by mixing existing checkpoints per query, with no retraining. This makes it more useful, not less, for larger models, where each training run is expensive.
Across InternVL2.5-8B, LLaVA-1.5-7B and the newer Qwen2.5-VL-3B, every answer-only frontier shows the same knee: forgetting drops first while retention stays close to the parent, and retention falls only at stronger settings. The shape and the ranking of methods are unchanged under two LLM judges, and the gaps between objectives at matched forgetting are clear on every scorer. We also traced the frontier on LLaVA-1.5-13B, the larger model of the same family as LLaVA-1.5-7B. It shows the same knee, and its frontier lies above that of the 7B model, so moving to a larger model changes the level of the frontier but not its shape.
The forget and retain queries come from the public UMU-Bench benchmark. We inject the facts by fine-tuning so that we can verify the parent knows every fact before unlearning; without this step, forgetting cannot be measured.