acceptodds
Under review as a conference paper at ICLR 2027

Can MLLMs Retain More Without Unlearning Less?

Abstract

Multimodal large language model (MLLM) unlearning aims to make MLLMs for- get chosen facts, such as a biography recalled from a portrait or a name, while retaining the rest of their knowledge. Existing benchmarks take a static, single- point view, scoring each method once at the end of its training. Even a fully converged run marks one position on the trade-off between forgetting and reten- tion and cannot characterize the method as a whole. We instead take a dynamic view and introduce iso-forget evaluation, which characterizes each method by its forget–retain frontier, the best retention it attains at every forgetting level. Unlike a single score, the frontier compares methods at the same forgetting level. It sep- arates a genuine improvement, which shifts the frontier, from a gain that merely trades forgetting for retention along it. Specifically, we start from a few anchor checkpoints at different unlearning strengths and reach any target forgetting level exactly, without retraining, by randomizing between two adjacent anchors with a closed-form probability. Across six MLLMs from three families, forgetting costs almost no retention up to a knee, whereas beyond it retention drops sharply. Tun- ing the training procedure or switching the objective mostly moves a model along the frontier, so the algorithm chiefly picks a point on it. The frontier itself is set by how the model represents and retrieves the forgotten facts, since forgetting spills over onto look-alike people and blocks mainly the cue it was trained on. MLLMs can therefore retain more without unlearning less by reaching a fact through every cue that recalls it, rather than by pushing harder on one.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Machine Unlearning, Auxiliary Supervision, Supervision Weight Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Dav5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fair evaluation of multimodal large language model unlearning, where higher knowledge retention may simply come at the expense of weaker forgetting. The authors introduce iso-forget evaluation, which obtains checkpoints at different unlearning strengths and randomly selects between adjacent checkpoints to compare methods at the same expected forgetting level. The framework characterizes methods through forget–retain frontiers and decomposes retention gains into movement along a reference frontier and improvement relative to it. Experiments on six models from three families examine how identity similarity, acquisition frequency, portrait- and name-based queries, training configurations, and unlearning objectives affect these frontiers.

Critical strengths
  1. Comparing retention at the same forgetting level is a clear and valuable idea. It directly addresses whether higher retention in conventional single-point comparisons merely reflects weaker forgetting. The paper also demonstrates how this evaluation changes practical judgments: format-weighted training ranks highly by retention at its default setting but drops to last under RMF@50, whereas SMFA remains competitive. This shows that the framework can distinguish genuine improvements from apparent gains.

  2. Separating rehearsed retain knowledge from other knowledge absent during unlearning provides a useful observation. The latter begins to deteriorate before the former suffers substantial damage. This suggests that evaluating only the retain set used during training can overestimate preservation of other knowledge, and provides a useful design for future unlearning evaluations.

Critical weaknesses
  1. The reliability of the scorer underlying the main conclusions has not been sufficiently validated. The main experiments assess forgetting by checking whether a response contains the complete reference answer. However, the paper's own appendix shows that this scorer misses paraphrased factual leakage and can even indicate the opposite direction of change from human judgments. The main frontiers and method rankings are not rechecked using semantic scoring. A parent model's ability to produce the complete reference answer does not ensure that the unlearned model will not continue leaking the fact in different words. This directly affects confidence in the core conclusions.

  2. The evaluation idea is strong, but the paper does not sufficiently establish whether a general relationship governs forgetting and retention. A more fundamental question is whether changing the forget-loss weight, training steps, learning rate, trainable parameters, or training objective leads to approximately the same R whenever the resulting F is the same. If so, the frontier would have substantial predictive value, making this one of the most promising directions to explore more deeply. Although the authors examine some of these factors, the evidence does not yet adequately show which factors follow this relationship and which break it. Moreover, the main experiments focus on retrieving personal facts through portraits or names and use relatively old model families, further limiting the generality of these conclusions.

Questions for authors
  1. Do the main frontier shapes and method rankings under RMF still hold when using semantic scoring that recognizes paraphrased answers? Given the appendix's finding that textual matching and human judgments can disagree in direction, please verify this on at least the main models and representative methods.

  2. The paper finds that various training choices mainly move models along the frontier. Does this suggest a more general relationship: regardless of which factor changes unlearning strength, approximately the same R is obtained whenever the final F is the same? Drawing on the factors already studied and a broader range of training configurations, please clarify how broadly this relationship holds and which factors cause it to break down.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Dav5

We thank the reviewer for the review, and we apologize for the late response: we ran additional experiments and analyzed them carefully.

W1 and Q1: Do frontiers and rankings hold under semantic scoring?

Yes. Under two LLM judges that credit paraphrased answers, the frontiers keep their shape and the ranking of methods is unchanged. We re-scored all runs with Qwen3-8B and Mistral-Nemo, which rate agreement with the reference from 0 to 10 and give faithful paraphrases full credit, so paraphrased leakage counts as leakage. Retention at 50% forgetting on the answer-only frontiers:

ModelOccurrenceQwen3-8BMistral-Nemo
InternVL2.5-8B87.887.286.4
LLaVA-1.5-7B77.474.577.0

Both frontiers keep their knee under every scorer: at the weakest strength, forgetting has already dropped while retention stays close to the parent. We also swept four more objectives on InternVL2.5-8B (gradient ascent, gradient difference, KL minimization and NPO). Their ranking together with answer-only at 50% forgetting is identical under occurrence and both judges.

W2 and Q2: Is there a general relationship between F and R?

No single relationship holds across objectives: the same forgetting level can come with very different retention. On InternVL2.5-8B, retention at 50% forgetting differs widely across objectives:

ObjectiveRMF@50
Answer-only87.8
Gradient difference72.3
KL minimization59.6
NPO50.9
Gradient ascent50.3

This is why one checkpoint per method is not enough: a good unlearning evaluation should trace each method's frontier and compare methods along it.

W3: Generality.

The evaluation protocol is independent of the model, and its conclusions hold across model families, scorers and objectives. Iso-forget evaluation needs only a strength parameter and scores that average over a fixed set of queries, which every model and benchmark provides. Its extra cost is a few strength settings per method, and matching is done afterwards by mixing existing checkpoints per query, with no retraining. This makes it more useful, not less, for larger models, where each training run is expensive.

Across InternVL2.5-8B, LLaVA-1.5-7B and the newer Qwen2.5-VL-3B, every answer-only frontier shows the same knee: forgetting drops first while retention stays close to the parent, and retention falls only at stronger settings. The shape and the ranking of methods are unchanged under two LLM judges, and the gaps between objectives at matched forgetting are clear on every scorer. We also traced the frontier on LLaVA-1.5-13B, the larger model of the same family as LLaVA-1.5-7B. It shows the same knee, and its frontier lies above that of the 7B model, so moving to a larger model changes the level of the frontier but not its shape.

The forget and retain queries come from the public UMU-Bench benchmark. We inject the facts by fine-tuning so that we can verify the parent knows every fact before unlearning; without this step, forgetting cannot be measured.

Thank you for the response and additional experiments. They address some of my concerns, and I will keep my original rating.

Reviewer_yYBfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the iso-forget evaluation method to compare multimodal unlearning methods at the same level of forgetting. It constructs forgetting-retention curves by changing the unlearning strength and randomly choosing between two adjacent checkpoints. Through controlled experiments, mainly using refusal training, it studies how knowledge relatedness, modality, and training choices affect this trade-off.

Critical strengths

The paper develops an analysis tool to compare knowledge retention at the same level of forgetting.

Critical weaknesses
  1. The novelty and contribution are limited. Prior work, such as RULE [1], already analyzes forget–retain frontiers. The main methodological addition is comparison at matched forgetting levels.
  2. Most analyses focus on a reference refusal-training method and a small dataset. The comparison in Figure 7(a) does not test whether the key findings on knowledge relatedness and modality hold across different unlearning methods. [1] Chenlong Zhang et al. RULE: Reinforcement UnLEarning Achieves Forget–retain Pareto Optimality. NeurIPS, 2025.
Questions for authors
  1. The presentation of figures could be clearer. For example, Figure 7(a) uses unlabeled gray curves for several methods, making their differences and the resulting conclusions difficult to assess.
  2. The reported frontiers are constructed from only a few checkpoints. Their shape and the
1 comment
AuthorsOpenReview ↗

Response to Reviewer yYBf

We thank the reviewer for the review, and we apologize for the late response: we ran additional experiments and analyzed them carefully.

W1: Novelty relative to RULE.

RULE summarizes each method's trade-off curve by its area; we compare methods at the same forgetting level, and every point we compare at is attained by an actual system. RULE is an unlearning method for text-only LLMs, and it supports its claim with Pareto plots and the area under them. An area mixes all forgetting levels into one number, so it cannot say which method retains more when both must forget a fact to the same degree, which is the requirement in deployment. We answer that question directly. Answering each query with one of two adjacent checkpoints, with a closed-form probability, reaches any target forgetting level without retraining or extrapolation. This changes conclusions: at their common default setting, several established objectives have the best retention only because they forget nothing, and they fall well behind refusal training once forgetting is matched.

Two further parts have no counterpart in RULE. A retention difference in a results table splits into movement along a reference frontier, which comes from forgetting less, and a shift of that frontier, which is a real improvement. In the multimodal setting, matching forgetting within each modality reveals that a fact forgotten through the portrait still leaks through the name, as we show in our answer to W2.

W2: Do the findings hold beyond refusal training?

Yes. The multimodal finding, that a fact forgotten through one modality still leaks through the other, holds across objectives and models. Each fact in our benchmark is queried both with the person's portrait and with the person's name in text only. For every objective and model, we take the point where the portrait question recovers half of the facts, and report how often the name question still recovers them (name recall).

ModelObjectiveName recall
InternVL2.5-8BAnswer-only77
InternVL2.5-8BNPO72
InternVL2.5-8BGradient difference66
InternVL2.5-8BKL minimization55
LLaVA-1.5-13BAnswer-only58
Qwen2.5-VL-3BAnswer-only58
LLaVA-1.5-7BAnswer-only50

On InternVL2.5-8B, every objective leaves the fact more recoverable through the name than through the portrait, so measuring forgetting through one modality overstates it. The size of this gap depends on the model, which is why a multimodal evaluation should trace the frontier for each modality and compare methods at matched forgetting within it. This dimension does not exist for text-only models. Gradient ascent is not listed because it has no intermediate checkpoint between no forgetting and collapse.

Q1: Labels in Figure 7(a).

Every curve in Figure 7(a) will carry a method label, and a table will list each objective's retention at matched forgetting.

Q2: Frontiers built from a few checkpoints.

The question appears to be cut off on OpenReview. We read it as asking whether a frontier from a few checkpoints is reliable: whether the checkpoints are chosen on a principled basis and whether they cover the shape of the curve.

The checkpoints are fixed in advance, span the whole range from no forgetting to strong forgetting, and every point between them is attained by an actual system. For refusal training, the strengths are set before any scoring, from zero to full strength with the nonzero strengths on a roughly geometric scale, so that the sweep runs from the parent's level, through the knee, to the point where most facts are forgotten. For the newly added objectives, whose forgetting changes abruptly, we placed extra checkpoints inside the range where it changes, chosen from forget scores only. Between two adjacent checkpoints, answering each query with one of them realizes every point on the connecting segment in expectation, so the frontier involves no fitted curve, and adding checkpoints can only add attainable points.

Reviewer_WoSqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that unlearning methods should be compared at matched forgetting. It traces a forget–retain frontier by sweeping a strength parameter, reaches intermediate levels by randomly mixing two adjacent checkpoints per query, and splits a retention gain into movement along a reference frontier and a shift of it. On six MLLMs, it finds that every frontier has a knee, that look-alike retain sets and name cues lower the frontier, and that training choices and objectives mostly move runs along it.

Critical strengths
  1. The question is a useful one. Single-checkpoint tables mix up "retains more" and "forgets less", and the fix costs only a small strength sweep per method. It changes conclusions too: format-weighted training has the best reported retention on InternVL2.5-8B (Table 15) yet sits 18 pp below the answer-only frontier at its own forgetting level, and 25 pp below on LLaVA-1.5-7B (Table 12).

  2. The experiments are well controlled (parent knowledge checked, a held-out cohort, paired runs, identity-bootstrap intervals, three seeds), and unfavorable results are reported (App. F, G).

Critical weaknesses
  1. I don't think the data support the abstract's claim that "the algorithm chiefly picks a point on" the frontier. "Mostly along" is a share of the table difference. The along term is fixed by the two forgetting levels and the reference frontier, so the share can be large even when the shift is not small. Fig. 7b puts the objective's shift at about 23–25 pp, close to the relatedness effect (26–36 pp). At 50% forget occurrence, retention across the seven established objectives ranges from 78.9 (GA) to 92.8 (SMFA) on InternVL2.5-8B (Table 15). The low rank agreement (τb 0.36/0.40) is driven by the authors' format-weighted variant; without it the printed values give τb ≈ 0.78/0.71. Objectives seem to matter at matched forgetting.

  2. Wang et al. (ICLR 2025, "Towards Effective Evaluations and Comparisons for LLM Unlearning Methods") already compare methods at a calibrated common level, using weight mixing to fix retention, and are not cited. The name-cue leak in Sec. 4.4 was reported by UMU-Bench; what is new here is measuring it at matched forgetting. Per-query mixing and the along/shift decomposition appear to be the main new parts.

  3. The setup behind these general conclusions is fairly narrow: facts injected by fine-tuning, two attributes, 50 queries per cohort, four sweep points, a string-occurrence scorer, and two models for the factor studies. The format effect did not transfer to the public recipe (App. G).

Questions for authors
  1. How far apart are the seven established objectives at τ = 25%? Fig. 7a suggests the frontiers spread out there, and the answer bears on W1.

  2. Does the Table 15 ranking hold under the two LLM judges from App. F? Occurrence misses rewordings, and gradient ascent and refusal training may fail differently.

  3. The along/shift split depends on the reference frontier R⋆: Sec. 3 uses a reference procedure's frontier for two objectives and A's frontier for two conditions. How much do the shares in Fig. 7b change with another choice, e.g. B's frontier, or NPO's instead of answer-only? If they move a lot, sorting the factors into "along" and "shift" is less firm than Sec. 4.5 suggests.

1 comment
AuthorsOpenReview ↗

Response to Reviewer WoSq

We thank the reviewer for the review, and we apologize for the late response: we ran additional experiments and analyzed them carefully.

W1 and Q1: Do objectives only pick a point on the frontier?

A single checkpoint is not enough to judge an unlearning objective; comparing whole frontiers at matched forgetting shows that objectives differ substantially. We swept the learning rate of gradient ascent (GA), gradient difference (GD), KL minimization (KL) and NPO on InternVL2.5-8B with three seeds, using the same data, LoRA setup and readout as answer-only training. Each objective jumps from almost no forgetting to near-total collapse within a narrow range of learning rates, so we added points inside that range, chosen from forget scores only. We compare objectives by their retention at a fixed forgetting level, which measures the difference in points rather than as a share.

ObjectiveRMF@50RMF@25
Answer-only87.854.2
GD72.3not reached
KL59.629.8
NPO50.925.2
GA50.325.2

Retention differs widely across objectives at both forgetting levels. One checkpoint per method would have told a different story: at the common default learning rate, GD, KL and NPO keep full retention because they forget nothing, while GA forgets everything and keeps nothing. The objective therefore shifts the frontier, and tracing the frontier is what measures by how much.

W2: Relation to Wang et al. (ICLR 2025) and UMU-Bench.

Wang et al. fix retention and compare forgetting; we fix forgetting and compare retention, with exact matching and a decomposition. Their UWC framework mixes the weights of the original and the unlearned model to bring methods to a common retention level. Our protocol fixes forgetting instead, which matches the requirement that a fact be forgotten to a stated level. It mixes two adjacent checkpoints per query rather than in weight space, so the matching probability has a closed form and needs no search. It also splits a retention gain into movement along a reference frontier and a shift of that frontier. UMU-Bench first reported the name-cue leak; our contribution is to measure it at matched forgetting.

W3: Scope of the setup.

The evaluation protocol is independent of the model, and its conclusions hold across model families, scorers and objectives. Iso-forget evaluation needs only a strength parameter and scores that average over a fixed set of queries, which every model and benchmark provides. Its extra cost is a few strength settings per method, and matching is done afterwards by mixing existing checkpoints per query, with no retraining. This makes it more useful, not less, for larger models, where each training run is expensive.

Across InternVL2.5-8B, LLaVA-1.5-7B and the newer Qwen2.5-VL-3B, every answer-only frontier shows the same knee: forgetting drops first while retention stays close to the parent, and retention falls only at stronger settings. The shape and the ranking of methods are unchanged under two LLM judges, and the gaps between objectives at matched forgetting are clear on every scorer. We also traced the frontier on LLaVA-1.5-13B, the larger model of the same family as LLaVA-1.5-7B. It shows the same knee, and its frontier lies above that of the 7B model, so moving to a larger model changes the level of the frontier but not its shape.

The forget and retain queries come from the public UMU-Bench benchmark. We inject the facts by fine-tuning so that we can verify the parent knows every fact before unlearning; without this step, forgetting cannot be measured.

Q2: Does the ranking hold under the LLM judges?

Yes, the ranking is identical under both judges. We re-scored every run with two open LLM judges, Qwen3-8B and Mistral-Nemo, which rate agreement with the reference from 0 to 10 and give faithful paraphrases full credit. The ranking of the five objectives at RMF@50 matches the occurrence ranking under both judges, and retention at matched forgetting changes little.

Q3: Sensitivity to the reference frontier R⋆.

The comparison of objectives does not depend on R⋆. We compare objectives by each one's own retention at a fixed forgetting level, as in the table above, which needs no reference frontier.

Comments on the paper

1 comment

Dear Reviewers,

The Author-Reviewer Discussion period is coming to an end soon. Please actively participate in the rebuttal discussion with the authors after carefully reading the authors' responses.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.