acceptodds
Under review as a conference paper at ICLR 2027

RouteMask: Unlearning Identities Along Access Routes in VLMs

Abstract

Vision–language models (VLMs) learn from large collections of web images and text and are now widely used in assistants and search. Some of what they learn concerns real people: a single photograph can be enough for a model to reveal a person's birthplace, occupation, or education. Machine unlearning aims to remove such information after training. Existing methods for VLMs fine-tune the model with gradient ascent, preference optimization, or refusal targets, but they mostly change what the model says, not what it knows. After refusal training, the model still picks the true answer in multiple choice, and forgetting trained with names does not carry over to photographs. We address this in two steps. First, we locate where identity is accessed: injecting name states into frozen models while blocking attention to the image reveals an entity state that faces and names share in middle layers, and a visual bypass through which late layers read the image again. Second, we propose RouteMask, which updates only the MLP gate rows in these layers. At each step, a row mask compares the attribution of the current forget answers with fixed retain and perception profiles and scales the parameter update, and every forget term is paired with a retain term under both name and image cues. Experiments on MLLMU-Bench and S-MLLMUn with LLaVA-1.5 and Qwen2.5-VL show that both cues must be supervised and that forget and retain accuracy can only be separated when updates are restricted in this way.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal unlearning, machine unlearning, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Q7AwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multimodal machine unlearning and identifies the limitations of existing works, behavioral forgetting and the inconsistency of unlearning across textual and visual identity cues. The paper develops RouteMask which selectively updates MLP gate rows using attribution-guided masks and presents the empirical evaluation across two VLMs (LLaVA-1.5 and Qwen2.5-VL) and two benchmarks (MLLMU-Bench and S-MLLMUn).

Critical strengths

S1. The paper focuses on both factual knowledge and cross-modal identity access.

S2. The proposed method appears compelling as it edits only MLP gate rows within layers identified by the mechanistic investigation.

Critical weaknesses

W1. The authors argue that existing methods primarily change what a model says rather than what it knows. However, the developed RouteMask also optimizes behavioral objectives, including changes in answer probabilities and desired forgetting responses. Restricting parameter updates does not necessarily establish that the underlying identity information has been eliminated. The model may still retain forgotten facts in representations or parameter regions not modified by RouteMask.

W2. More datasets such as FIUBench should be included in the evaluation. The paper also needs to discuss whether the proposed approach could generalize to other settings than forgetting personal facts associated with identities that can be referenced through names or photographs.

Questions for authors

W1 and W2 in the above weakness section.

Reviewer_PsGUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates identity unlearning in vision–language models and observes that refusal training can leave factual recognition intact, while forgetting learned from names transfers poorly to photographs. Through name-state injection and visual-attention blocking, the paper identifies a shared identity representation in middle layers and a visual bypass through which later layers access identity information from the image again. To address these access routes, the paper proposes RouteMask, which restricts updates to MLP gate rows in the identified layers. A training-only mask, recomputed for each forget example, scales parameter updates by comparing forget-answer attribution with fixed retain and perception profiles, while paired forgetting and retention objectives cover both name and image cues. Experiments on MLLMU-Bench and S-MLLMUn with LLaVA-1.5 and Qwen2.5-VL indicate improved forgetting–retention trade-offs, with lower forget factuality on MLLMU-Bench.

Critical strengths
  • Clear problem framing with mechanistic observations. The paper studies multimodal machine unlearning, focusing on the observation that suppressing model outputs does not necessarily prevent the retrieval of targeted information. The distinction between name-based and image-based identity access is noted and analyzed. Furthermore, the causal intervention experiments suggest that identity-related information may be accessed through multiple computational pathways, which leads to the proposed selective editing approach.

  • Control experiments supporting selective parameter updates. In addition to standard baselines, the evaluation includes control settings such as location-only, step-matched uniform, and shuffled-mask updates. These comparisons suggest that the choice of specific updated rows has an effect beyond simply altering update magnitude. For instance, on MLLMU-Bench under image cues, RouteMask obtains a forget factuality score of 1.16, compared to 3.52 and 3.71 for the step-matched uniform and shuffled controls, respectively. These results provide reasonable empirical support for attribution-guided row selection.

Critical weaknesses
  • Although the paper includes several ablation studies, the contributions of its key components remain unclear. It lacks comparisons of entity-band-only, re-entry-only, and combined updates under the full RouteMask objective, as well as a direct comparison between dynamic and fixed attribution-based masks. These omissions make it difficult to isolate the sources of improvement.

  • The S-MLLMUn evaluation omits several important metrics from the original benchmark, including Meaningful Score, ROUGE-L, and aggregate forgetting-retention metrics. This is concerning because low forgetting scores may reflect degenerate or incoherent outputs rather than successful selective unlearning.

  • Comparisons with more recent multimodal unlearning methods under consistent evaluation protocols would further strengthen the experimental evidence.

Questions for authors
  • Could the authors explain the discrepancies between the SMFA results in Table 5 and those reported in the original SMFA paper (including the differences in the Original model scores)?
1 comment

Thank you for your rebuttal, which has largely addressed some of my concerns. I will raise my score to 3.

Reviewer_4HW7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the unlearning of personal facts about fictitious identities in VLMs, where a fact can be queried either by name or by photograph.

First, it localizes how identity reaches attribute extraction, using interventions on frozen benchmark checkpoints: at layer ℓ\ell, BB's name state is injected at the last question token of AA's photo prompt, and attention to the image tokens is blocked in all later layers. This reveals two regions:

  • an entity band of middle layers, where the injected name state can replace the photographed identity;
  • a re-entry window of later layers, where the model reads identity from the image again (a "visual bypass").

Second, it proposes RouteMask, which fine-tunes only the MLP gate rows Wg,ℓW_{g,\ell} in these layers.

  • At step tt, the AdamW proposal ut,ℓu_{t,\ell} is scaled row-wise: Wg,ℓt+1=Wg,ℓt+Mℓ(Bt)⊙ut,ℓW_{g,\ell}^{t+1} = W_{g,\ell}^{t} + M_\ell(B_t) \odot u_{t,\ell}.
  • The mask compares the gradient ×\times activation attribution of the current forget answers with fixed retain and perception profiles: mc,ℓ(x)=a~c,ℓ(x)/(a~c,ℓ(x)+r~c,ℓ+p~c,ℓ)m_{c,\ell}(x) = \tilde a_{c,\ell}(x) / (\tilde a_{c,\ell}(x) + \tilde r_{c,\ell} + \tilde p_{c,\ell}).
  • The objective pairs each forget term (refusal; flattening the likelihood over candidate answer contents) with a retain "mirror", and adds a multiple-choice format term and a perception KL term.

Experiments on MLLMU-Bench (LLaVA-1.5-7B) and S-MLLMUn (Qwen2.5-VL-7B) report three main results:

  • the lowest forget factuality on MLLMU-Bench;
  • improvement over SMFA on all six S-MLLMUn scores;
  • more forgetting than uniform or shuffled updates of matched norm.
Critical strengths

1. A careful localization study with a useful practical finding.

  • The frozen-model interventions are well controlled:
    • noise of equal norm;
    • injecting AA's own name as a positive control;
    • an untrained base model as the reference for cross-modal alignment;
    • candidate sets restricted by gender and birth cohort.
  • The re-entry criterion was fixed before the curves were inspected.
  • Together with Table 8, which shows that forgetting trained under one cue barely transfers to the other, this gives a clear and actionable message: multimodal unlearning should be supervised and evaluated under both name and image cues.

I expect this finding to be useful beyond this specific method.

2. The mask is tested against strong, matched controls.

  • Table 3 compares RouteMask with uniform and shuffled updates whose per-layer update norm is matched step by step.
  • At equal norm, RouteMask reaches an image-cue forget Fact of 1.161.16, against 3.523.52 and 3.713.71 for the controls, with similar retain scores.
  • Such controls are rare in the unlearning literature. They convincingly show that the choice of rows matters, not only the step size.

This is the strongest evidence for the method.

Critical weaknesses

1. The "knows vs. says" claim would benefit from knowledge-level evidence.

The paper is motivated by the observation that existing methods change what the model says rather than what it knows. However, the evaluation of RouteMask is behavioral, and its main forget term is refusal.

  • The edited model assigns log⁡p(I don’t know.∣x)≈−0.09\log p(\text{I don't know.} \mid x) \approx -0.09 to forget questions (App. C).
  • The paper's own unmasked refusal-only run reaches nearly the same forget Fact as RouteMask (1.201.20 vs. 1.161.16, Table 6), because the judge gives refusals the minimum score.
  • Multiple-choice accuracy is the main measure that refusal does not affect. On this 4-way task (chance 25%) it drops only from 46.9% to 37.4%, so part of the above-chance knowledge seems to remain.
  • Since only the gate rows WgW_g are edited and WuW_u, WdW_d stay unchanged, it is natural to ask whether the facts are still recoverable.

I see two simple ways to address this:

  • report a knowledge-level measure; the forced-prefix scoring of Sec. 4, applied to the edited models, would suffice;
  • or reframe the claim as more selective behavioral forgetting.

Either option would largely resolve this concern.

2. The contribution of the localization is not yet separated from the objective.

Table 3 isolates the effect of the mask well. However:

  • The gains over external baselines (Tables 4–5) combine the mask with a richer objective.
  • No experiment shows that the identified layers work better than other layer sets of the same size.
    • This matters because the eligible set covers 19 of Qwen's 28 layers, and for LLaVA it is a single contiguous block (layers 5–17).
    • As a result, the link between the localization analysis (Sec. 4) and the method is currently argued rather than tested.

Two additions would make both contributions considerably more convincing:

  • placing the refusal-only run next to RouteMask in the main tables;
  • adding one layer-set comparison.
Questions for authors

Main questions (these could change my assessment):

  1. Knowledge-level check. Could you apply the forced-prefix scoring of Sec. 4 to the RouteMask models and compare forget and retain facts? For example, report the rank or log-probability of the true value after "This person was born in". This reuses existing infrastructure and requires no retraining. If this is not feasible, would you consider rephrasing the claim as behavioral forgetting?

  2. Data separation (clarification).

    • Do the MLLMU-Bench multiple-choice options overlap with the candidate sets ArA_r used by the Recognize term?
    • Are the 900 development questions (App. F) disjoint from the evaluation set in Table 5?

    A short clarification is sufficient.

  3. Localization vs. other layers. Could you add one layer-set comparison under the full RouteMask setup on one benchmark? Possible variants are the entity band only, the re-entry window only, or random layers of the same count. A single seed is fine.

  4. Uncertainty (clarification). App. A describes bootstrap 95% intervals. Could you report them for Tables 3–5?

If questions 1 and 2 are addressed satisfactorily, I would be inclined to raise my score. Question 3 would further strengthen the paper.

Minor suggestions (no need to address in the rebuttal):

  • Report refusal rates on retain and perception questions.
  • Briefly describe how the baselines were tuned. GA-Diff collapses on S-MLLMUn.
  • Clarify how the LLaVA pairs were selected (App. B vs. Sec. 4).
  • Unify the definition of the re-entry window between Sec. 4 and App. B.
  • In Eq. (2), ℓ\ell denotes both the layer index and the loss ℓ(x,y)\ell(x, y); please use separate symbols.
  • State in the caption of Figure 1a that the refusal model is interpolated (α=0.5\alpha = 0.5).
  • Possible future work: relearning and quantization attacks, general VLM benchmarks, and larger models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.