acceptodds
Under review as a conference paper at ICLR 2027

MIRRA: Identity-Routed Refusal Adaptation for Unlearning in MLLMs

Abstract

Multimodal large language models (MLLMs) can memorize personal information and reveal it through text and image queries, motivating multimodal machine unlearning to protect individual privacy. However, existing methods that modify shared model parameters can degrade performance on unrelated queries, while refusal training alone does not ensure that target individuals are consistently recognized across modalities. To address these limitations, we propose MIRRA, a multimodal unlearning framework that separates identity recognition from refusal training. The key idea is to use identity recognition to determine when to activate a lightweight refusal adapter. MIRRA combines textual identity matching with face verification using a single reference image per identity, drawing on both forget and retain identity prototypes to distinguish target individuals from similar non-targets. A match in either modality activates the adapter to refuse the query, while other queries are handled directly by the base model. Experiments on MLLMU-Bench and CLEAR across three MLLMs show that MIRRA effectively suppresses responses about target individuals while preserving model utility. On MLLMU-Bench, MIRRA reduces forget-set classification accuracy to 0.42–1.25%, with retain-set classification accuracy decreasing by at most 1.23 percentage points relative to the original models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal large language models, machine unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FZRCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MIRRA, a multimodal machine unlearning framework that separates target-identity recognition from refusal generation. MIRRA identifies whether a query refers to a target identity through text matching or face verification; target queries activate a LoRA refusal adapter, while non-target queries are handled by the base model. The method is evaluated on MLLMU-Bench and CLEAR with three MLLMs and compared against several unlearning baselines. Results show that MIRRA strongly suppresses responses about target identities while largely preserving performance on retained and general queries. Ablation studies further confirm that multimodal identity routing and the refusal adapter are both important for effective selective forgetting.

Critical strengths
  1. Simple and modular design. By separating target identification from refusal generation, MIRRA avoids applying the forgetting mechanism indiscriminately to all inputs. This modularity makes the method easy to interpret and helps explain why non-target utility is largely preserved.

  2. Effective multimodal target identification. The combination of text matching and face verification allows MIRRA to recognize forget identities from either textual or visual evidence. This is important for multimodal unlearning because the same identity can be referenced through different modalities.

  3. Good control of false refusals. The use of both forget and retain identity prototypes helps reduce incorrect activation of the refusal mechanism on non-target individuals. This is consequential because excessive refusal would undermine the practical usefulness of the system.

Critical weaknesses
  1. The method depends heavily on the robustness of the identity router. If the text or face gate fails to recognize a target identity, the query bypasses the refusal adapter and is answered by the base model. This creates a direct failure mode in which imperfect routing can lead to information leakage. The ablation results themselves show that removing either gate substantially increases leakage.

  2. The evaluation does not sufficiently test adversarial or evasive queries. The paper evaluates paraphrased questions and altered portraits, but it does not appear to systematically test whether users can bypass the router through name obfuscation, indirect references, adversarial images, or other routing attacks. This matters because MIRRA’s guarantees depend on reliable target detection at inference time.

  3. The benchmark setting is relatively narrow. The main experiments use the 5% forgetting setting on MLLMU-Bench and CLEAR. It remains unclear whether the strong forget-retain trade-off would hold when a much larger fraction of identities must be forgotten.

  4. The paper does not fully characterize failure cases of false-positive routing. The sensitivity analysis shows that increasing the routing FPR improves forgetting but reduces retained utility, indicating a non-trivial trade-off. However, the paper provides limited qualitative analysis of which non-target identities are most likely to be incorrectly routed and refused, which would be important for understanding practical reliability.

Questions for authors

How robust is the identity router to evasive or adversarial references to a forget identity, such as name obfuscation, indirect textual descriptions, heavily altered faces, or adversarial image perturbations? Because routing failure directly exposes the base model response, this could materially affect the claimed forgetting guarantees.

5 comments
AuthorsOpenReview ↗

Response to Reviewer FZRC(Part 1)

Thank you for your comments. We address each concern below.

W1&W2&Q1: Robustness of the identity router to evasive and adversarial queries

Weaknesses 1 and 2 and Question 1 concern the same fundamental issue that whether the identity router can still activate the refusal path when a forget identity is presented through evasive, indirect, or perturbed inputs. We agree that if both the text and face gates fail to recognize the target identity, the query may bypass the refusal adapter. Routing failure therefore constitutes a direct failure mode of MIRRA. To address this concern, we conducted a robustness evaluation.

Specifically, we constructed 11 types of controlled identity-preserving perturbations using the existing benchmark samples, resulting in 130 test queries. The text-side tests include name capitalization, punctuation changes, minor spelling errors, aliases, and adversarial text prompts. The image-side tests include indirect visual references without explicitly providing the identity name, eye-region occlusion, lower-face occlusion, half-face occlusion, strong blur, and adversarial image prompts. All visual transformations were applied directly to existing test images and did not rely on newly generated face images. In addition, the original test sets of MLLMU-Bench and CLEAR already contain natural variations in viewpoints, backgrounds, and clothing relative to the identity-enrollment images.

We evaluate the router using two metrics. Routing Recall measures the proportion of target queries that successfully activate the refusal path, while Answer Leakage measures the proportion of queries whose final responses disclose information about the target identity. The results are summarized below.

Test scenarioQueriesRouting Recall ↑Answer Leakage ↓
Name capitalization10100%0%
Name punctuation10100%0%
Minor name typo10100%0%
Alias10100%0%
Indirect visual reference10100%0%
Eye-region occlusion10100%0%
Lower-face occlusion10100%0%
Half-face occlusion10100%0%
Strong blur10100%0%
Adversarial text prompt20100%0%
Adversarial image prompt20100%0%
Overall130100%0%

Across all 130 evaluated queries, the complete dual-gate router achieves 100% routing recall and 0% answer leakage.

These results also clarify the gate-ablation findings. The text and face gates are complementary rather than redundant. Queries containing a name or an enrolled alias are primarily detected by the text gate, whereas image-based queries that do not explicitly mention the identity are handled by the face gate. MIRRA adopts a privacy-oriented OR fusion rule: the refusal adapter is activated whenever either gate detects a forget identity.

We include these results in Section 5.7 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer FZRC(Part 2)

W4: Qualitative analysis of false-positive routing

To address this concern, we conducted an identity-level audit of all false-positive routing decisions at the operating point used in our main experiments.

Among 11,280 non-target queries, 37 are incorrectly routed to the refusal adapter, corresponding to a query-level false-positive rate of 0.328%. These errors affect only 5 of the 628 non-target identities (0.80%). All five identities belong to the Retain set, while no false positives are observed among the 153 Real Celebrity identities. Of the 37 false-positive queries, 19 are triggered by the text gate and 18 by the face gate.

The audit identifies two recurring failure modes. Text-gate errors are caused by lexical name collisions: \emph{Eliza Thornton} is confused with the forget identity \emph{Emilia Thornton}, \emph{Lena Fitzgerald} with \emph{Elena Fitzgerald}, and \emph{Marco Delgado} with \emph{Mariana Delgado}. Face-gate errors are concentrated in two retained identities, \emph{Lucia Martinez} and \emph{Kimberly Nguyen}, whose face representations exceed the routing threshold because of their similarity to the forget prototypes of \emph{Sofia Delacroix} and \emph{Camila Vermeer}, respectively.

Triggering gateNon-target identityMatched forget identityFP queries
TextEliza ThorntonEmilia Thornton9
TextLena FitzgeraldElena Fitzgerald9
TextMarco DelgadoMariana Delgado1
FaceLucia MartinezSofia Delacroix9
FaceKimberly NguyenCamila Vermeer9
Total5 identities—37

These results show that false positives mainly occur when a retained identity has a name or face similar to a forget identity. Therefore, increasing the router sensitivity may also cause these similar non-target identities to be incorrectly refused. We have added this analysis to clarify when false-positive routing is most likely to occur.

We have added this analysis in Section 5.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer FZRC(Part 3)

W3: Scalability to larger forget sets

Thank you for pointing out the limited forgetting scale in our original evaluation. To address this concern, we added experiments with 10% and 15% forgetting ratios on MLLMU-Bench. We evaluated MIRRA and baselines using the corresponding Forget and Retain partitions. The complete results are shown below.

Qwen3-VL-4B-Instruct (10% Forget)

MethodForget Cls. ↓Forget R-L ↓Forget Cloze ↓Retain Cls. ↑Retain R-L ↑Retain Cloze ↑Real Celebrity Cls. ↑Real Celebrity R-L ↑Real Celebrity Cloze ↑
Vanilla58.780.60212.7658.580.6018.6373.680.52312.58
GA58.370.53811.7358.740.5288.3572.050.47210.95
GA-Difference58.160.51113.7853.430.52510.7569.760.43311.44
KL Minimization60.820.61414.2960.840.6018.6372.960.53312.09
NPO59.590.46511.7358.740.4638.2474.200.45310.95
MIRRA (Ours)1.020.0000.0058.960.6018.8073.220.52113.07

Qwen3-VL-4B-Instruct (15% Forget)

MethodForget Cls. ↓Forget R-L ↓Forget Cloze ↓Retain Cls. ↑Retain R-L ↑Retain Cloze ↑Real Celebrity Cls. ↑Real Celebrity R-L ↑Real Celebrity Cloze ↑
Vanilla58.110.60010.4760.890.60110.7973.680.52312.58
GA57.840.4945.0760.700.5035.9073.220.46011.27
GA-Difference50.410.5348.7853.610.54310.4467.860.42310.78
KL Minimization56.620.60911.4959.630.6119.5572.630.52812.58
NPO59.860.5415.7460.490.5498.1472.240.48011.27
MIRRA (Ours)1.890.0100.0060.700.59911.2073.020.52412.42

At both forgetting ratios, MIRRA reduces the Forget metrics to nearly zero while maintaining Retain and Real Celebrity performance close to Vanilla. These results demonstrate that MIRRA preserves its forget--retain trade-off as the number of forget identities increases. We have added these experiments and the corresponding discussion to Section 5.8 and Table 5 of the revised manuscript.

Reviewer_UsDYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MIRRA, which uses an identity router to determine whether an input involves a person targeted for unlearning and activates a LoRA refusal adapter when a match is detected. The method separates target identity recognition from refusal generation, using two gating branches: text-based identity matching and single-image face recognition. Experiments are conducted on MLLMU-Bench and CLEAR.

Critical strengths

1.Personal information can leak through both text and image inputs in multimodal models. Updating shared model parameters alone can damage knowledge unrelated to the target identities.

2.MIRRA separates identity recognition, routing, and refusal generation. In principle, this design can avoid highly destructive updates to the entire model.

3.Removing either the text gate or the visual gate reduces target recall, while removing the refusal adapter substantially increases leakage. These results suggest that the routing and refusal modules serve distinct functions.

Critical weaknesses

1.MIRRA is a guardrail rather than an unlearning method. It does not remove knowledge from model weights; it intercepts requests at inference time. When a user requests deletion of their data, GDPR requires removal at the weight level, rather than refusal to answer while retaining the knowledge. MIRRA therefore addresses a scenario closer to content moderation than the right to be forgotten.

2.The comparison with the baselines does not use equivalent information conditions. MIRRA receives additional external supervision unavailable to GA, NPO, MMUnlearner, and MIP-Editor: a complete list of names targeted for unlearning, one reference face per target identity, and reference faces for every retain identity to construct the prototype bank. This effectively provides explicit annotations of the boundary between forget and retain identities.

3.The baselines, by contrast, must infer this boundary from the forget data. Comparing a routing system with explicit knowledge of the boundary against weight editing methods that rely on gradient signals to reduce likelihood makes forget accuracy difficult to interpret. A substantial part of the performance gap may arise from unequal access to information rather than methodological superiority.

More fundamentally, routing requests to a fixed refusal template will almost inevitably drive classification accuracy toward zero. This demonstrates that the classifier works, rather than that the knowledge has been forgotten.

Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer UsDY

Rebuttal

Thank you for your comments. We address each concern below.

W1 & W3.2: Scope of MIRRA and weight-level erasure

MIRRA aims to reduce the disclosure of information about designated forget targets through a controlled multimodal inference interface while preserving the model’s ability to answer other queries. Specifically, MIRRA intervenes selectively based on identity, so the system can withhold information about designated individuals while still answering the same kinds of questions about other people.

Even after model weights have been updated, it remains difficult to verify whether the target information has been erased. Hu et al. [1] and Xu et al. [2] found that limited subsequent fine-tuning could restore some response behaviors that appeared to have been unlearned. In some experiments, the fine-tuning data did not explicitly include the target content. These results show that suppressing target responses after a weight update is insufficient evidence of durable erasure. In retrieval-augmented systems, the generator can also obtain the same information from external documents [8].

Other work has explored ways to suppress target information at inference time without updating the base model’s weights. ICUL changes model behavior through in-context examples using only query access [3]. SPUL learns soft prompts while keeping the base model frozen [4]. During generation, ULD uses an auxiliary model to adjust the target model’s logits [5], while SEGUE suppresses the output of target knowledge through entropy-guided decoding [6]. MIRRA studies selective disclosure control for text and image queries involving personal identities.

W2 & W3.1: Use of Forget and Retain data

In the MLLMU-Bench 5% setting [8], MIRRA uses the names of all 25 forget identities for text routing. In the verified visual-gate configuration, MIRRA uses one reference image per identity for all 25 forget identities and 79 of the 475 retain identities. The retain images form a separate prototype bank. Comparing a query image's similarity to both banks helps reduce false routing when a non-target person resembles a forget identity.

The Retain set is used to limit degradation in model utility during unlearning. GA-Difference and KL-Min use examples from both the Forget and Retain sets in their optimization objectives. We use a subset of the Retain split for the same purpose of preserving utility on non-target queries. In MIRRA, these examples provide reference faces for the visual gate, helping it distinguish retained individuals from forget targets and avoid unintended refusals.

The results below show that MIRRA suppresses correct answers to target queries while largely preserving performance on Retain and Real Celebrity sets. The Real Celebrity set isn't used to train for all evaluated methods. And MIRRA can perform well on the dataset.

Main-table results for Qwen3-VL-4B-Instruct (5% Forget)

SplitMetricVanillaGA-DifferenceMIRRA
Forget ↓Cls.58.3353.330.83
ROUGE-L0.6100.3970.000
Cloze12.5013.540.00
Retain ↑Cls.58.9451.6858.00
ROUGE-L0.6010.4630.601
Cloze9.8611.5012.76
Real Celebrity ↑Cls.73.6865.8473.29
ROUGE-L0.5230.3900.523
Cloze12.5813.0712.42

References

[1] Hu et al. [Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning]. ICLR 2025.

[2] Xu et al. [Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs]. ICML 2026.

[3] Pawelczyk et al. [In-Context Unlearning: Language Models as Few-Shot Unlearners]. ICML 2024.

[4] Bhaila et al. [Soft Prompting for Unlearning in Large Language Models]. NAACL 2025.

[5] Ji et al. [Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference]. NeurIPS 2024.

[6] Pu et al. [Decoding-Unlearning: Fact Forgetting via Entropy-Guided Inference] ACL 2026.

[8] Lewis et al. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks]. NeurIPS 2020.

Reviewer_qXzAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MIRRA, a multimodal identity-routed refusal framework for machine unlearning in MLLMs. It combines textual identity matching and face verification using forget/retain identity prototypes to identify target individuals. When either gate detects a target, a LoRA adapter trained on refusal responses is activated; otherwise, queries are processed by the retain-adapted base model. Experiments across 3 MLLMs demonstrate substantial reductions in target-answer accuracy while largely preserving retained performance. However, the approach primarily implements conditional refusal rather than explicitly removing learned information.

Critical strengths
  1. MIRRA achieves substantial reductions in forget-set performance across three MLLMs, while maintaining most retain-set metrics close to the original models. Experiments on two benchmarks provide evidence of the practical effectiveness of selective refusal.
  2. Separating identity detection from refusal generation is a practical design that limits unnecessary interventions on non-target queries. The ablations demonstrate the complementary contributions of textual and visual identity matching.
  3. Ablations demonstrate that text and face gates contribute substantially to target detection. The use of forget/retain identity prototypes also helps reduce false-positive routing.
Critical weaknesses
  1. The approach combines established identity verification, threshold-based routing, and LoRA refusal training. Conditional inference-time intervention and refusal-based unlearning have already been explored in works such as GUARD (ICMLW 2025) [1], MLLMEraser (arxiv 2025) [2], and ASRU (ICML 2026)[3]. Although the specific multimodal identity-routing combination is useful, the paper does not sufficiently establish its incremental contribution over these approaches.

  2. The experiments omit closely related inference-time multimodal unlearning approaches, particularly MLLMEraser [2] and ASRU [3]. More importantly, a simple identity-router-plus-fixed-refusal baseline is missing. Without this comparison, it is unclear whether training a dedicated refusal adapter offers meaningful benefits beyond identity detection and deterministic response suppression.

  3. MIRRA does not explicitly remove target knowledge from model parameters; it conditionally activates a refusal adapter when a target identity is recognized. Consequently, forgotten information may remain accessible when routing fails or is bypassed. This distinction is important given prior evidence that apparently forgotten knowledge can remain recoverable (Hu et al. [4]).

  4. he method depends heavily on accurate identity recognition, yet experiments do not adequately assess indirect identity references, aliases, multi-person images, occluded faces, or adversarial prompts. These are important failure cases because a missed identity match can allow target information to be disclosed through the ordinary generation path.

  5. Although aggregate retention results are promising, Real Face accuracy decreases by approximately 18–20 percentage points for the Qwen models. These substantial declines are not sufficiently analyzed and weaken the claim that MIRRA consistently preserves general utility.

  6. The evaluation focuses on 5% forgetting settings and identity-centered benchmarks. It remains unclear how the framework performs with substantially larger forget/retain identity banks, more visually similar individuals, or repeated unlearning requests. Its dependence on enrolled reference images also limits applicability to identities without suitable visual references.

References

[1] Deng et al. (2025). GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection. ICMLW 2025

[2] Ding et al. (2025). MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering. arxiv

[3] Guang et al. (2026). ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models. ICML 2026

[4] Hu et al. (2025). Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. ICLR 2025

Questions for authors
  1. How does MIRRA differ from identity-based access control, given that the underlying model may retain the designated information? What evidence establishes unlearning beyond conditional refusal?
  2. Why are closely related multimodal approaches such as MLLMEraser and ASRU not included in the experimental comparisons? How would MIRRA compare under matched evaluation settings?
  3. How does MIRRA handle indirect identity references, aliases, or multiple faces in an image? Can the authors report target-answer leakage specifically when routing fails?
  4. Additionally, what explains the substantial degradation in CLEAR Real Face accuracy for the Qwen models? How much is attributable to retain adaptation versus routing errors?
  5. How do false-positive rates, target detection, and inference costs change as the number of forget and retain identities increases?
4 comments
AuthorsOpenReview ↗

Response to Reviewer qXzA(Part 1)

W4 & Q3: Robustness of Identity Routing

Thank you for raising the important question of identity-routing robustness. To validate MIRRA under different identity expressions and challenging visual conditions, we conducted additional experiments on Qwen3-VL-4B covering name variations, aliases, indirect image references, facial occlusion, image blur, adversarial prompts, and multi-person images.

Test ScenarioRouting Recall ↑Answer Leakage Rate ↓
Uppercase name100%0%
Name with punctuation100%0%
Minor name misspelling100%0%
Alias100%0%
Indirect image reference100%0%
Eye occlusion100%0%
Lower-face occlusion100%0%
Half-face occlusion100%0%
Strong image blur100%0%
Adversarial text prompt100%0%
Adversarial image prompt100%0%
Multi-person image: target on the left100%0%
Multi-person image: target on the right100%0%
Multi-person image: small target face100%0%

For textual inputs, MIRRA handles name variations and aliases through text normalization and alias mapping. For indirect references without an explicit name, such as questions referring to “the person in the image,” MIRRA uses visual identity matching to determine the routing decision.

For multi-person images, we employ all-face routing, which matches every detected face against the target identity bank and activates the refusal path if any target identity is detected. This strategy achieves 100% routing recall and 0% answer leakage across all evaluated multi-person settings. It also maintains a 0% false-positive rate on multi-person images containing only retained identities.

Overall, all target queries in these additional evaluations were correctly routed. We include these results in Section 5.7of the revised manuscript.

W5 & Q4: Analysis of the CLEAR Real Face Degradation

We thank the reviewer for highlighting the degradation on CLEAR Real Face. We analyzed the existing per-example routing records and found that the refusal adapter is not activated for any of the 151 Real Face queries for either Qwen model. Therefore, routing errors and conditional adapter activation contribute 0 percentage points to the observed degradation.

When identity routing and the refusal adapter are applied directly to the Vanilla backbones, Real Face accuracy remains unchanged at 77.48% and 79.47%. The accuracy decreases to 59.60% only after retain adaptation, corresponding to reductions of 17.88 and 19.87 percentage points, respectively. The degradation is therefore attributable to retain adaptation rather than identity routing.

Retain adaptation substantially improves the in-domain CLEAR Retain metrics but introduces negative transfer to Real Face recognition. In contrast, Real World accuracy remains stable at 12.81% for the smaller Qwen model and changes from 11.99% to 12.26% for the larger model. We have added this attribution analysis in appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer qXzA(Part 2)

W6 & Q5: Scalability, Identity-Bank Size, and Deployment Cost

Thank you for the valuable suggestion. To evaluate MIRRA beyond the original 5% forgetting setting, we conduct additional full MLLMU-Bench experiments at 10% and 15% forgetting ratios.

Forget RatioForget IDsTarget Recall ↑FPR ↓Answer Leakage ↓Retain Cls. ↑Real Face Cls. ↑
5%2598.39%0.33%0.57%58.0073.29
10%5098.65%0.20%0.79%58.9673.22
15%7597.98%0.48%1.16%60.7073.02

As the number of forget identities increases from 25 to 75, MIRRA maintains approximately 98% target recall, an FPR below 0.5%, and stable retention performance.

We further conduct a controlled routing analysis by scaling the forget identity bank to 250 identities under a shared routing threshold.

Forget IDsRetain IDsTarget Recall ↑FPR ↓Matching Latency (p50)Bank Size
25100100.0%0.0%0.0149 ms250 KiB
5010098.0%0.0%0.0160 ms300 KiB
7510098.7%0.0%0.0171 ms350 KiB
10010098.0%0.0%0.0182 ms400 KiB
25010098.0%0.0%0.0249 ms700 KiB

The face-embedding latency is 38.15 ms at p50, whereas identity-bank matching remains below 0.025 ms with 250 forget identities. Thus, the primary routing cost comes from face encoding rather than bank search.

We also evaluate visually similar identities, sequential forget requests, and the number of reference images. MIRRA obtains a 0% FPR on the 50 most visually similar non-target identities. When the forget bank is sequentially expanded from 25 to 50 and 75 identities without retraining, MIRRA achieves 100% routing recall, 0% FPR, and zero answer leakage at all stages. Using one, two, and three reference images per identity yields routing recalls of 98.2%, 98.7%, and 98.7%, respectively, with a 0% FPR.

Our identity-level multimodal unlearning setting assumes that each forget request provides at least one reference face image. The one-reference result shows that MIRRA does not require a large reference collection for each target. We include the complete protocols, results, and cost analysis in the appendix D.6 of the revised manuscript.

W3 & Q1: Scope of MIRRA and Evidence of Unlearning

Machine unlearning is not limited to directly editing or erasing knowledge from model parameters. Several existing methods achieve unlearning without modifying the base-model weights. For example, ICUL changes model behavior through in-context examples [1], SPUL learns soft prompts while keeping the base model frozen [2], ULD adjusts the target model's logits using an auxiliary model [3], and SEGUE suppresses target knowledge through entropy-guided decoding [4]. MIRRA follows this line of inference-time unlearning and focuses on preventing the disclosure of information about designated identities through a controlled multimodal interface.

Hu et al. [5] study whether parameter-level unlearning in text-only LLMs can be reversed through subsequent fine-tuning. This setting does not directly apply to MIRRA, which neither edits the base-model parameters nor operates solely on text. For MIRRA, the relevant risk is routing bypass. As reported in our response to W4 & Q3, MIRRA achieves 100% routing recall and zero target-answer leakage across the evaluated textual, visual, and adversarial perturbations.

Therefore, MIRRA does not claim irreversible parameter erasure. It provides effective inference-time unlearning by preventing target-information disclosure through the evaluated multimodal interface.

References

[1] Pawelczyk et al. [In-Context Unlearning: Language Models as Few-Shot Unlearners]. ICML 2024.

[2] Bhaila et al. [Soft Prompting for Unlearning in Large Language Models]. NAACL 2025.

[3] Ji et al. [Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference]. NeurIPS 2024.

[4] Pu et al. Decoding-Unlearning: Fact Forgetting via Entropy-Guided Inference. ACL 2026.

[5] Hu et al. [Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning]. ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer qXzA(Part 3)

W1, W2.1 & Q2: Incremental Contribution and Comparisons with Closely Related Methods

Thank you for raising the concerns regarding methodological novelty and comparisons with closely related inference-time unlearning methods. We add matched MLLMU-Bench evaluations of MLLMEraser on all three backbones and ASRU on Qwen2.5-VL-3B. ASRU is evaluated using its checkpoint after 30 training rounds.

BackboneMethodForget: Cls. / R-L / Cloze ↓Retain: Cls. / R-L / Cloze ↑Real Celebrity: Cls. / R-L / Cloze ↑
Qwen2.5-VL-3BMLLMEraser57.08 / 0.568 / 13.5455.04 / 0.557 / 12.7671.59 / 0.493 / 10.46
ASRU58.75 / 0.598 / 7.2956.92 / 0.578 / 13.6172.05 / 0.516 / 11.27
MIRRA0.83 / 0.000 / 0.0058.00 / 0.601 / 12.7673.29 / 0.523 / 12.42
Qwen2.5-VL-7BMLLMEraser31.25 / 0.345 / 3.1326.28 / 0.324 / 2.6970.21 / 0.285 / 7.03
MIRRA0.42 / 0.000 / 0.0054.03 / 0.464 / 9.9776.22 / 0.464 / 20.75
LLaVA-1.5-7BMLLMEraser31.25 / 0.225 / 5.4334.49 / 0.208 / 7.3544.08 / 0.186 / 10.78
MIRRA1.25 / 0.028 / 0.0036.75 / 0.369 / 10.3654.87 / 0.371 / 19.44

Under the same evaluation protocol, MIRRA achieves substantially lower Forget scores while preserving stronger Retain and Real Celebrity performance.

The incremental contribution of MIRRA lies in identity-scoped multimodal selective intervention, rather than in any individual component alone. GUARD [1] is designed for text-only LLMs and suppresses forbidden tokens during generation after prompt classification. MLLMEraser [2] constructs multimodal erasure directions and modifies internal activations through input-aware steering. ASRU [3] first redirects activations and then optimizes refusal boundaries through reinforcement learning. In contrast, MIRRA explicitly verifies identity evidence from both text and faces, separates identity recognition from response control, and conditionally activates a modular refusal policy only for matched target identities. Non-target inputs follow the ordinary generation path, and new forget identities can be added by updating the identity bank without retraining the backbone or refusal adapter.

References

[1] Deng et al. [GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection]. ICMLW 2025.

[2] Ding et al. [MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering]. arXiv 2025.

[3] Guang et al. [ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models]. ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer qXzA(Part 4)

Thank you for the valuable comments.

W2.2: Identity Router + Fixed Refusal Baseline

We also add the requested identity-router-plus-fixed-refusal baseline using the same 48 Forget and 48 Retain queries.

ConfigurationForget Safe Refusal ↑Forget Answer Leakage ↓Retain False Refusal ↓
Vanilla0%50%0%
Router without Adapter0%50%0%
Refusal Adapter Always On100%0%100%
Router + Fixed Refusal100%0%0%
Router + Refusal Adapter100%0%0%

The unmodified MLLM does not exhibit reliable refusal behavior. Computing the identity route without activating a response intervention leaves the output unchanged from Vanilla and leaks target answers on 50% of Forget queries. Training the refusal adapter equips the MLLM with an explicit refusal behavior, reducing target leakage to zero. Activating it unconditionally, however, rejects all Retain queries; conditional routing is therefore necessary to obtain selective refusal.

The fixed-refusal baseline achieves the same binary safety and retention scores as the learned adapter in this diagnostic. We therefore do not claim that the refusal adapter outperforms deterministic suppression on these metrics. However, the Vanilla MLLM and the router without the adapter do not actively refuse target queries, resulting in a 50% answer-leakage rate in our diagnostic. The refusal adapter is therefore introduced to teach the MLLM to produce refusal responses without disclosing the target information. Unlike fixed refusal, which bypasses model generation and returns a hard-coded application-level response, the adapter equips the MLLM itself with a learnable, model-generated refusal policy. Accordingly, MIRRA’s empirical selectivity primarily comes from multimodal identity routing, while the refusal adapter enables the MLLM to execute the required refusal behavior.

We have added MLLMEraser and ASRU to the main comparison, clarified their methodological distinctions from MIRRA in the related-work and method sections, and provided the fixed-refusal analysis and the computational cost of the newly added baselines in the appendix, with the latter reported in Appendix B.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

General Response to All Reviewers

We thank reviewers qXzA, UsDY, and FZRC for their thoughtful comments. MIRRA studies identity-level multimodal unlearning at inference time. It uses text and face evidence to identify forget targets and activates a refusal adapter only for matched queries. This allows the system to avoid providing information about designated individuals while continuing to answer the same kinds of questions about others. In the revised manuscript, we clarify the scope of this contribution and add experiments on baseline comparisons, routing robustness, retention, and scalability.

  1. Scope of unlearning and use of reference data (qXzA, UsDY). MIRRA follows an established line of inference-time unlearning research, including ICUL, SPUL, and ULD, that changes model behavior while keeping the base model’s weights fixed. MIRRA applies this kind approach to designated identities in multimodal queries. Its unlearning claim concerns whether the system provides information about those identities in responses to text or image queries submitted through the MIRRA interface. Each forget request is assumed to include at least one reference face image. In the evaluated 5% MLLMU-Bench setting, the visual gate uses one reference image per identity for all 25 forget identities and for 79 of the 475 retain identities. The retain images form a separate reference bank that helps the router distinguish forget targets from visually similar non-targets. On Qwen3-VL-4B-Instruct, MIRRA reduces Forget classification accuracy from 58.33% to 0.83%, while classification accuracy on Retain and Real Celebrity remains close to that of the vanilla model.
  2. Methodological contribution and baseline comparisons (qXzA). MIRRA separates identity recognition from response generation. The router uses text and face evidence to determine which query needs refusal, and the refusal adapter enables the MLLM to generate a refusal without providing the requested information. In our diagnostic evaluation, the router-only configuration achieved 0% safe refusal and leaked target answers on 50% of Forget queries. With the adapter activated for matched identities, MIRRA achieved 100% safe refusal and 0% answer leakage on the 48 Forget queries, with 0% false refusal on the 48 Retain queries. We also added matched comparisons on MLLMU-Bench with MLLMEraser across all three backbones and with ASRU. MIRRA achieved lower scores on all three Forget metrics in these comparisons.
  3. Routing robustness and false positives (qXzA, FZRC). Across 130 controlled queries covering 11 scenarios, including name variants, aliases, indirect visual references, face occlusion, blur, and adversarial prompts, MIRRA achieves 100% routing recall and 0% observed answer leakage. We also evaluated multi-person images by checking every detected face against the target bank. A target missed by both gates can still bypass the refusal path. At the operating point used in the main experiments, our audit found 37 false positives among 11,280 non-target queries (0.328%), concentrated in five retain identities whose names or faces resembled forget targets. These analyses are reported in Sections 5.7 and 5.5.
  4. Attribution of the CLEAR Real Face degradation (qXzA). The refusal adapter was inactive for all 151 Real Face queries on both Qwen backbones. Adding the router and refusal adapter to the vanilla backbones preserved accuracies of 77.48% and 79.47%. Both declined to 59.60% after retain adaptation, identifying that adaptation step as the source of the observed degradation. We report this analysis in Appendix B.2.
  5. Scalability and deployment cost (qXzA, FZRC). In the added 10% and 15% MLLMU-Bench experiments, MIRRA maintains approximately 98% target routing recall, a false-positive rate below 0.5%, and Retain and Real Celebrity performance close to the vanilla model. In a separate controlled routing experiment with 250 forget identities, MIRRA achieves 98.0% recall with 0% observed false positives, and median bank-matching latency remains below 0.025 ms. We also evaluated sequential enrollment without retraining and found that one reference image per identity yielded 98.2% routing recall in the reference-count experiment. The additional forget ratio results are in Section 5.8 and Table 5, and the controlled routing and cost analyses are in Appendix D.6.

We appreciate the reviewers’ feedback and welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.