- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MIRRA, a multimodal machine unlearning framework that separates target-identity recognition from refusal generation. MIRRA identifies whether a query refers to a target identity through text matching or face verification; target queries activate a LoRA refusal adapter, while non-target queries are handled by the base model. The method is evaluated on MLLMU-Bench and CLEAR with three MLLMs and compared against several unlearning baselines. Results show that MIRRA strongly suppresses responses about target identities while largely preserving performance on retained and general queries. Ablation studies further confirm that multimodal identity routing and the refusal adapter are both important for effective selective forgetting.
Critical strengths
-
Simple and modular design. By separating target identification from refusal generation, MIRRA avoids applying the forgetting mechanism indiscriminately to all inputs. This modularity makes the method easy to interpret and helps explain why non-target utility is largely preserved.
-
Effective multimodal target identification. The combination of text matching and face verification allows MIRRA to recognize forget identities from either textual or visual evidence. This is important for multimodal unlearning because the same identity can be referenced through different modalities.
-
Good control of false refusals. The use of both forget and retain identity prototypes helps reduce incorrect activation of the refusal mechanism on non-target individuals. This is consequential because excessive refusal would undermine the practical usefulness of the system.
Critical weaknesses
-
The method depends heavily on the robustness of the identity router. If the text or face gate fails to recognize a target identity, the query bypasses the refusal adapter and is answered by the base model. This creates a direct failure mode in which imperfect routing can lead to information leakage. The ablation results themselves show that removing either gate substantially increases leakage.
-
The evaluation does not sufficiently test adversarial or evasive queries. The paper evaluates paraphrased questions and altered portraits, but it does not appear to systematically test whether users can bypass the router through name obfuscation, indirect references, adversarial images, or other routing attacks. This matters because MIRRA’s guarantees depend on reliable target detection at inference time.
-
The benchmark setting is relatively narrow. The main experiments use the 5% forgetting setting on MLLMU-Bench and CLEAR. It remains unclear whether the strong forget-retain trade-off would hold when a much larger fraction of identities must be forgotten.
-
The paper does not fully characterize failure cases of false-positive routing. The sensitivity analysis shows that increasing the routing FPR improves forgetting but reduces retained utility, indicating a non-trivial trade-off. However, the paper provides limited qualitative analysis of which non-target identities are most likely to be incorrectly routed and refused, which would be important for understanding practical reliability.
Questions for authors
How robust is the identity router to evasive or adversarial references to a forget identity, such as name obfuscation, indirect textual descriptions, heavily altered faces, or adversarial image perturbations? Because routing failure directly exposes the base model response, this could materially affect the claimed forgetting guarantees.
5 comments
Response to Reviewer FZRC(Part 2)
W4: Qualitative analysis of false-positive routing
To address this concern, we conducted an identity-level audit of all false-positive routing decisions at the operating point used in our main experiments.
Among 11,280 non-target queries, 37 are incorrectly routed to the refusal adapter, corresponding to a query-level false-positive rate of 0.328%. These errors affect only 5 of the 628 non-target identities (0.80%). All five identities belong to the Retain set, while no false positives are observed among the 153 Real Celebrity identities. Of the 37 false-positive queries, 19 are triggered by the text gate and 18 by the face gate.
The audit identifies two recurring failure modes. Text-gate errors are caused by lexical name collisions: \emph{Eliza Thornton} is confused with the forget identity \emph{Emilia Thornton}, \emph{Lena Fitzgerald} with \emph{Elena Fitzgerald}, and \emph{Marco Delgado} with \emph{Mariana Delgado}. Face-gate errors are concentrated in two retained identities, \emph{Lucia Martinez} and \emph{Kimberly Nguyen}, whose face representations exceed the routing threshold because of their similarity to the forget prototypes of \emph{Sofia Delacroix} and \emph{Camila Vermeer}, respectively.
| Triggering gate | Non-target identity | Matched forget identity | FP queries |
|---|---|---|---|
| Text | Eliza Thornton | Emilia Thornton | 9 |
| Text | Lena Fitzgerald | Elena Fitzgerald | 9 |
| Text | Marco Delgado | Mariana Delgado | 1 |
| Face | Lucia Martinez | Sofia Delacroix | 9 |
| Face | Kimberly Nguyen | Camila Vermeer | 9 |
| Total | 5 identities | — | 37 |
These results show that false positives mainly occur when a retained identity has a name or face similar to a forget identity. Therefore, increasing the router sensitivity may also cause these similar non-target identities to be incorrectly refused. We have added this analysis to clarify when false-positive routing is most likely to occur.
We have added this analysis in Section 5.6 of the revised manuscript.
Response to Reviewer FZRC(Part 3)
W3: Scalability to larger forget sets
Thank you for pointing out the limited forgetting scale in our original evaluation. To address this concern, we added experiments with 10% and 15% forgetting ratios on MLLMU-Bench. We evaluated MIRRA and baselines using the corresponding Forget and Retain partitions. The complete results are shown below.
Qwen3-VL-4B-Instruct (10% Forget)
| Method | Forget Cls. ↓ | Forget R-L ↓ | Forget Cloze ↓ | Retain Cls. ↑ | Retain R-L ↑ | Retain Cloze ↑ | Real Celebrity Cls. ↑ | Real Celebrity R-L ↑ | Real Celebrity Cloze ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Vanilla | 58.78 | 0.602 | 12.76 | 58.58 | 0.601 | 8.63 | 73.68 | 0.523 | 12.58 |
| GA | 58.37 | 0.538 | 11.73 | 58.74 | 0.528 | 8.35 | 72.05 | 0.472 | 10.95 |
| GA-Difference | 58.16 | 0.511 | 13.78 | 53.43 | 0.525 | 10.75 | 69.76 | 0.433 | 11.44 |
| KL Minimization | 60.82 | 0.614 | 14.29 | 60.84 | 0.601 | 8.63 | 72.96 | 0.533 | 12.09 |
| NPO | 59.59 | 0.465 | 11.73 | 58.74 | 0.463 | 8.24 | 74.20 | 0.453 | 10.95 |
| MIRRA (Ours) | 1.02 | 0.000 | 0.00 | 58.96 | 0.601 | 8.80 | 73.22 | 0.521 | 13.07 |
Qwen3-VL-4B-Instruct (15% Forget)
| Method | Forget Cls. ↓ | Forget R-L ↓ | Forget Cloze ↓ | Retain Cls. ↑ | Retain R-L ↑ | Retain Cloze ↑ | Real Celebrity Cls. ↑ | Real Celebrity R-L ↑ | Real Celebrity Cloze ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Vanilla | 58.11 | 0.600 | 10.47 | 60.89 | 0.601 | 10.79 | 73.68 | 0.523 | 12.58 |
| GA | 57.84 | 0.494 | 5.07 | 60.70 | 0.503 | 5.90 | 73.22 | 0.460 | 11.27 |
| GA-Difference | 50.41 | 0.534 | 8.78 | 53.61 | 0.543 | 10.44 | 67.86 | 0.423 | 10.78 |
| KL Minimization | 56.62 | 0.609 | 11.49 | 59.63 | 0.611 | 9.55 | 72.63 | 0.528 | 12.58 |
| NPO | 59.86 | 0.541 | 5.74 | 60.49 | 0.549 | 8.14 | 72.24 | 0.480 | 11.27 |
| MIRRA (Ours) | 1.89 | 0.010 | 0.00 | 60.70 | 0.599 | 11.20 | 73.02 | 0.524 | 12.42 |
At both forgetting ratios, MIRRA reduces the Forget metrics to nearly zero while maintaining Retain and Real Celebrity performance close to Vanilla. These results demonstrate that MIRRA preserves its forget--retain trade-off as the number of forget identities increases. We have added these experiments and the corresponding discussion to Section 5.8 and Table 5 of the revised manuscript.
Response to Reviewer FZRC(Part 1)
Thank you for your comments. We address each concern below.
W1&W2&Q1: Robustness of the identity router to evasive and adversarial queriesWeaknesses 1 and 2 and Question 1 concern the same fundamental issue that whether the identity router can still activate the refusal path when a forget identity is presented through evasive, indirect, or perturbed inputs. We agree that if both the text and face gates fail to recognize the target identity, the query may bypass the refusal adapter. Routing failure therefore constitutes a direct failure mode of MIRRA. To address this concern, we conducted a robustness evaluation.
Specifically, we constructed 11 types of controlled identity-preserving perturbations using the existing benchmark samples, resulting in 130 test queries. The text-side tests include name capitalization, punctuation changes, minor spelling errors, aliases, and adversarial text prompts. The image-side tests include indirect visual references without explicitly providing the identity name, eye-region occlusion, lower-face occlusion, half-face occlusion, strong blur, and adversarial image prompts. All visual transformations were applied directly to existing test images and did not rely on newly generated face images. In addition, the original test sets of MLLMU-Bench and CLEAR already contain natural variations in viewpoints, backgrounds, and clothing relative to the identity-enrollment images.
We evaluate the router using two metrics. Routing Recall measures the proportion of target queries that successfully activate the refusal path, while Answer Leakage measures the proportion of queries whose final responses disclose information about the target identity. The results are summarized below.
Across all 130 evaluated queries, the complete dual-gate router achieves 100% routing recall and 0% answer leakage.
These results also clarify the gate-ablation findings. The text and face gates are complementary rather than redundant. Queries containing a name or an enrolled alias are primarily detected by the text gate, whereas image-based queries that do not explicitly mention the identity are handled by the face gate. MIRRA adopts a privacy-oriented OR fusion rule: the refusal adapter is activated whenever either gate detects a forget identity.
We include these results in Section 5.7 of the revised manuscript.