- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MUSE, a lightweight framework for multimodal large language model unlearning through adaptive system-prompt embeddings. With the backbone frozen, a trainable Multimodal Embedding Adapter generates prompt embeddings conditioned on each image–question pair. An asymmetric dual-space objective steers forget-set representations toward a reference region associated with absent target knowledge, while preserving retain-set representations and anchoring their prompts to the original embeddings. Experiments on MLLMU-Bench and OFFSIDE report competitive forgetting–utility trade-offs, robustness to several prompt attacks, and reduced training time and GPU memory usage. The approach provides reversible control over knowledge access rather than removing knowledge from the frozen backbone’s parameters.
Critical strengths
-
MUSE generates system-prompt embeddings conditioned on each image–question pair, enabling input-specific behavioral control while keeping the backbone frozen.
-
Results on two model backbones show substantial suppression of targeted information while largely preserving performance on retain and real-celebrity subsets.
-
The evaluation includes prompt attacks, efficiency comparisons, component ablations, hyperparameter analysis, and hidden-state visualizations.
Critical weaknesses
-
MUSE learns an external adapter to induce representation shifts toward a presumed no-knowledge region. However, LoRA also introduces additional trainable parameters while keeping the original backbone weights frozen, and its intervention can likewise be disabled. The paper should clarify why adaptive prompt embeddings are preferable to LoRA and compare both approaches under the same representation-level objectives and comparable parameter budgets.
-
The reported MLLMU-Bench evaluation covers classification and generation but omits the cloze task. Moreover, although the experiments include multimodal VQA, corresponding text-only QA evaluations are not reported, leaving unclear whether suppression persists when the same target knowledge is queried without images.
-
The paper lacks concrete examples showing what MUSE actually produces compared with prior methods. If its outputs are predominantly templated refusals, the advantage over existing refusal-based approaches remains unclear. Representative case studies and explicit criteria for naturalness and informativeness would help establish whether MUSE provides more useful, context-sensitive responses.
-
The difference between forget-set and no-knowledge reference representations is assumed to define a meaningful forgetting direction, but the experiments do not isolate its contribution from generic representation disruption. An equal-norm random-direction baseline, and a baseline using random target representations, would help determine whether the proposed direction is necessary. Comparable performance from these controls would weaken the claimed interpretation of the direction as specifically encoding the absence of target knowledge.
Questions for authors
see weaknesses
5 comments
Response to Reviewer xGT1(W4)
W4: Necessity of the Learned Forgetting Direction.
We thank the reviewer for this valuable suggestion. We introduce two additional baselines:
Random Target Representations (RTR):
where controls the magnitude of the random target representation, and is a random unit vector.
Equal-norm Random Direction (ERD). We replace the learned refusal direction with a randomly sampled direction whose norm is identical to that of the original refusal direction in our method:
The corresponding forgetting target is:
Table 3. Comparison with random steering baselines on LLaVA-1.5-7B(5% forget).
| Model | Forget Gen.↓ | CR↑ | Forgetfulness↑ | Retain Gen.↑ |
|---|---|---|---|---|
| Vanilla | 0.580 | -- | -- | 0.490 |
| MUSE | 0.105 | 4.40 | 4.42 | 0.603 |
| RTR | 0.072 | 0.04 | 0.04 | 0.589 |
| ERD | 0.016 | 0.42 | 0.50 | 0.532 |
As shown in Table 3, both random baselines achieve even lower Forget Generation scores than MUSE. However, their Contextual Refusal and Forgetfulness scores are substantially lower (0.04/0.04 and 0.42/0.50 vs. 4.40/4.42). Our sample-level inspection further reveals that these baselines often produce incoherent or repetitive outputs (e.g., "when ETFS, ETFS, ETFS..."), indicating general generation degeneration rather than meaningful knowledge suppression.
In contrast, MUSE achieves coherent, context-aware refusals while maintaining better Retain performance . These results demonstrate that simply disrupting forget representations, even with comparable or random target shifts, is insufficient. The data-driven forgetting direction is important for guiding the model toward a meaningful refusal region rather than arbitrary representation disruption.
Response to Reviewer xGT1
Dear Reviewer xGT1,
I hope this message finds you well. Thank you very much for your valuable comments! We have carefully addressed each of your concerns in our rebuttal and hope that our additional experiments and clarifications have addressed your questions. We would be very grateful to hear your feedback. If any questions or concerns remain, we would be happy to discuss them further and provide additional clarifications.
Thank you again for your time and constructive suggestions!
Best regards, All Authors
Response to Reviewer xGT1(W1,W2,W3)
We appreciate your thoughtful comments and address your concerns below.
W1: Comparison with LoRA.
We introduce a LoRA baseline trained with the same representation-level objectives as MUSE.
Table 1. Comparison between MUSE and LoRA on LLaVA-1.5-7B (5% forget).
As shown in Table 1, although LoRA achieves some suppression, its forgetting effectiveness is substantially weaker than MUSE. Meanwhile, both methods maintain comparable Retain generation performance . These results highlight that representation-level steering alone is insufficient for effective knowledge suppression through parameter-based intervention. MUSE further leverages adaptive prompt embeddings and prompt-space constraints to achieve substantially more effective and selective knowledge intervention while keeping the backbone frozen.
W2.1 Regarding the Cloze Evaluation.
We additionally evaluate MUSE on the Cloze task using the officially released checkpoints provided by MLLMU-Bench. As shown in Table 2, the results are consistent with our main findings: MUSE substantially reduces Forget accuracy (31.25 to 8.33) while maintaining or improving performance on Retain (31.01 to 33.76) and Real (8.17 to 9.48). We did not include Cloze in the main paper because the released checkpoints exhibit relatively limited absolute performance on this task, particularly on Real samples. Nevertheless, these additional results further support the effectiveness of MUSE.
Table 2. Cloze accuracy (%) on LLaVA-1.5-7B.
W2.2 Regarding the Text-only QA Evaluation.
MUSE focuses on unlearning textual knowledge associated with visual inputs, where knowledge access is regulated through jointly conditioned image-question pairs. As stated in FIU-BENCH, since the goal of VLMs is to describe information provided in images, unlearning that involves only the text modality should instead be considered part of LLM unlearning rather than VLM unlearning.
Accordingly, the Pure_Text subset of MLLMU-Bench evaluates a different setting from the image-conditioned multimodal unlearning targeted by MUSE. We will clarify this distinction and explicitly specify the scope of our unlearning task in the revised manuscript. Exploring broader cross-modal unlearning methods that regulate knowledge access across modalities without modifying backbone parameters is an interesting direction for future work.
W3: Case Study
We have added a case study in Section 8 of the revised manuscript to provide representative examples of MUSE's responses on Forget and Retain samples.
As illustrated below, MUSE does not simply produce a fixed refusal template. Instead, it generates context-dependent responses that explain the lack of relevant information according to the specific query, while avoiding disclosure of the target facts. For Retain samples, MUSE continues to provide direct and accurate answers.
Case 1: Profession-related Knowledge
Forget (Image ID: 259)
Retain (Image ID: 333)
Case 2: Personal Background Knowledge
Forget (Image ID: 093)
Retain (Image ID: 486)
These examples illustrate three desirable properties of MUSE's responses: (1) Contextual naturalness: refusals are expressed in fluent, query-dependent language rather than a fixed response template; (2) Informativeness: refusals provide relevant explanations about the unavailable information without revealing the target facts; and (3) Utility preservation: MUSE continues to provide accurate and direct answers to Retain queries. These case studies further illustrate how MUSE achieves context-aware knowledge access control beyond simple fixed-template refusal.