acceptodds
Under review as a conference paper at ICLR 2027

MUSE: Adaptive Input-Conditioned System-Prompt Embeddings for Multimodal Large Language Model Unlearning

Abstract

Multimodal large language models (MLLMs) may retain sensitive or undesirable knowledge, motivating effective multimodal unlearning. Existing parameter-level methods often incur substantial training and storage overhead. We propose MUSE, a lightweight and adaptive framework that reformulates multimodal unlearning as system-prompt embedding intervention over a frozen MLLM. MUSE trains a Multimodal Embedding Adapter (MEA) to generate system-prompt embeddings conditioned on each image–question pair, enabling selective knowledge suppression or preservation according to the multimodal context. To explicitly control the forget–retain boundary, MUSE further introduces an asymmetric dual-space objective that jointly constrains prompt embeddings and internal representations. Experiments on MLLMU-Bench demonstrate strong forgetting, favorable forget–retain trade-offs, and robustness to prompt attacks. Moreover, MUSE achieves up to 4.9 faster training and 74.2% lower GPU memory usage than the strongest baseline, demonstrating effective and efficient multimodal unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Machine Unlearning, Multimodal Large Language Models, Adaptive Prompting, Embedding Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xGT1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MUSE, a lightweight framework for multimodal large language model unlearning through adaptive system-prompt embeddings. With the backbone frozen, a trainable Multimodal Embedding Adapter generates prompt embeddings conditioned on each image–question pair. An asymmetric dual-space objective steers forget-set representations toward a reference region associated with absent target knowledge, while preserving retain-set representations and anchoring their prompts to the original embeddings. Experiments on MLLMU-Bench and OFFSIDE report competitive forgetting–utility trade-offs, robustness to several prompt attacks, and reduced training time and GPU memory usage. The approach provides reversible control over knowledge access rather than removing knowledge from the frozen backbone’s parameters.

Critical strengths
  1. MUSE generates system-prompt embeddings conditioned on each image–question pair, enabling input-specific behavioral control while keeping the backbone frozen.

  2. Results on two model backbones show substantial suppression of targeted information while largely preserving performance on retain and real-celebrity subsets.

  3. The evaluation includes prompt attacks, efficiency comparisons, component ablations, hyperparameter analysis, and hidden-state visualizations.

Critical weaknesses
  1. MUSE learns an external adapter to induce representation shifts toward a presumed no-knowledge region. However, LoRA also introduces additional trainable parameters while keeping the original backbone weights frozen, and its intervention can likewise be disabled. The paper should clarify why adaptive prompt embeddings are preferable to LoRA and compare both approaches under the same representation-level objectives and comparable parameter budgets.

  2. The reported MLLMU-Bench evaluation covers classification and generation but omits the cloze task. Moreover, although the experiments include multimodal VQA, corresponding text-only QA evaluations are not reported, leaving unclear whether suppression persists when the same target knowledge is queried without images.

  3. The paper lacks concrete examples showing what MUSE actually produces compared with prior methods. If its outputs are predominantly templated refusals, the advantage over existing refusal-based approaches remains unclear. Representative case studies and explicit criteria for naturalness and informativeness would help establish whether MUSE provides more useful, context-sensitive responses.

  4. The difference between forget-set and no-knowledge reference representations is assumed to define a meaningful forgetting direction, but the experiments do not isolate its contribution from generic representation disruption. An equal-norm random-direction baseline, and a baseline using random target representations, would help determine whether the proposed direction is necessary. Comparable performance from these controls would weaken the claimed interpretation of the direction as specifically encoding the absence of target knowledge.

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer xGT1(W1,W2,W3)

We appreciate your thoughtful comments and address your concerns below.

W1: Comparison with LoRA.

We introduce a LoRA baseline trained with the same representation-level objectives as MUSE.

Table 1. Comparison between MUSE and LoRA on LLaVA-1.5-7B (5% forget).

ModelForget Class.↓Forget Gen.↓Retain Class. ↑Retain Gen.↑
Vanilla51.670.58046.340.490
MUSE22.500.10546.170.603
LoRA48.330.44744.520.613

As shown in Table 1, although LoRA achieves some suppression, its forgetting effectiveness is substantially weaker than MUSE. Meanwhile, both methods maintain comparable Retain generation performance . These results highlight that representation-level steering alone is insufficient for effective knowledge suppression through parameter-based intervention. MUSE further leverages adaptive prompt embeddings and prompt-space constraints to achieve substantially more effective and selective knowledge intervention while keeping the backbone frozen.

W2.1 Regarding the Cloze Evaluation.

We additionally evaluate MUSE on the Cloze task using the officially released checkpoints provided by MLLMU-Bench. As shown in Table 2, the results are consistent with our main findings: MUSE substantially reduces Forget accuracy (31.25 to 8.33) while maintaining or improving performance on Retain (31.01 to 33.76) and Real (8.17 to 9.48). We did not include Cloze in the main paper because the released checkpoints exhibit relatively limited absolute performance on this task, particularly on Real samples. Nevertheless, these additional results further support the effectiveness of MUSE.

Table 2. Cloze accuracy (%) on LLaVA-1.5-7B.

ModelForget↓Retain↑Real↑
Vanilla31.2531.018.17
GA_diff25.0025.424.25
NPO29.1727.428.16
KL_Min25.0021.527.84
MUSE8.3333.769.48

W2.2 Regarding the Text-only QA Evaluation.

MUSE focuses on unlearning textual knowledge associated with visual inputs, where knowledge access is regulated through jointly conditioned image-question pairs. As stated in FIU-BENCH, since the goal of VLMs is to describe information provided in images, unlearning that involves only the text modality should instead be considered part of LLM unlearning rather than VLM unlearning.

Accordingly, the Pure_Text subset of MLLMU-Bench evaluates a different setting from the image-conditioned multimodal unlearning targeted by MUSE. We will clarify this distinction and explicitly specify the scope of our unlearning task in the revised manuscript. Exploring broader cross-modal unlearning methods that regulate knowledge access across modalities without modifying backbone parameters is an interesting direction for future work.

W3: Case Study

We have added a case study in Section 8 of the revised manuscript to provide representative examples of MUSE's responses on Forget and Retain samples.

As illustrated below, MUSE does not simply produce a fixed refusal template. Instead, it generates context-dependent responses that explain the lack of relevant information according to the specific query, while avoiding disclosure of the target facts. For Retain samples, MUSE continues to provide direct and accurate answers.

Case 1: Profession-related Knowledge

  • Forget (Image ID: 259)

    • Question: What profession is depicted by the individual in the image?
    • Ground Truth: The individual is an architect.
    • MUSE: "I'm sorry, but I cannot provide an answer to this question. The image does not provide enough information to determine the individual's profession."
  • Retain (Image ID: 333)

    • Question: What profession is depicted in the image along with the person's features?
    • Ground Truth: The person in the image is a software engineer.
    • MUSE: "The image shows a person who is a software engineer."

Case 2: Personal Background Knowledge

  • Forget (Image ID: 093)

    • Question: Which country is the person in the image originally from?
    • Ground Truth: The person in the image is originally from Ireland.
    • MUSE: "I'm sorry, but I don't have any information about the person's origin."
  • Retain (Image ID: 486)

    • Question: Which city is the individual depicted in the image residing in?
    • Ground Truth: The individual resides in Berlin, Germany.
    • MUSE: "The individual resides in Berlin, Germany."

These examples illustrate three desirable properties of MUSE's responses: (1) Contextual naturalness: refusals are expressed in fluent, query-dependent language rather than a fixed response template; (2) Informativeness: refusals provide relevant explanations about the unavailable information without revealing the target facts; and (3) Utility preservation: MUSE continues to provide accurate and direct answers to Retain queries. These case studies further illustrate how MUSE achieves context-aware knowledge access control beyond simple fixed-template refusal.

AuthorsOpenReview ↗

Response to Reviewer xGT1(W4)

W4: Necessity of the Learned Forgetting Direction.

We thank the reviewer for this valuable suggestion. We introduce two additional baselines:

Random Target Representations (RTR):

Lf=Exf∈Dforget∥∣h(xf)−αu∥∣22L_{f} = E_{x_f \in D_{forget}} \|| h(x_f) - \alpha u \||_2^2

where α\alpha controls the magnitude of the random target representation, and ulu_l is a random unit vector.

Equal-norm Random Direction (ERD). We replace the learned refusal direction clc_l with a randomly sampled direction whose norm is identical to that of the original refusal direction in our method:

clrand=∥cl∥2ulc_l^{\mathrm{rand}}=\|c_l\|_2u_l

The corresponding forgetting target is:

Lf=Exf∈Dforget∥∣h(xf)−αclrand∥∣22L_{f} = E_{x_f \in D_{forget}} \|| h(x_f)-\alpha c_l^{\mathrm{rand}}\||_2^2

Table 3. Comparison with random steering baselines on LLaVA-1.5-7B(5% forget).

ModelForget Gen.↓CR↑Forgetfulness↑Retain Gen.↑
Vanilla0.580----0.490
MUSE0.1054.404.420.603
RTR0.0720.040.040.589
ERD0.0160.420.500.532

As shown in Table 3, both random baselines achieve even lower Forget Generation scores than MUSE. However, their Contextual Refusal and Forgetfulness scores are substantially lower (0.04/0.04 and 0.42/0.50 vs. 4.40/4.42). Our sample-level inspection further reveals that these baselines often produce incoherent or repetitive outputs (e.g., "when ETFS, ETFS, ETFS..."), indicating general generation degeneration rather than meaningful knowledge suppression.

In contrast, MUSE achieves coherent, context-aware refusals while maintaining better Retain performance . These results demonstrate that simply disrupting forget representations, even with comparable or random target shifts, is insufficient. The data-driven forgetting direction is important for guiding the model toward a meaningful refusal region rather than arbitrary representation disruption.

AuthorsOpenReview ↗

Response to Reviewer xGT1

Dear Reviewer xGT1,

I hope this message finds you well. Thank you very much for your valuable comments! We have carefully addressed each of your concerns in our rebuttal and hope that our additional experiments and clarifications have addressed your questions. We would be very grateful to hear your feedback. If any questions or concerns remain, we would be happy to discuss them further and provide additional clarifications.

Thank you again for your time and constructive suggestions!

Best regards, All Authors

Reviewer_8rN3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MUSE, which addresses the MLLM unlearning problem through system-prompt embeddings on each image-question pair. MUSE further designs an asymmetric dual-space objective to jointly constrain these prompt embeddings and internal representations. Experimental results on MLLMU-Bench demonstrate the effectiveness and efficiency of this work.

Critical strengths
  1. MUSE offers reversible intervention without altering the backbone. It generates system-prompt embeddings and conditions the intervention on image-question pairs, providing a meaningful extension beyond a shared soft prompt. The reported efficient training and memory savings make this design important, particularly when the original model must remain available.
  2. The authors have conducted various investigations and analyses. MUSE demonstrates useful suppression while preserving substantial retained capability across MLLMU-Bench and OFFSIDE using LLaVA-1.5-7B and Qwen-3-VL-8B-Instruct
Critical weaknesses
  1. MUSE constructs its steering direction from the mean representation difference between forget inputs and unseen synthetic identities from DigiFace1M. However, this difference may capture identity or input-distribution differences rather than knowledge absence specifically. The reported steering-strength and layer analyses cannot resolve this ambiguity.
  2. The attack experiments cover random prefixes, paraphrases, and jailbreak prompts. However, low lexical overlap under attack is insufficient to imply that target facts remain inaccessible, since leaked information can be expressed differently from the reference answers.
Questions for authors
  1. The ablations in Table 3 have highlighted the gains of both image and text inputs. I am curious whether the gains come from input-conditioned prompting or the dual-space training objectives. The author should provide an analysis to clarify this.
  2. How are the reference questions paired with DigiFace1M identities, and how closely do they match the forget queries? I am also confused that if it is proper to consider the resulting direction as knowledge absence rather than an identity or input-distribution difference?
  3. Does the low generation overlap under prompt attacks also correspond to low factual leakage? Could you apply the knowledge-leakage assessment to the attacked responses to clarify this distinction?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8rN3

We appreciate your thoughtful comments and address your concerns below.

W1&Q2: Clarification of the Steering Direction.

We thank the reviewer for raising this important concern.

Regarding the construction of reference questions, they are constructed based on the Forget queries and target similar types of personal information about individuals. Specifically, when constructing the steering direction, we randomly assign one reference question to each identity image from DigiFace1M and use the resulting image-question pair to compute its hidden representation. Thus, the reference questions are aligned with the Forget queries in terms of semantic intent and question type, although they are not explicitly paired with the corresponding real identities or matched one-to-one with individual Forget queries.

We clarify that the term "absence of knowledge" does not imply a universal knowledge-free direction. Instead, the no-knowledge reference set provides an empirical reference for the model's responses to image-question pairs for which it has no relevant prior knowledge. Accordingly, the derived steering direction serves as a reference direction for guiding forget samples toward a "no relevant knowledge and refuse" representation region, rather than explicitly disentangling knowledge absence from all identity or input-distribution factors.

Empirically, MUSE substantially reduces generation performance on the Forget set while achieving high contextual refusal scores, indicating that the learned direction effectively guides the model toward the intended refusal behavior. Moreover, our activation visualization (Fig. 6) shows a clearer separation between Forget and Retain representations after applying MUSE, further supporting the effectiveness of the learned intervention.

W2&Q3: Assessment of Responses under Attack

We further evaluate MUSE under three types of attacks using GPT-4o-mini as the judge, reporting Contextual Refusal and Forgetfulness scores to assess refusal behavior and semantic knowledge leakage beyond surface-level textual similarity.

Table 1: Assessment under Attack of LLaVA-1.5-7B(5% forget)

CR↑Forgetfulness↑
MUSE4.404.42
Jailbreak Prompt4.404.42
Paraphrase4.404.41
Random Prefix4.324.30

The results show that MUSE maintains consistently high refusal and forgetfulness scores across all three attack settings, with only marginal variations compared to the original setting. These results provide further evidence that MUSE's robustness is not merely due to reduced lexical overlap, but also extends to semantic-level knowledge suppression under different input perturbations.

Q1:Ablations

To disentangle their contributions, we conduct an additional ablation by retaining only self-attention over the system prompt while removing cross-attention to image and question inputs. The original dual-space training objectives remain unchanged.

Table 2: Ablation study on LLaVA-1.5-7B (5% forget).

ModelForget Gen. ↓Test Gen. ↓Retain Gen. ↑Real Gen. ↑
Vanilla0.5800.2300.4900.220
MUSE0.1050.1430.6030.336
variant0.4350.2760.4990.343

As shown in Table 2, compared with the input-independent variant, full MUSE substantially reduces Forget and Test generation scores (0.105 vs. 0.435 and 0.143 vs. 0.276, respectively), while achieving higher Retain performance (0.603 vs. 0.499) and comparable Real performance (0.336 vs. 0.343). This demonstrates that multimodal input conditioning is crucial for instance-specific knowledge intervention. Furthermore, the w/o Prompt Constraints ablation in Table 3 of our paper shows that removing the prompt-space objectives severely degrades Retain and Real performance (0.603 to 0.126 and 0.336 to 0.085, respectively), highlighting their importance in preserving non-target knowledge.

Together, these results demonstrate that multimodal input conditioning and dual-space objectives play complementary roles in achieving effective and selective multimodal unlearning.

I appreciate the response from the authors. Most of my concerns have been well addressed. Therefore, I keep my rating.

AuthorsOpenReview ↗

Response to Reviewer 8rN3

We sincerely thank you for your thoughtful assessment and positive feedback. We greatly appreciate your constructive comments, which have helped us improve the clarity and rigor of our work. We are glad that our clarifications and additional analyses have addressed your concerns. Thank you again for your time and consideration.

Reviewer_GNXQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MUSE keeps the MLLM frozen and trains a small transformer that rewrites the system-prompt embedding for each image–question pair. Forget activations at one layer are pushed along a "no-knowledge" direction estimated from unseen synthetic faces, retain activations are kept fixed, retain prompts stay near the original, and forget prompts are pushed away from it. The paper reports much lower forget-set accuracy than weight-editing baselines on MLLMU-Bench and OFFSIDE, better refusal quality, and lower training cost than ASRU.

Critical strengths
  1. The design is practical. The backbone is untouched, the adapter trains in under an hour (Table 2), and a new forget scope needs only a new adapter, not another fine-tuned model. That fits deployments where the provider controls inference.

  2. The reported forgetting is strong on two backbones (forget accuracy 51.67→22.50 on LLaVA-1.5-7B and 55.00→16.00 on Qwen3-VL-8B). A GPT judge scores the outputs as appropriate refusals (CR 4.44 on LLaVA-1.5-7B), and the ablations show that both modalities are needed.

Critical weaknesses
  1. MUSE controls access to the knowledge rather than removing it. The paper says as much ("knowledge access behavior") but does not draw the consequences. The knowledge stays in the weights, so an open-weight model can simply be run without the adapter. That is a legitimate setting, and ECO (NeurIPS 2024) states the same limitation, but the threat model should then be explicit and the gate tested where it can fail. MLLMU-Bench's text-only questions, which name the person, are not reported. At 15% forget on LLaVA-1.5-7B, Real-celebrity accuracy drops to 25.46, the lowest of all methods (Table 7), which looks like the gate firing on identities it was not trained on.

  2. All baselines edit weights. The comparison that matters for a gate is another gate, such as MLLMU-Bench's own system-prompt baseline or an (image, question) classifier that triggers a fixed refusal. With 25 forget identities, I can't tell how much the learned prompt adds over such a filter.

  3. The method is close to prior work. The adapter follows Sysformer (ICLR 2026), which rewrites the system-prompt embedding of a frozen LLM using self-attention over the prompt and cross-attention over the user input, and regularizes it toward the original prompt. L_forget and L_retain have the same form as ASRU's stage-1 objective (Guang et al., 2026): a knowledge-absence direction from unseen-image minus forget-set means, with targets a(x)+λv and a(x), here optimized through the adapter rather than a down-projection layer. The new parts are the image conditioning and the forget-prompt separation term.

Questions for authors
  1. Apart from MANU, the baseline rows of Table 1, including Vanilla, are identical to ASRU's Table 1. If they come from that paper, please say so and confirm that MUSE was run from the same fine-tuned checkpoints and scored with the same judge (ASRU used GPT-4o-mini; Sec. 5.1 says GPT-5.4-mini).

  2. Text-only results and a classifier-gate comparison would most change my assessment.

  3. How often does MUSE refuse on Retain and Real inputs at 5%, 10% and 15% forget? CR is reported only on forget queries. A rising refusal rate on Real identities at 15% would confirm that the drop to 25.46 comes from the gate firing on non-forget identities; if the rate does not rise, something else in the generated prompt is hurting utility.

6 comments
AuthorsOpenReview ↗

Response to Reviewer GNXQ(W1,W2&Q2)

We greatly appreciate your thoughtful comments and would also like to do our best to address your concerns below.

W1.1: Threat Model and Scope.

MUSE aims to achieve reversible, input-conditioned knowledge access control rather than permanently removing knowledge from the model parameters. Consequently, MUSE cannot provide protection if an attacker has direct access to the open-weight model and bypasses the MEA. Our threat model assumes a controlled deployment environment in which the model provider manages the inference pipeline and enforces the MEA as a mandatory component. End users can only submit queries through the controlled interface and cannot directly access the backbone model weights or disable the MEA. Under this setting, MUSE provides a flexible and reversible knowledge access control mechanism without modifying or retraining the underlying MLLM. We will explicitly clarify this assumption and its scope in the revised manuscript.

W1.2: Regarding the Pure_Text Evaluation.

MUSE focuses on unlearning textual knowledge associated with visual inputs, where knowledge access is regulated through jointly conditioned image-question pairs. As stated in FIU-Bench[1], since the goal of VLMs is to describe information provided in images, unlearning that involves only the text modality should instead be considered part of LLM unlearning rather than VLM unlearning.

Accordingly, the Pure_Text subset of MLLMU-Bench evaluates a different setting from the image-conditioned multimodal unlearning targeted by MUSE. We will clarify this distinction and explicitly specify the scope of our unlearning task in the revised manuscript.Exploring broader cross-modal unlearning methods that regulate knowledge access across modalities without modifying backbone parameters is an interesting direction for future work.

[1] Yingzi Ma, Jiongxiao Wang, et al. Benchmarking vision language model unlearning via fictitious facial identity dataset. In The Thirteenth International Conference on Learning Representations, ICLR 2025.

W1.3&Q3: Performance on Real Celebrity

To further investigate the preservation of non-target knowledge under different forgetting ratios, we report the generation performance (Gen., MUSE / Original) and refusal scores (CR, on a 5-point scale) on the Retain and Real Celebrity sets.

Table 1. Unlearning performance on LLaVA-1.5-7B.

ModelRetain Gen.↑Retain CR↓Real Gen.↑Real CR↓
MUSE (5%)0.603 / 0.4900.190.336/0.2200.76
MUSE (10%)0.603/0.4960.210.332/0.2250.81
MUSE (15%)0.520/0.5000.950.288/0.2241.42

Although the refusal scores fluctuate with the forgetting ratio, the generation performance does not exhibit a comparable decline, suggesting that the underlying knowledge remains largely preserved. To better understand the discrepancy between these two evaluation tasks, we conducted a sample-level analysis of the generated responses. We found that some classification errors stem from non-answer responses produced by the model, which are treated as empty predictions under the classification evaluation protocol. This suggests that classification results may be affected by task-specific prompting and answer-format requirements, rather than directly reflecting the model's ability to retain the underlying knowledge. In contrast, open-ended generation allows the model to express its acquired knowledge more flexibly, thereby providing a complementary perspective on knowledge retention.

W2&Q2: All baselines edit weights.

To directly address this concern, we introduce two additional input-level baselines: (1) the system-prompt baseline from MLLMU-Bench and (2) a multimodal classifier gate that uses a two-layer MLP to classify image-question pairs. For Forget inputs, it returns a fixed refusal response ("I don't know"), while Retain inputs are passed to the vanilla MLLM for normal generation.

Table 2. Generation performance on LLaVA-1.5-7B (5% Forget).

ModelForget Gen.↓Retain Gen.↑
Vanilla0.5800.490
MUSE0.1050.603
System-prompt0.5950.499
Classifier Gate0.0000.593

The system-prompt baseline provides limited suppression, with a Forget score comparable to Vanilla (0.595 vs. 0.580). Although the classifier gate reduces the Forget score to zero, it relies on fixed refusal responses("I don't know"), which may reveal the presence of an unlearning mechanism and degrade user experience. In contrast, MUSE uses adaptive prompt embeddings to generate natural, context-aware refusals rather than relying on fixed templates (e.g., "I'm sorry, but I cannot provide an answer to this question. The image does not provide enough information to determine the individual's profession."). Importantly, MUSE achieves effective knowledge suppression (0.105 on Forget) while attaining the best Retain performance (0.603), demonstrating its advantage over simple input filtering.

AuthorsOpenReview ↗

Response to Reviewer GNXQ(W3,Q1)

W3: Novelty of MUSE

MUSE introduces a unified framework for multimodal unlearning through input-conditioned system-prompt embedding intervention. Its novelty lies in jointly designing multimodal-conditioned prompt adaptation and asymmetric dual-space optimization to achieve fine-grained knowledge access control.

Unlike Sysformer, which adapts system-prompt embeddings based on textual user prompts for LLM safety alignment, MUSE jointly conditions prompt embeddings on visual and textual inputs to enable instance-specific knowledge control in frozen MLLMs. Furthermore, its dual-space objective jointly regulates internal representations and prompt embeddings to establish a more precise forget–retain boundary. In multimodal unlearning, although forget samples can be identified, their ideal post-unlearning representations are generally unavailable. directly defining target representations for individual forget samples is impractical. Representation steering provides a natural solution by deriving a data-driven forgetting direction toward a no-knowledge reference region. Importantly, MUSE leverages this representation-level objective to supervise an external multimodal prompt adapter, rather than directly modifying the backbone's internal projection parameters as in ASRU. This enables representation-guided, input-conditioned and reversible knowledge intervention while keeping the MLLM backbone frozen.

We will clarify these methodological distinctions and further highlight MUSE's contributions in the revised manuscript.

Q1: Evaluation Consistency and Baseline Provenance

We clarify that the Vanilla and baseline results (except MANU) in Table 1 were adopted from ASRU, which we will explicitly acknowledge and cite in the revised manuscript. Regarding the evaluation judge, we additionally evaluated MUSE using GPT-4o-mini, following ASRU's evaluation protocol. The results are highly consistent with those obtained using GPT-5.4-mini. The small differences confirm that our evaluation conclusions are robust to the choice of judge model.

Table 3. Comparison of different LLM judges (5% Forget).

BackboneMetricGPT-5.4-miniGPT-4o-miniDiff.
LLaVA-1.5-7BCR4.444.400.04
LLaVA-1.5-7BForgetfulness4.404.420.02
Qwen3-VL-8BCR3.843.920.08
Qwen3-VL-8BForgetfulness4.103.940.16

We will update the evaluation description and clarify the provenance of the baseline results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer GNXQ

Dear Reviewer GNXQ,

I hope this message finds you well. Thank you very much for your valuable comments! We have carefully addressed each of your concerns in our rebuttal and hope that our additional experiments and clarifications have addressed your questions. We would be very grateful to hear your feedback. If any questions or concerns remain, we would be happy to discuss them further and provide additional clarifications.

Thank you again for your time and constructive suggestions!

Best regards,
All Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

Dear ACs/SACs/PCs,

We hope this message finds you well. We would like to sincerely express our gratitude for your dedicated efforts in coordinating the review process for this conference. We also appreciate the constructive feedback and valuable suggestions from all reviewers. As the discussion phase comes to an end, we would like to provide a brief summary of our responses.

To address the reviewers' concerns, we provided the following additional experiments and clarifications in our responses:

Clarification of the threat model and scope: We clarified MUSE's reversible knowledge access control setting and the distinction between multimodal and text-only unlearning. (Reviewer GNXQ)

Additional input-level and parameter-level baselines: We introduced system-prompt, classifier-gate, and LoRA baselines to validate the advantages of MUSE. (Reviewers GNXQ, xGT1)

Expanded evaluation and quantitative analysis: We added Cloze evaluation, analyzed Retain and Real performance across forgetting ratios, compared different LLM judges, and clarified baseline provenance. (Reviewers GNXQ, xGT1)

Further validation of the forgetting mechanism: We introduced random-direction and random-target baselines and clarified the construction of no-knowledge reference representations. (Reviewer xGT1)

Additional ablation and robustness analysis: We disentangled the contributions of multimodal input conditioning and dual-space objectives, and evaluated semantic knowledge leakage under prompt attacks. (Reviewer 8rN3)

Representative case studies: We added Forget and Retain examples to demonstrate context-aware refusals and preserved non-target responses. (Reviewers 8rN3, xGT1)

Clarification of methodological contributions: We further clarified the distinctions between MUSE, Sysformer, and ASRU. (Reviewer GNXQ)

Overall, Reviewer xGT1 acknowledged that we had addressed their concerns and raised the rating to 3, while Reviewer 8rN3 considered most concerns addressed and maintained their rating of 3. Reviewer xGT1 acknowledged our clarification that MUSE focuses on unlearning textual knowledge associated with visual inputs through jointly conditioned image-question pairs, addressing the concern regarding text-only evaluation. For Reviewer GNXQ, we further elaborated on the scope of text-only evaluation and the increased CR scores on Retain and Real under the 15% forgetting setting. We explained that MUSE largely preserves informative responses on non-target samples despite a moderate increase in refusal tendency. Unfortunately, due to the limited time remaining following the reviewer's further feedback, we were unable to conduct additional experiments. Nevertheless, we hope our clarifications help address the remaining concerns and further strengthen the reviewer's recognition of our work.


We would also like to highlight the key strengths and contributions of our paper recognized by the reviewers:

Lightweight and reversible knowledge intervention: MUSE keeps the MLLM backbone frozen and enables input-conditioned knowledge access control through a lightweight Multimodal Embedding Adapter, avoiding costly backbone fine-tuning and preserving the original model parameters. (As noted by Reviewers GNXQ, 8rN3, and xGT1)

Fine-grained multimodal input conditioning: MUSE generates system-prompt embeddings conditioned on individual image-question pairs, enabling instance-specific knowledge suppression and preservation beyond shared prompts or fixed refusal mechanisms. (As noted by Reviewers 8rN3 and xGT1)

Effective forgetting-retention trade-off: Experiments across different MLLM backbones and benchmarks demonstrate effective suppression of target knowledge while preserving non-target capabilities, supported by comprehensive ablations and robustness evaluations. (As noted by Reviewers GNXQ, 8rN3, and xGT1)

Practical efficiency and deployment potential: By keeping the backbone frozen and training only a lightweight adapter, MUSE substantially reduces training costs and GPU memory consumption, while allowing different forgetting scopes to be handled with separate adapters. (As noted by Reviewers GNXQ and 8rN3)

Overall, the reviewers recognized the practicality of reversible knowledge intervention, the benefits of multimodal input-conditioned prompting, and the potential of MUSE for efficient and fine-grained multimodal unlearning.

Finally, we believe that MUSE makes a valuable contribution to multimodal unlearning by introducing a lightweight and adaptive framework for input-conditioned knowledge access control, providing an alternative to conventional parameter-level unlearning approaches. We sincerely hope that our additional experiments and clarifications have further strengthened the methodological rigor and empirical evidence of our work.

Thank you again for your time, effort, and consideration. We greatly appreciate your support.

Sincerely,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.