acceptodds
Under review as a conference paper at ICLR 2027

CUSE: Fine-Grained Multimodal Unlearning with Counterfactual Guidance and On-Policy Self-Distillation

Abstract

Multimodal large language models (MLLMs) can reproduce sensitive, copyrighted, or unsafe information from their training data. Effective unlearning must remove target information while preserving non-target content, even within the same response. Existing methods typically apply uniform forgetting objectives to complete responses and rely on fixed, off-policy supervision, resulting in coarse token control and misalignment with the model’s current behavior. We propose CUSE, a Counterfactual-guided Unlearning framework via on-policy SElf-distillation for multimodal large language models. It first constructs counterfactual inputs by removing target-related evidence and uses the resulting changes in token probabilities to identify which tokens require stronger forgetting. It then distills token-level forgetting and retention targets on responses sampled from the current policy, with a frozen copy of the original model serving as the teacher model augmented with the corresponding forgetting- and retention-oriented privileged contexts. This design jointly determines what to forget and what to preserve. On MLLMU-Bench, CUSE reduces forget-set ROUGE-L by 33.6% while achieving 1.2% higher retain-set ROUGE-L than the strongest baseline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
MLLM Unlearning, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_uVqbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CUSE combines counterfactual token weighting with on-policy self-distillation for multimodal unlearning. A frozen original model compares token probabilities under original and target-replaced inputs to estimate sensitivity. These scores weight distillation from a teacher conditioned on forgetting instructions, while a retention branch preserves non-target capabilities. Experiments on MLLMU-Bench and CoVUBench report improved forgetting–utility trade-offs across three backbones overall.

Critical strengths
  1. Sections 3.1–3.3 explain sensitivity estimation, dual-branch distillation, and their combination in a clear sequence. Equations (6)–(12) and Algorithm 1 make the main procedure understandable.
  2. The reported improvements are substantial. In Table 1, CUSE reduces LLaVA’s forget-set classification accuracy to 11.20%, compared with 30.40% for ASRU, while retaining 60.51% accuracy versus the original model’s 60.70%. Table 5 extends evaluation to copyright unlearning. The comparisons include recent ASRU (2026).
  3. Table 2 evaluates on-policy versus fixed offline responses and sensitivity weighting versus uniform and random weighting. Relative to Uniform, CUSE reduces forget-set ROUGE-L from 0.192 to 0.129 while slightly improving retain-set ROUGE-L.
Critical weaknesses
  1. The novelty positioning omits ViKeR—Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning (Cai et al., 2026, arXiv:2601.22020). ViKeR already uses replacement visual inputs and frozen-model token distributions to prioritize sensitive tokens. Its objective differs, but the overlap requires discussion and qualification of the introduction’s broad uniform-supervision characterization. The relationship to privileged-context on-policy safety distillation by Fu et al. (2026), already cited in Appendix A.3, also deserves a more explicit distinction.
  2. The central within-response preservation claim is not directly established. Figure 1 motivates suppressing identity while preserving attributes such as hair color. Appendix B.4 instead evaluates general attributes through additional questions, and Appendix H presents separate forgetting and retention examples. These support preservation but do not directly test selective behavior within mixed-content responses.
  3. Important construction and evaluation details remain unclear. Appendix B.4 does not specify how same-attribute, different-identity portraits are obtained or checked, or how counterfactuals are constructed for copyright concepts. Appendix F does not explain the construction of forgetting reference responses. Appendix F.3 also leaves the evaluator and aggregation into a single naturalness score unspecified.
Questions for authors
  1. What distinguishes CUSE from ViKeR and privileged-context on-policy distillation beyond combining these ideas?
  2. What do the forgetting references contain, and do existing evaluations directly test mixed-content responses?
  3. Please clarify counterfactual construction, the naturalness evaluator, and score aggregation.
7 comments
AuthorsOpenReview ↗

Response 1

We sincerely appreciate your positive feedback and thoughtful suggestions. Below, we provide detailed responses to address your remaining concerns.

Comment 1: Discussion about the related work

We thank the reviewers for pointing out the missing discussion of ViKeR in our related work. We apologize for the incomplete discussion. We have revised the Introduction and Related Work accordingly to better clarify the positioning and contributions of CUSE. We further discuss ViKeR [1] and OPSA [2] and highlight their differences from CUSE as follows.

ViKeR employs distribution regularization, which to some extent induces implicit token-level reweighting. However, ViKeR still differs substantially from CUSE in its reweighting mechanism, sensitivity estimation, counterfactual construction, and supervision paradigm, with the following limitations:

  • Explicit vs. implicit reweighting. ViKeR introduces an ideal post-unlearning token distribution estimated from irrelevant visual inputs and uses KL regularization to implicitly induce different token-level gradient magnitudes. In contrast, CUSE explicitly computes counterfactual sensitivity scores from token-level log-probability differences and transforms them into normalized weights that directly modulate the forgetting KL objective. This provides an explicit mechanism to control the relative optimization strength across generation steps.

  • Static vs. dynamic sensitivity estimation. ViKeR estimates its reference distributions using the frozen pre-unlearning model, and these distributions remain fixed throughout subsequent optimization. Consequently, its token-level guidance does not adapt to changes in the student model's generation behavior. In contrast, CUSE continuously samples responses from the evolving student policy and recomputes sensitivity scores along its current generation trajectories. Thus, the forgetting weights adapt to evolving rollouts and their corresponding prefixes, rather than relying on pre-estimated reference distributions.

  • Counterfactual construction. ViKeR replaces the original image with irrelevant reference images of different individuals to estimate identity-agnostic token distributions. In contrast, CUSE performs targeted counterfactual interventions by replacing target-related visual evidence (e.g., the face region) while preserving non-target attributes such as hairstyle, clothing, and background. It also supports interventions on target-related textual evidence. This design aims to isolate the effect of target-related evidence while minimizing unnecessary changes to non-target visual information.

  • ViKeR uses the estimated distributions to regularize the original unlearning objective, thereby encouraging the model to move away from target-specific token predictions. While this approach improves coherence over conventional gradient ascent, distributional regularization alone does not explicitly specify the desired semantic behavior of the resulting response. Consequently, suppressing target-related predictions may lead to semantically unsupported alternative tokens and potentially hallucinated responses. In contrast, CUSE employs privileged-context teachers to provide explicit forgetting- and retention-oriented behavioral supervision, with counterfactual sensitivity determining the strength of forgetting supervision at each generation step.

Regarding OPSA [2], it applies on-policy self-distillation (OPSD) to safety alignment in text-only LLMs, which differs from our task of multimodal unlearning. Moreover, OPSA employs token-level KL distillation but does not consider fine-grained token-level reweighting based on the sensitivity of individual tokens to target-related information. In contrast, CUSE introduces counterfactual sensitivity estimation to explicitly quantify token-level sensitivity and dynamically reweight forgetting supervision along evolving on-policy trajectories.

AuthorsOpenReview ↗

Response 2

Overall, our contributions can be summarized as follows:

  • Problem analysis. We identify two coupled limitations of existing MLLM unlearning objectives: forgetting supervision is often applied uniformly across tokens, while existing token-level approaches may rely on fixed reference distributions or responses rather than the model's evolving generations. Consequently, forgetting supervision is rarely both fine-grained and on-policy.

  • Counterfactual sensitivity estimation. We introduce a counterfactual-based mechanism to explicitly estimate the sensitivity of each generated token to target-related visual and textual evidence through changes in conditional log probabilities. By continuously recomputing sensitivity scores along the evolving policy's on-policy rollouts, CUSE dynamically adjusts token-level forgetting weights to reflect changes in the model's generation behaviors throughout unlearning.

  • Sensitivity-aware on-policy self-distillation for unlearning. Building on privileged-context on-policy self-distillation, we adapt the dual-branch scheme to targeted multimodal unlearning with sample-specific forgetting and retention contexts. Specifically, we incorporate the proposed counterfactual sensitivity weights into the forgetting branch to explicitly control the relative strength of supervision across generation steps, while the retention branch preserves non-target knowledge and capabilities. Together, the two branches enable CUSE to selectively suppress target-related information while maintaining general multimodal capabilities, achieving a better balance between forgetting effectiveness and capability preservation.


Comment 2: Within-Response Selective Preservation

We thank the reviewer for raising this important concern. We construct an additional within-response mixed-content evaluation set through data augmentation based on the Forget Set. Specifically, we create queries that simultaneously ask about a target-sensitive attribute (e.g., a person's profession) and a non-sensitive visual attribute. For the latter, we focus primarily on visually distinguishable attributes, such as hair color, that should remain accessible after unlearning. The expected behavior is to withhold the sensitive information while still providing an accurate description of the non-sensitive visual attribute within the same response.

We report the reduction in ROUGE-L for the sensitive-information component (Δ ROUGE-L) and the accuracy of the non-sensitive visual concept description (Visual Concept Accuracy). The results are shown below.

MethodΔ ROUGE-L ↑Visual Concept Accuracy (%) ↑
GA23.50.0
GA_Diff11.93.4
NPO30.26.2
KL_Min10.16.7
MMUnlearner18.424.0
MANU21.523.4
ASRU30.745.1
CUSE49.261.0

The results demonstrate that CUSE effectively balances sensitive-information suppression and non-sensitive attribute preservation within the same response. It withholds target-sensitive information while retaining the ability to accurately describe non-sensitive visual attributes.


AuthorsOpenReview ↗

Response 3

Comment 3: More Experimental Details

Discussion of reference answer

We thank the reviewer for raising this important concern. We clarify that CUSE uses different privileged-context designs depending on whether a query requests only target attributes or combines sensitive and non-sensitive information.

For attribute-only queries, as described in Appendix, we provide the forgetting teacher with a reference refusal response. Specifically, we prepare a set of general refusal expressions (e.g., "I'm not sure" and "I'm unable to provide an answer to that") and randomly select one as the reference response for each sample. We use these general refusal references because the desired post-unlearning behavior should prevent disclosure while maintaining response coherence and readability. We have included the complete reference-answer template in the revised manuscript. Although a generic refusal response is provided as a reference, CUSE does not directly fine-tune the student on this fixed response. Instead, the frozen teacher provides token-level distributional supervision conditioned on the original image, query, and generation prefix. Through on-policy distillation, the student learns to suppress target-related information in its own generated responses. As illustrated by the complete example in Appendix I, the resulting response can be adapted to the specific query rather than simply reproducing the reference refusal verbatim. More generally, the desired post-unlearning response need not be a refusal: a sample-specific alternative answer could also serve as the reference. We choose refusal references because they provide a coherent and readable way to withhold the target information without introducing fabricated or unsupported facts [3]. Simply generating arbitrary or seemingly plausible alternative answers may result in hallucinations and make the unlearning behavior less reliable.

For queries involving both sensitive and non-sensitive content, we explicitly design the privileged context to distinguish between the different types of information. It first instructs the teacher not to disclose the target-sensitive attributes, while emphasizing that unrelated visual descriptions should still be answered normally rather than being unnecessarily refused. We also provide the ground-truth answer to the entire query, covering both sensitive and non-sensitive information, as a reference in the privileged context. This allows the teacher to provide selective behavioral supervision: withholding the sensitive information while retaining useful visual descriptions within the same response.

Importantly, CUSE is not restricted to refusal-based unlearning. Its desired post-unlearning behavior is flexible and can be specified through the privileged context. By simply modifying the textual instructions and reference responses in the privileged context, we can guide the teacher toward different types of desired behaviors, such as refusing to disclose target information, providing alternative responses, or preserving selected non-target information. The underlying distillation framework remains unchanged.


Construction of Counterfactual Images

We thank the reviewer for this question. Across both benchmarks, counterfactual inputs follow the same principle: remove target-related evidence while preserving non-target information as much as possible. The two benchmarks differ only in what constitutes target-related evidence. All counterfactual images are pre-generated offline with an image-editing model (Nano Banana) and serve only for sensitivity estimation, not as supervision targets.

(1) MLLMU-Bench. The forgetting target is the identity of a fictitious individual and its profile, which is primarily carried by the face, whereas hairstyle, clothing, and background are identity-irrelevant. Since the profile images of MLLMU-Bench were generated with ThisPersonDoesNotExist, we sample substitute faces from the same source and keep only those whose face-recognition similarity (ArcFace) to every individual in MLLMU-Bench falls below the same-identity threshold. Each forget individual is paired with a substitute of similar gender and age, which serves only as the identity reference: the editing model replaces the face in the original image while keeping hairstyle, clothing, accessories, expression, and background unchanged. For questions that name the individual, the name is replaced with a new fictitious name.

Reviewer_ovQxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies unlearning for multimodal large language models (MLLMs). It argues that existing methods suffer from two interrelated limitations:

  • Lack of fine-grained forgetting control. Forgetting objectives are applied uniformly across all tokens of a response, even though only a subset of tokens (e.g., those revealing a person's identity) depends strongly on target-related evidence. Other tokens, such as those describing clothing or surroundings, mainly support visual grounding and linguistic coherence.
  • Supervision–behavior misalignment. Forgetting supervision is typically off-policy, relying on fixed forget-set responses or predefined refusal templates rather than on responses generated by the current model. Existing RL-based methods are on-policy but provide only sequence-level rewards.

To address these limitations, the paper proposes CUSE, which combines two components:

  1. Counterfactual sensitivity estimation. For each forget sample, a counterfactual input is constructed by replacing target-related evidence: same-attribute, different-identity portraits for visual inputs, and target-replaced entities for textual queries. Using the frozen original model, token-level sensitivity is measured as the log-probability difference Δt\Delta_t of each generated token under the original versus the counterfactual input. Positive shifts are thresholded by a batch-level percentile and converted into per-response normalized weights w^t\hat{w}_t.
  2. Dual-branch on-policy self-distillation. The student samples responses from its current policy. A frozen copy of the original model serves as the teacher, conditioned on privileged contexts: a forgetting-oriented context for forget samples and a retention-oriented context for retain samples, each containing behavioral instructions and a reference response. The student is trained to match the teacher via token-level forward KL, and the forgetting-branch KL terms are weighted by the counterfactual sensitivity weights.

Claimed contributions:

  • identifying the lack of token-level control and the off-policy nature of supervision as key limitations of current MLLM unlearning;
  • the CUSE framework, which integrates counterfactual sensitivity-aware token reweighting with dual-branch on-policy self-distillation;
  • improved forgetting and utility preservation over existing baselines.

Validation. Experiments are conducted on two benchmarks:

  • MLLMU-Bench (fictitious-profile privacy unlearning, with classification, generation, and cloze tasks), using LLaVA-1.5-7B and Qwen2.5-VL-7B-Instruct;
  • CoVUBench (copyright concept unlearning, with six metrics), using LLaVA-1.5-7B and Qwen3-VL-4B-Instruct.

Baselines include GA, GA-Diff, KL-Min, NPO, MMUnlearner, MANU, and ASRU. The paper reports results at 5%, 10%, and 15% forgetting ratios (on LLaVA), along with:

  • an ablation comparing off-policy, uniform-weight, and random-weight variants;
  • sensitivity to the reweighting coefficient γ\gamma and training dynamics;
  • robustness under random-prefix, paraphrase, jailbreak, and sampling-based perturbations;
  • LLM-judged forgetting naturalness and a visual concept accuracy evaluation;
  • qualitative case studies.

On MLLMU-Bench, the authors report a 33.6% reduction in forget-set ROUGE-L and a 1.2% increase in retain-set ROUGE-L relative to the strongest baseline.

Critical strengths
  • Relevant and well-motivated problem. The observation that forget-set responses mix target-related and non-target content is pertinent to MLLM unlearning, where an identity often co-occurs with visual descriptions that should be preserved. The two stated limitations, uniform token-level supervision and off-policy supervision, are clearly articulated and connected.
  • Novel combination of techniques. Introducing on-policy self-distillation (OPSD) with privileged contexts into MLLM unlearning appears new in this setting. Using counterfactual interventions on the input to estimate token-level sensitivity is intuitive and naturally exploits the multimodal structure of the task.
  • Informative ablation controls. Beyond the uniform-weight variant, the ablation includes a random-weight control. This helps rule out the explanation that any non-uniform weighting would yield similar gains, and supports the claim that the counterfactual signal itself is useful.
  • Broad experimental coverage. The evaluation spans two benchmarks with different unlearning targets (privacy-oriented profiles and copyrighted concepts), three MLLM backbones, multiple forgetting ratios, and a range of supplementary analyses: hyperparameter sensitivity, training dynamics, perturbation robustness, naturalness evaluation, and case studies.
  • Reproducibility. Implementation details, including LoRA configuration, learning rates, and the full privileged-context and evaluation prompt templates, are provided, and code is released.
Critical weaknesses
  1. The forgetting branch effectively distills a refusal template, which contradicts the paper's own motivation. The forgetting privileged context (Appendix F.2) contains a {reference} response that the teacher is instructed to treat as "the complete and only allowed answer." The paper never specifies what this reference is. However, the case study (Appendix H.1) shows CUSE answering "I'm sorry, but I cannot provide information regarding this individual's profession," which suggests a refusal response. If so, the forgetting branch amounts to distilling a predefined refusal into the model through the teacher's context. Yet the introduction explicitly criticizes existing methods for relying on "predefined refusal templates" that "can induce over-refusal." Similarly, the retention context includes the ground-truth answer, which makes the retention branch close to soft-label SFT on the retain set. The paper should specify the forget-side reference, explain how CUSE differs in substance from refusal-based unlearning, and evaluate over-refusal on non-target inputs.

  2. The central "fine-grained" claim is not demonstrated. The motivation (Fig. 1) is that an unlearned model should withhold the target identity while still describing non-target content in the same response, such as clothing, hairstyle, or surroundings. CUSE's resulting behavior, however, is a full-sentence refusal, so within-response preservation of non-target content is never shown. The token weights w^t\hat{w}_t modulate how strongly each position is pushed toward the teacher distribution during training, but they do not by themselves yield responses that omit only target-related content. Moreover, as the student increasingly samples refusals, Δt\Delta_t for refusal tokens should approach zero, so the weights may degenerate toward uniform. The evolution of the weight distribution during training is not reported. The only supporting evidence, Visual Concept Accuracy, relies on questions the authors added to MLLMU-Bench, but their construction, number, and relation to the forget images are not described.

  3. Key baselines are missing, and the off-policy ablation is underspecified. Since CUSE's end behavior is refusal, the most natural baselines are:

    • refusal ("I don't know") fine-tuning combined with retain-set SFT, as in the refusal-based baselines of TOFU [1];
    • preference optimization with refusal responses as preferred outputs;
    • sequence-level, off-policy context distillation, in which the privileged-context teacher generates responses once and the student is fine-tuned on them.

    None of these is included. Without them, it is unclear whether the gains stem from token-level on-policy distillation or simply from training toward a refusal target with strong retain supervision. In addition, the "Off-policy" ablation variant barely forgets anything (forget classification accuracy 57.00% vs. 59.20% for the original model), and "fixed offline responses" is not defined. If these are the ground-truth forget answers, the comparison does not isolate the benefit of on-policy sampling.

  4. The evaluation conflates refusal with knowledge removal.

    • Forget metrics reward refusal. CUSE's forget-set classification accuracy (11.20% on LLaVA, 13.60% on Qwen2.5-VL) falls below the chance level of the four-option multiple-choice format. This indicates refusal or systematic avoidance of the correct option rather than forgetting, and below-chance accuracy can itself leak information. Likewise, ROUGE-L against ground-truth answers trivially approaches zero for refusals.
    • No test of whether the knowledge remains. Because the teacher is the original model instructed to refuse, the target knowledge plausibly remains in the parameters. Refusal-based approaches are known to be vulnerable when the underlying knowledge persists [6]. Yet no relearning attacks [4, 5], membership inference, or representation probing are reported.
    • Weak robustness evaluation. The robustness evaluation (Table 3) reports only CUSE, without baselines, and uses relatively weak perturbations.
    • Naturalness evaluation favors refusal. The judge prompt instructs the judge not to penalize refusals and scores readability and helpfulness, which favors fluent refusals. Accordingly, the two refusal-based methods rank first and second (CUSE 4.64, ASRU 3.43). The judge model is also not specified.
  5. Reliability of results and missing related work.

    • Unverifiable claims and anomalous numbers. The abstract's claim of a 33.6% reduction in forget-set ROUGE-L relative to the strongest baseline cannot be derived from Table 1 (ASRU → CUSE is 0.358 → 0.129 on LLaVA and 0.305 → 0.142 on Qwen2.5-VL). Some values are anomalous: the LLaVA 5% Test ROUGE of 0.009 is far below the forget-set value (0.129) and the 10% / 15% Test values (0.114 / 0.122). On CoVUBench, Generality (99.75) exceeds Efficacy (99.23).
    • Incomplete experimental reporting. All results come from single runs without variance. Baseline hyperparameters are not reported. Results at 10% and 15% forgetting are given only for LLaVA. The computational cost of per-step sampling, teacher inference, and counterfactual forward passes is not quantified. How counterfactual inputs are obtained, especially "same-attribute, different-identity portraits" and counterfactuals for copyrighted concepts in CoVUBench, is not described.
    • Missing related work. The claim that existing methods treat all tokens uniformly overlooks token-level reweighting methods for LLM unlearning, such as WGA [2] and SatImp [3], which are neither cited nor compared. Liu et al. (2024c) [7], which frames targeted unlearning from a causal-intervention perspective, is cited but its relation to the proposed counterfactual intervention is not discussed.

References

[1] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, J. Zico Kolter. TOFU: A Task of Fictitious Unlearning for LLMs. COLM 2024.

[2] Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, Kilian Q. Weinberger. Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond. ICLR 2025.

[3] Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, Bo Han. Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning. ICML 2025.

[4] Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, Virginia Smith. Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. ICLR 2025.

[5] Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, Sijia Liu. Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond. ICML 2025.

[6] Puning Yang, Junchi Yu, Qizhou Wang, Philip Torr, Bo Han, Xiuying Chen. Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning. ICML 2026.

[7] Yujian Liu, Yang Zhang, Tommi S. Jaakkola, Shiyu Chang. Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective. EMNLP 2024.

Questions for authors
  1. Forget-side reference and over-refusal.

    • What exactly is the {reference} response in the forgetting privileged context (Appendix F.2)? Is it a fixed refusal template, a set of templates, or something generated per sample?
    • How does CUSE differ in substance from distilling a refusal response into the model?
    • Could you report over-refusal rates on the retain and real-celebrity sets?
    • Could you provide examples on queries that mix target and non-target information (e.g., asking for both a person's identity and their clothing), showing whether CUSE withholds only the target-related part, as motivated in Fig. 1?
  2. Behavior of the sensitivity weights.

    • How does the distribution of ata_t and w^t\hat{w}_t evolve during training? In particular, what fraction of positions has at>0a_t > 0 once the student mostly samples refusals?
    • Please describe the preliminary analysis behind the claim that about 20% of generated tokens are highly sensitive to target-related evidence. Which model, data, and threshold were used?
  3. Baselines and ablation definitions.

    • How does CUSE compare against refusal fine-tuning combined with retain-set SFT, preference optimization with refusal responses as preferred outputs, and sequence-level off-policy context distillation from the privileged-context teacher?
    • What exactly are the "fixed offline responses" in the Off-policy ablation variant?
    • What hyperparameters and tuning protocol were used for each baseline?
  4. Evidence of knowledge removal.

    • Could you evaluate CUSE and the baselines under relearning attacks [4, 5], and report representation probing or membership-inference results?
    • Could you add baseline results to the robustness evaluation in Table 3?
    • On the classification task, how are refusals scored, and how many options does each question have?
    • Which model serves as the judge in the naturalness evaluation, and how does the judge prompt treat refusals compared with informative but non-disclosing answers?
  5. Reported numbers and experimental details.

    • How is the 33.6% reduction in forget-set ROUGE-L in the abstract computed?
    • Could you check the LLaVA 5% Test ROUGE of 0.009, which is far below both the forget-set value and the Test values at 10% and 15%?
    • On CoVUBench, why does Generality (99.75) exceed Efficacy (99.23)?
    • Could you report mean and standard deviation over multiple seeds, as well as the computational cost (wall-clock time and memory) relative to the baselines?
    • How are the "same-attribute, different-identity portraits" obtained, how are counterfactual inputs constructed for copyrighted concepts in CoVUBench, and how were the Visual Concept Accuracy questions constructed?

Minor comments

  • Outdated references. NPO and TOFU are cited as arXiv preprints but have been published (both COLM 2024).

  • Missing citation. MMUnlearner is mentioned in Appendix A.2 without a citation.

  • Subjective case-study labels. The labels in Appendix H (e.g., "hallucination" vs. "fail to unlearn") are assigned without a stated criterion. Note also that a plausible but incorrect answer can be viewed as a form of non-disclosure rather than a failure.

    • Missing related work. Several directly relevant works are neither cited nor discussed:
      • Token-level reweighting. The claim that existing methods treat all tokens uniformly overlooks token-level reweighting methods for LLM unlearning, such as WGA [2] and SatImp [3].
      • Counterfactual intervention. Liu et al. (2024c) [7], which frames targeted unlearning from a causal-intervention perspective, is cited, but its relation to the proposed counterfactual intervention is not discussed.
      • MLLM privacy unlearning. FIUBench [8] studies essentially the same fictitious facial-identity unlearning setting for VLMs and provides membership-inference and adversarial privacy-extraction evaluations. These would directly test whether CUSE removes knowledge or only suppresses it.
      • Refusal-based training. Safety Mirage [9] shows that refusal-oriented supervised fine-tuning of VLMs learns spurious shortcuts between surface input patterns and refusals, causing over-prudence on benign queries. This bears directly on CUSE's forgetting branch.
      • Same mechanism as an existing OPSD method. Fu et al. (2026), cited in the paper only as an application of OPSD, uses a privileged-context self-teacher to elicit latent refusal for safety alignment. This is essentially the mechanism of CUSE's forgetting branch, so the distinction should be made explicit.
      • Specifying post-unlearning behavior. TRU [10] and FLAT [12] explicitly specify the post-unlearning response, which corresponds closely to CUSE's reference-conditioned privileged context. EUA [6] and DRAGON [11] discuss refusal- and detection-based unlearning, which is the natural point of comparison for positioning CUSE. In addition, [13] studies honesty in LLM unlearning, which is relevant to how the case study labels plausible but incorrect answers as "hallucination."

[8] Yingzi Ma et al. Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset. ICLR 2025.

[9] Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, Sijia Liu. Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning. ICLR 2026.

[10] Junfeng Liao, Qizhou Wang, Shanshan Ye, Xin Yu, Ling Chen, Zhen Fang. Explainable LLM Unlearning Through Reasoning. ICLR 2026.

[11] Yaxuan Wang, Chris Yuhao Liu, Quan Liu, Jinglong Pang, Wei Wei, Yujia Bao, Yang Liu. DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning. ICLR 2026.

[12] Yaxuan Wang, Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Parag Shah, Yujia Bao, Yang Liu, Wei Wei. LLM Unlearning via Loss Adjustment with Only Forget Data. ICLR 2025.

[13] Renjie Gu, Jiazhen Du, Yihua Zhang, Sijia Liu. Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning. ACL 2026.

9 comments
AuthorsOpenReview ↗

Response 1

We sincerely appreciate your positive feedback and thoughtful suggestions. Below, we provide detailed responses to address your remaining concerns.

Comment 1: Clarification of the Forgetting Mechanism and Over-refusal

We thank the reviewer for raising these important concerns. We clarify the rationale behind our choice of post-unlearning responses, the role of privileged-context references, and the distinction between CUSE and directly distilling predefined refusal templates.

On the choice of desired post-unlearning behavior

While preventing the disclosure of target information is the common objective of unlearning, what the model should generate after unlearning remains an important design choice. Direct optimization methods such as gradient ascent can severely disrupt the model's generation behavior, sometimes resulting in incoherent or unintelligible responses. Another possible strategy is to construct seemingly plausible alternative answers [1]. For example, when the original answer is "professor," the model may be encouraged to respond with another occupation, such as "engineer." Although such responses may remain fluent, they introduce fabricated factual information and can be considered hallucinations. Such fabricated answers may also serve as identifiable artifacts of the unlearning process.

In this work, we choose refusal responses as the desired behavior. This provides a coherent and readable way to withhold target information without introducing fabricated or unsupported facts. We adopt the preference strings from TOFU [2], randomly selecting one reference refusal for each forget sample. The complete set of reference responses has been added to Appendix G of the revised manuscript.


Why CUSE is not simply distilling predefined refusal templates

Importantly, the reference refusal is provided as privileged context to the frozen teacher, rather than being used as a fixed sequence for directly fine-tuning the student. The teacher generates token-level distributions conditioned on the original image, query, and the student's current generation prefix. Through on-policy self-distillation, the student learns from these context-dependent distributions along its own generation trajectories, rather than simply memorizing the predefined reference strings. As illustrated in our case study (Appendix I), although the reference responses are generic, the resulting responses can be tailored to the specific query. For example, the model produces a profession-specific refusal rather than mechanically reproducing a generic expression such as "I'm not sure."

More importantly, CUSE is not restricted to refusal-based unlearning. Its desired post-unlearning behavior is flexible and can be specified through the privileged context. By modifying the textual instructions and reference responses, we can guide the teacher toward different desired behaviors, such as withholding target information, providing alternative responses, or preserving selected non-target information, without changing the underlying distillation framework. Thus, refusal is a practical choice of reference behavior in our current experiments.


Why the retention branch is not equivalent to soft-label SFT

Similarly, the ground-truth answers in the retention privileged context serve as references for conditioning the teacher, rather than direct supervised targets used to compute the student's training loss. Following the OPSD [3] paradigm, the frozen teacher takes the privileged context and the student's current on-policy prefix as input and provides token-level distributional supervision for the student's ongoing generation. This differs from conventional SFT, which directly optimizes the student against fixed reference sequences. On-policy distillation instead aligns the teacher's distribution with the student's evolving generation trajectories and provides richer distributional supervision than fixed response-level targets.

To empirically examine this distinction, we introduce a Direct SFT baseline that directly fine-tunes the student on responses generated by the privileged teacher, without on-policy distillation or counterfactual sensitivity-aware weighting. We conduct this comparison on LLaVA-1.5-7B under the 5% forgetting setting of MLLMU-Bench.

SettingClassification Fgt ↓Classification Test ↓Classification Ret ↑Classification Cele ↑Generation Fgt ↓Generation Test ↓Generation Ret ↑Generation Cele ↑Cloze Fgt ↓Cloze Test ↓Cloze Ret ↑Cloze Cele ↑
Vanilla59.2044.8060.7040.210.6450.2890.5840.31254.0026.0055.478.17
CUSE11.2014.4060.5139.610.1290.1090.5540.31010.006.0055.898.17
Direct SFT32.8038.4041.7329.500.3290.2170.3800.21612.0014.0010.402.60
AuthorsOpenReview ↗

Response 2

The results reveal that Direct SFT provides limited forgetting improvements but substantially compromises the preservation of non-target knowledge. In contrast, CUSE achieves considerably stronger forgetting across classification, generation, and cloze tasks, while maintaining much better performance on the retain and celebrity sets. These results demonstrate that directly learning from fixed teacher-generated responses is insufficient to achieve the forgetting--retention balance.


Over-refusal and within-response selective preservation

To evaluate whether CUSE induces excessive refusal behavior, we calculate the average refusal rates across the Retain and real-celebrity (Cele) sets for each of the three evaluation tasks. The results are reported below.

TaskClozeClassificationGeneration
Refusal Rate (%) ↓0.210.381.84

These low refusal rates suggest that CUSE does not indiscriminately reject queries.

Furthermore, to directly examine whether CUSE preserves non-target information within the same response, we construct an additional mixed-content evaluation set through data augmentation based on the Forget Set. Each query simultaneously asks about a target-sensitive attribute (e.g., profession) and a non-sensitive visual attribute (e.g., hair color). We measure sensitive-information suppression using the reduction in ROUGE-L (Δ\Delta ROUGE-L) and non-sensitive visual preservation using Visual Concept Accuracy. The results are reported below.

MethodΔ ROUGE-L ↑Visual Concept Accuracy (%) ↑
GA23.50.0
GA_Diff11.93.4
NPO30.26.2
KL_Min10.16.7
MMUnlearner18.424.0
MANU21.523.4
ASRU30.745.1
CUSE49.261.0

The results demonstrate that CUSE effectively suppresses target-sensitive information while preserving non-sensitive visual descriptions within the same response, rather than simply refusing the entire query.

Overall, CUSE does not simply distill predefined refusal templates or perform soft-label SFT on ground-truth answers. Instead, it uses privileged contexts to specify desired behaviors, while the frozen teacher provides context-dependent distributional supervision along the student's evolving on-policy trajectories. Combined with counterfactual sensitivity-aware weighting, this framework enables selective forgetting while preserving non-target information and general response capabilities.


Comment 2: Discussion about Fine-Grained Unlearning

We thank the reviewer for raising these important concerns. We address them through additional token-level analyses and a more detailed description of our mixed-content evaluation.

Token-level sensitivity and weight evolution

We have added a detailed token-level analysis in Appendix E of the revised manuscript. To investigate whether counterfactual sensitivity aligns with the actual disclosure of sensitive information, we use DeepSeek-V4.1-Flash to annotate generated tokens in the model's on-policy rollouts into three categories: (1) sensitive tokens that disclose information targeted for unlearning, (2) other tokens that convey non-sensitive content, and (3) refusal tokens that express appropriate refusal behaviors (e.g., "cannot provide" or "cannot answer"). We then examine the sensitivity scores, corresponding optimization weights, and token-count distributions of these categories throughout unlearning.

The results reveal a clear correspondence between semantic annotations and counterfactual sensitivity. Tokens annotated as sensitive generally exhibit larger counterfactual log-probability differences (\Delta_t), and consequently receive higher sensitivity-aware token weights during unlearning. In contrast, other and refusal tokens receive relatively lower weights. Furthermore, as optimization progresses, the number of sensitive tokens in the on-policy rollouts progressively decreases, suggesting that the model's tendency to disclose target-related information is increasingly suppressed. The detailed visualizations of sensitivity scores, token weights, and token-count distributions are provided in Appendix E.

Regarding the approximately 20% proportion of highly sensitive tokens discussed in the paper, this estimate comes from our preliminary token-level analysis using LLaVA-1.5-7B on MLLMU-Bench. Specifically, based on the semantic annotations of generated responses, sensitive tokens account for approximately 20% of the tokens per response on average.


AuthorsOpenReview ↗

Response 3

Within-response selective preservation

To further demonstrate that CUSE can preserve non-target information within the same response, we construct an additional mixed-content evaluation set through data augmentation based on the Forget Set. Specifically, for each original forget query, we retain the original question about the target-sensitive attribute (e.g., a person's profession) and augment it with an additional question about a non-sensitive visual attribute (e.g., hair color or hairstyle). The additional visual questions focus on directly observable attributes that should remain accessible after unlearning. We create one mixed-content version for every original forget query, resulting in a mixed-content evaluation set with exactly the same number of queries as the original Forget Set.

For these mixed-content queries, we explicitly design the privileged context to distinguish between the different types of information. It instructs the teacher not to disclose target-sensitive attributes, while emphasizing that unrelated visual descriptions should still be answered normally and accurately. We also provide the ground-truth answer to the entire mixed-content query, covering both sensitive and non-sensitive information, as a sample-specific reference in the privileged context. This allows the teacher to provide selective behavioral supervision: withholding sensitive information while retaining useful visual descriptions within the same response.

Comment 3: More Comparisons with Additional Baselines

We thank the reviewer for these valuable suggestions. To provide a more comprehensive evaluation, we have included additional comparisons with three relevant baselines:

  • Refusal SFT with retain-set SFT: We combine refusal-based supervised fine-tuning on the forget set with standard SFT on the retain set, following the refusal-based training paradigm in TOFU.

  • Preference optimization with refusal responses: We introduce a preference optimization baseline that treats refusal responses as preferred outputs for forget samples, while retaining the corresponding supervision for non-target knowledge.

  • Sequence-level off-policy context distillation: We compare against a sequence-level distillation baseline in which the privileged-context teacher generates responses offline, which are then used as fixed targets for student fine-tuning.

ModelCls Fgt ↓Cls Test ↓Cls Ret ↑Cls Cele ↑Gen Fgt ↓Gen Test ↓Gen Ret ↑Gen Cele ↑Cloze Fgt ↓Cloze Test ↓Cloze Ret ↑Cloze Cele ↑
Vanilla59.2044.8060.7040.210.6450.2890.5840.31254.0026.0055.478.17
IDK SFT31.2039.2052.3030.500.3490.1800.3230.19718.0016.0020.703.30
DPO52.8042.4055.3035.500.5850.2770.5450.28436.0024.0053.425.20
Offline23.2040.8059.5033.400.1350.2470.5130.26516.0022.0052.107.84
CUSE11.2014.4060.5139.610.1290.1090.5540.31010.006.0055.898.17

The three baselines correspond to the three additional comparisons described above: IDK SFT represents refusal SFT combined with retain-set SFT, DPO represents preference optimization with refusal responses as preferred outputs, and Offline represents sequence-level off-policy context distillation.

The results reveal distinct limitations of these approaches. IDK SFT substantially compromises the model's ability to retain non-target knowledge, indicating that directly fine-tuning the model toward refusal responses can lead to considerable degradation in its general capabilities. DPO, on the other hand, exhibits limited forgetting effectiveness, suggesting that preference optimization alone is insufficient to reliably suppress the targeted information. Offline achieves relatively effective forgetting on the target set but generalizes poorly to unseen test samples. This suggests that distilling fixed responses generated by the privileged-context teacher is insufficient to adapt to diverse inputs, highlighting the importance of on-policy interaction during unlearning. In contrast, CUSE achieves a more favorable balance between forgetting effectiveness and knowledge retention. It effectively suppresses the targeted information while better preserving non-target knowledge and generalizing to unseen test inputs. These results further demonstrate that the improvements of CUSE do not simply stem from learning refusal responses or distilling fixed teacher-generated targets, but benefit from the proposed on-policy distillation and counterfactual sensitivity-aware optimization.

Reviewer_R6eEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CUSE, a fine-grained multimodal unlearning framework that combines counterfactual sensitivity-aware token reweighting with on-policy self-distillation. Specifically, CUSE constructs counterfactual inputs by removing target-related evidence and measures token-level probability changes to identify tokens requiring stronger forgetting supervision. It further employs a dual-branch on-policy self-distillation strategy, where a frozen teacher model conditioned on forgetting- and retention-oriented privileged contexts provides token-level supervision for responses generated by the current student policy.

Critical strengths
  1. CUSE integrates counterfactual sensitivity-aware token reweighting with dual-branch on-policy self-distillation. The overall framework is coherent, and the proposed components are designed to improve forgetting effectiveness while preserving non-target capabilities.

  2. The experiments demonstrate substantial improvements in forgetting effectiveness while largely maintaining retained utility across multiple MLLM backbones and benchmarks. The ablation studies and robustness evaluations further provide empirical support for the effectiveness of the proposed approach.

  3. The paper is generally well-organized and clearly written. The methodological descriptions, illustrations, and experimental analyses make the proposed framework easy to follow.

Critical weaknesses

W1: The proposed counterfactual sensitivity score measures how strongly a generated token depends on target-related evidence, but does not distinguish between tokens that disclose sensitive information and those that represent appropriate refusal or non-sensitive responses. Therefore, it remains unclear whether higher sensitivity necessarily indicates a greater need for forgetting-oriented supervision

W2: The existing off-policy ablation, which uses fixed offline responses, does not adequately address this concern, as it does not evaluate direct SFT on privileged teacher-generated outputs. Without this baseline, it remains unclear whether the improvements primarily arise from the privileged supervision itself or from the proposed on-policy distillation and counterfactual sensitivity weighting. This comparison is essential for establishing the necessity and practical advantage of the proposed framework over a considerably simpler alternative.

Questions for authors
  1. Can authors provide token-level analyses demonstrating that the proposed weighting mechanism preferentially targets sensitive disclosures rather than safe responses?

  2. Could the authors compare CUSE with a simpler SFT baseline that directly fine-tunes the student on responses generated by the privileged teacher?

  3. Could the authors conduct additional experiments on a more recent MLLM backbone, such as Qwen3.5-VL, to further validate the generalizability and effectiveness of CUSE across different model architectures?

I am willing to raise my overall assessment of the paper if the authors adequately address my concerns.

4 comments
AuthorsOpenReview ↗

Response 1

We sincerely appreciate your positive feedback and thoughtful suggestions. Below, we provide detailed responses to address your remaining concerns.

Comment 1: Token-level Analyses

We thank the reviewer for raising this important concern. To investigate whether counterfactual sensitivity aligns with the actual disclosure of sensitive information, we conduct an additional token-level analysis on the model's on-policy rollouts. Specifically, we use DeepSeek-V4.1-Flash to annotate generated tokens into three categories: (1) sensitive tokens that disclose information targeted for unlearning, (2) other tokens that convey non-sensitive content, and (3) refusal tokens that express appropriate refusal behaviors (e.g., "cannot provide" or "cannot answer"). We then examine the sensitivity scores, corresponding optimization weights, and token-count distributions of these categories throughout unlearning. The detailed visualizations and corresponding analyses are provided in Appendix E of the revised manuscript.

The results reveal a clear correspondence between semantic annotations and counterfactual sensitivity. Tokens annotated as sensitive generally exhibit larger counterfactual log-probability differences Δt\Delta_t, and consequently receive higher optimization weights during unlearning. In contrast, other and refusal tokens receive relatively lower weights, indicating that the proposed mechanism preferentially allocates forgetting supervision to tokens associated with sensitive information disclosure rather than indiscriminately emphasizing all generated tokens. Furthermore, as optimization progresses, the number of sensitive tokens in the on-policy rollouts progressively decreases, suggesting that the model's tendency to disclose target-related information is effectively suppressed.

Overall, these analyses indicate a correspondence between counterfactual sensitivity and sensitive information disclosure, suggesting its potential utility in guiding token-level forgetting supervision. Although sensitivity is not intended to serve as a perfect semantic classifier, it provides an effective signal for allocating stronger forgetting supervision to target-related generation steps while reducing unnecessary optimization of non-sensitive and refusal-related content.


Comment 2: Comparison with a Direct SFT Baseline

We thank the reviewer for this valuable suggestion. To directly address this concern, we introduce a simpler Direct SFT baseline, which fine-tunes the student model on responses generated by the privileged teacher, without the proposed on-policy distillation and counterfactual sensitivity-aware weighting mechanisms. We conduct this comparison on LLaVA-1.5-7B under the 5% forgetting setting of MLLMU-Bench. The results are reported below.

SettingClassification Fgt ↓Classification Test ↓Classification Ret ↑Classification Cele ↑Generation Fgt ↓Generation Test ↓Generation Ret ↑Generation Cele ↑Cloze Fgt ↓Cloze Test ↓Cloze Ret ↑Cloze Cele ↑
Vanilla59.2044.8060.7040.210.6450.2890.5840.31254.0026.0055.478.17
CUSE11.2014.4060.5139.610.1290.1090.5540.31010.006.0055.898.17
Direct SFT32.8038.4041.7329.500.3290.2170.3800.21612.0014.0010.402.60

The results reveal that direct SFT provides limited forgetting improvements but substantially compromises the preservation of non-target knowledge. In contrast, CUSE achieves considerably stronger forgetting across classification, generation, and cloze tasks, while maintaining much better performance on the retain and celebrity sets.

These results highlight that privileged teacher supervision alone is insufficient to achieve a favorable forgetting–retention trade-off. By combining on-policy self-distillation, which aligns supervision with the student's evolving generation behavior, and counterfactual sensitivity-aware token weighting, which allocates stronger forgetting supervision to target-sensitive tokens, CUSE achieves more effective and selective unlearning. The dual-branch design further helps preserve non-target knowledge through explicit retention supervision. Overall, this comparison demonstrates the practical advantage of CUSE over the considerably simpler direct SFT alternative. Together with our component-wise ablation studies, these results further support the effectiveness of the proposed framework.


AuthorsOpenReview ↗

Response 2

Comment 3: More Experimental Results Across Different MLLM Backbones

We thank the reviewer for this valuable suggestion. Due to time and computational resource constraints, we have not yet conducted experiments on Qwen3.5. Nevertheless, to further examine the generalizability of CUSE across different MLLM backbones, we conducted additional experiments on Qwen3-VL-4B.

We evaluate CUSE on both MLLMU-Bench and CoVUBench. For a fair comparison, we follow the same evaluation settings as ASRU [1] and adopt the corresponding baseline results reported in its Table 1. The results on MLLMU-Bench are presented below.

MethodClass. Forget ↓Class. Test ↓Gen. Forget ↓Gen. Test ↓Class. Retain ↑Gen. Retain ↑
Vanilla46.6750.000.5780.32741.950.541
GA40.8345.000.5110.28839.410.517
GA_diff45.8350.000.5480.31234.500.512
NPO40.0043.330.5610.32732.560.520
KL_Min44.8046.670.5490.31538.480.518
MMUnlearner40.0047.200.5520.29839.280.527
ASRU31.2034.400.3990.24539.450.531
CUSE12.8013.600.1140.10140.370.537

The results on CoVUBench are also reported below.

MethodEfficacy ↑Generality ↑Divergence ↑Fluency ↑Specificity ↑Capability ↑
Vanilla4.285.2932.1275.2398.6381.33
GA70.7365.3361.2758.4061.9768.47
GA_Diff66.1261.1859.6560.7469.0171.35
KL_Min64.9659.7757.4363.3567.8670.20
NPO77.4167.5463.9965.9268.5769.92
MMUnlearner81.5871.0269.5061.0476.4875.16
MANU79.0774.8875.2567.5785.3576.43
ASRU89.8486.4681.6971.6792.1677.31
CUSE (Ours)99.8299.3095.3873.8297.9879.00

These results provide further evidence that CUSE generalizes effectively across different MLLM backbones.


Summary

We sincerely thank the reviewer for the insightful comments and constructive suggestions. We have addressed the reviewer's concerns through the following additional analyses and experiments:

  • a detailed token-level analysis;
  • an additional direct SFT baseline;
  • additional experiments on Qwen3-VL-4B.

We sincerely hope that these additional analyses and experimental results adequately address the reviewer's concerns. We would greatly appreciate the reviewer's consideration in raising the score. We remain happy to address any further questions.

[1] Guang, Jiahui, et al. "ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models." Proceedings of the 43rd International Conference on Machine Learning (ICML). 2026.

Reviewer_R6eEOpenReview ↗

Response to author

Thank you for your reply. I believe these responses have alleviated my concerns and doubts, and I will improve my score.

Thank you for your reconsideration and for updating the score. We sincerely appreciate your constructive feedback.

Reviewer_YhJBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines Counterfactual Intervention with On-policy Self-Distillation, it controls the unlearning process through token-level weight adjustments, effectively improving the unlearning results.

Critical strengths

1: This work demonstrates substantial performance improvements over existing baselines, achieving a favorable balance between forgetting effectiveness and knowledge retention. 2: The paper provides extensive experiments, including comparisons with existing methods, ablation studies, and robustness evaluations, to demonstrate the effectiveness of the proposed approach.

Critical weaknesses

1: The main contribution of this work lies in its counterfactual-based token sensitivity estimation. However, the paper does not provide a systematic evaluation of the quality of the constructed counterfactual data, despite acknowledging this limitation in the Limitations section. Furthermore, the remaining components largely build upon existing techniques, offering limited methodological novelty beyond their integration. 2: Although the authors evaluate robustness against several input perturbations, including random prefixes, paraphrasing, and jailbreak prompts, they lack a systematic evaluation against stronger, targeted knowledge extraction attacks. For example, explicitly providing the forgotten answer among multiple-choice options, using binary verification questions, or employing likelihood-based answer ranking may reveal whether the model still retains the forgotten knowledge. Such evaluations are particularly important for distinguishing genuine knowledge unlearning from mere behavioral suppression. 3: This work's forgetting objective primarily operates on generated text tokens. It remains unclear whether sensitive information encoded in visual representations is effectively removed. For instance, can the model still recognize or recover private information embedded in images after unlearning?.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response 1

We sincerely appreciate your positive feedback and thoughtful suggestions. Below, we provide detailed responses to address your remaining concerns.

Comment 1: Evaluation of Counterfactual Data Quality

We thank the reviewer for raising this important concern. Our counterfactual construction follows the principle of removing target-related visual evidence while preserving non-target information as much as possible. All counterfactual images are generated offline using an image-editing model (Nano Banana), which replaces the original face with an unrelated face while preserving other visual attributes, such as hair color and hairstyle.

To further examine the quality of our counterfactual construction, we introduce an additional variant that directly blurs only the facial region in each image, rather than replacing it with an unrelated face. We evaluate the vanilla model across classification, cloze, and generation tasks using the same original questions paired with the two types of counterfactual images. We conduct this additional evaluation on LLaVA-1.5-7B. The results are reported below, where Original denotes the face-replacement strategy adopted by CUSE, and Variant 1 denotes the face-blurring strategy.

MethodClassification Accuracy(%)↓Cloze Accuracy(%)↓Generation ROUGE-L↓
Original26.67%18.75%0.3883
Variant 157.50%33.33%0.5215

The face-replacement strategy consistently yields lower scores across all three tasks than face blurring, suggesting that it removes target-related visual evidence more effectively. We further incorporate both types of counterfactual data into CUSE training and compare their resulting unlearning performance, as reported below.

SettingClassification Fgt ↓Classification Test ↓Classification Ret ↑Classification Cele ↑Generation Fgt ↓Generation Test ↓Generation Ret ↑Generation Cele ↑Cloze Fgt ↓Cloze Test ↓Cloze Ret ↑Cloze Cele ↑
Original11.214.460.539.60.1290.1090.5540.31010.06.055.98.2
Variant 121.620.859.338.90.2010.2200.5410.30420.016.054.77.5

These additional experiments evaluate our counterfactual construction from both input-level and training-level perspectives. The results consistently show that face replacement is more effective than face blurring in removing target-related information. Moreover, using face-replacement counterfactuals in CUSE training leads to better unlearning performance. These findings further support our counterfactual construction strategy and its effectiveness in token sensitivity estimation.


Comment 2: Clarification of Contributions

Overall, our contributions can be summarized as follows:

  • Problem analysis. We identify two coupled limitations of existing MLLM unlearning objectives: forgetting supervision is often applied uniformly across tokens, while existing token-level approaches may rely on fixed reference distributions or responses rather than the model's evolving generations. Consequently, forgetting supervision is rarely both fine-grained and on-policy.

  • Counterfactual sensitivity estimation. We introduce a counterfactual-based mechanism to explicitly estimate the sensitivity of each generated token to target-related visual and textual evidence through changes in conditional log probabilities. By continuously recomputing sensitivity scores along the evolving policy's on-policy rollouts, CUSE dynamically adjusts token-level forgetting weights to reflect changes in the model's generation behaviors throughout unlearning.

  • Sensitivity-aware on-policy self-distillation for unlearning. Building on privileged-context on-policy self-distillation, we adapt the dual-branch scheme to targeted multimodal unlearning with sample-specific forgetting and retention contexts. Specifically, we incorporate the proposed counterfactual sensitivity weights into the forgetting branch to explicitly control the relative strength of supervision across generation steps, while the retention branch preserves non-target knowledge and capabilities. Together, the two branches enable CUSE to selectively suppress target-related information while maintaining general multimodal capabilities, achieving a better balance between forgetting effectiveness and capability preservation.


AuthorsOpenReview ↗

Response 2

Comment 3: Further Experimental Evaluation of Robustness

We thank the reviewer for this valuable suggestion. We have already reported the four-option multiple-choice evaluation in Table 2, which examines whether the model can recover forgotten answers when they are explicitly provided as candidate options. To further evaluate targeted knowledge extraction, we conduct an additional binary verification evaluation on sensitive profile information. Specifically, we use binary questions to directly examine whether the model can verify forgotten facts. The results obtained with LLaVA-1.5-7B are reported below.

Modelbinary verification accuracy
Vanilla94.83%
CUSE17.00%

CUSE substantially reduces binary verification accuracy compared with the vanilla model, suggesting that it is much less likely to leak sensitive information when explicitly queried.

Furthermore, following the multi-attack evaluation approach of OpenMMUnlearning [1], we assess privacy using three likelihood-based membership inference attacks: LOSS, ZLIB, and Min-KK%. These attacks measure the average answer-token log-likelihood, its normalization by the compressed answer length, and the average log-likelihood of the least likely 20% of answer tokens, respectively. Each example is evaluated through teacher forcing with its ground-truth answer, with scores computed exclusively over answer tokens. We compute AUROC to quantify distinguishability between forget-set and held-out examples, and report O=1−2∣AUROC−0.5∣O=1-2|\mathrm{AUROC}-0.5|. Higher (O)(O) indicates lower distinguishability, with (O=1)(O=1) corresponding to chance-level discrimination. The results on LLaVA-1.5-7B are reported below.

MethodsLOSS ↑ZLIB ↑Min-20% ↑
Vanilla0.8460.7020.861
GA0.8770.7570.890
GD0.8750.9480.865
CUSE (Ours)0.9120.9690.916

CUSE achieves the highest (O)(O) scores across all three attacks, indicating reduced distinguishability between forget-set and held-out examples. These results further demonstrate the robustness of CUSE against diverse targeted knowledge extraction attacks.


Comment 4: Evaluation of Visual Knowledge Unlearning

We thank the reviewer for raising this important concern. First, we evaluate CUSE on the held-out Test Set of MLLMU-Bench, which consists of distribution-shifted variants of the Forget Set. This evaluation examines whether the model can still recover target information under changes in both textual queries and visual inputs, providing evidence of the generalization of forgetting beyond the original inputs. The corresponding results are reported in Table 1. CUSE achieves consistently strong forgetting performance on the Test Set, indicating that its effectiveness generalizes beyond the original forget samples and is robust to variations in both textual and visual inputs.

Furthermore, to directly investigate whether private information can be recovered from visual inputs alone, we evaluate profile-information leakage on the Forget Set using a portrait-only setting. The model is prompted with "Describe this image" without providing any names, reference answers, or profile information. We compare the generated responses against the corresponding identities' reference profiles. A response is counted as leakage if it discloses at least one supported nonvisual profile fact, while generic appearance descriptions and unsupported claims are excluded.

The results obtained with LLaVA-1.5-7B are presented below.

ModelLeakage (%) ↓
Vanilla72.00
CUSE6.67

The results suggest that CUSE substantially reduces the recovery of private profile information directly from portrait images, even without explicit identity-related textual queries. Together with the held-out Test Set evaluation, these findings provide further evidence that CUSE effectively mitigates visual information leakage and the extraction of sensitive profile information.


Summary

We sincerely thank the reviewer for the insightful comments and constructive suggestions. We have addressed the reviewer's concerns through the following clarifications and additional experiments:

  • a more comprehensive evaluation of counterfactual data quality;
  • a clearer articulation of CUSE's methodological contribution;
  • additional robustness evaluations against targeted knowledge extraction;
  • further evaluation of visual information unlearning.

We sincerely hope that these clarifications and additional experimental results address the reviewer's concerns. We remain happy to address any further questions.

[1] Junkai Chen et al. Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning. arXiv 2026.

Thank you for your response. My concerns have been resolved. Based on the overall quality of the paper, I will keep my score. Good luck!

Thank you for your time and thoughtful consideration. We sincerely appreciate your valuable feedback and are glad that we could address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We are encouraged that the reviewers recognized the importance of fine-grained multimodal unlearning, the coherent design of CUSE, and its strong empirical performance. During the rebuttal, we addressed the primary concerns through additional experiments and clarifications, including:

  • counterfactual quality and token-level sensitivity analyses;
  • additional mixed-content and robustness evaluations;
  • comparisons with new baselines;
  • experiments on Qwen3-VL-4B and additional statistical and efficiency analyses.

Our Strengths

Motivation and Practical Value

  • CUSE addresses the important challenge of selectively removing target-related information while preserving non-target knowledge and multimodal capabilities (Reviewers YhJB, R6eE, ovQx, and uVqb).
  • The paper identifies the limitations of uniform forgetting supervision and off-policy supervision, providing clear motivation for fine-grained on-policy unlearning (Reviewers R6eE and ovQx).

Methodological Design

  • CUSE combines counterfactual sensitivity-aware token weighting with dual-branch on-policy self-distillation to guide selective forgetting along evolving generation trajectories (Reviewers R6eE, ovQx, and uVqb).

Effectiveness and Generalization

  • CUSE achieves strong forgetting while largely preserving retain-set performance across multiple tasks and MLLM backbones (Reviewers YhJB, R6eE, and uVqb).
  • Extensive evaluations on MLLMU-Bench and CoVUBench, together with ablations and robustness tests, support the effectiveness of the proposed framework (Reviewers R6eE and ovQx).

Responses to the Primary Concerns

Counterfactual Quality and Fine-grained Unlearning (Reviewers YhJB, R6eE, and ovQx)

  • Additional comparisons demonstrate that face replacement is more effective than face blurring for removing target-related evidence.
  • Token-level analyses show that sensitive tokens receive higher sensitivity scores and optimization weights than non-sensitive and refusal tokens.
  • The new mixed-content evaluation demonstrates selective suppression of sensitive information while preserving non-sensitive visual descriptions within the same response.

Forgetting Robustness and Additional Baselines (Reviewers YhJB, R6eE, and ovQx)

  • We clarify the privileged-context design and report additional over-refusal, binary verification, relearning, and membership inference evaluations.
  • We introduce Direct SFT, refusal SFT, preference optimization, and sequence-level off-policy distillation baselines, demonstrating CUSE's more favorable forgetting–retention trade-off.

Experimental Reliability and Implementation Details (Reviewers YhJB and ovQx)

  • We report results over three independent runs, GPU memory consumption, and training time.
  • We clarify counterfactual construction, evaluation protocols, and qualitative case-study criteria.

Methodological Novelty and Related Work (Reviewers ovQx and uVqb)

  • We clarify the distinctions between CUSE and ViKeR, privileged-context on-policy distillation, token-level reweighting, and refusal-based unlearning.
  • We emphasize CUSE's explicit counterfactual sensitivity estimation and dynamic token reweighting on evolving on-policy trajectories.

Summary

  • We substantially strengthened the evaluation through new baselines, token-level analyses, mixed-content tests, and robustness evaluations.
  • The additional evidence supports CUSE's ability to selectively suppress target-related information while preserving non-target knowledge and general multimodal capabilities.

Overall, the new evidence supports CUSE as an effective, robust, and selective framework for fine-grained multimodal unlearning while preserving non-target knowledge and capabilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.