- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CUSE combines counterfactual token weighting with on-policy self-distillation for multimodal unlearning. A frozen original model compares token probabilities under original and target-replaced inputs to estimate sensitivity. These scores weight distillation from a teacher conditioned on forgetting instructions, while a retention branch preserves non-target capabilities. Experiments on MLLMU-Bench and CoVUBench report improved forgetting–utility trade-offs across three backbones overall.
Critical strengths
- Sections 3.1–3.3 explain sensitivity estimation, dual-branch distillation, and their combination in a clear sequence. Equations (6)–(12) and Algorithm 1 make the main procedure understandable.
- The reported improvements are substantial. In Table 1, CUSE reduces LLaVA’s forget-set classification accuracy to 11.20%, compared with 30.40% for ASRU, while retaining 60.51% accuracy versus the original model’s 60.70%. Table 5 extends evaluation to copyright unlearning. The comparisons include recent ASRU (2026).
- Table 2 evaluates on-policy versus fixed offline responses and sensitivity weighting versus uniform and random weighting. Relative to Uniform, CUSE reduces forget-set ROUGE-L from 0.192 to 0.129 while slightly improving retain-set ROUGE-L.
Critical weaknesses
- The novelty positioning omits ViKeR—Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning (Cai et al., 2026, arXiv:2601.22020). ViKeR already uses replacement visual inputs and frozen-model token distributions to prioritize sensitive tokens. Its objective differs, but the overlap requires discussion and qualification of the introduction’s broad uniform-supervision characterization. The relationship to privileged-context on-policy safety distillation by Fu et al. (2026), already cited in Appendix A.3, also deserves a more explicit distinction.
- The central within-response preservation claim is not directly established. Figure 1 motivates suppressing identity while preserving attributes such as hair color. Appendix B.4 instead evaluates general attributes through additional questions, and Appendix H presents separate forgetting and retention examples. These support preservation but do not directly test selective behavior within mixed-content responses.
- Important construction and evaluation details remain unclear. Appendix B.4 does not specify how same-attribute, different-identity portraits are obtained or checked, or how counterfactuals are constructed for copyright concepts. Appendix F does not explain the construction of forgetting reference responses. Appendix F.3 also leaves the evaluator and aggregation into a single naturalness score unspecified.
Questions for authors
- What distinguishes CUSE from ViKeR and privileged-context on-policy distillation beyond combining these ideas?
- What do the forgetting references contain, and do existing evaluations directly test mixed-content responses?
- Please clarify counterfactual construction, the naturalness evaluator, and score aggregation.
7 comments
Response 2
Overall, our contributions can be summarized as follows:
-
Problem analysis. We identify two coupled limitations of existing MLLM unlearning objectives: forgetting supervision is often applied uniformly across tokens, while existing token-level approaches may rely on fixed reference distributions or responses rather than the model's evolving generations. Consequently, forgetting supervision is rarely both fine-grained and on-policy.
-
Counterfactual sensitivity estimation. We introduce a counterfactual-based mechanism to explicitly estimate the sensitivity of each generated token to target-related visual and textual evidence through changes in conditional log probabilities. By continuously recomputing sensitivity scores along the evolving policy's on-policy rollouts, CUSE dynamically adjusts token-level forgetting weights to reflect changes in the model's generation behaviors throughout unlearning.
-
Sensitivity-aware on-policy self-distillation for unlearning. Building on privileged-context on-policy self-distillation, we adapt the dual-branch scheme to targeted multimodal unlearning with sample-specific forgetting and retention contexts. Specifically, we incorporate the proposed counterfactual sensitivity weights into the forgetting branch to explicitly control the relative strength of supervision across generation steps, while the retention branch preserves non-target knowledge and capabilities. Together, the two branches enable CUSE to selectively suppress target-related information while maintaining general multimodal capabilities, achieving a better balance between forgetting effectiveness and capability preservation.
Comment 2: Within-Response Selective Preservation
We thank the reviewer for raising this important concern. We construct an additional within-response mixed-content evaluation set through data augmentation based on the Forget Set. Specifically, we create queries that simultaneously ask about a target-sensitive attribute (e.g., a person's profession) and a non-sensitive visual attribute. For the latter, we focus primarily on visually distinguishable attributes, such as hair color, that should remain accessible after unlearning. The expected behavior is to withhold the sensitive information while still providing an accurate description of the non-sensitive visual attribute within the same response.
We report the reduction in ROUGE-L for the sensitive-information component (Δ ROUGE-L) and the accuracy of the non-sensitive visual concept description (Visual Concept Accuracy). The results are shown below.
| Method | Δ ROUGE-L ↑ | Visual Concept Accuracy (%) ↑ |
|---|---|---|
| GA | 23.5 | 0.0 |
| GA_Diff | 11.9 | 3.4 |
| NPO | 30.2 | 6.2 |
| KL_Min | 10.1 | 6.7 |
| MMUnlearner | 18.4 | 24.0 |
| MANU | 21.5 | 23.4 |
| ASRU | 30.7 | 45.1 |
| CUSE | 49.2 | 61.0 |
The results demonstrate that CUSE effectively balances sensitive-information suppression and non-sensitive attribute preservation within the same response. It withholds target-sensitive information while retaining the ability to accurately describe non-sensitive visual attributes.
Response 3
Comment 3: More Experimental Details
Discussion of reference answer
We thank the reviewer for raising this important concern. We clarify that CUSE uses different privileged-context designs depending on whether a query requests only target attributes or combines sensitive and non-sensitive information.
For attribute-only queries, as described in Appendix, we provide the forgetting teacher with a reference refusal response. Specifically, we prepare a set of general refusal expressions (e.g., "I'm not sure" and "I'm unable to provide an answer to that") and randomly select one as the reference response for each sample. We use these general refusal references because the desired post-unlearning behavior should prevent disclosure while maintaining response coherence and readability. We have included the complete reference-answer template in the revised manuscript. Although a generic refusal response is provided as a reference, CUSE does not directly fine-tune the student on this fixed response. Instead, the frozen teacher provides token-level distributional supervision conditioned on the original image, query, and generation prefix. Through on-policy distillation, the student learns to suppress target-related information in its own generated responses. As illustrated by the complete example in Appendix I, the resulting response can be adapted to the specific query rather than simply reproducing the reference refusal verbatim. More generally, the desired post-unlearning response need not be a refusal: a sample-specific alternative answer could also serve as the reference. We choose refusal references because they provide a coherent and readable way to withhold the target information without introducing fabricated or unsupported facts [3]. Simply generating arbitrary or seemingly plausible alternative answers may result in hallucinations and make the unlearning behavior less reliable.
For queries involving both sensitive and non-sensitive content, we explicitly design the privileged context to distinguish between the different types of information. It first instructs the teacher not to disclose the target-sensitive attributes, while emphasizing that unrelated visual descriptions should still be answered normally rather than being unnecessarily refused. We also provide the ground-truth answer to the entire query, covering both sensitive and non-sensitive information, as a reference in the privileged context. This allows the teacher to provide selective behavioral supervision: withholding the sensitive information while retaining useful visual descriptions within the same response.
Importantly, CUSE is not restricted to refusal-based unlearning. Its desired post-unlearning behavior is flexible and can be specified through the privileged context. By simply modifying the textual instructions and reference responses in the privileged context, we can guide the teacher toward different types of desired behaviors, such as refusing to disclose target information, providing alternative responses, or preserving selected non-target information. The underlying distillation framework remains unchanged.
Construction of Counterfactual Images
We thank the reviewer for this question. Across both benchmarks, counterfactual inputs follow the same principle: remove target-related evidence while preserving non-target information as much as possible. The two benchmarks differ only in what constitutes target-related evidence. All counterfactual images are pre-generated offline with an image-editing model (Nano Banana) and serve only for sensitivity estimation, not as supervision targets.
(1) MLLMU-Bench. The forgetting target is the identity of a fictitious individual and its profile, which is primarily carried by the face, whereas hairstyle, clothing, and background are identity-irrelevant. Since the profile images of MLLMU-Bench were generated with ThisPersonDoesNotExist, we sample substitute faces from the same source and keep only those whose face-recognition similarity (ArcFace) to every individual in MLLMU-Bench falls below the same-identity threshold. Each forget individual is paired with a substitute of similar gender and age, which serves only as the identity reference: the editing model replaces the face in the original image while keeping hairstyle, clothing, accessories, expression, and background unchanged. For questions that name the individual, the name is replaced with a new fictitious name.
Response 1
We sincerely appreciate your positive feedback and thoughtful suggestions. Below, we provide detailed responses to address your remaining concerns.
We thank the reviewers for pointing out the missing discussion of ViKeR in our related work. We apologize for the incomplete discussion. We have revised the Introduction and Related Work accordingly to better clarify the positioning and contributions of CUSE. We further discuss ViKeR [1] and OPSA [2] and highlight their differences from CUSE as follows.
ViKeR employs distribution regularization, which to some extent induces implicit token-level reweighting. However, ViKeR still differs substantially from CUSE in its reweighting mechanism, sensitivity estimation, counterfactual construction, and supervision paradigm, with the following limitations:
Explicit vs. implicit reweighting. ViKeR introduces an ideal post-unlearning token distribution estimated from irrelevant visual inputs and uses KL regularization to implicitly induce different token-level gradient magnitudes. In contrast, CUSE explicitly computes counterfactual sensitivity scores from token-level log-probability differences and transforms them into normalized weights that directly modulate the forgetting KL objective. This provides an explicit mechanism to control the relative optimization strength across generation steps.
Static vs. dynamic sensitivity estimation. ViKeR estimates its reference distributions using the frozen pre-unlearning model, and these distributions remain fixed throughout subsequent optimization. Consequently, its token-level guidance does not adapt to changes in the student model's generation behavior. In contrast, CUSE continuously samples responses from the evolving student policy and recomputes sensitivity scores along its current generation trajectories. Thus, the forgetting weights adapt to evolving rollouts and their corresponding prefixes, rather than relying on pre-estimated reference distributions.
Counterfactual construction. ViKeR replaces the original image with irrelevant reference images of different individuals to estimate identity-agnostic token distributions. In contrast, CUSE performs targeted counterfactual interventions by replacing target-related visual evidence (e.g., the face region) while preserving non-target attributes such as hairstyle, clothing, and background. It also supports interventions on target-related textual evidence. This design aims to isolate the effect of target-related evidence while minimizing unnecessary changes to non-target visual information.
ViKeR uses the estimated distributions to regularize the original unlearning objective, thereby encouraging the model to move away from target-specific token predictions. While this approach improves coherence over conventional gradient ascent, distributional regularization alone does not explicitly specify the desired semantic behavior of the resulting response. Consequently, suppressing target-related predictions may lead to semantically unsupported alternative tokens and potentially hallucinated responses. In contrast, CUSE employs privileged-context teachers to provide explicit forgetting- and retention-oriented behavioral supervision, with counterfactual sensitivity determining the strength of forgetting supervision at each generation step.
Regarding OPSA [2], it applies on-policy self-distillation (OPSD) to safety alignment in text-only LLMs, which differs from our task of multimodal unlearning. Moreover, OPSA employs token-level KL distillation but does not consider fine-grained token-level reweighting based on the sensitivity of individual tokens to target-related information. In contrast, CUSE introduces counterfactual sensitivity estimation to explicitly quantify token-level sensitivity and dynamically reweight forgetting supervision along evolving on-policy trajectories.