- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies selective unlearning in vision–language models: a deletion request removes only some attributes of an individual (e.g., date of birth, occupation, medical information), while other attributes of the same individual must be preserved, often under the same image. The authors re-partition MLLMU-Bench at the attribute level (6 forget attribute types, 11 retain attribute types) and propose SIEVE, which alternates two training phases and updates only LoRA adapters on the value projections :
- Forget phase: push the attention values of all decoder layers toward zero () and maximize sequence cross-entropy.
- Retain phase: match to a frozen pre-unlearning model () and minimize cross-entropy.
On LLaVA-1.5-7B and Qwen2-VL-7B, under both Visual-QA and Textual-QA, SIEVE achieves the lowest Forget-set classification and cloze scores among GA Diff, KL Min, NPO, and MMUnlearner. The appendix adds 13B results, loss-term ablations, a key-versus-value suppression comparison, visualizations, and local perturbation bounds.
Critical strengths
-
A practically relevant problem setting. Real deletion requests rarely cover everything known about a person. Forgetting someone's date of birth while keeping their hobby, when both queries use the same face image, is harder and more realistic than the identity-level splits used in prior multimodal unlearning benchmarks. I consider this setting the paper's most valuable contribution.
-
A simple, low-cost method with large forgetting gains. SIEVE needs only forward hooks, a frozen reference model, and LoRA on , with no saliency or localization preprocessing. Its training cost is comparable to GA Diff (11,091 s vs. 10,961 s). It achieves the lowest Forget classification and cloze scores in all four 7B settings (e.g., 20.06 vs. 36.59 for the best baseline on LLaVA Visual-QA classification). If the selectivity holds up, this makes the method attractive in practice.
Critical weaknesses
-
It is not yet clear whether the forgetting is association-level or category-level. The same six attribute categories are forgotten for every profile (Table 5), so the current results are also consistent with the model learning not to answer certain question types, rather than removing specific face-to-attribute associations. Two observations point in this direction:
- On LLaVA Visual-QA, Forget classification drops by 49.6 points, while the Test split (paraphrased questions, modified images) drops by only 5.7.
- Some Forget accuracies are at or below chance level.
A related question is how the losses treat image-token positions. Forget and retain examples of the same identity share these positions, so (target ) and (target ) would set opposite targets there. Because association-level selectivity is the central claim, this is the main factor in my rating. I believe it can be largely clarified with analyses on the existing checkpoints (Q2–Q3).
-
Some reported results need verification, and the comparisons need more detail.
- In Table 1, the KL Min rows for LLaVA-Textual and Qwen-Textual share four identical values (8.48, 0.619, 17.68, 0.587), and the NPO rows for LLaVA-Visual and Qwen-Visual share two. These may be transcription errors.
- Fig. 15 (captioned 7B) matches the 13B results in Table 8.
- The forget ratio used in Table 1 is not stated, and the split does not match the 5%/10% ratios in Fig. 3.
- Baseline configurations are not reported, and the closest representation-level method, RMU, is not compared.
These are mostly clarifications, and resolving them would substantially increase my confidence in the results.
Questions for authors
I have tried to limit these to questions whose answers would change my assessment. Q1–Q4 can be answered with clarifications or with analyses on existing models. I do not expect Q5 to be completed during the rebuttal period.
- (Clarification) Could you verify the duplicated values in Table 1 noted above and confirm which model Fig. 15 comes from? Which forget ratio does Table 1 use, and how does it relate to Fig. 3?
- (Analysis on existing outputs) Could you report Retain results separately for (i) permitted attributes of identities that also appear in and (ii) other identities? Could you also report Real-set results restricted to forget-category questions, such as celebrities' occupation or birthplace? Retained performance on (i), and no notable drop on that Real-set subset, would directly support the association-level claim.
- (Clarification) Are and applied to image-token and system-prompt positions? If so, how is the conflict between and resolved for forget/retain pairs that share an image? What exactly is plotted in Figs. 11–14?
- (Clarification) For each baseline, what were the trainable modules, learning rate, number of epochs, and tuning budget? Were all baselines trained on the same attribute-level split? If the baselines did not use -only LoRA, a single GA Diff run with -only LoRA would help separate the effect of the losses from the effect of parameter localization.
- (Would strengthen the paper; not required for rebuttal) A comparison with RMU under the same cross-entropy terms, and a simple residual-information check such as linear probing or a few-step relearning attack.
What would change my score: if Q1 confirms transcription errors and the corrected numbers support the conclusions, and Q2 shows that permitted attributes of forget-set identities are preserved, I would be inclined to raise my score.
Suggestions for future versions (not affecting my score):
- Report the distance to a retain-only oracle model.
- Report attribute-level factual accuracy alongside ROUGE-L.
- Consider an abstention objective. In Figs. 4–5, forgotten professions appear to be replaced by professions of other profiles, and such misattribution could be harmful for real individuals.
4 comments
Rebuttal to Reviewer LAGn (2/3)
Clarifications on Value Conflict
Are and applied to image-token and system-prompt position? If so, how is the conflict between $ V \to 0$ and resolved for forget/retain pairs that share an image?
Thank you for raising this important question. We clarify that and are applied to the value projections of forget and retain samples, respectively. Although both objectives optimize the same model parameters, they do not directly impose conflicting constraints on the same activations. Specifically, encourages the value representations associated with forget samples to approach zero, whereas constrains the value representations of retain samples to remain close to those of the reference model.
We acknowledge that applying these objectives to shared token positions, such as image tokens or system-prompt tokens, may introduce gradient interference because these tokens can encode common information across forget and retain samples. Nevertheless, such interference is not equivalent to imposing and simultaneously on identical activations. Instead, the two objectives encourage input-dependent suppression and preservation, allowing the model to distinguish between forget-related and retain-related representations.
What exactly is plotted in Figs. 11–14?
We include these visualizations to examine whether SIEVE produces the intended selective changes in attention-value representations. Figure 12 shows reduced value activations for forget examples, while Figure 11 and 13 show comparatively stable representations for retain examples. These contrastive patterns provide qualitative evidence that SIEVE suppresses forget representations while limiting disruption for retained representations, complementing the behavioral results with an internal view of the changes in model.
Figure 14 complements this analysis by evaluating different suppression targets. Among the tested settings, the zero-valued target achieves the lowest forget scores across all six evaluation metrics, while obtaining the highest retain scores in the five of the six metrics. This supports our design of a zero target. Collectively, these figures provide evidence for the intended representation changes and the empirical effectiveness of the target choice.
Clarifications on baselines
For each baseline, what were the trainable modules, learning rate, number of epochs, and tuning budget? Were all baselines trained on the same attribute-level split? If the baselines did not use -only LoRA, a single GA Diff run with -only LoRA would help separate the effect of the losses from the effect of parameter localization.
We thank the reviewer for pointing out the need to clarify the baseline training configurations to better isolate the effectiveness of the proposed value suppression mechanism.
To ensure a fair comparison, all baselines and our method are trained using the same LoRA configuration and their respective training objectives. Specifically, LoRA is applied to all linear layers within the language decoder, including the attention query (q_proj), key (k_proj), value (v_proj), and output (o_proj) projections, as well as the feed-forward network projections (gate_proj, up_proj, and down_proj). The vision encoder (vision_model), multimodal projector (multi_modal_projector), and language modeling head (lm_head) are excluded from LoRA adaptation. This configuration is consistent with previous works (MMUnlearner, MANU).
We also acknowledge an unclearness in the original manuscript, where we stated that LoRA was applied only to "All decoder value projection". By this, we meant that all values of the projections within language decoders. We have corrected this description in the revised manuscript to accurately reflect the actual implementation, in which LoRA is applied to all eligible linear layers of the language decoder. We emphasize that this correction concerns only the description of the training configuration and does not affect the experimental setup or reported results.
Rebuttal to Reviewer LAGn (3/3)
Association or Category Level
It is not yet clear whether the forgetting is association-level or category-level.
We thank the reviewer for the opportunity to clarify the granularity of SIEVE's unlearning objective. SIEVE is formulated as association-level unlearning, because it targets selected identity-attribute associations while preserving permitted information within the same identity. The six categories in Table 7 specify which associations are selected for the deletion request in our experiments. To clarify, they are used to construct the forget set only, they will not constitute a model-wide rule prohibiting those questions categories. The objectives in Eq. 4 to 7 optimize the designated forget examples and preserve the corresponding reference behaviour on retained examples.
Our existing evaluation examines this selectivity within identities. Figure 4 presents a profession question designated for forgetting and a hobby question retained for the same individual. The forget classification for LLaVa Visual-QA decreases from 69.67 to 20.06, while Retain accuracy remains at 39.20 compared with 43.75 before unlearning. These results support selective forgetting of designated attributes alongside preservation of permitted knowledge.
Regarding whether the same attribute category remains available for other identities, our Real set already evaluates celebrity identities outside the deletion request. Reporting its occupation and birthplace subsets separately would test this dimension more directly, while separating Retain results by identity overlap would show the evidence for preservation within affected identities in a more explicit way. We will clarify this distinction in the manuscript, where the deletion target is an identity-attribute association, whereas the shared category list defines the particular deletion request evaluated in this benchmark. The additional breakdown assesses how specifically the learned suppression follows the request.
Dear Reviewer LAGn,
Thank you again for your time and constructive feedback on our paper. We have carefully addressed each of the concerns raised in your review through detailed responses, additional clarifications, and experimental evidence. If any questions remain or any points require further clarification, we would be happy to address them during the remaining discussion period.
Thank you very much for your time and consideration again. We look forward to your feedback!
Best regards, The Authors
Rebuttal to Reviewer LAGn (1/3)
We thank the reviewer for noting that “A practically relevant problem setting.”
We appreciate the reviewer's careful evaluation of our work. Our responses follow.
Transcription Error
We sincerely thank reviewer for pointing out the reporting details. For the duplicate values in Table 1, the two matching NPO scores arise from rounding distinct underlying values to three decimal places. For KL-Min, the revised Qwen Textual-QA reports 9.33 for Forget Cloze, 0.623 for Retain Generation, 18.59 for Retain Cloze, and 0.589 for Test Generation. These replace the four repeated entries identified.
For the model identification in Figure 15, the value ablation figure matches the LLaVA1.5-13B results reported in Table 8, including the Visual-QA Forget scores of 31.53, 0.437 and 5.76. The 7B caption is therefore a transcription error, which now we have already fixed it in the manuscript.
We have addressed this concern in the General Rebuttal. We kindly refer the reviewer to the corresponding response for further details.
Analysis on existing outputs
We thank the reviewer for this suggestion. We first clarify that requested additional experiment results are already part of our experimental setting, the details of this clarification is reported in the general comment.
In Appendix B.3, Eq. 42, 43, and 44 further explain this restructuring. Each profile consisting of an image and multiple pairs of question-answer tuple, is then flattened into individual examples such that ((xi,qi,yi)). These examples are assigned to forget and retain according to the designated attribute.
The Forget results measure the unlearning effect on the 500 identities, where the Retain results in Table 1 are also measured under this construction and include evaluation of permitted information about the same 500 identities affected by unlearning. Section 4.3 and Figure 4 explicitly illustrate this through a forgotten profession and a retained hobby for the same individual. With this setting, same-identity retention is already part of our evaluation.
For Real set, Appendix B.3 specifies that the celebrity evaluation set remains unchanged, which means the set consists of both permitted and non-permitted attributes. This evaluates preservation of knowledge in terms of a separate population of non-target identities. The deletion request concerns designated person-attribute associations, which does not require suppressing the corresponding attribute category for every identity. The requested restriction to celebrities' occupation, birthplace and other questions belongs to forget category would provide a more focused disaggregation of this existing Real evaluation. The current Real columns report aggregate celebrity performance, so we distinguish those results from the category-restricted preservation claim requested.
RMU Results
We have addressed this question in the General Rebuttal. We kindly refer the reviewer to the corresponding response for further details.