- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to make multimodal models forget specified identities across both image and text inputs while preserving performance on other data. It proposes Blindfold-to-Forgetting, which retargets forget-image representations toward a generic-face representation and suppresses target answers on both input routes. Experiments on three benchmarks and three MLLMs show improved forgetting–retention balance in many, but not all, settings.
Critical strengths
The method is motivated by the model architecture. The visual projector provides an image-specific intervention point, while text-only inputs bypass it. The method accounts for this difference by treating the two routes separately.
The design considers the trade-off between forgetting and utility. Stage 1 constrains retain-image representations, and Stage 2 includes retain supervision. The hinge suppression objective is also intended to avoid unnecessarily suppressing target answers beyond a threshold.
The evaluation covers multiple benchmarks and models. In addition to the main results, the paper includes component ablations, forget-ratio and threshold analyses, relearning experiments, and general-capability evaluations.
The ablations provide some support for the design choices. Removing text-route training weakens text-only forgetting, while changing the visual route used for suppression harms retained performance. These findings are broadly consistent with the authors’ explanation of the components’ roles.
Critical weaknesses
The evaluation shows reduced target-answer accuracy on the tested benchmark prompts, but does not establish how well this reduction generalizes to paraphrased, indirect, or otherwise varied queries. This limits the strength of claims about forgetting beyond the evaluated prompts. The results would be more convincing with prompt-variation or extraction-robustness tests, or with a clearer qualification that the evidence demonstrates benchmark-level behavioral unlearning.
BTF obtains the highest Forget–Retain H-Mean in 8 of 12 settings on UMU-Bench and CLEAR, but some baselines perform better in the remaining settings. For example, GD achieves higher H-Mean than BTF on CLEAR with Qwen2-VL-7B. More analysis of the differences across models and benchmarks would help clarify where the method is most effective.
Some general capabilities decline. The average score across the reported general-capability benchmarks decreases from 61.74 for Vanilla to 60.44 for BTF, and POPE drops from 83.7 to 76.4. Retain VQA accuracy also declines as the forget ratio increases.
Questions for authors
Have you evaluated BTF using paraphrased or indirect queries about forget identities? If not, how would you characterize the scope of the current evidence: forgetting on the benchmark prompts, or more general robustness to alternative queries?
What factors might explain the settings where BTF does not achieve the highest H-Mean, particularly CLEAR with Qwen2-VL-7B?
Have you analyzed the decline on POPE or other general-capability benchmarks by error type? Can the observed degradation be linked to either visual representation retargeting or Stage 2 suppression?
In the relearning experiments, how do the number of examples and optimization steps relate to the intended threat model? What conclusions do you believe these experiments support?
3 comments
Response to Reviewer a4qT (2/3)
W3 & Q3. General-capability degradation and the POPE decrease
We thank the reviewer for this careful observation. We agree that the POPE decrease is a real limitation of our method, and we analyzed it further.
(1) Scope of the degradation
The average over the five benchmarks changes from 61.74 to 60.44. MMLU (47.1 → 46.8), MMBench (64.7 → 65.6), and GQA (51.8 → 53.7) are maintained, while SEED-Bench (61.4 → 59.7) and POPE decline. Although GA and KL collapse (averages of 27.74 and 47.82), we acknowledge that GD, NPO, PO, MANU, and MIP-Editor preserve POPE better than our method, and we do not claim that BTF fully preserves general capabilities.
(2) Error-type and component-wise analysis
We re-evaluated all 1,200 POPE items (600 positive) using likelihood-based yes/no scoring. Because this protocol differs from the greedy generation used in Table 5, the numbers differ slightly.
| Variant | Acc. | Precision | Recall | Yes-ratio | FP | FN |
|---|---|---|---|---|---|---|
| Vanilla | 83.33 | 82.26 | 85.00 | 51.67 | 110 | 90 |
| Stage 1 only | 81.00 | 78.70 | 85.00 | 54.00 | 138 | 90 |
| Stage 2 only | 84.33 | 90.71 | 76.50 | 42.17 | 47 | 141 |
| Full BTF | 78.75 | 79.19 | 78.00 | 49.25 | 123 | 132 |
- The degradation arises from combining the two stages, not from either stage alone. Stage 1 alone (81.00) and Stage 2 alone (84.33) remain close to Vanilla (83.33), but combining them lowers accuracy to 78.75.
- The degradation reflects reduced discrimination rather than bias. Stage 1 shifts predictions toward “yes” and Stage 2 toward “no”; these opposite biases largely cancel out when combined, bringing the yes-ratio close to Vanilla's. However, both FP (+13) and FN (+42) increase, and the additional errors are mainly “no” answers for objects that are present.
We attribute this degradation to Stage 2’s retain supervision consisting only of QA about retain identities, so object discrimination on general images is not preserved during training. Including general-image VQA data in the retain set is a natural direction for improvement. The POPE degradation and its component-wise analysis are now discussed in Section 4.7 and Appendix F.8 (Table 13) of the revised manuscript.
(3) Effect of the forget ratio
Retain VQA decreases as the forget ratio increases, with most of the decrease occurring between 5% and 10%. Over the same range, Forget VQA and Retain QA remain stable, so the image-route F-R H-Mean stays nearly unchanged (62.3 → 59.3 → 60.1). This decrease is not excessive relative to other methods: GD, MANU, and NPO preserve retention but barely forget, PO loses more retention than BTF, and KL collapses. Scalability to larger forget ratios is now discussed as a limitation in Section 4.4 and Appendix A.
Response to Reviewer a4qT (3/3)
Q4. Threat model and interpretation of the relearning experiments
The threat model assumed in these experiments is the benign relearning of Hu et al. (2025): after unlearning, a small amount of data related to the forget targets is included in further fine-tuning. In particular, we focus on whether relearning only a subset of the forget identities also restores responses about other forget identities that are not included in the relearning data.
Accordingly, the experiments use small amounts of data and short training budgets. On UMU-Bench with LLaVA-1.5-7B, we relearned 5, 25, or 50 of the 50 forget identities and measured accuracy on the full forget set. Relearning used all 12 QA examples per identity and ran for 8 epochs with a learning rate of 1e-4. In the training-budget analysis (Figure 4c), we trained on a single forget example for up to 8 epochs with the same learning rate and measured accuracy at each epoch. The experiments thus cover data scales from a single example to the full forget set, and training budgets of up to 8 epochs.
Under these settings, the experiments support the following conclusions.
- Recovery is limited under small-scale relearning. When 5 identities are relearned, BTF's Forget accuracy is 40.0 on the image route and 51.0 on the text route. Under the same condition, Vanilla reaches 84 / 75 and PO reaches 87 / 73 (Figure 4b).
- Recovery remains limited as the training budget increases. With a single example and increasing epochs, PO’s image-route Forget accuracy rises from 68.0 before the attack to 87 after 8 epochs. In contrast, BTF stays at or below its pre-attack level (46.0) at every evaluated point and returns to 46.0 after 8 epochs (Figure 4c).
- Low recovery is also maintained on other benchmarks and models. Across the six benchmark–model combinations, BTF’s Forget accuracy after relearning ranges from 20.0 to 40.0 (Figure 4a). Some baselines show lower values than BTF in some settings, but most of these are models whose Retain accuracy had already dropped substantially right after unlearning (Table 1).
- Forgetting weakens as the scope of relearning widens, and recovery is faster on the text route. When 25 identities are relearned, text-route Forget accuracy rises to 77.0, while the image route stays at 46.0. When all 50 are relearned, accuracy rises to 78.0 on the image route and 81.0 on the text route.
These results therefore provide evidence that target-response recovery is limited under small data and short training budgets. They do not guarantee resistance to stronger attacks that use sufficient forget data or prolonged additional training. The threat model, experimental settings, and scope of our conclusions are now clarified in Section 4.6 of the revised manuscript.
Response to Reviewer a4qT (1/3)
We thank the reviewer for recognizing our route-aware design, the training objectives that account for the forgetting–retention trade-off, and the validity of our component analysis. We address the reviewer’s concerns below.
W1 & Q1. Forgetting generalization to varied queries
We agree that it is important to verify whether forgetting persists under rephrased or indirect queries. Because each benchmark’s standard protocol evaluates with the same questions and images used for unlearning, the results in the paper alone cannot answer this question.
To address this, we added two controlled evaluations on MLLMU-Bench with LLaVA-1.5-7B. First, we generated three paraphrases per question, preserving meaning and answer. Second, for each identity, we used three Test Set images of the same person that were never used during unlearning, and confirmed by comparing hashes of the decoded pixels that none of them overlaps with the original benchmark images.
Since accuracy drops for all models, including Vanilla, when the query or image changes, we compare reductions relative to Vanilla. The baselines reduce accuracy by at most 5.0 points relative to Vanilla in all three conditions. In contrast, BTF reduces it by 12.7 points on the original setting, 10.0 points on unseen images, and 11.0 points on paraphrased questions. BTF’s forgetting effect is thus not confined to the query phrasings used in training, and is largely maintained on paraphrases and new images. These results are included in Table 2.
W2 & Q2. Relatively lower H-Mean in some settings
We acknowledge that BTF does not achieve the highest H-Mean in every setting. BTF achieves the highest F-R H-Mean in 8 of the 12 settings on UMU-Bench and CLEAR. The four remaining settings are:
Why GD leads on CLEAR with Qwen2-VL-7B. For this setting, we added a Retain-only model, trained without the forget identities, as a reference.
We have incorporated this analysis into Section 4.2 and Appendix F.5 of the revised manuscript.