acceptodds
Under review as a conference paper at ICLR 2027

Blindfold-to-Forgetting: Route-Aware Multimodal Unlearning

Abstract

Multimodal unlearning aims to remove the influence of specific training data from models that process images and text while preserving performance on non-target data. Since the same target information can be elicited through distinct image- and text-based routes, intervention should account for their differences. Whereas prior work has primarily focused on editing modality-specific neurons or internal paths, we propose Blindfold-to-Forgetting, which uses the visual-to-language interface as a selective intervention point. The first stage shifts forget-image representations toward the mean representation of generic face images while preserving retained-image representations. The second stage suppresses target responses under image inputs processed by the original projector and text-only inputs, using a hinge objective and retain supervision to limit excessive suppression and utility degradation. Across three benchmarks and three MLLMs, our method achieves the highest Forget-Retain harmonic mean in 8 of 12 evaluation settings on UMU-Bench and CLEAR, with improvements of up to 18.1 points over the strongest baseline. Relearning experiments further show limited target-response recovery under the tested low-data conditions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Multimodal Large Language Models, Multimodal Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_a4qTOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to make multimodal models forget specified identities across both image and text inputs while preserving performance on other data. It proposes Blindfold-to-Forgetting, which retargets forget-image representations toward a generic-face representation and suppresses target answers on both input routes. Experiments on three benchmarks and three MLLMs show improved forgetting–retention balance in many, but not all, settings.

Critical strengths

The method is motivated by the model architecture. The visual projector provides an image-specific intervention point, while text-only inputs bypass it. The method accounts for this difference by treating the two routes separately.

The design considers the trade-off between forgetting and utility. Stage 1 constrains retain-image representations, and Stage 2 includes retain supervision. The hinge suppression objective is also intended to avoid unnecessarily suppressing target answers beyond a threshold.

The evaluation covers multiple benchmarks and models. In addition to the main results, the paper includes component ablations, forget-ratio and threshold analyses, relearning experiments, and general-capability evaluations.

The ablations provide some support for the design choices. Removing text-route training weakens text-only forgetting, while changing the visual route used for suppression harms retained performance. These findings are broadly consistent with the authors’ explanation of the components’ roles.

Critical weaknesses

The evaluation shows reduced target-answer accuracy on the tested benchmark prompts, but does not establish how well this reduction generalizes to paraphrased, indirect, or otherwise varied queries. This limits the strength of claims about forgetting beyond the evaluated prompts. The results would be more convincing with prompt-variation or extraction-robustness tests, or with a clearer qualification that the evidence demonstrates benchmark-level behavioral unlearning.

BTF obtains the highest Forget–Retain H-Mean in 8 of 12 settings on UMU-Bench and CLEAR, but some baselines perform better in the remaining settings. For example, GD achieves higher H-Mean than BTF on CLEAR with Qwen2-VL-7B. More analysis of the differences across models and benchmarks would help clarify where the method is most effective.

Some general capabilities decline. The average score across the reported general-capability benchmarks decreases from 61.74 for Vanilla to 60.44 for BTF, and POPE drops from 83.7 to 76.4. Retain VQA accuracy also declines as the forget ratio increases.

Questions for authors

Have you evaluated BTF using paraphrased or indirect queries about forget identities? If not, how would you characterize the scope of the current evidence: forgetting on the benchmark prompts, or more general robustness to alternative queries?

What factors might explain the settings where BTF does not achieve the highest H-Mean, particularly CLEAR with Qwen2-VL-7B?

Have you analyzed the decline on POPE or other general-capability benchmarks by error type? Can the observed degradation be linked to either visual representation retargeting or Stage 2 suppression?

In the relearning experiments, how do the number of examples and optimization steps relate to the intended threat model? What conclusions do you believe these experiments support?

3 comments
AuthorsOpenReview ↗

Response to Reviewer a4qT (1/3)

We thank the reviewer for recognizing our route-aware design, the training objectives that account for the forgetting–retention trade-off, and the validity of our component analysis. We address the reviewer’s concerns below.

W1 & Q1. Forgetting generalization to varied queries

We agree that it is important to verify whether forgetting persists under rephrased or indirect queries. Because each benchmark’s standard protocol evaluates with the same questions and images used for unlearning, the results in the paper alone cannot answer this question.

To address this, we added two controlled evaluations on MLLMU-Bench with LLaVA-1.5-7B. First, we generated three paraphrases per question, preserving meaning and answer. Second, for each identity, we used three Test Set images of the same person that were never used during unlearning, and confirmed by comparing hashes of the decoded pixels that none of them overlaps with the original benchmark images.

Forget originalForget unseenForget paraphrase
Vanilla68.761.859.8
GA68.361.457.0
GD71.760.159.2
KL68.361.760.1
NPO65.057.054.8
PO75.062.960.7
MANU68.360.759.2
MIP-Editor71.761.358.0
Ours56.051.848.8

Since accuracy drops for all models, including Vanilla, when the query or image changes, we compare reductions relative to Vanilla. The baselines reduce accuracy by at most 5.0 points relative to Vanilla in all three conditions. In contrast, BTF reduces it by 12.7 points on the original setting, 10.0 points on unseen images, and 11.0 points on paraphrased questions. BTF’s forgetting effect is thus not confined to the query phrasings used in training, and is largely maintained on paraphrases and new images. These results are included in Table 2.

W2 & Q2. Relatively lower H-Mean in some settings

We acknowledge that BTF does not achieve the highest H-Mean in every setting. BTF achieves the highest F-R H-Mean in 8 of the 12 settings on UMU-Bench and CLEAR. The four remaining settings are:

  • UMU-Bench, Qwen2-VL-7B, image route: GD (49.2) is marginally ahead of BTF (49.0), but GD’s Retain VQA (35.9) is far below BTF’s (66.0).
  • CLEAR, LLaVA-1.5-7B, text route: KL achieves a higher H-Mean, but at the cost of a large drop in Retain QA from 92.0 (Vanilla) to 62.0. BTF’s Retain QA is 89.0.
  • CLEAR, Qwen2-VL-7B, both routes: In the setting noted by the reviewer, GD achieves a VQA/QA H-Mean of 67.5/81.8, compared with 53.1/27.3 for BTF.

Why GD leads on CLEAR with Qwen2-VL-7B. For this setting, we added a Retain-only model, trained without the forget identities, as a reference.

Forget VQAForget QARetain VQARetain QA
Vanilla45.097.035.095.0
Retain-only2.053.031.095.0
GD0.027.051.093.0
Ours6.084.037.094.0
  • Image route: Both methods forget successfully (Forget VQA: 0.0 for GD, 6.0 for BTF, 2.0 for the reference); the gap comes from retention. BTF’s Retain VQA (37.0) is close to Vanilla (35.0) and the reference (31.0), whereas GD’s is 51.0, 20 points above the reference. Its Real-World VQA, on identities never used in training, also rises from 57.0 to 93.0. The gap therefore does not stem from forgetting ability.
  • Text route: We acknowledge that BTF’s forgetting is insufficient: its Forget QA of 84.0 is 31 points above the reference (53.0). This weakness appears on CLEAR in general (69.0 and 54.0 on LLaVA-1.5-7B and 13B), compared with UMU-Bench (55.0, 20.0, 16.0). GD's 27.0, in turn, is 26 points below the reference. Since a model that never learned the forget identities still answers 53.0 correctly, this is suppression beyond the reference level, which H-Mean cannot distinguish because it rewards lower Forget accuracy.

We have incorporated this analysis into Section 4.2 and Appendix F.5 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer a4qT (2/3)

W3 & Q3. General-capability degradation and the POPE decrease

We thank the reviewer for this careful observation. We agree that the POPE decrease is a real limitation of our method, and we analyzed it further.

(1) Scope of the degradation

The average over the five benchmarks changes from 61.74 to 60.44. MMLU (47.1 → 46.8), MMBench (64.7 → 65.6), and GQA (51.8 → 53.7) are maintained, while SEED-Bench (61.4 → 59.7) and POPE decline. Although GA and KL collapse (averages of 27.74 and 47.82), we acknowledge that GD, NPO, PO, MANU, and MIP-Editor preserve POPE better than our method, and we do not claim that BTF fully preserves general capabilities.

(2) Error-type and component-wise analysis

We re-evaluated all 1,200 POPE items (600 positive) using likelihood-based yes/no scoring. Because this protocol differs from the greedy generation used in Table 5, the numbers differ slightly.

VariantAcc.PrecisionRecallYes-ratioFPFN
Vanilla83.3382.2685.0051.6711090
Stage 1 only81.0078.7085.0054.0013890
Stage 2 only84.3390.7176.5042.1747141
Full BTF78.7579.1978.0049.25123132
  • The degradation arises from combining the two stages, not from either stage alone. Stage 1 alone (81.00) and Stage 2 alone (84.33) remain close to Vanilla (83.33), but combining them lowers accuracy to 78.75.
  • The degradation reflects reduced discrimination rather than bias. Stage 1 shifts predictions toward “yes” and Stage 2 toward “no”; these opposite biases largely cancel out when combined, bringing the yes-ratio close to Vanilla's. However, both FP (+13) and FN (+42) increase, and the additional errors are mainly “no” answers for objects that are present.

We attribute this degradation to Stage 2’s retain supervision consisting only of QA about retain identities, so object discrimination on general images is not preserved during training. Including general-image VQA data in the retain set is a natural direction for improvement. The POPE degradation and its component-wise analysis are now discussed in Section 4.7 and Appendix F.8 (Table 13) of the revised manuscript.

(3) Effect of the forget ratio

Retain VQA decreases as the forget ratio increases, with most of the decrease occurring between 5% and 10%. Over the same range, Forget VQA and Retain QA remain stable, so the image-route F-R H-Mean stays nearly unchanged (62.3 → 59.3 → 60.1). This decrease is not excessive relative to other methods: GD, MANU, and NPO preserve retention but barely forget, PO loses more retention than BTF, and KL collapses. Scalability to larger forget ratios is now discussed as a limitation in Section 4.4 and Appendix A.

AuthorsOpenReview ↗

Response to Reviewer a4qT (3/3)

Q4. Threat model and interpretation of the relearning experiments

The threat model assumed in these experiments is the benign relearning of Hu et al. (2025): after unlearning, a small amount of data related to the forget targets is included in further fine-tuning. In particular, we focus on whether relearning only a subset of the forget identities also restores responses about other forget identities that are not included in the relearning data.

Accordingly, the experiments use small amounts of data and short training budgets. On UMU-Bench with LLaVA-1.5-7B, we relearned 5, 25, or 50 of the 50 forget identities and measured accuracy on the full forget set. Relearning used all 12 QA examples per identity and ran for 8 epochs with a learning rate of 1e-4. In the training-budget analysis (Figure 4c), we trained on a single forget example for up to 8 epochs with the same learning rate and measured accuracy at each epoch. The experiments thus cover data scales from a single example to the full forget set, and training budgets of up to 8 epochs.

Under these settings, the experiments support the following conclusions.

  • Recovery is limited under small-scale relearning. When 5 identities are relearned, BTF's Forget accuracy is 40.0 on the image route and 51.0 on the text route. Under the same condition, Vanilla reaches 84 / 75 and PO reaches 87 / 73 (Figure 4b).
  • Recovery remains limited as the training budget increases. With a single example and increasing epochs, PO’s image-route Forget accuracy rises from 68.0 before the attack to 87 after 8 epochs. In contrast, BTF stays at or below its pre-attack level (46.0) at every evaluated point and returns to 46.0 after 8 epochs (Figure 4c).
  • Low recovery is also maintained on other benchmarks and models. Across the six benchmark–model combinations, BTF’s Forget accuracy after relearning ranges from 20.0 to 40.0 (Figure 4a). Some baselines show lower values than BTF in some settings, but most of these are models whose Retain accuracy had already dropped substantially right after unlearning (Table 1).
  • Forgetting weakens as the scope of relearning widens, and recovery is faster on the text route. When 25 identities are relearned, text-route Forget accuracy rises to 77.0, while the image route stays at 46.0. When all 50 are relearned, accuracy rises to 78.0 on the image route and 81.0 on the text route.

These results therefore provide evidence that target-response recovery is limited under small data and short training budgets. They do not guarantee resistance to stronger attacks that use sufficient forget data or prolonged additional training. The threat model, experimental settings, and scope of our conclusions are now clarified in Section 4.6 of the revised manuscript.

Reviewer_XiY4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-stage method for identity unlearning in MLLMs. It changes image features and trains LoRA adapters to reduce target responses from both image and text inputs. Experiments show promising results, but performance varies across models and settings.

Critical strengths

The visual projector is a well-motivated intervention point, and explicitly addressing both image-conditioned and text-only routes is important for consistent identity unlearning across modalities.

Critical weaknesses
  1. The method does not consistently achieve the best F-R H-Mean across settings.
  2. The paper does not clearly show whether forgetting works on unseen images of the same identities. More ablation studies are also needed to test the effects of δ*, the hinge loss, and the anchor set G.
  3. The experiments only cover LLaVA-1.5 and Qwen2-VL. Since the method relies on an explicit visual projector, it is unclear whether it works on newer MLLMs or models with different architectures.
Questions for authors
  1. How were the hyperparameters selected?
  2. Are the evaluation images different from the Stage 1 training images? Can you report results on unseen images of the same identities?
  3. Can you provide more ablation studies on Stage 1 alone, δ*, the hinge loss, and the anchor set G?
  4. Can you evaluate the method on a newer MLLM, such as Qwen3.5?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XiY4 (1/3)

We thank the reviewer for recognizing the validity of our visual-projector-based intervention and the importance of addressing both image-conditioned and text-only input routes. We address the reviewer’s concerns below.

W1. Consistency of F-R H-Mean performance

We agree that BTF does not achieve the highest F-R H-Mean in every setting. However, BTF achieves the highest H-Mean in most settings, and the settings where it does not rank first can be explained concretely.

In Table 1, BTF achieves the highest F-R H-Mean in 8 of the 12 settings on UMU-Bench and CLEAR, with improvements of up to 18.1 points over the strongest baseline. On UMU-Bench in particular, it achieves the highest H-Mean on both routes for both LLaVA-1.5-7B and 13B. In the newly added Qwen3.5-4B experiment (see W3 below), BTF also achieves the highest H-Mean on both routes.

The four settings where BTF does not rank first are:

  • UMU-Bench, Qwen2-VL-7B, image route: GD (49.2) is marginally ahead of BTF (49.0). However, GD’s Retain VQA is 35.9, far below BTF’s 66.0.
  • CLEAR, LLaVA-1.5-7B, text route: KL achieves a higher H-Mean, but at the cost of a large drop in Retain QA from 92.0 (Vanilla) to 62.0. BTF’s Retain QA is 89.0.
  • CLEAR, Qwen2-VL-7B, both routes: GD achieves a higher H-Mean. We added a Retain-only reference for this setting (see our response to Reviewer a4qT, W2). On the image route, BTF is close to the reference (Forget VQA 6.0 vs. 2.0; Retain VQA 37.0 vs. 31.0), whereas GD’s Retain VQA (51.0) is 20 points above it, which raises its H-Mean. On the text route, BTF’s forgetting is insufficient (Forget QA 84.0 vs. 53.0 for the reference), which we acknowledge as a limitation; GD’s 27.0, in turn, falls 26 points below the reference. Moreover, GD’s advantage does not hold in other settings: on UMU-Bench with LLaVA-1.5-7B its H-Mean is only 31.5 / 35.8, and on MLLMU-Bench its Forget accuracy is even higher than Vanilla’s (71.7 vs. 68.7).

Therefore, although BTF does not rank first in every setting, it provides a stable forgetting-retention balance across input routes and models. We have incorporated this analysis into Section 4.2 and Appendix F.5 of the revised manuscript.

W2 & Q2. Forgetting generalization to images not used in training

We agree that it is important to verify whether forgetting persists on new images of the same identities.

To answer the question directly: our main experiments follow each benchmark’s standard protocol, so the evaluation images are the same as those used for unlearning. The existing results therefore cannot answer this question.

To address this, we added a controlled evaluation with non-overlapping images on MLLMU-Bench with LLaVA-1.5-7B. For each identity, we used three Test Set images of the same person that were never used during unlearning, and confirmed by comparing hashes of the decoded pixels that none of them overlaps with the original benchmark images. We also generated three paraphrases per question, preserving meaning and answer, to evaluate generalization to query variations.

MethodForget originalForget unseenForget paraphrase
Vanilla68.761.859.8
GA68.361.457.0
GD71.760.159.2
KL68.361.760.1
NPO65.057.054.8
PO75.062.960.7
MANU68.360.759.2
MIP-Editor71.761.358.0
Ours56.051.848.8

Since accuracy drops for all models on unseen images, including Vanilla, we compare reductions relative to Vanilla. The baselines reduce accuracy by at most 5.0 points relative to Vanilla in all three conditions. In contrast, BTF reduces it by 12.7 points on the original images, 10.0 points on unseen images, and 11.0 points on paraphrased questions. Most of the forgetting effect obtained on the training images thus carries over to new images of the same identities and to rephrased questions.

This generalization is consistent with the design of BTF. Stage 1 does not memorize outputs for individual images. Instead, it trains the parameters of the shared visual projector to shift the visual representations of forget identities toward the generic-face representation. Retargeting can therefore apply to new images whose encoder features belong to the same identity. In addition, Stage 2 suppresses target answers not only at z0(x)z_0(x) but also at its perturbed counterpart z0(x)+δ∗z_0(x)+\delta^*, which improves robustness to variations in the visual representation. These results have been incorporated into Table 2 and Section 4.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer XiY4 (2/3)

W2 & Q3. Additional component analysis

Following the reviewer’s suggestion, we extended the ablation on UMU-Bench with LLaVA-1.5-7B.

VariantForget VQAForget QARetain VQARetain QAH-Mean VQAH-Mean QA
Ours46.055.065.870.959.355.1
Stage 1 only61.876.062.069.847.335.7
w/o δ∗\delta^*63.165.063.163.646.645.2
Hinge → unbounded GA44.548.460.049.857.750.7
Hinge → NPO47.051.657.853.955.351.0
w/o G in preservation term43.655.656.056.456.249.7

Among the added variants, the full BTF achieves the highest H-Mean on both routes. The role of each component is as follows.

  • Stage 1 only: Stage 1 alone reduces Forget VQA from 81.5 to 61.8, but Retain VQA also drops from 75.9 to 62.0. Adding Stage 2 further lowers Forget VQA to 46.0 while Retain VQA recovers to 65.8. Moreover, since Stage 1 modifies only the projector, text-only performance is identical to Vanilla (Forget QA 76.0, Retain QA 69.8). This shows that Stage 2’s text-route suppression is necessary for text-only forgetting.
  • δ∗\delta^*: Removing δ∗\delta^* increases Forget VQA from 46.0 to 63.1 and causes the largest drop in VQA H-Mean among all variants (59.3 → 46.6). Among the components of Stage 2, the adversarial perturbation contributes most to image-route forgetting. Although δ∗\delta^* is used only in the image-route loss, Forget QA also increases from 55.0 to 65.0. We attribute this to the two routes sharing the language model’s LoRA adapters.
  • Hinge loss: Replacing the hinge with unbounded GA or NPO yields similar or slightly lower Forget accuracy, but Retain QA drops by 21.1 and 17.0 points, respectively. This is consistent with the design intent: by stopping suppression once the answer falls below the threshold, the hinge prevents degradation from excessive suppression.
  • Anchor set G: Removing G from the preservation term slightly lowers Forget VQA (43.6) but reduces Retain VQA by 9.8 points. This shows that G prevents generic-face representations from shifting toward the retargeting target together with the forget identities, thereby helping preserve visual information about non-target identities.

These additional ablation results and the corresponding analysis are included in Appendix F.6 (Table 11) of the revised manuscript.

Q1. Hyperparameter selection

We selected hyperparameters on the evaluation set of UMU-Bench with LLaVA-1.5-7B, without a separate validation split, based on the balance between forgetting and Retain accuracy. We then applied the same configuration without re-tuning to all other benchmarks and models, including the newly added Qwen3.5-4B. The results in those settings therefore show whether the configuration transfers to settings that were not used for tuning.

  • Stage 1 epochs: We compared 10–150 epochs (Appendix F.4) and used 150 epochs, which gave the best balance between forgetting and retention.
  • Stage 2 epochs: We compared 2, 4, and 8 epochs (Appendix F.4). Eight epochs gave only marginal forgetting gains while Retain VQA dropped substantially (65.8 → 53.4), so we used 4 epochs.
  • Hinge threshold τ\tau: We compared −2.0, −1.5, and −1.0 (Figure 3). τ=−1.5\tau = −1.5 achieved the highest Retain accuracy on both routes. The other two thresholds strengthen text-only forgetting but reduce retention.
  • Other settings. We use LoRA with rank r=16r = 16 and α=32\alpha = 32, a learning rate of 1×10−41 \times 10^{-4}, and a forget-to-retain data ratio of 1:2. Results for other ratios are reported in Appendix F.2.

The hyperparameter selection protocol is clarified in Section 4.1, with the supporting sensitivity analyses provided in Section 4.4 (Figure 3) and Appendix F.4 (Figure 5) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer XiY4 (3/3)

W3 & Q4. Extension to recent MLLMs

Following the reviewer’s suggestion, we added UMU-Bench experiments on Qwen3.5-4B. Unlike Qwen3-VL, Qwen3.5 has no DeepStack mergers (deepstack_visual_indexes = []), so the vision encoder’s output reaches the language model only through a single patch merger. We applied Stage 1 of BTF to this merger and kept the vision encoder frozen. All other hyperparameters are the same as in the main paper.

Qwen3.5-4BForget VQAForget QARetain VQARetain QAF-R H-Mean VQAF-R H-Mean QAReal-World VQAReal-World QA
Vanilla68.068.060.457.441.841.158.462.7
GA28.028.029.429.441.841.825.225.2
GD69.066.060.157.440.942.758.864.0
KL24.024.026.026.038.738.726.526.5
NPO39.067.059.355.860.141.558.163.0
PO15.030.024.033.037.444.919.321.9
MANU43.071.058.656.157.838.257.161.7
MIP-Editor37.058.060.157.661.548.659.162.4
Ours29.022.059.357.664.666.358.260.2

BTF achieves the highest H-Mean on both routes on Qwen3.5-4B. The gap is especially large on the text route, where BTF exceeds the strongest baseline, MIP-Editor, by 17.7 points. GA, KL, and PO achieve low Forget accuracy, but their Retain and Real-World accuracy collapse as well. In contrast, BTF maintains Retain and Real-World accuracy at Vanilla levels. This shows that BTF is applicable to recent architectures with an explicit visual-to-language interface.

That said, defining an equivalent intervention point may be less straightforward for architectures that process vision and language tokens within a single backbone without a separate connector. The Qwen3.5-4B results are now reported in Appendix F.7 (Table 12) and referenced in Section 4.2, while the architectural applicability and limitations of BTF are discussed in Appendix A of the revised manuscript.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer XnzT (1/2)

We thank the reviewer for recognizing our route-aware perspective on multimodal unlearning and the validity of intervening at the visual-to-language interface. We address the reviewer's concerns below.

W1 & Q1. Comparison with a Retain-only retrained model

We agree that a model retrained only on retain data is the primary reference for unlearning, and that this comparison is particularly important for a method like Stage 2, which directly lowers the probability of target answers. We therefore added a Retain-only model on LLaVA-1.5-7B, trained without the forget identities. Using the same configuration as the memorized checkpoint, we fine-tuned the pretrained backbone on the retain identities only.

Forget VQAForget QARetain VQARetain QA
UMU-Bench
Vanilla81.576.075.969.8
Retain-only44.024.078.675.0
Ours46.055.065.870.9
CLEAR
Vanilla47.092.057.092.0
Retain-only1.060.067.096.0
Ours3.069.058.089.0

This comparison shows three things.

  • On the image route, BTF's Forget accuracy is nearly identical to that of the Retain-only model. BTF thus reaches the reference level without over-suppressing below it.
  • On the text route, BTF's forgetting falls short of the reference. The gap is smaller on CLEAR (69.0 vs. 60.0) than on UMU-Bench, where it is large (55.0 vs. 24.0). A substantial amount of information about the forget identities remains on the text route. However, this gap is related to the trade-off with retain supervision: under the original setting with less retain data, BTF's Forget QA drops to 4.0, below the reference, but its Retain QA also drops to 57.8 (Table 7).
  • BTF's Retain accuracy is lower than that of the Retain-only model, particularly on the image route.

The reference also changes how low Forget accuracy should be interpreted. For example, on CLEAR, GD's Forget QA is 11.0, far below the Retain-only model's 60.0, but this is a consequence of its Retain QA collapsing to 29.0 (Table 1). Since Forget accuracy closer to the reference is preferable to simply lower Forget accuracy, this comparison complements our evaluation.

Because the forget identities in these benchmarks are injected via LoRA fine-tuning, retraining is inexpensive, which allowed us to construct the Retain-only model as a reference. Based on this comparison, we have qualified our claims: our results provide behavioral evidence that responses about forget identities are suppressed across both input routes, but they do not certify that the influence of the training data has been completely removed at the parameter level. The Retain-only comparison is now included in Section 4.5 (Table 4), and our claims are qualified in the Introduction of the revised manuscript.

W2 & Q2. Fairness of the baseline comparison

We acknowledge that we did not apply the same search space and selection procedure to all methods. We describe the actual procedure and its implications below.

(1) How each method's configuration was determined

  • Baselines: We used the implementation settings provided by each benchmark (Appendix D). The only exceptions are GA and KL. Under their original settings, Retain accuracy collapses to 0.0 and H-Mean becomes 0.0, so we used the checkpoints just before collapse. This adjustment raised their VQA/QA H-Mean from 0.0/0.0 to 37.9/35.5 and 48.6/43.9, respectively (Table 6); it therefore favors the baselines.
  • BTF: We selected its configuration on the evaluation set of UMU-Bench with LLaVA-1.5-7B, without a separate validation split, based on the balance between forgetting and retention. BTF therefore has a selection advantage in this one setting. We applied the same configuration without re-tuning to the remaining five benchmark–model combinations and to MLLMU-Bench.

(2) Results in untuned settings

Excluding UMU-Bench with LLaVA-1.5-7B, where BTF was tuned, BTF achieves the highest F-R H-Mean in 6 of the remaining 10 settings (Table 1). In the Qwen3.5-4B UMU-Bench experiment added at Reviewer XiY4's request, BTF also achieves the highest H-Mean on both routes without re-tuning.

(3) Supervision budget

All methods use the same forget set for forget supervision. Retain supervision is controlled in Table 7. Under the matched setting, where all methods use 1,200 retain QA examples per route, BTF achieves a VQA/QA H-Mean of 59.3/55.1, while the strongest baseline, MANU, achieves 43.0/40.6. Even when each baseline is allowed the more favorable of its original and matched settings, BTF remains higher.

The selection procedure and baseline comparisons are now clarified in Section 4.1 and Appendices D, F.1, and F.2 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer XnzT (2/2)

W3. Limited statistical evidence

We extended the repeated runs to the remaining five benchmark–model combinations. For each combination, we trained BTF and the top two baselines from Table 1 with three seeds. The five-run results for UMU-Bench with LLaVA-1.5-7B are reported in Table 8.

Forget VQAForget QARetain VQARetain QAReal-World VQAReal-World QA
LLaVA-1.5-7BCLEAR
GD3.3 ± 4.611.3 ± 0.623.7 ± 1.629.0 ± 0.023.4 ± 0.333.0 ± 0.1
KL15.0 ± 4.057.3 ± 0.222.0 ± 3.562.0 ± 0.316.0 ± 2.436.3 ± 5.0
Ours3.0 ± 0.068.7 ± 0.358.0 ± 0.789.7 ± 1.277.0 ± 0.044.0 ± 1.0
LLaVA-1.5-13BUMU-Bench
PO51.3 ± 1.236.7 ± 0.648.1 ± 0.134.2 ± 0.852.5 ± 0.854.8 ± 0.4
MANU48.3 ± 0.839.3 ± 1.548.4 ± 4.035.6 ± 1.344.8 ± 1.259.4 ± 0.7
Ours41.5 ± 2.119.9 ± 0.246.1 ± 0.935.9 ± 0.148.9 ± 0.552.0 ± 0.5
CLEAR
GD3.7 ± 1.84.3 ± 2.125.0 ± 0.944.0 ± 0.822.0 ± 1.140.0 ± 7.1
KL13.3 ± 3.35.3 ± 1.532.0 ± 0.534.7 ± 6.038.0 ± 2.641.0 ± 4.2
Ours6.0 ± 0.553.7 ± 0.246.7 ± 1.095.0 ± 0.275.0 ± 1.042.0 ± 1.6
Qwen2-VL-7BUMU-Bench
GD24.0 ± 2.624.0 ± 3.536.1 ± 4.439.3 ± 4.439.1 ± 2.039.2 ± 6.3
PO60.0 ± 6.635.7 ± 2.659.2 ± 3.360.4 ± 1.047.8 ± 3.344.1 ± 3.9
Ours57.0 ± 3.516.3 ± 0.266.6 ± 1.066.3 ± 0.348.5 ± 0.950.5 ± 2.5
CLEAR
GD1.0 ± 0.627.0 ± 0.350.3 ± 0.892.0 ± 1.793.0 ± 0.142.0 ± 2.6
KL5.0 ± 3.643.3 ± 2.121.7 ± 3.556.0 ± 0.924.0 ± 2.533.7 ± 0.6
Ours6.7 ± 0.586.0 ± 1.737.0 ± 0.190.7 ± 2.150.3 ± 1.541.0 ± 2.6
  • The conclusions of Table 1 hold under repeated runs. Across the 10 settings, the settings where BTF leads and those where a baseline leads (the text route of CLEAR with LLaVA-1.5-7B and both routes of CLEAR with Qwen2-VL-7B) match Table 1. The only difference is the image route of UMU-Bench with Qwen2-VL-7B: GD led by 0.2 points in Table 1, but BTF leads on average.
  • The large variance in text-route forgetting is not a general phenomenon. The Forget QA standard deviation of 8.1 observed in Table 7 is specific to UMU-Bench with LLaVA-1.5-7B; in the other five combinations, BTF’s Forget QA standard deviation ranges from 0.2 to 1.7. Moreover, the value reported for that setting in Table 1 (Forget QA 55.0) comes from one of the five runs and is less favorable than the five-run mean (47.6).

These additional repeated-run results and the corresponding analysis are now included in Appendix F.3 (Tables 8, 9) of the revised manuscript.

AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their constructive feedback. We have uploaded a revised version of the paper, with all changes marked in blue. The main changes are as follows.

  • Retain-only reference (XnzT W1, a4qT W2): We added a comparison with models trained without the forget identities (Section 4.5, Table 4) and analyzed the settings in which our method does not rank first (Appendix F.5).
  • Generalization (XiY4 W2, a4qT W1): We added evaluations on unseen images and paraphrased questions (Table 2, Section 4.3).
  • Additional ablations (XiY4 Q3): We analyzed Stage 1 alone, δ∗\delta^*, the hinge loss, and the anchor set G\mathcal{G} (Appendix F.6, Table 11).
  • Recent MLLM (XiY4 W3): We added results on Qwen3.5-4B (Appendix F.7, Table 12).
  • Hyperparameter selection (XiY4 Q1, XnzT W2): We stated the selection procedure in Section 4.1.
  • Statistical evidence (XnzT W3): We extended the repeated runs to the remaining five benchmark–backbone combinations (Appendix F.3, Table 9).
  • General capability (a4qT W3): We added an error analysis on POPE (Appendix F.8, Table 13) and noted the limitation regarding the forget ratio in Section 4.4.
  • Relearning (a4qT Q4): We stated the threat model and experimental settings in Section 4.6.

We hope the revision addresses your concerns. If any concerns remain or further questions arise, we would be glad to address them during the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.