acceptodds
Under review as a conference paper at ICLR 2027

What Does It Mean to Forget a Person? Individual-Level Unlearning in Vision-Language Models

11 views

Abstract

Erasing individual identities from Vision-Language Models (VLMs) is uniquely challenging because personal data is entangled across modalities rather than stored as isolated attributes. However, existing multimodal unlearning benchmarks primarily evaluate attribute-centric forgetting, overlooking the more critical objective of individual-level unlearning: eliminating a model's ability to access, link, and reconstruct target-related information across modalities. To address this gap, we propose IDUnlearn-Bench, the first benchmark for individual-level multimodal unlearning in VLMs. It represents each individual as connected multimodal evidence and evaluates four task families: attribute access, identity access, identity binding, and identity reconstruction. Experiments on representative VLMs and unlearning methods show that successful attribute-centric forgetting often leaves substantial identity-level knowledge intact and can be non-monotonic: reducing one form of risk may amplify another. Models may suppress selected information while still identifying the target, linking records, or reconstructing the individual. These findings reveal a fundamental gap between forgetting information about a person and forgetting the person as a whole.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Unlearning, Vision-Language Models, Individual-Level Unlearning, Machine Unlearning, Privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_fRLqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IDUnlearn-Bench, a benchmark for evaluating individual-level unlearning in VLMs. Beyond attribute-centric forgetting, it assesses whether information about a target individual remains accessible, linkable, or reconstructable through alternative multimodal and relational evidence. The benchmark comprises 60 synthetic identities, 10,050 fine-tuning QA pairs, and 4,860 evaluation probes organized into four task families and nine subtasks: Attribute Access, Identity Access, Identity Binding, and Identity Reconstruction. This paper evaluate seven unlearning method families across six VLMs from three model families. The experiments reveal that suppressing direct attribute retrieval often leaves alternative identity-access and relational reconstruction paths intact. They further identify asymmetric forgetting across query directions and modalities, along with trade-offs between forgetting effectiveness and retained utility. These findings highlight the limitations of existing unlearning evaluations and motivate more comprehensive assessment of identity-level knowledge removal.

Critical strengths
  • The identity-centric framework expands unlearning evaluation to reverse lookup, evidence binding, and relational reconstruction, providing a valuable basis for auditing residual personal information.

  • The cross-path experiments demonstrate practical diagnostic value by revealing differences in recoverability across query directions and identity keys.

Critical weaknesses
  • The positive-only Identity Binding protocol conflates identity-linking ability with affirmative-response bias: an always-Yes model receives the maximum score. This paper acknowledge this issue for LLaVA, while analogous response shifts remain a plausible explanation for increased binding recovery on other backbones . Matched negative pairs and joint TPR/FPR measurements are essential to substantiate the claimed amplification of identity-linkage risk [1].

  • Initial acquisition varies substantially across tasks, and base models already recover some identity-related information, making the contribution of target-specific learning ambiguous. ReMem [2] demonstrates how inadequate prior memorization can compromise unlearning evaluation, while SALMUBench [3] provides a relevant precedent through clean and compromised reference models. Extending the existing Vanilla-conditioned analyses and evaluating the retain-only reference described in Appendix D.3 would help distinguish acquired personal knowledge from pre-existing capabilities and evidence-supported inference. These controls are important for substantiating selective removal of learned personal information.

[1] Li, Yifan, et al. "Analyzing and mitigating object hallucination: A training bias perspective." AAAI 2026.

[2] Kwon, JuneHyoung, et al. "Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks." ACL 2026 Findings.

[3] Selvas-Sala, Cai, Lei Kang, and Lluis Gomez. "SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning." CVPR 2026.

Questions for authors
  • Could you evaluate matched negative pairs in a representative risk-amplification setting and report TPR and FPR before and after unlearning? This would clarify whether the observed binding increase reflects improved identity discrimination or response bias.

  • How does the evaluation distinguish acquired personal knowledge from inference supported by the supplied evidence? Could you report results from the retain-only reference described in Appendix D.3?

  • Do the main conclusions persist when Vanilla-conditioned recovery is reported systematically across task families? Please include eligible sample counts and both correct-to-incorrect and incorrect-to-correct transitions to clarify the effects of heterogeneous initial acquisition.

3 comments

We thank the reviewer for recognizing the value of our identity-centric benchmark. We have addressed Yes-response bias, initial knowledge acquisition, and residual recovery after unlearning through additional analyses in Sections 3.3 and 4.3.2 and Appendices E.1 and E.5. All revisions are highlighted in blue in the updated manuscript.


(W1, Q1) Identity Binding with Matched Negative Pairs

Identity-linking risks remain after unlearning, even when matched negative pairs are considered. We extend Identity Binding with 40 matched negative pairs on Qwen3-VL-8B under Forget-5 to distinguish residual identity associations from Yes-response bias.

(1) Positive and negative evaluation.

MethodPaper IB+TPRFPRCorrect matched pairs
Vanilla65.00%95.00%97.50%0/40
MMUN-Both35.00%60.00%67.50%1/40
MMUN-Lang77.50%55.00%42.50%6/40
MMUN-Vis10.00%27.50%25.00%3/40

After MMUN-Lang, 6/40 matched pairs are correctly distinguished, compared with 0/40 for Vanilla. This shows that some identity associations remain distinguishable beyond affirmative-response bias.

These results provide evidence of residual identity-linking capability after unlearning. We will report negative-pair controls alongside positive-pair recovery to distinguish genuine identity associations from response bias.

(W2, Q2) Initial Knowledge Acquisition and Retain-only Control

The retain-only control confirms substantial target-data-related knowledge acquisition, while identity reconstruction remains highly recoverable after unlearning. We evaluate the Base-initialized retain-only reference from Appendix D.3 on Qwen3-VL-4B under Forget-5.

(1) Target knowledge acquisition. We compare Base, Vanilla, and Retain-only on the same target queries.

TaskBaseVanillaRetain-onlyAcquisition gap
AA10.48%98.57%22.38%+76.19 pp
IA20.91%70.00%25.45%+44.55 pp
IR24.44%71.11%26.67%+44.44 pp

Retain-only preserves strong general task competence. On retained identities, its AA, IA, and IR scores reach 99.91%, 71.24%, and 68.69%, compared with 100.00%, 72.81%, and 71.92% for Vanilla. The large target-set gaps therefore cannot be explained simply by inadequate reference-model learning.

(2) Acquisition-controlled recovery. We further restrict evaluation to probes correctly answered by Vanilla but incorrectly answered by Retain-only, controlling for baseline inference and initial acquisition differences.

MethodAA (n=161)IA (n=49)IR (n=21)
GA-Diff122/161 (75.78%)47/49 (95.92%)20/21 (95.24%)
MMUN-Lang52/161 (32.30%)26/49 (53.06%)19/21 (90.48%)
MMUN-Vis70/161 (43.48%)39/49 (79.59%)15/21 (71.43%)

MMUN-Lang suppresses AA to 32.30%, yet IR remains at 90.48%. This includes 8/8 relational and 11/11 relational-attribute reconstruction probes. Thus, the discrepancy between attribute forgetting and identity reconstruction persists even after controlling for initial knowledge acquisition.

(W2, Q3) Vanilla-Conditioned Recovery and Knowledge Transitions

Yes. Our central finding remains that suppressing attribute-level retrieval does not necessarily eliminate other forms of identity recovery. We verify this using Vanilla-conditioned recovery across 96 valid configurations, covering six VLMs, nine unlearning configurations, and both Forget-1 and Forget-5.

(1) Controlling for initial acquisition. We restrict recovery to initially correct predictions and separately report correctness transitions. Results below use Qwen3-VL-4B with MMUN-Lang under Forget-5.

TaskVanilla correctC→CC→II→CConditional recovery
AA20794113045.41%
IA775423070.13%
IR32302493.75%
IB+404000100.00%

Conditional recovery counts only C→C among Vanilla-correct probes. MMUN-Lang suppresses 113/207 AA probes, while 30/32 IR probes remain recoverable. The four newly correct IR predictions are excluded, so they do not inflate residual recovery.

(2) Cross-model and systematic validation. The same analysis is extended across models and methods. Each cell reports recovery among Vanilla-correct probes under Forget-5.

Model / MethodAAIAIR
Qwen-2B / GA-Diff93/207 (44.93%)60/75 (80.00%)30/35 (85.71%)
Qwen-2B / MMUN-Lang83/207 (40.10%)44/75 (58.67%)19/35 (54.29%)
Qwen-4B / GA-Diff168/207 (81.16%)75/77 (97.40%)31/32 (96.88%)
Qwen-4B / MMUN-Lang94/207 (45.41%)54/77 (70.13%)30/32 (93.75%)
Qwen-4B / MMUN-Vis106/207 (51.21%)61/77 (79.22%)20/32 (62.50%)
Qwen-8B / MMUN-Lang30/207 (14.49%)22/77 (28.57%)12/32 (37.50%)
Qwen-8B / MMUN-Vis40/207 (19.32%)17/77 (22.08%)12/32 (37.50%)
LLaVA-7B / MMUN-Lang89/207 (43.00%)13/77 (16.88%)3/34 (8.82%)
LLaVA-13B / MMUN-Lang89/207 (43.00%)24/75 (32.00%)10/32 (31.25%)
Llama-11B / GA122/207 (58.94%)62/62 (100%)33/37 (89.19%)

Across all 96 configurations, 53/96 (55.21%) exhibit at least a 10 percentage-point difference among AA, IA, and IR, including 29/30 (96.67%) MMUN configurations. The difference reaches 20 points in 32/96 and 17/30 configurations, respectively.

The direction varies across models. Qwen-4B/MMUN-Lang retains substantially more IR than AA, whereas LLaVA-7B/MMUN-Lang shows the opposite pattern. These results confirm that direct attribute suppression alone is insufficient to characterize individual-level forgetting, with residual access depending on the model and unlearning method.

Thank you for the additional experiments. The retain-only and Vanilla-conditioned analyses largely address my concern about initial acquisition. But the evidence for increased linkage risk is more limited. It would great to reflect that qualification in the main text. Also, I see the broader evaluation of identity recoverability as a strength, but I still find the “first benchmark” claim too broad. MLLMU-Bench already uses individual profiles as forgetting targets, FIUBench focuses on facial identity, and ReMem probes residual information through indirect multimodal queries. A short related work section in the main paper would help clarify what the benchmark adds and explain the distinction between Profile and Individual in Table 7. Therefore, I will maintain my current score.

Reviewer_jQocOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IDUnlearn-Bench for evaluating individual-level multimodal unlearning in vision-language models. The benchmark contains 60 synthetic individuals, 10,050 training QA pairs, and 4,860 evaluation probes covering four task families: attribute access, identity access, identity binding, and identity reconstruction. Experiments with seven unlearning method families across six models reveal inconsistent forgetting across access paths: reducing direct attribute retrieval does not necessarily eliminate other identity associations, while stronger forgetting may damage retained knowledge.

Critical strengths
  1. The research question is practically relevant and supported by evidence of asymmetric recovery. The paper examines whether failure to retrieve information through one query path adequately reflects individual-level forgetting. Section 2.2 and Figure 3 extend the evaluation to identity access, identity binding, and relational reconstruction. Figure 4(b) illustrates the relevance of this distinction: among person–attribute pairs that Vanilla answers correctly in both directions, MMUN-Lang on Qwen3-VL-4B leaves 40% forward recovery but 100% reverse recovery. This demonstrates that suppressing one lookup direction can leave the same identity-attribute association accessible in the other direction, motivating evaluation beyond a single query path.
  2. The evaluation covers complementary tasks and a broad range of model settings. Table 5 documents nine subtasks spanning four task families, with 4,860 evaluation queries across 60 synthetic individuals. Sections 4.1-4.3 evaluate seven unlearning method families across six VLMs from three model families. This coverage provides useful evidence about both residual access and retained utility. For example, Table 2 shows that MMUN-Lang on Qwen3-VL-4B reduces forget-set AA from 98.57% to 44.76% while preserving 93.68% retain-set AA, yet forget-set IR remains 75.56%. Reporting these measures separately helps distinguish suppression of direct answers from persistent reconstruction ability and avoids judging forgetting solely by a reduction in one aggregate score.
Critical weaknesses
  1. The conditional extent of alternative-path recovery remains insufficiently quantified. Table 2 mainly reports aggregate recovery rates across task families, which do not directly quantify how often alternative paths succeed specifically when a direct query fails on the same fact. Figure 4(b) partially addresses this concern by comparing forward and reverse lookup on person-attribute pairs correctly answered in both directions before unlearning. However, the paper does not explicitly report the conditional recovery rate among cases where the designated direct query fails, together with the corresponding sample counts. Such an analysis would quantify how often direct-only evaluation incorrectly suggests successful forgetting and would strengthen the central claim about residual access after direct-answer suppression.
  2. The incremental diagnostic value over existing evaluation protocols remains insufficiently demonstrated. Table 4 identifies differences in task coverage, and the experiments illustrate complementary recovery patterns within the proposed benchmark. However, the paper does not directly compare a representative existing evaluation protocol with the expanded task suite on the same identities and unlearned checkpoints. Consequently, it remains unclear how many additional failures the proposed probes detect beyond those captured by existing evaluations, and which task families contribute these detections. Quantifying this additional diagnostic coverage is important for establishing the benchmark’s practical value, as broader task coverage alone does not demonstrate its ability to detect failures missed by existing evaluations.
Questions for authors
  1. Among facts correctly recalled before unlearning but no longer retrieved through direct queries afterward, what proportion can still be recovered through alternative paths?
  2. Which unlearning failures do the proposed probes detect that existing evaluations miss, and how much additional coverage do they provide?
1 comment

We thank the reviewer for recognizing the value of our benchmark in evaluating individual-level unlearning. We have strengthened the analysis of recovery after direct-query failure and additional detection beyond existing benchmarks in Section 4.3.1 and Appendix E.4. All revisions are highlighted in blue in the updated manuscript.


(W1, Q1) Additional Recovery after Direct-Access Failure

Yes. Alternative paths recover substantial identity-attribute associations even after direct-access failure. We align 69 associations correctly retrieved through name, face, and fingerprint queries before unlearning. A direct failure requires all three queries to fail after unlearning. We then test recovery through previously correct reverse and relational probes.

(1) Conditional recovery.

Model / MethodDirect failuresRecoveredRecovery rate
Qwen-2B / MMUN-Lang20525.00%
Qwen-4B / MMUN-Lang231356.52%
Qwen-8B / MMUN-Lang521528.85%
LLaVA-7B / MMUN-Lang2600.00%
LLaVA-13B / MMUN-Lang24416.67%
Llama-11B / GA14428.57%
Qwen-4B / GA-Diff5360.00%
Qwen-4B / MMUN-Vis20840.00%

(2) Recovery through alternative paths. The following counts distinguish associations recovered exclusively by reverse lookup, exclusively by relational queries, or by both.

Model / MethodReverse onlyRelational onlyBothTotal
Qwen-4B / MMUN-Lang51713
Qwen-8B / MMUN-Lang101415
Qwen-4B / MMUN-Vis5128
LLaVA-13B / MMUN-Lang1304
Llama-11B / GA1304

Alternative probes cover six attributes per identity. We retain all direct failures in the denominator, so the reported rates measure observed recovery without assuming untested associations are forgotten.

These paired results confirm that direct-access failure does not imply removal of the same identity-attribute association.

(W2, Q2) Incremental Diagnostic Value

IDUnlearn-Bench detects residual identity knowledge missed by existing extraction-based evaluations, particularly through relational reconstruction. Among eligible missed cases, our relational probes recover 11/20 (55.00%), demonstrating additional identity recovery beyond conventional attribute extraction.

(1) Matched evaluation and additional detection. We adapt MLLMU-Bench and FIUBench evaluation protocols to the same five identities and 12 unlearned checkpoints across Qwen3-VL-2B/4B/8B. We align queries by individual-attribute association and apply a unified answer parser. Additional detections require successful recovery both before and after unlearning.

Model / MethodLearned pairsExisting missesAdditional Attribute AccessAdditional Identity ReconstructionJoint recovery
Qwen-2B / GA-Diff2917202/17
Qwen-2B / MMUN-Lang295101/5
Qwen-2B / MMUN-Vis299000/9
Qwen-2B / NPO292000/2
Qwen-4B / GA-Diff339224/9
Qwen-4B / MMUN-Lang338000/8
Qwen-4B / MMUN-Vis338101/8
Qwen-4B / NPO336033/6
Qwen-8B / GA-Diff367033/7
Qwen-8B / MMUN-Lang3622000/22
Qwen-8B / MMUN-Vis3622000/22
Qwen-8B / NPO364033/4
Total39211961117/119 (14.29%)

(2) Complementary diagnostic value. We further distinguish attribute extraction, relational reconstruction, and identity recognition.

Evaluation targetEligible missesAdditional detections
Attribute Access beyond existing extraction656/65 (9.23%)
Identity Reconstruction through relational evidence2011/20 (55.00%)
Combined association detection11917/119 (14.29%)
Identity recovery after failed name extraction208/20 (40.00%)
Combined detection with multiple-choice control232/23 (8.70%)

The 11 relational detections correspond to six distinct associations across two target identities. Separately, alternative biometric, masked-document, and relational queries recover 8/20 identity cases missed by name extraction.

When multiple-choice recognition is also counted as an existing detection, 2/23 missed observations remain recoverable through relational probes, representing one underlying association across two checkpoints.

These results demonstrate that identity-level probes reveal residual associations missed by conventional extraction.

Comments on the paper

3 comments

We thank the reviewer for recognizing the importance of individual-level unlearning and the broader coverage of our benchmark. We have added experiments addressing Yes-response bias, identity selection robustness, and alternative knowledge acquisition in Appendices E.1–E.3, with further clarification of external validity in Appendix G. All revisions are highlighted in blue in the updated manuscript.


(W1, Q1) Identity Binding with Matched Negative Pairs

Negative-pair evaluation reveals that some identity links remain distinguishable after unlearning, beyond simple Yes-response bias. We evaluate 40 positive and 40 matched negative pairs on Qwen3-VL-8B under Forget-5.

(1) Identity Binding with negative controls.

MethodTPRFPRCorrectly distinguished pairsAccuracy
Vanilla95.00%97.50%0/4048.75%
MMUN-Both60.00%67.50%1/4046.25%
MMUN-Lang55.00%42.50%6/4056.25%
MMUN-Vis27.50%25.00%3/4051.25%

Vanilla shows substantial Yes-response bias, accepting 39/40 mismatched pairs. After MMUN-Lang, 6/40 identity pairs are correctly distinguished, compared with 0/40 before unlearning. Overall accuracy increases by 7.50 percentage points, while FPR decreases from 97.50% to 42.50%.

These results demonstrate residual identity-linking capability under negative-pair controls. We will report positive-pair recovery and identity discrimination separately to provide a more precise interpretation of the risk-transfer phenomenon.

(W2, Q2) Robustness to Forget-5 Identity Selection

Our main finding persists across different identity subsets. GA-Diff consistently suppresses attribute access across five new Forget-5 splits, while substantial identity access remains recoverable. Reconstruction shows greater identity-dependent variability, highlighting the limitations of evaluating forgetting through direct attributes alone.

(1) Repeated identity subsets. We repeat Qwen3-VL-4B with GA-Diff on five additional disjoint subsets covering 25 distinct identities, using the same Vanilla checkpoint and training hyperparameters.

SubsetVanilla AAUnlearned AAAA reductionConditional IAConditional IR
S185.24%56.67%28.57 pp27/45 (60.00%)0/2 (0.00%)
S280.95%57.14%23.81 pp31/44 (70.45%)5/11 (45.45%)
S389.05%54.76%34.29 pp39/65 (60.00%)7/13 (53.85%)
S490.95%56.19%34.76 pp43/55 (78.18%)2/5 (40.00%)
S590.95%51.90%39.05 pp55/67 (82.09%)20/24 (83.33%)

Across all five subsets, AA decreases by 23.81 to 39.05 pp, yet conditional IA recovery remains between 60.00% and 82.09%. Thus, direct attribute suppression does not imply comparable removal of identity access.

(2) Sensitivity across privacy tasks. We quantify variability across the five new subsets, reporting sample standard deviations.

TaskOriginal forgetNew subsets, Mean ± SDConditional recovery, Mean ± SD
AA62.38%55.33 ± 2.11%62.93 ± 4.92%
IA41.82%36.18 ± 10.33%70.15 ± 10.16%
IR26.67%20.44 ± 19.01%44.53 ± 29.99%

AA exhibits relatively consistent suppression, while IA and IR are more sensitive to identity composition. IR also shows substantial variation in initial acquisition, with only 2 to 24 Vanilla-correct probes per subset.

These findings reinforce the need for identity-level evaluation. Consistent direct attribute suppression does not guarantee equally consistent forgetting across alternative identity access paths.

(W3, Q3) External Validity and Generalization

We further investigate document-based knowledge acquisition beyond direct profile fine-tuning. This experiment examines whether residual personal information remains accessible through different identity-related tasks after unlearning.

(1) Document-based acquisition. We fine-tune Qwen3-VL-4B on 1,860 document images and 9,780 VQA samples, followed by GA-Diff unlearning.

Forget-set evaluationBaseDoc-VanillaGA-Diff
Document QA26.34%77.07%45.85%
Attribute Access6.19%7.62%4.76%
Identity Access0.00%2.73%1.82%
Identity Reconstruction0.00%0.00%0.00%

Document QA improves by 50.73 percentage points after acquisition. GA-Diff retains 8/16 (50.00%) initially correct Attribute Access probes and 2/3 (66.67%) Identity Access probes. These results show residual task-level recovery even without successful Identity Reconstruction.

(2) Rationale for synthetic identities. Synthetic identities are widely adopted in multimodal unlearning benchmarks, including MLLMU-Bench [1], FIUBench [2], and CLEAR [3].

BenchmarkData design
MLLMU-Bench [1]Fictitious personal profiles
FIUBench [2]Fictitious facial identities
CLEAR [3]Synthetic character identities
IDUnlearn-BenchConnected multimodal identities

Synthetic identities avoid exposing real personal information during knowledge acquisition and unlearning experiments. They also enable controlled knowledge acquisition, reproducible forgetting targets, and reliable ground truth.

Our benchmark extends this established approach to individual-level access, binding, and reconstruction.

References

[1] Liu et al. Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench. NAACL 2025.

[2] Ma et al. Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset. ICLR 2025.

[3] Dontsov et al. CLEAR, Character Unlearning in Textual and Visual Modalities. Findings of ACL 2025.

I have reviewed the detailed responses and additional experiments provided in the authors’ rebuttal, which have addressed some of my concerns. Overall, I find this to be an interesting paper, but some of the claims go beyond what is supported by the current evidence, and I hope the authors to tone down these claims. For example,I am not entirely convinced by the conclusion regarding “risk transfer.” Moreover, I think the paper presentation quality could be improved. So, I will maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.