- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces IDUnlearn-Bench, a benchmark for evaluating individual-level unlearning in VLMs. Beyond attribute-centric forgetting, it assesses whether information about a target individual remains accessible, linkable, or reconstructable through alternative multimodal and relational evidence. The benchmark comprises 60 synthetic identities, 10,050 fine-tuning QA pairs, and 4,860 evaluation probes organized into four task families and nine subtasks: Attribute Access, Identity Access, Identity Binding, and Identity Reconstruction. This paper evaluate seven unlearning method families across six VLMs from three model families. The experiments reveal that suppressing direct attribute retrieval often leaves alternative identity-access and relational reconstruction paths intact. They further identify asymmetric forgetting across query directions and modalities, along with trade-offs between forgetting effectiveness and retained utility. These findings highlight the limitations of existing unlearning evaluations and motivate more comprehensive assessment of identity-level knowledge removal.
Critical strengths
-
The identity-centric framework expands unlearning evaluation to reverse lookup, evidence binding, and relational reconstruction, providing a valuable basis for auditing residual personal information.
-
The cross-path experiments demonstrate practical diagnostic value by revealing differences in recoverability across query directions and identity keys.
Critical weaknesses
-
The positive-only Identity Binding protocol conflates identity-linking ability with affirmative-response bias: an always-Yes model receives the maximum score. This paper acknowledge this issue for LLaVA, while analogous response shifts remain a plausible explanation for increased binding recovery on other backbones . Matched negative pairs and joint TPR/FPR measurements are essential to substantiate the claimed amplification of identity-linkage risk [1].
-
Initial acquisition varies substantially across tasks, and base models already recover some identity-related information, making the contribution of target-specific learning ambiguous. ReMem [2] demonstrates how inadequate prior memorization can compromise unlearning evaluation, while SALMUBench [3] provides a relevant precedent through clean and compromised reference models. Extending the existing Vanilla-conditioned analyses and evaluating the retain-only reference described in Appendix D.3 would help distinguish acquired personal knowledge from pre-existing capabilities and evidence-supported inference. These controls are important for substantiating selective removal of learned personal information.
[1] Li, Yifan, et al. "Analyzing and mitigating object hallucination: A training bias perspective." AAAI 2026.
[2] Kwon, JuneHyoung, et al. "Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks." ACL 2026 Findings.
[3] Selvas-Sala, Cai, Lei Kang, and Lluis Gomez. "SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning." CVPR 2026.
Questions for authors
-
Could you evaluate matched negative pairs in a representative risk-amplification setting and report TPR and FPR before and after unlearning? This would clarify whether the observed binding increase reflects improved identity discrimination or response bias.
-
How does the evaluation distinguish acquired personal knowledge from inference supported by the supplied evidence? Could you report results from the retain-only reference described in Appendix D.3?
-
Do the main conclusions persist when Vanilla-conditioned recovery is reported systematically across task families? Please include eligible sample counts and both correct-to-incorrect and incorrect-to-correct transitions to clarify the effects of heterogeneous initial acquisition.
3 comments
(W2, Q3) Vanilla-Conditioned Recovery and Knowledge Transitions
Yes. Our central finding remains that suppressing attribute-level retrieval does not necessarily eliminate other forms of identity recovery. We verify this using Vanilla-conditioned recovery across 96 valid configurations, covering six VLMs, nine unlearning configurations, and both Forget-1 and Forget-5.
(1) Controlling for initial acquisition. We restrict recovery to initially correct predictions and separately report correctness transitions. Results below use Qwen3-VL-4B with MMUN-Lang under Forget-5.
| Task | Vanilla correct | C→C | C→I | I→C | Conditional recovery |
|---|---|---|---|---|---|
| AA | 207 | 94 | 113 | 0 | 45.41% |
| IA | 77 | 54 | 23 | 0 | 70.13% |
| IR | 32 | 30 | 2 | 4 | 93.75% |
| IB+ | 40 | 40 | 0 | 0 | 100.00% |
Conditional recovery counts only C→C among Vanilla-correct probes. MMUN-Lang suppresses 113/207 AA probes, while 30/32 IR probes remain recoverable. The four newly correct IR predictions are excluded, so they do not inflate residual recovery.
(2) Cross-model and systematic validation. The same analysis is extended across models and methods. Each cell reports recovery among Vanilla-correct probes under Forget-5.
| Model / Method | AA | IA | IR |
|---|---|---|---|
| Qwen-2B / GA-Diff | 93/207 (44.93%) | 60/75 (80.00%) | 30/35 (85.71%) |
| Qwen-2B / MMUN-Lang | 83/207 (40.10%) | 44/75 (58.67%) | 19/35 (54.29%) |
| Qwen-4B / GA-Diff | 168/207 (81.16%) | 75/77 (97.40%) | 31/32 (96.88%) |
| Qwen-4B / MMUN-Lang | 94/207 (45.41%) | 54/77 (70.13%) | 30/32 (93.75%) |
| Qwen-4B / MMUN-Vis | 106/207 (51.21%) | 61/77 (79.22%) | 20/32 (62.50%) |
| Qwen-8B / MMUN-Lang | 30/207 (14.49%) | 22/77 (28.57%) | 12/32 (37.50%) |
| Qwen-8B / MMUN-Vis | 40/207 (19.32%) | 17/77 (22.08%) | 12/32 (37.50%) |
| LLaVA-7B / MMUN-Lang | 89/207 (43.00%) | 13/77 (16.88%) | 3/34 (8.82%) |
| LLaVA-13B / MMUN-Lang | 89/207 (43.00%) | 24/75 (32.00%) | 10/32 (31.25%) |
| Llama-11B / GA | 122/207 (58.94%) | 62/62 (100%) | 33/37 (89.19%) |
Across all 96 configurations, 53/96 (55.21%) exhibit at least a 10 percentage-point difference among AA, IA, and IR, including 29/30 (96.67%) MMUN configurations. The difference reaches 20 points in 32/96 and 17/30 configurations, respectively.
The direction varies across models. Qwen-4B/MMUN-Lang retains substantially more IR than AA, whereas LLaVA-7B/MMUN-Lang shows the opposite pattern. These results confirm that direct attribute suppression alone is insufficient to characterize individual-level forgetting, with residual access depending on the model and unlearning method.
Thank you for the additional experiments. The retain-only and Vanilla-conditioned analyses largely address my concern about initial acquisition. But the evidence for increased linkage risk is more limited. It would great to reflect that qualification in the main text. Also, I see the broader evaluation of identity recoverability as a strength, but I still find the “first benchmark” claim too broad. MLLMU-Bench already uses individual profiles as forgetting targets, FIUBench focuses on facial identity, and ReMem probes residual information through indirect multimodal queries. A short related work section in the main paper would help clarify what the benchmark adds and explain the distinction between Profile and Individual in Table 7. Therefore, I will maintain my current score.
We thank the reviewer for recognizing the value of our identity-centric benchmark. We have addressed Yes-response bias, initial knowledge acquisition, and residual recovery after unlearning through additional analyses in Sections 3.3 and 4.3.2 and Appendices E.1 and E.5. All revisions are highlighted in blue in the updated manuscript.
(W1, Q1) Identity Binding with Matched Negative Pairs
Identity-linking risks remain after unlearning, even when matched negative pairs are considered. We extend Identity Binding with 40 matched negative pairs on Qwen3-VL-8B under Forget-5 to distinguish residual identity associations from Yes-response bias.
(1) Positive and negative evaluation.
After MMUN-Lang, 6/40 matched pairs are correctly distinguished, compared with 0/40 for Vanilla. This shows that some identity associations remain distinguishable beyond affirmative-response bias.
These results provide evidence of residual identity-linking capability after unlearning. We will report negative-pair controls alongside positive-pair recovery to distinguish genuine identity associations from response bias.
(W2, Q2) Initial Knowledge Acquisition and Retain-only Control
The retain-only control confirms substantial target-data-related knowledge acquisition, while identity reconstruction remains highly recoverable after unlearning. We evaluate the Base-initialized retain-only reference from Appendix D.3 on Qwen3-VL-4B under Forget-5.
(1) Target knowledge acquisition. We compare Base, Vanilla, and Retain-only on the same target queries.
Retain-only preserves strong general task competence. On retained identities, its AA, IA, and IR scores reach 99.91%, 71.24%, and 68.69%, compared with 100.00%, 72.81%, and 71.92% for Vanilla. The large target-set gaps therefore cannot be explained simply by inadequate reference-model learning.
(2) Acquisition-controlled recovery. We further restrict evaluation to probes correctly answered by Vanilla but incorrectly answered by Retain-only, controlling for baseline inference and initial acquisition differences.
MMUN-Lang suppresses AA to 32.30%, yet IR remains at 90.48%. This includes 8/8 relational and 11/11 relational-attribute reconstruction probes. Thus, the discrepancy between attribute forgetting and identity reconstruction persists even after controlling for initial knowledge acquisition.