- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces UNLINK-VL, a benchmark for evaluating cross-modal knowledge unlearning in vision-language models.
Critical strengths
S1. The paper focuses a practically relevant problem and provide a benchmark to support the future research, as unlearning in one modality does not necessarily prevent knowledge recovery through another.
S2. The benchmark provides a reasonably comprehensive evaluation, covering different VLMs, unlearning methods, training strategies, and knowledge retention.
Critical weaknesses
W1. Closely related benchmarks, particularly UMU-Bench and PPU-Bench, have already investigated cross-modal unlearning and real-world knowledge forgetting. The main contribution appears to be an extension and combination of existing evaluation settings rather than a substantially new perspective. The authors are encouraged to claim the importance and necessity for such extension and combination.
W2. How to make sure the paired text-only and text-image questions test the same knowledge, with the input modality being the only difference. However, this assumption does not always hold. For example, in Table C.3, the text-only question asks about a specific Grammy Award won by Lady Gaga, while the image-text question simply asks what award she received, which could have multiple valid answers. Such inconsistencies may introduce evaluation bias and make it difficult to attribute performance differences solely to cross-modal unlearning.
W3. The paper measures forgetting mainly through answer accuracy, but a lower accuracy does not necessarily indicate that the target knowledge has been forgotten. For image-text queries, the model may fail because it can no longer recognize the entity in the image, rather than because it has forgotten the corresponding facts. Similarly, incorrect answers or refusals may reflect changes in output behavior rather than knowledge removal.
Questions for authors
See weaknesses.
2 comments
R3-W3. Distinguishing knowledge forgetting from visual recognition failure
Reduced image-conditioned accuracy could potentially result from impaired visual entity recognition rather than forgetting of the associated factual knowledge.
To distinguish these possibilities, we conduct an additional entity recognition experiment on all 250 target entities. Specifically, we use the corresponding held-out Image1 images from the image-conditioned factual evaluation as inputs and test whether each checkpoint can still correctly recognize the target entity.
The results are:
| Checkpoint | Image1 Entity Recognition |
|---|---|
| Vanilla | 250/250 (100%) |
| GA-T / GA-M | 250/250 (100%) / 249/250 (99.6%) |
| DPO-T / DPO-M | 250/250 (100%) / 244/250 (97.6%) |
| NPO-T / NPO-M | 248/250 (99.2%) / 250/250 (100%) |
| RT-T / RT-M | 250/250 (100%) / 250/250 (100%) |
The recognition accuracy remains between 97.6% and 100% across all evaluated checkpoints.
These results show that the target entities remain recognizable in the held-out Image1 evaluation images after unlearning. The decline in image-conditioned factual accuracy therefore reflects more than a general failure to recognize the target entities.
We will incorporate this experiment and clarify the distinction between visual entity recognition and factual knowledge retrieval in the revised manuscript.
We sincerely thank Reviewer Wbns for the careful review and constructive suggestions, and especially appreciate the positive assessment of the practical relevance of cross-modal knowledge unlearning and the comprehensive evaluation coverage of UNLINK-VL.
Response to R3-W1.
We acknowledge that these benchmarks have established important foundations for studying cross-modal knowledge unlearning. Our contribution is not to claim an entirely new unlearning paradigm, but to extend the evaluation scope and systematically investigate cross-modal forgetting across diverse real-world visual entity categories.
Specifically, the existing benchmarks primarily focus on person-centric knowledge settings: UMU-Bench mainly investigates synthetic persons, while PPU-Bench focuses on real-world public figures.
However, visual knowledge in VLMs is not limited to associations between human faces and biographical facts. Different entity categories involve fundamentally different visual representations and associated knowledge relations:
Consequently, observations obtained exclusively from person-centric benchmarks cannot establish whether cross-modal forgetting behaviors generalize to other types of visual entities.
UNLINK-VL addresses this gap by extending the evaluation to diverse real-world entities and examining forgetting, multi-hop forgetting, retention, and rewriting across text-only and text-image settings.
This broader coverage allows us to investigate whether directional cross-modal forgetting transfer is a general characteristic of VLM unlearning or is restricted to particular entity categories.
We will clarify this distinction in the revised manuscript and more explicitly position UNLINK-VL as an extension of the evaluation scope and a systematic empirical study, rather than claiming that its individual evaluation components are entirely new.
The paired-question construction raises an important semantic-consistency issue.
Our original data construction was designed to preserve semantic equivalence between text-only and text-image questions. Specifically, Prompt H in Appendix B.3 instructs the model to preserve the original answer, entity relation, queried fact, and reasoning chain while replacing the entity name with an image-grounded reference.
However, we acknowledge that prompt-level constraints alone cannot guarantee strict semantic equivalence for every generated pair.
The example in Table C.3 illustrates this limitation: the image-text question inadvertently drops the specific constraint related to visual media songwriting, making it less restrictive than its text-only counterpart.
To systematically investigate this issue, we conduct an alignment audit over all paired questions. Specifically, we replace the entity name in each text-only question and the corresponding image-grounded entity reference in each text-image question with a common placeholder, [ENTITY], and then perform exact textual matching.
The results are:
This audit confirms that 98.53% of the paired questions have identical textual structures after entity normalization, while 718 pairs contain additional textual differences.
In the revised manuscript, we will replace free-form question rewriting with deterministic entity replacement, preserving the original question structure and changing only the target entity name to an image-grounded reference.
For example:
We will exclude the 718 mismatched pairs and report the updated results in the revised manuscript. The present results still use the original question pairs. Because the mismatches account for 1.47% of all pairs, we do not expect them to overturn the main qualitative direction of transfer; this expectation will be checked in the corrected evaluation. The planned construction change is intended to make the retained pairs differ only in their entity references.