acceptodds
Under review as a conference paper at ICLR 2027

UNLINK-VL: Evaluating Cross-Modal Transfer of Knowledge Unlearning in Vision-Language Models

Abstract

Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpora. Removing such knowledge is essential for building trustworthy AI systems. However, existing studies primarily focus on forgetting within individual modalities. Although recent work has begun to explore cross-modal consistency in unlearning, the cross-modal transfer of real-world knowledge unlearning remains insufficiently studied. To address this gap, we introduce UNLINK-VL, a real-world benchmark for cross-modal knowledge unlearning in VLMs. Under a post-hoc unlearning setting in which the original forget and retain corpora are unavailable, UNLINK-VL selects visually identifiable real-world entities as unlearning targets and associates them with corresponding images and one-hop and multi-hop facts derived from Wikidata. The benchmark comprises four complementary subsets that evaluate direct forgetting of target knowledge, the propagation of forgetting through relational knowledge, the preservation of related non-target knowledge, and robustness to semantically equivalent queries. We train models under text-only and text-image unlearning settings and evaluate forgetting effectiveness and retained utility across text-only and text-image scenarios. Extensive experiments reveal a pronounced asymmetry in cross-modal transfer: text-image unlearning remains effective under text-only evaluation, whereas text-only unlearning transfers poorly to text-image scenarios. Meanwhile, the evaluated methods largely preserve the models’ general capabilities. These findings demonstrate that relying solely on intra-modal evaluation, particularly text-only evaluation, may substantially overestimate the effectiveness of knowledge unlearning in VLMs, underscoring the need for cross-modal unlearning and evaluation.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Machine Unlearning, Cross-Modal Unlearning, Multimodal Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WbnsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UNLINK-VL, a benchmark for evaluating cross-modal knowledge unlearning in vision-language models.

Critical strengths

S1. The paper focuses a practically relevant problem and provide a benchmark to support the future research, as unlearning in one modality does not necessarily prevent knowledge recovery through another.

S2. The benchmark provides a reasonably comprehensive evaluation, covering different VLMs, unlearning methods, training strategies, and knowledge retention.

Critical weaknesses

W1. Closely related benchmarks, particularly UMU-Bench and PPU-Bench, have already investigated cross-modal unlearning and real-world knowledge forgetting. The main contribution appears to be an extension and combination of existing evaluation settings rather than a substantially new perspective. The authors are encouraged to claim the importance and necessity for such extension and combination.

W2. How to make sure the paired text-only and text-image questions test the same knowledge, with the input modality being the only difference. However, this assumption does not always hold. For example, in Table C.3, the text-only question asks about a specific Grammy Award won by Lady Gaga, while the image-text question simply asks what award she received, which could have multiple valid answers. Such inconsistencies may introduce evaluation bias and make it difficult to attribute performance differences solely to cross-modal unlearning.

W3. The paper measures forgetting mainly through answer accuracy, but a lower accuracy does not necessarily indicate that the target knowledge has been forgotten. For image-text queries, the model may fail because it can no longer recognize the entity in the image, rather than because it has forgotten the corresponding facts. Similarly, incorrect answers or refusals may reflect changes in output behavior rather than knowledge removal.

Questions for authors

See weaknesses.

2 comments

We sincerely thank Reviewer Wbns for the careful review and constructive suggestions, and especially appreciate the positive assessment of the practical relevance of cross-modal knowledge unlearning and the comprehensive evaluation coverage of UNLINK-VL.

R3-W1. Novelty and relationship to UMU-Bench / PPU-Bench

Response to R3-W1.

We acknowledge that these benchmarks have established important foundations for studying cross-modal knowledge unlearning. Our contribution is not to claim an entirely new unlearning paradigm, but to extend the evaluation scope and systematically investigate cross-modal forgetting across diverse real-world visual entity categories.

Specifically, the existing benchmarks primarily focus on person-centric knowledge settings: UMU-Bench mainly investigates synthetic persons, while PPU-Bench focuses on real-world public figures.

However, visual knowledge in VLMs is not limited to associations between human faces and biographical facts. Different entity categories involve fundamentally different visual representations and associated knowledge relations:

  • People: facial appearances and personal facts.
  • Landmarks: architectural appearances and geographical facts.
  • Organizations: visual logos and organizational attributes.
  • Cultural works: visual covers and associated creators or publication information.

Consequently, observations obtained exclusively from person-centric benchmarks cannot establish whether cross-modal forgetting behaviors generalize to other types of visual entities.

UNLINK-VL addresses this gap by extending the evaluation to diverse real-world entities and examining forgetting, multi-hop forgetting, retention, and rewriting across text-only and text-image settings.

This broader coverage allows us to investigate whether directional cross-modal forgetting transfer is a general characteristic of VLM unlearning or is restricted to particular entity categories.

We will clarify this distinction in the revised manuscript and more explicitly position UNLINK-VL as an extension of the evaluation scope and a systematic empirical study, rather than claiming that its individual evaluation components are entirely new.

R3-W2. Semantic consistency between paired questions

The paired-question construction raises an important semantic-consistency issue.

Our original data construction was designed to preserve semantic equivalence between text-only and text-image questions. Specifically, Prompt H in Appendix B.3 instructs the model to preserve the original answer, entity relation, queried fact, and reasoning chain while replacing the entity name with an image-grounded reference.

However, we acknowledge that prompt-level constraints alone cannot guarantee strict semantic equivalence for every generated pair.

The example in Table C.3 illustrates this limitation: the image-text question inadvertently drops the specific constraint related to visual media songwriting, making it less restrictive than its text-only counterpart.

To systematically investigate this issue, we conduct an alignment audit over all paired questions. Specifically, we replace the entity name in each text-only question and the corresponding image-grounded entity reference in each text-image question with a common placeholder, [ENTITY], and then perform exact textual matching.

The results are:

Alignment StatusNumber of PairsPercentage
Total pairs48,811100%
Exact matches48,09398.53%
Mismatches7181.47%

This audit confirms that 98.53% of the paired questions have identical textual structures after entity normalization, while 718 pairs contain additional textual differences.

In the revised manuscript, we will replace free-form question rewriting with deterministic entity replacement, preserving the original question structure and changing only the target entity name to an image-grounded reference.

For example:

  • Text-only: "Which visual media songwriting work earned Lady Gaga a Grammy Award?"
  • Text-image: "Which visual media songwriting work earned the person in the image a Grammy Award?"

We will exclude the 718 mismatched pairs and report the updated results in the revised manuscript. The present results still use the original question pairs. Because the mismatches account for 1.47% of all pairs, we do not expect them to overturn the main qualitative direction of transfer; this expectation will be checked in the corrected evaluation. The planned construction change is intended to make the retained pairs differ only in their entity references.

R3-W3. Distinguishing knowledge forgetting from visual recognition failure

Reduced image-conditioned accuracy could potentially result from impaired visual entity recognition rather than forgetting of the associated factual knowledge.

To distinguish these possibilities, we conduct an additional entity recognition experiment on all 250 target entities. Specifically, we use the corresponding held-out Image1 images from the image-conditioned factual evaluation as inputs and test whether each checkpoint can still correctly recognize the target entity.

The results are:

CheckpointImage1 Entity Recognition
Vanilla250/250 (100%)
GA-T / GA-M250/250 (100%) / 249/250 (99.6%)
DPO-T / DPO-M250/250 (100%) / 244/250 (97.6%)
NPO-T / NPO-M248/250 (99.2%) / 250/250 (100%)
RT-T / RT-M250/250 (100%) / 250/250 (100%)

The recognition accuracy remains between 97.6% and 100% across all evaluated checkpoints.

These results show that the target entities remain recognizable in the held-out Image1 evaluation images after unlearning. The decline in image-conditioned factual accuracy therefore reflects more than a general failure to recognize the target entities.

We will incorporate this experiment and clarify the distinction between visual entity recognition and factual knowledge retrieval in the revised manuscript.

Reviewer_NF49OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UNLINK-VL benchmarks cross-modal knowledge unlearning in vision-language models using real-world entities and Wikidata facts. It contains 250 forget entities, 500 retain entities, and 97,622 paired text-only and image-text QA instances covering direct forgetting, two-hop knowledge, paraphrase robustness, and neighboring-knowledge retention. The authors evaluate GA, DPO, NPO, and refusal tuning under independently varied training and evaluation modalities, with full-parameter and LoRA updates. Experiments on Qwen3-VL and Llama-3.2-Vision report asymmetric transfer, with multimodal unlearning transferring better to text-only queries than the reverse direction. General-capability evaluations suggest that Qwen3-VL-8B LoRA variants largely preserve broad utility, although forgetting remains incomplete and some capabilities decline.

Critical strengths

1、Fact-aligned text and image-text probes independently vary training and evaluation modalities, exposing cross-modal retrieval failures that evaluation restricted to the training modality can miss. 2、The benchmark combines real-world, model-recognized entities with direct, relational, paraphrased, and retention probes, enabling a broader assessment of targeted forgetting and collateral knowledge loss.

Critical weaknesses

1、The evaluation protocol leaves answer matching and the timing of incorrect-answer/refusal filtering unspecified, making accuracy denominators and the treatment of successful refusals difficult to verify. 2、The claimed universal multimodal advantage in intra-modal forgetting relies on raw accuracies despite unequal baselines; DPO LoRA shows the opposite ordering under the paper's NFD metric. 3、The claim that LoRA better preserves broad utility lacks a full-parameter counterpart in Table 4, which reports only Qwen3-VL-8B LoRA variants across general-capability benchmarks. 4、RT is labeled the best forgetting-retention balance without a defined trade-off criterion or matched-retention comparison; intermediate Forget and Retain scores alone do not establish optimality.

Questions for authors

1、What exact rule scores generated answers, including aliases, partially correct answers, and refusals? At what stage are incorrect/refusal instances excluded, and are identical fact IDs and denominators retained across models, modalities, and runs? 2、How should the universal intra-modal advantage claim be revised after baseline normalization? For Qwen3-VL-8B DPO LoRA, NFD is 18.77% for M→M and 19.51% for T→T, reversing the raw-accuracy ordering. 3、What explicit criterion makes RT the best forgetting-retention balance? Does RT remain preferable under a fixed Retain threshold or a clearly specified weighting of forgetting and retention?

4 comments

C2 / Q2. Baseline-normalized intra-modal forgetting

Our original claim regarding the universal advantage of multimodal training in intra-modal forgetting was too strong.

We agree that raw accuracy comparisons are insufficient when the vanilla models have different baseline accuracies.

In particular, for Qwen3-VL-8B with DPO LoRA, the NFD values are: NFDM→M=18.77NFD_{M\rightarrow M}=18.77\\%,

NFDT→T=19.51NFD_{T\rightarrow T}=19.51\\%.

This result indeed reverses the ordering suggested by the corresponding raw accuracy comparison.

We therefore acknowledge that the original claim of a universal multimodal advantage in intra-modal forgetting is not supported by all experimental settings.

Importantly, our additional Cross-modal Forgetting Transfer Ratio analysis addresses a different question: how much of the normalized intra-modal forgetting transfers to the other modality. As reported in our response to R1, the directional transfer asymmetry remains consistent across all three evaluated backbones.

We will revise the manuscript to distinguish these two observations and remove the overly broad claim of universal multimodal superiority in intra-modal forgetting.

R2-C3. LoRA versus full-parameter fine-tuning

To address this concern, we evaluate full-parameter unlearning with all four methods under both text-only and multimodal training settings on the same ten general-capability benchmarks.

The additional results show that full-parameter unlearning maintains relatively stable performance across most evaluated benchmarks. Specifically, the eight full-parameter configurations in the table below achieve MME scores of 1722.04–1745.71, OCRBench scores of 89.19–93.15, MIA-Bench scores of 91.71–97.26, and POPE scores of 85.89–90.59. Although performance varies on certain benchmarks, these rows do not indicate uniform capability degradation.

Together with our original LoRA evaluation, these findings suggest that both LoRA and full-parameter fine-tuning largely preserve general model capabilities after unlearning, although the degree of preservation depends on the unlearning objective and evaluation benchmark.

We will incorporate the full-parameter results into the revised manuscript and clarify that both tuning strategies can maintain broad model utility, rather than claiming that LoRA is universally superior in general-capability preservation.

ModalityMethodMME↑MMBench↑SEED↑OCRBench↑ScienceQA↑MIA-Bench↑MathVista↑MathVision↑POPE↑HallusionBench↑
–Vanilla1734.7775.4367.2490.9074.2793.8378.9049.0188.4473.50
TGA (Full)1733.2973.4264.6993.1571.0592.6176.1950.5186.1673.93
TDPO (Full)1734.8172.4363.8989.4172.9291.8678.5650.9288.9176.05
TNPO (Full)1737.1476.6863.1292.5368.3395.2179.7251.2287.7075.85
TRT (Full)1727.2477.4566.7291.2973.4196.1780.2850.6788.4371.91
MGA (Full)1722.0472.6465.1992.0165.8095.9279.7449.6185.8974.91
MDPO (Full)1745.7175.7067.3692.6078.4792.0979.0549.4490.2471.68
MNPO (Full)1739.4477.7665.9989.1968.4991.7176.9151.9490.5972.20
MRT (Full)1744.8774.1467.7391.7975.8897.2677.0849.6886.0071.68

R2-C4 / Q3. RT forgetting-retention trade-off

We agree that intermediate Forget and Retain scores alone are insufficient to establish an optimal trade-off. To address this concern, we conduct two complementary analyses using the results reported in Tables 3 and 5: (1) Pareto frontier analysis and (2) explicitly weighted forgetting–retention evaluation.

Our analysis covers 20 matched configurations across three backbones (Qwen3-VL-8B, Llama-3.2-Vision-11B, and Qwen3-VL-32B), two parameter-update strategies (Full and LoRA), and four training–evaluation modality settings.

1. Pareto Frontier Analysis

We consider two objectives: minimizing Forget accuracy ((F↓F\downarrow)) and maximizing Retain accuracy ((R↑R\uparrow)).

Method (AA) Pareto-dominates method (BB) if: FA≤FB,RA≥RB,F_A\leq F_B,\qquad R_A\geq R_B, with at least one strict inequality. A method is Pareto-efficient if no other evaluated method dominates it within the same configuration.

To examine RT's relative position, we calculate six pairwise differences: ΔF1=FGA−FRT,ΔR1=RRT−RGA,ΔF2=FRT−FNPO,ΔR2=RRT−RNPO,ΔF3=FDPO−FRT,ΔR3=RDPO−RRT.\begin{aligned} \Delta F_1 &= F_{GA}-F_{RT}, & \Delta R_1 &= R_{RT}-R_{GA},\\\\ \Delta F_2 &= F_{RT}-F_{NPO}, & \Delta R_2 &= R_{RT}-R_{NPO},\\\\ \Delta F_3 &= F_{DPO}-F_{RT}, & \Delta R_3 &= R_{DPO}-R_{RT}. \end{aligned} Positive ((ΔF1,ΔR1)(\Delta F_1,\Delta R_1)) indicates that RT dominates GA. Positive ((ΔF2,ΔR2)(\Delta F_2,\Delta R_2)) indicates that NPO achieves stronger forgetting while RT preserves more neighboring knowledge. Similarly, positive ((ΔF3,ΔR3)(\Delta F_3,\Delta R_3)) indicates that RT achieves stronger forgetting while DPO preserves more neighboring knowledge.

Results across all 20 configurations.

All differences are reported in accuracy percentage points (pp).

BackboneTuningSettingΔF1\Delta F_{1}ΔR1\Delta R_{1}ΔF2\Delta F_{2}ΔR2\Delta R_{2}ΔF3\Delta F_{3}ΔR3\Delta R_{3}
Qwen3-VL-8BFullT→M1.67.91.62.83.21.2
Qwen3-VL-8BFullM→M3.68.43.63.87.20.9
Qwen3-VL-8BFullM→T2.57.41.42.64.11.4
Qwen3-VL-8BFullT→T2.26.93.12.76.01.5
Qwen3-VL-8BLoRAT→M2.24.81.42.12.81.1
Qwen3-VL-8BLoRAM→M4.85.62.12.56.61.7
Qwen3-VL-8BLoRAM→T2.74.61.61.73.80.9
Qwen3-VL-8BLoRAT→T3.44.01.81.64.71.7
Llama-3.2-Vision-11BFullT→M1.67.81.42.72.91.0
Llama-3.2-Vision-11BFullM→M2.98.33.53.86.60.9
Llama-3.2-Vision-11BFullM→T2.27.11.72.64.11.5
Llama-3.2-Vision-11BFullT→T2.37.02.92.95.81.4
Llama-3.2-Vision-11BLoRAT→M2.24.31.21.82.70.9
Llama-3.2-Vision-11BLoRAM→M4.55.22.32.36.41.3
Llama-3.2-Vision-11BLoRAM→T2.74.01.61.74.00.8
Llama-3.2-Vision-11BLoRAT→T3.43.91.61.54.71.4
Qwen3-VL-32BLoRAT→M2.14.91.42.12.61.0
Qwen3-VL-32BLoRAM→M4.45.82.12.56.31.6
Qwen3-VL-32BLoRAM→T2.24.81.61.73.51.0
Qwen3-VL-32BLoRAT→T2.84.31.91.74.11.7

All six differences are strictly positive across all 20 configurations.

This establishes a consistent relationship: FNPO<FRT<FDPOF_{NPO} < F_{RT} < F_{DPO}

RNPO<RRT<RDPOR_{NPO} < R_{RT} < R_{DPO}

while RT simultaneously achieves lower Forget accuracy and higher Retain accuracy than GA.

Consequently, RT Pareto-dominates GA, whereas RT, NPO, and DPO are mutually non-dominated.

The overall Pareto-efficiency statistics are:

MethodPareto-EfficientDominated
GA0/2020/20
DPO20/200/20
NPO20/200/20
RT20/200/20

Thus, the Pareto frontier is consistent across all evaluated configurations: P=NPO,RT,DPO\mathcal{P}=\\{NPO,RT,DPO\\} These results demonstrate that RT consistently provides a non-dominated compromise between forgetting effectiveness and neighboring-knowledge preservation, although Pareto efficiency alone does not establish a uniquely optimal method.

2. Weighted Forgetting–Retention Evaluation

To further quantify the trade-off under an explicit preference criterion, we define a weighted score: λ(F0−Fm)−(1−λ)(R0−Rm)\lambda(F_0-F_m)- (1-\lambda)(R_0-R_m) where (F0F_0) and (R0R_0) denote the Vanilla model's Forget and Retain accuracies, respectively, and (FmF_m) and (RmR_m) denote the corresponding results after applying unlearning method (m).

The first term rewards stronger forgetting relative to Vanilla, while the second penalizes degradation in neighboring-knowledge retention. Both terms are measured in accuracy percentage points.

The parameter (λ∈[0,1]\lambda\in[0,1]) controls their relative importance: larger values emphasize forgetting, whereas smaller values emphasize retention. Higher scores indicate a more favorable trade-off under the specified weighting.

We first consider (λ=0.4\lambda=0.4), assigning 40% weight to forgetting and 60% to retention: 0.4(F0−Fm)−0.6(R0−Rm)0.4(F_0-F_m)-0.6(R_0-R_m) Illustrative calculation.

Consider Qwen3-VL-8B with LoRA under the M→M configuration:

MethodForget ↓Retain ↑
Vanilla82.680.8
GA65.370.6
DPO67.177.9
NPO58.473.7
RT60.576.2

The weighted scores are:

S0.4(GA)=0.4(82.6−65.3)−0.6(80.8−70.6)=0.80,S0.4(DPO)=0.4(82.6−67.1)−0.6(80.8−77.9)=4.46,S0.4(NPO)=0.4(82.6−58.4)−0.6(80.8−73.7)=5.42,S0.4(RT)=0.4(82.6−60.5)−0.6(80.8−76.2)=6.08.\begin{aligned} S_{0.4}(\mathrm{GA}) &=0.4(82.6-65.3)-0.6(80.8-70.6)\\\\ &=0.80,\\\\[6pt] S_{0.4}(\mathrm{DPO}) &=0.4(82.6-67.1)-0.6(80.8-77.9)\\\\ &=4.46,\\\\[6pt] S_{0.4}(\mathrm{NPO}) &=0.4(82.6-58.4)-0.6(80.8-73.7)\\\\ &=5.42,\\\\[6pt] S_{0.4}(\mathrm{RT}) &=0.4(82.6-60.5)-0.6(80.8-76.2)\\\\ &=6.08. \end{aligned}

Therefore,

S0.4(RT)>S0.4(NPO)>S0.4(DPO)>S0.4(GA)S_{0.4}(\mathrm{RT}) > S_{0.4}(\mathrm{NPO}) > S_{0.4}(\mathrm{DPO}) > S_{0.4}(\mathrm{GA})

RT achieves the highest score, exceeding the second-best method, NPO, by 0.66 points.

Reviewer_MauBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UNLINK-VL, a benchmark for evaluating cross-modal knowledge unlearning in pretrained vision-language models. It pairs real-world entities with Wikidata facts and constructs aligned text-only and image-text queries across Forget, Retain, Multi-Hop, and Rewrite subsets. Experiments independently vary unlearning and evaluation modalities, where the results show incomplete cross-modal forgetting and a consistent directional asymmetry, alongside trade-offs between forgetting strength and neighboring-knowledge preservation.

Critical strengths
  1. UNLINK-VL studies pre-existing real-world knowledge and independently varies training and evaluation modalities. Keeping the queried relation and answer fixed while changing entity access from a name to an image provides a useful framework for testing whether apparent forgetting survives a change in input modality.
  2. The study covers two model families, multiple model sizes, four objectives, and both full-parameter and LoRA updates. The recurring gap between intra-modal and cross-modal forgetting provides substantive evidence that evaluation in the unlearning modality alone can overestimate suppression.
Critical weaknesses
  1. The transfer penalty subtracts intra-modal accuracy from cross-modal accuracy, but the two directions start from different intra-modal forgetting levels and different evaluation baselines. Since multimodal training already achieves stronger intra-modal forgetting, comparing the raw penalties does not cleanly isolate directional transferability.
  2. The benchmark presents multi-hop performance as evidence about forgetting through relational composition. However, the training-data construction also incorporates two-hop knowledge when available, and the paper does not clearly separate directly supervised two-hop content from paths reserved for testing propagation. Moreover, a valid Wikidata path does not mean its verbalized question requires composing both edges. Consequently, Multi-Hop accuracy is less clearly interpretable as relational propagation than as suppression of another collection of target-associated facts.
Questions for authors
  1. Does the directional asymmetry remain when transfer is measured as the fraction of intra-modal, baseline-normalized forgetting preserved in the other modality, rather than as a raw accuracy difference?
  2. What overlap exists between the two-hop paths incorporated into training passages and those used in the Multi-Hop Set?
  3. Because multimodal queries replace the entity name with an image, have you examined target-entity recognition after unlearning?
  4. For M-trained models, does lower M→M accuracy reflect forgotten facts or impaired visual entity recognition?
1 comment

Accordingly, we will revise our original statement from:

"RT provides the best forgetting-retention balance."

to:

"RT consistently achieves a Pareto-efficient forgetting–retention trade-off across all 20 evaluated configurations and attains the highest weighted score under moderate forgetting–retention preferences."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.