acceptodds
Under review as a conference paper at ICLR 2027

Task-Agnostic Domain Unlearning for Vision-Language Models via Contrastive Editing

Abstract

Vision-Language Models (VLMs) such as CLIP acquire domain-general image–text alignments, allowing the same semantic concept to be accessed across visual domains. This behavior becomes undesirable when a semantic association should be suppressed only in selected domains while remaining usable elsewhere. Existing domain unlearning approaches for CLIP are primarily evaluated through classification behavior and feature-level domain separation, which need not directly control the cross-modal ranking geometry used by both prompt-based classification and retrieval. We introduce Domain-Aware Contrastive Editing (DACE), a framework for approximate domain unlearning that directly edits domain-conditioned image–text relations. DACE preserves alignment on retain domains and applies one of three forgetting operators on forget domains: absolute similarity suppression (DACE-S), mismatch reassignment (DACE-M), or hard-negative margin reversal (DACE-H). We formalize the connection between alignment editing and downstream forgetting through cross-modal ranking margins: zero-shot classification and Recall@ are characterized by order statistics of the same similarity geometry, and the DACE-H hinge objective upper-bounds residual top-1 success on forget samples. This makes the edit task-independent at the level of the shared similarity geometry, with formal guarantees for prompt-based classification and top- retrieval, without implying certified information removal. Across Office-Home, Mini DomainNet, and DomainNet, DACE suppresses forget-domain behavior while preserving retain-domain utility across classification and bidirectional retrieval with both CLIP and SigLIP. Representation and relearning analyses further indicate that behavioral forgetting can result from disrupting cross-modal accessibility without necessarily erasing all class-discriminative visual information.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Machine Unlearning, Vision-Language Models, Domain Unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4FHtOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper extends the classification-centered evaluation of approximate domain unlearning to bidirectional image-text retrieval for more reliable forgetting of domain-specific information. To achieve this, the authors propose domain-aware contrastive editing (DACE). By preserving image-text alignment for retaining domains and modifying it for domains to be forgotten through similarity suppression, mismatch reassignment, and rank reversal. The experimental results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. Novel problem setting -- This paper addresses the novel and important problem that has not been tackled in previous works. This paper extends the domain unlearning based on classification to bidirectional retrieval.

  2. Clear alignment between motivation and method -- This paper is motivated by domain unlearning beyond classification by considering image-text retrieval. The proposed method directly controls the cross-modal alignment, which is effective for both classification and retrieval. These motivation and method align well.

Critical weaknesses
  1. Limited ablation study -- The proposed method employs the caption-based training for retaining domains, but it is unclear whether the baselines use the same training. Comparing the objectives under otherwise identical training settings would help to identify the source of performance gain.
Questions for authors
  1. Can authors provide the ablation study that is mentioned in Weakness 1?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4FHt (Part 1)

We thank the reviewer for the positive assessment of our paper's novelty, soundness, and motivation, particularly the recognition of extending domain unlearning beyond classification to bidirectional retrieval and the alignment between our motivation and proposed method. We also appreciate the constructive feedback.

W1 & Q1: Text supervision and matched comparisons


The source of text supervision depends on the dataset. On DomainNet, paired BLIP-2-generated captions are available and DACE uses the caption associated with each image. On Office-Home and Mini-DomainNet, image-specific captions are unavailable, so DACE uses the standard class-label-derived prompt as the text target. We will make this distinction explicit in the revised manuscript.

  • DomainNet: All methods use the same images and class labels, but they do not use caption supervision in the same way. DACE and CLIPErase directly optimize image-caption alignment, whereas ADU is classification-oriented: its retain and forgetting losses act on class-prompt predictions and do not optimize instance-level caption alignment. We will therefore avoid suggesting that the methods exploit captions identically and this difference is part of the methodological comparison.

  • Office-Home and Mini-DomainNet: These datasets contain no image-specific captions. ADU and DACE are trained with the same images, class labels, and class-prompt targets. The classification results in Table 3 of Appendix B therefore provide a matched-supervision comparison that is not confounded by access to image-specific captions.

Scope of the evidence. Office-Home and Mini-DomainNet evaluate classification only. DomainNet is the dataset on which we evaluate both zero-shot classification and bidirectional image-text retrieval. On DomainNet, all methods follow the same downstream evaluation protocol, while their training objectives use the available supervision differently by design. We will revise the paper to distinguish these two conclusions: the additional datasets establish classification performance under matched text supervision, whereas DomainNet provides the evidence for task-spanning domain unlearning.

Additional controlled comparisons with ADU. We also interchanged the retain, forgetting, and DDL objectives of ADU and DACE under identical DomainNet data, training budgets, and forget-domain configurations. Replacing ADU's retain objective, forgetting operator, or DDL with the corresponding DACE component independently increases the bidirectional-retrieval harmonic mean in the reported comparison. The strongest retrieval results arise from combining DACE's contrastive retain objective with DACE-H forget objective. The effect of retain-forget-aware DDL is operator-dependent: it is most useful for the weaker DACE-S suppression objective and is smaller, or can be negative, for DACE-M and DACE-H. Adding the auxiliary class-prompt retain loss recovers classification performance at a modest retrieval cost. These controlled results support the contribution of the alignment-editing objectives without attributing all gains to caption supervision or to DDL.

AuthorsOpenReview ↗

Response to Reviewer 4FHt (Part 2)

The complete component-wise results on DomainNet, averaged over all possible single forget-domain configurations (∣Df∣|\mathcal{D}_f|=1), are reported below and will be incorporated into the revised manuscript.

DACE ComponentsLr\mathcal{L}_rLf\mathcal{L}_fLDDL\mathcal{L}_{DDL}Cls Mem↑Cls For↑Cls H↑T2I Mem↑T2I For↑T2I H↑I2T Mem↑I2T For↑I2T H↑
BaselineADUADUADU59.9679.1468.111.5398.952.992.0499.173.99
RetainDACEADUADU47.0990.5761.9476.7471.6672.4273.6079.1375.16
ForgetADUDACE-SADU61.7375.1167.4436.0489.5749.8138.8886.8351.70
ForgetADUDACE-MADU59.2989.0671.1631.2090.9044.4529.6395.1342.97
ForgetADUDACE-HADU58.1586.0369.3837.4796.1050.9048.2194.7261.67
DDLADUADUDACE61.4777.4168.2547.5492.5861.5245.8191.8959.55
Retain, ForgetDACEDACE-S–54.9862.4956.8862.3662.3557.4462.1261.4356.78
Retain, ForgetDACEDACE-M–54.6790.2968.0658.8985.6068.1960.5089.3670.72
Retain, ForgetDACEDACE-H–51.5791.5465.8678.8495.6385.2078.5793.4484.48
Retain, Forget, DDLDACEDACE-SDACE56.1167.3760.0561.5668.5261.8361.5565.1959.39
Retain, Forget, DDLDACEDACE-MDACE51.0292.9665.7554.9487.3965.8252.8393.3765.79
Retain, Forget, DDLDACEDACE-HDACE50.0393.3865.0174.8894.2382.1174.9493.5982.34
Retain, Forget, DDLDACE+CPDACE-SDACE63.4582.2671.4758.0086.0267.7557.4584.1866.49
Retain, Forget, DDLDACE+CPDACE-MDACE61.8690.1273.3556.6887.9567.3754.8290.9266.71
Retain, Forget, DDLDACE+CPDACE-HDACE60.0589.4171.8476.3090.5681.8676.9987.8781.34

Dear Reviewer,

Thank you again for your time and for reviewing our paper. We have provided detailed responses to your comments and updated the manuscript accordingly. We would appreciate your feedback on whether our responses and revisions have addressed your concerns or if any points require further clarification.

We would be happy to provide any additional details.

Thank you for your time and consideration.

Reviewer_yPkMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the domain unlearning problem for VLM. Existing methods achieve forgetting solely by lowering classification accuracy, which also damages retrieval ability on the retained domains. The authors therefore propose DACE, which directly edits image-text similarity. it maintains correct pairings in the retained domain and weakens or inverts correct pairings in the forgotten domain. As a result, it achieves a better balance between forgetting and retention on both classification and bidirectional retrieval.

Critical strengths

None

Critical weaknesses

I noticed the authors’ statement at L488: "No generative AI tools were used in the preparation of this manuscript."

However, their statement is very likely inaccurate. The following overly defensive and ambiguous wording is very strange and appears to have been AI-generated without human review.

  1. Introduction, L71: "We do not claim task-agnosticity for arbitrary VLM capabilities such as generation or reasoning."

  2. Contribution, L97-98: "without claiming guarantees for arbitrary functions of the similarity ranking".

  3. Sec 3.2, L238: "The Reproducing Kernel Hilbert Space (RKHS) notation above leaves the particular positive-definite kernel and empirical estimator implementation-dependent; these must be kept identical to the implementation used to produce the reported experiments."

In addition, there are mismatches between the cross-references in the appendix and those in the main text:

  1. At L1488, it should cite Eq. (7) instead: "This is the pairwise-sigmoid counterpart of the cosinesimilarity suppression in Eq. (6) of the main paper."
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer yPkM

We confirm that the statement at L488 is accurate for the manuscript as submitted. We appreciate the opportunity to clarify the passages you identified.

1. L71 and L97–98. We wrote both passages deliberately to clarify the scope of our claims. At L71, we wanted to make explicit that our use of “task-agnostic” applies within the setting studied in the paper. Generation and reasoning illustrate capabilities outside that scope. At L97–98, we wanted to distinguish the guarantee we establish for the similarity ranking from the stronger claim that it would extend to every function of that ranking.

2. L238. This sentence was an internal editing note from the thesis supervisor to the student. It was inadvertently left as ordinary manuscript text, and we overlooked it during proofreading. We will replace it with a direct description of the kernel and empirical estimator used in the reported experiments.

3. L1488. You are correct that the appendix should refer to Eq. (7) of the main paper. We will correct the cross-reference.

In light of these clarifications, we respectfully ask the reviewer to reconsider their recommendation on the scientific merits of the submission. If there are specific concerns about our assumptions, methodology, or results that contributed to your score, we would welcome the opportunity to address them during the discussion.

Dear Reviewer,

Thank you again for your time and for reviewing our paper. We have provided detailed responses to your comments and updated the manuscript accordingly. We would appreciate your feedback on whether our responses and revisions have addressed your concerns or if any points require further clarification.

We would be happy to provide any additional details.

Thank you for your time and consideration.

Reviewer_3D6nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of domain unlearning for vision-language models (e.g., CLIP), which aims to suppress the model's recognition ability in selected domains while preserving it in others. To this end, the authors reformulate domain unlearning as domain-conditioned alignment editing in the shared image–text embedding space, rather than as manipulating classification behavior. They propose the DACE family — three forgetting operators (DACE-S, which directly weakens the similarity of true image–text pairs; DACE-M, which reassigns samples to mismatched targets; and DACE-H, which imposes a margin constraint against the hardest negative) — combined with a retain-side contrastive loss and an optional domain-disentanglement loss. Only lightweight adapters are trained on 8-shot data, with the backbones kept frozen. Experiments on three datasets spanning two task types (classification and retrieval) demonstrate the effectiveness of the proposed methods.

Critical strengths
  1. Machines unlearning is an interesting problem, especailly for LLM/VLMs, as their pre-training and post-training contains massive data, which may contains some sensitive or dangerous information that need to be removed.
  2. The proposed method is straightforward and easy to follow. The DACE framework consists of three losses: a forgetting loss, a retain loss, and a domain-disentanglement loss, all operating on the image–text embedding space.
Critical weaknesses
  1. Incremental modification. The proposed DACE is very similar with previous work, ADU, which also contains a retain loss, a forgetting loss and a domain-disentanglement loss. The differences are: (1) the retain loss in ADU is calculated with a classification CE loss, while DACE calculates as constractive loss; (2) the forgetting loss is DACE contains 3 variations. I suggest the author add a complete comparsion with ADU by ablation each modification, which can significantly increases the novelty of this work.
  2. Generalization ability of propose method. In Table 1, ADU achieves comparable or even better performance than DACE-S/M/H on classification tasks. It appears that DACE-S/M/H excel mainly on retrieval tasks, particularly in the retain domains. Could the authors comment on this discrepancy, or provide additional analysis of when DACE is preferable to ADU?
Questions for authors
  1. I believe that machine unlearning is an interesting and valuable problem for LLMs/VLMs. For example, users may want to remove personal or sensitive information (e.g., API keys or home addresses) from a trained model. However, domain unlearning for CLIP models seems to lack clear real-world applications that would justify its value: it is feasible to train a domain- or application-specific model from scratch, rather than removing domain knowledge from a general-purpose model. Could the authors clarify the practical scenarios in which domain unlearning for CLIP is preferable to training a specialized model?

If the author can address the above weakness and questions, I would like to increase my score.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3D6n (Part 1)

We thank the reviewer for the careful reading of our paper, the positive assessment of its clarity, and the constructive comments. We appreciate the opportunity to clarify the distinction from ADU, explain the classification-retrieval trade-off, and strengthen the practical motivation.

W1: Incremental modification relative to ADU


We agree that both ADU and DACE contain retain, forget, and optional domain-disentanglement components. Such components are common to machine-unlearning methods; the substantive distinction lies in what is edited and how forgetting is defined.

ADU defines forgetting through classification-oriented objectives: cross-entropy on retain domains, entropy maximization on forget domains, and a domain-disentanglement loss acting on feature distributions. DACE instead formulates domain unlearning as domain-conditioned editing of individual image-text relations in CLIP's shared embedding space. Its retain objective preserves paired cross-modal alignment, while its three forgetting operators implement different geometric interventions:

  • DACE-S suppresses the absolute similarity of the true image-text pair.
  • DACE-M reassigns the sample to a mismatched target.
  • DACE-H reverses the relative ranking margin against the hardest negative.

This distinction allows the same edited model to support forgetting in both zero-shot classification and bidirectional retrieval, without introducing a separate forgetting objective for each task. Our theoretical analysis makes this connection explicit: prompt-based classification and Recall@K are threshold events of order statistics of the same similarity geometry, and the DACE-H hinge loss bounds residual top-1 success on forget samples. We have revised the wording to avoid claiming task independence beyond such similarity-based tasks.

We strengthened the empirical comparison in two ways:

1. Existing component ablations. Figure 2 compares DACE without DDL, with ADU's global DDL, and with our retain-forget-aware DDL. Table 2 separately ablates the retain, forget, and DDL components. Notably, DACE-M and DACE-H remain effective without DDL, showing that their forgetting behavior is primarily induced by the proposed alignment-editing operators and DDL acts as an auxiliary regularizer.

2. New controlled ADU-DACE ablation. Following the reviewer's suggestion, we systematically interchange the ADU and DACE retain, forget, and disentanglement objectives under identical data, training, and forget-domain configurations. This isolates the effect of replacing ADU's classification retain objective with contrastive alignment preservation, replacing entropy maximization with each DACE forgetting operator, and replacing global DDL with retain-forget-aware DDL.

The complete component-wise results on DomainNet, averaged over all possible single forget-domain configurations (∣Df∣|\mathcal{D}_f|=1), are reported below and will be incorporated into the revised manuscript.

DACE ComponentsLr\mathcal{L}_rLf\mathcal{L}_fLDDL\mathcal{L}_{DDL}Cls Mem↑Cls For↑Cls H↑T2I Mem↑T2I For↑T2I H↑I2T Mem↑I2T For↑I2T H↑
BaselineADUADUADU59.9679.1468.111.5398.952.992.0499.173.99
RetainDACEADUADU47.0990.5761.9476.7471.6672.4273.6079.1375.16
ForgetADUDACE-SADU61.7375.1167.4436.0489.5749.8138.8886.8351.70
ForgetADUDACE-MADU59.2989.0671.1631.2090.9044.4529.6395.1342.97
ForgetADUDACE-HADU58.1586.0369.3837.4796.1050.9048.2194.7261.67
DDLADUADUDACE61.4777.4168.2547.5492.5861.5245.8191.8959.55
Retain, ForgetDACEDACE-S–54.9862.4956.8862.3662.3557.4462.1261.4356.78
Retain, ForgetDACEDACE-M–54.6790.2968.0658.8985.6068.1960.5089.3670.72
Retain, ForgetDACEDACE-H–51.5791.5465.8678.8495.6385.2078.5793.4484.48
Retain, Forget, DDLDACEDACE-SDACE56.1167.3760.0561.5668.5261.8361.5565.1959.39
Retain, Forget, DDLDACEDACE-MDACE51.0292.9665.7554.9487.3965.8252.8393.3765.79
Retain, Forget, DDLDACEDACE-HDACE50.0393.3865.0174.8894.2382.1174.9493.5982.34
Retain, Forget, DDLDACE+CPDACE-SDACE63.4582.2671.4758.0086.0267.7557.4584.1866.49
Retain, Forget, DDLDACE+CPDACE-MDACE61.8690.1273.3556.6887.9567.3754.8290.9266.71
Retain, Forget, DDLDACE+CPDACE-HDACE60.0589.4171.8476.3090.5681.8676.9987.8781.34
AuthorsOpenReview ↗

Response to Reviewer 3D6n (Part 2)

The new results show that both the contrastive retain objective and the proposed forgetting operators contribute substantial improvements in bidirectional retrieval. Their combination preserves cross-modal utility while producing strong forgetting, whereas DDL provides a smaller, configuration-dependent additional benefit. We will include this controlled comparison and discuss each modification separately in the revised manuscript.

W2: Generalization and classification-retrieval trade-off


The reviewer correctly identifies an important trade-off in Table 1. ADU is optimized directly for classification over a fixed class-prompt vocabulary and is therefore competitive on that task. By contrast, the original DACE retain objective preserves instance-level image-text alignment. This explains why DACE's clearest advantage appears in retrieval, whereas its original classification performance can be lower.

The retrieval results also reveal a limitation of using classification performance alone to assess a general-purpose VLM. Although ADU strongly suppresses forget-domain classification, its retain-domain retrieval performance is nearly zero in Table 1. Thus, successful classification forgetting does not necessarily preserve the model's underlying cross-modal utility.

To determine whether the classification gap is intrinsic to alignment editing, we added two auxiliary retain objectives: Class-Prompt alignment (CP), which uses bidirectional contrastive alignment between retain-domain images and their class prompts; and Cross-Entropy (CE), which performs image-to-text classification over class-prompt similarities.

With either auxiliary objective, DACE-H exceeds ADU's classification harmonic mean for all different number of forget domain setting, using the same configurations in the main paper:

∣Df∣\lvert \mathcal{D}_f \rvertMethodCls Mem↑Cls For↑Cls H↑T2I Mem↑T2I For↑T2I H↑I2T Mem↑I2T For↑I2T H↑
1ADU59.9679.1468.111.5398.952.992.0499.173.99
1DACE-H51.5791.5465.8678.8495.6385.2078.5793.4484.48
1DACE-H + CP60.0589.4171.8476.3090.5681.8676.9987.8781.34
1DACE-H + CE60.6189.2272.1875.8591.0781.6377.4088.5681.96
2ADU59.9781.2268.981.7499.183.412.7499.255.28
2DACE-H41.6491.3056.8976.0195.2383.4573.7594.4682.12
2DACE-H + CP59.1387.8770.5873.9690.9880.3675.3889.4581.06
2DACE-H + CE60.5385.3670.7975.5491.9082.1275.5590.4281.54
3ADU57.4482.1767.612.0899.474.052.9299.285.66
3DACE-H44.6485.6958.7073.2098.2983.3369.7399.0781.22
3DACE-H + CP56.4287.0468.4770.1196.6180.5968.2893.6278.20
3DACE-H + CE57.2085.8868.6659.4896.4271.5168.0294.4578.26
4ADU68.7385.6776.272.9899.505.764.4099.628.38
4DACE-H61.8886.8072.2591.2697.7894.1587.3998.9492.50
4DACE-H + CP68.2688.6977.1484.2595.1488.9384.5995.6789.46
4DACE-H + CE70.5886.6677.8085.1795.3689.5887.0094.9090.48
5ADU61.1891.4772.893.3599.686.386.8699.5812.64
5DACE-H51.7991.8964.8889.7097.5193.1384.0398.9690.11
5DACE-H + CP64.0489.3274.4088.9896.5692.3285.3198.4290.97
5DACE-H + CE66.4787.1675.2486.6897.0891.2583.4997.9889.72

Crucially, these classification gains preserve the large bidirectional-retrieval advantage of DACE. The updated results reflect the completed 50-epoch runs using the same checkpoint-selection criterion. We will add the complete results to Table 1 and analyze the auxiliary retain objectives in the ablation section.

These findings clarify when each method is preferable. ADU is competitive when the sole downstream objective is classification over a predetermined label set. DACE is preferable when the edited VLM must preserve cross-modal retrieval, serve several similarity-based downstream tasks, or remain usable without committing to a fixed class vocabulary. The CP and CE variants show that DACE can achieve stronger classification performance while maintaining its advantages in bidirectional retrieval.

AuthorsOpenReview ↗

Response to Reviewer 3D6n (Part 3)

Q1: Practical motivation for domain unlearning in VLMs


We agree that removing an individual API key, address, or training record concerns instance-level or factual unlearning. This is distinct from the problem studied here. DACE addresses domain-conditioned accessibility: a semantic concept should remain available in retain domains while its association with a designated visual domain is suppressed.

One concrete example arises in autonomous driving, where a model should continue recognizing real vehicles, pedestrians, and traffic signs while avoiding inappropriate responses to their depictions on billboards or advertisements [1]. Related scenarios include suppressing associations with copyrighted, licensed, synthetic, unsafe, or institution-restricted visual domains after deployment, while retaining the same semantic concepts in permitted domains.

Retraining a specialized model does not generally provide an equivalent solution:

1. Scale and access. CLIP was trained on 400 million image-text pairs [2], and modern VLMs require substantial data and computation [3,4]. A deployer may possess pretrained weights without access to the original corpus or training infrastructure.

2. Preserving general-purpose capabilities. A specialized model replaces the broad functionality of the foundation model. Domain unlearning instead updates an existing model while preserving its utility across retain domains.

3. Selective forgetting is not obtained through retain-only training. Because visual domains share semantic classes, a model trained or fine-tuned only on retain domains may continue recognizing forget-domain samples through cross-domain generalization.

To test the last point, we added retain-only adaptation baselines using both full-model and lightweight fine-tuning. Full fine-tuning with few-shot data substantially degrades general utility, while lightweight adaptation preserves retain-domain performance more effectively. However, neither approach produces reliable forgetting: substantial forget-domain accessibility remains in classification and bidirectional retrieval. An explicit domain-conditioned forgetting objective is therefore necessary. We will include the complete numerical results for both retain-only baselines in the revised manuscript.

DACE addresses this setting using only eight samples per class and domain, updating lightweight modules while keeping the pretrained backbones frozen. We emphasize that this constitutes approximate behavioral suppression of domain-conditioned associations, rather than certified deletion of individual training records.

∣Df∣\lvert\mathcal{D}_f\rvertMethodCls Mem↑Cls For↑Cls H↑T2I Mem↑T2I For↑T2I H↑I2T Mem↑I2T For↑I2T H↑
1Fine-tuning (Full)47.5954.2740.8569.2729.2319.3068.8730.2420.24
1Fine-tuning (Lightweight)55.6846.2048.9284.5914.3719.7484.1016.5622.41
1DACE-H51.5791.5465.8678.8495.6385.2078.5793.4484.48
2Fine-tuning (Full)1.1498.312.224.4393.758.133.4694.776.48
2Fine-tuning (Lightweight)55.8245.8849.8785.4715.4723.7684.8717.1326.06
2DACE-H41.6491.3056.8976.0195.2383.4573.7594.4682.12
3Fine-tuning (Full)0.4899.710.960.9099.161.781.0099.291.97
3Fine-tuning (Lightweight)52.6642.0746.7780.069.6116.0578.8410.9918.21
3DACE-H44.6485.6958.7073.2098.2983.3369.7399.0781.22
4Fine-tuning (Full)71.7452.5060.6395.0222.1434.6195.2223.5936.76
4Fine-tuning (Lightweight)71.1654.1061.4796.5222.1234.9896.3324.3737.94
4DACE-H61.8886.8072.2591.2697.7894.1587.3998.9492.50
5Fine-tuning (Full)5.6296.438.6515.2392.8716.9513.7493.2214.99
5Fine-tuning (Lightweight)64.6746.4953.9694.9920.6532.7194.5522.2034.81
5DACE-H51.7991.8964.8889.7097.5193.1384.0398.9690.11

References:

[1] Kawamura et al., "Approximate domain unlearning for vision-language models," Advances in Neural Information Processing Systems (NeurIPS), 2025.

[2] Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” International Conference on Machine Learning (ICML), 2021.

[3] Schuhmann et al., “LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models,” Advances in Neural Information Processing Systems (NeurIPS), 2022.

[4] Cherti et al., “Reproducible Scaling Laws for Contrastive Language-Image Learning,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

Summary of Revisions and Additional Evidence


We thank the reviewers for the constructive feedback. In response, we made the following revisions and added the corresponding experimental evidence to the manuscript.

1. Scope and interpretation of DACE. We clarified that “task-agnostic” refers to editing the shared image-text similarity geometry used by zero-shot classification and bidirectional retrieval. It does not claim coverage of generation or reasoning tasks. We also stated explicitly that DACE provides approximate behavioral suppression of domain-conditioned associations, rather than certified deletion of individual training records.

2. Text supervision and evaluation scope. We documented that DomainNet uses paired BLIP-2-generated captions and supports both classification and bidirectional retrieval, whereas Office-Home and MiniDomainNet use class-label-derived prompts and are evaluated for classification only. The latter datasets therefore provide matched-supervision classification comparisons with ADU, while DomainNet supplies the task-spanning evidence.

3. Controlled component-wise comparison with ADU. Under identical DomainNet data, training budgets, and forget-domain configurations, we interchange ADU's retain, forgetting, and DDL objectives with the corresponding DACE components (Table 3). The original ADU “Vision + text” updates both encoders, while the “vision only” variant updates the vision encoder, matching our methods and distinguishing the ADU loss comparison from the matched encoder-update comparison. Each replacement increases the bidirectional-retrieval harmonic mean relative to the ADU baseline in the reported comparison, and combining DACE's contrastive retain objective with DACE-H produces the strongest retrieval results. The effect of retain-forget-aware DDL is operator-dependent: it is most useful for DACE-S and smaller, or negative, for DACE-M and DACE-H.

4. Auxiliary classification-oriented retain objectives. We reported the class-prompt (CP) and cross-entropy (CE) variants, as well as their combination (CP+CE), in Tables 1-3 and provided their formulations in Appendix B. On DomainNet, both variants improve DACE-H classification and exceed ADU in classification harmonic mean for all five evaluated forget-domain cardinalities, while preserving a large bidirectional-retrieval advantage. We discussed the associated classification-retrieval trade-off rather than presenting these variants as uniformly dominant.

5. Standardized experimental updates. We reran the DomainNet DACE-H + CP and DACE-H + CE variants with a common 50-epoch budget and the MiniDomainNet variants with a common 150-epoch budget, using the same checkpoint-selection protocol. We updated Tables 1-3 in the main paper and Appendices B, F and N accordingly. These changes improve comparability without altering the method, data, or evaluation protocol.

6. Retain-only baselines and reproducibility. We added full-model and lightweight adaptation baselines trained only with DACE's retain objective, with the results reported in Table 5 of Appendix D. Neither produces reliable selective forgetting, supporting the need for an explicit forget objective.

7. AI use and reproducibility statement. We updated the AI Use Statement to clarify that generative AI tools were used only to improve writing, grammar, clarity, and table formatting. All research ideas, methods, code, experiments, analyses, and reported numerical results were developed and produced by the authors. We also added a Reproducibility Statement indicating where readers can find the methodological details, theoretical analysis, experimental settings, hyperparameters, and additional results in the paper and appendices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.