acceptodds
Under review as a conference paper at ICLR 2027

Forget at Sight: Training-Free Domain Unlearning in Vision-Language Models

Abstract

Vision-language models such as CLIP are designed to recognize the same object across different visual domains. This makes domain unlearning difficult: an intervention that suppresses recognition in one domain can also damage the same classes in other domains. Existing approaches rely on iterative optimization and access to retain-domain images, limiting their use when model updates are costly or retain data are unavailable. We introduce GATE-DU (Gated Activation Steering for Targeted Erasure in Domain Unlearning), a training-free framework that separates when to erase from how to erase. For each input, a domain gate first estimates whether it belongs to the forget domain. Only then does GATE-DU contract its representation toward a class-agnostic forget-domain centroid, suppressing class recognition while leaving retain-domain inputs unchanged. GATE-DU obtains its gating evidence in three ways. When retain images are unavailable, GATE-DU (T) compares the forget-domain description against known retain-domain descriptions using the frozen text branch. GATE-DU (O) starts from this text gate and progressively grounds it in visual evidence from the unlabeled evaluation stream, first through a class-paired prototype and then through domain-LDA. When labeled retain images are available, GATE-DU (V) directly constructs a class-paired domain-LDA gate and calibrates it using a closed-form Gaussian posterior. All variants keep the model, prompts, and erasure payload frozen, requiring no gradients, adapters, or replay. Experiments on PACS, OfficeHome, and Mini-DomainNet across class fractions \(\gamma\in{0.25,0.5,0.75,1.0}\) show that GATE-DU is competitive with or outperforms optimization-based methods, particularly in low-\(\gamma\) open-vocabulary settings. Our results highlight conditional intervention as a practical principle for domain unlearning: effective erasure requires deciding not only how to forget, but also when to forget.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Domain unlearning, Conditional Activation Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8qP7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates domain forgetting in hosted CLIP services, aiming to suppress the recognition of classes in a specified visual domain (e.g., painting) while preserving the recognition of those classes in other domains. It also considers an open-vocabulary setting where only a small subset of classes are observed during forgetting.

With both the encoder and cue frozen, the method uses a domain gate gg to control the fixed intervention, shifting activation toward the forgetting support centroid: h~=h+λg(μF−h)\tilde{h}=h+\lambda g(\mu_F-h). Three variants construct the gate from different sources of evidence: T relies on class-conditional text without preserving images; O also uses prototype and domain LDA evidence from the unlabeled stream, making each prediction before updating its statistics; V derives a domain-LDA/Gaussian gate from labeled forgetting and retention support. Experiments evaluate CLIP ViT-B/16 on OfficeHome, Mini-DomainNet, and PACS, using eight support images for each selected category.

Critical strengths

The support settings and achievable service level objectives are clearly defined. The data requirements for the three variants are clearly differentiated: T does not use preserved images, O includes an unlabeled data stream, and V uses labeled support based on optimization available from competitors (page 8, lines 390–399). The service level objectives make selective suppression a reasonable design choice, as the task does not require the permanent removal of information from the model parameters (page 1, lines 44–50). The online protocol also includes several useful safeguards: prediction precedes statistical updates, and admission decisions rely on a fixed text teacher (page 14, A.2.4–A.2.5). Performance for both full-stream and late-stream streams is reported in Table 3, helping to reveal the adaptive reduction in accuracy (page 9).

Performance is good when forgetting learning observes only a small subset of classes. At γ=0.25 and γ=0.50, V outperforms the reported strongest optimization baseline on all three datasets. When γ=0.25, HM achieves an improvement of 15.67 percentage points on the OfficeHome dataset, 3.89 percentage points on the Mini-DomainNet dataset, and 44.68 percentage points on the PACS dataset (page 7, Table 1). Therefore, this advantage is present on all datasets, although the largest improvement occurs on the PACS dataset, which is described as a small-sample stress test (page 8, rows 382-389). The single-run evaluation in Figure 1 and the ViT-L/14 experiments in Tables 5-6 provide further evidence under different support budgets and backbone sizes. This advantage becomes less stable as the proportion of categories increases: TMS performs better on the Mini-DomainNet dataset at γ=0.75 and 1.0, while OfficeHome performs similarly at γ=0.75 (V = 71.98, TMS = 71.75), but slightly worse at γ=1.0 (V = 74.83 vs. 75.42). Variability estimation will help determine whether these narrow gaps support meaningful rankings (Page 7, Table 1; Page 9, Figure 1; Page 18, Tables 5–6).

Critical weaknesses

1: Inconsistent reporting and incomplete standardization limited the reproducibility of the results. At γ=0.50, the DomainNet T results showed significant differences between the two tables: Table 1 listed F/R/HM as 8.45/62.55/74.32%, while Table 7 showed average ViT-B/16 as 3.37/47.90/63.71%. These correspond to differences of 5.08, 14.65, and 10.61 percentage points, respectively (pages 7 and 18). Both tables mention a frozen ViT-B/16 model, 8 support images per selected class, evaluation across all classes, and λ=0.5, but they do not specify whether the extraction, calibration, and implementation of the support images are identical (page 8, §4.1; page 13, A.2.1–A.2.3; page 17, Table 4). Another HM aggregation rule fails to account for the differences between F and R. Therefore, this paper needs to clarify the configuration behind each table and explain whether direct comparisons are valid. This issue only applies to T; it does not negate the performance reported by V.

Furthermore, the intervention is not adequately described at the tensor level. Equation (11) operates on the activation vector, but A.2.1 constructs gating features by averaging the residual labels. To reproduce this operation, the intervention layer, the affected label locations, and the rule for mapping the updated vector back to the label tensor are required (page 7, lines 363–373; page 13, lines 673–681). More details are needed regarding the prompt template and η_prompt, scatter/covariance scaling and regularization, candidate blocks of V, and numerical thresholds for controlling LDA shading and submission decisions (pp. 4–6, Equations (1)–(9); pp. 15–17, A.2.7–A.2.11). The use of “multiple independent runs” in the experiments should also include the number of repetitions and their variability (p. 8, line 408).

2: The current comparisons fail to adequately separate gating, intervention, and retention effects. Keep gating unchanged and simplify the intervention. For inputs with existing gating choices, replace activation contraction with fixed or random, class-independent outputs while keeping the data protocol unchanged. Comparing F/R/HM can show how much the payload contributes beyond identifying the input to be suppressed. If performance remains similar, gating may still have some effect, but the evidence for the contraction mechanism will be weaker.

Keep the intervention unchanged and simplify the gating. Suitable controls include: merge regularized LDA or nearest centroid classification for V, and basic domain-text gating for T. These comparisons should use matching support and consider the difference between hard gating in T/O and soft gating in V (page 17, A.2.11).

Online adaptive experiments and oracle-AlphaSteer comparisons cannot replace these controls, as they test different parts of the method (page 9, Table 3; pages 18-19, Table 8 and Figures 2-3). Novelty claims also need to consider the pre-separation of detection and gating interventions, as described in Section 4 of GSS.

OfficeHome demonstrates the trade-offs of T's reduced support requirements. Its retention accuracy ranges from 36.02% to 40.99%, while TMS's retention accuracy ranges from 63.24% to 75.45%, and its HM is lower than TMS across all reported class scores. When γ = 0.50, 0.75, and 1.0, the HM values ​​for the T method are 55.90/51.74/56.12, and for the TMS method are 64.27/71.75/75.42. Due to the varying support levels of the different methods, these values ​​describe cross-method comparisons, not memory retention loss relative to the unmodified CLIP method (page 7, Table 1; page 8, rows 390-399). Tables 1-2 omit the F/R results for the original CLIP method, making it impossible to directly measure this loss. Similarly, the ViT-L/14 entries marked "baseline" in Tables 5-6 require more precise definitions. Incorporating the corresponding original model results will make the variations in memory retention more easily interpretable (pages 7 and 18).

The evaluation of all categories includes those not trained with forgetting, but the overall performance does not reveal the transferability of the suppression method in these categories. Calculating the F/R results and gate error rates for the observed and unobserved classes separately directly answers this question (page 8, lines 400–407). Claims regarding service cost also require measuring setup and refresh times, latency per query, and peak memory usage on matched hardware. Since gate computation uses the final embedding from the unguided process before modifying intermediate activations, execution scheduling can significantly impact latency. Maintaining complete scatter statistics introduces additional O(L<sub>C</sub>d²) storage costs. Therefore, gradient-free operations alone are insufficient to demonstrate lower overall service overhead (page 14, A.2.4; page 15, A.2.6).

Questions for authors

What causes the differences between Table 1 and Table 7? Which configuration should the reader consider the default? Please indicate any differences in support sampling, calibration, or implementation, and provide the number of runs and variability for both tables.

How is Equation (11) executed in the implementation? From calculating the unguided gate to modifying the relevant layers and label positions? In a single V setting, which pooling and regularization procedures make it possible to estimate the shared domain/class covariance?

Could the revised version include the aforementioned matching gate/payload controls and report performance for the observed and unobserved classes, including gate errors?

What flow construction and admission strategy is used for O in Table 1? If this is the weak text strategy discussed after Table 3, how was it chosen without labeled deployment data? For PACS with γ=0.25, the fixed forgetting support contains only 2×8=16 samples, below the minimum of 32 for online LDA. Please indicate whether O is still in the text/prototype stage in this setting, and report when LDA is activated in other settings (Page 13, A.2.2; Page 14, A.2.5; Page 16, A.2.9).

8 comments

We thank the reviewer for their thoughtful reviews and feedback. Below we address each of their concerns:

Q1: Why do Tables 1 and 7 differ, and which configuration is the default?

Table 1 is the default configuration and follows Table 4. Text-calibration coverage is 0.90 throughout, except in Table 7, where it is 0.95. For the shared ViT-B/16 backbone, this coverage change is the sole configuration difference; support sampling, seeds, and other implementation settings remain unchanged.

Higher coverage lowers the gate's acceptance threshold, admitting more forget-support images and producing stronger forgetting, potentially at the expense of retention. Table 7 deliberately tests this more aggressive regime across backbones. At the same 0.95 coverage, HM increases monotonically from ViT-B/16 to ViT-L/14 and MobileCLIP2-B, showing that stronger backbones improve the forgetting–retention balance.

Both tables use three independent repetitions. The maximum observed HM deviation from the mean was below 1.5 percentage points, comparable to benchmark variability. We will explicitly state the default configuration, Table 7's coverage change, and repetition statistics in the revision.

Q2: How is Eq. (11) implemented, and how is shared covariance estimated?

Several requested details already appear in the appendix: token pooling in A.2.1; test-then-update processing and admission in A.2.4–A.2.5; streaming statistics in A.2.6; prototypes and reliability weighting in A.2.7–A.2.8; and LDA readiness, shadowing, and deployment in A.2.9–A.2.11 and Table 4. We will clarify these connections and add the details below.

Execution of Eq. (11).

The affected positions are vision tokens, not class labels. Each block produces a residual tensor [B, T, d], containing CLS and patch tokens. An unedited pass computes the gate. During the payload pass, the same per-image gate acts on every token at each selected block:

H′=H+λg(μ−H)H' = H + \lambda g (\mu - H)

Here, H is a token activation, H′ its edited value, μ the layer’s forget centroid, and g the image’s gate value. The centroid, pooled over forget-support images and token positions, broadcasts as [1, 1, d]; the gate broadcasts as [B, 1, 1]. The hook preserves tensor shape. No inverse-pooling or label-tensor mapping is needed: subsequent blocks process the edited activations, and the unchanged text classifier scores the final image embedding.

Gate and payload blocks are selected separately. V searches all vision blocks for the largest valid LDA objective. Its payload uses the top round(0.6L) blocks ranked by:

∥μFℓ−μglobalℓ∥2.\Vert \mu_F^{\ell} - \mu_{\mathrm{global}}^{\ell} \Vert_2.

T/O intervenes at every block strictly after its separately selected visual anchor. V uses a continuous posterior; T/O uses binary acceptance.

Prompts and disagreement coefficient.

The three domain-conditioned/neutral template pairs are:

Domain-conditionedNeutral
a {domain} image of a {class}an image of a {class}
a {class} in {domain} stylea depiction of a {class}
a {domain} depiction of a {class}a representation of a {class}

For DomainNet, {domain} is clean flat vector clipart, a watercolor painting, natural real-world photographic, or detailed monochrome hand-drawn pencil-sketch. Aggregation follows Eqs. (1)–(3). We tested the prompt-disagreement coefficient at 0.5, 0.75, 0.9, and 1.0, observed little variation, and defaulted to 1.0.

Shared covariance and regularization.

All-token-mean features are grouped into forget/retain–class cells, pooling the selected retain domains. The shared covariance pools residuals centered at their cell means:

SW=1N∑g,c,i(hg,c,i−μg,c)(hg,c,i−μg,c)⊤.S_W = \frac{1}{N}\sum_{g,c,i}(h_{g,c,i}-\mu_{g,c})(h_{g,c,i}-\mu_{g,c})^\top.

N is the total contributing sample count. This is one shared within-cell covariance, not a separate covariance per class. We use:

SW,reg=SW+0.05tr(SW)dI,S_{W,\mathrm{reg}} = S_W + 0.05\frac{\mathrm{tr}(S_W)}{d}I,

C=Wk⊤SW,regWk,C = W_k^\top S_{W,\mathrm{reg}}W_k,

Σ=C+0.05tr(C)kI.\Sigma = C + 0.05\frac{\mathrm{tr}(C)}{k}I.

The retained rank targets 90% eigenvalue energy, excludes eigenvalues below 0.001 times the leading eigenvalue, and respects the class-count cap. The Gaussian gate uses equal priors and a linear solve (A.2.10). Online LDA reconstructs this covariance from Welford statistics (A.2.6).

LDA shadowing and deployment.

Admission, readiness, and refresh settings appear in A.2.5, A.2.7–A.2.10, and Table 4. Four leave-one-shard-out fits must pass readiness and numerical checks. A candidate completes a shadow interval without affecting predictions and is deployed only after another valid proposal selects the same block.

There is no fixed minimum reliability cutoff. The target weight is the minimum of fold-direction stability, fold-decision agreement, one minus four times fold-posterior variance, and a Gaussian boundary-offset penalty. Its increase/decrease rates are 0.10/0.50.

Q3(a): Matched gate and payload controls.

We conducted both requested ablations: changing the gate with contraction fixed, and changing the payload with the original gate fixed. Support and evaluation protocols remain matched. Bold marks the best result within each family; lower F/FP and higher R/HM are preferred.

Gate ablation with the payload fixed.

OfficeHome results use class fraction 0.50. Within each family, support, payload layers, centroids, and strength remain fixed. V also uses the same feature layer. T retains binary actuation; V retains continuous actuation.

FamilyGateF ↓R ↑HM ↑Retain FP ↓
TBasic domain text7.8736.5647.4256.39
TProposed text gate9.5643.1855.6147.65
VNearest centroid35.3955.4559.6732.54
VClass-paired LDA27.4769.8871.1115.03

Values are percentages, averaged over four forget configurations. Comparisons are within families because support requirements differ. V's 0.5 threshold is diagnostic only.

T improves HM by 8.19 points, retention by 6.62 points, and retain FP by 8.74 points over basic text similarity. Paired domain/neutral prompts contrast domain evidence against class content; class marginalization and template aggregation provide a more structured score. Unseen HM also improves from 48.47% to 57.62%.

V improves both forgetting and retention over nearest centroid, raising HM from 59.67% to 71.11% and reducing retain FP from 32.54% to 15.03%. This supports covariance-aware discrimination rather than centroid distance alone.

Pooled LDA slightly improves overall HM (71.52% versus 71.11%), while unseen HM is nearly identical (70.20% versus 70.15%). Class pairing performs better when forgetting Art: HM increases from 66.28% to 67.55%, and unseen HM from 63.99% to 65.65%.

Art is poorly resolved by the frozen text representation: domain recall is only 3.77%, with 60% of Art images classified as Product. Matching object classes across domains helps isolate rendering differences from semantic content, offering an explanation for class pairing's benefit here.

Both follow the same covariance-aware linear-discriminant philosophy and produce Gaussian posterior gates. Pooled LDA uses domain-group covariance; class pairing centers within domain–class cells and selects a subspace from matching-class differences. They are alternative estimators within the same gating framework, rather than separate end-to-end methods.

Class pairing particularly suits GATE-DU(O), where admitted pseudo-retains can have different class proportions from fixed forget support. Matched comparisons reduce semantic-composition confounding during subspace estimation. We extend this construction to V for consistency; their close overall performance with balanced support accords with their shared structure.

Payload ablation with the gate fixed.

On DomainNet at class fraction 0.50, we replace contraction with fixed or independently sampled random class outputs, independent of image content and labels. Replacement probabilities mix with original CLIP probabilities using the unchanged gate, preserving T's binary decisions and V's continuous scores without activation edits.

VariantPayloadF ↓R ↑HM ↑
TFixed output8.5262.5774.21
TRandom output8.4962.5774.21
TContraction8.4562.5574.21
VFixed output16.1772.7577.88
VRandom output16.1872.7477.87
VContraction16.3973.3978.14

Results aggregate four configurations; replacement controls average 20 prespecified randomizations.

Gating is the dominant contributor. T's aggregate HM is unchanged to two decimals. V's contraction gains approximately 0.27 HM and 0.64 retention points, with slightly weaker forgetting. Contraction is therefore competitive under hard gating and modestly beneficial under soft gating, rather than providing a large independent advantage.

The gate primarily determines which inputs receive suppression, allowing the class-agnostic payload to reduce recognition without explicitly constructing a retain-preserving subspace. Payload behavior still affects collateral damage when retain inputs receive nonzero activation. We will clarify this distinction, qualify contraction-specific claims, and include all controls, including pooled LDA, in the appendix of the revised version of the paper.

Q3(b): Seen/unseen performance, original CLIP, and gate errors.

We evaluate CLIP and V on identical images using the full vocabulary. Each accuracy cell reports F/R/HM; adjacent cells report FN/FP. Bold marks the better F/R/HM between methods within each dataset, fraction, and class group. All values are percentages. DN = DomainNet; OH = OfficeHome; V = GATE-DU(V).

DatasetFractionMethodSeen F/R/HMFN/FPUnseen F/R/HMFN/FPAll F/R/HMFN/FP
DN0.25CLIP83.91/83.91/27.01—82.34/82.34/29.08—82.74/82.74/28.56—
DN0.25V20.94/75.73/77.3622.81/9.7922.07/71.95/74.8225.90/12.8521.79/72.91/75.4725.12/12.08
DN0.50CLIP82.46/82.46/28.93—83.02/83.02/28.20—82.74/82.74/28.56—
DN0.50V17.38/75.42/78.8618.57/8.0717.14/75.66/79.1019.92/9.6017.26/75.54/78.9819.25/8.84
DN0.75CLIP83.03/83.03/28.18—81.88/81.88/29.68—82.74/82.74/28.56—
DN0.75V14.84/77.94/81.3915.59/6.3815.00/75.52/79.9819.22/8.4414.88/77.33/81.0416.51/6.90
DN1.00CLIP82.74/82.74/28.56———82.74/82.74/28.56—
DN1.00V13.93/77.31/81.4615.16/6.72——13.93/77.31/81.4615.16/6.72
OH0.25CLIP77.94/77.94/34.39—83.98/83.98/26.91—82.50/82.50/28.88—
OH0.25V25.10/67.79/71.1726.31/13.6528.83/67.25/69.1632.49/19.4527.96/67.35/69.6231.03/18.04
OH0.50CLIP79.08/79.08/33.09—85.75/85.75/24.44—82.50/82.50/28.88—
OH0.50V24.99/69.54/72.1728.02/11.8829.79/70.32/70.2732.90/17.8827.47/69.88/71.1830.55/15.03
OH0.75CLIP81.73/81.73/29.87—84.78/84.78/25.81—82.50/82.50/28.88—
OH0.75V22.85/71.60/74.2724.49/13.3526.06/69.56/71.6828.34/17.4023.67/71.08/73.6125.46/14.38
OH1.00CLIP82.50/82.50/28.88———82.50/82.50/28.88—
OH1.00V20.49/72.26/75.7123.28/12.22——20.49/72.26/75.7123.28/12.22
PACS0.25CLIP94.12/94.12/11.06—96.16/96.16/7.38—95.55/95.55/8.51—
PACS0.25V53.27/93.63/62.347.87/0.6920.46/92.77/85.6420.54/3.4530.66/92.85/79.3917.40/2.78
PACS0.50CLIP94.47/94.47/10.46—98.74/98.74/2.50—95.55/95.55/8.51—
PACS0.50V27.11/93.83/82.044.86/0.7815.00/94.36/89.4415.25/4.4222.62/93.33/84.619.06/2.30
PACS0.75CLIP94.79/94.79/9.88—98.49/98.49/2.97—95.55/95.55/8.51—
PACS0.75V18.68/94.37/87.363.14/0.557.21/97.14/94.927.60/1.4215.58/94.82/89.324.64/0.83
PACS1.00CLIP95.55/95.55/8.51———95.55/95.55/8.51—
PACS1.00V16.61/94.70/88.695.99/0.95——16.61/94.70/88.695.99/0.95

Frozen ViT-B/16 uses eight support images per seen class/domain and strength 0.5. F/R average four forget configurations; R is macro over retain domains. HM is recomputed from aggregate F/R, unlike the configuration-averaged gate-control table. FN/FP use threshold 0.5 only diagnostically; V remains continuous. At fraction 1.00, no unseen classes exist.

At fraction 0.50, unseen F decreases from 83.02% to 17.14% on DN, 85.75% to 29.79% on OH, and 98.74% to 15.00% on PACS, demonstrating transfer beyond support classes. DN's seen/unseen F is particularly close: 17.38%/17.14%.

Unseen retention losses are 7.35, 15.43, and 4.38 points, respectively. OH's larger loss accompanies higher unseen FP: 17.88%, versus 9.60%/4.42%. At PACS fraction 0.25, unseen F is lower despite higher FN, distinguishing payload effectiveness from gate detection.

We will add the full decomposition to the revised version of the paper(in the main paper section itself). “Baseline” in main-paper Tables 5–6 denotes NegGrad+Fisher, not unmodified CLIP; we will label it explicitly.

Reviewer_cyM4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces gated activation steering for targeted erasure in domain unlearning (GATE-DU), a training-free framework for domain unlearning. Authors argue that modifying representations without distinguishing between the forgetting and retaining inputs can degrade the unlearning performance. The proposed method uses three variants of domain gates to selectively suppress the recognition for forgetting domain inputs. The experimental results show the effectiveness of the proposed method.

Critical strengths
  1. Sound problem setting -- This paper addresses the limitation of existing works on domain unlearning -- their reliance on additional training and labeled retaining domain images. The proposed method is training-free and includes the variants that do not require domain labeled retaining domain images.

  2. Straightforward method -- Authors point out that unconditional steering impairs the domain-unlearning performance. Based on this observation, they propose the method to separate when to intervene from how to intervene. This proposed method aligns well with the observation.

Critical weaknesses
  1. Limited analysis of domain similarity -- Although the paper examines overlap between forgetting and retaining domain feature spaces, the experiments do not systematically discuss how well the method preserves the domain-unlearning performance when forgetting and retaining domains are similar.

  2. Unclear criteria for choosing online adaptation -- The choice between the visual variant and the retain-image-free variants follows naturally from data availability. However, it is less clear when GATE-DU (O) should be used over GATE-DU (T).

Questions for authors
  1. Can authors provide the domain similarity or overlap for individual forget-retain domain pairs and examine how it relates to the domain-unlearning performance.

  2. Can authors provide the guidance for how to determine which variants (i.e., GATE-DU (T, O, V)) to use?

4 comments

We thank the reviewer for their thoughtful reviews and feedback. Below we address each of their concerns:

Q1: How does pairwise domain overlap relate to unlearning performance?

We evaluated all 12 directed forget–retain pairs on DomainNet at class fraction 0.50, keeping support, gates, and payloads unchanged. Feature separation is measured using class-averaged regularized Mahalanobis distance between matching-class domain means, with pooled within-domain/class covariance. Smaller distance denotes greater overlap. Diagnostic statistics and intervention outcomes use disjoint image subsets.

Feature overlap alone does not explain retention loss.

FP is the percentage of retain images classified as forget; ΔR is original CLIP retain accuracy minus post-intervention accuracy, in percentage points. V uses threshold 0.5 only for FP diagnostics; payload application remains continuous.

ForgetRetainDistanceT FPV FPT ΔRV ΔR
ClipartPainting3323.66.885.034.503.70
ClipartReal3725.812.1710.0510.587.94
ClipartSketch2906.543.3912.4334.669.79
PaintingClipart3323.612.965.0310.324.23
PaintingReal3707.354.5012.9648.6811.11
PaintingSketch3294.221.965.5615.615.03
RealClipart3725.810.326.087.944.76
RealPainting3707.331.2213.7624.8711.90
RealSketch3917.14.232.912.652.12
SketchClipart2906.516.9311.1112.438.73
SketchPainting3294.224.077.1419.314.76
SketchReal3917.127.257.4124.347.41

Distance uses frozen final CLIP embeddings. Each gate retains the original one-forget/three-retain setting, without pairwise refitting.

For example, clipart–sketch has the smallest distance (2906.5), but T's retention loss for clipart → sketch is 34.66 points, below the 48.68-point loss for painting → real, whose distance is larger (3707.3). Across directed pairs, final-embedding distance has little association with retention loss (Pearson r = −0.05 for T and −0.09 for V). Block-10 token-mean correlations are also near zero (0.02 and 0.01).

In contrast, retain-side gate false positives closely track retention loss (r = 0.994 for T and 0.979 for V). This matches the intervention mechanism: retain images assigned forget evidence receive the payload. The deployed gate's discrimination is therefore more informative about collateral damage than class-averaged feature proximity alone. Large class-specific domain differences need not align into a consistent domain-discriminative direction, explaining why feature distance and gate performance need not coincide.

Unlearning difficulty is directional.

Painting → real and real → painting have the same distance, yet T incurs retention losses of 48.68 and 24.87 points. Asymmetry remains in V: clipart → real loses 7.94 points, whereas real → clipart loses 4.76 points despite identical distance. Thus, difficulty depends on which domain is suppressed and which is protected, not solely on symmetric proximity.

Unequal domain representation during pretraining is a plausible additional contributor. Original accuracy is 89.42% on real, versus 81.75% on clipart, 76.72% on painting, and 75.40% on sketch. These differences motivate the hypothesis that representations also differ in susceptibility to suppression. Representation strength, gate decisions, and payload geometry can jointly produce directional outcomes.

How GATE-DU(V) addresses this difficulty.

V searches intermediate layers for class-paired domain-discriminative evidence using the generalized-eigenvalue objective in Section 3.4, following Eq. (8). It exploits a usable domain boundary rather than assuming overall feature proximity determines intervention difficulty. V reduces gate false positives and retention loss on every directed pair, including clipart–sketch. Mean retention loss decreases from 19.66 points for T to 7.62 for V, accompanied by less aggressive forgetting: mean forget accuracy is 16.53% for V versus 8.47% for T.

Under the measured feature geometry, overlap alone does not predict retention damage; directional gate discrimination provides the stronger empirical explanation. These correlations are descriptive because pairs share domains and contain six unique distances. We will include the pairwise table and analysis in the revised version in the appendix.

Q2: How should users choose between GATE-DU(T, O, V)?

When labeled retain support is available, GATE-DU(V) directly constructs the visual discriminator. Without labeled retain support, Sections 3.2–3.3 and Table 3 distinguish three practical choices:

  • Streaming adaptation is unavailable: Use GATE-DU(T), which provides a fixed gate without online statistics or warm-up.

  • Text-domain evidence is weak: Use the full GATE-DU(O) progression, allowing the prototype to contribute before sufficient evidence exists for LDA. Table 3 shows prototype improvements in all 12 OfficeHome Art, Product, and Real World configurations.

  • Text-domain evidence is strong: Retain T's decisions while collecting pseudo-retain statistics, bypass prototype deployment, and activate LDA only when its readiness and stability requirements are satisfied. This avoids replacing an effective text boundary with an immature prototype: prototype-only adaptation hurts all 16 DomainNet configurations and all four OfficeHome Clipart configurations.

The third choice is the conditional policy motivated by Table 3. It transitions directly from text decisions to a V-like visual discriminator, while remaining distinct from V because its retain statistics use pseudo-labels from the unlabeled stream rather than labeled retain support. Table 3's AUROC is retrospective and is not used during deployment.

Table 7 further shows that stronger backbones can make T more effective: mean HM increases from 63.71% with ViT-B/16 to 75.53% with ViT-L/14 and 79.95% with MobileCLIP2-B. Label-free margins and prompt consistency may provide heuristic indicators of text-gate ambiguity. When text evidence is strong, we recommend retaining T's decisions; when weak, the full O progression provides visual grounding.

Thus, for commercial settings permitting streaming adaptation, we recommend early prototype deployment when text evidence is weak, and continued text gating with background statistics collection when strong. Without streaming adaptation, T is the default retain-image-free option. We will state this guidance explicitly in the revised manuscript.

Dear Reviewer, as the discussion period is nearing its end, we wanted to kindly ask whether you had a chance to review our rebuttal. We would greatly appreciate any feedback and, if our clarifications address your concerns, your reconsideration of the score.

Dear authors,

Thank you for the clarification. Authors have addressed my concerns, and I keep my positive score.

Reviewer_ksnEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses domain unlearning in vision-language models (VLMs), specifically the open-vocabulary setting where only a fraction of classes is available during unlearning. The authors propose GATE-DU (Gated Activation Steering for Targeted Erasure in Domain Unlearning), a training-free framework that decouples when to forget from how to forget. A domain gate first estimates whether an input belongs to the forget domain; only then does a class-agnostic centroid-collapse payload suppress class recognition, leaving retain-domain inputs unchanged. Three variants are introduced depending on available supervision: GATE-DU(T) uses only text (retain-image-free), GATE-DU(O) progressively grounds the text gate in visual evidence from the unlabeled evaluation stream, and GATE-DU(V) constructs a fully visual gate from labeled retain images. Experiments on PACS, OfficeHome, and MiniDomainNet show competitiveness with or improvement over optimization-based methods, especially in low-γ settings.

Critical strengths

Practical, deployment-friendly orientation. By requiring no gradients, adapters, or replay, the method targets hosted/controlled inference deployments where frequent unlearning requests and limited support data make retraining costly. This relevance to real-world serving is a notable strength.

Systematic handling of supervision regimes. The T/O/V trichotomy is clean and maps onto realistic data-availability scenarios, which improves the paper's generality and readability.

Sound experimental coverage. Evaluation spans three standard domain-generalization benchmarks and multiple class fractions γ ∈ {0.25, 0.5, 0.75, 1.0}, with supervision-matched comparisons against optimization-based baselines.

Critical weaknesses

Theoretical grounding is thin. The centroid-collapse payload and the Gaussian-posterior calibration are presented largely empirically. A more formal statement of when conditional intervention provably preserves retain-domain performance (or the conditions under which it fails) would strengthen the contribution beyond a heuristic.

The text-only gate (GATE-DU(T)) is inherently limited. Its reliability depends entirely on the quality of domain descriptions, which may not be available or discriminative in practice. The paper should discuss sensitivity to prompt/description quality and failure modes when forget and retain domains are semantically close.

Causal online protocol is assumed but not deeply analyzed. GATE-DU(O)'s prequential updates are described as strictly causal, yet the paper offers limited analysis of cold-start behavior, drift, or the risk of error accumulation from confident-but-wrong pseudo-retains.

Benchmark scope is somewhat narrow. While three datasets are used, all are domain-shift benchmarks for object recognition. It remains unclear whether the approach generalizes to other unlearning targets (e.g., classes, concepts, or generative VLMs), which the related-work section gestures at but does not validate.

Limited sensitivity analysis. The paper would benefit from ablations on key hyperparameters (e.g., the reliability thresholds governing text→prototype→LDA transitions, the Gaussian prior strength, and the choice of steering layer in GATE-DU(V)).

Questions for authors

Please see the weakness.

7 comments

We thank the reviewer for their thoughtful reviews and feedback. Below we address each of their concerns:

Q1: Limited theoretical grounding for centroid contraction, Gaussian calibration, and retain-performance preservation.

We agree that the retention claim should be stated formally. Conditional intervention admits a distribution-free guarantee for hard gating and a margin-based guarantee for soft gating, with Gaussian assumptions needed only to characterize the gate's activation probability.

Proposition: retain-performance preservation.

Let AR0A_R^0 and ARgA_R^g denote the original and intervened retain accuracies, respectively, expressed as probabilities. The gate is computed on an unedited forward pass and held fixed during intervention.

Hard gating.

For GATE-DU(T/O), predictions are unchanged whenever g(x)=0g(x)=0. Consequently,

AR0−ARg≤Pr⁡R(g(x)=1, f0(x)=y)≤Pr⁡R(g(x)=1).A_R^0 - A_R^g \leq \Pr_R\left(g(x)=1,\ f_0(x)=y\right) \leq \Pr_R(g(x)=1).

Thus, retain-accuracy loss is bounded by the gate's retain-side false-positive rate. In particular, zero false positives imply exact preservation of retain predictions, independently of the payload's behavior on accepted inputs.

Soft gating.

For GATE-DU(V), let s0(x)s_0(x) and sg(x)s_g(x) be the original and intervened class-logit vectors, and define

m(x)=s0,y(x)−max⁡c≠ys0,c(x).m(x) = s_{0,y}(x) - \max_{c \neq y}s_{0,c}(x).

At payload block ℓ\ell, let MFℓM_F^\ell be the forget centroid broadcast across token positions, and Hpreℓ(x)H_{\mathrm{pre}}^\ell(x) the activation immediately before its edit. Suppose the unedited network suffix from this block to the logits is LℓL_\ell-Lipschitz from the activation Frobenius norm to the logit ℓ∞\ell_\infty norm over the relevant activation region. Define

B(x)=∑ℓ∈LLℓ∥MFℓ−Hpreℓ(x)∥F.B(x) = \sum_{\ell \in \mathcal{L}} L_\ell \Vert M_F^\ell - H_{\mathrm{pre}}^\ell(x) \Vert_F.

Then

∥sg(x)−s0(x)∥∞≤λg(x)B(x).\Vert s_g(x) - s_0(x) \Vert_\infty \leq \lambda g(x)B(x).

An originally correct retain prediction is therefore preserved whenever

m(x)>2λg(x)B(x),\boxed{m(x) > 2\lambda g(x)B(x)},

and

AR0−ARg≤Pr⁡R(0<m(x)≤2λg(x)B(x)).A_R^0 - A_R^g \leq \Pr_R\left(0 < m(x) \leq 2\lambda g(x)B(x)\right).

Compare successive networks obtained by introducing payload edits in forward layer order. Each edit has magnitude λg(x)∥MFℓ−Hpreℓ(x)∥F\lambda g(x)\Vert M_F^\ell - H_{\mathrm{pre}}^\ell(x) \Vert_F; propagating it through the corresponding unedited suffix and summing gives the logit bound. A perturbation bounded by ϵ\epsilon in every logit can reduce a class margin by at most 2ϵ2\epsilon, which establishes the preservation condition.

Gaussian-posterior interpretation.

The Gaussian construction provides an additional distributional characterization, rather than an unconditional calibration guarantee. Under a correctly specified shared-covariance Gaussian model with equal priors and population parameters, the posterior is

g(u)=σ((μF−μR)⊤Σ−1[u−12(μF+μR)]).g(u) = \sigma\left((\mu_F - \mu_R)^\top \Sigma^{-1}\left[u - \frac{1}{2}(\mu_F + \mu_R)\right]\right).

Writing δ2=(μF−μR)⊤Σ−1(μF−μR)\delta^2 = (\mu_F - \mu_R)^\top \Sigma^{-1}(\mu_F - \mu_R), for δ>0\delta > 0 and 0<ϵ<10 < \epsilon < 1,

Pr⁡R(g>ϵ)=1−Φ(logit(ϵ)+δ2/2δ).\Pr_R(g > \epsilon) = 1 - \Phi\left(\frac{\mathrm{logit}(\epsilon) + \delta^2/2}{\delta}\right). Here, Φ\Phi denotes the standard normal cumulative distribution function.

If B(x)≤Bmax⁡B(x) \leq B_{\max} on retain inputs, this yields

AR0−ARg≤1−Φ(logit(ϵ)+δ2/2δ)+Pr⁡R(0<m(x)≤2λϵBmax⁡).A_R^0 - A_R^g \leq 1 - \Phi\left(\frac{\mathrm{logit}(\epsilon) + \delta^2/2}{\delta}\right) + \Pr_R\left(0 < m(x) \leq 2\lambda\epsilon B_{\max}\right).

The bound separates substantial retain-side gate activation from insufficient classification margin.

Implications and failure conditions.

These statements formalize the framework's division of responsibility. Hard gating protects rejected inputs exactly; soft gating protects inputs whose classification margin exceeds the propagated intervention magnitude. The centroid payload is itself a contraction:

∥H~ℓ−MFℓ∥F=(1−λg(x))∥Hℓ−MFℓ∥F.\Vert \widetilde{H}^{\ell} - M_F^{\ell} \Vert_F = (1 - \lambda g(x)) \Vert H^{\ell} - M_F^{\ell} \Vert_F.

However, contraction alone does not guarantee either retention or semantic erasure. Retention can fail through gate confusion, small classification margins, large activation displacements, or high downstream sensitivity. The Gaussian-tail characterization additionally depends on distributional fit and parameter estimation; the hard-gate and margin results do not.

We will add these propositions and proofs to the appendix and explicitly distinguish these conditional guarantees from the empirical performance of the fitted gates.

Q2: Description sensitivity and semantically close domains.

GATE-DU(T) requires domain-discriminative text evidence, not elaborate descriptions. Our new ablation separates sensitivity to wording from failures of image–text domain alignment.

Description sensitivity.

Equation (2) averages evidence across three paired templates and penalizes disagreement. Together with domain-conditioned/neutral prompt subtraction, this reduces dependence on individual templates, although it cannot correct consistently misleading descriptions.

For DomainNet's ordered domains clipart, painting, real, sketch, we compare:

  • Full: “clean flat vector clipart”; “a watercolor painting”; “natural real-world photographic”; “detailed monochrome hand-drawn pencil-sketch”.
  • Shortened: “clean flat vector”; “a watercolour”; “natural photographic”; “detailed monochrome hand-drawn”.
  • Names only: “clipart”; “painting”; “real”; “sketch”.

Only these phrases change; templates, support, evaluation images, payload, and calibration remain matched.

(a) DomainNet description ablation

Class fractionDescriptionF ↓R ↑HM ↑Gate FP ↓
0.25Full8.6965.8676.5220.66
0.25Shortened8.3364.0675.4223.21
0.25Names only10.0060.3772.2727.12
0.50Full6.7163.1675.3324.05
0.50Shortened7.8662.3074.3425.44
0.50Names only8.1357.5170.7430.91
0.75Full9.7267.9277.5218.19
0.75Shortened9.2165.0875.8222.05
0.75Names only10.5662.0873.2925.33
1.00Full8.6166.0376.6720.50
1.00Shortened8.8964.2375.3522.87
1.00Names only8.6558.1571.0629.85

Values are percentages, aggregated over four forget configurations; HM is recomputed from aggregate F/R. Experiments use frozen QuickGELU CLIP ViT-B/16, up to eight forget-support images per seen class, no retain support, calibration coverage 0.90, and steering strength 0.5. FP denotes retain-side gate false positives.

Shortening descriptions changes HM by only 1.0–1.7 percentage points. Names alone achieve 70.74–73.29% HM, 8.13–10.56% forget accuracy, and 71.94–74.94% unseen-class HM. Thus, detailed descriptions improve retention but are unnecessary for useful open-vocabulary suppression.

Regarding descriptions being unavailable, the forget and retain domains must first be specified to define the task. T can use their names directly, without additional descriptions.

Failure mode: poorly resolved domain evidence.

We independently evaluate four-way domain classification using the same backbone and the generic prompt a photo in {} style, filled with domain names.

DatasetDomainTop-1 (%)Top-2 (%)
DomainNetClipart98.1099.21
DomainNetPainting66.0384.60
DomainNetReal33.1788.10
DomainNetSketch22.5473.17
PACSArt painting98.54100.00
PACSCartoon90.4198.72
PACSPhoto92.7997.90
PACSSketch30.5371.76
OfficeHomeArt3.7715.47
OfficeHomeClipart69.7498.25
OfficeHomeProduct93.8399.15
OfficeHomeReal world2.3724.52

PACS exceeds 90% Top-1 on three domains. DomainNet retains informative rankings across all domains: even real and sketch achieve 88.10% and 73.17% Top-2 despite low Top-1 recall.

OfficeHome exhibits a more severe mismatch: art and real-world recall remains low even at Top-2 (15.47%/24.52%). Moreover, 60.00% of art and 83.44% of real-world images are classified as product. This skewed domain evidence helps explain T's weaker OfficeHome retention in Table 1. Consistently, mean retain-side gate FP in the full-description ablation is 49.11% on OfficeHome, versus 20.85% on DomainNet and 12.85% on PACS.

The relevant failure indicator is therefore whether the VLM resolves the intended domain distinction, rather than semantic proximity alone. Semantically close descriptions can still provide exploitable visual evidence.

Backbone capability.

Table 7 further shows that T's mean HM increases from 63.71% with ViT-B/16 to 75.53% with ViT-L/14 and 79.95% with MobileCLIP2-B; retention increases from 47.90% to 63.11% and 69.50%. Stronger backbones can therefore exploit textual domain evidence more effectively, rather than T having a fixed performance ceiling.

We will add these ablations, per-domain diagnostics, and the failure-mode discussion to the revised manuscript.

Q3: Online causality, cold start, drift, and pseudo-retain errors.

The manuscript analyzes cold start and pseudo-retain errors in Table 3, with safeguards detailed in A.2.4–A.2.10. We will make these connections clearer.

Causality and cold start.

A.2.4 specifies strict test-then-update processing: each image is predicted using the preceding immutable state before updating statistics for future images. O starts with frozen text gating. The first prototype candidate receives zero stability weight (A.2.8), and LDA requires sufficient samples and paired classes (A.2.9).

Table 3 explicitly demonstrates that adaptation is not immediately beneficial. Prototype-only adaptation hurts all 16 DomainNet and all four OfficeHome Clipart configurations, but improves all 12 OfficeHome Art, Product, and Real World configurations. On DomainNet, enabling LDA raises HM from 0.656 to 0.728; in the final stream quartile, the full pipeline exceeds frozen text by 3.1 percentage points on average. This exposes both the early-stage cost and the benefit of mature visual statistics.

Confident-but-wrong pseudo-retains.

Admission requires normalized class entropy ≤ 0.35 and retain-side evidence from the frozen text boundary (A.2.5). The adaptive visual gate never generates its own admission labels, and labeled forget statistics remain fixed. This prevents direct self-reinforcement through adaptive admissions, although frozen-teacher errors can still contaminate statistics.

Our oracle-admission control tests this empirically: replacing DomainNet's retain-domain admissions with oracle labels improves prototype-only HM only from 0.649 to 0.658, still below frozen text at 0.742. Incorrect domain admissions alone therefore do not explain prototype degradation, although this control does not isolate pseudo-class assignment errors.

Stability and drift.

A.2.8–A.2.10 monitor direction/boundary stability, use four leave-one-shard-out reconstructions, and evaluate LDA candidates in shadow mode before deployment. Reliability decreases faster than it increases (0.50 versus 0.10 update rates), limiting unstable estimates. These safeguards address evolving-statistics instability. We will clarify this scope alongside the existing cold-start and admission-error analyses in the appendix. Additional sensitivity analysis regarding GATE-DU(O) is provided in the response to Q5.

Q4: Benchmark scope and generalization beyond domain unlearning.

Our contribution targets open-vocabulary domain unlearning: suppressing recognition across classes in one visual domain while preserving those same classes elsewhere. This differs from removing a semantic category everywhere.

Why domain unlearning requires separate study.

The manuscript includes class-unlearning controls to establish this distinction. In Table 8, AlphaSteer achieves (F, R, HM) = (13.41, 73.17, 79.32)% when forget/retain classes share domains, and (10.48, 75.87, 82.14)% when both classes and domains differ.

In contrast, unconditional domain intervention reduces forget and retain accuracy together (Figure 2). Figure 3 shows stronger subspace selectivity for the class controls but substantial overlap for domain unlearning. Applying the same AlphaSteer payload with an oracle domain gate achieves 2.1% forget accuracy and 80.4% retention, isolating the domain-selectivity bottleneck motivating GATE-DU.

Thus, our contribution addresses a setting where demonstrated class-unlearning success does not transfer. Conditional gating could also target classes, but existing subspace steering already exploits useful class-discriminative structure in these controls. Additional class benchmarks would evaluate an extension rather than the central domain-specific difficulty. We do not claim that class unlearning is universally solved, but strong existing results on object-recognition benchmarks suggest that conditional gating would offer an incremental benefit, unlike its central role in our domain-unlearning setting.

Benchmark choice and broader targets.

OfficeHome, Mini-DomainNet, and PACS follow the few-shot domain-unlearning protocols of Kawamura et al. (2026) as specified in Section 4.1. This enables support-matched comparisons with relevant baselines. Varying class exposure additionally tests transfer to classes absent from support.

Concept removal and generative VLMs require different target definitions and evaluations. Overlapping concepts require compositional suppression and preservation tests. Generative VLMs require assessment across autoregressive outputs, paraphrased requests, and retained generation quality; classification F/R/HM cannot establish these properties.

Related work contextualizes our method without claiming validated generalization to these settings. We will state this scope explicitly and identify broader targets as future work. The demonstrated contribution remains training-free conditional domain unlearning with held-out-class transfer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.