- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates domain forgetting in hosted CLIP services, aiming to suppress the recognition of classes in a specified visual domain (e.g., painting) while preserving the recognition of those classes in other domains. It also considers an open-vocabulary setting where only a small subset of classes are observed during forgetting.
With both the encoder and cue frozen, the method uses a domain gate to control the fixed intervention, shifting activation toward the forgetting support centroid: . Three variants construct the gate from different sources of evidence: T relies on class-conditional text without preserving images; O also uses prototype and domain LDA evidence from the unlabeled stream, making each prediction before updating its statistics; V derives a domain-LDA/Gaussian gate from labeled forgetting and retention support. Experiments evaluate CLIP ViT-B/16 on OfficeHome, Mini-DomainNet, and PACS, using eight support images for each selected category.
Critical strengths
The support settings and achievable service level objectives are clearly defined. The data requirements for the three variants are clearly differentiated: T does not use preserved images, O includes an unlabeled data stream, and V uses labeled support based on optimization available from competitors (page 8, lines 390–399). The service level objectives make selective suppression a reasonable design choice, as the task does not require the permanent removal of information from the model parameters (page 1, lines 44–50). The online protocol also includes several useful safeguards: prediction precedes statistical updates, and admission decisions rely on a fixed text teacher (page 14, A.2.4–A.2.5). Performance for both full-stream and late-stream streams is reported in Table 3, helping to reveal the adaptive reduction in accuracy (page 9).
Performance is good when forgetting learning observes only a small subset of classes. At γ=0.25 and γ=0.50, V outperforms the reported strongest optimization baseline on all three datasets. When γ=0.25, HM achieves an improvement of 15.67 percentage points on the OfficeHome dataset, 3.89 percentage points on the Mini-DomainNet dataset, and 44.68 percentage points on the PACS dataset (page 7, Table 1). Therefore, this advantage is present on all datasets, although the largest improvement occurs on the PACS dataset, which is described as a small-sample stress test (page 8, rows 382-389). The single-run evaluation in Figure 1 and the ViT-L/14 experiments in Tables 5-6 provide further evidence under different support budgets and backbone sizes. This advantage becomes less stable as the proportion of categories increases: TMS performs better on the Mini-DomainNet dataset at γ=0.75 and 1.0, while OfficeHome performs similarly at γ=0.75 (V = 71.98, TMS = 71.75), but slightly worse at γ=1.0 (V = 74.83 vs. 75.42). Variability estimation will help determine whether these narrow gaps support meaningful rankings (Page 7, Table 1; Page 9, Figure 1; Page 18, Tables 5–6).
Critical weaknesses
1: Inconsistent reporting and incomplete standardization limited the reproducibility of the results. At γ=0.50, the DomainNet T results showed significant differences between the two tables: Table 1 listed F/R/HM as 8.45/62.55/74.32%, while Table 7 showed average ViT-B/16 as 3.37/47.90/63.71%. These correspond to differences of 5.08, 14.65, and 10.61 percentage points, respectively (pages 7 and 18). Both tables mention a frozen ViT-B/16 model, 8 support images per selected class, evaluation across all classes, and λ=0.5, but they do not specify whether the extraction, calibration, and implementation of the support images are identical (page 8, §4.1; page 13, A.2.1–A.2.3; page 17, Table 4). Another HM aggregation rule fails to account for the differences between F and R. Therefore, this paper needs to clarify the configuration behind each table and explain whether direct comparisons are valid. This issue only applies to T; it does not negate the performance reported by V.
Furthermore, the intervention is not adequately described at the tensor level. Equation (11) operates on the activation vector, but A.2.1 constructs gating features by averaging the residual labels. To reproduce this operation, the intervention layer, the affected label locations, and the rule for mapping the updated vector back to the label tensor are required (page 7, lines 363–373; page 13, lines 673–681). More details are needed regarding the prompt template and η_prompt, scatter/covariance scaling and regularization, candidate blocks of V, and numerical thresholds for controlling LDA shading and submission decisions (pp. 4–6, Equations (1)–(9); pp. 15–17, A.2.7–A.2.11). The use of “multiple independent runs” in the experiments should also include the number of repetitions and their variability (p. 8, line 408).
2: The current comparisons fail to adequately separate gating, intervention, and retention effects. Keep gating unchanged and simplify the intervention. For inputs with existing gating choices, replace activation contraction with fixed or random, class-independent outputs while keeping the data protocol unchanged. Comparing F/R/HM can show how much the payload contributes beyond identifying the input to be suppressed. If performance remains similar, gating may still have some effect, but the evidence for the contraction mechanism will be weaker.
Keep the intervention unchanged and simplify the gating. Suitable controls include: merge regularized LDA or nearest centroid classification for V, and basic domain-text gating for T. These comparisons should use matching support and consider the difference between hard gating in T/O and soft gating in V (page 17, A.2.11).
Online adaptive experiments and oracle-AlphaSteer comparisons cannot replace these controls, as they test different parts of the method (page 9, Table 3; pages 18-19, Table 8 and Figures 2-3). Novelty claims also need to consider the pre-separation of detection and gating interventions, as described in Section 4 of GSS.
OfficeHome demonstrates the trade-offs of T's reduced support requirements. Its retention accuracy ranges from 36.02% to 40.99%, while TMS's retention accuracy ranges from 63.24% to 75.45%, and its HM is lower than TMS across all reported class scores. When γ = 0.50, 0.75, and 1.0, the HM values for the T method are 55.90/51.74/56.12, and for the TMS method are 64.27/71.75/75.42. Due to the varying support levels of the different methods, these values describe cross-method comparisons, not memory retention loss relative to the unmodified CLIP method (page 7, Table 1; page 8, rows 390-399). Tables 1-2 omit the F/R results for the original CLIP method, making it impossible to directly measure this loss. Similarly, the ViT-L/14 entries marked "baseline" in Tables 5-6 require more precise definitions. Incorporating the corresponding original model results will make the variations in memory retention more easily interpretable (pages 7 and 18).
The evaluation of all categories includes those not trained with forgetting, but the overall performance does not reveal the transferability of the suppression method in these categories. Calculating the F/R results and gate error rates for the observed and unobserved classes separately directly answers this question (page 8, lines 400–407). Claims regarding service cost also require measuring setup and refresh times, latency per query, and peak memory usage on matched hardware. Since gate computation uses the final embedding from the unguided process before modifying intermediate activations, execution scheduling can significantly impact latency. Maintaining complete scatter statistics introduces additional O(L<sub>C</sub>d²) storage costs. Therefore, gradient-free operations alone are insufficient to demonstrate lower overall service overhead (page 14, A.2.4; page 15, A.2.6).
Questions for authors
What causes the differences between Table 1 and Table 7? Which configuration should the reader consider the default? Please indicate any differences in support sampling, calibration, or implementation, and provide the number of runs and variability for both tables.
How is Equation (11) executed in the implementation? From calculating the unguided gate to modifying the relevant layers and label positions? In a single V setting, which pooling and regularization procedures make it possible to estimate the shared domain/class covariance?
Could the revised version include the aforementioned matching gate/payload controls and report performance for the observed and unobserved classes, including gate errors?
What flow construction and admission strategy is used for O in Table 1? If this is the weak text strategy discussed after Table 3, how was it chosen without labeled deployment data? For PACS with γ=0.25, the fixed forgetting support contains only 2×8=16 samples, below the minimum of 32 for online LDA. Please indicate whether O is still in the text/prototype stage in this setting, and report when LDA is activated in other settings (Page 13, A.2.2; Page 14, A.2.5; Page 16, A.2.9).
8 comments
Q3(a): Matched gate and payload controls.
We conducted both requested ablations: changing the gate with contraction fixed, and changing the payload with the original gate fixed. Support and evaluation protocols remain matched. Bold marks the best result within each family; lower F/FP and higher R/HM are preferred.
Gate ablation with the payload fixed.
OfficeHome results use class fraction 0.50. Within each family, support, payload layers, centroids, and strength remain fixed. V also uses the same feature layer. T retains binary actuation; V retains continuous actuation.
| Family | Gate | F ↓ | R ↑ | HM ↑ | Retain FP ↓ |
|---|---|---|---|---|---|
| T | Basic domain text | 7.87 | 36.56 | 47.42 | 56.39 |
| T | Proposed text gate | 9.56 | 43.18 | 55.61 | 47.65 |
| V | Nearest centroid | 35.39 | 55.45 | 59.67 | 32.54 |
| V | Class-paired LDA | 27.47 | 69.88 | 71.11 | 15.03 |
Values are percentages, averaged over four forget configurations. Comparisons are within families because support requirements differ. V's 0.5 threshold is diagnostic only.
T improves HM by 8.19 points, retention by 6.62 points, and retain FP by 8.74 points over basic text similarity. Paired domain/neutral prompts contrast domain evidence against class content; class marginalization and template aggregation provide a more structured score. Unseen HM also improves from 48.47% to 57.62%.
V improves both forgetting and retention over nearest centroid, raising HM from 59.67% to 71.11% and reducing retain FP from 32.54% to 15.03%. This supports covariance-aware discrimination rather than centroid distance alone.
Pooled LDA slightly improves overall HM (71.52% versus 71.11%), while unseen HM is nearly identical (70.20% versus 70.15%). Class pairing performs better when forgetting Art: HM increases from 66.28% to 67.55%, and unseen HM from 63.99% to 65.65%.
Art is poorly resolved by the frozen text representation: domain recall is only 3.77%, with 60% of Art images classified as Product. Matching object classes across domains helps isolate rendering differences from semantic content, offering an explanation for class pairing's benefit here.
Both follow the same covariance-aware linear-discriminant philosophy and produce Gaussian posterior gates. Pooled LDA uses domain-group covariance; class pairing centers within domain–class cells and selects a subspace from matching-class differences. They are alternative estimators within the same gating framework, rather than separate end-to-end methods.
Class pairing particularly suits GATE-DU(O), where admitted pseudo-retains can have different class proportions from fixed forget support. Matched comparisons reduce semantic-composition confounding during subspace estimation. We extend this construction to V for consistency; their close overall performance with balanced support accords with their shared structure.
Payload ablation with the gate fixed.
On DomainNet at class fraction 0.50, we replace contraction with fixed or independently sampled random class outputs, independent of image content and labels. Replacement probabilities mix with original CLIP probabilities using the unchanged gate, preserving T's binary decisions and V's continuous scores without activation edits.
| Variant | Payload | F ↓ | R ↑ | HM ↑ |
|---|---|---|---|---|
| T | Fixed output | 8.52 | 62.57 | 74.21 |
| T | Random output | 8.49 | 62.57 | 74.21 |
| T | Contraction | 8.45 | 62.55 | 74.21 |
| V | Fixed output | 16.17 | 72.75 | 77.88 |
| V | Random output | 16.18 | 72.74 | 77.87 |
| V | Contraction | 16.39 | 73.39 | 78.14 |
Results aggregate four configurations; replacement controls average 20 prespecified randomizations.
Gating is the dominant contributor. T's aggregate HM is unchanged to two decimals. V's contraction gains approximately 0.27 HM and 0.64 retention points, with slightly weaker forgetting. Contraction is therefore competitive under hard gating and modestly beneficial under soft gating, rather than providing a large independent advantage.
The gate primarily determines which inputs receive suppression, allowing the class-agnostic payload to reduce recognition without explicitly constructing a retain-preserving subspace. Payload behavior still affects collateral damage when retain inputs receive nonzero activation. We will clarify this distinction, qualify contraction-specific claims, and include all controls, including pooled LDA, in the appendix of the revised version of the paper.
Q3(b): Seen/unseen performance, original CLIP, and gate errors.
We evaluate CLIP and V on identical images using the full vocabulary. Each accuracy cell reports F/R/HM; adjacent cells report FN/FP. Bold marks the better F/R/HM between methods within each dataset, fraction, and class group. All values are percentages. DN = DomainNet; OH = OfficeHome; V = GATE-DU(V).
| Dataset | Fraction | Method | Seen F/R/HM | FN/FP | Unseen F/R/HM | FN/FP | All F/R/HM | FN/FP |
|---|---|---|---|---|---|---|---|---|
| DN | 0.25 | CLIP | 83.91/83.91/27.01 | — | 82.34/82.34/29.08 | — | 82.74/82.74/28.56 | — |
| DN | 0.25 | V | 20.94/75.73/77.36 | 22.81/9.79 | 22.07/71.95/74.82 | 25.90/12.85 | 21.79/72.91/75.47 | 25.12/12.08 |
| DN | 0.50 | CLIP | 82.46/82.46/28.93 | — | 83.02/83.02/28.20 | — | 82.74/82.74/28.56 | — |
| DN | 0.50 | V | 17.38/75.42/78.86 | 18.57/8.07 | 17.14/75.66/79.10 | 19.92/9.60 | 17.26/75.54/78.98 | 19.25/8.84 |
| DN | 0.75 | CLIP | 83.03/83.03/28.18 | — | 81.88/81.88/29.68 | — | 82.74/82.74/28.56 | — |
| DN | 0.75 | V | 14.84/77.94/81.39 | 15.59/6.38 | 15.00/75.52/79.98 | 19.22/8.44 | 14.88/77.33/81.04 | 16.51/6.90 |
| DN | 1.00 | CLIP | 82.74/82.74/28.56 | — | — | — | 82.74/82.74/28.56 | — |
| DN | 1.00 | V | 13.93/77.31/81.46 | 15.16/6.72 | — | — | 13.93/77.31/81.46 | 15.16/6.72 |
| OH | 0.25 | CLIP | 77.94/77.94/34.39 | — | 83.98/83.98/26.91 | — | 82.50/82.50/28.88 | — |
| OH | 0.25 | V | 25.10/67.79/71.17 | 26.31/13.65 | 28.83/67.25/69.16 | 32.49/19.45 | 27.96/67.35/69.62 | 31.03/18.04 |
| OH | 0.50 | CLIP | 79.08/79.08/33.09 | — | 85.75/85.75/24.44 | — | 82.50/82.50/28.88 | — |
| OH | 0.50 | V | 24.99/69.54/72.17 | 28.02/11.88 | 29.79/70.32/70.27 | 32.90/17.88 | 27.47/69.88/71.18 | 30.55/15.03 |
| OH | 0.75 | CLIP | 81.73/81.73/29.87 | — | 84.78/84.78/25.81 | — | 82.50/82.50/28.88 | — |
| OH | 0.75 | V | 22.85/71.60/74.27 | 24.49/13.35 | 26.06/69.56/71.68 | 28.34/17.40 | 23.67/71.08/73.61 | 25.46/14.38 |
| OH | 1.00 | CLIP | 82.50/82.50/28.88 | — | — | — | 82.50/82.50/28.88 | — |
| OH | 1.00 | V | 20.49/72.26/75.71 | 23.28/12.22 | — | — | 20.49/72.26/75.71 | 23.28/12.22 |
| PACS | 0.25 | CLIP | 94.12/94.12/11.06 | — | 96.16/96.16/7.38 | — | 95.55/95.55/8.51 | — |
| PACS | 0.25 | V | 53.27/93.63/62.34 | 7.87/0.69 | 20.46/92.77/85.64 | 20.54/3.45 | 30.66/92.85/79.39 | 17.40/2.78 |
| PACS | 0.50 | CLIP | 94.47/94.47/10.46 | — | 98.74/98.74/2.50 | — | 95.55/95.55/8.51 | — |
| PACS | 0.50 | V | 27.11/93.83/82.04 | 4.86/0.78 | 15.00/94.36/89.44 | 15.25/4.42 | 22.62/93.33/84.61 | 9.06/2.30 |
| PACS | 0.75 | CLIP | 94.79/94.79/9.88 | — | 98.49/98.49/2.97 | — | 95.55/95.55/8.51 | — |
| PACS | 0.75 | V | 18.68/94.37/87.36 | 3.14/0.55 | 7.21/97.14/94.92 | 7.60/1.42 | 15.58/94.82/89.32 | 4.64/0.83 |
| PACS | 1.00 | CLIP | 95.55/95.55/8.51 | — | — | — | 95.55/95.55/8.51 | — |
| PACS | 1.00 | V | 16.61/94.70/88.69 | 5.99/0.95 | — | — | 16.61/94.70/88.69 | 5.99/0.95 |
Frozen ViT-B/16 uses eight support images per seen class/domain and strength 0.5. F/R average four forget configurations; R is macro over retain domains. HM is recomputed from aggregate F/R, unlike the configuration-averaged gate-control table. FN/FP use threshold 0.5 only diagnostically; V remains continuous. At fraction 1.00, no unseen classes exist.
At fraction 0.50, unseen F decreases from 83.02% to 17.14% on DN, 85.75% to 29.79% on OH, and 98.74% to 15.00% on PACS, demonstrating transfer beyond support classes. DN's seen/unseen F is particularly close: 17.38%/17.14%.
Unseen retention losses are 7.35, 15.43, and 4.38 points, respectively. OH's larger loss accompanies higher unseen FP: 17.88%, versus 9.60%/4.42%. At PACS fraction 0.25, unseen F is lower despite higher FN, distinguishing payload effectiveness from gate detection.
We will add the full decomposition to the revised version of the paper(in the main paper section itself). “Baseline” in main-paper Tables 5–6 denotes NegGrad+Fisher, not unmodified CLIP; we will label it explicitly.
We thank the reviewer for their thoughtful reviews and feedback. Below we address each of their concerns:
Q1: Why do Tables 1 and 7 differ, and which configuration is the default?
Table 1 is the default configuration and follows Table 4. Text-calibration coverage is 0.90 throughout, except in Table 7, where it is 0.95. For the shared ViT-B/16 backbone, this coverage change is the sole configuration difference; support sampling, seeds, and other implementation settings remain unchanged.
Higher coverage lowers the gate's acceptance threshold, admitting more forget-support images and producing stronger forgetting, potentially at the expense of retention. Table 7 deliberately tests this more aggressive regime across backbones. At the same 0.95 coverage, HM increases monotonically from ViT-B/16 to ViT-L/14 and MobileCLIP2-B, showing that stronger backbones improve the forgetting–retention balance.
Both tables use three independent repetitions. The maximum observed HM deviation from the mean was below 1.5 percentage points, comparable to benchmark variability. We will explicitly state the default configuration, Table 7's coverage change, and repetition statistics in the revision.
Q2: How is Eq. (11) implemented, and how is shared covariance estimated?
Several requested details already appear in the appendix: token pooling in A.2.1; test-then-update processing and admission in A.2.4–A.2.5; streaming statistics in A.2.6; prototypes and reliability weighting in A.2.7–A.2.8; and LDA readiness, shadowing, and deployment in A.2.9–A.2.11 and Table 4. We will clarify these connections and add the details below.
Execution of Eq. (11).
The affected positions are vision tokens, not class labels. Each block produces a residual tensor
[B, T, d], containing CLS and patch tokens. An unedited pass computes the gate. During the payload pass, the same per-image gate acts on every token at each selected block:H′=H+λg(μ−H)
Here, H is a token activation, H′ its edited value, μ the layer’s forget centroid, and g the image’s gate value. The centroid, pooled over forget-support images and token positions, broadcasts as
[1, 1, d]; the gate broadcasts as[B, 1, 1]. The hook preserves tensor shape. No inverse-pooling or label-tensor mapping is needed: subsequent blocks process the edited activations, and the unchanged text classifier scores the final image embedding.Gate and payload blocks are selected separately. V searches all vision blocks for the largest valid LDA objective. Its payload uses the top
round(0.6L)blocks ranked by:∥μFℓ−μglobalℓ∥2.
T/O intervenes at every block strictly after its separately selected visual anchor. V uses a continuous posterior; T/O uses binary acceptance.
Prompts and disagreement coefficient.
The three domain-conditioned/neutral template pairs are:
a {domain} image of a {class}an image of a {class}a {class} in {domain} stylea depiction of a {class}a {domain} depiction of a {class}a representation of a {class}For DomainNet,
{domain}isclean flat vector clipart,a watercolor painting,natural real-world photographic, ordetailed monochrome hand-drawn pencil-sketch. Aggregation follows Eqs. (1)–(3). We tested the prompt-disagreement coefficient at 0.5, 0.75, 0.9, and 1.0, observed little variation, and defaulted to 1.0.Shared covariance and regularization.
All-token-mean features are grouped into forget/retain–class cells, pooling the selected retain domains. The shared covariance pools residuals centered at their cell means:
SW=N1∑g,c,i(hg,c,i−μg,c)(hg,c,i−μg,c)⊤.
N is the total contributing sample count. This is one shared within-cell covariance, not a separate covariance per class. We use:
SW,reg=SW+0.05dtr(SW)I,
C=Wk⊤SW,regWk,
Σ=C+0.05ktr(C)I.
The retained rank targets 90% eigenvalue energy, excludes eigenvalues below 0.001 times the leading eigenvalue, and respects the class-count cap. The Gaussian gate uses equal priors and a linear solve (A.2.10). Online LDA reconstructs this covariance from Welford statistics (A.2.6).
LDA shadowing and deployment.
Admission, readiness, and refresh settings appear in A.2.5, A.2.7–A.2.10, and Table 4. Four leave-one-shard-out fits must pass readiness and numerical checks. A candidate completes a shadow interval without affecting predictions and is deployed only after another valid proposal selects the same block.
There is no fixed minimum reliability cutoff. The target weight is the minimum of fold-direction stability, fold-decision agreement, one minus four times fold-posterior variance, and a Gaussian boundary-offset penalty. Its increase/decrease rates are 0.10/0.50.