acceptodds
Under review as a conference paper at ICLR 2027

Forgetting Must Ignite: Overcoming Under-Forgetting in Machine Unlearning with Pair-Free Cauchy–Schwarz Angular Repulsion

Abstract

Approximate machine unlearning seeks to remove the influence of a forget set at substantially lower cost than full retraining. Yet strong baselines such as fine-tuning often under-forget: they preserve utility while leaving forget-set representations close to their original, memorized state. We address this failure mode with CSD-Unlearning, a feature-level objective based on Cauchy–Schwarz divergence that preserves retain-set representations while selectively repelling forget-set features from the original model. We show that the resulting objective admits a geometric interpretation as a negative log-cosine divergence between kernel mean embeddings. This analysis reveals that the repulsive gradient vanishes for deterministic feature extraction when the current and reference representations coincide. A capped one-dimensional surrogate further exhibits a sharp transition between locally suppressing and amplifying nonzero displacements, providing a mechanistic explanation for the threshold-like behavior observed in several training sweeps. Across CIFAR-10, CIFAR-100, and Tiny ImageNet with ResNet-18 and vision transformers, CSD-Unlearning consistently reduces the gap to retraining when added to fine-tuning. We also characterize its regime of effectiveness: the method is most useful for under-forgetting baselines, provides limited gains once a baseline already over-forgets, and degrades in selectivity at large forget ratios or outside a narrow stability range on vision transformers.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Machine Unlearning, Approximate Unlearning, Representation Unlearning, Membership Inferenc

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aWJLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CSD-Unlearning, a machine unlearning method that uses Cauchy–Schwarz divergence to improve forgetting. The approach combines two objectives: keeping representations of retained samples close to those of the original model and pushing representations of forgotten samples away from their original state. They motivate this saying that previous methods, in particular fine-tuning, “underforget”. The paper provides a theoretical analysis showing that the repulsive gradient vanishes at initialization (as the model has not moved yet), and they identify a threshold-like behavior in a simplified optimization setting. Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet with ResNet-18 and Vision Transformers show that adding CSD to fine-tuning can bring the model closer to a retrained reference.

Critical strengths
  • The motivation is clear and puts the focus on the representation space that methods usually do not take a look at in approximate unlearning, with a nice distribution-level approach.

  • Simple and flexible: The CSD objective can be integrated into other existing unlearning methods without additional adaptations.

  • Interesting theoretical analysis: The paper provides a geometric interpretation of CSD in representation space and investigates the optimization dynamics, with a threshold-like effect in a simplified setting.

Critical weaknesses
  • Experiments

ViT architecture and scalability: Which ViT architecture is used? Tiny, Small, Base? This is not clear from the paper. The authors should specify the architecture and number of parameters. I would also like to see experiments with larger models, at least ViT-Base, to better assess scalability.

Test accuracy degradation: The proposed method seems to degrade test accuracy considerably more than some of the other approximate unlearning baselines. The authors should discuss this trade-off more explicitly, explain why it happens, and explain how it could be alleviated.

  • Missing baselines

Contrastive Unlearning: The paper cites Contrastive Unlearning: A Contrastive Approach to Machine Unlearning, but does not experimentally compare against it. This is a particularly relevant baseline, as it also directly modifies forget-set representations. I would expect a comparison to understand whether CSD actually provides an improvement over existing representation-based methods.

Limited comparisons with existing methods: In general, improving over fine-tuning alone is not enough, given the large body of work developing stronger unlearning methods. While the authors include other baselines, these comparisons are limited. CoUn is evaluated using a different pipeline, the SCRUB and SalUn reproductions perform surprisingly poorly, and the MMD comparison uses only a single seed. Moreover, several comparisons appear only in the appendix. I would like to see more comprehensive, fair, shared-protocol comparisons in the main results.

Missing simple repulsion baselines: Another important missing ablation is a comparison against simpler feature-level repulsion objectives, such as cosine distance or squared Euclidean distance between the original and current forget-set representations. It is not clear how much benefit comes specifically from using CSD rather than simply pushing the representations apart. This comparison seems particularly important to establish the contribution of the proposed objective.

  • Limited scope

Forgetting scenarios: Almost all experiments use 10% random-subset forgetting, with 50% evaluated only in a limited setting. I would like to see additional forgetting ratios, including smaller ones (e.g., 2%), which can be relevant to practical deletion requests. More importantly, class-wise unlearning should also be evaluated. Currently, the experiments are too homogeneous to assess how well the method generalizes to different forgetting scenarios.

Limited to image classification: All experiments are conducted on relatively small image-classification benchmarks. Given the general framing of the proposed method, I would like to see results on other modalities, particularly language models. Otherwise, the authors should make the scope of their empirical claims clearer and explicitly limit their conclusions to image-classification unlearning.

  • Theoretical results

Theorem 1: I find this result rather weak. The ignition threshold essentially follows from a sign change in the coefficient of a quadratic objective, and it does not seem to establish anything specific to Cauchy–Schwarz divergence. A simple squared-distance repulsion competing against a quadratic regularizer would lead to a similar threshold.

Moreover, the theorem studies a highly simplified optimization problem, whereas the actual method operates on high-dimensional neural-network representations. The cap also appears inactive in the reported experiments, unlike in the theoretical analysis. This makes it difficult to see how directly the theorem explains the observed optimization behavior. I see it more as an interesting illustrative analysis that could be moved to the appendix, rather than a major theoretical contribution.

Propositions: Proposition 1 is rather standard, and Proposition 3 is also expected. Proposition 2, on the other hand, provides an interesting insight.

Gradient boundedness: The authors state that the boundedness of the divergence under feature normalization prevents unbounded repulsive gradients. However, a bounded function does not necessarily have bounded gradients. This argument needs a clarifying justification. The authors should either derive a proper gradient bound under explicit assumptions or weaken the statement.

In general, none of the theoretical results establishes that maximizing CSD removes memorized information, guarantees approximation to retraining, or produces the proposed threshold in actual neural networks. I understand that the paper addresses approximate unlearning, but the theoretical contribution and its limitations should be framed more carefully.

  • MIA results

Insufficient stronger MIA evaluation: The stronger LiRA-style attack is evaluated only on a subset of the experimental settings. I do not think the simple SVC attack is sufficiently strong to be the main privacy evaluation and it cannot be used to conclude that the method is forgetting the instances. I would like to see LiRA-style results across the main experiments, rather than relying primarily on the SVC metric.

Additionally, the shadow models appear to be trained for only 50 epochs, whereas the original models are trained for longer. Why is this the case? Could this mismatch affect the reliability of the attack? Please clarify.

Missing representation-level forgetting evaluation: This is an important concern given the paper's motivation. The authors repeatedly emphasize forgetting in representation space, but they do not directly evaluate whether information has actually been forgotten from these representations. A representation change does not necessarily mean that memorized information has been removed. I would like to see an independent evaluation, such as feature-based membership inference attacks or other representation-level information-leakage measures, ideally compared against the retrained model.

  • General comments

Framing around fine-tuning: The motivation and experimental narrative focus heavily on improving fine-tuning. However, many existing unlearning approaches already go beyond FT and address its limitations. While improving FT is valuable, I think the authors should better position their contribution relative to these methods and clarify whether CSD provides a general improvement or mainly addresses the specific under-forgetting behavior of FT. I also think that it is very nice that the authors take CSD as a potential plug-in to existing unlearning methods. I encourage the authors for future iterations to add more results on this "plug-in" nature as it is very interesting and adds a lot of value to the community!

Assumption about retrained representations: The authors state: "A retrained model keeps retain representations essentially unchanged while forget representations are re-encoded from the retained structure."

Why should this be the case? An independently retrained model may learn substantially different representations, even if its predictions remain similar. I would like to see a direct comparison between the original, retrained, and unlearned models in representation space to support this assumption.

Minor — Figure 1: The figure would be better for the reader with a more descriptive caption. Some elements also become pixelated when zooming in, and the placement of the "Repulsive force" label could be improved for readability.

Questions for authors

Please see the weaknesses section and clarify or add the new experimental results of each point I raised.

12 comments

We appreciate the reviewer’s detailed and technically demanding assessment. We agree with several of the points, particularly that the original paper over-emphasized CSD-specific geometry, that the utility trade-off required a stronger control, and that representation displacement should not be equated with proof of information erasure. The new experiments materially change how we intend to position the paper.

1. Test-accuracy degradation and fair operating-point comparison.

We agree that the best-gap CSD configuration sacrifices substantial TA. We therefore added a utility-matched experiment on CIFAR-100/ResNet-18. With β=0.5\beta=0.5 and the FT learning rate,

TACSD=76.313±0.495\mathrm{TA}_{\mathrm{CSD}}=76.313\pm0.495,

compared with

TAFT=76.750±0.430\mathrm{TA}_{\mathrm{FT}}=76.750\pm0.430

under the same retain-only BN recalibration, while

Avg.GapCSD=1.959±0.148\mathrm{Avg.Gap}_{\mathrm{CSD}}=1.959\pm0.148

versus

Avg.GapFT=3.003±0.178\mathrm{Avg.Gap}_{\mathrm{FT}}=3.003\pm0.178.

Thus, CSD can improve retrain matching without relying on a large additional utility sacrifice. We will add this comparison and explicitly present CSD as an operating-point trade-off rather than a method that uniformly improves every metric.

We also audited BN fairness. On the same setting, applying the same retain-only BN recalibration to all directly compared BN-based methods gives

Avg.GapFT=3.003±0.178\mathrm{Avg.Gap}_{\mathrm{FT}}=3.003\pm0.178,

Avg.GapCT=1.453±0.205\mathrm{Avg.Gap}_{\mathrm{CT}}=1.453\pm0.205,

Avg.GapCSD=1.435±0.287\mathrm{Avg.Gap}_{\mathrm{CSD}}=1.435\pm0.287,

and

Avg.GapMSG=10.371±0.417\mathrm{Avg.Gap}_{\mathrm{MSG}}=10.371\pm0.417.

All learnable parameters remain unchanged during this post-hoc recalibration. These results make the comparison more nuanced: CT and CSD are essentially comparable in aggregate retrain-matching gap, with CT preserving test utility more strongly and CSD matching the retrained UA/MIA targets more closely. We will revise the text accordingly rather than imply uniform dominance.

2. Simple feature-repulsion baselines and what is CSD-specific.

We performed the requested comparison against cosine and squared-Euclidean forget-feature repulsion, using the same normalized representations, retain-side anchor, learning rate, and 50-epoch protocol.

Over three seeds,

Avg.GapCSD=1.079±0.142\mathrm{Avg.Gap}_{\mathrm{CSD}}=1.079\pm0.142,

Avg.Gapcosine=0.607±0.124\mathrm{Avg.Gap}_{\mathrm{cosine}}=0.607\pm0.124,

and

Avg.Gapℓ22=0.617±0.226\mathrm{Avg.Gap}_{\ell_2^2}=0.617\pm0.226.

The simple paired objectives are stronger in this setting. We regard this as an important negative result, and agree that the empirical improvement should not be attributed uniquely to CSD’s angular geometry.

For normalized features,

∣u−v∣22=2(1−cos⁡(u,v))|u-v|_2^2=2(1-\cos(u,v)),

so cosine and squared-Euclidean repulsion belong to essentially the same paired family. The distinction we retain for CSD is narrower: it provides a set-level distributional discrepancy and does not require explicit instance correspondence.

We tested this directly by removing all same-instance current/reference pairs. In the strict disjoint-batch variant, the current network receives BfB_f while the frozen reference receives a non-overlapping Bf′B_f'. Over five seeds,

Avg.Gapstandard=1.200±0.196\mathrm{Avg.Gap}_{\mathrm{standard}}=1.200\pm0.196

and

Avg.Gapdisjoint=1.411±0.428\mathrm{Avg.Gap}_{\mathrm{disjoint}}=1.411\pm0.428.

Thus, same-instance terms provide useful signal, but they are not required for CSD to remain effective. We will therefore narrow the novelty claim to a pair-free set-level realization of feature repulsion with an RKHS angular interpretation, rather than claim that CSD is intrinsically the strongest repulsion objective.

3. Theoretical claims and correspondence with actual training.

We agree that Theorem 1 is intentionally simple and that the threshold itself is not unique to CSD. A similar competition between quadratic anchoring and repulsion can arise with other discrepancy functions. We will therefore not present the existence of a threshold as a distinctive CSD advantage.

To assess whether the qualitative behavior appears in the actual network, we added a trajectory diagnostic on CIFAR-100/ResNet-18 with the implemented adaptive median bandwidth.

For β=0.3\beta=0.3, the forget-side divergence remains near DCSf≈0.015D_{\mathrm{CS}}^f\approx0.015 throughout all 50 epochs.

For β=1\beta=1, it remains near the low-discrepancy regime for much of training and then activates late, reaching approximately 0.4150.415.

For β=3\beta=3, it activates within the first few epochs and stabilizes around 0.650.65--0.690.69.

This gives three qualitatively different temporal regimes: inactive, delayed activation, and early activation.

Importantly, the cap is never active in any recorded training step:

Pr⁡(DCSf>G)=0\Pr(D_{\mathrm{CS}}^f>G)=0,

with

max⁡DCSf=0.740≪G=5\max D_{\mathrm{CS}}^f=0.740\ll G=5.

Thus, the empirical transition is not caused by clipping. We will explicitly characterize GG as a numerical safeguard in this normalized regime and Theorem 1 as an illustrative one-dimensional surrogate rather than a quantitative theory of full-network optimization.

We also sampled the total parameter-gradient norm during these trajectories. Its magnitude changes qualitatively when the β=1\beta=1 and β=3\beta=3 runs leave the low-discrepancy regime, although we emphasize that this is the total optimization gradient rather than an isolated CSD-gradient measurement.

We also agree with the reviewer’s gradient-boundedness criticism. Boundedness of the divergence does not by itself imply bounded parameter gradients. We will remove that statement and claim only the bounded divergence established by Proposition 4 unless additional assumptions are imposed.

4. ViT architecture, optimizer, and scalability.

We agree that the original manuscript underspecified the transformer architecture. The model used in our experiments is not a standard ViT-Tiny/Small/Base configuration. It is a small-image Vision Transformer using Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA).

For 32×3232\times32 CIFAR inputs, the model uses 4×44\times4 patches, a 512-dimensional token embedding, 6 Transformer blocks, a 1024-dimensional feed-forward layer, dropout 0.10.1, and 15,963,210 trainable parameters (15.96M). The positional embedding contains 64 patch tokens plus one class token. We will add the full architectural specification to the implementation section so that the transformer setting is unambiguous.

We agree that this does not constitute a ViT-Base-scale experiment. We therefore will not use these experiments to make a large-scale transformer claim. The intended conclusion is limited to the tested 15.96M-parameter small-image SPT-LSA transformer.

We also tested whether the reported ViT behavior is merely caused by switching from AdamW to SGD during unlearning. On CIFAR-100/ViT, using AdamW for both FT and CSD gives

Avg.GapFT+AdamW=16.046±0.221\mathrm{Avg.Gap}_{\mathrm{FT+AdamW}}=16.046\pm0.221

and

Avg.GapCSD+AdamW=12.619±0.476\mathrm{Avg.Gap}_{\mathrm{CSD+AdamW}}=12.619\pm0.476,

corresponding to a 21.4% reduction. The relative CSD effect therefore persists under the optimizer used to train the original/retrained ViT.

However, CSD+AdamW still strongly overshoots the retrained MIA target, with 99.053±0.32399.053\pm0.323 versus the retrained target 56.8656.86. We therefore will not claim that AdamW removes the ViT instability. The more accurate conclusion is that the tested transformer has an optimizer-dependent and narrow operating range.

We have not added ViT-Base during the rebuttal. We agree that larger-scale transformers would be informative, but the current evidence does not support such a scalability claim. We will make this scope explicit rather than extrapolate from the tested 15.96M-parameter architecture.

5. Corrected ViT references.

The reviewer’s concern about the transformer experiments is also affected by an evaluation bookkeeping issue identified during the review process. The stored reference fields for CIFAR-100/ViT and Tiny ImageNet/ViT mixed forget-set accuracy with our convention

UA=100−Acc(Df)\mathrm{UA}=100-\mathrm{Acc}(D_f).

After correcting the references, CIFAR-100/ViT uses

UA∗=38.440\mathrm{UA}^*=38.440

and Tiny ImageNet/ViT uses

UA∗=56.190\mathrm{UA}^*=56.190.

The corrected Avg.Gap results are

CIFAR-100/ViT:18.742±0.124→12.600±1.147\mathrm{CIFAR\text{-}100/ViT}: 18.742\pm0.124 \rightarrow 12.600\pm1.147,

and

TinyImageNet/ViT:9.565±0.281→7.499±0.638\mathrm{TinyImageNet/ViT}: 9.565\pm0.281 \rightarrow 7.499\pm0.638.

Thus, the numerical values in the submitted version require correction, but the narrower conclusion that CSD reduces Avg.Gap in these tested ViT settings remains. We will correct all affected tables and figures.

6. Contrastive Unlearning and broader baseline coverage.

We agree that Contrastive Unlearning is a highly relevant representation-based comparison. We do not want to present a hurried reproduction under a different training and tuning pipeline as if it were a definitive fair comparison.

The current study includes CoUn as the closest contrastive contextual comparison and shared-protocol checks with CT, MSG, SCRUB, and SalUn. However, these do not replace a direct reproduction of Contrastive Unlearning. We will therefore make this missing comparison explicit and remove any wording that could be interpreted as claiming superiority over all representation-space unlearning methods.

We view this as a limitation of comparative completeness rather than evidence that the controlled FT/CSD comparison is invalid.

7. Stronger MIA and representation-level forgetting.

We agree that the confidence-based SVC metric alone cannot establish instance erasure. Our stronger LiRA-style audit on CIFAR-100/ResNet-18 gives

AUCretrain=0.504\mathrm{AUC}_{\mathrm{retrain}}=0.504,

AUCFT=0.571\mathrm{AUC}_{\mathrm{FT}}=0.571,

and

AUCCSD=0.499\mathrm{AUC}_{\mathrm{CSD}}=0.499,

under the same target pool and shadow-model pipeline.

At low false-positive rates, Retrain, FT, and CSD also give different profiles, with CSD remaining close to the retrained reference in the audited setting. We interpret this only as evidence of closer matching to retraining under this attack, not as a privacy guarantee.

The 50-epoch shadow training is identical for all compared targets, so the comparison itself is controlled. We agree, however, that this does not establish attack optimality or replace stronger-MIA coverage across every experimental setting.

More importantly, we agree with the reviewer that representation displacement is not equivalent to information erasure. We will remove language that could suggest otherwise and use the narrower terms representation displacement and matching to the retrained reference. An independent feature-level leakage attack would be valuable additional evidence, but it is not established by the experiments currently reported.

8. Assumption about retrained representations and plug-in framing.

We agree that an independently retrained network need not preserve the original retain representations. The sentence stating that a retrained model “keeps retain representations essentially unchanged” is therefore too strong.

We will rewrite this as a motivating heuristic for the retain anchor rather than an assumption or theoretical property of retraining. The anchor is intended to limit unnecessary movement of retained representations during the unlearning update; it is not intended to reproduce the internal representation of an independently retrained model.

Similarly, our baseline-compatibility experiments indicate that CSD is most useful when the residual failure mode is specifically under-forgetting. For the tested over-forgetting NegGrad+ configuration, adding CSD produces only marginal improvement. We will therefore replace broad “plug-in improvement” language with the narrower claim that CSD provides additional forgetting pressure when the base method still under-forgets.

9. Scope of forgetting scenarios and modalities.

We agree that our empirical scope is narrower than the most general framing might suggest. The current study evaluates random-subset image-classification unlearning, mainly at 10% forgetting, with an additional 50% stress test.

We have not added 2% deletion, class-wise unlearning, language models, or foundation-model experiments during the rebuttal. Rather than imply untested generalization, we will explicitly limit the empirical claims in the abstract, conclusion, and limitations to the tested random-subset image-classification settings and model families.

The 50% experiment already demonstrates that the mechanism has an important operating boundary: retain/forget selectivity weakens substantially at large deletion ratios. We regard broader deletion structures and modalities as important future tests rather than evidence presently established by the paper.

10. Figure 1 and presentation.

We agree with the presentation comments. We will expand the Figure 1 caption so that the three panels and the intended “under-forgetting → representation repulsion → operating regimes” logic can be understood without reconstructing it from the main text. We will also replace rasterized elements with vector-quality graphics and improve the placement of the “Repulsive force” label.

Reviewer_HZ1cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses under-forgetting in approximate machine unlearning. It proposes CSD-Unlearning, which augments a base unlearning objective with two feature-distribution terms: an anchor that keeps retain-set representations close to those of the frozen original model, and a repulsive term that moves forget-set representations away from that model. Both terms use Cauchy–Schwarz divergence. The authors interpret the divergence as an angular discrepancy between kernel mean embeddings, establish a zero-gradient condition at matched deterministic initialization, and analyze a threshold effect in a one-dimensional surrogate. Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet show improved aggregate matching to retrained models relative to fine-tuning, with utility losses and instability in some settings. Additional evaluations include other unlearning baselines, component ablations, an MMD comparison, membership inference attacks, and computational overhead.

Critical strengths

1.The improvement over under-forgetting fine-tuning is supported by several useful controls. The ResNet experiments include five-seed results, a matched-learning-rate comparison, and a shared-protocol reevaluation with split-matched retraining references in Table 14. Together, these provide reasonably convincing evidence that the improvement is not simply an artifact of learning-rate selection or a single favorable run.

2.The method is straightforward to incorporate, and the paper reports its limitations rather than presenting only favorable results. The objective avoids explicit positive/negative pair construction, and the experiments document utility losses, increased variance on ViT, and reduced effectiveness in other forgetting regimes. This makes the paper potentially useful to researchers seeking a practical way to address under-forgetting.

Critical weaknesses

1.Two ViT retraining references show unexplained discrepancies that affect the interpretation of headline results. Under the stated definition of UA, Table 5 implies a forget-set accuracy of 38.44% for the CIFAR-100/ViT retrained model, compared with 61.40% test accuracy. For Tiny ImageNet/ViT, the corresponding values are 56.19% and 44.11%. In a random-subset deletion setting, such large differences between held-out forget examples and test examples require an explanation, particularly if evaluation transforms are consistent. Appendix C.1 explicitly verifies the CIFAR-10/ViT UA but does not provide the same verification for these two settings. Since the reference values directly determine Avg.Gap, this needs to be resolved before the reported ViT improvements can be fully assessed.

2.The evidence for a distinctive advantage of CSD remains limited. The paper does compare against several unlearning methods; the issue is the strength of the comparison rather than the number of baselines. CoUn, the closest related method, uses a different evaluation pipeline, while the shared-protocol SCRUB and SalUn comparison covers one setting. On CIFAR-100/ResNet-18, CSD improves UA and MIA matching but reduces test accuracy from 76.52% for FT to 72.24%, below the retrained reference of 75.56%. It is therefore unclear whether CSD offers a better forgetting–utility trade-off at comparable utility. The single-seed MMD comparison and the qualitative surrogate analysis do not yet establish that the benefit depends specifically on CSD’s angular geometry rather than more general feature repulsion.

Questions for authors

1.Reference verification: Could you reevaluate the CIFAR-100/ViT and Tiny ImageNet/ViT retrained checkpoints using consistent evaluation transforms and report forget-set accuracy, UA, and test accuracy together? What explains the discrepancies identified above? If any reference values change, how do the corresponding headline results change?

2.Utility-matched comparison: Do the existing hyperparameter sweeps contain configurations showing that CSD improves retrain matching over FT and other directly comparable baselines at similar test accuracy? In particular, does the advantage remain when test accuracy is constrained to stay close to that of the retrained model? This would help distinguish an improved trade-off from a stronger willingness to sacrifice utility.

3.Evidence for the proposed mechanism: Do training trajectories support the claimed ignition mechanism under the actual implementation? The surrogate assumes fixed bandwidth and obtains its nonzero optimum at the cap boundary, whereas the implemented bandwidth is adaptive and the cap is usually inactive. Evidence on early divergence, gradient magnitudes, and cap activation across β values would help establish how much of the observed behavior the surrogate explains.

7 comments

We appreciate the reviewer’s focus on three decision-critical questions: whether the ViT references are valid, whether CSD improves the forgetting–utility trade-off at comparable utility, and whether the proposed activation picture is visible in the actual network optimization. We performed targeted checks for each and have also narrowed the claims where the new evidence does not support the broader interpretation.

1. ViT reference verification.

We re-evaluated the CIFAR-100/ViT and Tiny ImageNet/ViT retrained checkpoints with consistent deterministic evaluation transforms. The previous reference table mixed forget-set accuracy with our UA convention, UA=100−Acc(Df)\mathrm{UA}=100-\mathrm{Acc}(D_f).

The corrected reference values are:

CIFAR-100/ViT: ForgetAcc=61.560\mathrm{ForgetAcc}=61.560, UA=38.440\mathrm{UA}=38.440, and TA=61.400\mathrm{TA}=61.400.

Tiny ImageNet/ViT: ForgetAcc=43.810\mathrm{ForgetAcc}=43.810, UA=56.190\mathrm{UA}=56.190, and TA=44.112\mathrm{TA}=44.112.

After correction, CIFAR-100/ViT changes from Avg.Gap=18.742±0.124\mathrm{Avg.Gap}=18.742\pm0.124 for FT to 12.600±1.14712.600\pm1.147 for CSD, a 32.8% reduction. Tiny ImageNet/ViT changes from 9.565±0.2819.565\pm0.281 to 7.499±0.6387.499\pm0.638, a 21.6% reduction.

Thus, the original reference fields were incorrect and will be corrected throughout the paper, but the qualitative conclusion that CSD reduces Avg.Gap in these tested ViT settings remains.

2. Utility-matched comparison.

We agree that the original best-gap CSD point alone does not establish an improved trade-off, because it reduces test accuracy substantially below the FT value.

We therefore evaluated a utility-matched CSD operating point at the FT learning rate with β=0.5\beta=0.5. Over five seeds, the result is RA=99.975±0.003\mathrm{RA}=99.975\pm0.003, UA=17.920±0.360\mathrm{UA}=17.920\pm0.360, TA=76.313±0.495\mathrm{TA}=76.313\pm0.495, MIA=47.704±0.422\mathrm{MIA}=47.704\pm0.422, and Avg.Gap=1.959±0.148\mathrm{Avg.Gap}=1.959\pm0.148.

After applying the same retain-only BN recalibration to FT, FT obtains TA=76.750±0.430\mathrm{TA}=76.750\pm0.430 and Avg.Gap=3.003±0.178\mathrm{Avg.Gap}=3.003\pm0.178.

The CSD point is therefore within 0.44 percentage points of FT in TA and within 0.75 points of the retrained target TA∗=75.56\mathrm{TA}^*=75.56, while reducing Avg.Gap by approximately 34.8%.

This directly answers the reviewer’s concern: the benefit is not only a consequence of allowing CSD to sacrifice substantially more utility. We will add this utility-matched result and distinguish it from the minimum-gap operating point.

We also audited the directly comparable CT baseline under the same BN-recalibrated evaluator. CT obtains Avg.Gap=1.453±0.205\mathrm{Avg.Gap}=1.453\pm0.205, while the original formal CSD checkpoints obtain 1.435±0.2871.435\pm0.287 after the same post-hoc recalibration.

We therefore will not claim that CSD uniformly dominates CT. The more accurate interpretation is that the two have similar aggregate retrain-matching gaps but different profiles: CT preserves TA more strongly, while CSD matches UA/MIA more closely.

3. Is the benefit distinctive to CSD’s angular geometry?

We agree that the original evidence was insufficient to establish this stronger claim. We therefore added simple forget-feature repulsion controls using the same normalized representation, retain-side anchor, learning rate, and training budget.

Over three seeds, CSD obtains Avg.Gap=1.079±0.142\mathrm{Avg.Gap}=1.079\pm0.142, cosine repulsion obtains 0.607±0.1240.607\pm0.124, and squared-Euclidean repulsion obtains 0.617±0.2260.617\pm0.226.

Because features are normalized, ∥u−v∥22=2(1−cos⁡(u,v))\|u-v\|_2^2=2(1-\cos(u,v)), so the cosine and squared-Euclidean controls belong to essentially the same paired repulsion family.

These simple paired objectives are stronger than CSD in this setting. We regard this as an important negative result and will revise the paper accordingly: the broad performance benefit comes from adding feature-level repulsive pressure, not from a demonstrated universal superiority of CSD’s angular geometry.

The structural distinction we retain for CSD is that it is a set-level distributional discrepancy that does not require instance correspondence. In a separate strict disjoint-batch experiment, all same-instance current/reference pairs are removed, and CSD remains effective with Avg.Gap=1.411±0.428\mathrm{Avg.Gap}=1.411\pm0.428 over five seeds.

We will therefore narrow the novelty claim to a pair-free set-level formulation with an RKHS angular interpretation, rather than claim that angular CSD is intrinsically the optimal repulsion objective.

4. Evidence for the proposed mechanism in actual training.

We performed the trajectory analysis suggested by the reviewer using the actual CIFAR-100/ResNet-18 implementation with adaptive per-batch bandwidth.

The trajectories show three qualitatively different regimes.

For β=0.3\beta=0.3, DCSfD_{\mathrm{CS}}^f remains near 0.0150.015 throughout all 50 epochs.

For β=1\beta=1, it stays near the low-discrepancy regime for much of training and then activates late, reaching approximately 0.4150.415.

For β=3\beta=3, it activates within the first few epochs and stabilizes around 0.650.65--0.690.69.

Thus, the final hyperparameter sweep corresponds to qualitatively different optimization trajectories rather than merely different terminal scalar values.

We also measured cap activation at every training step. The result is unambiguous: Pr⁡(DCSf>G)=0\Pr(D_{\mathrm{CS}}^f>G)=0, and the maximum observed raw divergence is max⁡DCSf=0.740\max D_{\mathrm{CS}}^f=0.740, far below G=5G=5.

Therefore, the actual transition is not produced by the hard cap. We will state explicitly that the cap acts only as a numerical safeguard in the evaluated normalized regime.

We additionally sampled the total parameter-gradient norm during optimization. The β=0.3\beta=0.3 run decays toward a low-gradient regime, while the β=1\beta=1 and β=3\beta=3 runs show renewed or sustained gradient activity as they leave the low-discrepancy regime. We emphasize, however, that this is the total optimization gradient and not an isolated measurement of the CSD-gradient component.

We therefore agree with the reviewer on the appropriate theoretical scope. Theorem 1 is not a quantitative characterization of the full network with adaptive bandwidth. Its role is to provide an illustrative surrogate for an anchoring–repulsion competition. The new trajectories show that an inactive/delayed/early activation pattern is present in the actual optimization, while also demonstrating that the cap itself is not the empirical mechanism.

5. Revised theoretical positioning.

The simple-repulsion result also clarifies what the theory can and cannot establish. The threshold-like behavior is not unique to CSD; related anchoring–repulsion objectives can exhibit similar competition. We will therefore weaken any language suggesting that the threshold is a uniquely CSD-specific phenomenon.

The CSD-specific analytical content that remains is the RKHS angular interpretation of the empirical estimator and the stationary repulsive gradient under exact current/reference coincidence. The scalar threshold theorem will be presented as a mechanistic surrogate rather than a central performance guarantee.

Taken together, the corrected references, utility-matched comparison, simple-repulsion controls, and actual optimization trajectories directly address the main uncertainties identified in the review. They also motivate a narrower claim that we believe is better supported by the evidence: CSD is a pair-free set-level way to supply representation-level forgetting pressure in an under-forgetting regime, not a universally superior discrepancy or a theoretical guarantee of retraining equivalence. We hope this additional evidence and revised framing support a reassessment of the current recommendation.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address the concerns and their transparency in acknowledging the limitations.

The corrected ViT references largely resolve my concern about the reliability of the reported results. The new utility-matched experiment is also convincing, showing that CSD can improve Avg.Gap over FT without substantially sacrificing test accuracy. I also appreciate the additional CT comparison and optimization trajectories.

However, I still have reservations about the distinctiveness of CSD. In particular, the simpler cosine and squared-Euclidean repulsion objectives outperform CSD in the reported comparison, suggesting that the benefits may come primarily from feature-level repulsion rather than CSD's specific angular geometry. While the pair-free, set-level formulation remains a useful distinction, its advantage over simpler alternatives is not yet sufficiently established.

The new training trajectories provide useful qualitative evidence, but the theoretical threshold remains a simplified surrogate rather than a demonstrated CSD-specific mechanism in actual training. I appreciate the authors' willingness to narrow these claims.

Overall, the rebuttal substantially addresses my original empirical concerns, but I remain unconvinced that the CSD-specific methodological and theoretical contributions are sufficiently strong.

Reviewer_yHdCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the under-forgetting failure mode of retain-set fine-tuning (FT) in approximate machine unlearning, where a model preserves test utility on Dr\mathcal{D}_r while leaving penultimate representations of the forget set Df\mathcal{D}_f close to their memorized state in θ0\theta_0.

Critical strengths
  1. Conceptual Simplicity and Pair-Free Formulation

  2. Candor on Operating Boundaries and Negative Regimes

  3. Strong Membership Inference Auditing and Controlled Discrepancy Ablations

Critical weaknesses
  1. Calculation problem in ViT Retrained Reference Values (Table 5). Unlearning Accuracy (UAUA) is defined throughout the paper as classification error on the forget set, UA=100−Acc(Df)UA = 100 - \text{Acc}(\mathcal{D}_f). Because a model retrained from scratch on Dr\mathcal{D}_r never sees the random forget subset Df\mathcal{D}_f or the test set, both sets act as unseen i.i.d. evaluation data, meaning forget-set error plus test accuracy must sum to roughly 100%100\% (UA∗+TA∗≈100%UA^* + TA^* \approx 100\%). In Table 5, this identity holds across all ResNet-18 benchmarks and for CIFAR-10/ViT (UA∗=12.18%,TA∗=87.39%UA^* = 12.18\%, TA^* = 87.39\%), which the authors explicitly re-verified in Appendix C.1 by converting its 87.82%87.82\% forget-set accuracy into 12.18%12.18\% error. However, for Tiny ImageNet/ViT (UA∗=43.81%,TA∗=44.11%UA^* = 43.81\%, TA^* = 44.11\%) and CIFAR-100/ViT (UA∗=61.56%,TA∗=61.40%UA^* = 61.56\%, TA^* = 61.40\%), the authors left the original pipeline values untouched, where $UA^* $ nearly equals $TA^* $ because the pipeline accidentally recorded forget-set accuracy instead of forget-set error. Consequently, the true retrained targets should be UA∗≈56.19%UA^* \approx 56.19\% for Tiny ImageNet/ViT and UA∗≈38.44%UA^* \approx 38.44\% for CIFAR-100/ViT, meaning all headline ViT Avg.Gap numbers in Table 1, Table 6, Table 9, Figure 2b, and Figure 3 are miscalculated against uninverted reference targets.

  2. The problem Between Theorem 1 and the Actual Algorithm. Theorem 1 attempts to explain the empirical "sharp ignition threshold" using a 1D scalar surrogate, but its derivation in Appendix A.5 removes the two core mathematical properties of CSD-Unlearning: multi-sample kernel estimation and ℓ2\ell_2 feature normalization. Specifically, the proof sets the sample size to a single point (n=m=1n = m = 1), where the Cauchy-Schwarz divergence reduces to simple squared Euclidean distance (x22σ2\frac{x^2}{2\sigma^2}) and no longer represents a distribution-level angular divergence. Furthermore, the proof drops the unit-sphere normalization constraint and relies on the hard cap GG to halt feature displacement at a boundary solution xG=σ2Gx_G = \sigma\sqrt{2G}. In the actual algorithm, ℓ2\ell_2 normalization naturally bounds the divergence by 2/σ22/\sigma^2 (Proposition 4), causing the empirical forget-set divergence to saturate around 0.690.69, well below the cap GG. As shown in Table 8, varying G∈{2,5,10}G \in \{2, 5, 10\} produces identical numerical results (0.68760.6876), demonstrating that Theorem 1 attributes saturation to an inactive hyperparameter rather than the unit-sphere geometry that actually governs optimization.

  3. Hidden Sample Pairing Inside the "Pair-Free" Objective Although the paper positions CSD-Unlearning as a purely "pair-free," distribution-level method, Algorithm 1 computes the forget repulsion $D_{CS}(\Phi_\theta(B_f), \Phi_{\theta_0 }(B_f )) $ by evaluating both the current model θ\theta and the frozen reference model $\theta_0 $ on the exact same mini-batch $B_f $. Consequently, in the cross-kernel matrix $\frac{1}{m^2}\sum_{i,j} k_\sigma(z_i, z_j^0) $ of Equation (6), the diagonal entries (i=ji = j) directly compute instance-paired self-similarity $k_\sigma(\Phi_\theta(x_i), \Phi_{\theta_0}(x_i)) $ between each sample and its own frozen representation. Because the forget batch size is m=64m = 64 while CIFAR-100 and Tiny ImageNet contain 100100 and 200200 classes, most off-diagonal pairs (i≠ji \neq j) belong to different classes and are already nearly orthogonal at initialization, meaning the initial repulsive gradient is heavily driven by the 11-to-11 paired diagonal entries. since random 50%50\% retain and forget splits follow the exact same underlying data distribution, pushing the marginal forget distribution away from θ0\theta_0 inevitably perturbs the retain distribution to the same degree (DCSr=1.024D_{\text{CS}}^r = 1.024 vs. DCSf=1.014D_{\text{CS}}^f = 1.014).

  4. Steep Test Accuracy Drop and Evaluation Inconsistencies While CSD-Unlearning improves forgetting metrics (UA and MIA), it incurs a substantial drop in Test Accuracy (TA) and relies on inconsistent evaluation protocols across baselines. On CIFAR-100 (ResNet-18, 10%), CSD-Unlearning reduces test accuracy to 72.24%72.24\%, which is 3.32%3.32\% below the retrained reference (75.56%75.56\%) and 5.64%5.64\% below CoUn (77.88%77.88\%), and on Tiny ImageNet (ResNet-18, 10%) it drops test accuracy 2.14%2.14\% below the retrained model (61.68%61.68\% vs. 63.82%63.82\%).

Questions for authors
  1. Correction of ViT Reference $UA^* $ in Table 5: Can you confirm that the retrained reference values UA∗=43.81%UA^* = 43.81\% (Tiny ImageNet/ViT) and UA∗=61.56%UA^* = 61.56\% (CIFAR-100/ViT) in Table 5 are indeed uncomplemented forget-set accuracies Acc(Df)\text{Acc}(\mathcal{D}_f) rather than forget-set errors 100−Acc(Df)100 - Acc(D_f), and provide the updated Avg.Gap values for Tables 1, 6, and 9 using the corrected UA∗=56.19%UA^* = 56.19\% and UA∗=38.44%UA^* = 38.44\%?

  2. Disjoint-Batch Ablation for $D_{CS}(Z_f, Z_f^0) $: If you compute Zf=Φθ(Bf)Z_f = \Phi_\theta(B_f) and $Z_f^0 = \Phi_{\theta_0}(B_f') $ using two independent forget mini-batches Bf,Bf′⊂DfB_f, B_f' \subset \mathcal{D}_f at each step (removing the i=ji=j same-sample diagonal terms kσ(Φθ(xi),Φθ0(xi))k_\sigma(\Phi_\theta(x_i), \Phi_{\theta_0}(x_i)) from the cross-kernel matrix), does forgetting still ignite, or does the method rely on implicit instance-level pairing along the Gram matrix diagonal?

  3. BN Recalibration Parity in Main Tables: In Tables 1, 2, and 4, was the 50-batch retain-only BN recalibration (Algorithm 1) applied strictly to FT+CSD, or was it also applied to plain FT, CT, MSG, and CoUn?

  4. Unlearning ViT with AdamW: Have you tested running CSD-Unlearning on ViT using the native AdamW optimizer (matching the original/retrained ViT recipe) to see whether the narrow stability window and ±9.41%\pm 9.41\% seed variance on ViT are artifacts of switching from AdamW to SGD during unlearning?

6 comments

We appreciate the reviewer’s careful audit. Several of the concerns identify real issues in our presentation or evaluation protocol, and we have used them to substantially tighten both the evidence and the scope of the paper. Importantly, the additional checks do not change the central empirical conclusion: CSD supplies useful forgetting pressure when FT under-forgets, but its operating point is optimization-dependent and should not be interpreted as a universal or theory-certified unlearning mechanism.

1. ViT retrained-reference correction.

The reviewer is correct that the stored ViT reference fields for CIFAR-100 and Tiny ImageNet mixed forget-set accuracy with our UA convention, UA=100−Acc(Df)\mathrm{UA}=100-\mathrm{Acc}(D_f).

We re-evaluated the retrained checkpoints with consistent deterministic evaluation transforms. The corrected references are:

CIFAR-100/ViT: ForgetAcc=61.560\mathrm{ForgetAcc}=61.560, UA=38.440\mathrm{UA}=38.440, and TA=61.400\mathrm{TA}=61.400.

Tiny ImageNet/ViT: ForgetAcc=43.810\mathrm{ForgetAcc}=43.810, UA=56.190\mathrm{UA}=56.190, and TA=44.112\mathrm{TA}=44.112.

Using these corrected targets, the CIFAR-100/ViT Avg.Gap becomes 18.742±0.12418.742\pm0.124 for FT and 12.600±1.14712.600\pm1.147 for CSD, corresponding to a 32.8% reduction. For Tiny ImageNet/ViT, the corrected values are 9.565±0.2819.565\pm0.281 for FT and 7.499±0.6387.499\pm0.638 for CSD, corresponding to a 21.6% reduction.

We will correct all affected tables and figures and explicitly state that the previous values resulted from a metric-orientation bookkeeping error.

2. Scope of Theorem 1 and the inactive cap.

We agree that the scalar theorem should not be read as a derivation of the full multi-sample, normalized neural-network dynamics. In particular, the one-sample reduction removes the distribution-level character of the estimator, and the scalar surrogate drops the unit-sphere constraint used by the actual implementation.

We will therefore revise the role of Theorem 1. Its intended purpose is to provide a one-dimensional mechanistic surrogate for the competition between local anchoring and repulsive displacement, not a quantitative prediction of the full-network threshold or saturation value.

We additionally ran the requested actual-training trajectory diagnostic on CIFAR-100/ResNet-18 using the implemented adaptive median bandwidth. The trajectories show three qualitatively different regimes. For β=0.3\beta=0.3, DCSfD_{\mathrm{CS}}^f remains near 0.0150.015 throughout all 50 epochs. For β=1\beta=1, it stays close to the low-discrepancy regime for much of training and then activates late, reaching approximately 0.4150.415. For β=3\beta=3, it rises within the first few epochs and stabilizes around 0.650.65--0.690.69.

Crucially, the cap is never activated in any recorded training step: Pr⁡(DCSf>G)=0\Pr(D_{\mathrm{CS}}^f>G)=0, while the maximum observed raw divergence is max⁡DCSf=0.740≪G=5\max D_{\mathrm{CS}}^f=0.740\ll G=5.

Thus, the empirical activation is not caused by clipping. We will explicitly describe the cap as a numerical safeguard in this normalized regime and avoid attributing the observed plateau to the capped surrogate.

3. Does CSD rely on hidden same-instance pairing?

We directly tested this concern. In a strict disjoint-batch variant, the current model receives BfB_f while the frozen reference receives an independently sampled, non-overlapping Bf′B_f'. Therefore no current/reference identity match appears in the cross-kernel matrix.

In a controlled five-seed rerun, standard CSD obtains Avg.Gap=1.200±0.196\mathrm{Avg.Gap}=1.200\pm0.196, while disjoint-batch CSD obtains 1.411±0.4281.411\pm0.428.

The disjoint variant is somewhat worse, so same-instance terms can provide useful signal, but the forgetting effect clearly persists after all identity correspondence is removed. We will therefore sharpen our use of “pair-free”: CSD does not require explicit positive/negative pair construction or instance correspondence, rather than implying that evaluating the same minibatch has no effect.

4. BN-recalibration parity and evaluation fairness.

The reviewer is also correct that BN post-processing should be applied consistently across directly comparable BN-based methods. We therefore re-evaluated the existing five checkpoints of FT, CT, MSG, and CSD using the same retain-only BN recalibration. This operation changes no learnable parameters; we verified their hashes remain unchanged.

On CIFAR-100/ResNet-18 with 10% forgetting, the revised Avg.Gap values are:

FT: 3.003±0.1783.003\pm0.178.

CSD: 1.435±0.2871.435\pm0.287.

CT: 1.453±0.2051.453\pm0.205.

MSG: 10.371±0.41710.371\pm0.417.

The correction improves FT and CT and slightly worsens MSG, so BN recalibration is not a CSD-specific advantage. It also makes the comparison with CT more nuanced: CSD and CT have essentially comparable aggregate retrain-matching gaps, but different metric profiles. CT preserves test utility more strongly, while CSD matches the retrained UA and MIA targets more closely.

We will revise the comparison accordingly and apply the same BN-recalibration protocol consistently in the directly shared pipeline.

5. Test-accuracy trade-off and utility-matched comparison.

We agree that the original best-gap CSD point pays a substantial TA cost. To test whether the improvement requires this utility sacrifice, we added a utility-matched CSD operating point at the FT learning rate with β=0.5\beta=0.5.

Over five seeds, CSD obtains RA=99.975±0.003\mathrm{RA}=99.975\pm0.003, UA=17.920±0.360\mathrm{UA}=17.920\pm0.360, TA=76.313±0.495\mathrm{TA}=76.313\pm0.495, MIA=47.704±0.422\mathrm{MIA}=47.704\pm0.422, and Avg.Gap=1.959±0.148\mathrm{Avg.Gap}=1.959\pm0.148.

Under the same BN-recalibrated evaluation, FT has TA=76.750±0.430\mathrm{TA}=76.750\pm0.430 and Avg.Gap=3.003±0.178\mathrm{Avg.Gap}=3.003\pm0.178.

Thus, the utility-matched CSD point is within 0.44 percentage points of FT in TA and within 0.75 points of the retrained TA target, while still reducing Avg.Gap by approximately 34.8%. This shows that the improvement is not solely obtained by accepting a larger utility loss.

We will add this operating-point comparison and make the utility–forgetting trade-off more explicit.

6. ViT unlearning with AdamW.

We additionally repeated CIFAR-100/ViT unlearning with AdamW for both FT and CSD. Over three seeds, FT+AdamW obtains Avg.Gap=16.046±0.221\mathrm{Avg.Gap}=16.046\pm0.221, while CSD+AdamW obtains 12.619±0.47612.619\pm0.476, a 21.4% reduction.

Therefore, the relative CSD effect does not disappear when unlearning uses the optimizer employed for the original/retrained ViT models.

However, the CSD+AdamW run strongly overshoots the retrained MIA target, with 99.053±0.32399.053\pm0.323 versus 56.8656.86. We therefore do not claim that AdamW resolves the ViT instability. The more accurate conclusion is that the tested transformer has an optimizer-dependent operating point, rather than the original SGD result being purely an optimizer artifact.

Overall, the reviewer’s concerns led us to correct the ViT references, unify BN evaluation, directly test identity pairing, and substantially narrow the theoretical interpretation. We believe these revisions resolve the main soundness questions while preserving the central empirical observation that set-level repulsive pressure can address under-forgetting. We hope the corrected evidence supports a reassessment of the current recommendation.

Reviewer yHdC,

We have now submitted the revised manuscript and incorporated the issues raised in your review directly into the paper.

The main revisions corresponding to your concerns are:

  1. ViT reference correction. We re-evaluated the CIFAR-100/ViT and Tiny ImageNet/ViT retrained references and corrected the UA convention. All affected headline Avg.Gap values, tables, figures, and discussion have been updated. The corrected five-seed results remain favorable to CSD: CIFAR-100/ViT changes from 18.742±0.12418.742\pm0.124 for FT to 12.600±1.14712.600\pm1.147 for CSD, and Tiny ImageNet/ViT from 9.565±0.2819.565\pm0.281 to 7.499±0.6387.499\pm0.638.

  2. Pair-free formulation. We now define “pair-free” precisely as requiring no explicit positive/negative pair construction or identity correspondence. We added a strict disjoint-batch experiment in which the current and frozen networks receive non-overlapping forget batches. CSD remains effective, with Avg.Gap 1.411±0.4281.411\pm0.428 versus 1.200±0.1961.200\pm0.196 for the corresponding standard controlled rerun. Thus, same-instance terms are useful but not necessary.

  3. BN evaluation parity. We re-evaluated the directly comparable CIFAR-100/ResNet-18 checkpoints with the same retain-only BN recalibration. Under this common protocol, Avg.Gap is 3.003±0.1783.003\pm0.178 for FT, 1.453±0.2051.453\pm0.205 for CT, 1.435±0.2871.435\pm0.287 for CSD, and 10.371±0.41710.371\pm0.417 for MSG. The revised manuscript no longer treats BN recalibration as a CSD-specific advantage.

  4. Utility trade-off. We added a utility-matched CSD operating point. It obtains TA 76.313±0.49576.313\pm0.495 versus 76.750±0.43076.750\pm0.430 for FT, while reducing Avg.Gap from 3.003±0.1783.003\pm0.178 to 1.959±0.1481.959\pm0.148. This shows that the gain does not require the large TA drop of the minimum-gap operating point.

  5. ViT optimizer control. We added a three-seed AdamW control. FT+AdamW gives Avg.Gap 16.046±0.22116.046\pm0.221, while CSD+AdamW gives 12.619±0.47612.619\pm0.476. We also report the remaining MIA overshoot, so the revised claim is that the relative effect persists under AdamW, not that optimizer choice removes ViT instability.

  6. Theory/algorithm correspondence. The scalar theorem is now explicitly presented as a qualitative anchoring--repulsion surrogate rather than a quantitative theory of the implemented network. We added actual training trajectories showing inactive, delayed-activation, and early-activation regimes, and verified that the explicit cap is never activated in the recorded runs. We also removed the incorrect implication that bounded divergence guarantees bounded parameter gradients.

These changes materially tighten the paper, but they do not alter its central empirical conclusion: explicit set-level repulsive pressure can address the under-forgetting failure mode of retain-only fine-tuning, with identifiable operating limits.

Your review focused primarily on correctness, protocol consistency, and the correspondence between the stated mechanism and the actual implementation. Those points are now directly addressed in the revised manuscript. We would therefore ask that the revised version be assessed on the corrected evidence, and that the score be reconsidered if the soundness concerns motivating the original weak-reject assessment are resolved.

Thanks!

Yours Sincerely,

Submission 10823 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.