- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CSD-Unlearning, a machine unlearning method that uses Cauchy–Schwarz divergence to improve forgetting. The approach combines two objectives: keeping representations of retained samples close to those of the original model and pushing representations of forgotten samples away from their original state. They motivate this saying that previous methods, in particular fine-tuning, “underforget”. The paper provides a theoretical analysis showing that the repulsive gradient vanishes at initialization (as the model has not moved yet), and they identify a threshold-like behavior in a simplified optimization setting. Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet with ResNet-18 and Vision Transformers show that adding CSD to fine-tuning can bring the model closer to a retrained reference.
Critical strengths
-
The motivation is clear and puts the focus on the representation space that methods usually do not take a look at in approximate unlearning, with a nice distribution-level approach.
-
Simple and flexible: The CSD objective can be integrated into other existing unlearning methods without additional adaptations.
-
Interesting theoretical analysis: The paper provides a geometric interpretation of CSD in representation space and investigates the optimization dynamics, with a threshold-like effect in a simplified setting.
Critical weaknesses
- Experiments
ViT architecture and scalability: Which ViT architecture is used? Tiny, Small, Base? This is not clear from the paper. The authors should specify the architecture and number of parameters. I would also like to see experiments with larger models, at least ViT-Base, to better assess scalability.
Test accuracy degradation: The proposed method seems to degrade test accuracy considerably more than some of the other approximate unlearning baselines. The authors should discuss this trade-off more explicitly, explain why it happens, and explain how it could be alleviated.
- Missing baselines
Contrastive Unlearning: The paper cites Contrastive Unlearning: A Contrastive Approach to Machine Unlearning, but does not experimentally compare against it. This is a particularly relevant baseline, as it also directly modifies forget-set representations. I would expect a comparison to understand whether CSD actually provides an improvement over existing representation-based methods.
Limited comparisons with existing methods: In general, improving over fine-tuning alone is not enough, given the large body of work developing stronger unlearning methods. While the authors include other baselines, these comparisons are limited. CoUn is evaluated using a different pipeline, the SCRUB and SalUn reproductions perform surprisingly poorly, and the MMD comparison uses only a single seed. Moreover, several comparisons appear only in the appendix. I would like to see more comprehensive, fair, shared-protocol comparisons in the main results.
Missing simple repulsion baselines: Another important missing ablation is a comparison against simpler feature-level repulsion objectives, such as cosine distance or squared Euclidean distance between the original and current forget-set representations. It is not clear how much benefit comes specifically from using CSD rather than simply pushing the representations apart. This comparison seems particularly important to establish the contribution of the proposed objective.
- Limited scope
Forgetting scenarios: Almost all experiments use 10% random-subset forgetting, with 50% evaluated only in a limited setting. I would like to see additional forgetting ratios, including smaller ones (e.g., 2%), which can be relevant to practical deletion requests. More importantly, class-wise unlearning should also be evaluated. Currently, the experiments are too homogeneous to assess how well the method generalizes to different forgetting scenarios.
Limited to image classification: All experiments are conducted on relatively small image-classification benchmarks. Given the general framing of the proposed method, I would like to see results on other modalities, particularly language models. Otherwise, the authors should make the scope of their empirical claims clearer and explicitly limit their conclusions to image-classification unlearning.
- Theoretical results
Theorem 1: I find this result rather weak. The ignition threshold essentially follows from a sign change in the coefficient of a quadratic objective, and it does not seem to establish anything specific to Cauchy–Schwarz divergence. A simple squared-distance repulsion competing against a quadratic regularizer would lead to a similar threshold.
Moreover, the theorem studies a highly simplified optimization problem, whereas the actual method operates on high-dimensional neural-network representations. The cap also appears inactive in the reported experiments, unlike in the theoretical analysis. This makes it difficult to see how directly the theorem explains the observed optimization behavior. I see it more as an interesting illustrative analysis that could be moved to the appendix, rather than a major theoretical contribution.
Propositions: Proposition 1 is rather standard, and Proposition 3 is also expected. Proposition 2, on the other hand, provides an interesting insight.
Gradient boundedness: The authors state that the boundedness of the divergence under feature normalization prevents unbounded repulsive gradients. However, a bounded function does not necessarily have bounded gradients. This argument needs a clarifying justification. The authors should either derive a proper gradient bound under explicit assumptions or weaken the statement.
In general, none of the theoretical results establishes that maximizing CSD removes memorized information, guarantees approximation to retraining, or produces the proposed threshold in actual neural networks. I understand that the paper addresses approximate unlearning, but the theoretical contribution and its limitations should be framed more carefully.
- MIA results
Insufficient stronger MIA evaluation: The stronger LiRA-style attack is evaluated only on a subset of the experimental settings. I do not think the simple SVC attack is sufficiently strong to be the main privacy evaluation and it cannot be used to conclude that the method is forgetting the instances. I would like to see LiRA-style results across the main experiments, rather than relying primarily on the SVC metric.
Additionally, the shadow models appear to be trained for only 50 epochs, whereas the original models are trained for longer. Why is this the case? Could this mismatch affect the reliability of the attack? Please clarify.
Missing representation-level forgetting evaluation: This is an important concern given the paper's motivation. The authors repeatedly emphasize forgetting in representation space, but they do not directly evaluate whether information has actually been forgotten from these representations. A representation change does not necessarily mean that memorized information has been removed. I would like to see an independent evaluation, such as feature-based membership inference attacks or other representation-level information-leakage measures, ideally compared against the retrained model.
- General comments
Framing around fine-tuning: The motivation and experimental narrative focus heavily on improving fine-tuning. However, many existing unlearning approaches already go beyond FT and address its limitations. While improving FT is valuable, I think the authors should better position their contribution relative to these methods and clarify whether CSD provides a general improvement or mainly addresses the specific under-forgetting behavior of FT. I also think that it is very nice that the authors take CSD as a potential plug-in to existing unlearning methods. I encourage the authors for future iterations to add more results on this "plug-in" nature as it is very interesting and adds a lot of value to the community!
Assumption about retrained representations: The authors state: "A retrained model keeps retain representations essentially unchanged while forget representations are re-encoded from the retained structure."
Why should this be the case? An independently retrained model may learn substantially different representations, even if its predictions remain similar. I would like to see a direct comparison between the original, retrained, and unlearned models in representation space to support this assumption.
Minor — Figure 1: The figure would be better for the reader with a more descriptive caption. Some elements also become pixelated when zooming in, and the placement of the "Repulsive force" label could be improved for readability.
Questions for authors
Please see the weaknesses section and clarify or add the new experimental results of each point I raised.
12 comments
3. Theoretical claims and correspondence with actual training.
We agree that Theorem 1 is intentionally simple and that the threshold itself is not unique to CSD. A similar competition between quadratic anchoring and repulsion can arise with other discrepancy functions. We will therefore not present the existence of a threshold as a distinctive CSD advantage.
To assess whether the qualitative behavior appears in the actual network, we added a trajectory diagnostic on CIFAR-100/ResNet-18 with the implemented adaptive median bandwidth.
For , the forget-side divergence remains near throughout all 50 epochs.
For , it remains near the low-discrepancy regime for much of training and then activates late, reaching approximately .
For , it activates within the first few epochs and stabilizes around --.
This gives three qualitatively different temporal regimes: inactive, delayed activation, and early activation.
Importantly, the cap is never active in any recorded training step:
,
with
.
Thus, the empirical transition is not caused by clipping. We will explicitly characterize as a numerical safeguard in this normalized regime and Theorem 1 as an illustrative one-dimensional surrogate rather than a quantitative theory of full-network optimization.
We also sampled the total parameter-gradient norm during these trajectories. Its magnitude changes qualitatively when the and runs leave the low-discrepancy regime, although we emphasize that this is the total optimization gradient rather than an isolated CSD-gradient measurement.
We also agree with the reviewer’s gradient-boundedness criticism. Boundedness of the divergence does not by itself imply bounded parameter gradients. We will remove that statement and claim only the bounded divergence established by Proposition 4 unless additional assumptions are imposed.
4. ViT architecture, optimizer, and scalability.
We agree that the original manuscript underspecified the transformer architecture. The model used in our experiments is not a standard ViT-Tiny/Small/Base configuration. It is a small-image Vision Transformer using Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA).
For CIFAR inputs, the model uses patches, a 512-dimensional token embedding, 6 Transformer blocks, a 1024-dimensional feed-forward layer, dropout , and 15,963,210 trainable parameters (15.96M). The positional embedding contains 64 patch tokens plus one class token. We will add the full architectural specification to the implementation section so that the transformer setting is unambiguous.
We agree that this does not constitute a ViT-Base-scale experiment. We therefore will not use these experiments to make a large-scale transformer claim. The intended conclusion is limited to the tested 15.96M-parameter small-image SPT-LSA transformer.
We also tested whether the reported ViT behavior is merely caused by switching from AdamW to SGD during unlearning. On CIFAR-100/ViT, using AdamW for both FT and CSD gives
and
,
corresponding to a 21.4% reduction. The relative CSD effect therefore persists under the optimizer used to train the original/retrained ViT.
However, CSD+AdamW still strongly overshoots the retrained MIA target, with versus the retrained target . We therefore will not claim that AdamW removes the ViT instability. The more accurate conclusion is that the tested transformer has an optimizer-dependent and narrow operating range.
We have not added ViT-Base during the rebuttal. We agree that larger-scale transformers would be informative, but the current evidence does not support such a scalability claim. We will make this scope explicit rather than extrapolate from the tested 15.96M-parameter architecture.
5. Corrected ViT references.
The reviewer’s concern about the transformer experiments is also affected by an evaluation bookkeeping issue identified during the review process. The stored reference fields for CIFAR-100/ViT and Tiny ImageNet/ViT mixed forget-set accuracy with our convention
.
After correcting the references, CIFAR-100/ViT uses
and Tiny ImageNet/ViT uses
.
The corrected Avg.Gap results are
,
and
.
Thus, the numerical values in the submitted version require correction, but the narrower conclusion that CSD reduces Avg.Gap in these tested ViT settings remains. We will correct all affected tables and figures.
6. Contrastive Unlearning and broader baseline coverage.
We agree that Contrastive Unlearning is a highly relevant representation-based comparison. We do not want to present a hurried reproduction under a different training and tuning pipeline as if it were a definitive fair comparison.
The current study includes CoUn as the closest contrastive contextual comparison and shared-protocol checks with CT, MSG, SCRUB, and SalUn. However, these do not replace a direct reproduction of Contrastive Unlearning. We will therefore make this missing comparison explicit and remove any wording that could be interpreted as claiming superiority over all representation-space unlearning methods.
We view this as a limitation of comparative completeness rather than evidence that the controlled FT/CSD comparison is invalid.
7. Stronger MIA and representation-level forgetting.
We agree that the confidence-based SVC metric alone cannot establish instance erasure. Our stronger LiRA-style audit on CIFAR-100/ResNet-18 gives
,
,
and
,
under the same target pool and shadow-model pipeline.
At low false-positive rates, Retrain, FT, and CSD also give different profiles, with CSD remaining close to the retrained reference in the audited setting. We interpret this only as evidence of closer matching to retraining under this attack, not as a privacy guarantee.
The 50-epoch shadow training is identical for all compared targets, so the comparison itself is controlled. We agree, however, that this does not establish attack optimality or replace stronger-MIA coverage across every experimental setting.
More importantly, we agree with the reviewer that representation displacement is not equivalent to information erasure. We will remove language that could suggest otherwise and use the narrower terms representation displacement and matching to the retrained reference. An independent feature-level leakage attack would be valuable additional evidence, but it is not established by the experiments currently reported.
8. Assumption about retrained representations and plug-in framing.
We agree that an independently retrained network need not preserve the original retain representations. The sentence stating that a retrained model “keeps retain representations essentially unchanged” is therefore too strong.
We will rewrite this as a motivating heuristic for the retain anchor rather than an assumption or theoretical property of retraining. The anchor is intended to limit unnecessary movement of retained representations during the unlearning update; it is not intended to reproduce the internal representation of an independently retrained model.
Similarly, our baseline-compatibility experiments indicate that CSD is most useful when the residual failure mode is specifically under-forgetting. For the tested over-forgetting NegGrad+ configuration, adding CSD produces only marginal improvement. We will therefore replace broad “plug-in improvement” language with the narrower claim that CSD provides additional forgetting pressure when the base method still under-forgets.
9. Scope of forgetting scenarios and modalities.
We agree that our empirical scope is narrower than the most general framing might suggest. The current study evaluates random-subset image-classification unlearning, mainly at 10% forgetting, with an additional 50% stress test.
We have not added 2% deletion, class-wise unlearning, language models, or foundation-model experiments during the rebuttal. Rather than imply untested generalization, we will explicitly limit the empirical claims in the abstract, conclusion, and limitations to the tested random-subset image-classification settings and model families.
The 50% experiment already demonstrates that the mechanism has an important operating boundary: retain/forget selectivity weakens substantially at large deletion ratios. We regard broader deletion structures and modalities as important future tests rather than evidence presently established by the paper.
10. Figure 1 and presentation.
We agree with the presentation comments. We will expand the Figure 1 caption so that the three panels and the intended “under-forgetting → representation repulsion → operating regimes” logic can be understood without reconstructing it from the main text. We will also replace rasterized elements with vector-quality graphics and improve the placement of the “Repulsive force” label.
We appreciate the reviewer’s detailed and technically demanding assessment. We agree with several of the points, particularly that the original paper over-emphasized CSD-specific geometry, that the utility trade-off required a stronger control, and that representation displacement should not be equated with proof of information erasure. The new experiments materially change how we intend to position the paper.
1. Test-accuracy degradation and fair operating-point comparison.
We agree that the best-gap CSD configuration sacrifices substantial TA. We therefore added a utility-matched experiment on CIFAR-100/ResNet-18. With β=0.5 and the FT learning rate,
TACSD=76.313±0.495,
compared with
TAFT=76.750±0.430
under the same retain-only BN recalibration, while
Avg.GapCSD=1.959±0.148
versus
Avg.GapFT=3.003±0.178.
Thus, CSD can improve retrain matching without relying on a large additional utility sacrifice. We will add this comparison and explicitly present CSD as an operating-point trade-off rather than a method that uniformly improves every metric.
We also audited BN fairness. On the same setting, applying the same retain-only BN recalibration to all directly compared BN-based methods gives
Avg.GapFT=3.003±0.178,
Avg.GapCT=1.453±0.205,
Avg.GapCSD=1.435±0.287,
and
Avg.GapMSG=10.371±0.417.
All learnable parameters remain unchanged during this post-hoc recalibration. These results make the comparison more nuanced: CT and CSD are essentially comparable in aggregate retrain-matching gap, with CT preserving test utility more strongly and CSD matching the retrained UA/MIA targets more closely. We will revise the text accordingly rather than imply uniform dominance.
2. Simple feature-repulsion baselines and what is CSD-specific.
We performed the requested comparison against cosine and squared-Euclidean forget-feature repulsion, using the same normalized representations, retain-side anchor, learning rate, and 50-epoch protocol.
Over three seeds,
Avg.GapCSD=1.079±0.142,
Avg.Gapcosine=0.607±0.124,
and
Avg.Gapℓ22=0.617±0.226.
The simple paired objectives are stronger in this setting. We regard this as an important negative result, and agree that the empirical improvement should not be attributed uniquely to CSD’s angular geometry.
For normalized features,
∣u−v∣22=2(1−cos(u,v)),
so cosine and squared-Euclidean repulsion belong to essentially the same paired family. The distinction we retain for CSD is narrower: it provides a set-level distributional discrepancy and does not require explicit instance correspondence.
We tested this directly by removing all same-instance current/reference pairs. In the strict disjoint-batch variant, the current network receives Bf while the frozen reference receives a non-overlapping Bf′. Over five seeds,
Avg.Gapstandard=1.200±0.196
and
Avg.Gapdisjoint=1.411±0.428.
Thus, same-instance terms provide useful signal, but they are not required for CSD to remain effective. We will therefore narrow the novelty claim to a pair-free set-level realization of feature repulsion with an RKHS angular interpretation, rather than claim that CSD is intrinsically the strongest repulsion objective.