acceptodds
Under review as a conference paper at ICLR 2027

Machine Unlearning under an Uncertain Forget Set via Distributional Optimization

Abstract

Machine unlearning, which aims to remove the influence of specific training samples, has received increasing attention as demands grow to remove harmful or unwanted data. Existing unlearning methods typically assume that the forget set is known exactly. However, this assumption is often impractical when detectors provide uncertain sample-wise evidence rather than a verified forget set. Under such uncertainty, prior methods reduce this evidence to retain-or-forget decisions and discard how strongly each sample should be forgotten. To address this, we propose Uncertainty-Aware Distributional Unlearning (UADU), which directly operates on continuous forgetting likelihoods. UADU (i) introduces a Likelihood-Conditioned Unlearning Objective (LCUO) that combines the training label probabilities of retain and reference-based forget targets according to forgetting likelihoods, and (ii) develops Entropy-Constrained Distributional Optimization (ECDO) to preserve the LCUO-specified training label probability and redistribute the remaining probability mass under an entropy criterion. We further provide theoretical guarantees for target approximation and local redistribution optimality. Extensive experiments on three benchmark datasets demonstrate the effectiveness and efficiency of UADU under uncertain forget sets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Uncertain Forget Set, Data Privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iti8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies machine unlearning when forget-set membership is uncertain and represented by continuous sample-wise likelihoods. The proposed UADU framework first constructs likelihood-conditioned training-label targets using retention-weighted reference predictions and projected feature similarities. It then redistributes non-label probability mass through entropy-constrained optimization and fits the resulting targets. Experiments cover mislabeled-data removal on three image-classification datasets, with additional detector, architecture, and backdoor evaluations.

Critical strengths
  1. UADU achieves forget accuracy closer to Retrain than the reported baselines across the three main datasets, with complementary membership-inference measurements.
  2. The paper includes component ablations and sensitivity experiments.
Critical weaknesses
  1. Figure 1 looks AI-generated. I recommend redrawing it with a clearer and more consistent technical presentation.
  2. The related-work discussion should include relevant recent studies, including those published at CVPR 2026. Additional comparisons with these methods would help establish the contribution.
  3. Can UADU be extended to unlearning in generative models? Please discuss its applicability and the modifications required, particularly given its reliance on class probabilities and a linear classification head.
  4. In Section 5.1, conventional baselines receive thresholded forget/retain assignments, whereas UADU uses continuous forgetting likelihoods. The threshold sweep in Appendix C.3 does not resolve this difference. Comparisons with softly weighted baselines using the same likelihoods are needed to isolate the benefit of the proposed distributional formulation.
Questions for authors

see weakness

1 comment

We sincerely thank the reviewer for their careful reading and attention to detail. Please find our response below. (W for weakness, Q for Questions) Revisions are marked in blue in the updated PDF, locations follow each reply.

1. Clarity and Consistency of Figure 1 [W1]

We have redrawn Figure 1 to provide a clearer and more consistent technical presentation. The revised Figure 1 is available at the provided link and is included in the revised manuscript.

https://drive.proton.me/urls/W1GX06FEP8#OqcjhF7J0ssO

The revised figure simplifies the visual design and more explicitly shows the computational flow of UADU, including projected reference similarity, retention-weighted aggregation, the likelihood-conditioned objective, and the subsequent entropy-constrained redistribution and target fitting.

Revision: Figure 1.

2. Recent Related Work and Additional Comparisons [W2]

We thank the reviewer for this suggestion. Among recent CVPR 2026 unlearning studies, few address instance-level unlearning with publicly available code. We therefore compare MUpHT [1], which satisfies both conditions and enables a reproducible evaluation. We follow the authors' official implementation and reported main parameter settings.

DatasetMethodArA_r (%)AfA_f (%)MIA (%)
CIFAR-10Retrain100.00±0.000.67±0.0675.75±0.48
MUpHT96.57±0.4438.18±2.3531.14±1.50
UADU98.55±0.135.93±0.4751.01±2.49
CIFAR-100Retrain99.98±0.000.23±0.0144.07±0.28
MUpHT90.24±6.8859.89±19.3832.73±1.12
UADU90.21±1.815.62±0.7633.58±1.20
TinyImageNetRetrain99.98±0.000.18±0.0233.59±0.76
MUpHT93.86±2.5867.77±9.8828.75±1.81
UADU93.28±1.0715.04±2.7230.66±2.42

At comparable retain accuracy, UADU achieves substantially lower AfA_f than MUpHT across all three datasets. Its MIA is also consistently closer to Retrain. These results support the effectiveness of UADU relative to a recent CVPR 2026 instance-level unlearning method.

Revision: Sections 2.1 and 5.1--5.2 (Table 1).

3. Applicability to Generative Models [W3]

UADU can be adapted to autoregressive generative models, which predict a categorical distribution through a softmax head. In this case, next-token probabilities replace class probabilities, and the observed next token serves as the training label. LCUO can construct token-level targets from references with similar hidden representations, using the observed token’s output-head weight as the projection direction. ECDO can then preserve the prescribed token probability while redistributing the remaining vocabulary mass under the same entropy criterion. We will discuss this extension in the revised manuscript.

Revision: Section 6.

4. Comparison with Softly Weighted Baselines [W4]

We agree that the threshold sweep in Appendix C.3 varies only the hard retain–forget assignments and therefore does not isolate the effect of using continuous forgetting likelihoods. We accordingly add two baselines that use the same pip_i as UADU. Soft-weighted Losses weights standard retain-loss minimization and forget-loss maximization by 1−pi1-p_i and pip_i, respectively. Direct Distillation directly fits reference-based soft targets obtained by combining original and reference predictions according to the forgetting likelihoods. Both baselines use the same original models and forgetting likelihoods as UADU. We follow their core training settings, where applicable, and compare at similar mean retain accuracy.

DatasetMethodArA_r (%)AfA_f (%)MIA (%)
CIFAR-10Retrain100.00±0.000.67±0.0675.75±0.48
Soft-weighted Losses99.25±0.1814.17±0.1147.35±1.46
Direct Distillation97.84±0.2514.60±0.225.75±0.79
UADU98.55±0.135.93±0.4751.01±2.49
CIFAR-100Retrain99.98±0.000.23±0.0144.07±0.28
Soft-weighted Losses91.97±6.6229.91±4.9132.05±1.93
Direct Distillation90.75±0.949.54±1.025.50±0.16
UADU90.21±1.815.62±0.7633.58±1.20
TinyImageNetRetrain99.98±0.000.18±0.0233.59±0.76
Soft-weighted Losses91.21±0.6119.18±0.8831.37±0.62
Direct Distillation92.22±1.1817.40±3.146.22±1.31
UADU93.28±1.0715.04±2.7230.66±2.42

At comparable mean retain accuracy, UADU achieves the AfA_f closest to Retrain across all three datasets. Its MIA is also closest to Retrain except against Soft-weighted Losses on TinyImageNet. These results show that UADU's gains are not explained by access to continuous likelihoods alone, supporting the benefit of its target construction and distributional optimization.

Revision: Appendix C.6.

[1] : Qian, Chengyao, et al. "pH-Strips for Selective Forgetting: A Blunt but Fast Diagnostic Baseline for Machine Unlearning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Reviewer_tXLLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the machine unlearning when the forget data is uncertain. The proposed UADU resorts to the continuous forgetting probabilities instead of binary decisions. It obtains the traget predictions using similar training samples and adjust the remaining class probabilities under an entropy constraint. The experiments results are promising.

Critical strengths

(1) The paper study the forgetting scenario where the samples to be forgotten are not known with certainty and uses the continuous probabilities rather than the binary decisions to handle the uncertainty.

(2) The paper involved several robustness experiment setting to demonstrate its robustness.

(3) The paper provides the theoretical analysis of the approximation bounds and constrained optimization.

Critical weaknesses

(1) For the motivation of handling the uncertain forget data, the authors should give more clarification why such task setting is necessary in general task scenarios.

(2) The authors utilize the reference predictions obtained from the original model, and study the output adjustment around the unlearned teacher. However, the the original model has already learned from the forget set, especially in the random data forgetting setting. Thus, such predictions can also contain the information of forget data.

(3) The author should provide the ablation study to analyze the contribution of involved continuous likelihoods, feature projection, reference aggregation, and entropy redistribution.

(4) Without a clear definition of the MIA metric and its direction, it is difficult to determine whether these results indicate successful forgetting. The authors should give more explanation.

(5) The authors should give more explanation of the training objective and Equation 5 fail to clearly define the cross-entropy objective.

Questions for authors

(1) Can the authors evaluate the distance between the constructed targets and the predictions of a retrained model? This maybe useful to clarify the effectiveness.

(2) The authors can add one ablation study on using the binary probabilities.

4 comments

Thanks for your recognition and the valuable suggestions. Please find our response below. (W for weakness, Q for Questions) Revisions are marked in blue in the updated PDF, locations follow each reply.

1. Motivation for Uncertain Forget Sets [W1]

We appreciate the suggestion. Uncertainty in identifying the forget set can arise across common machine unlearning applications, including privacy-driven data deletion, unlearning mislabeled training samples, and backdoor removal. The removal objective may be clear, but identifying the corresponding training images requires imperfect detectors. For privacy-driven deletion, visual privacy detectors provide confidence scores for sensitive information in images [1]. For mislabeled-data unlearning, detectors estimate labeling errors using training-dynamics scores [2][3]. For backdoor removal, detectors identify potentially poisoned samples using transformation-sensitivity or prediction-shift scores [4][5]. These scores provide uncertain evidence rather than verified forget-set membership. However, existing unlearning methods are not designed for these practical scenarios. In contrast, UADU directly operates on such evidence through continuous forgetting likelihoods.

Revision: Section 1.

2. Reference Predictions and Comparison with Retraining [W2&Q1]

We acknowledge that the original model may retain information from forget samples. LCUO reduces this influence by weighting each reference by (1−pj)(1-p_j), so samples with higher forgetting likelihoods contribute less to the forget target.

We compare the original predictions and UADU targets with Retrain using Jensen–Shannon divergence on the forget set (lower is better).

DatasetOriginalUADU TargetReduction
CIFAR-100.6610.11882.2%
CIFAR-1000.6680.25761.6%
TinyImageNet0.6670.29755.5%

On average, UADU's targets reduce the JSD to Retrain by 66.4% across the three datasets, indicating substantially closer agreement with retrained predictions than the original model.

Revision: Appendix C.5.

3. Contribution of Individual Components [W3&Q2]

We extend our component-wise ablation to continuous likelihoods, feature projection, reference aggregation, and entropy redistribution. Binary Likelihoods replaces pip_i with 1[pi≥0.5]\mathbb{1}[p_i\geq0.5], w/o Feature Projection removes the orthogonal projection in reference similarity, w/o Reference Aggregation uses the highest-weight reference prediction, and w/o Entropy Constraint removes entropy preservation.

MethodCIFAR-10CIFAR-100
ArA_r(%)AfA_f(%)MIA(%)ArA_r(%)AfA_f(%)MIA(%)
Retrain100.00±0.000.67±0.0675.75±0.4899.98±0.000.23±0.0144.07±0.28
Binary Likelihoods98.34±0.185.38±0.5851.03±2.2189.11±4.1845.11±4.8122.83±2.78
w/o Feature Projection98.84±0.127.36±0.8736.37±2.8590.59±1.645.56±0.7332.38±1.04
w/o Reference Aggregation98.53±0.156.11±0.5947.05±1.3390.23±1.425.94±0.9033.03±1.07
w/o Entropy Constraint98.61±0.146.01±0.3548.03±2.5990.08±1.835.48±0.8529.95±1.22
UADU98.55±0.135.93±0.4751.01±2.4990.21±1.815.62±0.7633.58±1.20

The largest degradation appears when continuous likelihoods are replaced by hard assignments on CIFAR-100, where AfA_f increases from 5.62% to 45.11%. Removing feature projection or reference aggregation also worsens forgetting or MIA, while the entropy constraint mainly improves MIA. Overall, these results support the complementary roles of all components.

Revision: Section 5.3.

4. Definition and Interpretation of the MIA Metric [W4]

We follow the logistic regression-based MIA implementation in [6,7]. The attacker uses predictive entropy to distinguish retain samples, treated as members, from held-out test samples. We report the percentage of forget samples classified as members:

\sum_{x_i\in\mathcal D_f}\hat m_\theta(x_i),$$ where $\hat m_\theta(x_i)\in\{0,1\}$ is the attacker's membership prediction. A higher MIA means more forget samples are classified as members. Since unlearning aims to approximate retraining, we assess MIA by its closeness to Retrain. *Revision: Section 5.1.* ## 5. Clarification of the Training Objective [W5] We clarify that LCUO uses binary cross-entropy on the training-label probability, defined as $$\ell_y(\mathbf a,\mathbf b) =-b_y\log a_y-(1-b_y)\log(1-a_y),$$ where $a_y$ and $b_y$ denote the predicted and target probabilities of the training label. LCUO applies this loss to the retain and forget targets, weighted by $1-p_i$ and $p_i$, respectively. This is equivalent to fitting the likelihood-conditioned target $$\tilde q_{i,y_i}=(1-p_i)q^r_{i,y_i}+p_iq^f_{i,y_i}.$$ ECDO then preserves this target probability while determining the remaining class probabilities under an entropy criterion and fitting the resulting full distribution through KL minimization. *Revision: Section 4.1.*

[1] : Hu, Xiaobin, et al. "VPD-100K: Towards Generalizable and Fine-grained Visual Privacy Protection." Forty-third International Conference on Machine Learning. 2026.

[2] : Pleiss, Geoff, et al. "Identifying mislabeled data using the area under the margin ranking." Advances in Neural Information Processing Systems 33 (2020): 17044-17056.

[3] : Kim, Suyeon, et al. "Learning discriminative dynamics with label corruption for noisy label detection." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] : Chen, Weixin, Baoyuan Wu, and Haoqian Wang. "Effective backdoor defense by exploiting sensitivity of poisoned samples." Advances in Neural Information Processing Systems 35 (2022): 9727-9737.

[5] : Li, Wei, et al. "PSBD: Prediction shift uncertainty unlocks backdoor detection." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[6] : Chundawat, Vikram S., et al. "Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.

[7] : Foster, Jack, Stefan Schoepf, and Alexandra Brintrup. "Fast machine unlearning without retraining through selective synaptic dampening." Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 11. 2024.

Reviewer_tXLLOpenReview ↗

Thank you for the response.

I still have some reservations regarding the residual influence of forgotten information in the reference predictions and the relatively limited contribution of the entropy constraint.

We sincerely appreciate your continued consideration and thoughtful feedback. We would like to further clarify how the experimental results presented in our earlier response address these two concerns.

1. Residual Influence in Reference Predictions

We acknowledge that the original model may retain information from forgotten samples, including in the predictions of other reference samples. LCUO does not assume that these predictions are free from such influence. Instead, it excludes the queried sample and reduces the contribution of likely-forgotten references through retention-weighted aggregation. As reported in our earlier response, the resulting targets show substantially closer agreement with Retrain predictions on the forget set.

DatasetOriginalUADU TargetJSD Reduction
CIFAR-100.6610.11882.2%
CIFAR-1000.6680.25761.6%
TinyImageNet0.6670.29755.5%

Despite relying on original-model references, UADU reduces the JSD to Retrain by 66.4% on average across all three datasets. Since Retrain is trained without the forget samples, this substantial improvement indicates that the constructed targets considerably reduce the influence of forgotten samples on the resulting predictions. While complete elimination of indirect influence is not guaranteed, these results provide empirical evidence that LCUO effectively mitigates its impact on the constructed targets.

2. Contribution of the Entropy Constraint

The entropy constraint serves a different purpose from improving forgetting accuracy. LCUO specifies the training-label probability, while ECDO controls the remaining class probabilities under an entropy criterion. As shown in our earlier ablation results, this constraint improves agreement with Retrain in membership behavior.

MethodCIFAR-10 MIA Gap (pp)CIFAR-100 MIA Gap (pp)
w/o Entropy Constraint27.7214.12
UADU24.7410.49

Our MIA evaluation uses logistic regression based on predictive entropy, which depends on the full output distribution. The entropy constraint serves to preserve predictive uncertainty when redistributing the remaining probability mass after LCUO determines the training-label probability. Without this constraint, the redistribution does not explicitly preserve the post-LCUO teacher's entropy and may introduce unnecessary changes in predictive uncertainty. Our ablation results confirm its additional contribution. The entropy constraint reduces the MIA gap to Retrain by 2.98 and 3.63 percentage points on CIFAR-10 and CIFAR-100, respectively. These consistent improvements support the benefit of preserving predictive uncertainty during unlearning

We will further clarify these findings and the distinct roles of both components in the camera-ready version.

Reviewer_UP18OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles machine unlearning with soft/uncertain forget sets characterized by sample-wise forgetting likelihoods. It introduces UADU, a two-stage framework: LCUO first interpolates training-label probabilities between the original model and feature-similarity-weighted reference outputs; ECDO then completes the target distribution via entropy-based probability reallocation while preserving the target label mass. Backed by approximation bounds and local minimum-distortion guarantees, UADU proves effective for erasing both mislabeled data and backdoor triggers.

Critical strengths

1. Realistic and practical problem formulation. The paper tackles an important and realistic setting where forget-set membership is inherently uncertain. By leveraging continuous forgetting likelihoods directly in target construction rather than relying on arbitrary hard thresholding, the framework effectively bridges detection uncertainty with the unlearning objective.

2. Consistent forgetting efficacy and efficiency. Across multiple benchmarks, UADU achieves forget-set accuracy and membership inference defense closer to full retraining than existing baselines, while remaining computationally efficient. Evaluations across multiple detectors and backdoor unlearning tasks provide solid empirical support for its effectiveness.

Critical weaknesses
  1. Unclear advantage over simpler soft baselines. The compared baselines use hard-thresholded retain and forget sets, whereas UADU uses continuous likelihoods. Without comparisons against straightforward alternatives, such as soft-weighted retain–forget losses or direct distillation from reference-based soft targets, it remains unclear how much of the improvement comes from UADU’s specific design beyond using continuous membership information.

  2. The forgetting–retention trade-off needs stronger evidence. UADU’s retain accuracy drops substantially relative to Retrain, from 99.98% to 90.21% on CIFAR-100. Without comparisons at matched retention levels, it is difficult to assess whether the improved forgetting metrics reflect a better overall trade-off. Moreover, retain accuracy is reported on training data, leaving the impact on clean test performance unclear.

Questions for authors
  1. Have you compared UADU with simple soft-weighted retain–forget losses or direct distillation from reference targets using the same likelihoods?

  2. How does UADU compare with the strongest baselines at matched retain accuracy, and could you report clean test accuracy to assess generalization?

3 comments

We thank the reviewer for their detailed review and their insightful questions. Please find our response below. (W for weakness, Q for Questions) Revisions are marked in blue in the updated PDF, and each reply concludes with their location.

1. Comparison with Simpler Soft Baselines [W1&Q1]

We appreciate the suggestion. To distinguish UADU's contribution from the use of continuous forgetting likelihoods alone, we introduce two simple baselines using the same likelihoods. Soft-weighted Losses applies continuous likelihood weights to the standard retain-loss minimization and forget-loss maximization objective. Direct Distillation directly fits reference-based soft targets obtained by combining original and reference predictions according to the forgetting likelihoods. Both baselines use the same original models and forgetting likelihoods as UADU. We retain their main settings, following prior work [1], and adjust only the learning rates to compare at similar retain accuracy.

DatasetMethodArA_r (%)AfA_f (%)MIA (%)
CIFAR-10Retrain100.00±0.000.67±0.0675.75±0.48
Soft-weighted Losses99.25±0.1814.17±0.1147.35±1.46
Direct Distillation97.84±0.2514.60±0.225.75±0.79
UADU98.55±0.135.93±0.4751.01±2.49
CIFAR-100Retrain99.98±0.000.23±0.0144.07±0.28
Soft-weighted Losses91.97±6.6229.91±4.9132.05±1.93
Direct Distillation90.75±0.949.54±1.025.50±0.16
UADU90.21±1.815.62±0.7633.58±1.20
TinyImageNetRetrain99.98±0.000.18±0.0233.59±0.76
Soft-weighted Losses91.21±0.6119.18±0.8831.37±0.62
Direct Distillation92.22±1.1817.40±3.146.22±1.31
UADU93.28±1.0715.04±2.7230.66±2.42

At comparable mean retain accuracy, UADU achieves the AfA_f closest to Retrain across all three datasets. UADU also yields MIA results closest to Retrain, except against Soft-weighted Losses on TinyImageNet. These results support the benefit of UADU's target construction and distributional optimization beyond simply using continuous forgetting likelihoods.

Revision: Appendix C.6 .

2. Forgetting–Retention Trade-off and Clean Test Performance [W2&Q2]

We agree that matched-retention comparisons and clean test accuracy are necessary to assess the trade-off. We compare UADU with SalUn and LAF+R, the two baselines with the lowest average AfA_f in Table 1. The baselines are adjusted to approximately match UADU’s mean retain accuracy. We additionally report clean test accuracy to assess generalization.

DatasetMethodArA_r (%)AfA_f (%)MIA (%)Clean test (%)
CIFAR-10Retrain100.00±0.000.67±0.0675.75±0.4894.46±0.20
SalUn99.26±0.0912.82±1.4334.87±4.6592.04±0.44
LAF+R98.35±0.3521.52±1.074.24±0.7387.57±1.04
UADU98.55±0.135.93±0.4751.01±2.4992.07±0.16
CIFAR-100Retrain99.98±0.000.23±0.0144.07±0.2875.68±0.23
SalUn89.37±2.2235.30±3.792.89±0.5866.66±1.46
LAF+R90.31±6.9676.99±15.6330.47±7.2164.34±3.74
UADU90.21±1.815.62±0.7633.58±1.2067.37±1.04
TinyImageNetRetrain99.98±0.000.18±0.0233.59±0.7662.97±0.28
SalUn92.36±2.3330.15±4.326.15±1.1352.87±1.66
LAF+R92.41±1.0719.29±8.2134.79±4.3555.23±1.01
UADU93.28±1.0715.04±2.7230.66±2.4255.18±0.88

At comparable mean retain accuracy, UADU achieves the AfA_f closest to Retrain across all three datasets. Its MIA results are also closest to Retrain, except against LAF+R on TinyImageNet. Moreover, UADU matches or exceeds the baselines in clean test accuracy. These results indicate that UADU achieves more effective forgetting without sacrificing additional retention or generalization relative to the compared baselines, supporting a more favorable forgetting–retention trade-off.

Revision: Appendix C.7.

[1] : Kurmanji, Meghdad, et al. "Towards unbounded machine unlearning." Advances in neural information processing systems 36 (2023): 1957-1987.

Thank you for the additional experiments. The soft baselines, approximately matched-retention comparisons, and clean test results substantially alleviate my original empirical concerns. My remaining reservation concerns novelty: although the results support combining LCUO’s soft target interpolation with ECDO’s entropy-constrained redistribution, I view this as a relatively modest extension of existing weighting and distillation ideas. On balance, I will keep my current rating, while appreciating the authors’ efforts to strengthen the paper.

We sincerely appreciate your careful reading of our rebuttal and your thoughtful feedback. We are glad that the additional experiments helped address your empirical concerns. We would also like to clarify how UADU differs from existing weighting and distillation approaches.

Our framework is designed to determine what a model should predict under uncertain forgetting evidence rather than simply adjusting the strength of existing objectives. Conventional soft weighting scales predefined retain–forget losses. In contrast, LCUO constructs reference-based forget targets using retention-weighted aggregation and feature projection. It then determines each sample's training-label probability according to its forgetting likelihood. Direct distillation fits a given soft target. ECDO instead determines the remaining class probabilities while preserving the LCUO-specified training-label probability under the post-LCUO teacher's entropy criterion.

The following CIFAR-10 ablation results further illustrate the roles of reference construction and entropy preservation.

MethodArA_r (%)AfA_f (%)MIA (%)
Retrain100.000.6775.75
w/o Feature Projection98.847.3636.37
w/o Reference Aggregation98.536.1147.05
w/o Entropy Constraint98.616.0148.03
UADU98.555.9351.01

These ablations further support the intended roles of each component. Removing feature projection or reference aggregation increases the deviation from Retrain in forgetting accuracy or MIA. Removing the entropy constraint leaves forgetting accuracy nearly unchanged but increases the MIA deviation from Retrain. Together, these results provide empirical evidence that LCUO's reference construction and ECDO's entropy-constrained redistribution serve distinct purposes and contribute to the effectiveness of UADU, rather than simply combining existing weighting and distillation ideas. Our theoretical results further support these constructions through reference-target approximation and local minimum second-order KL distortion.

We will make these methodological differences clearer in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.