acceptodds
Under review as a conference paper at ICLR 2027

Pseudo-Label-Aware Unlearning of Labeled Data in Self-Training

Abstract

Self training fits a labeling model on a small labeled dataset and uses its predictions to assign pseudo labels and select unlabeled examples for downstream training. After deletion of a labeled record, its influence may remain in this generated supervision. Removing the record from downstream training while keeping stored pseudo labels and selection decisions can therefore fail to reproduce full retraining without that record. We propose Pseudo label Aware Labeled data Machine Unlearning (PALMU). PALMU updates the labeling model on retained labeled data and regenerates pseudo labels for the unlabeled pool. Examples whose predicted class probability meets the confidence threshold enter downstream training. PALMU then updates the downstream classifier without fitting both classifiers to convergence. For fixed representations, we bound PALMU’s distance from full retraining using remaining downstream optimization error and supervision differences from the fully retrained labeling model. Across four datasets at deletion ratios of 5%, 10%, 20%, and 30%, PALMU meets predefined agreement and utility criteria in 397 of 400 fixed representation requests. Its mean predictive total variation is 4.3 to 56.6 times lower than the closest baseline evaluated against the same retraining reference. On CommitBERT, it retains 1.56 to 1.83 macro F1 points over labeled only retraining. For MLPs with learned representations, updating only the final classifier does not match retraining.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Machine Unlearning, Semi-Supervised Learning, Pseudo-level Contamination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4bUJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PALMU for machine unlearning in one-round self-training. It updates the labeling classifier on retained data, regenerates pseudo-labels and acceptance decisions, and corrects the downstream classifier through Newton updates under fixed representations.

Critical strengths
  • Considering the influence of deleted records on generated supervision is reasonable.
  • The method combines pseudo-label regeneration and downstream correction coherently.
  • The theoretical analysis and experiments support closer agreement with full retraining under fixed representations.
Critical weaknesses
  • The motivation and formulation need clearer explanations, particularly the roles of the two classifiers, fixed representations, and replacement loss contributions.
  • The method assumes that representations are independent of deletion-affected data. Its applicability is limited when deleted records also influence the encoder.
  • The evaluation mainly establishes predictive agreement with retraining. Neither the forgetting score nor predictive total variation establishes that information about deleted records has been removed.
  • Hyperparameter selection is described, but sensitivity analyses of the confidence threshold, pseudo-label weight, and regularization strengths are missing.
  • Random deletion experiments do not sufficiently establish performance on imbalanced data or when deleted records contain unique characteristics.
  • Some abbreviations are not defined at first use, and punctuation errors remain. Figure 1 also needs more consistent fonts and clearer spacing.
Questions for authors
  • Can the authors better explain the roles of the two classifiers and how updating the labeling classifier reduces the influence of deleted records?
  • Can PALMU handle deletion-affected representations and deletion of unique samples from imbalanced datasets?
  • Can the authors provide stronger evidence of information removal beyond predictive agreement and the reported membership attack?
  • Can the authors provide sensitivity analyses of the main hyperparameters?
Reviewer_MEHAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces unlearning in the self-training setup, where the forget set not only has impact on the direct supervision of the downstream classifier but also on the labelling classifier which was trained to provide pseudo-labels for the unlabelled data. It introduces PALMU, a unlearning method for this setup, where they first update the downstream classifier, then the labelling model and regenerate the pseudo-labels for the unlabelled set, and only readjust the classifier without full retraining.

Critical strengths
  1. The paper introduces a new setting for unlearning, in self-training, where the forget samples influence both the classifier and labelling model.
  2. The method remains within TV tolerance and has comparable F1 score showing agreement with full retraining, across 4 datasets across text and images.
Critical weaknesses
  1. The method's effect does not transfer well when the feature extractor is not frozen and the deleted set also affects the representations. The setting studied in the paper is very simple, with only the MLP trained.
  2. In section 5.5 and Appendix H, in table 5, it is shown that stage 1 does not add any advantage and can be skipped, this ablation questions the need for stage 1 in the first place, when it is faster and more accurate (with lower TV) without it. The stage 1 is not well-motivated, this is one of the core steps of the entire design.
  3. Labelled only retraining in most datasets, have similar F1 score as full retraining, so the need for pseudo labels is not well motivated in this setup.
Questions for authors
  1. The datasets used in evaluation are not extensive enough, it would be helpful to see it's effect on a variety of tasks, with varying sizes and classes to better assess the strengths and the weaknesses. 2.The motivation behind stage 1 should be explained further, why not fully skip it?.
  2. Why don't we just retrain the classifier on labelled data, as it's results are comparable to full retraining? This is the safest way to delete, so can we also check agreement with this model as well.
  3. The self-training setting studied here is simple, how does it expand to other settings of self-training where the labelling model is trained with the classifier, in this case, it would be interesting to see how the deleted data would affect the pseudo-label supervision and how it's effect propagates to the classifier.
  4. How does the method perform under targeted attacks, where an entire class is removed, to see how the class wise accuracy dropped and also analysis on what kind of samples are deleted and how correlated they are to the retained set in terms of feature similarity, and it's effect on these on the final metrics.
Reviewer_FParOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PALMU as an unlearning approach to address the removal of labeled records in a self-training setup. The main challenge is to remove the influence of the forgotten samples on the downstream classifier. The setup consists of a labeler model that is trained on a small labeled set and used to produce pseudo-labels and confidence scores to build the downstream set of (pseudo-)labeled data. The proposed approach consists of several stages: updating the labeler, regenerating pseudo-labels and confidence-based selection decisions, and then correcting the downstream classifier. The analysis and main experimental section focus on frozen representations with linear classifiers. MLPs are also tested and used to evaluate the classifier-only update setup. The experiments span several text and image datasets.

Critical strengths
  • The paper identifies a valuable research direction: deleted labels may continue to influence the downstream classifier. It proposes a method and an experimental analysis that address this issue. The experimental section establishes robust baselines that show the limitations of adapting a classifier fitted on frozen representations when compared with retraining that classifier on the new downstream set.
  • The results in terms of fidelity to full retraining are competitive. Compared with other approaches, PALMU obtains a significantly lower discrepancy, suggesting better alignment of its predictions with those of the retrained classifier.
  • The evaluation design is robust, as it measures agreement with the retrained classifier, classification utility, and deletion-update time.
Critical weaknesses
  • The scope of the paper is somewhat limited, since the theory and the stronger experimental results focus on frozen representations and linear classifiers. This is particularly important because the MLP experiments show that updating only the MLP does not match retraining when the frozen representations have been affected by the deleted data.
  • Similarly, the study only addresses one-round self-training, and it is not clear how the corrections would extend to iterative rounds of self-training, where regenerated pseudo-labels would affect subsequent rounds.
  • It seems that Stage 1 is informative about the effect of deletions, but it is presented as a main part of the approach. The authors should clarify throughout the paper the role of this stage and when it is recommended to be applied.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.