acceptodds
Under review as a conference paper at ICLR 2027

Behavioral Guarantees for Proxy Based Unlearning

10 views

Abstract

This paper generalizes recent proxy based unlearning methods and proves theoretical guarantees about the behavior of the resulting unlearned model: upper bounds on the Kullback-Leibler divergence from the ideal retrained. We model approximate unlearning as a constrained optimization problem and interpret the family of solutions as introducing a scaled unlearning signal in the output space. The unlearning signal arises from proxies of the posterior data distributions. Its scale is adapted to the proxies to ensure the behavioral upper bounds. This framework relies on the structure of the data distributions in order to create proxies. If need be, the solution serves as a teacher to distill the update in the weights. Our approach is experimentally validated over three forgetting scenarios as reaching the closest classifier to the ideal retrained model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_47e8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for machine unlearning. It uses two proxy models to adjust an existing model’s predictions. One represents the original data. The other represents the retained data. The method automatically selects the strength of this adjustment. The authors provide theoretical analysis and evaluate class and subclass forgetting on image-classification datasets.

Critical strengths

S1. Automatic Selection of the Adjustment Strength

The method provides a concrete procedure for choosing how strongly to change the model's predictions. This addresses a practical choice that otherwise requires tuning.

S2. Comparison With Retrained Models

The evaluation compares output distributions with those of retrained models. This addresses the intended outcome of unlearning more directly than forgotten-example accuracy alone.

Critical weaknesses

M1. What does each new component add?

The paper acknowledges that earlier methods already correct predictions using auxiliary models. UCD uses two auxiliary models. T3-Unlearning also studies distribution corrections for unlearning.

The main advance therefore appears to be the joint proxy construction and automatic adjustment strength. I would like clearer evidence of what each adds. In particular, it is difficult to judge the automatic scaling without comparing adjustment strategies using the same proxies.

M2. What happens when the proxies are poor?

The correction depends on the proxies. Their behaviour when inaccurate is therefore important. Proposition 1 assumes that the retain proxy is closer to the true retain posterior than the original proxy. Algorithm 2 checks the normalisation function. It does not check this assumption. The claim that incorrect proxies lead to zero scaling therefore does not follow from this check alone. Automatic scale selection is a central contribution. I would like to see when inaccurate proxies cause the method to harm performance, and whether a practitioner could recognise these cases before applying the correction.

M3. The comparison with Datamodel Matching

Datamodel Matching uses data attribution to predict retrained outputs. It then adjusts the original model to match those predictions. This differs from fitting a density model to retained examples.

The argument that modelling only retained data is suboptimal does not establish an advantage over Datamodel Matching. The distinction needs more care. A direct comparison is needed only if the authors retain a claim of superiority.

M4. What this means for individual-data removal

The medical application I have in mind is removing a patient's contribution after their data are withdrawn. The current experiments do not establish that outcome.

Matching a retrained model's predictions is useful evidence. It does not, by itself, establish protection against privacy attacks. The paper acknowledges this distinction, but it should remain clear when discussing the practical guarantees. This requires narrower wording, not a medical benchmark.

Questions for authors

Post-rebuttal update — Weak acceptance. (editing to make it more structured)

Thank you for the substantial revisions. I am raising my rating to weak acceptance.

--The matched-proxy experiments and expanded Datamodel Matching discussion meaningfully address M1 and M3, and the new robustness experiments strengthen the evidence for adaptive scaling.

--My remaining concern is the interpretation of the guarantee (M2): the retained-data diagnostic does not verify the full theoretical condition. Please distinguish this empirical safeguard from the CONDITIONAL guarantee, and describe zero scaling as abstaining from unlearning. I think that the paper should have "conditional guarantees" qualified throughout. I am on the fence for whether this should be in the title but err on no.

--For M4, please keep the distribution-level scope explicit in the main claims. Individual-example privacy guarantees are not required for this contribution.


Original questions and assessment (before rebuttal):

  • M1: Identify which results isolate the benefits of joint proxy construction and automatic scaling. If this is not already tested, a matched comparison of adjustment strategies using the same proxies would help.
  • M2: Distinguish the conditional guarantee from what the implemented procedure can verify. Revise the safety claims accordingly. A small sensitivity experiment with inaccurate proxies would help. A general failure-detection method is not required. State whether harmful updates can be recognised before use.
  • M3: Clarify the distinction from Datamodel Matching. Support any claim of superiority with a direct comparison, or narrow the claim.
  • M4: State what the evaluation establishes about forgetting. Separate this from what remains untested about individual-data removal and privacy.

M1 and M2 were my main reasons for weak rejection before the rebuttal. The updated assessment above supersedes that recommendation.

5 comments

We thank the reviewer for their detailed and insightful comments, leading to targeted revisions (in blue) in the manuscript.

M1. What does each component add?

Identify which results isolate the benefits of joint proxy construction and automatic scaling.

First, we thought that Fig. 2 (right) sufficiently illustrates the crucial role of the automatic scaling. It shows the trajectory of the unlearned model qηq_{\eta} when the scaling η\eta increases starting from 0. This trajectory is pictured in terms of proximity (measured with empirical Kullback-Leibler divergences) to the ideal model (x-axis) and the initial models (y-axis).

We emphasize this more in the revision:

  • Without automatic scaling, say η=1\eta=1, q1q_{1} is far away from both reference models.
  • On the contrary, the automatic scaling is able to locate the sweet spot of the trajectory closer to the ideal models.

We now mark the point on the trajectory at η=1\eta=1 with a red circle and add a comment in the revised version.

a matched comparison of adjustment strategies using the same proxies would help

Second, in relation to your comments M2 and M3, the new App. G.2 proposes a new experiment with the proxies proposed in the Datamodel matching paper plus our adaptive scaling. In short, adaptive scaling becomes more essential whenever the proxies are inaccurate.

Finally, regarding the joint proxy construction, we emphasize it more in the text. We explain the interest of the 2C construction (i) for efficiency purposes (a single fit) and (ii) for focusing on the modeling shift (Prop.2 and Prop.3).

M2. What happens when the proxies are poor?

Algorithm 2 checks the normalization function. It does not check this assumption.

You are absolutely right. We added Appendix D to address your concern, as well as in the ETAMAX procedure (Alg. 2) in this sense.

State whether harmful updates can be recognised before use.

We propose a detection of those cases. After identifying the proxies, the practitioner computes this estimator on a held-out validation set of retained data. The new App. D.1 explains how this estimator works. Note that this test is now integrated into Alg. 2 (ETAMAX): If the test fails, ηmax⁡\eta_{\max} is set to 0.

I would like to see when inaccurate proxies cause the method to harm performance.

A new experiment in App. D.2 considers several proxies for prp_r learned with less and less data. This artificially makes the assumption less likely to hold. It shows that the new estimator yields little to no undetected violation.

A small sensitivity experiment with inaccurate proxies would help.

Also, we used undertrained datamodels to create inaccurate proxies in App. G.2. Their use for unlearning is detrimental when used without scaling (ie. η=1\eta=1). Our automatic scaling succeeds in limiting this effect.

M3. The comparison with Datamodel Matching

Clarify the distinction from Datamodel Matching.

The way we cited this paper was very confusing. This work deserves much more attention, and we develop Appendix G in this sense.

  • We rewrite the way we cite this paper in the main body, pinpointing the similarities and differences with our proposal.
  • App. G.1 shows that our theoretical framework also encompasses Datamodel Matching
  • App. G.2 presents new experiments based on Datamodel Matching with our adaptive scaling. It shows that:
    • When the datamodels are accurate (trained over large subgroups until convergence), our scaling ηmax⁡\eta_{\max} goes to 1. We are back to the Datamodel matching method (ie. no scaling).
    • However, when the datamodels are inaccurate (early stop of the training before convergence), our adaptive scaling ηmax⁡\eta_{\max} is bigger than 1. Indeed, setting η=1\eta=1 is detrimental.

M4. What this means for individual-data removal

It does not, by itself, establish protection against privacy attacks. The paper acknowledges this distinction, but it should remain clear when discussing the practical guarantees

We agree. Be it the behavioral guarantees (main propositions) or the application to privacy auditing (see App. I), our results concern only averaged statistics. For instance, App. I estimates the difficulty for an attacker to distinguish the unlearned model from an ideal retrained model that never saw the forget data, on average, ie. over typical data. This doesn't mean that the attacker fail on a non-typical data sample. We clearly do not have any individual sample guarantees. This is clearly stated in App. I.2.

State what the evaluation establishes about forgetting.

Of note, we consider scenarios (subclass and class) where the forget data are correlated because they share the same semantics. Our work focus more on unlearning this specific meaning: a distribution and not a specific example. The original verbatim DMM, for instance, cannot by design handle this kind of scenario (see App. G.2).

Dear Reviewer,

Please do not let us down. We have devoted a great deal of time and effort to revising the submission. We dare think that this new version meets your expectations.

Regards

Hi all,

Thank you and congratulations for the substantial work. I can imagine you burning the midnight oil, similar to our team to ensure the questions were addressed.

I am upping this to a weak accept. "accept" because the work is a solid effort to carefully describe the concept with appropriate experimentation and ablations.

"weak" because it is not have a clear practical application. --The guarantees are still quite conditional and narrow. --The positioning within existing literature is a bit weak --The methodological advance is incremental (not bad just a bit small)

Do you think you can better distinguish the work from existing output-correction approaches?

Reviewer_7eTuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a proxy-based machine unlearning framework to address the instability, lack of clear stopping criteria, and limited theoretical guarantees of existing gradient-based unlearning methods. The proposed approach constructs proxy distributions for the original and retained data and uses their log-probability difference to derive an unlearning signal that directly adjusts the model's output predictions. The authors further introduce an adaptive mechanism for determining the unlearning strength and a 2C-Lift proxy modeling strategy. They also establish theoretical upper bounds on the KL divergence between the unlearned model and the ideal retained-data distribution. Experiments covering both class-wise and subclass unlearning scenarios demonstrate that the proposed method can achieve behavior close to that of an ideally retrained model at a relatively low computational cost.

Critical strengths
  1. This paper addresses an important problem: existing approximate unlearning methods lack theoretical characterizations of the behavior of unlearned models and guarantees of stability. By using KL divergence to measure the discrepancy between the unlearned model and the ideal distribution, the paper formulates the problem as a well-defined mathematical optimization objective.
  2. The proposed method presents a well-structured and theoretically grounded unlearning framework.
Critical weaknesses
  1. Theoretical guarantees rely on assumptions that are difficult to verify in practice. Proposition 1 assumes that the retain proxy better approximates the true retained-data distribution, but this condition is difficult to verify. And the adaptive scaling mechanism does not guarantee that this assumption holds, raising concerns about the practical reliability of the theoretical guarantees.
  2. Limited evaluation of proxy robustness. The experiments do not sufficiently examine how inaccurate proxy distributions or less informative feature representations affect unlearning performance, despite the theoretical guarantees relying on proxy quality.
Questions for authors

See Weaknesses.

2 comments

Many thanks for your valuable comments. Revisions appear in blue in the new version of our submission.

Weakness #1

Proposition 1 assumes that the retain proxy better approximates the true retained-data distribution, but this condition is difficult to verify.

We agree. To address your concern, we propose an estimation of this assumption. After identifying the proxies, the practitioner computes this estimator on a held-out validation set of retained data. The new App.D.1 explains how this estimator works.

Linking Weakness #1 with Weakness #2, we further study the impact of inaccurate modeling on Prop. 1's assumption (see App. D.2) by measuring whether it holds and the efficacy of the practitioner estimator in flagging violations. The new experiment considers several proxies for p^r\hat p_r, either learned with less data or from noisier features. This artificially makes the assumption less likely to hold. It shows that this estimator yields few to no undetected violations.

raising concerns about the practical reliability of the theoretical guarantees.

To improve reliability, we add this additional estimated safeguard (App. D.1.) to the ETAMAX procedure (Alg. 2.).

Weakness #2

The experiments do not sufficiently examine how inaccurate proxy distributions or less informative feature representations affect unlearning performance

Some clues were already present in our initial submission:

  • inaccurate proxy: We have proposed some families of proxies with various accuracies.
    • Prop. 4 in App. E derives a closed-form expression for the cost of using the less accurate LDA proxy in place of the more refined QDA proxy.
    • Experimental results (Fig. 2, Tab. 2, Fig. 5) show the hierarchy: LDA-2C > LDA-Mix or LDA, QDA-Mix > QDA.
  • less informative feature:
    • Our experiments use DINO and RESNET models, which provide features of different quality: DINO offers better features (ie., distributions easier to model) than RESNET. This has a clear impact in Fig.3, for instance.
    • The ablation study over layer representation in RESNET (Fig. 12 App. H) shows that features are what matters most in practice when modeling the data distribution in order to improve the unlearning performance

Yet, we agree these clues were not clearly pinpointed. The revision does a better job.

Moreover, we propose a new experiment (See App. G.2.) based on datamodels (K. Georgiev et al., ICLR2025) proxies, where our proxies are crafted from inaccurate models (up to 80% training accuracy instead of 99%). This shows the real interest and adaptiveness of our ETAMAX procedure.

Dear Reviewer,

Please do not let us down. We have devoted a great deal of time and effort to revising the submission. We dare think that this new version meets your expectations.

Regards

Reviewer_1AryOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

the papers attempts to provide proxy-based machine unlearning methods by introducing a theoretically grounded framework that provides the upper bound of the algorithm with KL-divergence guarantees relative to retraining and achieving closer alignment with the ideally retrained model.

Critical strengths

discussing the proxy unlearning by relying on the data distribution.

Critical weaknesses

the research question is not clear. from the abstract you provide a very description about what is this research about and you inroduce your research question. However, reading the abstract is confusing and doesn't provide a clear desciption about the research question. the research question is not clearly defined, the prelimanaries are not well structured or described to properly and clearly formulate the research.

the description seems incohisive, poorly written without any revising from the authors. it is extermely hard to follow the idea and their description and the reader should constantly make assumptions.

the introduction is too bad:

  • incoherent description
  • lack of reference and citations
  • claims and references to the unlearning algorithms without any citations.

line 107) what is l_theta? the problem formulation is clearly and properly formulated. line 109) disconneted and incoherent writing.

the plots and figures are not up to the conference standard.

Questions for authors

Unfortunately the papers need to be havily revised to actually give a fair chance for review.

Comments on the paper

2 comments

Dear reviewers,

Thanks to your reviews, we have uploaded a revised version of our paper with the following main improvements:

  • A practical estimation testing whether the main assumption required by Prop. 1 holds (see App. D). This test is now integrated into Alg.2 ETAMAX to improve our theoretical reliability. This addresses a concern shared by reviewers 7eTU and 47e8
  • Integration into our theoretical framework of the paper (DMM) ATTRIBUTE-TO-DELETE: MACHINE UNLEARNING VIA DATAMODEL MATCHING, K. Georgiev et al., ICLR2025; and study of the improvement due to ETAMAX. (See App. G.1 and G.2)
  • App. D and G also contain new experiments to outline:
    • the impact of inaccurate proxies or less informative features
    • the need to test the assumption related to Prop. 1
    • the need for adaptive scaling by comparing the adjustment strategies for the same proxies.

We rephrase or clarify ambiguous claims pinpointed by the reviewers; all revisions appear in blue in the new version of our submission.

Dear reviewers,

We’re about halfway through the discussion period and the authors have responded to your reviews. This is a friendly reminder to check these responses and respond if necessary.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.