acceptodds
Withdrawn from ICLR 2027

Bounding Retraining Equivalence and the Deletion Floor in Materials Machine Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In materials machine learning, closely related retained structures can support accurate property predictions even after a specific record is removed. Post-deletion prediction error is therefore an ambiguous measure of machine unlearning. We define the deletion floor as the expected loss on a deletion request under a specified retraining procedure. Standard indistinguishability constraints give a sharp interval bounding an unlearning update’s target loss around this reference. A conditional neighbor bound links a low deletion floor to retained-data fit, prediction regularity, and local label agreement; an exact ridge identity separates residual fit from the prediction change caused by deletion. In controlled redundancy sweeps, retaining just one related structure reduces median normalized retraining loss by approximately eightfold. In a paired Materials Project study spanning two fitting regimes, the regime with the lower deletion floor also shows a larger prediction change on more than 50% of shared requests. Comparisons with approximate updates and the original model distinguish deliberate suppression of target predictions from preservation of overall utility. Request-level unlearning evaluations should therefore report retraining reference loss, prediction change, and retained utility together, interpreting post-deletion accuracy in light of what retraining itself leaves behind.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
machine unlearning, retraining equivalence, data redundancy, leave-one-out stability, exact unlearning, ridge regression, materials informatics, materials property prediction, formation energy, crystal structure representation, deletion floor, retained utility, bounded-loss transfer, target suppression

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PkcXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies unlearning for materials-property regression. It start from the assumption that similar structures remain after deletion, so a retrained model can still predict the deleted record well. The authors define the “deletion floor” as the retrained model’s expected loss on that record. They attempted to prove that an update close to retraining must have a loss near this floor. They used an exact ridge formula to separate the floor from the prediction change caused by deletion. They also propose a per-request reporting protocol. Experiments mainly use ridge regression on synthetic data.

Critical strengths

There are two highlights about this paper.

First, the paper makes concrete the point: in regression unlearning, high post-deletion accuracy is not a failure in itself.

Second, Near-duplicate structures (e.g., ionic substitutions) are common in materials-property data. This makes the domain a reasonable and under-explored test case for studying redundancy effects in machine unlearning evaluation.

Critical weaknesses

Severn concerns about the paper

First, the theory involved is largely known. For example, Theorem 1 is a standard step in differential privacy (bounding expectations of bounded functions), and its “sharpness” follows from a two-point construction. Beyond restating these known results, the paper does not explain how they help with the core difficulties of unlearning in materials-property prediction.

Second, the referenced baseline seems not to be a true retraining. Feature standardization and the RFF bandwidth are computed with the deleted record and kept fixed. It is suspected that the “retrained” reference therefore still depends on the deleted record through preprocessing.

Also, there is little connection between theory and experiments.

Questions for authors
  1. Why is this unlearning study framed around of the materials? So far as the review know, the deletion floor, the bounds and the protocol look fully generic and would apply to any regression task with redundant data.

  2. Following question 1, does any part of the method or analysis depend on materials-specific structure?

  3. How many retrains does each request need to estimate the floor for a NN model?

  4. What decision does the module FloorScore support (given its weak correlation and its use of the deleted label)?

Reviewer_ACfhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses an important question in how to evaluate Machine Unlearning algorithms. Given a set of data points to be removed from a pre-trained model, the authors argue that an evaluation protocol needs to distinguish between predictability (model performance) and sensitivity to deletion. The core intuition is that an example will still be accurately predicted (e.g. classified) because retained examples provide information, or act as support vectors. The theoretical contribution is proving that an unlearned model should have a similar average error (assumed to be loss) to retraining. The assumption is the error measure is bounded and algorithms produce similar distributions (weight spaces). To validate the theory under said assumptions, the authors develop a synthetic dataset with similar examples (e.g. one forgotten example would have some support vectors in the retain dataset). The authors find leaving support vectors in the retain set leads to a reduce in the loss for the forgotten example, which matches intuition. The paper compares a range of unlearning algorithms on a materials dataset as well, making the analysis use case specific. Finally, an analysis is done to check the loss relationships between similar input points, but the associations are modest and linked to the model type (ridge, graph neural networks).

Critical strengths

The strengths of the paper:

  1. The authors derive a theoretical explanation of why accurate predictions can survive deletion. This result was previously observed in empirical studies, and the mathematical result matches the intuition. They use this result to clarify in Machine Unlearning evaluation, high accuracy on forgotten sample is an artefact of the retain set and shouldn't be qualified as unsuccessful unlearning.
  2. An empirical evaluation spanning paired deletion requests. The experiments aim to show that prediction accuracy is not the same as deletion sensitivity. The main takeaway is that aggresively forgetting points (through the lens of the forget loss) is damaging if the objective is to approximate retrain from scratch (which may still contain similar examples to the ones forgotten aka. support vectors in the retain set).
Critical weaknesses

The weaknesses are discussed below:

  1. The mathematical framework is sound. However, several building blocks are classic tools: the triangle inequality argument, ridge for leave-one-out. The paper organises them into an argument for unlearning, but the mathematical novelty is modest, and the result is not new in machine unlearning literature. It is known in literature that similar points will retain information and thus, if the target is to be as close as possible to the retrain-from-scratch, then no action is a valid unlearning routine for low memorised data points. In general, outliers are highly memorised, and thus points with very few support points in the retain. However, Machine Unlearning is target-subjective: for example, a valid target is to scrub the information from a model (without considering the retrain-from-scratch proxy). An example of that is removing from NLP models copyright content, regardless of how much surrounding context exists in the retain-set. Therefore, the contributions can be narrowed down to an improvement in Machine Unlearning evaluation when the goal is to approximate retrain-from-scratch, and not a general rule for Machine Unlearning applications.
  2. The empirical validation is insufficient. The experiments do not account for the assumptions behind the theoretical bounds. The results validate deleted examples can remain predictable, which is a known result. Previous literature looked at this through the lens of memorisation. There exists a disconnect between the theory and the experimental analysis: for example, the distributional equivalence, smoothness assumptions around neighbour bound, for the real-world materials use case. This makes the empirical validation modest, and opens the questions whether an algorithm can be built based on the theoretical framework, rather than applying existing ones.
Questions for authors
  1. Is there a way to demonstrate the bounded-loss interval in a certified unlearning setting, with known epsilon delta bounds? This would prove how tight the theoretical framing is, and whether an algorithm can be derived based on the characterisation, ideally with a certificate.
Reviewer_xGT2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that accurate prediction after deletion does not by itself imply failed unlearning and proposes to define a deletion floor, i.e., the expected loss of a specified retraining procedure on a deleted record. It derives bounds for it based on some assumptions e.g., local smoothness, retraining equivalence and perform experiments on synthetic redundant data with random-feature ridge models on Materials Project.

Critical strengths
  1. The paper carefully separates three questions: whether a deleted point remains predictable, whether deleting it changes the model, and whether the updated model matches retraining, which is important in machine unlearning.
  2. The ridge-regression examples provide a straightforward retraining reference.
Critical weaknesses
  1. To my knowledge, the finding/motivation that accurate prediction after deletion does not by itself imply failed unlearning has been pointed out and well-studied in toy examples, and the challenge lies in providing a comparably strict analysis for deep models. In other words, I think this paper does not tackle a sufficiently challenging problem and still focuses on models like ridge regression, which undermines its contribution.
  2. Theoretically, I am afraid this paper might lack some important related work comparisons and does not clearly isolate its theoretical novelty. In particular, it should better clarify its contribution when compared with retraining-based analysis and certified unlearning theory [1,2]. To my understanding , the deletion floor is the retrained model's loss on the forget data, thus theorem 1 appears to be a bounded-observable consequence of well-known (\epsilon ,\delta)-indistinguishbility. Could the authors further clarify their main distinction and novelty? Also, I think the leave-on-out in ridge regression has been well-studied, and it seems not propose a new analysis perspective/modeling approach. Therefore I hope the authors could further clarify and better highlighted their theorectical contribution by comparing with related works.

[1] Forget Unlearning: Towards True Data-Deletion in Machine Learning, 2023, ICML [2] Certified Unlearning for Neural Networks, 2025, ICML

Questions for authors
  1. Could the author explain what concrete materials-science deletion request motivates this setting, and clarify whether machine unlearning in material area is special and different from other unlearning problem or it just serves as a redundant regression dataset? From existing paper, it seems that there is no materials-specific problem or algorithm design is proposed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.