acceptodds
Under review as a conference paper at ICLR 2027

HERA: An Empirical Audit of Partition Guarantees on Unlearnable Data

Abstract

Unlearnable examples perturb training images to impair learning, but recovery accuracy alone does not establish stability to later record edits or equality to retraining after deletion. We use HERA, a testbed of deterministic hash-partitioned ridge ensembles over fixed image views or frozen ImageNet-pretrained ResNet-18 features, to audit these properties on four archived CIFAR-10 protection instances. The partition guarantees are established; our contribution is an empirical comparison of clean-label allocation, regularization and shard count. Equal-weight ensembles combining 40,000 protected records with 2,000 clean records improve mean accuracy and certification over protected-only ensembles at all 32 tested five-shard instance–penalty settings, without uniformly dominating clean-only. Their certificates cover edits to the combined training set. Training-only shard-local cross-validation retains higher combined mean one-record certified accuracy than both separate options for every instance, although the NTGA difference from protected-only remains unresolved. The selector maximizes fold accuracy on each condition's training distribution using a previously explored penalty grid; it does not optimize ensemble certification. Tuned combined CUDA reaches 71.30% certified accuracy versus clean-only's 69.71%, but remains 4.07 accuracy points lower. Separate-resource comparisons expose strong regularization sensitivity: clean-only certification peaks among tested shared penalties at 70.21%, versus CUDA's 64.92%, before declining at larger penalties. Small clean shards motivate a near-interpolation hypothesis, not an established mechanism. Separate pooled additions reveal clean-budget dependence, with CUDA reducing accuracy at 2,000 clean labels despite helping at 250. Sparse deletion checks match local and fresh states and predictions while reusing untouched shards. These fixed-feature results motivate joint accuracy/certification reporting with shard-local selection, not claims of optimal allocation or robustness to dense perturbations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Unlearnable Examples, Machine Unlearning, Certified Robustness, Data Deletion, Sharded Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hiFTOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper proposes an empirical framework, HERA, for assessing different methods aiming to ensure guarantees on "unlearnable data". I found this paper extremely difficult to read -- I have done substantial prior work, e.g., on ensemble classification methods, but I found it very difficult to parse this paper, primarily because the authors do not do a good job of making the paper reader-accessible. There is a lot of jargon that is not defined and a lot of motivation that is missing.

Critical strengths

Again, I had great difficulty following this paper.. the main strength of the paper is in defining a sound test-bed/testing framework for evaluating "unlearnable example" approaches. The authors have also conducted extensive experiments.

Critical weaknesses

There is much terminology and motivation that is not well-defined in this paper: shard is nowhere defined.; CA(q) is used throughout the paper but is only defined in sec. 5.3. In Figure 1, what is the purpose of the clean routing set ? This is not explained.

Both Theorem 1 and Theorem 2 appear very straightforward.

Some experimental results are not given convincing explanation by the authors (gain of 23.49 points)

Questions for authors

Line 42: "For an authorized curator, however, recovery answers..." -- What data does an authorized curator have access to ? Original data, or only corrupted data ? Line 43: "After training, a few records may need to be corrected or deleted" -- please give scenarios where correction/deletion may be needed. Line 49: "correct certificate coverage" -- please explain what certification "coverage" means here. "savings when deletions touch many shards" -- shards are never truly defined in this paper -- and "savings" with respect to what ? Line 74: CA(1) -- CA(q) is used throughout the paper, but isn't defined until sec. 5.3 -- is this what is meant by "certified coverage" ? Line 96: "error-minimizing noise" -- unclear what this is. Line 106: What is the main contribution/distinction of Hera, compared to DPA ? P. 131: what is the "frozen variant" here ? Line 178: For each shard, are you selecting one of the possible views (based on "routing accuracy" -- that is, the routing set is treated as a validation set for selecting the view ?) Line 266: "how much correct accuracy is certifiable" -- please define what certifiable means here.

Gain of 23.49 points -- "locates the gain with identifying its cause" and "HERA's bagging advantage remains unresolved" -- what are the causes for these findings ? -- Can you explain the behavior in Figure 2 (as mean-loss penalty lambda_j is varied ?)

Reviewer_srkSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates prediction recovery, certified stability against training-record modifications, and exact unlearning in the context of UEs. The HERA framework leverages shard-level voting for prediction certification and localized retraining for exact data deletion. Experiments are conducted on four UE instances from CIFAR-10 to examine how various settings affect prediction accuracy and certified accuracy.

Critical strengths

The paper provides relatively thorough experimental evaluations, including comparisons between clean and protected data, sensitivity analyses of regularization strength and shard count, and additional experiments examining feature dimensionality and model configurations. These ablations help identify how different settings affect prediction accuracy and certified accuracy.

Critical weaknesses
  1. My primary concern is limited methodological novelty. The core methodology largely builds upon existing mechanisms for deterministic data partitioning, vote-based certification, and localized retraining. Although the paper applies these mechanisms to the recovery of UE and provides extensive empirical analyses, the findings remain largely descriptive, focusing on emprirical observations. The paper does not sufficiently explain why these phenomena occur or identify the underlying data or model properties that lead to these behaviors.
  2. The experimental evaluation is largely restricted to four pre-generated UE instances on CIFAR-10 and ridge regression classifiers trained on fixed feature representations. The current evidence is insufficient to establish whether the main findings generalize to modern deep neural networks with end-to-end representation learning.
Questions for authors

How would the results be affected if the experiments were conducted using an end-to-end deep neural network for representation learning? Would the primary empirical findings still hold?

Reviewer_T4SmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The topic of the paper is recovery from unlearnable examples (UE). To this end, they build an empirical testbed to evaluate UE protection instances that are derived from CIFAR-10. There are three metrics that are evaluated:

  • recovery accuracy on held-out data
  • prediction stability against sparse training data edits
  • algorithm-relative machine unlearning using isolated shard retraining (SISA).
Critical strengths

The experiments seem reasonably done.

Critical weaknesses
  • Not sure how well motivated is the problem - UE typically perturb almost all of the training set. But the paper deals with DPA, which shows prediction invariance certificates for a few tens of edits. Not sure if these two are matched.
  • The presentation is extremely difficult to follow, with almost 100% of the paper is AI-generated content/text. Just try parsing the first sentence of the abstract. It is the responsibility of the authors to ensure that the paper is accessible and self-contained for the reviewer.
Questions for authors

Why didn't you generate new UE datasets using the publicly available official repositories for EMIN, REM, NTGA, and CUDA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.