acceptodds
Under review as a conference paper at ICLR 2027

A Few Bits Break Forgetting: Bit-Flip Relearning Attack against Machine Unlearning

Abstract

Machine unlearning (MU) is an emerging paradigm to remove specified knowledge from trained models. Despite significant progress, recent studies reveal that unlearned knowledge can be recovered through relearning attacks, raising concerns about MU’s robustness. However, existing relearning attacks typically rely on fine-tuning and model update capability for dense parameter modifications, making them detectable by deployment-time integrity verification and less practical. This raises a critical question: can relearning attacks be achieved through only minimal runtime bit-level modifications? As bit-level updates are sparse and discrete, two challenges arise: balancing relearning with utility and efficiently identifying effective bit flips. Existing bit-flip attacks mainly degrade model behavior, which fundamentally differs from relearning and fails to address the challenges. In this paper, we propose **BFRA**, a **B**it **F**lip **R**elearning **A**ttack that is applicable across tasks and MU methods, enabling fine-tuning-free relearning through minimal bit flips. To balance relearning and utility, BFRA formulates relearning as a constrained discrete optimization problem that maximizes forgotten knowledge recovery while constraining retain utility drop. To improve efficiency, BFRA adopts a progressive bit-flip strategy combining First-Order Candidate Selection and Trial-Loss Bit Picking. Extensive experiments across 3 tasks, 14 model-dataset settings, and 12 MU methods show that BFRA substantially recover forgotten knowledge with only a few dozen bit flips while keeping high retain utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Bit Flip Attack, Relearning Attack, Robustness, Privacy, Image Classification, LLM, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nXFCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Machine unlearning removes knowledge associated with a forget set from a learned model. The unlearned knowledge can resurface through relearning attacks. Typical relearning attacks require fine-tuning the unlearned model, which leads to significant changes to the model parameters. The paper proposes a method, BFRA, to recover unlearned knowledge by flipping a small number of bits in the model parameters. They formulate the goal of identifying a small number of bits to flip that minimizes loss on the forget set while maintaining small loss on the retain set as a discrete optimization problem. They select candidate bits by solving a first-order approximation of the problem. Then they select the actual set of bit flips based on the actual realized loss reduction while maintaining the retain accuracy constraints. To make the method more efficient, the set of eligible bit flips is restricted to sign and exponent bits. In the experiments, restricting to this subset of bits was shown to be effective. High forget accuracy was achieved while maintaining retain loss by only flipping a small number of sign and exponent bits.

Critical strengths

With only a small number of bit flips, BFRA recovers a high forget accuracy while maintaining comparable retain accuracy to the original unlearned model. The empirical measurements of the runtime of BFRA demonstrate that solving the first-order approximation of the optimization problem is not too expensive. Restricting the bit search space to sign and exponent bits increases efficiency, and with this restricted search space, BFRA can still achieve high forget accuracy.

Critical weaknesses

In the main experiments, BFRA has access to the entire forget set, and then the evaluation metric is the accuracy on that forget set. This doesn't necessarily mean that BFRA has managed to recover unlearned "knowledge" that generalizes beyond the exact examples in the forget set. For example, for the TOFU experiments, could the authors evaluate the accuracy of the various methods on reworded or related versions of the questions in the forget set? There are some experiments where BFRA is only given a subset of the forget dataset, and then forget accuracy is evaluated on a held-out portion. I think these experiments should be expanded to demonstrate that BFRA can resurface forgotten knowledge, rather than just specific forgotten examples.

Since BFRA solves an approximation to the discrete optimization problem, there's no guarantee that the retain accuracy constraint will be satisfied. The procedure simply keeps flipping bits (increasing T) until the retain accuracy drops below a desired level. It needs a manual check to ensure that the retain accuracy is satisfied. It seems that this retain accuracy constraint and manual check could be added (or incorporated with a slight change in objective) to any standard bit-flipping attack.

Questions for authors

In the comparison with baseline bit-flip attack methods (Table 4), were the baseline methods also restricted to sign and exponent bits? In the ablations, the authors establish that the bit search space can significantly impact the effectiveness of the attack. It would be good to compare BFRA to other bit-flip attacks under the same bit search space to see how much of the improvement can be attributed to this restricted search space versus the optimization of BFRA?

The baseline bit-flip attacks target a different objective than BFRA. Can the baseline objectives be easily modified to target the same objective as BFRA? With the modified objective, how do these methods compare to BFRA?

See the weakness section for my concern on the distinction between recovering knowledge vs exact examples. How does fine-tuning compare to BFRA on recovering forgotten knowledge beyond the exact examples in the forget set? How does fine-tuning compare to BFRA when only a subset of the forget set is given?

Reviewer_FKZGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Bit-Flip Relearning Attack (BFRA) against machine unlearning. BFRA uses a first-order approximation to estimate the loss change caused by each bit flip. BFRA then selects candidates through trial-loss evaluation to recover forgotten knowledge while preserving model utility. Experiments across image classification, image generation, and LLMs show that a small number of bit flips can substantially recover forgotten knowledge.

Critical strengths

None

Critical weaknesses

First-order Taylor approximation. The key methodological novelty of the paper is using a first-order Taylor approximation to identify promising bit flips. However, I am not convinced that this approximation is good, especially when δw\delta w is relatively large. Neural networks are generally highly nonlinear, so I wonder if there is a way to justify this assumption better (beyond Appendix C. 4). An empirical evaluation of the approximation error would make the analysis more convincing.

Baselines. It would be useful to compare Stage 1 with simpler candidate-selection strategies, such as random selection or ranking candidates by weight magnitude, to show the effectiveness of the approximation.

Threat model. The threat model seems not realistic and a bit not convincing to me. A more plausible and practical scenario would be to make a deployed, unlearned model reveal forgotten knowledge without requiring the attacker to have access to the forgotten knowledge (without access to forget set/forget-related set).

Questions for authors

Please see the Weaknesses.

Reviewer_BdvWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BFRA, a whitebox bit flip attack that aims to recover forgotten knowledge from unlearned models while preserving utility on retained knowledge. Given access to the model and forgotten dataset and retained dataset, BFRA formulates recovery as constrained discrete parameter optimization. Each iteration computes gradients, then uses first order estimates with pruning and layer quotas to shortlist candidate bit flips, and evaluates their actual loss changes before applying one flip under retain loss constraints. The search focuses on sign and exponent bits. The authors evaluate BFRA across image classification, image generation, and language modeling, covering 14 model dataset settings and 13 unlearning methods. Comparisons with conventional finetuning and existing bit-flip attacks, together with ablation studies, report substantial recovery with tens of bit flips in many settings.

Critical strengths

1.The paper is well structured and easy to follow.

2.The authors design a systematic search procedure to identify important weights and select bit flips in their sign and exponent bits. In multiple experimental settings, the method effectively recovers forgotten knowledge with tens of bit flips while well preserving performance on retained data.

Critical weaknesses
  1. Unclear “fine-tuning-free” claim. BFRA still uses data and repeated gradient calculations to update weights. The distinction from fine-tuning is not clearly explained.

  2. Unclear advantage over fine-tuning. In the LLM comparison, five epochs of fine-tuning achieve better recovery and preserve other abilities better than BFRA.

  3. Missing baselines and cost comparisons. The paper lacks a baseline that fine-tunes only weights selected by gradient magnitude. It also does not compare runtime, memory usage, or FLOPs against fine-tuning.

  4. Insufficient evidence of practical feasibility. The attack requires substantial model and data access, but the paper does not demonstrate that its selected bit flips can be carried out on real hardware. Its practical benefit under these assumptions remains unclear.

Questions for authors
  1. BFRA still needs both forgotten dataset and retain dataset, computes gradients in every round, and uses them to update model weights. It appears to replace optimizer.step() with a search for which bit to flip after loss.backward(). The forward and backward passes still require substantial computation and memory, and testing candidate flips adds further cost. Could the authors explain why this should be considered “fine-tuning-free,” rather than a different way of updating weights during finetuning?

  2. In the LLM comparison, BFRA recovers more forgotten knowledge than one epoch of finetuning, but five epochs recover more than BFRA and preserve other abilities better. It is therefore unclear when BFRA would be the better choice.

  3. BFRA changes a few weights, why not compare it with a simple method that selects weights with a proportion largest gradients and finetunes only those weights? Using the same data and changing a similar number of weights would help show whether searching for specific bits is actually necessary. Also, there is no running time, memory usage, FLOPs reported when compared with finetuning methods. It is hard to detect if the BFRA is a more advanced technique than finetuning.

  4. BFRA requires white-box access to the model, both forget-related and retain-related data, and repeated gradient calculations. These assumptions already allow the attacker to fine-tune a local copy of the model. It is therefore important to explain what practical benefit BFRA provides over ordinary fine-tuning or fine-tuning only a few selected weights. If the benefit is lower cost with better recovery, this should be demonstrated through direct comparisons. If the benefit is that the attacker can flip a few bits in the deployed model but cannot replace it with a finetuned model, the paper should provide stronger evidence for this scenario and show that the selected flips can actually be achieved on real hardware. Without such evidence, the practical value of the additional search remains unclear.

If these questions are not well addressed, the score will be dropped.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.