acceptodds
Under review as a conference paper at ICLR 2027

Towards Probabilistic Unlearning Attacks on Large Language Models

Abstract

Evaluating large language model (LLM) unlearning requires determining whether target information remains accessible after its removal. However, unlearning is commonly evaluated and attacked through single responses, even though target information may remain exposed under repeated sampling. This probabilistic exposure creates an additional attack surface beyond single-response relearning attacks. We introduce the first probabilistic unlearning attack that directly optimizes target-fact disclosure over sampled responses. We derive finite-sample confidence bounds, characterize its idealized dynamics, and establish conditions for greater disclosure than single-reference supervised fine-tuning (SFT). Across Harry Potter QA, TOFU, and WMDP, we show that single-response metrics can miss target exposure, GRPO recovers unlearned information more effectively than state-of-the-art SFT, and it substantially compromises RULE-NPO, a recent method designed to resist probabilistic leakage. Our results reveal probabilistic outputs as an important attack surface for LLM unlearning. Anonymous source code: https://anonymous.4open.science/r/relearning_attack_grpo-321B

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, LLM Memorization, Relearning Attack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TJoKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies probabilistic relearning attacks on unlearned LLMs. It introduces GRPO with a binary fact-disclosure reward to increase relearning information's exposure and evaluates attack results on questions across Harry Potter, TOFU, and WMDP. It also provides finite-sample leakage bounds and an idealized analysis of amplification and comparisons with single-response SFT.

Critical strengths
  1. Relevant problem setup: The paper highlights that single-response evaluation and sampling without subsequent attacks may not be sufficient for attacks. Introducing GRPO as a multi-turn attack is a very interesting and intuitive idea.
  2. Clear objective: The design of the reward provides an intuitive connection between reinforcement learning and target-fact exposure probability. Also, this paper provides proof for corresponding ideas and theoretical grounds.
  3. Useful empirical evidence: Several evaluation and cross-attribute probes examine recovery beyond training answers, while retained-utility and attack performance also help emphasize the balance of the proposed methods.
Critical weaknesses
  1. Several notations and proof problems: In Line 162, the denominator is $\bar{\sigma}$, when all rewards in the group are the same, it could become 0 for a denominator. In Line 272, the meaning of $\eta$ is not defined. The Clopper–Pearson interval in lines 1113–1114 seems to correspond to Lemma 1/Theorem 1 instead of proposition 1. In line 887, the proposition 3 didnt show up earlier.
  2. Novelty issues: This paper is built on the relearning attack that is proposed by [1] and introduce GRPO with grouped sampling and a binary fact-disclosure reward. The shift from single response supervision to GRPO's group response is interesting, but it remains unclear that this forms a methodological advance beyond a straightforward application of an existing RL algorithm. For example, could the two-stage approach, SFT warm up + GRPO RL, achieve stronger attack performance?
  3. Experiments: Hardware requirements and training timeare not reported for SFT and GRPO. Ablations on GRPO group size could be helpful. Adding experiments on MUSE mught help evaluate generalization to corpus-level forgetting and verbatim memorization beyond current QA-focused evaluation. Also there are several works specifically focusing on robustness to relearning attacks, following up on SAM+NPO. Adding them could be helpful on evaluating further performance.

[1] Hu, Shengyuan, et al. "Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors

See weaknesses. Specifically,

  1. Could SFT+GRPO improve overall performance than merely using GRPO or SFT?
  2. What's the computational overhead compared to SFT?
  3. What's the ablation results on using different group size in GRPO?
  4. What's the result on MUSE dataset and more recent works that focus on robustness to relearning attacks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TJoK

We thank the reviewer for recognizing the novelty of the problem we study, the clarity of our objectives, and the experimental evidence supporting our claims.

W1. Notations and proof problems

Thank you for identifying these notation and cross-reference issues. We address each point explicitly:

a) Zero reward variance (line 162)

The displayed advantage normalization omitted the numerical safeguard used in implementation. We now write the stabilized expression explicitly, where εnum\varepsilon_{\mathrm{num}} is a small positive constant. Ai=ri−rˉσˉ+εnum,εnum>0.A_i = \frac{r_i - \bar{r}}{\bar{\sigma} + \varepsilon_{\mathrm{num}}}, \qquad \varepsilon_{\mathrm{num}} > 0. When all rewards in a group are identical (ri=rˉr_i = \bar{r} for all ii), every numerator is zero, so Ai=0A_i = 0 and the group contributes no reward-based policy-gradient signal (only the KL regularization term remains active).

b) Notation near Equation (11) (line 272)

We now explicitly define η>0\eta > 0 at Equation (11) as the step-size parameter of the idealized KL-proximal update (distinct from the neural optimizer learning rate).

c) Clopper–Pearson reference in Table 5 caption (lines 1113–1114)

We have corrected the caption of Table 5 to reference Lemma 1 (which defines the Clopper–Pearson interval endpoints) and Theorem 1 (which states its finite-sample coverage guarantee), rather than Proposition 1.

d) Amplification reference in Appendix A.5 (line 887)

We have corrected "Proposition 3" on line 887 to "Theorem 3", as Equation (23) follows directly from the convergence bound in Equation (13) using the matched coefficient βΔ\beta_\Delta.

We have also audited all remaining theorem and equation cross-references across the paper and appendix.

W2/Q1. Methodological contribution and sequential SFT + GRPO results

Methodological and theoretical advance: Beyond applying GRPO, our work makes two core contributions:

  • (1) We identify a fundamental mismatch between how unlearning is increasingly evaluated (probabilistically across sampled outputs) and how relearning attacks optimize (maximizing single-reference likelihood), showing that even defenses explicitly designed for probabilistic robustness (RULE-NPO) fail under distributional optimization;
  • (2) We prove in Theorems 2–4 and Proposition 2 that single-reference SFT wastes part of its KL divergence budget altering conditional probabilities within the target-exposing set (Cq(π)>0C_q(\pi) > 0 in Eq. 22), whereas event-level reward optimization allocates the full KL budget toward shifting total probability mass into Lq\mathcal{L}_q.

Sequential SFT + GRPO evaluation: Following your suggestion, we evaluated a two-stage SFT warm-up + GRPO pipeline across four unlearning methods (WGA, SatImp, PDU, and SimNPO):

ModelAttackMean leakageLeak@128 /20Greedy /20
WGASFT + GRPO0.7561916
WGAGRPO0.7711916
SatImpSFT + GRPO0.7321916
SatImpGRPO0.6981815
PDUSFT + GRPO0.7411918
PDUGRPO0.7471916
SimNPOSFT + GRPO0.5951513
SimNPOGRPO0.6021513

SFT + GRPO does not consistently outperform standalone GRPO: while warm-up improves recovery on SatImp (19/20 vs. 18/20) and greedy recovery on PDU (18/20 vs. 16/20), standalone GRPO achieves equal Leak@128 coverage and slightly higher mean leakage on WGA, PDU, and SimNPO. This aligns with Theorem 4: because SFT warm-up initially concentrates policy mass toward a single reference string, it can reduce sampling diversity prior to RL exploration while adding extra compute overhead.

W3/Q2. Computational overhead and hardware requirements

All TOFU experiments (Llama-3.2-1B-Instruct, LoRA rank 8) were run on a single NVIDIA RTX A6000 (48 GB) GPU. The wall-clock training time and peak memory usage are:

AttackGPU-minutesPeak memory (GB)
SFT0.48.1
GRPO (G=16G=16)6.214.0
SFT + GRPO7.216.0

Although GRPO incurs higher relative cost than SFT due to generating G=16G=16 on-policy rollouts per prompt, its absolute cost remains very low (6.2 minutes and 14 GB VRAM on a single GPU), making it practical for standard unlearning robustness audits. We have added these hardware and runtime details to Appendix D.4.

W3/Q3. GRPO group-size ablation

As suggested, we evaluated GRPO across group sizes G∈{4,8,16,32}G \in \{4, 8, 16, 32\} on TOFU SimNPO checkpoint A.

Group size GGLeak@128 /20Keyword matches /2,560
4161,635
8161,424
16171,723
32181,791

Increasing the group size GG monotonically improves Leak@128 coverage from 16/20 (at G=4,8G=4, 8) to 17/20 (at G=16G=16) and 18/20 (at G=32G=32), with G=32G=32 also achieving the highest total keyword-match count (1,791/2,560). Larger groups reduce variance in the group-normalized advantage estimate and increase the likelihood of sampling at least one positive-reward disclosure on harder prompts, whereas G=16G=16 offers a strong balance between recovery and runtime.

W3/Q4. Results on MUSE and recent robustness work to relearning attack.

To evaluate generalization to corpus-level forgetting beyond QA benchmarks, we tested SFT and GRPO on MUSE using both standard (SimNPO) and sharpness-aware relearning-resilient (NPO-SAM, Fan et al., 2025a) checkpoints, measuring held-out ROUGE-L recovery:

Unlearning MethodUnlearned (No attack)SFT (LR=10−3)(\text{LR} = 10^{-3})GRPO (LR=10−3)(\text{LR} = 10^{-3})
NPO-SAM0.4410.4880.524
SimNPO0.4860.5020.549

GRPO achieves substantially higher held-out ROUGE-L recovery than SFT on both SimNPO (0.549 vs. 0.502) and the relearning-resilient NPO-SAM checkpoint (0.524 vs. 0.488).

Combined with our evaluations on NPO-SAM and TAR-Bio-v2 on WMDP (Table 4), RULE-NPO on TOFU (Table 2), and the newly added WGA, SatImp, and PDU results (shown in Response to Q1 above), these findings demonstrate that probabilistic policy optimization consistently recovers more residual information than single-reference SFT even against recent defenses designed to resist relearning and probabilistic decoding.

We also identified two additional recent works studying robustness to relearning attacks following up on SAM+NPO:

  1. Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimization (Wu et al., 2025)
  2. Robust LLM Unlearning Against Relearning Attacks: The Minor Components in Representations Matter (Xiao et al., 2026)

Unfortunately, (1) does not provide public code or checkpoints, and (2) does not release unlearned model checkpoints or provide code for the TOFU or MUSE datasets. Because training unlearning defenses from scratch requires substantial hyperparameter optimization that is difficult to validate fairly during the rebuttal period, we focus our empirical evaluation on publicly released unlearned checkpoints and have added both works to Section 6 (Related Work) and Section 5 (Discussion) as important directions for future evaluation.

AuthorsOpenReview ↗

Response by Authors

Dear Reviewer TJoK,

As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.

Best regards,
Authors

Reviewer_HePBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is an attack on machine unlearning for large language models. The success of machine unlearning lies in uncovering the supposedly unlearnt knowledge from the model. In the current practice, it is mostly done by testing for a single response (per unlearning query). This paper (correctly) argues that this is a limited view of testing for the success of an attack. The target unlearned information could possible be uncovered under repeated sampling. Based on this argument, it proposes a new probabilistic unlearning attack. that does an optimization of the disclosure of the target unlearned fact over multiple sampled responses. The proposed attack uses GRPO (unlike the use of SFT in many other attacks) and is shown to be successful where other normal unlearning and probabilistic unlearning attacks fail. The paper provides some confidence bounds by repurposing existing statistical results in the particular setting. The empirical evidence presented provides sufficient evidence for the claims.

Critical strengths

(1) The paper correctly argues that single response testing for machine unlearning attack success is a very limited view of the phenomenon. This is an important observation.

(2) The proposed GRPO based probabilistic attack that optimizes for a distributional fact uncovering response as characterize by the description via equations (7), (8) and (9) is a novel and interesting contribution.

(3) The amplification bounds theoretical results in Sections 3.3 and 3.4 are interesting, even though they are re-stating of some classical statistical results for this particular problem setting.

Critical weaknesses

(1) The experimental results are limited. In Tables 2, 3 and 4, the comparison is against only 4 methods. Ideally, the strongest, current unlearning methods must be identified. The proposed method should be tested against them. This would be a more principled approach towards evaluation. There are methods which do not fall under the "representative" categories mentioned in the first paragraph of Section 4.1 introduction.

(2) The related works section is not comprehensive. Many works are missing.

(3) The claim that it is the "first" probabilistic unlearning attack is undermined by the statements on line 046 about Scholten 2025 and on line 049 about Reisizadeh 2026 works. Perhaps, the claim should be toned down.

Questions for authors

(1) Please provide a more detailed evaluation against other unlearning methods, not covered in the current draft. This reviewer is deliberately not listing them for anonymity reasons.

(2) Minor point: in Section 4.1, it is not clear why Table 3 results are discussed before Table 2 results.

2 comments
AuthorsOpenReview ↗

Response to Reviewer HePB

We thank the reviewer for the positive assessment of our problem formulation, theoretical bounds, and empirical findings, as well as the helpful suggestions to broaden our unlearning method comparisons and related work.

W1/Q1. Evaluation against additional recent unlearning methods

In addition to the 8 unlearning methods evaluated across our original manuscript—which already include recent defense-focused methods such as RULE-NPO (Reisizadeh et al., 2026, designed to resist probabilistic Leak@k decoding), NPO-SAM (Fan et al., 2025a, sharpness-aware minimization for relearning robustness), TAR-Bio-v2 (Tamirisa et al., 2025, tamper-resistant safeguards), and IDK-AP (Yuan et al., 2025), alongside NPO, SimNPO, IdkDPO, RMU, and AltPO—we have expanded our TOFU evaluation to three additional recent unlearning methods: WGA (Weighted Gradient Ascent), SatImp, and PDU (Primal-Dual Unlearning), bringing our evaluation to 12 unlearning methods in total. We observe that GRPO still consistently outperforms SFT.

ModelAttackMean leakageLeak@128 /20Greedy /20
WGANo attack0.3701611
WGASFT0.7681811
WGAGRPO0.7711916
SatImpNo attack0.4721511
SatImpSFT0.7361815
SatImpGRPO0.6981815
PDUNo attack0.3991512
PDUSFT0.7271815
PDUGRPO0.7471916

W2. Expanding the Related Work section

Thanks for the suggestion! We have expanded Section 6 (Related Work) to provide a more comprehensive discussion of recent unlearning objectives—including token-importance and weighted-gradient methods (WGA, SatImp), primal-dual constrained optimization (PDU), and sharpness/meta-learning defenses against relearning—as well as recent developments in multi-sample auditing and model-tampering attacks.

W3. Clarifying the "first probabilistic attack" claim

Thanks you for pointing this out. We have removed the word "first" from the abstract. To clarify the distinction in Table 1: prior probabilistic works (Scholten et al., 2025; Reisizadeh et al., 2026) perform passive evaluation of a frozen unlearned model under repeated sampling (No attack in Table 1), without updating model parameters. Our work is the first to formulate an active parameter-updating relearning attack that directly optimizes target disclosure over the sampling distribution. We have revised the text to state this distinction precisely.

Q2. Ordering of Table 2 and Table 3 in Section 4.1

Thank you for catching this. We have corrected the ordering issue in the revised PDF.

AuthorsOpenReview ↗

Response by Authors

Dear Reviewer HePB,

As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.

Best regards, Authors

Reviewer_Mz7bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the knowledge leakage problem after forgetting in large language models, pointing out that traditional single-response evaluation may underestimate the information remaining in the model. The authors propose a probabilistic forgetting attack based on GRPO, which increases the exposure probability of forgotten knowledge by rewarding samples containing target facts. The paper also provides a confidence bound for the leakage probability under finite sampling and theoretically analyzes the potential advantages of this attack over traditional SFT. Experiments were conducted on Harry Potter QA, TOFU, and WMDP, showing that GRPO can recover more target knowledge than SFT in most settings and effectively attacks forgetting methods such as RULE-NPO.

Critical strengths
  1. This paper points out that a single answer may underestimate the knowledge remaining in the forgetting model and proposes the GRPO attack, which directly optimizes the probability of target fact leakage. Compared to the traditional SFT attack, this method is not limited to a fixed reference answer and can more comprehensively evaluate the robustness of the forgetting model. Confidence interval analysis under finite sampling also makes the evaluation results more evidence-based.

  2. The authors validated the method on Harry Potter, TOFU, and WMDP. GRPO recovered more target knowledge than SFT in most settings while maintaining relatively good model utility. In particular, the attack results on RULE-NPO show that even forgetting methods designed for probability leakage can still be vulnerable to subsequent optimization attacks, which has practical reference value for improving forgetting evaluation.

Critical weaknesses
  1. The conclusions of Theorem 4 are based on idealized distribution optimization and the same KL budget. However, actual experiments did not rigorously control the KL shift or computational overhead of the two attacks. GRPO uses multiple sampling and reward optimization, while SFT mainly fits a single reference answer and lacks comparison with stronger methods such as multi-reference answer SFT. Furthermore, the TOFU original question experiments in Appendix D also show that GRPO is not always superior to SFT. Therefore, it is currently difficult to determine whether the performance improvement comes from the probabilistic optimization objective or differences in training strategies and resource investment.

  2. The paper mainly uses keyword matching to determine whether the target fact is leaked. However, the appearance of the target word in the answer does not necessarily mean that the model has correctly recovered the relevant knowledge, especially in the case of negation, citation, or misstatement. Appendix D.3.1 also shows a significant difference between keyword matching and correct answer matching. In addition, the WMDP experiments still use traditional multiple-choice accuracy, rather than the probabilistic leakage metric emphasized in this paper.

Questions for authors
  1. The advantage of Theorem 4 is based on the same KL budget, but the actual experiments did not report the KL shift of the two attacks relative to the forgetting model, nor did they compare the training computation. Could the authors explain whether the resource investment of the two methods is comparable? If the same KL budget or computational cost is controlled, does GRPO still have an advantage? Furthermore, would using multiple reference answers for SFT narrow this gap?

  2. The paper considers the appearance of target keywords in the answers as knowledge leakage, but this approach cannot distinguish between correct answers, incorrect attributions, and negative statements. In Appendix D.3.1, NPO's 786 keyword matches actually only resulted in 613 exact answer matches. Could the authors report whether the relative performance of GRPO versus SFT still holds after using a more rigorous factual correctness judgment?

  3. Figure 4 in Appendix D shows that GRPO is not consistently superior to SFT on the original question. In Table 7, the training question matching rate of Llama-3.2-1B improved from 0.242 to 0.622, but the test question matching rate remained at 0.160. In contrast, the text using the new attribute questions achieved a more significant recovery effect. Can the authors explain the difference between the two evaluation methods? How can they determine whether the performance improvement comes from the recovery of forgotten knowledge, or from the fact that a specific question format is more likely to trigger existing answer patterns?

  4. Section 2 states that ideal forgetting should be close to a retrained model without the target data. However, the experiments mainly compare the models before and after forgetting, without directly evaluating the additional leakage of the attack relative to the retrained baseline. How did the authors rule out that some recovery results come from relearning during the attack phase, general knowledge inference, or accidental guessing, rather than residual target information in the model?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Mz7b

We thank the reviewer for the constructive feedback and insightful questions regarding KL and compute control, multi-reference SFT, factual verification, and the retain90 baseline.

W1/Q1. a) KL-constrained SFT and GRPO comparison; b) Multi-reference SFT compared with GRPO; c) Computational cost comparison

a) Comparison between KL-constrained SFT and GRPO

Thank you for requesting a direct comparison of distributional shift and computational cost. We report the realized training KL divergence (measured relative to the frozen unlearned model π0\pi_0) alongside held-out attack coverage (Leak@128) and retained model utility on TOFU SimNPO.

Before attack

CheckpointKeyword /2,560Greedy /20Leak@128 /20Model utility
SimNPO530590.687

After attack

KL coefficient β\betaAttack LREpochsSFT training KLGRPO training KLSFT Leak@128 /20GRPO Leak@128 /20SFT utilityGRPO utility
0.0110−410^{-4}10.1400.171890.7010.682
0.0110−410^{-4}30.4400.67011150.6520.502
0.0410−410^{-4}30.4500.50810120.6410.611

Across all three configurations, GRPO achieves higher Leak@128 than KL-regularized SFT. Specifically, across all tested SFT settings, SFT peaks at 11/20 (KL = 0.440), whereas GRPO at β=0.04\beta = 0.04 (KL = 0.508) reaches 12/20 and at β=0.01\beta = 0.01 (KL = 0.670) reaches 15/20, demonstrating a consistent coverage advantage at nearby KL budgets.

b) Multi-reference SFT compared with SFT and GRPO

To test whether multi-reference supervision narrows the gap predicted by Theorem 4, we construct multi-reference SFT by augmenting each relearning question in the TOFU dataset with three additional reference answers: paraphrases of the gold answer generated by the retain-only model (retain90), filtered to keep only responses that contain every target keyword and mention no held-out birth city. We then fine-tune the unlearned model with standard SFT over all four references per question (240 examples per epoch) using the same LoRA configuration and learning rate sweep as single-reference SFT.

Model / attackLeak@128 /20Greedy /20
SimNPO (No attack)95
+ Single-reference SFT119
+ Multi-reference SFT1410
+ GRPO1513

These results highlight two key findings:

  1. Empirical validation of Theorem 4: Multi-reference SFT substantially improves over single-reference SFT (14/20 vs. 11/20 Leak@128), directly supporting our theoretical analysis in Section 3.4: single-reference SFT underperforms because it spends KL budget concentrating probability mass onto one specific realization y⋆y^\star rather than the broader disclosure event Lq\mathcal{L}_q.
  2. Advantage of GRPO without external paraphrase generation: Even against this stronger baseline, GRPO achieves higher Leak@128 (15/20 vs. 14/20) and substantially stronger greedy recovery (13/20 vs. 10/20). Moreover, Multi-reference SFT requires an external oracle to generate and verify multiple clean paraphrases per query, whereas GRPO discovers diverse target-exposing completions directly from the unlearned model's own output distribution using only the target keyword.

c) Computational cost comparison and equal-compute behavior

The wall-clock time and peak GPU memory required to attack the TOFU SimNPO model (Llama-3.2-1B-Instruct, LoRA rank 8) on a single NVIDIA RTX A6000 GPU are:

AttackGPU-minutesPeak memory (GB)
SFT0.48.1
GRPO (G=16G=16)6.214.0

While GRPO requires more compute than single-reference SFT due to sampling G=16G=16 rollouts per prompt, both attacks are lightweight in absolute terms (under 7 GPU-minutes on a single GPU). Importantly, giving SFT an equal compute budget (15×15\times more training steps/epochs) does not close the recovery gap: as noted in Section 4.1, SFT is already trained to convergence of the supervised loss, and additional epochs simply overfit the single reference answers without improving held-out disclosure.

AuthorsOpenReview ↗

Response to Reviewer Mz7b

W2/Q2. a) Keyword matching vs. Exact match and LLM-judge verification and b) Why WMDP uses multiple-choice evaluation

a) Keyword matching vs. Exact match and LLM-judge verification

In Appendix D.3.1, the difference between keyword matches (786) and exact matches (613) on NPO arises from response length rather than factual negation or misattribution.

Leaks count any response containing the target city as a contiguous whole-word phrase, whereas Exact matches require the entire normalized completion to equal only the city name. Even when prompted to "Answer with only the city name", models frequently include the country or a full sentence, which fails string-level exact match while still genuinely disclosing the target fact:

Model response (Target: Santiago)Leak?Exact match?
SantiagoYesYes
Santiago, Chile.YesNo
Hermosa de Santiago, ChileYesNo
Carmen Montenegro was born in Santiago, ChileYesNo

To verify that keyword matches reflect true factual disclosure rather than negations (e.g., "was not born in Santiago") or misattributions, we evaluated the full-sentence post-attack responses on TOFU using an LLM judge DeepSeek-V4.1-Flash-1017. The judge found zero instances of negation across the reported TOFU completions, confirming that the relative advantage of GRPO over SFT (15/20 vs. 11/20 Leak@128) holds under semantic factual correctness verification.

b) Why WMDP uses multiple-choice evaluation

We follow the standard WMDP-Deduped protocol (Li et al., 2024; Deeb & Roger, 2025), where questions are formatted as 4-way multiple-choice questions (A/B/C/D) rather than open-ended generation. In a 4-choice setting, random guessing has a 25%25\% per-sample success rate, meaning repeated sampling over k=128k=128 draws would trivially yield Leak@128≈1−0.75128≈1\text{Leak}@128 \approx 1 - 0.75^{128} \approx 1 on nearly all questions. Instead, calibrated MCQ accuracy directly measures the probability mass assigned by the policy to the correct option relative to the distractors. Crucially, during the attack phase on the 20% relearning split, GRPO still operates probabilistically by sampling groups of G=16G=16 completions and rewarding selections of the correct answer, which transfers to higher held-out accuracy and MMLU utility than SFT on the remaining 80% evaluation split (Table 4).

Q3. Difference between the two evaluation setups

In the Original-QA protocol (Table 7, Figure 4), relearning and evaluation use the original TOFU forget-set questions. Because unlearning methods (such as SimNPO and NPO) directly suppressed the exact surface phrasing of these specific QA pairs, fine-tuning on a subset of them causes the model to memorize the training pairs (0.242 →\to 0.622) while remaining suppressed on the held-out original surface forms (0.160 →\to 0.160).

Motivated by Hu et al. (2025), who showed that suppressed entity attributes remain recoverable when probed through alternative phrasings, our Attribute protocol (Table 2, Figure 3) separates the attributes used for attack supervision from the attribute used for evaluation: the attacker trains on 60 questions covering other attributes (genre, book title, award, parents' profession) and evaluates on 20 held-out birth-city questions.

Three pieces of evidence confirm that the recovery in Table 2 reflects revived forgotten knowledge rather than a question format that triggers generic city-name patterns:

  1. Identical evaluation prompts before and after attack: Both the unlearned model (π0\pi_0) and the attacked model (πA\pi_{\mathcal{A}}) are evaluated on the exact same 20 birth-city prompts.
  2. Zero birth-city supervision during attack: Birth-city prompts and city names are completely excluded from the 60 attack-training examples, so the attack never trains the model on the birth-city question format.
  3. Negative control on retain90 (Q4 below): Applying the exact same attack to a control model that was never trained on the forget authors (retain90) yields no increase in birth-city matches (10/20 →\to 8/20), ruling out format-driven guessing or general-knowledge inference.
AuthorsOpenReview ↗

Response to Reviewer Mz7b

Q4. Attack performance relative to the TOFU retain90 control

We evaluated the baseline exposure and GRPO attack on the retain90 control checkpoint (trained from scratch without the 20 forget-set authors) using the exact same 60-question relearning set and 20 held-out birth-city probes:

Model / attackMean leakageLeak@128 /20Greedy /20
SimNPO (No attack)0.20795
+ SFT on SimNPO0.453119
+ GRPO on SimNPO0.6021513
retain90 control (No attack)0.386109
+ GRPO Attack on retain900.32185
Full model (Pre-unlearning)0.6202013

Because several TOFU fictitious authors have prominent world capitals as birth cities (e.g., Tokyo, Seoul, Tehran), retain90 exhibits a baseline chance guess rate of 0.386 (10/20 Leak@128) when forced to output a city name across 128 samples. Crucially, running our GRPO attack on retain90 does not increase leakage—in fact, it reduces mean leakage from 0.386 to 0.321 and Leak@128 from 10/20 to 8/20 because retain90 contains no latent associations between the fictitious authors' attributes and their birth cities.

In contrast, on SimNPO, GRPO boosts mean leakage from 0.207 to 0.602 (nearly matching the 0.620 pre-unlearning full model) and Leak@128 to 15/20, well above the retain90 control. This confirms that GRPO recovers residual target information retained in the unlearned model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: Revision & New Experiments

Dear AC and PCs,

We have uploaded a revised manuscript with changes highlighted in blue. We have also expanded our empirical evaluation to three additional unlearning methods (WGA, SatImp, and PDU), a new benchmark (MUSE), and controlled ablations (KL-constrained SFT vs. GRPO, multi-reference SFT, retain90 negative controls, sequential SFT + GRPO, and rollout group-size ablations) on TOFU Dataset.

Below we summarize the major changes in the revised PDF:

  • Section 4 (Experiments): Table 3 to include WGA, SatImp, and PDU alongside existing checkpoints.
  • Section 5 (Discussion and Limitations): Added future work on relearning attack against emerging relearning-resilient methods.
  • Section 6 (Related Work): Expanded coverage of recent 2025–2026 unlearning methods (including WGA, SatImp, PDU, and sharpness/representation defenses against relearning), multi-sample probabilistic auditing, and model-tampering attacks.
  • Appendix A (Proofs): Fixed cross-references (updating "Proposition 3" to "Theorem 3" in Appendix A.5) and audited all theorem/equation citations.
  • Appendix D (Additional Experimental Setup and Results): Fixed the Clopper–Pearson reference in Table 5 and added six new subsections:
    • Appendix D.4 (new): Hardware specifications, wall-clock runtime, and peak GPU memory comparisons (Table 9).
    • Appendix D.5 (new): KL-constrained SFT vs. GRPO comparisons under controlled distributional shift (Table 10).
    • Appendix D.6 (new): Multi-reference SFT comparisons and retain90 negative control experiments (Table 11).
    • Appendix D.7 (new): Keyword matching vs. exact string matching analysis and LLM-judge verification (Table 12).
    • Appendix D.8 (new): Full TOFU evaluation on WGA, SatImp, and PDU plus sequential SFT + GRPO warm-up comparisons (Table 13).
    • Appendix D.9 (new): GRPO rollout group-size ablations on TOFU (Table 14).
    • Appendix D.10 (new): Attack performance on the MUSE benchmark (Table 14).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.