acceptodds
Under review as a conference paper at ICLR 2027

Widen the Gap, Hide the Gap: Auditable and Indistinguishable Unlearning for LLMs

Abstract

Machine unlearning aims to remove a user's data from a trained model after a deletion request. Existing methods are majorly evaluated on three axes - how well they forget, how much capability they retain, and how robust they are to membership inference attacks, but not on whether the provider can offer the user any evidence that the deletion occurred. To fill this gap, we introduce auditability as a fourth axis, defined as the separation between requested (forget set) and retained (retain set) data under a statistic the provider computes. We show that across eleven unlearning methods on TOFU and MUSE no method satisfies all four axes, with different methods falling short on different axes. To address this, we propose a two stage method that can be used on top of any unlearning method, in the the first stage we propose GapWiden which uses a hinge loss to increase the difference between audit statistics of forget set and retain set. In the second stage we propose GapVeil, an inference time rejection-sampler which uses the base model as the drafter and re-samples based on the audit statistics of the drafted response. We observe that using our method, auditability and leakage reach oracle parity for every unlearning method, and forget performance converges to the oracle; retain performance improves for nine of eleven methods but remains short of parity. Our results show that evidence of deletion or auditability can be constructed by design, and that treating it as an explicit requirement changes the viability of existing unlearning methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Unlearning, Auditability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kTxhOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a two-stage LLM unlearning framework that constructs an audit signal through likelihood-gap optimization and suppresses sensitive outputs through inference-time sampling.

Critical strengths
  1. The distinction between provider-side audit information and adversary-visible behavior is interesting.

  2. The experimental comparison includes many established unlearning baselines and multiple model scales.

Critical weaknesses
  1. The sampling formulation is inconsistent with the implementation. GapVeil claims a rejection-sampling distribution conditioned on an energy threshold. However, its adaptive token suppression changes subsequent proposal distributions, invalidating the exact sampling interpretation. The algorithm can also return a rejected candidate when its budget is exhausted.

  2. Evaluation metrics and experimental configurations are problematic. The leakage metric assigns a perfect score to attacks with AUROC below 0.5, ignoring the possibility of reversing the attack decision. Moreover, Section 4.1 states that GapWiden is applied only to methods with audit AUROC below 0.95, whereas Appendix D.2 reports applying it to all eleven methods on MUSE. This protocol discrepancy requires clarification.

  3. The conclusions are overstated. The method substantially decreases retain performance on MUSE-News, contradicting the broad claim of simultaneous improvement across evaluation dimensions. The experiments also lack convincing adaptive attacks against the inference-time filtering mechanism.

Questions for authors
  1. What evidence shows that GapWiden's audit signal corresponds to actual data removal rather than deliberately engineered score separation?

  2. What is the true output distribution of the adaptive GapVeil sampler?

  3. How often does GapVeil exhaust its sampling budget and return a response that violates the acceptance threshold?

3 comments

We thank the reviewer for their comments, please find our response below.

How does Auditability validate Unlearning?

Thank you. Our goal is to make existing unlearning methods auditable, which we define as a statistical gap between the model’s behaviour on the forget and retain sets. We agree that such a gap alone does not rule out a model that still answers correctly when a question is reworded. We address this with two experiments:

  1. GapWiden + GapVeil on the base model θ0\theta_0 (Table-1, general comment). Without any prior unlearning method, our approach still achieves strong forget performance, so the effect does not rely on an existing unlearning signal.
  2. Paraphrased TOFU questions (Table-2, general comment). Our method does not produce the ground-truth answer to paraphrased questions either, so the effect extends beyond the exact training tokens to the underlying semantic facts.

What is the Output distribution of GapVeil?

Thank you for this question. Eq. 4 is the ideal distribution we wish to sample from. Sampling from it directly is impractical because of the cold-start problem described in Sec. 3.3. On forget queries, θ0\theta_0 places most of its mass on the memorized answer, so independent proposals are repeatedly rejected. GapVeil therefore draws each new proposal from θ0\theta_0 with the content tokens of earlier rejected drafts masked, so the proposal changes from one iteration to the next. In the revised draft (Appendix F, Prop. 1), we derive the exact output distribution of the sampler as implemented:

p^(y)=∑k=0KE[1[N≥k]qRk(y)] 1[y∈A]+Pexh ρ(y),\hat{p}(y) = \sum_{k=0}^{K} \mathbb{E}\big[\mathbb{1}[N \ge k] q_{\mathcal{R}_k}(y)\big]\, \mathbb{1}[y \in \mathcal{A}] + P_{\mathrm{exh}}\, \rho(y), where qRkq_{\mathcal{R}_k} is the masked proposal at iteration kk, NN is the index of the first accept, and ρ\rho is the fallback used when the budget is exhausted. Conditional on acceptance at iteration kk, the response is distributed as πRk∝cRk π 1[no masked token]\pi_{\mathcal{R}_k} \propto c_{\mathcal{R}_k}\,\pi\,\mathbb{1}[\text{no masked token}]. This is Eq. 4 restricted to responses that do not reuse content from rejected drafts, re-weighted by cRkc_{\mathcal{R}_k}. Each iteration is therefore an exact rejection sampler for its own target, and the output of GapVeil is a mixture of these targets and the fallback. The first iteration uses the unmasked proposal θ0\theta_0, so its target is exactly Eq. 4, and it is selected with probability Z=pθ0(A)Z = p_{\theta_0}(\mathcal{A}). As a result, the TV(p^,π)\mathrm{TV}(\hat{p},\pi) is at most 1−Z1 - Z, and the same bound holds for the distance between p^\hat{p} and θ0\theta_0 (Appendix F, Cor. F.1). On retain queries, where GapWiden keeps the gap near zero and ZZ is close to one, the deployed sampler is therefore close both to Eq. 4 and to the original model.

Exhaustion Rate

We report the exhaustion rate in Table-3 of general comment and add it to Appendix as well. We observe that an exhaustion rate of ~65%. The reason for this is that by design, θ0\theta_0 and θu\theta_u don't agree on forget set queries and hence, have a high energy (large gap GG) becuase of which after several iterations, the gap doesn't go down to 0. Hence, after exhaustion we return the response with lowest energy across all steps. Note that this is by design and is expected. On the other hand, the exhaustion rate for retain set is very low, again because by design they are pinned to lower energy.

One sided MIA eval

Thanks a lot for pointing this out, in our earlier results, we had presented a one-sided leakage score in tables. However, after incorporating your comment, we have changed all the values as per Eq. 21 of the revised draft, which takes into account both sides of attack. Furthermore, we present the raw AUROC values in tables in Appendix for clarity. Broadly, our observations still hold that the proposed method improves the leakage score across all benchmarks: on average, from 0.790.79 to 0.960.96 on TOFU 1B, 0.760.76 to 0.960.96 on 3B, 0.770.77 to 0.950.95 on 8B, and 0.810.81 to 0.870.87 on MUSE. Further, we also see that the raw AUROC scores remain in the range of (0.4,0.6) avoiding attack from either sides.

GapWiden is applied to all methods on MUSE

We apply GapWiden to all methods on MUSE because we found that none of the methods were auditable on MUSE. This is in line with our protocol where we apply GapWiden only if the underlying method is not auditable.

Overstated claims

Thanks for the constructive critique, we would like to point that we have mentioned these limitations in detail in Appendix G including black-box and adaptive attacks. However, the focus of this work is to fundamentally question how unlearning should be approached?, where we emphasize on auditable unlearning which can be verified and presented as a certificate to deletion requests. We show such auditability is achievable without compromising downstream privacy, to our knowledge this is the first work to look at unlearning from this perspective.

the responses have addressed some of my concerns, so i have adjusted my scores,

Dear Reviewer kTxh,

Thanks a lot for your time and comments. We appreciate you for raising the score. We hope we have answered your questions. We would be happy to answer any other questions you might have.

Authors

Reviewer_EsXbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an auditable and indistinguishable machine unlearning framework for large language models. The authors argue that existing unlearning approaches primarily focus on forgetting effectiveness, utility preservation, and membership inference robustness, while overlooking auditability as an additional evaluation dimension. To address this limitation, the paper introduces a two-stage framework consisting of GapWiden and GapVeil. GapWiden explicitly enlarges the log-probability gap between the original and unlearned models on the forget set while preserving similar behavior on the retain set. GapVeil subsequently uses this gap as an inference-time criterion to filter potentially sensitive responses generated by the original model.

Critical strengths
  • Meaningful research problem. The paper addresses an important challenge in machine unlearning: how model providers can demonstrate that a deletion request has been conducted.

  • Interesting framework design. The proposed two-stage approach combines post-unlearning optimization with inference-time response filtering.

  • Extensive experimental evaluation. The authors evaluate the framework on eleven existing unlearning methods across TOFU and MUSE, covering multiple model sizes and forgetting configurations.

Critical weaknesses
  • The proposed auditability metric does not sufficiently establish evidence of actual unlearning.

My primary concern is the fundamental assumption underlying the proposed definition of auditability. In Section 3.1, the authors define auditability as the statistical separability between forget and retain sets based on the log-probability gap between the original and unlearned models. However, it is unclear why such separation should constitute reliable evidence that the requested information has actually been removed. In Section 3.2, GapWiden explicitly optimizes the model to enlarge this gap on the forget set, and the resulting separation is subsequently used to evaluate auditability. Consequently, achieving a high auditability score primarily demonstrates that the model has been successfully optimized to satisfy the proposed statistical criterion, rather than independently verifying successful unlearning. A model could exhibit a large likelihood gap on the specified forget examples while still retaining the corresponding knowledge, which might be recoverable through alternative prompts or related queries. Furthermore, the proposed auditability mechanism is computed entirely by the model provider using privileged access to both models. It does not provide a mechanism through which users can independently verify the claimed deletion. Therefore, the paper does not sufficiently establish the connection between the constructed audit signal and its broader claim of providing verifiable evidence of data removal. The authors should more clearly distinguish statistical evidence of behavioral modification from evidence of successful unlearning, and provide independent validation of the proposed auditability metric.

  • The inference-time filtering mechanism raises concerns about whether the framework achieves genuine unlearning.

In Section 3.3, GapVeil uses the original model, which still contains the information requested for deletion, to generate candidate responses, while the unlearned model primarily serves as a detector for filtering these responses. This design appears closer to inference-time output suppression than to removing the influence of the deleted data from the deployed system. Although such filtering may reduce observable information leakage, the paper does not sufficiently explain how this approach satisfies the broader objective of machine unlearning when the original model remains necessary for inference. Additionally, there is a discrepancy between the theoretical formulation and the actual implementation. Equation (4) defines a restricted output distribution that assigns zero probability to responses exceeding the acceptance threshold. However, Algorithm 1 returns the best available candidate when the sampling budget is exhausted, even if that candidate does not satisfy the threshold. The adaptive token suppression strategy also changes the proposal distribution from the independent rejection sampling described in the theoretical formulation.

  • The experimental evaluation does not sufficiently support the claimed improvements in privacy and unlearning effectiveness.

Although the experiments cover many baseline methods, several aspects of the evaluation require further investigation. First, the leakage evaluation in Appendix C.3 primarily considers behavioral abnormality signals, such as refusals and degenerate responses, rather than stronger adaptive attacks designed to recover forgotten information. Moreover, Equation (21) uses a one-sided transformation of the attack AUROC, which may report high leakage robustness even when an attacker can obtain nontrivial discrimination by reversing the decision direction. This appears inconsistent with the symmetric attacker advantage discussed in Section 4.5 and requires clarification. Second, the improvements are not consistent across benchmarks. In particular, Table 3 shows that average retain performance on MUSE-News decreases from 0.569 to 0.380 after applying the proposed framework, indicating a substantial utility trade-off. Finally, the proposed forget metric primarily measures whether reference-answer entities appear in generated responses, which may not adequately capture whether the underlying knowledge remains recoverable. More comprehensive evaluations using alternative queries, stronger extraction attacks, and independent unlearning metrics would be necessary to substantiate the claimed effectiveness of the framework.

Questions for authors
  1. What evidence supports the assumption that the artificially enlarged log-probability gap constitutes reliable proof of successful unlearning rather than merely reflecting optimized behavioral differences?

  2. How does the proposed framework guarantee that forgotten information is not exposed when GapVeil continues to use the original model for generation and may return a response that violates the acceptance threshold?

1 comment

We thank the reviewer for their comments, please find our response below.

How does Auditability validate Unlearning?

Thank you. Our goal is to make existing unlearning methods auditable, which we define as a statistical gap between the model’s behaviour on the forget and retain sets. We agree that such a gap alone does not rule out a model that still answers correctly when a question is reworded. We address this with two experiments:

  1. GapWiden + GapVeil on the base model θ0\theta_0 (Table-1, general comment). Without any prior unlearning method, our approach still achieves strong forget performance, so the effect does not rely on an existing unlearning signal.
  2. Paraphrased TOFU questions (Table-2, general comment). Our method does not produce the ground-truth answer to paraphrased questions either, so the effect extends beyond the exact training tokens to the underlying semantic facts.

User cannot verify the claimed Deletion

Thanks for bringing this point. Indeed a user cannot verify such a claim, that is why such auditability has to be executed via independent third parties or government agencies. Recently, there have been such examples where third parties like METR have investigated and audited closed proprietary models [1].

The original model is still present

Thanks for this comment. Indeed, our method only modifies the output distribution. Such class of methods are often refered to as 'Weak Unlearning' [2], where the deletion presents itself in the final output distribution rather than finetuned weights itself.

Output after budget exhaustion

As mentioned in Alg. 1, we output the response with the least energy over the iteration if we exhaust the budget.

Proposal change every iteration

We agree, and we thank the reviewer for pointing this. Since the proposal is drawn from θ0\theta_0 with tokens from previous iterations suppressed, the new proposal is different for each step. In the new Appendix F, Prop. 1, we characterize the output of the sampler as implemented. Conditional on acceptance at stage kk, the returned response is distributed as πRk∝cRk π 1[no masked token]\pi_{\mathcal{R}_k} \propto c_{\mathcal{R}_k}\,\pi\,\mathbb{1}[\text{no masked token}]. This is Eq. 4 restricted to responses that do not reuse content from the rejected drafts, re-weighted by the normalizing factor cRkc_{\mathcal{R}_k}. Hence, each iteration is an exact rejection sampler with its own target, and the final output distribution of GapVeil is the mixture of these individual targets. We also formalize the closeness between this distribution and ideal distribution in Corr. F.1.

Leakage Scores

Thanks a lot for pointing this out, in our earlier results, we had presented a one-sided leakage score in tables. However, after incorporating your comment, we have changed all the values as per Eq. 21 of the revised draft, which takes into account both sides of attack. Furthermore, we present the raw AUROC values in tables in Appendix for clarity. Broadly, our observations still hold that the proposed method improves the leakage score across all benchmarks: on average, from 0.790.79 to 0.960.96 on TOFU 1B, 0.760.76 to 0.960.96 on 3B, 0.770.77 to 0.950.95 on 8B, and 0.810.81 to 0.870.87 on MUSE. Further, we also see that the raw AUROC scores remain in the range of (0.4,0.6) avoiding attack from either sides.

Robustness experiments

Thanks for this suggestion. To robustly evaluate our method on forget set, we evaluate our method on paraphrased questions of TOFU as mentioned above. This result simultaneously shows that our method is robust to paraphrasing of related questions.

Exhaustion Rate

We report the exhaustion rate in Table-3 of general comment and add it to Appendix as well. We observe that an exhaustion rate of ~65%. The reason for this is that by design, θ0\theta_0 and θu\theta_u don't agree on forget set queries and hence, have a high energy (large gap GG) because of which after several iterations, the gap doesn't go down to 0. Hence, after exhaustion we return the response with lowest energy across all steps. Note that this is by design and is expected. On the other hand, the exhaustion rate for retain set is very low, again because by design they are pinned to lower energy.

How forgotten information is not exposed

Thanks for the comment. While θ0\theta_0 is used to draw the proposals, because of the in-built auditability (through GapWiden), the forget set energy should be greater than zero. Leading to masking/suppresion of the proposal tokens. Now, since θ0\theta_0 is prone to generating ground-truth (GT), it is the GT tokens that get suppressed. Hence, in subsequent iterations, as more of these tokens are suppresed, it is unlikely that the GT response is generated at the end. We provide a detailed theoretical analysis of this in Appendix F of revised draft.

[1] Independent Assessment of AI R&D Acceleration at Anthropic in October 2026, METR.
[2] Xu, Heng et al. “Machine Unlearning: A Survey.” ACM Computing Surveys 56 (2023)

Reviewer_e6JyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new evaluation metric, auditability, for LLM unlearning. It complements existing measures of forgetting, retained utility, and membership leakage. Auditability measures how well forget and retain examples can be separated using scores computed from the original and unlearned models. The authors then propose a framework with two stages, GapWiden and GapVeil, to improve existing unlearning methods. GapWiden increases the log-probability gap for forget examples while keeping it near zero for retain examples. GapVeil aims to reduce membership leakage during inference. It uses the audit score to screen responses generated by the original model. Rejected responses are regenerated with their content tokens suppressed. The framework is evaluated with eleven unlearning methods on TOFU and MUSE. The authors report average improvements across all four metrics on TOFU. On MUSE, average retained utility drops, especially on the News split.

Critical strengths
  1. The evaluation covers a range of settings and shows improvements in forgetting and retention on TOFU. The authors test the framework with eleven unlearning methods on TOFU and MUSE. The TOFU experiments include three model sizes and three forget-set sizes. With the 8B model on TOFU-forget05, the average forget score across the eleven methods increases from 0.749 to 0.934. The average retain score also improves, from 0.742 to 0.966. The ablation experiments examine the contribution of each stage and how the margin, threshold, and sampling budget affect the results.
  2. The audit score connects evaluation with the design of the method. The main statistic is the average token log-probability gap between the original and unlearned models. The paper also considers KL divergence, JS divergence, and an entropy gap. Comparing these scores on forget and retain examples gives a concrete way to measure how model behavior changes across the two sets. GapWiden directly optimizes the log-probability gap during training, while GapVeil uses the selected audit statistic to screen generated responses. This provides a practical link between measuring model changes and controlling the deployed output.
Critical weaknesses
  1. The paper does not show that its audit can distinguish effective forgetting from answer suppression. The audit AUROC in Section 3.1 and Appendix C.2 measures how well the gap separates forget examples from retain examples. These labels specify which examples were selected for deletion; they do not specify whether deletion succeeded. GapWiden then directly trains the model to enlarge this gap on forget examples while keeping it small on retain examples. A high audit score therefore establishes that the model was changed differently on the two sets. However, a model could lower the probability of the designated answers and obtain a large gap while still revealing the same facts when the questions are rephrased. The paper reports separate forget scores, but these do not validate the audit’s ability to distinguish this form of answer suppression from effective unlearning. This missing comparison weakens the claim that the constructed signal provides verifiable evidence of deletion.
  2. The paper uses inconsistent definitions of leakage robustness. Appendix C.3, Eq. (21), defines the leakage score as MIA = min{2(1 - AUROC),1}. This gives the highest robustness score to every AUROC below 0.5. For example, an AUROC of 0.1 receives a score of 1, although reversing the attack score gives an AUROC of 0.9. Section 4.5 explicitly recognizes this possibility and reports the absolute distance of AUROC from 0.5, which handles both attack directions. This differs from Equation (21). It is unclear whether the issue is limited to the written definition or also affects the tables. Without the underlying AUROCs and consistent treatment of both directions, the near-oracle leakage scores in the tables are difficult to assess. Figure 4 also does not show a reduction in leakage for every method. The reported attack advantage stays at 0.11 for CEU and rises from 0.10 to 0.14 for GradAscent. The uncertainty of these estimates is not reported. A comparison with the oracle under the same evaluation, including confidence intervals, is needed to assess the claim of oracle parity.
Questions for authors
  1. What evidence shows that a high audit AUROC reflects deletion of the target information? Could a model pass the audit by suppressing the tested answers while still revealing the same facts under reworded questions? Do any control experiments or tests on new questions address this concern? Also, were the audit statistic and threshold chosen using data separate from the final evaluation set?
  2. How does the output distribution of Algorithm 1 relate to Equation (4)? Token suppression changes the proposal distribution after each rejection, and the algorithm can return a response that still fails the energy threshold when the budget runs out. Please clarify whether Equation (4) is only an ideal target. Also, report the fraction of all returned responses that fail the threshold, separately for forget and retain queries.
  3. Were the leakage scores in the tables computed using Equation (21)? How were attacks with AUROC below 0.5 handled, given that Section 4.5 allows the attacker to reverse the score direction? Please also clarify whether Figure 4 uses the default TOFU-forget10/3B setting. If so, the AUROC of 0.61 for CEU + Ours gives a score of about 0.78 under Equation (21). Table 8 reports 1.000 for the same method and setting. Please provide raw AUROCs with confidence intervals for the methods and the oracle, and clarify whether oracle parity still holds when both attack directions are considered.
  4. What gains does GapVeil provide over the inference-time methods discussed in Appendix B.2, such as CP-k and conformal unlearning? Can you provide comparisons in the same setting with a comparable inference budget? A comparison with independent resampling, using the same reference model, threshold, and budget, would also help show the benefit of adaptive token suppression. Please report the average number of proposals and end-to-end latency, including the cost of reference-model scoring.
4 comments

We thank the reviewer for their comments, please find our response below.

How does Auditability validate Unlearning?

Thank you. Our goal is to make existing unlearning methods auditable, which we define as a statistical gap between the model’s behaviour on the forget and retain sets. We agree that such a gap alone does not rule out a model that still answers correctly when a question is reworded. We address this with two experiments:

  1. GapWiden + GapVeil on the base model θ0\theta_0 (Table-1, general comment). Without any prior unlearning method, our approach still achieves strong forget performance, so the effect does not rely on an existing unlearning signal.
  2. Paraphrased TOFU questions (Table-2, general comment). Our method does not produce the ground-truth answer to paraphrased questions either, so the effect extends beyond the exact training tokens to the underlying semantic facts.

How are the thresholds chosen?

We follow the protocols presented in OpenUnlearning [1] where the evaluation is done on a perturbed version of retain and forget set. For thresholds and margins, we don't use the perturbed version which is used for evaluation. Instead we use the original QA pairs for GapWiden training as well as for GapVeil threshold parameters.

What is the Output distribution of GapVeil?

Thanks for this question. The distribution in Eq. 4 is the ideal distribution that we want to sample from, however, due to the cold-start problem mentioned in Sec. 3.3 it is practically difficult to directly sample from. To alleviate this, we use the suppression strategy as described in draft. We derive the output distribution of this strategy in Appendix F of revised draft, specifically, it is given by- p^(y)=∑k=0KE[1[N≥k]qRk(y)] 1[y∈A]+Pexh ρ(y)\hat{p}(y) = \sum_{k=0}^{K} \mathbb{E}\big[1[N \ge k] q_{R_k}(y)\big]\, 1[y \in A] + P_{exh}\, \rho(y). Pls see Appendix F, Prop. 1 for details. Essentially, the output distribution is a mixture of target distributions of each loop of GapVeil. Further, we show that the total variation distance of p^\hat{p} and π\pi (in Eq. 4) is bounded above by 1−Z1-Z (see Corr. F.1.1).

Exhaustion Rate

We show exhaustion rate in Table-3 of general comment and add it to Appendix. We observe that an exhaustion rate of ~65%. The reason for this is that, by design θ0\theta_0 and θu\theta_u don't agree on forget set queries and hence, have a high energy (large gap G()G()) becuase of which after several steps, the gap doesn't go down to 0. Hence, after exhaustion we return the response with lowest energy across all steps. Note that this is by design and is expected. On the other hand, the exhaustion rate for retain set is very low, again because they are pinned to lower energy.

Leakage Scores

Thanks a lot for pointing this out, in our earlier results, we had presented a one-sided leakage score in tables. However, after incorporating your comment, we have changed all the values as per Eq. 21 of the revised draft, which takes into account both sides of attack. Furthermore, we present the raw AUROC values in tables in Appendix for clarity. Broadly, our observations still hold that the proposed method improves the leakage score across all benchmarks: on average, from 0.790.79 to 0.960.96 on TOFU 1B, 0.760.76 to 0.960.96 on 3B, 0.770.77 to 0.950.95 on 8B, and 0.810.81 to 0.870.87 on MUSE. Further, we also see that the raw AUROC scores remain in the range of (0.4,0.6) avoiding attack from either sides.

Comparison with Independent Sampling

We present the results (averaged across all methods) with different sampling strategies below (full table can be found in Appendix), we can see that independent re-sampling leads to degraded forget performance because of the cold-start issue. Whereas using θu\theta_u as fallback leads to low leakage score.

MethodFrgRetLeak
GapVeil (ours)0.8820.7130.974
Resampling0.5590.7380.946
GapVeil+θu\theta_u as fallback0.9300.7180.646

Comparison with other samplers

We compare our method with CP-kk [2] and UCD [3] below. We observe that compared to our method the forget performance of CP-kk and UCD is consistently lower. While UCD performs better on retain set by 6 points, it degrades by 32 points on forget set.

MethodFrgRetLeak
GapVeil (ours)0.9040.7230.970
CP-k (k=5)0.6940.6650.982
UCD (β=0.5)0.5810.7840.936

Additional results

We report the average number of proposals and the end-to-end latency per query, averaged across all methods in Table 14 of Appendix. Retain and unseen queries stay close to the cost of θ0\theta_0 alone, and only forget queries use the full budget. Scoring accounts for 5–6% of the latency.

[1] Dorna et al. "Openunlearning: Accelerating llm unlearning via unified benchmarking of methods and metrics." NeurIPS 2026.
[2] Chowdhury et al. "Inference-time Unlearning Using Conformal Prediction." 2026.
[3] Suriyakumar et al. "UCD: Unlearning in LLMs via Contrastive Decoding." 2025.

Thank you for the response and the additional experiments. The results on paraphrased questions ease my concern that the method may only suppress answers to the original questions.

I still have reservations about how the audit score is interpreted. The audit AUROC measures how well the forget and retain examples can be separated, while GapWiden directly increases the difference between the two sets. I am therefore still not convinced how much a high audit score tells us about whether the target information has been forgotten.

For the leakage results, the authors state that they have recalculated the scores and added the raw AUROCs. However, the response does not directly explain the relation between Figure 4 and Table 8 in the original paper. If they use the same setting, attack, and generated responses, the AUROC of 0.61 for CEU + Ours in Figure 4 gives a score of about 0.78 under the original formula, while Table 8 reports 1.000. From the response alone, I am still unsure how well the corrected results support the claim that leakage is comparable to the retrained model.

Overall, I will keep my rating of 2 (weak reject) for now.

Thank you for following up, and we are glad the paraphrase results addressed your concern.

On the interpretation of the audit score. We agree with the reviewer: a high audit AUROC does not by itself show that the information has been forgotten, and after GapWiden the separation on the training pairs holds by construction. We do not intend the audit score as a measure of forgetting, and we mention it in draft. Forgetting is measured by the forget axis, which does not use the audit statistic, including on paraphrased questions (Table 13).

The audit score answers a different question: can the provider show, per record, that the model was changed on the requested data and not elsewhere? Particularly, two of our results indicate that it is more than a fitted label. (i) It transfers/generalizes to inputs GapWiden never optimized: on paraphrased questions the audit AUROC is 0.955 on average (0.988 on the original; Table 13). (ii) It is tied to deployed behaviour: GapVeil’s acceptance test uses the same statistic, so a certified record can be returned only when the budget is exhausted (Prop. F.2). This happens on 67.9% of forget queries (Table 10), where the lowest-energy draft is returned, and the forget scores are measured on those outputs.

We therefore present auditability as an accountability axis that complements the forget axis and does not substitute for it.

On Figure 4 and Table 8. Thanks for this, your calculation is correct, and the two do not use the same generated responses. Fig. 4 was produced for the original submission, and was not regenerated in the revision for rebuttal. The leakage results in the tables come from new runs carried out during the rebuttal, with the two-sided score of Eq. 21 and the raw AUROC reported beside each entry. We apologise for the confusion; the figure should have been updated together with the tables. We will regenerate Fig. 4 and Fig. 9 from the same runs as the tables, so that the figures and tables report the same quantities throughout.

We again thank you for your time and consideration, we value your feedback and would be happy to answer any further questions you might have.

AuthorsOpenReview ↗

Updated Fig. 4 and Table 8

Dear Reviewer e6Jy,

We thank you again for your careful comments and suggestions. Following up on your comments, we have updated our draft with one consistent run for Fig. 4 and Table 8. Both are now obtained from same run as suggested. We would be happy to provide any further clarification or answer any question you might have.

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We would like to thank all the reviewers for their time and sincere review of our work. These comments have helped us enormously in revising and improving our manuscript. We have uploaded a revised version of our manuscript with changes denoted in blue color. The major changes are as follows:

  1. We revise the leakage score by changing from the earlier one-sided score to the new double-sided score which takes into account the attack from either side (presented in Eq. 21). Additionally, we also provide the raw AUROC scores for leakage in all tables in Appendix for transparency and clarity.
  2. We add additional results as requested by the reviewers regarding the budget exhaustion, sampling strategies, comparison with other samplers and robustness evaluation of proposed method. These results are presented in Appendix E.
  3. We add a detailed theoretical analysis of the proposed method, elucidating the exact proposal distribution and output distribution of our sampler. Further, we also quantify the closeness of the sampler's output with the ideal output distribution through Total Variation distance. These analysis are presented in Appendix F.

Further, we present some results here in the general comment for common reference for all the reviewers. Unless explicitly mentioned the results are presented on TOFU-forget10 with Llama3-3B.

Table-1: Our method applied on the base model

SystemForgetRetainLeakageAuditability
Base model θ0\theta_00.1030.8910.8980.500
θ0\theta_0 + GapWiden0.9570.4490.5820.997
θ0\theta_0 + GapWiden + GapVeil0.9140.7400.9680.997
Retrained oracle0.8610.8950.9210.998

Table-2: Performance of proposed method on original/paraphrased queries

SystemForgetRetainLeakageAuditability
Retrained oracle0.874 / 0.8920.862 / 0.3350.964 / 0.959–
CEU + Ours0.773 / 0.8580.703 / 0.2840.994 / 0.9980.999 / 0.996
DPO + Ours0.896 / 0.9060.654 / 0.2190.981 / 0.9590.997 / 0.932
GradAscent + Ours0.921 / 0.8950.701 / 0.3000.958 / 0.9571.000 / 1.000
GradDiff + Ours0.786 / 0.8640.576 / 0.2150.957 / 0.9370.989 / 0.981
NPO + Ours0.923 / 0.9190.661 / 0.2760.988 / 0.9510.997 / 0.990
PDU + Ours0.774 / 0.8210.704 / 0.2970.971 / 0.9571.000 / 1.000
RMU + Ours0.860 / 0.8810.683 / 0.2380.988 / 0.9800.983 / 0.988
SatImp + Ours0.731 / 0.8220.612 / 0.2090.987 / 0.9490.967 / 0.883
SimNPO + Ours0.728 / 0.8070.656 / 0.2420.999 / 0.9790.993 / 0.932
UNDIAL + Ours0.892 / 0.8770.449 / 0.1710.995 / 0.9540.962 / 0.906
WGA + Ours0.866 / 0.8720.563 / 0.1490.968 / 0.9790.986 / 0.901
Average (Ours)0.832 / 0.8660.633 / 0.2360.981 / 0.9640.988 / 0.955

Table-3: Exhaustion rate of GapVeil (%)

MethodForget (K=0K=0)Forget (K=3K=3)Forget (K=5K=5)Retain (K=0K=0)Retain (K=3K=3)Retain (K=5K=5)
CEU98.293.293.00.00.00.0
DPO94.246.833.827.310.26.8
GradAscent98.095.094.50.00.00.0
GradDiff99.299.299.20.80.80.8
NPO81.531.222.01.00.00.0
PDU98.296.296.04.52.52.2
RMU98.095.895.80.00.00.0
SatImp95.575.070.20.00.00.0
SimNPO80.819.810.531.55.01.5
UNDIAL73.20.00.08.50.00.0
WGA98.295.094.80.00.00.0
Average92.367.964.56.71.71.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.