acceptodds
Under review as a conference paper at ICLR 2027

AAPO: Adaptive Adversarial Prompt Orchestration for Robust Unlearning in Large Language Models

Abstract

Machine unlearning aims to selectively remove specific knowledge from large language models (LLMs) to meet privacy, safety, and compliance requirements. However, existing unlearning methods are primarily evaluated under non-adversarial settings and often fail to maintain the unlearning state when faced with adaptive adversarial prompt attacks at inference time. This fragility of the unlearning state severely undermines the practical value of machine unlearning and may pose latent risks for real-world deployment. Existing studies have attempted to introduce defensive mechanisms to improve model robustness. Nevertheless, the majority of current approaches are dependent on parameter updates, which not only introduce computational overhead but also significantly limit the generalization capability to diverse unknown attacks in the future. To address these challenges, we propose Adaptive Adversarial Prompt Orchestration(AAPO), a model-decoupled, inference-stage defense framework. AAPO formulates robust unlearning as a closed-loop adversarial procedure involving Attacker, Judge, and Defender agents. It enables the dynamic evolution of defense prompts via multi-agent self-play to adaptively address a wide range of continuously evolving potential attacks. Extensive experiments on the RWKU benchmark across multiple mainstream LLMs demonstrate that AAPO significantly improves unlearning robustness under adversarial attacks, achieves up to a 70% relative reduction in forget-set leakage across multiple baseline methods, while preserving overall model utility. Additional experiments on the WMDP benchmark further validate the generalizability of AAPO across different unlearning settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Large Language Models, Adversarial Robustness, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jrfpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AAPO, an inference-time defense that uses attacker–judge–defender interactions to generate and select defensive prompts without modifying model parameters. The method combines reactive guard generation with an Exp3-style selection policy and evaluates forgetting and utility on RWKU and WMDP.

Critical strengths
  1. The framework is practical and compatible with existing unlearning methods. AAPO operates through system prompts without updating model parameters. Table 1 shows consistent reductions in forget-set scores across two model families and four unlearning methods.

  2. The comparison with simpler prompt defenses is useful. Table 6 includes length-matched prompts, prompt concatenation, iterative refinement, and filtering. AAPO improves both forgetting and neighbor retention over iterative refinement, supporting the value of adaptive prompt orchestration.

Critical weaknesses
  1. The prompt-selection update is insufficiently justified. Equation (14) divides the shared reward by (p_i), but the WDGS rule in Equation (12) selects multiple prompts without establishing that (p_i) represents their actual inclusion probabilities. This leaves the connection between prompt selection and the Exp3-style reward update unclear.

  2. The related-work discussion overlooks adversarial prompt optimization. Robust Prompt Optimization (RPO; Zhou et al., 2024) already optimizes defensive suffixes against adaptive attacks without updating model weights, but is not discussed in this paper. A direct comparison of objectives, model-access assumptions, and adaptation mechanisms is needed to clarify AAPO’s additional contribution.

  3. The analysis understates the loss of model utility. In Table 1, adding AAPO reduces LLaMA-3 factuality from 54.6 to 38.5 for DPO and the neighbor score from 66.7 to 53.7 for RT. These substantial drops weaken the utility-preservation claim and require a clearer analysis of the forgetting–utility trade-off.

Questions for authors
  1. Clarify the prompt-selection update. How are prompt subsets sampled under WDGS, and why is (p_i) the appropriate denominator in Equation (14)? How does the update account for interactions among prompts that receive the same joint reward?

  2. Clarify the distinction from RPO. How does AAPO differ from RPO in its optimization objective, model-access requirements, and adaptation mechanism? What evidence establishes the benefit of these differences for robust unlearning?

  3. Explain the utility losses. What causes the substantial factuality and neighbor-score drops in Table 1? Does AAPO still preserve utility better than simpler prompt defenses when compared at matched forgetting levels?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jrfp

Thank you for your time and constructive feedback! We are glad that you recognized the practicality of AAPO and its compatibility with existing unlearning methods. This is a great encouragement to us.

AAPO models robust unlearning as a closed-loop adversarial process among the Attacker, the Judge, and the Defender. Unlike methods that rely on parameter updates, AAPO works at the system-prompt level and requires neither access to model weights nor additional training. Therefore, it can serve as a plug-and-play module on top of different unlearning methods.

We respond to your comments on the prompt-selection update, the difference from RPO, and the utility loss point by point below, and we will revise the paper accordingly.

Comment 1: The prompt-selection update is insufficiently justified. Equation (14) divides the shared reward by (p_i), but the WDGS rule in Equation (12) selects multiple prompts without establishing that (p_i) represents their actual inclusion probabilities. This leaves the connection between prompt selection and the Exp3-style reward update unclear. Clarify the prompt-selection update. How are prompt subsets sampled under WDGS, and why is (p_i) the appropriate denominator in Equation (14)? How does the update account for interactions among prompts that receive the same joint reward?

Response 1: Thank you for this insightful question. We answer it one by one below.

• Sampling in WDGS. In each round, the Defender first determines the combination length ktk_t according to Eq. 11. It then samples the first prompt according to the probability p(t)p^{(t)}, and selects the remaining kt−1k_t-1 prompts one by one according to Eq. 12. In the revised version, we will provide the full selection procedure as pseudocode.

• Denominator in Eq. 14. When kt=1k_t=1, the inclusion probability of prompt ii is pip_i, and Eq. 14 is consistent with the standard Exp3 importance-weighted estimator. When kt>1k_t>1, prompt ii can be sampled as the first prompt (with probability pip_i), or selected later by Eq. 12. Therefore, its actual inclusion probability satisfies qi≥piq_i \ge p_i. In this case, Eq. 14 should be viewed as an Exp3-inspired heuristic update. The ablation study in Table 2 compares AAPO with two simpler settings. In w/o Exp3, the prompt weights are not updated, and all prompts keep the same weight. In Fixed-k1, only one prompt is selected in each round.

SettingForget Set↓Neighbor Set↑
Full (AAPO)43.576.8
w/o Exp360.978.5
Fixed-k159.379.0

Compared with w/o Exp3, the weight update in Eq. 14 reduces the forget-set score by 17.4 points. Compared with the single-prompt setting (Fixed-k1), which matches the standard Exp3 estimator, extending to multi-prompt combinations further reduces the forget-set score by 15.8 points. Therefore, although Eq. 14 is a heuristic update when kt>1k_t>1, it is effective in practice.

• Credit assignment among multiple prompts. AAPO does not explicitly model the interactions among prompts. Instead, it handles them implicitly through two mechanisms. First, since the first prompt is randomly sampled according to p(t)p^{(t)}, the same prompt appears together with different prompts in different rounds. After many rounds of updates, its weight reflects the average performance of the various combinations it takes part in, which can be seen as an approximation of its marginal effect. Second, the diversity term in WDGS lowers the chance that semantically similar prompts are selected together, which reduces confusion among redundant prompts. Appendix Table 4 shows that the forget-set score is the lowest when β is 0.5. Leaning too much toward historical weights (β=0.8) or toward diversity (β=0.2) increases the forget-set score to 47.3 and 52.4, respectively. This shows that balancing effectiveness and diversity is necessary for handling the interactions among prompts.

We will add the above analysis to the revised version. Thank you very much!

AuthorsOpenReview ↗

Response to Reviewer jrfp [part 2]

Comment 2: The related-work discussion overlooks adversarial prompt optimization. Robust Prompt Optimization (RPO; Zhou et al., 2024) already optimizes defensive suffixes against adaptive attacks without updating model weights, but is not discussed in this paper. A direct comparison of objectives, model-access assumptions, and adaptation mechanisms is needed to clarify AAPO’s additional contribution. Clarify the distinction from RPO. How does AAPO differ from RPO in its optimization objective, model-access requirements, and adaptation mechanism? What evidence establishes the benefit of these differences for robust unlearning?

Response 2: Thank you very much for your valuable comment. RPO [1] is an excellent work. Below we compare the two methods from the three aspects you mentioned.

RPOAAPO
Optimization objectiveTargets jailbreak defense, making the model refuse harmful instructions under worst-case attacksTargets robust unlearning, preventing leakage of unlearned information while keeping normal answers on retained knowledge
Model accessRequires gradients from a white-box model to optimize the suffixRequires no gradients or weights
Adaptation mechanismIncludes worst-case attacks during optimization, and the suffix is fixed after deploymentKeeps generating new guards from leakage cases, and dynamically selects and combines them

These differences are also reflected in the experiments. For the optimization objective, unlearning requires not only blocking leakage but also avoiding harm to related knowledge. In Appendix Table 6, refusal-based prompt defenses reduce the forget-set score, but their neighbor scores also drop to 40.1–49.0. In contrast, AAPO achieves 76.8, almost the same as the original model.

For the adaptation mechanism, we further tested two adaptive attacks against the final defense. The schema-aware attack can access AAPO's full defense prompt pool, and the PAIR attack iteratively searches for attack prompts based on model responses. On the original model + AAPO, we used these attacks as feedback for 10 rounds of adaptation. The changes in ASR are shown below:

Defense stateSchema-awarePAIR
ASR0\mathrm{ASR}_00.580.67
ASR1\mathrm{ASR}_10.410.29
Δ\Delta ASR-0.17-0.38

Without retraining the target model, AAPO reduces the ASR of the two attacks by 17 and 38 percentage points, respectively. In contrast, the defensive suffix of RPO stays fixed after deployment, and it needs to be re-optimized to handle new attacks. We will add a discussion of and comparison with RPO to the related work in the revised version. Thanks again!

[1] Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

AuthorsOpenReview ↗

Response to Reviewer jrfp [part 3]

Comment 3: The analysis understates the loss of model utility. In Table 1, adding AAPO reduces LLaMA-3 factuality from 54.6 to 38.5 for DPO and the neighbor score from 66.7 to 53.7 for RT. These substantial drops weaken the utility-preservation claim and require a clearer analysis of the forgetting–utility trade-off. Explain the utility losses. What causes the substantial factuality and neighbor-score drops in Table 1? Does AAPO still preserve utility better than simpler prompt defenses when compared at matched forgetting levels?

Response 3: Thank you for your comment. We address your questions one by one below.

• Cause of the factuality drop. This drop mainly comes from AAPO making the target model answer more conservatively. The defense prompts of AAPO mainly use refusal as the form of defense. Since DPO has already changed the model's behavior, this refusal tendency can spread to some factual questions. This phenomenon mainly occurs on LLaMA-3-8B-Instruct. On Qwen2.5-14B-Instruct, the factuality of DPO(+AAPO) instead increases from 32.5 to 35.8.

• Neighbor drop of RT(+AAPO). This drop is closely related to the nature of RT itself. RT uses predefined refusal responses as its optimization target, so it already tends to refuse conservatively. In Table 1, RT alone reduces the neighbor score from 77.4 to 66.7, which is the largest drop among the four unlearning methods. On top of this, AAPO forms multi-layer protection through the dynamic selection, combination, and iterative update of defense prompts, which makes RT(+AAPO) more conservative than the other combinations.

• Comparison with prompt defenses at matched forgetting levels. We compare the AAPO results under different λ in Appendix Table 3 with the prompt-defense baselines in Table 6. Compared with Iterative (47.0 / 67.0), AAPO (λ=0.5) is better on both the forget-set score and the neighbor score (43.5 / 76.8). When λ=0.8, AAPO achieves a forget-set score of 40.9, which is close to that of Guardrail, Multi-Guardrail, and Length-Matched (33.0–34.7), while its neighbor score of 70.1 is 21.1–30.0 points higher than these baselines. Therefore, at similar forgetting levels, AAPO better preserves related knowledge.

We will add the above analysis to the revised version and correct the statements about utility preservation in the main text. Thank you again!

In summary, we are truly grateful for your insightful comments, and we sincerely hope you will consider the overall value of our paper. Thank you sincerely!

Reviewer_gAVcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AAPO learns defensive prompt compositions through attacker–defender self-play while keeping the target model frozen. It combines adaptive selection, diversity-based composition, and reward updates, and evaluates forgetting and utility alongside existing unlearning methods.

Critical strengths
  1. The prompt-based defense works with a frozen target model and can complement existing unlearning methods.
  2. The evaluation separates RWKU test probes from self-play and includes relevant prompt-based controls.
  3. The appendices provide agent prompts and sensitivity experiments that help explain the implementation.
Critical weaknesses
  1. Eq. 14 uses normalized prompt weight as an importance-correction probability, although WDGS selects a set of prompts whose inclusion probabilities generally differ from those weights. The stated Exp3 rationale therefore does not justify the update. The authors should specify the selection law or evaluate the update explicitly as a heuristic.
  2. Table 1 shows substantial utility losses that conflict with the utility-preservation claim. LLaMA factuality drops from 51.4 to 38.8 with GA and from 54.6 to 38.5 with DPO. The evaluation needs to show which operating points preserve utility while retaining the forgetting gains.
  3. The feedback pipeline is insufficiently specified. The judge checkpoint, reference-answer construction, and the point at which the learned guards are frozen remain unclear. Because judge feedback drives optimization, these details and an independent semantic-leakage evaluation are needed to assess the reported gains.
Questions for authors
  1. How does the probability in Eq. 14 correspond to WDGS inclusion probabilities? Does the update outperform simpler rules at equal budgets?
  2. Which operating points preserve utility while retaining the forgetting gains in Table 1?
  3. Which judge and reference answers drive self-play, and when are guards frozen before evaluation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gAVc

Thank you for your constructive and valuable feedback! We are glad that you recognized the applicability of AAPO to frozen models and its complementarity with existing unlearning methods. This is a great encouragement to us.

AAPO is an inference-time defense framework for LLM unlearning. Through the Attacker–Judge–Defender loop, it keeps exposing failure cases of the current defense, and then generates new defense prompts and updates the selection policy based on these cases. In this way, it continuously improves the robustness of unlearning without modifying model parameters. As a result, AAPO requires neither access to model weights nor additional fine-tuning. It can be applied directly to frozen and deployed models, and can be combined with existing unlearning methods.

Your comments on the update rule in Eq. 14, utility preservation, and the feedback pipeline are very much to the point, and they have helped us further improve the paper. We respond to each point below.

Comment 1: Eq. 14 uses normalized prompt weight as an importance-correction probability, although WDGS selects a set of prompts whose inclusion probabilities generally differ from those weights. The stated Exp3 rationale therefore does not justify the update. The authors should specify the selection law or evaluate the update explicitly as a heuristic. How does the probability in Eq. 14 correspond to WDGS inclusion probabilities? Does the update outperform simpler rules at equal budgets?

Response 1: Thank you for your careful and insightful comment. We truly appreciate your precise understanding of the method details. We answer your questions one by one below.

• Selection rule of WDGS. In each round, the Defender first determines the combination length ktk_t according to Eq. 11. It then samples the first prompt according to the probability p(t)p^{(t)}, and selects the remaining kt−1k_t-1 prompts one by one according to Eq. 12. In the revised version, we will provide the full selection procedure as pseudocode.

• Relation between Eq. 14 and the actual inclusion probability. When kt=1k_t=1, the inclusion probability of prompt ii is pip_i, and Eq. 14 is consistent with the standard Exp3 importance-weighted estimator. When kt>1k_t>1, prompt ii can be sampled as the first prompt (with probability pip_i), or selected later by Eq. 12. Therefore, its actual inclusion probability satisfies qi≥piq_i \ge p_i. In this case, Eq. 14 should be viewed as an Exp3-inspired heuristic update. We will revise the description in Section 4.3 accordingly and explicitly evaluate this update with the experiments below.

• Comparison with simpler update rules. The ablation study in Table 2 compares AAPO with two simpler settings. In w/o Exp3, the prompt weights are not updated, and all prompts keep the same weight. In Fixed-k1, only one prompt is selected in each round.

SettingForget Set↓Neighbor Set↑
Full (AAPO)43.576.8
w/o Exp360.978.5
Fixed-k159.379.0

Compared with w/o Exp3, the weight update in Eq. 14 reduces the forget-set score by 17.4 points. Compared with the single-prompt setting (Fixed-k1), which matches the standard Exp3 estimator, extending to multi-prompt combinations further reduces the forget-set score by 15.8 points, while the neighbor score drops by only 1.7 and 2.2 points, respectively. Therefore, although Eq. 14 is a heuristic update when kt>1k_t>1, it is effective in practice. We will add this analysis to the revised version. Thanks again!

AuthorsOpenReview ↗

Response to Reviewer gAVc [part 2]

Comment 2: Table 1 shows substantial utility losses that conflict with the utility-preservation claim. LLaMA factuality drops from 51.4 to 38.8 with GA and from 54.6 to 38.5 with DPO. The evaluation needs to show which operating points preserve utility while retaining the forgetting gains. Which operating points preserve utility while retaining the forgetting gains in Table 1?

Response 2: Thank you for pointing out this issue. We address it from two aspects: the cause of the drop, and the settings that preserve utility.

• Cause of the factuality drop. The clear factuality drop mainly occurs on LLaMA-3-8B-Instruct. It is mainly because AAPO makes the target model answer more conservatively. The defense prompts of AAPO mainly use refusal as the form of defense. Since GA and DPO have already changed the model's behavior, this refusal tendency can spread to some factual questions.

• Settings that preserve utility. In Table 1, on Qwen2.5-14B-Instruct, GA(+AAPO), DPO(+AAPO), and RT(+AAPO) reduce the forget-set score by 60.1%–70.3% (relative). At the same time, factuality increases from 28.8, 32.5, and 30.4 to 33.3, 35.8, and 34.8, respectively, while the other utility metrics remain largely unchanged. In addition, AAPO can adjust the trade-off between forgetting and utility through λ in Eq. 13, without retraining the target model. Appendix Table 3 shows that when λ is 0.2 or 0.3, the neighbor scores are 79.1 and 78.5, both higher than the 77.4 of the original model, while the forget-set score is still reduced from 72.6 to 58.3 and 51.7, respectively.

In the revised version, we will correct the statements about utility preservation in the main text and add the above analysis. Thank you very much!


Comment 3: The feedback pipeline is insufficiently specified. The judge checkpoint, reference-answer construction, and the point at which the learned guards are frozen remain unclear. Because judge feedback drives optimization, these details and an independent semantic-leakage evaluation are needed to assess the reported gains. Which judge and reference answers drive self-play, and when are guards frozen before evaluation?

Response 3: Thank you for your valuable suggestions. We answer your questions one by one below.

• Judge model and reference answers. The Judge is implemented with qwen-flash, with the temperature set to 0. It follows the judging rules in Appendix D.2 (Table 9) to decide whether a model response leaks the unlearned information. Each round of self-play is built around a base question about the unlearning target and its reference answer (see Appendix Table 12 for an example). The Attacker rewrites the base question into an adversarial prompt, and the Judge compares the target model's response with the reference answer to decide whether leakage occurs. The base questions and reference answers come from the training forget set provided by RWKU, which is separate from the evaluation probes.

• When the defense state is frozen. After 40 rounds of self-play for each unlearning target, the defense prompt pool and selection weights are frozen. A fixed combination of defense prompts is then determined from them and used for evaluation. No updates are made during evaluation.

• Independent semantic-leakage evaluation. The results in Table 1 use the ROUGE-L metric, which is independent of the Judge used in self-play. On this basis, to further evaluate leakage at the semantic level, we use qwen-flash, deepseek-v4-flash, and gpt-5.4-mini as LLM judges to re-evaluate the outputs of GA and DPO on LLaMA-3-8B-Instruct in Table 1, before and after adding AAPO.

Methodsqwen-flashdeepseek-v4-flashgpt-5.4-mini
GA78.478.479.1
GA(+AAPO)21.321.621.5
DPO83.484.283.5
DPO(+AAPO)22.221.422.0

The agreement rate among the three Judges is 98.2%. Under each Judge, AAPO greatly reduces the leakage rate, showing that its gains do not depend on the Judge used in self-play.

We will add the above details and experiments to the revised version. Thank you again!

In summary, we greatly appreciate your deep understanding and detailed suggestions, and we sincerely hope you will take the value of our paper into consideration. Thanks again!

Reviewer_D6t4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AAPO, an inference-time prompt evolution algorithm that mitigates the leakage of unlearned information under adversarial prompting. The defense prompts are optimized through a multi-agent framework consisting of an attacker, a judge, and a defender, without modifying model parameters. The method is evaluated mainly on the RWKU benchmark with two LLMs (LLaMA-3-8B-Instruct and Qwen2.5-14B-Instruct), with additional results on WMDP in the appendix.

Critical strengths
  1. The topic of designing robust unlearning methods is timely and important.
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. Missing key baselines: There is no comparison with existing robust unlearning methods, including training-based robust unlearning and inference-based robust unlearning.
  2. Underspecified method and cost: The source and size of the self-play data, the Judge model, and the number of targets are unclear. No training or inference cost is reported.
  3. Weak empirical rigor: All results are single runs with no variance. Hyperparameters appear to be tuned on the test set, and convergence is not convincingly shown.
  4. Limited scope: The evaluation covers only two models on one main benchmark, with WMDP relegated to the appendix with few details. It uses only fixed attack probes, with no adaptive attacks against the final defense.
  5. Conceptual limitation: A prompt-level defense does not remove knowledge and is easily bypassed on open-weight models.
Questions for authors

Please refer to the Critical Weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer D6t4

Thank you for your time and valuable comments! Your feedback is very helpful for further improving our paper.

To address your concerns about baseline comparisons, implementation details and cost, empirical rigor, evaluation scope, and the positioning of our method, we have added the corresponding experiments and clarifications. We respond to each point below.

Comment 1: Missing key baselines: There is no comparison with existing robust unlearning methods, including training-based robust unlearning and inference-based robust unlearning.

Response 1: Thank you for your comment. For inference-based methods, Appendix C.2 (Table 6) already compares AAPO with five inference-based methods that do not modify model parameters: Guardrail, Length-Matched, Multi-Guardrail, Iterative, and LLM-Filter. Building on this, we further add the inference-based unlearning method ICUL [1] and the training-based robust unlearning method LAU [2]. Since AAPO can be combined with parameter-level methods, we also report the results of LAU(+AAPO). All experiments are conducted on LLaMA-3-8B-Instruct.

MethodsForget Set↓Neighbor Set↑
Original72.677.4
ICUL36.054.0
AAPO43.576.8
LAU37.869.2
LAU(+AAPO)15.370.4

As shown in the table, ICUL and LAU reduce the forget-set score, but their neighbor scores drop to 54.0 and 69.2, respectively. In contrast, AAPO keeps the neighbor score at 76.8 without updating model parameters, which is close to the 77.4 of the original model. With AAPO on top of LAU, the forget-set score drops from 37.8 to 15.3 and the neighbor score slightly increases. This shows that AAPO can complement training-based robust unlearning methods. We will add this comparison to the revised version. Thanks again!

[1] In-Context Unlearning: Language Models as Few-Shot Unlearners

[2] Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Models


Comment 2: Underspecified method and cost: The source and size of the self-play data, the Judge model, and the number of targets are unclear. No training or inference cost is reported.

Response 2: Thank you for your questions. We answer them one by one below.

• Source and size of the self-play data. The Attacker generates self-play data from the unlearning target, attack strategy, and context (Appendix D.1). Self-play is performed for each unlearning target, i.e., for each target person on RWKU and for each domain on WMDP. In terms of size, we run 40 rounds of self-play for each target. In each round, the Attacker makes at most 3 attack attempts. If an attack succeeds, the process moves on to the next round. Otherwise, the Attacker adjusts its strategy based on the previous attacks and model responses, and then attacks again (see Appendix Table 12 for a complete example). As a result, at most 120 adversarial prompts are generated for each target.

• Judge model settings. The Judge is qwen-flash (temperature 0), following the rules in Appendix D.2 (Table 9) to decide whether a response leaks unlearned information.

• Number of unlearning targets. RWKU contains 200 target persons (Appendix A.2), and WMDP contains two domains, Bio and Cyber (Appendix C.3).

• Computational cost. The cost of AAPO mainly comes from agent calls during self-play. In each self-play interaction, the Attacker and the Judge are each called once. The Guard Generator is called only when leakage is detected, which happens 0.38 times per interaction on average. The average token usage and cost of each agent are shown below:

ComponentAvg. Input TokensAvg. Output TokensCost
Attacker Agent447286$0.00007
Judge Agent493112$0.00003
Guard Generator682314$0.00008
Total (avg. per interaction)1201519$0.00014

Based on this, the agent-call cost for each unlearning target is less than $0.02.

The average latency per interaction is shown below. The extra latency mainly comes from the agents called through the qwen-flash API. The policy update and prompt selection of the Defender take less than 5 ms per round, which is negligible.

MethodAvg. Latency (s)Overhead vs. Vanilla
Vanilla Inference4.44--
AAPO (per self-play interaction)9.17+4.73s

The defense prompt pool and selection policy obtained from self-play can be reused for later requests on the same target, so self-play does not need to be rerun for each request. When new leakage feedback appears, only the external defense state needs to be updated, without retraining the target model. Compared with parameter-modification methods, AAPO requires no additional training. We therefore consider the above cost acceptable.

We will add the above content to Appendix B in the revised version. Thank you again!

AuthorsOpenReview ↗

Response to Reviewer D6t4 [part 2]

Comment 3: Weak empirical rigor: All results are single runs with no variance. Hyperparameters appear to be tuned on the test set, and convergence is not convincingly shown.

Response 3: Thank you for your valuable comments. The issues you raised are very important for improving the rigor of our paper. We address them one by one below.

• Variance analysis. During evaluation, the target model uses greedy decoding, and the temperatures of the Attacker and the Guard Generator are also set to 0. Thus, the randomness of AAPO mainly comes from the Defender sampling prompts according to the probability p(t)p^{(t)}. We therefore ran self-play 3 times independently and evaluated each run. The results (mean ± std) are shown below.

MethodsForget Set↓Neighbor Set↑
AAPO44.2 ± 1.277.3 ± 2.0
GA(+AAPO)15.9 ± 1.470.7 ± 1.8

As shown in the table, the standard deviations of AAPO are small, and the means are close to the results reported in the paper. This shows that the results are stable.

• Hyperparameter setting. The default hyperparameters of AAPO were set before the experiments. λ and β are set to 0.5, which gives equal weight to the two objectives, and η is set to the commonly used value of 0.1. The same set of hyperparameters is used for both models, all four unlearning methods, and both benchmarks, without tuning for any specific setting. In the sensitivity analysis in Appendix C, the default values are not the best on every metric. For example, λ=0.8 gives a lower forget-set score, and η=0.01 gives a higher neighbor score. Across all tested values, AAPO reduces the forget-set score from 72.6 (original model) to 40.9–58.3, while keeping the neighbor score within 70.1–79.1. This shows that its effectiveness does not depend on a specific set of hyperparameters.

• Policy convergence. Figure 4 shows that, as self-play proceeds, the selection probability gradually shifts from the initial prompts to a few prompts generated by GuardGen. Figure 5 shows that the combination length kk gradually decreases from 5 at round 10 to 3 at round 40. Since kk is determined by the policy entropy (Eq. 11), this indicates that the policy entropy decreases over rounds and the defense policy gradually converges to a stable prompt combination.

We will add the above analysis to the revised version. Thank you very much!

AuthorsOpenReview ↗

Response to Reviewer D6t4 [part 3]

Comment 4: Limited scope: The evaluation covers only two models on one main benchmark, with WMDP relegated to the appendix with few details. It uses only fixed attack probes, with no adaptive attacks against the final defense.

Response 4: Thank you for your comment. We address it in two parts.

• Coverage of models and benchmarks. Our evaluation covers two models from different families and of different sizes (LLaMA-3-8B-Instruct and Qwen2.5-14B-Instruct), as well as two benchmarks (RWKU and WMDP). In the revised version, we will move the WMDP results to the main text and add the corresponding experimental details.

• Adaptive attacks against the final defense. To evaluate the robustness of the final defense more directly, we conducted additional evaluations on RWKU. The evaluation includes four types of attacks that were not used in self-play: (i) multilingual attacks, covering French, Spanish, German, Russian, Chinese, Japanese, Arabic, and mixed Chinese-English; (ii) many-shot attacks [1], using prompts with N∈{1,4,8,16,32}N\in\{1,4,8,16,32\} examples of compliant answers; (iii) schema-aware attacks, where the attacker can access AAPO's full defense prompt pool and its structural patterns; and (iv) PAIR attacks [2], which perform iterative black-box prompt search based on the responses of the defended model. Among them, schema-aware and PAIR attacks are adaptive attacks that directly target the final defense.

DefenseMultilingualMany-shotSchema-awarePAIRAverage ASR
Original0.860.870.911.000.91
GA0.830.770.901.000.88
GA(+AAPO)0.150.080.390.370.25
DPO0.820.790.920.930.87
DPO(+AAPO)0.100.100.670.570.36

As shown in the table, after adding AAPO, the ASR drops clearly for all attack types, including the two adaptive attacks. This shows that the defense state learned by AAPO can transfer to attacks unseen during self-play, as well as to attacks that know the defense content.

In addition, AAPO does not rely on keeping a single static defense prompt secret. Its defense state consists of a dynamic prompt pool, learned selection weights, and new guards generated from past failures. Therefore, even if the defense content becomes known to the attacker, AAPO can still use the newly exposed failure modes as feedback to keep updating the defense. To verify this, we used schema-aware and PAIR attacks as feedback for 10 rounds of adaptation on the original model + AAPO, and compared the ASR before and after adaptation.

Defense stateSchema-awarePAIR
ASR0\mathrm{ASR}_00.580.67
ASR1\mathrm{ASR}_10.410.29
Δ\Delta ASR-0.17-0.38

After adaptation, the ASR of the two attacks drops by 17 and 38 percentage points, respectively. This shows that exposing the defense information does pose a threat, but after observing the new attack distribution, AAPO can partially recover its defense ability without retraining the target model. We will add these experiments and analysis to the revised version. Thanks again!

[1] Many-shot Jailbreaking

[2] Jailbreaking Black Box Large Language Models in Twenty Queries


Comment 5: Conceptual limitation: A prompt-level defense does not remove knowledge and is easily bypassed on open-weight models.

Response 5: Thank you for your comment. AAPO indeed does not remove the unlearned knowledge from the model parameters. Instead, it defends at inference time through the Attacker–Judge–Defender loop. We believe this does not conflict with the core goal of LLM unlearning. Parameter modification is one way to achieve unlearning, but its effectiveness depends on whether the specified information can be elicited again from the model outputs. AAPO requires neither fine-tuning nor access to model weights, so it also applies to frozen and closed-source models, and it dynamically updates its defense as attacks evolve.

For open-weight models, if an attacker can access the weights and remove the system prompt, AAPO's defense may indeed be bypassed. AAPO therefore mainly targets settings where a service provider deploys the model and users cannot modify the system prompt. AAPO can also complement parameter-level unlearning. The parameter-level method establishes the basic unlearned state, and AAPO further improves robustness against adaptive prompt elicitation at inference time. In Table 1, AAPO reduces forget-set leakage on top of GA, DPO, NPO, and RT, which reflects this complementarity.

In the revised version, we will clarify this complementarity in the Introduction and Method sections, and describe AAPO's scope of application in the Limitations section. Thank you again!

In summary, we greatly appreciate your careful review and valuable comments, and we sincerely hope that the above responses address your concerns. Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.