acceptodds
Under review as a conference paper at ICLR 2027

Learning to Forget Together: Prompt-Conditioned Adapter Generation for Multi-Domain LLM Unlearning

Abstract

Large language models (LLMs) can memorize and disclose sensitive information, making machine unlearning an important tool for mitigating privacy and safety risks. However, removing knowledge across multiple domains can cause interference between unlearning updates and degrade general model capabilities. The need to retrain for different combinations of target domains further complicates deployment. We introduce PUMA (Prompt-conditioned Unlearning via Multi-domain Adapter Generation), a framework that generates low-rank adaptation (LoRA) weights from prompts representing the knowledge to be forgotten. A shared parameter generator learns to translate instance-level prompts into unlearning adapters, enabling flexible requests expressed through examples. Once trained, PUMA supports instance-based, multi-domain unlearning without further optimization for each request, enabling zero-shot deployment. It combines forgetting objectives across domains within a single generated adapter, eliminating the need to train a separate adapter for every domain combination. Experiments show that PUMA achieves a better balance between forgetting efficacy and retained utility than existing domain-level unlearning methods while accommodating multiple target domains. These results highlight the potential of prompt-conditioned parameter generation for flexible and efficient LLM unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine learning, parameter generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4tioOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces PUMA, a framework for multi-domain unlearning in large language models that generates low-rank adapters from prompts, enabling flexible unlearning without retraining for each domain. It combines forgetting objectives across domains and demonstrates improved performance over existing methods.

Critical strengths
  1. The paper proposes a novel framework, PUMA, which addresses the challenge of multi-domain unlearning by generating adapters from prompts, enabling flexible and efficient unlearning without retraining for each domain combination.
  2. The experiments are comprehensive, evaluating PUMA across multiple LLM backbones and domains, showing superior performance in balancing forgetting efficacy and retained utility compared to existing methods like Single, EUL and ALKN.
  3. The method’s efficiency is highlighted, as PUMA avoids retraining for each request and supports zero-shot deployment through a shared generator. This is a practical advantage for real-world applications.
  4. The paper incorporates structural embeddings and module-specific residual heads, which enhance the model’s ability to adapt to domain-specific updates, as demonstrated through ablation studies.
Critical weaknesses
  1. The theoretical analysis of PUMA’s parameter generation mechanism is limited. While the framework is empirically validated, the paper lacks a rigorous mathematical proof or derivation for the generator’s behavior.
  2. The experiments do not explicitly address the scalability of PUMA to domains outside the tested ones (e.g., unseen domains). The paper assumes that the generator generalizes well but does not provide evidence for this claim.
  3. The paper does not compare PUMA’s computational efficiency (e.g., memory usage or inference speed) with existing methods, which is critical for practical deployment.
Questions for authors
  1. PUMA’s parameter generator has only empirical support, without rigorous mathematical derivation to explain its behavior.
  2. The experiments do not verify whether PUMA can generalize to unseen domains, and the assumption of cross-domain transfer lacks supporting evidence.
  3. The paper omits comparisons of PUMA’s memory cost and inference speed against prior work, which restricts assessment of its practical usability.
Reviewer_hPDvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PUMA, a multi-domain LLM unlearning method based on prompts-conditional generation of LoRA parameters. The authors first train forgetting adapters for different domains, then use a shared parameter generator to learn the mapping from textual cues to LoRA weights. This allows the model to directly generate the corresponding adapter based on the input forgetting request without additional optimization. The method also introduces structure embeddings and module-specific residual modules to improve parameter generation. The authors evaluated PUMA on four LLMs and nine forgetting domains, demonstrating a balance between forgetting performance and non-target knowledge preservation compared to independently trained domain adapters.

Critical strengths
  1. PUMA directly maps textual forgetting requests to LoRA adapters via a shared parameter generator, avoiding the overhead of retraining the model for each request. This design provides a more flexible solution for forgetting tasks across multiple known domains.

  2. The authors validated PUMA on four different LLMs, demonstrating improvements in average forgetting performance and knowledge retention compared to independently trained domain adapters.

Critical weaknesses
  1. Although the paper emphasizes multi-domain unlearning, PUMA actually generates a LoRA for each domain for each request, rather than forgetting multiple domains simultaneously in a single adapter. The results in Table 1 are also averaged after evaluating each domain separately, and are not entirely comparable to baselines that require processing all forgotten domains simultaneously.

  2. During training, different text groups from the same domain are used to predict LoRA checkpoints for that domain. Therefore, the generator may primarily be identifying the domain rather than generating differentiated updates based on specific forgotten content. Currently, there are no comparisons with simpler approaches such as domain-label conditional generation or directly selecting existing adapters, nor has selective forgetting for specific instances been verified.

  3. The forgetting effect is mainly measured by the decrease in MMLU accuracy, making it difficult to determine whether the target knowledge has truly been forgotten.

Questions for authors
  1. The results for the full model (+SE + MRH) in Table 2 are 29.3, 31.8, 55.6, and 59.4, which are completely consistent with the single results for Qwen2.5-3B in Table 1, not the results for PUMA. The same values ​​appear when using 9 training domains in Table 3. Can the authors explain the specific settings of these experiments? Why do the ablation results of the full model not correspond to the performance of PUMA in Table 1?

  2. According to Section 3.2, different groups of randomly sampled texts within the same domain are paired to LoRA checkpoints for that domain. Does this training method cause the generator to primarily learn domain recognition, rather than adjusting forgetting behavior based on specific text content? Currently, the visualization results in Figure 2 are insufficient to demonstrate instance-level forgetting capabilities.

  3. The paper emphasizes multi-domain unlearning, but PUMA in Table 1 actually generates adapters separately for each domain and then aggregates the evaluation results. Can the authors clarify whether PUMA supports specifying multiple forgetting domains in a single request and generating an adapter capable of forgetting all of these domains simultaneously?

  4. Appendix A mentions that some generator experiments used the full benchmark results to select update coefficients, while others used approximately 20% of the data for calibration. Can the authors clarify which settings were used in Tables 1, 2, and 3 respectively? Did Single and other baselines use the same level of parameter selection? If the generator adapter still needs to use labeled data from the target task to select update strength, what is the relationship between this setting and the zero-shot deployment emphasized in the paper?

  5. The paper primarily uses the accuracy decrease of MMLU and WMDP to measure the forgetting effect. However, accuracy decrease does not necessarily mean that relevant knowledge has been deleted; it may simply mean that the model's answering ability has been interfered with. Have the authors verified the stability of forgetting through question rewriting, open-ended question answering, or other knowledge extraction methods?

Reviewer_yZBsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PUMA, a prompt-conditioned generator for multi-domain LLM unlearning. The method first trains domain-specific unlearning LoRA adapters and then learns a shared generator to predict LoRA weights from textual requests. This allows the model to generate unlearning adapters without additional optimization at inference time. Experiments are conducted on several LLM backbones across nine forgetting domains.

Critical strengths
  1. The paper addresses a practical problem in multi-domain unlearning, and the motivation is reasonable.
  2. The method is clearly presented. Generating LoRA weights from textual requests is an interesting direction that could reduce the need for repeated optimization.
  3. The experiments cover multiple LLM backbones, and PUMA shows consistent improvements over independently trained LoRA adapters.
Critical weaknesses
  1. The multi-domain comparison is not entirely fair. PUMA generates a separate adapter for each requested domain, while several baselines (e.g., EUL, Arithmetic, and TIES) handle multiple forgetting domains within a single model. These are different settings, and the baselines may face more interference between forgetting objectives. As a result, the current comparison does not fully support the claim that PUMA effectively addresses interference in multi-domain unlearning. I would like to see comparisons under the same setting, especially when multiple domains need to be forgotten simultaneously.

  2. The improvement is relatively limited, while retained utility still drops substantially. Although PUMA improves over Single LoRA, the retained utility remains considerably lower than that of the original model. For example, on Qwen2.5-7B, retain accuracy drops from 72.4% to 63.5%, while the improvement over Single LoRA is only 1.7 percentage points. Given the additional cost of training a parameter generator, I am not fully convinced that these gains are sufficient to justify the proposed framework.

  3. The paper lacks robustness evaluation for unlearning. The evaluation mainly relies on accuracy drops on MMLU and WMDP. However, lower accuracy alone does not necessarily indicate that the target knowledge has been reliably forgotten. The paper does not evaluate whether the forgotten knowledge can be recovered through paraphrased prompts, adversarial prompting, or relearning attacks. While the authors study stability across different numbers of training domains and cross-domain effects, these experiments do not directly measure unlearning robustness. I consider this an important missing evaluation.

Questions for authors
  1. Can PUMA generate a single adapter that handles forgetting requests from multiple domains simultaneously?
  2. Can the generator distinguish different forgetting targets within the same domain? For example, can it forget a specific subset of knowledge while preserving other knowledge from that domain?
  3. Could the authors clarify whether the adapter scaling factors were selected using a separate calibration set, and whether the same tuning protocol was applied to all baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.