- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how LLM unlearning transfers across QA and continuation formats, languages, and prompt-based attacks. AUGUNLEARNING provides synthetic variants and a score combining forget-set removal with retain-set preservation. The experiments identify a pronounced QA/continuation boundary, examine module-specific updates, and show that cross-form augmentation and MLP-only training can improve the trade-off for some methods.
Critical strengths
I find the empirical study convincing, particularly the distinction between generalization to paraphrases within the same form and transfer across QA and continuation formats. The evaluation separates these effects, and the module-specific experiments show that similar forgetting on the training format can conceal substantially different cross-form performance. The checks that frozen parameters remain unchanged further strengthen this comparison. The figures present these patterns clearly, making it easy to connect the evidence to the paper’s claims about the limits of unlearning generalization.
Critical weaknesses
My main concern is that the additional insight over prior work is not sufficiently clear. The experimental coverage is substantial, but several of the central observations and evaluation choices have close precedents.
For cross-form generalization, RWKU (Jin et al., 2024) already reports that fill-in-the-blank and adversarial probes can recover knowledge that appears forgotten under QA evaluation. RULE (Zhang et al., 2025, Appendix A.1) also explicitly trains on both completion-style inputs and derived QA queries. Although its refusal-based setting differs, this is a close precedent for the proposed cross-form augmentation. More directly, the statement in Section 2 that existing benchmarks “train and score within a single form” is too broad: MUSE evaluates both verbatim continuation and QA derived from the same source text, while RWKU includes fill-in-the-blank, QA, and nine types of adversarial probes.
The other findings also need a sharper comparison with existing evidence. Choi et al. (2024) and Farashah et al. (2026) already study uneven cross-language transfer and resource-related differences. The paper’s stronger transfer result may be informative, but I would like a clearer account of what explains the difference. For module localization, Guo et al. (2025) already show that targeting fact-lookup MLPs improves robustness across input formats and against relearning, distinguishing this from interventions on output-extraction mechanisms.
I appreciate the bidirectional QA/continuation matrix, gradient analyses, and extension to a hybrid architecture. However, the paper needs to explain more precisely which conclusions these experiments add beyond the existing observations, rather than relying mainly on broader coverage to establish the contribution.
Questions for authors
See weakness.
We thank the reviewer for the precise pointers. Regarding the comment that "the paper needs to explain more precisely which conclusions these experiments add beyond the existing observations," we admit that they are not stated explicitly in one place, as they surface as the paper progresses, from Section 4 through Section 6. We list them below.
What the paper adds beyond precedents
o_projalone matches the MLP on the trained form Q = 0.91 with cross-form Q = 0.09, also in a linear-attention hybrid (Table 8).What explains the stronger language transfer Transfer across languages is uneven in our grid too, with Burmese and Yoruba the weakest sources, but it is consistently stronger than transfer across forms. This contradicts Choi et al.\ and Farashah et al., who find that unlearning transfers weakly across languages and leaves low-resource languages exposed. We think the contradiction comes from how performance is measured. The length-normalized answer probability used in earlier work isn't fair across languages. Besides capturing whether the model knows a fact, it also rewards the model for being fluent in the script, which depends heavily on tokenization. In our own runs, after unlearning in English, 0.47 of the average token probability survives in Burmese but only 0.21 in Spanish. Burmese, however, averages 0.71 characters per token, roughly three times as many tokens per character as most languages, so most of that high token probability comes from the model predicting the next piece of a Burmese word well, not from remembering the fact (Appendix~G). Under this metric, heavy tokenization makes low-resource languages look leaky even when the knowledge is gone. That is why we built our metrics to measure knowledge only for a fairer cross-language metric. Measured this way, much of the cross-lingual leakage reported before turns out to come from the metric rather than from knowledge that survived.
In a nutshell. From prior work: add other languages, forms, and paraphrases to the forget set, and train every objective the same way. After our study: no paraphrases, no extra languages, at most one added form, and a different decision for each objective family (Table 2):
Wording we will fix. The Section 2 sentence that benchmarks "train and score within a single form" is too broad, since MUSE and RWKU probe more than one form. The accurate statement is that they unlearn on one form and probe others in one direction, and none varies the training form with the objective held fixed. Section 5 will no longer say that we "revise" earlier conclusions. We will cite RULE and discuss Guo et al. as a direct precedent.
The revision will state these additions explicitly in the introduction, each against its closest precedent. If this clarifies the increment, we would be grateful if the reviewer would consider raising the score.