- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of cross-lingual loophole in LLM unlearning and provides a large-scale systematic analysis of when forgotten knowledge remains accessible across languages, scripts, and paraphrased expressions. The authors introduce the Cross-lingual Unlearning Tensor, covering 174 language-script pairs and 25 paraphrase types, and use it to analyze unlearning across different query and answer routes. They further define language-budgeted multilingual unlearning and propose COVER, a source-language selection method based on cross-language coverage estimated from benign calibration data. COVER improves held-out forgetting in several settings, although the gains vary across models and datasets and are more limited on the real-world LORELEI evaluation. Experiments on Aya, Qwen, and Llama, together with analyses of learning and unlearning transfer, language bottlenecks, script and paraphrase sensitivity, and source-set stability, show that cross-lingual forgetting varies greatly across linguistic routes and that effective multilingual unlearning depends on carefully selecting a limited set of source languages.
Critical strengths
-
The paper introduces a broad and well-structured benchmark for cross-lingual unlearning. The Cross-lingual Unlearning Tensor covers 174 language-script pairs and 25 paraphrase types, while separately evaluating query and answer language routes. This provides a much more systematic basis for studying cross-lingual forgetting than prior evaluations on a small number of languages.
-
The paper provides a systematic study of language-budgeted multilingual unlearning. By variously comparing source-language subsets across different models, language inventories, training orders, and disjoint forget requests, the experiments clearly show that source-language choice has a substantial and partly stable effect on held-out forgetting. This gives strong support for the need to carefully select a limited set of source languages.
Critical weaknesses
The limited advantage of COVER over simpler selection strategies is a central weakness of the paper. On TOFU, COVER selects the same source subsets as the fixed coverage rule for Aya and Qwen, indicating that it adds little benefit over a simpler heuristic. For Llama, COVER differs from fixed coverage, but its gains are mixed and it does not consistently outperform alternatives such as the historical-best subset. More importantly, on the human-translated LORELEI news data, COVER again coincides with fixed coverage and delivers only modest improvements, leaving limited evidence that the method provides a substantial advantage on more realistic data.
Questions for authors
-
Why does the learned COVER selector often choose exactly the same source subsets as fixed coverage, and what information learned from historical unlearning outcomes actually changes the ranking when the two methods differ?
-
Why are COVER’s gains substantially smaller on the human-translated LORELEI news data than on the synthetic benchmarks, especially for document-interior prompts? Could the authors provide further analysis of what causes this degradation in the more realistic setting?
4 comments
2. What information learned from historical unlearning outcomes actually changes the ranking (Response to Reviewer xjZW)
On our LUME forget request experiment, learning changes the selected source set and reduces residual access from 39.26 to 32.13, ranking first in both realizations. 4. What from historical unlearning outcomes actually changes the ranking when the two methods differ decomposes the exact scoring contributions responsible for that decision. Please see 4. What from historical unlearning outcomes actually changes the ranking when the two methods differ for the concrete examples and feature decomposition. The learned ranker is described in Section 4.1.
3. Smaller LORELEI/Interior gain (Response to Reviewer xjZW)
We investigated two aspects of the smaller LORELEI gains. (1) whether forgetting generalizes across document segments, and (2) how much improvement is available through source selection.
For document interiors, our stopping protocol leaves an additional generalization challenge that we did not sufficiently detail. In our experiment, unlearning stops based on source language forgetting at document opening, whereas interior evaluation uses different contexts and continuation targets within those documents. Across all 20 source triples and two training orders, mean oracle normalized source removal reaches 95.7% for Aya and 95.6% for Qwen on leads, compared with 86.9% and 89.0% on interiors. Thus, comparable forgetting at document openings does not imply comparable forgetting throughout the document, even in the directly supervised languages.
To put the smaller absolute selection gains in context, we also compared COVER with the best source triple identified in hindsight (via repeated empirical rankings). The best triple across both orders improves over uniform selection by 6.96 points for Aya and 3.91 for Qwen. COVER achieves 4.60 and 2.31 points, capturing 66% and 59% of that improvement. COVER therefore remains competitive among source selections on leads, while its benefit transfers less reliably to document interiors.
Response to author
Thank you for your detailed response to my concerns. I will keep my positive rating.
1. Additional baselines (Response to Reviewer xjZW)
We added geographic coverage, cosine-similarity coverage, English constrained coverage, and script diversity. COVER achieves 32.13 residual access versus 36.84 for the strongest added baseline on LUME. 3. Simple baselines presents the comparisons, including retention tradeoffs, alongside the existing baselines. Please see 4. Simple baselines. We have added the four baselines to Table 4b and their definitions to Appendix D.6 (“Additional source selection baselines”).