- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Summary
- In multilingual fine-tuning, LATTE aligns cross-lingual representations so that subsequent single-language unlearning transfers more effectively to other languages.
- The idea is intuitive, and there is a particularly large improvement in cross-lingual forgetting for NPO and GradDiff.
Overall
- The idea and the results are interesting and have practical value.
- The biggest issue is the lack of a baseline to separate the LATTE-specific contribution.
- Currently, relying mainly on NLI/ROUGE-centered evaluations is somewhat limited for making a strong knowledge erasure claim.
- Including stronger baselines and additional unlearning evaluations could significantly strengthen the paper.
Critical strengths
- The problem setting is important, and the motivation is clear.
- The approach is natural, first analyzing the relationship between representation alignment and unlearning transfer before designing the method.
- Qwen shows a large improvement in cross-lingual forgetting for NPO/GradDiff.
- It is practical that the method can be used without modifying existing unlearning objectives.
- The ablation and retrieval analyses support the proposed mechanism to some extent.
Critical weaknesses
1. The comparison baseline is insufficient
- The current main comparison focuses on Standard fine-tuning and LATTE.
- It is difficult to tell whether this is a LATTE-specific effect or simply the result of improved cross-lingual alignment.
- Comparing against generic alignment baselines and major cross-lingual unlearning methods would clarify its novelty and advantages.
- Approach is different though, performance should be compared against related existing methods.
Examples of related methods:
- Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
- Cross-lingual Unlearning of Selective Knowledge in Multilingual Language Models
- UNLEARN: Efficient Removal of Knowledge in Large Language Models
- Layer-Targeted Multilingual Knowledge Erasure in Large Language Models
2. The evaluation of unlearning is somewhat limited
- The main evaluations are Forget/Retain NLI and ROUGE-L.
- These metrics show whether the target answer is still being generated, but it is difficult to confirm whether the underlying knowledge has actually been removed.
- Auxiliary evaluations such as paraphrased queries, Truth Ratio/likelihood, recovery, or steering tests would be more convincing.
3. The dataset range is limited
- TOFU is based on fictitious-author synthetic knowledge, and LUME uses only the synthetic-story task rather than the entire benchmark.
- It is unclear whether the same effect occurs with actual pretrained factual/private knowledge.
Questions for authors
- Does the same trend hold for larger models or real pretrained knowledge?
- Does applying only generic middle-layer alignment produce a similar unlearning gain?
5 comments
Official Comment by Authors
W3/Q1: TOFU is based on fictitious-author synthetic knowledge, and LUME uses only the synthetic-story task rather than the entire benchmark. It is unclear whether the same effect occurs with actual pretrained factual/private knowledge.
LATTE is designed to make knowledge acquired during multilingual fine-tuning easier to unlearn across languages. Its alignment intervention acts while that knowledge is learned, before future forget requests are known. Removing knowledge already encoded during pretraining is outside the setting LATTE targets. TOFU and LUME's synthetic-story task introduce facts during fine-tuning and therefore directly evaluate the setting our method is designed to address.
W3/Q1: Does the same trend hold for larger models?
In addition to the 3B models in the paper, we extended the evaluation to Qwen2.5-7B-Instruct on TOFU using English-only NPO. We report both forgetting and retention below, since the practical benefit depends on the trade-off achieved by each unlearning method.
| Method | Model | Forget NLI | Retain NLI |
|---|---|---|---|
| NPO | Standard | 0.78 | 0.94 |
| NPO | LATTE | 0.50 | 0.92 |
LATTE lowers forget NLI in all ten languages. NPO reduces mean forget NLI by 0.28 while mean retain NLI decreases by 0.018. The NPO results demonstrate that LATTE's multilingual forgetting benefits extend to a 7B model with mean retain NLI of 0.92. Appendix A.7, Table 16 reports the full results.
If the above addresses the reviewer's concerns, we would be grateful if they could update their score accordingly.
Official Comment by Authors
As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. To address their comments, we have additionally:
-
Compared LATTE directly with middle-layer alignment. On Qwen/TOFU, LATTE achieves stronger forgetting (0.16 versus 0.40) and higher retention (0.88 versus 0.84), supporting the value of its complete training objective (Appendix A.5, Table 14).
-
Added multilingual NPO, multilingual LINGTEA, and MUTE comparisons. LATTE achieves similar forgetting to multilingual NPO with higher retention while using only English unlearning examples (Appendix A.6, Table 15).
-
Broadened evaluation to unseen paraphrases, answer likelihood, and Truth Ratio. Mean non-English forget reduction on paraphrases increases from 0.32 to 0.74, and the relative Truth Ratio decrease reaches 42.6% versus 4.9% (Appendices A.3 and A.4).
-
Added steering and retraining attacks. These show partial reversibility, while LATTE maintains lower mean forget NLI after retraining, at 0.49 versus 0.66 (Appendix A.10).
-
Extended evaluation to Qwen2.5-7B-Instruct, where LATTE lowers forget NLI in all ten languages under NPO, and clarified the scope of knowledge acquired during fine-tuning (Appendix A.7, Table 16).
We believe these additions provide substantially stronger baseline comparisons and evidence beyond NLI and ROUGE-L. We would appreciate your reconsideration of the rating, including an increase if the new experiments address your concerns.
Official Comment by Authors
We thank the reviewer for their careful assessment and constructive suggestions. We appreciate their recognition of the importance of the problem, the clear motivation, and the practical value of improving cross-lingual forgetting without modifying existing unlearning objectives.
To distinguish the contribution of LATTE's combined objective from gains due to alignment alone, we compared LATTE with generic middle-layer alignment on Qwen2.5-3B-Instruct using the same multilingual TOFU data and identical English-only NPO settings. NLI scores are averaged across all ten languages, and Tℓ across the nine non-English languages.
LATTE achieves stronger forgetting alongside higher average retention and normalized transfer. Our loss ablations in Table 5 further show that removing either preservation term weakens both forgetting and retention across all five alignment targets. Together, these results support the benefit of combining alignment with knowledge preservation for subsequent single-language unlearning, beyond the gains from generic middle-layer alignment. The comparison is reported in Appendix A.5, Table 14.
Based on the reviewer's feedback, we added comparisons using the same underlying forget facts and ten evaluation languages, reporting both forgetting and retention to make the trade-offs explicit. Appendix A.6 and Table 15 provide the configurations and results.
LATTE achieves similar forget NLI to multilingual NPO (0.16 versus 0.17), with higher retain NLI (0.88 versus 0.72). Compared with multilingual LINGTEA, LATTE achieves lower forget NLI (0.16 versus 0.63), with similar retain NLI (0.88 versus 0.89). These results demonstrate strong multilingual forgetting using only English unlearning examples. We did not include a quantitative comparison with UNLEARN because the authors did not provide a code implementation.
We broadened the analyses on Qwen2.5-3B-Instruct/TOFU after English-only NPO. These evaluations test whether LATTE's gains extend to unseen question wording and probability-based measures. The table reports averages over the nine non-English languages.
Retain-answer negative log-likelihood increases by less than 0.03 for both models in every language. These results show that LATTE's forgetting gains extend to unseen question wording and probability-based evaluations beyond NLI and ROUGE-L. Appendix A.4, Tables 12 and 13 report the full results.
We also evaluated steering and retraining attacks after English-only NPO. The following table reports mean forget NLI across all ten languages. Lower values indicate stronger forgetting.
Steering produces only a small additional increase in LATTE's mean forget NLI over the random-direction control under the tested strengths. Retraining recovers part of the forget performance in both models and a larger fraction of LATTE's initial reduction in most languages, showing that some of its additional forgetting remains reversible. Nevertheless, LATTE retains lower mean forget NLI after retraining. In seven languages, every LATTE run remains below every standard fine-tuning run. Appendix A.10 reports the attack settings, controls, and per-language results.