acceptodds
Under review as a conference paper at ICLR 2027

Language-Aligned Training for Transferable Erasure in Multilingual LLMs

Abstract

Large language models are nowadays multilingual. This means they can process and generate text in many languages, and the same fact can be represented across languages. While this behavior is desired, it also makes it harder to unlearn data, such as outdated, private, or harmful facts. This is because even when a fact is unlearned in one language, this fact can often still be accessed through prompts in other languages. Prior work either characterizes this gap or addresses it during unlearning itself, by adapting the unlearning objective per language, restricting updates to language-agnostic layers, or projecting out shared knowledge subspaces. These approaches often require forget data in several languages and still yield non-uniform forget and retain accuracies across languages. In this work, we observe that English-only unlearning transfers more strongly to languages whose prompt representations are better aligned with their English counterparts, as measured by cross-language retrieval accuracy. Based on this observation, we propose LATTE (Language-Aligned Training for Transferable Erasure), a method that aligns the representations of the same facts across different languages already during fine-tuning. Afterwards, any standard unlearning method can be applied without modification, using only single-language forget data. Our evaluation on Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct across up to ten languages shows that LATTE learns multilingual facts as well as standard fine-tuning. We additionally show that ours can be combined with any state-of-the-art single-language unlearning method and improves cross-lingual forgetting while maintaining retain performance, both in the unlearning language and in the others. Thereby, LATTE makes a significant step towards reliable unlearning in multilingual state-of-the-art models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
unlearning, alignment, multilingual

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Aj6qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Summary

  • In multilingual fine-tuning, LATTE aligns cross-lingual representations so that subsequent single-language unlearning transfers more effectively to other languages.
  • The idea is intuitive, and there is a particularly large improvement in cross-lingual forgetting for NPO and GradDiff.

Overall

  • The idea and the results are interesting and have practical value.
  • The biggest issue is the lack of a baseline to separate the LATTE-specific contribution.
  • Currently, relying mainly on NLI/ROUGE-centered evaluations is somewhat limited for making a strong knowledge erasure claim.
  • Including stronger baselines and additional unlearning evaluations could significantly strengthen the paper.
Critical strengths
  • The problem setting is important, and the motivation is clear.
  • The approach is natural, first analyzing the relationship between representation alignment and unlearning transfer before designing the method.
  • Qwen shows a large improvement in cross-lingual forgetting for NPO/GradDiff.
  • It is practical that the method can be used without modifying existing unlearning objectives.
  • The ablation and retrieval analyses support the proposed mechanism to some extent.
Critical weaknesses

1. The comparison baseline is insufficient

  • The current main comparison focuses on Standard fine-tuning and LATTE.
  • It is difficult to tell whether this is a LATTE-specific effect or simply the result of improved cross-lingual alignment.
  • Comparing against generic alignment baselines and major cross-lingual unlearning methods would clarify its novelty and advantages.
  • Approach is different though, performance should be compared against related existing methods.

Examples of related methods:

  • Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
  • Cross-lingual Unlearning of Selective Knowledge in Multilingual Language Models
  • UNLEARN: Efficient Removal of Knowledge in Large Language Models
  • Layer-Targeted Multilingual Knowledge Erasure in Large Language Models

2. The evaluation of unlearning is somewhat limited

  • The main evaluations are Forget/Retain NLI and ROUGE-L.
  • These metrics show whether the target answer is still being generated, but it is difficult to confirm whether the underlying knowledge has actually been removed.
  • Auxiliary evaluations such as paraphrased queries, Truth Ratio/likelihood, recovery, or steering tests would be more convincing.

3. The dataset range is limited

  • TOFU is based on fictitious-author synthetic knowledge, and LUME uses only the synthetic-story task rather than the entire benchmark.
  • It is unclear whether the same effect occurs with actual pretrained factual/private knowledge.
Questions for authors
  • Does the same trend hold for larger models or real pretrained knowledge?
  • Does applying only generic middle-layer alignment produce a similar unlearning gain?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for their careful assessment and constructive suggestions. We appreciate their recognition of the importance of the problem, the clear motivation, and the practical value of improving cross-lingual forgetting without modifying existing unlearning objectives.

W1: It is difficult to tell whether this is a LATTE-specific effect or simply the result of improved cross-lingual alignment.

Q2: Does applying only generic middle-layer alignment produce a similar unlearning gain?

To distinguish the contribution of LATTE's combined objective from gains due to alignment alone, we compared LATTE with generic middle-layer alignment on Qwen2.5-3B-Instruct using the same multilingual TOFU data and identical English-only NPO settings. NLI scores are averaged across all ten languages, and TℓT_\ell across the nine non-English languages.

Training methodForget NLIRetain NLIMean TℓT_\ell
Generic middle-layer alignment0.400.840.80
LATTE0.160.880.91

LATTE achieves stronger forgetting alongside higher average retention and normalized transfer. Our loss ablations in Table 5 further show that removing either preservation term weakens both forgetting and retention across all five alignment targets. Together, these results support the benefit of combining alignment with knowledge preservation for subsequent single-language unlearning, beyond the gains from generic middle-layer alignment. The comparison is reported in Appendix A.5, Table 14.

W1: Comparing against major cross-lingual unlearning methods would clarify its novelty and advantages.

W1: Approach is different though, performance should be compared against related existing methods.

Based on the reviewer's feedback, we added comparisons using the same underlying forget facts and ten evaluation languages, reporting both forgetting and retention to make the trade-offs explicit. Appendix A.6 and Table 15 provide the configurations and results.

MethodForget-data languagesForget NLIRetain NLIMean UℓU_\ell
Standard FT + Multilingual NPOAll ten0.170.720.80
Standard FT + Multilingual LINGTEAAll ten0.630.890.28
Standard FT + MUTEAll ten0.550.740.35
LATTE + NPOEnglish0.160.880.83

LATTE achieves similar forget NLI to multilingual NPO (0.16 versus 0.17), with higher retain NLI (0.88 versus 0.72). Compared with multilingual LINGTEA, LATTE achieves lower forget NLI (0.16 versus 0.63), with similar retain NLI (0.88 versus 0.89). These results demonstrate strong multilingual forgetting using only English unlearning examples. We did not include a quantitative comparison with UNLEARN because the authors did not provide a code implementation.

W2: Auxiliary evaluations such as paraphrased queries, Truth Ratio/likelihood,

We broadened the analyses on Qwen2.5-3B-Instruct/TOFU after English-only NPO. These evaluations test whether LATTE's gains extend to unseen question wording and probability-based measures. The table reports averages over the nine non-English languages.

MetricStandard fine-tuningLATTE
Unseen-paraphrase forget reduction UℓU_\ell0.320.74
Increase in forget-answer negative log-likelihood0.142.88
Relative decrease in Truth Ratio score4.9%42.6%

Retain-answer negative log-likelihood increases by less than 0.03 for both models in every language. These results show that LATTE's forgetting gains extend to unseen question wording and probability-based evaluations beyond NLI and ROUGE-L. Appendix A.4, Tables 12 and 13 report the full results.

W2: recovery, or steering tests would be more convincing.

We also evaluated steering and retraining attacks after English-only NPO. The following table reports mean forget NLI across all ten languages. Lower values indicate stronger forgetting.

EvaluationStandard fine-tuningLATTE
After unlearning, without attack0.690.16
Steering attack0.330.20
Random-direction steering control0.400.19
Retraining attack0.660.49

Steering produces only a small additional increase in LATTE's mean forget NLI over the random-direction control under the tested strengths. Retraining recovers part of the forget performance in both models and a larger fraction of LATTE's initial reduction in most languages, showing that some of its additional forgetting remains reversible. Nevertheless, LATTE retains lower mean forget NLI after retraining. In seven languages, every LATTE run remains below every standard fine-tuning run. Appendix A.10 reports the attack settings, controls, and per-language results.

AuthorsOpenReview ↗

Official Comment by Authors

W3/Q1: TOFU is based on fictitious-author synthetic knowledge, and LUME uses only the synthetic-story task rather than the entire benchmark. It is unclear whether the same effect occurs with actual pretrained factual/private knowledge.

LATTE is designed to make knowledge acquired during multilingual fine-tuning easier to unlearn across languages. Its alignment intervention acts while that knowledge is learned, before future forget requests are known. Removing knowledge already encoded during pretraining is outside the setting LATTE targets. TOFU and LUME's synthetic-story task introduce facts during fine-tuning and therefore directly evaluate the setting our method is designed to address.

W3/Q1: Does the same trend hold for larger models?

In addition to the 3B models in the paper, we extended the evaluation to Qwen2.5-7B-Instruct on TOFU using English-only NPO. We report both forgetting and retention below, since the practical benefit depends on the trade-off achieved by each unlearning method.

MethodModelForget NLIRetain NLI
NPOStandard0.780.94
NPOLATTE0.500.92

LATTE lowers forget NLI in all ten languages. NPO reduces mean forget NLI by 0.28 while mean retain NLI decreases by 0.018. The NPO results demonstrate that LATTE's multilingual forgetting benefits extend to a 7B model with mean retain NLI of 0.92. Appendix A.7, Table 16 reports the full results.

If the above addresses the reviewer's concerns, we would be grateful if they could update their score accordingly.

AuthorsOpenReview ↗

Official Comment by Authors

As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. To address their comments, we have additionally:

  1. Compared LATTE directly with middle-layer alignment. On Qwen/TOFU, LATTE achieves stronger forgetting (0.16 versus 0.40) and higher retention (0.88 versus 0.84), supporting the value of its complete training objective (Appendix A.5, Table 14).

  2. Added multilingual NPO, multilingual LINGTEA, and MUTE comparisons. LATTE achieves similar forgetting to multilingual NPO with higher retention while using only English unlearning examples (Appendix A.6, Table 15).

  3. Broadened evaluation to unseen paraphrases, answer likelihood, and Truth Ratio. Mean non-English forget reduction on paraphrases increases from 0.32 to 0.74, and the relative Truth Ratio decrease reaches 42.6% versus 4.9% (Appendices A.3 and A.4).

  4. Added steering and retraining attacks. These show partial reversibility, while LATTE maintains lower mean forget NLI after retraining, at 0.49 versus 0.66 (Appendix A.10).

  5. Extended evaluation to Qwen2.5-7B-Instruct, where LATTE lowers forget NLI in all ten languages under NPO, and clarified the scope of knowledge acquired during fine-tuning (Appendix A.7, Table 16).

We believe these additions provide substantially stronger baseline comparisons and evidence beyond NLI and ROUGE-L. We would appreciate your reconsideration of the rating, including an increase if the new experiments address your concerns.

Reviewer_r79POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns the cross-lingual machine learning problem that the forgetting knowledge in one language does not necessarily remove the same knowledge in other languages. The authors observe that stronger cross-lingual representation alignment is associated with better unlearning transfer. And based on this observation, the authors propose LATTE, which is a training framework that aligns representations of parallel prompts across languages during fine-tuning. LATTE combines representation alignment with knowledge preservation objectives and allows existing single-language unlearning methods to be applied without modification. Experiments on multilingual TOFU and LUME using Llama-3.2-3B and Qwen2.5-3B demonstrate improved cross-lingual forgetting while generally preserving retained knowledge.

Critical strengths
  1. This paper studies an important and meaningful research problem in multilingual machine unlearning, and proposes a method that improves the cross-lingual forgetting with knowledge retained.

  2. The proposed method is simple and easy to implement. Considering the cross-lingual alignment during the fine-tuning stage is interesting and practical. The proposed method can be combined with existing unlearning frameworks without requiring forget examples at the unlearning stage.

  3. The authors evaluated two model families, across multiple languages, compared their method with several unlearning methods, and did ablation studies and representation analysis supporting the proposed method.

  4. The authors provide code in the supplementary materials, which is helpful for the reproducibility.

Critical weaknesses
  1. Novelty: The novelty of this paper is its application to cross-lingual unlearning, and the core idea of aligning middle-layer representations during fine-tuning has been explored in previous studies, such as Liu and Niehues (2025) [1]. To clarify the contribution of LATTE, a more direct comparison with existing alignment-based approaches would be helpful.

Reference: [1] Middle-layer representation alignment for cross-lingual transfer in fine-tuned LLMs. Danni Liu, Jan Niehues. ACL 2025.

  1. Experiments:

2.1 The experiments focus on two small 3B models and mainly on synthetic benchmarks. Evaluation on larger models and real unlearning tasks is needed to support the generalizability of the proposed method.

2.2 The experiment does not evaluate how well LATTE works when future forget examples are not included in the alignment data.

  1. Effectiveness: The improvements are not consistent across all evaluated unlearning tasks, and the alternative anchor-language results are largely weaker than those with English. Further discussion on these limitations would be helpful to clarify the practical usage of the proposed method.
Questions for authors
  1. How does the proposed method perform compared with alignment-based baselines?

  2. How does LATTE perform on larger models and real knowledge unlearning tasks?

  3. How well does LATTE perform when different forget subsets are outside the alignment pool?

  4. Discuss the limitation mentioned in [Critical Weaknesses 3] together with the additional training cost of LATTE.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for their careful assessment and constructive suggestions. We appreciate their recognition of the importance of cross-lingual unlearning, as well as LATTE's simplicity, practicality, and compatibility with existing unlearning frameworks. We also appreciate their positive assessment of our evaluation across model families and languages, the supporting ablation and representation analyses, and the code provided for reproducibility.

W1/Q1: The novelty of this paper is its application to cross-lingual unlearning, and the core idea of aligning middle-layer representations during fine-tuning has been explored in previous studies, such as Liu and Niehues (2025). To clarify the contribution of LATTE, a more direct comparison with existing alignment-based approaches would be helpful.

LATTE builds on the training-time alignment approach of Liu and Niehues (2025). Our contribution is to prepare multilingual factual knowledge for subsequent single-language unlearning before the forget set is known, combining prompt alignment with preservation of anchor-language representations and next-token predictions. To assess the contribution of this objective, we added a comparison with middle-layer representation alignment under the same multilingual training data and identical English-only NPO settings.

Training methodForget NLIRetain NLIMean TℓT_\ell
Generic middle-layer alignment0.400.840.80
LATTE0.160.880.91

LATTE achieves stronger forgetting, higher average retention, and improved mean normalized transfer. Our loss ablations in Table 5 show that removing either preservation term weakens both forgetting and retention across all five alignment targets. Together, these comparisons support the contribution of LATTE's complete objective to subsequent cross-lingual unlearning. The baseline comparison is reported in Appendix A.5, Table 14.

W2.1/Q2: The experiments focus on two small 3B models.

We extended the evaluation to Qwen2.5-7B-Instruct on TOFU using English-only NPO. Below, we report both forgetting and retention to make the trade-offs at this model scale explicit.

MethodModelForget NLIRetain NLI
NPOStandard0.780.94
NPOLATTE0.500.92

LATTE lowers forget NLI in all ten languages. NPO reduces mean forget NLI by 0.28 while mean retain NLI decreases by 0.02. The NPO results demonstrate that LATTE's multilingual forgetting benefits extend to a 7B model with mean retain NLI of 0.92. Appendix A.7, Table 16 reports the full results.

W2.1/Q2: How does LATTE perform on real knowledge unlearning tasks?

LATTE is designed to make knowledge acquired during multilingual fine-tuning easier to unlearn across languages. Its alignment intervention acts while that knowledge is learned, before future forget requests are known. Removing knowledge already encoded during pretraining is outside the setting LATTE targets. TOFU and LUME introduce new facts during fine-tuning and therefore directly evaluate this setting. Across both benchmarks, LATTE improves forgetting with English-only unlearning. The held-out author experiments further show that these gains extend to facts learned during fine-tuning but excluded from alignment (Appendix A.2, Table 10).

W2.2: The experiment does not evaluate how well LATTE works when future forget examples are not included in the alignment data.

Q3: How well does LATTE perform when different forget subsets are outside the alignment pool?

Based on the reviewer's suggestion, we evaluated three disjoint TOFU forget sets on Qwen2.5-3B-Instruct. These authors' facts were learned during multilingual fine-tuning but excluded from the alignment objective. After English-only NPO, LATTE improves forgetting and normalized transfer on all three sets.

Forget setModelForget NLIRetain NLIMean UℓU_\ellMean TℓT_\ell
Set 1Standard0.710.900.190.44
Set 1LATTE0.170.900.810.86
Set 2Standard0.690.900.250.61
Set 2LATTE0.200.890.790.87
Set 3Standard0.720.900.180.46
Set 3LATTE0.230.890.770.80

These results show that LATTE's benefits extend to facts never directly used in its alignment objective and hold across different forget subsets. Appendix A.2 describes the setup, and Table 10 reports the results.

AuthorsOpenReview ↗

Official Comment by Authors

W3: The improvements are not consistent across all evaluated unlearning tasks, and the alternative anchor-language results are largely weaker than those with English.

Q4: Discuss the limitation together with the additional training cost of LATTE.

LATTE requires 8.51 GPU-hours compared with 5.55 for standard fine-tuning on one NVIDIA A100, a 1.53 cost with matched data and language-model updates. The benefit of this additional training depends on the unlearning objective and anchor language. On Qwen2.5-3B-Instruct/TOFU, English-anchored NPO, GradDiff, and LINGTEA* improve forgetting in all ten languages. The following NPO results show how the gains vary across anchors, averaging NLI over all ten languages.

AnchorModelForget NLI
EnglishStandard0.69
EnglishLATTE0.16
ChineseStandard0.72
ChineseLATTE0.68
GermanStandard0.76
GermanLATTE0.72

English-anchored NPO provides the clearest benefit for the additional training, substantially improving forgetting. The smaller gains with Chinese and German show that the benefit is configuration-dependent. The additional computation occurs during fine-tuning and is not repeated for each unlearning request. Subsequent unlearning uses the baseline objective and update budget, and merging the adapters introduces no inference overhead.

If the clarifications and additional experiments have addressed the reviewer's comments, we would be grateful if they could update their score accordingly.

AuthorsOpenReview ↗

Official Comment by Authors

As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. To address their comments, we have additionally:

  1. Added a direct comparison with middle-layer alignment on Qwen/TOFU. LATTE reduces mean forget NLI from 0.40 to 0.16 while increasing retain NLI from 0.84 to 0.88, supporting the contribution of its combined alignment and preservation objective (Appendix A.5, Table 14).

  2. Extended evaluation to Qwen2.5-7B-Instruct. Under English-only NPO, LATTE improves forgetting in all ten languages, reducing mean forget NLI from 0.78 to 0.50, with mean retain NLI of 0.92 (Appendix A.7, Table 16).

  3. Tested three disjoint forget sets whose facts were learned during fine-tuning but excluded from alignment. LATTE improves forgetting and normalized transfer on all three, achieving transfer of 0.80–0.87 versus 0.44–0.61 (Appendix A.3, Table 11).

  4. Quantified the 1.53× fine-tuning cost and discussed its benefits alongside method and anchor dependence. We also clarified the scope of knowledge acquired during fine-tuning. The additional training is reusable across future unlearning requests and introduces no inference overhead.

We believe these comparisons and generalization experiments strengthen LATTE’s contribution and clarify its practical scope. If they address your concerns, we would be grateful if you could consider increasing your rating.

Reviewer_pNAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LATTE, a training-time intervention for improving cross-lingual unlearning transfer in multilingual LLMs. Motivated by the correlation between cross-lingual prompt retrieval and unlearning transfer, LATTE alternates between multilingual fine-tuning and representation alignment using rank-8 LoRA adapters. The adapters are merged into the base model before applying existing unlearning methods, requiring only English forget and retain examples at unlearning time. Experiments on TOFU and LUME, using Llama-3.2-3B and Qwen2.5-3B across up to ten languages, show substantial improvements with NPO, GradDiff, and LINGTEA. The paper presents an interesting approach, but several questions remain regarding the proposed mechanism, evaluation methodology, and practical advantages over multilingual unlearning baselines.

Critical strengths
  1. LATTE does not require modifying existing unlearning algorithms, and its merged adapters introduce no additional inference overhead. The evaluation across different models, benchmarks, and unlearning methods provides encouraging empirical evidence.

  2. Removing the alignment objective substantially degrades forgetting performance, while the LSAR analysis shows improved cross-language retrieval. These experiments provide useful support for the proposed approach.

Critical weaknesses
  1. The paper motivates LATTE using the normalized transfer metric Tℓ=Uℓ/UenT_\ell=U_\ell/U_{\mathrm{en}}, but the main results primarily report absolute forget NLI. This is important because LATTE also substantially improves English forgetting. Consequently, improvements in non-English forgetting do not necessarily imply improved relative transfer. Moreover, several non-target languages also benefit substantially, while languages with nearly saturated retrieval accuracy still exhibit different forgetting performance. This suggests that representation alignment alone may not explain the results. The retrieval evaluation also uses prompts included in fine-tuning, leaving its generalization to unseen prompts unclear.

  2. Retention is not consistently maintained. For example, English retain NLI on Qwen/TOFU decreases from 0.97 to 0.81 under GradDiff and from 0.98 to 0.80 under LINGTEA. Thus, the reported forgetting improvements may involve a shift in the forgetting–retention trade-off.

  3. Minor Presentation & Bibliographic Issues: There are a few minor formatting and citation inconsistencies that can be easily fixed in a revision: Inconsistent In-Text Citations: The RMU paper is cited as (N. Li et al., 2024) on pages 1–3, but as (Li et al., 2024) on pages 7 and 16. Bibliography Updates: Several references listed as arXiv preprints have been formally published in conference proceedings (e.g., Brinkmann et al., 2025 in NAACL 2025; Dehghanpour Farashah et al., 2026 in EACL 2026).

Questions for authors
  1. Could the authors report Tℓ for both standard fine-tuning and LATTE? Since English forgetting also improves substantially, absolute forget NLI alone cannot establish improved transfer. How should Tℓ be interpreted when English forgetting approaches saturation?

  2. Does the proposed alignment mechanism generalize to unseen forget examples?

  3. Could the authors compare LATTE against standard fine-tuning followed by unlearning with translated forget examples, as well as full multilingual configurations of methods such as LINGTEA, MUTE, and UNLEARN?

  4. Have the authors tried varying the number of unlearning steps or the learning rate to plot forgetting–retention Pareto curves? Have they also tested how unlearning affects the model’s general capabilities? Please report standard deviations or confidence intervals and clarify how checkpoints were selected and which random seeds were used.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for their careful assessment and constructive suggestions. We appreciate their recognition of LATTE's compatibility with existing unlearning algorithms and lack of additional inference overhead, as well as their description of our evaluation. We also appreciate their positive assessment of the ablation and retrieval analyses as providing useful support for our approach.

W1: Consequently, improvements in non-English forgetting do not necessarily imply improved relative transfer.

Q1: Could the authors report TℓT_\ell for both standard fine-tuning and LATTE? How should TℓT_\ell be interpreted when English forgetting approaches saturation?

To address this concern, we report TℓT_\ell for both standard fine-tuning and LATTE, together with the proportional forget NLI reductions UℓU_\ell and UenU_{\mathrm{en}}:

Tℓ=UℓUenT_\ell = \frac{U_\ell}{U_{\mathrm{en}}}

MetricStandard fine-tuningLATTE
UenU_{\mathrm{en}}0.500.90
Mean non-English UℓU_\ell0.350.82
Mean non-English TℓT_\ell0.690.91

When Uen→1U_{\mathrm{en}} \to 1, Tℓ→UℓT_\ell \to U_\ell. Near English saturation, Tℓ=0.80T_\ell = 0.80 therefore corresponds to approximately an 80% reduction in the target language's forget NLI. English saturation does not inflate TℓT_\ell. We added the comparison in Section 5.2, Table 4.

W1: Moreover, several non-target languages also benefit substantially, while languages with nearly saturated retrieval accuracy still exhibit different forgetting performance. This suggests that representation alignment alone may not explain the results.

LATTE combines alignment with knowledge preservation, and our ablations in Section 5.3 support contributions from both. Removing alignment alone raises Korean forget NLI from 0.15 to 0.84. Removing either preservation term also weakens forgetting and retention. Moreover, alignment improves in non-target languages too, with German retrieval accuracy increasing from 0.906 to 0.984, consistent with updates affecting shared parameters. Near-saturated top-1 retrieval indicates that the correct translation ranks first, but does not imply identical representations or forgetting outcomes. These results support alignment's contribution within LATTE's combined objective. We clarified this interpretation in Section 5.4.

W1: The retrieval evaluation also uses prompts included in fine-tuning, leaving its generalization to unseen prompts unclear.

To test whether the alignment gains extend beyond the training prompts, we repeated the evaluation on 217 TOFU real-author and world-fact questions excluded from both fine-tuning and alignment.

ModelHITHZHJAKODEESFRVIMean
Standard fine-tuning11.143.396.880.665.092.695.997.777.973.4
LATTE88.592.297.795.487.695.499.197.792.694.0

LATTE increases mean retrieval accuracy from 73.4% to 94.0%, improving retrieval in eight languages and matching standard fine-tuning in French. These results show that the alignment gains extend beyond the training prompts. We added these results in Appendix A.3, Table 10.

Q2: Does the proposed alignment mechanism generalize to unseen forget examples?

Based on the reviewer's feedback, we have added experiments to examine generalization beyond the examples used for alignment. On questions about authors whose facts were learned during fine-tuning but excluded from alignment, LATTE increases mean retrieval accuracy from 75.2% to 99.4%. We also evaluated English-only NPO on unseen paraphrases and three author sets excluded from alignment. The table reports forget and retain NLI after unlearning.

EvaluationModelForget NLI ↓\downarrowRetain NLI ↑\uparrowMean UℓU_\ell ↑\uparrowMean TℓT_\ell ↑\uparrow
Unseen paraphrasesStandard0.340.900.320.81
Unseen paraphrasesLATTE0.140.880.740.81
Author set 1Standard0.710.900.190.44
Author set 1LATTE0.170.900.810.86
Author set 2Standard0.690.900.250.61
Author set 2LATTE0.200.890.790.87
Author set 3Standard0.720.900.180.46
Author set 3LATTE0.230.890.770.80

Forget NLI, UℓU_\ell, and TℓT_\ell are averaged over the nine non-English languages. Retain NLI uses the original retain questions across all ten languages, with pre-unlearning means of 0.92 for standard fine-tuning and 0.94 for LATTE.

LATTE achieves stronger forgetting on unseen paraphrases with similar normalized transfer, and improves both forgetting and normalized transfer on all three author sets. These results show that the alignment and unlearning gains extend beyond the prompts and facts used for alignment. The experimental details and full results are reported in Appendix A.3 and Table 11.

AuthorsOpenReview ↗

Official Comment by Authors

W2: Retention is not consistently maintained. For example, English retain NLI on Qwen/TOFU decreases from 0.97 to 0.81 under GradDiff and from 0.98 to 0.80 under LINGTEA. Thus, the reported forgetting improvements may involve a shift in the forgetting–retention trade-off.

LATTE's forgetting gains extend across methods and languages. On Qwen/TOFU, it lowers forget NLI in all ten languages under NPO, GradDiff, and LINGTEA*. Averaged across languages, forget NLI decreases by 0.44–0.53, while retain NLI decreases by 0.02–0.06. On Qwen/LUME, retain NLI improves in nine of ten languages under each method, alongside forgetting improvements in nine or ten languages. These results demonstrate LATTE's benefits across multiple unlearning methods, languages, and benchmarks.

W3: Inconsistent In-Text Citations: The RMU paper is cited as "(N. Li et al., 2024)" on pages 1–3, but as "(Li et al., 2024)" on pages 7 and 16.

W3: Bibliography Updates: Several references listed as arXiv preprints have been formally published in conference proceedings (e.g., Brinkmann et al., 2025 in NAACL 2025; Dehghanpour Farashah et al., 2026 in EACL 2026).

We corrected the RMU citation throughout the manuscript and updated Brinkmann et al. (2025) and Dehghanpour Farashah et al. (2026) to their NAACL 2025 and EACL 2026 proceedings versions, respectively.

Q3: Could the authors compare LATTE against standard fine-tuning followed by unlearning with translated forget examples, as well as full multilingual configurations of methods such as LINGTEA, MUTE, and UNLEARN?

Based on the reviewer's feedback, we have added comparisons on Qwen/TOFU using the same underlying forget facts and ten evaluation languages, with all baselines starting from standard fine-tuning. Appendix A.6 and Table 15 provide the configurations and results.

MethodForget-data languagesForget NLI ↓\downarrowRetain NLI ↑\uparrowMean UℓU_\ell ↑\uparrow
Standard FT + Multilingual NPOAll ten0.170.720.80
Standard FT + Multilingual LINGTEAAll ten0.630.890.28
Standard FT + MUTEAll ten0.550.740.35
LATTE + NPOEnglish0.160.880.83

LATTE achieves similar forget NLI to multilingual NPO (0.16 versus 0.17), with higher retain NLI (0.88 versus 0.72). Compared with multilingual LINGTEA, LATTE achieves lower forget NLI (0.16 versus 0.63), with similar retain NLI (0.88 versus 0.89). These results demonstrate strong multilingual forgetting using only English unlearning examples. We did not include a quantitative comparison with UNLEARN because the authors did not provide a code implementation.

Q4: Have the authors tried varying the number of unlearning steps or the learning rate to plot forgetting–retention Pareto curves?

We varied both the learning rate and the number of NPO steps on Qwen/TOFU and added the resulting per-language curves. At a learning rate of 1×10−51 \times 10^{-5}, LATTE after 50 steps achieves comparable or higher relative forget reduction across all ten languages than standard fine-tuning after 100 steps, while the mean retain NLI drop is approximately 0.07 rather than 0.15. This provides a concrete configuration with stronger multilingual forgetting and a smaller retention decrease. Section A.8 and Figures 2 and 3 present the per-language forgetting–retention curves and the experimental settings.

Q4: Please report standard deviations or confidence intervals and clarify how checkpoints were selected and which random seeds were used.

We now report means and sample standard deviations across five unlearning runs using seeds 0, 1, 2, 3, and 4 for the main experiments. We evaluate the final checkpoint after each method's specified unlearning budget, using identical settings for standard fine-tuning and LATTE within each comparison.

Q4: Have the authors tested how unlearning affects general capabilities?

We evaluated a fixed MMLU/MMMLU subset of 2,850 questions per language across eight languages. After English-only NPO, mean accuracy changes from 52.2% to 51.9% for standard fine-tuning and from 50.9% to 51.6% for LATTE. Both fine-tuned models score below the original Instruct model at 57.5%, so the capability decrease occurs primarily during fine-tuning. LATTE's subsequent unlearning introduces no additional mean accuracy loss on this evaluation. Appendix A.9 reports the setup and per-language results.

If the clarifications have addressed the reviewer's concerns, we would be grateful if they could update their score accordingly.

AuthorsOpenReview ↗

Official Comment by Authors

As the rebuttal period is coming to an end, we would like to thank the Reviewer once again for their constructive feedback. To address their comments, we have additionally:

  1. Reported English forgetting alongside normalized transfer. On Qwen/TOFU, LATTE increases mean non-English transfer from 0.69 to 0.91 (Section 5.2, Table 4).

  2. Tested unseen prompts and three forget sets excluded from alignment. LATTE improves forgetting on all three sets, while retrieval on questions excluded from both fine-tuning and alignment increases from 73.4% to 94.0% (Appendix A.3).

  3. Added multilingual NPO, multilingual LINGTEA, and MUTE comparisons. LATTE achieves similar forgetting to multilingual NPO (0.16 versus 0.17) with higher retention (0.88 versus 0.72), using only English unlearning examples (Appendix A.6, Table 15).

  4. Added learning-rate and budget sweeps. LATTE after 50 NPO steps achieves comparable or higher forget reduction than standard fine-tuning after 100 steps in all ten languages, with a mean retain NLI decrease of 0.07 versus 0.15 (Appendix A.8).

  5. Evaluated general capabilities on MMLU/MMMLU across eight languages, showing no additional mean accuracy loss from LATTE’s subsequent unlearning in this evaluation (Appendix A.9). We also reported standard deviations and clarified checkpoint selection.

We believe these additions substantially strengthen the evidence addressing your concerns. We would be grateful if you could review the new results and consider increasing your rating if your concerns have been addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their constructive feedback. We appreciate their recognition of the importance and practical relevance of cross-lingual unlearning (r79P, Aj6q), LATTE’s compatibility with existing methods and lack of inference overhead (pNAX), and the supporting ablation and representation analyses. The experiments we ran to address the reviewers’ comments, and experiments we ran beyond are summarized below. All changes in the revised paper are marked in blue.

Rebuttal Highlights based on the Reviewers’ comments

  • Alignment and multilingual baselines. Following the requests of r79P and Aj6q, we compared LATTE with middle-layer alignment. LATTE reduces forget NLI from 0.40 to 0.16 while increasing retain NLI from 0.84 to 0.88. For pNAX and Aj6q, we also evaluated multilingual NPO, multilingual LINGTEA, and MUTE. LATTE achieves similar forgetting to multilingual NPO (0.16 versus 0.17) with higher retention (0.88 versus 0.72), using only English unlearning data. We present these results in Appendices A.5 and A.6, Tables 14 and 15.

  • Relative transfer and forgetting–retention trade-offs. To address pNAX’s concern about stronger English forgetting, we report normalized transfer, which increases from 0.69 to 0.91. Learning-rate and budget sweeps also show that LATTE after 50 NPO steps achieves comparable or higher forget reduction in all ten languages than standard fine-tuning after 100 steps, with a mean retain NLI decrease of 0.07 versus 0.15. Section 5.2, Table 4 reports the transfer results, while Appendix A.8, Figures 2 and 3 shows the forgetting–retention curves.

  • Generalization beyond alignment data. As requested by pNAX and r79P, we tested three forget sets learned during fine-tuning but excluded from alignment. LATTE improves forgetting and normalized transfer on all three, achieving transfer of 0.80–0.87 versus 0.44–0.61. Retrieval accuracy on questions excluded from both fine-tuning and alignment increases from 73.4% to 94.0%. The evaluation setup and results appear in Appendix A.3.

  • Evaluations beyond NLI and ROUGE-L. Following Aj6q’s request, we evaluated paraphrases, answer likelihood, Truth Ratio, steering, and retraining attacks. Mean non-English forget reduction on unseen paraphrases increases from 0.32 to 0.74, and the relative Truth Ratio decrease reaches 42.6% versus 4.9%. Retraining recovers a larger fraction of LATTE’s initial reduction, but LATTE retains lower mean forget NLI after the attack, at 0.49 versus 0.66. We detail the likelihood and Truth Ratio results in Appendix A.4, Tables 12 and 13, and the attack evaluations in Appendix A.10.

  • Larger models and general capabilities. Responding to r79P and Aj6q, we evaluated Qwen2.5-7B-Instruct. LATTE improves NPO forgetting in all ten languages, reducing mean forget NLI from 0.78 to 0.50, with retain NLI of 0.92 versus 0.94. For pNAX’s capability question, MMMLU accuracy across eight languages changes from 50.9% before unlearning to 51.6% afterward, showing no additional mean accuracy loss in this evaluation. The 7B-model results are reported in Appendix A.7, Table 16, and the capability evaluation in Appendix A.9.

  • Training cost and scope. For r79P’s cost question, LATTE requires 8.51 versus 5.55 GPU-hours, a 1.53× cost on the same A100. This cost occurs during fine-tuning and is not repeated for future unlearning requests. We clarified method and anchor dependence, the scope of knowledge acquired during fine-tuning, and the reporting of variability and checkpoint selection. Merging the adapters preserves the original architecture and introduces no inference overhead.

Additional extensions of the paper

To further strengthen our submission, beyond the requested experiments, we also made the following additions.

  • Alignment layer. We tested alignment depth. Mean target-language transfer ranges from 0.62 to 0.85 across layers 15 to 27, compared with 0.33 for the standard fine-tuning control. Layer 18 achieves the highest reported target and non-target transfer, supporting the default choice while showing gains at several other depths. Appendix A.12 reports the full comparison.

  • Alignment frequency. We also tested the update interval. Updating adapters every second base-model step achieves target-language transfer of 0.85, compared with 0.42 every step and 0.60 every fourth step. The default therefore achieves stronger transfer than more frequent alignment while using half as many adapter updates. Appendix A.13 presents the results supporting this choice.

These additions strengthen the evidence for preparing multilingual knowledge for future English-only unlearning while retaining compatibility with existing methods. We thank the reviewers again for helping us sharpen LATTE’s contribution and its empirical support, and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.