acceptodds
Under review as a conference paper at ICLR 2027

MAPP: Cross-Lingual Transfer through Representation Alignment with Performance-Gap Probing

Abstract

Cross-lingual representation alignment — training a Large Language Model (LLM) to encode semantically equivalent inputs similarly across languages — is increasingly used to improve LLM performance on low-resource languages, motivated by the observation that a language's performance gap to English correlates with the distance between their hidden states, particularly at middle layers. State-of-the-art alignment methods optimize distance functions or contrastive losses selected a priori, all of which are minimized in the limit of complete alignment. We argue that these objectives are only proxies for optimal cross-lingual transfer learning by showing that complete alignment both eliminates language identity and yields sub-optimal performance gains. By training oracle models and examining their multilingual representation geometry, we establish the performance gap and the language gap as separable components of cross-lingual representation difference. We therefore propose MAPP (Multilingual Alignment through Performance-gap Probing), which trains probes on middle-layer hidden representations to predict cross-lingual performance gaps and uses them as reward models to align the LLM. Across three tasks, MAPP yields larger performance gains than state-of-the-art alignment signals while preserving output language identity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multilingual language models, cross-lingual transfer, representation alignment, low-resource languages, probing, reward modeling, sparse autoencoders, activation steering, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i7dYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the relationship between cross-lingual representation differences and cross-lingual performance differences in multilingual LLMs. The authors train probes that predict performance gaps between languages from internal representations and then use these probes as reward models to improve cross-lingual alignment.

I find the setup and the core idea very appealing. The paper itself is at times extremely hard to follow because of very dense writing and presentational issues. In many places I had to read sentences five to ten times to extract their meaning. In its current form, the presentation falls short of the level required for publication.

Critical strengths
  • The paper addresses an important problem, the improvement of multilingual representations, and proposes a clever idea: using a performance probe as a reward signal for alignment. This idea is the main reason I see potential in the work.
  • The experimentation is thorough, which gives the empirical claims a solid basis.
Critical weaknesses
  • Writing and clarity (major). The presentation significantly hinders understanding of the paper.
    • Several passages are very dense and hard to parse, for example lines 054-060 and 067-070. They read like the product of AI rewriting. The writing could be much clearer and more concise.
    • The purpose of Section 3.1 is unclear to me. I assume it is meant to show that representational difference alone is insufficient to predict performance. Measuring representational distance via a pivot language seems unmotivated, and the intuition behind this choice is missing.
    • I was unable to decipher Figure 4.
  • Motivation for the probe architectures (major). The reasoning behind the two complex probe architectures remains unclear. A linear or bilinear classifier on the concatenated reference and target representations seems like the natural baseline. The use of SAEs is also unmotivated: SAEs are lossy approximations of the residual stream, so probing the residual stream directly seems preferable.
  • Evaluation (minor): The evaluation is restricted to classification tasks. I understand that task availability is limited for low-resource languages, but machine translation on FLORES would have been a feasible addition.
Questions for authors

What is the intuition behind measuring representational distance via a pivot language in Section 3.1, and what role does this section play in the overall argument? Why are SAE features used as probe inputs instead of the residual stream directly? How do the proposed probes compare to a simple linear or bilinear probe on the concatenated reference and target representations? What does Figure 4 show, and how should it be read?

2 comments

Several passages are very dense and hard to parse, for example lines 054–060 and 067–070 of the original submission.

We apologize for the imperfect presentation. We devoted about a year to the research and left only a week for writing. We should have allowed more time. We have refined the wording throughout the paper and rewritten the introduction and the transition from the steering study to MAPP. We hope the revised manuscript reads better. See top-level response.

What is the intuition behind measuring representational distance via a pivot language in Section 3.1, and what role does this section play in the overall argument?

The original section 3.1 was addressing the claim made by prior work that "representation distance between a low resource language and English predicts downstream performance in that language". Liu and Niehues (2025, Fig. 1b) measured alignment by translation retrieval and found that it correlated with downstream slot-filling F1 across languages. Through experiments, we found that that is only true on average. Across individual queries, there's large variance and the actual query-level correlation is weak. These findings motivated us to develop MAPP, which aims at learning an alignment signal that actually predicts performance gap, rather than relying on raw representation distance.

But we do realize that it is loosely related to the central argument, so we've revised it to focus on the representation distribution study, where we show that representations cluster by language, and that representation distance is largely language gaps with pretty small performance gaps (§3.1, Figs. 1 and 2a, lines 204–214 and 227–229). These results show that it is thus improper to use those distances for alignment directly, and motivates a language-agnostic, performance-based signal.

What does Figure 4 show, and how should it be read?

We apologize for putting the figures in the wrong order. The probe architecture diagram is Figure 5 in the revised manuscript, and §4.2 explains its two panels. On the left, each target token attends to the reference tokens before the probe predicts a performance gap. On the right, the SAE encodes both sentences and scores the pair by the distance between their pooled codes (Fig. 5, lines 395–399). Figure 4 now introduces the training procedure before we describe either architecture (lines 310–313).

Why are SAE features used as probe inputs instead of the residual stream directly?

We chose an SAE because prior work found that some sparse features are voluntarily language-agnostic, making them a good fit to signify cross-lingual content matching. Some prior work had used it to align the LLM (Sin Jing Xuan et al., 2025). We further show that it can do better if we finetune the SAE with a deliberate language-agnostic loss term. (§4.2, lines 360–377 and 410–412, Fig. 5, lines 395–399).

How do the proposed probes compare to a simple linear or bilinear probe on the concatenated reference and target representations?

We report a linear probe for comparison in the ablation section, alongside other ablation studies to demonstrate the selected, complex probe structure is necessary (Appendix C, Table 4, lines 847–855). On six-language MASSIVE, the mean-pooled MLP scored 54.99 slot F1, compared with 56.99 for the full cross-attention probe. Giving every reference token the same attention weight reduced F1 to 55.39, suggesting that learning which tokens to attend to improves downstream slot filling.

The choice of nonlinear probes is because prior work had established that some cross-lingual information in LLM representations cannot be recovered by a linear readout. Gonen et al. (2020), for example, found that nonlinear methods could recover substantially more word-translation information from mBERT than linear methods could. If language and task-relevant content are entangled in this way, a linear probe will miss information that could help predict the performance gap (§1, lines 087–092).

Machine translation on FLORES would have been a feasible addition.

Thank you for suggesting a machine translation evaluation on FLORES. We did not include this workflow (finetuning on machine translated queries) as a baseline due to the limit of time. We do find it an interesting baseline that is widely used for middle to high resource languages where high-quality MT systems are available. However, we find the machine translated data for our tested low-resource languages a bit unsatisfying and prone to translation errors, and were not able to establish a good instantiation of this baseline for comparison. We will consider implementing this baseline as future work.

Thanks a lot for the extensive answer! I see the paper has been heavily rewritten and most of my weaknesses have been addressed (I still don't agree with the SAE-centric probes; I think the residual would make sense to include as a baseline at least), so I'm raising my score.

Reviewer_cvyWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-lingual transfer when task labels in the target language are unavailable. It argues that the difference between two languages' representations combines a performance gap that should be reduced with a language gap that should be preserved. To construct supervision, the authors train models with sufficient target-language task supervision and use frozen language models to force-decode the same pivot-language translation from reference- and target-language inputs. The resulting normalized log-likelihood difference is used as a proxy for the performance gap. They fit either an attention probe or a sparse autoencoder (SAE) probe to this signal. During alignment, the probe is frozen; the LLM alternates reference-language task SFT with alignment on parallel sentences, using a loss that penalizes negative predicted performance gaps and includes a KL penalty to the initial SFT model, with gradients backpropagated through the target-language branch. The probe is removed at inference, so the trained LLM is used directly. Experiments use Llama-3.1-8B-Instruct on ARC-Challenge, MASSIVE, and HellaSwag, with English-only or six high-resource supervised languages. The best MAPP variants improve target-language averages over SFT by up to 7.0 points in the English-only setting, and MAPP-attn preserves target-language output better than InfoNCE.

Critical strengths
  1. The failure-mode analysis is concrete. Sections 3.2–3.3 and Figures 2–3 distinguish the oracle-defined performance gap from the language gap and intervene on them separately. Steering along the performance gap raises accuracy from 32.2% to 40.0% while leaving target-language output largely unchanged, whereas steering along the combined gap produces a smaller gain and more off-language output. This supports the motivation that not all representation differences should be removed.

  2. The end-to-end evaluation covers multiple tasks and exposes trade-offs across variants. Under both English-only and six-language supervision, MAPP variants obtain higher target-language averages than the listed distance baselines on ARC-Challenge, MASSIVE, and HellaSwag.

Critical weaknesses
  1. The translation-likelihood proxy is not shown to measure the downstream performance gap. Equation (1) defines g from the relative likelihood of a shared pivot-language translation, and Equations (3) and (7) train and optimize a probe for this quantity. The held-out ρ = 0.614 in Figure 1 only validates prediction in that translation setting; it does not show that the probe remains valid after alignment or tracks ARC-Challenge accuracy and MASSIVE slot F1. The SAE variant uses g only to reweight training pairs, with no constant- or shuffled-g control. Thus, the gains in Table 1 may reflect general alignment regularization rather than performance-gap disentanglement. The paper should either narrow this mechanistic claim or measure proxy–task agreement before and after alignment.

  2. The experiments do not isolate performance-gap supervision from the training procedure. Prior work [1] already uses alternating alignment, and [2] uses SAE-based activation alignment, so the novelty should be attributed to the performance-gap signal. Yet MAPP uses one alignment epoch followed by two task epochs per round, whereas InfoNCE uses 1,000 contrastive and 1,000 task micro-batches per epoch; matched task exposure is not shown. SAE-L2 shares MAPP's schedule, but MAPP additionally trains a probe and uses three-way translation supervision. The reported gains over SAE-L2 are small in some settings, and Table 3 reports one seed without a continued-SFT control. The results, therefore, do not separate the effects of the new labels, probe capacity, schedule, and budget.

[1] Danni Liu and Jan Niehues. Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs. ACL 2025. ACL Anthology.

[2] Richmond Sin Jing Xuan, Jalil Huseynov, and Yang Zhang. Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders. arXiv:2507.18918, 2025. arXiv.

Questions for authors
  1. Before and after each alignment round, can you report the frozen probe's predictions and the re-measured g on the same pairs, together with their relationship to the actual downstream cross-lingual performance gap? Are there samples or languages for which probe loss decreases while the task gap does not?

  2. Under the same data and training budget, what happens when the SAE g-weights are replaced by constant weights or shuffled weights? What happens when the translation performance-gap labels used for the attention probe are shuffled? These controls would distinguish performance-gap supervision from a general representation regularizer.

  3. Can you match task tokens, alignment tokens, optimizer updates, and the KL term across MAPP, InfoNCE, and SAE-L2, and add a continued-SFT-without-alignment control at the same budget? Please report variation across training seeds, especially for the small gains over SAE-L2.

1 comment

Does the translation proxy still predict performance after alignment? Can a lower probe loss coincide with a downstream task gap that stays the same or grows?**

Yes, the frozen probe still predicts the measured translation gap after further LLM training. We trained it on general-domain translation triples and reported its predicted performance gap on 180 bilingual queries from the LLM alignment data at an early and late checkpoint during LLM alignment. At epoch 2 its predictions correlate with the measured gaps at r = 0.52. At epoch 5, after three further epochs of updates to the LLM and none to the probe, the correlation is 0.50. Mean-shifted L₂ distance (a baseline predictor of performance gap suggested by another reviewer) gives −0.07 and −0.06 at those checkpoints (Appendix H, lines 1298–1302 and 1307–1316, Fig. 12, lines 1346–1348). See top-level response.

Our choice of machine translation data for training the probe is motivated by prior work that found a strong correlation between LLM translation ability and LLM performance on downstream reasoning tasks in that language (Issaka et al. (2026)). The benefit of using these translation data is that the output labels for different languages are the same (a shared output language), therefore the performance gap is directly measurable (§4.1, lines 324–329).

What happens with constant or shuffled SAE weights, or shuffled attention-probe labels?**

We added the shuffled-label ablation to isolate performance-gap supervision from the attention architecture. For this ablation, we kept the probe architecture and LLM alignment procedure fixed and shuffled which measured translation gap was assigned to each probe-training example. Six-language MASSIVE slot F1 falls from 56.99 to 53.23. The model sees the same set of labels in both runs, so the 3.76-point drop measures the contribution of pairing each example with its own gap (Appendix C, Table 4, lines 847–849 and 854–855). Also see our top-level response.

Reviewer_sugnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces MAPP to improve cross-lingual transfer by relying on representation alignment that is guided by predicted performance differences. The paper argues that aligning representations using distance-based objectives removes language information. Instead, they attempt to separate task performance from language information and align based on task performance. The authors use two probe architectures for this purpose and show improvement on MASSIVE, ARC, and HellaSwag.

Critical strengths

The paper addresses an important limitation of representation alignment where minimizing the distance between parallel inputs in different languages may remove language-specific information necessary for correct generation/language fidelity. The steering experiments show that intervening on different representation directions can affect downstream accuracy and output language differently. The proposed performance-aware objective also shows improvements over the evaluated baselines, making the approach potentially useful despite the limitations discussed below.

Critical weaknesses
  1. The central argument in this paper is that cross-lingual representation differences can be decomposed into separable language-related and performance-related components. In Section 3.2, the performance gap is defined as the difference between base-model and task-fine-tuned oracle representations. In practice though, these differences can reflect other effects of fine-tuning, like task adaptation, knowledge retrieval, and language-selection behavior. Therefore, the theoretical decomposition does not establish that these components are actually independent. More importantly, the proposed probe is trained to predict differences in translation log-likelihood. This can reflect translation/linguistic abilities instead of shared knowledge or reasoning abilities. Additionally, the authors describe the multilingual probe trained across languages as language-agnostic, but do not provide evidence that its predictions are independent of language. Related work has also studied language and task seperation using factual knowledge retrieval and mathematical reasoning (Bandarkar et al., 2024; Bandarkar and Peng, 2025; Rahmati et al., 2025; Zhao et al., 2025). These papers show the importance of evaluating task performance and language adherence separately. In contrast, MAPP relies on translation-based supervision and evaluates primarily on multiple-choice tasks and slot filling. Although ARC-C tests knowledge and reasoning, its multiple-choice format does not show whether the model can retrieve and generate the correct information in the correct language in open-ended generation. As a result, the theory and experiments do not properly establish that the learned probe isolates transferable task performance from language-specific information, as claimed.

  2. The paper proposes a rather complex alignment method that requires learned probes and alternating optimization, but does not show whether this complexity is really necessary. Section 3.1 shows that subtracting a mean language offset improves the correlation between representation distance and performance differences. Relying on this, a simpler training method could be to subtract the language offset before alignment. Related work has also examined representation projection and language-specific component ablation (Xu et al., 2023; Zhao et al., 2025). The evaluation is also limited to one model, Llama-3.1-8B-Instruct, and one seed. Additionally, the advantage of the proposed method has not been sufficiently established. The authors motivate MAPP by the absence of target-language task data, but such data can often be generated by translating existing high-resource-language training examples, especially for tasks involving factual knowledge, science, or general reasoning. Table 1 shows that target-language SFT substantially outperforms MAPP, especially on MASSIVE (75.8 vs. 46.6 F1). Although the target-language SFT uses a different computation budget and does not provide a controlled comparison, the results raise an important question about the practical value of the proposed method. The paper does not compare MAPP’s data requirements, computational costs, or downstream performance against machine-translated task supervision with comparable budgets. In result, it is unclear what MAPP’s advantages are over this simpler, and potentially more effective approach. Furthermore, language-preservation improvements are not consistent. For example, on ARC-C, MAPP-SAE achieves only 70.0% target-language token retention, compared with 83.7% for the simpler SAE-L2 baseline suggesting that language preservation is not necessarily a general property of the proposed approach.

Bandarkar et al., 2024, Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models

Bandarkar et al., 2025, The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs

Rahmati et al., 2025, CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMs

Xu et al., 2023, Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?

Zhao et al., 2025, When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners

Questions for authors

My main questions and concerns are the same as weaknesses:

  1. What is the justification for using translation-based performance gaps as a measure of transferable task performance?
  2. What evidence supports the separation of task-related and language-related information, both theoretically and empirically?
  3. What evidence supports characterizing the multilingual probe as language-agnostic?
  4. How does MAPP compare against simpler, potentially competitive baselines, like removing the mean language offset before alignment?
  5. Are the results consistent across different models and in open-ended factual knowledge or reasoning tasks, where both answer correctness and language fidelity can be evaluated?
  6. How does MAPP compare with SFT on machine-translated task data under comparable data and computational budgets?
2 comments

What is the justification for using translation-based performance gaps as a measure of transferable task performance?

We added an experiment showing that a probe trained on general-domain translation triples from FLORES-200 and TED2020 predicts measured performance gaps on the bilingual queries actually used to align the LLM (§5.1, lines 473–476). Across 180 alignment queries, the frozen probe reaches r = 0.52 at epoch 2 and still reaches r = 0.50 after three more epochs of LLM training. Mean-shifted representation distance gives r = −0.07 and −0.06 on those same queries (Appendix H, lines 1307–1316, Fig. 12, lines 1346–1348). See top-level response.

Our choice of machine translation data for training the probe is motivated by prior work that found a strong correlation between LLM translation ability and LLM performance on downstream reasoning tasks in that language (Issaka et al. (2026)). The benefit of using these translation data is that the output labels for different languages are the same (a shared output language), therefore the performance gap is directly measurable (§4.1, lines 324–329). Our new experiment shows that a probe trained on that signal can guide useful LLM updates. Our main result also support this: with English-only task supervision, the best MAPP variant improves over SFT by 2.6 accuracy points on ARC-Challenge, 5.8 slot F1 on MASSIVE, and 7.0 accuracy points on HellaSwag (Table 1, lines 436–441, §6.1, lines 498–502).

What evidence supports the separation of task-related and language-related information, both theoretically and empirically?

The central argument in this paper is that cross-lingual representation differences can be decomposed into separable language-related and performance-related components.

Our proposed method does not assume that a representation difference can be decomposed into a content vector and a language vector. In fact, we consider this as a major strength of MAPP. The LLM uses its hidden states to produce an answer, and MAPP learns from those states which differences between translations predict a performance gap. The probe can learn that relationship regardless of whether the relevant information is linearly separable (§1, lines 075–092, §4, lines 286–292), and does not involve separating a “language-agnostic vector” in that process.

We did make that assumption of separability in the motivating steering experiment; which could confuse the audience that our proposed method makes the same assumption. However, the motivating steering experiment’s purpose was to test whether moving the target representation toward its translation is a good alignment objective; we show that it is not, and make the argument that “simply decompose into language and content vectors and use the content vector does a much better job” (§3.2, Fig. 3, lines 266–274). We apologize that the original text blurred the distinction. The revision states the assumptions and limitations of this experiment (§3.1, lines 167–195) and explains why its result motivates learning the alignment signal (§§3.2–4, lines 273–292). See top-level response.

What evidence supports characterizing the multilingual probe as language-agnostic?

Thank you for raising this concern. We now report probe predictions separately for each of nine target languages, to provide insight into how the probe handles input language. We show that the distributions of scores for each language are highly similar, while probe predicted scores within each language vary widely. In other words, knowing the language alone tells us little about the predicted gap for a particular query, and the probe’s prediction is hardly determined by language (Appendix I, lines 1352–1357, Fig. 13, lines 1389–1391). See top-level response.

How does MAPP compare against simpler, potentially competitive baselines, like removing the mean language offset before alignment?

Appendix C already compares cross-attention with a simpler probe that mean-pools each sentence before applying an MLP. On six-language MASSIVE, the simpler probe gives 54.99 slot F1 and cross-attention gives 56.99 (Table 4, lines 847–855).
We have also added a report of your suggested metric, the mean-shifted representation distance, against the ground truth performance gap on the queries used during LLM alignment, alongside our probe’s predicted score against ground truth performance gap. The L₂ distance after subtracting the average language offset is essentially uncorrelated with the measured gap at both checkpoints (r = −0.07 and −0.06). The probe reaches 0.52 and 0.50 on the same queries (Appendix H, lines 1307–1316, Fig. 12, lines 1346–1348). The offset-corrected distance thus gives little indication of which queries have a large performance gap.

Do the results extend to other models and open-ended tasks? How does MAPP compare with SFT on machine-translated task data at a matched budget?

We evaluated Llama-3.1-8B-Instruct on three benchmarks (§§5.1–5.2, lines 466–469 and 489–491). We haven’t tested a second model family or evaluated open-ended factual or reasoning questions due to the limit of time. We agree that open-ended factual and reasoning tasks would let us test whether the model can both generate the correct answer and express it in the requested language.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their feedback and for carefully identifying issues in the manuscript. We devoted about a year to the research and left only a week for writing. We apologize for the imperfect presentation. We have continued revising the manuscript and summarize the four major changes below.

  1. Refined the wording throughout the paper. We rewrote the motivation and method sections to make our approach easier to follow. In particular, we now explain why the steering experiment assumes a linear decomposition, how its results motivate MAPP, and how MAPP learns an alignment signal without making that assumption (§1, lines 069–092, §3.1, lines 167–195, and §4, lines 286–292). We hope the revised manuscript reads better.
  2. Isolated the contribution of performance-gap supervision. A reviewer asked whether the gains come from the measured translation gaps or from “a general representation regularizer”. We kept the attention architecture and alignment procedure fixed and shuffled the gap labels across probe-training examples. Shuffling lowers downstream MASSIVE slot F1 from 56.99 to 53.23 under six-language supervision. The 3.76-point drop shows that it matters which gap label is assigned to each example (Appendix C, Table 4, lines 847–849 and 854–855).
  3. Reported the probe output on alignment queries to show that its learning on MT tasks transfers to alignment queries. The probe was trained on general-domain translation triples. On 180 queries from the bilingual data used to align the LLM, its predictions correlate with measured performance gaps at r = 0.52 at epoch 2 and r = 0.50 at epoch 5. Mean-shifted representation distance gives r = −0.07 and −0.06 on the same queries. We believe that the probe trained on translation data transfers to the data on which it guides alignment and continues to outperform the distance baseline as the LLM changes (Appendix H, lines 1298–1302 and 1307–1316, Fig. 12, lines 1346–1348).
  4. Reported per-language probe output. We report the distribution of predicted gaps for each of nine target languages. Predictions vary substantially within each language, and the distributions largely overlap, although Bengali and Tamil tend to receive higher scores. This supports our characterization of the signal as language-agnostic, since knowing the language alone tells us little about the score of a particular query (Appendix I, lines 1352–1357, Fig. 13, lines 1389–1391).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.