- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the relationship between cross-lingual representation differences and cross-lingual performance differences in multilingual LLMs. The authors train probes that predict performance gaps between languages from internal representations and then use these probes as reward models to improve cross-lingual alignment.
I find the setup and the core idea very appealing. The paper itself is at times extremely hard to follow because of very dense writing and presentational issues. In many places I had to read sentences five to ten times to extract their meaning. In its current form, the presentation falls short of the level required for publication.
Critical strengths
- The paper addresses an important problem, the improvement of multilingual representations, and proposes a clever idea: using a performance probe as a reward signal for alignment. This idea is the main reason I see potential in the work.
- The experimentation is thorough, which gives the empirical claims a solid basis.
Critical weaknesses
- Writing and clarity (major). The presentation significantly hinders understanding of the paper.
- Several passages are very dense and hard to parse, for example lines 054-060 and 067-070. They read like the product of AI rewriting. The writing could be much clearer and more concise.
- The purpose of Section 3.1 is unclear to me. I assume it is meant to show that representational difference alone is insufficient to predict performance. Measuring representational distance via a pivot language seems unmotivated, and the intuition behind this choice is missing.
- I was unable to decipher Figure 4.
- Motivation for the probe architectures (major). The reasoning behind the two complex probe architectures remains unclear. A linear or bilinear classifier on the concatenated reference and target representations seems like the natural baseline. The use of SAEs is also unmotivated: SAEs are lossy approximations of the residual stream, so probing the residual stream directly seems preferable.
- Evaluation (minor): The evaluation is restricted to classification tasks. I understand that task availability is limited for low-resource languages, but machine translation on FLORES would have been a feasible addition.
Questions for authors
What is the intuition behind measuring representational distance via a pivot language in Section 3.1, and what role does this section play in the overall argument? Why are SAE features used as probe inputs instead of the residual stream directly? How do the proposed probes compare to a simple linear or bilinear probe on the concatenated reference and target representations? What does Figure 4 show, and how should it be read?
2 comments
Thanks a lot for the extensive answer! I see the paper has been heavily rewritten and most of my weaknesses have been addressed (I still don't agree with the SAE-centric probes; I think the residual would make sense to include as a baseline at least), so I'm raising my score.
We apologize for the imperfect presentation. We devoted about a year to the research and left only a week for writing. We should have allowed more time. We have refined the wording throughout the paper and rewritten the introduction and the transition from the steering study to MAPP. We hope the revised manuscript reads better. See top-level response.
The original section 3.1 was addressing the claim made by prior work that "representation distance between a low resource language and English predicts downstream performance in that language". Liu and Niehues (2025, Fig. 1b) measured alignment by translation retrieval and found that it correlated with downstream slot-filling F1 across languages. Through experiments, we found that that is only true on average. Across individual queries, there's large variance and the actual query-level correlation is weak. These findings motivated us to develop MAPP, which aims at learning an alignment signal that actually predicts performance gap, rather than relying on raw representation distance.
But we do realize that it is loosely related to the central argument, so we've revised it to focus on the representation distribution study, where we show that representations cluster by language, and that representation distance is largely language gaps with pretty small performance gaps (§3.1, Figs. 1 and 2a, lines 204–214 and 227–229). These results show that it is thus improper to use those distances for alignment directly, and motivates a language-agnostic, performance-based signal.
We apologize for putting the figures in the wrong order. The probe architecture diagram is Figure 5 in the revised manuscript, and §4.2 explains its two panels. On the left, each target token attends to the reference tokens before the probe predicts a performance gap. On the right, the SAE encodes both sentences and scores the pair by the distance between their pooled codes (Fig. 5, lines 395–399). Figure 4 now introduces the training procedure before we describe either architecture (lines 310–313).
We chose an SAE because prior work found that some sparse features are voluntarily language-agnostic, making them a good fit to signify cross-lingual content matching. Some prior work had used it to align the LLM (Sin Jing Xuan et al., 2025). We further show that it can do better if we finetune the SAE with a deliberate language-agnostic loss term. (§4.2, lines 360–377 and 410–412, Fig. 5, lines 395–399).
We report a linear probe for comparison in the ablation section, alongside other ablation studies to demonstrate the selected, complex probe structure is necessary (Appendix C, Table 4, lines 847–855). On six-language MASSIVE, the mean-pooled MLP scored 54.99 slot F1, compared with 56.99 for the full cross-attention probe. Giving every reference token the same attention weight reduced F1 to 55.39, suggesting that learning which tokens to attend to improves downstream slot filling.
The choice of nonlinear probes is because prior work had established that some cross-lingual information in LLM representations cannot be recovered by a linear readout. Gonen et al. (2020), for example, found that nonlinear methods could recover substantially more word-translation information from mBERT than linear methods could. If language and task-relevant content are entangled in this way, a linear probe will miss information that could help predict the performance gap (§1, lines 087–092).
Thank you for suggesting a machine translation evaluation on FLORES. We did not include this workflow (finetuning on machine translated queries) as a baseline due to the limit of time. We do find it an interesting baseline that is widely used for middle to high resource languages where high-quality MT systems are available. However, we find the machine translated data for our tested low-resource languages a bit unsatisfying and prone to translation errors, and were not able to establish a good instantiation of this baseline for comparison. We will consider implementing this baseline as future work.