acceptodds
Under review as a conference paper at ICLR 2027

Measuring the Language Share of Multilingual Representations

Abstract

Multilingual language models can represent the same content in different languages, but the balance between language and sentence differences inside these models remains difficult to measure. We introduce Language Factor Share (LFS), which uses translations of the same sentences to measure average language differences relative to sentence differences at each layer, without choosing a reference language. Across 33 open base models, language share is lowest at an intermediate layer and higher at the input and output. This pattern repeats on a second collection of translations for 17 models, and the decrease varies widely among similarly sized models. Lower language share is also associated with higher average accuracy on two multilingual benchmarks. In a preregistered study of 19 models, lower language share goes with greater benefit from relevant context in another language when predicting English text, after accounting for a proxy for language data availability, language family, script, and tokenization. Reporting the two parts the share is built from shows which one changed, or whether both did. Our training experiments did not establish that lowering the share improves performance. LFS therefore helps researchers examine how multilingual representations are organized and interpret their changes alongside direct tests of model performance.

Primary area:
interpretability and explainable AI
Keywords:
multilingual language models, representation analysis, analysis of variance, interpretability, cross-lingual transfer, layerwise analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PEXwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LFS (Language Factor Share), a metric comparing "language effect" and "sentence effect" for sentence representations (translated into multiple languages). This metric does not require choosing a reference language, and can be leveraged for cross-layer and cross-model comparison of internal multilingual representations. The authors find that LFS is lowest in intermediate layers. LFS is correlated with higher accuracy on multilingual benchmarks, but optimizing model training so as to lower LFS did not improve model performance.

Critical strengths

I share the authors' motivations -- understanding how different languages are internally represented is an interesting and useful research question.

The definition of LFS appears sound and its explanation is clear, although I would appreciate a deeper interpretation of "sentence effect" -- how do syntactic and semantic differences created by translation affect it?

I appreciate the negative results regarding the training effect experiment mentioned in section 7, showing that LFS is not a good training objective.

Critical weaknesses

Overall, I found this paper hard to follow and unclear. I am unsure how and why researchers should use the proposed metric, as opposed to MEXA or other baselines. However, I am willing to attribute some of my concerns to lack of deep familiarity with research on multilingual representations. I hope discussion with the authors can be constructive and address some of my questions.

Structure and writing style

The paper refers to "protocols" frequently, and it is not clear what this refers to. There seem to be different definitions/usages of "protocols" across the paper, which is confusing to the reader. Citing a few: Line 1042 of the appendix mentions "layer-selection protocols". Section H is titled as "context-transfer protocols". Line 1440 mentions an "expansion protocol": "LFS-VC at the layer of minimum language share from the expansion protocol (never relabeled as standard LFS; a linear fit is unchanged by using 1 − LFS-VC instead)". This particular sentence also mentions relabeling; it reads like the output of a coding agent.

The paper refers to a "development phase" on multiple occasions (starting on Line 85). From my understanding, LFS does not require a set of "development" models to be calculated. The authors mention a "development-grid" of 10 models in the Appendix, which is contrasted against Falcon and SmolLM in the text (line 277). However, Figure 1 depicts results for all models. The authors state "we used ten [models] while developing the method and added seven later under predictions fixed before their runs." Why? I can understand working with a limited suite while developing an experimental setup, but why are these models considered to be in different sets? There is also a "[...] second expansion set of 19 models". Is this "expansion set" related to the "expansion protocols" from line 1440?

In general, the main text often introduces unexplained concepts (e.g. Line 85 mentions "failed predictions" without establishing a task) and results are underspecified (e.g. Line 193 states "measured grids lie far above it" -- why not giving any specific numbers).

UNCLEAR CONTRIBUTIONS

The authors make clear what questions LFS does not address. The section "What LFS can and cannot see." is disappointing -- it makes explicit what LFS cannot see (uniform changes in representation magnitude, and how two languages differ), but not what LFS can see. The only positive experimental results seem to be the correlation between LFS and benchmark accuracy, but there is a wide confidence interval for these correlations. Regarding use of context in another language, LFS also does not surpass MEXA. Could the authors make explicit what they found through LFS that could not be achieved with other metrics?

The motivation behind this proposed metric is also underspecified. The authors state "helps researchers examine how multilingual representations are organized and interpret their changes alongside direct tests of model performance." If by representation of multilingual representation the authors mean cross-layer analysis, why couldn't MEXA be used? If direct tests of model performance are accessible, why not simply evaluate models on those, instead of using LFS as a proxy?

DATA RELATED CONCERNS

All parallel data used in this work was translated from English. An exploration of how this introduces biases in the present results (and comparisons with non-English-centric data) would be interesting. I believe this should be addressed in the Limitations section.

Questions for authors

Are there specific languages (or sentences) that correlate with a stronger/weaker LFS?

Why did the authors create a "noise-corrected" variant of LFS? My intuition regarding multilingual representations would be to keep variance across representations, as they might encode language-specific or context-specific information. Why consider this to be noise?

3 comments
AuthorsOpenReview ↗

Response to reviewer PEXw

Thank you for identifying where the terminology obscured the main question. The words you flagged ("protocols", "development phase", "expansion set") are no longer in the paper, and the introduction no longer mentions failed predictions. LFS can be computed for a single model. We distinguished the first ten models from the next seven because predictions for the seven were written before their runs. Section 4 now introduces each model set by what it was used to test.

What LFS found that other measures could not: Figure 2 shows that the language share is lowest at an intermediate layer mainly because content variation grows there, in all 50 models we checked. In BLOOM-1.7B the language component even rose (52.9% to 62.2% of total variation), and LFS still fell because the content component doubled. A single similarity score cannot show this. In Figure 3, we applied either a common shift to one language or equally large perturbations that varied from sentence to sentence. MEXA and standardized cosine fell under both changes. The common shift increased the language component, while sentence-specific perturbations increased the remainder.

For model accuracy, we use direct evaluation; LFS is not offered as a proxy for it. Section 3 also gives the numbers you requested: the unstructured reference is 0.298 on the 128-language grid, and the measured minima range from 0.60 to 0.90 across the 17 primary models.

We agree that the intervals are wide and that LFS does not surpass MEXA on context use. With 10 to 13 model families the intervals cannot be narrow; Table 2 shows the correlations with each family counted once.

We use content effect for what the translations of an item share. This can include meaning, length, names, and numbers. On WMT13 news originally written in six languages, model rankings were nearly identical across source languages (rank correlation 0.996 to 1.000). A language contributed less to the remainder as an original than as a translation in 79 of 84 comparisons. This locates a translation-associated difference while leaving its syntactic or semantic origin unresolved (Appendix O). The limitations now state that these checks cover six European languages.

On your question about individual languages and sentences: median LFS was 0.48 on 24 high-resource languages and 0.88 on 24 low-resource ones (Figure 4). Dhivehi accounted for a median 3.4% of between-language variation across 14 models, and Macedonian for 0.37%. Shorter sentences contributed more to the content component and the remainder (median correlations with English word count of -0.60 and -0.90, Appendix R).

On your second question: we no longer call the second estimator "noise-corrected"; one supplementary figure still shows the old label. LFS-VC estimates how much finite sampling contributes to the two measured effects. It does not treat variation as noise, and the remainder can still contain meaningful language-specific differences. Standard LFS and LFS-VC selected the same layer in all 19 context-study models (Appendix C).

The negative result on training that you appreciated is kept, in the abstract, the limitations and an appendix.

I appreciate the clarifications provided by the authors. My main concerns remain. LFS does not seem to have an advantage over other measures regarding prediction of model behaviour (Table 1). I share the authors' interest and excitement regarding language representation, and I believe the results reported on Figure 1 may be indicative of interesting patterns in multilingual model representations. However, I do not think the applications of this metric are clear (I share the concerns of reviewer dCK8 in that sense) and I do not believe this paper is ready for acceptance at ICLR.

The authors write "median LFS was 0.48 on 24 high-resource languages and 0.88 on 24 low-resource ones (Figure 4)", but Figure 4 measures results for different training conditions (am I misreading?). An analysis of how LFS differs across different languages (especially comparing high and low-resource languages) would be quite interesting and might provide directions for future research.

AuthorsOpenReview ↗

response to PEXw

Thank you for reading our response. Please refer to the revised PDF, which is now available on this page. We posted our responses before the revised PDF was uploaded, so the paper you had in front of you was the original submission. You are not misreading: in the original, Figure 4 shows the training conditions. All figure, table and page numbers in our responses refer to the revised PDF. We apologize for the confusion.

Below we answer your points in order, with the page of the revised PDF for each.

First, the comparison of high and low resource languages that you suggest is already in the main text. Figure 4 on page 7 shows content share on 24 high resource and 24 low resource languages for 14 models. Every model shares far more among the high resource languages. Median LFS is 0.48 on the high resource languages and 0.88 on the low resource ones. On three groups of 26 languages it is 0.45, 0.77 and 0.87 from high to low, in that order in all 14 models. The same page also shows what separates languages: at the input, 31% of the differences lie between writing systems, and at the most shared layer the part between resource levels doubles from 5% to 10%.

Second, on prediction of model behaviour, we agree with you. The revised abstract says that for accuracy, similarity measures computed over all language pairs do about as well as LFS (Table 1, page 8). We do not offer LFS as a better predictor.

Third, Figure 2 on page 5 explains the pattern you found interesting in Figure 1. The language share is lowest at an intermediate layer mainly because the content part grows there, and not because the languages come closer. In all 50 models we checked, the content part is a larger share of total variation at that layer than at the input. In the smaller BLOOM model the language part even grows, from 52.9% to 62.2% of total variation, and LFS still falls because the content part doubles. A single similarity score cannot show this.

Fourth, on applications. The use we claim is diagnostic. Figure 3 on page 6 shows one example. We added one constant vector to every sentence of a language, or noise of the same total size to each sentence. LFS rose by 0.118 and 0.151 in two Qwen3 models under the constant vector, and by at most 0.011 under the noise. MEXA for that language fell under both, so MEXA cannot say which of the two happened. The split can.

Fifth, we share your view and that of Reviewer dCK8 that an application to task accuracy would be stronger. We tested it and report the result in Section 5 on page 6. Removing the language effect brings translations closer, from 0.41 to 0.55, but accuracy fell by 0.7 points over 14 models. We do not claim that application.

To summarize: LFS is not a better predictor, and the revised paper says so. What it adds is the split, which shows which part of the variation changes and how strongly the result depends on the languages measured. These results are in Figures 2, 3 and 4 of the revised PDF, on pages 5, 6 and 7.

We understand if you still find the contribution insufficient, and we thank you for engaging with our response.

Reviewer_jFJTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Language Factor Share (LFS), a two-way variance decomposition that compares average language effects with average sentence effects in parallel-text representations. Across 33 base models, it finds lower language share at intermediate layers, replicates the pattern on another corpus, and studies associations with multilingual accuracy and cross-language context use. Training experiments do not show that reducing LFS improves performance. The contribution is a descriptive diagnostic and a broad empirical study, not a new training method or a general measure of multilingual ability.

Overall assessment: The transparent validation and informative negative results outweigh the metric's modest novelty. Acceptance is justified as a careful representation study, not as evidence that lower LFS means a better multilingual model.

Critical strengths
  • The empirical study is broad and unusually careful about interpretation. It covers 33 models, replication across corpora, sensitivity checks, and a preregistered 19-model context-transfer study. The paper distinguishes model-level evidence from repeated model-language observations and reports unsuccessful predictions and training interventions. This makes the descriptive findings more credible.
  • The decomposition provides interpretable information beyond a single similarity score. Reporting language, sentence, and residual variation helps explain why a ratio changes. The demonstrations that language remains decodable and that representation magnitude can shrink without a large LFS change are useful cautions for representation analysis (Secs. 3 and 7).
Critical weaknesses
  • The added practical value over existing diagnostics is still limited. Sec. 6 finds nearly equivalent within-model language rankings, no demonstrated advantage over standardized cosine, and no established predictive gain from adding LFS to MEXA. The underlying decomposition is classical ANOVA. Thus, the contribution rests mainly on interpretation and empirical coverage; a concrete diagnostic conclusion unavailable from simpler measures would strengthen its significance.
  • Cross-model rankings depend substantially on measurement choices. Appendix J reports ranking correlations of only 0.60 under length-matched truncation and 0.45 under whitening. Appendix K also notes that rotating raw representations before coordinate standardization can change LFS. These are acknowledged limitations, not hidden flaws, but they restrict comparisons to a fixed measurement protocol rather than an intrinsic ordering of models.
Questions for authors
  1. Clarification: Which substantive model comparisons remain stable across the tested pooling and preprocessing choices? Can the paper distinguish those robust conclusions from rankings specific to the default estimator?
  2. Clarification: Does the main context-transfer association remain after controlling for model size or general English capability? Language-level covariates and normalization by English context benefit do not by themselves resolve model-level confounding.
1 comment
AuthorsOpenReview ↗

Response to reviewer jFJT

Thank you for asking which individual model comparisons survive changes in measurement. Across seven mean-pooled comparisons, Qwen3-4B and Qwen3-8B stayed among the four models with the highest content share, and OLMo-2-0425-1B stayed among the four lowest. Overall, 39 of 91 model pairs retained their order. Changing the pooling rule could reverse even a large difference: in the five-model pooling check, final-token pooling reversed Qwen3-8B and BLOOM-1.7B (Appendix N). What is robust is the pattern within a model: under every pooling scheme and on both main corpora, the content part is larger at the lowest-LFS layer than at the input. The limitations now have a paragraph, "What is stable and what is not", that separates the two. We agree that the ordering of models holds for a fixed way of measuring and is not an intrinsic ordering.

The association with normalized cross-language context benefit remained after controlling for model size and English accuracy. The partial rank correlation with content share was 0.75 given size (19 models), 0.68 given English accuracy (16 models), and 0.74 given both (16 models, 95% family-bootstrap interval [0.22, 0.95]). For raw context benefit, the interval with both controls included zero (Section 6 and Appendix M).

Figure 2 gives a concrete example of the diagnostic value you asked about. In BLOOM-1.7B, the language component rose from 52.9% to 62.2% of total variation. LFS still fell because the content component doubled from 3.5% to 7.0%. Reporting the components reveals why the ratio fell. Figure 3 adds a controlled test: a shift of a whole language and noise of the same size both lower MEXA and cosine, and the components show which of the two occurred.

We agree that LFS shows no advantage over standardized cosine for predicting accuracy. Computed over all language pairs, cosine orders the 30 models almost as content share does (rank correlation 0.95), and the paper now says so. For predicting context benefit, we found no clear gain from adding LFS to MEXA.

The training experiments of the former Section 7 moved to an appendix to make room. Their result stays in the abstract and the limitations: lowering LFS by fine-tuning did not raise accuracy.

Reviewer_dCK8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes a score, language factor share (LFS), which quantifies the degree of language specific impact to the sentence-wise representation in relation to the overall drift by language and semantic of sentences. The layer-wise visualization of diverse models shows that LFS is high in the bottom and top layers, while low in the middle but different tendencies observed across models. The correlation of the end-to-end task accuracies on QA benchmarks (i.e., Belebele and INCLUDE) shows moderate correlation when measured on the layer where LFS is the minimum for each model.

Critical strengths
  • The proposed score can quantify the language specific representations observed in each layer, although not intended to relate the end-to-end accuracies. At least, it can identify the language neutral layer for each model by searching for the layer with the minimum LFS.
  • The score is very simple yet intuitive in that the differences is measured by taking the difference of mean representations and the differences are aggregated language-wise and sentence-wise to represent the impact of language-specific and sentence semantic-specific representations, respectively.
Critical weaknesses
  • The manuscript is hard to follow: Several important discussions about the proposed score rely on figures and tables in appendices, and it is hard to justify the explanation in the manuscript.
  • The assumption of uniform language: Language models are usually influenced by the amount of training data, and language imbalance give bias to the language representations. Thus, it is not clear whether it is feasible to simply taking an average across languages without considering a prior of language resource distribution. At leas, a controlled study will be necessary to measure LFS on a subset of languages to see if there is an impact of low-resource or high-resource languages. Alternatively, it may be more beneficial to measure the difference from English representations, not from the average across languages to measure the bias encoded in each language, assuming that English is a dominant language for the training resource.
  • Application of the score: An application of the score is not clear in that it can simply trace the impact of languages in each layer. At least, I'd expect an application of reducing language biases in end-task performance.
Questions for authors
  • Discussion on mean representation: I'd like to find a justification of taking a mean across languages.
  • Additional experiments: It is better to investigate the difference of the language subset to investigate the impact of training resources for language models. In addition. A difference from English representation might be more informative to quantify the biases encoded for each language when compared with English.
1 comment
AuthorsOpenReview ↗

Response to Reviewer dCK8

Thank you for suggesting the resource-level and English-reference comparisons. We weight languages equally because our target is the average language in the evaluated set. Your proposed subset test shows why that choice matters. On equally sized panels of 24 languages, all 14 models had higher content share on the high-resource panel. Median LFS was 0.48 on the high-resource panels and 0.88 on the low-resource panels (Figure 4). Weighting languages by corpus token count changed the level substantially, while preserving the lowest-LFS saved layer in 13 of 14 models.

We tested English as a reference in two ways. Across 30 models, MEXA and standardized cosine against English went less strongly with accuracy than the same measures averaged over all language pairs, by 0.10 to 0.17 on Belebele, INCLUDE, and Global-MMLU (Table 1). Within one model, your suggestion works well: how much a language shares with English alone goes with how much of the English accuracy the model keeps in that language (rank correlation 0.89, Table 3). So an English reference is informative for comparing languages within a model, and weaker for comparing models.

We also tested whether removing the language effect helps in practice. Subtracting the average language offset from saved vectors improved translation retrieval from 0.41 to 0.55. Applying a similar shift inside 14 models lowered mean non-English Belebele accuracy by 0.7 percentage points. In the two Qwen3 follow-ups, the English accuracy gap widened in both. Section 5 now presents the retrieval and task results together, and we do not claim that LFS reduces differences in end-task accuracy.

On presentation: these results are now in the main text with their figures and tables (Figures 2 to 4, Tables 1 to 3), not only in appendices.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised PDF uploaded: figure and table numbers refer to the revision

A note to all reviewers: we posted our responses before the revised PDF was uploaded, and we apologize for the confusion. The revised PDF is now on this page. All figure, table and section numbers in our responses refer to it, not to the original submission. In the revision: Figure 2 (page 5) shows the three parts across layers, Figure 3 (page 6) the shifted against the noisy language, Figure 4 (page 7) high- against low-resource languages, Tables 1 and 2 (page 8) the accuracy results and their checks, and Table 3 and Figure 5 (page 9) the gap to English and context use.

AuthorsOpenReview ↗

Note to all reviewers: please refer to the revised PDF

We apologize to all reviewers. We posted our responses before the revised paper was uploaded. If you read our responses against the original submission, the figure, table and page numbers will not have matched. The revised PDF is now available on this page. All numbers in our responses refer to it.

The revised paper contains the following, none of which was in the main text of the original.

  1. Figure 2 on page 5 shows the three parts of the split across layers. The language share is lowest at an intermediate layer mainly because the content part grows there. This holds in all 50 models we checked.

  2. Figure 3 on page 6 shows a controlled test. A shift of a whole language raises LFS by 0.118 and 0.151 in two models. Noise of the same size moves it by at most 0.011. MEXA falls under both, so it cannot tell the two apart.

  3. Figure 4 on page 7 compares high and low resource languages, as Reviewer dCK8 asked. Median LFS is 0.48 on 24 high resource languages and 0.88 on 24 low resource ones. The same page reports three resource levels and a version that weights languages by how much text they have.

  4. Table 1 on page 8 now covers six test sets and 30 models. It compares LFS with MEXA and cosine computed against English and over all language pairs. Over all pairs the baselines do about as well as LFS, and the paper says so.

  5. Table 2 on page 8 gives the checks that Reviewer jFJT and Reviewer PEXw asked about: one value per model family, only models above the guessing level, and controls for model size and English accuracy.

  6. Table 3 and Figure 5 on page 9 report the gap to English and the use of context in another language. The context result holds given model size and English accuracy (0.74, interval 0.22 to 0.95).

  7. Section 5 on page 6 reports two tests on other text: text written in six languages, and a test set released after the models were trained.

  8. Section 5 also reports the application Reviewer dCK8 asked about. Removing the language effect brings translations closer, from 0.41 to 0.55, but it did not raise accuracy. We state this as a negative result.

Three further changes. The words that Reviewer PEXw found confusing are removed, and the model sets are described once, by what each is used for. The main text has five figures and three tables, where the original had two figures and one table. The training experiments moved to an appendix, and their result stays in the abstract and the limitations.

We do not claim that LFS predicts accuracy better than other measures. The contribution is the split into a language part, a content part and a remainder, and what it shows about how a model organizes language and content.

We thank the reviewers for their time, and we are sorry for the confusion that the order of our uploads caused.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.