- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes LFS (Language Factor Share), a metric comparing "language effect" and "sentence effect" for sentence representations (translated into multiple languages). This metric does not require choosing a reference language, and can be leveraged for cross-layer and cross-model comparison of internal multilingual representations. The authors find that LFS is lowest in intermediate layers. LFS is correlated with higher accuracy on multilingual benchmarks, but optimizing model training so as to lower LFS did not improve model performance.
Critical strengths
I share the authors' motivations -- understanding how different languages are internally represented is an interesting and useful research question.
The definition of LFS appears sound and its explanation is clear, although I would appreciate a deeper interpretation of "sentence effect" -- how do syntactic and semantic differences created by translation affect it?
I appreciate the negative results regarding the training effect experiment mentioned in section 7, showing that LFS is not a good training objective.
Critical weaknesses
Overall, I found this paper hard to follow and unclear. I am unsure how and why researchers should use the proposed metric, as opposed to MEXA or other baselines. However, I am willing to attribute some of my concerns to lack of deep familiarity with research on multilingual representations. I hope discussion with the authors can be constructive and address some of my questions.
Structure and writing style
The paper refers to "protocols" frequently, and it is not clear what this refers to. There seem to be different definitions/usages of "protocols" across the paper, which is confusing to the reader. Citing a few: Line 1042 of the appendix mentions "layer-selection protocols". Section H is titled as "context-transfer protocols". Line 1440 mentions an "expansion protocol": "LFS-VC at the layer of minimum language share from the expansion protocol (never relabeled as standard LFS; a linear fit is unchanged by using 1 − LFS-VC instead)". This particular sentence also mentions relabeling; it reads like the output of a coding agent.
The paper refers to a "development phase" on multiple occasions (starting on Line 85). From my understanding, LFS does not require a set of "development" models to be calculated. The authors mention a "development-grid" of 10 models in the Appendix, which is contrasted against Falcon and SmolLM in the text (line 277). However, Figure 1 depicts results for all models. The authors state "we used ten [models] while developing the method and added seven later under predictions fixed before their runs." Why? I can understand working with a limited suite while developing an experimental setup, but why are these models considered to be in different sets? There is also a "[...] second expansion set of 19 models". Is this "expansion set" related to the "expansion protocols" from line 1440?
In general, the main text often introduces unexplained concepts (e.g. Line 85 mentions "failed predictions" without establishing a task) and results are underspecified (e.g. Line 193 states "measured grids lie far above it" -- why not giving any specific numbers).
UNCLEAR CONTRIBUTIONS
The authors make clear what questions LFS does not address. The section "What LFS can and cannot see." is disappointing -- it makes explicit what LFS cannot see (uniform changes in representation magnitude, and how two languages differ), but not what LFS can see. The only positive experimental results seem to be the correlation between LFS and benchmark accuracy, but there is a wide confidence interval for these correlations. Regarding use of context in another language, LFS also does not surpass MEXA. Could the authors make explicit what they found through LFS that could not be achieved with other metrics?
The motivation behind this proposed metric is also underspecified. The authors state "helps researchers examine how multilingual representations are organized and interpret their changes alongside direct tests of model performance." If by representation of multilingual representation the authors mean cross-layer analysis, why couldn't MEXA be used? If direct tests of model performance are accessible, why not simply evaluate models on those, instead of using LFS as a proxy?
DATA RELATED CONCERNS
All parallel data used in this work was translated from English. An exploration of how this introduces biases in the present results (and comparisons with non-English-centric data) would be interesting. I believe this should be addressed in the Limitations section.
Questions for authors
Are there specific languages (or sentences) that correlate with a stronger/weaker LFS?
Why did the authors create a "noise-corrected" variant of LFS? My intuition regarding multilingual representations would be to keep variance across representations, as they might encode language-specific or context-specific information. Why consider this to be noise?
3 comments
I appreciate the clarifications provided by the authors. My main concerns remain. LFS does not seem to have an advantage over other measures regarding prediction of model behaviour (Table 1). I share the authors' interest and excitement regarding language representation, and I believe the results reported on Figure 1 may be indicative of interesting patterns in multilingual model representations. However, I do not think the applications of this metric are clear (I share the concerns of reviewer dCK8 in that sense) and I do not believe this paper is ready for acceptance at ICLR.
The authors write "median LFS was 0.48 on 24 high-resource languages and 0.88 on 24 low-resource ones (Figure 4)", but Figure 4 measures results for different training conditions (am I misreading?). An analysis of how LFS differs across different languages (especially comparing high and low-resource languages) would be quite interesting and might provide directions for future research.
response to PEXw
Thank you for reading our response. Please refer to the revised PDF, which is now available on this page. We posted our responses before the revised PDF was uploaded, so the paper you had in front of you was the original submission. You are not misreading: in the original, Figure 4 shows the training conditions. All figure, table and page numbers in our responses refer to the revised PDF. We apologize for the confusion.
Below we answer your points in order, with the page of the revised PDF for each.
First, the comparison of high and low resource languages that you suggest is already in the main text. Figure 4 on page 7 shows content share on 24 high resource and 24 low resource languages for 14 models. Every model shares far more among the high resource languages. Median LFS is 0.48 on the high resource languages and 0.88 on the low resource ones. On three groups of 26 languages it is 0.45, 0.77 and 0.87 from high to low, in that order in all 14 models. The same page also shows what separates languages: at the input, 31% of the differences lie between writing systems, and at the most shared layer the part between resource levels doubles from 5% to 10%.
Second, on prediction of model behaviour, we agree with you. The revised abstract says that for accuracy, similarity measures computed over all language pairs do about as well as LFS (Table 1, page 8). We do not offer LFS as a better predictor.
Third, Figure 2 on page 5 explains the pattern you found interesting in Figure 1. The language share is lowest at an intermediate layer mainly because the content part grows there, and not because the languages come closer. In all 50 models we checked, the content part is a larger share of total variation at that layer than at the input. In the smaller BLOOM model the language part even grows, from 52.9% to 62.2% of total variation, and LFS still falls because the content part doubles. A single similarity score cannot show this.
Fourth, on applications. The use we claim is diagnostic. Figure 3 on page 6 shows one example. We added one constant vector to every sentence of a language, or noise of the same total size to each sentence. LFS rose by 0.118 and 0.151 in two Qwen3 models under the constant vector, and by at most 0.011 under the noise. MEXA for that language fell under both, so MEXA cannot say which of the two happened. The split can.
Fifth, we share your view and that of Reviewer dCK8 that an application to task accuracy would be stronger. We tested it and report the result in Section 5 on page 6. Removing the language effect brings translations closer, from 0.41 to 0.55, but accuracy fell by 0.7 points over 14 models. We do not claim that application.
To summarize: LFS is not a better predictor, and the revised paper says so. What it adds is the split, which shows which part of the variation changes and how strongly the result depends on the languages measured. These results are in Figures 2, 3 and 4 of the revised PDF, on pages 5, 6 and 7.
We understand if you still find the contribution insufficient, and we thank you for engaging with our response.
Response to reviewer PEXw
Thank you for identifying where the terminology obscured the main question. The words you flagged ("protocols", "development phase", "expansion set") are no longer in the paper, and the introduction no longer mentions failed predictions. LFS can be computed for a single model. We distinguished the first ten models from the next seven because predictions for the seven were written before their runs. Section 4 now introduces each model set by what it was used to test.
What LFS found that other measures could not: Figure 2 shows that the language share is lowest at an intermediate layer mainly because content variation grows there, in all 50 models we checked. In BLOOM-1.7B the language component even rose (52.9% to 62.2% of total variation), and LFS still fell because the content component doubled. A single similarity score cannot show this. In Figure 3, we applied either a common shift to one language or equally large perturbations that varied from sentence to sentence. MEXA and standardized cosine fell under both changes. The common shift increased the language component, while sentence-specific perturbations increased the remainder.
For model accuracy, we use direct evaluation; LFS is not offered as a proxy for it. Section 3 also gives the numbers you requested: the unstructured reference is 0.298 on the 128-language grid, and the measured minima range from 0.60 to 0.90 across the 17 primary models.
We agree that the intervals are wide and that LFS does not surpass MEXA on context use. With 10 to 13 model families the intervals cannot be narrow; Table 2 shows the correlations with each family counted once.
We use content effect for what the translations of an item share. This can include meaning, length, names, and numbers. On WMT13 news originally written in six languages, model rankings were nearly identical across source languages (rank correlation 0.996 to 1.000). A language contributed less to the remainder as an original than as a translation in 79 of 84 comparisons. This locates a translation-associated difference while leaving its syntactic or semantic origin unresolved (Appendix O). The limitations now state that these checks cover six European languages.
On your question about individual languages and sentences: median LFS was 0.48 on 24 high-resource languages and 0.88 on 24 low-resource ones (Figure 4). Dhivehi accounted for a median 3.4% of between-language variation across 14 models, and Macedonian for 0.37%. Shorter sentences contributed more to the content component and the remainder (median correlations with English word count of -0.60 and -0.90, Appendix R).
On your second question: we no longer call the second estimator "noise-corrected"; one supplementary figure still shows the old label. LFS-VC estimates how much finite sampling contributes to the two measured effects. It does not treat variation as noise, and the remainder can still contain meaningful language-specific differences. Standard LFS and LFS-VC selected the same layer in all 19 context-study models (Appendix C).
The negative result on training that you appreciated is kept, in the abstract, the limitations and an appendix.