“OLMoTrace analysis of pretraining data sources reveals little direct reproduction of training text across languages, while the corpus composition, training stage, and cultural provenance of retrieved training evidence vary substantially by response language.” I honestly don’t know what is meant by this. Are they saying training data is typically only in one language and not multilingual?
“To what extent can LLMs generalize moral reasoning across languages and cultures? More specifically, what moral assumptions embedded in their training data limit their cross-cultural ethical competence?” It is unclear whether this is a good or a bad thing. Specifically, what does generalizing moral reasoning mean? Does it mean applying the same morals no matter the culture, or does it mean adapting to fit the culture? I’m assuming they mean the latter, but they should define what is meant by generalizable moral reasoning. Also, why are the authors asserting generalizable moral reasoning to be a desirable trait? I would assume that a model should have consistent, unshakable morals to avoid issues with safety. The authors should defend this stance, or at least clarify their motives to be exploration-based rather than argument-based.
“English dominates regardless of gold truth origin.” It is extremely unclear what is meant by this sentence. Do the authors mean that the moral values associated with English moral value datasets are most likely to be reproduced from models? If so, then stating “english dominates” does not convey this.
What are the number of samples in MoralExceptQA + ETHICS versus CMoralEval and UniMoral?
In the description for RQ3: “Do the moral framing of inputs in different languages and models’ inherent values” This seems like two questions. What is meant by moral framing? What are the models inherent values (are models’ inherent values these the preferences described in RQ1 or something different)? It seems like the authors are really asking about the first part, the aspect of models’ inherent values seems irrelevant to this question (though perhaps it is a justification for the results).
“retrieved evidence varies substantially” Unclear what is meant by “retrieved evidence”
It is unclear what is meant by “heterogeneous and often localized training-data pathways.” Heterogeneous linguistically?
“The answers to RQ1 to RQ4 reveal systematic cross-linguistic variation in LLM moral judgment, reasoning, and value weighting, suggesting that current models rely more on shared training abstractions than on culturally grounded moral norms.” Don’t the results state the opposite? If English-related moral values are dominant and there is low verbatim overlap, doesn’t that imply that there is low cross-linguistic variation?
“LLMs exhibit substantial cross-lingual variation, yet English generally leads regardless of the source of ground-truth judgments.” The meaning of this sentence is unclear. Do they exhibit cross-lingual variation in their preferences? If so, then why does the sentence also imply that there is little variation because English leads. What is meant by “English generally leads,” is this talking about the values from English datasets? It seems like there is an equivocation of moral judgments and preferences here, when they are not the same. You should at some point define what is meant by an ethical/moral preference and a judgment. Based on my reading, you use judgment to mean a single output, while a preference is a trend across outputs (i.e., your judgments will trend toward your preferences). If you are using the definitions from UniMoral, then state that.
“Despite Chinese being the source language, it does not yield the strongest performance, and English still outperforms it.” Did the authors look into the data to understand potential reasons why?
“further shows that cross-lingual behavioral similarity does not systematically reorganize around the language from which the ground truth originates.” Its unclear what is meant by “systematically reorganize.”
“we adopt an “LLM-as-a-judge” approach to automatically annotate explanations with fifteen reasoning phases and scores across ten normative ethical frameworks.” Did you verify these annotations?
“We have the generated responses for all eight datasets in English (n = 26,382 traced records)” 8 datasets? There were only two discussed? If you are treating each of the dataset splits separately, then you should be consistent throughout the paper or clarify here.
“Which training corpora are associated with moral reasoning, and does their contribution differ across languages?” This doesn’t make much sense. How would the training data be specifically associated with reasoning? If the data were more likely to be produced from the model, that says nothing about its link to the models reasons and the ways that it uses that data.
“Although OLMo-2 is primarily an English language model (Appendix Table 4), it successfully generates responses in all six languages evaluated here” How good are the responses?
“Arabic and Spanish contain substantially larger fractions whose cultural provenance cannot be confidently identified.” How was the cultural provenance of the data identified? In the appendix it describes pulling from page metadata, but this should also be mentioned within the core paper since it is fundamental to the research question being answered.
“our results reveal shared abstraction but heterogeneous training-data pathways across languages.” What is meant by this? The authors should be very clear about the level of abstraction that is shown. Specifically, they only show the models are not exactly reproducing data from their many datasets, not that the models are actually learning to generalize from the samples provided.
“localization of retrieved training evidence does not necessarily translate into localization of the moral abstractions expressed by the model.” It is unclear what is the exact link the authors aim to make across moral abstractions and moral reasoning. In addition, you can’t make this claim because different models have different sources and different proportions of these sources. The authors have already shown in Figure 1 that models tend to perform the best when producing outputs in English, so models that were trained on a majority of English data would exhibit this pattern where the localization of retrieved training evidence does not necessarily translate into localization of the moral abstractions (unless the authors mean for “moral abstractions” to mean the literal moral reasoning text, which is not the definition of moral abstraction).
In A.1.2: “native speakers additionally verify ambiguous or low-confidence items,” Its unclear how/if you recruited native speakers. Did you just use all-MiniLM-L6-v2 instead of a native speaker?
Also in A.1.2: “The average similarity scores across languages are: Hindi (0.81), Russian (0.78), Spanish (0.76), Chinese (0.74), and Arabic (0.68).” These similarity scores do not seem great, especially not for Arabic. You cannot use these as valid translations, in addition to the many other issues I mentioned above. The authors should also seek to explain (or at least address as an open question) why the similarity scores for similarly structured languages like English and Spanish, were still less similar than very different languages like Hindi and English.
What is the authors’ approach to hierarchical clustering? Single, complete, or average. This could change a lot.
Additional issues:
Combining model outputs without having a section detailing the variances in model performance/training data makes the results quite noisy. In particular, the results from RQ4 are quite inconclusive for a number of reasons, this being one.
“(Kohlberg, 1981; Haidt & Graham, 2007)” These are quite odd citations if you’re just talking about the foundations of moral reasoning. The authors should describe what they mean by moral reasoning, then use their citations to support this definition.
“moral reasoning itself is far from universal” Once again, a definition regarding what is meant by moral reasoning/the author’s stance on morality would go a long way toward supporting this statement. This seems very hand-wavey right now. For example, a moral objectivist might view moral reasoning as the attempt to approximate true morality while a relativist might treat this reasoning process as true morality in itself. Even an antirealist might need to clarify that this moral reasoning only exist as a human concept rather than a universal one.
“(Kumar & Jurgens, 2025)” This is not a good paper to cite here, the goal is to show the philosophical justification for the computational methods you will be discussing. This paper is already computational.
The primary difference between the authors’ translations in comparison to UniMoral’s translations, were that the UniMoral dataset validated the outputs with human evaluators. The translations the authors created for MoralExceptQA and ETHICS are still not valid.
UniMoral is not a “native benchmark.” The data was created in English then translated to other languages and verified using human evaluators. Data of a particular origin should be pulled from that country. UniMoral would be a dataset with annotations from native annotators.
The findings for RQ2 are not consistent with the graphs shown. Specifically, the graph does not show that “Utilitarianism dominates in 5/6 languages, with Hindi uniquely favoring Deontology.” Instead, it shows models vary in the frameworks their reasoning support. In addition, this method of quantifying moral reasoning does not show whether they systematically reason in different ways, it only shows their patterns of morally-salient language use across language.
The authors should change use the term “moral foundation” or “moral theory” to clarify what they mean in different contexts, instead of “moral framing.” A large portion of the time they just mean moral foundations in MFT.
In RQ4: “To what extent does pretraining data shape LLMs’ moral orientations, and does it lead them to generalize or merely reproduce content?” This is a very big question that the data/evaluation does not support (see more below). Specifically, the claim about generalization is too broad for what the authors show.
In the evaluation for RQ4, “verbatim overlap is only ≈ 3%” is this low overlap not just explained by the model’s temperature and other forms of exploration of the generation space? How can you assert that the outputs are not semantically equivalent, because the question specifically asks how the data shapes model morals. I would even argue that semantics are not enough if you are aiming to assert full generalizability, but verbatim overlap is certainly not enough. If this is not what the authors aim to show, then they should reduce the scope.
“The answers to RQ1 to RQ4 reveal systematic cross-linguistic variation in LLM moral judgment, reasoning, and value weighting, suggesting that shared moral abstractions emerge through heterogeneous and often localized training-data pathways” These research questions don’t suggest this. Moral abstraction is not at all related to verbatim overlap, it is concerned with the moral values embedded within moral judgments/decisions. This means that RQ4 is irrelevant.
“We translate English MoralExceptQA and ETHICS, and Chinese CMoralEval using SeamlessM4T-v2 (Communication et al., 2023) in all the languages in L.” You need to validate these translations.
You need to show the English F1 score on all of these datasets. Just knowing that it performs the best isn’t enough.
“These three results show that the English advantage cannot be explained solely by English-origin ground truth or translation effects. It persists when ground-truth judgments originate in Chinese” I don’t think you can assert that because the error is so great and the difference in average so small in CMoralEval. This is magnified by the fact that it is unclear how many samples are used to get these error amounts. Either emphasize statistical significance, or don’t make this claim.
In the caption for Figure 2b: “Utilitarianism dominates across languages except Hindi, as shown by item-weighted ethical-framework scores.” That is not at all what this shows? Different languages lean toward different theories.
“We obtain eMFD dictionaries for the languages in L by translating the English eMFD using a standard translation and back-translation strategy (Singh et al., 2025). Further details on dictionary translation and verification are provided in Appendix §A.1.2.” This seems like an approach that would lose a high amount of information since the dataset was only originally produced in English. Different languages have different nuances in the words that are used, and that does not get captured unless you re-annotate it.
“Most languages nevertheless exhibit a sharp change at the final step, suggesting that the moral-value composition of the concluding stage differs from the preceding reasoning trajectory.” It is also still not clear how much of this error is due to poor translation/semantic equivalence. The plots shown to validate the semantics of the translations are not shown to be statistically significant (discussed below).
“40 model reasonings (20 English and 20 Hindi) using the same phase and framework definitions. Reasoning-phase annotations show substantial agreement with the LLM judge.” This is quite a low number, especially if one is doing a correlation analysis (discussed below). In addition, it is never stated how many samples there are in total.
“Cohen’s κ = 0.73 for English and 0.68 for Hindi” 20 samples per language is just too noisy (even though there are 15 categories). The authors should annotate more samples, and show the proportion of categories that exist in the samples to ensure sample rarity isn’t the main cause of a high agreement.
In Figure 2b, the meaning of the colors should be labelled.
“Utilitarianism emerges as the dominant framework,” Figure 2b does not show this. What it might show is that the aggregated performance across languages leans toward utilitarianism; however, it does not show that utilitarianism is a “dominant” framework, because each language individually leans toward different frameworks (with certain languages having extremely similar values for multiple frameworks). This conclusion is not valid.
“0.38-0.56 of item-weighted framework assignments” What is this metric? It is very unclear what is meant by this, and nowhere is item-weighted framework assignments defined.
“moral foundations predict permissibility in strongly language-specific ways, with only partial correspondence to established human cultural patterns.” Figures 3a and 3b are also measuring different things. Figure 3a shows the amount of morally-salient words that are used in the text, while 3b is not related to the morally-salient words used in human decisions because UniMoral does not contain human reasoning traces of a similar form to what the models generated.
In Figure 5 you should justify why you’ve chosen to not have the x and y axes be mirrored in language order
In A.4: “Chinese does not form a distinct source-language-centered cluster” Yes it does, in CMoralEval the chinese and english rows are closest then spanish is the next closest to this cluster, meaning that there is a source-language-centered cluster. In addition, these plots are simply showing the languages where the model is less likely to disagree with itself when switching language. This means that plotting results for UniMoral is invalid, because responses are different for different languages. This means that that disagreement is meaningless for that dataset. At the very least there should be a “disagreement baseline” that measures the disagreement in ground truth evaluations. Even then, it is a dubious metric.
Also in A.4: “Moreover, disagreement persists on natively authored UNIMORAL, where translation is absent.” Translation is not absent in UniMoral, it is just validated by humans.
In A.5: “Across all three subsets, we observe that translations of the same scenario cluster closely together regardless of language, while remaining distinct from other scenarios.” 10 samples per group is not statistically valid. The authors should either improve how they justify their subset selection and take a statistically valid sample of each (they must provide the total number of samples in each subset), or they should plot all of the samples.
In Figure 7: It is not detailed whether the color coding is applied ad-hoc or not. If it is not applied ad hoc, then a legend should be included. Despite this, the authors should still have some way to show that the groups are actually comprised of samples that are semantically the same (the legend could serve this function). Also, describe the basis/meaning of the hierarchical clustering shown in this figure.
“This indicates that the semantic content is preserved across translations” This doesn’t show what the first sentence claimed: “whether the automatic translation process influences the performance of LLM predictions across languages”
Response to reviewer
We thank the reviewer for their review. Below we address each weakness and answer the questions asked. Changes are highlighted in blue in the manuscript.
[UniMoral “natively authored” vs SeamlessM4T] We agree the UniMoral paper states that scenarios and action options were translated from English with SeamlessM4T-v2 and manually verified, while human judgments and questionnaire profiles were collected independently in each language. When our manuscript said “natively authored,” we meant “natively annotated”. The majority-vote decisions (and associated value ratings) are produced by native speakers of each language of L, not by translating an English gold label into the target language. Because decision selection is annotated natively, there is no translation error in the evaluation labels, the target of weighted F1 and of cross-lingual label/model disagreement. That is the property that distinguishes UniMoral from MEQ/ETHICS (and from non-Chinese CMoralEval), where the gold decision is fixed in a source language and travels with a translated item. We do not claim that Seamless stimuli are free of English framing; we claim that human decision selection on those items is not a translated label, so UniMoral still functions as our native-label control for RQ1 alignment analyses. Change in manuscript: We explicitly elaborate on UniMoral’s scenario and options being translated wherever it is mentioned in this context. Specifically, in Section 3, Section 5, Appendix A.1, and Appendix A.4.
[Parse failures, prediction distributions, answer-only prompts] We agree that differences in weighted F1 may reflect not only variation in moral judgments but also differences in language comprehension, instruction following, and adherence to the required JSON output format. To help disentangle these factors, we report the mean parsing success rates across all evaluation suites: Arabic (88%), Chinese (93%), English (97%), Hindi (85%), Russian (91%), and Spanish (94%). These relatively high rates suggest that formatting failures alone are unlikely to fully explain the observed cross-lingual performance differences, although they may still contribute. We additionally report the gold-label and model prediction distributions across languages (percentages), with predictions computed over successfully parsed responses:
While predictions are relatively balanced in English, Spanish, Chinese, and Russian, we observe a stronger tendency toward Label 2 in Arabic and particularly Hindi, despite Label 1 being more prevalent in the gold distributions. This suggests that systematic prediction biases may contribute to the lower performance observed in these languages, beyond parsing failures alone. Change in manuscript: We incorporated these distributions and the parsing statistics in Section 3 and a new Appendix A.4. Finally, we agree that answer-only prompting would provide a useful control for potential differences in reasoning and formatting requirements. We will aim to complete these additional experiments before the camera-ready submission and include the results if available.
[Generated explanations and LLM-as-judge validation] Our RQ2 analysis examines the moral frameworks and reasoning phases expressed in model-generated explanations, rather than attempting to recover the models' internal decision processes. We use a predefined taxonomy of 10 ethical frameworks and 15 reasoning phases to systematically compare these explanations across languages under controlled prompting conditions. The analysis focuses on Qwen-3.5-27B, with annotations produced by Llama-3.3-70B and preliminary human agreement assessments conducted on English (n=20) and Hindi (n=20) responses. Importantly, applying the same prompts and annotation framework across languages enables a controlled comparison of expressed reasoning patterns, even though generated explanations need not faithfully reflect internal computations. Our findings therefore concern the consistency of expressed moral reasoning across languages, rather than identical underlying decision mechanisms. Similarly, our characterization of predominantly Western-oriented reasoning refers specifically to the prevalence of frameworks captured by our predefined, Western-rooted taxonomy; it does not imply that models lack culturally specific moral concepts outside this taxonomy. Change in manuscript: Clarified the scope of RQ2's analysis of expressed reasoning and the limitations of LLM-as-judge annotations, and reported preliminary human validation results in Section 4.