acceptodds
Under review as a conference paper at ICLR 2027

Language Shapes Decisions in Large Language Models

Abstract

Large language models are deployed globally as multilingual decision-makers under the assumption that language is a neutral doorway to the same underlying system. Although leading models often reach similar factual accuracy across languages, it remains unclear whether they make the same choices when the language changes. We tested this question with 52,800 controlled forced-choice trials covering six model families, eleven scenarios, and seven behavioral domains. Each model was compared with itself in matched English and Chinese conditions, and we also asked it to respond in a language different from the language of the scenario. Across 66 comparisons, 37 showed a statistically significant shift after Benjamini–Hochberg correction at , or 56.1% of all comparisons. The arcsine-transformed contrast reached . The direction of change depended on the model and sometimes reversed completely. In a loss-framed risk task, one model moved from complete risk tolerance in English to complete risk aversion in Chinese, while another model showed the opposite pattern. Agent attribution was the only domain with the same directional shift across all six models. When the requested answer language was crossed with the scenario language, mixed-language responses frequently departed from the monolingual envelope by more than , with marked suppression in some pairs and marked amplification in others. These results show that the distribution of observable decisions can change with language and support evaluating model behavior separately by language before deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multilingual Large Language Models, Behavioral Invariance, Language-Conditioned Decision Shifts, Behavioral Auditing, Linguistic Relativity, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qx2rOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This work studies whether the language in which an LLM’s prompt is written affects what decisions they make in tasks taken from cognitive science and behavioral economics experiments. Specifically, they compare behaviors between English and Chinese language prompts. They ran 11 scenarios with 6 LLMs, and just over half showed statistically significant differences between the languages, with the affect varying across LLMs. Around a quarter of trials showed differences that surpassed the large effect threshold that the authors specify.

Critical strengths

The paper’s focus is interesting and relevant to broad applications of LLMs. The finding that prompt language frequently affects decision making is useful to both users and developers of LLMs.

Critical weaknesses

(1) A paragraph in the paper’s appendix suggests that there may have been a critical methodological mistake, though I would like clarification from the authors on exactly what happened. The paragraph says:

“The merged JSONL does not retain a trusted mapping from each response letter to its semantic option. Reconstruction agreed with the displayed letter in 23,269 of 52,607 records with complete metadata, or 44.2%. The expected random-agreement rate under the observed scenario mix (seven two-choice and four three-choice) is approximately (7/11)(1/2)+(4/11)(1/3) = 43.9%, so the observed agreement sits only marginally above the random floor.”

It appears that this is saying that the authors lost track of the ground truth labels in their evaluation data, tried to reconstruct the ground truth, but the reconstruction performed as random. Is this a correct interpretation? Does this affect all of the data presented in the paper?

(2) Some of the wording may overstate the strength of the effect. While 37 of 66 comparisons had a statistically significant difference, only 15 of 66 were above the authors’ large effect threshold. Statistical significance appears to not be especially meaningful in this context because the authors collected a very large number of samples for each scenario, implying that very small differences in decision rates can be statistically significant. This point could be made clearer.

(3) The paper should provide the prompts for each scenario in both English and Chinese. A concern is that question framings or connotations might differ between the English and Chinese translations. Could the authors discuss and address to what extent this might affect the results?

(4) With only 11 scenarios, it’s difficult to draw broad conclusions about difference in decision making between languages.

(5) Some of the text in the figures is very small and should be made larger.

Questions for authors

I have no additional questions beyond those stated above.

4 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer

We thank the Reviewer for the constructive feedback and address each point below.

Comment [1]: Appendix A.4 & Ground-Truth Label Integrity

No—ground-truth semantic labels (extracted_choice and score) were never lost or reconstructed post hoc, and none of the paper's data or results are affected. All tables, figures, and tests read the online-logged extracted_choice and score fields directly from original experimental files.

  1. Online Resolution: At runtime (run_trial), each trial rotated the K∈{2,3}K \in \{2,3\} options by offset r∈{0,…,K−1}r \in \{0,\dots,K-1\}, mapped the returned letter (A/B/C) in memory to extracted_choice and score (∈{0,0.5,1}\in \{0,0.5,1\}), and saved both alongside raw_response.
  2. Why the Post-Hoc MD5 Check Was 44.2%: Merging batches omitted option_rotation (rr), standardized model IDs (moonshotai/Kimi-K2.5 →\to kimi-k2.5), and kept batch-local trial indices (0..99). A post-hoc script tried to re-predict rr from post-merge strings (MD5(model_id_scenario_id_language_trial) mod K), matching 44.2%44.2\% (23,269/52,60723{,}269/52{,}607, vs. 43.9%43.9\% random floor) because post-merge strings no longer matched pre-merge seeds. Unclear phrasing in Appendix A.4 made this failed seed-hash guess sound like a failed label reconstruction.
  3. Three Direct Proofs (Sec. 3.4 & Appendix A.4): Comparing canonical index iorigi_{\mathrm{orig}} (extracted_choice) with displayed letter index idispi_{\mathrm{disp}} (raw_response) recovers the exact runtime shift r=(iorig−idisp) mod Kr = (i_{\mathrm{orig}} - i_{\mathrm{disp}}) \bmod K across all 52,60752{,}607 records without any hash: (i) Balance: rr splits 50.4%/49.6%50.4\%/49.6\% in binary (n=33,510,V=0.016n=33{,}510, V=0.016) and 33.5%/33.7%/32.8%33.5\%/33.7\%/32.8\% in ternary tasks (n=19,097,V=0.019n=19{,}097, V=0.019); (ii) Unanimous Cells: in 85/264 cells, a model chose the identical semantic option in 200/200200/200 trials (100%100\%) while displayed letters split evenly (Kimi-K2.5 Loss Risk ZH: "certain" 200/200200/200, A 106106/B 9494; Claude Opus 4.6 Moral Luck EN: "intent_same" 200/200200/200, A 6161/B 6969/C 7070, p<10−60p < 10^{-60} under random mapping); (iii) Text Audit: in 223223 Gain Risk trials echoing option text ("B. 战略2" vs. "B. 战略1"), extracted_choice matched in 222/223222/223 cases (99.6%99.6\%).

Comment [2]: Statistical Significance vs. Effect Size under Large NN

We agree. With n≈200n \approx 200 trials/condition (400400/comparison), high power makes small shifts (∣Δ∣=0.035,∣h∣=0.18|\Delta|=0.035, |h|=0.18) reach padj<0.05p_{\text{adj}} < 0.05. In the Abstract, Intro, Sec. 3.4, Sec. 4.1, and Sec. 5.2, we decouple statistical significance from practical magnitude across all 6666 cells:

  • Large (∣h∣≥0.8|h| \ge 0.8): 15/6615/66 (22.7%22.7\%; 32.6%32.6\% of 4646 non-boundary), up to ∣h∣=3.02|h| = 3.02.
  • Medium (0.5≤∣h∣<0.80.5 \le |h| < 0.8): 9/669/66 (13.6%13.6\%; 19.6%19.6\% of non-boundary).
  • Small (0.2≤∣h∣<0.50.2 \le |h| < 0.5): 15/6615/66 (22.7%22.7\%; 1212 BH-significant).
  • Trivial/Null (∣h∣<0.2|h| < 0.2): 27/6627/66 (40.9%40.9\%, 2020 boundary + 77 non-boundary; 11 BH-significant).

Comment [3]: Verbatim Bilingual Prompts & Translation Connotations

We added Appendix A.5 (Tables S3–S4) with verbatim English and Chinese prompts, options, and scores for all 1111 scenarios, and expanded Sec. 3.2 & Sec. 5.1:

  1. Opposite Reversals on Identical Prompts: In the 6 quantitative tasks (Risk, Disease, Trust, Public Goods) with identical numbers and neutral labels (Strategy 1/战略1), a fixed Chinese wording bias would push models in the same direction. Instead, on the exact same prompts, models flip in opposite directions (Loss Risk: Kimi-K2.5 h=−3.02h=-3.02 vs. Claude Opus 4.6 h=+3.02h=+3.02; Trust: Kimi-K2.5 h=+0.89h=+0.89 vs. Claude Opus 4.6 h=−0.82h=-0.82).
  2. Zero-Translation Controls (EN vs. EN->ZH, ZH vs. ZH->EN): Holding input text 100% identical and varying only output language still triggers large shifts (Gemini 3.1 Pro Public Goods: 0.980.98 in EN vs. 0.260.26 in EN->ZH; 13/4613/46 non-boundary cells are Coupled).
  3. Corpus-Prior Interactions: Where all 6 models shift unidirectionally (Causal Attribution, h∈[+0.45,+1.50]h \in [+0.45, +1.50] in ZH), syntactic/lexical framing ("While..." vs. "虽然……但……", "most responsible" vs. "主要责任") interacts with pretraining priors (Sec. 5.1).

Comment [4]: Generalizability from 11 Scenarios

We agree and will revise the Intro, Sec. 3.2, and Sec. 5.4 to frame our 1111-scenario benchmark (4,8004{,}800 trials/scenario; 52,80052{,}800 total) as a diagnostic falsification test of cross-lingual invariance on canonical paradigms rather than a broad survey of "English vs. Chinese decision-making." Opposite reversals across models and boundary locking in 20/6620/66 cells refute sweeping generalizations and show that consistency must be audited per task.

Comment [5]: Figure Text Legibility

We will regenerate Figures 1, 2, and 3 at full textwidth with font sizes increased by 25%–40% across axis labels, titles, and annotations/legends.

Thank you for the careful reading. Here is how the revision addresses each of your points.

(1) Appendix A.4. That interpretation is not correct. The ground-truth labels were not lost. Each letter was decoded online against that trial's rotated menu, and the semantic choice was logged with the raw response. All results use these logged labels. What failed was a later attempt to re-derive the rotation seed from merged fields whose trial counters had been reused, and the old paragraph described this poorly. Sec. 3.4 and App. A.4 now explain it and add two checks that do not depend on any mapping. In 85 of 264 cells, the model chose the same option in 200/200 trials while the displayed letters split. All 1,598 letter-plus-text echoes match the logged choice.

(2) Significance vs. effect size (Abstract, Intro, Sec. 3.4, 4.1, 5.2). We agree. The paper now states that n≈200n\approx200 per condition makes shifts as small as ∣Δ∣=0.035|\Delta|=0.035 significant, and it reports effect-size tiers. 15/66 shifts are large (∣h∣≥0.8|h|\ge0.8), 9 are medium, and the remaining 13 significant contrasts have ∣h∣<0.5|h|<0.5.

(3) Bilingual prompts (App. A.5, Tables S3–S4; Sec. 5.1). All English and Chinese prompts, options, and scores are now given verbatim, together with the mixed-condition suffix. A single translation bias cannot explain the quantitative tasks, because the same prompt pair shifts different models in opposite directions. In Loss Risk, Kimi-K2.5 moves by −3.02-3.02 and Claude Opus 4.6 by +3.02+3.02. In Trust Game, Kimi-K2.5 moves by +0.89+0.89 and GLM-5.1 by −0.82-0.82. In Causal Attribution, where all six models shift the same way, wording differences such as clause order may contribute, and we say so.

(4) Eleven scenarios (Intro, Sec. 5.4, App. A.7). We agree. The title and claims are narrowed. The study shows that language invariance fails on these canonical probes. It does not support broad conclusions about English vs. Chinese decision-making.

(5) Figures. Figures 1–3 were regenerated at full text width with larger fonts.

We hope these changes resolve the label-integrity concern, which we understand to be your main soundness issue. We would be grateful if you would reconsider your score.

Thank you for the detailed explanations. I will increase my score to a 2.

Thank you very much for reading our clarifications and increasing your score to a 2. Since our revision addresses your earlier concerns, we would be deeply grateful if you could consider raising your score further, as every point makes a decisive difference for our submission. Please let us know if anything remains unresolved before discussion closes.

Reviewer_ocxsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper evaluates whether language changes the decisions made by a fixed language model. The paper studies six models, eleven scenarios across seven behavioral domains, and four conditions crossing English /Chinese scenario language with requested response language. the authors find the evidence for language-specific behavioral evaluation and the effects of language are heterogeneous across models.

Critical strengths
  1. The paper focus on crossed-language design, provide insights the effects of language on model decisions that monolingual comparisons miss.
  2. The paper reports results over a relatively comprehensive comparisons..
Critical weaknesses
  1. The experiment analysis is carried out based on records that match the reconstructed results. Of course, this doesn’t mean that the operation limits the accuracy of those results; rather, its accuracy remains unresolved.
  2. In order to evaluate the language effects robustly, the experiments provide more of a qualitative analysis regarding the sensitivity to specific prompt implementations. The language amount limitations, as well as the dependence on task scenarios, aren’t fully considered.
Questions for authors
  1. Can the authors provide exact semantic choice for every trial from raw prompts and responses?
  2. In the experiments, which effects survive independently authored paraphrases and alternative bilingual translations? For the mixed-language results, how many departures remain supported after accounting jointly for uncertainty in the mixed and monolingual means?
3 comments

We thank the Reviewer for the constructive feedback. Below is our point-to-point reply.

1. Critical Weakness 1: Reconstructed Records & Label Accuracy

No primary result uses reconstructed records; main analyses read all 52,613 online-logged choices (extracted_choice, score), verified on raw logs.

  1. Why Post-Hoc MD5 Was 44.2%: At runtime, workers rotated options by rt∈{0,…,K−1}r_t\in\{0,\dots,K-1\} and logged extracted_choice and score in memory. Merging omitted rtr_t, standardized model_id, and reset wave trial (0-99), so static MD5 re-hashing hit the random floor (23,269/52,607=44.2%23,269/52,607=44.2\% vs. 43.9%43.9\%). That 44.2% subset was only an auxiliary check (33/66 vs. 37/66 BH-sig.; 0/66 sign flips).
  2. Ground-Truth Proofs (Sec. 3.4, App. A.4): (i) 85/264 unanimous cells chose one option in 200/200 (100%) trials across evenly split letters (Kimi-K2.5 Loss Risk ZH: 200/200 "certain", A:106/B:94; Claude Moral Luck EN: A:61/B:69/C:70, p<10−60p<10^{-60}); (ii) 1,598/1,598 (100.0%) single-line letter+text echoes ("A/B. 战略1", "全部10美元") matched extracted_choice.

2. Critical Weakness 2: Prompt Sensitivity, Language Scope, & Scenarios

  1. Prompt Sensitivity: (i) Within-language option rotation (rtr_t) gives median ∣h∣=0.17|h|=0.17 (∣Δ∣=0.032|\Delta|=0.032) vs. cross-lingual informative median 0.540.54, mean 0.7750.775 (95% CI [0.569,0.999][0.569,0.999], max 3.023.02); (ii) Zero-translation controls (EN vs. EN->ZH; ZH vs. ZH->EN) hold text 100% verbatim while changing only answer language, yet yield mean ∣h∣=0.54|h|=0.54 (27/66 >0.50>0.50) and 0.510.51 (20/66 >0.50>0.50).
  2. Scenario Dependence: Two-way SS shares of signed hh (6666 cells) are 25.9% Scenario, 3.5% Model, and 70.6% Model ×\times Scenario (86.7% LMM variance; ∣h∣|h|: 25.3%, 16.3%, 58.4%), with 20/66 ceiling/floor and 15/66 (22.7%) large shifts (∣h∣≥0.8|h|\ge 0.8). The 11 tasks thus act as diagnostic falsification probes of task/model contingency (Sec. 4.1-4.2, 5.4).
  3. Language Scope (EN-ZH): Testing 2 languages traded breadth for 4-condition depth (52,800 trials isolating input vs. output language), so claims do not generalize to other language families or separate script/tokenization from alignment (Sec. 1, 5.4; EN-FR, ZH-JA, EN-AR in App. A.6).

3. Question 1: Exact Semantic Choice per Trial from Raw Prompts & Responses

Yes. The experimental logs extracted_choice, score, and raw_response for 52,613/52,800 (99.65%) trials (187 excluded; Table 2). Pairing verbatim templates (App. A.5) with canonical index i0i_0 and letter index idi_d recovers each trial's shift rt=(i0−id) mod Kr_t=(i_0-i_d)\bmod K across 52,607 records (K=2K=2: 50.4%/49.6%, V=0.016V=0.016; K=3K=3: 33.5%/33.7%/32.8%, V=0.019V=0.019) and menu options[r_t:]+options[:r_t]. Multi-line text re-parsing differs in 287/52,800 trials (0.54%, "Procedure A/B"), leaving all 15 large effects intact; we will release the 52,800-row table.

4. Question 2: Paraphrases/Translations & Mixed-Language Uncertainty

  1. Paraphrases & Translations: Each task used one back-translated template (Sec. 5.4), leaving multi-item banks to future work. Three contrasts test rewording/translation survival: (i) Parallel frames (8 tasks in 4 domains, including Gain/Loss rewordings of identical payoffs): shifts persist (GPT-5.4 Disease: h=+0.61,+0.79h=+0.61,+0.79; Kimi-K2.5 Disease: +2.63,+1.48+2.63,+1.48, Temporal: +0.61,+0.59+0.61,+0.59; Qwen Temporal: +0.49,+0.48+0.49,+0.48), with cross-lingual ∣h∣>|h|> within-language Gain/Loss ∣h∣|h| in 4/6 models on Disease, while Kimi-K2.5 Risk flips sign (+0.93+0.93 vs. −3.02-3.02); (ii) Opposite reversals on identical translations (Loss Risk: Kimi-K2.5 −3.02-3.02 vs. Claude +3.02+3.02; Trust: +0.89+0.89 vs. −0.82-0.82) rule out a fixed slant; (iii) Zero-translation shifts (Gemini Public Goods: 0.98→0.260.98\to 0.26 on identical English text). Causal Attribution (6/6 models, h∈[−2.59,−0.32]h\in[-2.59,-0.32]; t(5)=−3.05,p=0.0285t(5)=-3.05, p=0.0285) is prompt-conditional (Sec. 4.3, 5.1).
  2. Mixed-Language Uncertainty: Across 29 informative pairs (∣Δ∣>0.10|\Delta|>0.10, 58 mixed cells), we accounted jointly for uncertainty in McM_c and (MEN,MZH)(M_{EN},M_{ZH}) via 95% Joint Bootstrap CIs (B=10,000B=10,000) on Mc−Mmin⁡M_c-M_{\min}, Mc−Mmax⁡M_c-M_{\max} and Intersection-Union Tests (IUT) (H0:μc∈[μmin⁡,μmax⁡]H_0:\mu_c\in[\mu_{\min},\mu_{\max}], pIUT=max⁡(p(Yc≷YEN),p(Yc≷YZH))p_{IUT}=\max(p(Y_c\gtrless Y_{EN}),p(Y_c\gtrless Y_{ZH})); BH across 58 cells, α=0.05\alpha=0.05). Result: Both identify the same 17/29 pairs (58.6%)—11 below, 6 above—and 29/58 cells (50.0%) (pBH<0.05p_{BH}<0.05, joint CIs excluding 0), vs. 20/29 (33/58) under δ=0.02\delta=0.02 (4 cells of 0.0230.023-0.0660.066 cross zero), e.g., GLM-5.1 Public Goods ZH->EN (−0.140,[−0.190,−0.095],pBH=3.8×10−8-0.140, [-0.190,-0.095], p_{BH}=3.8\times 10^{-8}), Kimi-K2.5 Public Goods ZH->EN (−0.438,[−0.512,−0.365],pBH=2.7×10−23-0.438, [-0.512,-0.365], p_{BH}=2.7\times 10^{-23}), and GPT-5.4 Vertical Time EN->ZH (+0.482,[+0.412,+0.553],pBH=4.2×10−31+0.482, [+0.412,+0.553], p_{BH}=4.2\times 10^{-31}).

Thank you for the review. Here is how the revision answers your questions.

1. Reconstructed records (Sec. 3.4, App. A.4). All analyses use the 52,613 semantic choices that were decoded online and logged with the raw responses. None use the reconstructed hash mapping. The hash subset (44.2%) is reported only as an auxiliary check: 35/66 comparisons are significant, and the one sign change occurs in a non-significant cell. Two raw-log checks are independent of the mapping. In 85 of 264 cells, the model chose one option in all 200 trials across split letters. All 1,598 single-line letter-plus-text echoes match the logged choice.

2. Exact semantic choice per trial. The semantic choice, score, and raw response are logged for all 52,613 parsed trials, and we will release the full 52,800-row table. The displayed option order can only be derived from the logged choice, because the offset was not stored. We state this limitation explicitly.

3. Paraphrases and translations (Sec. 4.5, 5.4). No effect has been tested under independently authored paraphrases or alternative translations, because each scenario uses one back-translated template. We added the evidence that is available:

  • Option-order variation is about one third of the language effect (median ∣h∣|h| 0.16 vs. 0.54).
  • Same-input contrasts still show large shifts (mean ∣h∣|h| 0.54 / 0.51), although they also change the reasoning instruction.
  • Effects persist across the paired gain/loss vignettes for several models and reverse sign for others.

4. Scenario dependence and language scope (Sec. 4.1, 5.4). Most variance is model × scenario specific (70.6%), and 20 of 66 cells are at ceiling or floor. The suite is therefore framed as a diagnostic test of language invariance, not a general survey. EN–ZH is the only pair tested, and extensions to other language pairs are listed in App. A.6.

5. Mixed-language uncertainty (Sec. 4.4, App. A.3). After accounting jointly for uncertainty in the mixed and monolingual means, 17 of 29 pairs and 29 of 58 cells remain supported under both joint bootstrap CIs and BH-corrected IUT.

We hope the revision addresses both of your soundness concerns, and we would be grateful if you would consider raising your score.

Thank you again for your constructive review. As discussion ends shortly, we wanted to ask if our clarifications on label reliability and study scope address your concerns. A score increase would mean a great deal to our paper at this stage, and we would sincerely appreciate it if you could raise your rating.

Reviewer_uv5ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether the language of a decision-making prompt changes the choices large language models make. The authors test six model families on eleven forced-choice scenarios across seven domains: causal attribution, temporal reasoning, risk preference, framing, cooperation, moral judgment, and counterfactual reasoning. Over 52,800 sampled trials, they compare matched English and Chinese prompts and add two mixed conditions where the response language differs from the scenario language. After multiple-testing correction, 37 of 66 model–scenario pairs show significant English–Chinese differences. Some are large, including risk preferences that reverse between models, and causal attribution shifts in the same direction for all six models. Mixed-language outcomes sometimes fall outside the range of both monolingual conditions. The authors conclude that decisions can depend heavily on prompt language and recommend language-specific behavioral evaluation before multilingual deployment. Sensitivity analyses cover response parsing, option position, and boundary effects.

Critical strengths
  1. Broad comparison across models and domains. The paper addresses a practical question: do multilingual models make the same decisions when a scenario is presented in different languages? Six model families, eleven scenarios, and seven domains give useful coverage. The mixed-language conditions are especially valuable because they reveal variability that a plain English–Chinese comparison would miss.

  2. Fine-grained analysis of language effects. Instead of aggregate scores, the authors report the size and direction of each model–scenario effect. With multiple-testing correction and sensitivity checks, the finding that models shift in opposite directions shows that consistent factual accuracy across languages does not imply consistent decisions.

Critical weaknesses
  1. The outcome measure is not validated. Appendix A.4 admits that the records lack a reliable mapping from response letters to semantic options. The reconstructed mapping agrees with the displayed letter in only 44.2% of 52,607 records, compared with roughly 43.9% expected by chance. Because the scores depend on which option was chosen, errors here could drive the reported language contrasts. The position-checked subset, where 33 of 66 comparisons remain significant, is selected with the same flawed reconstruction, so it cannot validate the pipeline. A proper audit would recover each trial’s displayed option order and check the chosen option against the raw response. Without that, the reported prevalence, effect sizes, and reversals are hard to trust.

  2. Language effects are not separated from prompt artifacts. The 52,800 trials are repeated samples from just eleven prompt pairs. That gives precise estimates for each fixed prompt but says nothing about other wordings of the same problem. Back-translation confirms that meaning is preserved, but not framing or pragmatics, which matter for risk, moral, temporal, and social scenarios. Switching language also changes tokenization, word choice, and cultural associations together. The Benjamini–Hochberg correction handles multiple testing but not the lack of variation at the scenario level. The evidence therefore shows differences between specific translated prompts, not that language shapes decisions in general. Several independent paraphrases per domain, with uncertainty estimated at the prompt level, would be needed for the broader claim.

Questions for authors
  1. The reconstructed letter-to-option mapping agrees with displayed letters at roughly chance level (44.2% vs. 43.9%). How were the semantic scores verified, and can each trial’s option order and chosen option be recovered from the raw outputs?

  2. Do the effects hold under independent translations or paraphrases, with uncertainty estimated at the prompt level rather than over repeated samples?

  3. Mixed-language departures use a fixed margin (δ = 0.02) around sampled monolingual means. Can the authors report confidence intervals or formal tests for them?

3 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer

We thank the reviewer for the rigorous evaluation and address each point below with new audits and formal tests.


1. Outcome Validation & Option-Order Recovery (Weakness 1 & Q1)

The 44.2%44.2\% rate in Appendix A.4 reflects an offline post-hoc seed mismatch across merged execution waves, not an error in the stored choices (extracted_choice) or scores (YY).

  1. Why 44.2%44.2\% occurred: In format_prompt(), each trial’s rotated options and mapping πt\pi_t were generated together, and extracted_choice =πt(L)= \pi_t(L) and YY were decoded online at runtime before saving. However, the 200 trials per cell were merged from 2–4 execution waves where within-wave trial indices reset (0–490\text{--}49) without logging rtr_t or the wave seed. Re-evaluating one un-batched MD5 hash post-hoc across merged waves compared one sequence against four wave schedules, matching at chance (44.2%44.2\% vs. 43.9%43.9\%). Within each wave, rotation is 100% (50/50) deterministic.
  2. Runtime option-order recovery (n=52,607n = 52{,}607): Inverting the runtime cyclic shift recovers every trial’s exact rotation rt=(idx(choice)−idx(L)) mod Kr_t = (\text{idx}(\text{choice}) - \text{idx}(L)) \bmod K, confirming balanced option orders across conditions (K=2K=2: χ2(3)=8.55,V=0.016\chi^2(3)=8.55, V = 0.016; K=3K=3: χ2(6)=14.10,V=0.019\chi^2(6)=14.10, V = 0.019).
  3. Direct check against raw outputs: In 1,522/1,5221{,}522 / 1{,}522 (100.0%100.0\%) of single-line responses echoing both letter and option text without embedded A/B tokens (e.g., "A. 战略1", "B. 战略1"), extracted_choice matches the literal option text across rotated slots (e.g., Kimi-K2.5 Loss Risk ZH: 106 "A. 战略1" and 94 "B. 战略1" trials all decoded to certain, MZH=0.00M_{\text{ZH}}=0.00). Across all 52,80052{,}800 trials, only 287287 (0.54%0.54\%) differed between line-1 text and the trailing-letter parser (due to "程序A/B" in explanations); overriding all 287 leaves all 15 large effects (∣h∣>0.8|h| > 0.8) and all reversals (∣h∣=3.02|h| = 3.02) unchanged (35/6635/66 full-grid and 37/4437/44 informative cells remain BH-significant). We updated Appendix A.4 with this audit.

2. Prompt-Level Uncertainty, Paraphrases, & Claim Scope (Weakness 2 & Q2)

We agree that N=200N = 200 replicates measure within-prompt precision rather than cross-paraphrase generalizability, and revised our inference and claim scope:

  1. Prompt/cell-level uncertainty (n=1n = 1 per cell): Collapsing 200 replicates into one cell mean Yˉm,s,c\bar{Y}_{m,s,c} (n=66n = 66; n=46n = 46 informative) removes replicate inflation:
    • Cell-cluster bootstrap (10,00010{,}000 draws, n=46n=46) yields mean ∣Δ∣=0.268|\Delta| = 0.268 (95% CI [0.196,0.349][0.196, 0.349]) and mean ∣h∣=0.775|h| = 0.775 (95% CI [0.569,0.999][0.569, 0.999]).
    • In Causal Attribution (n=6n = 6 cell means), the Chinese shift remains significant at the cell level (Δˉ=−0.409\bar{\Delta} = -0.409, paired t(5)=−3.05,p=0.0285t(5) = -3.05, p = 0.0285; Wilcoxon p=0.0312p = 0.0312).
    • A random-effects LMM on signed hh shows scenario main effects explain 13.3% of variance (σscen2=0.110\sigma^2_{\text{scen}} = 0.110), whereas the model ×\times prompt interaction explains 86.7% (σres2=0.719\sigma^2_{\text{res}} = 0.719).
  2. Cross-instantiation contrast & scope: Four domains include two distinct canonical vignettes (Risk vs. Disease; Trust vs. Public Goods; Horizontal vs. Vertical Time). Some models shift consistently across parallel vignettes (GPT-5.4: h=+1.04h = +1.04 vs. +0.79+0.79), whereas others reverse direction in the same domain (Kimi-K2.5: h=−3.02h = -3.02 vs. +1.48+1.48). We retitled the paper and scoped claims to conditional instability on matched canonical probes rather than a paraphrase-invariant language prior.

3. Formal Tests & CIs for Mixed-Language Departures (Q3)

We replaced the δ=0.02\delta = 0.02 margin across all 29 informative pairs (∣Δ∣>0.10|\Delta| > 0.10, 58 mixed-language cells) with two formal tests (Section 3.3 & Table 3):

  1. 95% Joint Bootstrap CIs (B=10,000B = 10{,}000): Resampling (EN,ZH,EN→ZH,ZH→EN)(\text{EN}, \text{ZH}, \text{EN}\to\text{ZH}, \text{ZH}\to\text{EN}) jointly to capture variance in both McM_c and the order statistics Mmin⁡,Mmax⁡M_{\min}, M_{\max} for Mc−Mmin⁡M_c - M_{\min} and Mc−Mmax⁡M_c - M_{\max}.
  2. Intersection–Union Tests (IUT): Testing H0:μc∈[μmin⁡,μmax⁡]H_0: \mu_c \in [\mu_{\min}, \mu_{\max}] via pIUT=max⁡(p(Yc≷YEN),p(Yc≷YZH))p_{\text{IUT}} = \max(p(Y_c \gtrless Y_{\text{EN}}), p(Y_c \gtrless Y_{\text{ZH}})), BH-corrected across all 58 cells (α=0.05\alpha = 0.05).

Result: Both methods identify the exact same 17 of 29 pairs (58.6%58.6\%)—11 below the envelope and 6 above it (29/58 cells)—as significant (pBH<0.05p_{\text{BH}} < 0.05) with 95% joint CIs excluding zero (vs. 20/29 under δ=0.02\delta = 0.02; only 3 borderline cases of 0.030–0.0660.030\text{--}0.066 cross zero), e.g., GLM-5.1 Public Goods (−0.140-0.140, CI [−0.190,−0.095],pBH=3.8×10−8[-0.190, -0.095], p_{\text{BH}} = 3.8\times 10^{-8}), Kimi-K2.5 Public Goods (−0.438-0.438, CI [−0.510,−0.365],pBH=2.7×10−23[-0.510, -0.365], p_{\text{BH}} = 2.7\times 10^{-23}), and GPT-5.4 Vertical Time (+0.482+0.482, CI [+0.412,+0.553],pBH=6.7×10−27[+0.412, +0.553], p_{\text{BH}} = 6.7\times 10^{-27}).

Thank you for the rigorous review. Here is how the revision addresses your two main concerns and Q3.

1. Outcome validation (Sec. 3.2, 3.4, App. A.4). The scores were not derived from the reconstructed mapping. Each letter was decoded online against that trial's rotated menu, and the semantic choice and score were logged with the raw response. You asked whether each trial's option order can be recovered. The rotation offset was not logged, so the order can only be derived from the logged choice. The paper reports this derivation as a consistency check, not as validation. Two checks are independent of the mapping:

  • 85 of 264 cells are unanimous over 200 trials while the displayed letters split across positions (p<10−60p<10^{-60} if decoding had ignored the rotation).
  • 1,598 of 1,598 single-line letter-plus-text echoes match the logged choice.

The 44.2% figure comes from re-hashing merged fields whose batch-local trial counters were reused. On that subset, 35/66 comparisons are BH-significant, and one non-significant cell changes sign. A parser re-scoring of 138 ambiguous trials leaves all 15 large effects and both ∣h∣=3.02|h|=3.02 reversals unchanged. Two Kimi-K2.5 temporal effects lose significance, and we report this as a sensitivity result.

2. Prompt-level uncertainty (Sec. 4.1, 4.3, 4.5, 5.4). We agree that 200 replicates measure within-prompt precision, and we did not test independent paraphrases or translations. We therefore narrowed the claim, including in the title, to falsifying invariance on matched canonical probes. We also report the following:

  • Treating each cell mean as one observation, mean ∣h∣=0.775|h|=0.775 (95% CI [0.57, 1.00]).
  • The Causal Attribution shift holds across the six models (t(5)=−3.05t(5)=-3.05, p=0.029p=0.029).
  • 70.6% of the signed-hh sum of squares lies in the Model × Scenario interaction.

3. Formal tests for mixed-language departures (Sec. 4.4, App. A.3, Table S2). We added 95% joint bootstrap CIs that resample all four conditions, and BH-corrected intersection–union tests across the 58 mixed cells. Both methods support the same 17 of 29 pairs (11 below, 6 above) and 29 of 58 cells. Four small δ-departures (0.023–0.066) are not supported.

We believe the measurement concern is now resolved and the claims match the evidence. We would appreciate it if you would reconsider your soundness and overall scores.

Thank you again for your detailed review. With the discussion period closing soon, we wanted to check whether our response resolved your soundness and statistical-testing concerns. If so, raising your score would make a critical difference to our submission, and we would be deeply grateful.

Reviewer_js1kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether LLMs make the same decisions when a task is presented in different languages. The authors test six models on eleven forced-choice scenarios across seven behavioral domains, including risk, cooperation, and moral judgment. They compare English and Chinese prompts, and also test what happens when the requested answer language differs from the scenario language. The authors run 52,800 trials and find significant differences in 37 of 66 English–Chinese comparisons. The direction of these differences varies across models, and in some cases reverses completely. They also find that mixed-language conditions can lead to choices outside the range observed in the two monolingual settings. The main finding is that a model's decisions are not always consistent across languages.

Critical strengths
  1. A controlled comparison across models and languages. I liked that the authors compare each model with itself rather than comparing different models in different languages. They also cover several types of decisions instead of focusing on one task. The results show significant differences in 37 of 66 comparisons, even after correcting for multiple tests. I think this provides useful evidence that changing the language of a prompt can affect a model's choices.

  2. Looking at the direction of changes and mixed-language conditions. One of the more interesting findings is that the same language change can have opposite effects on different models. For example, in the loss-framed risk task, Claude and Kimi move in completely opposite directions (Section 4.2). I also liked that the authors test scenarios where the input and requested answer languages differ. These experiments show that multilingual behavior can be more complicated than simply following the language of the input.

Critical weaknesses

The results may depend too much on the specific wording of the scenarios. Although the paper includes 52,800 trials, these are repeated evaluations of only eleven fixed scenarios in English and Chinese. The authors check the translations, which is helpful, but they do not test alternative phrasings or compare different wordings within the same language. Since previous work has already shown that model decisions can vary across languages, I think it is important to check whether these differences hold beyond the specific prompts used. A comparison with alternative phrasings would make the findings more convincing and help show how well they generalize.

Questions for authors
  1. Could the authors compare the language effects with the variation caused by different phrasings of the same scenarios within one language? This would help show whether the findings are specific to changing languages.

  2. Appendix A.4 states that the merged data do not retain a trusted mapping between response letters and their corresponding options. The authors provide a sensitivity check with similar results, which is helpful, but it does not directly verify the original scoring. Since the main results depend on this mapping, I would have liked a clearer explanation of how the choices were scored and how the mapping was checked. Could the authors clarify how response letters were mapped to semantic choices after rotating the options, and how the original scoring was validated?

  3. Could the authors provide a deeper analysis of the patterns across scenarios and decision domains? For example, Table 3 shows that mixed-language conditions substantially reduce Public Goods scores for several models, even when the two monolingual conditions are similar. It would be interesting to understand whether such patterns are specific to certain tasks and what might explain them.

5 comments

We thank the Reviewer for the constructive feedback. Below is our point-to-point reply.

Comment [1] — Critical Weakness: Scenario Wording Sensitivity

  1. Opposite-Polarity Reversals on Identical Prompts: Wording bias would pull all models in one direction, yet identical prompts induce opposite shifts across models (Loss Risk: Kimi-K2.5 h=−3.02h=-3.02 vs. Claude Opus 4.6 h=+3.02h=+3.02; Trust Game: Kimi-K2.5 h=+0.89h=+0.89 vs. GLM-5.1 h=−0.82h=-0.82; Intertemporal: Qwen3.6-plus h=+1.77h=+1.77 vs. Claude Opus 4.6 h=−0.35h=-0.35).
  2. Zero-Translation Control (EN vs. EN->ZH; ZH vs. ZH->EN): Keeping scenario and option text 100% verbatim identical while switching only output language yields mean ∣h∣=0.54|h|=0.54 (27/6627/66 cells >0.50>0.50) and ∣h∣=0.51|h|=0.51 (20/6620/66 cells >0.50>0.50).
  3. Revision: We will add to Sections 1 & 4.5, with a paraphrase variance decomposition (σlang2\sigma^2_{\text{lang}} vs. σpara2\sigma^2_{\text{para}}) in Section 5.5 & Appendix A.5.

Comment [2] — Question 1: Cross-Lingual vs. Within-Language Variation

  1. Vs. Within-Language Gain/Loss Rephrasing: Asian Disease and Monetary Risk reword the same payoff matrix in Gain vs. Loss terms. In 4/6 models on Asian Disease, switching language under a fixed frame yields a larger ∣h∣|h| than within-language Gain/Loss rewording: Kimi-K2.5 (cross-lingual ∣h∣=2.63,1.48|h|=2.63, 1.48 vs. within-language 0.00,1.150.00, 1.15), GPT-5.4 (0.61,0.790.61, 0.79 vs. 0.35,0.540.35, 0.54), GLM-5.1 (0.410.41 vs. ≤0.31\le 0.31), and Qwen3.6-plus (0.430.43 vs. ≤0.25\le 0.25). In Monetary Risk, Claude Opus 4.6 is invariant in EN (h=0.00h=0.00) yet flips in ZH (h=+3.02h=+3.02); Kimi-K2.5 inverts its framing sign (+3.02+3.02 in EN vs. −0.93-0.93 in ZH).
  2. Vs. Surface-Order Variants (3.2×3.2\times): Across rotated option orders (A/B/C) within each language, median ∣h∣=0.17|h|=0.17 (∣Δ∣=0.032|\Delta|=0.032) vs. cross-lingual median ∣h∣=0.54|h|=0.54 (∣Δ∣=0.220|\Delta|=0.220, mean ∣h∣=0.77|h|=0.77, max 3.023.02).
  3. Revision: We will add to Section 4.5.

Comment [3] — Question 2: Option-Rotation Mapping & Scoring Validation (Appendix A.4)

  1. Online Scoring vs. Post-Merge Indexing: De-rotation and scoring ran online in memory during collection, not post hoc. At runtime (run_trial), each worker shifted options by r=MD5(model, scenario, lang, trial) mod Kr = \text{MD5}(\text{model, scenario, lang, trial}) \bmod K, held in memory, and saved extracted_choice and score to disk. Merging concurrent ThreadPoolExecutor workers re-indexed trial (0..1990..199) without saving MtrialM_{\text{trial}}, so post-hoc trial re-hashing matches at the random floor (44.2%44.2\% vs. 43.9%43.9\%).
  2. Three Direct Audits: (i) Score Integrity: extracted_choice matches score in 52,613/52,613 (100.00%) valid trials; (ii) Verbatim Text Audit: when responses echoed both letter and option text (e.g., "A/B/C. 全部10美元"), extracted_choice matches the verbatim option in 1,510/1,510 (100.00%) direct responses across all 8 non-aliased scenarios and 545/575 (94.8%) CoT restatements (re-scoring 151/9,600 Disease "程序A" echoes leaves all 66 BH decisions/signs unchanged); (iii) Hash-Matched Subset (n=23,269n=23,269): retains 33/6633/66 BH-significant shifts with 0/66 sign flips.
  3. Revision: We will add clarify in Section 3.2 & Appendix A.4.

Comment [4] — Question 3: Domain Patterns & Mixed-Language Collapse in Public Goods

  1. Why Public Goods & Trust Game Collapse: Monolingual (EN, ZH) prompts elicit 1-letter answers (mean 1.8 chars) following cooperative norms (0.74,0.620.74, 0.62). Mixed-language (EN->ZH, ZH->EN) prompts trigger explicit deliberation (mean 692 and 1,096 chars): all 5 collapsing models (Gemini 3.1 Pro, GPT-5.4, GLM-5.1, Kimi-K2.5, Qwen3.6-plus) compute MPCR=1.5/10=0.15<1\text{MPCR} = 1.5/10 = 0.15 < 1 ($-$0.85$ loss/$$1$) and choose the $$0$ Nash equilibrium (0.25,0.190.25, 0.19). Backward induction similarly drops Trust Game (0.30/0.28→0.12/0.140.30/0.28 \to 0.12/0.14). Natural Control: Claude Opus 4.6 is the only model without collapse (Public Goods 1.001.00, Trust Game 0.500.50) and the only model suppressing CoT ("FINAL ANSWER: A", 7–217\text{--}21 chars).
  2. Four-Regime Taxonomy: (i) Below-Envelope Collapse (Public Goods/Trust Game: MPCR=0.15\text{MPCR}=0.15 & backward induction drive $$0$ Nash play, 0.74∣0.62→0.25∣0.190.74\mid 0.62 \to \mathbf{0.25\mid 0.19}); (ii) Above-Envelope Amplification (Time Metaphor/Vertical Time: timeline rules raise means above baselines, 0.80∣0.87→0.96∣0.960.80\mid 0.87 \to \mathbf{0.96\mid 0.96}; 0.59∣0.71→0.87∣0.940.59\mid 0.71 \to \mathbf{0.87\mid 0.94}); (iii) EV Attenuation (Loss Risk/Disease: E[A]=E[B]E[A]=E[B] dampens risk-seeking, 0.35∣0.42→0.23∣0.170.35\mid 0.42 \to \mathbf{0.23\mid 0.17}); (iv) Input-Language Tracking (Causal Attribution: 6/66/6 models shift in ZH, 0.98→0.570.98\to 0.57; mixed outputs track input language, 0.88,0.510.88, 0.51).
  3. Revision: We will add to Sections 4.4 & 5.3.

Thank you for the constructive review. The revision addresses your three questions as follows.

1. Wording sensitivity (Sec. 4.5, 5.4). We compare the language effect with three internal sources of prompt variation:

  • Splitting each cell by option order gives median ∣h∣=0.16|h|=0.16, against 0.540.54 for informative EN–ZH contrasts.
  • Same-input contrasts keep the scenario text verbatim and change the answer language together with the reasoning instruction. They give mean ∣h∣=0.54|h|=0.54 and 0.510.51.
  • For gain/loss rewording of identical payoffs, the EN–ZH shift exceeds the within-language framing shift in 4 of 6 models on Disease. On Risk, the framing shift is at least as large as the EN–ZH shift in all six models.

We did not test independently authored paraphrases. The claims are therefore scoped to these prompts (title, abstract, Sec. 5.4, App. A.7), and the paraphrase decomposition is listed as the most important extension (App. A.6).

2. Letter-to-option mapping (Sec. 3.2, 3.4, App. A.4). Each trial's letter was decoded in memory against that trial's rotated menu, and the semantic choice and score were logged with the raw response. The rotation offset itself was not logged. Two checks do not depend on the logged mapping:

  • In 85 of 264 cells, the model chose one option in 200/200 trials while the displayed letters split (e.g., 106 A / 94 B).
  • All 1,598 single-line responses that echo both a letter and the option text agree with the logged choice.

The 44.2% figure comes from re-hashing merged fields after batch-local trial counters were reused. It reflects lost seeds, not lost labels. Re-scoring 138 parser-ambiguous trials leaves all 15 large effects unchanged.

3. Domain patterns (Sec. 4.4, 5.3). In monolingual Public Goods, models mostly answer with a bare letter (pooled means 0.74 EN / 0.62 ZH). Under the mixed-language instruction, the five non-Claude models write 646–1,081-character derivations, and 66–100% of responses per model compute the per-capita return 0.15<10.15<1. Scores fall to 0.25 / 0.19. Claude Opus 4.6 returns only the answer line and stays at 1.00. Across domains we find four patterns: collapse in social dilemmas, amplification in temporal mapping, attenuation in Loss Disease, and input-language tracking in Causal Attribution. We state that the requested language cannot be separated from the reasoning instruction.

You rated the work as clear and the core finding as useful. If the revision resolves your soundness concerns, we would be grateful if you would consider maintaining or even raising your score.

Thank you again for your supportive and constructive review. As the discussion period closes soon, we wanted to check whether our clarifications on phrasing sensitivity, label mapping, and domain patterns addressed your questions. If your remaining concerns are now resolved, we would be deeply grateful if you could consider raising your score or supporting our paper in the discussion, as every bit of support makes a critical difference at this stage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.