- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This work studies whether the language in which an LLM’s prompt is written affects what decisions they make in tasks taken from cognitive science and behavioral economics experiments. Specifically, they compare behaviors between English and Chinese language prompts. They ran 11 scenarios with 6 LLMs, and just over half showed statistically significant differences between the languages, with the affect varying across LLMs. Around a quarter of trials showed differences that surpassed the large effect threshold that the authors specify.
Critical strengths
The paper’s focus is interesting and relevant to broad applications of LLMs. The finding that prompt language frequently affects decision making is useful to both users and developers of LLMs.
Critical weaknesses
(1) A paragraph in the paper’s appendix suggests that there may have been a critical methodological mistake, though I would like clarification from the authors on exactly what happened. The paragraph says:
“The merged JSONL does not retain a trusted mapping from each response letter to its semantic option. Reconstruction agreed with the displayed letter in 23,269 of 52,607 records with complete metadata, or 44.2%. The expected random-agreement rate under the observed scenario mix (seven two-choice and four three-choice) is approximately (7/11)(1/2)+(4/11)(1/3) = 43.9%, so the observed agreement sits only marginally above the random floor.”
It appears that this is saying that the authors lost track of the ground truth labels in their evaluation data, tried to reconstruct the ground truth, but the reconstruction performed as random. Is this a correct interpretation? Does this affect all of the data presented in the paper?
(2) Some of the wording may overstate the strength of the effect. While 37 of 66 comparisons had a statistically significant difference, only 15 of 66 were above the authors’ large effect threshold. Statistical significance appears to not be especially meaningful in this context because the authors collected a very large number of samples for each scenario, implying that very small differences in decision rates can be statistically significant. This point could be made clearer.
(3) The paper should provide the prompts for each scenario in both English and Chinese. A concern is that question framings or connotations might differ between the English and Chinese translations. Could the authors discuss and address to what extent this might affect the results?
(4) With only 11 scenarios, it’s difficult to draw broad conclusions about difference in decision making between languages.
(5) Some of the text in the figures is very small and should be made larger.
Questions for authors
I have no additional questions beyond those stated above.
4 comments
Thank you for the careful reading. Here is how the revision addresses each of your points.
(1) Appendix A.4. That interpretation is not correct. The ground-truth labels were not lost. Each letter was decoded online against that trial's rotated menu, and the semantic choice was logged with the raw response. All results use these logged labels. What failed was a later attempt to re-derive the rotation seed from merged fields whose trial counters had been reused, and the old paragraph described this poorly. Sec. 3.4 and App. A.4 now explain it and add two checks that do not depend on any mapping. In 85 of 264 cells, the model chose the same option in 200/200 trials while the displayed letters split. All 1,598 letter-plus-text echoes match the logged choice.
(2) Significance vs. effect size (Abstract, Intro, Sec. 3.4, 4.1, 5.2). We agree. The paper now states that per condition makes shifts as small as significant, and it reports effect-size tiers. 15/66 shifts are large (), 9 are medium, and the remaining 13 significant contrasts have .
(3) Bilingual prompts (App. A.5, Tables S3–S4; Sec. 5.1). All English and Chinese prompts, options, and scores are now given verbatim, together with the mixed-condition suffix. A single translation bias cannot explain the quantitative tasks, because the same prompt pair shifts different models in opposite directions. In Loss Risk, Kimi-K2.5 moves by and Claude Opus 4.6 by . In Trust Game, Kimi-K2.5 moves by and GLM-5.1 by . In Causal Attribution, where all six models shift the same way, wording differences such as clause order may contribute, and we say so.
(4) Eleven scenarios (Intro, Sec. 5.4, App. A.7). We agree. The title and claims are narrowed. The study shows that language invariance fails on these canonical probes. It does not support broad conclusions about English vs. Chinese decision-making.
(5) Figures. Figures 1–3 were regenerated at full text width with larger fonts.
We hope these changes resolve the label-integrity concern, which we understand to be your main soundness issue. We would be grateful if you would reconsider your score.
Thank you for the detailed explanations. I will increase my score to a 2.
Thank you very much for reading our clarifications and increasing your score to a 2. Since our revision addresses your earlier concerns, we would be deeply grateful if you could consider raising your score further, as every point makes a decisive difference for our submission. Please let us know if anything remains unresolved before discussion closes.
Point-to-Point Response to Reviewer
We thank the Reviewer for the constructive feedback and address each point below.
Comment [1]: Appendix A.4 & Ground-Truth Label Integrity
No—ground-truth semantic labels (
extracted_choiceandscore) were never lost or reconstructed post hoc, and none of the paper's data or results are affected. All tables, figures, and tests read the online-loggedextracted_choiceandscorefields directly from original experimental files.run_trial), each trial rotated the K∈{2,3} options by offset r∈{0,…,K−1}, mapped the returned letter (A/B/C) in memory toextracted_choiceandscore(∈{0,0.5,1}), and saved both alongsideraw_response.option_rotation(r), standardized model IDs (moonshotai/Kimi-K2.5→kimi-k2.5), and kept batch-local trial indices (0..99). A post-hoc script tried to re-predict r from post-merge strings (MD5(model_id_scenario_id_language_trial) mod K), matching 44.2% (23,269/52,607, vs. 43.9% random floor) because post-merge strings no longer matched pre-merge seeds. Unclear phrasing in Appendix A.4 made this failed seed-hash guess sound like a failed label reconstruction.extracted_choice) with displayed letter index idisp (raw_response) recovers the exact runtime shift r=(iorig−idisp)modK across all 52,607 records without any hash: (i) Balance: r splits 50.4%/49.6% in binary (n=33,510,V=0.016) and 33.5%/33.7%/32.8% in ternary tasks (n=19,097,V=0.019); (ii) Unanimous Cells: in 85/264 cells, a model chose the identical semantic option in 200/200 trials (100%) while displayed letters split evenly (Kimi-K2.5Loss RiskZH:"certain"200/200,A106/B94;Claude Opus 4.6Moral LuckEN:"intent_same"200/200,A61/B69/C70, p<10−60 under random mapping); (iii) Text Audit: in 223 Gain Risk trials echoing option text ("B. 战略2"vs."B. 战略1"),extracted_choicematched in 222/223 cases (99.6%).Comment [2]: Statistical Significance vs. Effect Size under Large N
We agree. With n≈200 trials/condition (400/comparison), high power makes small shifts (∣Δ∣=0.035,∣h∣=0.18) reach padj<0.05. In the Abstract, Intro, Sec. 3.4, Sec. 4.1, and Sec. 5.2, we decouple statistical significance from practical magnitude across all 66 cells:
Comment [3]: Verbatim Bilingual Prompts & Translation Connotations
We added Appendix A.5 (Tables S3–S4) with verbatim English and Chinese prompts, options, and scores for all 11 scenarios, and expanded Sec. 3.2 & Sec. 5.1:
Strategy 1/战略1), a fixed Chinese wording bias would push models in the same direction. Instead, on the exact same prompts, models flip in opposite directions (Loss Risk:Kimi-K2.5h=−3.02 vs.Claude Opus 4.6h=+3.02; Trust:Kimi-K2.5h=+0.89 vs.Claude Opus 4.6h=−0.82).ENvs.EN->ZH,ZHvs.ZH->EN): Holding input text 100% identical and varying only output language still triggers large shifts (Gemini 3.1 ProPublic Goods: 0.98 inENvs. 0.26 inEN->ZH; 13/46 non-boundary cells areCoupled).ZH), syntactic/lexical framing ("While..." vs. "虽然……但……", "most responsible" vs. "主要责任") interacts with pretraining priors (Sec. 5.1).Comment [4]: Generalizability from 11 Scenarios
We agree and will revise the Intro, Sec. 3.2, and Sec. 5.4 to frame our 11-scenario benchmark (4,800 trials/scenario; 52,800 total) as a diagnostic falsification test of cross-lingual invariance on canonical paradigms rather than a broad survey of "English vs. Chinese decision-making." Opposite reversals across models and boundary locking in 20/66 cells refute sweeping generalizations and show that consistency must be audited per task.
Comment [5]: Figure Text Legibility
We will regenerate Figures 1, 2, and 3 at full textwidth with font sizes increased by 25%–40% across axis labels, titles, and annotations/legends.