- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates which characteristics of non-native English prompts affect LLM responses. It introduces FABLE, constructed by translating Chinese conversational prompts into English variants representing different ESL profiles, and evaluates multiple open-weight models. The analysis relates five dimensions of prompt fluency to response fluency and quality. The authors report that surface errors are generally not propagated, whereas some higher-level linguistic characteristics are reflected in responses, alongside small reductions in response quality associated with lower prompt fluency.
Critical strengths
-
The paper addresses a relevant question about how language variation affects access to LLM capabilities. Examining individual linguistic dimensions offers a more informative perspective than treating native versus non-native English as a binary distinction.
-
The study combines conversational prompts from WildChat and ShareLM with systematically generated variants and evaluates multiple models. This provides a useful framework for investigating whether different prompt characteristics affect response language and task performance differently.
Critical weaknesses
-
The manipulation does not clearly separate linguistic fluency from preservation of task information. The Content rubric includes vagueness, missing information, and off-topic content, so lower scores may reflect changes in task specification rather than language proficiency alone. Since Content contributes substantially to the reported effects, pooled semantic-preservation validation is insufficient to rule out this confound. This weakens the interpretation of the observed quality penalty as an effect of non-native English.
-
The quality evaluation and statistical analysis do not sufficiently establish the claimed effects. Responses are judged against a clean reference translation rather than the actual input, making it important to distinguish success on the original intended task from faithful execution of the supplied prompt. The small reported effects also require uncertainty estimates accounting for repeated variants and models, together with evidence about judge variability. Finally, the reported negative coefficients for some dimensions need reconciliation with the broad mirroring claim.
Questions for authors
-
Can you report semantic-preservation results separately by fluency level, especially for low-Content prompts? How often do these variants omit or change constraints present in the reference translation?
-
Does the quality evaluation measure success on the original intended task or compliance with the actual prompt supplied to the model? How sensitive are the conclusions to this distinction and to using an independent judge outside the evaluated model panel?
-
Please reconcile the total prompt pool, language-specific subsets, variants per source prompt, filtering stages, and the final 190,911 variants. What role do the English prompts play?
-
Which models generated the responses used in Figures 3 and 5? Please reconcile the model names in the appendix with the reported 34-model evaluation panel.
-
How do the negative associations for Language Use and Mechanics support the mirroring conclusion? How does dropping Organization because of multicollinearity affect interpretation and comparisons across analyses?
-
Do the uncertainty estimates account for repeated observations sharing a base prompt or model? Can you provide confidence intervals for the predicted quality differences and assess their magnitude relative to judge variability?
2 comments
Response to Reviewer mBNk (Part 2)
- “The reported negative coefficients for some dimensions need reconciliation with the broad mirroring claim.”
- We do discuss the negative coefficients for the Language Use and Mechanics dimensions, explicitly stating “Language Use and Mechanics exhibit negative associations” (section 5.2.1).
- “Figure 3 shows that “higher levels of Content, Organization, and Vocabulary show [statistically significant] positive associations with response proficiency, demonstrating that prompt fluency does pattern with response fluency.
- “Can you report semantic-preservation results separately by fluency level, especially for low-Content prompts?”: Appendix A.3 reports that “the Spearman correlation between the average instructed [prompt fluency] level and similarity [of the resulting translated prompt variants] to the original Chinese prompt is 0.06.” In other words, the fluency level of the prompt is not correlated with any semantic changes, as measured by embedding similarity.
- “Does the quality evaluation measure success on the original intended task or compliance with the actual prompt supplied to the model? How sensitive are the conclusions to this distinction [ . . .]?”:
- The quality evaluation measures success on the original intended task.
- We motivate our quality evaluation setup in section 5.1 of the original paper.
- We are interested in the gap between the performance a non-native user would receive when prompting in native English compared to the performance they actually receive when prompting in non-native English.
- Therefore, the judge needs to see a consistent reference native prompt as context for rating the responses from the disfluent prompts.
- “[How sensitive are the conclusions ]to using an independent judge outside the evaluated model panel?”
- Our work as it stands already demonstrates generalization of our independent judges outside the evaluated model panel.
- Section 4.1: “For Rhetorical and Lexical Qualities . . . We use google/gemma-4-26B-A4B as judge.”
- Section 5.1: “We develop a response-quality evaluation protocol using Qwen3.8-27B.”
- Appendix Table 4: We included 14 model families, and most of the model panel is disjoint from the judges used.
- “Please reconcile the total prompt pool, language-specific subsets, variants per source prompt, filtering stages, and the final 190,911 variants.”: See Appendix A.10 and the updated section 3.
- “What role do the English prompts play?”: Every English prompt in FABLE is a translation of a Chinese source prompt at a specific instructed fluency level.
- “How does dropping Organization because of multicollinearity affect interpretation and comparisons across analyses?”: Dropping correlated predictors allows us to compute better estimates of the effects of the remaining predictors in our regressions (Ford 2025).
- Can you provide confidence intervals for the predicted quality differences and assess their magnitude relative to judge variability?”
- Figures 4/5 already include confidence intervals for predicted quality differences.
- Figure 6 already indicates which predicted quality differences are statistically significant.
References:
Hellman, A.B. (2011), Vocabulary size and depth of word knowledge in adult-onset second language acquisition. International Journal of Applied Linguistics, 21: 162-182. https://doi.org/10.1111/j.1473-4192.2010.00265.x
Ford, C. (2025), Addressing Multicollinearity. UVA Library StatLab. https://library.virginia.edu/data/articles/addressing-multicollinearity
Response to Reviewer mBNk (Part 1)
We thank Reviewer mBNk for their thoughtful review recognizing several critical strengths of our work, including:
We address reviewer concerns below: