acceptodds
Under review as a conference paper at ICLR 2027

Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models

Abstract

Large language models (LLMs) are increasingly used by people whose first language is not English, yet these users have been shown to receive systematically lower-quality responses than fluent speakers. Which specific features of non-native English drive this gap remains unclear, because fluency is itself a composite of mechanical accuracy, vocabulary use, organization, and discourse coherence. Here, we introduce FABLE, a controlled dataset of 190,911 English prompt variants derived from 174K real user prompts for writing-related tasks. Evaluating responses from 34 open-weight LLMs, we find a clear asymmetry; while models do not propagate surface errors such as misspellings into their outputs, models do mirror higher-level rhetorical and lexical qualities present in the user’s prompt. Further, the overall quality of responses differs substantially between the least- and most-fluent prompts. These results highlight a key LLM performance disparity for non-native English LLM users, resulting in both lower-quality and less-fluent answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Multilingual NLP, Prompt Engineering, Language Fluency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mBNkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates which characteristics of non-native English prompts affect LLM responses. It introduces FABLE, constructed by translating Chinese conversational prompts into English variants representing different ESL profiles, and evaluates multiple open-weight models. The analysis relates five dimensions of prompt fluency to response fluency and quality. The authors report that surface errors are generally not propagated, whereas some higher-level linguistic characteristics are reflected in responses, alongside small reductions in response quality associated with lower prompt fluency.

Critical strengths
  1. The paper addresses a relevant question about how language variation affects access to LLM capabilities. Examining individual linguistic dimensions offers a more informative perspective than treating native versus non-native English as a binary distinction.

  2. The study combines conversational prompts from WildChat and ShareLM with systematically generated variants and evaluates multiple models. This provides a useful framework for investigating whether different prompt characteristics affect response language and task performance differently.

Critical weaknesses
  1. The manipulation does not clearly separate linguistic fluency from preservation of task information. The Content rubric includes vagueness, missing information, and off-topic content, so lower scores may reflect changes in task specification rather than language proficiency alone. Since Content contributes substantially to the reported effects, pooled semantic-preservation validation is insufficient to rule out this confound. This weakens the interpretation of the observed quality penalty as an effect of non-native English.

  2. The quality evaluation and statistical analysis do not sufficiently establish the claimed effects. Responses are judged against a clean reference translation rather than the actual input, making it important to distinguish success on the original intended task from faithful execution of the supplied prompt. The small reported effects also require uncertainty estimates accounting for repeated variants and models, together with evidence about judge variability. Finally, the reported negative coefficients for some dimensions need reconciliation with the broad mirroring claim.

Questions for authors
  1. Can you report semantic-preservation results separately by fluency level, especially for low-Content prompts? How often do these variants omit or change constraints present in the reference translation?

  2. Does the quality evaluation measure success on the original intended task or compliance with the actual prompt supplied to the model? How sensitive are the conclusions to this distinction and to using an independent judge outside the evaluated model panel?

  3. Please reconcile the total prompt pool, language-specific subsets, variants per source prompt, filtering stages, and the final 190,911 variants. What role do the English prompts play?

  4. Which models generated the responses used in Figures 3 and 5? Please reconcile the model names in the appendix with the reported 34-model evaluation panel.

  5. How do the negative associations for Language Use and Mechanics support the mirroring conclusion? How does dropping Organization because of multicollinearity affect interpretation and comparisons across analyses?

  6. Do the uncertainty estimates account for repeated observations sharing a base prompt or model? Can you provide confidence intervals for the predicted quality differences and assess their magnitude relative to judge variability?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mBNk (Part 1)

We thank Reviewer mBNk for their thoughtful review recognizing several critical strengths of our work, including:

  • focusing on a problem with societal implications for LLM access
  • providing deeper insights by “examining individual linguistic dimensions” instead of “treating native versus non-native English as a binary distinction”
  • providing a useful framework for studying the effect of prompt characteristics on task performance and linguistic qualities of responses, including multiple models, real-world human-LLM conversational data, and systematically synthesized variants of real prompts

We address reviewer concerns below:

  • “The manipulation does not clearly separate linguistic fluency from preservation of task information.”: We respectfully disagree with Reviewer mBNk’s interpretation of our methods and results. As stated in section 4.1, “we consider semantic preservation during translation” by conducting the human evaluation described in detail in A.3. We find that “the translations generally preserve the original semantics with limited information loss on average.”
  • “The Content rubric includes vagueness, missing information, and off-topic content, so lower scores may reflect changes in task specification rather than language proficiency alone. Since Content contributes substantially to the reported effects, pooled semantic-preservation validation is insufficient to rule out this confound. This weakens the interpretation of the observed quality penalty as an effect of non-native English.”:
    • We respectfully disagree with Reviewer mBNk’s interpretation of our methods and results. As stated in section 4.3, “we consider semantic preservation during translation” by conducting the human evaluation described in detail in A.3. We find that “the translations generally preserve the original semantics with limited information loss on average.”
    • In section 5.2.2 of the original paper, we also reported the results of our semantic matched pair analysis, which found that “even with similar semantic intent, more fluent prompts generally receive higher-quality responses.”
    • Fluency cannot be fully reduced to stylistic features, but also must consider content. Non-native speakers know fewer total lexical items, lack conceptual depth of understanding of lexical items compared to native speakers, and have more difficulty producing lexical items in their non-native language (Hellmann 2011). As a result, it is harder for them to find as specific and detailed wording as native speakers, even while the basic semantics of the prompt intent are preserved.
  • “Responses are judged against a clean reference translation rather than the actual input, making it important to distinguish success on the original intended task from faithful execution of the supplied prompt.”:
    • We are interested in the gap between the performance a non-native user would receive when prompting in native English compared to the performance they actually receive when prompting in non-native English. Therefore, the judge needs to see a consistent reference native prompt as context for rating the responses from the disfluent prompts.
    • Section 5.1 of the original paper states an additional reason for using a clean reference translation: “to mitigate the potential inconsistencies caused by varying translation quality in the user prompts” and “[allow] the evaluation to focus on differences in model responses rather than artifacts introduced by translation noise.”
  • “The small reported effects also require uncertainty estimates accounting for repeated variants and models, together with evidence about judge variability.”
    • Appendix A.7 shows that we did account for repeated models in our regression formulas.
    • Figure 4 does not draw on many repeated variants; it uses 4,992 translations of 4,292 distinct source prompts, 85% of which contribute a single prompt variant.
    • The matched-pair analysis (Figure 5) compares pairs of different real-world source prompts with similar meaning (embedding similarity ≥ 0.85), translated under different fluency profiles, and regresses within-pair differences in response quality on within-pair differences in fluency.
    • For the fluency judge, per-dimension agreement with human ICNALE scores is in Table 5 with 95% confidence intervals.
    • We have also re-scored 200 responses for response quality with the same judge. We were able to reproduce the overall score for 190 responses; scores differed by one point for the remaining 10 responses. This demonstrates that our judge is consistent. We have added this note about response quality judge variability to Appendix A.6.
AuthorsOpenReview ↗

Response to Reviewer mBNk (Part 2)

  • “The reported negative coefficients for some dimensions need reconciliation with the broad mirroring claim.”
    • We do discuss the negative coefficients for the Language Use and Mechanics dimensions, explicitly stating “Language Use and Mechanics exhibit negative associations” (section 5.2.1).
    • “Figure 3 shows that “higher levels of Content, Organization, and Vocabulary show [statistically significant] positive associations with response proficiency, demonstrating that prompt fluency does pattern with response fluency.
  • “Can you report semantic-preservation results separately by fluency level, especially for low-Content prompts?”: Appendix A.3 reports that “the Spearman correlation between the average instructed [prompt fluency] level and similarity [of the resulting translated prompt variants] to the original Chinese prompt is 0.06.” In other words, the fluency level of the prompt is not correlated with any semantic changes, as measured by embedding similarity.
  • “Does the quality evaluation measure success on the original intended task or compliance with the actual prompt supplied to the model? How sensitive are the conclusions to this distinction [ . . .]?”:
    • The quality evaluation measures success on the original intended task.
    • We motivate our quality evaluation setup in section 5.1 of the original paper.
      • We are interested in the gap between the performance a non-native user would receive when prompting in native English compared to the performance they actually receive when prompting in non-native English.
      • Therefore, the judge needs to see a consistent reference native prompt as context for rating the responses from the disfluent prompts.
  • “[How sensitive are the conclusions ]to using an independent judge outside the evaluated model panel?”
    • Our work as it stands already demonstrates generalization of our independent judges outside the evaluated model panel.
    • Section 4.1: “For Rhetorical and Lexical Qualities . . . We use google/gemma-4-26B-A4B as judge.”
    • Section 5.1: “We develop a response-quality evaluation protocol using Qwen3.8-27B.”
    • Appendix Table 4: We included 14 model families, and most of the model panel is disjoint from the judges used.
  • “Please reconcile the total prompt pool, language-specific subsets, variants per source prompt, filtering stages, and the final 190,911 variants.”: See Appendix A.10 and the updated section 3.
  • “What role do the English prompts play?”: Every English prompt in FABLE is a translation of a Chinese source prompt at a specific instructed fluency level.
  • “How does dropping Organization because of multicollinearity affect interpretation and comparisons across analyses?”: Dropping correlated predictors allows us to compute better estimates of the effects of the remaining predictors in our regressions (Ford 2025).
  • Can you provide confidence intervals for the predicted quality differences and assess their magnitude relative to judge variability?”
    • Figures 4/5 already include confidence intervals for predicted quality differences.
    • Figure 6 already indicates which predicted quality differences are statistically significant.

References:

Hellman, A.B. (2011), Vocabulary size and depth of word knowledge in adult-onset second language acquisition. International Journal of Applied Linguistics, 21: 162-182. https://doi.org/10.1111/j.1473-4192.2010.00265.x

Ford, C. (2025), Addressing Multicollinearity. UVA Library StatLab. https://library.virginia.edu/data/articles/addressing-multicollinearity

Reviewer_8CXuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Authors investigate how the fluency of non-native English prompts shapes the responses that LLMs produce. To enable controlled experimentation, authors curate FABLE, a collection of 190,911 writing task–related prompts that vary systematically in fluency while preserving semantic intent. In experiments with responses from 34 popular open-source models, authors find that LLMs mirror the rhetorical and lexical qualities of the prompt, and their overall response quality is consistently lower for less fluent prompts.

Critical strengths

An interdisciplinary work, linguistic approaches to prompt engineering. The authors investigate which specific components of language literacy affect the quality of the response while preserving the user's semantic intent.

Critical weaknesses

It is generally obvious that the less clearly you write a question to the model, the worse its answer will be. What seems most questionable to me is that the authors do not separate the effect of language literacy as such from the effect of knowing how to properly frame a task for the model. These are in fact two different skills.

The authors generate artificial translation variants with different levels of language literacy, but they do not compare the behavior of real users with different levels of English. Therefore, in my view, the conclusion about inequality between user groups is premature.

Authors report a Spearman correlation with human ratings of ρ = 0.55. This is actually a small correlation that raise questions to methodological correctness and conclusions

Questions for authors

Gemma is used to evaluate the linguistic characteristics of prompts and responses, while Qwen is used for overall quality. How robust are the conclusions to replacing the judge model? Does the judge favor responses with more complex vocabulary and structure, even when they do not actually perform the task better? Do the effects hold under independent human evaluation on a substantially larger sample?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8CXu (Part 1)

We thank Reviewer 8CXu for their insightful review that recognizes our interdisciplinary approach and investigation of “which specific components of language literacy affect the quality of the response while preserving the user's semantic intent” as strengths. We address the Reviewer’s concerns below and in our updated uploaded paper draft.

  • “It is generally obvious that the less clearly you write a question to the model, the worse its answer will be”: We respectfully disagree. For example, we may expect more detailed instructions to increase prompt clarity and result in better model performance, but Yang et al. (2026) find that this is not the case.
  • “What seems most questionable to me is that the authors do not separate the effect of language literacy as such from the effect of knowing how to properly frame a task for the model. These are in fact two different skills.”
    • We disagree, because fluency cannot be fully reduced to stylistic features, but also must consider content.
    • Non-native speakers know fewer total lexical items, lack conceptual depth of understanding of lexical items compared to native speakers, and have more difficulty producing lexical items in their non-native language (Hellmann 2011). As a result, it is harder for them to find as specific and detailed wording as native speakers, even while the basic semantics of the prompt intent are preserved.
  • “The authors generate artificial translation variants with different levels of language literacy, but they do not compare the behavior of real users with different levels of English. Therefore, in my view, the conclusion about inequality between user groups is premature.”:
    • Using synthesized data in our framework is more scalable and controllable than relying on in-the-wild samples, because publicly available corpora of real-world human-LLM conversations generally do not label users’ language fluency levels.
    • “Compar[ing] with real non-native English” would be problematic because there is no reason to think that the characteristics of non-native fluency are universal. Deviations from native-level fluency depend on the source language. Here, we show how particular types/categories of deviations matter for response quality, rather than narrowing our focus to a specific kind of non-native fluency. Our approach is therefore more generalizable.
  • “Authors report a Spearman correlation with human ratings of ρ = 0.55. This is actually a small correlation that raise questions to methodological correctness and conclusions.”:
    • See section 5.1: “Given the challenging and subjective nature of evaluating open-ended responses, this moderate positive correlation provides encouraging evidence that the judge captures meaningful aspects of human assessments of response quality.”
    • Our reported Spearman correlation is well within the norms for subjective evaluation tasks (see Bavaresco et al. 2025 for examples of typical ranges).
AuthorsOpenReview ↗

Response to Reviewer 8CXu (Part 2)

  • Gemma is used to evaluate the linguistic characteristics of prompts and responses, while Qwen is used for overall quality. How robust are the conclusions to replacing the judge model?
    • We ensure the robustness of our conclusions by validating our judges against human ratings.
    • Overall, our human validation procedures provide evidence that our results are not idiosyncratic artifacts of the particular judges used, but rather correlate with human groundtruth measurements of our constructs of interest.
    • The fluency judge achieves a Spearman’s correlation of 0.61 with human scores on ICNALE (n = 640), and the quality judge reaches a Spearman’s correlation of 0.55 with the mean score of two human annotators on 100 responses (inter-annotator Krippendorff's alpha = 0.71).
    • The quality judge is also stable under re-runs: re-judging 200 stored items reproduces the original Overall score exactly in 95% of cases (Spearman’s correlation = 0.99).
  • “Does the judge favor responses with more complex vocabulary and structure, even when they do not actually perform the task better?”: “Perform the task better” can mean many different things in practice. Our judging rubric for evaluating response quality in appendix A.6 instructs the judge to consider helpfulness, relevance, accuracy, creativity, and level of detail when scoring responses. Therefore, “vocabulary and structure” complexity are both relevant parts of assessing task performance. For example, more complex structure may indicate a more detailed response, while more complex vocabulary may be appropriate for creative writing tasks.
  • “Do the effects hold under independent human evaluation on a substantially larger sample?”
    • Our sample sizes are already large enough to report statistically significant results.
    • Our human evaluation of the response quality judge in section 5.1 from the original paper already uses an appropriate sample size of n = 100.
      • At an alpha = 0.05 significance level, this results in a 95% confidence interval [0.38, 0.68] for Spearman’s correlation coefficient (Bonett et al. 2000).
      • As we state in 5.1, “Given the challenging and subjective nature of evaluating open-ended responses, this moderate positive correlation provides encouraging evidence that the judge captures meaningful aspects of human assessments of response quality.”
    • We report 95% confidence intervals for our human validation of the fluency judge on the ICNALE dataset, demonstrating that we achieve moderate to strong correlations with human groundtruth fluency scores (appendix A.9).

References:

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–255, Vienna, Austria. Association for Computational Linguistics.

Bonett et al. (2000), Sample Size Requirements for Estimating Pearson, Kendall and Spearman Correlations. In Psychometrika (Vol. 65, No. 1), pages 23-28, Psychometric Society. doi:10.1007/BF02294183

Yang, C. et al. (2026), What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts. Findings of the Association for Computational Linguistics: ACL 2026. https://aclanthology.org/2026.findings-acl.441/

Reviewer_HgFdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks which aspects of non-native English lower the quality of LLM responses. It takes first turn writing prompts in Chinese from WildChat and ShareLM and has a 32B Qwen3 model translate each into English (under target levels on the five dimensions of the ESL Composition Profile), giving FABLE, 190K English variants. Prompt and response fluency are scored by a Gemma-4 judge and by LanguageTool, and response quality by a 27B Qwen3.8 judge that is given a DeepSeek golden translation of the original prompt. 30+ open-weight models answer 5K sampled variants. Paper reports that surface errors are not copied into responses, that responses mirror higher-level qualities of the prompt. Also, each level of a prompt fluency dimension raises response quality by 0.04-0.14 points on a 5-point scale, compared to 0.21 for a 10-fold increase in model size.

Critical strengths

The question matters, yes, and the shift from a binary native vs. non-native contrast to an established analytic rubric (Jacobs et al., 1981) is a thoughtful way to ask which aspects of a prompt influence the answer. The prompts is from real first turn requests and the restriction to writing tasks is justified by the task distribution in Fig. 1; and the observation in Fig. 2 that models largely do not copy surface errors into their answers is useful for users who rely on LLMs to write. I think it matters for the assessment because a well-founded dataset of this kind would let the community study a real and under-measured equity problem.

The scale is large enough and the validation is openly reported. 30+ models from many families answer the same prompts at temperature 0, the generation and judging prompts are given in full, and the paper reports its validation numbers even where they are moderate (0.47 on ICNALE, 0.55 against human quality scores, 4.13 of 5 for semantic preservation). Again, I think this matters because it makes clear what the measurements can and cannot carry.

Critical weaknesses

However, the manipulated variable and the judging reference do not yet isolate non-native fluency. FABLE prompts are written by an LLM following essay-grading descriptors, and they are not compared with real non-native English, although synthetic and natural noise can affect models differently (Belinkov and Bisk, ICLR 2018), and real learner corpora with proficiency labels exist, including the ICNALE data the paper already uses. The Content dimension changes the information in the request rather than its fluency. L1 asks for ideas that are "vague, off-topic, or insufficient to evaluate" while L4 for "specific, relevant details" (see App. A1), and Content carries a large share of the quality effect in Fig 4 and the largest in Fig 5. The quality judge is then given the golden translation as the question (App A6), not the variant the model answered, so a requirement lost in translation counts as a model failure. Fig. 8 also suggests that the instructions do not specifically shift the judged dimensions. I think this matters because the main claim is that less fluent users receive worse answers, and the measured gap is not yet separated from information loss and from properties of LLM-simulated disfluency.

The dimension-level conclusions do not follow from the analyses as reported. Fig 4 shows Mechanics, which covers spelling and punctuation, with the largest fluency coefficient on quality (about 0.145, above Content at 0.11), yet the Conclusion states that the disparity "is driven less by surface mistakes than by the higher-level discourse and lexical qualities". Fig 3, by its colour scale, shows a pattern by prompt dimension rather than along the diagonal, with negative coefficients for prompt Language Use and Mechanics that are not discussed, and the responses behind Fig. 3/5 come from the 14B Qwen3, the 8B Llama-3.1 and the 20B variant of GPT-OSS, none of which is among the open models in Table 4. The regressions also omit source-prompt fixed effects despite the paired design, include response length as a covariate although it may be caused by prompt fluency, and do not state whether standard errors are clustered by prompt. I do think this matters because the paper's stated aim is to identify which features of non-native English drive the gap, and that attribution is not really supported.

Questions for authors
  • How do the quality results change if each response is judged against the prompt the model received rather than the golden translation, and if Content is excluded or modelled separately from the four language dimensions?
  • Are the regression predictors the instructed levels or the judged levels? Also, how do the estimates change with source-prompt fixed effects, standard errors clustered by prompt, and without response length as a covariate?
  • Which models generated the responses behind Figures 3/5, and do those patterns hold for the 30+ models in Table 4?
3 comments
AuthorsOpenReview ↗

Response to Reviewer HgFd (Part 1)

We thank Reviewer HgFd for their detailed review pointing out critical strengths of our work, including its focus on a “real and under-measured equity problem,” interdisciplinary grounding, use of real-world data, useful insights for end users, dataset contribution, experimental and analytical depth and breadth, and steps taken to ensure reproducible and reliable results. We address the reviewer’s fixable concerns as follows:

  • “The manipulated variable and the judging reference do not yet isolate non-native fluency. FABLE prompts are written by an LLM following essay-grading descriptors, and they are not compared with real non-native English, although synthetic and natural noise can affect models differently (Belinkov and Bisk, ICLR 2018), and real learner corpora with proficiency labels exist, including the ICNALE data the paper already uses.”:
    • “Compar[ing] with real non-native English” would be problematic because there is no reason to think that the characteristics of non-native fluency are universal. Deviations from native-level fluency depend on the source language. Here, we show how particular types/categories of deviations matter for response quality, rather than narrowing our focus to a specific kind of non-native fluency. Our approach is therefore more generalizable.
    • Using synthesized data in our framework is more scalable and controllable than relying on in-the-wild samples, because publicly available corpora of real-world human-LLM conversations generally do not label users’ language fluency levels.
    • We conduct multiple experiments and analyses to isolate non-native fluency. To validate how effectively we are able to control fluency levels of our translated prompts, we carry out the evaluation described in section 4.3 of the original paper. As we report there, “We found that higher instructed proficiency levels generally yield corresponding improvements in measured fluency.” We also validate our fluency judge using the human-annotated ICNALE dataset, as described in section 4.1.
  • “The Content dimension changes the information in the request rather than its fluency. L1 asks for ideas that are "vague, off-topic, or insufficient to evaluate" while L4 for "specific, relevant details" (see App. A1), and Content carries a large share of the quality effect in Fig 4 and the largest in Fig 5.”:
    • We respectfully disagree with Reviewer HgFd’s interpretation of our methods and results. As stated in section 4.1, “we consider semantic preservation during translation” by conducting the human evaluation described in detail in A.3. We find that “the translations generally preserve the original semantics with limited information loss on average.”
    • In section 5.2.2 of the original paper, we also reported the results of our semantic matched pair analysis, which found that “even with similar semantic intent, more fluent prompts generally receive higher-quality responses.”
    • Fluency cannot be fully reduced to stylistic features, but also must consider content. Non-native speakers know fewer total lexical items, lack conceptual depth of understanding of lexical items compared to native speakers, and have more difficulty producing lexical items in their non-native language (Hellmann 2011). As a result, it is harder for them to find as specific and detailed wording as native speakers, even while the basic semantics of the prompt intent are preserved.
  • “The quality judge is then given the golden translation as the question (App A6), not the variant the model answered, so a requirement lost in translation counts as a model failure.”:
    • We motivate our quality evaluation setup in section 5.1 of the original paper.
    • Using the golden translation of the prompt to judge the quality of responses to non-fluent versions of that prompt is appropriate for our goals.
    • We are interested in the gap between the performance a non-native user would receive when prompting in native English compared to the performance they actually receive when prompting in non-native English.
    • Therefore, the judge needs to see a consistent reference native prompt as context for rating the responses from the disfluent prompts.
    • An additional reason for using a clean reference translation: “to mitigate the potential inconsistencies caused by varying translation quality in the user prompts” and “[allow] the evaluation to focus on differences in model responses rather than artifacts introduced by translation noise.”
AuthorsOpenReview ↗

Response to Reviewer HgFd (Part 2)

  • “Fig. 8 also suggests that the instructions do not specifically shift the judged dimensions. I think this matters because the main claim is that less fluent users receive worse answers, and the measured gap is not yet separated from information loss and from properties of LLM-simulated disfluency.”:
    • To verify that substantial information loss does not occur, we perform additional validation and analysis in section 4.3 and section 5.2.2 of the original paper.
    • Figure 8 shows that our instructed Rhetorical and Lexical Quality levels are significantly associated with the judged levels in the translated prompts, confirming that our data synthesis instructions do shift the judged dimensions.
    • For ecological validity, we should not aim for fluency dimensions to be fully independent, but rather recognize that they interact in the real world. For example, in order to score high on Language Use (which measures grammatical correctness and complexity), a speaker must also have a good grasp of Vocabulary, since in English, verb tenses are highly irregular and require substantial memorization of conjugation patterns for individual lexical items.
  • “The dimension-level conclusions do not follow from the analyses as reported. Fig 4 shows Mechanics, which covers spelling and punctuation, with the largest fluency coefficient on quality (about 0.145, above Content at 0.11), yet the Conclusion states that the disparity "is driven less by surface mistakes than by the higher-level discourse and lexical qualities":
    • Another reviewer (HgFd) agrees with our conclusion, stating “models largely do not copy surface errors into their answers” (Figure 2). Therefore, surface mistakes are insufficient to explain the disparity in response quality.
    • Recap: we measure fluency from two perspectives in section 4.1.
      • Surface Form Features include spelling, grammatical, lexical, and stylistic errors, and Flesch Reading Ease.
      • Rhetorical and Lexical Qualities are Content, Organization, Vocabulary, Language Use, and Mechanics.
      • We update section 4.1 to make this distinction as clear as possible in the newly uploaded paper draft.
      • As seen from the rubric in appendix A.1, the Surface Form Features are not equivalent to the Rhetorical and Lexical Quality “Mechanics” dimension.
  • “Fig 3, by its colour scale, shows a pattern by prompt dimension rather than along the diagonal, with negative coefficients for prompt Language Use and Mechanics that are not discussed.”:
    • We do discuss the negative coefficients for the Language Use and Mechanics dimensions, explicitly stating “Language Use and Mechanics exhibit negative associations” (section 5.2.1).
    • “Figure 3 shows that “higher levels of Content, Organization, and Vocabulary show [statistically significant] positive associations with response proficiency,” demonstrating that prompt fluency does pattern with response fluency.
  • “The responses behind Fig. 3/5 come from the 14B Qwen3, the 8B Llama-3.1 and the 20B variant of GPT-OSS, none of which is among the open models in Table 4.”
    • Note: The surface-form result (Figure 2) and the response quality results (Figures 4 and 6) are computed on all 34 models from Table 4.
    • Two of the models used for Figures 3 and 5 come from families (Qwen, Llama) included in Table 4, and their sizes (8-20B parameters) fall inside the range of those 34 models as well (0.8-70B).
    • We have revised the captions in the uploaded paper to clarify which data was used to create each figure.
  • “The regressions also omit source-prompt fixed effects despite the paired design, include response length as a covariate although it may be caused by prompt fluency, and do not state whether standard errors are clustered by prompt.”:
    • Our semantically matched pair analysis (Section 5, Appendix A.7.4) regresses within-pair differences in response quality on within-pair differences in fluency. This is equivalent to a pair fixed-effects estimator.
    • Appendices A.7.1, A.7.3, and A.7.5 of the original paper include “source-prompt fixed effects” in the form of prompt length for each respective regression.
    • Including response length as a covariate is a valid choice, given that our prompts cover a diverse array of tasks and certain task types would be associated with longer or shorter responses.
    • Our standard errors are not clustered by prompt.
AuthorsOpenReview ↗

Response to Reviewer HgFd (Part 3)

  • “[How do the quality results change] if Content is excluded or modelled separately from the four language dimensions?”: Excluding or separately modeling the Content dimension from our regressions would be problematic because it would create omitted-variable bias.
  • “Are the regression predictors the instructed levels or the judged levels?”:
    • The regression predictors in Figures 2 and 3 are the judged fluency levels of the translated prompt. We updated the paper draft to make this clear.
    • As stated in the caption for Figure 7 of the original paper, the regression shown in this figure predicts surface form features from instructed prompt fluency levels.
    • As stated in the caption for Figure 8 of the original paper, the regression shown in this figure predicts the judged Rhetorical and Lexical Qualities of the translated prompts from instructed prompt fluency levels.

References:

Hellman, A.B. (2011), Vocabulary size and depth of word knowledge in adult-onset second language acquisition. International Journal of Applied Linguistics, 21: 162-182. https://doi.org/10.1111/j.1473-4192.2010.00265.x

Comments on the paper

1 comment
AuthorsOpenReview ↗

Top-Level Summary for ACs and Reviewers

We thank the reviewers for their thoughtful comments. Please find a summary of the reviews and our responses below. We have also uploaded an updated draft of the paper with our updates highlighted in blue:

  • Strengths of our work:
    • Reviewers HgFd, 8CXu, and mBNk:
      • We incorporate interdisciplinary insights to our study of the concept of “fluency,” “shift[ing] from a binary native vs. non-native contrast to an established analytic rubric (Jacobs et al., 1981),” which “is a thoughtful way to ask which aspects of a prompt influence the answer.” (Reviewer HgFd)
    • Reviewer HgFd:
      • We contribute a “well-founded dataset” that “would let the community study a real and under-measured equity problem.” (Reviewer HgFd)
      • We investigate a societally impactful question (Which specific features of non-native English drive the gap between performance for native speakers and non-native speakers?).
      • We use real human-LLM conversational data for our experiments reflecting realistic common real-world use cases (Reviewer HgFd says “the restriction to writing tasks is justified by the task distribution in Fig. 1”).
      • Our work provides useful insights for end users, e.g., “the observation in Fig. 2 that models largely do not copy surface errors into their answers is useful for users who rely on LLMs to write.” (Reviewer HgFd)
      • We validate our measurements and provide exact prompts used for data synthesis and judging, supporting reproducibility and allowing readers to better judge the reliability of our work.
    • Reviewers HgFd and mBNk: Our experimentation and analysis is both broad and deep. Reviewer HgFd writes, “The scale is large enough” and we feature “30+ models from many families,” while Reviewer mBNk additionally notes the “systematically generated [prompt] variants.”
    • Reviewer mBNk: Our work “provides a useful framework for investigating whether different prompt characteristics affect response language and task performance differently.”
  • Reviewer concerns addressed in responses:
    • Reviewers HgFd, 8CXu, and mBNk:
      • We explain how our experimental design isolates non-native fluency.
      • We explain how our experimental design and analyses allow us to conclude that the driving factor behind differences in response quality and fluency is prompt fluency, rather than information loss due to the data synthesis process.
    • Reviewers HgFd and 8CXu:
      • We further justify our use of synthetic data.
    • Reviewers 8Cxu and mBNk
      • We provide mathematical justification for our human evaluation sample sizes.
      • We explain the robustness checks we performed on our judges.
    • Reviewers HgFd and mBNk
      • We justify our choice of the groundtruth reference prompt translation as context for judging response quality.
      • We explain how our analyses account for fixed effects of the prompts and models used.
    • Reviewer HgFd
      • We explain our interpretations of the paper figures.
      • We clarify which models generated the responses shown in each figure.
      • We clarify which figures show instructed v. judged fluency levels.
    • Reviewer mBnk
      • We clarify how our figures illustrate the statistical significance of our results.
      • We clarify our data filtering procedures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.