- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper does an end-to-end pipeline evaluation of the difference in how American and British English are treated in different aspects of model construction, from data to tokenization to perplexity to generation. They also introduce DiaAlign, a metric for understanding how much text leans towards either American or British English.
Critical strengths
This is an interesting paper and I appreciate the topic it covers. I like that it examines a wide breadth of systems at every stage of the pipeline to demonstrate that these patterns are broader than just any single system. The analysis is thorough and well written.
Critical weaknesses
Many of your analyzes are in some way entangled, which makes it hard to understand the goal. Data influences tokenization and both influence the perplexity of the final model. I think in general, I would have done some sort of controlled study, systematically ablating each stage in the pipeline to understand its influence. While I understand that is difficult to do on a large scale, one such study with a single, small LLM (say, Qwen 0.6B) would have improved the strength of the work. I think this is my biggest sticking point with the paper. You claim you are examining how "linguistic asymmetries persist across the LLM pipeline," but from my perspective it seems to basically come down to data, which in turn has these downstream impacts. If your argument is that multiple stages of training are biased towards AmEng, I don't believe that point can be made with systems trained on this unbalanced data. If your argument is that unbalanced training data impacts many aspects of performance, then this paper aligns with that argument but that is a less novel argument.
I also think this paper would also be greatly improved by including Indian English (or another largely-spoken dialect) -- British English benefits from many systems that might be hiding how significantly these issues impact performance and Indian English is a better proxy for a very widely spoken but understudied dialect.
Some smaller concerns:
-
Your method of defining DiaAlign concerns me in a few points, especially since it is one of the primary metrics used across the paper. First, the dependence on n-gram frequency from Google Books feels as though it may result in bias due to the register/domain, especially when examining highly casual/conversational text as is done in a lot of the pre- and post-training corpora. I also am concerned by the choice to upweight samples that contain terms that are part of your curated lexicon. Since this curated lexicon is part of the construction of the test set, it seems like it could introduce issues and should largely be unnecessary if the lexicon is as binary as claimed (with one dialect showing significantly higher usage of some of these words). I similarly don't see a point in downweighting samples that are not highly dialectal since in theory, that should be automatic if the fraction of f_am/f_be is near 1. I would like to see it confirmed that the results hold without this unnecessary weighting. I would also like to see it confirmed that the DiaAlign metric is equally accurate on more diverse data, rather than the formal written register seen in books and news articles.
-
For the prediction cost and generation samples, I would like to see if the results hold when the input context is in the target dialect. While I understand the point that when the input is "neutral" the output leans American, I am curious whether that is flipped by the user simply speaking in British English, with that raising the likelihood of British orthography enough to overcome the differences.
Questions for authors
- Do you find that the tokenization differences are localized more to specific categories? Intuitively, I would imagine "words with sublexical spelling variation" would be the majority of cases, since both -ise and -ize likely appear enough to be tokenized as suffixes. I also notice that you include "jewelry vs. jewellry" in this category rather than as part of group 5, why?
- What is Velvet? You bring it up for the first time in the tokenization section but I don't know what that is. Did you mean GPT-4o?
- Is there a reason for using GPT-4o tokenizer instead of the GPT-5 tokenzier, considering they are different?
- Can you confirm to me that non highly-dialectal text ("My name is X") is not assigned to AmEng by your metric?
Thank you! I have updated my score.