- Rating
- 1
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper makes an important, interesting, and well-supported point that the ratio alone between dialects is not the only factor in performance discrepancies. In other words, even at the same data ratio, data selection can result in massive difference in the performance gaps between AAE and white English. They perform pre-training experiments on GPT 2 using a dialect-tagged twitter corpus as well as we corpora. Their findings are significant when it comes to convincing those who are pre-training to assess data makeup beyond frequency, especially because for many dialects frequency is not something that is easy to fix. I think with a complete re-write to make it more clear, but with largely the same experiments, this would make a good conference paper but at this stage it is too confusing.
Critical strengths
The experiments in this paper are very well done - many seeds, lots of baselines, and diverse data sources. I believe that the findings are also novel and interesting. While their results are discussing just AAE, it is easy to see that they would likely transfer to other dialects/varieties. They primarily focus on the NLL gap between the dialects, which concerned me at first that part of the reduced gap was due to worse white English performance, but it seems that they address that in the main body and for the most part it is not a concern.
Critical weaknesses
-
Honestly, my biggest gripe is how difficult this paper was for me to parse. I found myself having to jump around a lot to figure out what was being referred to and relying heavily on figures/tables to decode the prose. The issue starts with the abstract, which I found to be very dense with numeric results shown without sufficient background to understand them. Through the introduction and results, there is no unified list of all the data variations with their differences, leading me to need to put it together with a mix of close reading and making assumptions based on the names of the methods. I'm still not sure what ClusterRandom is and whether this is a new method proposed by the authors. As someone not familiar with pre-training data filtering, I had no background on DataComp-LM and the paper provided no insight, forcing me to go read that paper to understand what the filtering process was. This issue continues on throughout the paper and means I can't really verify the claims.
-
I can't tell if this is an argument against filtering in general, filtering in a manner that restricts AAE more heavily due to bias, or against combining filtering and uneven data distributions. For example, uniform, unfiltered seems to perform the best suggesting that we shouldn't filter and instead just use all the data. While the paper seems to make an argument for not filtering, the restored-share baseline seems to suggest that the filtering isn't as much of a problem as the ratio, with fixed-share performing worse. This is especially true because it isn't stated how much the filter cut non-AAE data.
-
The web comparison section seems to not have any explicit information about dialect - even from a dialect classification system. As a result, it's hard to connect these results to the broader argument from the previous section. I believe your argument would be stronger if you explicitly were able to point towards specific attested AAE constructions that are filtered out and demonstrate that the performance drop is due to that. Otherwise its hard to say whether this is due to the filtering causing a ratio discrepancy (the ratio-share comparison) or the filtering removing specific features that are in the evaluation set (the fixed-share comparison).
Questions for authors
- For Table 2, it seems that you have only sampled one dataset for the comparison, but you ran multiple seeds (with different data sampling?), so do these differences hold when looking at the standard deviation of the values over sampling runs?
- What is the motivation behind ClusterRandom? You are semantically embedding these documents and proportionally sampling from them, which introduces a whole different axis. I would understand more if you ensured each cluster contributed uniform dialectal proportions, to basically control for topic and dialect when creating the dataset, but that doesn't seem to be the case. Also you need to introduce this earlier if it isn't a pre-established method.