- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Many natural language number systems are regular (e.g. 78 is pronounced seven + ty + eight, i.e. 7 * 10 + 8). This paper hypothesizes that this is because such systems are easier to learn. It tests this hypothesis using LSTMs trained either in supervised learning or reinforcement learning, and in general finds support for the hypothesis. Another very important predictor is whether the number system follows PEMDAS (the usual order of precedence of parentheses, multiplication, addition).
Critical strengths
- The paper is very well written and provides an excellent overview of the topic.
- The experiments and their statistical analysis are thorough.
Critical weaknesses
Some of the findings do not necessarily require an experiment; for example, it is fairly obvious that to find effects of regularity you want the distribution of the test set to be uniform rather than identical to the training distribution (power law), so it seems like you could just skip the IID experiment. But more fundamentally, I find it hard to imagine a case where regularity wouldn't affect learnability; of course if all of the numbers in English' are regular but 9338 is pronounced "avocado" it will be very hard to learn that 9338 is pronounced "avocado". I don't think the paper does the most compelling job of explaining what specific outcomes of the experiment would be empirically illuminating (which non-obvious possibilities are ruled out). To be fair, this is something that may be resolvable during the discussion period, and I'd be happy to engage with the authors on this.
Questions for authors
-
I don't really know what to make of the cognitive plausibility of the particular modeling choices (LSTM, RL vs. SL). Are some of these more like what we imagine humans to be? Is there really such thing as "learnability" that's not relativized to a particular learner?
-
I got a little lost in the discussion of PEMDAS, which seems particularly important in the statistical model. Do we imagine humans and LSTMs as having an inductive bias that favors number systems that follow PEMDAS?
We thank the reviewer for their comments and we appreciate their compliments for our presentation and thoroughness of experiments and analysis.
Critical Weakness 1: To address your comments (and a similar comment from another reviewer), we agree that "regularity facilitates learnability" is not particularly a novel insight. However, we would like to point out that our work adds important nuances to this claim. First, in our view, the lack of learnability advantage in Experiment 1 (especially for RL) was not at all obvious a priori--perhaps a small exposure to rare numerals would have sufficed for a regularity advantage. Only by doing both Experiment 1 and 2 did we learn that the capacity to generalise to rare numerals is key. The non-linear shape of the learnability curve in Experiment 2 (and 3) also suggests that regularity has little to no effect on learnability once the system becomes sufficiently irregular; in our view, this is also not obvious. Second, our work highlights a crucial distinction (often conflated in prior work on language efficiency analysis) between frequency of use and communicative need, the latter of which actually shapes linguistic systems. In the case of recursive numerals, the use of numerals is highly skewed, and yet we place a lot of importance in expressing rare numerals precisely. This distinction may also explain systems in other domains where a bias for certain meanings has been found, such as obligation modals in modal verbs (Uegaki et al., Deontic priority in the lexicalization of impossibility modals, 2023). Also, our multivariate regression modelling shows, non-trivially, that regularity remains important and key even when incorporating other possible factors. Finally, the new main section we have added in response to reviewers (Experiment 3, page 9) reveals that among human and human-like systems, irregularity does not reduce learnability over the whole system (ρ=0.27) but does reduce generalisation to unseen numerals (ρ=−0.51) (both Spearman ρs, LSTM under RL), corroborating the connection between regularity and learnability (as generalisability).
To answer your specific questions:
Q1: We agree with the reviewer that learnability is learner-relative. Both LSTMs and transformers (which we have added experiments for in the main paper, see Experiment 3, page 9) are high-level analogues for human learning--at least to the extent that all algorithms learning the same target system exploit data similarly; we remain agnostic as to whether they are truly realistic cognitive models. That said, between the two training regimes, our results tentatively point to RL as the closer analogue in this setting: it exploits structure in the data--something we know that humans do--whereas SL also learns highly idiosyncratic, unattested systems efficiently (§4.2). Comparing the two architectures, we can safely say that the overall trend and separation between regular and irregular systems hold for both (Spearman ρ=0.84 under RL and 0.94 under SL across systems), but the two agree only partly on the ordering of individual human(-like) systems (ρ≈0.5, cf. Appendix J). Our claims are therefore about general learnability trends.
Q2: We actually view the PEMDAS bias primarily as a property of the data. We made the operational assumption that the space of possible numeral grammars will be those generating numerals that will almost always evaluate to the correct numbers if one follows PEMDAS (Hurford, The linguistic theory of numerals, 1975), and deviations from PEMDAS are unsystematic. The systems with consistent evaluation patterns are those whose correct way of evaluating numerals consistently conform to PEMDAS, resulting in an apparent "PEMDAS bias". Humans and models alike can take advantage of the consistency of evaluation patterns (which happens to be PEMDAS-conforming in this case), e.g., through overhypothesis learning, and thus learn PEMDAS-conforming systems more easily.