- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to measure an LLM’s language proficiency and what structure it learns. It fine-tunes pretrained LLMs on strings generated by known formal grammars. It proposes a local discrimination test, comparing valid strings with nearby invalid variants that allows to answer whether or not the language has been assimilated by the model. It also finds that models learn local patterns more reliably than global structure. Natural-language experiments report a similar local/global asymmetry.
Critical strengths
Clearly stated research questions. Separating language proficiency from implicit language structure gives the paper a useful focus.
Controlled experimental environment. Formal languages provide explicit grammar rules and known valid strings, making it possible to investigate questions that are difficult to isolate in natural language.
Useful idea behind local discrimination. Comparing a correct string with nearby incorrect strings is an appealing way to assess whether the model distinguishes grammatical structure. Figure 3 helps illustrate why the choice and difficulty of incorrect examples matter.
Critical weaknesses
-
The paper would benefit from an accessible introduction to probabilistic formal languages and a running example connecting the grammar, training data, and evaluations. The experimental workflow is also difficult to follow: what the model sees during training, what researchers modify during evaluation, and how each measurement answers its research question should be stated explicitly. Although the appendix provides more experimental details, the main text should summarize better the general setup
-
The implemented criterion requires a correct string’s current loss to beat the lowest loss any incorrect neighbor reaches anywhere in the training run. This differs from checking whether the model prefers the correct string to its neighbors at the current checkpoint. Consequently, the reported proficiency depends on other checkpoints as well as the model’s current weights. The paper should show whether the main findings hold under the more direct, same-checkpoint criterion or at least justify better their current choice
-
The interpretation of OOD generalization needs greater precision. The experiments provide evidence that learned local patterns remain useful on certain strings from modified grammars. However, high local proficiency alone does not establish that the model follows the modified grammars’ full structure. A stronger evaluation would test each modified grammar’s own global constraints, using incorrect strings that preserve familiar local chunks but violate those constraints, similar to what's been done in Section 4 for instance but on the edited grammar.
-
The natural-language evidence has interpretive limitations Word-, sentence-, and paragraph-level edits can affect grammaticality, meaning, or coherence in different ways. Some sentence or paragraph edits may leave the text grammatically valid. The paper should clarify what these experiments measure and how closely they support the conclusions drawn from formal grammars.
Minor concern: The connection between the creativity discussion in Section 6 and the paper’s main findings was unclear to me.
Questions for authors
How much do the results depend on prior natural-language training? Would a model trained from random initialization show the same local/global asymmetry?
Why use a threshold drawn from the entire training run? What advantage does this provide over comparing correct and incorrect strings at the same checkpoint?
Are edited examples checked for grammaticality and coherence? How do the authors account for differences in difficulty between word-, sentence-, and paragraph-level edits?
3 comments
The interpretation of OOD generalization needs greater precision
Based on the reviewer comments, we have now directly tested whether the LLM learns the full structure of the modified OOD languages (Appendix I): we measure rule proficiency within the OOD grammar itself, using incorrect strings obtained by editing the OOD grammar's own rules (constraints) at each level. We find the same asymmetry holds: the OOD grammar's local rules are learned better than global rules (Figure 45), mirroring the asymmetry in the original grammar.
Therefore, when an LLM learns a language, it learns its local structure better than its global structure, and generalizes to globally modified OOD languages. In each of these OOD languages, the LLM again learns the local structure better than the global structure, providing an explanation of why they show OOD generalization in the first place. This deeper point characterizes the implicit world model of the LLM.
I thank the authors for their detailed answer. I appreciate the additions and clarifications. I have no further questions. Will maintain my weak-accept rating.
We thank the reviewer for the careful review and for highlighting the paper's focus and experimental design.
Yes: during rebuttal we observe the same local/global asymmetry in pre-training from random initialization (Figures 42–43, Appendix F). The LLM generalizes better to OOD languages that relax global structure and learns local structure better within the language, consistent with fine-tuning. The only difference is the rate of learning: fine-tuning learns faster and overfits sooner, while pre-training is slower under the same hyperparameters. Therefore, local-global learning asymmetry is a general inductive bias of the LLM.
We use the local boundary threshold from the entire training run because it gives a single threshold of learning and forgetting per string, rather than varying the threshold from one epoch to another. For example, in Figure 4(a), the string is learned at epoch 2 and forgotten at epoch 18 under an all-epoch threshold. If we apply a per-epoch (or single checkpoint) threshold, the string is still learned at epoch 2, but never forgotten because its neighbor's loss rises alongside it. As such, after epoch 18, the LLM generates the string worse than its best-case incorrect neighbor, yet the string is still not called forgotten under the per-epoch threshold. Therefore, we apply an all-epoch threshold whenever the entire training trace is available to us (e.g., formal language experiments) and a per-epoch threshold otherwise (e.g., natural language experiments; Figures 5c, 8b).
However, our main conclusion doesn't depend on this choice: the natural-language setup, using a per-epoch threshold, shows the same local/global asymmetry as the formal-language setup with an all-epoch threshold.
Yes. During rebuttal, we apply an LLM-judge (Claude Sonnet 5) to check the coherence of edited strings, since unlike a formal grammar, natural language has no ground-truth grammar to check against. The coherence of edited strings depends on the edit operations and the levels of edits (word/sentence/paragraph). Specifically, insertions and replacements almost always make edits incoherent regardless of level; deletions and swaps break coherence more at the word level than at the sentence/paragraph level (Appendix F.2). This mirrors the formal-language result: global edits are less likely to break coherence and are thus discriminated less well by the LLM.
In the updated paper, we expand the existing example (Figure 1, Section 3) with the vocabulary, non-terminal set, and local/global rules and describe how strings are sampled by iteratively expanding non-terminals to terminals.
We summarize the experimental setup in Section 3: we sample disjoint training/test strings from the ground-truth grammar, edit each test string into a grammatically incorrect one, and measure local discrimination between the two to gauge how well the LLM learns the language. To characterize its implicit grammar, we similarly edit grammar rules to build OOD grammars and measure discrimination there; uneven proficiency across OOD languages reveals the LLM's implicit grammar.
We are currently experimenting with whether the LLM learns the full structure in the OOD languages as well. Our intuition is that it does not: we expect the OOD grammar will show a similar trend to the given grammar, where local rules are learned better than global ones. We will follow up with direct experimental results.
We have provided a deeper analysis on OOD generalization based on n-gram overlap (see Reviewer qeDy).
The LLM generalizes well to OOD languages that relax global structure, which we anthropomorphize a bit and read as a form of creative learning. Our position is that creativity is a natural inductive bias in how the LLM generalizes, rather than something invoked only at inference time (prompting, constrained decoding). This connection is still exploratory, and we leave it as future work.