acceptodds
Under review as a conference paper at ICLR 2027

Implicit Language Representation of LLMs: A Formal Language Learning Perspective

Abstract

When a large language model (LLM) is trained to learn a language, is it learning the underlying language representation (henceforth, also structure) precisely, or is it also becoming proficient in other related languages? This question offers insight into the LLM’s implicit world model. To answer this question precisely, we need a learning environment for LLMs where we can define the language and its underlying structure. We propose probabilistic formal language as the learning environment, which provides greater control than natural language. Next, we need a proficiency test that considers the possibility of generating new strings both inside and outside the language. Our answer is a local discriminative test that operates on individual strings: a string is learned if the LLM generates it with higher probability than every neighboring out-of-language string, and language proficiency is the fraction of strings learned this way. Empirically, within a language, an LLM learns strings better if they are structurally closer to the training data. Beyond individual strings, we characterize the LLM’s implicit structure, i.e., its formal grammar here, by the languages it generalizes to beyond the target. We find that the LLM generalizes better to out-of-distribution languages that relax global structure than to ones that relax local structure. Simultaneously, the implicit grammar is close to grammar rules governing local structure, but deviates from rules governing global structure, a pattern we also confirm empirically in natural language, where it learns local word structure better than global sentence- or paragraph-level structure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Model, LLMs, Formal Language, Formal Grammar, Discriminative Test

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KzmhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to measure an LLM’s language proficiency and what structure it learns. It fine-tunes pretrained LLMs on strings generated by known formal grammars. It proposes a local discrimination test, comparing valid strings with nearby invalid variants that allows to answer whether or not the language has been assimilated by the model. It also finds that models learn local patterns more reliably than global structure. Natural-language experiments report a similar local/global asymmetry.

Critical strengths

Clearly stated research questions. Separating language proficiency from implicit language structure gives the paper a useful focus.

Controlled experimental environment. Formal languages provide explicit grammar rules and known valid strings, making it possible to investigate questions that are difficult to isolate in natural language.

Useful idea behind local discrimination. Comparing a correct string with nearby incorrect strings is an appealing way to assess whether the model distinguishes grammatical structure. Figure 3 helps illustrate why the choice and difficulty of incorrect examples matter.

Critical weaknesses
  1. The paper would benefit from an accessible introduction to probabilistic formal languages and a running example connecting the grammar, training data, and evaluations. The experimental workflow is also difficult to follow: what the model sees during training, what researchers modify during evaluation, and how each measurement answers its research question should be stated explicitly. Although the appendix provides more experimental details, the main text should summarize better the general setup

  2. The implemented criterion requires a correct string’s current loss to beat the lowest loss any incorrect neighbor reaches anywhere in the training run. This differs from checking whether the model prefers the correct string to its neighbors at the current checkpoint. Consequently, the reported proficiency depends on other checkpoints as well as the model’s current weights. The paper should show whether the main findings hold under the more direct, same-checkpoint criterion or at least justify better their current choice

  3. The interpretation of OOD generalization needs greater precision. The experiments provide evidence that learned local patterns remain useful on certain strings from modified grammars. However, high local proficiency alone does not establish that the model follows the modified grammars’ full structure. A stronger evaluation would test each modified grammar’s own global constraints, using incorrect strings that preserve familiar local chunks but violate those constraints, similar to what's been done in Section 4 for instance but on the edited grammar.

  4. The natural-language evidence has interpretive limitations Word-, sentence-, and paragraph-level edits can affect grammaticality, meaning, or coherence in different ways. Some sentence or paragraph edits may leave the text grammatically valid. The paper should clarify what these experiments measure and how closely they support the conclusions drawn from formal grammars.

Minor concern: The connection between the creativity discussion in Section 6 and the paper’s main findings was unclear to me.

Questions for authors

How much do the results depend on prior natural-language training? Would a model trained from random initialization show the same local/global asymmetry?

Why use a threshold drawn from the entire training run? What advantage does this provide over comparing correct and incorrect strings at the same checkpoint?

Are edited examples checked for grammaticality and coherence? How do the authors account for differences in difficulty between word-, sentence-, and paragraph-level edits?

3 comments

We thank the reviewer for the careful review and for highlighting the paper's focus and experimental design.

How much do the results depend on prior natural-language training? Would a model trained from random initialization show the same local/global asymmetry?

Yes: during rebuttal we observe the same local/global asymmetry in pre-training from random initialization (Figures 42–43, Appendix F). The LLM generalizes better to OOD languages that relax global structure and learns local structure better within the language, consistent with fine-tuning. The only difference is the rate of learning: fine-tuning learns faster and overfits sooner, while pre-training is slower under the same hyperparameters. Therefore, local-global learning asymmetry is a general inductive bias of the LLM.

Why use a threshold drawn from the entire training run? What advantage does this provide over comparing correct and incorrect strings at the same checkpoint?

We use the local boundary threshold from the entire training run because it gives a single threshold of learning and forgetting per string, rather than varying the threshold from one epoch to another. For example, in Figure 4(a), the string is learned at epoch 2 and forgotten at epoch 18 under an all-epoch threshold. If we apply a per-epoch (or single checkpoint) threshold, the string is still learned at epoch 2, but never forgotten because its neighbor's loss rises alongside it. As such, after epoch 18, the LLM generates the string worse than its best-case incorrect neighbor, yet the string is still not called forgotten under the per-epoch threshold. Therefore, we apply an all-epoch threshold whenever the entire training trace is available to us (e.g., formal language experiments) and a per-epoch threshold otherwise (e.g., natural language experiments; Figures 5c, 8b).

However, our main conclusion doesn't depend on this choice: the natural-language setup, using a per-epoch threshold, shows the same local/global asymmetry as the formal-language setup with an all-epoch threshold.

Are edited (natural language) examples checked for grammaticality and coherence? How do the authors account for differences in difficulty between word-, sentence-, and paragraph-level edits?

Yes. During rebuttal, we apply an LLM-judge (Claude Sonnet 5) to check the coherence of edited strings, since unlike a formal grammar, natural language has no ground-truth grammar to check against. The coherence of edited strings depends on the edit operations and the levels of edits (word/sentence/paragraph). Specifically, insertions and replacements almost always make edits incoherent regardless of level; deletions and swaps break coherence more at the word level than at the sentence/paragraph level (Appendix F.2). This mirrors the formal-language result: global edits are less likely to break coherence and are thus discriminated less well by the LLM.

Working example and experimental details.

In the updated paper, we expand the existing example (Figure 1, Section 3) with the vocabulary, non-terminal set, and local/global rules and describe how strings are sampled by iteratively expanding non-terminals to terminals.

We summarize the experimental setup in Section 3: we sample disjoint training/test strings from the ground-truth grammar, edit each test string into a grammatically incorrect one, and measure local discrimination between the two to gauge how well the LLM learns the language. To characterize its implicit grammar, we similarly edit grammar rules to build OOD grammars and measure discrimination there; uneven proficiency across OOD languages reveals the LLM's implicit grammar.

The interpretation of OOD generalization needs greater precision.

We are currently experimenting with whether the LLM learns the full structure in the OOD languages as well. Our intuition is that it does not: we expect the OOD grammar will show a similar trend to the given grammar, where local rules are learned better than global ones. We will follow up with direct experimental results.

We have provided a deeper analysis on OOD generalization based on n-gram overlap (see Reviewer qeDy).

Creativity and language learning

The LLM generalizes well to OOD languages that relax global structure, which we anthropomorphize a bit and read as a form of creative learning. Our position is that creativity is a natural inductive bias in how the LLM generalizes, rather than something invoked only at inference time (prompting, constrained decoding). This connection is still exploratory, and we leave it as future work.

The interpretation of OOD generalization needs greater precision

Based on the reviewer comments, we have now directly tested whether the LLM learns the full structure of the modified OOD languages (Appendix I): we measure rule proficiency within the OOD grammar itself, using incorrect strings obtained by editing the OOD grammar's own rules (constraints) at each level. We find the same asymmetry holds: the OOD grammar's local rules are learned better than global rules (Figure 45), mirroring the asymmetry in the original grammar.

Therefore, when an LLM learns a language, it learns its local structure better than its global structure, and generalizes to globally modified OOD languages. In each of these OOD languages, the LLM again learns the local structure better than the global structure, providing an explanation of why they show OOD generalization in the first place. This deeper point characterizes the implicit world model of the LLM.

I thank the authors for their detailed answer. I appreciate the additions and clarifications. I have no further questions. Will maintain my weak-accept rating.

Reviewer_ufBVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes to use a PCFG to evaluate what a pretrained language model has learnt after finetuning on the synthetic data. They fine-tune a model trained on natural language for several epochs (up to 20) on a small set (16-1024) of strings sampled from the grammar. It is then proposed to look at a Levenshtein-ball around some string from the grammar and consider the string learned if its length-normalized loss is lower at a given timestep than its grammatically incorrect neighbours for all training epochs, past and future. A claim is made that this is a somehow a more informative measure than the more commonly used Kullback-Leibler divergence used to compare two distributions. The work also proposes using this setting to study OOD generalisation by sampling strings from grammars that have small edit distance in their rules to the original one. If the loss of the string sampled from the new grammar is better “learned” (lower loss than neighbours), they claim the LLM is “not only learning language L but also learning nearby languages”. This is then speculated to be connected to succinctness and creativity in the conclusion, and the claims that “when an LLM learns a language, it is capable of learning other related languages as well”.

Critical strengths

None

Critical weaknesses

The paper's argument as I understand it is flawed. Seemingly arbitrary grammars are used to fine-tune a pretrained language model to say something about its internal structure, and the start of the paper hints at this being about natural language representations. The experiments say something about how well the model studied can be fine-tuned on a small set of samples from the chosen PCFG. The neighborhood idea is adapted from Ghosh et al (2026) but without sufficient justification for the measure as such (see e.g. Jacobs and Wallach, 2021). The critique on the KL also seems a bit misplaced, saying that the forward KL ignores probability assigned to invalid strings doesn't make sense to me, it just correctly has terms with zero mass on them under the PCFG. But mass outside its language leaves less mass inside its language and so assigning mass outside impacts it. And this doesn’t prevent e.g. a simple Monte Carlo estimate of how much mass the trained model assigns to the PCFG language.

Questions for authors

None

5 comments

We appreciate the reviewer's detailed critique and address the core concerns below.

Arbitrary grammars -- natural language connection -- fine-tune only experiments

We do not pick any arbitrary grammars but hierarchical PCFGs, which have previously been argued to be inspired by the recursive structure of natural language (Allen-Zhu & Li, 2023; Ghosh et al., 2026). We find several observations to independently hold in natural language, e.g., the local/global asymmetry of learning (Figure 8b).

Our goal in the paper is to investigate the implicit language representation of an LLM when it learns a language, since the LLM may learn a different representation than the training language. To this end, we specifically focus on controlled formal language because its underlying representation (i.e., formal grammar) is known to us, unlike natural language.

Separately, during rebuttal we extend our findings to pre-training setup (see Reviewer Kzmh), and Appendix E contains experiments across multiple formal languages and LLMs.

The neighborhood idea is adapted from Ghosh et al (2026) but without sufficient justification for the measure as such (see e.g. Jacobs and Wallach, 2021).

The neighborhood idea from Ghosh et al. (2026) cannot answer whether an individual string in the language is learned or not. This is a critical limitation since not all strings are learned equally during training. We therefore consider a local per-string test for learnability based on a similar neighborhood idea: if an LLM learns a string, then it should fix even small grammatical mistakes in the string within its neighborhood of incorrect strings.

Our understanding is that the reviewer points us to the paper by Jacobs and Wallach, 2021 to point out that, oftentimes, there is a gap between a theoretical construct (of a measure) and its operationalization. While we believe that this is often the case, we also believe that in our case we have both a theoretical understanding of what we want to measure, but also a clear procedure of how to measure it. Specifically, if a string is learned by our theoretical definition, it must have the lowest loss within the neighborhood of incorrect strings, so starting from an incorrect string, one can traverse to the direction that strictly reduces loss, eventually ending at a grammatically correct string (Section 4.1). We have added this discussion and citation to Appendix A.

The critique on the KL also seems a bit misplaced ..

When comparing PCFG vs. LLM distribution, KL divergence is only informative on strings inside the language; all strings outside the language are treated identically in KL divergence. Elaborately, an LLM is designed to generate any string in the universe of all strings with some probability, whereas the PCFG has strict zero probability on strings outside the language. Among strings outside the language, an incorrect string closer in edit distance to a correct one can be generated with higher probability by the LLM than one farther away (Figure 3); however, the PCFG cannot differentiate them, e.g., both have zero probability in the grammar.

We agree that Monte Carlo estimation can compute how much probability mass is inside vs. outside the language, but this is not natively included in KL divergence. Our local discriminative test, on the other hand, is appropriate for how an LLM generates strings, including those outside the language, and defines the learnability at the level of individual strings, unlike an aggregated measure such as KL.

The main message of the paper is that the LLM “learns” out-of-distribution languages with relaxed global structure, and that this reveals its implicit grammar. The experiment cannot prove this point. The OOD languages are generated by modifying a grammar rule, but the model’s proficiency in them is tested with respect to single-token modifications. A string from a language in which a high-level grammar rule is modified is composed of chunks which are all valid in the original language, and a one-token modification which takes the string out of the language destroys such a chunk. Therefore, passing the test needs only knowledge of the original local rules. The edited rule itself is never tested; therefore, it is expected that a model with only local statistics will pass it as well.

This interpretation is corroborated by the rebuttal. Namely, in the new Appendix H it is stated that these languages are more similar to the original language in terms of n-gram overlap and have lower plain generation loss. Hence, the result reflects the local similarity between the two languages and not learning of an out-of-distribution language. The authors themselves state in the rebuttal that their intuition is that the model does not learn the full structure of the OOD languages.

All that remains is that the higher-level rules are learned less well from at most 1,024 strings. As the authors acknowledge, this result agrees with Cagnetta & Wyart (2024). By itself it says nothing about an implicit world model.

I keep my rating.

Thanks for engaging with us.

We agree with the reviewer that being able to locally discriminate individual strings in a language does not entail that the LLM has learned the underlying grammar exactly, which is in fact the key insight of our paper.

We have shown that when learning a language, the LLM additionally generalizes to globally relaxed OOD languages. Whether the LLM learns the full structure of these OOD languages is the same question we addressed in our response to Reviewer Kzmh. Our answer is negative. Specifically, we perform an additional experiment, where we take a grammar $ G_{\text{OOD}} $, edit its rules at both local and global levels, and find that local rules are learned better than the global ones (Appendix I). This result is consistent with the same asymmetry observed in the original grammar $ G $ of a language $ L $.

Therefore, when an LLM learns a language, it learns its local structure better than its global structure and generalizes to globally relaxed OOD languages. In each of these OOD languages, the LLM again learns the local structure better than the global structure, providing an explanation of why they show OOD generalization in the first place. This deeper point characterizes the implicit world model of the LLM and, to our best knowledge, offers a new perspective on this question.

Finally, we believe our original rebuttal already addressed the reviewer's mentioned weakness of our paper; we are happy to clarify further if anything remains unresolved.

This interpretation is corroborated by the rebuttal. Namely, in the new Appendix H it is stated that these languages are more similar to the original language in terms of n-gram overlap and have lower plain generation loss. Hence, the result reflects the local similarity between the two languages and not learning of an out-of-distribution language. The authors themselves state in the rebuttal that their intuition is that the model does not learn the full structure of the OOD languages. All that remains is that the higher-level rules are learned less well from at most 1,024 strings. As the authors acknowledge, this result agrees with Cagnetta & Wyart (2024). By itself it says nothing about an implicit world model.

We are a bit confused about the reviewer's argument above, which relies on the same evidence we provided in our response to Reviewer qeDy, interpreted differently. Given this new result (Appendix I), we would like to ask the reviewer if there is any remaining concern.

Reviewer_qeDyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how small transformer-based LLMs learn simple probabilistic context-free grammars (PCFGs) (simple here meaning finite / non-recursive), introducing a new point of view of “correctness” as “the language modeling having a gap in the probability assigned to sentences generable from the grammar vs out-of-language sentences obtained by small edits to correct sentences). Practically, incorrect strings are generated in two ways — by string edits (Hamming distance like) and rule edits (randomly edit a rule somewhere in the grammar). Then each string has an associated “string proficiency” — the gap between probability assigned that grammatical string and edits of it (ie loss of grammatical string being less than “boundary-loss”, which is a min loss over the ungrammatical small edits). Overall, language proficiency is defined as average prob that a string is “learned” in this sense. In experiments on the small LLM trained on synthetic simple PCFGs, they found that during training, learns indeed occurs in their sense (of increases gap between training strings and edits of training strings) up until overfitting. However the most surprising / interesting finding is related to rule-based edits. They make “OOD languages” by editing a rule at a specific level of the grammar, and find that when this rule is ‘higher up’ in the PCFG (i.e. more global), that the OOD language is better modeled (in their sense of correctness) than an OOD language obtained by modifying rules closer to the bottom / leaves.

Critical strengths
  • I am personally very interested in studying LLMs via how they learn/represent formal languages and CFGs in particular. In other words I agree with their justifications in 3.2 (though it's a bit long / is it fully necessary? see nits in Questions for Authors). It gives a principles and formal approach that can tell us about how LLMs work. Their idea of defining correctness via "gap between prob assigned to grammatical string and non-grammatical edited strings" is an interesting / deep one -- I agree with them it's surprising it hasn't been asked / studied as much! This is the key conceptual contribution
  • the main key finding that the for the LLMs / grammars they tested, their notion of correctness holds more on modified versions of the language that deeply depart from the training set / intended language (by changing a deep rule) vs a more surface-level rule is very interesting / surprising, it's the key empirical finding of the paper
Critical weaknesses
  1. the key empirical finding is interesting (as said above) but I would like to see a bit more investigation of this finding to make the paper's contribution more substantial (overall it feels just a little bit weak as of now). The comparison to pure edit-based non-grammatical sentences with the same number of edits as you get from changing the rule (Figure 9b) is an important example of this. But I feel like with relatively little work more can be said. One issue with the their new notion of correctness (which I do like) when applied to "out of distribution strings" is that if the out of distribution strings are very unlikely, the neg log loss can be huge, so then maybe differences between additional edits to those strings can be very big. Is this what accounts for this? In other words, it is unclear what is the interpretation of their finding that changing earlier/deeper rules results in a language more accurately modeled by the LM. I'd like to see the LM's original probabilities of those modified languages, vs. the edited versions of those modified languages, etc. Another thing -- LLMs generate their outputs autoregressively, so a very local violation (token X coming after a token Y which never happens in training) can lead to a huge drop in log prob. This should be explored more (they discuss about the possibility that their global vs. local finding has to do with this-- that changing global rules preserves local structure) -- can we somehow measure how "correct" a globally-modified OOD language looks locally (maybe just counts of valid 2 or 3-token continuations)? This would give more mechanistic / explanatory "meat" to their empirical finding.

Also more generally -- how do your findings compare to the setting where correctness is just measured as log prob? does that give you the opposite of your findings -- that changing the earlier/deeper rules results in language more poorly modeled by the model, rather than if you change shallow/leaf rules? (That's what I expect). That would be an interesting finding with respect to your new notion of correctness, that it captures something different. (Maybe this is reported / discussed somewhere but in my reading I did not see it!)

  1. the main text makes it sound like the main experiments are done with just 1 model and grammar. I didn't read the appendix in full but there it seems like more models were used and more grammars were used. can this be clarified, in the main text? If the finding was found with 5 models x 2 grammars, the main text should clearly state: the exact same phenomenon is observed across the 10 combinations (i.e., that changing a global rule resulted in a "more correctly" modeled language than changing a local rule) -- knowing this occurs over 10 combos is more convincing / important than just for 1 model x language.
Questions for authors

The main weaknesses contain respective main questions, see above. Some additional questions:

  • One issue is that "levels" in (P)CFGs are not necessarily unique -- this is actually quite a deep topic. There are usually (infinitely) many CFGs representation the same grammar. In particular, as far as I understand your PCFGs are all finite ones / they have no recursion -- as such they are all actually just finite languages (very simple regular languages). This makes the interpretations about deep/shallow rules a bit tenuous (I do see it / agree with it overall -- but basically it means that rather than being about PCFG rules, it's really about the total edit distance you get in the modified language and/or about comptactness of representation) -- anyways there is something unclear about all of this (even though I don't think it majorly affects the point of the paper)
  • How is experiment of 8b done?? This is the first "natural language" thing in the paper -- some detail should definitely belong in the main text about this! Also, for natural language, maybe the farther away / global edits are not ungrammatical (whereas for small PCFGs they can be made to be so).
  • there are several very recent ICML papers on LMs and PCFGs that could be relevant / added to related work / compared with. One is Schulz et al Unraveling Syntax on the Substructure of Grammar -- they formalize the idea of shallower and deeper sub-grammars / rules (in their setting the CFGs are infinite / not reducible to weaker classes, but they only study the in-distribution notion of correctness) -- how this connects to this work is interesting / important. there is also "Towards a theory of how the structure of language is acquired" which studies similar finite PCFGs and shows how actually the global / deeper rules are acquired later / need the most data to be generalized to, seems directly related to findings of this work.

Finally since there is no other place for it I will include the minor nits / questions here too:

  • 41 “which lacks a well-defined structure” — controversial, I would remove/rephrase. It has a well-defined structure, but it is complicated/big/not fully described (ie the CFG for English is enormous and not written out anywhere)
  • 50 “what other related languages it also learns” -> formally any language model represents exactly one language.. rephrase. eg how well it can model / how close its output distribution looks to other languages close to the target (input data) language
  • Related work cite Schulz et al 2026
  • 226 strictly speaking this is incorrect, the language proficiency of an LLM can be defined just by generation loss of strings inside the language; as the total probability of all such strings -> 1, the probability of the remainder -> 0… maybe rephrase a bit
  • The discussion of 3.2 is fine but a bit superfluous — if needed, this could be be shortened to give extra space (because someone reading this, myself included, is likely already interested in formal languages). Also a whole page for discussion + conclusion -- could be shortened if space is needed!
  • 276/277 is confusingly phrased “learn to generate .. nearby grammatically incorrect strings”, I think you meant “but also distinguish / correct nearby grammatically incorrect strings”
  • 321 “both formal language and natural language support this claim” - ambiguous but interesting / expand what you mean?
  • 355 weird to use “learnable” here (“any OOD grammars that are equally different from G should be equally learnable”… similar to previous comments, should be “equally well modeled / approximated” or something like that
  • 358 relax global structure … it’s really break global structure
1 comment

We thank the reviewer for the thoughtful review and for recognizing the paper's key conceptual and empirical contributions.

An interpretation on OOD generalization

Based on the reviewer comments, we expand our experimental findings in Appendix H: we compute n-gram overlap between strings in the OOD language and the original language (Figure 44), where the overlap is the highest for a string in the globally modified OOD language, i.e., it is more similar locally. As a result, when we only look at their generation loss (Figure 45a), strings in the globally modified OOD language have the lowest loss. Moreover, the gap between loss of grammatically correct and incorrect strings in the globally modified OOD language is also the maximum (Figure 45d). All these results provide a (mechanistic) interpretation of why globally modified OOD languages generalize best. Furthermore, we do not see any conflict between our local discriminative test and the traditional generative loss-based test, contrary to what the reviewer expected.

Experimental summary

We've added a brief description of our experiments in Section 3. To clarify the scope precisely: language proficiency (RQ1, discriminative vs. generative) is measured across 6 LLMs and 12 languages. The local-global asymmetry finding itself (RQ2, editing a global vs. local rule, evaluating a family of OOD languages) is currently shown for one model (Qwen2.5-1.5B) across 8 grammar variants.

Equivalent grammars & finite PCFG

The reviewer is raising interesting points. We have thought about the fact that the same set of strings can be generated by different grammars. However, such equivalent grammars generate the same set of strings despite having very different structures. Our approach has therefore been to look beyond the language, i.e., what OOD languages the LLM also learns, giving a distinct approach to ask the implicit language representation question.

Our PCFGs are indeed finite, but while editing a grammar to construct OOD languages, we can introduce recursion by repeating a rule's left-hand-side non-terminal in its own right-hand side, turning $ A \to \beta $ into $ A \to \beta A $. In our perspective, such recursion leads more to a global change than a local one. We'd guess the LLM would generalize well to such a recursive grammar too, consistent with our local-global asymmetry, though we have not tested this directly.

Natural language experiment: Figure 8(b)

In the natural language experiment (Figure 8b), we test whether the same local-global asymmetry holds: we edit text at three levels—word, sentence, and paragraph—using one of four operations (replace, insert, delete, swap). The LLM discriminates local word-level edits better than global sentence/paragraph-level edits, mirroring how it learns local rules better than global rules within the language.

Agreeing with the reviewer’s point, we find global edits in natural language to be more grammatical depending on the edit operation: when we delete or swap a sentence or paragraph, an LLM-judge finds the edits more coherent than the same edits on words. Insertions or replacements, however, almost always make an edit ungrammatical. See Appendix F.2 for the full results (also discussed under Reviewer Kzmh).

Related work

Schulz et al. (2026) indeed consider recursive PCFGs and theoretically establish that the language modeling (training) loss recurses linearly over irreducible subgrammars. If we think from an empirical perspective, we only provide finite strings to an LLM, and any set of finite strings can be modelled as a regular grammar. Therefore, learning the original recursive representation from finite strings can be a challenge in practice. Separately, Cagnetta & Wyart (2024) find that deeper (non-recursive) grammar rules require more training data to generalize, which corroborates our local/global result. Our paper complements this growing field by showing that the internal representation of the LLM may not exactly match the given language’s structure. Therefore, deeper structures (in our case global) are harder to learn exactly, resulting in a generalization of the LLM on out-of-distribution languages where the global structure is allowed to be relaxed. We've added both citations and this discussion to Appendix A.2.

Minor comments.

We have addressed minor comments in the updated pdf. Notably, we have moved Section 3.2 to Appendix D. In Section 4.2 (Figure 5), we show that strings that are easier to generate (having lower loss) are not necessarily discriminated well (having lower loss than local boundary). In the context of natural language, we ask the same LLM to generate two minimally different strings from the BLiMP dataset (Warstadt et al., 2020) and find that a lower generation loss for the grammatically correct string does not guarantee it is assigned lower loss than its ungrammatical minimal-pair counterpart (Figure 5c).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We have updated our paper based on the constructive suggestions of the reviewers. All changes to the PDF are highlighted in blue.

  • Natural-language edit coherence/grammaticality check (Appendix F.2): We apply an LLM-judge to check the coherence and grammaticality of natural-language edits (reviewers Kzmh's and qeDy's question of whether edits are checked for grammaticality across levels).

  • Local-global asymmetry at random initialization (Appendix G): Local-global learning asymmetry applies to randomly initialized models, demonstrating it as a general inductive bias of language learning in LLMs (reviewer Kzmh's question of whether the asymmetry depends on natural-language pretraining).

  • Out-of-distribution (OOD) generalization mechanism via n-gram overlap (Appendix H): shows globally relaxed OOD strings are locally more similar to the original grammar (addresses qeDy's request for a mechanistic explanation of why global edits generalize better).

  • Rule proficiency on the OOD grammar's own rules (Appendix I): tests whether the full structure of the OOD grammar is learned by the LLM or not, showing a consistent observation where local structure is learned better than global structure (reviewers Kzmh's W3 and ufBV's point that the edited rule itself is never tested).

We have also revised the writing in several places (e.g., motivation, related work, and definitions) for clarity, in response to reviewer comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.