acceptodds
Under review as a conference paper at ICLR 2027

Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorization

Abstract

In this study, we demonstrate that backpropagation optimization updates the whole continuous landscape of the representation space from the beginning of learning. We train a series of decoder-only Transformers on controlled synthetic grammars and analyze changes in the representation geometry and models’ behaviour over the learning trajectory. We find that empty regions of the representation space are also systematically updated during early-stage optimization, even though these regions are not occupied by observed data points. This process progressively forms global geometrical structures in the whole continuous representation space. As a consequence of this optimization property, we find that early-stage learning is dominated by generalization, while memorization becomes increasingly dominant only at later stages of learning. From the earliest stage of learning, models progressively form categories that capture the true underlying data-generating process, rather than only memorizing specific input sequences. These categories are not simply sets of tokens, but continuous regions in the representation space. Model behaviour is driven by such categorization knowledge until the transition toward a more memorization-based representation. When memorization begins to dominate learning and representation, optimization in the representation space becomes increasingly localized around discrete observed data points. At the same time, the global continuous geometrical structures in the representation space are progressively destroyed. Consequently, model behaviour becomes less driven by category-based generalization and increasingly driven by the memorization of specific input data points. This learning trajectory contrasts with memorization-based theories of neural network cognitive representations. Our findings suggest that NLMs are inherently categorization-driven models: they achieve generalization by forming categories from distributional statistics, and this categorization behaviour begins from the earliest stage of optimization.

Primary area:
interpretability and explainable AI
Keywords:
Neural language models, generalization, memorization, categorization, learning dynamics, statistical learning, synthetic grammar

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vETsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This submission seeks to investigate the question of how generalization and memorization evolve as learning unfolds in neural language models. The submission contains two studies based on training networks to learn synthetic language datasets.

The first set of experiments seeks to address the question of whether learning affects the continua global structure formed by the model. They assess the performance of the models in predicting the category of the next token, when provided with tokens along the path defined by the centroids of the representations of the two categories. They compared the results from models of different sizes and lengths of training, and found that there were some differences.

In the second set of experiments, they seek to investigate “the temporal relationship between generalization and memorization along the learning trajectory” and whether observed sequences leads to stronger compression. The compression was evaluated using a quantify from information theory.

Overall, while the paper conducts several numerical experiments on synthetic datasets and reported some notable observations, the results are too preliminary. It wasn’t clear how these experimental results speak to or against the bigger questions they claim to study. The quality of the writing would need substantial improvement to meet the bar for ICLR.

Critical strengths
  • The question of generalization vs. memorization is an important problem in language modeling, particularly in the context of large language models (LLMs). I thought the creation of the two synthetic datasets and some of the task designs are thoughtful.
Critical weaknesses
  1. The results are preliminary. The experimental results were not carefully interpreted. It was not clear how they indeed support the main claims on generalization and memorization. It is uncertain to me if the observations reported could indeed speak in favor (or against) the issues of memorization and generalizations in language modeling in general.

  2. It is unclear how these results would generalize to other regimes, in particular for large language models trained on a large amounts of text. Even if the authors’ interpretations on the generalization and memorization on these synthetic datasets are correct, there seem still a large gap for understanding generalization and memorization in LLMs.

Questions for authors

The paper claims that, early in training, there is evidence for the emergence of geometric structure that would support generalization. As training progresses for much longer, memorization start to dominate. Does this mean that models trained for longer would not generalize or generalize worse?

Line 52-32, “early stage optimization constitutes a comprehensive update on the whole continuous representation space.” The paper seems to imply that this means generalization. Can the authors unpack the ideas? What does variable c represent in Section 2?

Would the results generalize to large-scale datasets with larger models?

Can the authors better explain what “pure generalization” and “pure memorization” would predict in the context of their numerical experiments? I think explaining these early on before showing the results would help the readers.

Reviewer_i7zcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how generalization and memorization evolve during training in small decoder-only Transformers trained on synthetic grammars. Across two studies, it argues that early training organizes broad, continuous regions of embedding space that support category-level generalization, while prolonged training increasingly localizes representations around observed items, disrupting this global structure. In Study 1, models trained on M --> N and P --> Q dependencies develop an early smooth category boundary between the two embedding regions, which weakens or disappears with continued training in larger models. In Study 2, models trained on a six-token grammar initially compress both familiar and held-out grammatical strings toward a category-level baseline, but later increasingly favor familiar strings, with larger models compressing observed strings more and assigning more information to held-out ones. Together, the results are interpreted as a shift from categorization-based generalization toward exemplar-specific memorization.

Critical strengths
  1. The paper directly tracks the emergence and degradation of category-level generalization. Combining interpolation-based probes, held-out grammatical sequences, and a compression baseline provides a developmental picture beyond static checkpoint evaluation. The finding that familiar and held-out strings initially improve together before diverging in larger models is clear and potentially useful.

  2. The experimental controls appropriately isolate compositional generalization. In both studies, held-out items combine familiar tokens in unattested sequences, distinguishing lexical familiarity from category-level generalization. The multi-seed appendix further supports the robustness of the qualitative patterns.

Critical weaknesses
  1. The interpolation analysis does not establish the claimed geometry of "empty regions". Measuring next-token behavior along a line between input-embedding centroids is a useful probe, but it does not establish a continuous category manifold or show that late training destroys global geometry. The results could reflect the specific interpolation path, shifting centroids, or the output layer's response to inputs far from training tokens. The authors need a more principled geometric analysis or substantially more cautious claims.

  2. The transition from generalization to memorization is not cleanly demonstrated. Study 2 interprets familiar-string compression below the perfect-grammar baseline as memorization and increased held-out information as degraded generalization. However, these are relative likelihood changes, not direct measures of memorization or lost grammatical knowledge. Controls involving non-grammatical and shuffled-category sequences, repeated presentations of individual training strings, and analyses separating token-, position-, and bigram-level probability changes would help distinguish these explanations. Without such evidence, the central developmental claim remains plausible but underdetermined.

  3. Figures are impossible to read. While the authors did link to a demo, all figures in the paper are impossible to read.

Questions for authors
  1. Why is interpolation between the static input-embedding centroids of M and P an appropriate probe of the "global geometric structure" supporting generalization? Do contextual hidden states, rather than input embeddings, show the same trajectory?

  2. How sensitive are the Study 1 results to the choice of interpolation path?

  3. In Study 2, can you rule out that the later familiar-string advantage reflects optimization-related overconfidence or token/position-level regularities rather than exemplar-specific memorization?

  4. The main text says the larger model's global structure is "totally destructed", while the appendix says the response becomes "weaker and more fragmented". Could you provide a quantitative measure of structure degradation, with seed-level uncertainty, rather than relying on visual heatmaps?

5/ It would strengthen the paper to test whether interventions on embedding geometry causally affect the familiar/held-out divergence.

1 comment

The authors did not engage in a discussion, so my concerns were not addressed and thus I am keeping my score.

Reviewer_NV7YOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate generalization and memorization in Transformers by constructing synthetic grammars (either categorical pairs that consistently follow each other or longer categorical sequences). They find that the hidden space in between each category's centroids meaningfully relates to model performance, starting early during training, demonstrating that representations are well-structured and suited for generalization early on. This can deteriorate later on, especially for larger models. They similarly find that larger models increasingly memorize the specific seen sequences over the general sequence structure.

Critical strengths
  • The paper is well written overall: the experiments are clearly designed and the writing is easy to follow.
  • I liked the idea of probing the transition between the different category centroids as a way to understand how well-structured the representational space is. This provides a useful way to measure the generalization the authors are interested in.
Critical weaknesses

I think this paper is pursuing a thought-provoking direction and has a number of interesting ideas. However, at present, I do not think the contributions merit acceptance. My concerns are both with respect to the overall scope and support for the current insights:

  • The authors present two small synthetic studies across three random seeds and three models. The presented synthetic grammars are extremely simple and as a result, the findings here do not merit the kinds of sweeping conclusions made in the conclusion (e.g., "NLM learning is fundamentally category-driven", l. 449). Overall, this limits the scope of the conclusions that can be made from this substantially. Similarly, the authors do not demonstrate that this perspective is applicable to larger models (ideally trained on real data) as well.
  • The analysis of the synthetic studies lacks a more detailed analysis: The transition from lack of structure to smooth transition structure to irregular transition structure is visually clear, but how exactly are the authors proposing to quantify this? What do the authors propose a good versus bad transition structure looks like (e.g., monotonicity, symmetry, ...)? Such quantitative measures would also allow the authors to more systematically compare different seeds, whereas currently the evaluation is limited in many places to a broader qualitative insight that the reader then needs to validate for the other seeds in the appendix. This would also let the authors more concretely test whether the representational structure is concretely associated with the generalization performance: does one deteriorate at the same time as the other drops? Notably, on study 2, the authors do not seem to analyze the representational structure at all.

Taken together, though it contains some promising ideas, I think this paper lacks validation in more complex tasks or in more realistic models and a more detailed analysis of the concrete setups considered here.

Questions for authors
  • How do you think your chosen synthetic setups may interact with generalization arising before memorization and do you expect that other task setups might result in different findings?
  • What would a proposed quantitative measure for the transition between the two categorical regions look like. Would you expect that its evolution correlates with the generalization performance?
  • Can you please make the figure text bigger? It is currently really difficult to understand these figures due to the small text.

Minor comments:

  • l. 52: “backpropagation”
  • l. 50: “account of”?
  • l. 149: can you clarify how much these empty regions are actually not filled with data points?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.